Evaluating the Prediction Accuracy of Generated Performance Models - - PowerPoint PPT Presentation

▶
evaluating the prediction accuracy of generated
SMART_READER_LITE
LIVE PREVIEW

Evaluating the Prediction Accuracy of Generated Performance Models - - PowerPoint PPT Presentation

Aug 19, 2023 372 likes •682 views

Stuttgart, Germany, 2014-11-27 Evaluating the Prediction Accuracy of Generated Performance Models in Up- and Downscaling Scenarios Symposium on Software Performance (SOSP) 2014 Andreas Brunnert 1 , Stefan Neubig 1 , Helmut Krcmar 2 1 fortiss

slide-1
SLIDE 1

fortiss GmbH An-Institut Technische Universität München Stuttgart, Germany, 2014-11-27

Evaluating the Prediction Accuracy of Generated Performance Models in Up- and Downscaling Scenarios

Symposium on Software Performance (SOSP) 2014 Andreas Brunnert1, Stefan Neubig1, Helmut Krcmar2

1fortiss GmbH, 2Technische Universität München

slide-2
SLIDE 2

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 2

  • Motivation and Vision
  • Performance Model Generation

– Data Collection – Data Aggregation – Model Generation

  • Evaluation

– SPECjEnterprise2010 – Overhead Evaluation – Experiment Setup – Scenario Description – Scenario Results

  • Upscaling
  • Downscaling
  • Future Work

Agenda

slide-3
SLIDE 3

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 3

  • Motivation and Vision
  • Performance Model Generation

– Data Collection – Data Aggregation – Model Generation

  • Evaluation

– SPECjEnterprise2010 – Overhead Evaluation – Experiment Setup – Scenario Description – Scenario Results

  • Upscaling
  • Downscaling
  • Future Work

Agenda

slide-4
SLIDE 4

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 4

  • Numerous performance modeling approaches are available to evaluate the

performance (i.e., response time, throughput, resource utilization) of enterprise applications

  • Performance models are especially useful for scenarios that cannot be

tested on real systems, e.g.:

– Scaling a system up or down in terms of the available hardware resources (e.g., number of CPU cores) during the capacity planning process.

  • Creating a performance model requires considerable manual effort

–  low adoption rates of performance models in practice

Motivation & Vision

slide-5
SLIDE 5

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 5

  • To increase the adoption rates of performance models in practice!

– …For that purpose, we have proposed an automatic performance model generation approach for Java Enterprise Edition (EE) applications. – …This work improves the existing approach by further reducing the effort and time for the model generation. – ... This work evaluates the prediction accuracy of these generated performance models in up- and downscaling scenarios, i.e.:

  • Increased and reduced number of CPU cores
  • Increased and reduced number of concurrent users

Motivation & Vision

slide-6
SLIDE 6

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 6

  • Motivation and Vision
  • Performance Model Generation

– Data Collection – Data Aggregation – Model Generation

  • Evaluation

– SPECjEnterprise2010 – Overhead Evaluation – Experiment Setup – Scenario Description – Scenario Results

  • Upscaling
  • Downscaling
  • Future Work

Agenda

slide-7
SLIDE 7

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 7

Overview

Performance Model Generation

CSV MBeans PMWT Agent CSV CSV Performance Model PMWT Connector Performance Model Generator Java EE Application Monitoring Database Monitoring Data Persistence Service

  • 1. Data Collection
  • 2. Data Aggregation
  • 3. Model Generation

(adapted from Willnecker et al. (2014))

slide-8
SLIDE 8

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 8

Data Collection

Performance Model Generation

Java EE Application Web Tier Business Tier Enterprise Information Systems Tier

Servlet Filters EJB Interceptors JDBC Wrappers Web Components (Servlets/JSPs) Enterprise JavaBeans (EJBs) Data collected:

  • EJB and Web components
  • EJB and Web component
  • perations
  • EJB and Web component

relationships on the level

  • f single component
  • perations
  • Resource demands for

single component operations (CPU, Memory)

CSV MBeans PMWT Agent CSV CSV Performance Model PMWT Connector Performance Model Generator Java EE Application Monitoring Database Monitoring Data Persistence Service

slide-9
SLIDE 9

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 9

Data Aggregation

Performance Model Generation

  • type : string(idl)
  • componentName : string(idl)
  • operationName : string(idl)

OperationIdentifier JavaEEComponentOperationMBean BranchDescriptor

  • invocationCount : long(idl)
  • totalCPUDemand : long(idl)
  • totalResponseTime : long(idl)
  • totalAllocatedHeapBytes : long(idl)

BranchMetrics ExternalOperationCall ParentOperationBranch

  • loopCount : long(idl)
  • loopCountOccurrences : long(idl)

OperationCallLoopCount 1 * 1 * 1 1 1 * 1 * 1 1 1 * 1 * {ordered}

CSV MBeans PMWT Agent CSV CSV Performance Model PMWT Connector Performance Model Generator Java EE Application Monitoring Database Monitoring Data Persistence Service

(Brunnert et al. 2014_1)

slide-10
SLIDE 10

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 11

Model Generation

  • Except for the usage model, default models for

all model layers of the Palladio Component Model (PCM) are generated automatically:

– Repository model containing the components of an EA, their relationships and resource demands – System model containing the deployment units detected during the data collection (no single components) – A simple resource environment with one server and an allocation model that maps all deployment units to this server

Performance Model Generation

Repository Model Resource Environment System Model Allocation Model Usage Model

CSV MBeans PMWT Agent CSV CSV Performance Model PMWT Connector Performance Model Generator Java EE Application Monitoring Database Monitoring Data Persistence Service

(adapted from a presentation for Brunnert et al. (2014_2)

slide-11
SLIDE 11

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 12

  • Motivation and Vision
  • Performance Model Generation

– Data Collection – Data Aggregation – Model Generation

  • Evaluation

– SPECjEnterprise2010 – Overhead Evaluation – Experiment Setup – Scenario Description – Scenario Results

  • Upscaling
  • Downscaling
  • Future Work

Agenda

slide-12
SLIDE 12

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 13

SPECjEnterprise2010

Evaluation

  • Benchmark Driver / Emulator

– Simulates interactions with the SUT – Defines workload

  • Automobile Manufacturer

– Orders Domain (CRM) – Manufacturing Domain – Supplier Domain (SCM)

  • Database Server

SPECjEnterprise 2010 Architecture (SPEC, 2009)

slide-13
SLIDE 13

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 14

SPECjEnterprise2010

Evaluation

(Standard Performance Evaluation Corporation 2009)

  • Workload defined by driver
  • Business Transactions

– Browse, Manage, Purchase – Predefined sequence of HTTP requests including probabilities

SPECjEnterprise 2010 Architecture (SPEC, 2009) Orders Domain Architecture (Brunnert et al. 2013)

slide-14
SLIDE 14

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 15

Overhead Evaluation 1/2

  • Monitoring code is called before and after each monitored invocation

 Considerable instrumentation overhead!

  • Overhead Evaluation: CPU & Heap vs. CPU only

– 4 CPU cores – 20 GB RAM – 600 Users – Only steady state data is collected

Evaluation

slide-15
SLIDE 15

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 16

Overhead Evaluation 2/2

Evaluation

CPU & Heap CPU only Component Operation All levels Top level All levels Top level CPU Heap CPU Heap CPU CPU 1 app.sellinventory 1.023 ms 33,650 B 3.001 ms 225,390 B 0.756 ms 3.003 ms 2 CustomerSession.sellInventory 0.785 ms 60,450 B

  • 0.731 ms
  • 3

CustomerSession.getInventories 0.594 ms 49,540 B

  • 0.548 ms
  • 4

OrderSession.getOpenOrders 0.954 ms 70,600 B

  • 0.878 ms
  • 5

dealerinventory.jsp.sellinventory 0.108 ms 16,660 B

  • 0.103 ms
  • Total Resource Demand

3.464 ms 230,900 B 3.001 ms 225,390 B 3.015 ms 3.003 ms Mean Data Collection Overhead

0.116 ms 1378 B

0.003 ms

(Brunnert et al. 2014_1)

slide-16
SLIDE 16

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 17

Overhead Evaluation 2/2

Evaluation

CPU & Heap CPU only Component Operation All levels Top level All levels Top level CPU Heap CPU Heap CPU CPU 1 app.sellinventory 1.023 ms 33,650 B 3.001 ms 225,390 B 0.756 ms 3.003 ms 2 CustomerSession.sellInventory 0.785 ms 60,450 B

  • 0.731 ms
  • 3

CustomerSession.getInventories 0.594 ms 49,540 B

  • 0.548 ms
  • 4

OrderSession.getOpenOrders 0.954 ms 70,600 B

  • 0.878 ms
  • 5

dealerinventory.jsp.sellinventory 0.108 ms 16,660 B

  • 0.103 ms
  • Total Resource Demand

3.464 ms 230,900 B 3.001 ms 225,390 B 3.015 ms 3.003 ms Mean Data Collection Overhead

0.116 ms 1378 B

0.003 ms

(Brunnert et al. 2014_1)

slide-17
SLIDE 17

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 18

Overhead Evaluation 2/2

Evaluation

CPU & Heap CPU only Component Operation All levels Top level All levels Top level CPU Heap CPU Heap CPU CPU 1 app.sellinventory 1.023 ms 33,650 B 3.001 ms 225,390 B 0.756 ms 3.003 ms 2 CustomerSession.sellInventory 0.785 ms 60,450 B

  • 0.731 ms
  • 3

CustomerSession.getInventories 0.594 ms 49,540 B

  • 0.548 ms
  • 4

OrderSession.getOpenOrders 0.954 ms 70,600 B

  • 0.878 ms
  • 5

dealerinventory.jsp.sellinventory 0.108 ms 16,660 B

  • 0.103 ms
  • Total Resource Demand

3.464 ms 230,900 B 3.001 ms 225,390 B 3.015 ms 3.003 ms Mean Data Collection Overhead 0.116 ms 1378 B 0.003 ms

Without heap monitoring, the Mean Data Collection Overhead drops dramatically!  We focus on collecting CPU demand only!

(Brunnert et al. 2014_1)

slide-18
SLIDE 18

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 19

Experiment Setup

  • Model generation

– Collect only CPU demands using Servlet Filters and EJB Interceptors – Only steady state data is collected – SUT under moderate load (~ 40-60 %)

  • Model simulation

– Generated PCM models are taken as input for SimuCom – Results compared with benchmark runs

  • Benchmark runs

– Servlet Filter at front controller to measure response times and throughput – JMX connection to measure CPU utilization of JVM – Only steady state data is collected – Performed three times (same weight)

Evaluation

slide-19
SLIDE 19

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 20

Scenario Description

  • Upscaling: How many resources do I need for an increasing workload?

– Workload: 600 Users  900 Users  1200 Users – CPU: 4 Cores  6 Cores  8 Cores – Model generation: 4 cores (~52 % load)

  • Downscaling: How many resources do I need for the given workload?

– Workload: 800 Users – CPU: 8 Cores  6 Cores  4 Cores – Model generation: 8 cores (~39 % load)

Evaluation

slide-20
SLIDE 20

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 21

Scenario Results: Upscaling 1/2

Evaluation

CPU Cores Number

  • f Users

Business Transaction Measured RT Simulated RT Prediction Error Measured Throughput Simulated Throughput Prediction Error Measured CPU Simulated CPU Prediction Error

4 600

Browse 63.23 ms 65.06 ms 2.91 % 1820.6 1813.1 0.41 % 48.76 % 46.87 % 3.88 % Manage 11.58 ms 13.28 ms 14.71 % 906.8 917.3 1.16 % Purchase 8.27 ms 9.73 ms 17.67 % 904.9 900.3 0.50 %

6 900

Browse 69.25 ms 57.56 ms 16.89 % 2708.3 2721.5 0.49 % 51.72 % 46.85 % 9.42 % Manage 12.54 ms 11.95 ms 4.69 % 1354.3 1354.4 0.01 % Purchase 8.95 ms 8.72 ms 2.60 % 1352.4 1368.1 1.16 %

8 1200

Browse 88.82 ms 56.25 ms 36.66 % 3617.8 3641.9 0.67 % 57.34 % 46.97 % 18.09 % Manage 14.13 ms 11.64 ms 17.67 % 1806.4 1795.0 0.63 % Purchase 9.31 ms 8.46 ms 9.15 % 1811.6 1819.2 0.42 %

(Brunnert et al. 2014_1)

slide-21
SLIDE 21

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 22

Scenario Results: Upscaling 1/2

Evaluation

CPU Cores Number

  • f Users

Business Transaction Measured RT Simulated RT Prediction Error Measured Throughput Simulated Throughput Prediction Error Measured CPU Simulated CPU Prediction Error

4 600

Browse 63.23 ms 65.06 ms 2.91 % 1820.6 1813.1 0.41 % 48.76 % 46.87 % 3.88 % Manage 11.58 ms 13.28 ms 14.71 % 906.8 917.3 1.16 % Purchase 8.27 ms 9.73 ms 17.67 % 904.9 900.3 0.50 %

6 900

Browse 69.25 ms 57.56 ms 16.89 % 2708.3 2721.5 0.49 % 51.72 % 46.85 % 9.42 % Manage 12.54 ms 11.95 ms 4.69 % 1354.3 1354.4 0.01 % Purchase 8.95 ms 8.72 ms 2.60 % 1352.4 1368.1 1.16 %

8 1200

Browse 88.82 ms 56.25 ms 36.66 % 3617.8 3641.9 0.67 % 57.34 % 46.97 % 18.09 % Manage 14.13 ms 11.64 ms 17.67 % 1806.4 1795.0 0.63 % Purchase 9.31 ms 8.46 ms 9.15 % 1811.6 1819.2 0.42 %

(Brunnert et al. 2014_1)

slide-22
SLIDE 22

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 23

Scenario Results: Upscaling 2/2

Evaluation

4 CPU cores, 600 users 6 CPU cores, 900 users 8 CPU cores, 1200 users

30 60 90 120 150 MRT Brow se SRT Brow se response time [ms] 5 10 15 20 25 30 35 MRT Manage SRT Manage response time [ms] 5 10 15 20 25 30 35 MRT Purchase SRT Purchase response time [ms] 30 60 90 120 150 MRT Brow se SRT Brow se response time [ms] 5 10 15 20 25 30 35 MRT Manage SRT Manage response time [ms] 5 10 15 20 25 30 35 MRT Purchase SRT Purchase response time [ms] 30 60 90 120 150 MRT Brow se SRT Brow se response time [ms] 5 10 15 20 25 30 35 MRT Manage SRT Manage response time [ms] 5 10 15 20 25 30 35 MRT Purchase SRT Purchase response time [ms]

(Brunnert et al. 2014_1)

slide-23
SLIDE 23

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 24

Scenario Results: Downscaling 1/2

Evaluation

CPU Cores Number

  • f Users

Business Transaction Measured RT Simulated RT Prediction Error Measured Throughput Simulated Throughput Prediction Error Measured CPU Simulated CPU Prediction Error

8 800

Browse 71.54 ms 64.03 ms 10.50 % 2413.9 2415.8 0.08 % 37.41 % 35.17 % 5.99 % Manage 12.96 ms 12.64 ms 2.49 % 1203.5 1209.2 0.48 % Purchase 9.36 ms 9.33 ms 0.25 % 1215.9 1228.7 1.05 %

6 800

Browse 67.62 ms 66.03 ms 2.35 % 2413.9 2425.4 0.48 % 46.38 % 46.94 % 1.21 % Manage 12.52 ms 13.08 ms 4.45 % 1202.0 1196.6 0.45 % Purchase 9.05 ms 9.64 ms 6.57 % 1208.2 1215.0 0.56 %

4 800

Browse 71.15 ms 87.46 ms 22.92 % 2437.0 2420.8 0.66 % 65.60 % 70.27 % 7.12 % Manage 12.98 ms 17.04 ms 31.29 % 1199.7 1193.5 0.51 % Purchase 8.93 ms 12.88 ms 44.33 % 1211.6 1212.1 0.04 %

(Brunnert et al. 2014_1)

slide-24
SLIDE 24

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 25

Scenario Results: Downscaling 1/2

Evaluation

CPU Cores Number

  • f Users

Business Transaction Measured RT Simulated RT Prediction Error Measured Throughput Simulated Throughput Prediction Error Measured CPU Simulated CPU Prediction Error

8 800

Browse 71.54 ms 64.03 ms 10.50 % 2413.9 2415.8 0.08 % 37.41 % 35.17 % 5.99 % Manage 12.96 ms 12.64 ms 2.49 % 1203.5 1209.2 0.48 % Purchase 9.36 ms 9.33 ms 0.25 % 1215.9 1228.7 1.05 %

6 800

Browse 67.62 ms 66.03 ms 2.35 % 2413.9 2425.4 0.48 % 46.38 % 46.94 % 1.21 % Manage 12.52 ms 13.08 ms 4.45 % 1202.0 1196.6 0.45 % Purchase 9.05 ms 9.64 ms 6.57 % 1208.2 1215.0 0.56 %

4 800

Browse 71.15 ms 87.46 ms 22.92 % 2437.0 2420.8 0.66 % 65.60 % 70.27 % 7.12 % Manage 12.98 ms 17.04 ms 31.29 % 1199.7 1193.5 0.51 % Purchase 8.93 ms 12.88 ms 44.33 % 1211.6 1212.1 0.04 %

(Brunnert et al. 2014_1)

slide-25
SLIDE 25

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 26

Scenario Results: Downscaling 2/2

Evaluation

30 60 90 120 150 MRT Brow se SRT Brow se response time [ms] 5 10 15 20 25 30 35 MRT Manage SRT Manage response time [ms] 5 10 15 20 25 30 35 MRT Purchase SRT Purchase response time [ms] 30 60 90 120 150 MRT Brow se SRT Brow se response time [ms] 5 10 15 20 25 30 35 MRT Manage SRT Manage response time [ms] 5 10 15 20 25 30 35 MRT Purchase SRT Purchase response time [ms] 30 60 90 120 150 MRT Brow se SRT Brow se response time [ms] 5 10 15 20 25 30 35 MRT Manage SRT Manage response time [ms] 5 10 15 20 25 30 35 MRT Purchase SRT Purchase response time [ms]

8 CPU cores, 800 users 6 CPU cores, 800 users 4 CPU cores, 800 users

(Brunnert et al. 2014_1)

slide-26
SLIDE 26

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 27

  • Motivation and Vision
  • Performance Model Generation

– Data Collection – Data Aggregation – Model Generation

  • Evaluation

– SPECjEnterprise2010 – Overhead Evaluation – Experiment Setup – Scenario Description – Scenario Results

  • Upscaling
  • Downscaling
  • Future Work

Agenda

slide-27
SLIDE 27

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 28

  • Usage model generation

– Master Thesis recently finished based on session traces – Christian Vögele currently extends this approach

  • Representing distributed systems

– Distributing the transaction ID within distributed Java EE environments using the Application Response Measurement (ARM) standard or similar means – Aggregating data from multiple Java EE instances

  • Recently finished a prototype for distributed deployments communicating over SOAP

– Setting thread limits for each tier (Web, EJB, JDBC Conn. Pools)

  • Supporting new component types:

– JSF, web-services, Message driven beans, Other Java EE 7.0 enhancements, e.g. Batch Jobs, ..

Future Work

slide-28
SLIDE 28

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 29

  • Using APM data for the model generation (collaboration with Dynatrace)

Future Work

dynaTrace Agent CSV MBeans PMWT Agent CSV CSV Performance Model dynaTrace Server PMWT Connector dynaTrace Connector Performance Model Generator Java EE Application dynaTrace Performance Warehouse Monitoring Database Session Store Monitoring Data Persistence Service

(Willnecker et al. 2014)

slide-29
SLIDE 29

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 30

Brunnert, A.; Vögele, C.; Krcmar, H. (2013): Automatic performance model generation for java enterprise edition (ee) applications, in: M. S. Balsamo, W. J. Knottenbelt, A. Marin (Eds.), Computer Performance Engineering, Vol. 8168 of Lecture Notes in Computer Science, Springer Berlin Heidelberg, 2013, pp. 74-88. Brunnert, A.; Neubig, S.; Krcmar, H. (2014): Evaluating the prediction accuracy of generated performance models in up- and downscaling scenarios, in: Proceedings of the Symposium on Software Performance, SOSP 14, 2014, pp. 113-130. Brunnert, A.; Wischer, K.; Krcmar, H. (2014): Using architecture-level performance models as resource profiles for enterprise applications, in: Proceedings of the 10th International ACM Sigsoft Conference on Quality of Software Architectures, QoSA '14, ACM, New York, NY, USA, 2014, pp. 53-62. Standard Performance Evaluation Corporation (SPEC) (2009): SPECjEnterprise2010 Design Document. http://www.spec.org/jEnterprise2010/docs/DesignDocumentation.html, accessed at 2014-11-26 Willnecker, F.; Brunnert, A.; Gottesheim, W.; Krcmar, H. (2014): Using dynatrace monitoring data for generating performance models of Java EE applications. Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, Austin, TX, USA, (accepted for publication, to appear).

Bibliography

slide-30
SLIDE 30

pmw.fortiss.org SOSP 2014, Stuttgart, Germany, 2014-11-27 31

Q&A

Andreas Brunnert, Stefan Neubig

performancegroup@fortiss.org pmw.fortiss.org