Landscaping Performance Research at the ICPE and its Predecessors: A - - PowerPoint PPT Presentation

▶
landscaping performance research at the icpe and its
SMART_READER_LITE
LIVE PREVIEW

Landscaping Performance Research at the ICPE and its Predecessors: A - - PowerPoint PPT Presentation

Nov 30, 2022 127 likes •435 views

Austin, TX, USA, 2015-02-02 Landscaping Performance Research at the ICPE and its Predecessors: A Systematic Literature Review Short Paper International Conference on Performance Engineering (ICPE) 2015 Alexandru Danciu 1 , Johannes Kro 1 ,

slide-1
SLIDE 1

Austin, TX, USA, 2015-02-02

Landscaping Performance Research at the ICPE and its Predecessors: A Systematic Literature Review

Short Paper International Conference on Performance Engineering (ICPE) 2015 Alexandru Danciu 1, Johannes Kroß 1, Andreas Brunnert 1, Felix Willnecker 1, Christian Vögele 1, Anand Kapadia 2, Helmut Krcmar 2

1fortiss GmbH, 2Technische Universität München

slide-2
SLIDE 2

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 2

  • Methodology
  • Results

– Topics at the ICPE – Research and Contribution Facets – Evaluation Methods – Geographical and Organizational Perspective

  • Conclusion

Agenda

slide-3
SLIDE 3

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 3

  • Methodology
  • Results

– Topics at the ICPE – Research and Contribution Facets – Evaluation Methods – Geographical and Organizational Perspective

  • Conclusion

Agenda

slide-4
SLIDE 4

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 4

Research Questions

  • The systematic literature review in this work is conducted following the

guidelines provided by Kitchenham/Charters (2007) and is guided by the following research questions (RQ):

– RQ 1: Which topics have been addressed in the papers published at the ICPE (respectively at its predecessors) in the time period from 1998 to 2014? – RQ 2: Which research facets, contribution facets and evaluation methods have been used in papers published at the ICPE and its predecessors? – RQ 3: Who are the top ten countries and organizations in terms of the quantity of articles published at the ICPE and its predecessors?

Methodology

slide-5
SLIDE 5

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 5

Data Sources and Paper Selection

Methodology

slide-6
SLIDE 6

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 6

  • Methodology
  • Results

– Topics at the ICPE – Research and Contribution Facets – Evaluation Methods – Geographical and Organizational Perspective

  • Conclusion

Agenda

slide-7
SLIDE 7

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 7

Frequency of Terms

Results - RQ1: Topics at the ICPE

UML Web LQN Power Cloud Power PCM

slide-8
SLIDE 8

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 8

Most Frequent Keyword Combinations (WOSP/SIPEW)

Results - RQ1: Topics at the ICPE

values need to be multiplied by 10 -3

slide-9
SLIDE 9

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 9

Most Frequent Keyword Combinations (ICPE)

Results - RQ1: Topics at the ICPE

values need to be multiplied by 10 -3

slide-10
SLIDE 10

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 10

  • Methodology
  • Results

– Topics at the ICPE – Research and Contribution Facets – Evaluation Methods – Geographical and Organizational Perspective

  • Conclusion

Agenda

slide-11
SLIDE 11

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 11

Distribution of Research Facets Over the Years

Research facets based on Petersen et al. (2008)

Results - RQ2: Research Facets

slide-12
SLIDE 12

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 12

Distribution of Contribution Facets Over the Years

RQ2: Contribution Facet

slide-13
SLIDE 13

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 13

  • Methodology
  • Results

– Topics at the ICPE – Research and Contribution Facets – Evaluation Methods – Geographical and Organizational Perspective

  • Conclusion

Agenda

slide-14
SLIDE 14

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 14

Distribution of Evaluation Methods Over the Years

Results - RQ2: Evaluation Methods

slide-15
SLIDE 15

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 15

  • Methodology
  • Results

– Topics at the ICPE – Research and Contribution Facets – Evaluation Methods – Geographical and Organizational Perspective

  • Conclusion

Agenda

slide-16
SLIDE 16

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 16

Top 10 Contributing Countries

Results - RQ3: Geographical Perspective

slide-17
SLIDE 17

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 17

Top 10 Contributing Organizations

Results - RQ3: Organizational Perspective

slide-18
SLIDE 18

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 18

  • Methodology
  • Results

– Topics at the ICPE – Research and Contribution Facets – Evaluation Methods – Geographical and Organizational Perspective

  • Conclusion

Agenda

slide-19
SLIDE 19

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 19

  • Most articles are focused on the system development phase.
  • Since 2010, an increasing number of papers address the system operation

phase which results in a well-balanced conference profile.

  • Constant shift of the conference focus towards the latest technologies such

as cloud computing.

  • The community would greatly benefit from more research which provides

taxonomies for the generated knowledge and summarizes existing findings.

  • Positive influence of hosting a conference on the number of publications of

this country.

Conclusion

slide-20
SLIDE 20

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 20

Q&A

Alexandru Danciu, Andreas Brunnert

performancegroup@fortiss.org pmw.fortiss.org

slide-21
SLIDE 21

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 21

  • A general overview of prevailing topics and methods within the community

does not exist.

  • The principle conference that is focused on the performance of software

systems and related questions is the International Conference on Performance Engineering (ICPE).

  • Performance engineering research at the ICPE and its predecessors is

analyzed in a systematic literature review.

Motivation

slide-22
SLIDE 22

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 22

N-Gram Analysis

  • The N-Gram analysis is employed to reveal trends within the conference

from 1998 to 2014 (Soper/Turel 2012, Demeyer et al. 2013):

– An N-Gram is a sequence of n words extracted from a body of text. – For example, the phrase “software performance management” can be divided into:

  • three 1-Grams (“software", "performance", "management"),
  • two 2-Grams (“software performance", "performance management"), and
  • ne 3-Gram (“software performance management").

– In order to prevent distortion of results we removed in several post-processing steps any unnecessary data such as author information, keyword lists, the bibliography, the appendix, page numbers and citation references.

Results - RQ1: Topics at the ICPE

slide-23
SLIDE 23

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 23

Method descriptions (Hevner et al. 2004)

Results - RQ2: Evaluation Methods

slide-24
SLIDE 24

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 24

Distribution of Research Facets Over Countries

Results - RQ3: Geographical Perspective

slide-25
SLIDE 25

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 25

Distribution of Contribution Facets Over Countries

Results - RQ3: Geographical Perspective

slide-26
SLIDE 26

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 26

Distribution of Evaluation Methods Over Countries

Results - RQ3: Geographical Perspective

slide-27
SLIDE 27

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 27

Distribution of Research Facets Over Countries

Results - RQ3: Organizational Perspective

slide-28
SLIDE 28

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 28

Distribution of Contribution Facets Over Organizations

Results - RQ3: Organizational Perspective

slide-29
SLIDE 29

pmw.fortiss.org ICPE 2015, Austin, TX, USA, 2015-02-02 29

Distribution of Evaluation Methods Over Countries

Results - RQ3: Organizational Perspective