clean performance test results By Vivek Venkatachalam - PowerPoint PPT Presentation

PerfCells: A case study in achieving clean performance test results By Vivek Venkatachalam ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 1

Performance is key for quality. ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 2

Inconsistent results makes it hard to do performance testing well. ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 3

The real problems are hard to find ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 4

What if you could eliminate the haystack ? ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 5

First, an overview of our performance engineering system - Planning - Instrumentation - Testing - Analysis ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 6

Planning - Key customer scenarios ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 7

Planning - Metrics ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 8

Instrumentation – Inserting code markers ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 9

Testing - Manual ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 10

Testing - Automation ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 11

Analysis – Processing the results ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 12

The problem: Lots of noise in the system. ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 13

Unexplainable variations in results led to wasted time and effort ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 14

Needed a quick fix ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 15

Decided to focus only on large anomalies ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 16

The long term solution ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 17

Brainstorming ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 18

Realization : We had conflicting goals Prevent Verify that regressions real world however performance small. goals were being met. ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 19

The solution: Perf cells ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 20

Key point: one machine ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 21

Advantages - Isolated ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 22

Advantages - Reliable behavior ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 23

Advantages - Easy management ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 24

Advantages - Scale Out ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 25

Our results ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 26

Histogram of elapsed times from tests on pre-production topology ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 27

Histogram of elapsed times from tests on a PerfCell ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 28

Key Takeaways • Steady losses add up. • PerfCells increased test sensitivity. • Allowed us to accept variability in real world testing. ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 29

Some possible improvements • Test builds simultaneously • Partition tests • PerfCells in the cloud ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 30

Credits My co-authors Marcelo and Shirley. Our reviewers Bob and Mano Microsoft Clip Art for the images. ________________________________________________________________________________________________________________ Copies may not be made or distributed for commercial use Excerpt from PNSQC 2011 PNSQC.ORG 31