Using Non-Redundant Mutation Operators and Test Suite Prioritization - - PowerPoint PPT Presentation

▶

Jan 01, 2023 310 likes •1.18k views

Using Non-Redundant Mutation Operators and Test Suite Prioritization to Achieve Efficient and Scalable Mutation Analysis Ren Just 1 , 2 & Gregory M. Kapfhammer 3 & Franz Schweiggert 2 1 University of Washington, USA 2 Ulm University,

SLIDE 1

Using Non-Redundant Mutation Operators and Test Suite Prioritization to Achieve Efficient and Scalable Mutation Analysis

René Just1,2 & Gregory M. Kapfhammer3 & Franz Schweiggert2

1University of Washington, USA 2Ulm University, Germany 3Allegheny College, USA

23rd International Symposium on Software Reliability Engineering November 28, 2012

SLIDE 2

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis Background

Mutation analysis assesses the quality of a test suite with artificial faults (mutants)

SLIDE 3

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis Background

Mutation analysis assesses the quality of a test suite with artificial faults (mutants)

Program Test suite

SLIDE 4

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis Background

Mutation analysis assesses the quality of a test suite with artificial faults (mutants)

Program Test suite Generate mutants Mutants

SLIDE 5

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis Background

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; }

Mutation analysis assesses the quality of a test suite with artificial faults (mutants)

Program Test suite Generate mutants Mutants

Original

SLIDE 6

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis Background

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; }

Mutation analysis assesses the quality of a test suite with artificial faults (mutants)

Program Test suite Generate mutants Mutants

Original

SLIDE 7

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis Background

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } public int max(int a, int b){ int max = a; if (b>=a){ max=b; } return max; }

Mutation analysis assesses the quality of a test suite with artificial faults (mutants)

Program Test suite Generate mutants Mutants

Contains a small syntactic change Original Mutant

SLIDE 8

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis Background

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } public int max(int a, int b){ int max = a; if (b>=a){ max=b; } return max; }

Mutation analysis assesses the quality of a test suite with artificial faults (mutants)

Program Test suite Generate mutants Mutants Execute mutants Mutation score

Contains a small syntactic change Original Mutant

SLIDE 9

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis is Expensive

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original

SLIDE 10

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis is Expensive

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original if (b < a) if (b <= a) if (b >= a) if (b != a) if (b == a)

SLIDE 11

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis is Expensive

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original if (b < a) if (b <= a) if (b >= a) if (b != a) if (b == a)

Many mutants can be generated for large programs

SLIDE 12

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis is Expensive

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original if (b < a) if (b <= a) if (b >= a) if (b != a) if (b == a)

Many mutants can be generated for large programs Large programs include comprehensive test suites

SLIDE 13

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Analysis is Expensive

public int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original if (b < a) if (b <= a) if (b >= a) if (b != a) if (b == a)

Many mutants can be generated for large programs Large programs include comprehensive test suites Executing the entire test suite for all mutants in large programs is prohibitive!

SLIDE 14

Introduction Reduction Characteristics Prioritization Conclusion

Overview: Efficient Mutation Analysis

Execute fewer mutants fewer times

SLIDE 15

Introduction Reduction Characteristics Prioritization Conclusion

Overview: Efficient Mutation Analysis

Execute fewer mutants fewer times Mutant reduction Generate fewer mutants Execute fewer mutants

SLIDE 16

Introduction Reduction Characteristics Prioritization Conclusion

Overview: Efficient Mutation Analysis

Execute fewer mutants fewer times Mutant reduction Generate fewer mutants Execute fewer mutants Test suite prioritization Test suite characteristics Reordering and splitting

SLIDE 17

Introduction Reduction Characteristics Prioritization Conclusion

Overview: Efficient Mutation Analysis

Execute fewer mutants fewer times Mutant reduction Generate fewer mutants Execute fewer mutants 27% Test suite prioritization Test suite characteristics Reordering and splitting 29% Empirical evaluation of 10 open-source projects with 560,000 mutants

SLIDE 18

Introduction Reduction Characteristics Prioritization Conclusion

Reduction of Mutants

Execute fewer mutants fewer times Mutant reduction Generate fewer mutants Execute fewer mutants 27% Empirical evaluation of 10 open-source projects with 560,000 mutants

SLIDE 19

Introduction Reduction Characteristics Prioritization Conclusion

Reduce Number of Generated Mutants

Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b → a - b

(replace binary operator) a + b → a + (-b) (insert unary operator)

Use only non-redundant mutation operators

Avoid the generation of such subsumed mutants

SLIDE 20

Introduction Reduction Characteristics Prioritization Conclusion

Reduce Number of Generated Mutants

Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b → a - b

(replace binary operator) a + b → a + (-b) (insert unary operator)

Use only non-redundant mutation operators

Avoid the generation of such subsumed mutants

Number of generated mutants reduced by 27%

SLIDE 21

Introduction Reduction Characteristics Prioritization Conclusion

Reduce Number of Generated Mutants

Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b → a - b

(replace binary operator) a + b → a + (-b) (insert unary operator)

Use only non-redundant mutation operators

Avoid the generation of such subsumed mutants

Number of generated mutants reduced by 27% More than 410,000 gen- erated mutants remaining

SLIDE 22

Introduction Reduction Characteristics Prioritization Conclusion

Reduce Number of Generated Mutants

Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b → a - b

(replace binary operator) a + b → a + (-b) (insert unary operator)

Use only non-redundant mutation operators

Avoid the generation of such subsumed mutants

Number of generated mutants reduced by 27% More than 410,000 gen- erated mutants remaining Executing all non-redundant mutants is still prohibitive!

SLIDE 23

Introduction Reduction Characteristics Prioritization Conclusion

Reduce Number of Executed Mutants

Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants

SLIDE 24

Introduction Reduction Characteristics Prioritization Conclusion

Reduce Number of Executed Mutants

Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Total reduction of executed mutants of more than 50%

SLIDE 25

Introduction Reduction Characteristics Prioritization Conclusion

Reduce Number of Executed Mutants

Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Total reduction of executed mutants of more than 50% Mutation analysis runtime still up to 13 hours

SLIDE 26

Introduction Reduction Characteristics Prioritization Conclusion

Reduce Number of Executed Mutants

Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Total reduction of executed mutants of more than 50% Mutation analysis runtime still up to 13 hours Further optimizations beyond the reduction of mutants are necessary!

SLIDE 27

Introduction Reduction Characteristics Prioritization Conclusion

Optimized Workflow for Mutation Analysis

Execute fewer mutants fewer times Test suite prioritization Test suite characteristics Reordering and splitting 29% Empirical evaluation of 10 open-source projects with 560,000 mutants

SLIDE 28

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5

SLIDE 29

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Test case t1: 5 seconds Test case t2: 2 seconds Test case t3: 1 second

SLIDE 30

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Test case t1: 5 seconds 1, 2, 3, 4, 5 Test case t2: 2 seconds 1, 3, 4, 5 Test case t3: 1 second 1, 2, 3

SLIDE 31

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3

SLIDE 32

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1t2t3 :

SLIDE 33

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1t2t3 : 1 2 3 4 5

SLIDE 34

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1t2t3 : 1 2 3 4 5 3 4

SLIDE 35

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1t2t3 : 1 2 3 4 5 3 4 3

SLIDE 36

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1t2t3 : 1 2 3 4 5 3 4 3 t3t2t1 :

SLIDE 37

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1t2t3 : 1 2 3 4 5 3 4 3 t3t2t1 : 1 2 3

SLIDE 38

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1t2t3 : 1 2 3 4 5 3 4 3 t3t2t1 : 1 2 3 1 4 5

SLIDE 39

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Reordering

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1t2t3 : 1 2 3 4 5 3 4 3 t3t2t1 : 1 2 3 1 4 5 2 5

SLIDE 40

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Splitting

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5

SLIDE 41

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Splitting

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t′

1:

3 seconds 1, 2, 3, 4 1, 2 Test case t′′

1 :

2 seconds 2, 3, 4, 5 2, 5

SLIDE 42

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Splitting

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t′

1:

3 seconds 1, 2, 3, 4 1, 2 Test case t′′

1 :

2 seconds 2, 3, 4, 5 2, 5 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 : 1 2 3 4 5

SLIDE 43

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Splitting

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t′

1:

3 seconds 1, 2, 3, 4 1, 2 Test case t′′

1 :

2 seconds 2, 3, 4, 5 2, 5 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 : 1 2 3 4 5 t′

1 t′′ 1 :

SLIDE 44

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Splitting

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t′

1:

3 seconds 1, 2, 3, 4 1, 2 Test case t′′

1 :

2 seconds 2, 3, 4, 5 2, 5 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 : 1 2 3 4 5 t′

1 t′′ 1 :

1 2 3 4

SLIDE 45

Introduction Reduction Characteristics Prioritization Conclusion

Motivating Example for Splitting

Mutants: 1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t′

1:

3 seconds 1, 2, 3, 4 1, 2 Test case t′′

1 :

2 seconds 2, 3, 4, 5 2, 5 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 : 1 2 3 4 5 t′

1 t′′ 1 :

1 2 3 4 3 4 5

SLIDE 46

Introduction Reduction Characteristics Prioritization Conclusion

Runtime Distribution of Tests within Test Suites

trove

chart itext math time lang jdom jaxen io num4j 5 10 15 20 Test runtime in seconds

SLIDE 47

Introduction Reduction Characteristics Prioritization Conclusion

Runtime Distribution of Tests within Test Suites

trove

chart itext math time lang jdom jaxen io num4j 5 10 15 20 Test runtime in seconds

Most tests have short runtime

SLIDE 48

Introduction Reduction Characteristics Prioritization Conclusion

Runtime Distribution of Tests within Test Suites

trove

chart itext math time lang jdom jaxen io num4j 5 10 15 20 Test runtime in seconds

Most tests have short runtime A few long- running outliers

SLIDE 49

Introduction Reduction Characteristics Prioritization Conclusion

Runtime Distribution of Tests within Test Suites

trove

chart itext math time lang jdom jaxen io num4j 5 10 15 20 Test runtime in seconds

Most tests have short runtime A few long- running outliers

A few tests constitute most of the total runtime: Reduce number of executions for these tests

SLIDE 50

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Coverage Overlap

Overlap measures the similarity of a test case with its enclosing test suite Pair-wise comparison of test cases is infeasible

SLIDE 51

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Coverage Overlap

Overlap measures the similarity of a test case with its enclosing test suite Pair-wise comparison of test cases is infeasible Definition: Overlap O(ti, T), ti ∈ T O(ti, T) ≔

      

1,

|Cov(ti)| = 0

|Cov(ti)∩Cov(T\ti)| |Cov(ti)|

, |Cov(ti)| > 0

SLIDE 52

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Coverage Overlap

Overlap measures the similarity of a test case with its enclosing test suite Pair-wise comparison of test cases is infeasible Definition: Overlap O(ti, T), ti ∈ T O(ti, T) ≔

      

1,

|Cov(ti)| = 0

|Cov(ti)∩Cov(T\ti)| |Cov(ti)|

, |Cov(ti)| > 0

Most of the test cases exhibit high overlap: Does test runtime correlate with overlap?

SLIDE 53

Introduction Reduction Characteristics Prioritization Conclusion

Correlation of Test Runtime and Mutation Coverage

1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 10 20 30 40 50 60 Index of test in original test suite 50 100 150 200 250 Runtime of test in milliseconds

SLIDE 54

Introduction Reduction Characteristics Prioritization Conclusion

Correlation of Test Runtime and Mutation Coverage

1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 10 20 30 40 50 60 Index of test in original test suite 50 100 150 200 250 Runtime of test in milliseconds

Test case with longest runtime

SLIDE 55

Introduction Reduction Characteristics Prioritization Conclusion

Correlation of Test Runtime and Mutation Coverage

1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 10 20 30 40 50 60 Index of test in original test suite 50 100 150 200 250 Runtime of test in milliseconds

Test case with longest runtime Overlapping test cases

SLIDE 56

Introduction Reduction Characteristics Prioritization Conclusion

Correlation of Test Runtime and Mutation Coverage

1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 10 20 30 40 50 60 Index of test in original test suite 50 100 150 200 250 Runtime of test in milliseconds

Test case with longest runtime Overlapping test cases

Reorder to exploit mutation coverage overlap

SLIDE 57

Introduction Reduction Characteristics Prioritization Conclusion

Correlation of Test Runtime and Mutation Coverage

1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 10 20 30 40 50 60 Index of test in original test suite 50 100 150 200 250 Runtime of test in milliseconds

Test case with longest runtime Overlapping test cases

Reorder to exploit mutation coverage overlap

Large mutation coverage

SLIDE 58

Introduction Reduction Characteristics Prioritization Conclusion

Correlation of Test Runtime and Mutation Coverage

1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 10 20 30 40 50 60 Index of test in original test suite 50 100 150 200 250 Runtime of test in milliseconds

Test case with longest runtime Overlapping test cases

Reorder to exploit mutation coverage overlap

Large mutation coverage

Split test cases to increase coverage precision

SLIDE 59

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Coverage of Test suites

test suite class#1 method#1 ... ... ... class#m ... method#n

SLIDE 60

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Coverage of Test suites

test suite class#1 method#1 ... ... ... class#m ... method#n Higher precision

SLIDE 61

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Coverage of Test suites

test suite class#1 method#1 ... ... ... class#m ... method#n Lower overhead Higher precision

SLIDE 62

Introduction Reduction Characteristics Prioritization Conclusion

Mutation Coverage of Test suites

test suite class#1 method#1 ... ... ... class#m ... method#n Lower overhead Higher precision

Only split long-running test classes

SLIDE 63

Introduction Reduction Characteristics Prioritization Conclusion

Splitting Test Classes

Two splitting strategies

SLIDE 64

Introduction Reduction Characteristics Prioritization Conclusion

Splitting Test Classes

Two splitting strategies Split entire long- running test class High overhead and coverage precision

SLIDE 65

Introduction Reduction Characteristics Prioritization Conclusion

Splitting Test Classes

Two splitting strategies Split entire long- running test class High overhead and coverage precision Extract only long- running test methods Lower overhead and coverage precision

SLIDE 66

Introduction Reduction Characteristics Prioritization Conclusion

Splitting Test Classes

Two splitting strategies Split entire long- running test class High overhead and coverage precision Extract only long- running test methods Lower overhead and coverage precision Trade-off between overhead and precision: Splitting based on threshold for test runtime

SLIDE 67

Introduction Reduction Characteristics Prioritization Conclusion

Optimized workflow

Original program Generate mutants Set of non- redundant mutants

SLIDE 68

Introduction Reduction Characteristics Prioritization Conclusion

Optimized workflow

Original program Generate mutants Set of non- redundant mutants Execute test suite Original test suite

SLIDE 69

Introduction Reduction Characteristics Prioritization Conclusion

Optimized workflow

Original program Generate mutants Set of non- redundant mutants Execute test suite Original test suite Runtime of test cases Mutation coverage

SLIDE 70

Introduction Reduction Characteristics Prioritization Conclusion

Optimized workflow

Original program Generate mutants Set of non- redundant mutants Execute test suite Original test suite Runtime of test cases Mutation coverage Order/split test cases Prioritized test suite

SLIDE 71

Introduction Reduction Characteristics Prioritization Conclusion

Optimized workflow

Original program Generate mutants Set of non- redundant mutants Execute test suite Original test suite Runtime of test cases Mutation coverage Order/split test cases Prioritized test suite Mutation analysis

SLIDE 72

Introduction Reduction Characteristics Prioritization Conclusion

Example with Original Test Suite

0.2 0.4 0.6 0.8 1 Mutation score Original test suite 7 14 21 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes

SLIDE 73

Introduction Reduction Characteristics Prioritization Conclusion

Example with Original Test Suite

0.2 0.4 0.6 0.8 1 Mutation score Original test suite 7 14 21 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes Total runtime of test executing all covered, yet not killed, mutants

SLIDE 74

Introduction Reduction Characteristics Prioritization Conclusion

Example with Original Test Suite

0.2 0.4 0.6 0.8 1 Mutation score Original test suite 7 14 21 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes Total runtime of test executing all covered, yet not killed, mutants

SLIDE 75

Introduction Reduction Characteristics Prioritization Conclusion

Example with Original Test Suite

0.2 0.4 0.6 0.8 1 Mutation score Original test suite 7 14 21 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes Total runtime of test executing all covered, yet not killed, mutants

Reorder

SLIDE 76

Introduction Reduction Characteristics Prioritization Conclusion

Example with Original Test Suite

0.2 0.4 0.6 0.8 1 Mutation score Original test suite 7 14 21 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes Total runtime of test executing all covered, yet not killed, mutants

Reorder Split

SLIDE 77

Introduction Reduction Characteristics Prioritization Conclusion

Example with Prioritized Test Suite

0.2 0.4 0.6 0.8 1 Mutation score Prioritized test suite 7 14 21 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes

SLIDE 78

Introduction Reduction Characteristics Prioritization Conclusion

Empirical Results

Reordering: Reordering decreases the runtime by 20% Splitting strategies: Extracting long test methods reduces the runtime by 29% Splitting entire test classes increases the runtime by 27% Splitting may increase runtime if: Test suite has a very low mutation detection rate Test methods exhibit huge mutation coverage overlap

SLIDE 79

Introduction Reduction Characteristics Prioritization Conclusion

Empirical Results

Reordering: Reordering decreases the runtime by 20% Splitting strategies: Extracting long test methods reduces the runtime by 29% Splitting entire test classes increases the runtime by 27% Splitting may increase runtime if: Test suite has a very low mutation detection rate Test methods exhibit huge mutation coverage overlap Prioritizing test suites improves the efficiency

f mutation analysis by 29% on average!

29%

SLIDE 80

Introduction Reduction Characteristics Prioritization Conclusion

Related Work

Reduction of generated mutants: Sufficient mutation operators

Offutt et al., TOSEM’96 Namin et al., ICSE’08

Non-redundant mutation operators

Kaminski et al., AST’11 Just et al., Mutation’12

Mutation-based test suite optimization: Test case prioritization

Elbaum et al. TSE’02 Do and Rothermel, TSE’06

SLIDE 81

Introduction Reduction Characteristics Prioritization Conclusion

Related Work

Reduction of generated mutants: Sufficient mutation operators

Offutt et al., TOSEM’96 Namin et al., ICSE’08

Non-redundant mutation operators

Kaminski et al., AST’11 Just et al., Mutation’12

Mutation-based test suite optimization: Test case prioritization

Elbaum et al. TSE’02 Do and Rothermel, TSE’06

Still contain redundancies

SLIDE 82

Introduction Reduction Characteristics Prioritization Conclusion

Related Work

Reduction of generated mutants: Sufficient mutation operators

Offutt et al., TOSEM’96 Namin et al., ICSE’08

Non-redundant mutation operators

Kaminski et al., AST’11 Just et al., Mutation’12

Mutation-based test suite optimization: Test case prioritization

Elbaum et al. TSE’02 Do and Rothermel, TSE’06

Still contain redundancies Used in empirical study

SLIDE 83

Introduction Reduction Characteristics Prioritization Conclusion

Related Work

Reduction of generated mutants: Sufficient mutation operators

Offutt et al., TOSEM’96 Namin et al., ICSE’08

Non-redundant mutation operators

Kaminski et al., AST’11 Just et al., Mutation’12

Mutation-based test suite optimization: Test case prioritization

Elbaum et al. TSE’02 Do and Rothermel, TSE’06

Still contain redundancies Used in empirical study Do not address efficiency

SLIDE 84

Introduction Reduction Characteristics Prioritization Conclusion

Conclusions

Reduction of mutants: Non-redundant operators reduce number of mutants by 27% Test suite characteristics: Most of the tests exhibit mutation coverage overlap Notable difference in runtime of tests Optimized workflow: Exploits mutation coverage overlap and runtime differences Further reduces total runtime of mutation analysis by 29%

SLIDE 85

Introduction Reduction Characteristics Prioritization Conclusion

Conclusions

Reduction of mutants: Non-redundant operators reduce number of mutants by 27% Test suite characteristics: Most of the tests exhibit mutation coverage overlap Notable difference in runtime of tests Optimized workflow: Exploits mutation coverage overlap and runtime differences Further reduces total runtime of mutation analysis by 29% Non-redundant operators and optimized workflow implemented in the MAJOR mutation system