Software Testing with Evolutionary Strategies Introduction - - PowerPoint PPT Presentation

▶

Sep 24, 2023 186 likes •416 views

Rapid Integration of Software Engineering Techniques, 2005 1/22 Software Testing with Evolutionary Strategies Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work Enrique Alba

SLIDE 1

1/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Software Testing with Evolutionary Strategies

Enrique Alba and J. Francisco Chicano

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 2

2/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Introduction

After codification, software products require a test phase
The objective is to find errors and to ensure software correctness
Software companies dedicate 50% of resources to this task
We propose an automatic tool based on Metaheuristics to generate the

input data for the test 1.0, 2.3 1.0, 2.3 10.5 Ok! 2.7, 5.4 2.7, 5.4 15.0 Wrong! Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 3

3/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Test Adequacy Criteria

Test data generator objective: to propose input data finding a

maximum number of errors Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 4

4/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Test Adequacy Criteria

Test data generator objective: to propose input data finding a

maximum number of errors

Test data generator objectives (test adequacy criteria)

DIFFICULT TO CHECK Statement Coverage Branch Coverage Condition-Decision Coverage All the statements executed All the branches taken All the atomic conditions true and false Stronger Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 5

5/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Three main paradigms:

Random test data generation Symbolic test data generation Dynamic test data generation

Previous Work

Inputs: a,b c = a+b c < 0 true false 1.2, 0.7 α, β α+β < 0 α+β ≥ 0 1.0, -2.0 3.5, 1.2

1.0, -0.5

1.0, -0.5 Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 6

6/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Previous Work

The global objective is broken down in small sub-objectives

c1 true c1 false c2 true c3 false c3 true c2 false Six sub-objectives condition-decision coverage Function minimization problem Input data Distance Sub-objective (c3 true) Current input data Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 7

7/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Test Data Generator

Test Program Select sub-objective Evolutionary Strategy End Continue? yes no Test Data Generator Input data Fitness Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 8

8/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Evolutionary Strategy

Genetic Algorithm Evolutionary Strategy Genetic Programming Evolutionary Programming

Evolutionary Algorithms

Optimization problem Population-based Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 9

9/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Evolutionary Strategy

t := 0; P(0) := Generate(); Evaluate (P(0)); while stop criterion not met do P’(t) := Select (P(t)); P’’(t) := Gaussian_Mutation (P’(t)); Evaluate(P’’(t)); P(t+1) := Replacement (P(t), P’’(t)); t := t+1; end while; return the best solution;

Pseudo-code of an Evolutionary Strategy

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 10

10/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Selected solution New solution

Evolutionary Strategy

Evolutionary Strategy

Individual Gaussian Mutation Replacement → (μ+λ) and (μ,λ) Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work (0.2, -1.4, 3.5) (1.0, 10.3, 7.2) (1.7, 0.3, 2.1) → Solution vector → Angles → Standard deviations

Previous population New individuals

New population μ individuals

SLIDE 11

11/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Evolutionary Strategy

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work Population Gaussian Mutation Test Program (0.1, -5.2, 3.0, ...) 7.3 (μ+λ) Replacement New population μ individuals λ individuals

SLIDE 12

12/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Experiments: Benchmark

Benchmark: 11 programs in C language

Program Conds. LOC Args. Source

triangle 21 53 3 Michael et al., 2001 gcd 5 38 2 Authors calday 11 72 3 C-Recipes crc 9 82 13 C-Recipes insertion 5 47 20 C-Recipes shell 7 58 20 C-Recipes quicksort 18 143 20 C-Recipes heapsort 10 72 20 C-Recipes select 28 200 21 C-Recipes bessel 21 245 2 C-Recipes sa 30 332 23 C-Recipes

We perform 30 independent runs to get statistical confidence

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 13

13/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Experiments: First Results

Program Avg. Cov.(%) Avg. Evals. Avg. Time(s)

triangle 97.54 1975 10.6 gcd 100.00 21 0.0 calday 91.82 1182 6.8 crc 94.44 1114 28.7 insertion 100.00 10 0.0 shell 100.00 10 0.0 quicksort 88.89 1110 8.4 heapsort 90.00 1110 9.4 select 53.57 1120 8.9 bessel 95.08 1306 7.6 sa 97.06 1329 5082.1

Algorithm: (10+1)-ES
Maximum iterations: 100
Condition-decision is not a good metric for measuring accuracy

while(1<2) { /* something */ }

Code-dependent coverage loss

p = malloc(4); if (!p) { error(); }

Environment-dependent coverage loss Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 14

14/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Experiments: First Results

(10+1)-ES Corrected Coverage (%) Program Cov.(%) Avg.(%) Max(%)

Std. Dev.

triangle 97.54 99.92 100.00 91.82 100.00 100.00 100.00 94.12 100.00 83.33 97.40 98.70 gcd 100.00 0.0044 100.00 100.00 100.00 100.00 100.00 100.00 94.12 100.00 83.33 calday 91.82 97.56 0.0000 0.0246 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0061 crc 94.44 insertion 100.00 shell 100.00 quicksort 88.89 heapsort 90.00 select 53.57 bessel 95.08 100.00 0.0072 sa 97.06

Corrected coverage = reached sub-objectives reachable sub-objectives Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 15

15/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

10 20 30 40 50 60 70 80 90 100

triangle gcd calday crc insertion shell quicksort heapsort select bessel sa

Test Programs Corrected Coverage (%) 1 2 3

Number of offspring:

Experiments: Influence of λ

Influence of λ Coverage: negligible Number of evaluations: important Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 16

16/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

10 20 30 40 50 60 70 80 90 100

triangle gcd calday crc insertion shell quicksort heapsort select bessel sa

Test Programs Corrected Coverage (%) 1 2 3

Number of offspring:

500 1000 1500 2000 2500 3000 3500 4000 4500

triangle gcd calday crc insertion shell quicksort heapsort select bessel sa

Test Programs Evaluations 1 2 3

Number of offspring:

Experiments: Influence of λ

Influence of λ Coverage: negligible Number of evaluations: important Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 17

17/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Experiments: Influence of μ

10 20 30 40 50 60 70 80 90 100

triangle gcd calday crc insertion shell quicksort heapsort select bessel sa

Test Programs Corrected Coverage (%) 1 5 10 20 30

Population size:

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work Influence of μ Coverage: negligible for μ ≥ 5 Number of evaluations: slight

SLIDE 18

18/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Experiments: Influence of μ

10 20 30 40 50 60 70 80 90 100

triangle gcd calday crc insertion shell quicksort heapsort select bessel sa

Test Programs Corrected Coverage (%) 1 5 10 20 30

Population size:

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

500 1000 1500 2000 2500

triangle gcd calday crc insertion shell quicksort heapsort select bessel sa

Test Programs Evaluations 1 5 10 20 30

Population size:

Influence of μ Coverage: negligible for μ ≥ 5 Number of evaluations: slight

SLIDE 19

19/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Experiments: ES vs GA

(10+1)-ES (10+1)-GA Program Avg. Cov.(%) Avg. Evals. Avg. Cov.(%) Avg. Evals.

triangle 99.92 1975 95.20 2009 252 1217 1121 10 10 1110 1110 1339 1557 1110 gcd 100.00 21 100.00 calday 91.82 1182 90.91 crc 100.00 1114 100.00 insertion 100.00 10 100.00 shell 100.00 10 100.00 quicksort 94.12 1110 94.12 heapsort 100.00 1110 100.00 select 83.33 1120 83.33 bessel 97.40 1306 96.91 sa 98.70 1329 96.61

ES outperforms the results of the Genetic Algorithm

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 20

20/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Experiments: Previous Results

triangle Jones et al.,1996 Michael et al., 2001 Sagarna et al., 2003 Sagarna et al., 2005 (5+1)-ES (this work) Coverage(%) 100.00 94.29 100.00 100.00 100.00 Evaluations 18000 ≈8000 608 3439 2010

Two main issues to compare against different works:
The implementation → http://tracer.lcc.uma.es/problems/testing
The coverage measures

Branch coverage Corrected condition-decision coverage Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work DeepCover coverage

The (5+1)-ES gets total coverage with less number of evaluations

(exception: Sagarna et al., 2003)

SLIDE 21

21/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Conclusions & Future Work

We propose a test data generator using an ES for software testing
Condition-decision coverage is not a good measure of the generator

accuracy → Corrected condition-decision coverage

Increasing the number of offspring has a negligible impact in the

coverage but it increases the number of evaluations needed

The population size must be larger than one for a good coverage
The ES-based test data generator outperforms the results of the GA-

based one

Conclusions Future Work

Apply the test data generator to telecommunication software
Use parallel evolutionary strategies
Integrate some static analysis techniques

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work

SLIDE 22

22/22 Heraklion Crete, Greece, September 8-9, 2005 Rapid Integration of Software Engineering Techniques, 2005

Thanks for your attention !!!

THE END

Introduction Previous Work Test Data Generator Evolutionary Strategy Experiments Conclusions & Future Work