Effective Black-Box Testing with Genetic Algorithms Mark Last and - - PowerPoint PPT Presentation
Dr. Mark Last (BGU) Effective Black-Box Testing with Genetic Algorithms Effective Black-Box Testing with Genetic Algorithms Mark Last and Shay Eyal Department of Information Systems Engineering Ben-Gurion University of the Negev, Beer-Sheva,
Effective Black-Box Testing with Genetic Algorithms
Mark Last and Shay Eyal Department of Information Systems Engineering Ben-Gurion University of the Negev, Beer-Sheva, Israel Abraham Kandel National Institute for Systems Test and Productivity (NISTP) University of South Florida, Tampa, FL, USA IBM Verification Conference 2005
2
Effective Black-Box Testing with Genetic Algorithms
3
Effective Black-Box Testing with Genetic Algorithms
– College of Engineering: PI - Prof. Abraham Kandel (Executive Director) – College of Business Administration: CO-PI - Prof. Alan R. Hevner
– Arizona State University: PI - Dr. Wei-Tek Tsai – Drexel University: PI - Dr. Rosina Weber – QTSI: PI - Chick Garcia – Ben Gurion University: PI - Dr. Mark Last – Echelon: PIs - Dr. Jay Bayne, Dr. Norm Brown
Funding Source: SPAWAR - US Space and Naval Warfare Systems Command (Grant No. N00039-01-1-2248). WWW: http://nistp.csee.usf.edu/ The Institute's mission is to engage in innovative research and to provide direct support to government agencies (and their development agents) which will enable them to substantially reduce cost and improve the development schedule and reliability of large-scale systems
4
Effective Black-Box Testing with Genetic Algorithms
– Random Testing (within / without the
– Special value testing – Boundary value testing – Equivalence classes – Decision tables – Input-Output Analysis
I n p u t s O u t p u t s
5
Effective Black-Box Testing with Genetic Algorithms
– A particular choice of input data to be used in testing a program
– A test set that has a high probability of detecting faults presenting in a computer program
– Generate good test cases
yet undiscovered error
might lead to the real cause of the bug
– Prioritize test cases
case detect faults during the testing process
6
Effective Black-Box Testing with Genetic Algorithms
– Develop novel methodology for generation of effective black-box test cases with genetic algorithms
– Eliminate "bad" test cases that are unlikely to expose any error, while increasing the number of "good" test cases that have a high probability of producing an erroneous output
– Define a general framework for evolving an effective set of test cases with a genetic algorithm – Enhance the proposed GA-based methodology by the novel Fuzzy- Based Age Extension of Genetic Algorithms (FAexGA) – Evaluate the effectiveness and the efficiency of the proposed approach using synthetic and real computer programs
7
Effective Black-Box Testing with Genetic Algorithms
– Optimization: numerical, combinatorial (job-shop scheduling). – Machine learning tasks: classification, prediction (weights for neural networks). – Economics: bidding strategies.
Holland J. H. Holland J. H.
8
Effective Black-Box Testing with Genetic Algorithms
Initialize Initialize population population Selection Selection Mutation Mutation Crossover Crossover Insert into Insert into population population
Select individuals for mating Mate individuals Mutate offspring
Insert offspring into population Stop criteria satisfied ?
Answer
Yes No
9
Effective Black-Box Testing with Genetic Algorithms
– How to create an initial population of potential solutions?
– Selection method (ranking, roulette wheel…). – Crossover-operator (one-point, uniform…). – Mutation-operator (inversion, flip…).
– Rate the candidate solutions quality
– N - population size – Pc - crossover probability – Pm - mutation probability – Ngen – number of generations
10
Effective Black-Box Testing with Genetic Algorithms
100000011101000 100111100011100 100110011101000 100001100011100
000
001 010
011
100 101 111
110
?
11
Effective Black-Box Testing with Genetic Algorithms
Parameters: population size, crossover probability, etc. Exploration – overall search in the solution space Exploitation – localized search in the promising
regions discovered so far in that space
12
Effective Black-Box Testing with Genetic Algorithms
Where A and B are two chromosomes chosen for crossover Where A and B are two chromosomes chosen for crossover
13
Effective Black-Box Testing with Genetic Algorithms
Knowledge Base Knowledge Base Fuzzification interface Fuzzification interface GA GA Defuzzification interface Defuzzification interface Inference System Inference System
State Variables Control Parameters
14
Effective Black-Box Testing with Genetic Algorithms
Young Middle-age Old
0.2 0.5 1
Membership Function
1
Relative Age
(= age/avg. lifetime)
Young: 0-30% Middle-age: 20-80% Old: 70-100%
0.3 0.8 0.7
Lifetime = number of generations a chromosome stays alive
15
Effective Black-Box Testing with Genetic Algorithms
Low Medium High
0.15 0.5 1
Membership Function
1
Pc Low: 0-20% Medium: 15-85% High: 80-100%
0.2 0.85 0.8
16
Effective Black-Box Testing with Genetic Algorithms
Random Random Initialize P(t) Initialize P(t)
P(t) = population on generation t.
Evaluate P(t) Evaluate P(t) Increase Age Increase Age Selection Selection Assign Pc Assign Pc Mutation Mutation Crossover Crossover Remove Individuals Remove Individuals with age > lifetime with age > lifetime Evaluate P(t) Evaluate P(t)
Stop criteria satisfied ? Call Fuzzy Logic Controller
17
Effective Black-Box Testing with Genetic Algorithms
Low Medium Low “Old” Medium High Medium “middle-age” Low Medium Low “Young” “Old” “middle-age” “Young” Parent I Parent II
18
Effective Black-Box Testing with Genetic Algorithms
– Each test case can be represented as a vector of binary or continuous values related to the inputs of the tested software
– An initial test set can be generated randomly in the space of possible input values.
– Selection, crossover, and mutation operators can be adapted to representation of test cases for a specific program.
– Test cases can be evaluated by their fault-exposing-potential using buggy (mutated) versions of the original program.
19
Effective Black-Box Testing with Genetic Algorithms
– Simple GA [Goldberg]. – GA with varying population size (GAVaPS). – Crisp Age extension of GA. – Fuzzy-based Age extension of GA (three configurations).
80% 20%
FAexGA#3
75% 70%
FLC – “Old” lower limit
25% 30%
FLC – “Young” upper limit FAexGA#2 FAexGA#1 Parameter
20
Effective Black-Box Testing with Genetic Algorithms
max
min max min
best
November 13, 2005 Shay Eyal 24
f = fitness f = fitness Phenotypical Phenotypical diversity diversity d= distance from the best solution d= distance from the best solution Genotypical diversity Genotypical diversity g = # of g = # of generations for the first solution generations for the first solution Unuse Unuse factor factor Solution Quality Solution Quality
Formula Formula Performance Performance measure measure
#runs yielding the optimal solution total number of runs
max
1 g u g = −
min max min
best
f PD f =
21
Effective Black-Box Testing with Genetic Algorithms
attributes and three logical operators: AND, OR, and NOT.
– This expression is generated randomly using an external application – Number of combinatorial test cases: 2100
by randomly selecting an OR operator and flipping it to the AND one (or vice versa).
– Number of generations = 200. – Population size = 100. – Chromosome length = 100. – Crossover probability = 0.9 (for simple GA and GAVaPS). – Mutation probability = 0.01 – Discrete evaluation function (Error = Yes / No)
22
Effective Black-Box Testing with Genetic Algorithms
differences compared to the others algorithms.
locating the error).
Measure\Algor ithm SimpleGA GAVaPS FAexGA#1 FAexGA#2 FAexGA#3 % Runs with solution 70% 60% 95% 70% 75% % Solutions in the final population 1.27% 81.52% 99.93% 98.86% 87.18% Unuse Factor 65.57% 62.27% 70.66% 68.71% 46.27%
23
Effective Black-Box Testing with Genetic Algorithms
%Solutions
0.2 0.4 0.6 0.8 1 1.2 1 200 Generations % Solutions
SimpleGA GAVaPS FAex#1 FAex#2 FAex#3
24
Effective Black-Box Testing with Genetic Algorithms
– Five nominal and three continuous
– Decision (accept / reject) – Credit limit (continuous)
25
Effective Black-Box Testing with Genetic Algorithms
Average of 10 runs
0.2 0.4 0.6 0.8 1 1 5 9 13 17 21 25 29 33 37 41 45 49 Generation #Solution/Total_Pop FAex#1
26
Effective Black-Box Testing with Genetic Algorithms
effective black-box test cases
appears to be much more efficient for this problem than the two other evaluated algorithms (SimpleGA and GAVaPS):
– FAexGA has a much higher probability to find an error in the tested software – The error would be found much faster, which should result in saving a lot
– In addition, the number of distinct solutions produced by FAexGA is significantly higher, which may be useful for investigation and identification of the error itself by the software programmers
– Experimentation with larger software programs – Development of more sophisticated, possibly continuous evaluation functions
27
Effective Black-Box Testing with Genetic Algorithms
28
Effective Black-Box Testing with Genetic Algorithms
–
Software Testing”. World Scientific, Series in Machine Perception and Artificial Intelligence, Vol. 56, 2004.
–
Network”, T.M. Khoshgoftaar (Ed.), Software Engineering with Computational Intelligence, Kluwer Academic Publishers, pp. 235 – 258, 2003. –
Software Testing", Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2003), pp. 388 - 396, Washington, DC, USA August 24 - 27, 2003 –
Software Testing", International Journal of Software Engineering and Knowledge Engineering (IJSEKE), Special Issue on Data Mining for Software Engineering and Knowledge Engineering, Vol. 14, No. 4, pp. 369-393, August 2004. –
Fuzzy Sets and Systems, Vol. 149, Issue 1, January 2005, 131-147.