TUTORIAL Andrea Arcuri, Simula Research Labs, Norway Gordon Fraser, - - PowerPoint PPT Presentation

▶
tutorial
SMART_READER_LITE
LIVE PREVIEW

TUTORIAL Andrea Arcuri, Simula Research Labs, Norway Gordon Fraser, - - PowerPoint PPT Presentation

Jan 29, 2023 728 likes •1.64k views

Gordon Fraser, University of Sheffield, UK TUTORIAL Andrea Arcuri, Simula Research Labs, Norway Gordon Fraser, University of Sheffield Acknowledgements Andrea Arcuri Andre Mis Jos Campos Daniel Muth Benjamin Friedrich Annibale Panichella

slide-1
SLIDE 1

Gordon Fraser, University of Sheffield, UK Andrea Arcuri, Simula Research Labs, Norway

Gordon Fraser, University of Sheffield

TUTORIAL

slide-2
SLIDE 2

Acknowledgements

Andrea Arcuri José Campos Benjamin Friedrich Florian Gross Juan Pablo Galeotti Alessandra Gorla Mat Hall Fitsum Meshesha Kifitew Merlin Lang Yanchuan Li Eva May Phil McMinn Andre Mis Daniel Muth Annibale Panichella David Paterson Jeremias Roessler Jose Miguel Rojas Kaloyan Rusev Sina Shamshiri Sebastian Steenbuck Andrey Tarasevich Mattia Vivanti Thomas White

slide-3
SLIDE 3

Why should you care?

  • Are you writing Java code?


Learn how to use EvoSuite to help you testing

  • Are you doing research on testing?


Learn how to use EvoSuite for experiments
 Learn how to extend EvoSuite

  • Are you doing research on SBST?


Hear about our experiences in developing an SBST tool

slide-4
SLIDE 4

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-5
SLIDE 5

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-6
SLIDE 6

History

μTest

class Foo { int bar(int x) { return 2 * x; } } class Foo { int bar(int x) { return 2 * x; } } void test() { f = new Foo(); y = f.bar(10); }

+

assert(y==20

Mutation-based Generation of Tests and Oracles, ISSTA 2010

slide-7
SLIDE 7

LocalDate date = new LocalDate(2010, 7, 15); assertEquals(date.getDayOfMonth(), ...); date.plusYears(1); assertEquals(date.getYear(), 2011); assertEquals(date.size(), 3); assertEquals(date.getValue(YEAR), 2010); assertEquals(date.getValue(MONTH_OF_YEAR), 7); assertEquals(date.getValue(DAY_OF_MONTH), 15); assertEquals(date.getLocalMillis(), ...); assertEquals(date, date); assertEquals(date.compareTo(date), 0); assertEquals(date.getYearOfCentury(), ...); assertEquals(date.getYear(), 2010); assertEquals(date.getWeekyear(), ...); assertEquals(date.getMonthOfYear(), 7); assertEquals(date.getWeekOfWeekyear(), ...); assertEquals(date.getDayOfWeek(), ...);

slide-8
SLIDE 8

LocalDate date = new LocalDate(2010, 7, 15); assertEquals(date.getDayOfMonth(), ...); date.plusYears(1); assertEquals(date.getYear(), 2011); assertEquals(date.size(), 3); assertEquals(date.getValue(YEAR), 2010); assertEquals(date.getValue(MONTH_OF_YEAR), 7); assertEquals(date.getValue(DAY_OF_MONTH), 15); assertEquals(date.getLocalMillis(), ...); assertEquals(date, date); assertEquals(date.compareTo(date), 0); assertEquals(date.getYearOfCentury(), ...); assertEquals(date.getYear(), 2010); assertEquals(date.getWeekyear(), ...); assertEquals(date.getMonthOfYear(), 7); assertEquals(date.getWeekOfWeekyear(), ...); assertEquals(date.getDayOfWeek(), ...); assertEquals(date.size(), 3);

slide-9
SLIDE 9

assertEquals(date.getYear(), 2011); assertEquals(date.getDayOfMonth(), ...); assertEquals(date.getDayOfMonth(), ...); date.plusYears(1); assertEquals(date.size(), 3); assertEquals(date.getValue(YEAR), 2011); assertEquals(date.getValue(MONTH_OF_YEAR), 7); assertEquals(date.getValue(DAY_OF_MONTH), 15); assertEquals(date.getLocalMillis(), ...); assertEquals(date, date); assertEquals(date.compareTo(date), 0); assertEquals(date.getYearOfEra(), ...); assertEquals(date.getYearOfCentury(), ...); assertEquals(date.getWeekyear(), ...); assertEquals(date.getMonthOfYear(), 7); assertEquals(date.getWeekOfWeekyear(), ...); assertEquals(date.getDayOfWeek(), ...);

slide-10
SLIDE 10

assertEquals(date.getYear(), 2011); assertEquals(date.getDayOfMonth(), ...); assertEquals(date.getDayOfMonth(), ...); date.plusYears(1); assertEquals(date.size(), 3); assertEquals(date.getValue(YEAR), 2011); assertEquals(date.getValue(MONTH_OF_YEAR), 7); assertEquals(date.getValue(DAY_OF_MONTH), 15); assertEquals(date.getLocalMillis(), ...); assertEquals(date, date); assertEquals(date.compareTo(date), 0); assertEquals(date.getYearOfEra(), ...); assertEquals(date.getYearOfCentury(), ...); assertEquals(date.getWeekyear(), ...); assertEquals(date.getMonthOfYear(), 7); assertEquals(date.getWeekOfWeekyear(), ...); assertEquals(date.getDayOfWeek(), ...);

slide-11
SLIDE 11

LocalDate date = new LocalDate(2010, 7, 15); date.plusYears(1); assertEquals(date.getYear(), 2011); assertEquals(date.getValue(YEAR), 2011);

slide-12
SLIDE 12

LocalDate date = new LocalDate(2010, 7, 15); date.plusYears(1); assertEquals(date.getYear(), 2011);

slide-13
SLIDE 13

History

April 9, 2010

“Evolutionary Generation of Whole Test Suites,” 
 11th International Conference on Software Quality (QSIC 2011)

slide-14
SLIDE 14

Stats

  • 6,865 commits
  • 229,889 LOC
  • 2,420 tests
slide-15
SLIDE 15

@Test public void test() { } int x = 2; int y = 2; int result = x + y; assertEquals(4, result);

slide-16
SLIDE 16

@Test public void test() { }

DateTime var3 = var1.toDateTime(var2); DateTime var4 = var3.minus(var0); TimeOfDay var2 = new TimeOfDay(); YearMonthDay var1 = new YearMonthDay(var0); int var0 = 10 DateTime var5 = var4.plusSeconds(var0);

slide-17
SLIDE 17 DateTime var3 = var1.toDateTime(var2); DateTime var4 = var3.minus(var0); TimeOfDay var2 = new TimeOfDay(); YearMonthDay var1 = new YearMonthDay(var0); int var0 = 10 DateTime var5 = var4.plusSeconds(var0);
slide-18
SLIDE 18
slide-19
SLIDE 19

Crossover

slide-20
SLIDE 20

Mutation

slide-21
SLIDE 21

Mutation

slide-22
SLIDE 22

Fitness

public int gcd(int x, int y) { int tmp; while (y != 0) { tmp = x % y; x = y; y = tmp; } return x; }

slide-23
SLIDE 23

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-24
SLIDE 24

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-25
SLIDE 25

Getting EvoSuite

http://www.evosuite.org/downloads

  • Jar release - for command line usage
  • Maven plugin
  • IntelliJ plugin
  • Eclipse plugin
  • Jenkins plugin
slide-26
SLIDE 26

Testing a Class

  • Demo - command line
  • Main options:

  • projectCP

  • class

  • criterion
slide-27
SLIDE 27

Properties

  • -Dproperty=value
  • Search budget (s)

  • Dsearch_budget=60
  • Assertion generation

  • Dassertions=false

  • Dassertion_strategy=all
  • Minimisation (length and values)

  • Dminimize=false
  • Inlining

  • Dinline=false
slide-28
SLIDE 28

EvoSuite Sandbox

  • Demo - Nondeterministic class
  • Runtime library to execute tests
slide-29
SLIDE 29

Testing multiple classes

Demo:

  • Target / prefix
  • Continuous
  • Maven
  • Jenkins
  • IntelliJ
slide-30
SLIDE 30

Experiment Exercise

  • EvoSuite by default uses a combination of

different coverage criteria.

  • RQ1: Does the combination lead to larger

test suites than just using branch coverage?

  • RQ2: Does the combination lead to better

test suites than just using branch coverage?

slide-31
SLIDE 31

Setup

  • Download and unzip:


http://evosuite.org/files/tutorial/Tutorial_Experiments.zip

  • Maven project, but zip includes compiled code and

dependencies
 (mvn compile dependency:copy-dependencies)

  • Where is EvoSuite?


export EVOSUITE=“java -jar /path/to/evosuite.jar”

slide-32
SLIDE 32

Treatments

  • Default criteria combination:


$EVOSUITE -class tutorial.Person

  • Only branch coverage:


$EVOSUITE -class tutorial.Person -criterion branch

  • Resulting data:


evosuite-report/statistics.csv

slide-33
SLIDE 33

Generating Data

http://evosuite.org/files/tutorial/Tutorial_Experiments.zip

  • Commons Options:

  • prefix tutorial -Dsearch_budget=20 

  • Doutput_variables=configuration_id,TARGET_CLASS,


Size,Length,MutationScore

  • Treatment 1:


$EVOSUITE -Dconfiguration_id=Default <common options>

  • Treatment 2:


$EVOSUITE -Dconfiguration_id=Branch -criterion branch <common options>

slide-34
SLIDE 34

Cluster Experiments

  • Demo - Sun Grid Engine
slide-35
SLIDE 35

Analysing Data

  • R


analysis_scripts/analyse.R

  • Python


analysis_scripts/analysis.py
 analysis_scripts/plots.py

Libs:


easy_install numpy
 easy_install matplotlib
 easy_install pandas
 easy_install scipy

slide-36
SLIDE 36

Experiment Results

  • RQ1: Does the combination lead to larger

test suites than just using branch coverage?


  • RQ2: Does the combination lead to better

test suites than just using branch coverage?


slide-37
SLIDE 37

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-38
SLIDE 38

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-39
SLIDE 39

Building EvoSuite

  • Git repository:


git clone https://github.com/EvoSuite/evosuite.git

  • Maven


mvn package
 (mvn -DskipTests package)

  • Where is EvoSuite now?


master/target/evosuite-master-1.0.4-SNAPSHOT.jar

  • Why is the jar file so huge?
slide-40
SLIDE 40

Module Structure

  • master
  • client
  • runtime
  • standalone-runtime
  • plugins
  • generated
  • shaded
slide-41
SLIDE 41

Testing EvoSuite

public class NullStringSystemTest extends SystemTestBase { @Test public void testNullString() { EvoSuite evosuite = new EvoSuite(); String targetClass = NullString.class.getCanonicalName(); Properties.TARGET_CLASS = targetClass; String[] command = new String[] { "-generateSuite", "-class", targetClass }; Object result = evosuite.parseCommandLine(command); GeneticAlgorithm<?> ga = getGAFromResult(result); TestSuiteChromosome best = (TestSuiteChromosome) ga.getBestIndividual(); System.out.println("EvolvedTestSuite:\n" + best); int goals = TestGenerationStrategy.getFitnessFactories().
 get(0).getCoverageGoals().size(); // assuming single fitness function Assert.assertEquals("Wrong number of goals: ", 3, goals); Assert.assertEquals("Non-optimal coverage: ", 1d, best.getCoverage(), 0.001); } }

Example Test:

slide-42
SLIDE 42

Extending EvoSuite

  • (Artificial) Example: Middle point crossover
slide-43
SLIDE 43

Extending EvoSuite

  • (Artificial) Example: Middle point crossover
slide-44
SLIDE 44

Extending EvoSuite

  • (Artificial) Example: Middle point crossover

package org.evosuite.ga.operators.crossover; import org.evosuite.ga.Chromosome; import org.evosuite.ga.ConstructionFailedException; public class MiddleCrossOver extends CrossOverFunction { @Override public void crossOver(Chromosome parent1, Chromosome parent2) throws ConstructionFailedException { // TODO } }

slide-45
SLIDE 45

Extending EvoSuite

  • (Artificial) Example: Middle point crossover

@Test public void testSinglePointCrossOver() throws ConstructionFailedException { DummyChromosome parent1 = new DummyChromosome(1, 2, 3, 4, 5, 6); DummyChromosome parent2 = new DummyChromosome(7, 8, 9, 10); MiddleCrossOver xover = new MiddleCrossOver(); xover.crossOver(parent1, parent2); assertEquals(Arrays.asList(1, 2, 3, 9, 10), parent1.getGenes()); assertEquals(Arrays.asList(7, 8, 4, 5, 6), parent2.getGenes()); }

slide-46
SLIDE 46

Extending EvoSuite

  • (Artificial) Example: Middle point crossover

@Override
 public void crossOver(Chromosome parent1, Chromosome parent2) 
 throws ConstructionFailedException {
 int middle1 = parent1.size()/2;
 int middle2 = parent2.size()/2;
 
 Chromosome t1 = parent1.clone();
 Chromosome t2 = parent2.clone();
 
 parent1.crossOver(t2, middle1, middle2);
 parent2.crossOver(t1, middle2, middle1);
 }

slide-47
SLIDE 47

Extending EvoSuite

  • (Artificial) Example: Middle point crossover

public enum CrossoverFunction { SINGLEPOINTRELATIVE, SINGLEPOINTFIXED, SINGLEPOINT, COVERAGE, MIDDLE }

slide-48
SLIDE 48

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-49
SLIDE 49

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-50
SLIDE 50

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to test my own Java code?
  • Yes, of course
slide-51
SLIDE 51

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to implement my ideas on unit test

generation?

  • Yes, of course
slide-52
SLIDE 52

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to study developer behaviour?
  • Yes, of course
slide-53
SLIDE 53

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to generate unit tests for my experiment
  • n X?
  • Yes, of course
slide-54
SLIDE 54

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to build a unit test generator for a

different programming language?

  • EvoSuite is 90% JVM handling code
  • Would need to reimplement

representation, search operators, fitness functions, test execution, …

slide-55
SLIDE 55

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to create an Android testing tool?
  • Android uses Java / Dalvik bytecode
  • Can also compile to Java bytecode
  • How to handle Android dependencies?
slide-56
SLIDE 56

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to create a GUI testing tool?
  • If you want to test Java/Swing

applications…

  • But whole test suite optimisation may not

be the right choice

slide-57
SLIDE 57

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to create a web app testing tool?
  • If it’s based on JEE, unit testing already

works (JEE support is not complete yet)

  • System testing…see GUI testing
slide-58
SLIDE 58

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to implement a non-test generation SBSE

tool?

  • GA implementation is quite test specific
  • Using for other purposes would need

refactoring


But then, is it better than using existing generic GA libraries?

  • If the tool uses Java, why not?
slide-59
SLIDE 59

When to use and not to use EvoSuite

  • Should I use EvoSuite…
  • …to implement a tool that requires tests?
  • E.g., specification mining, fault localisation,

program repair, GI, …

  • Sure, integrating EvoSuite should be easy
slide-60
SLIDE 60

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-61
SLIDE 61

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-62
SLIDE 62
  • 1. Java

…is a weird language and never ceases to surprise me My personal enemy: Java Generics Bytecode over sourcecode - yes!

slide-63
SLIDE 63
  • 2. Corner Cases

The more corner cases you cover ...the more can go wrong ...the more new corner cases you will find ...the slower EvoSuite becomes

slide-64
SLIDE 64
  • 2. Corner Cases
  • Constant Seeding: +5%
  • Virtual FS: +1.4%
  • Mocking +4.7%
  • JEE support: +3%
  • DSE: +1.2%
slide-65
SLIDE 65
  • 3. Developers

…some really care only about coverage …others don’t care about coverage:


"I wouldn’t normally in real life be aiming for 100% coverage. I’d probably end up with fewer tests without this tool but I couldn’t tell you if they would be all the right tests.”

…do not want their tests to be generated …hate ugly tests …don’t like waiting Talk to them!

slide-66
SLIDE 66
  • 3. Developers

public class Example { private Example() {} // … }

slide-67
SLIDE 67
  • 4. Testing

Testing randomised algorithms is difficult Make the implementation deterministic Always use LinkedHashSet over HashSet, LinkedHashMap over HashMap Java reflection is not deterministic Avoid static state (e.g. singletons)

slide-68
SLIDE 68
  • 4. Testing

EvoSuite uses one central random number generator Any change will affect something at a completely different part of the program Change seeds frequently during testing to find flaky tests

slide-69
SLIDE 69
  • 5. Documentation

I don’t comment my code Students struggle I spend more time explaining things than it would take me to implement them

slide-70
SLIDE 70
  • 6. Tool Comparisons

Reviewers want to see them I don’t like doing them It’s impossible to make them fair Contact tool authors Report bugs Make your own tools usable

slide-71
SLIDE 71
  • 7. Open Source

“The source code will be released under an open source library (most likely GPL2) at a later point, as soon as a number of refactorings are completed.” — FSE’11 tool paper appendix Public GitHub repo: 2015 It will never be clean enough, just release it!

slide-72
SLIDE 72
  • 8. Licensing

License matters Google will not touch GPL BSD, MIT - do you want others to become rich with your idea? Gnu Lesser Public License, Apache

slide-73
SLIDE 73
  • 9. Tool Papers

The first one will be cited The rest no one will cite It shouldn’t be this way

slide-74
SLIDE 74
  • 10. Robustness

Creating a robust tool… …is a huge effort …it will never be finished EvoSuite is a black hole swallowing all my time!

slide-75
SLIDE 75

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-76
SLIDE 76

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-77
SLIDE 77

Stuff we are working on...

  • Increasing coverage…
  • Readability optimisation
  • Better environment handling
  • Mocking and private reflection
  • Finding out how developers benefit most

from using test generation

  • User studies, replications
slide-78
SLIDE 78

Method Names

@Test(timeout = 4000) public void testFooReturningFalse() throws Throwable { StringExample stringExample0 = new StringExample(); boolean boolean0 = stringExample0.foo(""); assertFalse(boolean0); } @Test(timeout = 4000) public void test3() throws Throwable { StringExample stringExample0 = new StringExample(); boolean boolean0 = stringExample0.foo(""); assertFalse(boolean0); }

slide-79
SLIDE 79

Variable Names

@Test(timeout = 4000) public void testFooReturningFalse() throws Throwable { StringExample stringExample0 = new StringExample(); boolean boolean0 = stringExample0.foo(""); assertFalse(boolean0); } @Test(timeout = 4000) public void testFooReturningFalse() throws Throwable { StringExample invokesFoo = new StringExample(); boolean resultFromFoo = invokesFoo.foo(""); assertFalse(resultFromFoo); }

slide-80
SLIDE 80

Variable Names

public class Foo { public void foo() { StringExample sx = new StringExample(); boolean bar = sx.foo(""); } } @Test(timeout = 4000) public void testFooReturningFalse() throws Throwable { StringExample sx = new StringExample(); boolean bar = sx.foo(""); assertFalse(bar); }

slide-81
SLIDE 81

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-82
SLIDE 82

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-83
SLIDE 83
  • 1. SBST is Slow
  • Fitness evaluation means executing tests
  • Executing tests is slow
  • How to reduce the number of fitness

evaluations?

  • How to improve search operators?
  • Can we use ML to predict test execution

results?

slide-84
SLIDE 84
  • 2. OO Guidance
  • Object oriented code has a terrible search

landscape

  • Complex dependency objects are a problem
  • Include dependency objects in fitness

functions?

  • Better testability transformations?
  • Better fitness functions?
slide-85
SLIDE 85
  • 3. New Features
  • Integration testing
  • Concurrent code
  • GUI handling code
  • Database dependent code
  • Prioritising tests
slide-86
SLIDE 86
  • 4. SBST Usability
  • Assertion/contract testing code?
  • Coverage isn’t a great objective
  • Usability as optimisation goal
  • Study developers using SBST tools
slide-87
SLIDE 87

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-88
SLIDE 88

Outline

  • History
  • Using EvoSuite
  • Extending EvoSuite
  • When to use and not to use EvoSuite
  • Lessons learned building an SBST tool
  • Things we are working on
  • Ideas for future work
slide-89
SLIDE 89

Online Tutorials

  • Using EvoSuite on the command line:


http://www.evosuite.org/documentation/tutorial-part-1/

  • Using EvoSuite with Maven:


http://www.evosuite.org/documentation/tutorial-part-2/

  • Running experiments with EvoSuite:


http://www.evosuite.org/documentation/tutorial-part-3/

  • Extending EvoSuite:


http://www.evosuite.org/documentation/tutorial-part-4/