Test Generation For Higher-Order Test Generation For Higher-Order - - PowerPoint PPT Presentation

Test Generation For Higher-Order Test Generation For Higher-Order Functions In Dynamic Languages Functions In Dynamic Languages

Marija Selakovic Marija Selakovic, Michael Pradel, Rezwana Karim, Frank Tip , Michael Pradel, Rezwana Karim, Frank Tip OOPSLA 2018 OOPSLA 2018

Automatic Test Generation Automatic Test Generation

1

Testing

Automatic Test Generation Automatic Test Generation

1

Testing Limited resources

Automatic Test Generation Automatic Test Generation

1

Testing

Automatic Test Generation Automatic Test Generation

1

Testing

Generates test cases in an automated way Reduces manual effort Effective in finding programming erros

Automatic Test Generation Automatic Test Generation

1

Testing

Generates test cases in an automated way Reduces manual effort Effective in finding programming erros

Example Example

function includes(str, s) { if (s === '') return true; return str.indexOf(s) != -1; };

Tests: Function:

includes('car','a'); // true includes('car',''); // true includes('car', 'km'); //false function size(arr) { if (!arr) return 0; return arr.length; }; size([1,2,3]); // 3 size(undefined); // 0 size([]); //0

2

Higher-Order Functions (HOF) Higher-Order Functions (HOF)

HOF: takes other functions as inputs that are called back

hof(x,cb){ var r = cb(x); if (r && x.a){ ..... } .... }

Test case: Function:

?

3

Challenges Challenges

Goal: Generate Effective Tests for Higher-Order Functions

hof(x,cb){ var r = cb(x); if (r && x.a){ ..... } .... }

?

4

Challenges Challenges

Goal: Generate Effective Tests for Higher-Order Functions

hof(x,cb){ var r = cb(x); if (r && x.a){ ..... } .... }

?

Challenge 1: callback position?

4

Challenges Challenges

Goal: Generate Effective Tests for Higher-Order Functions

hof(x,cb){ var r = cb(x); if (r && x.a){ ..... } .... } cb(x){ }

?

Challenge 1: callback position? Challenge 2: callback that interacts with program?

4

Challenges Challenges

Goal: Generate Effective Tests for Higher-Order Functions

hof(x,cb){ var r = cb(x); if (r && x.a){ ..... } .... } cb(x){ }

?

Challenge 1: callback position? Challenge 2: callback that interacts with program? Challenge 3: Chaining multiple calls?

4

Challenges Challenges

Goal: Generate Effective Tests for Higher-Order Functions

hof(x,cb){ var r = cb(x); if (r && x.a){ ..... } .... } cb(x){ }

?

Challenge 1: callback position? Challenge 2: callback that interacts with program? Challenge 3: Chaining multiple calls? Challenge 4: Callback related differences?

4

LambdaTester: Framework For LambdaTester: Framework For Testing Higher-Order Functions Testing Higher-Order Functions

Inference of callback position Functions + setup code Generate method call feedback Execute test Store test Tests

5

Inference of Callback Positions Inference of Callback Positions

higherOrderFunction(arg1, arg2,...,argn)

Is callback executed? Positions

6

Inference of Callback Positions Inference of Callback Positions

higherOrderFunction(arg1, arg2,...,argn) callback?

Is callback executed? Positions [1] Yes

6

Inference of Callback Positions Inference of Callback Positions

higherOrderFunction(arg1, arg2,...,argn) callback? callback?

Is callback executed? Positions [1] [1, 2] Yes Yes

6

Inference of Callback Positions Inference of Callback Positions

higherOrderFunction(arg1, arg2,...,argn) callback? callback?

Is callback executed? Positions [1] [1, 2] [1, 2]

callback?

Yes Yes No

6

Background: Feedback-Directed Background: Feedback-Directed Test Generation Test Generation

Add method call

7

Feedback : no crash return values

1

crash or max size? Execute sequence Store test

[1] Carlos Pacheco, Shuvendu K. Lahiri, Thomas Ball, Michael D. Ernst. Feedback-Directed Random Test Generation, ICSE '07

yes no

Background: Feedback-Directed Background: Feedback-Directed Test Generation Test Generation

Test: Setup code + Sequence of method calls Add method call

7

Feedback : no crash return values

1

crash or max size? Execute sequence Store test

[1] Carlos Pacheco, Shuvendu K. Lahiri, Thomas Ball, Michael D. Ernst. Feedback-Directed Random Test Generation, ICSE '07

yes no

Callback Generation Approaches Callback Generation Approaches

function () { }

cb-empty

function (err) { if (err) throw err; }

cb-mined

function () { return 17; }

cb-quick

2

cb-writes

8

[2] Koen Claessen and John Hughes. 2011. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. SIGPLAN Not. 46

function () { }

cb-empty

function (err) { if (err) throw err; }

cb-mined

8

no computation no return value

Callback Generation Approaches Callback Generation Approaches

8

Callback Generation Approaches Callback Generation Approaches

function () { return 17; }

cb-writes returns random values no additional computation

[2] Koen Claessen and John Hughes. 2011. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. SIGPLAN Not. 46

cb-quick

2

Callback Generation Approaches Callback Generation Approaches

function () { }

cb-empty

function (err) { if (err) throw err; }

cb-mined

8

extracted from existing code function expressions passed to methods with the same name

Callback Generation Approaches Callback Generation Approaches

function () { }

cb-empty

function (err) { if (err) throw err; }

cb-mined

function () { return 17; }

cb-quick cb-writes

8

Dynamic Analysis Dynamic Analysis

hof(x,cb){ ... var r = cb(x); if (r && x.a){ ... } ... } cb(x){ }

?

9

Dynamic Analysis Dynamic Analysis

hof(x,cb){ ... var r = cb(x); if (r && x.a){ ... } ... } cb(x){ }

?

Analysis of memory reads after callback executes

9

Dynamic Analysis Dynamic Analysis

hof(x,cb){ ... var r = cb(x); if (r && x.a){ ... } ... } cb(x){ }

?

Analysis of memory reads after callback executes

9

Dynamic Analysis Dynamic Analysis

hof(x,cb){ ... var r = cb(x); if (r && x.a){ ... } ... }

r x.a

cb(x){ }

?

Analysis of memory reads after callback executes

9

Dynamic Analysis Dynamic Analysis

hof(x,cb){ ... var r = cb(x); if (r && x.a){ ... } ... }

r x.a

cb(x){ }

?

Analysis of memory reads after callback executes

cb(x){ x.a = true; return 23; }

9

Application: Differential Application: Differential Testing of Polyfills Testing of Polyfills

Functions + setup code LambdaTester Tests

10

Application: Differential Application: Differential Testing of Polyfills Testing of Polyfills

Polyfill: implementation of an API not supported by older browsers Differential testing: testing the same program on different implementations Functions + setup code LambdaTester Differential Testing

• f Polyfills

Tests Behavioral differences

10

Application: Differential Application: Differential Testing of Polyfills Testing of Polyfills

Polyfill: implementation of an API not supported by older browsers Differential testing: testing the same program on different implementations Functions + setup code LambdaTester Differential Testing

• f Polyfills

Tests Behavioral differences Oracles: Error messages Non-termination Standard output Receiver and return object Callback arguments Number of callback invocations

10

Evaluation Evaluation

Setup: Polyfill.io, es5shim, mozilla and 10 Promise libraries Base approach: unaware of callbacks 1000 generated tests for each polyfill/approach combination Research Questions: Effectiveness? LambdaTester finds differences in 12 libraries Coverage? LambdaTester achieves on average 75% statement coverage Efficiency? Time to generate single test ranges between 0.12 and 12 seconds

11

Effectiveness Effectiveness

2 4 6 8 10 12 b a s e c b

• e

m p t y c b

• q

u i c k c b

• m

i n e d c b

• w

r i t e s Number of libraries

12

In how many libraries LambdaTester finds behavioral differences?

Effectiveness Effectiveness

2 4 6 8 10 12 b a s e c b

• e

m p t y c b

• q

u i c k c b

• m

i n e d c b

• w

r i t e s Number of libraries

cb-writes finds behavioral differences in 12 out of 13 libraries

12

In how many libraries LambdaTester finds behavioral differences?

Behavioral Differences Behavioral Differences

13

100 200 300 400 500 600 700 800 errors stout non-term receiver return

• c. arg
• c. inv

Number of differences cb-empty cb-quick cb-mined cb-writes

What types of behavioral differences LambdaTester finds?

Example: Q library Example: Q library

var Promise = require('q'); var p1 = Promise.resolve(18); var p2 = Promise.reject(17); var r1 = p1.then(function(){ return null; }, null); var r2 = p2.then(function(){ return r1; }); var r3 = r2.catch(function(){ return p2; }); var r4 = r1.then(function(){ return r4; }); //non-termination

14

Example: Q library Example: Q library

var Promise = require('q'); var p1 = Promise.resolve(18); var p2 = Promise.reject(17); var r1 = p1.then(function(){ return null; }, null); var r2 = p2.then(function(){ return r1; }); var r3 = r2.catch(function(){ return p2; }); var r4 = r1.then(function(){ return r4; }); //non-termination

Output: Native: TypeError: Chaining cycle detected Q: No output, non-termination

14

Example: Polyfill.io Example: Polyfill.io

var base = ["w", "I", 126]; base.map(function(a,b,c){ base['length'] = false; return a; });

15

write to location suggested by dynamic analysis

Example: Polyfill.io Example: Polyfill.io

var base = ["w", "I", 126]; base.map(function(a,b,c){ base['length'] = false; return a; });

Output : Native : return object = ["w", undefined, undefined] Polyfill.io : return object = ["w"]

15

write to location suggested by dynamic analysis

Coverage Coverage

10 20 30 40 50 60 70 80 b a s e c b

• e

m p t y c b

• q

u i c k c b

• m

i n e d c b

• w

r i t e s Statement coverage(%)

16

How much code is covered by generated tests?

Coverage Coverage

10 20 30 40 50 60 70 80 b a s e c b

• e

m p t y c b

• q

u i c k c b

• m

i n e d c b

• w

r i t e s Statement coverage(%)

cb-writes on average achieves 75% statement coverage

16

How much code is covered by generated tests?

Conclusions Conclusions

https://github.com/sola-da/LambdaTester

LambdaTester: Test generation for higher-order functions Automatic and feedback-directed Generates tests with non-trivial callbacks Several instances of the framework based on callback type Effective in: Finding unknown behavioral differences Increasing code coverage

17

Conclusions Conclusions

https://github.com/sola-da/LambdaTester

LambdaTester: Test generation for higher-order functions Automatic and feedback-directed Generates tests with non-trivial callbacks Several instances of the framework based on callback type Effective in: Finding unknown behavioral differences Increasing code coverage THANK YOU!

17

Setup code Setup code

var p1 = Promise.resolve(18); var p2 = Promise.reject(17); var p3 = Promise.resolve(null);

Promise libraries:

18

Array libraries:

var a1 = [1,2,3,4]; var a2 = new Array(10);