Strategic Automated Software Testing in the Absence of - - PowerPoint PPT Presentation

▶
strategic automated software testing in the absence of
SMART_READER_LITE
LIVE PREVIEW

Strategic Automated Software Testing in the Absence of - - PowerPoint PPT Presentation

Dec 23, 2023 280 likes •719 views

Strategic Automated Software Testing in the Absence of Specifications Tao Xie Dept. of Computer Science & Engineering University of Washington Parasoft Co. Nov. 2004 1 Motivation How do we generate useful tests automatically?

slide-1
SLIDE 1

1

Strategic Automated Software Testing in the Absence of Specifications

  • Dept. of Computer Science & Engineering

University of Washington

Parasoft Co. Nov. 2004

Tao Xie

slide-2
SLIDE 2

2

Motivation

  • How do we generate “useful” tests automatically?
  • With specifications, we can partition input space into subdomains

and generate samples from these subdomains [Myers 79]

  • Korat [Boyapati et al. 02]: repOk (partitioning input space into valid and

invalid subdomains)

  • AsmLT/SpecExplorer [MSR FSE]: abstract state machine
  • How do we know the generated tests run incorrectly in the

absence of uncaught exceptions?

  • With specifications, we know a fault is exposed when a

postcondition is violated by a precondition-satisfying input.

  • We know that specifications are often not written in practice
slide-3
SLIDE 3

3

Our Strategic Approaches

  • How do we generate “useful” tests automatically?
  • Detect and avoid redundant tests during/after test generation

[Xie, Marinov, and Notkin ASE 04]

  • Based on inferred equivalence properties among object states
  • Detected redundant tests do not improve reliability

– no changes in fault detection, structural coverage, confidence

  • How do we know the program runs incorrectly in the absence
  • f uncaught exceptions?
  • It is infeasible to inspect the execution of each single test
  • Select the most “valuable” subset of generated tests for inspection

[Xie and Notkin ASE 03]

  • Based on inferred properties from existing (manual) tests
  • Select any test that violates one of these properties (deviation from “normal”)
slide-4
SLIDE 4

4

Overview

  • Motivation
  • Redundant-test detection based on object

equivalence

  • Test selection based on operational violations
  • Conclusions
slide-5
SLIDE 5

5

Example Code

public class IntStack { private int[] store; private int size; public IntStack() { … } public void push(int value) { … } public int pop() { … } public boolean isEmpty() { … } public boolean equals(Object o) { … } } [Henkel&Diwan 03]

slide-6
SLIDE 6

6

Example Generated Tests

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

slide-7
SLIDE 7

7

Same inputs ⇒ Same behavior

Method Execution

  • bject state @entry
  • bject state @exit

Method arguments Method return Input = + Output = + Testing a method with the same inputs is unnecessary Assumption: deterministic method

We developed five techniques for representing and comparing object states

slide-8
SLIDE 8

8

Redundant Tests Defined

  • Equivalent method executions
  • the same method names, signatures, and input

(equivalent object states @entry and arguments)

  • Redundant test:
  • Each test produces a set of method executions
  • Testj is redundant for a test suite (Test1 ... Testi)
  • if the method executions produced by Testj is a subset of the

method executions produced by Test1 ... Testi Test 1 … Test i

methodexec 1

Redundant Test j

methodexec 1 subset

slide-9
SLIDE 9

9

Comparison with Traditional Definition

  • Traditionally redundancy in tests was largely

based on structural coverage

  • A test was redundant with respect to a set of other

tests if it added no additional structural coverage (no statements, no edges, no paths, no def-use edges, etc.)

  • Unlike our new definition, this structural-

coverage-based definition is not safe.

  • A redundant test (in the traditional definition) can

expose new faults

slide-10
SLIDE 10

10

Five State-Representation Techniques

  • Method-sequence representations
  • WholeSeq
  • The entire sequence
  • ModifyingSeq
  • Ignore methods that don’t modify the state
  • Concrete-state representations
  • WholeState
  • The full concrete state
  • MonitorEquals
  • Relevant parts of the concrete state
  • PairwiseEquals
  • equals() method used to compare pairs of states
slide-11
SLIDE 11

11

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

slide-12
SLIDE 12

12

<init>( ).state

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

slide-13
SLIDE 13

13

<init>( ).state

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

isEmpty( ).state

slide-14
SLIDE 14

14

<init>( ).state

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

isEmpty( ).state push( , 3).state s1.push 2

slide-15
SLIDE 15

15

<init>( ).state

WholeSeq Representation

Notation: methodName(entryState, methodArgs).state [Henkel&Diwan 03] Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Method sequences that create objects

isEmpty( ).state push( , 3).state s1.push 2 s3.push push(<init>( ).state, 3).state 2

slide-16
SLIDE 16

16

s1.push s3.push push(<init>( ).state, 3).state push(isEmpty(<init>( ).state).state, 3).state

ModifyingSeq Representation

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

State-modifying method sequences that create objects

2 2

slide-17
SLIDE 17

17

WholeState Representation

s1.push s2.push

store.length = 3 store[0] = 3 store[1] = 2 store[2] = 0 size = 1

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5);

store.length = 3 store[0] = 3 store[1] = 0 store[2] = 0 size = 1 5 5

The entire concrete state reachable from the object

Comparison by isomorphism

slide-18
SLIDE 18

18

MonitorEquals Representation

s1.push s2.push store.length = 3 store[0] = 3 store[1] = 2 store[2] = 0 size = 1 Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5); store.length = 3 store[0] = 3 store[1] = 0 store[2] = 0 size = 1

5 5

The relevant part of the concrete state defined by equals (invoking

  • bj.equals(obj) and monitor field accesses)

Comparison by isomorphism

slide-19
SLIDE 19

19

PairwiseEquals Representation

s1.push s2.push Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5);

5 5

The results of equals invoked to compare pairs of states

s1.equals(s2) == true

slide-20
SLIDE 20

20

Redundant-Test Detection

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

slide-21
SLIDE 21

21

Redundant-Test Detection

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Using last four techniques: ModifyingSeq, WholeState, MonitorEquals, PairwiseEquals

slide-22
SLIDE 22

22

Redundant-Test Detection

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Using last four techniques: ModifyingSeq, WholeState, MonitorEquals, PairwiseEquals

slide-23
SLIDE 23

23

Redundant-Test Detection

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Using last four techniques: ModifyingSeq, WholeState, MonitorEquals, PairwiseEquals

slide-24
SLIDE 24

24

Redundant-Test Detection

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

Using last four techniques: ModifyingSeq, WholeState, MonitorEquals, PairwiseEquals Test 3 is redundant w.r.t Test 1

slide-25
SLIDE 25

25

Detected Redundant Tests

T3, T2 MonitorEquals T3 WholeState T3, T2 PairwiseEquals T3 ModifyingSeq WholeSeq

detected redundant tests w.r.t. T1

technique

Test 1 (T1): IntStack s1 = new IntStack(); s1.isEmpty(); s1.push(3); s1.push(2); s1.pop(); s1.push(5); Test 2 (T2): IntStack s2 = new IntStack(); s2.push(3); s2.push(5); Test 3 (T3): IntStack s3 = new IntStack(); s3.push(3); s3.push(2); s3.pop();

slide-26
SLIDE 26

26

Experiment: Evaluated Test Generation Tools

  • ParaSoft Jtest 4.5 (both black and white box testing)
  • A commercial Java testing tool
  • Generates tests with method-call lengths up to three
  • JCrasher 0.2.7 (robustness testing)
  • An academic Java testing tool
  • Generates tests with method-call lengths of one
  • Use them to generate tests for 11 subjects from a

variety of sources

  • Most are complex data structures
slide-27
SLIDE 27

27

Answered Two Questions

  • How much do we benefit after applying Rostra on

tests generated by Jtest and JCrasher?

  • The last three techniques detect around
  • 90% redundant tests for Jtest-generated tests
  • 50% on half subjects for JCrasher-generated tests.
  • Detected redundancy in increasing order for five

techniques

  • Does redundant-test removal decrease test suite

quality?

  • The first three techniques preserve both branch cov and

mutation killing capability

  • Two equals-based techniques have very small loss
slide-28
SLIDE 28

28

Redundancy among Jtest-generated Tests

  • The last three techniques detect around 90% redundant tests
  • Detected redundancy in increasing order for five techniques

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% IntStack UBStack ShoppingCart BankAccount BinSearchTree BinomialHeap DisjSet FibonacciHeap HashMap LinkedList TreeMap WholeSeq ModifyingSeq WholeState MonitorEquals PairwiseEquals

slide-29
SLIDE 29

29

Overview

  • Motivation
  • Redundant-test detection based on object

equivalence

  • Test selection based on operational violations
  • Conclusions
slide-30
SLIDE 30

30

Operational Abstraction Generation

[Ernst et al. 01]

  • Goal: determine properties true at runtime

(e.g. in the form of Design by Contract)

  • Tool: Daikon (dynamic invariant detector)
  • Approach
  • 1. Run test suites on a program
  • 2. Observe computed values
  • 3. Generalize

http://pag.lcs.mit.edu/daikon

slide-31
SLIDE 31

31

Specification-Based Testing

  • Goal: generate test inputs and test oracles from

specifications

  • Tool: ParaSoft Jtest (both black and white box testing)
  • Approach:
  • 1. Annotate Design by Contract (DbC) [Meyer 97]
  • Preconditions/Postconditions/Class invariants
  • 2. Generate test inputs that
  • Satisfy preconditions
  • 3. Check if test executions
  • Satisfy postconditions/invariants

Jtest

slide-32
SLIDE 32

32

Basic Technique

Insert as

DbC comments Annotated program

Run

Data trace

Run & Check

Violating tests Automatically generated test inputs Violated OA Selected tests

Select Detect invariants

All OA OA: Operational Abstractions The existing test suite (manual tests) Program

slide-33
SLIDE 33

33

Precondition Removal Technique

@Pre @Post @Inv The existing test suite

Run

Data trace

Detect invariants Insert as

DbC comments program

  • Overconstrained preconditions may leave

(important) legal inputs unexercised

Program Annotated @Pre

  • Solution: precondition removal technique
slide-34
SLIDE 34

34

Motivating Example [Stotts et al. 02]

public class uniqueBoundedStack { private int[] elems; private int numberOfElements; private int max; public uniqueBoundedStack() { numberOfElements = 0; max = 2; elems = new int[max]; } public int getNumberOfElements() { return numberOfElements; } …… };

A manual test suite (15 tests)

slide-35
SLIDE 35

35

Operational Violation Example

public int top(){ if (numberOfElements < 1) { System.out.println("Empty Stack"); return -1; } else { return elems[numberOfElements-1]; } }

  • Precondition Removal Technique

@pre { for (int i = 0 ; i <= this.elems.length-1; i++) $assert ((this.elems[i] >= 0)); }

@post: [($result == -1) (this.numberOfElements == 0)]

Daikon generates from manual test executions:

uniqueBoundedStack THIS = new uniqueBoundedStack (); THIS.push (-1); int RETVAL = THIS.top ();

Jtest generates a violating test input:

slide-36
SLIDE 36

36

Iterations

The existing test suite

Run

Data trace

Detect invariants Insert as DbC comments Run & Check

Violating tests Annotated program

Automatically generated test inputs

Violated OA

Select

OA Selected tests

  • Iterates until
  • No operational violations
  • User-specified max number of iteration
  • The existing tests augmented by selected tests are

run to generate operational abstractions

Program

slide-37
SLIDE 37

37

Experiment: Subject Programs Studied

  • 12 programs from assignments and texts

(standard data structures)

  • Accompanying manual test suites
  • ~94% branch coverage
slide-38
SLIDE 38

38

Answered Questions

  • Is the number of tests selected by our approach

small enough?

  • if yes, affordable inspection effort
  • Range(0…25) Median(3)
  • Do the selected tests by our approach have a high

probability of exposing abnormal behavior?

  • if yes, select a good subset of generated tests
  • Iteration 1: 20% (Basic) 68% (Pre_Removal)
  • Iteration 2: 0% (Basic) 17% (Pre_Removal)
slide-39
SLIDE 39

39

More Strategic Approaches-I

  • How do we generate “useful” tests automatically?
  • Exhaustively exercise N arguments up to N method call

length N

  • Breadth-first search of concrete-object state space:

(limit: N = 6) [UW-CSE-04-01-05]

  • Breadth-first search of symbolic-object state space:

(limit: N = 8) using symbolic execution to build up symbolic states [UW-CSE-04-10-02]

  • Longer method call length
  • Higher branch coverage
  • Generate representative arguments automatically
slide-40
SLIDE 40

40

More Strategic Approaches - II

  • How do we know the program runs incorrectly in

the absence of uncaught exceptions?

  • Test selection: infer universal and common properties

and identify common and special tests [OOPSLA Companion 04, UW-CSE-04-08-03]

  • Test abstraction: recover succinct object-state-transition

information for inspection [ICFEM 04, SAVCBS 04]

  • Regression testing: detect behavior deviation of two

versions by comparing value spectra (defined based on program states) [ICSM 04]

slide-41
SLIDE 41

41

Conclusions

  • Specifications can help automated software testing
  • However, specifications are often not written in

practice

  • Developed strategic approaches to enjoy some

benefits of specification-based testing by using inferred program properties

  • Redundant test detection
  • Test generation
  • Test selection
  • Test abstraction
  • Regression testing
slide-42
SLIDE 42

42

Questions?

http://www.cs.washington.edu/homes/taoxie/publications.htm