Acceptance Test Optimization Optimization Mohamed Mussa, Ferhat - - PowerPoint PPT Presentation

Acceptance Test Optimization Optimization

Mohamed Mussa, Ferhat Khendek

SAM’2014

Outline Outline

• Background
• Problem Statement
• Overall Approach

– Integration test cases selection – Integration test cases selection – Comparing test models

• Conclusion

2

Background

Test process consists of several phases

Unit Integration Acceptance

• Unit

Testing Integration Testing Acceptance Testing Test Test Model Model Test Test Model Model Test Test Models Models Test Test Model Model Test Test Model Model Test Test Models Models Test Test Model Model

Background - Model Based Testing

Model CODE

Development Process Development Process

• Test Model

CODE

Testing Process Testing Process

Test Test Execution Execution

Background - Model Based Testing

• Different approaches for different test phases

– Unit, integration, acceptance

• Different notations/languages

– Different subsets of the same language – Different subsets of the same language

• Many test models are produced during different phases

– Redundancy – Test generation/planning: no reusability – Test Execution: no optimization

5

• Goals

– Provide a systematic transition between the test phases

• Framework

Background - Model Based Test Framework

Framework

– Strengthen the collaboration between the development and the testing teams – Well know standards & reuse – Improve the test process

• Enable reusability & optimization

Background - Model Based Test Framework

Unit Unit Integration Integration Acceptance Acceptance

Unit test modelcn Integration test modelint(n-1)

• Unit test

modelc1 Integration test modelint1

Optimized acceptance test model Acceptance test model

• Test cases may be exercised several times across the

testing phases

– Integration vs. Acceptance

• Goal: remove redundant acceptance test cases

Reduce test execution time

Problem Statement

– Reduce test execution time

• Obvious solution

– Compare integration test cases and acceptance test cases

• Problem

– Some integration test cases may include stubs for subsequent system components

Problem Statement

system components – Cannot be substituted to acceptance test cases

• Comp1

Comp2 SbSys1 Comp3 Test Test Model Model1

1

SbSys2 Comp4 Test Test Model Model2

2

SbSysn-1 Compn Test Test Model Modeln

n-

• 1

1

System Test Test Model Modeln

n

• Integration test cases selection
• Compare integration test cases to

Overall Approach

• Compare integration test cases to

acceptance test cases

10

• Test cases of last integration round are applied on complete

system

• Compare the behavior of test stubs of each test case to the

behavior of CUTs of test cases of subsequent integration rounds

• No additional information beside the test models

Integration test cases selection

• No additional information beside the test models
• Comp1

Comp2 SbSys1 Comp3 Test Test Model Model1

1

SbSys2 Comp4 Test Test Model Model2

2

SbSysn-1 Compn Test Test Model Modeln

n-

• 1

1

System Test Test Model Modeln

n

Test stubs can be specified explicitly, or

Integration test cases selection

«TestContext» TCk «SUT» SbSysk «SUT» CUTk+1 m1 m2 m3 m4 m5 m6 «TestComponent» stubk

12

specified implicitly

«TestContext» TCk «SUT» SbSysk «SUT» CUTk+1 m1 m2 m3 m4 m5 m6 m5 m6

• Event based comparison
• Not instance based comparison

– Instances are different

Integration test cases selection

• Not event name based

– but message – event types: message, time, miscellaneous

13

Selection condition

– Let

• Tk = {Ik, Ek, Rk} be an integration test case at integration round k,
• Ti = {Ii, Ei, Ri} be an integration test case at integration round i,

Integration test cases selection

i i i i

• i > k

– Tk does not use a test stub for the CUT of Ti if and only if

14

( ) ( ) ( ) ( ) ( ).

. . | , , SUT st

• wner

e e e e e E e E e e e

i k i k i k k i i k i

≠ ∧ = ∨ ≠ ∈ ∈ ⋅ ∀

Comparing integration test cases

Integration test cases selection

«TestContext» TCm «SUT» SbSysm «SUT» CUTm+1

Round k+1 Subsequent rounds

• «TestContext»

TCk «SUT» SbSysk «SUT» CUTk+1 m1 m2 m3 m4 m5 m6 TCm SbSysm CUTm+1 s1 s2 s3 s4 «TestContext» TCn «SUT» SbSysn «SUT» CUTn+1 m1 m3 m4 m6

Test cases, which their stubs do not match with subsequent CUTs, are

Integration test cases selection

match with subsequent CUTs, are compared to acceptance test cases

16

Comparing Test Models Comparing Test Models

• A lot of work has been done

– Compared models are evolved from the same source – Two-Way vs. Three-Way – Look up for differences (Add/Delete/Modify) – Look up for differences (Add/Delete/Modify) – Structure vs. Behavior

• Our case

– Models did not necessary evolve from the same source

17

Comparing Test Models Comparing Test Models

• Comparing MSCs or Sequence Diagrams is not

straightforward

– Several researchers have tackled this issue

• But this is not difficult for test cases

– Finite behaviors – Finite behaviors

18

Comparing Test Models Comparing Test Models

• A test case T is a tuple (I, E, R), where

– I : a set of instances – E : a set of events – R ⊆ (E x E): a partial order reflecting the transitive closure

• f the order relation between events on the same axis and

the sending and reception events of the same message the sending and reception events of the same message

• Test case inclusion

– Tacc = {Ia, Ea, Ra} and Tint = {Ii, Ei, Ri} – Tacc is included in Tint iff

• Ea ⊆ Ei
• Ra ⊆ Ri

19

Comparing Test Models Comparing Test Models

Comparing test cases

«TestContext» TCm «SUT» SbSysm «SUT» CUTm+1

Acceptance test case Integration test cases

• «TestContext»

TCa «SUT» Sys m1 m6 TCm SbSysm CUTm+1 s1 s2 s3 s4 «TestContext» TCn «SUT» SbSysn «SUT» CUTn+1 m1 m3 m4 m6

Conclusion

• We proposed an optimization approach that

reduces the acceptance test suite length

– already done at integration phase

• Implemented and completed the framework
• Implemented and completed the framework
• What kind of systems would benefit ?
• Requires evaluation of the gain

21