Killing Bugs in a Black Box with Model-based Mutation Testing - - PowerPoint PPT Presentation

▶
killing bugs in a black box with model based mutation
SMART_READER_LITE
LIVE PREVIEW

Killing Bugs in a Black Box with Model-based Mutation Testing - - PowerPoint PPT Presentation

Mar 24, 2023 369 likes •1.49k views

Institute of Software Technology tugraz Killing Bugs in a Black Box with Model-based Mutation Testing Bernhard K. Aichernig Institute of Software Technology Graz University of Technology, Austria MT CPS Workshop Vienna, 11 Apr 2016 B.K.

slide-1
SLIDE 1

tugraz

Institute of Software Technology

Killing Bugs in a Black Box with Model-based Mutation Testing

Bernhard K. Aichernig

Institute of Software Technology Graz University of Technology, Austria

MT CPS Workshop Vienna, 11 Apr 2016

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 1 / 64

slide-2
SLIDE 2

tugraz

Institute of Software Technology

Acknowledgements

Joint work with

  • J. Auer · H. Brandl · W. Herzner · E. Jöbstl · W. Krenn · R. Korosec ·
  • F. Lorber · D. Nickovic · A. Rosenmann · R. Schlick · B.V. Schmidt ·
  • M. Tappler · S. Tiran

Strong Collaboration: Since 2008 with AIT Since 2011 with AVL

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 2 / 64

slide-3
SLIDE 3

tugraz

Institute of Software Technology

Projects

Past:

◮ CREDO: FP6, MBT of distributed systems ◮ MOGENTES: FP7, MBT of embedded systems, mutation testing,

qualitative reasoning for testing hybrid systems

◮ TRUFAL: national, scalability of test-case generators via symbolic

analysis

◮ MBAT: FP7, integration of methods and tools, MBT +

consistency checking Ongoing:

◮ CRYSTAL: FP7, integration of tools, MBT + requirements

engineering

◮ TRUCONF: national, MBT + non-functional requirements +

systems of systems

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 3 / 64

slide-4
SLIDE 4

tugraz

Institute of Software Technology

Agenda

◮ Model-based Mutation Testing ◮ Real-Time Systems ◮ Hybrid Systems ◮ Discrete Systems

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 4 / 64

slide-5
SLIDE 5

tugraz

Institute of Software Technology

Mutation Testing I

Step 1: Create mutants

Mutation Process Source Code Mutant Mutation Operator

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 5 / 64

slide-6
SLIDE 6

tugraz

Institute of Software Technology

Mutation Testing II

Step 2: Try to kill mutants

A test case kills a mutant if its run shows different behaviour.

Quality of tests: How many mutants survived? [Lipton71, Hamlet77, DeMillo et al.78]

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 6 / 64

slide-7
SLIDE 7

tugraz

Institute of Software Technology

Objective

Don’t write test cases, generate them!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 7 / 64

slide-8
SLIDE 8

tugraz

Institute of Software Technology

Objective

Don’t write test cases, generate them!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 7 / 64

slide-9
SLIDE 9

tugraz

Institute of Software Technology

Timed Automata Model of a Car Alarm System

start q1 q2 q3 flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 close?

  • pen?

lock? unlock? unlock? lock? c := 0

  • pen?

close? c := 0 c == 20 armedOn!

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff! unlock?

◮ Car alarm system model ◮ and a mutation representing a

fault

◮ leading to non-conformance

representing an observable failure

◮ resulting in a test case

triggering this fault

◮ and propagating it to a visible

failure

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 8 / 64

slide-10
SLIDE 10

tugraz

Institute of Software Technology

Timed Automata Model of a Car Alarm System

start q1 q2 q3 flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 close?

  • pen?

lock? unlock? unlock? lock? c := 0

  • pen?

close? c := 0 c == 20 armedOn!

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff! unlock? unlock? (mutation)

◮ Car alarm system model ◮ and a mutation representing a

fault

◮ leading to non-conformance

representing an observable failure

◮ resulting in a test case

triggering this fault

◮ and propagating it to a visible

failure

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 8 / 64

slide-11
SLIDE 11

tugraz

Institute of Software Technology

Timed Automata Model of a Car Alarm System

start q1 q2 q3 flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 close?

  • pen?

lock? unlock? unlock? lock? c := 0

  • pen?

close? c := 0 c == 20 armedOn!

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff! unlock? unlock? (mutation) lock? close? c := 0 c == 20 armedOn!

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! e == 30 soundOff! e == 300 soundOff! flashOff! unlock?

◮ Car alarm system model ◮ and a mutation representing a

fault

◮ leading to non-conformance

representing an observable failure

◮ resulting in a test case

triggering this fault

◮ and propagating it to a visible

failure

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 8 / 64

slide-12
SLIDE 12

tugraz

Institute of Software Technology

Timed Automata Model of a Car Alarm System

start q1 q2 q3 flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 close?

  • pen?

lock? unlock? unlock? lock? c := 0

  • pen?

close? c := 0 c == 20 armedOn!

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff! unlock? unlock? (mutation) lock? close? c := 0 c == 20 armedOn!

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! e == 30 soundOff! e == 300 soundOff! flashOff! unlock? close? wait close? f := 0 armedOn!

◮ Car alarm system model ◮ and a mutation representing a

fault

◮ leading to non-conformance

representing an observable failure

◮ resulting in a test case

triggering this fault

◮ and propagating it to a visible

failure

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 8 / 64

slide-13
SLIDE 13

tugraz

Institute of Software Technology

Timed Automata Model of a Car Alarm System

start q1 q2 q3 flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 close?

  • pen?

lock? unlock? unlock? lock? c := 0

  • pen?

close? c := 0 c == 20 armedOn!

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff! unlock? unlock? (mutation) lock? close? c := 0 c == 20 armedOn!

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! e == 30 soundOff! e == 300 soundOff! flashOff! unlock? close? wait close? f := 0 armedOn!

◮ Car alarm system model ◮ and a mutation representing a

fault

◮ leading to non-conformance

representing an observable failure

◮ resulting in a test case

triggering this fault

◮ and propagating it to a visible

failure What is a failure?

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 8 / 64

slide-14
SLIDE 14

tugraz

Institute of Software Technology

Fault-Propagation in Models

Abstract 5-place buffer model: Counter variable n is internal!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 9 / 64

slide-15
SLIDE 15

tugraz

Institute of Software Technology

Fault-Propagation in Models

Let’s inject a fault: How does this fault propagate?

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 10 / 64

slide-16
SLIDE 16

tugraz

Institute of Software Technology

A Good Test Case

... triggers this fault and propagates it to a (visible) failure:

!setEmptyOn, ?Enqueue, !setEmptyOff, ?Enqueue, ?Enqueue, ?Enqueue, ?Enqueue, !setFullOn, ?Dequeue, !setFullOff, ?Enqueue, !setFullOn

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 11 / 64

slide-17
SLIDE 17

tugraz

Institute of Software Technology

Model-Based Testing

Test Case Generator SUT Test Driver

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-18
SLIDE 18

tugraz

Institute of Software Technology

Model-Based Testing

Model Test Case Generator SUT Test Driver

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-19
SLIDE 19

tugraz

Institute of Software Technology

Model-Based Testing

Model Test Case Generator Abstract Test Case SUT Test Driver

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-20
SLIDE 20

tugraz

Institute of Software Technology

Model-Based Testing

Model Test Case Generator Abstract Test Case SUT Test Driver

pass / fail

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-21
SLIDE 21

tugraz

Institute of Software Technology

Model-Based Testing

Model Test Case Generator Abstract Test Case SUT Test Driver

then pass if conforms

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-22
SLIDE 22

tugraz

Institute of Software Technology

Model-Based Testing

Model Test Case Generator Abstract Test Case SUT Test Driver

then pass/fail if ¬conforms

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-23
SLIDE 23

tugraz

Institute of Software Technology

Model-Based Mutation Testing

Model Mutation Tool Test Case Generator: Conformance Checker Abstract Test Case SUT Test Driver

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-24
SLIDE 24

tugraz

Institute of Software Technology

Model-Based Mutation Testing

Model Mutation Tool Model Mutant Test Case Generator: Conformance Checker Abstract Test Case SUT Test Driver

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-25
SLIDE 25

tugraz

Institute of Software Technology

Model-Based Mutation Testing

Model Mutation Tool Model Mutant Test Case Generator: Conformance Checker Abstract Test Case SUT Test Driver

then pass/fail if ¬conforms

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-26
SLIDE 26

tugraz

Institute of Software Technology

Model-Based Mutation Testing

Model Mutation Tool Model Mutant Test Case Generator: Conformance Checker Abstract Test Case SUT Test Driver

then fail if ¬conforms if conforms

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-27
SLIDE 27

tugraz

Institute of Software Technology

Model-Based Mutation Testing

Model Mutation Tool Model Mutant Test Case Generator: Conformance Checker Abstract Test Case SUT Test Driver

then fail if ¬conforms if conforms then ¬ conforms

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 12 / 64

slide-28
SLIDE 28

tugraz

Institute of Software Technology

MoMuT Tools

MoMuT

◮ is a family of tools implementing Model-based Mutation Testing. ◮ is jointly developed and maintained by AIT and TU Graz ◮ supports different modelling styles:

◮ MoMuT::UML (UML state machines) ◮ MoMuT::OOAS (OO Action Systems) ◮ MoMuT::QAS (Qualitative Action Systems) ◮ MoMuT::TA (Timed Automata) ◮ MoMuT::TAS (Timed Action Systems) ◮ MoMuT::REQs (Synchronous Requirement Interfaces)

www.momut.org

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 13 / 64

slide-29
SLIDE 29

tugraz

Institute of Software Technology

Agenda

◮ Model-based Mutation Testing ◮ Real-Time Systems ◮ Hybrid Systems ◮ Discrete Systems

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 14 / 64

slide-30
SLIDE 30

tugraz

Institute of Software Technology

Conformance Relation of Timed Systems

... defines in a testing theory what constitutes a failure. Definition (Timed input-output conformance – tioco [Krichen&Tripakis09]) Given a timed automaton Model and a Mutant with inputs and outputs Mutant tioco Model iff ∀σ ∈ L(Model) : out(Mutant after σ) ⊆ out(Model after σ)

S ... set of all states s0 ... initial state σ ... timed trace of labels ΣO ...

  • utput labels

A after σ = {s ∈ S | s0

σ

− → s} elapse(s) = {t > 0 | s

t

− →}

  • ut(s)

= {a ∈ ΣO | s

a

− →} ∪ elapse(s)

  • ut(S)

=

  • s∈S out(s)

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 15 / 64

slide-31
SLIDE 31

tugraz

Institute of Software Technology

tioco and Language Inclusion

Theorem ([Krichen&Tripakis09]) L(Mutant) ⊆ L(Model) ⇒ Mutant tioco Model Theorem ([Krichen&Tripakis09]) If Model is input-enabled, then Mutant tioco Model ⇒ L(Mutant) ⊆ L(Model)

start ... x! a?

b? a? c < 2 ΣI ∪ ΣO a? c ≥ 2 b?

Demonic completion for deterministic TA For deterministic TA, reduce tioco check to language inclusion check (PSPACE-complete).

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 16 / 64

slide-32
SLIDE 32

tugraz

Institute of Software Technology

k-Bounded Language Inclusion

◮ Construct a formula ϕk

AI ,AS that is satisfiable if L(AI) ⊆ L(AS)

◮ providing a timed trace as witness

ϕk

AI ,AS

≡ k

i=1(di ≥ 0 ∧ 1 ≤ αi ≤ |Σ|) ∧ i ≥ 1 ∧ i ≤ k

∧ (delays and actions) 1 ≤ i ≤ k ∧ (in i steps) initAI (XI, CI) ∧ path1,i−1

AI

(A, D, XI, CI) ∧ (reach in mutant) initAS (XS, CS) ∧ path1,i−1

AS

(A, D, XS, CS) ∧ (reach in model) pathi,i

AI (A, D, XI, CI) ∧ ¬pathi,i AS (A, D, XS, CS)

(failure)

Variable sets: xi ∈ X ... location at step i αi ∈ A ... ith discrete action di ∈ D ... ith time delay {ci, c∗,i} ⊆ C ... clock valuation after ith time and discrete step

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 17 / 64

slide-33
SLIDE 33

tugraz

Institute of Software Technology

Experimental Results I

◮ Bounded language inclusion check for deterministic Uppaal TA ◮ Implemented in Scala calling SMT solver Z3 ◮ Car alarm system characteristics: deterministic,

◮ 5 clock variables, 16 locations, 25 transitions.

◮ 8 mutation operators → 1,320 mutants ◮ Overall runtime: 30 minutes (k = 12)

Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 12 1.4s 1.1s 33s 0.07s Runtime details

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 18 / 64

slide-34
SLIDE 34

tugraz

Institute of Software Technology

Experimental Results I

◮ Bounded language inclusion check for deterministic Uppaal TA ◮ Implemented in Scala calling SMT solver Z3 ◮ Car alarm system characteristics: deterministic,

◮ 5 clock variables, 16 locations, 25 transitions.

◮ 8 mutation operators → 1,320 mutants ◮ Overall runtime: 30 minutes (k = 12)

Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 12 1.4s 1.1s 33s 0.07s Runtime details

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 18 / 64

slide-35
SLIDE 35

tugraz

Institute of Software Technology

Timed Action Systems

1 types{ 2

State = [ ... | Flash | FlashSound | Silent | SwitchOffAlarm | ... ]; }

3

state{

4

loc : State; }

5

clocks [Real]{ c;d;e; f ;g }

6

init {

7

loc := OpenAndUnlocked;}

8

invariant {

9

if loc == Flash then e <= 0;

10

if loc == FlashSound then e <= 30;

11

if loc == Silent then e <= 300;

12

... }

13

actions {

14

!soundOn#1() if loc == Flash && e == 0 then { loc := FlashSound; };

15 16

!soundOff#1() if loc == FlashSound && e == 30 then { loc := Silent ; };

17 18

?unlock#6() resets g if loc == FlashSound && e < 30 then { loc := SwitchOffAlarm; };

19

... }

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 19 / 64

slide-36
SLIDE 36

tugraz

Institute of Software Technology

Symbolic Execution of Timed Action Systems

s0

pc = ↼

  • pc

path conditions . . . blue symbolic (clock) states . . . red

qc = {e → ↼

  • d , . . .}

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

slide-37
SLIDE 37

tugraz

Institute of Software Technology

Symbolic Execution of Timed Action Systems

s0

pc = ↼

  • pc

path conditions . . . blue symbolic (clock) states . . . red

qc = {e → ↼

  • d , . . .}

s1

!soundOn

pc = ↼

  • pc ∧ Flash = Flash

q = {loc → FlashSound, . . .}

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

slide-38
SLIDE 38

tugraz

Institute of Software Technology

Symbolic Execution of Timed Action Systems

s0

pc = ↼

  • pc

path conditions . . . blue symbolic (clock) states . . . red

qc = {e → ↼

  • d , . . .}

s1

!soundOn

pc = ↼

  • pc ∧✭✭✭✭

✭

Flash = Flash q = {loc → FlashSound, . . .}

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

slide-39
SLIDE 39

tugraz

Institute of Software Technology

Symbolic Execution of Timed Action Systems

s0

pc = ↼

  • pc

path conditions . . . blue symbolic (clock) states . . . red

qc = {e → ↼

  • d , . . .}

s1

!soundOn

pc = ↼

  • pc

q = {loc → FlashSound, . . .}

s2

delay(d)

pc = ↼

  • pc ∧ Flash = Flash → ↼
  • d + d ≤ 30 ∧ . . .

qc = {e → ↼

  • d + d, . . .}

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

slide-40
SLIDE 40

tugraz

Institute of Software Technology

Symbolic Execution of Timed Action Systems

s0

pc = ↼

  • pc

path conditions . . . blue symbolic (clock) states . . . red

qc = {e → ↼

  • d , . . .}

s1

!soundOn

pc = ↼

  • pc

q = {loc → FlashSound, . . .}

s2

delay(d)

pc = ↼

  • pc ∧ ↼
  • d + d ≤ 30

qc = {e → ↼

  • d + d, . . .}

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

slide-41
SLIDE 41

tugraz

Institute of Software Technology

Symbolic Execution of Timed Action Systems

s0

pc = ↼

  • pc

path conditions . . . blue symbolic (clock) states . . . red

qc = {e → ↼

  • d , . . .}

s1

!soundOn

pc = ↼

  • pc

q = {loc → FlashSound, . . .}

s2

delay(d)

pc = ↼

  • pc ∧ ↼
  • d + d ≤ 30

qc = {e → ↼

  • d + d, . . .}

s3 s4

?unlock !soundOff delay(d′) delay(d′)

pc = ↼

  • pc ∧ ↼
  • d + d < 30

pc = ↼

  • pc ∧ ↼
  • d + d = 30

qc = {g → 0} q = {loc → SwitchOffAlarm} q = {loc → Silent}

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

slide-42
SLIDE 42

tugraz

Institute of Software Technology

Symbolic Execution of Timed Action Systems

s0

pc = ↼

  • pc

path conditions . . . blue symbolic (clock) states . . . red

qc = {e → ↼

  • d , . . .}

s1

!soundOn

pc = ↼

  • pc

q = {loc → FlashSound, . . .}

s2

delay(d)

pc = ↼

  • pc ∧ ↼
  • d + d ≤ 30

qc = {e → ↼

  • d + d, . . .}

s3 s4

?unlock !soundOff delay(d′) delay(d′)

pc = ↼

  • pc ∧ ↼
  • d + d < 30

pc = ↼

  • pc ∧ ↼
  • d + d = 30

qc = {g → 0} q = {loc → SwitchOffAlarm} q = {loc → Silent}

Provides all symbolic timed traces through model!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 20 / 64

slide-43
SLIDE 43

tugraz

Institute of Software Technology

Conformance Checking via Symbolic Execution

◮ Bounded implicit product graph exploration ◮ Simultaneous symbolic execution of all model traces ◮ Non-conformance checks (stioco) of the form: pcq . . . path condition of symbolic state q

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 21 / 64

slide-44
SLIDE 44

tugraz

Institute of Software Technology

Conformance Checking via Symbolic Execution

◮ Bounded implicit product graph exploration ◮ Simultaneous symbolic execution of all model traces ◮ Non-conformance checks (stioco) of the form:

∃qfail ∈ ModelStates

  • all symbolic states after current trace

pcqfail

state reachable (model) pcq . . . path condition of symbolic state q

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 21 / 64

slide-45
SLIDE 45

tugraz

Institute of Software Technology

Conformance Checking via Symbolic Execution

◮ Bounded implicit product graph exploration ◮ Simultaneous symbolic execution of all model traces ◮ Non-conformance checks (stioco) of the form:

∃qfail ∈ ModelStates

  • all symbolic states after current trace

, ∃λ ∈ Observations : pcqfail

state reachable (model)

∧

  • s ∈ MutantStates

pcs ∧ guardsλ[states]

  • bservation possible (mutant)

∧ ¬  

  • q ∈ ModelStates

pcq ∧ guardsλ[stateq]  

  • bservation not possible (model)

pcq . . . path condition of symbolic state q

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 21 / 64

slide-46
SLIDE 46

tugraz

Institute of Software Technology

Experimental Results II

◮ Symbolic execution tioco check for deterministic Timed Action

Systems

◮ Implemented in Scala calling SMT solver Z3 ◮ Car alarm system characteristics: deterministic,

◮ 5 clock variables, 16 locations, 25 transitions.

◮ 8 mutation operators → 986 mutants ◮ Overall runtime: 27.5 minutes (k = 12)

Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 12 1.4s 1.1s 33s 0.07s 1.7s 0.02s 38.83s ∼ 0s Runtime details

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 22 / 64

slide-47
SLIDE 47

tugraz

Institute of Software Technology

Experimental Results II

◮ Symbolic execution tioco check for deterministic Timed Action

Systems

◮ Implemented in Scala calling SMT solver Z3 ◮ Car alarm system characteristics: deterministic,

◮ 5 clock variables, 16 locations, 25 transitions.

◮ 8 mutation operators → 986 mutants ◮ Overall runtime: 27.5 minutes (k = 12)

Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 12 1.4s 1.1s 33s 0.07s 1.7s 0.02s 38.83s ∼ 0s Runtime details

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 22 / 64

slide-48
SLIDE 48

tugraz

Institute of Software Technology

Experimental Results III

◮ Symbolic tioco checker also for

non-deterministic models

◮ Car Alarm System: silent transition

with non-deterministic delay

◮ Plus underspecification in switching

  • n alarm

◮ 3 equivalent mutants timed out after

10min

start flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 lock? unlock? close? c := 0 close? c == 20 armedOn! lock? c := 0

  • pen?
  • pen?

unlock?

0 < c < 2 ǫ

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! soundOn! flashOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! unlock? close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 23 / 64

slide-49
SLIDE 49

tugraz

Institute of Software Technology

Experimental Results III

◮ Symbolic tioco checker also for

non-deterministic models

◮ Car Alarm System: silent transition

with non-deterministic delay

◮ Plus underspecification in switching

  • n alarm

◮ 3 equivalent mutants timed out after

10min Depth Symbolic Execution Mean Median Max Min 12 0.79s 0.06s 360.84s ∼ 0s

start flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 lock? unlock? close? c := 0 close? c == 20 armedOn! lock? c := 0

  • pen?
  • pen?

unlock?

0 < c < 2 ǫ

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! soundOn! flashOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! unlock? close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 23 / 64

slide-50
SLIDE 50

tugraz

Institute of Software Technology

Experimental Results III

◮ Symbolic tioco checker also for

non-deterministic models

◮ Car Alarm System: silent transition

with non-deterministic delay

◮ Plus underspecification in switching

  • n alarm

◮ 3 equivalent mutants timed out after

10min Depth Symbolic Execution Mean Median Max Min 12 0.79s 0.06s 360.84s ∼ 0s ... and the bounded model checking?

start flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 lock? unlock? close? c := 0 close? c == 20 armedOn! lock? c := 0

  • pen?
  • pen?

unlock?

0 < c < 2 ǫ

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! soundOn! flashOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! unlock? close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 23 / 64

slide-51
SLIDE 51

tugraz

Institute of Software Technology

Bounded Determinisation of Timed Automata

q0 start q1 q4 q2 q3 HEATING IDLE EMPTY GRAINING BREWING

coin {x} beep x = 2 beep 0 < x < 3 ǫ 1 < x < 2 {x} refund x < 4 coffee x = 1 B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 24 / 64

slide-52
SLIDE 52

tugraz

Institute of Software Technology

Bounded Determinisation of Timed Automata

q0 start q1 q4 q2 q3 HEATING IDLE EMPTY GRAINING BREWING

coin {x} beep x = 2 beep 0 < x < 3 ǫ 1 < x < 2 {x} refund x < 4 coffee x = 1

q0 start q1 q4 q2 q3 q5 q6

coin {x1} beep x1 = 2 {x2} beep 0 < x1 < 3 {x2} ǫ 1 < x1 < 2 {x2,0} refund x1 < 4 {x3} coffee x2,0 = 1 {x3}

unfolding

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 24 / 64

slide-53
SLIDE 53

tugraz

Institute of Software Technology

Bounded Determinisation of Timed Automata

q0 start q1 q4 q2 q3 HEATING IDLE EMPTY GRAINING BREWING

coin {x} beep x = 2 beep 0 < x < 3 ǫ 1 < x < 2 {x} refund x < 4 coffee x = 1

q0 start q1 q4 q2 q3 q5 q6

coin {x1} beep x1 = 2 {x2} beep 0 < x1 < 3 {x2} ǫ 1 < x1 < 2 {x2,0} refund x1 < 4 {x3} coffee x2,0 = 1 {x3}

q0 start q1 q4 q3 q5 q6

coin {x1} beep x1 = 2 {x2} beep 0 < x1 < 3 ∧ x1 < 2 {x2} refund x1 < 4 {x3} coffee 2 < x1 < 3 ∧ 1 < x1 {x3}

unfolding ǫ-removal

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 24 / 64

slide-54
SLIDE 54

tugraz

Institute of Software Technology

Bounded Determinisation of Timed Automata

q0 start q1 q4 q2 q3 HEATING IDLE EMPTY GRAINING BREWING

coin {x} beep x = 2 beep 0 < x < 3 ǫ 1 < x < 2 {x} refund x < 4 coffee x = 1

q0 start q1 q4 q2 q3 q5 q6

coin {x1} beep x1 = 2 {x2} beep 0 < x1 < 3 {x2} ǫ 1 < x1 < 2 {x2,0} refund x1 < 4 {x3} coffee x2,0 = 1 {x3}

q0 start q1 q¬acc q5 q6

coin {x1} beep (0 < x1 < 3∧ x1 < 2) ∨ x1 = 2 ∨ 0 < x1 < 3 {x2} refund x1 < 4∧ x1 − x2 = 2 {x3} coffee 2 < x1 < 3∧ 1 < x1∧ 0 < x1 − x2 < 3∧ x1 − x2 < 2 {x3}

q0 start q1 q4 q3 q5 q6

coin {x1} beep x1 = 2 {x2} beep 0 < x1 < 3 ∧ x1 < 2 {x2} refund x1 < 4 {x3} coffee 2 < x1 < 3 ∧ 1 < x1 {x3}

unfolding ǫ-removal determinisation

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 24 / 64

slide-55
SLIDE 55

tugraz

Institute of Software Technology

Experimental Results IV

◮ Bounded determinization

→ 13,545 locations (depth 12) → bounded model check fails

◮ Partial models!

start lock? unlock? close?

  • pen?
  • pen?

close? {c} unlock? lock? {c})

  • pen?

unlock? c = 20 armedOn! 0 < c < 2 ǫ start flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 lock? close? c := 0 c == 20 armedOn! 0 < c < 2 ǫ

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! soundOn! flashOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! unlock? close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff!

Model D. Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min Partial 1 8 9.7s 8.0s 85.1s 0.3s 0.28s 0.04s 16.78s ∼ 0s Partial 2 12 1.6s 1.63s 37.3s 0.08s 0.08s 0.03s 2.28s ∼ 0s Complete 12 x x x x 0.79s 0.06s 360.84s ∼ 0s

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 25 / 64

slide-56
SLIDE 56

tugraz

Institute of Software Technology

Experimental Results IV

◮ Bounded determinization

→ 13,545 locations (depth 12) → bounded model check fails

◮ Partial models!

start lock? unlock? close?

  • pen?
  • pen?

close? {c} unlock? lock? {c})

  • pen?

unlock? c = 20 armedOn! 0 < c < 2 ǫ start flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 lock? close? c := 0 c == 20 armedOn! 0 < c < 2 ǫ

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! soundOn! flashOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! unlock? close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff!

Model D. Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min Partial 1 8 9.7s 8.0s 85.1s 0.3s 0.28s 0.04s 16.78s ∼ 0s Partial 2 12 1.6s 1.63s 37.3s 0.08s 0.08s 0.03s 2.28s ∼ 0s Complete 12 x x x x 0.79s 0.06s 360.84s ∼ 0s

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 25 / 64

slide-57
SLIDE 57

tugraz

Institute of Software Technology

Experimental Results IV

◮ Bounded determinization

→ 13,545 locations (depth 12) → bounded model check fails

◮ Partial models!

start lock? unlock? close?

  • pen?
  • pen?

close? {c} unlock? lock? {c})

  • pen?

unlock? c = 20 armedOn! 0 < c < 2 ǫ start flashOff! c ≤ 20 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 0 e ≤ 30 e ≤ 300 g ≤ 0 g ≤ 0 d ≤ 0 f ≤ 0 e ≤ 300 lock? close? c := 0 c == 20 armedOn! 0 < c < 2 ǫ

  • pen?

e := 0 e == 0 armedOff! flashOn! soundOn! soundOn! flashOn! e < 30 unlock? g := 0 e == 30 soundOff! unlock? g := 0 e == 300 soundOff! flashOff! unlock? close? f := 0 armedOn! soundOff! unlock? d := 0 armedOff!

Model D. Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min Partial 1 8 9.7s 8.0s 85.1s 0.3s 0.28s 0.04s 16.78s ∼ 0s Partial 2 12 1.6s 1.63s 37.3s 0.08s 0.08s 0.03s 2.28s ∼ 0s Complete 12 x x x x 0.79s 0.06s 360.84s ∼ 0s

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 25 / 64

slide-58
SLIDE 58

tugraz

Institute of Software Technology

Experimental Results V

◮ Adding data variable and parameters to

◮ deterministic Car Alarm System with one clock ◮ 3-digit PIN code for unlocking

◮ No negative effects, even with higher digit PIN codes ◮ Symbolic execution faster with 1 clock (0.24s) than with 5 clocks

(1.7s)

Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 8 1.46s 0.28s 59.41s 0.12s 0.07s 0.05s 0.82s ∼ 0s 12 4.12s 0.35s 35.41s 0.13s 0.24s 0.05s 3.67s ∼ 0s

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 26 / 64

slide-59
SLIDE 59

tugraz

Institute of Software Technology

Experimental Results V

◮ Adding data variable and parameters to

◮ deterministic Car Alarm System with one clock ◮ 3-digit PIN code for unlocking

◮ No negative effects, even with higher digit PIN codes ◮ Symbolic execution faster with 1 clock (0.24s) than with 5 clocks

(1.7s)

Depth Bounded Model Checking Symbolic Execution Mean Median Max Min Mean Median Max Min 8 1.46s 0.28s 59.41s 0.12s 0.07s 0.05s 0.82s ∼ 0s 12 4.12s 0.35s 35.41s 0.13s 0.24s 0.05s 3.67s ∼ 0s

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 26 / 64

slide-60
SLIDE 60

tugraz

Institute of Software Technology

Real-Time Systems Summary

Symbolic execution (SE) seems to perform better, but no clear winner!

◮ Number of clocks:

◮ BMC: small impact (was faster in deterministic case) ◮ SE: high impact

◮ Non-determinism: is an obstacle for conformance checking

◮ BMC: state-space explosion → partial models ◮ SE: lowered performance (40s vs. 6min) → 3 mutants timed out

◮ Statistical outliers: due to equivalent mutants

◮ BMC: runtime almost equal ◮ SE: extreme differences due to optimisations B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 27 / 64

slide-61
SLIDE 61

tugraz

Institute of Software Technology

Agenda

◮ Model-based Mutation Testing ◮ Real-Time Systems ◮ Hybrid Systems ◮ Discrete Systems

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 28 / 64

slide-62
SLIDE 62

tugraz

Institute of Software Technology

A Hybrid System: Two Tank System

Full Empty T1 P1 G1 Empty Reserve Full T2 P2 G2

inout

  • ut

in

P1, P2 . . . water pumps G1, G2 . . . water-level sensors Requirements:

◮ P1 starts pumping, if T2 below

Reserve and T1 is full

◮ until T1 is empty or T2 is full ◮ P2 is controlled by button

WaterRequest

◮ runs if there is water in T2. ◮ Note: T1 may overflow

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 29 / 64

slide-63
SLIDE 63

tugraz

Institute of Software Technology

Related Work

◮ Hybrid Systems

◮ Hybrid Automata (Alur,Courcoubetis,Henzinger,Ho 93) ◮ Action Systems [Back,Kurki-Suonio 83] ◮ Hybrid Action Systems [Rönkkö,Ravn,Sere 03] ◮ Qualitative Reasoning [Kuipers 94]

◮ Testing

◮ Mutation Testing [Hamlet 77, De Millo et al. 78] ◮ Input-Output Conformance [Brinksma,Tretmans 92] B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 30 / 64

slide-64
SLIDE 64

tugraz

Institute of Software Technology

Abstraction 1: Action Systems

Modeling the Controller

Controller: |[ var P1_running, P2_running : Bool,

  • ut*, inout* : Real
  • P1_running := false;

P2_running := false;

  • ut := 0; inout := 0;

do g1 → P1_running := true; inout := (0, Max]

  • g2 → P1_running := false; inout := 0
  • g3 → P2_running := true; out := (0, Max]
  • g4 → P2_running := false; out := 0
  • d

]| : WaterRequest, x1, x2 Guards: ◮ g1 =

df

x2 ≤ Reserve ∧ x1 = Full ∧ ¬P1_running ◮ g2 =

df

P1_running ∧ (x1 ≤ Empty ∨ x2 = Full) ◮ g3 =

df

WaterRequest ∧ ¬P2_running ∧ x2 > Reserve ◮ g4 =

df

P2_running ∧ (¬WaterRequest∨ x2 = Empty)

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 31 / 64

slide-65
SLIDE 65

tugraz

Institute of Software Technology

Abstraction 1: Action Systems

Modeling the Controller

Controller: |[ var P1_running, P2_running : Bool,

  • ut*, inout* : Real
  • P1_running := false;

P2_running := false;

  • ut := 0; inout := 0;

do g1 → P1_running := true; inout := (0, Max]

  • g2 → P1_running := false; inout := 0
  • g3 → P2_running := true; out := (0, Max]
  • g4 → P2_running := false; out := 0
  • d

]| : WaterRequest, x1, x2 Guards: ◮ g1 =

df

x2 ≤ Reserve ∧ x1 = Full ∧ ¬P1_running ◮ g2 =

df

P1_running ∧ (x1 ≤ Empty ∨ x2 = Full) ◮ g3 =

df

WaterRequest ∧ ¬P2_running ∧ x2 > Reserve ◮ g4 =

df

P2_running ∧ (¬WaterRequest∨ x2 = Empty)

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 31 / 64

slide-66
SLIDE 66

tugraz

Institute of Software Technology

Why Action Systems?

◮ Well-suited for embedded systems modeling ◮ Action view maps naturally to LTS testing theories ◮ Solid foundation:

◮ precise semantics ◮ refinement

◮ Compositional modeling ◮ Many extensions available:

◮ object-orientation ◮ hybrid systems B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 32 / 64

slide-67
SLIDE 67

tugraz

Institute of Software Technology

Hybrid Action Systems

Environment:

|[ var x1*, x2* : Real

  • x1 := 0; x2 := 0

alt g1 → . . .

  • . . .

with ¬(g1 ∨ . . . ) :→ ˙ x1 = (in − inout)/A1 ∧ ˙ x2 = (inout − out)/A2 ]| : inout, out ◮

Rönkkö, M., Ravn, A.P., Sere, K.: Hybrid action systems. Theoretical Computer Science 290 (2003) 937–973. B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 33 / 64

slide-68
SLIDE 68

tugraz

Institute of Software Technology

Abstraction 2: Qualitative Flows

t-abs.f.t

1 2 1 2

t

zero med high max

g.s t v-abs.f.t f.t

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 34 / 64

slide-69
SLIDE 69

tugraz

Institute of Software Technology

Example Qualitative Flow of Water Tanks

Empty Full

x1 x2

Zero Empty Full Zero Empty Reserve Full P1/P2 OFF P1 ON P2 ON P1 OFF

x1 x2 t

x1 x2

Empty Reserve Full B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 35 / 64

slide-70
SLIDE 70

tugraz

Institute of Software Technology

Qualitative Reasoning (QR)

◮ QR originates from Artificial Intelligence ◮ Common sense reasoning about physical systems with possibly

incomplete knowledge.

◮ Ordinary Differential Equations (ODE)

→ Qualitative Differential Equations (QDE): ˙ x1 = (in − inout)/A1 → d/dt(x1, diff1) ∧ add(diff1, inout, in)

◮ Arithmetic is reduced to sign algebra:

5 − 1 = 4 → [+] + [−] = [+] | [−] −3 ∗ 2 = −6 → [−] ∗ [+] = [−]

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 36 / 64

slide-71
SLIDE 71

tugraz

Institute of Software Technology

Qualitative Action Systems

|[ var x1*, x2* : Real

  • x1 := 0; x2 := 0

alt g1 → . . .

  • . . .

with ¬(g1 ∨ . . . ) :→ d/dt(x1, diff1) ∧ d/dt(x2, diff2) ∧ add(diff2, out, inout) ∧ add(diff1, inout, in) ]| : inout, out

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 37 / 64

slide-72
SLIDE 72

tugraz

Institute of Software Technology

Qualitative Simulation

◮ Implementations:

◮ QSIM (Lisp) ◮ Garp3 (SWI-Prolog) ◮ ASIM (GNU-Prolog) B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 38 / 64

slide-73
SLIDE 73

tugraz

Institute of Software Technology

Model-based Mutation Testing

Action System Model Mutants IOLTSS IOLTSM for every mutant ioco? discriminating test case ioco . . . input-output conformance

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 39 / 64

slide-74
SLIDE 74

tugraz

Institute of Software Technology

Model-based Mutation Testing

Action System Model Mutants IOLTSS IOLTSM for every mutant ioco? discriminating test case ioco . . . input-output conformance

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 39 / 64

slide-75
SLIDE 75

tugraz

Institute of Software Technology

Model-based Mutation Testing

Action System Model Mutants IOLTSS IOLTSM for every mutant ioco? discriminating test case ioco . . . input-output conformance

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 39 / 64

slide-76
SLIDE 76

tugraz

Institute of Software Technology

Model-based Mutation Testing

Action System Model Mutants IOLTSS IOLTSM for every mutant ioco? discriminating test case ioco . . . input-output conformance

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 39 / 64

slide-77
SLIDE 77

tugraz

Institute of Software Technology

Conformance Checking

◮ Event-view: labeled actions ◮ Input and Output Labels

  • Def. IOCO [Tretmans 96]

∀σ ∈ Straces(Model) : out(Mutant after σ) ⊆ out(Model after σ)

  • ut ... outputs labels + quiescence

after ... reachable states after trace

◮ ioco supports: partial, non-deterministic models ◮ ioco-checker Ulysses

◮ implemented in GNU Prolog ◮ explores discrete actions + qualitative flows ◮ builds synchronous product modulo ioco ◮ highly non-deterministic → on-the-fly determinization B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 40 / 64

slide-78
SLIDE 78

tugraz

Institute of Software Technology

Conformance Checking

◮ Event-view: labeled actions ◮ Input and Output Labels

  • Def. IOCO [Tretmans 96]

∀σ ∈ Straces(Model) : out(Mutant after σ) ⊆ out(Model after σ)

  • ut ... outputs labels + quiescence

after ... reachable states after trace

◮ ioco supports: partial, non-deterministic models ◮ ioco-checker Ulysses

◮ implemented in GNU Prolog ◮ explores discrete actions + qualitative flows ◮ builds synchronous product modulo ioco ◮ highly non-deterministic → on-the-fly determinization B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 40 / 64

slide-79
SLIDE 79

tugraz

Institute of Software Technology

Generating a Testcase: Original Model

System = |[ var x1 : T1, x2 : T2, out, inout : FR, diff1, diff2 : NZP, p1_running, p2_running, wr : Bool

  • x1 := (0, 0); x2 := (0, 0);
  • ut := (0, 0); inout := (0, 0); wr := false

p1_running := false; p2_running := false alt

  • bs pump1_on : g1 → p1_running := true;

inout := (0..Max, 0)

  • bs pump1_off : g2 → p1_running := false;

inout := (0, 0)

  • bs pump2_on : g3 → p2_running := true;
  • ut := (0..Max, 0)
  • bs pump2_off : g4 → p2_running := false;
  • ut := (0, 0)
  • ctr water_req(X) : g5 → wr := X

with ¬(g1 ∨ g2 ∨ g3 ∨ g4 ∨ g5) :⇁ add(diff2, out, inout) ∧ add(diff1, inout, in)∧ d/dt(x1, diff1) ∧ d/dt(x2, diff2) ]| : in

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 41 / 64

slide-80
SLIDE 80

tugraz

Institute of Software Technology

Generating a Testcase II: Mutated Model

System = |[ var x1 : T1, x2 : T2, out, inout : FR, diff1, diff2 : NZP, p1_running, p2_running, wr : Bool

  • x1 := (0, 0); x2 := (0, 0);
  • ut := (0, 0); inout := (0, 0); wr := false

p1_running := false; p2_running := false alt

  • bs pump1_on : g1 → p1_running := true;

inout := (0..Max, 0)

  • bs pump1_off : g2 → p1_running := true;

inout := (0, 0)

  • bs pump2_on : g3 → p2_running := true;
  • ut := (0..Max, 0)
  • bs pump2_off : g4 → p2_running := false;
  • ut := (0, 0)
  • ctr water_req(X) : g5 → wr := X

with ¬(g1 ∨ g2 ∨ g3 ∨ g4 ∨ g5) :⇁ add(diff2, out, inout) ∧ add(diff1, inout, in)∧ d/dt(x1, diff1) ∧ d/dt(x2, diff2) ]| : in

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 42 / 64

slide-81
SLIDE 81

tugraz

Institute of Software Technology

Generating a Testcase III: Product Graph

Part of the result of the conformance check between the original and the mutated specification.

1

  • bs qual([x1:full/inc,x2:zero/std])

28 29

  • bs pump1_off

30 (pass)

  • bs qual([x1:empty..full/inc,x2:empty..reserve/dec])

31 (fail)

  • bs pump1_off

27

  • bs pump2_on

32

  • bs pump1_off
  • bs pump2_on

33 (fail)

  • bs pump1_off

35 (pass) 36 (fail) 34 ctr water_req(1) obs delta

  • bs pump1_off

26 ctr water_req(1)

  • bs pump1_off

... ... 2

  • bs qual([x1:empty..full/inc,x2:full/inc])
  • bs qual([x1: ...,x2: ...])
  • bs pump1_on

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 43 / 64

slide-82
SLIDE 82

tugraz

Institute of Software Technology

Results

Mut. No. Avg.Time Average No. = Op. Mutants [s] States Trans. = No. Perc. ASO 10 13.9 64 117 7 3 30% ENO 6 7.6 68 120 5 1 17% ERO 20 12.9 62 110 20 0% LRO 13 12.8 93 168 9 4 31% MCO 16 12.8 70 126 10 6 38% RRO 12 12.0 40 73 10 2 17% Total 77 12.0 66 119 61 16 21%

ASO ... Association Shift Operator ENO ... Expression Negation Operator ERO ... Event Replacement Operator LRO ... Logical Operator Replacement MCO ... Missing Condition Operator RRO ... Relational Replacement Operator

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 44 / 64

slide-83
SLIDE 83

tugraz

Institute of Software Technology

Generating a Testcase IV: Linear TC

Selecting one path for each unsafe state leading to failure.

1

  • bs qual([x1:full/inc,x2:zero/std])

106 (pass) 104

  • bs qual([x1:empty..full/inc,x2:empty..reserve/dec])

102

  • bs out_pump2_on

99 ctr in_water_req(1) 98

  • bs out_pump1_off

... (inconcl) 2

  • bs qual([x1:empty..full/inc,x2:full/inc])
  • bs qual([x1: ...,x2: ...])
  • bs out_pump1_on

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 45 / 64

slide-84
SLIDE 84

tugraz

Institute of Software Technology

Generating a Testcase V: Adaptive TC

A test graph including all paths to a given unsafe state leading to failure.

1 2 3 4 5 6 7 8 9 10 11 12 13

  • bs out_pum

p1_on

  • bs out_pum

p1_off

  • bs out_pum

p1_on

  • bs de

lta ctr in_wa te r_re q(1)

  • bs out_pum

p2_on ctr in_wa te r_re q(0)

  • bs out_pum

p2_off

  • bs out_pum

p1_off

  • bs out_pum

p1_on

  • bs de

lta

  • bs inconc
  • bs out_pum

p1_off

  • bs out_pum

p1_on

  • bs de

lta ctr in_wa te r_re q(1)

  • bs pa

ss

Qualitative events are internal (not visible).

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 46 / 64

slide-85
SLIDE 85

tugraz

Institute of Software Technology

Hybrid Systems Summary

◮ AI meets FM: qualitative reasoning ◮ Requirements → incomplete qualitative models ◮ Model exploration: controller (discrete) + environment (qualitative) ◮ TCG based on mutation testing and ioco conformance checking ◮ Different strategies for selecting test case

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 47 / 64

slide-86
SLIDE 86

tugraz

Institute of Software Technology

Agenda

◮ Model-based Mutation Testing ◮ Real-Time Systems ◮ Hybrid Systems ◮ Discrete Systems

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 48 / 64

slide-87
SLIDE 87

tugraz

Institute of Software Technology

Discrete Systems: MoMuT::UML

Applications:

◮ Car Alarm System (Ford) ◮ Railway Interlocking System (Thales) ◮ Automotive Meassurement Device: Particle Counter (AVL)

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 49 / 64

slide-88
SLIDE 88

tugraz

Institute of Software Technology

SUT: AVL489 Particle Counter

◮ One of AVL’s automotive

measurement devices

◮ Measures particle number

concentrations in exhaust gas

◮ Focus: testing of the control logic ◮ AVL uses virtual test-beds with

simulated devices for integration and regression testing.

◮ We tested a simulation of the

particle counter:

◮ Matlab Simulink model

compiled to real-time executable

◮ Same interface as real

device!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 50 / 64

slide-89
SLIDE 89

tugraz

Institute of Software Technology

SUT: AVL489 Particle Counter

◮ One of AVL’s automotive

measurement devices

◮ Measures particle number

concentrations in exhaust gas

◮ Focus: testing of the control logic ◮ AVL uses virtual test-beds with

simulated devices for integration and regression testing.

◮ We tested a simulation of the

particle counter:

◮ Matlab Simulink model

compiled to real-time executable

◮ Same interface as real

device!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 50 / 64

slide-90
SLIDE 90

tugraz

Institute of Software Technology

SUT: AVL489 Particle Counter

◮ One of AVL’s automotive

measurement devices

◮ Measures particle number

concentrations in exhaust gas

◮ Focus: testing of the control logic ◮ AVL uses virtual test-beds with

simulated devices for integration and regression testing.

◮ We tested a simulation of the

particle counter:

◮ Matlab Simulink model

compiled to real-time executable

◮ Same interface as real

device!

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 50 / 64

slide-91
SLIDE 91

tugraz

Institute of Software Technology

UML Test Model of AVL489

AVL489 isReady isBusy Pause_0 send SPAU state /entry send StatusBusy; set Busy /exit Standby_1 send STBY_state /entry send StatusBusy; set Busy /exit Active Purging_Pause_12 send SPUL_state /entry Purging_Standby_12 send SPUL_state /entry Response_14 send SEGA_state /entry Leakage_11 send SLEC_state /entry Integral_9 send SINT_state /entry send StatusBusy; set Busy /exit Measurement_2 send SMGA_state /entry send StatusBusy; set Busy /exit ZeroGas_10 send SNGA_state /entry send StatusBusy; set Busy /exit Manual set Manual /entry Remote unset Manual /entry DilutionSelection [ not Manual and not Busy ] / set Dilution LeakageTest, ResponseCheck [ not (oclIsInState(Standby_1 SetPurge [ not (oclIsInState(Pause_0) or oclIsInState(Standby SetZeroPoint [ not oclIsInState(Active::Measurement_2) and StopIntegralMeasurement [ not oclIsInState(Active::Integral_ StartMeasurement [ not (oclIsInState(Standby_1) or oclIsInState StartIntegralMeasurement [ not (oclIsInState(Active::Measurement_2) when Busy 30 [ not (oclIsInState(Active::Response_14)

  • r oclIsInState(Active::Purging_Standby_12)
  • r oclIsInState(Active::Leakage_11)
  • r oclIsInState(Active::ZeroGas_10)
  • r oclIsInState(Active::Purging_Pause_12)) ] / set not Busy - send StatusReady

LeakageTest, ResponseCheck, SetPurge, SetZeroPoint, StopIntegralMeasurement, SetStandby, StartMeasurement, StartIntegralMeasurement, SetPause, DilutionSelection SetStandby [ not Busy and not Manual ] SetPurge [ not Busy and not Manual ] SetPause [ not Busy and not Manual ] SetPause [ not Busy and not Manual ] SetPause [ not Busy and not Manual ] 10 SetStandby [ not Busy and not Manual ] SetPurge [ not Busy and not Manual ] LeakageTest [ not Busy and not Manual ] StartMeasurement [ not Busy and not Manual ] ResponseCheck [ not Busy and not Manual ] 10 10 10 SetStandby [ not Busy and not Manual ] StartIntegralMeasurement, StopIntegralMeasurement, StartMeasurement [ not Busy and not Manual ] StartIntegralMeasurement [ not Busy and not Manual ] SetZeroPoint [ not Busy and not Manual ] 10 / send Offline SetRemote / send Online DilutionSelection, LeakageTest, ResponseCheck, SetPurge, SetZeroPoint, StopIntegralMeasurement, SetPause, SetStandby, StartMeasurement, StartIntegralMeasurement / send RejectOF SetManual SetManual / send Offline SetRemote

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 51 / 64

slide-92
SLIDE 92

tugraz

Institute of Software Technology

MoMuT::UML

◮ Test-case generator of AIT and TU Graz ◮ Implementing model-based mutation testing for UML state machines MoMuT::UML( ( ( ( ( ( ( (

(

! UML2OOAS( Java! OOAS2AS( Java! Enumera3ve(TCG( Prolog! Symbolic(TCG( Prolog! SMT!Solver! Z3!

AlarmSystem_StateMachine Alarm Activate Alarms /entry Deactivate Alarms /exit Flash FlashAndSound Armed Show Armed /entry Show Unarmed /exit ClosedAndLocked OpenAndUnlocked ClosedAndUnlocked OpenAndLocked SilentAndOpen Unlock 30 / Deactivate Sound 300 Open Unlock 20 Close Unlock Open Lock Close Close Lock Open Unlock

UML(model( Papyrus MDT/! Visual Paradigm! abstract(test(cases( Aldebaran aut format!

Java! frontend! backend!

Architecture of the MoMuT::UML tool chain

AS ... Action Systems [Back83] OOAS ... Object-Oriented Action Systems

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 52 / 64

slide-93
SLIDE 93

tugraz

Institute of Software Technology

Abstract Test Case of AVL489

  • bs StatusReady(0)
  • bs SPAU_state(0)
  • bs Offline(0)

ctr SetStandby(0)

  • bs StatusBusy(0)
  • bs STBY_state(0)
  • bs Online(0)
  • bs StatusReady(30)

ctr StartMeasurement(0)

  • bs StatusBusy(0)
  • bs SMGA_state(0)
  • bs StatusReady(30)

ctr StartIntegralMeasurement(0)

  • bs SINT_state(0)

ctr SetStandby(0)

  • bs STBY_state(0)

pass

Abstract test cases → concrete C# NUnit test cases.

ctr ... controllable event (input)

  • bs ... observable event (output)

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 53 / 64

slide-94
SLIDE 94

tugraz

Institute of Software Technology

Test Execution on Particle Counter

We found several bugs in the SUT:

◮ Forbidden changes of operating state while busy

◮ Pause → Standby ◮ Normal Measurement → Integral Measurement

◮ Ignoring high-frequent input without error-messages ◮ Loss of error messages in client for remote control of the device

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 54 / 64

slide-95
SLIDE 95

tugraz

Institute of Software Technology

Refinement + ioco Conformance Checking

Refinement:

◮ state-based ◮ predicative semantics

  • Def. Refinement [Hoare & He 98]

∀s, s′ : Mutant(s, s′) ⇒ Model(s, s′) s ... state before s’ ... state after execution

Input-Output Conformance:

◮ event-based ◮ io labelled transition systems

  • Def. IOCO [Tretmans 96]

∀σ ∈ traces(Model) :

  • ut(Mutant after σ) ⊆ out(Model after σ)
  • ut ... outputs labels + quiescence

after ... reachable states after trace

New combined conformance checking:

◮ Refinement checker searches for faulty state (fast) ◮ Ioco checker looks if faulty state propagates to different observations

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 55 / 64

slide-96
SLIDE 96

tugraz

Institute of Software Technology

Refinement + ioco Conformance Checking

Refinement:

◮ state-based ◮ predicative semantics

  • Def. Refinement [Hoare & He 98]

∀s, s′ : Mutant(s, s′) ⇒ Model(s, s′) s ... state before s’ ... state after execution

Input-Output Conformance:

◮ event-based ◮ io labelled transition systems

  • Def. IOCO [Tretmans 96]

∀σ ∈ traces(Model) :

  • ut(Mutant after σ) ⊆ out(Model after σ)
  • ut ... outputs labels + quiescence

after ... reachable states after trace

New combined conformance checking:

◮ Refinement checker searches for faulty state (fast) ◮ Ioco checker looks if faulty state propagates to different observations

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 55 / 64

slide-97
SLIDE 97

tugraz

Institute of Software Technology

Refinement + ioco Conformance Checking

Refinement:

◮ state-based ◮ predicative semantics

  • Def. Refinement [Hoare & He 98]

∀s, s′ : Mutant(s, s′) ⇒ Model(s, s′) s ... state before s’ ... state after execution

Input-Output Conformance:

◮ event-based ◮ io labelled transition systems

  • Def. IOCO [Tretmans 96]

∀σ ∈ traces(Model) :

  • ut(Mutant after σ) ⊆ out(Model after σ)
  • ut ... outputs labels + quiescence

after ... reachable states after trace

New combined conformance checking:

◮ Refinement checker searches for faulty state (fast) ◮ Ioco checker looks if faulty state propagates to different observations

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 55 / 64

slide-98
SLIDE 98

tugraz

Institute of Software Technology

Symbolic Refinement Checking

Is non-refinement reachable?

∃ s, s′, tr, tr ′ : reachable(s, tr) ∧ Mutant(s, s′, tr, tr ′) ∧ ¬Model(s, s′, tr, tr ′)

s ... state before s’ ... states after execution tr ... trace of labels before tr’ ... trace of labels after execution

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 56 / 64

slide-99
SLIDE 99

tugraz

Institute of Software Technology

TCG Particle Counter

stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles stles 928+ 189+ 68+ notYconformingY (nonZref.Y&YnotYioco)Y conformingY (refining)Y conformingY (nonZref.,YbutYioco)Y

(a) Breakup into conforming and

not conforming model mutants.

stles stles stles stles stles stles stles stles stles stles stles stles stles stles 111+ 817+ uniqueYTCsY duplicateYTCsY

(b) Breakup into unique and

duplicate test cases.

1+ 1+ 1+ 1+ 12+ 1+ 12+ 2+ 2+ 26+ 10+ 9+ 13+ 4+ 12+ 4+ 0Y 5Y 10Y 15Y 20Y 25Y 30Y 1Y 2Y 3Y 4Y 5Y 6Y 7Y 8Y 9Y 10Y 11Y 12Y 13Y 14Y 15Y 16Y unique+test+cases+[#]+ length+

(c) Lengths of the unique test cases.

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 57 / 64

slide-100
SLIDE 100

tugraz

Institute of Software Technology

Fault Propagation

452' 423' 44' 6' 3' 01 1001 2001 3001 4001 5001 11 21 31 41 51 mutants'[#]' ioco'depth'

Figure: Number of steps from fault to failure (ioco depths)

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 58 / 64

slide-101
SLIDE 101

tugraz

Institute of Software Technology

Run-times

... for combined conformance checking (in min., max. depth 15+5) :

conforming conforming not conforming total (refining) (non-ref., but ioco) (non-ref. & not ioco) mutants [#] 189 68 928 1185

  • ref. check

Σ 6.1 h 7.7 7.1 h 13.3 h φ 1.9 6.8 sec 27 sec 40 sec max 4.3 1.8 3.9 4.3 ioco check Σ

  • 0.7 h

1.7 h 2.4 h φ

  • 38 sec

7 sec 7.4 sec max

  • 2

27 sec 2 tc constr. Σ

  • 22.9

22.9 φ

  • 1.5 sec

1.2 sec max

  • 3.7 sec

3.7 sec total without logging Σ 6.1 h 0.9 h 9.2 h 16.2 h φ 1.9 0.8 0.6 0.8 max 4.3 2.2 4.1 4.3

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 59 / 64

slide-102
SLIDE 102

tugraz

Institute of Software Technology

Run-times

... comparison to stand-alone ioco check (in min., max. depth 10):

not ioco ioco total mutants [#] 719 466 1185 time – ioco check Σ 9.8 h 22.8 h 32.6 h φ 0.8 2.9 1.7 max 3.9 5.2 5.2 time – tc constr. Σ 19

  • 19

φ 1.6 sec

  • 1 sec

max 5.8 sec

  • 5.8 sec

total without logging Σ 10.1 h 22.8 h 32.9 h φ 0.8 2.9 1.7 max 3.9 5.2 5.2

  • appr. 16h vs. 33h

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 60 / 64

slide-103
SLIDE 103

tugraz

Institute of Software Technology

Discrete Systems Summary

◮ Fault propagation important for test-case design ◮ Faster test-case generator

◮ find fault fast (refinement check) ◮ analyze if fault propagates to failure (ioco check)

◮ Optimized refinement check

◮ incremental SMT solving, state caching ◮ exploiting the location of mutation ◮ checking if existing test cases cover next fault

◮ Applied at AVL: many bugs found [TAP 2014]

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 61 / 64

slide-104
SLIDE 104

tugraz

Institute of Software Technology

Discrete Systems Summary

◮ Fault propagation important for test-case design ◮ Faster test-case generator

◮ find fault fast (refinement check) ◮ analyze if fault propagates to failure (ioco check)

◮ Optimized refinement check

◮ incremental SMT solving, state caching ◮ exploiting the location of mutation ◮ checking if existing test cases cover next fault

◮ Applied at AVL: many bugs found [TAP 2014]

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 61 / 64

slide-105
SLIDE 105

tugraz

Institute of Software Technology

Discrete Systems Summary

◮ Fault propagation important for test-case design ◮ Faster test-case generator

◮ find fault fast (refinement check) ◮ analyze if fault propagates to failure (ioco check)

◮ Optimized refinement check

◮ incremental SMT solving, state caching ◮ exploiting the location of mutation ◮ checking if existing test cases cover next fault

◮ Applied at AVL: many bugs found [TAP 2014]

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 61 / 64

slide-106
SLIDE 106

tugraz

Institute of Software Technology

Synchronous Systems – MoMuT::REQs

Contract-based Requirement Interfaces:

◮ Synchronous assume-guarantee pairs ◮ Combined via conjunction ◮ Efficient SMT solving

Application: Airbag Chip (Infineon)

Inputs coin, teabutton, coffeebutton; Outputs coffee, tea; Internals paid; {I} not paid and not coffee and not tea {R1} assume coin’ guarantee paid ’ {R2} assume paid and teabutton’ and not coffeebutton ’ guarantee tea’ and not paid ’ {R3} assume paid and coffeebutton ’ and not teabutton’ guarantee coffee ’ and not paid ’ {R4} assume teabutton’ and coffeebutton ’ guarantee skip Bernhard K. Aichernig, Klaus Hörmaier, Florian Lorber, Dejan Nickovic, Stefan Tiran. Require, Test and Trace IT, FMICS 2015 Bernhard K. Aichernig and Dejan Nickovic and Stefan Tiran. Scalable Incremental Test-case Generation from Large Behavior Models, TAP 2015. Bernhard K. Aichernig, Klaus Hörmaier, Florian Lorber, Dejan Nickovic, Rupert Schlick, Didier Simoneau, Stefan Tiran. Integration of Requirements Engineering and Test-Case Generation via OSLC, QSIC 2014

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 62 / 64

slide-107
SLIDE 107

tugraz

Institute of Software Technology

Synchronous Systems – MoMuT::REQs

Contract-based Requirement Interfaces:

◮ Synchronous assume-guarantee pairs ◮ Combined via conjunction ◮ Efficient SMT solving

Application: Airbag Chip (Infineon)

Inputs coin, teabutton, coffeebutton; Outputs coffee, tea; Internals paid; {I} not paid and not coffee and not tea {R1} assume coin’ guarantee paid ’ {R2} assume paid and teabutton’ and not coffeebutton ’ guarantee tea’ and not paid ’ {R3} assume paid and coffeebutton ’ and not teabutton’ guarantee coffee ’ and not paid ’ {R4} assume teabutton’ and coffeebutton ’ guarantee skip Bernhard K. Aichernig, Klaus Hörmaier, Florian Lorber, Dejan Nickovic, Stefan Tiran. Require, Test and Trace IT, FMICS 2015 Bernhard K. Aichernig and Dejan Nickovic and Stefan Tiran. Scalable Incremental Test-case Generation from Large Behavior Models, TAP 2015. Bernhard K. Aichernig, Klaus Hörmaier, Florian Lorber, Dejan Nickovic, Rupert Schlick, Didier Simoneau, Stefan Tiran. Integration of Requirements Engineering and Test-Case Generation via OSLC, QSIC 2014

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 62 / 64

slide-108
SLIDE 108

tugraz

Institute of Software Technology

Summary

◮ Model-based Mutation Testing

◮ Automatically test against anticipated faults ◮ TCG via conformance checks

◮ Real-Time Systems: Timed Automata ◮ Hybrid Systems: Action Systems + Qualitative Reasoning ◮ Discrete Systems: UML ◮ Synchronous Systems: Assume-Guarantee Contracts ◮ Ongoing projects:

◮ DSL for easier modelling, performance testing (AVL) ◮ Event-B refinement checker including sets, maps (Thales) ◮ Dependable Internet of Things: test-based model learning B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 63 / 64

slide-109
SLIDE 109

tugraz

Institute of Software Technology

Summary

◮ Model-based Mutation Testing

◮ Automatically test against anticipated faults ◮ TCG via conformance checks

◮ Real-Time Systems: Timed Automata ◮ Hybrid Systems: Action Systems + Qualitative Reasoning ◮ Discrete Systems: UML ◮ Synchronous Systems: Assume-Guarantee Contracts ◮ Ongoing projects:

◮ DSL for easier modelling, performance testing (AVL) ◮ Event-B refinement checker including sets, maps (Thales) ◮ Dependable Internet of Things: test-based model learning B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 63 / 64

slide-110
SLIDE 110

tugraz

Institute of Software Technology

Summary

◮ Model-based Mutation Testing

◮ Automatically test against anticipated faults ◮ TCG via conformance checks

◮ Real-Time Systems: Timed Automata ◮ Hybrid Systems: Action Systems + Qualitative Reasoning ◮ Discrete Systems: UML ◮ Synchronous Systems: Assume-Guarantee Contracts ◮ Ongoing projects:

◮ DSL for easier modelling, performance testing (AVL) ◮ Event-B refinement checker including sets, maps (Thales) ◮ Dependable Internet of Things: test-based model learning B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 63 / 64

slide-111
SLIDE 111

tugraz

Institute of Software Technology

References

Real-Time Systems ◮ B.K. Aichernig, F. Lorber, M. Tappler: Conformance Checking of Real-Time Models - Symbolic Execution vs. Bounded Model Checking. Theory and Practice of Formal Methods 2016: 15-32 ◮ F. Lorber, A. Rosenmann, D. Nickovic, B.K. Aichernig: Bounded Determinization of Timed Automata with Silent Transitions. FORMATS 2015: 288-304 ◮ B.K. Aichernig, F. Lorber, D. Nickovic: Time for Mutants - Model-Based Mutation Testing with Timed Automata. TAP 2013: 20-38 Hybrid Systems ◮ B.K. Aichernig, H. Brandl, E. Jöbstl, W. Krenn: Model-Based Mutation Testing of Hybrid

  • Systems. FMCO 2009: 228-249

◮ B. K. Aichernig, H. Brandl, W. Krenn: Qualitative Action Systems. ICFEM 2009: 206-225 Discrete Systems ◮ B.K. Aichernig, J. Auer, E. Jöbstl, R. Korosec, W. Krenn, R. Schlick, B.V. Schmidt: Model-Based Mutation Testing of an Industrial Measurement Device. TAP 2014: 1-19 ◮ Willibald Krenn, Rupert Schlick, Stefan Tiran, Bernhard K. Aichernig, Elisabeth Jöbstl, Harald Brandl: MoMut: : UML Model-Based Mutation Testing for UML. ICST 2015: 1-8

B.K. Aichernig Killing Bugs in a Black Box with Model-based Mutation Testing 64 / 64