Complementary Criteria for Testing Temporal Logic Properties - - PowerPoint PPT Presentation

▶
complementary criteria for testing temporal logic
SMART_READER_LITE
LIVE PREVIEW

Complementary Criteria for Testing Temporal Logic Properties - - PowerPoint PPT Presentation

Sep 02, 2022 282 likes •1.06k views

Complementary Criteria for Testing Temporal Logic Properties Gordon Fraser and Franz Wotawa Graz University of Technology, Austria Model-Based Development Process Informal ideas Specification Design Code Realization Model-Based

slide-1
SLIDE 1

Complementary Criteria for Testing Temporal Logic Properties

Gordon Fraser and Franz Wotawa Graz University of Technology, Austria

slide-2
SLIDE 2

Model-Based Development Process

Informal ideas Realization Code Design Specification

slide-3
SLIDE 3

Model-Based Development Process

Informal ideas Realization Code Design Specification

Real world World of models Informal world

slide-4
SLIDE 4

Model-Based Development Process

Informal ideas Realization Code Design Specification

Real world World of models Informal world Validation

slide-5
SLIDE 5

Model-Based Development Process

Informal ideas Realization Code Design Specification

Real world World of models Informal world Validation Formal verification

slide-6
SLIDE 6

Model-Based Development Process

Informal ideas Realization Code Design Specification

Real world World of models Informal world Validation Formal verification Model−based testing

slide-7
SLIDE 7

Specification Based Testing

Informal ideas Realization Code Design Specification

Real world World of models Informal world Validation Formal verification Specification based testing

slide-8
SLIDE 8

Specification Based Testing

Informal ideas Realization Code Design Requirement Properties Executable Specification

Formal verification

Real world World of models Informal world

slide-9
SLIDE 9

Specification Based Testing

Informal ideas Realization Code Design Requirement Properties Executable Specification

Formal verification

Real world World of models Informal world Specification based testing

slide-10
SLIDE 10

Test cases derived from executable spec

✎ When is a property covered? ✎ How is it covered? ✎ How to generate tests for properties? ✎ Which tests to generate? ✎ ✎

slide-11
SLIDE 11

Test cases derived from executable spec

✎ When is a property covered? ✎ How is it covered? ✎ How to generate tests for properties? ✎ Which tests to generate? ✎ Contribution: 2 new criteria to measure and

generate tests

✎

slide-12
SLIDE 12

Test cases derived from executable spec

✎ When is a property covered? ✎ How is it covered? ✎ How to generate tests for properties? ✎ Which tests to generate? ✎ Contribution: 2 new criteria to measure and

generate tests

✎ Assumption: Properties specified in

temporal logic

slide-13
SLIDE 13

Model Checking

satisfies

Model Property

G (x −> F y)

?

slide-14
SLIDE 14

Model Checking

satisfies

Model Property

G (x −> F y)

slide-15
SLIDE 15

Model Checking

violates

Model Property

G (x −> F y) Counterexample

slide-16
SLIDE 16

Specification Based Testing

Informal ideas Realization Code Design Requirement Properties Executable Specification

Formal verification

Real world World of models Informal world Specification based testing

slide-17
SLIDE 17

Specification Based Testing

slide-18
SLIDE 18

Specification Based Testing

Model Checker Counterexamples LTL Properties

slide-19
SLIDE 19

Test Case Generation with Model Checkers

SPECIFICATION Test predicates Model checker Counter example Trap property never(tp) Test Suite generator Cex = sequence

  • f states

that cover tp = test Coverage Criteria Test + coverage info Test suite

slide-20
SLIDE 20

Example Specification: NuSMV

▼❖❉❯▲❊ ♠❛✐♥ ❱❆❘ ❛❝❝❡❧❡r❛t❡✿ ❜♦♦❧❡❛♥❀ ❜r❛❦❡✿ ❜♦♦❧❡❛♥❀ ✈❡❧♦❝✐t②✿ ④ st♦♣✱ s❧♦✇✱ ❢❛st ⑥❀ ❆❙❙■●◆ ✐♥✐t✭✈❡❧♦❝✐t②✮ ✿❂ st♦♣❀ ♥❡①t✭✈❡❧♦❝✐t②✮ ✿❂ ❝❛s❡ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ ❢❛st❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ ❢❛st ✿ s❧♦✇❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ st♦♣❀ ❜r❛❦❡✿ st♦♣❀ ❚❘❯❊ ✿ ✈❡❧♦❝✐t②❀ ❡s❛❝❀

slide-21
SLIDE 21

Example Specification: NuSMV

slide-22
SLIDE 22

Temporal Logics

slide-23
SLIDE 23

Temporal Logics Example Property

G ✿✭ velocity ❂ fast✮

slide-24
SLIDE 24

Temporal Logics Example Property

X velocity ❂ slow

slide-25
SLIDE 25

Temporal Logics Example Property

F ✿ accelerate

slide-26
SLIDE 26

Temporal Logics Example Property

accelerate U brake

slide-27
SLIDE 27

Example: NuSMV

▼❖❉❯▲❊ ♠❛✐♥ ❱❆❘ ❛❝❝❡❧❡r❛t❡✿ ❜♦♦❧❡❛♥❀ ❜r❛❦❡✿ ❜♦♦❧❡❛♥❀ ✈❡❧♦❝✐t②✿ ④ st♦♣✱ s❧♦✇✱ ❢❛st ⑥❀ ❆❙❙■●◆ ✐♥✐t✭✈❡❧♦❝✐t②✮ ✿❂ st♦♣❀ ♥❡①t✭✈❡❧♦❝✐t②✮ ✿❂ ❝❛s❡ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ ❢❛st❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ ❢❛st ✿ s❧♦✇❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ st♦♣❀ ❜r❛❦❡✿ st♦♣❀ ❚❘❯❊ ✿ ✈❡❧♦❝✐t②❀ ❡s❛❝❀

slide-28
SLIDE 28

Example: NuSMV

▼❖❉❯▲❊ ♠❛✐♥ ❱❆❘ ❛❝❝❡❧❡r❛t❡✿ ❜♦♦❧❡❛♥❀ ❜r❛❦❡✿ ❜♦♦❧❡❛♥❀ ✈❡❧♦❝✐t②✿ ④ st♦♣✱ s❧♦✇✱ ❢❛st ⑥❀ ❆❙❙■●◆ ✐♥✐t✭✈❡❧♦❝✐t②✮ ✿❂ st♦♣❀ ♥❡①t✭✈❡❧♦❝✐t②✮ ✿❂ ❝❛s❡ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ ❢❛st❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ ❢❛st ✿ s❧♦✇❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ st♦♣❀ ❜r❛❦❡✿ st♦♣❀ ❚❘❯❊ ✿ ✈❡❧♦❝✐t②❀ ❡s❛❝❀

slide-29
SLIDE 29

Coverage Criteria

❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀

Predicate Coverage

✎ ✭ ❫ ✿ ❫ ❂ ✦ ✿✭ ❂ ✮✮ ✎ ✭✿✭ ❫ ✿ ❫ ❂ ✮ ✦ ✭ ❂ ✮✮

slide-30
SLIDE 30

Coverage Criteria

❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀

Predicate Coverage

✎ G ✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop ✦ X ✿✭ velocity ❂ slow✮✮ ✎ ✭✿✭ ❫ ✿ ❫ ❂ ✮ ✦ ✭ ❂ ✮✮

slide-31
SLIDE 31

Coverage Criteria

❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀

Predicate Coverage

✎ G ✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop ✦ X ✿✭ velocity ❂ slow✮✮ ✎ G ✭✿✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop✮ ✦ X ✭ velocity ❂ slow✮✮

slide-32
SLIDE 32

Coverage Criteria

❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀

Predicate Coverage

✎ G ✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop ✦ X ✿✭ velocity ❂ slow✮✮ ✎ G ✭✿✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop✮ ✦ X ✭ velocity ❂ slow✮✮ ✎ 15 specification based test criteria ✎ 2 property based criteria

1 Based on vacuity 2 Based on MCDC

slide-33
SLIDE 33

Unique First Cause Coverage

✎ MCDC: Test cases where clauses affect

predicates

✎

slide-34
SLIDE 34

Unique First Cause Coverage

✎ MCDC: Test cases where clauses affect

predicates

✎ UFC [Whalen et al., 2006]:

Clause c is the unique first cause of a formula A, if in the first state along a path where A is satisfied, it is satisfied because of c

slide-35
SLIDE 35

Unique First Cause Coverage

✭ ❴ ✮

slide-36
SLIDE 36

Unique First Cause Coverage

F ✭a ❴ b✮

slide-37
SLIDE 37

Unique First Cause Coverage

✎ Defined as set of rules ✎ Apply rules to property ✦ set of test

predicates

✎ Rules applied to all properties ✎ Test predicates: Measure coverage and

generate tests

slide-38
SLIDE 38

Unique First Cause Coverage is not enough

G ✭✭c1 ❫ c2✮ ✩ X d✮

✎

✎ ✎ ✎

❫ ✎ ✎

✿ ❫ ✿

✎ ✦

slide-39
SLIDE 39

Unique First Cause Coverage is not enough

G ✭✭c1 ❫ c2✮ ✩ X d✮

✎ UFC covers:

✎ Where c1 causes d to be false. ✎ Where c2 causes d to be false. ✎ Where c1 ❫ c2 causes d to be true.

✎ ✎

✿ ❫ ✿

✎ ✦

slide-40
SLIDE 40

Unique First Cause Coverage is not enough

G ✭✭c1 ❫ c2✮ ✩ X d✮

✎ UFC covers:

✎ Where c1 causes d to be false. ✎ Where c2 causes d to be false. ✎ Where c1 ❫ c2 causes d to be true.

✎ What if d represents a safety critical value? ✎

✿ ❫ ✿

✎ ✦

slide-41
SLIDE 41

Unique First Cause Coverage is not enough

G ✭✭c1 ❫ c2✮ ✩ X d✮

✎ UFC covers:

✎ Where c1 causes d to be false. ✎ Where c2 causes d to be false. ✎ Where c1 ❫ c2 causes d to be true.

✎ What if d represents a safety critical value? ✎ Is ✿c1 ❫ ✿c2 covered? ✎ ✦

slide-42
SLIDE 42

Unique First Cause Coverage is not enough

G ✭✭c1 ❫ c2✮ ✩ X d✮

✎ UFC covers:

✎ Where c1 causes d to be false. ✎ Where c2 causes d to be false. ✎ Where c1 ❫ c2 causes d to be true.

✎ What if d represents a safety critical value? ✎ Is ✿c1 ❫ ✿c2 covered? ✎ ✦ Property Inactive Clause Coverage

slide-43
SLIDE 43

Property Inactive Clause Coverage

✚✭x✮ ❂ ❢x❀ ✿x❣ ✚✭A ❫ B✮ ❂ ❢a ❫ ✿B ❥ a ✷ ✚✭A✮❣ ❬ ❢✿A ❫ b ❥ b ✷ ✚✭B✮❣ ✚✭A ❴ B✮ ❂ ❢a ❫ B ❥ a ✷ ✚✭A✮❣ ❬ ❢A ❫ b ❥ b ✷ ✚✭B✮❣ ✚✭✿A✮ ❂ ✚✭A✮ ✚✭ G ✭A✮✮ ❂ ❢A U ✭a ❫ G ✭A✮✮ ❥ a ✷ ✚✭A✮❣ ✚✭ F ✭A✮✮ ❂ ❢✿A U ✭a ❫ ✭A ❴ F A✮✮ ❥ a ✷ ✚✭A✮❣ ✚✭ X ✭A✮✮ ❂ ❢ X ✭a✮ ❥ a ✷ ✚✭A✮❣ ✚✭A U B✮ ❂ ❢✭A ❫ ✿B✮ U ✭a ❫ ✭B ❴ ✭A U B✮✮✮ ❥ a ✷ ✚✭A✮❣❬ ❢✭A ❫ ✿B✮ U ✭b ❫ ✭B ❴ ✭A U B✮✮✮ ❥ b ✷ ✚✭B✮❣

slide-44
SLIDE 44

Property Inactive Clause Coverage Property

✣ ❂ G ✭x ✦ X y✮ ✚✭ ✭ ✮✮ ❂ ❢ ✭ ❫ ✭ ✮✮ ❥ ✷ ✚✭ ✮❣ ✭ ✦ ✮ ✭ ❫ ✭ ✦ ✮✮ ❥ ✷ ✚✭ ✦ ✮

slide-45
SLIDE 45

Property Inactive Clause Coverage Property

✣ ❂ G ✭x ✦ X y✮

Rule

✚✭ G ✭A✮✮ ❂ ❢A U ✭a ❫ G ✭A✮✮ ❥ a ✷ ✚✭A✮❣ ✭ ✦ ✮ ✭ ❫ ✭ ✦ ✮✮ ❥ ✷ ✚✭ ✦ ✮

slide-46
SLIDE 46

Property Inactive Clause Coverage Property

✣ ❂ G ✭x ✦ X y✮

Rule

✚✭ G ✭A✮✮ ❂ ❢A U ✭a ❫ G ✭A✮✮ ❥ a ✷ ✚✭A✮❣

Result

✭x ✦ X y✮ U ✭a ❫ G ✭x ✦ X y✮✮ ❥ a ✷ ✚✭x ✦ X y✮

slide-47
SLIDE 47

Property Inactive Clause Coverage Property

x ✦ X y

slide-48
SLIDE 48

Property Inactive Clause Coverage Property

x ✦ X y

Rule

✚✭A ❴ B✮ ❂ ❢a ❫ B ❥ a ✷ ✚✭A✮❣ ❬ ❢A ❫ b ❥ b ✷ ✚✭B✮❣ ✚✭✿A✮ ❂ ✚✭A✮

slide-49
SLIDE 49

Property Inactive Clause Coverage Property

x ✦ X y

Rule

✚✭A ❴ B✮ ❂ ❢a ❫ B ❥ a ✷ ✚✭A✮❣ ❬ ❢A ❫ b ❥ b ✷ ✚✭B✮❣ ✚✭✿A✮ ❂ ✚✭A✮

Result

✚✭x ✦ X y✮ ❂ ✚✭✿x ❴ X y✮ ❂ ❢a ❫ X y ❥ a ✷ ✚✭✿x✮❣ ❬❢✿x ❫ b ❥ b ✷ ✚✭ X y✮❣

slide-50
SLIDE 50

Property Inactive Clause Coverage Property

X y

slide-51
SLIDE 51

Property Inactive Clause Coverage Property

X y

Rule

✚✭ X ✭A✮✮ ❂ ❢ X ✭a✮ ❥ a ✷ ✚✭A✮❣

slide-52
SLIDE 52

Property Inactive Clause Coverage Property

X y

Rule

✚✭ X ✭A✮✮ ❂ ❢ X ✭a✮ ❥ a ✷ ✚✭A✮❣

Result

✚✭ X y✮ ❂ ❢ X ✭a✮ ❥ a ✷ ✚✭y✮❣ ❂ ❢ X ✭y✮❀ X ✭✿y✮❣

slide-53
SLIDE 53

Property Inactive Clause Coverage Property

✣ ❂ G ✭x ✦ X y✮

Result

❢✭x ✦ X y✮ U ✭✭x ❫ X y✮ ❫ G ✭x ✦ X y✮✮❀ ✭x ✦ X y✮ U ✭✭✿x ❫ X y✮ ❫ G ✭x ✦ X y✮✮❀ ✭x ✦ X y✮ U ✭✭✿x ❫ X ✭y✮✮ ❫ G ✭x ✦ X y✮✮❀ ✭x ✦ X y✮ U ✭✭✿x ❫ X ✭✿y✮✮ ❫ G ✭x ✦ X y✮✮❣

slide-54
SLIDE 54

PICC Summary

✎ When do clauses not affect result? ✎ PICC extends UFC like RCDC extends

MCDC

✎ Apply rules to all properties ✦ test

predicates

slide-55
SLIDE 55

Mutation Testing for Properties

G (a −> X !b) G (b −> X !c) G (c −> X !d) .... Mutants Model Checker Test cases Model G (a −> X !b) G (b −> X !c) G (c −> X !d) .... Requirements x & !c

slide-56
SLIDE 56

Mutation Testing for Properties

Informal ideas Realization Code Design Requirement Properties Executable Specification

UFC, PICC: Are the properties implemented right?

slide-57
SLIDE 57

Mutation Testing for Properties

Informal ideas Realization Code Design Requirement Properties Executable Specification

UFC, PICC: Are the properties implemented right? Are the right properties implemented?

slide-58
SLIDE 58

Mutation Testing for Properties

✎ Test whether wrong properties are not

implemented

✎ Coupling effect ✎ If mutants are not implemented, complex

wrong properties are not implemented either

slide-59
SLIDE 59

Property Mutation Example Property

G ✭x ✦ X y✮

slide-60
SLIDE 60

Property Mutation Example Property

G ✭x ✦ X y✮

Example mutants

✎ LRO: G ✭x ✥ X y✮

slide-61
SLIDE 61

Property Mutation Example Property

G ✭x ✦ X y✮

Example mutants

✎ LRO: G ✭x ✥ X y✮ ✎ ORO: G ✭x ✦ X x ✮

slide-62
SLIDE 62

Property Mutation Example Property

G ✭x ✦ X y✮

Example mutants

✎ LRO: G ✭x ✥ X y✮ ✎ ORO: G ✭x ✦ X x ✮ ✎ ...

slide-63
SLIDE 63

Mutation Operators for LTL

G ✭x ✦ X y✮

slide-64
SLIDE 64

Mutation Operators for LTL

G ✭x ✦ X y✮

Temporal Operator Insertion

G ✭ X x ✦ X y✮

slide-65
SLIDE 65

Mutation Operators for LTL

G ✭x ✦ X y✮

Temporal Operator Insertion

G ✭ X x ✦ X y✮

Temporal Operator Replacement

G ✭x ✦ G y✮

slide-66
SLIDE 66

Mutation Operators for LTL

G ✭x ✦ X y✮

Temporal Operator Insertion

G ✭ X x ✦ X y✮

Temporal Operator Replacement

G ✭x ✦ G y✮

Missing Temporal Operator

G ✭x ✦ y✮

slide-67
SLIDE 67

Property Mutation Summary

✎ A mutant of a property is a trap property ✎ Each property results in several mutants for

each mutation operator

✎ Test case generation: Create mutants for all

properties

✎ Property mutation measures sensitivity wrt

implemented properties

slide-68
SLIDE 68

Case Study

✎ PICC and Mutation add new, previously

uncovered test predicates

✎ ✎ ✎ ✎

slide-69
SLIDE 69

Case Study

✎ PICC and Mutation add new, previously

uncovered test predicates

✎ 25% infeasible test predicates average ✎ 45% equivalent property mutants ✎ ✎

slide-70
SLIDE 70

Case Study

✎ PICC and Mutation add new, previously

uncovered test predicates

✎ 25% infeasible test predicates average ✎ 45% equivalent property mutants ✎ 16% of literals are vacuously satisfied ✎ Mutation operators can be optimized to

reduce equivalent mutants

slide-71
SLIDE 71

Summary

✎ Test in addition to verification ✎ ✎ ✎ ✎ ✎

slide-72
SLIDE 72

Summary

✎ Test in addition to verification ✎ Derive tests from properties in addition to

specifications

✎ ✎ ✎ ✎

slide-73
SLIDE 73

Summary

✎ Test in addition to verification ✎ Derive tests from properties in addition to

specifications

✎ Test inactive clauses in addition to active

clauses

✎ ✎ ✎

slide-74
SLIDE 74

Summary

✎ Test in addition to verification ✎ Derive tests from properties in addition to

specifications

✎ Test inactive clauses in addition to active

clauses

✎ Test that wrong properties are not

implemented, in addition to testing that right properties are implemented

✎ ✎

slide-75
SLIDE 75

Summary

✎ Test in addition to verification ✎ Derive tests from properties in addition to

specifications

✎ Test inactive clauses in addition to active

clauses

✎ Test that wrong properties are not

implemented, in addition to testing that right properties are implemented

✎ Thank you for your attention! ✎ Questions?