Combining Verification and Conformance Testing for Validating - - PowerPoint PPT Presentation

▶
combining verification and conformance testing for
SMART_READER_LITE
LIVE PREVIEW

Combining Verification and Conformance Testing for Validating - - PowerPoint PPT Presentation

Apr 16, 2023 235 likes •633 views

Combining Verification and Conformance Testing for Validating Reactive Systems Vlad Rusu, Thierry Jron, and Herv Marchand First.Last@irisa.fr IRISA/INRIA Rennes, project Vertecs http://www.irisa.fr/vertecs Combining verification and

slide-1
SLIDE 1

Combining Verification and Conformance Testing for Validating Reactive Systems

Vlad Rusu, Thierry Jéron, and Hervé Marchand

First.Last@irisa.fr

IRISA/INRIA Rennes, project Vertecs http://www.irisa.fr/vertecs

Combining verification and conformance testing - ETR’05 – p.1/38

slide-2
SLIDE 2

Verification

Properties: P Specification: S S | = P

Combining verification and conformance testing - ETR’05 – p.2/38

slide-3
SLIDE 3

Conformance testing

Implementation: I

Specification: S I ioco S

Combining verification and conformance testing - ETR’05 – p.3/38

slide-4
SLIDE 4

Model-based development

Implementation: I Testing Verification Properties: P Specification: S

Combining verification and conformance testing - ETR’05 – p.4/38

slide-5
SLIDE 5

Model-based development

Implementation: I Testing Verification Properties: P Specification: S Verification?

Combining verification and conformance testing - ETR’05 – p.5/38

slide-6
SLIDE 6

Model-based development

Implementation: I Verification Properties: P Specification: S Testing Testing

Combining verification and conformance testing - ETR’05 – p.6/38

slide-7
SLIDE 7

Outline

verification conformance testing combining the two: consistency? test generation.

Combining verification and conformance testing - ETR’05 – p.7/38

slide-8
SLIDE 8

Verification

Properties: P Specification: S S | = P

properties: safety, liveness . . . specification: SDL, Statecharts, extended automata . . .

Combining verification and conformance testing - ETR’05 – p.8/38

slide-9
SLIDE 9

Verification techniques

model checking: automatic, finite-state, exact theorem proving: interactive, infinite-state, exact abstract interpretation: automatic, infinite-state, approximated (conservative)

Combining verification and conformance testing - ETR’05 – p.9/38

slide-10
SLIDE 10

Sample specification

MSG!(m) m = x ∧ x > 0 x := x − 1 x = 0 STOP! START?(p) x := p

Combining verification and conformance testing - ETR’05 – p.10/38

slide-11
SLIDE 11

Property: after START?(p) with p > 0, no STOP! without MSG!(m) in between

STOP! MSG!(m) Violate p > 0 START?(p)

∗ ∗

START?(p) MSG!(m) m = x ∧ x > 0 x := x − 1 x = 0 x := p STOP!

Combining verification and conformance testing - ETR’05 – p.11/38

slide-12
SLIDE 12

Verifying Property on Specification

check reachability of Violate in synchronous product (Spec × observer for property) undecidable (abstract interpretation: conservative) here, property is proved, but not proved if p ≥ 0 only.

Combining verification and conformance testing - ETR’05 – p.12/38

slide-13
SLIDE 13

Outline

verification conformance testing conformance relation test generation and execution combining the two: consistency? test generation.

Combining verification and conformance testing - ETR’05 – p.13/38

slide-14
SLIDE 14

Conformance testing

Implementation: I

Specification: S I ioco S

ioco ≅ after all traces of S, outputs of I ⊆ outputs of S.

Combining verification and conformance testing - ETR’05 – p.14/38

slide-15
SLIDE 15

Conformance and non-conformance

ST ART ?(0) ST OP ! I1ioco S ST ART ?(0) MSG!(0) ¬(I2ioco S) ST ART ?(0) MSG!(0) ST ART ?(0) I3ioco S x = 0 ST OP ! ST ART ?(p) x := p S m = x ∧ x > 0 MSG!(m) x := x − 1

Combining verification and conformance testing - ETR’05 – p.15/38

slide-16
SLIDE 16

Conformance?

MSG!(m) m = x ∧ x > 0 x := x − 1 START?(p) x := p x = 0 STOP! I S Quiet... START?(0)

Combining verification and conformance testing - ETR’05 – p.16/38

slide-17
SLIDE 17

Blocking and suspension

blocking δ!: deadlock, or waiting for input δ! observed on black-box I (using timers) computed on S: suspension operation δ(·) Traces(δ(S)) = Traces(S) + δ! when blocked.

Combining verification and conformance testing - ETR’05 – p.17/38

slide-18
SLIDE 18

Non-Conformance!

MSG!(m) m = x ∧ x > 0 x := x − 1 δ! x < 0 δ! START?(p) x := p x = 0 STOP! δ! δ(I) δ! δ(S) START?(0)

Combining verification and conformance testing - ETR’05 – p.18/38

slide-19
SLIDE 19

Conformance relation

I ioco S Traces(δ(I)) ∩ Traces(δ(S)) · (Λ!

S ∪ {δ}) ⊆ Traces(δ(S)).

“After all traces of δ(S), outputs of δ(I) ⊆ outputs of δ(S)”

Combining verification and conformance testing - ETR’05 – p.19/38

slide-20
SLIDE 20

Test generation & execution

Specification S Test Purposes Implementation I Verdicts (e.g., Fail, Pass, Inconclusive) Test Generation Test Execution Test Cases selects ioco

Combining verification and conformance testing - ETR’05 – p.20/38

slide-21
SLIDE 21

Test Purpose and Test Case

STOP? Test Case START!(0) START?(p) STOP! ∗ Test Purpose Pass Fail MSG?(m) δ?

Combining verification and conformance testing - ETR’05 – p.21/38

slide-22
SLIDE 22

Outline

verification conformance testing combining the two: consistency guide implementation towards property violation deal with “imperfect” verification test generation.

Combining verification and conformance testing - ETR’05 – p.22/38

slide-23
SLIDE 23

Using P in test generation

Specification S Safety Properties P Implementation I Verdicts (e.g., Fail, Violate, Inconclusive) Test Execution Test Cases | = Test Generation Verification/ ioco

Combining verification and conformance testing - ETR’05 – p.23/38

slide-24
SLIDE 24

Problem: S | = P undecidable

try to prove S | = P using abstract interpretation even if S | = P could not be proved, generate test case that correctly detects ¬ I ioco S, I | = P, or S | = P use symbolic techniques for test generation as well.

Combining verification and conformance testing - ETR’05 – p.24/38

slide-25
SLIDE 25

Outline

verification conformance testing combining the two: consistency? test generation: first define a canonical tester for S and ioco then compute product with observer for property finally, analyse product, eliminate irrelevant parts.

Combining verification and conformance testing - ETR’05 – p.25/38

slide-26
SLIDE 26

Canonical tester: determinisation

Determinisation: det(·) det(S) is deterministic Traces(det(S)) = Traces(S).

Combining verification and conformance testing - ETR’05 – p.26/38

slide-27
SLIDE 27

Canonical tester:

  • utput-completion

Output-completion: Σ!(·) add a new location: Fail & transitions to it on each missing output notation: canon(S) = Σ!(det(δ(S)))

Combining verification and conformance testing - ETR’05 – p.27/38

slide-28
SLIDE 28

canon(S)

MSG!(m) m = x ∧ x > 0 x := x − 1 δ! x < 0 δ! δ! START?(p) x := p x = 0 STOP!

Fail

*! *! *!

canon(S) = canonical tester [Brinksma] for S and ioco. ioco-conformance to S is a safety property!

Combining verification and conformance testing - ETR’05 – p.28/38

slide-29
SLIDE 29

Test generation: product with

  • bserver

For observer (ω, Violateω) let test(S, ω) = ω||canon(S). Three possible sets of violating locations: ViolateFail = Violateω × {Fail} Fail = Violateω × {Fail} Violate = Violateω × {Fail}

Combining verification and conformance testing - ETR’05 – p.29/38

slide-30
SLIDE 30

The ViolateFail verdict

δ(I) | = (test(S, ω), ViolateFail) ⇒ ¬(Iioco S) ∧ δ(I) | = (ω, Violateω) Implementation violates both property and conformance

Combining verification and conformance testing - ETR’05 – p.30/38

slide-31
SLIDE 31

The Fail verdict

δ(I) | = (test(S, ω), Fail) = ⇒ ¬(I ioco S) Implementation violate conformance only

Combining verification and conformance testing - ETR’05 – p.31/38

slide-32
SLIDE 32

The Violate verdict

δ(I) | = (test(S, ω), Violate) ⇒ δ(S), δ(I) | = (ω, Violateω) Specification & implementation violate property

Combining verification and conformance testing - ETR’05 – p.32/38

slide-33
SLIDE 33

Specification and property

MSG!(m) m = x ∧ x > 0 x := x − 1 δ! x < 0 δ! δ! START?(p) x := p x = 0 STOP! STOP! MSG!(m) Violate p ≥ 0 START?(p)

∗ ∗

Combining verification and conformance testing - ETR’05 – p.33/38

slide-34
SLIDE 34

Resulting test case

x = 0 STOP? x = 0 STOP? x := x − 1 MSG?(m) m = x ∧ x > 0 m = x ∧ x > 0 MSG?(m) x := x − 1 Violate δ? x < 0 δ? x < 0 δ? Violate x = 0 Fail Fail STOP? ?* ?* ?* ?* p ≥ 0 x := p START!(p)

Combining verification and conformance testing - ETR’05 – p.34/38

slide-35
SLIDE 35

Simplifying a test case

Eliminate states from which property cannot be violated (from which Violate and ViolateFail are not reachable). abstract interpretation strikes again! coreachability analysis to Violate and ViolateFail ❀ constraints in each transition strengthen guards of transitions with new constraints

  • nly non-coreachable states are eliminated (but some

may remain).

Combining verification and conformance testing - ETR’05 – p.35/38

slide-36
SLIDE 36

Simplified test case

Fail *? *? START!(p) x := 0 p ≥ 0 STOP? x = 0 Violate Fail Violate x > 0 STOP? MSG?(m) Inconc

Combining verification and conformance testing - ETR’05 – p.36/38

slide-37
SLIDE 37

Conclusion

combination of verification and conformance testing automatic, symbolic methods testing provides correct verdicts, regardless of success in verification step test case “guides” implementation towards violating property. Only one among many approaches!

Combining verification and conformance testing - ETR’05 – p.37/38

slide-38
SLIDE 38

Questions?

Combining verification and conformance testing - ETR’05 – p.38/38