Applying Random Testing to a Base Type Environment Experience - - PowerPoint PPT Presentation

▶
applying random testing to a base type environment
SMART_READER_LITE
LIVE PREVIEW

Applying Random Testing to a Base Type Environment Experience - - PowerPoint PPT Presentation

Nov 11, 2022 180 likes •740 views

Applying Random Testing to a Base Type Environment Experience Report Vincent St-Amour Neil Toronto PLT ICFP 2013 - September 27th, 2013 base e : base e : base e : base e : base e : first :

slide-1
SLIDE 1

Applying Random Testing to a Base Type Environment Experience Report Vincent St-Amour Neil Toronto PLT

ICFP 2013 - September 27th, 2013

slide-2
SLIDE 2

Γ

base ⊢ e : τ

slide-3
SLIDE 3

Γ

base ⊢ e : τ

slide-4
SLIDE 4

Γ

base ⊢ e : τ

slide-5
SLIDE 5

Γ

base ⊢ e : τ

slide-6
SLIDE 6

Γ

base ⊢ e : τ

slide-7
SLIDE 7

Γ

base ⊢ e : τ first : (Listof A) → A string-append : String * → String

slide-8
SLIDE 8

Γ

base ⊢ e : τ

+ :

slide-9
SLIDE 9

Γ

base ⊢ e : τ

+ :

/* This macro is used to implement most all binary math and comparison functions (!): */ #define GEN_BIN_THING(rettype, name, scheme_name, \ iop, fop, fsop, bn_op, rop, wrap, combineinf, \ ...

slide-10
SLIDE 10

Γ

base ⊢ e : τ

+ :

slide-11
SLIDE 11

Γ

base ⊢ e : τ

+ :

18% of Typed Racket bugs

{

10.9% numeric Γ

base

6.8% other Γ

base

slide-12
SLIDE 12

Γ

base ⊢ e : τ

+ :

18% of Typed Racket bugs

{

10.9% numeric Γ

base

6.8% other Γ

base

slide-13
SLIDE 13

Γ

base ⊢ e : τ

+ :

Use random testing

slide-14
SLIDE 14

What do these bugs look like? How do we find them? How well did that work?

slide-15
SLIDE 15

What do these bugs look like?

slide-16
SLIDE 16

A Type Environment Bug

(: sinh (case→ [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...))

slide-17
SLIDE 17

A Type Environment Bug

(: sinh (case→ [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...)) τ₁ ∩ τ₂ ∩ …

slide-18
SLIDE 18

A Type Environment Bug

(: sinh (case→ [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...)) τ₁ ∩ τ₂ ∩ …

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2

2 2 2 2 2 2 2 2 2

  • 10
  • 10
  • 10
  • 10
  • 10
  • 10
  • 10
  • 10
  • 10
  • 5
  • 5
  • 5
  • 5
  • 5
  • 5
  • 5
  • 5
  • 5

5 5 5 5 5 5 5 5 5 10 10 10 10 10 10 10 10 10

slide-19
SLIDE 19

A Type Environment Bug

(: sinh (case→ [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...)) τ₁ ∩ τ₂ ∩ …

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2

2 2 2 2 2 2 2 2 2

  • 10
  • 10
  • 10
  • 10
  • 10
  • 10
  • 10
  • 10
  • 10
  • 5
  • 5
  • 5
  • 5
  • 5
  • 5
  • 5
  • 5
  • 5

5 5 5 5 5 5 5 5 5 10 10 10 10 10 10 10 10 10

19 cases

(integers, complexes, exact rationals)

slide-20
SLIDE 20

A Type Environment Bug

(: sinh (case→ [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...))

slide-21
SLIDE 21

A Type Environment Bug

(: sinh (case→ [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...))

  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002

.0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004

  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002

.0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004

(sinh 1.2535e-17) ⇒ 0.0 : Float-Zero

slide-22
SLIDE 22

A Type Environment Bug

(: sinh (case→ [Float-Zero → Float-Zero] [Nonnegative-Float → Nonnegative-Float] [Nonpositive-Float → Nonpositive-Float] ...))

  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002

.0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004

  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0004
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002
  • .0002

.0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004

(sinh 1.2535e-17) ⇒ 0.0 : Float-Zero

slide-23
SLIDE 23

A Type Environment Bug

(: * (case→ ... [Positive-Real Positive-Real → Positive-Real] ...))

slide-24
SLIDE 24

A Type Environment Bug

(: * (case→ ... [Positive-Real Positive-Real → Positive-Real] ...))

(* 5/1241 4.9406564584125e-324) ⇒ 0.0 : Float-Zero

slide-25
SLIDE 25

A Type Environment Bug

(: * (case→ ... [Nonnegative-Real Nonnegative-Real → Nonnegative-Real] ...))

(* 5/1241 4.9406564584125e-324) ⇒ 0.0 : Float-Zero

slide-26
SLIDE 26

A Type Environment Bug

(: * (case→ ... [Nonnegative-Real Nonnegative-Real → Nonnegative-Real] ...))

(* 5/1241 4.9406564584125e-324) ⇒ 0.0 : Float-Zero (* +inf.0 0.0) ⇒ +nan.0 : Float-Nan

slide-27
SLIDE 27

A Type Environment Bug

(: * (case→ ... [Nonnegative-Real Nonnegative-Real → (U Nonnegative-Real Float-Nan)] ...))

(* 5/1241 4.9406564584125e-324) ⇒ 0.0 : Float-Zero (* +inf.0 0.0) ⇒ +nan.0 : Float-Nan

slide-28
SLIDE 28

How do we find them?

slide-29
SLIDE 29

Use random testing

PLT Redex

slide-30
SLIDE 30

PLT Redex

(define-language λv [e (e e ...) (if0 e e e) x v] [v (λ (x ...) e) number] [x (variable-except λ if0)])

slide-31
SLIDE 31

PLT Redex

(define-language λv [e (e e ...) (if0 e e e) x v] [v (λ (x ...) e) number] [x (variable-except λ if0)]) (define red (reduction-relation λv ...))

slide-32
SLIDE 32

PLT Redex

(define-language λv [e (e e ...) (if0 e e e) x v] [v (λ (x ...) e) number] [x (variable-except λ if0)]) (redex-check λv v (number? (term v))) counterexample found after 4 attempts: (λ () 1)

slide-33
SLIDE 33

PLT Redex

(define-language λv [e (e e ...) (if0 e e e) x v] [v (λ (x ...) e) number] [x (variable-except λ if0)]) (redex-check λv v (> (n-google-results (term v)) 20)) counterexample found after 15 attempts: (λ (x y) (+ (λ () 3) 2))

slide-34
SLIDE 34

Testing Type Preservation

e ::= n | (+ e e) | ... Generate arithmetic expressions

slide-35
SLIDE 35

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ Typecheck using Typed Racket e ⊬

slide-36
SLIDE 36

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Evaluate using Typed Racket e ⊬ e

slide-37
SLIDE 37

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' Typecheck the result e ⊬ e v

slide-38
SLIDE 38

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ Check consistency τ e ⊬ e v τ' <: ≮:

slide-39
SLIDE 39

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) e ⊬ e v τ' <: ≮:

slide-40
SLIDE 40

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) Positive-Float e ⊬ e v τ' <: ≮:

slide-41
SLIDE 41

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) Positive-Float 0.0 e ⊬ e v τ' <: ≮:

slide-42
SLIDE 42

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) Positive-Float 0.0 Float-Zero e ⊬ e v τ' <: ≮:

slide-43
SLIDE 43

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) Positive-Float 0.0 Float-Zero Float-Zero ≮: Positive-Float e ⊬ e v τ' <: ≮:

slide-44
SLIDE 44

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) Positive-Float 0.0 Float-Zero Float-Zero ≮: Positive-Float e ⊬ e v τ' <: ≮:

slide-45
SLIDE 45

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) Positive-Float 0.0 Float-Zero Float-Zero ≮: Positive-Float e ⊬ e v τ' <: ≮:

slide-46
SLIDE 46

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) Positive-Float 0.0 Float-Zero Float-Zero ≮: Positive-Float e ⊬ e v τ' <: ≮:

57.6% initial rejection rate 1.6% after grammar engineering

slide-47
SLIDE 47

Testing Type Preservation

e ::= n | (+ e e) | ... Γ ⊢ e : τ e →* v Γ ⊢ v : τ' τ' <: τ τ (sinh 1.2535e-17) Positive-Float 0.0 Float-Zero Float-Zero ≮: Positive-Float e ⊬ e v τ' <: ≮:

slide-48
SLIDE 48

Testing Type Preservation

Random floating-point number generation 25% Laplace distribution 25% Uniform random bits 17.5% Close to 0 8.75% Close to ∞ 8.75% Close to -∞ 5% ∞ 5% -∞ 5% NaN

slide-49
SLIDE 49

Testing Type Preservation

Random floating-point number generation 25% Laplace distribution 25% Uniform random bits 17.5% Close to 0 8.75% Close to ∞ 8.75% Close to -∞ 5% ∞ 5% -∞ 5% NaN

slide-50
SLIDE 50

How well did that work?

slide-51
SLIDE 51

Finding Bugs

  • Existing 10+ kloc test suite
  • Found bugs anyway
  • Small random test cases
  • Smaller than user bug reports
  • Even without test case reduction
slide-52
SLIDE 52

Confidence When Refactoring

  • Fact: programs evolve over time
  • Follow changes with random testing

Success stories

  • NaN refactoring
  • Optimizer rewrite
slide-53
SLIDE 53

The Take-Away Type environments have bugs too! Random testing can help. Redex makes random testing easy.

slide-54
SLIDE 54

The Take-Away Type environments have bugs too! Random testing can help. Redex makes random testing easy. Thank You