Applying Random Testing to a Base Type Environment Experience - PowerPoint PPT Presentation

Applying Random Testing to a Base Type Environment Experience Report Vincent St-Amour Neil Toronto PLT ICFP 2013 - September 27th, 2013

Γ base ⊢ e : τ

Γ base ⊢ e : τ

Γ base ⊢ e : τ

Γ base ⊢ e : τ

Γ base ⊢ e : τ

first : (Listof A) → A Γ base ⊢ e : τ string-append : String * → String

+ : Γ base ⊢ e : τ

+ : /* This macro is used to implement most all Γ base ⊢ e : τ binary math and comparison functions (!): */ #define GEN_BIN_THING(rettype, name, scheme_name, \ iop, fop, fsop, bn_op, rop, wrap, combineinf, \ ...

+ : Γ base ⊢ e : τ

+ : 18% of Typed Racket bugs Γ base ⊢ e : τ 10.9% numeric Γ { base 6.8% other Γ base

+ : 18% of Typed Racket bugs Γ base ⊢ e : τ 10.9% numeric Γ { base 6.8% other Γ base

+ : Γ base ⊢ e : τ Use random testing

What do these bugs look like? How do we fi nd them? How well did that work?

What do these bugs look like?

A Type Environment Bug (: sinh (case → [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...))

A Type Environment Bug τ₁ ∩ τ₂ ∩ … (: sinh (case → [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...))

A Type Environment Bug τ₁ ∩ τ₂ ∩ … (: sinh (case → [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] 10 10 10 10 10 10 10 10 10 ...)) 5 5 5 5 5 5 5 5 5 0 0 0 0 0 0 0 0 0 -5 -5 -5 -5 -5 -5 -5 -5 -5 -10 -10 -10 -10 -10 -10 -10 -10 -10 -2 -2 -2 -2 -2 -2 0 0 0 0 0 0 2 2 2 2 2 2 -2 -2 -2 0 0 0 2 2 2

A Type Environment Bug τ₁ ∩ τ₂ ∩ … (: sinh (case → [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] 10 10 10 10 10 10 10 10 10 ...)) 5 5 5 5 5 5 5 5 5 0 0 0 0 0 0 0 0 0 19 cases -5 -5 -5 -5 -5 -5 -5 -5 -5 -10 -10 -10 -10 -10 -10 -10 -10 -10 -2 -2 -2 -2 -2 -2 0 0 0 0 0 0 2 2 2 2 2 2 -2 -2 -2 0 0 0 2 2 2 (integers, complexes, exact rationals)

A Type Environment Bug (: sinh (case → [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...))

A Type Environment Bug (: sinh (case → [Float-Zero → Float-Zero] [Positive-Float → Positive-Float] [Negative-Float → Negative-Float] ...)) .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 (sinh 1.2535e-17) 0 0 0 0 0 0 0 0 0 ⇒ 0.0 : Float-Zero -.0002 -.0002 -.0002 -.0002 -.0002 -.0002 -.0002 -.0002 -.0002 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0002 -.0002 -.0002 0 0 0 .0002 .0002 .0002 .0004 .0004 .0004 -.0004 -.0004 -.0004 -.0002 -.0002 -.0002 0 0 0 .0002 .0002 .0002 .0004 .0004 .0004 -.0004 -.0004 -.0004 -.0002 -.0002 -.0002 0 0 0 .0002 .0002 .0002 .0004 .0004 .0004

A Type Environment Bug (: sinh (case → [Float-Zero → Float-Zero] [Nonnegative-Float → Nonnegative-Float] [Nonpositive-Float → Nonpositive-Float] ...)) .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0004 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 .0002 (sinh 1.2535e-17) 0 0 0 0 0 0 0 0 0 ⇒ 0.0 : Float-Zero -.0002 -.0002 -.0002 -.0002 -.0002 -.0002 -.0002 -.0002 -.0002 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0004 -.0002 -.0002 -.0002 0 0 0 .0002 .0002 .0002 .0004 .0004 .0004 -.0004 -.0004 -.0004 -.0002 -.0002 -.0002 0 0 0 .0002 .0002 .0002 .0004 .0004 .0004 -.0004 -.0004 -.0004 -.0002 -.0002 -.0002 0 0 0 .0002 .0002 .0002 .0004 .0004 .0004

A Type Environment Bug (: * (case → ... [Positive-Real Positive-Real → Positive-Real] ...))

A Type Environment Bug (: * (case → ... [Positive-Real Positive-Real → Positive-Real] ...)) (* 5/1241 4.9406564584125e-324) ⇒ 0.0 : Float-Zero

A Type Environment Bug (: * (case → ... [Nonnegative-Real Nonnegative-Real → Nonnegative-Real] ...)) (* 5/1241 4.9406564584125e-324) ⇒ 0.0 : Float-Zero

A Type Environment Bug (: * (case → ... [Nonnegative-Real Nonnegative-Real → Nonnegative-Real] ...)) (* 5/1241 4.9406564584125e-324) ⇒ 0.0 : Float-Zero (* +inf.0 0.0) ⇒ +nan.0 : Float-Nan

A Type Environment Bug (: * (case → ... [Nonnegative-Real Nonnegative-Real → (U Nonnegative-Real Float-Nan)] ...)) (* 5/1241 4.9406564584125e-324) ⇒ 0.0 : Float-Zero (* +inf.0 0.0) ⇒ +nan.0 : Float-Nan

How do we fi nd them?

Use random testing PLT Redex

PLT Redex (define-language λ v [e (e e ...) (if0 e e e) x v] [v ( λ (x ...) e) number] [x (variable-except λ if0)])

PLT Redex (define red (reduction-relation λ v ...)) (define-language λ v [e (e e ...) (if0 e e e) x v] [v ( λ (x ...) e) number] [x (variable-except λ if0)])

PLT Redex (define-language λ v (redex-check λ v v [e (e e ...) (if0 e e e) (number? (term v))) x v] counterexample found [v ( λ (x ...) e) after 4 attempts: number] ( λ () 1) [x (variable-except λ if0)])

PLT Redex (define-language λ v (redex-check λ v v [e (e e ...) (> (n-google-results (if0 e e e) (term v)) x 20)) v] counterexample found [v ( λ (x ...) e) after 15 attempts: number] ( λ (x y) (+ ( λ () 3) 2)) [x (variable-except λ if0)])

Testing Type Preservation e ::= n | (+ e e) | ... Generate arithmetic expressions

Testing Type Preservation e ::= n | (+ e e) | ... e ⊬ Γ ⊢ e : τ Typecheck using Typed Racket

Testing Type Preservation e ::= n | (+ e e) | ... e ⊬ Γ ⊢ e : τ e e → * v Evaluate using Typed Racket

Testing Type Preservation e ::= n | (+ e e) | ... e ⊬ Γ ⊢ e : τ e e → * v v Γ ⊢ v : τ ' Typecheck the result

Testing Type Preservation e ::= n | (+ e e) | ... e ⊬ Γ ⊢ e : τ e e → * v τ v Γ ⊢ v : τ ' τ ' ≮ : <: τ ' <: τ Check consistency

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e ⊬ Γ ⊢ e : τ e e → * v τ v Γ ⊢ v : τ ' τ ' ≮ : <: τ ' <: τ

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e ⊬ Γ ⊢ e : τ Positive-Float e e → * v τ v Γ ⊢ v : τ ' τ ' ≮ : <: τ ' <: τ

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e ⊬ Γ ⊢ e : τ Positive-Float e 0.0 e → * v τ v Γ ⊢ v : τ ' τ ' ≮ : <: τ ' <: τ

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e ⊬ Γ ⊢ e : τ Positive-Float e 0.0 e → * v τ v Γ ⊢ v : τ ' Float-Zero τ ' ≮ : <: τ ' <: τ

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e ⊬ Γ ⊢ e : τ Positive-Float e 0.0 e → * v τ v Γ ⊢ v : τ ' Float-Zero τ ' ≮ : <: τ ' <: τ Float-Zero ≮ : Positive-Float

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e ⊬ Γ ⊢ e : τ Positive-Float e 0.0 e → * v τ v Γ ⊢ v : τ ' Float-Zero τ ' ≮ : <: τ ' <: τ Float-Zero ≮ : Positive-Float

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e ⊬ Γ ⊢ e : τ Positive-Float e 0.0 e → * v τ v Γ ⊢ v : τ ' Float-Zero τ ' ≮ : <: τ ' <: τ Float-Zero ≮ : Positive-Float

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e 57.6% initial rejection rate ⊬ Γ ⊢ e : τ Positive-Float 1.6% after grammar engineering e 0.0 e → * v τ v Γ ⊢ v : τ ' Float-Zero τ ' ≮ : <: τ ' <: τ Float-Zero ≮ : Positive-Float

Testing Type Preservation e ::= n | (+ e e) | ... (sinh 1.2535e-17) e ⊬ Γ ⊢ e : τ Positive-Float e 0.0 e → * v τ v Γ ⊢ v : τ ' Float-Zero τ ' ≮ : <: τ ' <: τ Float-Zero ≮ : Positive-Float

Testing Type Preservation Random fl oating-point number generation 25% Laplace distribution 8.75% Close to - ∞ 17.5% Close to 0 5% - ∞ 25% Uniform random bits 5% ∞ 8.75% Close to ∞ 5% NaN

Testing Type Preservation Random fl oating-point number generation 25% Laplace distribution 8.75% Close to - ∞ 17.5% Close to 0 5% - ∞ 25% Uniform random bits 5% ∞ 8.75% Close to ∞ 5% NaN

How well did that work?

Finding Bugs • Existing 10+ kloc test suite • Found bugs anyway • Small random test cases • Smaller than user bug reports • Even without test case reduction

Con fi dence When Refactoring • Fact: programs evolve over time • Follow changes with random testing Success stories • NaN refactoring • Optimizer rewrite

The Take-Away Type environments have bugs too! Random testing can help. Redex makes random testing easy.

The Take-Away Type environments have bugs too! Random testing can help. Redex makes random testing easy. Thank You