Finding Code That Explodes Under Symbolic Evalua<on James - - PowerPoint PPT Presentation

Finding Code That Explodes Under Symbolic Evalua<on

James Bornholt Emina Torlak

University of Washington

unsat.org

Automated reasoning tools help us solve hard programming problems

Automated reasoning tools help us solve hard programming problems

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

Verifica8on

Automated reasoning tools help us solve hard programming problems

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

How do I compile code
 for this weird new architecture? [PLDI’14] 

Verifica8on Synthesis

Automated reasoning tools help us solve hard programming problems

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

How do I compile code
 for this weird new architecture? [PLDI’14]  How do I teach kids
 the rules of algebra effec8vely? [VMCAI’18]



Verifica8on Synthesis “Programs”

Symbolic evaluators

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

How do I compile code
 for this weird new architecture? [PLDI’14] 

Symbolic evaluators

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

How do I compile code
 for this weird new architecture? [PLDI’14]  Interpreter for file system

• pera8ons

Interpreter for new architecture instruc8ons

Symbolic evaluators

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

How do I compile code
 for this weird new architecture? [PLDI’14] 

Symbolic evaluator

Sketch, RoseWe, …

Interpreter for file system

• pera8ons

Interpreter for new architecture instruc8ons

Symbolic evaluators

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

How do I compile code
 for this weird new architecture? [PLDI’14] 

Symbolic evaluator

Sketch, RoseWe, …

Verifica8on Synthesis Angelic Execu8on for free! Interpreter for file system

• pera8ons

Interpreter for new architecture instruc8ons

Symbolic evaluators: no free lunch

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

Symbolic evaluator

Sketch, RoseWe, …

Verifica8on Synthesis Angelic Execu8on for free! Interpreter for file system

• pera8ons

Symbolic evaluators: no free lunch

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

How do you make these tools scale?

Symbolic evaluator

Sketch, RoseWe, …

Verifica8on Synthesis Angelic Execu8on for free! Interpreter for file system

• pera8ons

Symbolic evaluators: no free lunch

Does my program s8ll work a:er the file system crashes? [ASPLOS’16]

👸

Searching all paths through the interpreter Searching all paths through the interpreter

How do you make these tools scale?

Symbolic evaluator

Sketch, RoseWe, …

Verifica8on Synthesis Angelic Execu8on for free! Interpreter for file system

• pera8ons

Symbolic profiling iden<fies performance issues in symbolic evalua<on

Symbolic profiling iden<fies performance issues in symbolic evalua<on

Symbolic profiling

Data structures and analyses

Symbolic profiling iden<fies performance issues in symbolic evalua<on

Symbolic profiling

Data structures and analyses

Symbolic evalua8on an8-paWerns

Common issues and source-level repairs

10 20 1500

Symbolic profiling iden<fies performance issues in symbolic evalua<on

Symbolic profiling

Data structures and analyses

Symbolic evalua8on an8-paWerns

Common issues and source-level repairs

Empirical results

300× speedup on real-world tools

10 20 1500

Symbolic profiling iden<fies performance issues in symbolic evalua<on

Symbolic profiling

Data structures and analyses

Symbolic evalua8on an8-paWerns

Common issues and source-level repairs

Empirical results

300× speedup on real-world tools

10 20 1500

Symbolic evalua8on

All-paths execu8on of programs

if (…) { … } ∀x. φ(…, x)

Symbolic evalua<on

All-paths execu8on of programs

Symbolic evalua<on executes all paths through a program

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k))

Symbolic evalua<on executes all paths through a program

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k)) Inputs are unknown (trying to find values that violate spec)

Symbolic evalua<on executes all paths through a program

(filter even? ‘(x0 x1))

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k)) Inputs are unknown (trying to find values that violate spec)

Symbolic evalua<on executes all paths through a program

(filter even? ‘(x0 x1)) ‘() ‘(x0)

¬(even? x0) (even? x0)

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k)) Inputs are unknown (trying to find values that violate spec)

Symbolic evalua<on executes all paths through a program

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k)) Inputs are unknown (trying to find values that violate spec)

Symbolic evalua<on executes all paths through a program

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k)) Inputs are unknown (trying to find values that violate spec)

Symbolic evalua<on executes all paths through a program

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k))

‘() ‘()

k=0

Inputs are unknown (trying to find values that violate spec)

‘(x1) ‘() ‘(x0) ‘() ‘(x0) ‘(x0 x1)

k=0 k=1 k=0 k=1 k=0 k=1 k=2

Symbolic evalua<on executes all paths through a program

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k))

‘() ‘()

k=0

Inputs are unknown (trying to find values that violate spec) take runs 22 8mes

‘(x1) ‘() ‘(x0) ‘() ‘(x0) ‘(x0 x1)

k=0 k=1 k=0 k=1 k=0 k=1 k=2

Symbolic evalua<on executes all paths through a program

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

#lang rosette (define (first-k-even lst k) (define xs (filter even? lst)) (take xs k))

‘() ‘()

k=0

Inputs are unknown (trying to find values that violate spec) take runs 22 8mes because filter ran

• n a list of size 2

‘(x1) ‘() ‘(x0) ‘() ‘(x0) ‘(x0 x1)

k=0 k=1 k=0 k=1 k=0 k=1 k=2

Blaming filter even though it’s not the slowest

Symbolic profiling

Data structures and metrics

Two data structures to summarize symbolic evalua<on

‘() ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

(even? x0) ∧ ¬ ∧ ∧ ∧ ¬ (even? x1)

Symbolic evalua<on graph Reflects the evaluator’s strategy
 for all-paths execu8on of the program Symbolic heap Shape of all symbolic values
 created by the program

Any symbolic evalua<on technique can be summarized by these two data structures

The symbolic evalua4on graph summarizes branching and merging

Symbolic evalua<on graph

• Nodes are program states
• Edges are transi8ons

between states

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

‘() ‘()

k=0

‘(x1) ‘() ‘(x0) ‘() ‘(x0) ‘(x0 x1)

k=0 k=1 k=0 k=1 k=0 k=1 k=2

The symbolic evalua4on graph summarizes branching and merging

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

The symbolic evalua4on graph summarizes branching and merging

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

Symbolic execu8on

The symbolic evalua4on graph summarizes branching and merging

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

Symbolic execu8on

(filter even? ‘(x0 x1)) ‘() ‘(x0)

¬(even? x0) (even? x0)

Bounded model checking

The symbolic evalua4on graph summarizes branching and merging

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

Symbolic execu8on

(filter even? ‘(x0 x1)) ‘() ‘(x0)

¬(even? x0) (even? x0)

ys0

ys0 = (ite (even? x0) ‘() ‘(x0))

Bounded model checking

The symbolic evalua4on graph summarizes branching and merging

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

Symbolic execu8on

(filter even? ‘(x0 x1)) ‘() ‘(x0)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1)

ys0 ys0 ys1

ys0 = (ite (even? x0) ‘() ‘(x0)) ys1 = (append ys0 ‘(x1))

Bounded model checking

The symbolic evalua4on graph summarizes branching and merging

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

Symbolic execu8on

(filter even? ‘(x0 x1)) ‘() ‘(x0)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1)

ys0 ys0 ys1 ys2

ys0 = (ite (even? x0) ‘() ‘(x0)) ys1 = (append ys0 ‘(x1)) ys2 = (ite (even? x1) ys1 ys0)

Bounded model checking

The symbolic evalua4on graph summarizes branching and merging

(filter even? ‘(x0 x1)) ‘() ‘(x0) ‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1) (even? x1) ¬(even? x1)

Symbolic execu8on

(filter even? ‘(x0 x1)) ‘() ‘(x0)

¬(even? x0) (even? x0) (even? x1) ¬(even? x1)

ys0 ys0 ys1 ys2

ys0 = (ite (even? x0) ‘() ‘(x0)) ys1 = (append ys0 ‘(x1)) ys2 = (ite (even? x1) ys1 ys0)

Bounded model checking

More states, but more concrete Fewer states but less concrete

The symbolic heap shows how symbolic values are used

(even? x0) ∧ ¬ ∧ ∧ ∧ ¬ (even? x1)

Symbolic execu8on

Symbolic heap

• Nodes are symbolic terms
• Edges are sub-terms

The symbolic heap shows how symbolic values are used

(even? x0) ∧ ¬ ∧ ∧ ∧ ¬ (even? x1) (even? x0) (even? x1) ite ‘(x0) ‘() ‘(x1) append ite

ys0 = ys1 = ys2 =

Symbolic execu8on Bounded model checking

ys0 = (ite (even? x0) ‘() ‘(x0)) ys1 = (append ys0 ‘(x1)) ys2 = (ite (even? x1) ys1 ys0)

The symbolic heap shows how symbolic values are used

Only condi8ons in the heap Condi8ons and values (lists etc.)
 in the heap

(even? x0) ∧ ¬ ∧ ∧ ∧ ¬ (even? x1) (even? x0) (even? x1) ite ‘(x0) ‘() ‘(x1) append ite

ys0 = ys1 = ys2 =

Symbolic execu8on Bounded model checking

ys0 = (ite (even? x0) ‘() ‘(x0)) ys1 = (append ys0 ‘(x1)) ys2 = (ite (even? x1) ys1 ys0)

Analyzing symbolic data structures

For each procedure, measure metrics that summarize the evolu8on of the symbolic evalua8on graph and symbolic heap

Analyzing symbolic data structures

For each procedure, measure metrics that summarize the evolu8on of the symbolic evalua8on graph and symbolic heap Summarize metrics as a score to rank procedures in the program

Analyzing symbolic data structures

Symbolic evalua<on an<-paJerns

Common issues and repairs

Common an<-paJerns and repairs in symbolic evalua<on

Algorithmic mismatch


Algorithms or op8miza8ons poorly suited to symbolic evalua8on

Common an<-paJerns and repairs in symbolic evalua<on

Algorithmic mismatch


Algorithms or op8miza8ons poorly suited to symbolic evalua8on

(define (list-set lst idx val) (match lst [(cons x xs) (if (= idx 0) (cons val xs) (cons x (list-set xs (- idx 1) val))] [_ lst]))

Terminates early

• nce idx is found

Common an<-paJerns and repairs in symbolic evalua<on

Algorithmic mismatch


Algorithms or op8miza8ons poorly suited to symbolic evalua8on

(define (list-set lst idx val) (match lst [(cons x xs) (if (= idx 0) (cons val xs) (cons x (list-set xs (- idx 1) val))] [_ lst]))

Terminates early

• nce idx is found

Common an<-paJerns and repairs in symbolic evalua<on

Algorithmic mismatch


Algorithms or op8miza8ons poorly suited to symbolic evalua8on

(define (list-set lst idx val) (match lst [(cons x xs) (cons (if (= idx 0) val x) (list-set xs (- idx 1) val))] [_ lst]))

Common an<-paJerns and repairs in symbolic evalua<on

Algorithmic mismatch


Algorithms or op8miza8ons poorly suited to symbolic evalua8on

(define (list-set lst idx val) (match lst [(cons x xs) (cons (if (= idx 0) val x) (list-set xs (- idx 1) val))] [_ lst]))

Always recurse to the end of lst

Common an<-paJerns and repairs in symbolic evalua<on

Algorithmic mismatch


Algorithms or op8miza8ons poorly suited to symbolic evalua8on

(define (list-set lst idx val) (match lst [(cons x xs) (cons (if (= idx 0) val x) (list-set xs (- idx 1) val))] [_ lst]))

Always recurse to the end of lst

Time (sec)

5 10 15 20

Length

500 1000 1500 2000

Original Repaired

Common an<-paJerns and repairs in symbolic evalua<on

Algorithmic mismatch


Algorithms or op8miza8ons poorly suited to symbolic evalua8on

Irregular representa8on


Data structures of different shapes create different paths

Missed concre8za8on


Lost opportuni8es to exploit concrete values

Empirical results

Case studies and evalua8on

Three symbolic profilers

We developed two implementa8ons:

• The RoseJe solver-aided language (Racket)
• The Jalangi dynamic analysis framework (JavaScript)

Since publica8on, based on our work:

• The Crucible symbolic simula8on library (C, Java, …)

by Galois

Three symbolic profilers

We developed two implementa8ons:

• The RoseJe solver-aided language (Racket)
• The Jalangi dynamic analysis framework (JavaScript)

Since publica8on, based on our work:

• The Crucible symbolic simula8on library (C, Java, …)

by Galois

Today

Ac<onable: real-world bugs

Tool Speedup Type system soundness checker [POPL’18] 1.35× Refinement type checker for Ruby [VMCAI’18] 6× File-system crash consistency verifier [ASPLOS’16] 24× Cryptographic protocol verifier [FM’18] 29× SQL query verifier [CIDR’17] 75× Safety-cri8cal radiotherapy system verifier [CAV’16] 290×

Mul8ple patches accepted by developers Case studies on published RoseWe-based tools

Ac<onable: real-world bugs

Tool Speedup Type system soundness checker [POPL’18] 1.35× Refinement type checker for Ruby [VMCAI’18] 6× File-system crash consistency verifier [ASPLOS’16] 24× Cryptographic protocol verifier [FM’18] 29× SQL query verifier [CIDR’17] 75× Safety-cri8cal radiotherapy system verifier [CAV’16] 290×

Mul8ple patches accepted by developers

Used in produc8on at the UW Medical Center

Case studies on published RoseWe-based tools

Explainable: study real users

Small user study: 8 RoseWe users, asked to find known performance bug in 4 programs Users solved every task more quickly when they had access to symbolic profiling

6 failures without symbolic profiling, none with

Qualita8ve feedback:

“gave insight into what RoseWe is doing” “even more useful on my own code”

Does my program work

• n all inputs?

Verifica8on 

Is there a program that does what I want?

Synthesis👸

https://unsat.org

Symbolic profiling iden<fies performance issues in symbolic evalua<on

raco symprofile file.rkt