Guarantees in in Program Synthesis Qinheping Hu , Jason Breck , - - PowerPoint PPT Presentation

β–Ά
guarantees in in program synthesis
SMART_READER_LITE
LIVE PREVIEW

Guarantees in in Program Synthesis Qinheping Hu , Jason Breck , - - PowerPoint PPT Presentation

Mar 24, 2024 630 likes β€’1.06k views

Guarantees in in Program Synthesis Qinheping Hu , Jason Breck , John Cyphert , Loris D'Antoni , Thomas Reps Program Synthesis Specification Solution Program Synthesizer Search space Program Synthesis is Un Unpredicta edictable ble

slide-1
SLIDE 1

Guarantees in in Program Synthesis

Qinheping Hu , Jason Breck , John Cyphert , Loris D'Antoni , Thomas Reps

slide-2
SLIDE 2

Program Synthesis

Synthesizer Specification Solution Program Search space

Program Synthesis is Un Unpredicta edictable ble

slide-3
SLIDE 3

Program Synthesis is Un Unpredictable edictable

Search space Solution space

slide-4
SLIDE 4

Program Synthesis is Un Unpredictable edictable

Synthesizer

Specification Solution Program Search space Ability to prefer a solution when there are multiple solution

slide-5
SLIDE 5

Program Synthesis is Un Unpredictable edictable

Search space constant programs: 1,2,3,… Solution space Specification: 𝑔 𝑦 = 𝑦

slide-6
SLIDE 6

Program Synthesis is Un Unpredictable edictable

Search space Solution space Search-based synthesizer

timeout Solution Program ?

slide-7
SLIDE 7

Gu Guarantees ntees in Program Synthesis

Synthesizer

Specification Solution Program Search space Ability to prefer a solution when there are multiple solution Ability to answer no solutions

Make program synthesis predictable

slide-8
SLIDE 8

Syntax-Guided Synthesis with Quantitative Objectives [CAV18]

slide-9
SLIDE 9

Start = +(Start, Start) | π½π‘ˆπΉ(BExpr, Start, Start) 𝑦 𝑧 0 1 BExpr = 𝑂𝑝𝑒(BExpr) | > (Start, Start) |π΅π‘œπ‘’(BExpr, BExpr)

Syntax-Guided Synthesis (SyGuS)

Synthesizer Specification Solution Program Search space

πœ’ 𝑛𝑏𝑦(𝑦, 𝑧), 𝑦, 𝑧 : 𝑛𝑏𝑦 𝑦, 𝑧 β‰₯ 𝑦 ∧ 𝑛𝑏𝑦 𝑦, 𝑧 β‰₯ 𝑧 ∧ (𝑛𝑏𝑦 𝑦, 𝑧 = 𝑦 ∨ 𝑛𝑏𝑦 𝑦, 𝑧 = 𝑧)

𝑓 ∈ 𝑀(𝐻) such that βˆ€π‘¦, 𝑧. πœ’ 𝑓, 𝑦, 𝑧

max 𝑦, 𝑧 = π½π‘ˆπΉ(> (𝑦, 𝑧), 𝑦, π½π‘ˆπΉ(< 𝑦, 𝑧 , 𝑧, 𝑦)

slide-10
SLIDE 10

What we expected Outp tput of f th the CVC4 solv lver

A limitation of SyGuS

slide-11
SLIDE 11

Readable Features you want Smallest size Efficient Least number of multiplication Most likely Largest probability Optim imization Obje jectives

slide-12
SLIDE 12

Gu Guarantees ntees in SyGuS

SyGuS solver

Specification Solution Program Search space Quantitative objectives

slide-13
SLIDE 13

Adding Quantitative Objective to SyGuS

Start = Start + Start | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start x 𝑧 1 BExpr = Start > Start | π‘œπ‘π‘’ BExpr | BExpr π‘π‘œπ‘’ BExpr

slide-14
SLIDE 14

Quantitative objective: Minimize number of if-statement

Adding Quantitative Objective to SyGuS

Start = Start + Start | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start x 𝑧 1 BExpr = Start > Start | π‘œπ‘π‘’ BExpr | BExpr π‘π‘œπ‘’ BExpr

slide-15
SLIDE 15

Quantitative objective: Minimize number of if-statement

Adding Quantitative Objective to SyGuS

Start = Start + Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 x/0 𝑧/0 0/0 1/0 BExpr = Start > Start /0 | π‘œπ‘π‘’ BExpr /0 | BExpr π‘π‘œπ‘’ BExpr /0

slide-16
SLIDE 16

ITE:1 >:0 X:0 Y:0 Y:0 ITE:1 >:0 X:0 X:0 0:0 X:0

If(x>y) then if(x > x) then 0 else x Else y

=

Weight = 2

slide-17
SLIDE 17

Weighted grammar Quantitative objective: Minimize number of if-statement

Adding Quantitative Objective to SyGuS

Start = Start + Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 x/0 𝑧/0 0/0 1/0 BExpr = Start > Start /0 | π‘œπ‘π‘’ BExpr /0 | BExpr π‘π‘œπ‘’ BExpr /0

slide-18
SLIDE 18

Weighted grammar Quantitative objective: Minimize number of if-statement Minimize weight

Adding Quantitative Objective to SyGuS

Start = Start + Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 x/0 𝑧/0 0/0 1/0 BExpr = Start > Start /0 | π‘œπ‘π‘’ BExpr /0 | BExpr π‘π‘œπ‘’ BExpr /0

QSyGuS: (specification,weighted grammar, Quantitative objective)

slide-19
SLIDE 19

Solving QSyGuS

slide-20
SLIDE 20

QSyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’

slide-21
SLIDE 21

QSyGuS SyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ CFG 𝐻 Constraint 𝜚

ignore weight

slide-22
SLIDE 22

QSyGuS SyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ CFG 𝐻 Constraint 𝜚 CFG 𝐻<𝑑1 Constraint 𝜚

Solution’s weight 𝑑1

slide-23
SLIDE 23

QSyGuS SyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ CFG 𝐻 Constraint 𝜚 CFG 𝐻<𝑑1 Constraint 𝜚 CFG 𝐻<𝑑2 Constraint 𝜚

Solution’s weight 𝑑2

slide-24
SLIDE 24

QSyGuS SyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ CFG 𝐻 Constraint 𝜚 CFG 𝐻<𝑑1 Constraint 𝜚 CFG 𝐻<𝑑2 Constraint 𝜚

Solution’s weight 𝑑2

β‹―

slide-25
SLIDE 25

WTG 𝑋 CFG 𝐻<2

Program 𝑄 has π‘₯π‘“π‘—π‘•β„Žπ‘’ < 2 iff 𝐻<2 accept 𝑄

QSyGuS SyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ CFG 𝐻 Constraint 𝜚 CFG 𝐻<2 Constraint 𝜚

Solution’s weight 2

slide-26
SLIDE 26

Grammar Reduction

Id Idea: keep tr track of f th the weight in in th the non-terminals

WTG 𝑋 CFG 𝐻<2 Start = Start + Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 x/0 𝑧/0 0/0 1/0 BExpr = Start > Start /0 | π‘œπ‘π‘’ BExpr /0 | BExpr π‘π‘œπ‘’ BExpr /0

π‘₯π‘“π‘—π‘•β„Žπ‘’ < 2

slide-27
SLIDE 27

Grammar Reduction

Id Idea: keep tr track of f th the weight in in th the non-terminals

WTG 𝑋 CFG 𝐻<2 (Start, < 2) = Start, 0 | (Start, 1)

π‘₯π‘“π‘—π‘•β„Žπ‘’ < 2

Start = Start + Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 x/0 𝑧/0 0/0 1/0 BExpr = Start > Start /0 | π‘œπ‘π‘’ BExpr /0 | BExpr π‘π‘œπ‘’ BExpr /0

slide-28
SLIDE 28

Grammar Reduction

Id Idea: keep tr track of f th the weight in in th the non-terminals

WTG 𝑋 CFG 𝐻<2 (Start, < 2) = Start, 0 | (Start, 1)

π‘₯π‘“π‘—π‘•β„Žπ‘’ < 2

Start, 1 = Start, 0 + Start, 1 | Start, 1 + (Start, 0) | 𝑗𝑔 BExpr, 0 π‘’β„Žπ‘“π‘œ (Start, 0) π‘“π‘šπ‘‘π‘“ (Start, 0) Start, 0 = Start, 0 + Start, 0 𝑦 𝑧 0 1 β‹― Start = Start + Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 x/0 𝑧/0 0/0 1/0 BExpr = Start > Start /0 | π‘œπ‘π‘’ BExpr /0 | BExpr π‘π‘œπ‘’ BExpr /0

slide-29
SLIDE 29

Handling complex weight constraints

Tree grammars are closed under Boolean operations

3 < π‘₯π‘“π‘—π‘•β„Žπ‘’ Complement of 𝐻<4 2 < π‘₯π‘“π‘—π‘•β„Žπ‘’ < 5 𝐻<5 ∩ 𝐻>2 3 < π‘₯π‘“π‘—π‘•β„Žπ‘’1 and π‘₯π‘“π‘—π‘•β„Žπ‘’2 < 0.5 𝐻π‘₯π‘“π‘—π‘•β„Žπ‘’1>3 ∩ 𝐻π‘₯π‘“π‘—π‘•β„Žπ‘’2<0.5 Minimization Minimization Linear search

slide-30
SLIDE 30

Evaluation

slide-31
SLIDE 31

Evaluation

CVC4 ESolver

QSyGuS SyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ CFG 𝐻 Constraint 𝜚 CFG 𝐻<𝑑1 Constraint 𝜚 CFG 𝐻<𝑑2 Constraint 𝜚

β‹―

SyGuS solvers

slide-32
SLIDE 32

Evaluation

Find solution with better cost for 16/26 SyGuS benchmarks Find optimal in 14/26 (couldn’t prove optimality for 2 benchmarks) Average time 3.1x Compared to SyGuS 26 Benchmarks taken from SyGuS 1.minimize number of specified operator, minimize solution size 2.maximize solution probability 3.find sorted optimal for (# of specified operators, size) 4.find Pareto optimal for (# of specified operators, size)

slide-33
SLIDE 33

Conclusion

  • ptimal

QSyGuS SyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ CFG 𝐻 Constraint 𝜚 CFG 𝐻<𝑑1 Constraint 𝜚 CFG 𝐻<𝑑2 Constraint 𝜚

Solution’s weight 𝐷2

β‹―

unrealizable

slide-34
SLIDE 34

Gu Guarantees ntees in SyGuS

SyGuS solver

Specification Solution Program Search space Ability to prefer a solution when there are multiple solution Ability to answer no solutions Quantitative objectives Answering unrealizable

slide-35
SLIDE 35

Proving Unrealizability for Syntax-Guided Synthesis [CAV19]

Tue 12:10 come to my CAV talk

slide-36
SLIDE 36

A Syntax-Guided Synthesis (SyGuS) is

Specification

πœ’ 𝑔(𝑦, 𝑧), 𝑦, 𝑧 :

𝑔 𝑦, 𝑧 β‰₯ 𝑦 ∧ 𝑔 𝑦, 𝑧 β‰₯ 𝑧 ∧ (𝑔 𝑦, 𝑧 = 𝑦 ∨ 𝑔 𝑦, 𝑧 = 𝑧)

Search space G:

Start = +(Start, Start) | π½π‘ˆπΉ(BExpr, Start, Start) 𝑦 𝑧 0 1 BExpr = 𝑂𝑝𝑒(BExpr) | > (Start, Start) |π΅π‘œπ‘’(BExpr, BExpr) max 𝑦, 𝑧 = π½π‘ˆπΉ(> (𝑦, 𝑧), 𝑦, 𝑧)

Goal: find a program 𝑓 ∈ 𝑀(𝐻) such that βˆ€π‘¦, 𝑧. πœ’ 𝑓, 𝑦, 𝑧

slide-37
SLIDE 37

Unrealizable SyGuS Problems

βˆ€π‘¦, 𝑧. max 𝑦, 𝑧 β‰₯ 𝑦 ∧ max 𝑦, 𝑧 β‰₯ 𝑧 ∧ (max 𝑦, 𝑧 = 𝑦 ∨ max 𝑦, 𝑧 = 𝑧) Start = +(Start, Start) 𝑦 𝑧 0 1

No Solution

slide-38
SLIDE 38

CEGIS-based Frameowrk

Example Set 𝐹

Nope

Verifier Z3 ESolver

𝑄 UNSAT

𝑄 is a solution

SAT new example e

slide-39
SLIDE 39

CEGIS-based Frameowrk

Example Set 𝐹

Nope

Verifier Z3 ESolver

Unreachable

UNREALIZABLE

𝑄 UNSAT

𝑄 is a solution

SAT new example e

Seahorn Reduction 𝑆𝑓𝐹

slide-40
SLIDE 40

SyGuS is unrealizable

↔ reachability problem 𝑆𝑓𝐹 is unsatisfiable

Unreachable

UNREALIZABLE Seahorn Reduction 𝑆𝑓𝐹 Reachability Verifier

slide-41
SLIDE 41

Gu Guarantees ntees in Program Synthesis

Synthesizer

Specification Solution Program Search space Ability to prefer a solution when there are multiple solution Ability to answer no solutions

More quantitative objectives 1.Semantic quantitative objectives 2.Resource bounded synthesis Answering unrealizable

  • 1. Tue 12:10 come to my talk
  • 2. Beyond SyGuS