Program Synthesis in the Industrial World: Inductive, Incremental, - - PowerPoint PPT Presentation

▶
program synthesis in the industrial world
SMART_READER_LITE
LIVE PREVIEW

Program Synthesis in the Industrial World: Inductive, Incremental, - - PowerPoint PPT Presentation

Mar 21, 2023 147 likes •608 views

Program Synthesis in the Industrial World: Inductive, Incremental, Interactive Alex Polozov Sumit Gulwani polozov@cs.washington.edu sumitg@microsoft.com And the rest of the PROSE team! prose-contact@microsoft.com July 18, 2016 SYNT-16,

slide-1
SLIDE 1

Program Synthesis in the Industrial World: Inductive, Incremental, Interactive

Alex Polozov polozov@cs.washington.edu Sumit Gulwani sumitg@microsoft.com

And the rest of the PROSE team! prose-contact@microsoft.com

July 18, 2016 SYNT-16, Toronto, Canada 1

slide-2
SLIDE 2

PROgram Synthesis using Examples

July 18, 2016 SYNT-16, Toronto, Canada 2

Vu Le Daniel Perelman Danny Simmons Adam Smith Mohammad Raza Abhishek Udupa Allen Cypher Sumit Gulwani Ranvijay Kumar Alex Polozov

R&D team, MSR → industrial Microsoft We are hiring! Interns or full-time 

slide-3
SLIDE 3

This talk

Lessons Challenges Solutions

July 18, 2016 SYNT-16, Toronto, Canada 3

slide-4
SLIDE 4

Outline

  • Programming by Examples (PBE) & PROSE:

Quick Background

  • Mass-Market Deployment

↪ Goals ↪ Challenges

↪ Solutions

  • Discussion

July 18, 2016 SYNT-16, Toronto, Canada 4

slide-5
SLIDE 5

PBE & PROSE

A 3-slide Background

July 18, 2016 SYNT-16, Toronto, Canada 5

slide-6
SLIDE 6

Motivation

99% of spreadsheet users do not know programming Data scientists spend 80% time wrangling raw data

July 18, 2016 SYNT-16, Toronto, Canada 6

slide-7
SLIDE 7

PROSE Timeline

July 18, 2016 SYNT-16, Toronto, Canada 7

…

2010-2012 [POPL 11]

FlashFill

(text transformations) 2012-2014 [PLDI 14]

FlashExtract

(text extraction) 2012-2015 [PLDI 15]

FlashRelate

(table transformations) 2014-2015 [OOPSLA 15]

FlashMeta

(PBE framework)

PROSE SDK

2015-present

slide-8
SLIDE 8

PBE Architecture

July 18, 2016 SYNT-16, Toronto, Canada 8

Program Synthesizer Debugging

Example-based intent spec 𝜒 Ranking function ℎ DSL ℒ Ranked program set ෩ 𝑂 Intended program 𝑄 ∈ ℒ Refined intent

Translator

Test inputs Ԧ 𝜏 Intended program in Python/C#/C++/…

slide-9
SLIDE 9

Mass-Market Deployment

Goals & Challenges

July 18, 2016 SYNT-16, Toronto, Canada 9

slide-10
SLIDE 10

July 18, 2016 SYNT-16, Toronto, Canada 10

User Experience Inductive

(intent is easily specified)

Interactive

(facilitates the debugging cycle)

Scalable

(snappy UI = responds in < 1 s)

Agile

(quick software development)

Ambiguity resolution Incremental synthesis Predictive synthesis Engineering practices

slide-11
SLIDE 11
slide-12
SLIDE 12

Engineering practices

  • Production-quality library code
  • Prototyping still exists, but it’s not the final form
  • Unit tests & TDD
  • Integration tests: real-life scenarios
  • Close to 8K for all DSLs in total
  • Most are mined from public sources (e.g. help forums)
  • In preparation: benchmark suite release for the community

July 18, 2016 SYNT-16, Toronto, Canada 13

slide-13
SLIDE 13

Performance-minded engineering

  • Parallelization of learning matters
  • E.g.: multi-user log file processing in Azure Log Analytics
  • Performance of program execution matters
  • E.g.: “Big Data” on an end-user’s machine
  • Smallest ≠ fastest!
  • (1) Synthesize many correct programs, then (2) optimize for the fast ones

July 18, 2016 SYNT-16, Toronto, Canada 14

Robustness-based ranking Performance-based ranking

slide-14
SLIDE 14

Development

  • DSL design: ≈ 10 months → ≈ 2 weeks
  • This is not a bottleneck!*
  • Ranking: bulk of the effort
  • Designing a score for an operator 𝐺 is 2-3x longer than

designing 𝐺 (incl. synthesis!)

  • E.g.: rock-paper-scissors among string processing operators

July 18, 2016 SYNT-16, Toronto, Canada 15

* Once you learn the skill…

Should I process the string “25-06-11” with regexes? Treat it as a numeric computation? A date?

slide-15
SLIDE 15
slide-16
SLIDE 16

From: all lines ending with “Number ∘ Dot” “Space ∘ Number ∘ Dot” starting with “Word ∘ Space ∘ CamelCase” Extract: the first “Number” before a “Dot” the last “Number” before a “Dot” the last “Number” before a “Dot ∘ LineBreak” the last “Number” text between the last “Space” and the last “Dot” the first “Comma ∘ Space” and the last “Dot ∘ LineBreak” …and up to 1020 more candidates

18 July 18, 2016 SYNT-16, Toronto, Canada

slide-17
SLIDE 17

Anecdotes

  • FlashFill was not accepted to Excel until it solved the most

common scenarios from 1 example

  • Some users still don’t know you can give 2!

July 18, 2016 SYNT-16, Toronto, Canada 19

Adam Smith Adam Alice Williams Alic

slide-18
SLIDE 18

Ambiguity resolution

Option 1: machine-learned robustness-based ranking [CAV 15]

  • Idioms/patterns from test data can influence search & ranking
  • E.g.: bucketing

July 18, 2016 SYNT-16, Toronto, Canada 20

100 76-100 51 51-75 86

slide-19
SLIDE 19

x ⇒ Concat(Round(x, Down, 25), Const(“-”), Round(x, Up, 25))

Ambiguity resolution

Option 1: machine-learned robustness-based ranking [CAV 15]

  • Idioms/patterns from test data can influence search & ranking
  • E.g.: bucketing

July 18, 2016 SYNT-16, Toronto, Canada 21

100 76-100 51 51-75 86

slide-20
SLIDE 20

x ⇒ Concat(Round(x, Down, 25), Const(“-”), Round(x, Up, 25))

Ambiguity resolution

Option 1: machine-learned robustness-based ranking [CAV 15]

  • Idioms/patterns from test data can influence search & ranking
  • E.g.: bucketing

July 18, 2016 SYNT-16, Toronto, Canada 22

100 76-100 51 51-75 86

slide-21
SLIDE 21

x ⇒ Concat(Round(x, Down, 25), Const(“-”), Round(x, Up, 25))

Ambiguity resolution

Option 1: machine-learned robustness-based ranking [CAV 15]

  • Idioms/patterns from test data can influence search & ranking
  • E.g.: bucketing

Option 2: interactive clarification

July 18, 2016 SYNT-16, Toronto, Canada 23

100 76-100 51 51-75 86

slide-22
SLIDE 22
slide-23
SLIDE 23

PBE Architecture

July 18, 2016 SYNT-16, Toronto, Canada 26

Program Synthesizer Debugging

Example-based intent spec 𝜒 Ranking function ℎ DSL ℒ Ranked program set ෩ 𝑂 Intended program 𝑄 ∈ ℒ Refined intent

Translator

Test inputs Ԧ 𝜏 Intended program in Python/C#/C++/…

slide-24
SLIDE 24

PBE Architecture

July 18, 2016 SYNT-16, Toronto, Canada 27

Program Synthesizer User

Example-based intent spec 𝜒 Ranking function ℎ DSL ℒ Ranked program set ෩ 𝑂 Intended program 𝑄 ∈ ℒ Refined intent

Translator

Test inputs Ԧ 𝜏 Intended program in Python/C#/C++/…

Hypothesizer

Questions

slide-25
SLIDE 25

PBE Architecture

July 18, 2016 SYNT-16, Toronto, Canada 28

Program Synthesizer User

Example-based intent spec 𝜒 Ranking function ℎ DSL ℒ Ranked program set ෩ 𝑂 Intended program 𝑄 ∈ ℒ Refined intent

Translator

Test inputs Ԧ 𝜏 Intended program in Python/C#/C++/…

Hypothesizer

Questions

slide-26
SLIDE 26

Hypothesizer

 Reduces the cognitive load on the user  Reduces the number of iterations by choosing the most effective disambiguating questions  Increases the user’s confidence in the system (“proactive = smart”)

July 18, 2016 SYNT-16, Toronto, Canada 29

Given a program set ෩ 𝑶, find program constraints (“hypotheses”) 𝝌 that best disambiguate among programs in ෩ 𝑶, and present them to the user as multiple-choice questions.

slide-27
SLIDE 27

Example

July 18, 2016 SYNT-16, Toronto, Canada 30

Missing page numbers, 1993 1993 64-67, 1995 64 … … …

Which output is correct here?

  • a. 64
  • b. 67
  • c. 1995

෩ 𝑂

1995 64 67

⊥

slide-28
SLIDE 28

Example

July 18, 2016 SYNT-16, Toronto, Canada 31

Missing page numbers, 1993 1993 64-67, 1995 64 … … …

Which output is correct here?

  • a. 64
  • b. 67
  • c. 1995

෩ 𝑂

1995 64 67

⊥ 𝜒𝑗+1: 𝑄 𝜏2 = "1995"

slide-29
SLIDE 29

Example

July 18, 2016 SYNT-16, Toronto, Canada 32

Missing page numbers, 1993 1993 64-67, 1995 1995 … … …

Which output is correct here?

  • a. 64
  • b. 67
  • c. 1995

𝜒𝑗+1: 𝑄 𝜏2 = "1995"

slide-30
SLIDE 30

Example – alternative

July 18, 2016 SYNT-16, Toronto, Canada 33

Missing page numbers, 1993 1993 64-67, 1995 64 … … …

෩ 𝑂

1995 64 67

⊥

64-67

Is this part of the input relevant?

  • a. Yes
  • b. No
  • c. Maybe
slide-31
SLIDE 31

Picking the right question

“Distinguishability” = effectiveness for disambiguation 1. An input is distinguishing if many top-ranked candidate programs disagree

  • n the intended output on it.
  • Any response will partition the program set well
  • 2. A question is distinguishing if the alternative candidate programs

corresponding to all potential responses have high ranks.

  • Any response will lead to a good alternative program

Preliminary results: good questions yield just 4-6 iterations until convergence

July 18, 2016 SYNT-16, Toronto, Canada 34

slide-32
SLIDE 32
slide-33
SLIDE 33

Big Data

July 18, 2016 SYNT-16, Toronto, Canada 37

slide-34
SLIDE 34

Big Data + Program Synthesis

July 18, 2016 SYNT-16, Toronto, Canada 38

slide-35
SLIDE 35

Problem definition

  • ℒ is an industrial DSL (e.g., FlashFill)
  • ෩

𝑂𝑗 ≈ 1020

  • Time limit: ≈ 1 sec

July 18, 2016 SYNT-16, Toronto, Canada 39

Given a program set ෩ 𝑶𝒋 ⊂ ℒ that satisfies the currently accumulated spec 𝝌𝒋, and a new constraint 𝝎𝒋+𝟐, learn a subset ෩ 𝑶𝒋+𝟐 ⊂ ෩ 𝑶𝒋 of programs that satisfy the new spec 𝝌𝒋+𝟐 = 𝝌𝒋 ∧ 𝝎𝒋+𝟐

slide-36
SLIDE 36

Background: Version Space Algebra

July 18, 2016 SYNT-16, Toronto, Canada 40

int positionIn[string s] := AbsPos(s, k) | RegPos(s, std.Pair(r, r), k);

slide-37
SLIDE 37

Background: Version Space Algebra

July 18, 2016 SYNT-16, Toronto, Canada 41

int positionIn[string s] := AbsPos(s, k) | RegPos(s, std.Pair(r, r), k);

Sharing #1: cross-product representation

slide-38
SLIDE 38

Background: Version Space Algebra

July 18, 2016 SYNT-16, Toronto, Canada 42

int positionIn[string s] := AbsPos(s, k) | RegPos(s, std.Pair(r, r), k);

Sharing #1: cross-product representation Sharing #2: equal sets are shared as the same DAG node Sharing #2: equal sets are shared as the same DAG node

slide-39
SLIDE 39

Background: Version Space Algebra

July 18, 2016 SYNT-16, Toronto, Canada 43

int positionIn[string s] := AbsPos(s, k) | RegPos(s, std.Pair(r, r), k);

Sharing #1: cross-product representation Sharing #2: equal sets are shared as the same DAG node Sharing #2: equal sets are shared as the same DAG node For, e.g., FlashFill, volume ෩ 𝑂 ≈ 105 when ෩ 𝑂 ≈ 1020. Runtime of VSA operations ∝ VSA volume.

slide-40
SLIDE 40

VSAs and CFGs are two isomorphic representations for a language

July 18, 2016 SYNT-16, Toronto, Canada 44

Filter ෩ 𝑂, 𝜔 ≡ Learn ℒ ෩ 𝑂 , 𝜔

slide-41
SLIDE 41

Incremental Inductive Synthesis

  • 1. Implicitly represent ෩

𝑂𝑗 (already a VSA!) as an isomorphic CFG ℒ ෩ 𝑂𝑗 .

  • 2. Analyze the descriptive power of 𝜔𝑗+1:
  • Definitive (e.g., examples, set membership, subsequence constraints):

Apply regular top-down deductive synthesis on ℒ ෩ 𝑂𝑗

  • Locally refining (e.g., datatypes, input relevance):

Re-run backpropagation only on relevant parts of ℒ ෩ 𝑂𝑗

  • Globally refining (e.g., negative examples):

Filter ෩ 𝑂𝑗 at the top level

July 18, 2016 SYNT-16, Toronto, Canada 45

slide-42
SLIDE 42

PBE Architecture

July 18, 2016 SYNT-16, Toronto, Canada 46

Program Synthesizer User

Example-based intent spec 𝜒 Ranking function ℎ DSL ℒ Ranked program set ෩ 𝑂 Intended program 𝑄 ∈ ℒ Refined intent

Translator

Test inputs Ԧ 𝜏 Intended program in Python/C#/C++/…

Hypothesizer

Questions Refined DSL ℒ′

slide-43
SLIDE 43

Preliminary results

  • Big improvement when VSA fragmentation is limited
  • Not the final results; work in progress has orders-of-magnitude improvements

July 18, 2016 SYNT-16, Toronto, Canada 47

FlashFill FlashExtract

slide-44
SLIDE 44

Lessons & Conclusions

  • Robust engineering is the key to PBE deployment
  • Ranking ≫ learning (in industrial PBE)
  • Interaction models should be first-class citizens in synthesis frameworks
  • Great theoretical results: e.g., OGIS [Jha & Seshia 2015]
  • Also need: HCI evaluations, comparison of query types, worst-case TD optimization
  • Proactive ambiguity analysis of current candidate programs
  • Incrementality: treat the previous program set as the new search space
  • Requires full program set computation (possibly in the background)

July 18, 2016 SYNT-16, Toronto, Canada 48

https://microsoft.github.io/prose prose-contact@microsoft.com

Thank you!