Program Boosting: Program Synthesis via Crowd-Sourcing Robert A. - - PowerPoint PPT Presentation

▶
program boosting program synthesis via crowd sourcing
SMART_READER_LITE
LIVE PREVIEW

Program Boosting: Program Synthesis via Crowd-Sourcing Robert A. - - PowerPoint PPT Presentation

Jan 18, 2024 97 likes •737 views

Program Boosting: Program Synthesis via Crowd-Sourcing Robert A. Cochran, Loris DAntoni, Benjamin Livshits, David Molnar and Margus Veanes Presented by: Sam Witty, Shehzaad Dhuliawala and Samer Nashed Outline Introduction (Research

slide-1
SLIDE 1

Program Boosting: Program Synthesis via Crowd-Sourcing

Presented by: Sam Witty, Shehzaad Dhuliawala and Samer Nashed

Robert A. Cochran, Loris D’Antoni, Benjamin Livshits, David Molnar and Margus Veanes

slide-2
SLIDE 2

Outline

Introduction (Research Question, Key Ideas, Contributions) Background (Genetic Programming and Regular Expressions) Motivating Example Evaluation Discussion

slide-3
SLIDE 3

Research Question

Can Crowd-Sourced solutions to programming tasks be combined automatically to boost performance?

slide-4
SLIDE 4

Key Idea

Many common programming tasks are 1. Surprisingly complex, such that even expert programmers may struggle

slide-5
SLIDE 5

Key Idea

Many common programming tasks are 1. Surprisingly complex, such that even expert programmers may struggle 2. Easily specified (at least to a good approximation) in English

slide-6
SLIDE 6

Key Idea

Many common programming tasks are 1. Surprisingly complex, such that even expert programmers may struggle 2. Easily specified (at least to a good approximation) in English 3. Nuanced enough that different programmers will fail in different ways These attributes make tasks prime candidates for genetic programming towards program synthesis

slide-7
SLIDE 7

Conventional Programming

Image sources - 4H, The Economist

Blood, sweat, and tears Mostly working program

slide-8
SLIDE 8

Program Boosting

Image sources - 4H, The Economist

Genetic Programming

Flawed programs The one true solution

slide-9
SLIDE 9

Contributions

1. Proposal of new technique: Program Boosting

slide-10
SLIDE 10

Contributions

1. Proposal of new technique: Program Boosting 2. Implementation of genetic programming for regular expressions using custom-designed crossover and mutation operations

slide-11
SLIDE 11

Contributions

1. Proposal of new technique: Program Boosting 2. Implementation of genetic programming for regular expressions using custom-designed crossover and mutation operations 3. Proposal of a new genetic programming paradigm in which the fitness function is evolved along with the candidate programs

slide-12
SLIDE 12

Contributions

1. Proposal of new technique: Program Boosting 2. Implementation of genetic programming for regular expressions using custom-designed crossover and mutation operations 3. Proposal of a new genetic programming paradigm in which the fitness function is evolved along with the candidate programs 4. Release of the tool, CROWDBOOST

slide-13
SLIDE 13

Contributions

1. Proposal of new technique: Program Boosting 2. Implementation of genetic programming for regular expressions using custom-designed crossover and mutation operations 3. Proposal of a new genetic programming paradigm in which the fitness function is evolved along with the candidate programs 4. Release of the tool, CROWDBOOST 5. First use of genetic programming on automata over complex alphabets, in this case UTF-16

slide-14
SLIDE 14

Contributions

1. Proposal of new technique: Program Boosting 2. Implementation of genetic programming for regular expressions using custom-designed crossover and mutation operations 3. Proposal of a new genetic programming paradigm in which the fitness function is evolved along with the candidate programs 4. Release of the tool, CROWDBOOST 5. First use of genetic programming on automata over complex alphabets, in this case UTF-16 6. Evaluation of the proposed method on 465 regular expressions

slide-15
SLIDE 15

Outline

Introduction (Research Question, Key Ideas, Contributions) Background (Genetic Programming and Regular Expressions) Motivating Example Evaluation Discussion

slide-16
SLIDE 16

Background: Genetic Programming

Genetic Programming is a technique wherein a computer program is evolved from some seed using a generic algorithm (often).

slide-17
SLIDE 17

Background: Genetic Programming

Genetic Programming is a technique wherein a computer program is evolved from some seed using a generic algorithm (often). Three main components

slide-18
SLIDE 18

Background: Genetic Programming

Genetic Programming is a technique wherein a computer program is evolved from some seed using a generic algorithm (often). Three main components Crossover - Merge candidate programs

slide-19
SLIDE 19

Background: Genetic Programming

Genetic Programming is a technique wherein a computer program is evolved from some seed using a generic algorithm (often). Three main components Crossover - Merge candidate programs Mutation - Stochastically alter candidate programs

slide-20
SLIDE 20

Background: Genetic Programming

Genetic Programming is a technique wherein a computer program is evolved from some seed using a generic algorithm (often). Three main components Crossover - Merge candidate programs Mutation - Stochastically alter candidate programs Fitness - Evaluate candidate programs

slide-21
SLIDE 21

Animation

Click Me

Source: Rafael Matsunaga

slide-22
SLIDE 22

Background: SFA and Regex

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 4 {-} [0-9]

Corresponding regex: [0-9]{3}(-)?[0-9]{7}

slide-23
SLIDE 23

Outline

Introduction (Research Question, Key Ideas, Contributions) Background (Genetic Programming and Regular Expressions) Motivating Example Evaluation Discussion

slide-24
SLIDE 24

Motivating Example

Determine whether a string is a valid phone number. Ex: 111-111-1111, 1111111111

slide-25
SLIDE 25

Method Overview

Crossover Mutation

Crowdsource the fitness

Evaluate New Gen

slide-26
SLIDE 26

A1 A2 B1 B2

slide-27
SLIDE 27

A1 A2 B1 B2

slide-28
SLIDE 28

A1 A2 B1 B2

slide-29
SLIDE 29

A1 A2 B1 B2

slide-30
SLIDE 30

Identifying components

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 8 [0-9] 9 [0-9]

10

[0-9] 11 4 {-} [0-9] [0-9] 1 [0-9] 2 [0-9] 3 [0-9] 4 [0-9] 5 [0-9] 6 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 7 {-} [0-9]

slide-31
SLIDE 31

Strongly Connected Components

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 8 [0-9] 9 [0-9]

10

[0-9] 11 4 {-} [0-9] [0-9] 1 [0-9] 2 [0-9] 3 [0-9] 4 [0-9] 5 [0-9] 6 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 7 {-} [0-9]

slide-32
SLIDE 32

Stretches

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 8 [0-9] 9 [0-9]

10

[0-9] 11 4 {-} [0-9] [0-9] 1 [0-9] 2 [0-9] 3 [0-9] 4 [0-9] 5 [0-9] 6 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 7 {-} [0-9]

slide-33
SLIDE 33

Stretches

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 8 [0-9] 9 [0-9]

10

[0-9] 11 4 {-} [0-9] [0-9] 1 [0-9] 2 [0-9] 3 [0-9] 4 [0-9] 5 [0-9] 6 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 7 {-} [0-9]

slide-34
SLIDE 34

Single Entry - Single Exit

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 8 [0-9] 9 [0-9]

10

[0-9] 11 4 {-} [0-9] [0-9] 1 [0-9] 2 [0-9] 3 [0-9] 4 [0-9] 5 [0-9] 6 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 7 {-} [0-9]

slide-35
SLIDE 35

Single Entry - Single Exit

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 8 [0-9] 9 [0-9]

10

[0-9] 11 4 {-} [0-9] [0-9] 1 [0-9] 2 [0-9] 3 [0-9] 4 [0-9] 5 [0-9] 6 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 7 {-} [0-9]

slide-36
SLIDE 36

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 8 [0-9] 9 [0-9]

10

[0-9] 11 4 {-} [0-9] [0-9] 1 [0-9] 2 [0-9] 3 [0-9] 4 [0-9] 5 [0-9] 6 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 7 {-} [0-9]

slide-37
SLIDE 37

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 8 [0-9] 9 [0-9] 10 [0-9] 11 7 {-} [0-9] 4 {-} [0-9]

Resulting Regex: [0-9]{3}-?[0-9]{2}-?[0-9]{4}

slide-38
SLIDE 38

Mutations

1. Diminishing 2. Augmenting

slide-39
SLIDE 39

Example Mutation

[0-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 9 [0-9] 10 [0-9] 11 [0-9] 12 8 {-} [0-9] 4 {-} [0-9] Negative example: 012-456-7890 Assume numbers cannot begin with 0

slide-40
SLIDE 40

Example Mutation

[1-9] 1 [0-9] 2 [0-9] 3 [0-9] 5 [0-9] 6 [0-9] 7 [0-9] 9 [0-9] 10 [0-9] 11 [0-9] 12 8 {-} [0-9] 4 {-} [0-9] Negative example: 012-456-7890 Assume numbers cannot begin with 0

slide-41
SLIDE 41

Fitness Function

  • A simple approach to calculate fitness for regular expressions would be to

calculate accuracy on the training set, but this doesn’t scale well.

slide-42
SLIDE 42

Fitness Function

  • A simple approach to calculate fitness for regular expressions would be to

calculate accuracy on the training set, but this doesn’t scale well.

  • Instead, evaluate cardinality of sets.
slide-43
SLIDE 43

Fitness Function

  • A simple approach to calculate fitness for regular expressions would be to

calculate accuracy on the training set, but this doesn’t scale well.

  • Instead, evaluate cardinality of sets.

Fitness(A) = (L(A ∩ P) + L(N - A) )/ L( P U N)

slide-44
SLIDE 44

Outline

Introduction (Research Question, Key Ideas, Contributions) Background (Genetic Programming and Regular Expressions) Motivating Example Evaluation Discussion

slide-45
SLIDE 45

Evaluation

Regular expressions were pulled from Regexlib.com, blogs, Stack Overflow, and a Bountify task set by the authors

slide-46
SLIDE 46

Evaluation

Regular expressions were pulled from Regexlib.com, blogs, Stack Overflow, and a Bountify task set by the authors In total, 465 program pairs were used for a variety of tasks (phone numbers, dates, email addresses, URLs)

slide-47
SLIDE 47

Evaluation

Regular expressions were pulled from Regexlib.com, blogs, Stack Overflow, and a Bountify task set by the authors In total, 465 program pairs were used for a variety of tasks (phone numbers, dates, email addresses, URLs) Mechanical turk was used to generate new examples, thus evolving the fitness

  • function. Examples were accepted of 60% of turkers reached consensus
slide-48
SLIDE 48

Evaluation

slide-49
SLIDE 49

Results - Accuracy

slide-50
SLIDE 50

Results - Accuracy

Crowd Boosting does not help with the Golden Set

slide-51
SLIDE 51

Results - Accuracy

Crowd Boosting does not help with the Golden Set

slide-52
SLIDE 52

Results - Runtime

Runtimes are reasonable, especially for synthesizing a program

slide-53
SLIDE 53

Results - Runtime

Runtimes are reasonable, especially for synthesizing a program This does not include the time required to source the RegExs

slide-54
SLIDE 54

Results - Runtime

Runtimes are reasonable, especially for synthesizing a program This does not include the time required to source the RegExs It is not clear from the paper whether this result includes time required for mechanical turkers

slide-55
SLIDE 55

Results - Cost

Overall, costs are reasonable

slide-56
SLIDE 56

Results - Cost

Phone and email are cheap. Why?

Overall, costs are reasonable

slide-57
SLIDE 57

Outline

Introduction (Research Question, Key Ideas, Contributions) Background (Genetic Programming and Regular Expressions) Motivating Example Evaluation Discussion

slide-58
SLIDE 58

Discussion Questions

  • 1. What kind of ethical issues might arise by involving crowd sourcing in

development?

slide-59
SLIDE 59

Discussion Questions

  • 2. What are some practical limitations of using large numbers of untrained workers

such as mechanical turkers?

slide-60
SLIDE 60

Discussion Questions

  • 3. What are some new research questions posed by this new paradigm wherein

the fitness function is evolves along with the population?

slide-61
SLIDE 61

Discussion Questions

  • 4. What are some weaknesses of genetic programming that persist even through

crowd-sourcing?

slide-62
SLIDE 62

Discussion Questions

  • 5. What other open problems might you imagine applying a similar (crowdsourcing

followed by some form of program synthesis) approach to?