Neutral Networks of Real-World Programs and their Application to - - PowerPoint PPT Presentation

▶
neutral networks of real world programs and their
SMART_READER_LITE
LIVE PREVIEW

Neutral Networks of Real-World Programs and their Application to - - PowerPoint PPT Presentation

Jan 16, 2023 41 likes •1.52k views

Neutral Networks of Real-World Programs and their Application to Automated Software Evolution Eric Schulte Department of Computer Science University of New Mexico Albuquerque, NM July 2014 Neutral Networks and Automated Software Evolution

slide-1
SLIDE 1

Neutral Networks of Real-World Programs and their Application to Automated Software Evolution

Eric Schulte

Department of Computer Science University of New Mexico Albuquerque, NM

July 2014

Neutral Networks and Automated Software Evolution Introduction

slide-2
SLIDE 2

Software Engineering

Motivation

Bugs cost estimated $312 billion per year

[Britton, University of Cambridge 2013]

Neutral Networks and Automated Software Evolution Introduction 1

slide-3
SLIDE 3

Software Engineering

Motivation

Bugs cost estimated $312 billion per year

[Britton, University of Cambridge 2013]

Security millions of dollars per exploit

[Van, OECD Publishing 2008]

Neutral Networks and Automated Software Evolution Introduction 1

slide-4
SLIDE 4

Software Engineering

Motivation

Bugs cost estimated $312 billion per year

[Britton, University of Cambridge 2013]

Security millions of dollars per exploit

[Van, OECD Publishing 2008]

Resources 1% of global energy to data centers in 2010

[Koomey, Analytics Press 2011]

Neutral Networks and Automated Software Evolution Introduction 1

slide-5
SLIDE 5

Software Engineering

Motivation

Bugs cost estimated $312 billion per year

[Britton, University of Cambridge 2013]

Security millions of dollars per exploit

[Van, OECD Publishing 2008]

Resources 1% of global energy to data centers in 2010

[Koomey, Analytics Press 2011]

Developers projected 1.5 million in US by 2018

[U.S Department of Labor, 2011]

Neutral Networks and Automated Software Evolution Introduction 1

slide-6
SLIDE 6

Software Engineering

Evolution and Automation

Evolved Ecosystem

◮ applications ◮ libraries ◮ compilers ◮ operating systems ◮ architectures

Neutral Networks and Automated Software Evolution Introduction 2

slide-7
SLIDE 7

Software Engineering

Evolution and Automation

Evolved Ecosystem

◮ applications ◮ libraries ◮ compilers ◮ operating systems ◮ architectures

Evolvable Software

◮ automated diversity ◮ assembly and binary repair ◮ patch closed source executable ◮ optimization

Neutral Networks and Automated Software Evolution Introduction 2

slide-8
SLIDE 8

Genprog

Automatically Fix Bugs in C Software

Neutral Networks and Automated Software Evolution Introduction 3

slide-9
SLIDE 9

Genprog

Automatically Fix Bugs in C Software

Collaboration between UNM and UVA

  • Dr. Stephanie Forrest
  • Dr. Westley Weimer

Neutral Networks and Automated Software Evolution Introduction 3

slide-10
SLIDE 10

Genprog

Automatically Fix Bugs in C Software

Input

X

Evolve

Fitness Evaluation Mutants

Output

  • Discard

Accept Minimize

[Le Goues, 2013, Figure 3.2]

Neutral Networks and Automated Software Evolution Introduction 3

slide-11
SLIDE 11

Genprog

Automatically Fix Bugs in C Software

Strengths

Effective Repaired 55/105 bugs for $8 each General Multiple classes of bugs and security defects Best Papers ICSE 2009, GECCO 2009, SBST 2009 Humies Gold 2009, Bronze 2012

Neutral Networks and Automated Software Evolution Introduction 3

slide-12
SLIDE 12

Genprog

Automatically Fix Bugs in C Software

But

Why doesn’t this break my software?

Neutral Networks and Automated Software Evolution Introduction 3

slide-13
SLIDE 13

Outline

Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion

Neutral Networks and Automated Software Evolution Program Representation and Transformation 4

slide-14
SLIDE 14

Program Representation

Source

if(a==0){ printf("%g\n", b); } else { while(b!=0){ if(a>b){ a=a-b; } else { b=b-a; }}} printf("%g\n", a);

Neutral Networks and Automated Software Evolution Program Representation and Transformation 5

slide-15
SLIDE 15

Program Representation

Source

if(a==0){ printf("%g\n", b); } else { while(b!=0){ if(a>b){ a=a-b; } else { b=b-a; }}} printf("%g\n", a);

Tree

if(a==0) printf("%g\n",b); while(b!=0) if(a>b) a=a-b; else b=b-a; printf("%g\n",a); Neutral Networks and Automated Software Evolution Program Representation and Transformation 5

slide-16
SLIDE 16

Program Representation

Source

if(a==0){ printf("%g\n", b); } else { while(b!=0){ if(a>b){ a=a-b; } else { b=b-a; }}} printf("%g\n", a);

Tree

if(a==0) printf("%g\n",b); while(b!=0) if(a>b) a=a-b; else b=b-a; printf("%g\n",a);

Vector

main: .cfi startproc pushq %rbp .cfi def cfa offset 16 .cfi offset 6, -16 movq %rsp, %rbp .cfi def cfa register 6 subq $48, %rsp Neutral Networks and Automated Software Evolution Program Representation and Transformation 5

slide-17
SLIDE 17

Program Representation

Representations

Rep. Type Languages CLang tree C-family source Cil tree C Int. Lang. LLVM vector LLVM SSA IR ASM vector x86 & ARM ELF vector x86 & MIPS

Tree: Cil, CLang

if(a==0) printf("%g\n",b); while(b!=0) if(a>b) a=a-b; else b=b-a; printf("%g\n",a);

Vector: ASM, LLVM, ELF

main: .cfi startproc pushq %rbp .cfi def cfa offset 16 .cfi offset 6, -16 movq %rsp, %rbp .cfi def cfa register 6 subq $48, %rsp Neutral Networks and Automated Software Evolution Program Representation and Transformation 5

slide-18
SLIDE 18

Program Mutation Operations

Tree Delete

→

Tree Insert

→

Tree Swap

→

Vector Delete Vector Insert Vector Swap

Neutral Networks and Automated Software Evolution Program Representation and Transformation 6

slide-19
SLIDE 19

Program Mutation Operations

Tree Delete

→

Tree Insert

→

Tree Swap

→

ELF Delete

0x0

Vector Insert Vector Swap

Neutral Networks and Automated Software Evolution Program Representation and Transformation 6

slide-20
SLIDE 20

Program Mutation Operations

Tree Delete

→

Tree Insert

→

Tree Swap

→

ELF Delete

0x0

ELF Replace Vector Swap

Neutral Networks and Automated Software Evolution Program Representation and Transformation 6

slide-21
SLIDE 21

LLVM Mutation Illustration

C source

int main(int argc , char *argv []){ int x=2; x+=3; x=x*x; printf("%d\n", x); return 0; }

Neutral Networks and Automated Software Evolution Program Representation and Transformation 7

slide-22
SLIDE 22

LLVM Mutation Illustration

LLVM

%x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %0, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x %3 printf

Neutral Networks and Automated Software Evolution Program Representation and Transformation 7

slide-23
SLIDE 23

LLVM Mutation Illustration

LLVM: Delete 4

%x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %0, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x %3 printf

Neutral Networks and Automated Software Evolution Program Representation and Transformation 7

slide-24
SLIDE 24

LLVM Mutation Illustration

LLVM: Delete 4

%x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %0, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x %3 printf

Neutral Networks and Automated Software Evolution Program Representation and Transformation 7

slide-25
SLIDE 25

LLVM Mutation Illustration

LLVM: Insert 9 4

%x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x %3 printf

Neutral Networks and Automated Software Evolution Program Representation and Transformation 7

slide-26
SLIDE 26

LLVM Mutation Illustration

LLVM: Insert 9 4

%x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 %add.insert = add nsw i32 %0, 3 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x %3 printf %add2 i32 3

Neutral Networks and Automated Software Evolution Program Representation and Transformation 7

slide-27
SLIDE 27

LLVM Mutation Illustration

LLVM: Insert 9 4

%x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 %add.insert = add nsw i32 %0, 3 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x %3 printf %add2 i32 3

Neutral Networks and Automated Software Evolution Program Representation and Transformation 7

slide-28
SLIDE 28

Conclusion

Representations

Rep. Type Languages CLang tree C-family source Cil tree C Int. Lang. LLVM vector LLVM SSA IR ASM vector x86 & ARM ELF vector x86 & MIPS

Differences

◮ Mutation operations and

Search Space

◮ Languages ◮ Requirements ◮ Expression as executables ◮ Communication of

mutations

Neutral Networks and Automated Software Evolution Program Representation and Transformation 8

slide-29
SLIDE 29

Outline

Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 9

slide-30
SLIDE 30

Software Mutational Robustness

[Schulte et al., GPEM 2013]

percentage of mutants which are functional

slide-31
SLIDE 31

Software Mutational Robustness

Program Space

Test Suite Specification Original Program

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 10

slide-32
SLIDE 32

Software Mutational Robustness

Program Space

Test Suite Specification Original Program Neutral Mutant

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 10

slide-33
SLIDE 33

Software Mutational Robustness

Program Space

Test Suite Specification Original Program Neutral Mutant Killed Mutant

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 10

slide-34
SLIDE 34

Definition

MutRB(P, T, M)

P program T test suite M mutation operators MutRB(P, T, M) = |{P′ | m ∈ M. P′ ← m(P) ∧ T(P′)}| |{P′ | m ∈ M. P′ ← m(P)}|

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 11

slide-35
SLIDE 35

Mutational Robustness by Test Suite

Sorters

%MutRB 10 20 30 40 50 b u b b l e

  • s
  • r

t i n s e r t i

  • n
  • s
  • r

t m e r g e

  • s
  • r

t q u i c k

  • s
  • r

t

Siemen’s

%MutRB 10 20 30 40 50 p r i n t t

  • k

e n s s c h e d u l e s e d s p a c e t c a s

Systems Programs

%MutRB 20 40 60 80 100 bzip2 1.0.2 – ccrypt 1.2 – imagemagick 6.5.2 jansson 1.3 leukocyte lighttpd 1.4.15 nullhttpd 0.5.0

  • ggenc 1.0.1

– potion 40b5f03 redis 1.3.4 tiff 3.8.2 vyquon 335426d grep Cil ASM

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 12

slide-36
SLIDE 36

Fitness Distribution by Representation

500 1000 1500 2000 2500 3000 3500 1 2 3 4 5 6 7 8 9 10 Number of Variants Fitness

CLANG CIL LLVM ASM ELF

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 13

slide-37
SLIDE 37

Fitness Landscapes and Neutral Networks

Program Space Test Suite Specification Original Program Neutral Mutant Killed Mutant

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14

slide-38
SLIDE 38

Fitness Landscapes and Neutral Networks

P r

  • g

r a m S p a c e T e s t S u i t e S p e c i fi c a t i

  • n

O r i g i n a l P r

  • g

r a m N e u t r a l M u t a n t K i l l e d M u t a n t

Fitness

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14

slide-39
SLIDE 39

Fitness Landscapes and Neutral Networks

P r

  • g

r a m S p a c e T e s t S u i t e S p e c i fi c a t i

  • n

K i l l e d M u t a n t O r i g i n a l P r

  • g

r a m N e u t r a l M u t a n t

Fitness

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14

slide-40
SLIDE 40

Fitness Landscapes and Neutral Networks

P r

  • g

r a m S p a c e T e s t S u i t e

Fitness

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14

slide-41
SLIDE 41

Fitness Landscapes and Neutral Networks

P r

  • g

r a m S p a c e T e s t S u i t e

Fitness

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14

slide-42
SLIDE 42

Fitness Landscapes and Neutral Networks

P r

  • g

r a m S p a c e T e s t S u i t e

Fitness

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14

slide-43
SLIDE 43

Fitness Landscapes and Neutral Networks

P r

  • g

r a m S p a c e T e s t S u i t e

Fitness

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14

slide-44
SLIDE 44

Span of Neutral Networks

Insertion Sort Neutral Variants

170 180 190 200 210 220 230 50 100 150 200 25012 13 14 15 16 17 18 19 20 21

  • Avg. LOC

% Neutral Variants Number of Applied Mutations

  • Avg. LOC

% Neutral Variants

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 15

slide-45
SLIDE 45

Span of Neutral Networks

Insertion Sort Neutral Variants: size controlled

172 173 174 175 50 100 150 200 2504 6 8 10 12 14 16 18 20

  • Avg. LOC

% Neutral Variants Number of Applied Mutations

  • Avg. LOC

% Neutral Variants

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 15

slide-46
SLIDE 46

Random Walks in Program Space

Quick Sort Random Variants

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 Fraction of Variants which are Neutral Number of Applied Mutations 0.35831x Fraction Found Neutral Experimentally

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 16

slide-47
SLIDE 47

Conclusion

Software Neutral Networks are

wide span large distances in program space thick consistent mutational robustness accessible automatically explorable

Program Space Test Suite

Fitness

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 17

slide-48
SLIDE 48

Conclusion

Software Neutral Networks are

wide span large distances in program space thick consistent mutational robustness accessible automatically explorable

Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 17

slide-49
SLIDE 49

Outline

Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion

Neutral Networks and Automated Software Evolution Application: Program Diversity 18

slide-50
SLIDE 50

Automated Diversity

Program Space

Test Suite Specification Original Program

Neutral Networks and Automated Software Evolution Application: Program Diversity 19

slide-51
SLIDE 51

Automated Diversity

Program Space

Test Suite Specification Original Program Mutant

Neutral Networks and Automated Software Evolution Application: Program Diversity 19

slide-52
SLIDE 52

Automated Diversity

Program Space

Test Suite Specification

Neutral Networks and Automated Software Evolution Application: Program Diversity 19

slide-53
SLIDE 53

Automated Diversity

Program Space

Test Suite Specification

Neutral Networks and Automated Software Evolution Application: Program Diversity 19

slide-54
SLIDE 54

Automated Diversity

Proactive Bug Repair

Program Space

Test Suite Specification Bug

Neutral Networks and Automated Software Evolution Application: Program Diversity 19

slide-55
SLIDE 55

Contribution

Automated Diversity

◮ Neutral networks expose meaningful diversity

[Baudry, ISSTA 2014]

◮ Diverse variants proactively fix defects ◮ Select small mutually diverse populations

Neutral Networks and Automated Software Evolution Application: Program Diversity 20

slide-56
SLIDE 56

Outline

Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion

Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 21

slide-57
SLIDE 57

Program Repair in Embedded Devices

[Schulte et al., ASPLOS 2013]

Resource Constraints

◮ Small disks ◮ Less memory ◮ Slow processors ◮ Slow, costly comm.

m a i a r g c , c h a r * a r g v [ ] ) m a i n ( i n t a r g c , [ ] ) i n t a ; i n t b ; i f ( a !

Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 22

slide-58
SLIDE 58

Genprog

Automatically Fix Bugs in C Software

Strengths

Effective Repaired 55/105 bugs for $8 each General Multiple classes of bugs and security defects Best Papers ICSE 2009, GECCO 2009, SBST 2009 Humies Gold 2009, Bronze 2012

Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 23

slide-59
SLIDE 59

Genprog

Automatically Fix Bugs in C Software

Strengths

Effective Repaired 55/105 bugs for $8 each General Multiple classes of bugs and security defects Best Papers ICSE 2009, GECCO 2009, SBST 2009 Humies Gold 2009, Bronze 2012

Requirements

◮

Source code

◮

Build tool chain

◮

Expensive fault localization

◮

Expensive fitness function (compilation, test execution)

Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 23

slide-60
SLIDE 60

Genprog: Assembler- and Binary-Level

Automatically Fix Bugs in C Software

Strengths

Effective Repaired 55/105 bugs for $8 each General Multiple classes of bugs and security defects Best Papers ICSE 2009, GECCO 2009, SBST 2009 Humies Gold 2009, Bronze 2012

Requirements

◮

Source code

◮

Build tool chain

◮

Expensive fault localization

◮

Expensive fitness function (compilation, test execution)

Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 23

slide-61
SLIDE 61

Embedded Repair Benchmark Programs

Program Program Description Bug atris graphical tetris game stack buffer exploit ccrypt encryption utility segfault deroff document processing segfault flex lexical analyzer generator segfault indent source code processing infinite loop look svr4 dictionary lookup infinite loop look ultrix dictionary lookup infinite loop merge merge sort duplicate inputs merge-cpp merge sort (in C++) duplicate inputs s3 sendmail utility buffer overflow uniq duplicate text processing segfault units metric conversion segfault zune embedded media player infinite loop

Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 24

slide-62
SLIDE 62

Contribution

ASM and ELF representation

◮ Reduce requirements of automated repair

◮ Effective: fixing 12 (ASM) and 11 (ELF) of 13 ◮ 62% faster runtime ◮ 95% smaller disk footprint ◮ 86% less memory

◮ Reduce resources needed to perform repair ◮ New program repair search spaces

Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 25

slide-63
SLIDE 63

Outline

Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion

Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 26

slide-64
SLIDE 64

NETGEAR Exploit

[Schulte et al., unpublished]

  • 1. URL starting with BRS bypasses authentication
  • 2. URL including unauth.cgi or securityquestions.cgi

bypass authentication

  • 3. unprotected page removes authentication for every page

http://router/BRS_02_genieHelp.html

Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 27

slide-65
SLIDE 65

Common Vulnerability

NETGEAR WNDR4700 D-Link DIR-100 Linksys WAG200G

Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 28

slide-66
SLIDE 66

NETGEAR Repair Overview

Mutate

QEMU

Select Insert Evaluate

QEMU QEMU QEMU QEMU

Virtual Machines Threads Population Firmware

binwalk Environment Binary

Minimize

Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 29

slide-67
SLIDE 67

NETGEAR Repair Overview

Mutate

QEMU

Select Insert Evaluate

QEMU QEMU QEMU QEMU

Virtual Machines Threads Population Firmware

binwalk Environment Binary

Minimize

Repair Runtime

evaluations ∼ 36, 000 runtime 86.6 min. threads 32

Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 29

slide-68
SLIDE 68

Contribution

Genprog Requirements

◮

Non-stripped ELF file

◮

Regression test suite

◮

Fault localization

Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 30

slide-69
SLIDE 69

Contribution

Genprog Requirements

◮

Non-stripped ELF file

◮

Regression test suite

◮

Fault localization

Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 30

slide-70
SLIDE 70

Contribution

Genprog Requirements

◮

Non-stripped ELF file

◮

Regression test suite

◮

Fault localization

Enables

◮ repair of black box executables ◮ repair in data sections ◮ patching proprietary software

Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 30

slide-71
SLIDE 71

Outline

Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 31

slide-72
SLIDE 72

Optimization Problem

[Schulte, Dorn et al., ASPLOS 2014]

Optimizing complex non-functional properties

properties × hardware × environment properties memory, network, energy, etc. . . hardware architectures, processors, memory stack, etc. . . environment variables, load, etc. . .

Every program transformation requires

a-priori reasoning manual implementation guaranteed correctness

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 32

slide-73
SLIDE 73

Genetic Optimization Algorithm (GOA)

Post-compiler, test-driven, Genetic Optimization Algorithm

Post-compiler source .s GOA .s .exe

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 33

slide-74
SLIDE 74

Genetic Optimization Algorithm (GOA)

Post-compiler, test-driven, Genetic Optimization Algorithm

Test driven

Use test cases to exercise program

◮ evaluate functionality ◮ measure runtime properties

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 33

slide-75
SLIDE 75

Genetic Optimization Algorithm (GOA)

Post-compiler, test-driven, Genetic Optimization Algorithm

Algorithm

Assembler Fitness Function Workload Population Mutate Profile Fitness Minimize Executable

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 33

slide-76
SLIDE 76

Genetic Optimization Algorithm (GOA)

Post-compiler, test-driven, Genetic Optimization Algorithm

Algorithm

Assembler Fitness Function Workload Population Mutate Profile Fitness Minimize Executable

pop 210 evals 218 time ∼ 16h

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 33

slide-77
SLIDE 77

Benchmark Applications

C/C++ ASM Program Lines of Code Description blackscholes 510 7,932 Finance modeling bodytrack 14,513 955,888 Human video tracking facesim no alternate inputs ferret 15,188 288,981 Image search engine fluidanimate 11,424 44,681 Fluid dynamics animation freqmine 2,710 104,722 Frequent itemset mining raytrace no testable output swaptions 1,649 61,134 Portfolio pricing vips 142,019 132,012 Image transformation x264 37,454 111,718 MPEG-4 video encoder total 225,467 1,707,068 Table: PARSEC benchmark applications.

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 34

slide-78
SLIDE 78

Hardware Platforms

AMD Server Intel Desktop

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 35

slide-79
SLIDE 79

Results: Energy Reduction

0% 20% 40% 60% 80% 100% b l a c k s c h

  • l

e s b

  • d

y t r a c k f e r r e t fl u i d a n i m a t e f r e q m i n e s w a p t i

  • n

s v i p s x 2 6 4 A v e r a g e

Energy Reduction

AMD Intel

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 36

slide-80
SLIDE 80

Functionality on Withheld Tests

Program AMD Intel blackscholes 100% 100% bodytrack 92% 100% ferret 100% 100% fluidanimate 6% 31% freqmine 100% 100% swaptions 100% 100% vips 100% 100% x264 27% 100%

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 37

slide-81
SLIDE 81

Anecdotes

Blackscholes

◮ 90% less energy ◮ removed redundant outer loop ◮ modified semantics

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 38

slide-82
SLIDE 82

Anecdotes

Blackscholes

◮ 90% less energy ◮ removed redundant outer loop ◮ modified semantics

Swaptions

◮ 42% less energy ◮ improved branch prediction ◮ hardware specific

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 38

slide-83
SLIDE 83

Anecdotes

Blackscholes

◮ 90% less energy ◮ removed redundant outer loop ◮ modified semantics

Swaptions

◮ 42% less energy ◮ improved branch prediction ◮ hardware specific

Vips

◮ 20% less energy ◮ substitution of memory access for calculation ◮ resource trade-off

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 38

slide-84
SLIDE 84

Contribution

Genetic Optimization Algorithm

  • 1. optimize complex runtime properties (energy)
  • 2. leverages particulars of hardware, and environment
  • 3. reveal compiler inefficiencies
  • 4. find efficiencies, e.g., loop elimination
  • 5. transformations presented as ASM diff

Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 39

slide-85
SLIDE 85

Outline

Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion

Neutral Networks and Automated Software Evolution Future Work 40

slide-86
SLIDE 86

Challenges

Interface

◮ Guarantee semantics preservation ◮ Communicate goals (fitness functions) ◮ Incorporate into development cycle

Power

◮ Novel functionality ◮ Combine components

Neutral Networks and Automated Software Evolution Future Work 41

slide-87
SLIDE 87

Future Work

Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr.

◮ Assembler diff chunks formally equivalent ◮ Static analysis to,

◮ find bugs ◮ measure code quality

◮ Invariant preservation

e.g., Daikon

Neutral Networks and Automated Software Evolution Future Work 42

slide-88
SLIDE 88

Future Work

Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr.

“Smarter” mutation operations

◮ Repair templates and external code libraries ◮ Type-aware operations ◮ Learned mutation heuristics

Neutral Networks and Automated Software Evolution Future Work 42

slide-89
SLIDE 89

Future Work

Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr.

Heterologous Crossover

between different

◮ compilation flags ◮ versions ◮ implementations ◮ programs

Homologous Crossover Heterologous Crossover ∼

Neutral Networks and Automated Software Evolution Future Work 42

slide-90
SLIDE 90

Future Work

Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr.

Evolution Threads Fuzz Tester e.g., Klee, S2E Population of Program Variants Test Suite

◮ regression tests ◮ fuzz test 1 ◮ fuzz test 2 ◮ etc.

Tournament Selection Fitness Evaluation Program p Passing all Tests Fuzz Test Failing Program p

Neutral Networks and Automated Software Evolution Future Work 42

slide-91
SLIDE 91

Future Work

Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr.

Augment tests to increase granularity

◮ Count unique values of program counter ◮ Real-valued oracle comparison (e.g., numdiff)

Neutral Networks and Automated Software Evolution Future Work 42

slide-92
SLIDE 92

Outline

Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion

Neutral Networks and Automated Software Evolution Conclusion 43

slide-93
SLIDE 93

Conclusion

Extant software is

  • 1. Mutationally robust with large neutral networks

Neutral Networks and Automated Software Evolution Conclusion 44

slide-94
SLIDE 94

Conclusion

Extant software is

  • 1. Mutationally robust with large neutral networks
  • 2. A product of evolution

Neutral Networks and Automated Software Evolution Conclusion 44

slide-95
SLIDE 95

Conclusion

Extant software is

  • 1. Mutationally robust with large neutral networks
  • 2. A product of evolution
  • 3. Amenable to automated evolutionary improvement

Neutral Networks and Automated Software Evolution Conclusion 44

slide-96
SLIDE 96

Conclusion

Extant software is

  • 1. Mutationally robust with large neutral networks
  • 2. A product of evolution
  • 3. Amenable to automated evolutionary improvement

Software evolution might

facilitate the automation of software engineering

Neutral Networks and Automated Software Evolution Conclusion 44

slide-97
SLIDE 97

Thank You

Eric Schulte

email eschulte@cs.unm.edu homepage https://cs.unm.edu/~eschulte dissertation https://cs.unm.edu/~eschulte/dissertation

Neutral Networks and Automated Software Evolution Conclusion 45

slide-98
SLIDE 98

References

Forrest, GECCO 2009 Stephanie Forrest, Westley Weimer, ThanhVu Nguyen and Claire Le Goues. A Genetic Programming Approach to Automatic Program Repair, in GECCO, 2009 Baudry, ISSTA 2014 Benoit Baudry, Simon Allier, and Martin Monperrus. Tailored source code transformations to synthesize computationally diverse program variants. In ISSTA, 2014. Draghi, Nature 2010 J.A. Draghi, T.L. Parsons, G.P. Wagner, and J.B. Plotkin. Mutational robustness can facilitate adaptation. Nature, 463(7279):353–355, 2010. Le Goues, ICSE 2012 Claire Le Goues, Michael Dewey-Vogt, Stephanie Forrest, and Westley Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In ICSE, 2012. Nguyen, SBST 2009 ThanhVu Nguyen, Westley Weimer, Claire Le Goues and Stephanie Forrest. Extended Abstract: Using Execution Paths to Evolve Software Patches, in SBST, 2009. Schulte, GPEM 2013 Eric Schulte, Zachary. Fry, Ethan Fast, Westley Weimer, and Stephanie Forrest. Software mutational robustness. Genetic Programming and Evolvable Machines, pages 1–32, 2013. Schulte, ASPLOS 2013 Eric Schulte, Jonathan DiLorenzo, Westley Weimer, and Stephanie Forrest. Automated repair of binary and assembly programs for cooperating embedded devices. In ASPLOS, 2013. Schulte, ASPLOS 2014 Eric Schulte, Jonathan Dorn, Stephen Harding, Stephanie Forrest, and Westley Weimer. Post-compiler software optimization for reducing energy. In ASPLOS, 2014. Weimer, ICSE 2009 Westley Weimer, ThanhVu Nguyen, Claire Le Goues and Stephanie Forrest. Automatically Finding Patches Using Genetic Programming, in ICSE, 2009 Neutral Networks and Automated Software Evolution References 46

slide-99
SLIDE 99

Backup Slides

◮ Program Space ◮ Program Expression ◮ Many Bugs ◮ LLVM SSA ◮ Robustness and Adaptation

Time

◮ Proactive Bug Repair ◮ Fault Localization ◮ Embedded Repair Results ◮ Distributed Repair ◮ Benefits of ASM ◮ Embedded Repair Space ◮ Profiling (GOA) ◮ Minimization (GOA) ◮ Energy Model (GOA) ◮ Runtime and Energy

Reduction (GOA)

◮ Physical Evolutionary

Computation

Neutral Networks and Automated Software Evolution Backup Slides 47

slide-100
SLIDE 100

Program Space

Back Total ni+1 = ni(d − i)(e − 1) (i + 1) Unique nz,i+1 = nz−1,i(d − i)1 + nz,i(d − i)(e − 2) i + 1 Ai =

  • z≤i

nz,i l z −1 Neutral R =

  • i≤d

r i

z≤i

nz,i l z −1

Constants: d=3, e=5

Step Total 1 1 2 3

Neutral Networks and Automated Software Evolution Backup Slides 48

slide-101
SLIDE 101

Program Space

Back Total ni+1 = ni(d − i)(e − 1) (i + 1) Unique nz,i+1 = nz−1,i(d − i)1 + nz,i(d − i)(e − 2) i + 1 Ai =

  • z≤i

nz,i l z −1 Neutral R =

  • i≤d

r i

z≤i

nz,i l z −1

Constants: d=3, e=5

Step Total 1 1 12 2 3

Neutral Networks and Automated Software Evolution Backup Slides 48

slide-102
SLIDE 102

Program Space

Back Total ni+1 = ni(d − i)(e − 1) (i + 1) Unique nz,i+1 = nz−1,i(d − i)1 + nz,i(d − i)(e − 2) i + 1 Ai =

  • z≤i

nz,i l z −1 Neutral R =

  • i≤d

r i

z≤i

nz,i l z −1

Constants: d=3, e=5

Step Total 1 1 12 2 48 3

Neutral Networks and Automated Software Evolution Backup Slides 48

slide-103
SLIDE 103

Program Space

Back Total ni+1 = ni(d − i)(e − 1) (i + 1) Unique nz,i+1 = nz−1,i(d − i)1 + nz,i(d − i)(e − 2) i + 1 Ai =

  • z≤i

nz,i l z −1 Neutral R =

  • i≤d

r i

z≤i

nz,i l z −1

Constants: d=3, e=5

Step Total 1 1 12 2 48 3 64

Neutral Networks and Automated Software Evolution Backup Slides 48

slide-104
SLIDE 104

Program Expression

Back Cil Source CLang LLVM ASM ELF Executable Serialize Serialize Compile Compile Link Serialize

Neutral Networks and Automated Software Evolution Backup Slides 49

slide-105
SLIDE 105

Systematic Evaluation

[Le Goues et al., ICSE 12] Back

Amazon EC2 Servers Version Control Repositories fbc gmp gzip libtiff . . . GenProg

Neutral Networks and Automated Software Evolution Backup Slides 50

slide-106
SLIDE 106

Fixing Bugs for $8 a Bug

Back

  • Avg. Cost
  • Avg. Cost

Defects per Non-Repair Per Repair Program Repaired Hours US$ Hours US$ fbc 1 / 3 8.52 5.56 6.52 4.08 gmp 1 / 2 9.93 6.61 1.60 0.44 gzip 1 / 5 5.11 3.04 1.41 0.30 libtiff 17 / 24 7.81 5.04 1.05 0.04 lighttpd 5 / 9 10.79 7.25 1.34 0.25 php 28 / 44 13.00 8.80 1.84 0.62 python 1 / 11 13.00 8.80 1.22 0.16 wireshark 1 / 7 13.00 8.80 1.23 0.17 total 55 / 105 11.22h 1.60h

Neutral Networks and Automated Software Evolution Backup Slides 51

slide-107
SLIDE 107

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-108
SLIDE 108

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-109
SLIDE 109

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-110
SLIDE 110

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-111
SLIDE 111

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-112
SLIDE 112

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-113
SLIDE 113

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-114
SLIDE 114

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x %3

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-115
SLIDE 115

LLVM SSA

Back %x = alloca i32, align 4 store i32 2, i32* %x, align 4 %0 = load i32* %x, align 4 %add = add nsw i32 %0, 3 store i32 %add, i32* %x, align 4 %1 = load i32* %x, align 4 %2 = load i32* %x, align 4 %mul = mul nsw i32 %1, %2 store i32 %mul, i32* %x, align 4 %3 = load i32* %x, align 4 %call = call ... @printf(@.str %3)

i32 2 x %0 i32 3 %add x %2 %1 %mul x %3 printf

Neutral Networks and Automated Software Evolution Backup Slides 52

slide-116
SLIDE 116

Robustness and Adaptation Time

Back

[Draghi et al., Nature 2010]

Neutral Networks and Automated Software Evolution Backup Slides 53

slide-117
SLIDE 117

Proactive Bug Repair

Back

1 2 3 4 5 6 7 8 2 4 6 8 10 12 14 16 1 10 100 1000 10000 Number of Unique Bugs Fixed Number of Selected Variants Number of Bugs Seeded Number of Selected Variants Number of Unique Bugs fixed

Figure: Bugs Fixed by Number of Variants in Potion

Neutral Networks and Automated Software Evolution Backup Slides 54

slide-118
SLIDE 118

Fault Localization

Back

◮ Focus Mutation ◮ Fault Localization

  • 1. instrument
  • 2. compile
  • 3. evaluate

if(a==0) puts(b); while(b!=0) if(a>b) a=a-b; else b=b-a; puts(a); Neutral Networks and Automated Software Evolution Backup Slides 55

slide-119
SLIDE 119

Fault Localization

Back

◮ Focus Mutation ◮ Fault Localization

  • 1. instrument
  • 2. compile
  • 3. evaluate

if(a==0) puts(b); while(b!=0) if(a>b) a=a-b; else b=b-a; puts(a); Neutral Networks and Automated Software Evolution Backup Slides 55

slide-120
SLIDE 120

Light Weight Fault Localization

1. Sample program counter. 2. Translate memory addresses to program offsets.

  • 3. Smooth sample with Gaussian convolution.

movq 8(%rdx), %rdi xorl %eax, %eax movl %eax, (%r15) addl $1, %r14d call atoi movq -80(%rbp), %rdx movq %rdx, -80(%rbp) addq $4, %r15 movq 8(%rdx), %rdi xorl %eax, %eax movl %eax, (%r15)

CPU Machine-code Instructions

Neutral Networks and Automated Software Evolution Backup Slides 56

slide-121
SLIDE 121

Light Weight Fault Localization

1. Sample program counter. 2. Translate memory addresses to program offsets.

  • 3. Smooth sample with Gaussian convolution.

movq 8(%rdx), %rdi xorl %eax, %eax movl %eax, (%r15) addl $1, %r14d call atoi movq -80(%rbp), %rdx movq %rdx, -80(%rbp) addq $4, %r15 movq 8(%rdx), %rdi xorl %eax, %eax movl %eax, (%r15)

CPU Machine-code Instructions

Neutral Networks and Automated Software Evolution Backup Slides 56

slide-122
SLIDE 122

Light Weight Fault Localization

1. Sample program counter. 2. Translate memory addresses to program offsets.

  • 3. Smooth sample with Gaussian convolution.

memory addr. to instruction

movq 8(%rdx), %rdi xorl %eax, %eax movl %eax, (%r15) addl $1, %r14d call atoi movq -80(%rbp), %rdx movq %rdx, -80(%rbp) addq $4, %r15 movq 8(%rdx), %rdi xorl %eax, %eax movl %eax, (%r15)

CPU Machine-code Instructions

Neutral Networks and Automated Software Evolution Backup Slides 56

slide-123
SLIDE 123

Light Weight Fault Localization

1. Sample program counter. 2. Translate memory addresses to program offsets.

  • 3. Smooth sample with Gaussian convolution.

memory addr. to instruction

movq 8(%rdx), %rdi xorl %eax, %eax movl %eax, (%r15) addl $1, %r14d call atoi movq -80(%rbp), %rdx movq %rdx, -80(%rbp) addq $4, %r15 movq 8(%rdx), %rdi xorl %eax, %eax movl %eax, (%r15)

CPU Machine-code Instructions

Neutral Networks and Automated Software Evolution Backup Slides 56

slide-124
SLIDE 124

Embedded Repair Results

Back

◮ Effective ◮ 62% faster runtime ◮ 95% smaller disk footprint ◮ 86% less memory

Total bugs repaired Rep.

  • Num. Bugs

AST 13 ASM 12 ELF 11 Average success rate 100 runs per bug Rep. Success Rate AST 78.17% ASM 70.75% ELF 65.83%

Neutral Networks and Automated Software Evolution Backup Slides 57

slide-125
SLIDE 125

Embedded Repair Results

Back

◮ Effective ◮ 62% faster runtime ◮ 95% smaller disk footprint ◮ 86% less memory

Expected fitness evaluations Rep. Evaluations AST 583.98 ASM 188.38 ELF 207.15 Total runtime Rep. Sec. AST 229.50 ASM 278.30 ELF 74.20

Neutral Networks and Automated Software Evolution Backup Slides 57

slide-126
SLIDE 126

Embedded Repair Results

Back

◮ Effective ◮ 62% faster runtime ◮ 95% smaller disk footprint ◮ 86% less memory

Example: Merge Sort

Repair by representation

◮ AST, 2 of 4900 Swaps ◮ ASM, 1 of 280 Deletes

merge.c

if(left[l-mid -1] <= right [0]){ result=list; } else{/* fix:swap branches */ result=merge(left ,l-mid , right ,mid ); }

merge.s

cmpl %eax , %edx ; fix: del. jg .L12 movq

  • 72(% rbp), %rax

Neutral Networks and Automated Software Evolution Backup Slides 57

slide-127
SLIDE 127

Embedded Repair Results

Back

◮ Effective ◮ 62% faster runtime ◮ 95% smaller disk footprint ◮ 86% less memory

Disk size Rep. Requirements AST Source code & build toolchain ASM Assembly code & linker ELF Compiled executable

Neutral Networks and Automated Software Evolution Backup Slides 57

slide-128
SLIDE 128

Embedded Repair Results

Back

◮ Effective ◮ 62% faster runtime ◮ 95% smaller disk footprint ◮ 86% less memory

Working memory Rep. MB AST 1402 ASM 756 ELF 200

Neutral Networks and Automated Software Evolution Backup Slides 57

slide-129
SLIDE 129

Distributed Embedded Repair

Back

1 2 3 4

Neutral Networks and Automated Software Evolution Backup Slides 58

slide-130
SLIDE 130

Distributed Embedded Repair

Back

1 2 3 4

Neutral Networks and Automated Software Evolution Backup Slides 58

slide-131
SLIDE 131

Distributed Embedded Repair

Back

1 2 3 4

delete(45) swap(23,86) delete(4) insert(4,23) Neutral Networks and Automated Software Evolution Backup Slides 58

slide-132
SLIDE 132

Distributed Embedded Repair

Back

1 2 3 4

delete(45) swap(23,86) delete(4) insert(4,23)

fault loc. active fault loc. active fault loc. active fault loc. active active

Neutral Networks and Automated Software Evolution Backup Slides 58

slide-133
SLIDE 133

Distributed Genetic Repair Evaluation

Relative performance of Distributed Algorithm

Expected Wall Clock # Nodes Fitness Evaluations Seconds w/SMS 1 1 1 1 2 0.94 0.89 1.07 3 0.84 0.67 0.81 4 0.80 0.55 0.63

Neutral Networks and Automated Software Evolution Backup Slides 59

slide-134
SLIDE 134

Distributed Genetic Repair Evaluation

Relative performance of Distributed Algorithm

Expected Wall Clock # Nodes Fitness Evaluations Seconds w/SMS 1 1 1 1 2 0.94 0.89 1.07 3 0.84 0.67 0.81 4 0.80 0.55 0.63

Neutral Networks and Automated Software Evolution Backup Slides 59

slide-135
SLIDE 135

Distributed Genetic Repair Evaluation

Relative performance of Distributed Algorithm

Expected Wall Clock # Nodes Fitness Evaluations Seconds w/SMS 1 1 1 1 2 0.94 0.89 1.07 3 0.84 0.67 0.81 4 0.80 0.55 0.63

Neutral Networks and Automated Software Evolution Backup Slides 59

slide-136
SLIDE 136

Benefits of ASM

Back

The ASM representation performs well.

Avoids

◮ direct addresses ◮ argumented instructions

Similarities to DNA

◮ Linear vector genome ◮ read sequentially ◮ reading frames ◮ start and stop codes ◮ padding ◮ bootstrapped compilers

Neutral Networks and Automated Software Evolution Backup Slides 60

slide-137
SLIDE 137

Embedded Repair Search Space

Back

Program size ≈ 3× more assembly instructions than C statements Search space size = |alphabet|program size Possible program coverage Possible Programs Reachable Programs Original Program

Neutral Networks and Automated Software Evolution Backup Slides 61

slide-138
SLIDE 138

Profiling

Back

Assembler Fitness Function Workload Population Mutate Profile Fitness Minimize Executable

Hardware Performance Counters

Neutral Networks and Automated Software Evolution Backup Slides 62

slide-139
SLIDE 139

Profiling

Back

Assembler Fitness Function Workload Population Mutate Profile Fitness Minimize Executable

$ perf stat -- ./blackscholes 1 input /tmp/output 6,864,315,342 cycles 5,062,293,918 instructions 2,944,060,039 r533f00 1,113,084,780 cache-references 1,122,960 cache-misses 3.227585368 seconds time elapsed

Neutral Networks and Automated Software Evolution Backup Slides 62

slide-140
SLIDE 140

Minimization

Back

Assembler Fitness Function Workload Population Mutate Profile Fitness Minimize Executable

Delta Debugging

5358 c5358 < .L808:

  • >

addl %ebx , %ecx 5416 c5416 < addl %ebx , %ecx

  • > .L808:

5463 c5463 < .L970:

  • >

.byte 0x33 5651 d5650 < .loc 1 457 0 is_stmt 0 discriminator 2 5841 d5839 < addq %rdx , %r14 6309 c6307 < xorpd %xmm1 , %xmm7

  • >

cmpq %r13 , %rdi 6413 a6412 > cmpl %ecx , %esi

Neutral Networks and Automated Software Evolution Backup Slides 63

slide-141
SLIDE 141

Minimization

Back

Assembler Fitness Function Workload Population Mutate Profile Fitness Minimize Executable

Delta Debugging

5358 c5358 < .L808:

  • >

addl %ebx , %ecx 5416 c5416 < addl %ebx , %ecx

  • > .L808:

5463 c5463 < .L970:

  • >

.byte 0x33 5651 d5650 < .loc 1 457 0 is_stmt 0 discriminator 2 5841 d5839 < addq %rdx , %r14 6309 c6307 < xorpd %xmm1 , %xmm7

  • >

cmpq %r13 , %rdi 6413 a6412 > cmpl %ecx , %esi

Neutral Networks and Automated Software Evolution Backup Slides 63

slide-142
SLIDE 142

Minimization

Back

Assembler Fitness Function Workload Population Mutate Profile Fitness Minimize Executable

Delta Debugging

5358 c5358 < .L808:

  • >

addl %ebx , %ecx 5416 c5416 < addl %ebx , %ecx

  • > .L808:

5463 c5463 < .L970:

  • >

.byte 0x33 5651 d5650 < .loc 1 457 0 is_stmt 0 discriminator 2 5841 d5839 < addq %rdx , %r14 6309 c6307 < xorpd %xmm1 , %xmm7

  • >

cmpq %r13 , %rdi 6413 a6412 > cmpl %ecx , %esi

Neutral Networks and Automated Software Evolution Backup Slides 63

slide-143
SLIDE 143

Minimization

Back

Assembler Fitness Function Workload Population Mutate Profile Fitness Minimize Executable

Delta Debugging

5358 c5358 < .L808:

  • >

addl %ebx , %ecx 5416 c5416 < addl %ebx , %ecx

  • > .L808:

5463 c5463 < .L970:

  • >

.byte 0x33 5651 d5650 < .loc 1 457 0 is_stmt 0 discriminator 2 5841 d5839 < addq %rdx , %r14 6309 c6307 < xorpd %xmm1 , %xmm7

  • >

cmpq %r13 , %rdi 6413 a6412 > cmpl %ecx , %esi

Neutral Networks and Automated Software Evolution Backup Slides 63

slide-144
SLIDE 144

Energy Model

Back

energy time = Cconst+Cins ins cycle +Cflops flops cycle +Ctca tca cycle +Cmem mem cycle

Intel AMD Coefficient Description (4-core) (48-core) Cconst constant power draw 31.530 394.74 Cins instructions 20.490

  • 83.68

Cflops floating point ops. 9.838 60.23 Ctca cache accesses

  • 4.102
  • 16.38

Cmem cache misses 2962.678

  • 4209.09

Table: Energy model coefficients.

Neutral Networks and Automated Software Evolution Backup Slides 64

slide-145
SLIDE 145

GOA: Runtime and Energy Reduction

Back

  • 10%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% b l a c k s c h

  • l

e s b

  • d

y t r a c k f e r r e t fl u i d a n i m a t e f r e q m i n e s w a p t i

  • n

s v i p s x 2 6 4 a v e r a g e

AMD Energy and Runtime Reduction

Energy Runtime

Neutral Networks and Automated Software Evolution Backup Slides 65

slide-146
SLIDE 146

Physical Evolutionary Computation

Back

Subpopulation

(64 individuals) ...79965+* ...99965+* ...9804++* ...-38*8*+ ...x7558** ...8679++* ...y894**5

RPN Evaluation

9728*** 34 12 stack

Tournament

*6*y-8*

Crossover

y + x 6 7 4 * 6 * y

  • 8

* 1 / 9

  • 4

2 9 9 6 5 + * * y+x6749965+** 1/9-42*6*y-8*

Mutation

0-51x7y58*** 0-51x7558***

Hardware Tile

◮ 70MHz ARM7TDMI MCU ◮ 32KB RAM, 16KB EEPROM ◮ 128KB problem data storage ◮ hardware timers ◮ power sharing, 4 serial ports

SoftwareProcess

∀ evolutionary tile randomly create an initial population for ever, concurrently with frequency fm select one individual and mutate with frequency fc select two individuals and crossover with frequency fs select one individual and share with frequency fd if collecting data report statistics to central collector

Real Time Genetic Operator Timelines

mutation (fm = 20Hz) crossover (fc = 10Hz) sharing (fs = 0.1Hz)

Computational Space

Sample geometry: Sixteen 4.75cm × 4.75cm tiles arranged in a 4 × 4 square.

Problem Space

Sample fitness function: Gravity data from the state of Montana. Each hardware tile covers 25′ latitude × 25′ longitude mapped using the ‘local disjoint’ spatial mapping. The size and shape of the total covered area is determined by the number and configuration of the hardware tiles, and by the scale and scheme used to map the hardware space to the problem space.

Neutral Networks and Automated Software Evolution Backup Slides 66

slide-147
SLIDE 147

The End

Back

Eric Schulte

email eschulte@cs.unm.edu homepage https://cs.unm.edu/~eschulte dissertation https://cs.unm.edu/~eschulte/dissertation

Neutral Networks and Automated Software Evolution Backup Slides 67