Finding and Understanding Bugs in Software Model Checkers Chengyu - - PowerPoint PPT Presentation

Finding and Understanding Bugs in Software Model Checkers

Chengyu Zhang, Ting Su, Yichen Yan, Fuyuan Zhang, Geguang Pu, Zhendong Su

Software Model Checking

2

Software Model Checking

P

3

Software Model Checking

P

ϕ

4

Software Model Checking

P

ϕ

Software Model Checker

5

Software Model Checking

P

ϕ

Software Model Checker

Determines whether ϕ Holds on P

6

Software Model Checking

P

ϕ

safe

Software Model Checker

7

Determines whether ϕ Holds on P

Software Model Checking

P

ϕ

Software Model Checker

safe

P’

¬ϕ

8

Determines whether ϕ Holds on P

Software Model Checking

P

ϕ

Software Model Checker

safe

P’

¬ϕ

unsafe

9

Determines whether ϕ Holds on P

Example: Reachability Safety Property

10

int main() { int i = 0; if (i > 0) { …… //unreachable __VERIFIER_error(); } return 0; }

Determines whether this function is reachable

Example: Reachability Safety Property

11

int main() { int i = 0; if (i > 0) { …… //unreachable __VERIFIER_error(); } return 0; }

Example: Reachability Safety Property

int main() { int i = 0; if (i > 0) { …… //unreachable __VERIFIER_error(); } return 0; }

12

SAFE!

Example: Reachability Safety Property

int main() { int i = 0; if (i > 0) { …… //unreachable __VERIFIER_error(); } return 0; }

12

Example: Reachability Safety Property

int main() { int i = 0; if (i == 0) { …… //reachable __VERIFIER_error(); } return 0; }

13

UNSAFE!

Example: Reachability Safety Property

int main() { int i = 0; if (i == 0) { …… //reachable __VERIFIER_error(); } return 0; }

13

Example: Reachability Safety Property

SAFE!

software model checker:

14

int main() { int i = 0; if (i == 0) { …… //reachable __VERIFIER_error(); } return 0; }

Example: Reachability Safety Property

SAFE!

software model checker:

15

int main() { int i = 0; if (i == 0) { …… //reachable __VERIFIER_error(); } return 0; }

Bug#529 in CPAchecker

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

16

Bug#529 in CPAchecker

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

i:0

17

Bug#529 in CPAchecker

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

i:0

18

Bug#529 in CPAchecker

i:1

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

19

Bug#529 in CPAchecker

i:1

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

20

Bug#529 in CPAchecker

i:1

!

UNSAFE

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

21

Bug#529 in CPAchecker

i:1

SAFE!

!

UNSAFE

22

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

Bug#529 in CPAchecker

i:0 *(&i):1

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

23

Bug#529 in CPAchecker

i:0 *(&i):1

void main() { int i = 0; while (1) { if (i > 0) { __VERIFIER_error(); break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

SAFE!

24

Workflow

Software Model Checker instrument Checking results profile Compare Bug reports [ inconsistent ] [ consistent ]

Seed programs Test programs

Intermediate Programs

Oracles

1 2

mutate

3 4 validate

25

Workflow

Software Model Checker instrument Checking results profile Compare Bug reports [ inconsistent ] [ consistent ]

Seed programs Test programs

Intermediate Programs

Oracles

1 2

mutate

3 4 validate

26

Approach I: Enumerative Reachability (ER)

void main() { int i = 0; while (1) {

void main() { int i = 0; while (1) { if (i > 0) {

void main() { int i = 0; while (1) { if (i > 0) { break; } if (i == 0){

if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } } break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

*(&i) = *(&i) + 1; } } }

Bug#529 in CPAchecker Bug#529 in CPAchecker Bug#529 in CPAchecker

27

Approach I: Enumerative Reachability (ER)

void main() { int i = 0; while (1) {

void main() { int i = 0; while (1) { if (i > 0) {

void main() { int i = 0; while (1) { if (i > 0) { break; } if (i == 0){

if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } } break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

*(&i) = *(&i) + 1; } } } __VERIFIER_error(); __VERIFIER_error(); __VERIFIER_error();

Intermediate Program 1 Intermediate Program 2 Intermediate Program 3

Bug#529 in CPAchecker Bug#529 in CPAchecker Bug#529 in CPAchecker

27

Workflow

Software Model Checker instrument Checking results profile Compare Bug reports [ inconsistent ] [ consistent ]

Seed programs Test programs

Intermediate Programs

Oracles

1 2

mutate

3 4 validate

28

Approach I: Enumerative Reachability (ER)

void main() { int i = 0; while (1) {

void main() { int i = 0; while (1) { if (i > 0) {

void main() { int i = 0; while (1) { if (i > 0) { break; } if (i == 0){

if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } } break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

*(&i) = *(&i) + 1; } } } __VERIFIER_error(); __VERIFIER_error(); __VERIFIER_error();

Intermediate Program 1 Intermediate Program 2 Intermediate Program 3

Actual execution: unsafe Actual execution: unsafe Actual execution: unsafe

Bug#529 in CPAchecker Bug#529 in CPAchecker Bug#529 in CPAchecker

29

Workflow

Software Model Checker instrument Checking results profile Compare Bug reports [ inconsistent ] [ consistent ]

Seed programs Test programs

Intermediate Programs

Oracles

1 2

mutate

3 4 validate

30

Approach I: Enumerative Reachability (ER)

void main() { int i = 0; while (1) {

void main() { int i = 0; while (1) { if (i > 0) {

void main() { int i = 0; while (1) { if (i > 0) { break; } if (i == 0){

if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } } break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

*(&i) = *(&i) + 1; } } } __VERIFIER_error(); __VERIFIER_error(); __VERIFIER_error();

Test program 1 Test program 2 Test program 3 Actual execution: unsafe Actual execution: unsafe Actual execution: unsafe Model checker: unsafe Model checker: safe Model checker: unsafe

Bug#529 in CPAchecker Bug#529 in CPAchecker Bug#529 in CPAchecker

31

Approach I: Enumerative Reachability (ER)

void main() { int i = 0; while (1) {

void main() { int i = 0; while (1) { if (i > 0) {

void main() { int i = 0; while (1) { if (i > 0) { break; } if (i == 0){

if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } } break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

*(&i) = *(&i) + 1; } } } __VERIFIER_error(); __VERIFIER_error(); __VERIFIER_error();

Test program 1 Test program 2 Test program 3 Actual execution: unsafe Actual execution: unsafe Actual execution: unsafe Model checker: unsafe Model checker: safe Model checker: unsafe

Bug#529 in CPAchecker Bug#529 in CPAchecker Bug#529 in CPAchecker

32

Approach

• Approach I: Enumerative Reachability (ER)

33

Bug#534 in CPAchecker

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

a[3]:{1,0,0}

34

Bug#534 in CPAchecker

a[3]:{1,1,1}

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

35

Bug#534 in CPAchecker

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

a[3]:{1,1,1}: unsafe a[3]:{1,0,0}: unsafe

36

Bug#534 in CPAchecker

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

__VERIFIER_error();

a[3]:{1,1,1}: unsafe a[3]:{1,0,0}: unsafe

36

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; } int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

Approach II: Enumerative Counting Reachability (ECR)

37

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; } int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0; br++; int br = 0; br++; GetValue(br) GetValue(br)

Approach II: Enumerative Counting Reachability (ECR)

37

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; } int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0; br++; int br = 0; br++; GetValue(br) GetValue(br)

Actual execution: br=1 Actual execution: br=3

Approach II: Enumerative Counting Reachability (ECR)

37

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; } int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0; br++; if(br != 1) __VERIFIER_error(); int br = 0; br++; if(br != 3) __VERIFIER_error();

Actual execution: br=1 Actual execution: br=3

Approach II: Enumerative Counting Reachability (ECR)

37

Approach II: Enumerative Counting Reachability (ECR)

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; } int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0; br++; if(br != 1) __VERIFIER_error(); int br = 0; br++; if(br != 3) __VERIFIER_error();

Test oracle: safe Test oracle: safe

38

Approach II: Enumerative Counting Reachability (ECR)

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0; br++; if(br != 1) __VERIFIER_error();

Model checker: unsafe Model checker: safe

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0; br++; if(br != 3) __VERIFIER_error();

39

Test oracle: safe Test oracle: safe

Approach

• Approach I: Enumerative Reachability (ER)
• Approach II: Enumerative Counting Reachability (ECR)

40

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

Approach III: Fused Counting Reachability (FCR)

41

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0;int br2 = 0; br1++; GetValue(br1)

Approach III: Fused Counting Reachability (FCR)

br2++; GetValue(br2)

Actual execution: br1=1; br2=3

41

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0;int br2 = 0; br1++; if(br1 != 3 || br2 != 1) __VERIFIER_error();

Approach III: Fused Counting Reachability (FCR)

br2++;

41

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0;int br2 = 0; br1++; if(br1 != 3 || br2 != 1) __VERIFIER_error();

Approach III: Fused Counting Reachability (FCR)

br2++;

Test oracle: safe

41

int main(void){ int a[3] = {1}; int i = 0; while(i < 3){ if(a[i] == 1) { …… } i++; } return 0; }

int br = 0;int br2 = 0; br1++; if(br1 != 3 || br2 != 1) __VERIFIER_error();

Approach III: Fused Counting Reachability (FCR)

br2++;

Test oracle: safe

Model checker: unsafe

41

Approach

• Approach I: Enumerative Reachability (ER)
• Approach II: Enumerative Counting Reachability (ECR)
• Approach III: Fused Counting Reachability (FCR)

42

Approach

• Approach I: Enumerative Reachability (ER)
• Approach II: Enumerative Counting Reachability (ECR)
• Approach III: Fused Counting Reachability (FCR)

42

find more kinds of bugs

Approach

• Approach I: Enumerative Reachability (ER)
• Approach II: Enumerative Counting Reachability (ECR)
• Approach III: Fused Counting Reachability (FCR)

42

find more kinds of bugs save more time

Evaluation Setup

43

Evaluation Setup

GCC test suite

43

Evaluation Setup

4,609 Files 219,636 Loc GCC test suite

43

Evaluation Setup

4,609 Files 219,636 Loc GCC test suite IC3 based

43

Evaluation Setup

4,609 Files 219,636 Loc GCC test suite IC3 based CEGAR based

43

Evaluation Setup

4,609 Files 219,636 Loc GCC test suite IC3 based CEGAR based BMC based

43

Evaluation Setup

4,609 Files 219,636 Loc GCC test suite IC3 based CEGAR based BMC based

43

RQ1: Can our approaches find bugs?

44

RQ1: Can our approaches find bugs?

45

RQ1: Can our approaches find bugs?

46

RQ1: Can our approaches find bugs?

47

RQ1: Can our approaches find bugs?

48

RQ1: Can our approaches find bugs?

49

RQ1: Can our approaches find bugs?

50

RQ1: Can our approaches find bugs?

51

RQ2: How many bugs can be found by each approach?

52

Approach I

52

53

RQ2: How many bugs can be found by each approach?

Approach I

52

Approach II

61

54

RQ2: How many bugs can be found by each approach?

Approach I

52

Approach II/III

61

55

RQ2: How many bugs can be found by each approach?

Approach I

52

Approach II/III

61

56

RQ2: How many bugs can be found by each approach?

Approach I

52

Approach II/III

61

56

RQ2: How many bugs can be found by each approach?

1 10 51

Approach I

Approach II/III

57

RQ2: How many bugs can be found by each approach?

Bug#529 in CPAchecker (False negative in approach II/III)

void main() { int i = 0; while (1) { if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

58

i:0 *(&i):1

Bug#529 in CPAchecker (False negative in approach II/III)

void main() { int i = 0; while (1) { if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

int br = 0; br++; if(br != 1) __VERIFIER_error();

Test oracle: safe

58

i:0 *(&i):1

Bug#529 in CPAchecker (False negative in approach II/III)

void main() { int i = 0; while (1) { if (i > 0) { break; } if (i == 0){ *(&i) = *(&i) + 1; } } }

int br = 0; br++; if(br != 1) __VERIFIER_error();

Test oracle: safe Buggy model checker: safe

58

i:0 *(&i):1

RQ3: How much time does each approach consume?

59

60

RQ3: How much time does each approach consume?

61

RQ3: How much time does each approach consume?

Save 89% of time

62

RQ3: How much time does each approach consume?

63

1 10 51

Approach I

Approach II/III

RQ3: How much time does each approach consume?

Assorted Bug Samples

• Front-end
• Memory model
• Pointer alias
• Third-party component
• C standard library
• Language feature
• Configuration

64

https://github.com/MCFuzzer/MCFuzz/issues

void f(int a, int b){ if (a == b) __VERIFIER_error(); } int main(){ int d = 0; int c = 4; int e = 2; f (d=c&&e, 1); return 0; }

Example: Front-end related bug in CPAchecker

65

Example: Front-end related bug in CPAchecker

d = c&&e = 1

Test oracle: unsafe

void f(int a, int b){ if (a == b) __VERIFIER_error(); } int main(){ int d = 0; int c = 4; int e = 2; f (d=c&&e, 1); return 0; }

66

__CPAchecker_TMP_0 = c&&e

d = __CPAchecker_TMP_0

Example: Front-end related bug in CPAchecker

Test oracle: unsafe

void f(int a, int b){ if (a == b) __VERIFIER_error(); } int main(){ int d = 0; int c = 4; int e = 2; f (d=c&&e, 1); return 0; }

67

__CPAchecker_TMP_0 = c&&e

d = __CPAchecker_TMP_0

Example: Front-end related bug in CPAchecker

Test oracle: unsafe

void f(int a, int b){ if (a == b) __VERIFIER_error(); } int main(){ int d = 0; int c = 4; int e = 2; f (d=c&&e, 1); return 0; }

67

__CPAchecker_TMP_0 = c&&e

d = __CPAchecker_TMP_0

Example: Front-end related bug in CPAchecker

Test oracle: unsafe : safe

void f(int a, int b){ if (a == b) __VERIFIER_error(); } int main(){ int d = 0; int c = 4; int e = 2; f (d=c&&e, 1); return 0; }

67

struct { int a:4; int :4; int b:4; int c:4; } x = { 2,3,4 }; int main (){ if (x.b != 3) __VERIFIER_error(); return 0; }

Example: Language feature related bug in CBMC

2 u 3 4

x

“Unnamed members of objects of structure type do not participate in initialization.” —— C standard

a c b _

Test oracle: safe

68

struct { int a:4; int :4; int b:4; int c:4; } x = { 2,3,4 }; int main (){ if (x.b != 3) __VERIFIER_error(); return 0; } 2 3 4 u

x a c b _

Example: Language feature related bug in CBMC

“Unnamed members of objects of structure type do not participate in initialization.” —— C standard

Test oracle: safe

69

struct { int a:4; int :4; int b:4; int c:4; } x = { 2,3,4 }; int main (){ if (x.b != 3) __VERIFIER_error(); return 0; } 2 3 4 u

x a c b _

Example: Language feature related bug in CBMC

“Unnamed members of objects of structure type do not participate in initialization.” —— C standard

: unsafe Test oracle: safe

69

void test(int x,int y, int q){ if ((x / y) != q ) __VERIFIER_error(); } int main (){ test (7, 6, 1); test (-7, -6, 1); return 0; }

Example: Configuration related bug in Seahorn

7/6 == 1

• 7/-6 == 1

Test oracle: safe

70

void test(int x,int y, int q){ if ((x / y) != q ) __VERIFIER_error(); } int main (){ test (7, 6, 1); test (-7, -6, 1); return 0; }

7/6 == 1

• 7/-6 == 1

sea pf file.c

Example: Configuration related bug in Seahorn

Test oracle: safe

71

void test(int x,int y, int q){ if ((x / y) != q ) __VERIFIER_error(); } int main (){ test (7, 6, 1); test (-7, -6, 1); return 0; }

7/6 == 1

• 7/-6 == 1

sea pf file.c

Example: Configuration related bug in Seahorn

Test oracle: safe : unsafe

71

7/6 == 1

• 7/-6 == 1

sea pf —inline file.c

Example: Configuration related bug in Seahorn

Test oracle: safe

void test(int x,int y, int q){ if ((x / y) != q ) __VERIFIER_error(); } int main (){ test (7, 6, 1); test (-7, -6, 1); return 0; }

72

7/6 == 1

• 7/-6 == 1

sea pf —inline file.c

Example: Configuration related bug in Seahorn

: safe Test oracle: safe

void test(int x,int y, int q){ if ((x / y) != q ) __VERIFIER_error(); } int main (){ test (7, 6, 1); test (-7, -6, 1); return 0; }

72

Evaluation on SV-COMP benchmarks

73

Evaluation on SV-COMP benchmarks

• SV-COMP benchmarks
• Benchmarks for software verification competition.

74

Evaluation on SV-COMP benchmarks

• SV-COMP benchmarks
• Benchmarks for software verification competition.
• Selected 1106 files from ReachSafety and SoftwareSystems.

75

Evaluation on SV-COMP benchmarks

• SV-COMP benchmarks
• Benchmarks for software verification competition.
• Selected 1106 files from ReachSafety and SoftwareSystems.
• Result
• Found 5 bugs via SV-COMP benchmark.

76

Evaluation on SV-COMP benchmarks

• SV-COMP benchmarks
• Benchmarks for software verification competition.
• Selected 1106 files from ReachSafety and SoftwareSystems.
• Result
• Found 5 bugs via SV-COMP benchmark.
• Undefined behavior
• It is not the bug, but leads to false alarms.

77

Usability of software model checkers

78

Usability of software model checkers

✓ Should not give wrong checking result. ✓ Should not let user to choose the right configuration. ✓ Better user manual.

79

Usability of software model checkers

✓ Should not give wrong checking result. ✓ Should not let user to choose the right configuration. ✓ Better user manual.

80

Usability of software model checkers

✓ Should not give wrong checking result. ✓ Should not let user to choose the right configuration. ✓ Better user manual.

81

Developers’ Feedback

82

Software model checker may have bugs

83

Software model checker may have bugs Proposed the approaches to find the bug

83

Software model checker may have bugs Proposed the approaches to find the bug Totally found 62 bugs in three model checkers

83

Software model checker may have bugs Proposed the approaches to find the bug Totally found 62 bugs in three model checkers Categorized the bugs into 7 categories

83

Thank you !

84

Software model checker may have bugs Proposed the approaches to find the bug Totally found 62 bugs in three model checkers Categorized the bugs into 7 categories

84