ReCrash Making crashes reproducible by preserving object states - - PowerPoint PPT Presentation

▶
recrash
SMART_READER_LITE
LIVE PREVIEW

ReCrash Making crashes reproducible by preserving object states - - PowerPoint PPT Presentation

Sep 02, 2023 246 likes •966 views

ReCrash Making crashes reproducible by preserving object states Shay Artzi, Sunghun Kim*, Michael D. Ernst MIT * now at HKUST Eclipse bug 30280: 2 days to reproduce, 4 minutes to fix 2003-01-27 08:01 User: Eclipse crashed I have no idea

slide-1
SLIDE 1

ReCrash

Making crashes reproducible by preserving object states

Shay Artzi, Sunghun Kim*, Michael D. Ernst MIT

* now at HKUST

slide-2
SLIDE 2

Eclipse bug 30280: 2 days to reproduce, 4 minutes to fix

2003-01-27 08:01 User: Eclipse crashed… I have no idea why... Here is the stack trace. 2003-01-27 08:26 Developer: What build are you using? Do you have a testcase to reproduce? 2003-01-27 08:39 Developer: Which JDK are you using? 2003-01-28 13:06 User: I’m running Eclipse 2.1, … I was not able to reproduce the crash. 2003-01-29 04:33 Developer: Reproduced. 2003-01-29 04:37 Developer: Fixed.

slide-3
SLIDE 3

Reproducing crashes

  • If a crash can’t be reproduced:

– Hard to fix – Hard to validate a solution

  • Reproducing a crash is hard!

– Nondeterminism – Configuration and system information – Steps to reproduce may be complex or long – In-field detection – Users rarely provide reproducible bug reports

slide-4
SLIDE 4

Approach 1: Postmortem analysis

Examples: stack trace, core dump Problems:

  • Fault (bug) may be far from failure (exception)

– Faulty method may not be in stack trace

  • Too much information

– Core dump: big; hard to interpret

  • Not enough information

– Shows effects (final values), not causes – Need initial values to reproduce the failure

slide-5
SLIDE 5

Approach 2: Record & replay

  • Logging: record interactions with environment
  • Replay: use log to reproduce the execution
  • Checkpoint: replay skips part of the execution

Problems:

  • Unrealistic overhead
  • Invasive changes to HW/OS/application
slide-6
SLIDE 6

Record & replay for OO programs

  • Object-oriented style uses only nearby state

– Unit testing depends on this property

  • ReCrash reproduces this nearby state

– Does not replay an execution – Static and dynamic analyses reduce the size

  • Lightweight: efficient, no harness, usable in-

field

  • Not guaranteed to reproduce every failure
slide-7
SLIDE 7

ReCrash technique

Goal: Convert a crash into a set of unit tests

  • 1. Monitoring: maintain a shadow stack

– Contains a copy of each method argument – On program crash, write the shadow stack to a file

  • 2. Test generation: create many unit tests

– For each stack frame, create one unit test:

  • Invoke the method using arguments from the shadow

stack

  • If the test does not reproduce the crash, discard the

test

slide-8
SLIDE 8

Maintaining the shadow stack

  • On method entry:

– Push a new shadow stack frame – Copy the actual arguments to the shadow stack

  • On non-exceptional method exit:

– Pop the shadow stack frame

  • On program failure (top-level exception):

– Write the shadow stack to a file

  • Serializes all state referenced by the shadow stack
slide-9
SLIDE 9

Create one JUnit test per stack frame

public void test_resolveType() { AllocExpr rec = (AllocExpr) shadowStack.getArg(0); BlockScope arg = (BlockScope) shadowStack.getArg(1); rec.resolveType(arg); }

We expect the method to fail as it did at run time

Test case for Eclipse bug 30280 Read arguments from the saved shadow stack Invoke the method from the stack frame

slide-10
SLIDE 10

Evaluating unit tests

  • Run each generated unit test
  • Discard the test if it does not reproduce the

run-time exception

slide-11
SLIDE 11

How a developer uses the tests

  • In a debugger, step through execution and

examine fields

  • Experiment by modifying the tests
  • Verify a fix
  • Create a regression test

– Replace deserialized objects by real objects or mock objects – More readable and robust

slide-12
SLIDE 12

Why create multiple tests?

  • Not all tests may reproduce the failure

– Due to state not captured on the shadow stack

  • Sockets, files, nondeterminism, distant program state
  • Does capture all values that are passed as arguments
  • Some tests may not be useful for debugging
slide-13
SLIDE 13

Not every test is useful

Stack trace:

NullPointerException at Class1.toString at Class2.myMethod ...

Tests:

void test_toString() { Class1 receiver = null; receiver.toString(); } void test_myMethod() { Class2 receiver = (Class2) shadowStack.getArg(0); receiver.myMethod(); }

slide-14
SLIDE 14

Other features of ReCrash

  • Non-crashing failures

– Add a ReCrash annotation

  • Caught exceptions that lead to later failures
  • Adding extra information to test cases

– Version number, configuration information

  • Reducing the serialized stack

– Size, privacy

slide-15
SLIDE 15

Cost of monitoring

Key cost: copying arguments to shadow stack Tradeoff: less information in shadow stack  lower chance of reproducing failures

  • 1. Depth of copy

– Deep, reference, or a hybrid

  • 2. Save less information about each argument

– Focus on important fields

  • 3. Monitor fewer methods

– Ignore methods not likely to crash or to be useful

slide-16
SLIDE 16

Original program execution

Real stack

slide-17
SLIDE 17

Original program execution

Real stack R: 17

slide-18
SLIDE 18

Original program execution

Real stack R: 17

slide-19
SLIDE 19

Original program execution

Real stack R: 18

slide-20
SLIDE 20

Original program execution

Real stack R: R: A1: 18

slide-21
SLIDE 21

Original program execution

Real stack R: R: A1: R: A1: A2: 18

slide-22
SLIDE 22
  • 1. Depth of copying
slide-23
SLIDE 23

Deep copy

Real stack Shadow stack

slide-24
SLIDE 24

Deep copy

Real stack R: 17 R:

17

Shadow stack

slide-25
SLIDE 25

Deep copy

Real stack R: 18 R:

17

Shadow stack

slide-26
SLIDE 26

Deep copy

Real stack R: R: A1: 18 R: R: A1:

17 18

Shadow stack

slide-27
SLIDE 27

Deep copy

R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18 18

Real stack Shadow stack

slide-28
SLIDE 28

Deep copy

R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18 18

Real stack

Multiple copies  quadratic cost Unusable in practice

Shadow stack

slide-29
SLIDE 29

Reference copy

Real stack Shadow stack

slide-30
SLIDE 30

Reference copy

Real stack R: 17 R: Shadow stack

slide-31
SLIDE 31

Reference copy

Real stack R: 18 R: Shadow stack

slide-32
SLIDE 32

Reference copy

Real stack R: R: A1: 18 R: R: A1: Shadow stack

slide-33
SLIDE 33

Reference copy

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2: Shadow stack

slide-34
SLIDE 34

Depth-1 copy

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18 18

Shadow stack

slide-35
SLIDE 35
  • 2. Ignoring some fields
slide-36
SLIDE 36

Depth-1 copy

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18 18

Shadow stack

slide-37
SLIDE 37

Used fields

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18 18

Shadow stack Analysis results

slide-38
SLIDE 38

Depth1 + used fields (= Depth2 – unused fields)

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18 18

Shadow stack

slide-39
SLIDE 39

Pure methods

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18 18

Shadow stack Analysis results

slide-40
SLIDE 40

Pure methods

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: A2:

17 18

A1: R: Shadow stack

slide-41
SLIDE 41

Immutable objects

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: A2:

17 18

A1: R: Shadow stack Analysis results

slide-42
SLIDE 42

Immutable objects

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18

Shadow stack

slide-43
SLIDE 43
  • 3. Ignoring some methods
slide-44
SLIDE 44

Ignored methods

Real stack R: R: A1: R: A1: A2: 18 R: R: A1: A2:

17

R: A1:

18

Shadow stack Analysis results

slide-45
SLIDE 45

Ignored methods

Real stack R: R: A1: R: A1: A2: R: R: A1: A2:

17

18 Shadow stack

slide-46
SLIDE 46

Methods that are unlikely to be useful

  • Trivial methods
  • Private methods
  • Library methods
  • Methods that are unlikely to crash
slide-47
SLIDE 47

Second chance mode

Idea: monitor only methods that are likely to crash

  • Initially, monitor no methods
  • After a crash, add monitoring for methods in the

stack trace

– Can update all clients, not just the one that crashed

  • Tradeoffs:

+ Very low overhead (no overhead until a crash) – Requires a failure to occur twice

slide-48
SLIDE 48

Experimental study

  • 1. Can ReCrash reproduce failures?
  • 2. Are the ReCrash-generated tests useful?
  • 3. How large are the test cases?
  • 4. What is the overhead of running ReCrash?
slide-49
SLIDE 49

Subject programs

Investigated 11 real crashes from:

– BST: .2 KLOC – SVNKit: 22 KLOC – Eclipse compiler: 83 KLOC – Javac-jsr308: 86 KLOC

slide-50
SLIDE 50

Q1: Can ReCrash reproduce failures?

Program Failure Candidate tests Reproducible tests reference copy depth 1 + used-fields deep copy BST Class cast 3 3 3 3 Class cast 3 3 3 3 Unsupported 3 3 3 3 SVNKit Index bounds 3 3 3 3 Null pointer 2 2 2 2 Null pointer 2 2 2 2 Eclipsec Null pointer 13 1 8 Javac- jsr308 Null pointer 17 5 5 5 Illegal arg 23 11 11 11 Null pointer 8 1 1 1 Index bounds 28 11 11 11

slide-51
SLIDE 51

Q1: Can ReCrash reproduce failures?

Program Failure Candidate tests Reproducible tests reference copy depth 1 + used-fields deep copy BST Class cast 3 3 3 3 Class cast 3 3 3 3 Unsupported 3 3 3 3 SVNKit Index bounds 3 3 3 3 Null pointer 2 2 2 2 Null pointer 2 2 2 2 Eclipsec Null pointer 13 1 8 Javac- jsr308 Null pointer 17 5 5 5 Illegal arg 23 11 11 11 Null pointer 8 1 1 1 Index bounds 28 11 11 11

slide-52
SLIDE 52

Q2: Are the ReCrash tests useful?

  • Developers found the tests useful

– Developer 1: “You don’t have to wait for the crash to occur again”; also liked multiple tests – Developer 2: “Using ReCrash, I was able to jump (almost directly) to the necessary breakpoint”

  • Developers found the stack trace insufficient

– Unable to reproduce – The failure may be far removed from the fault

slide-53
SLIDE 53

Q3: How large are the test cases?

  • The JUnit test suite uses the shadow stack
  • Serializes all reachable parts of the heap

Program Average shadow stack size (KB) BST 12 SVNKit 34 Eclipsec 62 Javac-jsr308 422

slide-54
SLIDE 54

Q4: Time overhead of ReCrash

0.5 1 1.5 2 Eclipsec SVNKit Original Reference Depth 1 + used-fields Second chance

Overhead of instrumented program in the field

slide-55
SLIDE 55

Q4: Memory overhead of ReCrash

Absolute memory overhead: .2M – 4.7 M

slide-56
SLIDE 56

Generating unit tests from system runs

  • Test factoring [Saff 2005, Elbaum 2006]

– Developer selects a portion of the program – System logs interactions with the environment – Unit test replays execution in a test harness

  • Contract-driven development [Leitner 2007]

– Reference copying, intended for durable tests

  • Backward-in-time debuggers [Lienhard 2008]

– Heavier-weight logging and checkpoints

slide-57
SLIDE 57

Future work

  • Capture more state

– Concurrency, timing, external resources

  • Other implementation tradeoffs

– Copy-on-write – Existing VM hooks – Logging/debugging techniques – These are probably orthogonal to ReCrash

slide-58
SLIDE 58

ReCrash converts failures into tests

  • ReCrash effectively reproduces failures
  • Replicates program states
  • Generates multiple unit tests
  • The unit tests are useful
  • Low overhead
  • Records only relevant parts of an execution
  • 4 program analyses; second chance mode
  • Can deploy instrumented programs in the field
  • Download: http://pag.csail.mit.edu/ReCrash/
slide-59
SLIDE 59

ReCrash converts failures into tests

  • ReCrash effectively reproduces failures
  • Replicates program states
  • Generates multiple unit tests
  • The unit tests are useful
  • Low overhead
  • Records only relevant parts of an execution
  • 4 program analyses; second chance mode
  • Can deploy instrumented programs in the field
  • Download: http://pag.csail.mit.edu/ReCrash/
slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62

Maintaining the shadow stack

Real stack R: R: A1: 18 R: R: A1:

17 18

Shadow stack

slide-63
SLIDE 63

Maintaining the shadow stack

R: R: A1: R: A1: A2: 18 R: R: A1:

17 18

Real stack Shadow stack On method entry

slide-64
SLIDE 64

Maintaining the shadow stack

R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18

Real stack Shadow stack On method entry:

  • 1. Push a new shadow stack frame
slide-65
SLIDE 65

Maintaining the shadow stack

R: R: A1: R: A1: A2: 18 R: R: A1: R: A1: A2:

17 18 18

Real stack Shadow stack On method entry:

  • 1. Push a new shadow stack frame
  • 2. Copy the actual arguments to the

shadow stack

slide-66
SLIDE 66

Maintaining the shadow stack

R: R: A1: 18 R: R: A1: R: A1: A2:

17 18 18

Real stack Shadow stack On method exit

slide-67
SLIDE 67

Maintaining the shadow stack

R: R: A1: 18 R: R: A1:

17 18

Real stack Shadow stack On method exit:

  • 1. Pop shadow stack frame
slide-68
SLIDE 68

Maintaining the shadow stack

R: R: A1: 18 R: R: A1:

17 18

Real stack Shadow stack On program failure (top-level exception):

slide-69
SLIDE 69

Shadow stack

Maintaining the shadow stack

R: R: A1: 18 R: R: A1:

17 18

Real stack On program failure (top-level exception):

  • 1. Write the shadow stack to a file
slide-70
SLIDE 70

Shadow stack

Maintaining the shadow stack

R: R: A1: 18 R: R: A1:

17 18

Real stack On program failure (top-level exception):

  • 1. Write the shadow stack to a file

Serializes all referenced state