Debugging Highly-Parallel Programs Joo M. Loureno , Jos C. Cunha and - - PowerPoint PPT Presentation

João M. Lourenço, José C. Cunha and Vitor Duarte CITI / Universidade Nova de Lisboa joao.lourenco@fct.unl.pt

1

Debugging Highly-Parallel Programs

Why do programs have errors?

Problem Problem solved! Write a computer program Devise a computational solution

What about parallel programs?

Problem Problem solved! Write a computer program Devise a computational solution Interleaving errors

All men are equal! What about errors?

Byzantine Performance Interleaving Synchronization Ordering Sequential errors

Violations of precedence or mutual exclusion relations Ordering failures and deadlocks Unwanted side effects caused by non-reentrant code and shared data Yields a correct result, although it takes longer than acceptable

Harder Easier

Non fail-stop errors

?

Parallel program Multicore system Observed behavior Expected behavior

Parallel computations

Parallel program Parallel computations

Multicore system

generates

running

Parallel computations

What? How?

Program histories

• Local History: hi – sequence of events

generated by executing the program “pi”

– hi = ei

0, ei 1, …, ei f

– kth event in hi (ei

k) produces the local state sk

• Global History: H – union of the local

histories of N processes

– H = h1 U h2 U … U hN

Parallel Computation

• A parallel computation is a partially ordered

set (poset) defined as CD = (H, )

– H = Global history –  = Lamport’s happens before relation

Cut of a parallel computation

• A cut of a parallel computation is a subset C
• f its global history H that contains an initial

prefix for each of its local histories C = {h1

x, h2 y, …, hn z}

P1 P2 P3 e1

1

e1

2

e1

3

e2

3

e2

2

e2

1

e3

1

e3

2

e3

3

e2

4

e1

4

e2

5

Frontier of a cut & Global state

• The frontier of a cut is the set of the last

states/events in a cut F = {s1

x, s2 y, …, sn z}

• The frontier of a cut defines a global state

P1 P2 P3 e1

1

e1

2

e1

3

e2

3

e2

2

e2

1

e3

1

e3

2

e3

3

e2

4

e1

4

e2

5

Consistent cut

• A cut is consistent if for all events in its

frontier, all their past events are also included in the cut

P1 P2 P3 e1

1

e1

2

e1

3

e2

3

e2

2

e2

1

e3

1

e3

2

e3

3

e2

4

e1

4

e2

5

F1 F2 Inconsistent cut Consistent cut

Inconsistent global state Consistent global state

Consistent cut

• A global state is consistent if it corresponds

to the frontier of a consistent cut

P1 P2 P3 e1

1

e1

2

e1

3

e2

3

e2

2

e2

1

e3

1

e3

2

e3

3

e2

4

e1

4

e2

5

F1 F2

Runs

20 30 40 50 60 51 61 52 62 05 15 25 00 10 03 04 14 24 34 35 45 55 65 01 11 02 12 13 21 31 41 42 32 22 23 33 43 53 44 54 63 64 P1 P2 e1

1

e1

2

e1

3

e2

3

e2

2

e2

1

e2

4

e1

4

e2

5

e1

5

e1

6

00 10 03 04 14 24 34 35 45 55 65 01 11 02 12 13 21 31 41 42 32 22 23 33 43 53 44 54 63 64 00 03 04 14 24 34 35 45 55 65 01 02 00 10 65 11 21 31 41 42 43 53 63 64 05 15 25 20 30 40 50 60 51 61 52 62 7 states 6 states 30 states

Observing a parallel program

Process (P1) Process (P2) Process (PN) Events / / states

internal & interaction events

Local histories Run

arbitrary total order

Consistent run Global history

union

Cut

subset

Consistent cut Frontier of a consistent cut parallel computation

casual precedence constraints

Observation

permutation

Consistent

• bservation

Observing a parallel program

Process (P1) Process (P2) Process (PN) Events / / states

internal & interaction events

Local histories Run

arbitrary total order

Consistent run Global history

union

Cut

subset

Consistent cut Frontier of a consistent cut parallel computation

casual precedence constraints

Observation

permutation

Consistent

• bservation

Program state Program execution Developer perspective

Observing and debugging

to obtain reproducible behavior to analyze alternative paths to evaluate correctness properties

interactive debugging

• f remote processes

state based debugging

• bservation of

program states

trace, replay and debugging

deterministic re-execution repeatable

• bservations

combined testing, steering and debugging

systematic state exploration alternative

• bservations

global predicate detection

global program properties

• bservation of

consistency

The scaling challenge

• How to deal with hundreds (or thousands) of threads?

– Collect, store and gather observations / logs

• What shall be the detail level?
• Logs may be huge

– Combining the logs – Reason about global observations

• Visualize large amounts of information
• Evaluate global predicates on the program state
• Evaluate global predicates on the program run

– Map observation points to the original program

• Dealing with code-generators
• Supporting high-level abstractions, DSLs

The End…

Happy debugging! 