THeME: A System for Testing by Hardware Monitoring Events Kristen - - PowerPoint PPT Presentation

▶
theme a system for testing by hardware monitoring events
SMART_READER_LITE
LIVE PREVIEW

THeME: A System for Testing by Hardware Monitoring Events Kristen - - PowerPoint PPT Presentation

Dec 18, 2022 473 likes •832 views

THeME: A System for Testing by Hardware Monitoring Events Kristen R. Walcott-Justice Jason Mars kwalcott@uccs.edu jom5x@cs.virginia.edu University of Colorado - University of California - Colorado Springs San Diego Mary Lou Soffa

slide-1
SLIDE 1

Issta 2012, July 17, Minneapolis, MN

THeME: A System for Testing by Hardware Monitoring Events

Kristen R. Walcott-Justice kwalcott@uccs.edu University of Colorado - Colorado Springs

Jason Mars jom5x@cs.virginia.edu University of California - San Diego

Mary Lou Soffa soffa@cs.virginia.edu University of Virginia

1 Wednesday, July 18, 12

slide-2
SLIDE 2

Developing Reliable Software

  • Measuring test quality:
  • Recompilation
  • High run time overheads
  • Large code growth

Software lifecycle

Require ment Design Release Maintenance and Patching Bug fix Implementation Testing

2 Wednesday, July 18, 12

slide-3
SLIDE 3

Expense of Traditional Test Coverage Analysis

  • Instrumentation
  • Probe
  • Payload
  • Branch analysis overheads:
  • Time: 10% - 30%
  • Code growth: 60% - 90%

Branch Executed? B1 B2 Will B2 execute? Will B1 execute?

√

3 Wednesday, July 18, 12

slide-4
SLIDE 4

Efficient Program Monitoring

Profiling Optimization Race Detection Software-Level Monitoring

4 Wednesday, July 18, 12

slide-5
SLIDE 5

Efficient Program Monitoring

Profiling Optimization Race Detection Software-Level Monitoring Hardware Monitoring

5 Wednesday, July 18, 12

slide-6
SLIDE 6

What is a Hardware Mechanism?

Sample read() System State Sample

6 Wednesday, July 18, 12

slide-7
SLIDE 7

Using Hardware Mechanisms

  • Developed for operating system performance

analysis

  • Widely available on nearly all processors
  • Low overhead
  • Short setup time (318µs)
  • Quick read time (3.5µs)
  • Use of samples
  • Estimate profiles
  • Reveal program execution behavior
  • Removes need for instrumentation

7 Wednesday, July 18, 12

slide-8
SLIDE 8

Hardware Mechanisms in Testing: Goals and Challenges

  • Structural testing requires more exact data
  • Can we capture ALL events with which

we are concerned?

  • Can we capture ONLY the events with

which we are concerned?

  • Tradeoff:
  • Amount of information collected
  • Overhead of sampling

8 Wednesday, July 18, 12

slide-9
SLIDE 9

THeME: Testing by Hardware Monitoring Events

Program modification Hardware Sampling/Monitoring Coverage Calculation Branch Sampler

9 Wednesday, July 18, 12

slide-10
SLIDE 10

THeME: Testing by Hardware Monitoring Events

Program modification Hardware Sampling/Monitoring Coverage Calculation Branch Sampler

9 Wednesday, July 18, 12

slide-11
SLIDE 11

THeME: Testing by Hardware Monitoring Events

Program modification Hardware Sampling/Monitoring Coverage Calculation Branch Sampler

9 Wednesday, July 18, 12

slide-12
SLIDE 12

THeME: Testing by Hardware Monitoring Events

Program modification Hardware Sampling/Monitoring Coverage Calculation Branch Sampler

9 Wednesday, July 18, 12

slide-13
SLIDE 13

Branch Vector Recording: Last Branch Record (LBR)

  • Mechanism for partial branch

profiling

  • Intended for OS performance

and debugging

  • Tracks set of executed branches
  • Branch source
  • Branch destination
  • Sample == Set of branches

“Branch Vector”

SAMPLE

Hardware Mechanism 1 2 3 4 5 6 7 8 2 4 6 . . . . . . . .

Branch Vector (≤16 branches)

Last Branch Record

10 Wednesday, July 18, 12

slide-14
SLIDE 14

THeME: Testing by Hardware Monitoring Events

Program modification Hardware Sampling/Monitoring Coverage Calculation Branch Sampler

11 Wednesday, July 18, 12

slide-15
SLIDE 15

Enabling Fall-through Visibility

Challenge: Hardware branch-based monitors can only see 1 of 2 branch edges

Jump FT 1 2 3 4 5 Jump FT 1 2 3 4 5 new new jump

  • Methods
  • Supplement with more samples
  • Use static analysis to infer branches
  • Minor program modification
  • Our Solution:

Insert innocuous unconditional branches

12 Wednesday, July 18, 12

slide-16
SLIDE 16

Enabling Fall-through Visibility

Challenge: Hardware branch-based monitors can only see 1 of 2 branch edges

Jump FT 1 2 3 4 5 Jump FT 1 2 3 4 5 new new jump

  • Methods
  • Supplement with more samples
  • Use static analysis to infer branches
  • Minor program modification
  • Our Solution:

Insert innocuous unconditional branches

12 Wednesday, July 18, 12

slide-17
SLIDE 17

THeME: Testing by Hardware Monitoring Events

Program modification Hardware Sampling/Monitoring Coverage Calculation Branch Sampler

13 Wednesday, July 18, 12

slide-18
SLIDE 18

THeME: Testing by Hardware Monitoring Events

Program modification Hardware Sampling/Monitoring Coverage Calculation Branch Sampler

14 Wednesday, July 18, 12

slide-19
SLIDE 19

THeME: Testing by Hardware Monitoring Events

Program modification Hardware Sampling/Monitoring Coverage Calculation Branch Sampler

15 Wednesday, July 18, 12

slide-20
SLIDE 20

Improving Branch Coverage

  • Sampling → Some missed data
  • Goal: Improve coverage using

static analysis

  • Dominator analysis
  • Associate seen branches with

control flow graph

  • Branch b executed → branch

c also executed

5 7 2 1 8 11

16 Wednesday, July 18, 12

slide-21
SLIDE 21

Experiment and System Design

  • Intel Core i7 860 quad-core processor
  • LBR size of 16 branches
  • Linux 2.6.34
  • Hardware access tools: libpfm4 (user-level), perf (kernel-level)
  • SPEC2006 C Benchmarks
  • Metrics:
  • Efficiency- time
  • Code growth size
  • Effectiveness- branch coverage
  • Instrumented vs Hardware Monitoring

17 Wednesday, July 18, 12

slide-22
SLIDE 22

Results: Enabling Fall-Through Visibility

  • Impact:
  • Increases time overhead
  • Increases code growth
  • How compared to instrumentation?

Time overhead

Benchmark bzip2 h264ref libquantum mcf sjeng Branch Time (s)

  • Mod. Time
  • Instr. Time

Cov. (s) (s) 64.20% 1499 1514 1599 35.72% 1753 1786 1890 39.07% 1056 1178 1236 74.01% 529 539 575 48.87% 1028 1162 1312

Avg: 5% increase Avg: 14% increase

18 Wednesday, July 18, 12

slide-23
SLIDE 23

Results: Enabling Fall-Through Visibility

  • Impact:
  • Increases time overhead
  • Increases code growth
  • How compared to instrumentation?

Benchmark Native Mod. Instr. Size (kB) % Increase % Increase bzip2 260 kB 1.52 32.65 h264ref 2892 kB 0.69 18.39 libquantum 208 kB 20.00 mcf 128 kB 17.95 sjeng 592 kB 0.67 30.05

Avg: 0.5% Avg: 24%

Code Growth

19 Wednesday, July 18, 12

slide-24
SLIDE 24

Results: Testing on a Single Core - Effectiveness

20 Wednesday, July 18, 12

slide-25
SLIDE 25

Results: Testing on a Single Core - Effectiveness

20 Wednesday, July 18, 12

slide-26
SLIDE 26

Results: Testing on a Single Core - Efficiency

libquantum mcf sjeng Percent time overhead Sample periods per benchmark Percent Time Overhead Using Interrupt Driven Approach on Ref Inputs

500K 1M 5M 10M 50M

−10% 0% 10% 20% 30% 40% 50% 60% bzip h264ref

21 Wednesday, July 18, 12

slide-27
SLIDE 27

Results: Better Coverage at High Sample Rates

22 Wednesday, July 18, 12

slide-28
SLIDE 28

Results: Better Coverage at High Sample Rates

22 Wednesday, July 18, 12

slide-29
SLIDE 29

Results: Better Coverage at High Sample Rates

71%

22 Wednesday, July 18, 12

slide-30
SLIDE 30

Results: Better Coverage at High Sample Rates

90% 72%

22 Wednesday, July 18, 12

slide-31
SLIDE 31

Results: Testing on a Multiple Cores - Efficiency

−20% −10% 0% 10% 20% 30% 40% bzip2 h264ef Percent time overhead Sample periods per benchmark Percent Time Overhead Splitting Inputs Across Cores

500K 1M 5M 10M 50M

23 Wednesday, July 18, 12

slide-32
SLIDE 32

Hardware Monitoring Benefits

  • Low overhead, effective branch testing technique
  • Up to 90% of branch coverage
  • 2% time improvement
  • 0.5% code growth (compared to 60% to 90%)
  • Test coverage approximation
  • Testing on resource constrained devices
  • “Imprecise” tasks (e.g. regression test prioritization)
  • Partial program monitoring
  • Significant benefits
  • Enable testing on resource constrained devices
  • Generates full picture of program execution

24 Wednesday, July 18, 12

slide-33
SLIDE 33

Conclusions and Future Work

  • Extensible, portable system for single or multiple cores
  • Up to 11.13% improvement in time overhead
  • Up to 90% of the coverage reported by

instrumentation

  • Reduced time overhead (~2%)
  • Negligible code growth
  • Future work:
  • Combine hardware monitoring with limited

instrumentation

  • Implement on resource constrained device
  • Extend system to other coverage metrics

25 Wednesday, July 18, 12

slide-34
SLIDE 34

Thank You!

Website: http://www.cs.virginia.edu/walcott Questions?

26 Wednesday, July 18, 12