Why Cant Johnny Fix Vulnerabilities: A Usability Evaluation of - - PowerPoint PPT Presentation

▶
why can t johnny fix vulnerabilities
SMART_READER_LITE
LIVE PREVIEW

Why Cant Johnny Fix Vulnerabilities: A Usability Evaluation of - - PowerPoint PPT Presentation

Aug 14, 2022 440 likes •610 views

Why Cant Johnny Fix Vulnerabilities: A Usability Evaluation of Static Analysis Tools for Security Justin Smith (Lafayette College) smithjus@lafayette.edu Lisa Nguyen Quang Do (Google) lisanqd@google.com Emerson Murphy-Hill (Google)

slide-1
SLIDE 1

Why Can’t Johnny Fix Vulnerabilities:

A Usability Evaluation of Static Analysis Tools for Security

Justin Smith (Lafayette College)

smithjus@lafayette.edu

Lisa Nguyen Quang Do (Google)

lisanqd@google.com

Emerson Murphy-Hill (Google)

emersonm@google.com

https://jssmith1.github.io/ smithjus@lafayette.edu @JustinSmith0903

slide-2
SLIDE 2

Static Analysis to the Rescue!

Static analysis tools detect vulnerabilities early

https://metier.jakarman.nl/design_sdlc/design_sdlc.html

Static Analysis

slide-3
SLIDE 3

Static Analysis to the Rescue?

Static analysis tools detect vulnerabilities early

https://metier.jakarman.nl/design_sdlc/design_sdlc.html

Static Analysis

slide-4
SLIDE 4

Unusable static analysis

Static analysis tools:

  • produce “bad warning messages” [Christakis, 2016];
  • “may not give enough information” [Johnson, 2013];
  • and “miscommunicate” [Johnson, 2016] with developers.

“Usable security for developers has been a critically under- investigated area” [Acar, 2016]. “[Improving] the usability of analysis results significantly increases the utility of analysis tools.” [Sadowski, 2015]

slide-5
SLIDE 5

What types of issues detract from the usability of security-oriented static analysis tools?

slide-6
SLIDE 6

Tools Evaluated

Three open-source tools

  • Find Security Bugs, RIPS, and Flawfinder

One commercial tool PHP RIPS FindSecBugs Commercial Tool Flawfinder

slide-7
SLIDE 7

Tools Evaluated

YOUR TOOL HERE

Replication Package

slide-8
SLIDE 8

Approach

Heuristic walkthrough evaluation

Phase 1: Cognitive walkthrough Phase 2: Heuristic evaluation

User study

Observed participants (n = 12) as they used the four tools

Analysis

Identified 194 (heuristic walkthroughs) + 140 (user study) usability issues Open card sort to group issues into unique themes for presentation

slide-9
SLIDE 9

Overview of Findings

Themes Subthemes Missing Affordances Managing Vulnerabilities Applying Fixes Missing or Buried Information Vulnerability Prioritization Fix Information Scalability of Interfaces Vulnerability Sorting Overlapping Vulnerabilities Scalable Visualizations Inaccuracy of Analysis Code Disconnect Mismatched Examples Immutable Code Workflow Continuity Tracking Progress Batch Processing

slide-10
SLIDE 10

Overview of Findings

Themes Subthemes Missing Affordances Managing Vulnerabilities Applying Fixes Missing or Buried Information Vulnerability Prioritization Fix Information Scalability of Interfaces Vulnerability Sorting Overlapping Vulnerabilities Scalable Visualizations Inaccuracy of Analysis Code Disconnect Mismatched Examples Immutable Code Workflow Continuity Tracking Progress Batch Processing

slide-11
SLIDE 11

Findings

Problem: Visual scalability over large programs

slide-12
SLIDE 12

Findings

Problem: Unclear severity scales

slide-13
SLIDE 13

Findings

Problem: Buried warnings

slide-14
SLIDE 14

Takeaways

Usability issues detract from security-oriented static analysis tools. Using relatively inexpensive heuristic walkthroughs, we can identify and address these issues!

slide-15
SLIDE 15

Takeaways

Usability issues detract from security-oriented static analysis tools. Using relatively inexpensive heuristic walkthroughs, we can identify and address these issues!

https://jssmith1.github.io/ smithjus@lafayette.edu @JustinSmith0903 Replication Package

you

}

^