A Human Factors Approach to Spam Factors Approach Filtering - - PowerPoint PPT Presentation

▶
a human factors approach to spam
SMART_READER_LITE
LIVE PREVIEW

A Human Factors Approach to Spam Factors Approach Filtering - - PowerPoint PPT Presentation

Mar 08, 2024 331 likes •506 views

Spam & HCI R. Beverly The Problem A Human A Human Factors Approach to Spam Factors Approach Filtering SpamGUI Parting Thoughts Summary Robert Beverly MIT CSAIL rbeverly@csail.mit.edu July 27, 2009 Conference on Email and

slide-1
SLIDE 1

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary

A Human Factors Approach to Spam Filtering

Robert Beverly

MIT CSAIL rbeverly@csail.mit.edu July 27, 2009

Conference on Email and Anti-Spam 2009

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 1 / 12

slide-2
SLIDE 2

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary The Problem

No spam classifier is perfect Okay in other ML fields, e.g. Handwriting recognition, search engines, music recommendation, etc. But with spam: Adaptable, adversarial inputs Complexion of dataset severely unbalanced High cost of false positives Getting from 99.9% to 99.999% Fighting a losing battle?

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 2 / 12

slide-3
SLIDE 3

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary The Problem

No spam classifier is perfect Okay in other ML fields, e.g. Handwriting recognition, search engines, music recommendation, etc. But with spam: Adaptable, adversarial inputs Complexion of dataset severely unbalanced High cost of false positives Getting from 99.9% to 99.999% Fighting a losing battle?

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 2 / 12

slide-4
SLIDE 4

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary The Problem

No spam classifier is perfect Okay in other ML fields, e.g. Handwriting recognition, search engines, music recommendation, etc. But with spam: Adaptable, adversarial inputs Complexion of dataset severely unbalanced High cost of false positives Getting from 99.9% to 99.999% Fighting a losing battle?

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 2 / 12

slide-5
SLIDE 5

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary The Problem

0.2 0.4 0.6 0.8 1

  • 15
  • 10
  • 5

5 10 15 20 25 30 35 40 45 Cumulative Fraction of Emails SpamAssassin Score Spam Ham

TREC 2007 dataset (∼75k messages) Classified with SpamAssassin How close are mails to the threshold (5)?

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 3 / 12

slide-6
SLIDE 6

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary The Problem

0.2 0.4 0.6 0.8 1

  • 15
  • 10
  • 5

5 10 15 20 25 30 35 40 45 Cumulative Fraction of Emails SpamAssassin Score Spam Ham

How close are mails to the threshold (5)? 99.72% of ham below threshold... good?

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 4 / 12

slide-7
SLIDE 7

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary The Problem

1e-05 0.0001 0.001 0.01 0.1 1

  • 15
  • 10
  • 5

5 10 15 20 25 30 35 40 45 50 55 60 Complimentary Cumulative Fraction of Emails SpamAssassin Score Spam Ham

No threshold gives zero FP/FN (well-known compromise) Deluge of spam implies this compromise is flawed 0.28% above → 71 false positives

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 5 / 12

slide-8
SLIDE 8

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary A Human Factors Approach

Approaching from a different direction... The User Agent: Users interact with their email via a Mail User Agent (MUA), e.g. Outlook, Hotmail, etc. Note that besides going graphical, MUAs have changed little over past ∼ 30 years Better incorporate human factors into a MUA

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 6 / 12

slide-9
SLIDE 9

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary A Human Factors Approach

Human Factors Approach – Potential:

1

Make email more useful to the user

How are emails presented?

2

Humans ultimate arbiter of any mail’s importance

How to better include, scale their decision process?

3

Remove burden of perfect classification from classifier

“good enough” filtering

4

Eliminate false positives Innovate in the user agent

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 7 / 12

slide-10
SLIDE 10

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary A Human Factors Approach

Human Factors Approach – Potential:

1

Make email more useful to the user

How are emails presented?

2

Humans ultimate arbiter of any mail’s importance

How to better include, scale their decision process?

3

Remove burden of perfect classification from classifier

“good enough” filtering

4

Eliminate false positives Innovate in the user agent

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 7 / 12

slide-11
SLIDE 11

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary SpamGUI

Position Separate classification from filtering The inbox: Rethink the inbox: use a single mail folder, don’t attempt to filter into spam, ham “folders” Use color, size, shade, order, and other human factors to present the inbox Presentation of email a function of importance Proof-of-concept: SpamGUI Thunderbird extension...

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 8 / 12

slide-12
SLIDE 12

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary SpamGUI

Position Separate classification from filtering The inbox: Rethink the inbox: use a single mail folder, don’t attempt to filter into spam, ham “folders” Use color, size, shade, order, and other human factors to present the inbox Presentation of email a function of importance Proof-of-concept: SpamGUI Thunderbird extension...

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 8 / 12

slide-13
SLIDE 13

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary SpamGUI

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 9 / 12

slide-14
SLIDE 14

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary SpamGUI

A Few Observations: A demarcation “line” naturally emerges to the eye, above which user (or UI) can ignore messages User part of filtering process, but only burdened by making spam decisions on a small number of emails around line Easy to scan for formerly false positive emails on the threshold border Lots of work remains: No user studies performed yet Experimenting with several approaches

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 10 / 12

slide-15
SLIDE 15

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary Parting Thoughts

More generally: Users inundated with information, how can UI help? Spam is just one class of very unimportant information Lots of unused input “features;” systems designers should use them Learn best way to present email to user Recognize that innovation is possible in the user agent

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 11 / 12

slide-16
SLIDE 16

Spam & HCI

  • R. Beverly

The Problem A Human Factors Approach SpamGUI Parting Thoughts Summary Summary

We’re fighting a losing battle trying to make spam classifiers perfect Separate act of classification from filtering As a community, think more about how HCI / human factors methods can help Thanks! http://www.rbeverly.net/spamgui/ Questions?

  • R. Beverly (MIT)

Spam & HCI CEAS 2009 12 / 12