The Impact of the Open Set Recognition Problem on Deep Learning - - PowerPoint PPT Presentation

▶

Aug 19, 2022 307 likes •572 views

The Impact of the Open Set Recognition Problem on Deep Learning Walter J. Scheirer Computer Vision Research Laboratory, De partment of Computer Science and Engineering Benchmarks in computer vision Assume we have examples from all classes:

SLIDE 1

The Impact of the Open Set Recognition Problem on Deep Learning

Walter J. Scheirer

Computer Vision Research Laboratory,

Department of Computer Science and Engineering

SLIDE 2

Benchmarks in computer vision

Assume we have examples from all classes: Places2 Data Set (part of ILSVRC 2016)

Airfield Campsite Water Park Mountain Gas Station

SLIDE 3

Out in the real world…

Detect the cars in this image while rejecting the trees, signs, telephone poles…

M. Milford, W.J. Scheirer, E. Vig, A. Glover, O. Baumann, J. Mattingley, and D.D. Cox, “Condition Invariant Top-Down Visual Place

Recognition,” ICRA 2014.

SLIDE 4

Open Set Recognition: incomplete knowledge

f the world is present at training time, and

unknown classes can be submitted to an algorithm during its operation.

SLIDE 5

“There are known knowns…”

known classes: the classes with distinctly labeled positive training examples (also serving as negative examples for other known classes) known unknown classes: labeled negative examples, not necessarily grouped into meaningful categories unknown unknown classes: classes unseen in training

SLIDE 6

Vision problems in order of “openness”

W. J. Scheirer, A. Rocha, A. Sapkota, and T. Boult, “Towards Open Set Recognition,” IEEE T-PAMI, 35(7) July 2013.

SLIDE 7

Fundamental multi-class recognition problem

argmin

⇢ RI(f) := Z

Rd×N

L(x, y, f(x))P(x, y)

Ideal Risk

Loss Function Joint Distribution Undefined for

pen set recognition!
A. Smola, “Learning with Kernels,” Ph.D. dissertation, Technische Universität Berlin, Berlin, Germany, November 1998.

SLIDE 8

Open Space

? ? ?

Positives Negatives

?

SLIDE 9

Open Set MNIST Benchmark

Closed set testing on MNIST Open set testing on MNIST

(a) (d)

0.55$ 0.7$ 0.85$ 1$ (6,6)$ (7,7)$ (8,8)$ (9,9)$ (10,10)$ Accuracy (##training#classes,###tes/ng#classes)# 1-vs-Rest SVM 0.55$ 0.7$ 0.85$ 1$ (6,6)$ (6,7)$ (6,8)$ (6,9)$ (6,10)$ Accuracy (##training#classes,###tes/ng#classes)# Proposed Approach 1-vs-Rest SVM w/ Threshold

Training images

f digits (0-5)

Testing images

f digits (0-9)

(b) (c)

PI -SVM

L. P. Jain, W. J. Scheirer, and T. Boult, “Multi-Class Open Set Recognition Using Probability of Inclusion,” ECCV 2014.

SLIDE 10

“Tangled”

Adapted from an image by D. D. Cox

SLIDE 11

“Untangled”

SLIDE 12

ineffective separating hyperplane individual 2 (”Joe”) individual 1 (”Sam”)

individual 2 (”Joe”) individual 1 (”Sam”) separating hyperplane

Linear separation of CNN feature representations

unknown individual

Female head #10 CC BY 2.0 Turinboy

unknown individual

Female head #3 CC BY 2.0 Turinboy

SLIDE 13

Read-out layer ?

Typical CNN architecture CC BY 4.0 Aphex34

Softmax

Sum over all of the classes

Linear SVM

subject to

Known positive or negative sample

Cosine Similarity

Threshold determined empirically via known pairs

A · B ||A|| ||B|| < δ

SLIDE 14

Evolving images to match CNN classes

A. Nguyen, J. Yosinski, and J. Clune, “Deep Neural Networks are Easily Fooled,” CVPR 2015.

SLIDE 15

A step towards a fix: OpenMax

A. Bendale and T. Boult, “Towards Open Set Deep Networks,” CVPR 2016.

SLIDE 16

How does OpenMax work?

Apply probability models derived from statistical extreme value theory to calculate class weights Apply rejection threshold Use weights to adjust activation

SLIDE 17

But you don’t have to use tricky manipulations

GoogleNet Output

Label: Hammerhead Shark Label: Blow Dryer Label: Mosque Label: Syringe Label: Trimaran Label: Missile

SLIDE 18

Are performance measures misleading us?

SLIDE 19

Psychophysics on the Model

Accuracy Face Signal

N = 427 Human performance* Google Picasa Facebook, mixed occluders Facebook, solid occluders

W.J. Scheirer, S. Anthony, K. Nakayama, and D. D. Cox, “Perceptual Annotation: Measuring Human Vision to Improve Computer Vision,” IEEE T-PAMI, 36(8) August 2014.

SLIDE 20

Psychophysics pipeline

Brandon RichardWebster

1. Render Class

Canonical View (CCV) Candidates

3. Manipulate Chosen

Variable

2. CCV Classifier
4. Classify Images

Plane Fish Skyscraper

5. Generate

Psychometric Curve

Area Visible Accuracy

SLIDE 21

SLIDE 22

SLIDE 23

SLIDE 24

Practical implications

http://goo.gl/78fglb

SLIDE 25

The Impact of the Open Set Recognition Problem on Deep Learning - - PowerPoint PPT Presentation

The Impact of the Open Set Recognition Problem on Deep Learning

Walter J. Scheirer

Benchmarks in computer vision

Assume we have examples from all classes: Places2 Data Set (part of ILSVRC 2016)

Out in the real world…

Detect the cars in this image while rejecting the trees, signs, telephone poles…

Open Set Recognition: incomplete knowledge

unknown classes can be submitted to an algorithm during its operation.

“There are known knowns…”

known classes: the classes with distinctly labeled positive training examples (also serving as negative examples for other known classes) known unknown classes: labeled negative examples, not necessarily grouped into meaningful categories unknown unknown classes: classes unseen in training

Vision problems in order of “openness”

Fundamental multi-class recognition problem

argmin

⇢ RI(f) := Z

L(x, y, f(x))P(x, y)

Loss Function Joint Distribution Undefined for

Open Space

? ? ?

?

Open Set MNIST Benchmark

“Tangled”

“Untangled”

Linear separation of CNN feature representations

Read-out layer ?

Softmax

Linear SVM

Cosine Similarity

A · B ||A|| ||B|| < δ

Evolving images to match CNN classes

A step towards a fix: OpenMax

How does OpenMax work?

But you don’t have to use tricky manipulations

GoogleNet Output

Are performance measures misleading us?

Psychophysics on the Model

Psychophysics pipeline

Practical implications

Thank you!

Read more: www.wjscheirer.com