In Search of a Unifying Theory for Image Interpretation Donald Gem - - PowerPoint PPT Presentation

In Search of a Unifying Theory for Image Interpretation

Donald Gem an

Department of Applied Mathematics and Statistics and Center for Imaging Science, Whitaker Institute Johns Hopkins University

Outline

Semantic Scene Interpretation Frameworks, Theories Hierarchical Testing The Efficiency of Abstraction

Orientation within Imaging

Sensors to Images Images to Images Images to Words

Images to Words

Computational vision remains a major challenge (and natural vision a mystery). Assume one grey-level image. Generally massive local ambiguity. But less so globally, e.g., at the semantic level of keywords.

Tasks

Object identification (find my car) and categorization (find all cars) Recognition of multiple objects, activities, contexts, etc. Ideally, a description machine from images to rich scene interpretations.

Scene

Slide credit: Li Fei Fei

Is that a picture of Mao?

Are there cars?

Multiple Object Categorization

sky building flag wall face banner street lamp bus bus cars

Scene Categorization

Confounding Factors

Clutter Invariance to

Viewpoint Photometry Variation

Invariance vs. Selectivity

Clutter

Clutter

Klimt, 1913

Viewpoint Variation

Michelangelo 1475-1564

Slide credit: Li Fei Fei

Lighting Variation

Slide credit: Shimon Ullman

Occlusion

Magritte, 1957

Occlusion

Xu, Beihong 1943

Within-Class Variation

Within-Class Variation

How Many Samples are Needed?

Where Things Stand

Reasonable performance for several classes of semi-rigid objects. Even for face detection, a large “ROC gap” with human vision. Full scene parsing is currently beyond reach.

Where Are the Faces?

The ROC Gap: Face Detection

Current Com puter Vision: Approximately one hallucination per scene at ninety percent detection.

Bruegel, 1564

Francisco’s Kitchen

Notation

I : greyscale image Y : distinguished descriptions of I Ex: strings of (class,pose) pairs Y∈ {0}∪ Y : hidden r.v. Ŷ(I) : estimated description(s) from Y L={(I,Y)}: finite training set, in theory i.i.d. under P(I,Y)

Description Machine Specs

DESIGN and LEARNING: An explicit set of instructions for building Ŷ involving from L. COMPUTATION: An explicit set of instructions for evaluating Ŷ(I) with as little computation as possible. ANALYSI S: A “supporting theory” which guides construction and predicts performance

Ground Truth

For Y sufficiently restricted, reasonable to assume a “true interpetation” of I:

• Y = { face} , Y = { indoor, outdoor} ,…
• More generally, Y = {(c1,θ1),…, (ck,θk)}, limited to

specific categories and rough poses.

Corresponds to Pemp(Y|I)=δf(I)(Y) where Pemp(I,Y) is the empirical distribution over a gigantic sample (I1 ,Y1), (I2 ,Y2),…

Outline

Semantic Scene Interpretation Frameworks, Theories Hierarchical Testing The Efficiency of Abstraction

Deceased Frameworks

Traditional “AI” (60’s, 70’s) Stepwise, bottom-up 3D metric reconstruction (80’s) Algebraic, geometric invariants (90’s)

… but who knows

Living Frameworks

Generative modeling Discriminative learning Information-theoretic

Generative Modeling

Not all observations and explanations are equally likely. Construct P(I,Y) from

A distribution P(Y) on interpretations. A data model P(I|Y).

Inference principle:

Ŷ(I) = arg max P(Y|I) = arg min {-log P(I|Y) – log P(Y)}

Examples

Deformable templates Hidden Markov models Probabilities on part hierarchies Graphical models, e.g., Bayesian networks Gaussian models (LDA, mixtures, etc.)

Generative: Critique

In principle, a very general framework. In practice, Diabolically hard to model and learn P(Y). Intense online computation. P(I|Y) alone (i.e., “templates-for- everything”) lacks selectivity and requires too much computation.

Discriminative Learning

Proceed (almost) directly from data to decision boundaries. Representation and learning:

Replace I by a fixed length feature vector X Quantize Y to a small number of classes Specify a family F of “classifiers” f(X) Induce f(X) directly from a training set L

Examples

In effect, learn P(Y|X) (or log posterior odds ratios) directly:

• Artificial neural networks
• k-NN with smart metrics
• Decision trees

Support vector machines (interpretation as Bayes rule via logistic regression) Multiple classifiers (e.g., random forests)

Learning : Critique

In principle:

• Universal learning machines which mimic natural

processes and “learn” everything (e.g., invariance).

• Solid foundations in statistical learning theory

(although |L| ↓ 1 is the interesting limit).

In practice, lacks a global structure to address:

• A very large number of classes (say 30,000)
• Small samples, bias vs. variance, invariance vs.

selectivity.

Information-theoretic

Established connections between IT and imaging, but mostly at the “tool” level and for “low-level vision.” Two emerging frameworks:

“Information scaling” (Zhu) Resource/ complexity tradeoffs and “information refinement” (O’Sullivan et al) Both tilted towards “theory”.

An Information Theory Constellation

Slide credit: Laurent Younes

Overall Critique

Current generative and discriminative methods lack efficiency. Problem-specific structure is absent, and hence so a global organizing principle for vision. Sparse theoretical support (especially for practical systems).

Hierarchical Vision

Exploit shared components among

• bjects and interpretations.

Incorporate discriminative and generative methods as necessary. Can yield efficient representation, learning and computation.

Simple Part Hierarchy

Examples

Compositional systems (S. Geman) Hierarchies of fragments (Ullman) Hierarchies of conj’s and disj’s (Poggio) Convolutional neural networks (LeCun) Hierarchical generative models (Amit; Torralba; Perona; etc.) Hierarchical Testing

Emerging Theory

“Theory of reusable parts” (S. Geman)

Inspired by MDL and speech technology. Non-Markovian (“context sensitive”) priors. Theoretical results on efficient representation and selectivity.

However, contextual constraints enforced at the expense of learning and computation.

Outline

Semantic Scene Interpretation Frameworks, Theories Hierarchical Testing The Efficiency of Abstraction

Hierarchical Testing

Coarse-to-fine modeling of both the interpretations and the computational process: Unites representation and processing. Concentrates processing on ambiguous areas. Evidence for CTF processing in neural systems. Scales to many categories.

Density of Work

Original image Spatial concentration

• f processing

Collaborators: Hierarchical Testing

Evgeniy Bart IMA Sachin Gangaputra Inductus Corp. Xiaodong Fan Microsoft François Fleuret EPFL, Lausanne Gilles Blanchard Fraunhofer Hichem Sahbi Cambridge U. Yali Amit

• U. Chicago

From Source Coding to Hierarchical Testing

Y: r.v. with distr p(y), y ∈ Y Code for p: a CTF exploration of Y :

Can ask all questions XA of the form: “Is Y∈ A ?”, A ⊂ Y All answers are exact. Y is the only source of uncertainty

From Source Coding to Hierarchical Testing (cont)

Constrained 20 questions:

Restrict to selected subsets A ⊂ Y Still, Y determines {XA} and vice-versa Still an errorless, unique path (root to leaf)

Realizable tests:

Make XA observable (XA= XA(I)) Requires appearance-based shared properties among elements of Y

From Source Coding to Hierarchical Testing (cont)

Accommodate mistakes:

Preserve P(XA =1|Y∈ A)=1 But allow P(XA =1|Y∉ A)≠0; hence, only negative answers eliminate hypotheses

Generalize paths to “traces”:

The outcome of processing is now a labeled subtree in a hierarchy of tests. Ŷ(I) is the union of leaves reached.

Representation of Y

Natural groupings A ⊂ Y based on shared parts or attributes.

• Ex: Shape similarities between (c,θ) and (c’,θ’) for

nearby poses.

In fact, natural nested coverings or hierarchies of attributes

Hattr= { Aξ , ξ ∈ T }

Two Attribute Hierarchies

Which Decomposition?

Another story ….

Statistical Structure

For each ξ ∈ T, consider a binary test Xξ =XA dedicated to H0:Y ∈ Aξ against

ξ Ha: Balt(ξ) ⊂ {Y ∉ Aξ}

Define Htest= { Xξ , ξ ∈ T } Constraint: Each Xξ satisfies inv(Xξ ) P(Xξ =1|Y ∈ Aξ ) ≅ 1 where P=Pemp estimated from L.

Summary

T ξ ξ∈T … … … Hattr Aξ Aξ⊂Y … … … Htest Xξ Xξ: ”Y∈Aξ?” … … …

Statistical Structure (cont)

Explore Htest under some “strategy”. Ŷ(I): y∈ Y not ruled out by any (performed) test: Ŷ = Y \ ∪ {Aξ : Xξ = 0} Our constraint implies Pemp(Y ∈ Ŷ(I)) ≅ 1 Could replace Pemp by a parametric model if available.

Example

A recursive partitioning of Y with four levels; there is a binary test for each of the 15 cells. (A): Positive tests are shown in black. (B): Ŷ is the union of leaves 3 and 4. (C): Tests performed under coarse-to-fine search.

Tests in Practice

Complex, e.g.,

Build an SVM from training examples

Simple, e.g.,

VQ 16x16 patches into k types; Ask two types of questions:

“Is there a patch in W with label k?” “Are there two patches in W with labels k,l ?”

Example: Face Detection

Detect and localize all instances of upright, frontal faces. Θ ={(z,σ,φ)} = {(position, scale, tilt)} Y : subsets of Θ Construct the hierarchy Hattr= { Aξ , ξ ∈ T } based on recursive partitions of Θ

Face Pose Hierarchy: Upper

First split: coarse scale (8-15, 16-31, 32-63, etc.) Second split: coarse position (disjoint 16x16 blocks B) These tests are “virtual” (always passed) without diffuse, extended attributes (e.g., color).

Face Pose Hierarchy: Lower

Construct Hattr from

{(z,σ,φ): z ∈ 16x16, 8 ≤ σ ≤ 15, -200 ≤ φ ≤ 200}

by splitting individual coordinates. The 64x64 subimage surrounding each B is then labeled as “face (θ)” or “background.”

Face Pose Hierarchy

{ (z,σ,φ): z ∈ 16x16, 8 ≤ σ ≤ 15, -200 ≤ φ ≤ 200 } { (z,σ,φ): z ∈ 2x2 14 ≤ σ ≤ 15, 100 ≤ φ ≤ 200 }

Detection Algorithm

Loop over resolution Loop over location (non-overlapping 16x16 blocks) Process the “reference” (lower) hierarchy Collect chains of positive responses

Scale 8 to 16 Scale 16 to 32 Scale 32 to 64

Examples

Learned hierarchy Manual hierarchy

False positives per image

Results

Face Tracking

Reading License Plates

37 classes. Synthesized training shapes from class prototypes. Nearly 99% classification rate per symbol on 380 plates.

Outline

Semantic Scene Interpretation Frameworks, Theories Hierarchical Testing The Efficiency of Abstraction

Efficient Computation

Strategy S: Any sequential, adaptive exploration of some or all of the tests. c(Xξ): cost of Xξ , sel(Xξ) = P(Xξ=0|Balt(Aξ)) Cost of Testing: Total Computation:

( ) ( )

s r

test H A s S r s

C S I c X

∈∂ ↓

= ∑

∑

ˆ ( ) * ( )

test

E C S c Y S ⎡ ⎤ + ⎣ ⎦

Assumptions

(i) “Background” is statistically dominant: P(Y=0) >> P(Y≠0) (Valid for sufficiently fine cells A, e.g., in the third level of the face hierarchy.) (ii) Total computation is driven by P(.|Y=0) (iii) The tests { Xξ , ξ ∈ T } are conditionally independent under this probability.

CTF Optimality Criterion

THEOREM: (G. Blanchard/ DG) CTF is optimal if

where C(ξ) = direct children of ξ in T. A numerical example:

( )

( ) ( ) ( ) ( )

C

c X c X sel X sel X

ξ η η ξ ξ η ∈

≤ ∑

, T ξ ∀ ∈

c(X1)=c(X2)=c(X3) sel(X1) = 1/2, sel(X2)= sel(X3)=9/10 Do X1 first !

Back to Twenty Questions

Optimal strategies are what we intuitively expect from playing 20Q:

A steady progression from high scope coupled with relatively poor selectivity to high selectivity coupled with dedication to specific interpretations.

“Right” Alternative Hypothesis

Due to CTF search Xξ is performed ⇔ all ancestor tests are performed and are positive. Hence, where = ancestors of node ξ in T.

ξ ξ η

η ξ = ∉ ∩ = ∀ ∈ A

( )

{ } { 1 ( )}

alt

B Y A X ξ A( )

Efficient Learning: Trace Model

Encodes the computational history of CTF search.

T: tree underlying the hierarchy S(I) : subtree of T determined by CTF search on image I Z(I) = { Xξ , ξ ∈ S(I) } , the “trace”, a labeled subtree.

Trace Representation

Tree hierarchy Realization of all tests Subtree from CTF search Trace

Trace Representation (cont)

n(T) = 5 23 = 8 n(T) = 26 27 = 128 Top: A 3 node hierarchy and its 5 possible traces Bottom: A 7 node hierarchy and 5 of its 26 possible traces

Trace Distributions

THEOREM: Let {pξ , ξ ∈ T} be any set of numbers with 0≤pξ≤1. Then defines a probability distribution on traces where Sz is the subtree identified with z and pξ(1)= pξ , pξ(0)= 1-pξ

( ) ( )

z

S

P z p x

ξ ξ ξ∈

∏

• Interpretation:

( ) ( | 1, ( )) p x P X x X

ξ ξ ξ ξ η

η ξ = = = ∀ ∈A

Learning

Needn’t model P(Xξ , ξ∈T); only need one parameter per node After CTF search, a trace-based likelihood ratio test. Interpretation: exact Bayesian inference with the trace as a (tree-structured) sufficient statistic: P(I|Y) = P(tr(I)|Y) / #{tr(I)}

Selectivity of the LRT

Top: Raw results of pure detection Bottom: False positives are eliminated with the trace model

Pruning Detections

Detection rate vs. false positives on the MIT+ CMU test set; Ex: 0.77 FPs/ image at 89.1% detection with | L| = 400

Conclusions: Hierarchical Testing

Can be seen as an adaptation of source coding to vision Hardwires invariance and computational efficiency. Limited by an impoverished contextual analysis. Resolve the particular, data-dependent, ambiguities by tests constructed on-line.

Conclusions: General

A dramatic “ROC gap” with natural vision. Also, no Shannon yet. Ambitious proposals center on hierarchical structures. None is simultaneously efficient and contextual.