3D-Assisted Image Feature Synthesis for Novel Views of an Object - - PowerPoint PPT Presentation

▶
3d assisted image feature synthesis for novel views of an
SMART_READER_LITE
LIVE PREVIEW

3D-Assisted Image Feature Synthesis for Novel Views of an Object - - PowerPoint PPT Presentation

Dec 04, 2023 348 likes •971 views

3D-Assisted Image Feature Synthesis for Novel Views of an Object Hao Su* Fan Wang* Li Yi Leonidas Guibas * Equal contribution View-agnostic Image Retrieval Retrieval using AlexNet features Query Cross-view Image Comparison Cross-view

slide-1
SLIDE 1

3D-Assisted Image Feature Synthesis for Novel Views of an Object

Hao Su* Fan Wang* Li Yi Leonidas Guibas

* Equal contribution

slide-2
SLIDE 2

View-agnostic Image Retrieval

Query Retrieval using AlexNet features

slide-3
SLIDE 3

Cross-view Image Comparison

slide-4
SLIDE 4

Cross-view Image Comparison

The comparison is between the underlying 3D objects

slide-5
SLIDE 5

Reconstruct 3D and then compare?

Kar et al, CVPR’15

Huang et al, SIGGRAPH’15 Su et al, SIGGRAPH’14

slide-6
SLIDE 6

Single-image based 3D Reconstruction is hard

Fg/bg segmentation 2D-3D Correspondence Non-convex iterative optimization Keypoint detection 2D image part segmentation 3D shape part segmentation

Common dependencies: Many dependencies Not Robust Slow

slide-7
SLIDE 7

Our Formulation: Novel View Feature Synthesis

Observed view

(HoG feature as an example)

slide-8
SLIDE 8

Our Novel View Feature Synthesis Results

(HoG feature as an example)

slide-9
SLIDE 9

Outline

Motivation Approach Method Diagnosis Applications Conclusion

slide-10
SLIDE 10

Key idea

Learn from a dataset of many objects with multi-view features

…

slide-11
SLIDE 11

Key idea

Learn from a dataset of multi-view features

d

The dataset is generated by rendering 3D models

slide-12
SLIDE 12

Key idea

The dataset is generated by rendering large-scale 3D models

http://shapenet.cs.stanford.edu

Learn from a dataset of multi-view features

slide-13
SLIDE 13

3D-assisted Feature Synthesis: Nearest Neighbour

Observed view image Novel view feature

(HoG feature as an example)

slide-14
SLIDE 14

Observed view image Novel view feature

Strong assumption: very similar model exists

(HoG feature as an example)

3D-assisted Feature Synthesis: Nearest Neighbour

slide-15
SLIDE 15

Observed view image Novel view feature

(HoG feature as an example)

...

3D-assisted Feature Synthesis: Multiple Shapes

slide-16
SLIDE 16

Attention: Brain games start!

3D-assisted Feature Synthesis: Multiple Shapes

slide-17
SLIDE 17

Pipeline

(HoG feature as an example)

Observed view image Novel view feature

slide-18
SLIDE 18

Observed view image Novel view feature

Pipeline

(HoG feature as an example)

slide-19
SLIDE 19

Observed view image Novel view feature

Pipeline

(HoG feature as an example)

slide-20
SLIDE 20

Observed view image Novel view feature

Pipeline

(HoG feature as an example) + + …

slide-21
SLIDE 21

Pipeline

(HoG feature as an example) + + …

Observed view image Novel view feature

slide-22
SLIDE 22

Pipeline

(HoG feature as an example) + + …

Observed view image Novel view feature + + …

0.1 0.4 0.3

Locally Linear Reconstruction

slide-23
SLIDE 23

Pipeline

(HoG feature as an example) + + …

Observed view image Novel view feature + + …

0.1 0.4 0.3

Locally Linear Reconstruction

slide-24
SLIDE 24

Pipeline

(HoG feature as an example) + + …

Observed view image Novel view feature + + …

0.1 0.4 0.3

Locally Linear Reconstruction

slide-25
SLIDE 25

+

+ +

+ …

…

0.1 0.4 0.3

Observed view image Novel view feature

Pipeline

(HoG feature as an example)

Locally Linear Reconstruction Inter-shape relationship

slide-26
SLIDE 26

Surrogate Relationship Discovery

+

+ +

+ …

…

0.1 0.4 0.3

Observed view image Novel view feature

?

Locally Linear Reconstruction Inter-shape relationship

(HoG feature as an example)

slide-27
SLIDE 27

Surrogate Relationship Discovery

Observed view Shape Collection Novel view

slide-28
SLIDE 28

Surrogate Relationship Discovery

Observed view Shape Collection Surrogate suitability matrix Novel view

slide-29
SLIDE 29

Formal Definition of Surrogate Suitability

Shape Collection

𝐵 𝐶

Novel view Observed view

Assume

A, 𝐶 are discrete random variables

slide-30
SLIDE 30

Formal Definition of Surrogate Suitability

Shape Collection

𝐵 𝐶

Novel view Observed view

Assume

A, 𝐶 are discrete random variables (𝑏1, 𝑐1), (𝑏2, 𝑐2), are i.i.d samples of (𝐵, 𝐶)

𝑏1 𝑐1

e.g.

𝑐2 𝑏2

slide-31
SLIDE 31

Formal Definition of Surrogate Suitability

Shape Collection

𝐵 𝐶

Novel view Observed view

Assume

A, 𝐶 are discrete random variables (𝑏1, 𝑐1), (𝑏2, 𝑐2), are i.i.d samples of (𝐵, 𝐶)

𝑏1 𝑐1

e.g.

𝑐2 𝑏2

𝛿 𝐵; 𝐶 = log 𝑄(𝑐1 = 𝑐2|𝑏1 = 𝑏2) Surrogate suitability:

slide-32
SLIDE 32

Formal Definition of Surrogate Suitability

Shape Collection

𝐵 𝐶

Novel view Observed view

Assume

A, 𝐶 are discrete random variables (𝑏1, 𝑐1), (𝑏2, 𝑐2), are i.i.d samples of (𝐵, 𝐶)

𝑏1 𝑐1

e.g.

𝑐2 𝑏2

𝛿 𝐵; 𝐶 = log 𝑄(𝑐1 = 𝑐2|𝑏1 = 𝑏2) Surrogate suitability:

How well can the sameness at A predict the sameness at B?

slide-33
SLIDE 33

Formal Definition of Surrogate Suitability

Shape Collection

𝐵 𝐶

Novel view Observed view

Assume

A, 𝐶 are discrete random variables (𝑏1, 𝑐1), (𝑏2, 𝑐2), are i.i.d samples of (𝐵, 𝐶)

𝑏1 𝑐1

e.g.

𝑐2 𝑏2

𝛿 𝐵; 𝐶 = log 𝑄(𝑐1 = 𝑐2|𝑏1 = 𝑏2) Surrogate suitability:

How well can the sameness at A predict the sameness at B? Cross-view transfer

  • f relationships
slide-34
SLIDE 34

Estimation of Surrogate Suitability

𝐼𝑆: Renyi-entropy Derivation shows

slide-35
SLIDE 35

Estimation of Surrogate Suitability

Sample complexity: tight bound Θ 𝑊

𝐵 + 𝑊 𝐶

Derivation shows where 𝑊

𝐵 and 𝑊 𝐶 are vocabulary size of 𝐵 and 𝐶

slide-36
SLIDE 36

Estimation of Surrogate Suitability

Sample complexity: tight bound Θ 𝑊

𝐵 + 𝑊 𝐶

Derivation shows where 𝑊

𝐵 and 𝑊 𝐶 are vocabulary size of 𝐵 and 𝐶

Theoretically optimal algorithm is proposed that reaches the bound

slide-37
SLIDE 37

Estimation of Surrogate Suitability

Sample complexity: tight bound Θ 𝑊

𝐵 + 𝑊 𝐶

Derivation shows where 𝑊

𝐵 and 𝑊 𝐶 are vocabulary size of 𝐵 and 𝐶

Strong connection with Mutual Information Theoretically optimal algorithm is proposed that reaches the bound

slide-38
SLIDE 38

More Visualization of Surrogate Suitability Matrix

Novel view Observed view

𝐶

slide-39
SLIDE 39

More Visualization of Surrogate Suitability Matrix

Novel view Observed view

𝐶

slide-40
SLIDE 40

More Visualization of Surrogate Suitability Matrix

Novel view Observed view

𝐶

slide-41
SLIDE 41

Review of Pipeline

+

+ +

+ …

…

0.1 0.4 0.3

Observed view image Novel view feature

slide-42
SLIDE 42

Review of Pipeline

Inter-shape relationship

+

+ +

+ …

…

0.1 0.4 0.3

Observed view image Novel view feature

Inter-shape relationship: Knowledge transfer from 3D shape database to new instance

slide-43
SLIDE 43

Review of Pipeline

Inter-shape relationship

+

+ +

+ …

…

0.1 0.4 0.3

Observed view image Novel view feature

Intra-shape relationship

Inter-shape relationship: Knowledge transfer from 3D shape database to new instance Intra-shape relationship: Knowledge transfer from observed view to novel view

slide-44
SLIDE 44

Outline

Motivation Approach Method Diagnosis Applications Conclusion

slide-45
SLIDE 45

Application: Cross-view localized image comparison

slide-46
SLIDE 46

Cross-view Image Retrieval

slide-47
SLIDE 47

HoG L2 Ours (combined HoG)

swivel base

Application: View-agnostic Image Retrieval

vertical bars

slide-48
SLIDE 48

HoG L2 Ours (combined HoG)

swivel base

Application: View-agnostic Image Retrieval

vertical bars

slide-49
SLIDE 49

HoG L2 Ours (combined HoG)

swivel base

Application: View-agnostic Image Retrieval

vertical bars

slide-50
SLIDE 50

Part-based View-agnostic Image Retrieval

slide-51
SLIDE 51

Generalizability to Many Feature Types

  • Task: fine-grained retrieval (images and annotations are from ImageNet)
  • Metric: Average Precision
slide-52
SLIDE 52

Outline

Motivation Approach Method Diagnosis Applications Conclusion

slide-53
SLIDE 53

How many shapes are sufficient?

(Measured by Average Precision on Fine-grained retrieval for Chairs)

200

slide-54
SLIDE 54

How many neighboring shapes for interpolation?

(Measured by Average Precision on Fine-grained retrieval for Chairs)

80

slide-55
SLIDE 55

How well can one view predict another view?

Cross-view retrieval rank

Controlled diagnosis on renderings

slide-56
SLIDE 56

Outline

Motivation Approach Method Diagnosis Applications Conclusion

slide-57
SLIDE 57

Conclusion

  • A novel framework for synthesizing object features at novel views
  • 3D shape database provides the knowledge of feature synthesis
  • For relationship transfer, surrogate suitability is defined, which is a type of

“predictability” between random variables.

  • A theoretically optimal estimator is proposed
slide-58
SLIDE 58

Thank you!

slide-59
SLIDE 59
slide-60
SLIDE 60