3D Photography: Stereo Vision Kalin Kolev, Marc Pollefeys Spring - - PowerPoint PPT Presentation

▶
3d photography stereo vision
SMART_READER_LITE
LIVE PREVIEW

3D Photography: Stereo Vision Kalin Kolev, Marc Pollefeys Spring - - PowerPoint PPT Presentation

Aug 20, 2023 210 likes •1.25k views

3D Photography: Stereo Vision Kalin Kolev, Marc Pollefeys Spring 2013 http://cvg.ethz.ch/teaching/2013spring/3dphoto/ Schedule (tentative) Feb 18 Introduction Feb 25 Lecture: Geometry, Camera Model, Calibration Mar 4 Lecture: Features,

slide-1
SLIDE 1

3D Photography: Stereo Vision

Kalin Kolev, Marc Pollefeys Spring 2013

http://cvg.ethz.ch/teaching/2013spring/3dphoto/

slide-2
SLIDE 2

Feb 18 Introduction Feb 25 Lecture: Geometry, Camera Model, Calibration Mar 4 Lecture: Features, Tracking/Matching Mar 11

Project Proposals by Students

Mar 18 Lecture: Epipolar Geometry Mar 25

Lecture: Stereo Vision

Apr 1 Easter Apr 8 Short lecture “SfM / SLAM” + 2 papers Apr 15

Project Updates

Apr 22 Short lecture “Active Ranging, Structured Light” + 2 papers Apr 29 Short lecture “Volumetric Modeling” + 2 papers May 6 Short lecture “Mesh-based Modeling” + 2 papers May 13 Short lecture “Shape-from-X” + 2 papers May 20 Pentecost / White Monday May 27

Final Demos

Schedule (tentative)

slide-3
SLIDE 3

Stereo & Multi-View Stereo

http://cat.middlebury.edu/stereo/ Tsukuba dataset

slide-4
SLIDE 4

Stereo

  • Standard stereo geometry
  • Stereo matching
  • Correlation
  • Optimization (DP, GC)
  • General camera configuration
  • Rectifications
  • Plane-sweep
  • Multi-view stereo
slide-5
SLIDE 5

Stereo

slide-6
SLIDE 6

Occlusions

(S lide from Pascal Fua)

slide-7
SLIDE 7

Exploiting scene constraints

slide-8
SLIDE 8

Ordering constraint

1 2 3 4,5 6 1 2,3 4 5 6 2 1 3 4,5 6 1 2,3 4 5 6

surface slice surface as a path

  • cclusion right
  • cclusion left
slide-9
SLIDE 9

Uniqueness constraint

  • In an image pair each pixel has at most
  • ne corresponding pixel
  • In general one corresponding pixel
  • In case of occlusion there is none
slide-10
SLIDE 10

Disparity constraint

surface slice surface as a path

bounding box

use reconstructed features to determine bounding box

constant disparity surfaces

slide-11
SLIDE 11

Stereo matching

Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints

  • epipolar
  • ordering
  • uniqueness
  • disparity limit

Trade-off

  • Matching cost (data)
  • Discontinuities (prior)

Consider all paths that satisfy the constraints pick best using dynamic programming

slide-12
SLIDE 12

Hierarchical stereo matching

Downsampling

(Gaussian pyramid)

Disparity propagation

Allows faster computation Deals with large disparity ranges

slide-13
SLIDE 13

Disparity map

image I(x,y) image I´(x´,y´) Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

slide-14
SLIDE 14
slide-15
SLIDE 15

Energy minimization

(S lide from Pascal Fua)

slide-16
SLIDE 16

Graph Cut

(S lide from Pascal Fua)

(general formulation requires multi-way cut!)

slide-17
SLIDE 17

(Boykov et al ICCV‘ 99) (Roy and Cox ICCV‘ 98)

Simplified graph cut

slide-18
SLIDE 18
slide-19
SLIDE 19

Stereo matching with general camera configuration

slide-20
SLIDE 20

Image pair rectification

slide-21
SLIDE 21

Planar rectification

Bring two views to standard stereo setup (moves epipole to ∞) (not possible when in/close to image)

~ image size

(calibrated)

Distortion minimization

(uncalibrated)

slide-22
SLIDE 22
slide-23
SLIDE 23

Polar re-paramet erizat ion around epipoles Requires only (orient ed) epipolar geomet ry Preserve lengt h of epipolar lines Choose ∆θ so t hat no pixels are compressed

  • riginal image

rectified image

Polar rectification

(Pollefeys et al. ICCV’99)

Works for all relative motions Guarantees minimal image size

slide-24
SLIDE 24

polar rectification planar rectification

  • riginal

image pair

slide-25
SLIDE 25

Example: Béguinage of Leuven

Does not work with standard Homography-based approaches

slide-26
SLIDE 26

Stereo camera configurations

(S lide from Pascal Fua)

slide-27
SLIDE 27
  • Multi-baseline, multi-resolution
  • At each depth, baseline and resolution

selected proportional to that depth

  • Allows to keep depth accuracy constant!

Variable Baseline/Resolution Stereo

(Gallup et al., CVPR08)

slide-28
SLIDE 28

Variable Baseline/Resolution Stereo: comparison

slide-29
SLIDE 29

Multi-view depth fusion

  • Compute depth for every

pixel of reference image

  • Triangulation
  • Use multiple views
  • Up- and down sequence
  • Use Kalman filter

(Koch, Pollefeys and Van Gool. ECCV‘ 98)

Allows to compute robust texture

slide-30
SLIDE 30

Plane-sweep multi-view matching

  • Simple algorithm for multiple cameras
  • no rectification necessary
  • doesn’t deal with occlusions

Collins’96; Roy and Cox’98 (GC); Yang et al.’02/’03 (GPU)

slide-31
SLIDE 31

Space Carving

slide-32
SLIDE 32

3D Reconstruction from Calibrated Images

Scene Volume

V

Input Images (Calibrated)

Goal: Determine transparency, radiance of points in V

slide-33
SLIDE 33

Discrete Formulation: Voxel Coloring

Discretized Scene Volume Input Images (Calibrated)

Goal: Assign RGBA values to voxels in V

photo-consistent with images

slide-34
SLIDE 34

Complexity and Computability

Discretized Scene Volume

N voxels C colors

3

All Scenes (CN3) Photo-Consistent Scenes True Scene

slide-35
SLIDE 35

Issues

Theoretical Questions

Identify class of all photo-consistent scenes

Practical Questions

How do we compute photo-consistent models?

slide-36
SLIDE 36
  • 1. C= 2 (silhouettes)

Volume intersection [Martin 81, Szeliski 93]

  • 2. C unconstrained, viewpoint constraints

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

slide-37
SLIDE 37

Reconstruction from Silhouettes (C = 2)

Binary Images

Approach:

Backproject each silhouette Intersect backprojected volumes

slide-38
SLIDE 38

Voxel Algorithm for Volume Intersection

Color voxel black if on silhouette in every image

O(MN3), for M images, N3 voxels Don’t have to search 2N3 possible scenes!

slide-39
SLIDE 39

Properties of Volume Intersection

Pros

  • Easy to implement, fast
  • Accelerated via octrees [Szeliski 1993]

Cons

  • No concavities
  • Reconstruction is not photo-consistent
  • Requires identification of silhouettes
slide-40
SLIDE 40
  • 1. C= 2 (silhouettes)

Volume intersection [Martin 81, Szeliski 93]

  • 2. C unconstrained, viewpoint constraints

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

slide-41
SLIDE 41
  • 1. Choose voxel
  • 2. Project and correlate
  • 3. Color if consistent

Voxel Coloring Approach

Visibility Problem: in which images is each voxel visible?

slide-42
SLIDE 42

Layers

Depth Ordering: visit occluders first!

Scene Traversal

Condition: depth order is view-independent

slide-43
SLIDE 43

Compatible Camera Configurations

Depth-Order Constraint

Scene outside convex hull of camera centers

Outward-Looking

cameras inside scene

Inward-Looking

cameras above scene

slide-44
SLIDE 44

Calibrated Image Acquisition

Calibrated Turntable

360° rotation (21 images)

Selected Dinosaur Images Selected Flower Images

slide-45
SLIDE 45

Voxel Coloring Results

Dinosaur Reconstruction

72 K voxels colored 7.6 M voxels tested 7 min. to compute

  • n a 250MHz SGI

Flower Reconstruction

70 K voxels colored 7.6 M voxels tested 7 min. to compute

  • n a 250MHz SGI
slide-46
SLIDE 46

Limitations of Depth Ordering

A view-independent depth order may not exist

p q

Need more powerful general-case algorithms

Unconstrained camera positions Unconstrained scene geometry/topology

slide-47
SLIDE 47
  • 1. C= 2 (silhouettes)

Volume intersection [Martin 81, Szeliski 93]

  • 2. C unconstrained, viewpoint

constraints

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

slide-48
SLIDE 48

Space Carving Algorithm

Space Carving Algorithm

Image 1 Image N

…...

Initialize to a volume V containing the true scene Repeat until convergence Choose a voxel on the current surface Carve if not photo-consistent Project to visible input images

slide-49
SLIDE 49

Convergence

Consistency Property

The resulting shape is photo-consistent

all inconsistent points are removed

Convergence Property

Carving converges to a non-empty shape

a point on the true scene is never removed

V’ V

p

slide-50
SLIDE 50

What is Computable?

The Photo Hull is the UNION of all photo-consistent scenes in V

  • It is a photo-consistent scene reconstruction
  • Tightest possible bound on the true scene
  • Computable via provable Space Carving Algorithm

True Scene V Photo Hull V

slide-51
SLIDE 51

Space Carving Algorithm

The Basic Algorithm is Unwieldy

  • Complex update procedure

Alternative: Multi-Pass Plane Sweep

  • Efficient, can use texture-mapping

hardware

  • Converges quickly in practice
  • Easy to implement
slide-52
SLIDE 52

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence

True Scene Reconstruction

slide-53
SLIDE 53

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-54
SLIDE 54

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-55
SLIDE 55

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-56
SLIDE 56

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-57
SLIDE 57

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-58
SLIDE 58

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-59
SLIDE 59

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-60
SLIDE 60

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-61
SLIDE 61

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-62
SLIDE 62

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-63
SLIDE 63

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-64
SLIDE 64

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-65
SLIDE 65

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-66
SLIDE 66

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-67
SLIDE 67

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-68
SLIDE 68

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-69
SLIDE 69

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-70
SLIDE 70

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-71
SLIDE 71

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-72
SLIDE 72

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-73
SLIDE 73

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-74
SLIDE 74

Space Carving Results: African Violet

I nput I mage (1 of 45) Reconstruction Reconstruction Reconstruction

slide-75
SLIDE 75

Space Carving Results: Hand

I nput I mage (1 of 100) Views of Reconstruction

slide-76
SLIDE 76

Other Features

Coarse-to-fine Reconstruction

  • Represent scene as octree
  • Reconstruct low-res model first, then refine

Hardware-Acceleration

  • Use texture-mapping to compute voxel

projections

  • Process voxels an entire plane at a time

Limitations

  • Need to acquire calibrated images
  • Restriction to simple radiance models
  • Bias toward maximal (fat) reconstructions
  • Transparency not supported
slide-77
SLIDE 77
slide-78
SLIDE 78

voxel occluded

Probal robalistic S Space pace C Carvi arving

Broadhurst et al. ICCV’01

slide-79
SLIDE 79

I

Light Intensity Object Color

N

Normal vector

L

Lighting vector

V

View Vector

R

Reflection vector

color of the light Diffuse color Saturation point 1 1 1

Reflected Light in RGB color space Dielectric Materials (such as plastic and glass)

C 

Space-carving for specular surfaces

(Yang, Pollefeys & Welch 2003)

Extended photoconsistency:

slide-80
SLIDE 80

Experiment

slide-81
SLIDE 81

Animated Views

Our result

slide-82
SLIDE 82

Volumetric Graph cuts

ρ(x)

  • 1. Outer surface
  • 2. Inner surface (at

constant offset)

  • 3. Discretize

middle volume

  • 4. Assign

photoconsistency cost to voxels

Slides from [Vogiatzis et al. CVPR2005]

slide-83
SLIDE 83

Volumetric Graph cuts

Source Sink

Slides from [Vogiatzis et al. CVPR2005]

slide-84
SLIDE 84

Volumetric Graph cuts

Source Sink Cost of a cut ≈ ∫∫ ρ(x) dS S

S

cut ⇔ 3D Surface S

[Boykov and Kolmogorov ICCV 2001]

Slides from [Vogiatzis et al. CVPR2005]

slide-85
SLIDE 85

Volumetric Graph cuts

Source Sink Minimum cut ⇔ Minimal 3D Surface under photo-consistency metric

[Boykov and Kolmogorov ICCV 2001]

Slides from [Vogiatzis et al. CVPR2005]

slide-86
SLIDE 86

Photo-consistency

  • Occlusion
  • 1. Get nearest point
  • n outer surface
  • 2. Use outer

surface for

  • cclusions

Slides from [Vogiatzis et al. CVPR2005]

slide-87
SLIDE 87

Photo-consistency

  • Occlusion

Self occlusion

Slides from [Vogiatzis et al. CVPR2005]

slide-88
SLIDE 88

Photo-consistency

  • Occlusion

Self occlusion

Slides from [Vogiatzis et al. CVPR2005]

slide-89
SLIDE 89

Photo-consistency

  • Occlusion

N threshold on angle between normal and viewing direction threshold= ~60°

Slides from [Vogiatzis et al. CVPR2005]

slide-90
SLIDE 90

Photo-consistency

  • Score

Normalised cross correlation Use all remaining cameras pair wise

Slides from [Vogiatzis et al. CVPR2005]

slide-91
SLIDE 91

Photo-consistency

  • Score

Average NCC = C Voxel score ρ = 1 - exp( -tan2[π(C-1)/4] / σ2 )

0 ≤ ρ ≤ 1 σ = 0.05 in all experiments

Slides from [Vogiatzis et al. CVPR2005]

slide-92
SLIDE 92

Example

Slides from [Vogiatzis et al. CVPR2005]

slide-93
SLIDE 93

Example - Visual Hull

Slides from [Vogiatzis et al. CVPR2005]

slide-94
SLIDE 94

Example - Slice

Slides from [Vogiatzis et al. CVPR2005]

slide-95
SLIDE 95

Example - Slice with graphcut

Slides from [Vogiatzis et al. CVPR2005]

slide-96
SLIDE 96

Example – 3D

Slides from [Vogiatzis et al. CVPR2005]

slide-97
SLIDE 97

Shrinking Bias

  • ‘Balooning’ force
  • favouring bigger volumes that fill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d

  • images. PAMI, 15(11):1131–1147, November 1993.

Slides from [Vogiatzis et al. CVPR2005]

slide-98
SLIDE 98

Shrinking Bias

  • ‘Balooning’ force
  • favouring bigger volumes that fill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131– 1147, November 1993.

∫∫

ρ(x) dS - λ ∫∫∫ dV S V

Slides from [Vogiatzis et al. CVPR2005]

slide-99
SLIDE 99

Shrinking Bias

Slides from [Vogiatzis et al. CVPR2005]

slide-100
SLIDE 100

Shrinking Bias

Slides from [Vogiatzis et al. CVPR2005]

slide-101
SLIDE 101

wij SOURCE

wb wb

Graph

h j i

wb = λh3 wij = 4/3πh2 * (ρi+ρj)/2

[Boykov and Kolmogorov ICCV 2001]

Slides from [Vogiatzis et al. CVPR2005]

slide-102
SLIDE 102

102

Address Memory and Computational Overhead

(Sinha et. al. 2007)

– Compute Photo-consistency only where it is needed – Detect Interior Pockets using Visibility

Graph-cut on Dual of Adaptive Tetrahedral Mesh

slide-103
SLIDE 103

Interesting comparison of multi- view stereo approaches

http://vision.middlebury.edu/mview/