EECS 504: Foundations of Computer Vision Andrew Owens Course staff - - PowerPoint PPT Presentation

▶
eecs 504 foundations of computer vision andrew owens
SMART_READER_LITE
LIVE PREVIEW

EECS 504: Foundations of Computer Vision Andrew Owens Course staff - - PowerPoint PPT Presentation

Apr 06, 2024 495 likes •1.09k views

EECS 504: Foundations of Computer Vision Andrew Owens Course staff Haozhu Wang Anthony Liang Bingqi Sun Graduate student (GSI) Instructional aide (IA) Instructional aide (IA) Interacting with us Ask questions on Piazza. - Sign up:

slide-1
SLIDE 1

EECS 504: Foundations of Computer Vision Andrew Owens

slide-2
SLIDE 2

Haozhu Wang Graduate student (GSI) Anthony Liang Instructional aide (IA) Bingqi Sun Instructional aide (IA)

Course staff

slide-3
SLIDE 3

Interacting with us

  • Ask questions on Piazza.
  • Sign up: https://bit.ly/36mfYeP
  • Submit written work to Gradescope
  • Office hours on website:
slide-4
SLIDE 4

Course website

https://web.eecs.umich.edu/~ahowens/eecs504/w20/

slide-5
SLIDE 5

Grading

  • Assignments (70%)
  • Final project (30%)
  • No exams!
slide-6
SLIDE 6

Assignments

  • Weekly homework assignments (≈10 total)
  • Due each Tuesday at midnight
  • Late submissions penalized 30% per day
  • You have 5 "late days”
  • Assignments should be done independently.
  • Encouraged to discuss them
  • Programming and writing should all be yours
slide-7
SLIDE 7

Assignments

  • Mix of programming and written problems
  • Python + numerical computing libraries (numpy, scipy, etc.)
  • PyTorch for deep learning
  • Linear algebra and multivariable calculus
  • Jupyter notebooks and Google Colab for problem sets
slide-8
SLIDE 8

Assignments

slide-9
SLIDE 9

Project

  • Open-ended! Example projects:
  • Implement and extend a recent computer vision paper
  • Use computer vision in your research
  • We’ll also provide a list of project ideas
  • Work in small groups (up to 4 people)
  • Complete in last month of class.
  • Project proposal (after spring break)
  • Short presentation (finals period)
  • Writeup (finals period)
slide-10
SLIDE 10

Readings

http://szeliski.org/Book

https://www.deeplearningbook.org

Manuscript chapters by Torralba, Freeman, and Isola (on course website). Class based

  • n this coursework.

And also occasional paper readings

slide-11
SLIDE 11

Class topics

slide-12
SLIDE 12

}

Signal processing Apple + orange = ?

Homework problem:

…

slide-13
SLIDE 13

…

Intro to deep learning

}

slide-14
SLIDE 14

Learning for vision

}

Spring break

Beach

…

slide-15
SLIDE 15

…

Cameras,

  • ptics, motion

}

Homework problem:

slide-16
SLIDE 16

Advanced topics and applications

}

slide-17
SLIDE 17

Today

  • 1. A bit of vision history
  • 2. Why vision is hard
  • 3. A simple visual system
slide-18
SLIDE 18

Medical applications Robotics Driving Mobile devices 3D modeling

Exciting times for computer vision

Accessibility

Slide credit: Torralba, Freeman, Isola

slide-19
SLIDE 19

To see

“What does it mean, to see? The plain man's answer (and Aristotle's, too) would be, to know what is where by looking.” To discover from images what is present in the world, where things are, what actions are taking place, to predict and anticipate events in the world.

Slides from MIT 6.869 class by Torralba, Freeman, and Isola

slide-20
SLIDE 20

Slide credit: Torralba, Freeman, Isola

slide-21
SLIDE 21

[“HOGgles”, Vondrick et al. , ICCV 2013]

Just a few years ago…

slide-22
SLIDE 22

[“Mask RCNN”, He et al., ICCV 2017]

Slide credit: Torralba, Freeman, Isola

slide-23
SLIDE 23

[“GauGAN”, Park et al., CVPR 2019]

slide-24
SLIDE 24

Different signals, same methods

Time Amplitude

Sound WiFi

(Zhao et al. 2019)

Touch

(Calandra et al. 2018)

slide-25
SLIDE 25

Input video

(Owens and Efros 2018)

slide-26
SLIDE 26

On-screen audio

(Owens and Efros 2018)

slide-27
SLIDE 27

(Owens and Efros 2018)

Off-screen audio

slide-28
SLIDE 28

What makes vision hard?

slide-29
SLIDE 29

To see: perception vs. measurement

Slide credit: Torralba, Freeman, Isola

slide-30
SLIDE 30

Slide credit: Torralba, Freeman, Isola

To see: perception vs. measurement

slide-31
SLIDE 31

Sinha & Adelson 93

Other ambiguities

Slide credit: Antonio Torralba

slide-32
SLIDE 32

Sinha & Adelson 93

Other ambiguities

Slide credit: Antonio Torralba

slide-33
SLIDE 33

A simple visual system

  • A simple world
  • A simple image formation model
  • A simple goal
slide-34
SLIDE 34

Slide credit: Antonio Torralba

A simple world

slide-35
SLIDE 35

A simple world

Slide credit: Antonio Torralba

slide-36
SLIDE 36

A simple image formation model

Simple world rules:

  • Surfaces can be horizontal or vertical.
  • Objects will be resting on a white horizontal ground plane

Slide credit: Antonio Torralba

slide-37
SLIDE 37

A simple image formation model

Camera plane World reference system

Slide credit: Antonio Torralba

slide-38
SLIDE 38

A simple image formation model

X + x0 cos(θ) Y – sin(θ) Z + y0 x = y = image coordinates World coordinates Image and projection of the world
 coordinate axes into the image plane World coordinates image 
 coordinates

Slide credit: Antonio Torralba

slide-39
SLIDE 39

A simple goal

Recover the 3D structure of the world

We want to recover X(x,y), Y(x,y), Z(x,y) using as input I(x,y)

Slide credit: Antonio Torralba

slide-40
SLIDE 40

Edges

Occlusion Change of Surface orientation Contact edge Shadow boundary Vertical 3D edge Horizontal 3D edge

Slide credit: Antonio Torralba

slide-41
SLIDE 41

Treating the image as a function

y x I(x,y)

255

Slide credit: Antonio Torralba

slide-42
SLIDE 42

Finding edges in the image

Image gradient: Approximation image derivative: Edge strength Edge orientation: Edge normal: I(x,y)

Slide credit: Antonio Torralba

slide-43
SLIDE 43

Finding edges in the image

E(x,y) n(x,y) and I(x,y)

Slide credit: Antonio Torralba

slide-44
SLIDE 44

Edge classification

  • Figure/ground segmentation

– Using the fact that objects have
 color

  • Occlusion edges

– Occlusion edges are

  • wned by 


the foreground

  • Contact edges

Slide credit: Antonio Torralba

slide-45
SLIDE 45

From edges to surface constraints

Y(x,y) Z(x,y) X(x,y)

?

Slide credit: Antonio Torralba

slide-46
SLIDE 46

From edges to surface constraints

  • Ground
  • Contact edge

… now things get a bit more complicated. Y(x,y) = 0 if (x,y) belongs to a ground pixel Y(x,y) = 0 if (x,y) belongs to foreground and is a contact edge

  • What happens inside the objects?

Slide credit: Antonio Torralba

slide-47
SLIDE 47

From edges to surface constraints

Vertical edges

Z = constant along the edge X + x0 cos(θ) Y – sin(θ) Z + y0 x = y = image coordinates World coordinates How can we relate the information in the pixels with 3D surfaces in the world? Given the image, what can we say about X, Y and Z in the pixels that belong
 to a vertical edge?

Slide credit: Antonio Torralba

slide-48
SLIDE 48

From edges to surface constraints

  • Horizontal edges

Y = constant along the edge Where t is the vector parallel to the edge X + x0 cos(θ) Y – sin(θ) Z + y0 x = y = image coordinates World coordinates Given the image, what can we say about X, Y and Z in the pixels that belong
 to an horizontal 3D edge?

Slide credit: Antonio Torralba

slide-49
SLIDE 49

From edges to surface constraints

  • What happens where there are no edges?

? Assumption of planar faces: Information has to be propagated from the edges

Slide credit: Antonio Torralba

slide-50
SLIDE 50

A simple inference scheme

All the constraints are linear!

Y(x,y) = 0 if (x,y) belongs to a ground pixel if (x,y) belongs to a vertical edge if (x,y) belongs to an horizontal edge if (x,y) is not on an edge

A similar set of constraints could be derived for Z

Slide credit: Antonio Torralba

slide-51
SLIDE 51

Discrete approximation

We can transform every differential constraint into a linear constraint on Y(x,y)

Y(x,y)

111 115 113 111 112 111 112 111 135 138 137 139 145 146 149 147 163 168 188 196 206 202 206 207 180 184 206 219 202 200 195 193 189 193 214 216 104 79 83 77 191 201 217 220 103 59 60 68 195 205 216 222 113 68 69 83 199 203 223 228 108 68 71 77

dY dx ≈ Y(x,y) – Y(x-1,y)

Slide credit: Antonio Torralba

slide-52
SLIDE 52

Discrete approximation

Y(x,y) Transform the “image” Y(x,y) into a column vector:

  • 1

1 x=0
 y=0

dY dx ≈ Y(x,y) – Y(x-1,y) = Y(2,2) – Y(1,2)= x=2, y=2

Slide credit: Antonio Torralba

slide-53
SLIDE 53

A simple inference scheme: solve for Y

=

A Y = b

Constraint weights Y b

Slide credit: Antonio Torralba

slide-54
SLIDE 54

Results

X Y Z Edge normals Edge strength 3D orientation Depth discontinuities Contact edges

Slide credit: Antonio Torralba

slide-55
SLIDE 55

Changing view point

Input New view points:

Slide credit: Antonio Torralba

slide-56
SLIDE 56

Failure cases… even in a simple world!

slide-57
SLIDE 57

Failure cases… even in a simple world!

slide-58
SLIDE 58

Failure cases… even in a simple world!

Edges Input image

Missing edges Extra edges

More on this next week!

slide-59
SLIDE 59

Questions about the course?