Interest point detection Nicolas ROUGON ARTEMIS Department - - PowerPoint PPT Presentation

▶
interest point detection
SMART_READER_LITE
LIVE PREVIEW

Interest point detection Nicolas ROUGON ARTEMIS Department - - PowerPoint PPT Presentation

Jan 15, 2024 167 likes •764 views

IMA4103 Extraction dInformation Multimdia Interest point detection Nicolas ROUGON ARTEMIS Department Nicolas.Rougon@telecom-sudparis.eu Institut Mines-Tlcom Problem statement We hereafter review methods for extracting local

slide-1
SLIDE 1

Institut Mines-Télécom

Interest point detection

Nicolas ROUGON

ARTEMIS Department

Nicolas.Rougon@telecom-sudparis.eu

IMA4103

Extraction d’Information Multimédia

slide-2
SLIDE 2

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ We hereafter review methods for extracting

local geometric features of interest in gray level images, useable in a variety of image matching problems

  • Image registration

► image stitching | augmented reality

  • Image retrieval & object recognition/categorization

► image & video indexing

  • 3D scene/object reconstruction

► vision-based 3D photogrammetry

  • Tracking & navigation

► simultaneous localization and mapping (SLAM)

slide-3
SLIDE 3

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ Motivation

Matching techniques using local features of interest are significantly more robust to large variations of scene geometry, including than approaches assessing similarity between image (sub)domains

  • strong viewpoint

change

  • partial occlusion
  • object deformation
slide-4
SLIDE 4

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

  • Structural properties

 generic  sparse ► compactness ► computational efficiency  numerous ► robustness

 uniformly distributed  occlusions | clutter | cropping

■ Requirements

Relevant features of interest should be distinctive, and satisfy properties ensuring stable and efficient detection / matching

slide-5
SLIDE 5

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ Requirements

Relevant features of interest should be distinctive, and satisfy properties ensuring stable and efficient detection / matching

  • Invariance properties

► repeatability  contrast transforms  sensor photometric calibration  scene lighting monotonic luminance transforms  spatial transforms

 sensor geometric calibration  viewpoint

isometries | scalings | affine transforms

slide-6
SLIDE 6

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

  • Robustness properties

► repeatability ► accuracy  sampling & quantization  digital image acquisition  coding scheme  noise  sensor model

■ Requirements

Relevant features of interest should be distinctive, and satisfy properties ensuring stable and efficient detection / matching

slide-7
SLIDE 7

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ Candidate features

slide-8
SLIDE 8

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ Candidate features

Edges are not eligible as features of interest

  • Generic, sparse, uniformly distributed
  • Reasonably invariant to contrast changes
  • Not distinctive

► matching ambiguity occurs along edges

  • Not invariant to spatial transforms

t n Ltarget = c Lsource = c

? ? ?

slide-9
SLIDE 9

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ Candidate features

Corners provide relevant features of interest

  • Distinctive
  • Generic, sparse, numerous, uniformly distributed
  • Invariant to contrast changes
  • Invariant to spatial transforms except scalings
slide-10
SLIDE 10

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ Interest point matching

  • Detection

 Extract a set of distinctive & repeatable interest points  Define an invariant interest patch around each keypoint

  • Description

 Normalize & transform patches into invariant local coordinates  Compute a patch local descriptor

  • Matching

 Match local descriptors based

  • n some similarity metrics

x1

1

x3

1 1

x2 x3

2 2

x2 x1

2 1

fi

2

fj

1

f1

1

fd

2

f1

2

fd ► ◄ ► ◄

1

fi d( , )

2

fj

▼ ▼

slide-11
SLIDE 11

Institut Mines-Télécom

Example applications

IMA 4103 - Nicolas ROUGON

■ Image stitching

View #1 View #2

slide-12
SLIDE 12

Institut Mines-Télécom

Example applications

IMA 4103 - Nicolas ROUGON

■ Image stitching

Corners #1 Corners #2

► Corners capture the geometry of textured shapes

slide-13
SLIDE 13

Institut Mines-Télécom

Example applications

IMA 4103 - Nicolas ROUGON

■ Image stitching

Corner matching

slide-14
SLIDE 14

Institut Mines-Télécom

Example applications

IMA 4103 - Nicolas ROUGON

■ Image stitching

View stitching

slide-15
SLIDE 15

Institut Mines-Télécom

IMA 4103 - Nicolas ROUGON

Strong viewpoint changes | Partial occlusions

■ Video Tracking

Example applications

slide-16
SLIDE 16

Institut Mines-Télécom

IMA 4103 - Nicolas ROUGON

Strong viewpoint changes | Partial occlusions | Object deformations

■ Video Tracking

Example applications

slide-17
SLIDE 17

Institut Mines-Télécom

Example applications

IMA 4103 - Nicolas ROUGON

■ Multi-view 3D scene reconstruction

Feature extraction

> corner points

Feature matching

> motion vectors

Camera + sparse depth estimation

> 3D point cloud

Surface reconstruction + texturing

> 3D textured mesh

slide-18
SLIDE 18

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

  • Good detection

■ Requirements

Expected performances of relevant interest point detectors ◄ generic framework for performance assessment

slide-19
SLIDE 19

Institut Mines-Télécom

Performance assessment

IMA 4103 - Nicolas ROUGON

■ Confusion matrix

Type I error Type II error

  • True Positives (TP)

Correct detections

  • True Negatives (TN)

Correct rejections

  • False Positives (FP)

Wrong detections False alarm | Type I error

  • False Negatives (FN)

Wrong rejections Miss | Type II error

Type I error Type II error

True Positives False Positives True Negatives False Negatives False True Ground Truth (X) Detection (Y) Negative Positive

Type I error Type II error

Success Y = X Failure Y ≠ X

slide-20
SLIDE 20

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

  • Good detection

 few Failures

 few false positives  few false negatives

■ Requirements

Expected performances of relevant interest point detectors ◄ dedicated error metrics

slide-21
SLIDE 21

Institut Mines-Télécom

Performance metrics

IMA 4103 - Nicolas ROUGON

■ Recall | Sensitivity | True Positive Rate

  • Probability of relevant samples

to be detected > P[Y=1|X=1]

𝑠𝑓𝑑𝑏𝑚𝑚 = TP TP + FN

  • 𝑠𝑓𝑑𝑏𝑚𝑚 ↗ 1 when FN ↘ 0

► assessment of false negatives ► false positives not addressed

Type I error Type II error

True Positives False Positives True Negatives False Negatives False True Ground Truth (X) Detection (Y) Negative Positive

Type I error Type II error

slide-22
SLIDE 22

Institut Mines-Télécom

Performance metrics

■ Precision | Positive Predicted Value

Type I error Type II error

𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜 = TP TP + FP

  • Probability of detections

to be relevant > P[X=1|Y=1]

Type I error Type II error

True Positives False Positives True Negatives False Negatives False True Ground Truth (X) Detection (Y) Negative Positive

Type I error Type II error

  • 𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜 ↗ 1 when FP ↘ 0

► assessment of false positives ► false negatives not addressed

IMA 4103 - Nicolas ROUGON

slide-23
SLIDE 23

Institut Mines-Télécom

Performance metrics

IMA 4103 - Nicolas ROUGON

■ Specificity | True Negative Rate

Type I error Type II error

  • Probability of irrelevant samples

to be rejected > P[Y=0|X=0]

𝑡𝑞𝑓𝑑𝑗𝑔𝑗𝑑𝑗𝑢𝑧 = TN TN + FP

Type I error Type II error

True Positives False Positives True Negatives False Negatives False True Ground Truth (X) Detection (Y) Negative Positive

Type I error Type II error

  • 𝑡𝑞𝑓𝑑𝑗𝑔𝑗𝑑𝑗𝑢𝑧 ↗ 1 when FP ↘ 0

► assessment of false positives

slide-24
SLIDE 24

Institut Mines-Télécom

Performance metrics

IMA 4103 - Nicolas ROUGON

■ Negative Predicted Value

Type I error Type II error

  • Probability of rejections

to be irrelevant > P[X=0|Y=0]

𝑂𝑄𝑊 = TN TN + FN

Type I error Type II error

True Positives False Positives True Negatives False Negatives False True Ground Truth (X) Detection (Y) Negative Positive

Type I error Type II error

  • 𝑂𝑄𝑊 ↗ 1 when FN ↘ 0

► assessment of false negatives

slide-25
SLIDE 25

Institut Mines-Télécom

Performance metrics

IMA 4103 - Nicolas ROUGON

𝐵𝐷𝐷 = TP + TN TP + TN + FP + FN

  • Probability of correct decision

(detection/rejection) > P[Y=X]

■ Accuracy

Type I error Type II error

True Positives False Positives True Negatives False Negatives False True Ground Truth (X) Detection (Y) Negative Positive

Type I error Type II error

  • 𝐵𝐷𝐷 ↗ 1 when (FP, FN) ↘ 0

► joint assessment of false positives & false negatives, from detections & rejections

slide-26
SLIDE 26

Institut Mines-Télécom

Performance metrics

IMA 4103 - Nicolas ROUGON

𝐺 = 2 𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜 ∙ 𝑠𝑓𝑑𝑏𝑚𝑚 𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜 + 𝑠𝑓𝑑𝑏𝑚𝑚

  • Harmonic mean of precision

and recall

■ F-score

Type I error Type II error

True Positives False Positives True Negatives False Negatives False True Ground Truth (X) Detection (Y) Negative Positive

Type I error Type II error

  • 𝐺 ↗ 1 when (FP, FN) ↘ 0

► joint assessment of false positives & false negatives, focusing on detections

= 2 TP 2 TP + FP + FN

slide-27
SLIDE 27

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

  • Good detection

 high sensitivity / recall = all (most) true positives  high specificity / precision

= few false positives

  • Robustness against noise
  • Good localization

► accuracy

  • Computational efficiency

■ Requirements

Expected performances of relevant interest point detectors

► repeatability

slide-28
SLIDE 28

Institut Mines-Télécom

Differential corner detection

IMA 4103 - Nicolas ROUGON

slide-29
SLIDE 29

Institut Mines-Télécom

Corner detection

IMA 4103 - Nicolas ROUGON

■ Basic idea

At corner points, comparing an image patch to its neighbors shows dissimilarity in all directions

  • Low-texture region ● Edge
  • Corner

► similar in all directions ► similar along edge direction ► dissimilar in all directions

slide-30
SLIDE 30

Institut Mines-Télécom

ρ Kρ

Corner detection

IMA 4103 - Nicolas ROUGON

■ Basic idea

At corner points, comparing an image patch to its neighbors shows dissimilarity in all directions

  • This requires defining a similarity metric between image patches,

and search for its local maxima over a space of admissible shifts u

  • A natural choice is (weighted) autocorrelation

 Kρ : kernel with extension ρ

► patch support unit | Gaussian  Opting for a unit kernel and 8-connected shifts yields the (early) Moravec detector

► not isotropic

x

slide-31
SLIDE 31

Institut Mines-Télécom

Corner detection

IMA 4103 - Nicolas ROUGON

■ Quadratic approximation of the autocorrelation metric

  • 1st-order Taylor expansion:
  • The matrix is known as the structure tensor

► Quadratic approximation

slide-32
SLIDE 32

Institut Mines-Télécom

  • A robust estimate of the structure tensor is obtained using

regularized image derivatives Gaussian: | Canny-Deriche

  • Expanded form

Corner detection

IMA 4103 - Nicolas ROUGON

■ Structure tensor

 σ = local scale  ρ = integration scale

slide-33
SLIDE 33

Institut Mines-Télécom

  • Pointwise patch (ρ = 0)

Corner detection

IMA 4103 - Nicolas ROUGON

■ Structure tensor

The information in the tensor (symmetric, positive definite) is described by its eigenvectors (dmax, dmin) and eigenvalues (λmax, λmin) ► same information in and

  • describes average gradient properties over patch support

 dmax (dmin) : dominant (anti-dominant) orientation  λmax , λmin : directional contrast values 

slide-34
SLIDE 34

Institut Mines-Télécom

Corner detection

IMA 4103 - Nicolas ROUGON

■ Structure tensor

  • Edge

 1 dominant direction – 1 large directional gradient  λmax large – λmin small

slide-35
SLIDE 35

Institut Mines-Télécom

Corner detection

IMA 4103 - Nicolas ROUGON

■ Structure tensor

  • Low-textured region

 no dominant direction – no gradient information  λmax small – λmin small

slide-36
SLIDE 36

Institut Mines-Télécom

Corner detection

IMA 4103 - Nicolas ROUGON

■ Structure tensor

  • High-textured region | Corner

 no dominant direction – large directional gradients  λmax large – λmin large

slide-37
SLIDE 37

Institut Mines-Télécom

Corner detection

IMA 4103 - Nicolas ROUGON

■ Structure tensor

  • At corner points, the smallest eigenvalue λmin of the structure

tensor is large enough

  • This property is exploited by 2 widely-used corner detectors

 Kanade-Lucas-Tomasi (KLT)  Harris-Förstner

slide-38
SLIDE 38

Institut Mines-Télécom

The KLT detector

IMA 4103 - Nicolas ROUGON

slide-39
SLIDE 39

Institut Mines-Télécom

Kanade-Lucas-Tomasi (KLT) detector

IMA 4103 - Nicolas ROUGON

■ Algorithm

Given a threshold λ on λmin and the size D of a square neighborhood

  • Compute image gradient
  • Initialize a point list L. For each x  Ω

− compute structure tensor using a unit (D x D) kernel − compute smallest eigenvalue λmin

− if λmin> λ, insert x into L

  • Sort L in decreasing order of λmin ► Lsort
  • Scan Lsort from top to bottom

− for each current point x  Lsort, discard all points after x in Lsort located in the (D x D) neighborhood of x

slide-40
SLIDE 40

Institut Mines-Télécom

Kanade-Lucas-Tomasi (KLT) detector

IMA 4103 - Nicolas ROUGON

■ Hyperparameters

  • The threshold λ controls the sensitivity of the detector

 λ can be estimated from the histogram of λmin which has usually an obvious valley near 0  however, this valley does not always exist

  • The kernel / neighborhood size D is estimated empirically

 in most cases: D  [2,10]  large values of D induce delocalization artifacts and neighboring corner fusion

slide-41
SLIDE 41

Institut Mines-Télécom

The Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

slide-42
SLIDE 42

Institut Mines-Télécom

  • The Harris-Förstner detector makes use of both eigenvalues
  • f the structure tensor via their ratio
  • To avoid computing (λmax, λmin) explicitly, similitude invariants
  • f are used

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Principles

slide-43
SLIDE 43

Institut Mines-Télécom

  • The latter are combined into a corner index

 setting a threshold on r induces a threshold on α

  • This yields the following corner metric

 hyperparameter α  [0, 0.25]

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Principles

− large at corner points − small in low-texture regions − negative at edge points

slide-44
SLIDE 44

Institut Mines-Télécom

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Algorithm

Given a value of α and a threshold tR on R

  • Compute image gradient
  • For each x  Ω

− compute structure tensor using a Gaussian kernel Gρ Standard choice: ρ = 2σ − compute corner metric R(x)

  • Threshold corner map R above tR and retain only local maxima*

* via Non-Maximal Suppression in the 8-connected neighborhood

  • Filter out weak*corners in the ρ-neighborhood of strong*corners

in a way similar to KLT

* w.r.t. the corner metric R

slide-45
SLIDE 45

Institut Mines-Télécom

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Hyperparameters

  • The parameter α controls the sensitivity of the detector

and is tuned empirically

 sensitivity  when α   in most cases: α  [0.04, 0.06]

  • The threshold R is tuned empirically

− usually, R is set close to 0

0.05 α 0.10 0.20 0.22 0.24

slide-46
SLIDE 46

Institut Mines-Télécom

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Hyperparameters

  • The parameter α controls the

sensitivity of the detector and is tuned empirically

 sensitivity  when α   in most cases: α  [0.04, 0.06]

  • The threshold R is tuned empirically

 usually, R is set close to 0

0.05 α 0.10 0.20 0.22 0.24

slide-47
SLIDE 47

Institut Mines-Télécom

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

  • riginal
slide-48
SLIDE 48

Institut Mines-Télécom

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

Corner metric R

slide-49
SLIDE 49

Institut Mines-Télécom

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

Thresholded corner metric R > R

slide-50
SLIDE 50

Institut Mines-Télécom

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

Corner metric local maxima

slide-51
SLIDE 51

Institut Mines-Télécom

Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

Harris points

slide-52
SLIDE 52

Institut Mines-Télécom

KLT vs. Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Properties

  • Isometry-invariance

Inherited from eigenvalues properties

  • Insensitive to affine intensity transforms

Local maxima of λmin / R are preserved R(x) x x R

φ(L) = aL + b

slide-53
SLIDE 53

Institut Mines-Télécom

KLT vs. Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Limitations

  • Not invariant to scaling

Using J at fixed scale ρ in both views

View #1 View #2

► multiple points detected as edges ► single point detected as a corner

ρ

slide-54
SLIDE 54

Institut Mines-Télécom

KLT vs. Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Performance comparison

KLT Harris

slide-55
SLIDE 55

Institut Mines-Télécom

KLT vs. Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ Performance comparison

KLT Harris

slide-56
SLIDE 56

Institut Mines-Télécom

KLT vs. Harris-Förstner detector

IMA 4103 - Nicolas ROUGON

■ KLT detector

  • Output is usually closer to human perception of corners
  • Often used for motion tracking

► widespread KLT Tracker

  • Mostly used in the US

■ Harris-Förstner detector

  • Good repeatability under varying rotation and lighting
  • Often used for 3D scene reconstruction and image retrieval
  • Mostly used in Europe
slide-57
SLIDE 57

Institut Mines-Télécom

Temporary conclusion

IMA 4103 - Nicolas ROUGON

■ Topics to be further addressed

  • Detection

 scale-invariance issue

Fixed by automatic scale selection ► Harris-Laplacian detector

 2nd-order detectors ► Hessian detector ► Hessian-Laplace detector

  • Description

Define / extract feature vector descriptors around interest points

  • Matching

Estimate correspondence between descriptors in each view

slide-58
SLIDE 58

Institut Mines-Télécom

Interest point detection

Nicolas ROUGON

ARTEMIS Department

Nicolas.Rougon@telecom-sudparis.eu

IMA4103

Extraction d’Information Multimédia