Geometric Registration for Deformable Shapes 3.5 Articulated - - PowerPoint PPT Presentation

▶
geometric registration for deformable shapes
SMART_READER_LITE
LIVE PREVIEW

Geometric Registration for Deformable Shapes 3.5 Articulated - - PowerPoint PPT Presentation

Apr 25, 2023 279 likes •798 views

Geometric Registration for Deformable Shapes 3.5 Articulated Registration Graph cuts and piecewise-rigid registration [CZ08] Articulated registration [CZ09] Implementation issues and alternatives Articulated registration Movement consists of

slide-1
SLIDE 1

Geometric Registration for Deformable Shapes

3.5 Articulated Registration

Graph cuts and piecewise-rigid registration [CZ08] Articulated registration [CZ09] Implementation issues and alternatives

slide-2
SLIDE 2

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Articulated registration

Movement consists of few parts

  • Material so far focused on matching individual corresp
  • Now: point groups move together
  • Each group according to a single rigid transformation

2

slide-3
SLIDE 3

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

How can we simplify the problem?

  • Before: Optimizing individual correspondence assignment
  • Articulated: Optimizing correspondence of groups
  • Q) What are the groups?
  • Generally: don’t know in advance.
  • If we know in advance: [PG08]
  • Q) What is the motion for each group?
  • We can guess well
  • ICP based search, feature based search

3

slide-4
SLIDE 4

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Basic idea

  • If we know the articulated movement (small set of

transformations {T} )

  • Reformulate optimization
  • Find an assignment of transformations to the points that

“minimizes registration error”

4

Transformations from finite set

Source Shape Target Shape

slide-5
SLIDE 5

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Basic idea

Find the assignment of transformations in {T} to points in P, that maximizes:

Transformations from finite set

Source Shape Target Shape

5

} { , ) ,..., (

1 , ) ( , 1 ) ( 1 ) (

T x P P x x P

i n j i compatible j i n i single i n match

∈ =

∏ ∏

= =

“Data” and “Smoothness” terms evaluate quality of assignment

slide-6
SLIDE 6

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

How to find transformations?

Global search / feature matching strategy [CZ08]

  • Sample transformations in advance by feature matching
  • Inspired by partial symmetry detection [MGP06]

Local search / refinement strategy [CZ09]

  • Start with initial part labeling, keep refining

transformations of each part via ICP

  • Refine part labels using transformations, repeat

alternation

6

slide-7
SLIDE 7

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Search by Feature Matching

Find transformations that move parts of the source to parts of the target

7

Source Shape Target Shape

slide-8
SLIDE 8

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Motion Sampling Illustration

Find transformations that move parts of the source to parts of the target

8

Source Shape Target Shape Sampled Points

slide-9
SLIDE 9

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Motion Sampling Illustration

Find transformations that move parts of the source to parts of the target

9

Source Shape Target Shape

slide-10
SLIDE 10

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Motion Sampling Illustration

Find transformations that move parts of the source to parts of the target

10

Translate Rotate and Translate Translate Rotations Translations

Source Shape Target Shape Transformation Space

slide-11
SLIDE 11

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Motion Sampling Illustration

Find transformations that move parts of the source to parts of the target

11

s2 s1 t1 t2

s1t1 s2t1 s1t2 s2t2

Source Shape Target Shape Transformation Space

Rotations Translations

slide-12
SLIDE 12

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Basic idea

Find the assignment of transformations in {T} to points in P, that maximizes: A discrete labelling problem  Graph Cuts for optimization

Transformations from finite set

Source Shape Target Shape

12

} { , ) ,..., (

1 , ) ( , 1 ) ( 1 ) (

T x P P x x P

i n j i compatible j i n i single i n match

∈ =

∏ ∏

= =

“Data” and “Smoothness” terms evaluate quality of assignment

slide-13
SLIDE 13

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Data Term

For each mesh vertex: Move close to target How to measure distance to target?

  • Apply assigned transformation for all =
  • Measure distance to closest point in target

13

slide-14
SLIDE 14

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

For each mesh edge: preserve length of edge

  • Both versions of fq(q) moved q close to the target
  • Disambiguate by preferring the one that preserves length

Smoothness Term

Original Length Transformed Length

14

slide-15
SLIDE 15

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Symmetric Cost Function

Swapping source / target can give different results

  • Optimize {T} assignment in both meshes
  • Assign {T} on source vertices, {T-1} on target vertices
  • Enforce consistent assignment: penalty when

15

, No Penalty

u p

f f = , Constant Penalty

u p

f f ≠

slide-16
SLIDE 16

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

 Data and smoothness terms apply to both shapes  Additional symmetric consistency term  Weights to control relative influence of each term

 Use “graph cuts” to optimize assignment

 [Boykov, Veksler & Zabih PAMI ’01]

Optimization Using Graph Cuts

Data Smoothness argmin

Assignment from a set

  • f transformations

+

Source Target

Data

Source Target

Smoothness + + + Symmetric ConsistencySource & Target

16

slide-17
SLIDE 17

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

17

Synthetic Dataset Example

Motion Segmentation (from Graph Cuts) Source Target Aligned Result

Registration Error

1.5% 0%

slide-18
SLIDE 18

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Synthetic Dataset w/ Holes

18

Source Aligned Result

Distance (from Target) to the closest point (% bounding box diagonal)

5.3% 0% Target

slide-19
SLIDE 19

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Arm Dataset Example

19

Source Noisy Target

Missing Data Missing Data

slide-20
SLIDE 20

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Arm Dataset Example

20

Aligned Result Motion Segmentation 5.4% 0% Distance (from Target) to the closest point (% bounding box diagonal)

slide-21
SLIDE 21

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Performance

Graph cuts optimization is most time-consuming step

  • Symmetric optimization doubles variable count
  • Symmetric consistency term introduces many edges

Performance improved by subsampling

  • Use k-nearest neighbors for connectivity

21

Dataset #Points # Labels Matching Clustering Pruning Graph Cuts Horse 8431 1500 2.1 min 3.0 sec (skip) 1.6 sec 1.1 hr Arm 11865 1000 55.0 sec 0.9 sec 12.4 min 1.2 hr Hand (Front) 8339 1500 14.5 sec 0.7 sec 7.4 min 1.2 hr Hand (Back) 6773 1500 17.3 sec 0.9 sec 9.4 min 1.6 hr

slide-22
SLIDE 22

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

How to find transformations?

Global search / feature matching strategy

  • Sample transformations in advance by feature matching
  • Inspired by partial symmetry detection [MGP06]

Local search / refinement strategy

  • Start with initial part labeling, keep refining

transformations of each part via ICP

  • Iterate between transformation refinement / part

assignment until convergence

  • Establish relationship between parts  preserve shape

connectivity & obtain deformable model

22

slide-23
SLIDE 23

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Part label representation

Part labelling: each point assigned vector of weights Transformations move each point according to its weights

23

[1,0] [0.5, 0.5] [0,1]

Bone 1 Bone 2

Weighted Blending Result

slide-24
SLIDE 24

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Weight Grid

Define weights on grid enclosing surface

  • Covers small holes, reduces variables
  • Provides regular structure for optimization
  • Trilinear interpolation inside grid cells – gives weights

everywhere inside the grid domain

24

Shape Grid

slide-25
SLIDE 25

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Optimization overview

25

Initialization Weight Refinement Final Result Main Optimization Loop

T-Step: Refine the transformations for each part W-Step: Optimize assignment of transformations

slide-26
SLIDE 26

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Optimization strategy

26

slide-27
SLIDE 27

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Optimization strategy

27

(Converged)

slide-28
SLIDE 28

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Optimization strategy

28

slide-29
SLIDE 29

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Optimization strategy

29

(Converged)

slide-30
SLIDE 30

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Optimization strategy

30

slide-31
SLIDE 31

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Optimization strategy

31

(Finished)

slide-32
SLIDE 32

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Distance Term

Fix weights & solve for transformations

32

Source Target

slide-33
SLIDE 33

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Distance Term

Fix weights & solve for transformations

  • Use closest point correspondences

33

Bone 1 Bone 2 Bone 3

slide-34
SLIDE 34

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Distance Term

Fix weights & solve for transformations

  • Use closest point correspondences

34

Bone 1 Bone 2 Bone 3

slide-35
SLIDE 35

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Distance Term

Fix weights & solve for transformations

  • Use closest point correspondences
  • Iterate further until convergence

35

Bone 1 Bone 2 Bone 3

slide-36
SLIDE 36

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Joint Constraint Term

Prevent neighboring bones from separating

36

Bone 1 Bone 2 Bone 3

slide-37
SLIDE 37

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Joint Constraint Term

Prevent neighboring bones from separating

  • Constrain overlapping weight regions

37

Unwanted stretch Bone 1 Bone 2 Bone 3

slide-38
SLIDE 38

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Joint Constraint Term

Prevent neighboring bones from separating

  • Constrain overlapping weight regions

38

Bone 1 Bone 2 Bone 3

slide-39
SLIDE 39

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Joint Constraint Term

Prevent neighboring bones from separating

  • Constrain overlapping weight regions

39

Bone 1 Bone 2 Bone 3

slide-40
SLIDE 40

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Identifying Overlapping Regions

Use the weight grid to find overlap

  • Identify overlap over the grid (including cell interior)
  • Multiply weight components to determine overlap
  • Constrain so that transformations map point to same

location

40

slide-41
SLIDE 41

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Joint Constraint Comparison

41

slide-42
SLIDE 42

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

T-Step: Optimization summary

Like rigid registration

  • Except multiple parts & joint constraints

Non-linear least squares optimization

  • Solving for a rotation matrix
  • Gauss-Newton algorithm
  • Solve by iteratively linearizing solution

Few variables  Fast performance

  • # variables = 6 x #bones
  • Typically 5~10 bones in our examples

42

slide-43
SLIDE 43

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

W-Step: Assign transformations

Assign transformations to grid Similar distance and smoothness terms

  • Distance: measures alignment for a given label (same as

before)

  • Smoothness: penalizes different labels for adjacent cells

(simpler than before)

Graph cuts for optimization  Good Performance

  • Only 1000~5000 grid cells (graph nodes) & 5~10 labels
  • Fast performance for graph cuts

43

slide-44
SLIDE 44

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Robot video (real-time recording)

44

Alignment Result Solved Weights 7 bones 1454 cells

slide-45
SLIDE 45

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Torso video (2x speed recording)

45

Alignment Result Solved Weights 7 bones 4890 cells

slide-46
SLIDE 46

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Interactive posing (real-time recording)

46

Interactive Posing Result Solved Weights (7 bones, 1598 cells)

slide-47
SLIDE 47

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Average performance statistics

47

Car Robot Walk Hand Bones 7 7 10 12 Corresp. 1200 1200 1000 1500 Vertices 5389 9377 4502 34342 Max Dist 20 40 20 30 Grid Res 60 65 50 40 Grid Cells 1107 1295 1014 814 Grid Points 2918 3366 2553 1884 Setup 0.185 sec 0.234 sec 0.136s ec 0.078 sec RANSAC 8.089 sec 20.001 sec 5.517 sec N/A Align 9.945 sec 19.644 sec 23.092 sec 49.918 sec Weight 6.135 sec 10.713 sec 10.497 sec 3.689 sec Total Time 24.355 sec 50.591 sec 39.242 sec 53.684 sec

slide-48
SLIDE 48

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Pros/Cons

Feature matching: Insensitive to initial pose

  • May fail to sample properly when too much missing data,

non-rigid motion

  • Hard assignment of transformations

48

Source Target Registration

slide-49
SLIDE 49

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Pros/Cons

Local Search: faster and more precise

  • Faster method: no “useless” transformations

corresponding to wrong feature matches

  • Can refine transformations until precise alignment is

achieved

  • Sensitive to initial pose
  • Topological issues with grid

We can combine the two methods

  • Initialize with first, refine with second
  • Also graph can be more robust than grid

49

slide-50
SLIDE 50

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Conclusions

We can simplify the problem for articulated shapes

  • Instead of searching for corresponding points, search for an

assignment of transformations

  • Explicitly sample a discrete set of transformations
  • Refine the transformations via local search
  • Optimize the assignment using graph cuts
  • No marker, template, segmentation information needed
  • Robust to occlusion & missing data

Thank you for listening!

50