[PPT] - Motion Estimation for Video Coding Motion-Compensated Prediction PowerPoint Presentation

SLIDE 1

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 1

Motion Estimation for Video Coding

Motion-Compensated Prediction
Bit Allocation
Motion Models
Motion Estimation
Efficiency of Motion Compensation Techniques

SLIDE 2

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 2

Intra-Frame Decoder Motion- Compensated Predictor Control Data DCT Coefficients Motion Data Intra/Inter Coder Control Decoder Motion Estimator Intra-Frame DCT Coder

]

, , [ t y x s ] , , [ t y x u ] , , [ ' t y x s ] , , [ ' t y x u ] , , [ ˆ t y x s

Hybrid Video Encoder

SLIDE 3

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 3

Motion-Compensated Prediction

Prediction for the luminance signal s[x,y,t] within the moving object:

Moving

bject

Displaced

bject

time t

x y

Previous frame Current frame Stationary background

t         

y x

d d ) , , ( ] , , [ ˆ t t d y d x s t y x s

y x

     

SLIDE 4

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 4

Motion-Compensated Prediction: Example

Referenced blocks in frame 1 Difference between motion- compensated prediction and current frame u[x,y,t] Frame 1 s[x,y,t-1] (previous) Frame 2 with displacement vectors Accuracy of Motion Vectors Frame 2 s[x,y,t] (current) Partition of frame 2 into blocks (schematic) Size of Blocks

SLIDE 5

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 5

y x y y x x

d d y x y x y x f y y d y x f x x d , , , , ) , , ( ), , , ( b a b a          

Motion in 3-D space corresponds to displacements in the image plane
Motion compensation in the image plane is conducted to provide a

prediction signal for efficient video compression

Efficient motion-compensated prediction often uses side information to

transmit the displacements

Displacements must be efficiently represented for video compression
Motion models relate 3-D motion to displacements assuming

reasonable restrictions of the motion and objects in the 3-D world

Motion Models

Motion Model

: location in previous image : location in current image : vector of motion coefficients : displacements

SLIDE 6

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 6

Decoded video signal is given as

Representation of Video Signal

] , , [ ] , , [ ˆ ] , , [ t y x u t y x s t y x s    

Motion-compensated prediction signal

) , , ( ] , , [ ˆ t t d y d x s t y x s

y x

     

) , ( ), , (

1 1

y x b d y x a d

i N i i y i N i i x

     

 

   

Prediction residual signal

) , ( ) , (

1

y x c y x u

j M j j 



 

   ,...) ( ,...), ( ), , ( b a f Rm    b a b a

Transmitted motion parameters Transmitted residual parameters

,...) ( ), ( c f Ru   c c

SLIDE 7

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 7

Optimum trade-off:
Displacement error variance can be influenced via
Block-size, quantization of motion parameters
Choice of motion model

Bit-rate Displacement error variance D Prediction error rate Ru Motion vector rate Rm

Rate-Constrained Motion Estimation

u m u m

R R R dR dD dR dD    ,

SLIDE 8

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 8

Lagrangian Optimization in Video Coding

Distortion after motion compensation Lagrange parameter Number of bits for motion vector

Rate-Constrained Motion Estimation [Sullivan, Baker 1991]:
Rate-Constrained Mode Decision [Wiegand, et al. 1996]:
A number of interactions are often neglected
Temporal dependency due to DPCM loop
Spatial dependency of coding decisions
Conditional entropy coding

Distortion after reconstruction Lagrange parameter Number of bits for coding mode

m m m

R D   min R D   min

SLIDE 9

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 9

Translational motion model
4-Parameter motion model: translation, zoom (isotropic Scaling),

rotation in image plane

Affine motion model:
Parabolic motion model

Motion Models

, b d a d

y x

  y a x a b d y a x a a d

y x 1 2 2 1

      y b x b b d y a x a a d

y x 2 1 2 1

      xy b y b x b y b x b b d xy a y a x a y a x a a d

x x 5 2 6 2 2 3 1 5 2 6 2 2 3 1

            ) , ( ), , (

1 1

y x b d y x a d

i N i i y i N i i x

     

 

   

SLIDE 10

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 10

Impact of the Affine Parameters

150
100
50

50 100 150

150
100
50

50 100 150 x y

150
100
50

50 100 150

150
100
50

50 100 150 x y

150
100
50

50 100 150

150
100
50

50 100 150 x y

150
100
50

50 100 150

150
100
50

50 100 150 x y

translation scaling sheering

  a dx   x a dx

1

  y a dx

3

SLIDE 11

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 11

150
100
50

50 100 150

150
100
50

50 100 150 x y

150
100
50

50 100 150

150
100
50

50 100 150 x y

150
100
50

50 100 150

150
100
50

50 100 150 x y

150
100
50

50 100 150

150
100
50

50 100 150 x y

Impact of the Parabolic Parameters

2 2x

a dx 

2 6y

a dx  xy a dx

5



SLIDE 12

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 12

Differential Motion Estimation

Assume small displacements dx,dy:
Aperture problem: several observations required
Inaccurate for displacements > 0.5 pel

 multigrid methods, iteration

Minimize

Displace frame difference Horizontal and vertical gradient of image signal S

y x y x

d y t t y x s d x t t y x s t t y x s t y x s d d t y x s t y x s t y x u                      ) , , ( ) , , ( ) , , ( ) , , ( ) , , , , ( ˆ ) , , ( ) , , (

) , , ( min

1 1 2

t y x u

By y Bx x



 

SLIDE 13

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 13

Gradient-Based Affine Refinement

Displacement vector field is represented as
Combination

yields a system of linear equations:

System can be solved using, e.g., pseudo-inverse,

by minimizing

y b x b b y y a x a a x

3 2 1 3 2 1

       

) ( ) ( ) , , ( ) , , ( ) , , (

3 2 1 3 2 1

y b x b b y s y a x a a x s t t y x s t y x s t y x u                 



  By y Bx x b b b a a a

t y x u

1 1 2 , , , , ,

) , , ( min arg

3 2 1 3 2 1

                                                 

3 2 1 3 2 1

, , , , , , b b b a a a y y s x y s y s y x s x x s x s s s u

SLIDE 14

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 14

Subdivide current frame

into blocks.

Find one displacement

vector for each block.

Within a search range, find

a “best match” that minimizes an error measure.

Intelligent search strategies

can reduce computation.

search range in previous frame block of current frame

Sk1

Sk

Block-matching Algorithm

SLIDE 15

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 15

Previous Frame Current Frame

Block of pixels is selected as a measurement window Measurement window is compared with a shifted block

f pixels in the other image,

to determine the best match

Block-matching Algorithm

SLIDE 16

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 16

Block-matching Algorithm

. . . process repeated for another block.

Previous Frame Current Frame

SLIDE 17

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 17

Error Measures for Block-matching

Mean squared error (sum of squared errors)
Sum of absolute differences
Approximately same performance

 SAD less complex for some architectures

2 1 1

)] , , ( ) , , ( [ ) , ( t t d y d x s t y x s d d SSD

y By y Bx x x y x

      

 

| ) , , ( ) , , ( | ) , (

1 1

t t d y d x s t y x s d d SAD

y By y Bx x x y x

      

 

SLIDE 18

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 18

Full search

All possible displacements within the search range

are compared.

Computationally expensive
Highly regular, parallelizable

Block-matching: Search Strategies

dx dy

SLIDE 19

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 19

Complexity of block-matching: evaluation of complex error measure for many candidates

Speedup of Block-matching

Combine both approaches: Choose starting point and search order that maximizes likelihood for efficient approximations, early terminations and excluding of candidates Reduce complexity

Approximations
Early terminations
Exclude candidates

Reduce number

Cover likely search areas
Unequal steps between

searched candidates

SLIDE 20

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 20

Approximations

Stop search, if match is “good enough”

(SSD, SAD < threshold or J=D+R is small enough)

Practical method in video conferencing for static

background: test zero-vector first and stop search if match is good enough static background

SLIDE 21

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 21

Early Termination

Partial distortion measure:
Previously computed minimum:
Early termination without loss:
Early termination with possible loss, but higher speedup:

p y K y B x x y x K

t t d y d x s t y x s d d D

x

)] , , ( ) , , ( [ ) , (

1 1

      

  y y x K

B N K d d D ... ), , (  ฀  Jmin

min

) , ( ) , ( if Stop, J d d R d d D

y x m y x K

  

min

) ( ) , ( ) , ( if Stop, J K d d R d d D

y x m y x K

     

2 / 1

) / ( ) ( e.g.,

y

B K K  

SLIDE 22

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 22

Block Comparison Speed-Ups

Triangle inequality for samples in block B (here SAD)
Strategy:
1. Compute partial sums for blocks in current and previous frame
2. Compare blocks based on partial sums
3. Omit full block comparison, if partial sums indicate worse error

measure than previous best result

Block partitioning
Choose blocks to be nested (4x4, 8x8, 16x16, …) – efficient

for modern variable block size video codecs

 

 

  

n B k k B k k

n

S S S S | | | |

1 1

  

 

n n n

B B B ,

 

 

  

B k k B k k

S S S S | | | | | |

1 1 n

B

SLIDE 23

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 23

Blockmatching: Search Order I

Iterative comparison of error

measure values at 5 neighboring points

Logarithmic refinement of the

search pattern if – best match is in the center of the 5-point pattern – center of search pattern touches the border of the search range

2D logarithmic search [Jain + Jain, 1981]

SLIDE 24

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 24

Diamond search [Li, Zeng, Liou, 1994] [Zhu, Ma, 1997]

dx d y

Start with large diamond pattern at (0,0) If best match lies in the center

f large diamond,

proceed with small diamond If best match does not lie in the center of large diamond, center large diamond pattern at new best match

Blockmatching: Search Order II

SLIDE 25

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 25

Hierarchical blockmatching

current previous fra Block matching Displacement vector field

SLIDE 26

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 26

Sub-pel Translational Motion

Motion vectors are often not restricted to only point into

the integer-pel grid of the reference frame

Typical sub-pel accuracies: half-pel and quarter-pel
Sub-pel positions are often estimated by „refinement“
1 •

1 • 1

1
1

d d

x y

1
1
1
1
2•

2

2 •

2

2
2
2•

2

SLIDE 27

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 27

Sub-pel Motion Compensation

Sub-pel positions are obtained via interpolation
Example: half-pixel accurate displacements
•
•
•
•
•
•
• • •
• • •
• • •
• • • •
•
4.5

4.5

x y

d d             

SLIDE 28

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 28

Bi-linear Interpolation

Interpolated Pixel Value brightness

SLIDE 29

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 29

History of Motion Compensation

Intraframe coding: only spatial correlation exploited

 DCT [Ahmed, Natarajan, Rao 1974], JPEG [1992]

Conditional replenishment

 H.120 [1984] (DPCM, scalar quantization)

Frame difference coding

 H.120 Version 2 [1988]

Motion compensation: integer-pel accurate displacements

 H.261 [1991]

Half-pel accurate motion compensation

 MPEG-1 [1993], MPEG-2/H.262 [1994]

Variable block-size (16x16 & 8x8) motion compensation

 H.263 [1996], MPEG-4 [1999]

Variable block-size (16x16 – 4x4) and multi-frame

motion compensation  H.264/MPEG-4 AVC [2003]

Complexity increase

SLIDE 30

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 30

Milestones in Video Coding

100 200 300 28 30 32 34 36 38 40 Rate [kbit/s] PSNR [dB]

Variable block size (16x16 – 8x8) (H.263, 1996) + quarter-pel motion compensation (MPEG-4, 1998) Variable block size (16x16 – 4x4) + quarter-pel + multi-frame motion compensation (H.264/AVC, 2003) Intraframe DCT coding (JPEG, 1990)

Foreman 10 Hz, QCIF 100 frames

Frame Difference coding (H.120 1988) Conditional Replenishment (H.120)

Half-pel motion compensation (MPEG-1 1993 MPEG-2 1994) Integer-pel motion compensation (H.261, 1991)

Bit-rate Reduction: 75%

35

SLIDE 31

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 31

Milestones in Video Coding

100 200 300 28 30 32 34 36 38 40 Rate [kbit/s] PSNR [dB]

Integer-pel motion compensation (H.261, 1991) Variable block size (16x16 – 8x8) (H.263, 1996) + quarter-pel motion compensation (MPEG-4, 1998) Variable block size (16x16 – 4x4) + quarter-pel + multi-frame motion compensation (H.264/AVC, 2003) Intraframe DCT coding (JPEG, 1990)

Foreman 10 Hz, QCIF 100 frames

Half-pel motion compensation (MPEG-1 1993 MPEG-2 1994)

Frame Difference coding (H.120 1988) Conditional Replenishment (H.120)

Visual Comparison

SLIDE 32

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 32

Visual Comparison

Foreman, QCIF, 10 Hz, 100 kbit/s JPEG H.264/AVC

SLIDE 33

T. Wiegand / B. Girod: EE398A Image and Video Compression

Motion estimation no. 33

Summary

Video coding as a hybrid of motion compensation and prediction

residual coding

Motion models can represent various kinds of motions
Lagrangian bit-allocation rules specify constant slope allocation to

motion coefficients and prediction error

In practice: affine or 8-parameter model for camera motion,

translational model for small blocks

Differential methods calculate displacement from spatial and

temporal differences in the image signal -

Block matching computes error measure for candidate

displacements and finds best match

Speed up block matching by fast search methods, approximations,

early terminations and clever application of triangle inequality

Hybrid video coding has been drastically improved by enhanced