In the name of Allah In the name of Allah the compassionate, the - - PowerPoint PPT Presentation

In the name of Allah In the name of Allah

the compassionate, the merciful

Digital Video Processing

• S. Kasaei
• S. Kasaei

Room: CE 307 Department of Computer Engineering Department of Computer Engineering Sharif University of Technology

E-Mail: skasaei@sharif.edu Web Page: http://sharif edu/ skasaei Web Page: http://sharif.edu/~skasaei http://ipl.ce.sharif.edu

Chapter 3 Chapter 3

Video Sampling Video Sampling

Vid S mplin M in C n rn Video Sampling Main Concerns

• 1. What are the necessary sampling

at a e t e ecessa y sa p g frequencies in the spatial & temporal directions?

• 2. Given an overall sampling rate (i.e.,

product of the horizontal, vertical, & temporal sampling rates), how do we sample in the 3-D space to obtain the best representation? best representation?

• 3. How can we avoid aliasing?

Kasaei 5

Video Sampling (A Bri f Di i n) (A Brief Discussion)

Review of Nyquist sampling theorem in 1-D Review of Nyquist sampling theorem in 1 D Extension to multi-dimensions Prefiltering in video cameras

g

Interpolation filtering in video displays

Kasaei 6

Nyquist Sampling Theorem in 1 D in 1-D

Given a band-limited signal with maximum

g frequency fmax, it can be sampled with a sampling rate fs>=2 fmax.

The original continuous signal can be exactly The original continuous signal can be exactly

reconstructed (interpolated) from the samples, by using an ideal low pass filter with cut-off frequency at fs /2. at fs /2.

Practical interpolation filters: replication

h

(sample-and-hold, 0th order), linear interpolation (1st order), cubic-spline (2nd

• rder).

Kasaei 7

)

Nyquist Sampling Theorem in 1 D in 1-D

Given the maximally feasible sampling rate fs, Given the maximally feasible sampling rate fs,

the original signal should be bandlimited to fs /2, to avoid aliasing.

The desired prefilter is an ideal low-pass filter with

cut-off frequency at fs /2.

Prefilter design: Trade-off between aliasing &

loss of high frequency content. g q y

Kasaei 8

E t n i n t M lti Dim n i n Extension to Multi-Dimensions

If the sampling grid is aligned in each If the sampling grid is aligned in each

dimension (rectangular in 2-D), and one performs sampling in each dimension t l th t i i t i htf d separately, the extension is straightforward:

Requirement: f s,i <= f max,i /2. Interpolation/prefilter: ideal low pass in each Interpolation/prefilter: ideal low-pass in each

dimension.

Kasaei 9

B i f L tti Th r Basics of Lattice Theory

A lattice, , in the real K-D space, ,is

k

R

Λ

A lattice, , in the real K D space, ,is

the set of all possible vectors that can be represented as integer-weighted

R

Λ

combinations of a set of K linearly independent basis vectors, that is:

1

| ,

K k k k k k

n n

=

  Λ = ∈ = ∀ ∈    

∑

x x v R Z

with generating matrix:

[ ] [ ]

V

k

 

Kasaei 10

[ ] [ ]

1 2

, , ,

k

= L V v v v

Basis Vectors Rectangular Hexagonal g Lattice g Lattice

Reciprocals

Lattice Points Voronoi Cell

Kasaei 11

B i f L tti Th r Basics of Lattice Theory

One can find more than one basis or One can find more than one basis or

generating matrix that can generate the same lattice.

Given a lattice, one can find a unit cell

, such that its translations to all lattice points form a tiling of the entire space.

Kasaei 12

L tti Unit C ll Lattice Unit Cells

Unit Cells

Kasaei 13

(in spatial domain)

L tti Unit C ll Lattice Unit Cells

Unit cells are of two types: fundamental Unit cells are of two types: fundamental

parallelepiped & Voronoi cell.

Th f d t l ll l i d

There are many fundamental parallelepipeds

associated with a lattice (because of the nonuniqueness of the generating matrix).

The volume of the unit cell is unique. A hexagonal lattice is more efficient than a

rectangular lattice (as it requires a lower li d it t bt i li f

Kasaei 14

sampling density to obtain an alias-free sampling).

V r n i C ll D t rmin ti n Voronoi Cell Determination

Determination of Voronoi cell:

Draw a straight line between the

• rigin & each one of the closest

l tti i t nonzero lattice points.

Draw a perpendicular line that is

the half way between the 2 points.

This line is the equidistance line

q between the origin & this lattice point.

Kasaei 15

R ipr l L tti Reciprocal Lattice

Given a lattice, its reciprocal lattice, , is defined

*

Λ

, p , , as a lattice that its basis vector is orthonormal to that of the lattice.

Λ

• r =I

The denser the lattice the sparser its reciprocal

[ ]

1

( )

T −

  =   U V

T

    V

[ ]

U

The denser the lattice, the sparser its reciprocal. A generalized Nyquist sampling theory exists,

g yq p g y , which governs the necessary density & structure

• f the sampling lattice for a given signal

spectrum

Kasaei 16

spectrum.

S mplin r L tti Sampling over Lattices

Fourier transform of a sampled signal over

• u e t a s o
• a sa

p ed s g a o e a lattice is called the sample-space Fourier transform (SSFT).

SSFT reduces to DTFT when the lattice is a

hypercube hypercube.

That is when [V] is a K-D identity matrix.

SSFT is periodic with a periodicity matrix

[U].

Kasaei 17

S mplin r L tti Sampling over Lattices

To avoid aliasing, the sampling lattice To avoid aliasing, the sampling lattice

must be designed so that the Voronoi cells

• f its reciprocal lattice completely cover

the signal spectrum.

To minimize the sampling density it To minimize the sampling density, it

should cover the signal spectrum as tightly as possible. tightly as possible.

Kasaei 18

S mplin r L tti Sampling over Lattices

Most real-world signals are symmetric in Most real world signals are symmetric in

frequency contents (spherical support).

Interlaced scan uses a non-rectangular

lattice in the vertical-temporal plane. p p

Kasaei 19

Input Signals Sampled Signals

Kasaei 20

S mplin Effi i n Sampling Efficiency

( ) d Λ

Sampling density: , # of lattice points

Kasaei 21

( ) d Λ

( ) ρ Λ

p g y , p Sampling efficiency:

S mplin f Vid Si n l Sampling of Video Signals

Most motion picture cameras sample a scene in

p p the temporal direction.

Store a sequence of analog frames on a film.

Most TV cameras capture a video sequence by

sampling it in temporal & vertical directions. sampling it in temporal & vertical directions.

1-D raster scan.

To obtain a full digital video, one should:

Sample analog frames in 2-D. Sample analog raster scan in 1-D.

Kasaei 22

p g

Acquire discrete video frames directly using a digital camera

(by sampling a scene in 3-D).

R q ir d S mplin R t Required Sampling Rates

Sampling frequency (frame rate & line rate): Sampling frequency (frame rate & line rate):

Frequency content of the underlying signal. Visual thresholds in terms of the spatial &

temporal cut-off frequencies.

Capture & display device characteristics.

Aff d bl i t & t i i

Affordable processing, storage, & transmission

costs.

Kasaei 23

Progressive S Interlaced Scan Sampling Scan Sampling Lattice Sampling Lattice

Nearest Aliasing Components

Kasaei 24

S mplin Vid in 2 D Sampling Video in 2-D

• 1. The same 2-D sampling density.

e sa e sa p g de s ty

• 2. The same 2-D nearest aliasing.
• 3. Different nearest aliasing along the

temporal frequency axis.

• Less flickering for interlaced.
• 4. Different mixed aliases.
• Nearest off-axis alias component.

5

For a signal with isotropic spectral

Kasaei 25

• 5. For a signal with isotropic spectral

support, the interlaced scan is more efficient.

Sampling Lattice Generating Matrix for Sampling Lattice Lattice Reciprocal Lattice

Kasaei 26

Filt rin Op r ti n Filtering Operations

How practical cameras & display devices

• p act ca ca

e as & d sp ay de ces accomplish the required prefiltering & reconstruction filters in a crude way.

How the HVS partially accomplishes the

required interpolation task.

Camera apertures consists of:

Temporal aperture. Temporal aperture. Spatial aperture. Combined aperture.

Kasaei 27

C m r Ap rt r Camera Apertures

Temporal aperture: Temporal aperture:

Intensity values read out at any frame instant

are not the sensed values at that time.

Rather they are the average of the sensed signal over

a certain time interval, known as exposure time.

Camera is applying a prefilter in the temporal domain, called temporal aperture function.

Modeled by a low pass filter:

) (t h ) (

, t

h

t p

Kasaei 28

C m r Ap rt r Camera Apertures

Spatial aperture: Spatial aperture:

Intensity value read out at any pixel is not the

• ptical signal at that point alone.

Rather it is a weighted integration of the signal in a

small window surrounding it, called aperture.

Camera is applying a prefilter in the spatial domain, called spatial aperture function.

Modeled by a circularly symmetric Gaussian

function:

) , (

, ,

y x h

y x p

Kasaei 29

C m r Ap rt r Camera Apertures

Combined aperture: Combined aperture:

Overall camera aperture function or prefilter

is: Wi h f

) , ( ) ( ) , , (

, , ,

y x h t h t y x h

y x p t p p

=

With frequency response:

) , ( ) ( ) , , (

, , , y x y x p t t p t y x p

f f H f H f f f H =

, , , y y p p y p

Kasaei 30

C Camera Aperture Function

Kasaei 31

Vid Di pl Video Display

The display device presents the analog or The display device presents the analog or

digital video on the screen to create the sensation of continuously varying signal in b th ti & both time & space.

With CRT, three electronic beams strike red,

green & blue phosphors with the desired green, & blue phosphors with the desired intensity at each pixel location.

No explicit interpolation filters are used.

p p

Spatial filtering is determined by the size of

scanning beam.

Temporal filtering is determined by the decaying

Kasaei 32

Temporal filtering is determined by the decaying

time of phosphors.

Vid Di pl Video Display

The eye performs the interpolation task: The eye performs the interpolation task:

Fuses discrete frames and pixels as continuously

varying (if the temporal and spatial sampling rates are sufficiently high) are sufficiently high).

Kasaei 33

The End