VIDEO SIGNALS Segmentation WHAT IS SEGMENTATION WHAT IS - - PowerPoint PPT Presentation

VIDEO SIGNALS

Segmentation

WHAT IS SEGMENTATION WHAT IS SEGMENTATION

 Segmentation is a fundamental low level operation on  Segmentation is a fundamental low-level operation on

images.

 If an image is already partitioned into segments  If an image is already partitioned into segments,

where each segment is a “homogeneous” region, then a number of subsequent image processing tasks b i become easier.

 A homogeneous region refers to a group of connected

pixels in the image that share a common feature This pixels in the image that share a common feature. This feature could be brightness, color, texture, motion, etc.

DIFFERENT KINDS OF SEGMENTATION DIFFERENT KINDS OF SEGMENTATION

Brightness segmentation

Color segmentation

DIFFERENT KINDS OF SEGMENTATION DIFFERENT KINDS OF SEGMENTATION

Motion segmentation

DIFFERENT KINDS OF SEGMENTATION DIFFERENT KINDS OF SEGMENTATION

Segmentation based on texture g

LUMINANCE BASED SEGMENTATION LUMINANCE-BASED SEGMENTATION

ERROR PROBABILITY FOR THRESHOLDING ERROR PROBABILITY FOR THRESHOLDING

OPTIMAL SUPERVISED THRESHOLDING OPTIMAL SUPERVISED THRESHOLDING

UNSUPERVISED THRESHOLDING UNSUPERVISED THRESHOLDING

UNSUPERVISED THRESHOLDING UNSUPERVISED THRESHOLDING

CHROMA KEYING CHROMA KEYING

C l i f l f i l i t ti 3 d 1 d

 Color is more powerful for pixel-wise segmentation: 3-d vs. 1-d

space

 Take picture in front of a blue screen (or green or orange)  Take picture in front of a blue screen (or green, or orange)

SOFT CHROMA KEYING SOFT CHROMA KEYING

LANDSAT IMAGE PROCESSING LANDSAT IMAGE PROCESSING

MULTIDIMENSIONAL MAP DETECTOR MULTIDIMENSIONAL MAP DETECTOR

L b l t g i i

 Label categories in

training set by hand

 Subdivide n dimensional  Subdivide n-dimensional

space into small bins

 Count frequency of  Count frequency of

• ccurrence for each bin

and class in training set g

 For test data: identify

bin, detect the more , probable category

MAP DETECTOR IN RGB SPACE MAP DETECTOR IN RGB-SPACE

LINEAR DISCRIMINANT FUNCTION LINEAR DISCRIMINANT FUNCTION

SELF SUPERVISED ROAD DETECTION SELF-SUPERVISED ROAD DETECTION

REGIONS VS BOUNDARIES REGIONS VS. BOUNDARIES

B d d t ti i th d l l f i t ti

 Boundary detection is the dual goal of image segmentation.  If the boundaries between segments are specified then it is

equivalent to identifying the individual segments q y g g themselves. BUT BUT In the process of image segmentation one obtains

 In the process of image segmentation, one obtains

regionwise information regarding the individual segments.

 This information can then be subsequently used to classify the

i di id l t individual segments.

 Detection of the boundaries between segments does not

automatically yield regionwise information about the individual segments.

 So, further image analysis is necessary before any segment-based

classification can be attempted.

WHY STATISTICAL METHODS? WHY STATISTICAL METHODS?

 They involve image features that are simple to

interpret by using a model.

 They also involve features that are easy to

compute from a given image compute from a given image

 They use merging methods that are firmly

rooted in statistical/mathematical inference.

WHAT ARE BOUNDARIES? WHAT ARE BOUNDARIES?

THE MATHEMATICAL PROBLEM THE MATHEMATICAL PROBLEM

 If we define  If we define

Ω={(m, , n): 1≤ m ≤ M and 1 ≤ n ≤ N } the domain where the image is defined.

 For any given point the segmentation g(m n)  For any given point the segmentation g(m,n)

at that point take a value from a set Γ. For l Γ {ξ ξ 0 1} f bi example Γ ={ξ: ξ= 0 or 1} for a binary

• segmentation. Γ ={ξ: ξ= 1,2,3,…k} for a

multiclass segmentation.

BOUNDARY SEGMENTATION BOUNDARY SEGMENTATION

 Of course, g could also denote a boundary

image:

 g(m,n) = 1 to denote the presence of a boundary.  While g(m n) = 0 to denote the absence  While g(m,n) = 0 to denote the absence.

GAUSSIAN STATISTICS GAUSSIAN STATISTICS

M th t f i bilit i th i l th t Measures the amount of variability in the pixels that comprise f1 along the (p,q) direction.

• If T is very small then that implies that f1 has little or no

variability along the (0,1)th (horizontal) direction.

• Computation of statistics is straightforward, as is merely

a quadratic operation.

FOURIER STATISTICS FOURIER STATISTICS

COVARIANCE STATISTICS COVARIANCE STATISTICS

LABEL STATISTICS LABEL STATISTICS

FISHER COLOR DISTANCE FISHER COLOR DISTANCE

VEHICLE TRACKING USING MAP

FISHER COLOR SEGMENTATION FISHER COLOR SEGMENTATION

A segmentation that yields all segments that do not contain g y g the color green.

TRACKING AN OBJECT OF INTEREST TRACKING AN OBJECT OF INTEREST

Tracking of a human hearth from frame to frame using Tracking of a human hearth from frame to frame using elastic deformation model

ILLUSORY BOUNDARY TRACKING ILLUSORY BOUNDARY TRACKING

Segmentation using texture phase in EdgeFlow Algorithm

SEGMENTATION USING TEXTURE ENERGY SEGMENTATION USING TEXTURE ENERGY

Segmentation using color and texture energy Segmentation using color and texture energy

MOTION FIELD SEGMENTATION MOTION FIELD SEGMENTATION

2D dense motion field from the second frame to the first and resulting segmentation

AN EXAMPLE OF CAR TRACKING AN EXAMPLE OF CAR TRACKING

• The vehicle is described by three parameters (Vb,Vl,Vw)

corresponding to the bottom edges, left edges and width of the p g g , g square.

• Vehicles seldom tend to be too big or small, and so depending on

the distance of the vehicle from the camera, in is possible to expect the width of the vehicle to be within a certain range : Wmin and Wmax

PARAMETER ESTIMATION PARAMETER ESTIMATION

 Let (vb,vl,vw) denote a specific hypothesis of the

b l w

unknown vehicle parameters (Vb,Vl,Vw): the merit of this hypothesis is decided by the likelyhood. Th it f thi h th i i d id d b th l

 The merit of this hypothesis is decided by the color

difference between pixels that are inside the square (i.e. pixels that are hypothesized to be the square (i.e. pixels that are hypothesized to be the vehicle) and pixels that are outside (pixels that are in the immediate background).

 The color difference evaluator is the Fisher

distance:

FISHER DISTANCE FISHER DISTANCE

 1 ad K1 are the mean and covariance of the pixels that  1 ad K1 are the mean and covariance of the pixels that

are inside the hypothesized square while 2 ad K2 are the mean and covariance of the pixels that are immediately surrounding the hypothesized square surrounding the hypothesized square.

 Hypotheses corresponding to a large color difference

between pixels inside and immediately surrounding the between pixels inside and immediately surrounding the square have more merit (and hence a higher probability of

• ccurrence) than those with smaller color difference.

 An optimal estimate of these parameters is the one that

maximizes the product of the prior and likelihood probabilities: the so-called maximum a pos posteriori eriori (MAP) probabilities: the so called maximum a pos posteriori eriori (MAP) estimate.

AERIAL IMAGE SEGMENTATION AERIAL IMAGE SEGMENTATION

Segmentation of an aerial image, a rural crop field area, using the texture-based maximum likelihood procedure

CHOOSING HOMOGENEOUS SEGMENTS FOR AERIAL IMAGES IMAGES

 The human operator examines the aerial image and chooses a

collection of polygons corresponding to various homogeneous segments of the image. g g

 By use of the pixels with these polygons as a training sample, a

statistical segmentation of the aerial image is effected;

 The segmentation procedure used for this map updating  The segmentation procedure used for this map updating

application is based on the Gaussian statistics.

 For each homogeneous polygonal region selected in the aerial

image by the human operator the Gaussian statistics for that image by the human operator, the Gaussian statistics for that polygon are automatically computed.



With these statistics, a model of probable variation in the ‘pixels' intensities within the polygon is subsequently created intensities within the polygon is subsequently created.

SEGMENTATION FOR OBJECT ORIENTED ENCODING SEGMENTATION FOR OBJECT-ORIENTED ENCODING Gi i i fi di id d i 8 8 bl k f

 Given an image is first divided into 8×8 blocks of

pixels.

 FFT is applied to each block (Fourier statistics).  If the pixels f1 within a single block have little or no

p

1

g variation then Ff1(0,0) will have a very large value.

 IF the blocks contain a vertical edge  IF the blocks contain a vertical edge

then will have a large value…

FOURIER DECOMPOSITION FOURIER DECOMPOSITION

If g denotes the collection of unknown block labels, g , then an estimate of g from f would correspond to an

• bject based segmentation of f.

RESULTS RESULTS

TEXTURE SEGMENTATION TEXTURE SEGMENTATION

 Texture features have been used in diverse applications

Texture features have been used in diverse applications such as satellite and aerial image analysis, medical image analysis for the detection of abnormalities, and more recently in image retrieval using texture as a descriptor recently, in image retrieval, using texture as a descriptor.

 In texture classification and segmentation, the objective is

to partition the given image into a set of homogeneous to partition the given image into a set of homogeneous textured regions.

 Aerial images are excellent examples of textured regions

where different areas such as water, sand, vegetation, etc. have distinct texture signatures.

 In many other cases such as in the classification of tissues  In many other cases, such as in the classification of tissues

in the magnetic resonance images of the brain, homogeneity is not that well defined.

WAVELET MODULATED WINDOW WAVELET MODULATED WINDOW

 An analytic wavelet can be constructed with a  An analytic wavelet can be constructed with a

frequency modulation of a real and symmetric window g g and it is a sort of a localized Fourier window g g and it is a sort of a localized Fourier Transform.

ONE DIMENSIONAL GABOR FUNCTION ONE DIMENSIONAL GABOR FUNCTION

GABOR FUNCTIONS DECOMPOSITION GABOR FUNCTIONS DECOMPOSITION

LOCAL BINARY PATTERN LOCAL BINARY PATTERN

A th l d t d t t d i t b t t

 Another commonly adopted texture descriptor robust to

luminance and contrast variations in the Local Binary Pattern (LBP) For every pixel a binary code is extracted comparing its (LBP). For every pixel a binary code is extracted comparing its value with the neighborhood. This operation is performed over the whole set of image pixels.

 The image (or a portion of it) is then converted into a matrix of

the same size where every element is the LBP value.

LOCAL BINARY PATTERN LOCAL BINARY PATTERN

 Different types of considered neighborhood:

Two different segmentation results shown above are the result of two different choices for the scale parameter in the algorithm.