Image Segmentation Machine Learning Study Group Presented by - - PowerPoint PPT Presentation

Image Segmentation

Machine Learning Study Group

Presented by Yaochen Xie Jan 25, 2018

Outline

• Overview
• Three Levels of Segmentation
• Basic Segmentation
• Semantic Segmentation (FCN, DeepLab)
• Instance Segmentation (Mask-RCNN)

Image Segmentation

• Process of partitioning a digital

image into multiple segments.

• Typically used to locate objects and

boundaries.

• More precisely, image segmentation

is the process of assigning a label to every pixel in an image such that pixels with the same label share certain visual characteristics.

http://imagej.net/Segmentation

Image Segmentation

• Definition:

Image segmentation partitions an image into regions called segments

• Image segmentation creates

segments of connected pixels by analyzing some similarity criteria: Intensity, color, texture, histogram, features…

Gestalt Theory

• Gestalt: whole or group
• The whole is greater than the sum of its parts
• Relationships between parts can yield new properties or

features

• Psychologists identified series of factors that predispose

set of elements to be grouped (by human visual system)

• Max Wertheimer (1880-1943)：

“I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of color. Do I have ”327”? No. I have sky, house, and trees.”

Image Segmentation

• Applications

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

(Mostly unsupervised learning)

• Automatically & Interactively
• Many approaches:
• Edge-based
• Graph-based
• Contour-based
• Energy-based
• ……

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Segmentation as a cluster

5 dimensions: 3 color channels & 2 space location

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Intelligent Scissors [Mortensen 95]

Approach answers a basic question - Q: how to find a path from seed to mouse that follows object boundary as closely as possible?

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Intelligent Scissors [Mortensen 95]

Basic Idea

• Define edge score for each pixel
• edge pixels have low cost
• Find lowest cost path from seed to mouse

Q: How to define cose? How to find the path? (Dijkstra)

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Intelligent Scissors [Mortensen 95]

How does really work? Treat the image as a graph:

Graph

• node for every pixel p


link between every adjacent

• pair of pixels p, q
• cost c for each link
• The link should follow the intensity edge:
• want intensity to change rapidly cross the link
• c related to difference of intensity cross the link

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Graph based segmentation

Felzenszwalb P F , Huttenlocher D P . Efficient Graph-Based Image Segmentation[J]. International Journal of Computer Vision, 2004, 59(2):167-181.

Basic Idea： Consider image as a graph, pixels as nodes and similarities as weights of edges. Cut edges with less weight in each iteration.

Source: Forsyth & Ponce

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Graph based segmentation

Break graph into segments To delete links that cross between segments (with low weight)

• Similar pixels should be in the same segment
• Dissimilar pixels should be in different segments

Source: Forsyth & Ponce

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Graph based segmentation

Measuring Affinity (weight):

Source: Forsyth & Ponce

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Graph based segmentation

Measuring Affinity (weight):

Source: Forsyth & Ponce

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Graph based segmentation

Measuring Affinity (weight):

Source: Forsyth & Ponce

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Graph based segmentation

Measuring Affinity (weight):

Source: Forsyth & Ponce

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Graph based segmentation

Steps: 1.Compute all the weights of edges. Put each node into separate components. 2.Sort edges by their weights. 3.In each iteration, for edge with the least weight: If its 2 ends belongs to different components and its weight is less than the threshold of both components, merge two ends. Set new threshold as min{weights}.

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Fully Convolutional Networks [CVPR 2015]

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Convolutionalization - why fully convolution?

Transforming fully connected layers into convolution layers enables a classification net to output a heat-map Fully Convolutional Networks [CVPR 2015]

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Upsampling (Deconvolution)

Convolution Deconvolution

Fully Convolutional Networks [CVPR 2015]

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Skip Architecture - combining WHAT and WHERE

Fully Convolutional Networks [CVPR 2015]

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• DeepLab [ICLR 2015]

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Atrous convolutions

Atrous convolution

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Atrous convolutions

500 x 500 16 x 16 Fully Convolutional Network DeepLab 67 x 67 514 x 514

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Conditional Random Field (CRF)

From output feature map From raw image

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Conditional Random Field (CRF)

Segmentation at Three Levels

❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

• Instance Segmentation