Support Vector Machines (SVMs). Semi-Supervised Learning. - - PowerPoint PPT Presentation

Maria-Florina Balcan

03/25/2015

• Support Vector Machines (SVMs).
• Semi-Supervised SVMs.
• Semi-Supervised Learning.

One of the most theoretically well motivated and practically most effective classification algorithms in machine learning.

Support Vector Machines (SVMs).

Directly motivated by Margins and Kernels!

Geometric Margin

Definition: The margin of example 𝑦 w.r.t. a linear sep. 𝑥 is the distance from 𝑦 to the plane 𝑥 ⋅ 𝑦 = 0.

𝑦1 w

Margin of example 𝑦1

𝑦2

Margin of example 𝑦2

If 𝑥 = 1, margin of x w.r.t. w is |𝑦 ⋅ 𝑥|.

WLOG homogeneous linear separators [w0 = 0].

+ + + +

• -
• 𝛿

𝛿

+

• w

Definition: The margin 𝛿 of a set of examples 𝑇 is the maximum 𝛿𝑥 over all linear separators 𝑥.

Geometric Margin

Definition: The margin 𝛿𝑥 of a set of examples 𝑇 wrt a linear separator 𝑥 is the smallest margin over points 𝑦 ∈ 𝑇. Definition: The margin of example 𝑦 w.r.t. a linear sep. 𝑥 is the distance from 𝑦 to the plane 𝑥 ⋅ 𝑦 = 0.

Both sample complexity and algorithmic implications.

Margin Important Theme in ML

Sample/Mistake Bound complexity:

• If large margin, # mistakes Peceptron makes

is small (independent on the dim of the space)!

• If large margin 𝛿 and if alg. produces a large

margin classifier, then amount of data needed depends only on R/𝛿 [Bartlett & Shawe-Taylor ’99].

+ + + +

• -
• 𝛿

𝛿

+

• w

Suggests searching for a large margin classifier… SVMs

Algorithmic Implications

Input: 𝛿, S={(x1, 𝑧1), …,(xm, 𝑧m)}; Output: w, a separator of margin 𝛿 over S

Support Vector Machines (SVMs)

Directly optimize for the maximum margin separator: SVMs

Find: some w where:

• w

2 = 1

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 𝛿

First, assume we know a lower bound on the margin 𝛿

+ + + +

• -
• 𝛿

𝛿

+

• w

Realizable case, where the data is linearly separable by margin 𝛿

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; Output: maximum margin separator over S

Support Vector Machines (SVMs)

Directly optimize for the maximum margin separator: SVMs

Find: some w and maximum 𝛿 where:

• w

2 = 1

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 𝛿

E.g., search for the best possible 𝛿

+ + + +

• -
• 𝛿

𝛿

+

• w

Support Vector Machines (SVMs)

Directly optimize for the maximum margin separator: SVMs

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; Maximize 𝛿 under the constraint:

• w

2 = 1

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 𝛿

+ + + +

• -
• 𝛿

𝛿

+

• w

Support Vector Machines (SVMs)

Directly optimize for the maximum margin separator: SVMs

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; Maximize 𝛿 under the constraint:

• w

2 = 1

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 𝛿

This is a constrained

• ptimization

problem.

• bjective

function constraints

• Famous example of constrained optimization: linear programming,

where objective fn is linear, constraints are linear (in)equalities

This constraint is non-linear.

Support Vector Machines (SVMs)

Directly optimize for the maximum margin separator: SVMs

+ + + +

• -
• 𝛿

𝛿

+

• w

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; Maximize 𝛿 under the constraint:

• w

2 = 1

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 𝛿

In fact, it’s even non-convex

𝑥1 𝑥2

𝑥1 + 𝑥2 2

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)};

Support Vector Machines (SVMs)

Directly optimize for the maximum margin separator: SVMs

Maximize 𝛿 under the constraint:

• w

2 = 1

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 𝛿

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; Minimize 𝑥′

2 under the constraint:

• For all i, 𝑧𝑗𝑥′ ⋅ 𝑦𝑗 ≥ 1

𝑥’ = 𝑥/𝛿, then max 𝛿 is equiv. to minimizing ||𝑥’||2 (since ||𝑥’||2 = 1/𝛿2). So, dividing both sides by 𝛿 and writing in terms of w’ we get:

+ + + +

• -
• 𝛿

𝛿

+

• w

+ + + +

• -
• +
• w’

𝑥’ ⋅ 𝑦 = −1 𝑥’ ⋅ 𝑦 = 1

Support Vector Machines (SVMs)

Directly optimize for the maximum margin separator: SVMs

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; argminw 𝑥

2 s.t.:

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 1

This is a constrained

• ptimization

problem.

• The objective is convex (quadratic)
• All constraints are linear
• Can solve efficiently (in poly time) using standard quadratic

programing (QP) software

Support Vector Machines (SVMs)

Question: what if data isn’t perfectly linearly separable?

𝑥

2 + 𝐷(# misclassifications)

Issue 1: now have two objectives

• maximize margin
• minimize # of misclassifications.

Ans 1: Let’s optimize their sum: minimize

NP-hard [Guruswami-Raghavendra’06]

where 𝐷 is some tradeoff constant. Issue 2: This is computationally hard (NP-hard).

[even if didn’t care about margin and minimized # mistakes]

+ + + +

• -
• +
• w

𝑥 ⋅ 𝑦 = −1 𝑥 ⋅ 𝑦 = 1

Support Vector Machines (SVMs)

Question: what if data isn’t perfectly linearly separable?

𝑥

2 + 𝐷(# misclassifications)

Issue 1: now have two objectives

• maximize margin
• minimize # of misclassifications.

Ans 1: Let’s optimize their sum: minimize

NP-hard [Guruswami-Raghavendra’06]

where 𝐷 is some tradeoff constant. Issue 2: This is computationally hard (NP-hard).

[even if didn’t care about margin and minimized # mistakes]

+ + + +

• -
• +
• w

𝑥 ⋅ 𝑦 = −1 𝑥 ⋅ 𝑦 = 1

Support Vector Machines (SVMs)

Question: what if data isn’t perfectly linearly separable?

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; argminw,𝜊1,…,𝜊𝑛 𝑥

2 + 𝐷 𝜊𝑗 𝑗

s.t.:

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 1 − 𝜊𝑗

Find 𝜊𝑗 ≥ 0 𝜊𝑗 are “slack variables”

Replace “# mistakes” with upper bound called “hinge loss”

+ + + +

• -
• +
• w

𝑥 ⋅ 𝑦 = −1 𝑥 ⋅ 𝑦 = 1

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; Minimize 𝑥′

2 under the constraint:

• For all i, 𝑧𝑗𝑥′ ⋅ 𝑦𝑗 ≥ 1

+ + + +

• -
• +
• w’

𝑥’ ⋅ 𝑦 = −1 𝑥’ ⋅ 𝑦 = 1

Support Vector Machines (SVMs)

Question: what if data isn’t perfectly linearly separable?

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; argminw,𝜊1,…,𝜊𝑛 𝑥

2 + 𝐷 𝜊𝑗 𝑗

s.t.:

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 1 − 𝜊𝑗

Find 𝜊𝑗 ≥ 0 𝜊𝑗 are “slack variables”

Replace “# mistakes” with upper bound called “hinge loss”

𝑚 𝑥, 𝑦, 𝑧 = max (0,1 − 𝑧 𝑥 ⋅ 𝑦)

+ + + +

• -
• +
• w

𝑥 ⋅ 𝑦 = −1 𝑥 ⋅ 𝑦 = 1

C controls the relative weighting between the twin goals of making the 𝑥

2 small (margin is

large) and ensuring that most examples have functional margin ≥ 1.

Support Vector Machines (SVMs)

Question: what if data isn’t perfectly linearly separable?

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; argminw,𝜊1,…,𝜊𝑛 𝑥

2 + 𝐷 𝜊𝑗 𝑗

s.t.:

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 1 − 𝜊𝑗

Find 𝜊𝑗 ≥ 0

Replace “# mistakes” with upper bound called “hinge loss”

𝑚 𝑥, 𝑦, 𝑧 = max (0,1 − 𝑧 𝑥 ⋅ 𝑦)

+ + + +

• -
• +
• w

𝑥 ⋅ 𝑦 = −1 𝑥 ⋅ 𝑦 = 1

𝑥

2 + 𝐷(# misclassifications)

Replace the number of mistakes with the hinge loss

Support Vector Machines (SVMs)

Question: what if data isn’t perfectly linearly separable?

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; argminw,𝜊1,…,𝜊𝑛 𝑥

2 + 𝐷 𝜊𝑗 𝑗

s.t.:

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 1 − 𝜊𝑗

Find 𝜊𝑗 ≥ 0

Replace “# mistakes” with upper bound called “hinge loss”

𝑚 𝑥, 𝑦, 𝑧 = max (0,1 − 𝑧 𝑥 ⋅ 𝑦)

+ + + +

• -
• +
• w

𝑥 ⋅ 𝑦 = −1 𝑥 ⋅ 𝑦 = 1

Total amount have to move the points to get them

• n the correct side of the lines 𝑥 ⋅ 𝑦 = +1/−1,

where the distance between the lines 𝑥 ⋅ 𝑦 = 0 and 𝑥 ⋅ 𝑦 = 1 counts as “1 unit”.

What if the data is far from being linearly separable?

Example: vs

No good linear separator in pixel representation.

SVM philosophy: “Use e a Ke Kernel”

Support Vector Machines (SVMs)

Input: S={(x1, 𝑧1), …,(xm, 𝑧m)}; argminw,𝜊1,…,𝜊𝑛 𝑥

2 + 𝐷 𝜊𝑗 𝑗

s.t.:

• For all i, 𝑧𝑗𝑥 ⋅ 𝑦𝑗 ≥ 1 − 𝜊𝑗

Which is equivalent to:

Find 𝜊𝑗 ≥ 0

Primal form

Input: S={(x1, y1), …,(xm, ym)}; argminα

1 2 yiyj αiαjxi ⋅ xj − αi i j i

s.t.:

• For all i,

Find 0 ≤ αi ≤ Ci

Lagrangian Dual

yiαi = 0

i

SVMs (Lagrangian Dual)

• Final classifier is: w = αiyixi

i

• The points xi for which αi ≠ 0

are called the “support vectors” Input: S={(x1, y1), …,(xm, ym)}; argminα

1 2 yiyj αiαjxi ⋅ xj − αi i j i

s.t.:

• For all i,

Find 0 ≤ αi ≤ Ci

yiαi = 0

i

+ + + +

• -
• +
• w

𝑥 ⋅ 𝑦 = −1 𝑥 ⋅ 𝑦 = 1

Kernelizing the Dual SVMs

• Final classifier is: w = αiyixi

i

• The points xi for which αi ≠ 0 are called the “support vectors”
• With a kernel, classify x using αiyiK(x, xi)

i

Input: S={(x1, y1), …,(xm, ym)}; argminα

1 2 yiyj αiαjxi ⋅ xj − αi i j i

s.t.:

• For all i,

Find 0 ≤ αi ≤ Ci

yiαi = 0

i

Replace xi ⋅ xj with K xi, xj .

One of the most theoretically well motivated and practically most effective classification algorithms in machine learning.

Support Vector Machines (SVMs).

Directly motivated by Margins and Kernels!

• The importance of margins in machine learning.
• The primal form of the SVM optimization problem

What you should know

• The dual form of the SVM optimization problem.
• Kernelizing SVM.
• Think about how it’s related to Regularized Logistic

Regression.

• Using Unlabeled Data and

Interaction for Learning

Modern (Partially) Supervised Machine Learning

Classic Paradigm Insufficient Nowadays

Modern applications: massive amounts of raw data. Only a tiny fraction can be annotated by human experts.

Billions of webpages Images Protein sequences

Expert Labeler

Semi-Supervised Learning

raw data

face not face

Labeled data

Classifier

Active Learning

face

O O O

Expert Labeler

raw data

Classifier

not face

Semi-Supervised Learning

Prominent paradigm in past 15 years in Machine Learning.

• Most applications have lots of unlabeled data, but

labeled data is rare or expensive:

• Web page, document classification
• Computer Vision
• Computational Biology,
• ….

Labeled Examples

Semi-Supervised Learning

Learning Algorithm Expert / Oracle Data Source Unlabeled examples Algorithm outputs a classifier Unlabeled examples

Sl={(x1, y1), …,(xml, yml)} Su={x1, …,xmu} drawn i.i.d from D xi drawn i.i.d from D, yi = c∗(xi) Goal: h has small error over D. errD h = Pr

x~ D(h x ≠ c∗(x))

Unlabeled data useful if we have beliefs not only about the form of the target, but also about its relationship with the underlying distribution.

Key Insight

Semi-supervised learning: no querying. Just have lots of additional unlabeled data. A bit puzzling since unclear what unlabeled data can do for you….

Combining Labeled and Unlabeled Data

• Several methods have been developed to try to use

unlabeled data to improve performance, e.g.:

– Transductive SVM [Joachims ’99] – Co-training [Blum & Mitchell ’98] – Graph-based methods [B&C01], [ZGL03]

Test of time awards at ICML!

Workshops [ICML ’03, ICML’ 05, …]

• Semi-Supervised Learning, MIT 2006
• O. Chapelle, B. Scholkopf and A. Zien (eds)

Books:

• Introduction to Semi-Supervised Learning,

Morgan & Claypool, 2009 Zhu & Goldberg

Example of “typical” assumption: Margins

• The separator goes through low density regions of

the space/large margin.

– assume we are looking for linear separator – belief: should exist one with large separation

+ + _ _ Labeled data only + + _ _ + + _ _ Transductive SVM SVM

Transductive Support Vector Machines

Optimize for the separator with large margin wrt labeled and unlabeled data. [Joachims ’99]

argminw w

2 s.t.:

• yi w ⋅ xi ≥ 1, for all i ∈ {1, … , ml}

Su={x1, …,xmu}

• yu

w ⋅ xu ≥ 1, for all u ∈ {1, … , mu}

• yu

∈ {−1, 1} for all u ∈ {1, … , mu}

+ + + +

• +
• w’

𝑥’ ⋅ 𝑦 = −1 𝑥’ ⋅ 𝑦 = 1

Input: Sl={(x1, y1), …,(xml, yml)} Find a labeling of the unlabeled sample and 𝑥 s.t. 𝑥 separates both labeled and unlabeled data with maximum margin.

Transductive Support Vector Machines

argminw w

2 + 𝐷 𝜊𝑗 𝑗

+ 𝐷 𝜊𝑣

𝑣

• yi w ⋅ xi ≥ 1-𝜊𝑗, for all i ∈ {1, … , ml}

Su={x1, …,xmu}

• yi

w ⋅ xu ≥ 1 − 𝜊𝑣 , for all u ∈ {1, … , mu}

• yu

∈ {−1, 1} for all u ∈ {1, … , mu}

+ + + +

• +
• w’

𝑥’ ⋅ 𝑦 = −1 𝑥’ ⋅ 𝑦 = 1

Input: Sl={(x1, y1), …,(xml, yml)} Find a labeling of the unlabeled sample and 𝑥 s.t. 𝑥 separates both labeled and unlabeled data with maximum margin.

Optimize for the separator with large margin wrt labeled and unlabeled data. [Joachims ’99]

Transductive Support Vector Machines

Optimize for the separator with large margin wrt labeled and unlabeled data.

argminw w

2 + 𝐷 𝜊𝑗 𝑗

+ 𝐷 𝜊𝑣

𝑣

• yi w ⋅ xi ≥ 1-𝜊𝑗, for all i ∈ {1, … , ml}

Su={x1, …,xmu}

• yi

w ⋅ xu ≥ 1 − 𝜊𝑣 , for all u ∈ {1, … , mu}

• yu

∈ {−1, 1} for all u ∈ {1, … , mu} Input: Sl={(x1, y1), …,(xml, yml)} NP-hard….. Convex only after you guessed the labels… too many possible guesses…

Transductive Support Vector Machines

Optimize for the separator with large margin wrt labeled and unlabeled data.

Heuristic (Joachims) high level idea: Keep going until no more improvements. Finds a locally-optimal solution.

• First maximize margin over the labeled points
• Use this to give initial labels to unlabeled points

based on this separator.

• Try flipping labels of unlabeled points to see if doing

so can increase margin

Experiments [Joachims99]

Highly compatible

+ +

_ _

Helpful distribution Non-helpful distributions

Transductive Support Vector Machines

1/°2 clusters, all partitions separable by large margin

Semi-Supervised Learning

Prominent paradigm in past 15 years in Machine Learning.

Unlabeled data useful if we have beliefs not only about the form of the target, but also about its relationship with the underlying distribution.

Key Insight – Transductive SVM [Joachims ’99] – Co-training [Blum & Mitchell ’98] – Graph-based methods [B&C01], [ZGL03]

Prominent techniques