[PPT] - y ou - User-sp eied e y = f ( x ) y P ( y | x ) in PowerPoint Presentation

SLIDE 1 Review

f

Le ture 4

Erro

r measures

User-sp

e ied e

h(x), f(x)
f

8 > < > : +1

y

u

−1

in truder

In-sample:

E

in(h) = 1

N

n=1

e

h(xn), f(xn)
Out-of-sample

E

ut(h) = Ex
e
h(x), f(x)
Noisy

ta rgets

y = f(x) − → y ∼ P(y | x)

P y ( | )

x

f: X Y

( )

x

P

TRAINING EXAMPLES

x y x y

N N 1 1

( , ), ... , ( , )

UNKNOWN TARGET DISTRIBUTION target function plus noise DISTRIBUTION UNKNOWN INPUT

(x1, y1), · · · , (xN, yN)

generated b y

P(x, y) = P(x)P(y|x)

E
ut(h)

is no w Ex,y

e
h(x), y

SLIDE 2 Lea rning F rom Data Y aser S. Abu-Mostafa Califo rnia Institute

f

T e hnology Le ture 5: T raining versus T esting Sp

nso

red b y Calte h's Provost O e, E&AS Division, and IST

T

uesda y , Ap ril 17, 2012

SLIDE 3 Outline

F

rom training to testing

Illustrative

examples

Key

notion: b reak p

int
Puzzle

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 2/20

SLIDE 4 The nal exam T esting: P P [|E in − E

ut| > ǫ] ≤

2 e−2ǫ2N

T raining: P P [|E in − E

ut| > ǫ] ≤ 2M e−2ǫ2N

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 3/20

SLIDE 5 Where did the M

me

from? The B ad events Bm a re |E in(hm) − E

ut(hm)| > ǫ

The union b

und:

P[B1 or B2 or · · · or BM]

B3 B1 B2

≤P[B1] + P[B2] + · · · + P[BM]

no
verlaps:

M terms

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 4/20

SLIDE 6 Can w e imp rove

n M

?

1 +1

up down

Y es, bad events a re very

verlapping!

∆E

ut

: hange in +1 and −1 a reas

∆E

in : hange in lab els

f

data p

ints

|E

in(h1) − E

ut(h1)| ≈ |E

in(h2) − E

ut(h2)|

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 5/20

SLIDE 7 What an w e repla e M with? Instead

f

the whole input spa e, w e

nsider

a nite set

f

input p

ints,

and

unt

the numb er

f

di hotomies

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 6/20

SLIDE 8 Di hotomies: mini-hyp

theses

A hyp

thesis

h : X → {−1, +1}

A di hotomy

h : {x1, x2, · · · , xN} → {−1, +1}

Numb er

f

hyp

theses |H|

an b e innite Numb er

f

di hotomies |H(x1, x2, · · · , xN)| is at most 2N Candidate fo r repla ing M

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 7/20

SLIDE 9 The gro wth fun tion The gro wth fun tion

unts

the most di hotomies

n

any N p

ints

mH(N)= max

x1,··· ,xN∈X |H(x1, · · · , xN)|

The gro wth fun tion satises:

mH(N) ≤ 2N

Let's apply the denition.

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 8/20

SLIDE 10 Applying mH(N) denition

p

er eptrons PSfrag repla ements 0.5 1 1.5 2 2.5 3 3.5 4 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 PSfrag repla ements 0.5 1 1.5 2 2.5 3 3.5 4 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 PSfrag repla ements 0.5 1 1.5 2 2.5 3 3.5 4 0.5 1 1.5 2 2.5 3

N = 3 N = 3 N = 4

mH(3) =

8

mH(4) =

14

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 9/20

SLIDE 11 Outline

F

rom training to testing

Illustrative

examples

Key

notion: b reak p

int
Puzzle

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 10/20

SLIDE 12 Example 1: p

sitive

ra ys PSfrag repla ements

x1 x2 x3 xN . . . h(x) = −1 h(x) = +1 a

0.2 0.4 0.6 0.8 1

0.1
0.08
0.06
0.04
0.02

0.02 0.04 0.06 0.08 0.1

H

is set

f h: R → {−1, +1}

h(x) = sign(x − a) mH(N) = N + 1

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 11/20

SLIDE 13 Example 2: p

sitive

intervals PSfrag repla ements

x1 x2 x3 xN . . . h(x) = −1 h(x) = −1 h(x) = +1

0.2 0.4 0.6 0.8 1

0.1
0.08
0.06
0.04
0.02

0.02 0.04 0.06 0.08 0.1

H

is set

f h: R → {−1, +1}

Pla e interval ends in t w

f N + 1

sp

ts

mH(N) =

N+1

2

+1 = 1

2N 2 + 1 2N + 1

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 12/20

SLIDE 14 Example 3:

nvex

sets

+ + + + + − − − − −

up bottom

H

is set

f h: R2 → {−1, +1}

h(x) = +1

is

nvex

mH(N) = 2N

The N p

ints

a re `shattered' b y

nvex

sets

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 13/20

SLIDE 15 The 3 gro wth fun tions

H

is p

sitive

ra ys:

mH(N) = N + 1

H

is p

sitive

intervals:

mH(N) = 1

2N 2 + 1 2N + 1

H

is

nvex

sets:

mH(N) = 2N

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 14/20

SLIDE 16 Ba k to the big pi ture Rememb er this inequalit y? P P [|E in − E

ut| > ǫ] ≤ 2M e−2ǫ2N

What happ ens if mH(N) repla es M ?

mH(N)

p

lynomial

= ⇒

Go

d!

Just p rove that mH(N) is p

lynomial?

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 15/20

SLIDE 17 Outline

F

rom training to testing

Illustrative

examples

Key

notion: b reak p

int
Puzzle

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 16/20

SLIDE 18 Break p

int
f H

PSfrag repla ements 0.5 1 1.5 2 2.5 3 3.5 4 0.5 1 1.5 2 2.5 3 Denition: If no data set

f

size k an b e shattered b y H , then k is a b reak p

int

fo r H

mH(k) < 2k

F

r

2D p er eptrons, k = 4 A bigger data set annot b e shattered either

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 17/20

SLIDE 19 Break p

int
the

3 examples

P
sitive

ra ys mH(N) = N + 1 b reak p

int k =

2

P
sitive

intervals mH(N) = 1

2N 2 + 1 2N + 1

b reak p

int k =

3

Convex

sets mH(N) = 2N b reak p

int k =

`∞'

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 18/20

SLIDE 20 Main result No b reak p

int

= ⇒ mH(N) = 2N

Any b reak p

int

= ⇒ mH(N)

is p

lynomial

in N

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 19/20

SLIDE 21 Puzzle

x1 x2 x3

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 5 20/20