Linear Classifiers and Perceptron E.D. And F. MAN TO BUY - - PowerPoint PPT Presentation

LinearClassifiersandPerceptron

CS678AdvancedTopicsinMachineLearning ThorstenJoachims Spring2003 Outline:

• Linearclassifiers
• Example:textclassification
• Perceptronlearningalgorithm
• MistakeboundforPerceptron
• Separationmargin
• Dualrepresentation

TextClassification

E.D.AndF.MANTOBUYINTOHONGKONG FIRM TheU.K.BasedcommodityhouseE.D.AndF.Man LtdandSingapore’ sYeoHiapSengLtdjointly announcedthatManwillbuyasubstantialstakein Yeo’ s71.1pctheldunit,YeoHiapSengEnterprises Ltd.Manwilldevelopthelocallylistedsoftdrinks manufacturerintoasecuritiesandcommodities brokeragearmandwillrenamethefirmManPacific (Holdings)Ltd.

Aboutacorportateacquisition? JA NEIN

LearningTextClassifiers

Goal:

• Learnerusestrainingsettofindclassifierwithlowpredictionerror.

TrainingSet NewDocuments Learner Classifier Real-World Process

Generativevs.DiscriminativeTraining

Process:

• Generator:Generatesdescriptions accordingtodistribution

.

• Teacher:Assignsavalue toeachdescription basedon

.

x P x ( ) y x P y x ( )

DiscriminativeTraining

• makeassumptionsaboutthe

setHofclassifiers

• estimateerrorofclassifiersin

Hfromthetrainingdata

• selectclassifierwithlowest

errorrate

• example:SVM,decisiontree

GenerativeTraining

• makeassumptionsaboutthe

parametricformof .

• estimatetheparametersof

fromthetrainingdata

• deriveoptimalclassifierusing

Bayes’ rule

• example:naiveBayes

P x y , ( ) P x y , ( )

=>Trainingexamples

x1 y1 , ( ) … xn yn , ( ) , , P x y , ( ) xi ℜ

N y

∈

i

∼ 1 1 – { , } ∈

RepresentingTextasAttributeVectors

Attributes:Words (Word-Stems) Values:Occurrence- Frequencies ==>Theorderingofwordsisignored!

graphics baseball specs references hockey car clinton unix space quicktime computer . . . 3 1 1 2

From: xxx@sciences.sdsu.edu Newsgroups: comp.graphics Subject: Need specs on Apple QT I need to get the specs, or at least a for QuickTime. Technical articles from be nice, too. have on ... very verbose interpretation of the specs,

• n a Unix or MS-Dos system. I can’t

do much with the QuickTime stuff they I also need the specs in a fromat usable magazines and references to books would

LinearClassifiers(Example)

D1: D2:

TextClassification:Physics(+1)versusReceipes(-1) ID nuclear (x1) atom (x2) salt (x3) pepper (x4) water (x5) heat (x6) and (x7) y D1 1 2 2 2 +1 D2 3 1 1

• 1

D3 2 1 3 +1 D4 1 1 1 1 1

• 1

w,b 2 3

• 1
• 3
• 1
• 1

b=1

wixi

i 1 = 7

• b

+ 2 1 3 2 1 – ( ) 0 3 – ( ) 0 1 – ( ) 2 1 – ( ) 0 0 2 ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ [ ] 1 + = wixi

i 1 = 7

• b

+ 2 0 3 0 1 – ( ) 0 3 – ( ) 3 1 – ( ) 0 1 – ( ) 1 0 1 ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + ⋅ [ ] 1 + =

LinearClassifiers

RulesoftheForm:weightvector ,threshold GeometricInterpretation(Hyperplane):

h x ( ) sign wixi

i 1 = N

• b

+ 1 if wixi

i 1 = N

• b

+ > 1 – else

• =

= w b

w b

Perceptron(Rosenblatt)

Input: (linearseparable)

• repeat
• fori=1ton
• if
• endif
• endfor
• untilnomistakesmadeintheforloop
• return

S x1 y1 , ( ) … xn yn , ( ) , ,

• =

xi ℜ

N y

∈

i

1 1 – { , } ∈ w0 0 b0 ; 0 k ; ← ← ← R maxi xi = yi wk xi bk + ⋅ ( ) ≤ wk

1 +

wk ηyixi + ← bk

1 +

bk ηyiR2 + ← k k ← 1 = wk bk , ( )

AnalysisofPerceptron

Definition(MarginofanExample):Themarginofanexample

• withrespecttothehyperplane

is Definition(MarginofanExample):Themarginofatrainingset withrespecttothehyperplane is Theorem(Novikoff):IfforatrainingsetSthereexistsaweightvector withmargin ,thentheperceptronmakesatmost mistakesbeforereturningaseparatinghyperplane.

xi yi , ( ) w b , ( ) δi yi w xi b + ⋅ ( ) = S x1 y1 , ( ) … xn yn , ( ) , ,

• =

w b , ( ) δ mini yi w xi b + ⋅ ( ) = δ 4 R2 δ2

• DualPerceptron
• Foreachexample

,countwith thenumberoftimesthe perceptronalgorithmmakesamistakeonit.Then

• and
• repeat
• fori=1ton
• if
• endif
• endfor
• untilnomistakesmadeintheforloop
• return

xi yi , ( ) αi w αiyixi

i 1 = n

• =

α = b0 ; ← R maxi xi = yi αiyi xj xi ⋅ ( )

j 1 = n

• bk

+

• ≤

αi αi 1 + ← b b yiR2 + ← α b , ( )

Experiment:PerceptronforTextClassification Trainon1000pos/1000negexamplesfor“ acq” (Reuters-21578).

5 10 15 20 25 30 1 2 3 4 5 6 7 8 9 10 Percent Training/Testing Errors Iterations Perceptron with eta=0.1 "perceptron_iter_trainerror.dat" "perceptron_iter_testerror.dat" hard_margin_svm_testerror.dat