Learning: Linear Methods CE417: Introduction to Artificial - - PowerPoint PPT Presentation
Learning: Linear Methods CE417: Introduction to Artificial Intelligence Sharif University of Technology Spring 2019 Soleymani Some slides are based on Klein and Abdeel, CS188, UC Berkeley. Paradigms of ML } Supervised learning (regression,
2
3
} E.g., 𝑧 ∈ [0,1]
} E.g.,𝑧 ∈ {1,2, … , 𝐷}
4
5
6
7
8
9
10
11
} Also known as ”steepest descent”
} 𝐾(𝒙) decreases fastest if one goes from 𝒙M in the direction of −𝛼𝐾 𝒙M } Assumption: 𝐾(𝒙) is defined and differentiable in a neighborhood of a
12
13
14 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
15 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
16
17
This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
18
19 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
20 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
21 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
22 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
23 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
24 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
25 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
26 This example has been adopted from: Prof. Ng’s slides (ML Online Course, Stanford)
27
28
Dear GlobalSCAPE Customer, GlobalSCAPE has partnered with ScanSoft to offer you the latest version of OmniPage Pro, for just $99.99* - the regular list price is $499! The most common question we've received about this offer is - Is this genuine? We would like to assure you that this offer is authorized by ScanSoft, is genuine and valid. You can get the . . . . . . To receive your $30 Amazon.com promotional certificate, click through to http://www.amazon.com/apparel and see the prominent link for the $30 offer. All details are
if you'd rather not receive future e-mails announcing new store launches, please click . . .
29
} Have you emailed the sender before? } Have 1M other people just gotten the same email? } Is the sending information consistent? } Is the email in ALL CAPS? } Do inline URLs point where they say they point? } Does the email address you by (your) name?
30
Hello, Do you want free printr cartriges? Why pay more when you can get them ABSOLUTELY FREE! Just # free : 2 YOUR_NAME : 0 MISSPELLED : 2 FROM_FRIEND : 0 ...
PIXEL-7,12 : 1 PIXEL-7,13 : 0 ... NUM_LOOPS : 1 ...
31
} Binary case: compare features to a weight vector to identify the class } Learning: figure out the weight vector from examples
# free : 2 YOUR_NAME : 0 MISSPELLED : 2 FROM_FRIEND : 0 ... # free : 4 YOUR_NAME :-1 MISSPELLED : 1 FROM_FRIEND :-3 ... # free : 0 YOUR_NAME : 1 MISSPELLED : 1 FROM_FRIEND : 1 ...
32
33
𝑦1 𝑦2
1 2 3 1 2 3 4
} Examples are points } Any weight vector is a hyperplane } One side corresponds toY=+1 } Other corresponds toY=-1
34
35
36
}
Least square loss penalizes ‘too correct’ predictions (that they lie a long way on the correct side
}
Least square loss also lack robustness to noise
*
37
38
39
40
41
42
43
44
}
}
}
}
}
45
46
47
} Start with weights = 0 } For each training instance: } Classify with current weights } If correct (i.e., y=y*), no change! } If wrong: adjust the weight vector by
48
49
50
} If we have multiple classes:
}
A weight vector for each class:
}
Score (activation) of a class y:
}
Prediction highest score wins
51
} Start with all weights = 0 } Pick up training examples one by one } Predict with current weights } If correct, no change! } If wrong: lower score of wrong answer,
52
53
} Separability: true if some parameters get the training set
} Convergence: if the training is separable, perceptron will
} Mistake Bound: the maximum number of mistakes (binary
54
55
56
57
58
59
60
61
62
63
64
65 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017
66 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017
67 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017
68 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017
} Maximizing the margin: good according to intuition, theory, practice } Support vector machines (SVMs) find the separator with max margin 69
70
* 𝒙
71
72
73
74
𝒙,—˜, š› ›œ•
ž
75
ž
… ≡ … 𝒚 : ; 𝑿
\𝒚(:)
76
… − 𝑡¡(–) …¢¡(–)
… = 𝒙… \𝒚(:)
77 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017
78
* Y ¤B'
Y ¤B'
This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017
79
} Linear } Polynomial (univariate) 80
81
82
83
*
A :B'
* ≫ 0
84
85
86
86
87
88
} statistical bounds on the difference between training and expected
89
90
91
92
} 𝐾¸ 𝒙 =
' ¸_”¹M ∑
*
93
} extra parameter (e.g., degree of polynomial) is fit to this set.
} performance of the selected model is finally evaluated on the test set
94
} Shuffle the dataset and randomly partition training data into 𝑙 groups of
} for 𝑗 = 1 to 𝑙
} Choose the 𝑗-th group as the held-out validation group } Train the model on all but the 𝑗-th group of data } Evaluate the model on the held-out group
} Performance scores of the model from 𝑙 runs are averaged.
} The average error rate can be considered as an estimation of the true
95
96
97
98
99
100
101
} 𝐾¸ 𝒙 =
' A_¸ ∑
*
102
103
104
105
*
*
106
107
*
A :B'
108
109