[PPT] - Generative Models and Nave Bayes Ke Chen Reading: [14.3, EA], PowerPoint Presentation

SLIDE 1

COMP24111 Machine Learning

Generative Models and Naïve Bayes

Ke Chen Reading: [14.3, EA], [3.5, KPM], [1.5.4, CMB]

SLIDE 2

COMP24111 Machine Learning

2

Outline

Background and Probability Basics
Probabilistic Classification Principle

– Probabilistic discriminative models – Generative models and their application to classification – MAP and converting generative into discriminative

Naïve Bayes – an generative model

– Principle and Algorithms (discrete vs. continuous) – Example: Play Tennis

Zero Conditional Probability and Treatment
Summary

SLIDE 3

COMP24111 Machine Learning

3

Background

There are three methodologies:

a) Model a classification rule directly

Examples: k-NN, linear classifier, SVM, neural nets, .. b) Model the probability of class memberships given input data Examples: logistic regression, probabilistic neural nets (softmax),…

c) Make a probabilistic model of data within each class

Examples: naive Bayes, model-based ….

Important ML taxonomy for learning models

probabilistic models vs non-probabilistic models discriminative models vs generative models

SLIDE 4

COMP24111 Machine Learning

4

Background

Based on the taxonomy, we can see different the essence of

learning models (classifiers) more clearly.

Probabilistic Non-Probabilistic Discriminative

Logistic Regression
Probabilistic neural nets
……..
K-nn
Linear classifier
SVM
Neural networks
……

Generative

Naïve Bayes
Model-based (e.g., GMM)
……

N.A. (?)

SLIDE 5

COMP24111 Machine Learning

5

Probability Basics

Prior, conditional and joint probability for random variables

– Prior probability: – Conditional probability: – Joint probability: – Relationship: – Independence:

Bayesian Rule

) | ) (

1 2 1

x P(x x | x P

2

,

) (x P

) ) ( ), , (

2 2

,x P(x P x x

1 1

= = x x ) ( ) | ( ) ( ) | ( )

2 2 1 1 1 2 2

x P x x P x P x x P ,x P(x1 = = ) ( ) ( ) ), ( ) | ( ), ( ) | (

2 1 2 1 2 1 2 1 2

x P x P ,x P(x x P x x P x P x x P

1

= = =

) ( ) ( ) ( ) ( x x x P c P c | P | c P =

Evidence Prior Likelihood Posterior × =

Discriminative Generative

SLIDE 6

COMP24111 Machine Learning

6

Probabilistic Classification Principle

Establishing a probabilistic model for classification

–

Discriminative model

) , , , (

2 1 n

x x x ⋅ ⋅ ⋅ = x

Discriminative Probabilistic Classifier

1

x

2

x

n

x

) | ( 1 x c P ) | (

2 x

c P ) | ( x

L

c P

To train a discriminative classifier

regardless its probabilistic or non- probabilistic nature, all training examples of different classes must be jointly used to build up a single discriminative classifier.

Output L probabilities for L class

labels in a probabilistic classifier while a single label is achieved by a non-probabilistic classifier .

) , , ) (

1 n 1 L

x (x c , , c c | c P ⋅ ⋅ ⋅ = ⋅ ⋅ ⋅ = , x x

SLIDE 7

COMP24111 Machine Learning

7

Probabilistic Classification Principle

Establishing a probabilistic model for classification (cont.)

–

Generative model (must be probabilistic)

) , , ) (

1 n 1 L

x (x c , , c c c | P ⋅ ⋅ ⋅ = ⋅ ⋅ ⋅ = , x x

Generative Probabilistic Model for Class 1

) | (

1

c P x

1

x

2

x

n

x

)

, , , (

2 1 n

x x x ⋅ ⋅ ⋅ = x

Generative Probabilistic Model for Class L

)

L

c P | (x

1

x

2

x

n

x

L probabilistic models have

to be trained independently

Each is trained on only the

examples of the same label

Output L probabilities for a

given input with L models

“Generative” means that

such a model produces data subject to the distribution via sampling.

SLIDE 8

COMP24111 Machine Learning

8

Probabilistic Classification Principle

Maximum A Posterior (MAP) classification rule

– For an input x, find the largest one from L probabilities output by a discriminative probabilistic classifier – Assign x to label c* if is the largest.

Generative classification with the MAP rule

– Apply Bayesian rule to convert them into posterior probabilities – Then apply the MAP rule to assign a label

..., , ). ( ) ( 1 x x | c P | c P

L

L i c P c | P P c P c | P | c P

i i i i i

for , , 2 , 1 ) ( ) ( ) ( ) ( ) ( ) ( ⋅⋅ ⋅ = ∝ = x x x x

) ( * x | c P

Common factor for all L probabilities

SLIDE 9

COMP24111 Machine Learning

9

Naïve Bayes

Bayes classification

Difficulty: learning the joint probability is infeasible!

Naïve Bayes classification

– Assume all input features are class conditionally independent! – Apply the MAP classification rule: assign to c* if

. ,..., for ) ( ) | , , ( ) ( ) ( ) (

1 1 L n

c c c c P c x x P c P c | P | c P = ⋅ ⋅ ⋅ = ∝ x x ) | , , ( 1 c x x P

n

⋅ ⋅ ⋅

L n n

c c c c c c P c a P c a P c P c a P c a P , , , ), ( )] | ( ) | ( [ ) ( )] | ( ) | ( [

1 * 1 * * * 1

⋅ ⋅ ⋅ = ≠ ⋅ ⋅ ⋅ > ⋅ ⋅ ⋅

) | ( ) | ( ) | ( ) | , , ( ) | ( ) | , , ( ) , , , | ( ) | , , , (

2 1 2 1 2 2 1 2 1

c x P c x P c x P c x x P c x P c x x P c x x x P c x x x P

n n n n n

⋅ ⋅ ⋅ = ⋅ ⋅ ⋅ = ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ = ⋅ ⋅ ⋅

) , , , ( '

2 1 n

a a a ⋅ ⋅ ⋅ = x

Applying the independence assumption

) | , , (

f

estimate

* 1

c a a P

n

⋅ ⋅ ⋅

) | , , (

f

esitmate

1

c a a P

n

⋅ ⋅ ⋅

SLIDE 10

COMP24111 Machine Learning

10

Naïve Bayes

S; in examples with estimate feature each

f

value feature every For S in examples with estimate

f

lue target va each For ) | ( ) | ( ˆ ) , 1 ; , , 1 ( ; ) ( ) ( ˆ

1 i jk i jk j j j jk i i L i i

c x P c x x P N , k F j x x c P c P ) c , , c (c c ← = ⋅ ⋅ ⋅ = ⋅ ⋅ ⋅ = ← ⋅ ⋅ ⋅ =

L i i i i n i n

c c c c c c P c a P c a P c P c a P c a P , , , ), ( ˆ )] | ( ˆ ) | ( ˆ [ ) ( ˆ )] | ( ˆ ) | ( ˆ [

1 * 1 * * * 1

⋅ ⋅ ⋅ = ≠ ′ ⋅ ⋅ ⋅ ′ > ′ ⋅ ⋅ ⋅ ′

) , , ( '

1 n

a a ′ ⋅ ⋅ ⋅ ′ = x

SLIDE 11

COMP24111 Machine Learning

11

Example

Example: Play Tennis

SLIDE 12

COMP24111 Machine Learning

12

Example

Learning Phase

Outlook Play=Yes Play=No

Sunny

2/9 3/5

Overcast

4/9 0/5

Rain

3/9 2/5

Temperature Play=Yes Play=No

Hot

2/9 2/5

Mild

4/9 2/5

Cool

3/9 1/5

Humidity Play=Yes Play=No

High

3/9 4/5

Normal

6/9 1/5

Wind Play=Yes Play=No

Strong

3/9 3/5

Weak

6/9 2/5 P(Play=Yes) = 9/14 P(Play=No) = 5/14

SLIDE 13

COMP24111 Machine Learning

13

Example

Test Phase

– Given a new instance, predict its label

x’=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong)

– Look up tables achieved in the learning phrase – Decision making with the MAP rule

Given the fact P(Yes|x’) < P(No|x’), we label x’ to be “No”.

SLIDE 14

COMP24111 Machine Learning

14

Naïve Bayes

Algorithm: Continuous-valued Features

– Numberless values taken by a continuous-valued feature – Conditional probability often modeled with the normal distribution – Learning Phase: Output: normal distributions and – Test Phase: Given an unknown instance

Instead of looking-up tables, calculate conditional probabilities with all the

normal distributions achieved in the learning phrase

Apply the MAP rule to assign a label (the same as done for the discrete case)

i j ji i j ji ji ji j ji i j

c c c x x c x P = =         − − = for which examples

f

x values feature

f

deviation standard : c for which examples

f

values feature

f

(avearage) mean : 2 ) ( exp 2 1 ) | ( ˆ

2 2

σ µ σ µ σ π

L F

c c C X X , , ), , , ( for

1 1

⋅ ⋅ ⋅ = ⋅ ⋅ ⋅ = X

L F ×

L i c C P

i

, , 1 ) ( ⋅ ⋅ ⋅ = = ) , , ( 1

n

a a ′ ⋅ ⋅ ⋅ ′ = ′ X

SLIDE 15

COMP24111 Machine Learning

15

Naïve Bayes

Example: Continuous-valued Features

– Temperature is naturally of continuous value. Yes: 25.2, 19.3, 18.5, 21.7, 20.1, 24.3, 22.8, 23.1, 19.8 No: 27.3, 30.1, 17.4, 29.5, 15.1 – Estimate mean and variance for each class –

Learning Phase: output two Gaussian models for P(temp|C)

∑ ∑

= =

µ − = σ = µ

N n n N n n

x N x N

1 2 2 1

) ( 1 , 1 09 . 7 , 88 . 23 35 . 2 , 64 . 21 = σ = µ = σ = µ

No No Yes Yes

        − − =         × − − =         − − =         × − − = 25 . 50 ) 88 . 23 ( exp 2 09 . 7 1 09 . 7 2 ) 88 . 23 ( exp 2 09 . 7 1 ) | ( ˆ 09 . 11 ) 64 . 21 ( exp 2 35 . 2 1 35 . 2 2 ) 64 . 21 ( exp 2 35 . 2 1 ) | ( ˆ

2 2 2 2 2 2

x x No x P x x Yes x P π π π π

SLIDE 16

COMP24111 Machine Learning

16

Zero conditional probability

If no example contains the feature value

– In this circumstance, we face a zero conditional probability problem during test – For a remedy, class conditional probabilities re-estimated with

) | ( ˆ , for = =

i jk jk j

c a P a x

) | ( ˆ ) | ( ˆ ) | ( ˆ

1

= ⋅ ⋅ ⋅ ⋅ ⋅ ⋅

i n i jk i

c x P c a P c x P

) 1 examples, virtual" "

f

(number prior weight to : )

f

values possible for / 1 (usually, estimate prior : for which examples training

f

number : and for which examples training

f

number : ) | ( ˆ ≥ = = = = + + = m m x t t p p c c n c c a x n m n mp n c a P

j i i jk j c c i jk

(m-estimate)

SLIDE 17

COMP24111 Machine Learning

17

Zero conditional probability

Example: P(outlook=overcast|no)=0 in the play-tennis dataset

– Adding m “virtual” examples (m: up to 1% of # training example)

In this dataset, # of training examples for the “no” class is 5.
We can only add m= 1 “virtual” example in our m-esitmate remedy.

– The “outlook” feature can takes only 3 values. So p=1/3. – Re-estimate P(outlook|no) with the m-estimate

SLIDE 18

COMP24111 Machine Learning

18

Summary

Probabilistic Classification Principle

– Discriminative vs. Generative models: learning P(c|x) vs. P(x|c) – Generative models for classification: MAP and Bayesian rule

Naïve Bayes: the conditional independence assumption

– Training and test are very efficient. – Two different data types lead to two different learning algorithms. – Working well sometimes for data violating the assumption!

Naïve Bayes: a popular generative model for classification

– Performance competitive to most of state-of-the-art classifiers even in presence of violating independence assumption – Many successful applications, e.g., spam mail filtering – A good candidate of a base learner in ensemble learning – Apart from classification, naïve Bayes can do more…