Thanks to R Parr, C Guesterin - - PowerPoint PPT Presentation

• Thanks to R Parr, C Guesterin

• !

"!# $$%&!

'

$('

)

• (*++'

, *-!#.

/ --00 1

, *,0

• +%%%%

/'2)3'221

4

"-

$

56θ

θ θ θ%567θ θ θ θ

...

+ !-!!*-

5%%%%6θ

θ θ θ θ θ θ θ (1 (1 (1 (1− − − − θ θ θ θ) (1 ) (1 ) (1 ) (1− − − − θ θ θ θ) ) ) ) θ θ θ θ = = = = θ θ θ θ3

3 3 3(1

(1 (1 (1− − − − θ θ θ θ) ) ) )2

2 2 2

28!" ,α α

9

$:(#'

• +" ,

α α

(*;-< θ = -

/>-?!+,!1

θ

:=

• -,-+

@

2 ;(*< A*

2++=

ln[ (1 ) ] [ ln ln (1 ) ] (1 )

h t t

h t h t θ θ θ θ θ θ θ θ ∂ ∂ − − = + − = + ∂ ∂ −

(1 ) h t t t h θ θ θ − + =

• =

− +

θ ˆ

So just average!!!

B

0, ;<1

)+ %θ$(' 6C96D.@

)+D D %θ$(' 6D.@

T H H MLE

α α α θ + = ˆ

E

3*&!

Normal( µ=0.6, σ=0.048)

&!

; <

%

$µ 6CF &!

σ σ σ σ µGµCF

%

µ6D.@ σ6D.D4E

D%D

µ6D.@ σ6D.DD4E

Normal( µ=0.6, σ=0.0048)

H

"-

5- 6D.@ !*#C % ,

n=5 n=50 P( D | θ θ θ θ ) for fixed θ θ θ θ=0.6

D

5--!

5- 6θ !* %

P( D | θ θ θ θ ) for fixed D

• ,,*> '8

5IJ2 G µ J < λ K≥ G 7λCΓ

• =

=

m i i m

X m S

1

1

∀ --...?... 2 +*#-!+

L ! ",-++*+.+. MD%N

5I2 Oµ Fλ KP7λQ

• 2 -

*,,*> 8

• L, 6αFα

2 +*6 (θR -

εOD

T H H MLE

α α α θ + = ˆ

• 5A(*

5A5--A :! #0-!# θ%

0ε 6D.% 0 --≥7δ 6D.H9

8L, O CδCε2

≈ 460.2

4

/- #0*1

2 #0 -!#θ

;!< 9D79D

!"#"$ *θ%

• -> + -+,θ

9

Two (related) Distributions: Parameter, Instances

Θ Θ Θ Θ = 0.1 Θ Θ Θ Θ = 0.1

T T T T H T

• Θ

Θ Θ Θ = 0.5

T T H T H H

• Θ

Θ Θ Θ = 0.8

H H T H H H

• Uniform density

Θ

1.0

Θ Θ Θ Θ

@

Two (related) Distributions: Parameter, Instances

Θ Θ Θ Θ = 0.1

T T T T H T

• Θ

Θ Θ Θ = 0.5

T T H T H H

• Θ

Θ Θ Θ = 0.8

H H T H H H

• Θ

1.0

Uniform density

Θ

1.0

Θ Θ Θ Θ

Θ

1.0

B

(*

3 8+0 *:θ 5θJ"

• #

E

(*,-!#

(#,! /- 1

: #0* 2 ,

?*

7, ,

• %#"&"'!(#)!"
• #

H

• G 5θ

5 (#,! )+ST%-

$C F- 3 ,%-O GCF-G &!-CF- F-G

D

5- ,

Prior P(θ) Likelihood P(D|θ) So Posterior is same form as Prior!! Conjugate!

• 5"-

5θ T α%α " % 5-

θ J T Fα% Fα

Prior + observe 1 head + observe 27 more heads; 18 tails

• ?*5

)+

5Θ Tα%α " 0

• #

5-

ΘJ Tα F%α F

5! 5θJα "'!(#,!

,**% !-0

5θJα>

,0, α>

• 3*5

5- ,! 0,θ

** ': !+

• *,!

4

$A5$: :

A-+%. #

,-$A5

U (#$('6*:θ5"Jθ

• ,;-+*< ≈ β7%β7:,

2 1 ) | ( max arg ˆ − + + + − + = =

T T H H H H MAP

D P β α β α β α θ θ

θ

) ˆ (

MAP

f θ ≈

$A5#

9

$A5,-

$A5# 8+:-!#, A ∞% ;,*< %+&"

2 1 ) | ( max arg ˆ − + + + − + = =

T T H H H H MAP

D P β α β α β α θ θ

θ

@

5!,

5Θ Tα%α

• + %

/ --:F, 1

Θ Θ × Θ = = =

• +

+

d D P D H X P D H X P

m m

) | ( ) , | ( ) | (

1 1 1

T T H H H H m m Beta T T H H

m m m E d m m Beta

T T H H

+ + + + = Θ = Θ + + Θ × Θ =

+ + Θ

• α

α α α α

α α

] [ ) , : (

) , : ( 1

B

A+;'!*<

%-≡ %µ

6F-

,,!+ =

µ 6CF-

'*

%6%D.9 D%D6D%D.9 B%6D%D.B

E

A !-+ 8+ =

8+:,

• A % ;,*<

#!&*+&

"

,!-"+

5 =-α%α.% > 8+ =α

Fix m’, change α Fix α, change m’

H

* 2*

m

• +

+

• +

= + + + = α α α α α α α α m m m m m m m m

H H H H

6 F α 6α Fα 8+ =

• θ$('

T T H H H H

m m m E + + + + = Θ α α α ] [

$(' *α

U $('%-0

6D $('

D

*,$

/,+#7-!#111

?'-!#?+

(#,!,!"

• 5S66θ

6..#

• θ 6θ ≥ D

/"'!( ,

• 0- % ,*%
• "!α F%%α# F#

)"

• +

+ = =

+ j j j i i m

m m D i X P ) ( ) | (

1

α α

• !
• "!#
• $$%&!
• '

$('

)

5 ,) (*5,)

• $+"-

A

!"-" )%

0µV+!σ

5" ---,!

• 25 ,)

A,,,

*-!*!

ST µ%σ W6S F-

WT µF-%σ

2,)

ST µS%σ

S

WT µW%σ

W

X6SFW

• XT µSFµW%σ

SFσ W

4

$+)

A7),-

• +!µ 6Iµ%µ K
• !+!:

0σ%?

6'I: G µ:? G µ?K

!+!

!%

; +7,<∀:: : ≥ D

• =

Σ

2 2 , 2 2 2 , 1 2 1 , 2 2 1 , 1

σ σ σ σ

$&)6$& ) 6)++-

9

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3 0.05 0.1 0.15 0.2 0.25

• =

Σ 1 1

• Σ

µ

@

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3

µ

• =

Σ 1 1

B

2 )

E

• µ
• =

Σ 1 5 . 5 . 1

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3 0.05 0.1 0.15 0.2 0.25

H

• −3

−2 −1 1 2 3 −3 −2 −1 1 2 3

µ

• =

Σ 1 5 . 5 . 1

4D

• µ
• =

Σ 1 8 . 8 . 1

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3 0.05 0.1 0.15 0.2 0.25

4

• −3

−2 −1 1 2 3 −3 −2 −1 1 2 3

µ

• =

Σ 1 8 . 8 . 1

4

&-

4

$+)':

44

': ,)

6α 6"*α% α 6"*α% α# )

Marginal…

49

3,5 ,)

2,!,8 --..

,D%!+!) *)

'+*)≡

)* ,,,

4@

3,5 ,)

A)-!

!,!-

+!, !+!:

8 ! 8

• ?-+ -

+-8 !

4B

3,5 ,)

$* ,)) )+ $*"- $*=-**

• Σ

Σ Σ Σ = Σ = =

bb ba ab aa b a b a x

x x ) , ( ), , ( µ µ µ

) , | ( ) (

aa a a a

x N x p Σ = µ

4E

3,5 ,)&

,)) "-

• Λ

Λ Λ Λ = Σ = Λ

− bb ba ab aa 1

) ( ) , | ( ) | (

1 | 1 | a b ab aa a b a aa b a a b a

x x N x x p µ µ µ µ − Λ Λ − = Λ =

− −

4H

&=*$*=Y *

9D

3,5 ,)&

A,,,,)+-

)

2 : ) 6A:F- )

3

• W ,-:

: ,-+*

9

3,5 ,)

(,*!*--

:0-)

(

,"+-0, +!00)

• (,70+**

?,*)-, *

9

(*)

!,%"

,7+...!

.*.%:!

(

$%µ &!%σ

99 75 82 … 93 :

9

$(',)

5-.,...!"6M:%%:N (*7#,

∏ ∏

= − − =

• =

=

N i x N N i i

i

e x P D P

1 2 ) ( 1

2 2

2 1 ) , | ( ) , | (

σ µ

π σ σ µ σ µ

94

$(',,)

/$(µT ,µ1

• −

= − =

• −

− =

• =

= =

µ σ µ σ σ µ µ N x x x d d

N i i N i i i N i 1 2 1 2 2 2 1

1 ) ( 2 2 1 2 ) (

• =

=

=

• =
• −
• =

N i i MLE N i i

x N N x D P d d

1 1

1 ˆ ) , | ( ln µ µ σ µ µ

MLE

µ ˆ Just empirical mean!!

99

$(',&!

A*%++=

• −

− − − =

i i

x N

3 2

2 ) ( 2 σ µ σ

… = 0

• −

=

i i MLE

x N

2 2

) ( 1 µ σ

• Just empirical variance!!

9@

• ' , -,,

'IK6

• +M:%%: N

0 !- 0!'I: K6µ

• =

=

N i i MLE

x N

1

1 ˆ µ

MLE

µ ˆ

µ µ µ = = =

• =
• =

= = N i N i i N i i MLE

N x E N x N E E

1 1 1

1 ] [ 1 1 ] ˆ [

9B

(*)

$(' $(',)+!)

': !,≠ Z 3-+!

Homework#1 !!

9E

/1

0 $V$('0 $

$ $

9H

'*$+)

)+M1%%1N%$(' !

T i i i MLE i i MLE

x x N x N ) ˆ ( ) ˆ ( 1 ˆ 1 ˆ µ µ µ − ⋅ − = Σ =

@D

*, )

P(µ |η,λ) 2λ η

?*

$) &!/ "-

5,

@

$A5,,)

) , | ( ) , | ( ) , , , | ( λ η µ σ µ α λ η σ µ P D P D P

2 ) (

2

) ( ) ( ln ) | ( ln ) ( ) | ( ln

λ η µ

σ µ µ µ µ µ µ µ µ

−

− − − = + =

• i

i

x P d d D P d d P D P d d

• +
• +
• =
• =
• 2

2 2 2

1 ˆ ... λ σ λ η σ µ N x

i i MAP

∝

@

$A5,,)

,#0*%λ → ∞

$A5$('Z

,λ P∞%

$A5/') A&'A)', $('; < η

• +
• +
• =
• 2

2 2 2

1 ˆ λ σ λ η σ µ N x

i i MAP

@

(,)

)

* #

!*: + ",*,)-

+!, 0!+

$ :%# -%!. 2 *,)*

@4

$:,)

/ :-

1

"0*!-,) !!!)%

,)

• -: +0*)
• =

=

= Σ =

K k k K k k k k

x N x p

1 1

1 ) , | ( ) ( π µ π

@9

$:,)':

@@

/#0

5--D 5'

$(' ,,* 85A *

%"! - )%

$A5

@B

@E

!

• 6-

∗ -6 F -

1 lnθ θ θ ∂ = ∂ 1 ln (1 ) (1 ) θ θ θ ∂ − − = ∂ −

@H

!!! ,+-

A!!*:%%: S%%S

• -5S%W65S5WJS

5SJW65WJS5SC5W

5S%%S6

5S5SJS… 5S#JS%%S#7… 5SJS%%S7

BD

• +> 8

S0,%+! &!*+!!,* 2

) ( ) | ) ( (| c X Var c X E X P ≤ ≥ −

B

+*!,2 $

A -+> 8

• 2

) ( ) | ) ( (| c X Var c X E X P ≤ ≥ − n X Var X Var n n X Var X Var

i i i i

) ( ) ( 1 ) (

2

= =

• =
• )

( lim ) | ) ( (| lim

2

= ≤ ≥ −

∞ → ∞ →

nc X Var c X E X P

n n

B

&-

'+! !G 0#--

A*%)%

+-,=-

)6A ,+MωΩ,)ω6AN

5 ,+%S

&S 6 -+,+ S !!*!6J&SJ5S6:6 !: S6::

6

B

$+)':