The EM Algorithm for Positive Linear Inverse Problems Bernard A. Mair - PowerPoint PPT Presentation

The EM Algorithm for Positive Linear Inverse Problems Bernard A. Mair Department of Mathematics University of Florida

Outline • Finite Dimensional Positive Linear Inverse Problems • Finite Dimensional EM Algorithm • Positive Linear Inverse Problems • The Infinite Dimensional EM Algorithm • Previous Convergence Results • New Developments

Finite Dimensional Poisson Linear Inverse Problems = = ≥ , where ( ) , , 0 y P x P p x y × , i j I J = T Data: [ , ,..., ] , where ~ Poiss[ ] d d d d d y 1 2 I i i ∑ = = … Normalization: 1, for 1,2, , p j J , i j i Application: Emission Tomography Maximum Likelihood Estimation Gaussian Errors ⇒ Least Squares Minimization Poisson Errors ⇒ Kullback-Leibler Minimization

PET Physics detector ring detector tube Emission-detection model Each annihilation results in 2 photons traveling in (nearly) opposite directions along a uniformly random line

Maximum Likelihood Estimation = log ( | ) Prob d y P x ∑ = − + − ( [ ] log [ ] !) Px d Px d i i i i i ∑ = − − + + ( log( [ ] ) [ ] ) d d Px d Px c i i i i i i = − + + � ( | ) ( ) KL d P x c L x c � ˆ MLE: arg max ( ) x x L ≥ x 0 = arg min ( | ) KL d P x ≥ x 0

Finite Dimensional EM Algorithm Analytic Derivation ∇ = ∇ ≤ � ˆ ˆ ˆ KT optimality: ( ) ; ( ) x L x 0 L x 0 d p ∑ + = > ( 1) ( ) i ij (0) n n EM Algorithm: , 0, flat x x x j j ( ) n [ ] Px i i ≥ For any data vector : d 0 ∑ ∑ + > = ≥ ( ) ( ) ( 1) ( ) n n n n (1) 0; ; ( ) ( ) x x d L x L x j i j i ( ) n (2) { } converges to a MLE (not unique) x L. Shepp and Y . Vardi (1982), IEEE Trans. Med. Imag..

Image De ‐ blurring Noisy data ⇒ EM converges to noisy image ⇒ Regularization needed − β � ˆ arg max ( ) ( ) x L x R x Penalized MLE: ≥ R x 0 � Automatic choice of β , R � Fast optimization algorithms Early Stopping of EM � Automatic choice of stopping iteration • EM converges rapidly to exact image if data is exact

Positive Linear Inverse Problems Vardi and Lee (1993), From image deblurring to optimal investments: maximum likelihood solutions for positive linear inverse problems, J. Royal Statist. Soc., B. ∫ = � ( ) ( , ) ( ) ( ), where g s p s x f x dx Pf s ∫ ≥ = , , 0; ( , ) 1, for all p f g p s x ds x ≈ � Given , estimate . g g f I nfinite Dimensional EM Algorithm � ( ) ( , ) g s p s x ∫ λ = λ λ > ( ) ( ) ; 0, const. x x ds + λ 1 0 n n ( ) P s n Kondor (1983), Nuclear Instruments and Methods. (MCW for f ,g on [0,1])

ML Estimation ∫ = ∈Σ � ( ) ( , ) ( ) ( ), ,where g s p s x f x dx Pf s s Ω ∫ ≥ = ∈Ω 0, bdd., ( , ) 1, for all p p s x ds x Σ ≈ ∈ Σ � 1 Given ( ), estimate . g g L f − � � ( ) ( | ) L f KL g Pf ( ) ( ) ( ) ∫ = − ⎡ − + ⎤ � � � log( ( )) ( ) g s g s Pf s g s Pf s ds ⎣ ⎦ Σ { } ∈ Ω ≥ = � 1 Max. ( ) over M ( ) : 0, L f f L f f g 0 1 1

ML Feasibility Set { } ∈ Ω ≥ = � 1 Max. ( ) over M ( ) : 0, L f f L f f g 0 1 1 Reasons: = ⇒ = = i Pf g Pf f g 1 1 1 i EM iterates satisfy the constraint. ( ) ( ) = − ∫ ∈ ⇒ � � M ( ) log( ( )) f L f g s g s Pf s ds 0 Σ • Is this norm constraint necessary? • Does not exist in finite dimensional case. • Does max. without constraint satisfy the constraint?

EM Convergence � ( ) ( , ) g s p s x ∫ λ = λ λ > λ ∈ ∞ Ω ( ) ( ) ; 0, ( ) x x ds L + λ 1 0 0 n n ( ) P s Ω n [Multhei and Schorr (1987,89,93)]: ∞ λ > λ ∈ Ω λ = (1) 0, ( ), L g n n n 1 1 { } λ (2) ( ) is incr. and convergent. L n λ → λ λ * 1 * (3) If in , is a max. of over . L L M 0 n Resmerita, Engl, Iusem (2007) = = λ � (4) If and has bdd. sol'n, then { } has a g g Pf g n ≥ p subseq. which conv. weakly in , 1. L p

Infinite Dimensional EM Algorithm: Issues • Existence of maximizer of L. • Definition of feasible set – different than finite dimensional case. • Convergence of EM algorithm for noisy data.

New Results All issues resolved (partially) in semi- infinite dimensional model � Existence of maximizer of L. � Definition of feasible set – similar to finite dimensional case. � Convergence of (subsequence of) EM iterates for noisy data.

Semi ‐ infinite Model ∫ = = � … ( ) ( ) , 1,2, , ,where g p x f x dx Pf i I i i i Ω I ∑ ≥ = ∈Ω 0, cont., ( ) 1, for all p p x x i i = 1 i ≈ ∈ � � I Given , estimate . g g f + Ω Extend to the set of finite Borel measures on . P S I ∑ λ − λ = − λ − + λ � � � � � ( ) ( | ) [ log( ) ] L KL g P g g P g P i i i i i = 1 i λ MLE: Maximize ( ) over . L S

Existence of MLE ∃ ⊂ λ = λ (1) wkly-cpct ( ) , s.t.sup ( ) sup ( ) S t S L L λ ∈ λ ∈ ( ) S S t λ ˆ = λ (2) arg max ( ) exists and satisfies L λ ∈ S ∑ ∑ ˆ ˆ λ Ω = λ ≤ ∈Ω � � ( ) , ( ) 1, for all g g p x P x i i i i i i ∑ ˆ λ = λ = λ ∈ λ Ω = � (3) arg max ( ) where { : ( ) } L S S g λ ∈ 0 S i 0 i λ ˆ (4) is MLE iff ∑ ˆ λ ≤ ∈Ω � ( ) ( ) 1, for all and i g p x P x i i i i ∑ � ˆ ˆ λ = λ − ∈Ω ( ) ( ) 1, for a.e. ii g p x P x i i i i Mair et al (1996), Inverse Problems

Semi ‐ infinite EM Algorithm ∑ � ∞ λ = λ λ λ > ( ) ( ) ( ) [ ] , 0, in x x g p x P L + 1 0 n n i i n i i Assumptions: i No is identically zero. p i ∞ → ∞ → ∞ i If , then [ ] for some . f Pf i n n i Results: ∗ ∞ λ λ ∈ i 1 { } has a subseq. conv. weakly in to some . L L n λ λ ∗ i { ( )} incr. to ( ) L L n λ ∗ λ i { ( | )} is a decreasing sequence. KL n ∗ ∗ λ → λ λ i 1 If { } in , is an MLE (w.r.t. ). L S n

END