Phase Transitions in Semidefinite Relaxations Andrea Montanari - - PowerPoint PPT Presentation

Phase Transitions in Semidefinite Relaxations

Andrea Montanari

[with Adel Javanmard, Federico Ricci-Tersenghi, Subhabrata Sen]

Stanford University

December 7, 2015

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 1 / 75

What is this talk about?

SDP for Matrix/Graph estimation

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 2 / 75

The hidden partition model

Vertices V , ❥V ❥ ❂ n, V ❂ V✰ ❬ V, ❥V✰❥ ❂ ❥V❥ ❂ n❂2 P

✟✭i❀ j ✮ ✷ E ✠ ❂ ✭

p if ❢i❀ j ❣ ✒ V✰ or ❢i❀ j ❣ ✒ V, q ❁ p

• therwise.

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 3 / 75

Of course entries are not colored. . .

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 4 / 75

. . . and rows/columns are not ordered

Problem: Detect/estimate the partition

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 5 / 75

What is this talk about?

SDP for Matrix/Graph estimation Exact phase transition(?)

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 6 / 75

Outline

1

Background

2

Near-optimality of SDP

3

How does SDP work ‘in practice’?

4

Proof ideas

5

Conclusion

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 7 / 75

Background

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 8 / 75

Hypothesis testing

Hypothesis H0: P

✟✭i❀ j ✮ ✷ E ✠ ❂ p ✰ q

2 Hypothesis H1: V ❂ V✰ ❬ V, ❥V✰❥ ❂ ❥V❥ ❂ n❂2 P

✟✭i❀ j ✮ ✷ E ✠ ❂ ✭

p if ❢i❀ j ❣ ✒ V✰ or ❢i❀ j ❣ ✒ V, q ❁ p

• therwise.

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 9 / 75

Hypothesis testing (p ❂ a❂n, q ❂ b❂n)

Hypothesis H0: P

✟✭i❀ j ✮ ✷ E ✠ ❂ a ✰ b

2n Hypothesis H1: V ❂ V✰ ❬ V, ❥V✰❥ ❂ ❥V❥ ❂ n❂2 P

✟✭i❀ j ✮ ✷ E ✠ ❂ ✭

a❂n if ❢i❀ j ❣ ✒ V✰ or ❢i❀ j ❣ ✒ V, b❂n

• therwise.

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 10 / 75

Information theory threshold

Theorem (Mossel, Neeman, Sly, 2012)

There is a test that succeed with high probability if and only if a ✰ b ❃ 2 and a b

♣

2✭a ✰ b✮ ❃ 1 ✿

[Proves conjecture by Decelle, Krzakala, Moore, Zdeborova, 2011]

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 11 / 75

Information theory threshold

Theorem (Mossel, Neeman, Sly, 2012)

There is a test that succeed with high probability if and only if a ✰ b ❃ 2 and a b

♣

2✭a ✰ b✮ ❃ 1 ✿

[Proves conjecture by Decelle, Krzakala, Moore, Zdeborova, 2011]

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 11 / 75

Computational threshold

◮ Dyer, Frieze 1989

p ❂ na ❃ q ❂ nb fixed.

◮ Condon, Karp 2001

a b ✢ n1❂2

◮ McSherry 2001

a b ✢ ♣b log n

◮ Coja-Oghlan 2010

a b ✢ ♣ b

◮ Massoulie 2013 and Mossel, Neeman, Sly, 2013

a b

♣

2✭a ✰ b✮ ❃ 1 Very ingenious spectral methods!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 12 / 75

Computational threshold

◮ Dyer, Frieze 1989

p ❂ na ❃ q ❂ nb fixed.

◮ Condon, Karp 2001

a b ✢ n1❂2

◮ McSherry 2001

a b ✢ ♣b log n

◮ Coja-Oghlan 2010

a b ✢ ♣ b

◮ Massoulie 2013 and Mossel, Neeman, Sly, 2013

a b

♣

2✭a ✰ b✮ ❃ 1 Very ingenious spectral methods!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 12 / 75

Computational threshold

◮ Dyer, Frieze 1989

p ❂ na ❃ q ❂ nb fixed.

◮ Condon, Karp 2001

a b ✢ n1❂2

◮ McSherry 2001

a b ✢ ♣b log n

◮ Coja-Oghlan 2010

a b ✢ ♣ b

◮ Massoulie 2013 and Mossel, Neeman, Sly, 2013

a b

♣

2✭a ✰ b✮ ❃ 1 Very ingenious spectral methods!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 12 / 75

What if I am not ingenious?

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 13 / 75

Maximum Likelihood

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 14 / 75

Adjacency matrix

Aij ❂

✭

1 if ✭i❀ j ✮ ✷ E,

• therwise.

A ❂ ✭Aij ✮1✔i❀j ✔n

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 15 / 75

Maximum likelihood

✛i ❂

✭

✰1 if i ✷ V✰, 1 if i ✷ V. maximize

n

❳

i❀j ❂1

Aij ✛i✛j ❀ subject to

n

❳

i❂1

✛i ❂ 0 ❀ ✛i ✷ ❢✰1❀ 1❣ ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 16 / 75

Maximum likelihood

✛i ❂

✭

✰1 if i ✷ V✰, 1 if i ✷ V. maximize

n

❳

i❀j ❂1

Aij ✛i✛j ❀ subject to

n

❳

i❂1

✛i ❂ 0 ❀ ✛i ✷ ❢✰1❀ 1❣ ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 16 / 75

Maximum likelihood

✛i ❂

✭

✰1 if i ✷ V✰, 1 if i ✷ V. maximize

n

❳

i❀j ❂1

Aij ✛i✛j ❀ subject to

n

❳

i❂1

✛i ❂ 0 ❀ ✛i ✷ ❢✰1❀ 1❣ ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 16 / 75

Lagrangian

maximize

n

❳

i❀j ❂1

Aij ✛i✛j ✌

✏ n ❳

i❂1

✛i

✑2

✿ subject to ✛i ✷ ❢✰1❀ 1❣ ✿ A good choice: ✌ ❂ a ✰ b 2n ✑ d n

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 17 / 75

Lagrangian

maximize

n

❳

i❀j ❂1

Aij ✛i✛j ✌

✏ n ❳

i❂1

✛i

✑2

✿ subject to ✛i ✷ ❢✰1❀ 1❣ ✿ A good choice: ✌ ❂ a ✰ b 2n ✑ d n

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 17 / 75

Centered adjacency matrix

Acen

ij

❂

✭

1 ✭d❂n✮ if ✭i❀ j ✮ ✷ E, ✭d❂n✮

• therwise.

Acen ❂ A d n 1 1T

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 18 / 75

Lagrangian

maximize ❤Acen❀ σσT✐ ❀ subject to σ ✷ ❢✰1❀ 1❣n ✿

◮ NP-hard ◮ SDP✭Acen✮ is a very natural convex relaxation

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 19 / 75

Lagrangian

maximize ❤Acen❀ σσT✐ ❀ subject to σ ✷ ❢✰1❀ 1❣n ✿

◮ NP-hard ◮ SDP✭Acen✮ is a very natural convex relaxation

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 19 / 75

Relaxation

maximize ❤Acen❀ σσT✐ ❀ subject to σ ✷ ❢✰1❀ 1❣n ✿

SDP✭A

cen✮:

maximize ❤Acen❀ X ✐ ❀ subject to X ✷ Rn✂n❀ X ✗ 0 ❀ Xii ❂ 1 ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 20 / 75

TSDP✭G✮ ❂

✭

1 if SDP✭Acen

G ✮ ✕ ✒✄,

• therwise.

◮ This is really off-the-shelf ◮ How well does it work?

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 21 / 75

Near-optimality of SDP

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 22 / 75

Before we pass to SDP

◮ What’s the problem with sparse graphs? ◮ What’s the problem vanilla PCA?

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 23 / 75

Why PCA?

Ground truth x0❀i ❂

✭

✰1 if i ✷ V✰, 1 if i ✷ V. Data ❂ RankOne ✰ Wigner 1 ♣ d Acen ❂ ✕ n x0x0T ✰ W ❀ ✕ ✑ a b

♣

2✭a ✰ b✮ E❢Wij ❣ ❂ 0 ❀ E❢W 2

ij ❣ ✷

♥ a

dn ❀ b dn

♦

✙ 1 n ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 24 / 75

Why PCA?

Ground truth x0❀i ❂

✭

✰1 if i ✷ V✰, 1 if i ✷ V. Data ❂ RankOne ✰ Wigner 1 ♣ d Acen ❂ ✕ n x0x0T ✰ W ❀ ✕ ✑ a b

♣

2✭a ✰ b✮ E❢Wij ❣ ❂ 0 ❀ E❢W 2

ij ❣ ✷

♥ a

dn ❀ b dn

♦

✙ 1 n ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 24 / 75

The right parametrization

d ❂ a ✰ b 2 ❀ ✕ ❂ a b

♣

2✭a ✰ b✮

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 25 / 75

Naive PCA

❜

xPCA✭Acen✮ ❂ ♣n v 1✭Acen✮ ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 26 / 75

Does it work?

1 ♣ d Acen ❂ ✕ n x0x0T ✰ W Naive idea: ❦W ❦2 ✔ const✿❀

✌ ✌ ✌ ✕

n x0x0T✌

✌ ✌

2 ❂ ✕

✮ Works for ✕ ❂ O✭1✮ Wrong!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 27 / 75

Does it work?

1 ♣ d Acen ❂ ✕ n x0x0T ✰ W Naive idea: ❦W ❦2 ✔ const✿❀

✌ ✌ ✌ ✕

n x0x0T✌

✌ ✌

2 ❂ ✕

✮ Works for ✕ ❂ O✭1✮ Wrong!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 27 / 75

Spectral relaxation bad in the sparse regime!

3 2 1 1 2 3

eigenvalues/

pγ 0.0 0.1 0.2 0.3 0.4 0.5 0.6 3 2 1 1 2 3

eigenvalues/

pγ 0.0 0.1 0.2 0.3 0.4 0.5 0.6

d ❂ 1✿5 d ❂ 15

Theorem (Krivelevich, Sudakov 2003+Vu 2005)

With high probability, ✕max✭Acen❂ ♣ d✮ ❂

✭

2 ✭1 ✰ o✭1✮✮ if d ✢ ✭log n✮4❀ C

♣

log n❂✭log log n✮✭1 ✰ o✭1✮✮ if d ❂ O✭1✮✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 28 / 75

Example: d ❂ 20, ✕ ❂ 1✿2, n ❂ 104

2000 4000 6000 8000 10000 0.04 0.03 0.02 0.01 0.00 0.01 0.02 0.03 0.04

v 1✭Acen✮

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 29 / 75

Example: d ❂ 3, ✕ ❂ 1✿2, n ❂ 104

2000 4000 6000 8000 10000 0.10 0.05 0.00 0.05 0.10

v 1✭Acen✮

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 30 / 75

Why should SDP work better?

maximize ❤Acen❀ X ✐ ❀ subject to X ✷ Rn✂n❀ X ✗ 0 ❀ Xii ❂ 1 ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 31 / 75

Recall the ultimate limit

G✭n❀ d❀ ✕✮ graph distribution with parameters d ❂ a ✰ b 2 ❃ 1 ❀ ✕ ❂ a b

♣

2✭a ✰ b✮

Theorem (Mossel, Neeman, Sly, 2012)

If ✕ ❁ 1, then lim sup

n✦✶

✌ ✌G✭n❀ d❀ 0✮ G✭n❀ d❀ ✕✮ ✌ ✌

TV ❁ 1 ✿

If ✕ ❃ 1, then lim

n✦✶

✌ ✌G✭n❀ d❀ 0✮ G✭n❀ d❀ ✕✮ ✌ ✌

TV ❂ 1 ✿ Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 32 / 75

SDP has nearly optimal threshold

Theorem (Montanari, Sen 2015)

Assume G ✘ G✭n❀ d❀ ✕✮. If ✕ ✔ 1, then, with high probability, 1 n ♣ d SDP✭Acen

G ✮ ❂ 2 ✰ od✭1✮ ✿

If ✕ ❃ 1, then there exists ✁✭✕✮ ❃ 0 such that, with high probability, 1 n ♣ d SDP✭Acen

G ✮ ❂ 2 ✰ ✁✭✕✮ ✰ od✭1✮ ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 33 / 75

Consequence

TSDP✭G✮ ❂

✭

1 if SDP✭Acen

G ✮ ✕ ✭2 ✰ ✍✮n

♣ d,

• therwise.

Corollary (Montanari, Sen 2015)

Assume ✕ ✕ 1 ✰ ✧. Then there exists d0✭✧✮ and ✍✭✧✮ such that the SDP-based test succeeds with high probability, provided d ✕ d0✭✧✮. Namely lim

n✦✶

✂P0✭TSDP✭G✮ ❂ 1✮ ✰ P1✭TSDP✭G✮ ❂ 0✮ ✄ ❂ 0 ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 34 / 75

Earlier/related work

Optimal spectral tests

◮ Massoulie 2013 ◮ Mossel, Neeman, Sly, 2013 ◮ Bordenave, Lelarge, Massoulie, 2015

SDP, d ❂ ✂✭log n✮

◮ Abbe, Bandeira, Hall 2014 ◮ Hajek, Wu, Xu 2015

SDP, detection

◮ Guédon, Vershynin, 2015 (requires ✕ ✕ 104, very different proof)

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 35 / 75

How does SDP work ‘in practice’?

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 36 / 75

Thresholds

◮ ✕opt c ✭d✮ ✑ Threshold for optimal test ◮ ✕SDP c

✭d✮ ✑ Threshold for SDP-based test

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 37 / 75

What we know

◮ ✕opt c ✭d✮ ❂ 1

[Mossel, Neeman, Sly, 2013]

◮ ✕SDP c

✭d✮ ❂ 1 ✰ od✭1✮

[Montanari, Sen, 2015]

How big is the od✭1✮ gap?

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 38 / 75

What we know

◮ ✕opt c ✭d✮ ❂ 1

[Mossel, Neeman, Sly, 2013]

◮ ✕SDP c

✭d✮ ❂ 1 ✰ od✭1✮

[Montanari, Sen, 2015]

How big is the od✭1✮ gap?

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 38 / 75

Simulations: d ❂ 5, Nsample ❂ 500 (with Javanmard and Ricci)

0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

(a−b)/

q

2(a +b)

0.0 0.2 0.4 0.6 0.8 1.0

Overlap GOE theory n =2,000 n =4,000 n =8,000 n =16,000

SDP estimator ˆ x SDP ✷ ❢✰1❀ 1❣n Overlapn✭❜ x✮ ❂ 1 n E

✟☞ ☞❤ˆ

x SDP✭G✮❀ x0✐

☞ ☞✠ ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 39 / 75

Simulations: d ❂ 5, Nsample ❂ 500

0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

(a−b)/

q

2(a +b)

0.0 0.2 0.4 0.6 0.8 1.0

Overlap GOE theory n =2,000 n =4,000 n =8,000 n =16,000

✕SDP

c

✭d ❂ 5✮ ✙ 1 ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 40 / 75

Simulations: d ❂ 10, Nsample ❂ 500

0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

a−b/

q

2(a +b)

0.0 0.2 0.4 0.6 0.8 1.0

Overlap GOE theory n =2,000 n =4,000 n =8,000 n =16,000

✕SDP

c

✭d ❂ 10✮ ✙ 1 ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 41 / 75

Simulations: d ❂ 10, Nsample ❂ 500

0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

a−b/

q

2(a +b)

0.0 0.2 0.4 0.6 0.8 1.0

Overlap GOE theory n =2,000 n =4,000 n =8,000 n =16,000

How to estimate ✕SDP

c

✭d✮ from data?

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 42 / 75

A technique from physics: Binder cumulant

Q✭G✮ ✑ 1 n ❤ˆ x SDP✭G✮❀ x0✐ ❀ Bind✭n❀ ✕❀ d✮ ✑ E

✟Q✭G✮4✠

E

✟Q✭G✮2✠2

CLT heuristics lim

n✦✶ Bind✭n❀ ✕❀ d✮ ❂

✭

3 if ✕ ❁ ✕SDP

c

✭d✮, 1 if ✕ ❃ ✕SDP

c

✭d✮.

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 43 / 75

A technique from physics: Binder cumulant

Q✭G✮ ✑ 1 n ❤ˆ x SDP✭G✮❀ x0✐ ❀ Bind✭n❀ ✕❀ d✮ ✑ E

✟Q✭G✮4✠

E

✟Q✭G✮2✠2

CLT heuristics lim

n✦✶ Bind✭n❀ ✕❀ d✮ ❂

✭

3 if ✕ ❁ ✕SDP

c

✭d✮, 1 if ✕ ❃ ✕SDP

c

✭d✮.

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 43 / 75

Simulations: d ❂ 5, Nsample ✕ 105!

1 1.5 2 2.5 3 0.7 0.8 0.9 1 1.1 1.2 Binder λ n=2000 n=4000 n=8000

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 44 / 75

Zoom

(✘ 2 years CPU time)

1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 1 1.005 1.01 1.015 1.02 1.025 1.03 d = 5 Binder λ n=2000 n=4000 n=8000 n=16000

Estimate ✕SDP

c

✭d✮ by the crossing point

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 45 / 75

Zoom

(✘ 2 years CPU time)

1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 1 1.005 1.01 1.015 1.02 1.025 1.03 d = 5 Binder λ n=2000 n=4000 n=8000 n=16000

Estimate ✕SDP

c

✭d✮ by the crossing point

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 45 / 75

✕

SDP

c ✭d✮

1 10 0.99 1.00 1.01 1.02

✕SDP

c

✭d✮ d

◮ Dots: Numerical estimates ◮ Line: Non-rigorous analytical approximation

(using statistical physics)

◮ At most 2✪ sub-optimal!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 46 / 75

One last question

Is this approach robust to model miss-specifications?

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 47 / 75

An experiment

◮ Select S ✒ V uniformly at random. with ❥S❥ ❂ n☛. ◮ For each i ✷ S, connect all of its neighbors.

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 48 / 75

An experiment

0.8 1 1.2 1.4 1.6 1.8 2 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SDP, α = 0 BH, α = 0 SDP, α = 0.025 BH, α = 0.025 SDP, α = 0.05 BH, α = 0.05

◮ Solid line:SDP ◮ Dashed line: Spectral

(Non-backtracking walk [Krzakala, Moore, Mossel, Neeman, Sly, Zdeborova, Zhang, 2013])

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 49 / 75

A simple robustness result

Lemma (Montanari, Sen, 2015)

If ❢ G is obtained from the hidden partition model by flipping at most n✧ edges, then

☞ ☞⑦

✕SDP

c

✭d✮ ✕SDP

c

✭d✮

☞ ☞ ✔ ✍✭✧✮ ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 50 / 75

Proof ideas

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 51 / 75

What we want to prove

Theorem (Montanari, Sen 2015)

Assume G ✘ G✭n❀ d❀ ✕✮. If ✕ ✔ 1, then, with high probability, 1 n ♣ d SDP✭Acen

G ✮ ❂ 2 ✰ od✭1✮ ✿

If ✕ ❃ 1, then there exists ✁✭✕✮ ❃ 0 such that, with high probability, 1 n ♣ d SDP✭Acen

G ✮ ❂ 2 ✰ ✁✭✕✮ ✰ od✭1✮ ✿

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 52 / 75

Strategy

• I. Prove equivalence to Gaussian model
• II. Analyze Gaussian model

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 53 / 75

Strategy

• I. Prove equivalence to Gaussian model
• II. Analyze Gaussian model

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 54 / 75

Gaussian model: x0 ✷ ❢✰1❀ 1❣n

B✭✕✮ ✑ ✕ n x0x0T ✰ W ✿ W ✘ GOE✭n✮:

◮ ✭Wij ✮i❁j ✘iid N✭0❀ 1❂n✮ ◮ W ❂ W T ◮ A lot is known about spectral properties of B

Need to characterize the SDP value with Gaussian data

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 55 / 75

Gaussian model: x0 ✷ ❢✰1❀ 1❣n

B✭✕✮ ✑ ✕ n x0x0T ✰ W ✿ W ✘ GOE✭n✮:

◮ ✭Wij ✮i❁j ✘iid N✭0❀ 1❂n✮ ◮ W ❂ W T ◮ A lot is known about spectral properties of B

Need to characterize the SDP value with Gaussian data

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 55 / 75

Gaussian model: x0 ✷ ❢✰1❀ 1❣n

B✭✕✮ ✑ ✕ n x0x0T ✰ W ✿ W ✘ GOE✭n✮:

◮ ✭Wij ✮i❁j ✘iid N✭0❀ 1❂n✮ ◮ W ❂ W T ◮ A lot is known about spectral properties of B

Need to characterize the SDP value with Gaussian data

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 55 / 75

Notation

s✭✕✮ ✑ lim sup

n✦✶

1 n SDP✭B✭✕✮✮ ❀ s✭✕✮ ✑ lim inf

n✦✶

1 n SDP✭B✭✕✮✮ ✿ SDP✭B✮ ✑ max

✟❤B❀ X ✐ ✿ X ✗ 0❀ Xii ❂ 1 ✽i ✠

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 56 / 75

Phase transition at ✕ ❂ 1!

Theorem (Montanari, Sen, 2015)

The following holds almost surely ✕ ✷ ❬0❀ 1❪ ✮ s✭✕✮ ❂ s✭✕✮ ❂ 2 ❀ ✕ ✷ ✭1❀ ✶✮ ✮ s✭✕✮ ❃ 2 (strictly) ✿

For explicit probability bounds, see the paper

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 57 / 75

Proof of Gaussian phase transition

Simple facts:

◮

s✭✕✮ ✔ lim

n✦✶ ✛max✭B✭✕✮✮ ❂

✭

2 if ✕ ✷ ❬0❀ 1❪, ✕ ✰ ✕1 if ✕ ✷ ✭1❀ ✶✮.

[Baik, Ben Arous, Peche, 2005]

◮

s✭✕✮ ✔ lim

n✦✶

1 n ❤1❀ B✭✕✮1✐ ❂ ✕

◮ s✭✕✮, s✭✕✮ are non-random, non-decreasing

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 58 / 75

Proof of Gaussian phase transition

Simple facts:

◮

s✭✕✮ ✔ lim

n✦✶ ✛max✭B✭✕✮✮ ❂

✭

2 if ✕ ✷ ❬0❀ 1❪, ✕ ✰ ✕1 if ✕ ✷ ✭1❀ ✶✮.

[Baik, Ben Arous, Peche, 2005]

◮

s✭✕✮ ✔ lim

n✦✶

1 n ❤1❀ B✭✕✮1✐ ❂ ✕

◮ s✭✕✮, s✭✕✮ are non-random, non-decreasing

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 58 / 75

Proof of Gaussian phase transition

Simple facts:

◮

s✭✕✮ ✔ lim

n✦✶ ✛max✭B✭✕✮✮ ❂

✭

2 if ✕ ✷ ❬0❀ 1❪, ✕ ✰ ✕1 if ✕ ✷ ✭1❀ ✶✮.

[Baik, Ben Arous, Peche, 2005]

◮

s✭✕✮ ✔ lim

n✦✶

1 n ❤1❀ B✭✕✮1✐ ❂ ✕

◮ s✭✕✮, s✭✕✮ are non-random, non-decreasing

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 58 / 75

Proof of Gaussian phase transition

Simple facts:

◮

s✭✕✮ ✔ lim

n✦✶ ✛max✭B✭✕✮✮ ❂

✭

2 if ✕ ✷ ❬0❀ 1❪, ✕ ✰ ✕1 if ✕ ✷ ✭1❀ ✶✮.

[Baik, Ben Arous, Peche, 2005]

◮

s✭✕✮ ✔ lim

n✦✶

1 n ❤1❀ B✭✕✮1✐ ❂ ✕

◮ s✭✕✮, s✭✕✮ are non-random, non-decreasing

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 58 / 75

Summarizing

0.0 0.5 1.0 1.5 2.0 2.5 3.0

λ

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

◮ Red: Upper bound ◮ Blue: Lower bound

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 59 / 75

Proof of Gaussian phase transition

Part 1: Prove that s✭✕ ❂ 0✮ ✕ 2

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 60 / 75

Hence

0.0 0.5 1.0 1.5 2.0 2.5 3.0

λ

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

◮ Red: Upper bound ◮ Blue: Lower bound ◮ Purple: Non-trivial lower bound

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 61 / 75

Proof of Gaussian phase transition

Part 1: Prove that s✭✕ ❂ 0✮ ✕ 2 Part 2: Prove that s✭✕ ❂ 1 ✰ ✧✮ ❃ 2

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 62 / 75

Hence

0.0 0.5 1.0 1.5 2.0 2.5 3.0

λ

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

◮ Red: Upper bound ◮ Blue: Lower bound ◮ Purple: Non-trivial lower bound

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 63 / 75

Proof of Gaussian phase transition

Part 1: Prove that s✭✕ ❂ 0✮ ✕ 2 Part 2: Prove that s✭✕ ❂ 1 ✰ ✧✮ ❃ 2 Technique: Construct feasible X , such that ❤A❀ X ✐ ✕ ✿ ✿ ✿ .

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 64 / 75

Part 1: ✕ ❂ 0

2 −2 Limiting Spectral Density

First idea:

◮ v 1 ❂ v 1✭B✮ ✑ principal eginvector of B ◮ Take X ❂ n v 1v T 1 ◮ Wrong: Xii ✙ N✭0❀ 1✮2 ✻❂ 1

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 65 / 75

Part 1: ✕ ❂ 0

2 −2 Limiting Spectral Density

First idea:

◮ v 1 ❂ v 1✭B✮ ✑ principal eginvector of B ◮ Take X ❂ n v 1v T 1 ◮ Wrong: Xii ✙ N✭0❀ 1✮2 ✻❂ 1

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 65 / 75

Part 1: ✕ ❂ 0

2 −2 Limiting Spectral Density

First idea:

◮ v 1 ❂ v 1✭B✮ ✑ principal eginvector of B ◮ Take X ❂ n v 1v T 1 ◮ Wrong: Xii ✙ N✭0❀ 1✮2 ✻❂ 1

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 65 / 75

Part 1: ✕ ❂ 0

2 −2 Limiting Spectral Density

Good idea:

◮ Let U ❂ ❬v 1❥v 2❥ ✁ ✁ ✁ ❥v n✍❪ ✷ Rn✂n✍, ✍ small. ◮ D ✑ Diag✭U U T✮ ✷ Rn✂n. Claim D ✙ ✍ I (Dii ✘ n1✤n✍) ◮ Set X ❂ D1❂2✭U U T✮D1❂2

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 66 / 75

Part 1: ✕ ❂ 0

2 −2 Limiting Spectral Density

Good idea:

◮ Let U ❂ ❬v 1❥v 2❥ ✁ ✁ ✁ ❥v n✍❪ ✷ Rn✂n✍, ✍ small. ◮ D ✑ Diag✭U U T✮ ✷ Rn✂n. Claim D ✙ ✍ I (Dii ✘ n1✤n✍) ◮ Set X ❂ D1❂2✭U U T✮D1❂2

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 66 / 75

Part 1: ✕ ❂ 0

2 −2 Limiting Spectral Density

Good idea:

◮ Let U ❂ ❬v 1❥v 2❥ ✁ ✁ ✁ ❥v n✍❪ ✷ Rn✂n✍, ✍ small. ◮ D ✑ Diag✭U U T✮ ✷ Rn✂n. Claim D ✙ ✍ I (Dii ✘ n1✤n✍) ◮ Set X ❂ D1❂2✭U U T✮D1❂2

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 66 / 75

Part 2: ✕ ❂ 1 ✰ ✧

2 −2 Limiting Spectral Density

Construction:

◮ T✭x✮ ✑ max✭min✭x❀ ✰1✮❀ 1✮, ϕ ✷ Rn

✬i ✑ T✭✧♣n v1❀i✮ ✿

◮ U ❂ ❬v 2❥v 3❥ ✁ ✁ ✁ ❥v n✍✰1❪ ✷ Rn✂n✍ ◮ D ✷ Rn✂n diagonal Dii ✑

q

1 ✬2

i ❂❦U ei❦2. ◮

X ✑ ϕϕT ✰ DU U TD

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 67 / 75

Part 2: ✕ ❂ 1 ✰ ✧

2 −2 Limiting Spectral Density

Construction:

◮ T✭x✮ ✑ max✭min✭x❀ ✰1✮❀ 1✮, ϕ ✷ Rn

✬i ✑ T✭✧♣n v1❀i✮ ✿

◮ U ❂ ❬v 2❥v 3❥ ✁ ✁ ✁ ❥v n✍✰1❪ ✷ Rn✂n✍ ◮ D ✷ Rn✂n diagonal Dii ✑

q

1 ✬2

i ❂❦U ei❦2. ◮

X ✑ ϕϕT ✰ DU U TD

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 67 / 75

Part 2: ✕ ❂ 1 ✰ ✧

2 −2 Limiting Spectral Density

Construction:

◮ T✭x✮ ✑ max✭min✭x❀ ✰1✮❀ 1✮, ϕ ✷ Rn

✬i ✑ T✭✧♣n v1❀i✮ ✿

◮ U ❂ ❬v 2❥v 3❥ ✁ ✁ ✁ ❥v n✍✰1❪ ✷ Rn✂n✍ ◮ D ✷ Rn✂n diagonal Dii ✑

q

1 ✬2

i ❂❦U ei❦2. ◮

X ✑ ϕϕT ✰ DU U TD

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 67 / 75

Part 2: ✕ ❂ 1 ✰ ✧

2 −2 Limiting Spectral Density

Construction:

◮ T✭x✮ ✑ max✭min✭x❀ ✰1✮❀ 1✮, ϕ ✷ Rn

✬i ✑ T✭✧♣n v1❀i✮ ✿

◮ U ❂ ❬v 2❥v 3❥ ✁ ✁ ✁ ❥v n✍✰1❪ ✷ Rn✂n✍ ◮ D ✷ Rn✂n diagonal Dii ✑

q

1 ✬2

i ❂❦U ei❦2. ◮

X ✑ ϕϕT ✰ DU U TD

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 67 / 75

Part 2: ✕ ❂ 1 ✰ ✧

2 −2 Limiting Spectral Density

Construction:

◮ T✭x✮ ✑ max✭min✭x❀ ✰1✮❀ 1✮, ϕ ✷ Rn

✬i ✑ T✭✧♣n v1❀i✮ ✿

◮ U ❂ ❬v 2❥v 3❥ ✁ ✁ ✁ ❥v n✍✰1❪ ✷ Rn✂n✍ ◮ D ✷ Rn✂n diagonal Dii ✑

q

1 ✬2

i ❂❦U ei❦2. ◮

X ✑ ϕϕT ✰ DU U TD

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 67 / 75

A parenthesis

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 68 / 75

The Gaussian model is very interesting

0.0 0.5 1.0 1.5 2.0 2.5 3.0

λ

0.0 0.2 0.4 0.6 0.8 1.0

MSE Max Likelihood Bayes SDP PCA SDP, n =200 SDP, n =400 SDP, n =800 SDP, n =1600 Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 69 / 75

Strategy

• I. Prove equivalence to Gaussian model
• II. Analyze Gaussian model

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 70 / 75

We want to prove

1 ♣ d SDP✭Acen✮ ✙ SDP✭B✭✕✮✮ Lindeberg method: Replace the entries one-by-one

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 71 / 75

We want to prove

1 ♣ d SDP✭Acen✮ ✙ SDP✭B✭✕✮✮ Lindeberg method: Replace the entries one-by-one

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 71 / 75

A simple Lindeberg lemma

◮ X1❀ X2❀ ✿ ✿ ✿ XM iid

Xi ❂

✽ ❁ ✿

1 ♣ d

✏

1 d

n

✑

with probability d

n ❀

• ♣

d n

with probability

✏

1 d

n

✑

✿ E❢Xi❣ ❂ 0, E❢X 2

i ❣ ❂ ✭1❂n✮ ✭d❂n2✮ ◮ Z1❀ Z2❀ ✿ ✿ ✿ ZM ✘i✿i✿d✿ N✭0❀ 1❂n✮

Lemma

Assume M ❂ n✭n 1✮❂2, F ✷ C3✭RM ✮, d ✔ n2❂3❂10. Then

☞ ☞ ☞EF✭X ✮ EF✭Z✮ ☞ ☞ ☞ ✔

n 3 ♣ d max

i✷❬M❪

✏✌ ✌❅2

i F

✌ ✌

✶ ❴

✌ ✌❅3

i F

✌ ✌

✶

✑

✿ where ❅❵

i F✭x✮ ✑ ❅❵F ❅x ❵

i , and ❦❅❵

i F❦✶ ✑ supx✷RM ❥❅❵ i F✭x✮❥.

Problem: F✭ ✁ ✮ ❂ SDP✭ ✁ ✮ ✻✷ C3✭RM ✮

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 72 / 75

A simple Lindeberg lemma

◮ X1❀ X2❀ ✿ ✿ ✿ XM iid

Xi ❂

✽ ❁ ✿

1 ♣ d

✏

1 d

n

✑

with probability d

n ❀

• ♣

d n

with probability

✏

1 d

n

✑

✿ E❢Xi❣ ❂ 0, E❢X 2

i ❣ ❂ ✭1❂n✮ ✭d❂n2✮ ◮ Z1❀ Z2❀ ✿ ✿ ✿ ZM ✘i✿i✿d✿ N✭0❀ 1❂n✮

Lemma

Assume M ❂ n✭n 1✮❂2, F ✷ C3✭RM ✮, d ✔ n2❂3❂10. Then

☞ ☞ ☞EF✭X ✮ EF✭Z✮ ☞ ☞ ☞ ✔

n 3 ♣ d max

i✷❬M❪

✏✌ ✌❅2

i F

✌ ✌

✶ ❴

✌ ✌❅3

i F

✌ ✌

✶

✑

✿ where ❅❵

i F✭x✮ ✑ ❅❵F ❅x ❵

i , and ❦❅❵

i F❦✶ ✑ supx✷RM ❥❅❵ i F✭x✮❥.

Problem: F✭ ✁ ✮ ❂ SDP✭ ✁ ✮ ✻✷ C3✭RM ✮

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 72 / 75

A simple Lindeberg lemma

◮ X1❀ X2❀ ✿ ✿ ✿ XM iid

Xi ❂

✽ ❁ ✿

1 ♣ d

✏

1 d

n

✑

with probability d

n ❀

• ♣

d n

with probability

✏

1 d

n

✑

✿ E❢Xi❣ ❂ 0, E❢X 2

i ❣ ❂ ✭1❂n✮ ✭d❂n2✮ ◮ Z1❀ Z2❀ ✿ ✿ ✿ ZM ✘i✿i✿d✿ N✭0❀ 1❂n✮

Lemma

Assume M ❂ n✭n 1✮❂2, F ✷ C3✭RM ✮, d ✔ n2❂3❂10. Then

☞ ☞ ☞EF✭X ✮ EF✭Z✮ ☞ ☞ ☞ ✔

n 3 ♣ d max

i✷❬M❪

✏✌ ✌❅2

i F

✌ ✌

✶ ❴

✌ ✌❅3

i F

✌ ✌

✶

✑

✿ where ❅❵

i F✭x✮ ✑ ❅❵F ❅x ❵

i , and ❦❅❵

i F❦✶ ✑ supx✷RM ❥❅❵ i F✭x✮❥.

Problem: F✭ ✁ ✮ ❂ SDP✭ ✁ ✮ ✻✷ C3✭RM ✮

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 72 / 75

Smoothing

SDP✭A✮: maximize

n

❳

i❀j ❂1

Aij ❤σi❀ σj ✐ ❀ subject to σ ❂ ✭σ1❀ σ2❀ ✿ ✿ ✿ ❀ σn✮T ✷ Rn✂n ❀ σi ✷ Rn ❀ ❦σi❦2 ❂ 1 ✿ Free energy ✟k✭☞❀ k❀ A✮ ✑ 1 ☞ log

✽ ❁ ✿ ❩

exp

✏

☞

n

❳

i❀j ❂1

Ai❀j ❤σi❀ σj ✐

✑

✗0❀k✭dσ✮

✾ ❂ ❀

◮ ✗0❀k✭dσ✮ ✑ uniform measure on S k1 ✂ ✁ ✁ ✁ ✂ S k1 ◮ Control ☞ ✦ ✶, k ✦ ✶

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 73 / 75

Smoothing

SDP✭A✮: maximize

n

❳

i❀j ❂1

Aij ❤σi❀ σj ✐ ❀ subject to σ ❂ ✭σ1❀ σ2❀ ✿ ✿ ✿ ❀ σn✮T ✷ Rn✂n ❀ σi ✷ Rn ❀ ❦σi❦2 ❂ 1 ✿ Free energy ✟k✭☞❀ k❀ A✮ ✑ 1 ☞ log

✽ ❁ ✿ ❩

exp

✏

☞

n

❳

i❀j ❂1

Ai❀j ❤σi❀ σj ✐

✑

✗0❀k✭dσ✮

✾ ❂ ❀

◮ ✗0❀k✭dσ✮ ✑ uniform measure on S k1 ✂ ✁ ✁ ✁ ✂ S k1 ◮ Control ☞ ✦ ✶, k ✦ ✶

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 73 / 75

Conclusion

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 74 / 75

Conclusion

◮ SDP ✢ PCA when data are heterogeneous ◮ Sharp information about eigenvalues of random matrices ◮ A lot of work on SDP with random data

[Srebro, Fazel, Parrillo, Candés, Recht, Gross, myself, . . . ]

◮ Little known about ‘sharp SDP properties’

and SDP vs PCA Thanks!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 75 / 75

Conclusion

◮ SDP ✢ PCA when data are heterogeneous ◮ Sharp information about eigenvalues of random matrices ◮ A lot of work on SDP with random data

[Srebro, Fazel, Parrillo, Candés, Recht, Gross, myself, . . . ]

◮ Little known about ‘sharp SDP properties’

and SDP vs PCA Thanks!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 75 / 75

Conclusion

◮ SDP ✢ PCA when data are heterogeneous ◮ Sharp information about eigenvalues of random matrices ◮ A lot of work on SDP with random data

[Srebro, Fazel, Parrillo, Candés, Recht, Gross, myself, . . . ]

◮ Little known about ‘sharp SDP properties’

and SDP vs PCA Thanks!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 75 / 75

Conclusion

◮ SDP ✢ PCA when data are heterogeneous ◮ Sharp information about eigenvalues of random matrices ◮ A lot of work on SDP with random data

[Srebro, Fazel, Parrillo, Candés, Recht, Gross, myself, . . . ]

◮ Little known about ‘sharp SDP properties’

and SDP vs PCA Thanks!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 75 / 75

Conclusion

◮ SDP ✢ PCA when data are heterogeneous ◮ Sharp information about eigenvalues of random matrices ◮ A lot of work on SDP with random data

[Srebro, Fazel, Parrillo, Candés, Recht, Gross, myself, . . . ]

◮ Little known about ‘sharp SDP properties’

and SDP vs PCA Thanks!

Andrea Montanari (Stanford) SDP Phase Transitions December 7, 2015 75 / 75