On projections onto polyhedral sets and applications to primal-dual - - PowerPoint PPT Presentation

SLIDE 1

On projections onto polyhedral sets and applications to primal-dual projection algorithms for solving maximally monotone inclusion problem

Krzysztof Rutkowski

Faculty of Mathematics and Information Science Warsaw University of Technology Systems Research Institute Polish Academy of Sciences

Paris 23.07.2018

SLIDE 2

Table of contents

1

Problem statement Operator inclusions problem Form of projection algorithms Projection onto polyhedral sets

2

Projection algorithms Fej´ er algorithm Best approximation algorithm Best approximation algorithms with memory

3

Extension to systems of monotone inclusions Extended Best Approximation Primal-Dual Algorithm with memory Block-Asynchronous Primal-Dual Algorithm with memory

4

Application

SLIDE 3

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Operator inclusions problem

Let H,G be Hilbert spaces, A : H → H, B : G → G be maximally monotone

• perators and L : H → G be a linear, bounded continuous operator. We are interested

in finding a point p ∈ H which solves the following inclusion problem 0 ∈ Ap + L∗BLp. (P) Within the generalized Fenchel - Rockafellar duality framework the dual inclusion problem to (P) is to find v∗ ∈ G such that 0 ∈ −LA−1(−L∗v∗) + B−1v∗. (D) A point p ∈ H solves (P) if and only if v∗ ∈ G solves (D) and (p, v∗) ∈ Z, where Z := {(p, v∗) ∈ H × G | − L∗v∗ ∈ Ap and Lp ∈ B−1v∗}. Z is a closed convex subset of H × G.

3 / 22

SLIDE 4

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Form of projection algorithms

In this presentation we consider two types of primal-dual projection algorithms: Fejer-type algorithms Haugazeau-type algorithms Fejer-type

1: Let x0 ∈ H × G 2: for n = 0, 1, . . . do 3:

Let Cn be closed convex

4:

such that Z ⊂ Cn

5:

xn+1 = PCn(xn) Haugazeau-type

1: Let x0 ∈ H × G 2: for n = 0, 1, . . . do 3:

Let Cn be closed convex

4:

such that Z ⊂ Cn

5:

xn+1 = PH(x0,xn)∩Cn(x0) where H(a, b) = {h ∈ H × G | h − a | a − b ≤ 0}. Weak convergence of Fejer-type algorithims and strong convergence of Haugazeau-type algorithms to ¯ z ∈ Z can be shown under some assumptions on the choice of Cn. When Cn are polyhedral sets (i.e intersection of halfspaces) exact or closed-form expression formulas need to be used to calculate the projection PCn(·).

4 / 22

SLIDE 5

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Projection onto polyhedral sets

In the literature following formulas for projections are known:

1

When C is a halfspace i.e C = {h ∈ H × G | h | s ≤ η} and x / ∈ C then PC (x) = x − x | s − η s2 s

2

When C is an intersection of two halfspaces closed-form formulas are provided by Bauschke Combettes

Heinz H. Bauschke and Patrick L. Combettes. Convex analysis and monotone operator theory in Hilbert spaces. CMS Books in Mathematics/Ouvrages de Math´ ematiques de la SMC. With a foreword by H´ edy Attouch. Springer, New York, 2011, pp. xvi+468. isbn: 978-1-4419-9466-0. doi: 10.1007/978-1-4419-9467-7. url: http://dx.doi.org/10.1007/978-1-4419-9467-7 3

General form of projection formula onto poyhedral set is given by Deutsch PC (x) = x −

m

• i=1

λisi, where λi ≥ 0, i = 1, . . . , m satisfy x −

m

• j=1

λjsj | si − ηi ≤ 0, i = 1, . . . , m, λi(x −

m

• j=1

λjsj | si − ηi) = 0, i = 1, . . . , m.

5 / 22

SLIDE 6

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

In the paper

Krzysztof E. Rutkowski. “Closed-Form Expressions for Projectors onto Polyhedral Sets in Hilbert Spaces”. In: SIAM Journal on Optimization 27.3 (2017), pp. 1758–1771. doi: 10.1137/16M1087540. eprint: https://doi.org/10.1137/16M1087540. url: https://doi.org/10.1137/16M1087540

we propose the closed-form formulas for general polyhedral in a form Cn = {h ∈ H × G | h | si ≤ ηi, i = 1, . . . , m} The formula requires to check at most 2m − 1 conditions for finding an exact

• projection. The number of conditions to check can be reduced dependently on the

structure of the polyhedral set.

6 / 22

SLIDE 7

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Notation

In the next presented algorithms we will use the following notation: For any monotone operator S : H → H the resolvent operator JS : H → H is given by JS(·) = (Id − S)−1(·). For any (a, a∗) ∈ graA, (b, b∗) ∈ graB, s∗

a,b := (a∗ + L∗b∗, b − La),

ηa,b := a | a∗ + b | b∗, Ha,b := {x ∈ H × G | x | s∗

a,b ≤ ηa,b}.

7 / 22

SLIDE 8

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Fej´ er algorithm

Let x0 = (p0, v∗

0 ) ∈ H × G, ε ∈ (0, 1) and iterate 1: for n = 0, 1, . . . do 2:

Pick (γn, µn) ∈ [ε, 1/ε]2

3:

an = JγnA(pn − γnL∗v∗

n ), a∗ n = 1 γn (pn − γnL∗v∗ n − an) 4:

bn = JµnB(Lpn + µnv∗

n ), b∗ n = 1 µn (Lpn + µnv∗ n − bn) 5:

if s∗

an,bn = 0 then 6:

¯ p = an, ¯ v∗ = b∗

n 7:

terminate

8:

else

9:

(pn+1, v∗

n+1) = PHan,bn (pn, v∗ n )

Theorem Suppose, that set Z is nonempty. Let x0 = (p0, v∗

0 ) ∈ H × G, ε ∈ (0, 1). The

following holds

(i)

(pn, v∗

n )n∈N is Fej´

er monotone with respect to Z,

(ii)

• n∈N pn+1 − pn2 < ∞ and

n∈N v∗ n+1 − v∗ n 2 < ∞,

(iii)

(pn)n∈N converges weakly to a point ¯ p, (vn)n∈N converges weakly to a point ¯ v∗ and (¯ p, ¯ v∗) ∈ Z.

• A. Alotaibi, P. L. Combettes, and N. Shahzad. “Solving Coupled Composite Monotone Inclusions by Successive Fej´

er Approximations of their Kuhn-Tucker Set.”. In: SIAM Journal on Optimization 24.4 (2014), pp. 2076–2095. doi: 10.1137/130950616. eprint: http://dx.doi.org/10.1137/130950616. url: http://dx.doi.org/10.1137/130950616 8 / 22

SLIDE 9

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Best approximation algorithm

Algorithm:

1: for n = 0, 1, . . . do 2:

Find (γn, µn) ∈ [ε, 1/ε]2

3:

an = JγnA(pn − γnL∗v∗

n ), a∗ n = 1 γn (pn − γnL∗v∗ n − an) 4:

bn = JµnB(Lpn + µnv∗

n ), b∗ n = 1 µn (Lpn + µnv∗ n − bn) 5:

if s∗

an,bn = 0 then 6:

¯ p = an, ¯ v∗ = b∗

n 7:

terminate

8:

else

9:

(pn+1, v∗

n+1) = PH(x0,(pn,v∗

n ))∩Han,bn (p0, v∗

0 )

Theorem Suppose, that set Z is nonempty. Let x0 = (p0, v∗

0 ) ∈ H × G, ε ∈ (0, 1). The

following holds

(i)

(pn+1, v∗

n+1) − x0 > (pn, v∗ n ) − x0 for all n = 0, 1 . . . ,

(ii)

• n∈N pn+1 − pn2 < ∞ and

n∈N v∗ n+1 − v∗ n 2 < ∞.

(iii)

(pn)n∈N converges to a point ¯ p, (vn)n∈N converges to a point ¯ v∗ and (¯ p, ¯ v∗) = PZ (x0).

Abdullah Alotaibi, Patrick L. Combettes, and Naseer Shahzad. “Best approximation from the Kuhn-Tucker set of composite monotone inclusions”. In: Numer. Funct. Anal. Optim. 36.12 (2015), pp. 1513–1532. issn: 0163-0563. doi: 10.1080/01630563.2015.1077864. url: http://dx.doi.org/10.1080/01630563.2015.1077864 9 / 22

SLIDE 10

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Best approximation algorithm with memory

1

Algorithm

1: for n = 0, 1, . . . do 2:

Find (γn, µn) ∈ [ε, 1/ε]2

3:

an = JγnA(pn − γnL∗v∗

n ), a∗ n = 1 γn (pn − γnL∗v∗ n − an) 4:

bn = JµnB(Lpn + µnv∗

n ), b∗ n = 1 µn (Lpn + µnv∗ n − bn) 5:

if s∗

an,bn = 0 then 6:

¯ p = an, ¯ v∗ = b∗

n 7:

terminate

8:

else

9:

Let Cn be closed convex set such that Z ⊂ Cn ⊂ Han,bn

10:

(pn+1, v∗

n+1) = PH(x0,(pn,v∗

n ))∩Cn(p0, v∗

0 )

2

Let Hn := Han,bn. Possible Cn choices:

1

Cn := Hn (without memory),

2

Cn := Hn ∩ Hn−1 for n ≥ 1 and C0 = H,

3

Cn := Hn ∩ H(x0, xn−1) for n ≥ 1 and C0 = H,

4

Cn := Hn ∩ H(x0, τnxn + (1 − τn)xn−1) for τn ∈ (0, 1), n ≥ 1 and C0 = H.

• E. M. Bednarczuk, A. Jezierska, and K. E. Rutkowski. “Proximal primal-dual best approximation algorithm with memory”. In: ArXiv

e-prints (Oct. 2016). arXiv: 1610.08697 [math.OC] 10 / 22

SLIDE 11

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Extension to systems of monotone inclusions

Let H1 × · · · × HM, G1 × · · · × GK be Hilbert spaces, E := H1 × · · · × HM × G1 × · · · × GK , Ai : Hi → Hi, Bk : Gk → Gk, i = 1, . . . , M, k = 1, . . . , K be maximally monotone operators and Lik : H → G, i = 1, . . . , M, k = 1, . . . , K be a linear, bounded continuous operators. Natural extension of P is the following 0 ∈ Aipi +

K

• k=1

L∗

kiBk M

• i=1

Likpi, i = 1 . . . , M (P1) Within the generalized Fenchel - Rockafellar duality framework the dual inclusion problem to (P1) is to find (v∗

1 , . . . , v∗ K ) ∈ G1 × · · · × GK such that

0 ∈ −

M

• i=1

LikA−1

i

(−

K

• k=1

L∗

kiv∗ k ) + B−1 k

v∗

k ,

k = 1 . . . , K (D1) A point (p1, . . . , pM) ∈ H1 × · · · × HM solves (P1) if and only if (v∗

1 , . . . , v∗ K ) ∈ G1 × · · · × GK solves (D1) and (p, v∗) ∈ Z, where

Z := {(p1, . . . , pM, v∗

1 , . . . , v∗ K ) ∈ H1 × . . . HM × G1 × · · · × GK |

−

K

• k=1

L∗

kiv∗ k ∈ Aipi,

i = 1 . . . , M

M

• i=1

Likpi ∈ B−1

k

v∗

k ,

k = 1 . . . , K}. (Z1)

11 / 22

SLIDE 12

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Subdifferential Case

Let Ai = ∂fi, i = 1, . . . , M and Bk = ∂gk, k = 1, . . . K, where fi, gk are proper l.s.c convex functions, i = 1, . . . , M, k = 1, . . . K. Then set Z1 have the following representation Z := {(p1, . . . , pM, v∗

1 , . . . , v∗ K ) ∈ H1 × . . . HM × G1 × · · · × GK |

−

K

• k=1

L∗

kiv∗ k ∈ ∂fipi,

i = 1 . . . , M

M

• i=1

Likpi ∈ (∂gk)−1(v∗

k ),

k = 1 . . . , K}. (Z2) If Z = ∅ then (p1, . . . , pM) solve the following primal problem min

p1,...,pM∈H1×...HM M

• i=1

fi(pi) +

K

• k=1

gk M

• i=1

Likpi

• and (v∗

1 , . . . , v∗ K ) solves the dual problem

min

v∗

1 ,...,v∗ K ∈G1×···×GK

M

• i=1

f ∗

i (− K

• k=1

L∗

kiv∗ k ) + K

• k=1

g∗

k (v∗ k )

12 / 22

SLIDE 13

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Extended Best Approximation Primal-Dual Algorithm with memory

1: for n = 0, 1 . . . do 2:

for i = 1, . . . , M do

3:

ai,n = proxγi,ci (n)fi (pi,ci (n) − γi,ci (n) K

k=1 L∗ kiv ∗ k,ci (n))

4:

a∗

i,n = γ−1 i,ci (n)(pi,ci (n) − ai,n) − K k=1 L∗ kiv ∗ k,ci (n)

5:

for k = 1, . . . , K do

6:

bk,n = proxµk,dk (n)gk (M

i=1 Likpi,dk (n) + µk,dk (n)v ∗ k,dk (n))

7:

b∗

k,n = µ−1 k,dk (n)(M i=1 Likpi,dk (n) − bk,n) + v ∗ k,dk (n)

8:

for i ∈ {1, . . . , M} do

9:

s∗

i,n = a∗ i,n + K k=1 L∗ kib∗ k,n

10:

for k ∈ {1, . . . , K} do

11:

s∗

M+k,n = bk,n − M i=1 Likai,n

12:

s∗

n = (s∗ 1,n, . . . , s∗ M,n, s∗ K+1,n, . . . , s∗ K+M,n)

13:

ηn =

i∈An ai,n | a∗ i,n + k∈Bn bk,n | b∗ k,n

14:

Hn = {h ∈ E | h | s∗

n ≤ ηn}

15:

if s∗

n = 0 then

16:

Terminate

17:

else xn+1/2 = PHn (xn)

18:

Take Cn convex, closed s.t. Z ⊂ Cn ⊂ Hn, xn+1 = PH(x0,xn)∩Cn (x0)

13 / 22

SLIDE 14

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Block-Asynchronous Primal-Dual Algorithm with memory

1: for n = 0, 1 . . . do 2:

Find In ⊂ {1, . . . , M}, Kn ⊂ {1, . . . , K}

3:

for i ∈ In do

4:

ai,n = proxγi,ci (n)fi (pi,ci (n) − γi,ci (n) K

k=1 L∗ kiv ∗ k,ci (n))

5:

a∗

i,n = γ−1 i,ci (n)(pi,ci (n) − ai,n) − K k=1 L∗ kiv ∗ k,ci (n)

6:

for i ∈ {1, . . . , M}\In do

7:

(ai,n, a∗

i,n) = (ai,n−1, a∗ i,n−1)

8:

for k ∈ Kn do

9:

bk,n = proxµk,dk (n)gk (M

i=1 Likpi,dk (n) + µk,dk (n)v ∗ k,dk (n))

10:

b∗

k,n = µ−1 k,dk (n)(M i=1 Likpi,dk (n) − bk,n) + v ∗ k,dk (n)

11:

for k ∈ {1, . . . , K}\Kn do

12:

(bk,n, b∗

k,n) = (bk,n−1, b∗ k,n−1)

13:

for i ∈ {1, . . . , M} do

14:

s∗

i,n = a∗ i,n + K k=1 L∗ kib∗ k,n

15:

for k ∈ {1, . . . , K} do

16:

s∗

M+k,n = bk,n − M i=1 Likai,n

17:

s∗

n = (s∗ 1,n, . . . , s∗ M,n, s∗ K+1,n, . . . , s∗ K+M,n)

18:

ηn =

i∈An ai,n | a∗ i,n + k∈Bn bk,n | b∗ k,n

19:

Hn = {h ∈ E | h | s∗

n ≤ ηn}

20:

if s∗

n = 0 then

21:

Terminate

22:

else xn+1/2 = PHn (xn)

23:

Take Cn convex, closed s.t. Z ⊂ Cn ⊂ Hn, xn+1 = PH(x0,xn)∩Cn (x0)

14 / 22

SLIDE 15

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Degrees of freedom of Block-Asynchronous Primal-Dual Algorithm

1

Parameters: ε > 0 and {γi,n}n∈N, {µk,n}n∈N, i ∈ {1, . . . , M}, k ∈ {1, . . . , K} sequences such that γi,n, µk,n > ε, i ∈ {1, . . . , M}, k ∈ {1, . . . , K}, n ∈ N.

2

Projection set: Cn — closed convex such that Z ⊂ Cn ⊂ Hn.

3

Sequence of block activations: {In}n∈N sequence of subsets of {1, . . . , M} and {Kn}n∈N sequence of subsets of {1, . . . , K} such that

1

In, Kn = ∅, n ∈ N,

2

I0 = {1, . . . , M}, K0 = {1, . . . , K},

3

There exists constants M, K such that

n+M

• n

In = {1, . . . , M},

n+K

• n

Kn = {1, . . . , K} for all n ∈ N,

4

Sequence of time-delays: D ∈ N, {ci(n)}n∈N and {dk(n)}n∈N, i ∈ {1, . . . , M}, k ∈ {1, . . . , K} sequences of natural numbers such that ∀n ∈ N n−D ≤ ci(n) ≤ n, n−D ≤ dk(n) ≤ n, ∀i ∈ {1, . . . , M}, ∀k ∈ {1, . . . , K}.

15 / 22

SLIDE 16

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Application - Image inpainting

The considered optimization problem is of the form min

p1,...,pM∈H M

• i=1

|pi| +

M

• i=1

ιyi (WiDpi) +

M

• i=1

ιS(Dpi) + λ

• (i,j)∈N

|pi − pj| (1) where

1

For i = 1, . . . , M, pi ∈ R2d2×1, i = 1 . . . , M are coefficients of patches d × d

2

dictionary matrix D learned via ITKrMM contains 2d2 normalised vectors (atoms) Di, stored as columns in D = (D1, . . . , D2d2) ∈ Rd2×2d2,

3

Wi ∈ Rd2×d2 is a diagonal matrix such that for k = 1, . . . , d2 we have Wi(k, k) = 0, if the pixel k in the corrupted image patch yi is lost and Wi(k, k) = 1, otherwise.

4

ιyi is the indicator function on set given by measured data yi ∈ Rd2×1 defined as: ιyi (x) =

• if

x = yi +∞

• therwise,

5

the indicator function ιS(x) imposes that x belongs to the set S = [0, 1]d2,

6

λ > 0 is a regularization parameter and N is a set of neighbourhood pairs of patches.

16 / 22

SLIDE 17

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

The considered grey-scale image is of the size 64 × 64. The corrupted image has 50% randomly missing pixels. We extract patches of size 8x8 with vertical/horizontal step

• f 2 pixels, which gives us M = 841 patches.

We compare the following variants of best approximation primal-dual synchronous algorithms:

1

C0 - as an non-memory Cn = Hn and In = I, Kn = K, n ∈ N,

2

C1 - as an memory Cn = Hn ∩ Hn−1 and In = I, Kn = K, n ∈ N,

3

C0random - as an non-memory Cn = Hn with block generated by Monte Carlo sampling technique,

4

C1random - as an memory Cn = Hn ∩ Hn−1 with block generated by Monte Carlo sampling technique.

17 / 22

SLIDE 18

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Block activation with Monte Carlo sampling technique

Let u ∈ RM be such that for all i ∈ {1, . . . , M}, ui = min{ǫ, 1 − Wi 1

d2

}, ǫ ∈ [0, 1].

1: Find In ⊂ {1, . . . , M} as realisation of random variable ¯

I with probability distribution function u using Monte Carlo sampling technique.

2: for i ∈ In do 3:

ai,n = proxγi,nfi (pi,n − γi,n K

k=1 L∗ kiv∗ k,n) 4:

a∗

i,n = γ−1 i,n (pi,n − ai,n) − K k=1 L∗ kiv∗ k,n 5: for i ∈ {1, . . . , M}\In do 6:

(ai,n, a∗

i,n) = (ai,n−1, a∗ i,n−1) 7: for k ∈ {1, . . . , K} do 8:

bk,n = proxµk,ngk (M

i=1 Likpi,n + µk,nv∗ k,n) 9:

b∗

k,n = µ−1 k,n(M i=1 Likpi,n − bk,n) + v∗ k,n

18 / 22

SLIDE 19

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Image inpaiting results

19 / 22

SLIDE 20

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Image inpaiting SNR

20 / 22

SLIDE 21

Problem statement Projection algorithms Extension to systems of monotone inclusions Application

Bibliography

1

Abdullah Alotaibi, Patrick L. Combettes, and Naseer Shahzad. “Best approximation from the Kuhn-Tucker set of composite monotone inclusions”. In:

• Numer. Funct. Anal. Optim. 36.12 (2015), pp. 1513–1532. issn: 0163-0563.

doi: 10.1080/01630563.2015.1077864. url: http://dx.doi.org/10.1080/01630563.2015.1077864

2

• E. M. Bednarczuk, A. Jezierska, and K. E. Rutkowski. “Proximal primal-dual

best approximation algorithm with memory”. In: ArXiv e-prints (Oct. 2016). arXiv: 1610.08697 [math.OC]

3

Patrick L. Combettes and Jonathan Eckstein. “Asynchronous block-iterative primal-dual decomposition methods for monotone inclusions”. In: Mathematical Programming 168.1 (2018), pp. 645–672. issn: 1436-4646. doi: 10.1007/s10107-016-1044-0. url: https://doi.org/10.1007/s10107-016-1044-0

4

• V. Naumova and K. Schnass. “Dictionary learning from incomplete data for

efficient image restoration”. In: 2017 25th European Signal Processing Conference (EUSIPCO). 2017, pp. 1425–1429. doi: 10.23919/EUSIPCO.2017.8081444

5

Krzysztof E. Rutkowski. “Closed-Form Expressions for Projectors onto Polyhedral Sets in Hilbert Spaces”. In: SIAM Journal on Optimization 27.3 (2017),

• pp. 1758–1771. doi: 10.1137/16M1087540. eprint:

https://doi.org/10.1137/16M1087540. url: https://doi.org/10.1137/16M1087540

21 / 22

SLIDE 22

Thank You for Your attention!