A Rank-Constrained Optimization approach: Application to Factor - - PowerPoint PPT Presentation

A Rank-Constrained Optimization approach: Application to Factor Analysis

Ramón A. Delgado, Juan C. Agüero, Graham C. Goodwin

School of Electrical Engineering and Computer Science, The University of Newcastle, Australia.

1

Handling the Rank Constraints

2

Some Existing Representations

3

A new representation of rank constraints

4

Rank-Constrained Optimisation

5

Application to Factor Analysis

6

Conclusions

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 2 / 22

Motivation

why include rank constraints?

Complexity of a model (e.g rank of a Hankel matrix) Low-Rank Decomposition (Principal Components Analysis) Recent interest on sparse representations.

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 4 / 22

Some Difficulties with rank constraints rank {X} ≤ r Non-differentiable Nonlinear Combinatorial nature in optimization Find a differentiable representation for the rank constraints, and hopefully reduce the number of nonlinearities.

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 5 / 22

Some existing representations

By construction

rank {X} ≤ r ⇐ ⇒ X = A Ir

• B.

rank {X} ≤ r ⇐ ⇒ X = UV. where A ∈ Rm×m, B ∈ Rn×n, U ∈ Rm×r, V ∈ Rr×n. Disadvantage Already assign a structure to X.

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 7 / 22

Some existing representations

Using the Characteristic Polynomial

Consider that ci(X), for i = 1, . . . , n are the coefficients of the characteristic polynomial of X. X ∈ Sn

+,

[Helmersson 2009] rank {X} ≤ r ⇐ ⇒ cn−r−1(−X) = 0. X ∈ Sn

+,

[d’Aspremont 2003] rank {X} = minv∈Rn n

i=1 vi

s.t. ci(G)(1 − vi) = 0, vi ≥ 0 for i = 1, . . . , n. Disadvantages Only valid for X ∈ Sn

+.

ci(X) is, in general, nonlinear.

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 8 / 22

Some existing representations used in optimization

Closely related to our representation

• I. Markovsky rank {X} ≤ r ⇐

⇒ ∃R ∈ R(m−r)×m such that RX = 0m−r×m and R is full row rank X ∈ Rm×n, but a rank constraint is now imposed on an auxiliary matrix.

• J. Dattorro X ∈ Sn

+,

rank {X} ≤ r ⇐ ⇒ ∃W ∈ Φn,r such that trace(WX) = 0. where Φn,r = {W ∈ Sn, 0 W I, trace(W) = n − r} Only valid for X ∈ Sn

+.

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 9 / 22

Main result1 Theorem Let X ∈ Rm×n then rank {X} ≤ r ⇐ ⇒ ∃ W ∈ Φn,r, such that GX = 0m×n where Φn,r = {W ∈ Sn, 0 W I, trace(W) = n − r}

1submitted for publication Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 11 / 22

Advantages Differentiable Freedom to impose a desired structure on X (e.g. Hankel). Generalisation of Dattorro’s result Avoid some difficulties in Markovsky and d’Aspremont’s results. Disadvantage We still have a bilinear condition WX = 0m×n.

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 12 / 22

Rank-Constrained Optimisation An equivalent representation for rank-constrained optimization problems Prco : min

θ∈Rp f(θ)

s.t. θ ∈ Ω rank {X(θ)} ≤ r ≡ Pbi : min

θ∈Rp min W∈Sn f(θ)

s.t. θ ∈ Ω X(θ)W = 0m×n W ∈ Φn,r

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 14 / 22

Rank-Constrained Optimisation We have developed a local optimisation method For the case X(θ) ∈ Sn

+, a global optimisation method.

The bilinear constraint can be imposed in several ways: X(θ) ∈ Rm×n X(θ)W = 0 ⇐ ⇒ X(θ)W = 0m×n X(θ) ∈ Sn

+

trace(X(θ)W) = 0 ⇐ ⇒ X(θ)W = 0n×n

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 15 / 22

Factor Analysis Consider a measured output yk ∈ RN, factors fk ∈ Rr, idiosyncratic noise vk ∈ RN, and a model: yk = Afk + vk (1) where A ∈ RN×n is a tall matrix. fk ∼ N(0, Φ) (2) vk ∼ N(0, Ψ) (3) Then, yk ∼ N(0, Σ), where Σ is given by Σ = AΦA⊤ + Ψ

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 17 / 22

Sparse Noise Covariance Existing approaches require Ψ diagonal.

◮ Ψ = σ2I, then there is a closed-form solution, e.g. PCA. ◮ Ψ diagonal, e.g. Maximum Likelihood (EM algorithm).

Considering the advances on sparse representations. We propose to assume that Ψ is sparse. Prcofa : min

Ψ∈SN Ψ1

subject to rank {Σ − Ψ} ≤ r Ψ 0 Σ − Ψ 0

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 18 / 22

Numerical Example

Local Optimization method

Consider r = 3 factors, N = 20 measured outputs and T = 100 Samples, Ψij = (0.7)|i−j|. and the performance index d(Pm) = 1 − trace(APmA⊤) trace(AA⊤) Method d(·) Total Time [s] PCA 0.1992 0.0136 RCO 0.1002 274.6732 EM 0.1330 0.0198

Table: Mean value over Nmc = 100 Monte Carlo simulations of d(·).

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 19 / 22

Conclusions We have developed a new representation of rank constraints.

◮ Second-order differentiable. ◮ Avoid several nonlinearities, excepting a bilinear constraint.

We have developed two optimization algorithms.

◮ Local Optimisation. ◮ Global optimisation.

We have applied the method:

◮ Factor Analysis with Correlated Errors. ◮ Sparse Control (tomorrow FrA04.2). Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 21 / 22

Thanks for your attention! Any questions?

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 22 / 22

Global Optimization Example Consider r = 1 factor, N = 3 measured outputs and T = 100 Samples. d(·)

• Ψ1

time [s] RCO 0.066 4.46 23.83 RCO-G 0.066 4.46 310.19 d(·) vec( Ψ)1 time [s] RCO 0.165 11.12 43.20 RCO-G 0.218 9.17 1768.31

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 23 / 22

Solving the Optimisation Problem Proposed method Solve Pbi iteratively. In each iteration, solves a feasibility problem that deals with the bilinear constraint. Given the estimate θm ∈ Ω, at iteration m min

θ∈Rp min W∈Sn X(θ)W

subject to f(θ) ≤ f(θm) − ηm θ ∈ Ω W ∈ Φn,r

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 24 / 22

Branch and Bound

Global Optimum 1st 2nd

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 25 / 22

ℓ1-norm Computationally efficient. Unintuitive to chose the sparse parameters Poor handling of group-constraints ℓ0-norm Computationally more expensive. ℓ0-norm constraints have a clear interpretation. Can handle Group constraints. ℓ0-norm equivalence

Pℓ0 : min

θ∈Rp f(θ)

s.t. θ ∈ Ω θ0 ≤ r ≡ Peq : min

θ∈Rp min w∈Rp f(θ)

s.t. θ ∈ Ω θ ◦ w = 0 0 ≤ w ≤ 1 1⊤w = p − r

Delgado, Agüero & Goodwin 2014 (IFAC 2014). Delgado, Agüero & Goodwin submitted to Automatica. Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 26 / 22

Local Optimization

Ψ (d) True (e) PCA (f) EM (g) RCO

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 27 / 22

Concave Minimisation Concave minimization may lead to a combinatorial problem. (e.g. matrix-rank function is quasi-concave on the Positive Semidefinite Cone, thus leads to Concave-minimisation and “reverse-convex” constraints)

Delgado, Agüero, Goodwin (UoN) IFAC 2014, Cape Town 28 / 22