Near optimal finite time identification of arbitrary linear - - PowerPoint PPT Presentation

Problem Definition Analysis and Techniques Results Main Results Poster Details

Near optimal finite time identification of arbitrary linear dynamical systems

Tuhin Sarkar & Alexander Rakhlin

Massachusetts Institute of Technology

June 12, 2019

Tuhin Sarkar & Alexander Rakhlin ICML 2019 1 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Plan

1 Problem Definition 2 Analysis and Techniques 3 Results 4 Main Results 5 Poster Details

Tuhin Sarkar & Alexander Rakhlin ICML 2019 2 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Linear Time Invariant (LTI) Systems

LTI systems appear in autoregressive processes, control and RL

• systems. Formally,

Xt+1 = AXt + ηt+1 (1) Xt, ηt ∈ Rn. Xt is state vector, ηt is noise vector. A is state transition matrix : characterizes the LTI system. Assume {ηt}∞

t=1 is isotropic and subGaussian.

Tuhin Sarkar & Alexander Rakhlin ICML 2019 3 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Linear Time Invariant (LTI) Systems

LTI systems appear in autoregressive processes, control and RL

• systems. Formally,

Xt+1 = AXt + ηt+1 (1) Xt, ηt ∈ Rn. Xt is state vector, ηt is noise vector. A is state transition matrix : characterizes the LTI system. Assume {ηt}∞

t=1 is isotropic and subGaussian.

Tuhin Sarkar & Alexander Rakhlin ICML 2019 3 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Linear Time Invariant (LTI) Systems

LTI systems appear in autoregressive processes, control and RL

• systems. Formally,

Xt+1 = AXt + ηt+1 (1) Xt, ηt ∈ Rn. Xt is state vector, ηt is noise vector. A is state transition matrix : characterizes the LTI system. Assume {ηt}∞

t=1 is isotropic and subGaussian.

Tuhin Sarkar & Alexander Rakhlin ICML 2019 3 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Learning A from data

Goal : Learn A from {Xt}T

t=1

ˆ A = inf

Ao T

• t=1

||Xt+1 − AoXt||2

2

Estimation error E = A − ˆ A = (

T

• t=1

ηt+1X⊤

t )( T

• t=1

XtX⊤

t )+

(2) Error analysis hard : {Xt}T

t=1 are not independent.

Tuhin Sarkar & Alexander Rakhlin ICML 2019 4 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Related Work

Faradonbeh et. al. (2017). Finite time identification in unstable linear systems. Simchowitz et. al. (2018). Learning without mixing : Towards a sharp analysis of linear system identification. Past works fail to capture correct behavior for all A.

Tuhin Sarkar & Alexander Rakhlin ICML 2019 5 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Main Technique

The analysis proceeds in two steps : Show invertibility of sample covariance matrix : T

t=1 XtX⊤ t ≈ f(T)I.

Show the following for self–normalized martingale term : (

T

• t=1

ηt+1X⊤

t )( T

• t=1

XtX⊤

t )−1/2 = O(1)

Tuhin Sarkar & Alexander Rakhlin ICML 2019 6 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Main Technique

The analysis proceeds in two steps : Show invertibility of sample covariance matrix : T

t=1 XtX⊤ t ≈ f(T)I.

Show the following for self–normalized martingale term : (

T

• t=1

ηt+1X⊤

t )( T

• t=1

XtX⊤

t )−1/2 = O(1)

Tuhin Sarkar & Alexander Rakhlin ICML 2019 6 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Sample Covariance Matrix

Let ρi(A) be the absolute value of ith eigenvalue of A with ρi(A) ≥ ρi+1(A). Then ρi ∈ S0 = ⇒ ρi(A) ≤ 1 − C/T ρi ∈ S1 = ⇒ ρi(A) ∈ [1 − C/T, 1 + C/T] ρi ∈ S2 = ⇒ ρi(A) ≥ 1 + C/T Theorem ρi(A) ∈ S0 = ⇒ T

t=1 XtX⊤ t = Θ(T)

ρi(A) ∈ S1 = ⇒ T

t=1 XtX⊤ t = Ω(T 2)

ρi(A) ∈ S2 = ⇒ T

t=1 XtX⊤ t = Ω(eaT ) (under necessary

and sufficient “regularity” conditions only)

Tuhin Sarkar & Alexander Rakhlin ICML 2019 7 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Sample Covariance Matrix

Let ρi(A) be the absolute value of ith eigenvalue of A with ρi(A) ≥ ρi+1(A). Then ρi ∈ S0 = ⇒ ρi(A) ≤ 1 − C/T ρi ∈ S1 = ⇒ ρi(A) ∈ [1 − C/T, 1 + C/T] ρi ∈ S2 = ⇒ ρi(A) ≥ 1 + C/T Theorem ρi(A) ∈ S0 = ⇒ T

t=1 XtX⊤ t = Θ(T)

ρi(A) ∈ S1 = ⇒ T

t=1 XtX⊤ t = Ω(T 2)

ρi(A) ∈ S2 = ⇒ T

t=1 XtX⊤ t = Ω(eaT ) (under necessary

and sufficient “regularity” conditions only)

Tuhin Sarkar & Alexander Rakhlin ICML 2019 7 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Sample Covariance Matrix

Let ρi(A) be the absolute value of ith eigenvalue of A with ρi(A) ≥ ρi+1(A). Then ρi ∈ S0 = ⇒ ρi(A) ≤ 1 − C/T ρi ∈ S1 = ⇒ ρi(A) ∈ [1 − C/T, 1 + C/T] ρi ∈ S2 = ⇒ ρi(A) ≥ 1 + C/T Theorem ρi(A) ∈ S0 = ⇒ T

t=1 XtX⊤ t = Θ(T)

ρi(A) ∈ S1 = ⇒ T

t=1 XtX⊤ t = Ω(T 2)

ρi(A) ∈ S2 = ⇒ T

t=1 XtX⊤ t = Ω(eaT ) (under necessary

and sufficient “regularity” conditions only)

Tuhin Sarkar & Alexander Rakhlin ICML 2019 7 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Self Normalized Martingale

Theorem (Abbasi-Yadkori et. al. 2011) Let V be a deterministic matrix with V ≻ 0. For any 0 < δ < 1 and {ηt, Xt}T

t=1 defined as before, we have with probability 1 − δ

||( ¯ YT−1)−1/2

T−1

• t=0

Xtη′

t+1||2

≤ R

• 8n log
• 5det( ¯

YT−1)1/2ndet(V )−1/2n δ1/n

• (3)

where ¯ Y −1

τ

= (Yτ + V )−1 and R2 is the subGaussian parameter

• f ηt.

Tuhin Sarkar & Alexander Rakhlin ICML 2019 8 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Main Result 1

Combining the previous results (and a few more matrix manipulations) we show Theorem ρi(A) ∈ S0 ∪ S1 ∪ S2 = ⇒ ||E||2 = O(T −1/2) ρi(A) ∈ S1 ∪ S2 = ⇒ ||E||2 = O(T −1) ρi(A) ∈ S2 = ⇒ ||E||2 = O(e−aT ) (under necessary and sufficient “regularity” conditions only)

Tuhin Sarkar & Alexander Rakhlin ICML 2019 9 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Main Result 1

Combining the previous results (and a few more matrix manipulations) we show Theorem ρi(A) ∈ S0 ∪ S1 ∪ S2 = ⇒ ||E||2 = O(T −1/2) ρi(A) ∈ S1 ∪ S2 = ⇒ ||E||2 = O(T −1) ρi(A) ∈ S2 = ⇒ ||E||2 = O(e−aT ) (under necessary and sufficient “regularity” conditions only)

Tuhin Sarkar & Alexander Rakhlin ICML 2019 9 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Main Result 1

Combining the previous results (and a few more matrix manipulations) we show Theorem ρi(A) ∈ S0 ∪ S1 ∪ S2 = ⇒ ||E||2 = O(T −1/2) ρi(A) ∈ S1 ∪ S2 = ⇒ ||E||2 = O(T −1) ρi(A) ∈ S2 = ⇒ ||E||2 = O(e−aT ) (under necessary and sufficient “regularity” conditions only)

Tuhin Sarkar & Alexander Rakhlin ICML 2019 9 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Main Result 2

Regularity condition : All eigenvalues greater than one should have geometric multiplicity one. Theorem If the regularity conditions are violated then OLS is inconsistent. OLS cannot learn A = ρI where ρ ≥ 1.5. E has a non–trivial probability distribution.

Tuhin Sarkar & Alexander Rakhlin ICML 2019 10 / 11

Problem Definition Analysis and Techniques Results Main Results Poster Details

Poster Details

Please come to our poster at Pacific Ballroom #193 at 6.30 pm today. Thank you !

Tuhin Sarkar & Alexander Rakhlin ICML 2019 11 / 11