Convergence of Cubic Regularization for Nonconvex Optimization under - - PowerPoint PPT Presentation
Convergence of Cubic Regularization for Nonconvex Optimization under ojasiewicz Property Cubic-regularization (CR) + 1 + 2 CR :
∗
2
∈ℝ
CR : 𝑦 ∈ argmin 𝑧 − 𝑦, 𝛼𝑔 𝑦 + 1 2 𝑧 − 𝑦 𝛼𝑔 𝑦 𝑧 − 𝑦 + 𝑁 6 𝑧 − 𝑦
nd
3
4
∗ on a compact
∗ ∗
5
Convergence rate 𝜄 = +∞ 𝜈 𝑦 = 0 finite-step 𝜄 ∈ 3 2 , +∞ 𝜈 𝑦 ≤ Θ exp − 2(𝜄 − 1) super-linear 𝜄 = 3 2 𝜈 𝑦 ≤ Θ exp −(𝑙 − 𝑙) linear 𝜄 ∈ 1, 3 2 𝜈 𝑦 ≤ Θ 𝑙 − 𝑙
()
Sharp Flat
6
Lojasiewicz exponent 𝜾 Convergence rate 𝜄 = +∞ 𝑔 𝑦 − 𝑔∗ = 0 𝜄 ∈ 3 2 , +∞ 𝑔 𝑦 − 𝑔∗ ≤ Θ exp −
2 𝑔 𝑦 − 𝑔∗ ≤ Θ exp −(𝑙 − 𝑙) 𝜄 ∈ 1, 3 2 𝑔 𝑦 − 𝑔∗ ≤ Θ 𝑙 − 𝑙
7
8
Lojasiewicz exponent 𝜾 Convergence rate 𝜄 = +∞ 𝑦 − 𝑦∗ = 0 𝜄 ∈ 3 2 , +∞ 𝑦 − 𝑦∗ ≤ Θ exp −
()
2 𝑦 − 𝑦∗ ≤ Θ exp −(𝑙 − 𝑙) 𝜄 ∈ 1, 3 2 𝑦 − 𝑦∗ ≤ Θ 𝑙 − 𝑙
()
9
Lojasiewicz exponent 𝜾 Gradient descent Cubic-regularization 𝜄 = +∞ finite-step finite-step 𝜄 ∈ 2, +∞ linear super-linear 𝜄 ∈ [
, 2)
sub-linear super-linear 𝜄 ∈ 1,
)
sub-linear Θ(𝑙
.)
10