Computational Optimization Constrained Optimization Algorithms - - PowerPoint PPT Presentation

Computational Optimization

Constrained Optimization Algorithms

Same basic algorithms

Repeat

Determine descent direction Determine step size Take a step

Until Optimal But now must consider feasibility, e.g. Pick a feasible descent directions Choose step size to maintain feasibility Need dual multipliers to check optimality

Builds on Prior Approaches

Unconstrained: Lots of approaches Linear Equality Constrained:

Convert to unconstrained and solve

min ( ) f x

min ( ) . . f x s t Ax b =

Prior Approaches (cont)

Linear Inequality Constrained:

Identify active constraints Solve subproblems

Nonlinear Inequality Constrained:

Linearize constraints Solve subproblems

min ( ) . . f x s t Ax b ≥ min ( ) . . ( ) f x s t g x ≥

Feasible Direction Methods

Use feasible descent directs. Consider If we have feasible point then all are feasible.

1 min ' ' 2 . . x Qx c x s t Ax b + =

ˆ x ˆ x x Zp = +

Reduced Problem

1 ˆ ˆ ˆ min ( )' ( ) '( ) 2 1 1 ˆ ˆ ˆ ˆ min ' ' ( )' ' 2 2 1 ˆ min ' ' ( )' 2

p p p

x Zp Q x Zp c x Zp p Z QZp Qx c Zp xQx c x p Z QZp Qx c Zp + + + + + + + + + +

• '

: Reduced Hessian ˆ '( ( ) ) : Reduced Gradient Z QZ Z Q x Zp c = + + =

Reduced Newton’s Equation

For general problems Reduced Newton Equations Then apply usual Newton’s Method

2

' ( ) : Reduced Hessian ' ( ) : Reduced Gradient Z f x Z Z f x ∇ = ∇ =

2 2 1 2 1

' ( )

• '

( ) ( ' ( ) ) ' ( ) in reduced space ( ' ( ) ) ' ( ) in original spac Z f x Z v Z f x v Z f x Z Z f x p Zv Z Z f x Z Z f x

− −

∇ = ∇ ⇒ = − ∇ ∇ ⇒ = = − ∇ ∇ e

Reduced Steepest Descent

Use reduced steepest descent direction Convergence rate depends on condition number of reduced Hessian

' ( ) in reduced space ' ( ) in original space v Z f x d Zv ZZ f x = − ∇ ⇒ = = − ∇

2 2

( ' ( ) ) ( ' ) ( ( )) Cond Z f x Z Cond Z Z Cond f x ∇ ≤ ∇

KEY POINT – Construct Z to have good conditioning!

Ways to Compute Z

Projection Method (orthogonal and non-orthogonal) Variable Reduction QR Factorization

What are they and how do they effect conditioning? How can you use them to compute multipliers?

Projection Method

Find closest feasible point Compute a KKT point: This is optimal projection since problem is convex

1 2 2

min

x

c x Ax − =

Projection Method

( )

1 2 1 2

min ( , ) ( )' ( , ) '

x x

c x Ax L x c x Ax L x c x A KKT are Ax b λ λ λ λ − = ⇒ = − + ∇ = − − + = ⇒ − =

( ) ( ) ( ) ( )

1 1 1 1

' ' ' ' ' ' ' Projection Matrix ' ' Ac Ax AA AA Ac AA Ac x c A AA Ac x I A AA A c P I A AA A λ λ λ

− − − −

− − = ⇒ = ⇒ = ⇒ = − ⎡ ⎤ ⇒ = − ⎣ ⎦ ⎡ ⎤ = − ⎣ ⎦

Project method

The matrix is a basis for the null space matrix of A. Check

( )

1

' ' Z I A AA A

−

= −

( )

1

' ' AZ A I A AA A A A

−

⎡ ⎤ = − = − = ⎣ ⎦

Get Lagrangian Multipliers for free!

The matrix is the right inverse matrix for A. For general problems

( ) ( )

1 1

' ' ' '

r r

A A AA where AA AA AA I

− −

= = =

'

min ( ) . . * ( *)

r

f x s t Ax b A f x λ = = ∇

Let’s try it

For Projection matrix

1 2 3 4

1 2 2 2 2 2 1 2 3 4

min ( ) . . 1 f x x x x x s t x x x x ⎡ ⎤ = + + + ⎣ ⎦ + + + =

( ) [ ] [ ]

1 1 3 1 1 1 4 4 4 4 1 3 1 1 4 4 4 4 1 1 3 1 4 4 4 4 1 1 1 3 4 4 4 4

1 1 1 1 1 1 ' ' 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Z I A AA A

− − − − − − − − − − − − − −

⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = − = − ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

Solve FONC for Optimal Point

FONC

1 2 4

1 2 3 4

1 1 ( ) ' 1 1 1 x x f x A x x x x x x λ λ ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ∇ − = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎢ ⎥ ⎣ ⎦ + + + =

Check Optimality Conditions

For Using Lagrangian

3 1 1 1 4 4 4 4 1 3 1 1 4 4 4 4 1 1 3 1 4 4 4 4 1 1 1 3 4 4 4 4

1 1 1 ( *) * 1 4 1 Z f x

− − − − − − − − − − − −

⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ∇ = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

* [1111]/ 4 ( *) [1111]/ 4 * x f x Ax b = ∇ = =

( )

1

1 ' ' [1111]' 4 ( *) 1/ 4 ( *) '

r r

A A AA A f x Clearly f x A λ λ

−

= = = ∇ = ∇ =

You try it

For Find projection matrix Confirm optimality conds are Z’Cx*=0, Ax* = b Find x* Compute Lagrangian multipliers Check Lagrangian form of the multipliers.

1 2

min ( ) ' . . 13 6 3 13 23 9 3 2 1 2 1 2 6 9 12 1 1 1 3 1 3 3 3 1 3 f x x Cx s t Ax b C A b = = − − − ⎡ ⎤ ⎢ ⎥ − − ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ = = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − − − − ⎣ ⎦ ⎣ ⎦ ⎢ ⎥ − ⎢ ⎥ ⎣ ⎦

Projected Gradient Method

Algorithm for min f(x) s.t. Ax=b Let Z=I-A’(AA’)-1A x0 given g =gradf(x0) While not optimal d=-Z’g xi+1 = xi+αd (use linesearch) g =gradf(xi+1) i=i+1

Projected Gradient Method

Equivalent to steepest descent on f(x+Zp) where x is some feasible point. So convergence rate depends on condition number of this problem The condition number is less than equal condition of Hessian of f * Compute Z’HZ for last example and check it’s conditioning:

Lagrangian Multipliers

Each null space method yields a corresponding calculation of the right inverse. Thus each yields calculation of corresponding dual multipliers.

Variable Reduction Method

Let A=[B N] A is m by n B is m by m assume m < n is a basis matrix for null space of A

[ ]

1 1 r r

B B A AA B N I

− −

⎡ ⎤ ⎡ ⎤ = ⇒ = = + ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

1

B N Z I

−

⎡ ⎤ − = ⎢ ⎥ ⎣ ⎦

Try on our example

Take for example first two columns for B Then Condition number of Z’ CZ = 158 better but not great

[ ]

2 1 2 1 2 1 2 1 1 1 3 1 1 1 3 1 A B N ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ = = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − − ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦

1 2 1 1 4 3 1 2 1 1

r

Z A − − ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ − − ⎢ ⎥ ⎢ ⎥ = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

QR Factorization

Use Gram-Schmidt algorithm to make

• rthogonal factorize A’=QR with Q
• rthogonal and R upper triangular

[ ]

1 1 2 1 2 1 2 1 1

' , , ( ),

T r

R A QR Q Q where A m n Q n m Q n n m R m m Z Q A Q R− ⎡ ⎤ = = ⎢ ⎥ ⎣ ⎦ ∈ × ∈ × ∈ × − ∈ × = =

QR on problem

Use matlab command QR [Q R ] = qr(A’) Q2 = Q(:,3:4) Cond(Q2’*C*Q2) = 9.79

Next Problem

Consider Next Hardest Problem How could we adapt gradient projection

• r other linear equality constrained

problem to this problem?

min ( ) . . f x s t Ax b ≥