Lecture 2: Linear Programming and Duality Lecture Outline Part I: - - PowerPoint PPT Presentation

Lecture 2: Linear Programming and Duality

Lecture Outline

• Part I: Linear Programming and Examples
• Part II: Von Neumann’s Minimax Theorem and

Linear Programming Duality

• Part III: Linear Programming as a Problem

Relaxation

Part I: Linear Programming, Examples, and Canonical Form

Linear Programming

• Linear Programming: Want to optimize a linear

function over linear equalities and inequalities.

• Example: Maximize f x, y, z = 3𝑦 + 4𝑧 + 5𝑨

when

1. 𝑦 + 𝑧 + 𝑨 = 1 2. 𝑦 ≥ 0 3. 𝑧 ≥ 0 4. 𝑨 ≥ 0

• Answer: 𝑦 = 𝑧 = 0, 𝑨 = 1, 𝑔 𝑦, 𝑧, 𝑨 = 5

Example: Directed Connectivity

• Directed connectivity: Is there a path from 𝑡 =

𝑦1 to 𝑢 = 𝑦𝑜 in a directed graph 𝐻?

• Linear program: Minimize 𝑦𝑜 subject to

1. 𝑦1 = 1 2. 𝑦𝑘 ≥ 𝑦𝑗 whenever x𝑗 → 𝑦𝑘 ∈ 𝐹(𝐻) 3. ∀𝑗, 𝑦𝑗 ≥ 0

• Answer is 1 if there is a path from 𝑡 to 𝑢 in 𝐻

and 0 otherwise.

Example: Maximum Flow

• Max flow: Given edge capacities 𝑑𝑗𝑘 for each

edge 𝑦𝑗 → 𝑦𝑘 in 𝐻, what is the maximum flow from 𝑡 = 𝑦1 to 𝑢 = 𝑦𝑜?

• Example:

6 s t 2 6 1 2 2 3 6 2 6 6 4 4 4 3 3 4 4 1 4

Example Answer: 15

• Answer: 15. Actual flow is red/purple, capacity

is blue/purple.

6 s t 2 6 1 2 2 3 4 2 5 2 1 3 4 3 2 3 5 6 6 1 4 4 6 3 4 4 4

Max Flow Equations

• Take 𝑦𝑗𝑘 = flow from 𝑗 to 𝑘
• Recall: 𝑑𝑗𝑘 is the capacity from 𝑗 to 𝑘
• Program: Maximize 𝑦𝑜1 subject to

1. ∀𝑗, 𝑘, 0 ≤ 𝑦𝑗𝑘 ≤ 𝑑𝑗𝑘 (no capacity is exceeded, no negative flow) 2. ∀𝑗, σ𝑘=1

𝑜

𝑦𝑘𝑗 = σ𝑘=1

𝑜

𝑦𝑗𝑘 (flow in = flow out)

In-class Exercise

• Shortest path problem: Given a directed graph

𝐻 with lengths 𝑚𝑗𝑘 on the edges, what is the shortest path from 𝑡 = 𝑦1 to 𝑢 = 𝑦𝑜 in 𝐻?

• Exercise: Express the shortest path problem as

a linear program.

In-class Exercise Answer

• Shortest path problem: How long is the shortest

path from 𝑡 = 𝑦1 to 𝑢 = 𝑦𝑜 in a directed graph 𝐻?

• Linear Program: Have variables 𝑒𝑗 representing

the distance of vertex 𝑦𝑗 from vertex 𝑡 = 𝑦1. Maximize 𝑒𝑜 subject to

1. 𝑒1 = 0 2. ∀𝑗, 𝑘, 𝑒𝑘 ≤ 𝑒𝑗 + 𝑚𝑗𝑘 where 𝑚𝑗𝑘 is the length of the edge from 𝑦𝑗 to 𝑦𝑘

Canonical Form

• Canonical form: Maximize cT𝑦 subject to

1. 𝐵𝑦 ≤ 𝑐 2. 𝑦 ≥ 0

Putting Things Into Canonical Form

• Canonical form: Maximize cT𝑦 subject to

1. 𝐵𝑦 ≤ 𝑐 2. 𝑦 ≥ 0

• To put a linear program into canonical form:
• 1. Replace each equality 𝑏𝑗

𝑈𝑦 = 𝑐𝑗 with two

inequalities 𝑏𝑗

𝑈𝑦 ≤ 𝑐𝑗 and −𝑏𝑗 𝑈𝑦 ≤ −𝑐𝑗

• 2. In each expression, replace 𝑦𝑘 with (𝑦𝑘

+−𝑦𝑘 −)

where 𝑦𝑘

+, 𝑦𝑘 − are two new variables.

Slack Form

• Slack form: Maximize cT𝑦 subject to

1. 𝐵𝑦 = 𝑐 2. 𝑦 ≥ 0

Putting Things Into Slack Form

• Slack form: Maximize cT𝑦 subject to

1. 𝐵𝑦 = 𝑐 2. 𝑦 ≥ 0

• To put a linear program into slack form from

canonical form, simply add a slack variable for each inequality. σ𝑘=1

𝑜

𝑏𝑗𝑘𝑦𝑘 ≤ 𝑐𝑗 ⬄ (σ𝑘=1

𝑜

𝑏𝑗𝑘𝑦𝑘) + 𝑡𝑗 = 𝑐𝑗, 𝑡𝑗 ≥ 0

Part II: Von Neumann’s Minimax Theorem and Linear Programming Duality

Linear Programming Duality

• Primal: Maximize cT𝑦 subject to

1. 𝐵𝑦 ≤ 𝑐 2. 𝑦 ≥ 0

• Dual: Minimize 𝑐𝑈𝑧 subject to

1. 𝐵𝑈𝑧 ≥ 𝑑 2. 𝑧 ≥ 0

• Observation: For any feasible 𝑦, 𝑧, 𝑑𝑈𝑦 ≤ 𝑐𝑈𝑧

because 𝑑𝑈𝑦 ≤ 𝑧𝑈𝐵𝑦 = 𝑧𝑈 𝐵𝑦 − 𝑐 + 𝑧𝑈𝑐 ≤ 𝑐𝑈𝑧

• Strong duality: 𝑑𝑈𝑦 = 𝑐𝑈𝑧 at optimal 𝑦, 𝑧

Heart of Duality

• Game: Have a function 𝑔: 𝑌 × 𝑍 → 𝑆.
• 𝑌 player wants to minimize 𝑔(𝑦, 𝑧), 𝑍 player

wants to maximize 𝑔(𝑦, 𝑧)

• Obvious: Better to go second, i.e

max

𝑧∈𝑍 min 𝑦∈𝑌 𝑔(𝑦, 𝑧) ≤ min 𝑦∈𝑌 max 𝑧∈𝑍 𝑔(𝑦, 𝑧)

• Minimax theorems: Under certain conditions,

max

𝑧∈𝑍 min 𝑦∈𝑌 𝑔(𝑦, 𝑧) = min 𝑦∈𝑌 max 𝑧∈𝑍 𝑔(𝑦, 𝑧) !

Von Neumann’s Minimax Theorem

• Von Neumann [1928]: If 𝑌 and 𝑍 are convex

compact subsets of 𝑆𝑛 and 𝑆𝑜 and 𝑔: 𝑌 × 𝑍 → 𝑆 is a continuous function which is convex in 𝑌 and concave in 𝑍 then max

𝑧∈𝑍 min 𝑦∈𝑌 𝑔(𝑦, 𝑧) = min 𝑦∈𝑌 max 𝑧∈𝑍 𝑔(𝑦, 𝑧)

• These conditions are necessary (see problem

set)

Example

• Let 𝑌 = 𝑍 = [−1,1] and consider the function

𝑔 𝑦, 𝑧 = 𝑦𝑧.

• If the 𝑦 player goes first and plays 𝑦 = .5, the 𝑧

player should play 𝑧 = 1, obtaining 𝑔 𝑦, 𝑧 = .5

• If the 𝑦 player goes first and plays 𝑦 = −.5, the

𝑧 player should play 𝑧 = −1, obtaining 𝑔 𝑦, 𝑧 = .5

• The best play for the 𝑦 player is 𝑦 = 0 as then

𝑔 𝑦, 𝑧 = 0 regardless of what 𝑧 is.

Connection to Nash Equilibria

• Recall: 𝑌 player wants to minimize 𝑔(𝑦, 𝑧), 𝑍

player wants to maximize 𝑔(𝑦, 𝑧).

• If (𝑦∗, 𝑧∗) is a Nash equilibrium then

𝑔 𝑦∗, 𝑧∗ ≤ max

𝑧∈𝑍 min 𝑦∈𝑌 𝑔 𝑦, 𝑧

≤ min

𝑦∈𝑌 max 𝑧∈𝑍 𝑔 𝑦, 𝑧 ≤ 𝑔(𝑦∗, 𝑧∗)

• Note: Since 𝑔 is convex in 𝑦 and concave in 𝑧,

pure strategies are always optimal.

• However, this is circular: proof that Nash

equilibria exist ≈ proof of minimax theorem

Minimax Theorem Proof Sketch

• Proof idea:
• 1. Define a function 𝑈: 𝑌 × 𝑍 → 𝑌 × 𝑍 so that

𝑈 𝑦, 𝑧 = (𝑦, 𝑧) if and only if (𝑦, 𝑧) is a Nash equilibrium.

• 2. Use Brouwer’s fixed point theorem to argue that T

must have a fixed point.

Attempt #1

• We could try to define 𝑈 as follows
• 1. Starting from (𝑦, 𝑧), take 𝑦′ to be the closest point

to 𝑦 which minimizes 𝑔(𝑦′, 𝑧).

• 2. Now take 𝑧′ to be the closest point to 𝑧 which

maximizes 𝑔(𝑦′, 𝑧′)

• 3. Take 𝑈 𝑦, 𝑧 = (𝑦′, 𝑧′)
• 𝑈 𝑦, 𝑧 = 𝑦, 𝑧 ⬄(𝑦, 𝑧) is a Nash equilibrium.

Brouwer’s Fixed Point Theorem

• Brouwer’s fixed point theorem: If 𝑌 is a convex,

compact subset of 𝑆𝑜 then any continuous map 𝑔: 𝑌 → 𝑌 has a fixed point

• Example: Any continuous function 𝑔: 𝐸2 → 𝐸2

has a fixed point.

Correct function T

• Problem: Previous 𝑈 may not be continuous!
• Correct 𝑈: Starting from 𝑦, 𝑧 :
• 1. Define Δ 𝑦2 = 𝑔 𝑦, 𝑧 − 𝑔 𝑦2, 𝑧 if 𝑔 𝑦2, 𝑧 <

𝑔(𝑦, 𝑧) and Δ 𝑦2 = 0 otherwise.

• 2. Take 𝑦′ =

𝑦+׬

𝑦2∈𝑌 Δ 𝑦2 𝑦2

1+׬

𝑦2∈𝑌 Δ 𝑦2

• 3. Define Δ 𝑧2 = 𝑔 𝑦′, 𝑧2 − 𝑔 𝑦′, 𝑧 if 𝑔 𝑦, 𝑧2 >

𝑔(𝑦, 𝑧) and Δ 𝑧2 = 0 otherwise.

• 4. Take 𝑧′ =

𝑧+׬

𝑧2∈𝑍 Δ 𝑧2 𝑧2

1+׬

𝑧2∈𝑍 Δ 𝑧2

and 𝑈 𝑦, 𝑧 = (𝑦′, 𝑧′)

Duality Via Minimax Theorem

• Idea: Instead of trying to enforce some of the

constraints, make the program into a two player game where the new player can punish any violated constraints.

• Example: Maximize cT𝑦 subject to

1. 𝐵𝑦 ≤ 𝑐 2. 𝑦 ≥ 0

• Game: Take 𝑔 𝑦, 𝑧 = 𝑑𝑈𝑦 + 𝑧𝑈 𝑐 − 𝐵𝑦 where

we have the constraint that 𝑧 ≥ 0 (here 𝑧 wants to minimize 𝑔(𝑦, 𝑧)).

• If 𝐵𝑦 𝑗 > 𝑐𝑗, 𝑧 can take 𝑧𝑗 → ∞ to punish this.

Strong Duality Intuition

• Canonical primal form: Maximize cT𝑦 subject to

1. 𝐵𝑦 ≤ 𝑐 2. 𝑦 ≥ 0

• = max

𝑦≥0 min 𝑧≥0 𝑑𝑈𝑦 + 𝑧𝑈 𝑐 − 𝐵𝑦

• = min

𝑧≥0 max 𝑦≥0 𝑧𝑈𝑐 + (𝑑𝑈 − 𝑧𝑈𝐵)𝑦

• Canonical dual form: Minimize 𝑐𝑈𝑧 subject to

1. 𝐵𝑈𝑧 ≥ 𝑑𝑈 2. 𝑧 ≥ 0

• Not quite a proof, domains of 𝑦, 𝑧 aren’t compact!

Slack Form Duality Intuition

• Slack primal form: Maximize cT𝑦 subject to

1. 𝐵𝑦 = 𝑐 2. 𝑦 ≥ 0

• = max

𝑦≥0 min 𝑧 𝑑𝑈𝑦 + 𝑧𝑈 𝑐 − 𝐵𝑦

• = min

𝑧 max 𝑦≥0 𝑧𝑈𝑐 + (𝑑𝑈 − 𝑧𝑈𝐵)𝑦

• Slack dual form: Minimize 𝑐𝑈𝑧 subject to

1. 𝐵𝑈𝑧 ≥ 𝑑𝑈

• See problem set for a true proof of strong duality.

Max-flow/Min-cut Theorem

• Classical duality example: max-flow/min-cut
• Max-flow/min-cut theorem: The maximum flow

from 𝑡 to 𝑢 is equal to the minimum capacity across a cut separating 𝑡 and 𝑢.

• Duality is a bit subtle (see problem set)

Max-flow/Min-cut Example

• Maximum flow was 15, this is matched by the

minimal cut shown below:

6 s t 2 6 1 2 2 3 6 2 6 6 4 4 4 3 3 4 4 1 4

Part III: Linear Programming as a Problem Relaxation

Convex Relaxations

• Often we want to optimize over a nonconvex

set, which is very difficult.

• To obtain an approximation, we can take a

convex relaxation of our set.

• Linear programming can give such convex

relaxations.

Bad Example: 3-SAT solving

• Actual problem: Want each 𝑦𝑗 ∈ {0,1}.
• A clause 𝑦𝑗 ∨ 𝑦𝑘 ∨𝑙 can be re-expressed as

𝑦𝑗 + 𝑦𝑘 + 𝑦𝑙 ≥ 1

• Negations can be handled with the equality

¬𝑦𝑗 = 1 − 𝑦𝑗

• Convex relaxation: Only require 0 ≤ 𝑦𝑗 ≤ 1
• Too relaxed: Could just take all 𝑦𝑗 =

1 2!

• Note: strengthening this gives cutting planes

Example: Maximum Matching

• Have a variable 𝑦𝑗𝑘 for each edge 𝑗, 𝑘 ∈ 𝐹(𝐻)
• Actual problem: Maximize σ𝑗,𝑘: 𝑗,𝑘 ∈𝐹(𝐻) 𝑦𝑗𝑘

subject to

1. ∀𝑗 < 𝑘: 𝑗, 𝑘 ∈ 𝐹 𝐻 , 𝑦𝑗𝑘 ∈ {0,1} 2. ∀𝑗, σ𝑘<𝑗: 𝑘,𝑗 ∈𝐹 𝐻 𝑦𝑘𝑗 + σ𝑘>𝑗: 𝑗,𝑘 ∈𝐹 𝐻 𝑦𝑗𝑘 ≤ 1

• Convex relaxation: Only require 0 ≤ 𝑦𝑗𝑘 ≤ 1
• Gives exact value for bipartite graphs, not in

general (see problem set)