The minimum Euclidean norm point in a polytope: Wolfe's method is - - PowerPoint PPT Presentation

The minimum Euclidean norm point in a polytope: Wolfe's method is exponential

Luis Rademacher, UC Davis Simons Institute Join work with Jesús de Loera, Jamie Haddock

Minimum norm point in a polytope

• Given 𝑞1, … , 𝑞𝑜 ∈ 𝑆𝑒, determine

argmin𝑦∈𝑄 𝑦 for 𝑄 = conv(𝑞1, … , 𝑞𝑜).

• A convex quadratic program.

Projection onto a simplex: not so easy

• Only polynomial time algorithms I know are general purpose convex

programming algorithms like the ellipsoid method.

• No strongly polynomial time algorithm known.

Wolfe’s method (Wolfe ’74, Lawson Hanson ‘74)

• A combinatorial algorithm to find the minimum norm point in a

polytope.

• Not the same as Frank-Wolfe

Motivation

• In machine learning:
• Fujishige-Wolfe algorithm
• one of the most practical algorithms for submodular function minimization.
• Wolfe’s method is a subroutine in it
• Optimal loading of recursive neural networks [CDKSV ‘95]
• Non-negative least squares

This talk:

• Complexity of Wolfe’s method
• Relationship between Linear Programming (LP) and minimum norm

point problem.

Related work

• [Lawson, Hanson ‘74] An algorithm very similar to Wolfe’s.
• [Fujishige, Hayashi, Isotani ’06]

Polynomial time reduction from LP to minimum norm point in a polytope.

• [Chakrabarty, Jain, Kothari ‘14]

[Lacoste-Julien, Jaggi ‘15] Rates of convergence of Wolfe’s method.

Our results

• LP reduces in strongly polynomial time to “minimum norm point in a

simplex”.

• A (strongly) polynomial time algorithm for minimum norm point in a simplex

would give a (strongly) polynomial time algorithm for general LPs.

• Step 1: LP reduces to “membership in a V-polytope”.
• Step 2: “Membership in a V-polytope” reduces to “distance to a simplex”.

Our results

• Wolfe’s method takes exponential time in the worst case.
• We construct explicit sets of points in every dimension.
• Similar in spirit to Klee-Minty cubes for LP.

Idea of Wolfe’s method

• Given points 𝑄 ⊆ 𝑆𝑒, maintain the following invariant:

A subset 𝑇 ⊆ 𝑄 whose vertices determine a simplex, and the minimum norm point, 𝑦, in the simplex.

• Start with any point in 𝑄 as 𝑇 and 𝑦.
• Alternate between the following two steps:
• Find point 𝑞 ∈ 𝑄 such that 𝑇 ← 𝑑𝑝𝑜𝑤 𝑇 ∪ 𝑞

contains a better point (entering rule). Add 𝑞 to 𝑇.

• Let 𝑦 “follow gravity” (towards 0) within 𝑑𝑝𝑜𝑤 𝑇 while dimension of current

face decreases. Let 𝑇 be the (vertices of the) current face of 𝑦.

Inefficiency of Wolfe’s method with “minnorm” entering rule

• A point, 𝑏, enters current set 𝑇, leaves, and then re-enters.
• 𝑏 < 𝑞 < 𝑟 < 𝑠 < |𝑡|

S . a ap apq pq pqr qr qrs rs rsa

Exponential lower bound

• Replace point 𝑏 and 1st coordinate by subspace

and a set of points in it, constructed recursively.

• Seq 𝑇: 𝑄 𝑒 − 2 → ⋯ → 𝑠

𝑒𝑡𝑒 → 𝑄 𝑒 − 2 𝑠 𝑒𝑡𝑒

Open questions

• Find exponential lower bound for other “entering rules.” (linopt)
• For submodular function minimization, polytopes are so-called “base

polytopes.”

• Our exponential example is not of that kind.
• Could Wolfe’s method be faster on base polytopes?
• Smoothed analysis of Wolfe’s method.