Empirically Comparing the Finite-Time Performance of - - PowerPoint PPT Presentation

▶
empirically comparing the finite time performance of
SMART_READER_LITE
LIVE PREVIEW

Empirically Comparing the Finite-Time Performance of - - PowerPoint PPT Presentation

Mar 10, 2023 551 likes •819 views

Empirically Comparing the Finite-Time Performance of Simulation-Optimization Algorithms Anna Dong, David Eckman , Xueqi Zhao, Shane Henderson Cornell University Matthias Poloczek University of Arizona Winter Simulation Conference December 5,

slide-1
SLIDE 1

Empirically Comparing the Finite-Time Performance

  • f Simulation-Optimization Algorithms

Anna Dong, David Eckman, Xueqi Zhao, Shane Henderson

Cornell University

Matthias Poloczek

University of Arizona

Winter Simulation Conference December 5, 2017

slide-2
SLIDE 2

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Simulation Optimization (SO)

Optimize a real-valued objective function, estimated via simulation,

  • ver a deterministic domain.

Challenges:

  • 1. Error in estimating objective function.
  • 2. Unknown topology (continuity/differentiability/convexity).

SO algorithms are designed to solve a broad class of problems.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 2/22

slide-3
SLIDE 3

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

SO Algorithm Performance

Many theoretical results for asymptotic performance,

  • . . . as simulation budget approaches infinity.

Examples:

  • Algorithm converges to local (global) optimizer.
  • Convergence rate, once within neighborhood of optimizer.

Required budget for such results may exceed practical budget! To decide which algorithm to use, a practitioner cares about finite-time performance.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 3/22

slide-4
SLIDE 4

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Evaluating Algorithms

SO community lags behind other optimization communities:

  • Established testbed of problems for benchmarking.
  • Metrics for empirical finite-time performance.
  • Comparison of algorithms on large testbed.

We implement several popular SO algorithms and test them on a subset of problems from the SimOpt library ✭✇✇✇✳s✐♠♦♣t✳♦r❣✮.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 4/22

slide-5
SLIDE 5

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Objectives

Near-term:

  • 1. Comparison of finite-time performance of different algorithms.
  • 2. Insights on the types of problems on which certain algorithms

work well. Long-term:

  • 1. More contributions to SimOpt library.
  • Problems and/or algorithms.
  • 2. Development of finite-time performance metrics.
  • E.g., adapting performance profiles.
  • 3. Motivate others to do similar comparisons.
  • 4. Development of algorithms with strong finite-time performance.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 5/22

slide-6
SLIDE 6

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Evaluating Finite-Time Performance

In deterministic optimization:

  • Measure computational effort needed to get to optimal solution

(or within specified tolerance).

  • Number of function evaluations or wall clock time.

Doesn’t work so well for SO.

  • Optimal solution is often unknown.
  • Often no certificate of optimality.
  • Estimation error makes it hard to check tolerance condition.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 6/22

slide-7
SLIDE 7

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Evaluating Finite-Time Performance

Idea

  • 1. Fix a simulation budget.
  • 2. Evaluate the objective function at the estimated best solution

found within the budget. We measure the budget in number of objective function evaluations.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 7/22

slide-8
SLIDE 8

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Evaluating Finite-Time Performance

Let Z(n) be the true objective function value of the estimated best solution visited in the first n objective function evaluations.

  • Z(n) is a random variable, because the estimated best solution,

X(n), is random.

  • Conditional on X(n), Z(n) is fixed, but needs to be estimated.

In our experiments, we estimate Z(n) (conditional on X(n)) in a post-processing step.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 8/22

slide-9
SLIDE 9

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Evaluating Finite-Time Performance

Figure: Pasupathy and Henderson (2006).

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 9/22

slide-10
SLIDE 10

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Evaluating Finite-Time Performance

Can obtain Z(n) curve from single macroreplication of an algorithm.

  • Unless the algorithm uses the budget in setting parameters.

Location of Z(n) curve is random, so take several macroreplications and look at:

  • mean,
  • median/quantile,
  • empirical cdf (hard to show on one plot).

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 10/22

slide-11
SLIDE 11

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Algorithms

  • 1. Random Search (non-adaptive)
  • 2. Gradient Search with Random Restarts
  • Central finite differences for gradient estimate.
  • 3. Simultaneous Perturbation Stochastic Approximation (SPSA)
  • Uses budget to set gain sequence.
  • 4. Stochastic Trust-Region Response-Surface Method (STRONG)
  • Didn’t use design of experiments for fitting.
  • Central finite differences for gradient estimate.
  • a. BFGS estimate of Hessian. (STRONG)
  • b. No second-order model. (STRONG-Stage1)
  • 5. Nelder-Mead
  • Simplicial method that doesn’t use gradient information.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 11/22

slide-12
SLIDE 12

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Benchmark Problems

Table: SimOpt benchmark problems and their characteristics.

Name on SimOpt Dimension Optimal Solution A Multimodal Function 2 Known Ambulances in a Square 6 Unknown Continuous Newsvendor 1 Known Dual Sourcing 2 Unknown Economic-Order-Quantity 1 Known Facility Location 4 Unknown GI/G/1 Queue 1 Unknown M/M/1 Metamodel 3 Known Optimal Controller for a POMDP 10 Unknown Optimization of a Production Line 3 Unknown Parameter Estimation: 2D Gamma 2 Known Rosenbrock’s Function 40 Known Route Prices for Mobility-on-Demand 12 Unknown SAN Duration 13 Unknown Toll Road Improvements 12 Unknown

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 12/22

slide-13
SLIDE 13

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Problems

Properties of all problems:

  • Continuous decision variables.
  • Deterministic (box) constraints or unbounded.

Initial solution is drawn from probability distribution over domain.

  • Uniform distribution for bounded variables.
  • Exponential/Laplace distribution for unbounded variables.

Took 30 replications at given solution to estimate its objective value.

  • Used common random numbers (CRN) across solutions.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 13/22

slide-14
SLIDE 14

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Experiments

For every problem:

  • Ran 30 macroreplications of each algorithm.
  • Recorded estimated best solution X(n) for range of n values.
  • Ran 30 function evaluations at each X(n) to estimate Z(n)

conditional on X(n).

  • Averaged 30 estimates of Z(n).
  • Constructed 95% normal confidence intervals.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 14/22

slide-15
SLIDE 15

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Rosenbrock (dim = 40)

Budget #104 2 4 6 8 10 12 Objective value #106 0.5 1 1.5 2 2.5 3 3.5 4 4.5 minimize Problem: Rosenbrock

Random Search Nelder-Mead Gradient Search RS SPSA STRONG STRONG-StageI

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 15/22

slide-16
SLIDE 16

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Facility Location (dim = 4)

Budget #104 0.5 1 1.5 2 2.5 3 Objective value 0.15 0.2 0.25 0.3 0.35 maximize Problem: FacilityLocation

Random Search Nelder-Mead Gradient Search RS SPSA STRONG STRONG-StageI

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 16/22

slide-17
SLIDE 17

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Ambulance (dim = 6)

Budget #104 0.5 1 1.5 2 2.5 3 Objective value 0.14 0.15 0.16 0.17 0.18 0.19 0.2 minimize Problem: Ambulance

Random Search Nelder-Mead Gradient Search RS SPSA STRONG STRONG-StageI

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 17/22

slide-18
SLIDE 18

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Route Prices (dim = 12)

Budget #104 1 2 3 4 5 Objective value 600 700 800 900 1000 1100 1200 1300 maximize Problem: RoutePrices

Random Search Nelder-Mead Gradient Search RS SPSA STRONG STRONG-StageI

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 18/22

slide-19
SLIDE 19

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Stochastic Activity Network (dim = 13)

Budget #104 2 4 6 8 10 Objective value 19 20 21 22 23 24 25 26 minimize Problem: SAN

Random Search Nelder-Mead Gradient Search RS SPSA STRONG STRONG-StageI

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 19/22

slide-20
SLIDE 20

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

POMDP Controller (dim = 10)

Budget #104 0.5 1 1.5 2 2.5 3 Objective value 136 137 138 139 140 141 142 143 minimize Problem: POMDPController

Random Search Nelder-Mead Gradient Search RS SPSA STRONG STRONG-StageI

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 20/22

slide-21
SLIDE 21

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Conclusions

Takeaways

  • 1. Robust performance of Nelder-Mead across problems.
  • 2. STRONG-Stage1 did as well as (or better than) STRONG.
  • 3. Random Search did better than expected.
  • 4. Performance of SPSA was sometimes highly variable.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 21/22

slide-22
SLIDE 22

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Future Work

Similar comparisons for:

  • High-dimensional problems.
  • Discrete/integer-ordered variables.
  • Stochastic constraints.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 22/22

slide-23
SLIDE 23

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Acknowledgments

This material is based upon work supported by the National Science Foundation under grants CMMI–1537394, CMMI–1254298, CMMI–1536895, and IIS–1247696, by the Air Force Office of Scientific Research under grants FA9550–12–1–0200, FA9550–15–1–0038, and FA9550–16–1–0046 and by the Army Research Office under grant W911NF–17–1–0094.

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 22/22

slide-24
SLIDE 24

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

MultiModal (dim = 2)

100 MultiModal Function Shape Demonstration Solution X1 50 Solution X2 50 0.5

  • 0.5
  • 1
  • 1.5
  • 2

100 Function value

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 22/22

slide-25
SLIDE 25

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

MultiModal (dim = 2)

Budget #105 0.5 1 1.5 2 2.5 3 Objective value

  • 2
  • 1.8
  • 1.6
  • 1.4
  • 1.2
  • 1
  • 0.8
  • 0.6
  • 0.4

minimize Problem: MultiModal

Random Search Nelder-Mead Gradient Search RS SPSA STRONG STRONG-StageI

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 22/22

slide-26
SLIDE 26

COMPARING SIMULATION-OPTIMIZATION ALGORITHMS DONG ET AL.

Ambulance (dim = 6)

0.5 1 1.5 2 2.5 3 x 10

4

0.14 0.15 0.16 0.17 0.18 0.19 0.2

Budget Objective value

minimize Problem: Ambulance

Nelder−Mead (IID) Nelder−Mead (CRN)

INTRODUCTION EVALUATION EXPERIMENTS PLOTS CONCLUSIONS 22/22