Hyperparameter Optimization Albert-Ludwigs-Universitt Freiburg - - PowerPoint PPT Presentation

▶
hyperparameter optimization
SMART_READER_LITE
LIVE PREVIEW

Hyperparameter Optimization Albert-Ludwigs-Universitt Freiburg - - PowerPoint PPT Presentation

Apr 04, 2024 26 likes •398 views

Surrogate Benchmarks for Hyperparameter Optimization Albert-Ludwigs-Universitt Freiburg Holger Hoos Katharina Eggensperger Kevin Leyton-Brown Frank Hutter University of British Columbia University of Freiburg {hoos,kevinlb}@cs.ubc.ca

slide-1
SLIDE 1

Albert-Ludwigs-Universität Freiburg

Surrogate Benchmarks for Hyperparameter Optimization

Katharina Eggensperger Frank Hutter

University of Freiburg {eggenspk,fh}@cs.uni-freiburg.de

Holger Hoos Kevin Leyton-Brown

University of British Columbia {hoos,kevinlb}@cs.ubc.ca

slide-2
SLIDE 2

Albert-Ludwigs-Universität Freiburg

Evaluation of Methods for Hyperparameter Optimization is expensive !

Problem:

slide-3
SLIDE 3
  • Benchmarking Hyperparameter Optimization

Methods

  • Constructing Surrogates
  • Using Surrogate Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

3

Outline

MetaSEL’14

slide-4
SLIDE 4
  • Benchmarking Hyperparameter Optimization

Methods

  • Constructing Surrogates
  • Using Surrogate Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

4

Outline

MetaSEL’14

slide-5
SLIDE 5

Bayesian Optimization Methods

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

5

MetaSEL’14

Configuration space Λ

Run algorithm with configuration λi

λi

Optimizer

Uses internal model M

Performance f(λi)

slide-6
SLIDE 6
  • Standard benchmark problems
  • Easy-to-use software

Then:

  • Run each optimizer on each benchmark X

multiple times

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

6

What do we need for an empirical comparison

slide-7
SLIDE 7
  • Standard benchmark problems
  • Easy-to-use software

Then:

  • Run each optimizer on each benchmark X

multiple times

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

6

What do we need for an empirical comparison

Evaluation of X is expensive

slide-8
SLIDE 8

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

7

Benchmarking hyperparameter

  • ptimization methods

Neural Network, configuration space Λ:

slide-9
SLIDE 9

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

7

Benchmarking hyperparameter

  • ptimization methods

Neural Network, configuration space Λ: categorical hyperparameter

slide-10
SLIDE 10

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

7

Benchmarking hyperparameter

  • ptimization methods

Neural Network, configuration space Λ: conditional hyperparameter

slide-11
SLIDE 11

Benchmarking hyperparameter optimization methods

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

8

MetaSEL’14

Neural network

slide-12
SLIDE 12

Benchmarking hyperparameter optimization methods

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

8

MetaSEL’14

Neural network

slide-13
SLIDE 13
  • Benchmarking Hyperparameter Optimization

Methods

  • Constructing Surrogates
  • Using Surrogate Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

9

Outline

MetaSEL’14

slide-14
SLIDE 14

Surrogate Benchmark 𝑌′

  • cheap-to-evaluate
  • Can be used like the real benchmark X
  • Behaves like X

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

10

𝑌

Configuration 𝜇 Performance 𝑔(𝜇)

MetaSEL’14

slide-15
SLIDE 15

Surrogate Benchmark 𝑌′

  • cheap-to-evaluate
  • Can be used like the real benchmark X
  • Behaves like X

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

11

Regression model 𝑌′

Configuration 𝜇 Performance 𝑔(𝜇)

MetaSEL’14

slide-16
SLIDE 16

Constructing a Surrogate for Benchmark X

  • 1. Collect data
  • 2. Choose a regression model
  • 3. Train and store model

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

12

slide-17
SLIDE 17

Training data: 𝜇1, 𝑔 𝜇1 , … , 𝜇𝑜, 𝑔 𝜇𝑜

  • Dense sampling in high performance regions
  • Good overall coverage

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

13

  • 1. Collect data for benchmark X
slide-18
SLIDE 18

Training data: 𝜇1, 𝑔 𝜇1 , … , 𝜇𝑜, 𝑔 𝜇𝑜

  • Dense sampling in high performance regions
  • Good overall coverage

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

13

  • 1. Collect data for benchmark X

Run optimizers on benchmark X

slide-19
SLIDE 19

Training data: 𝜇1, 𝑔 𝜇1 , … , 𝜇𝑜, 𝑔 𝜇𝑜

  • Dense sampling in high performance regions
  • Good overall coverage

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

13

  • 1. Collect data for benchmark X

Run optimizers on benchmark X Run random search on benchmark X

slide-20
SLIDE 20
  • 2. Choice of Regression Models

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

14

SVM K-nearest neighbour Gradient Boosting Random Forests Gaussian Processes Linear Regression Bayesian Neural Network Ridge Regression

MetaSEL’14

slide-21
SLIDE 21
  • 2. Choice of Regression Models

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

15

SVM K-nearest neighbour Gradient Boosting Random Forests Gaussian Processes Linear Regression Bayesian Neural Network Ridge Regression

MetaSEL’14

slide-22
SLIDE 22
  • 2. Choice of Regression Models

Can we quantify the performance of a new

  • ptimizer?
  • Leave-one-optimizer-out setting
  • Train model on data gathered by all but one
  • ptimizer
  • Test on remaining data

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

16

MetaSEL’14

slide-23
SLIDE 23

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

17

  • 2. Choice of Regression Models

Leave-one-optimizer-out setting

Neural Network True performance Random forest prediction

slide-24
SLIDE 24

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

17

  • 2. Choice of Regression Models

Leave-one-optimizer-out setting

Neural Network Random Forest

slide-25
SLIDE 25

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

17

  • 2. Choice of Regression Models

Leave-one-optimizer-out setting

Neural Network Gaussian Process k-nearest-neighbour Gradient Boosting nuSVR Random Forest

slide-26
SLIDE 26
  • 2. Choice of Regression Models

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

18

SVM K-nearest neighbour Gradient Boosting Random Forests Gaussian Processes Linear Regression Bayesian Neural Network Ridge Regression

MetaSEL’14

slide-27
SLIDE 27
  • Benchmarking Hyperparameter Optimization

Methods

  • Constructing Surrogates
  • Using Surrogate Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

19

Outline

MetaSEL’14

slide-28
SLIDE 28

Using Surrogate Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

20

MetaSEL’14

Neural Network

slide-29
SLIDE 29

Using Surrogate Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

20

MetaSEL’14

Real Benchmark

Neural Network

slide-30
SLIDE 30

Using Surrogate Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

20

MetaSEL’14

GP-based benchmark RF-based benchmark Real Benchmark

Neural Network

slide-31
SLIDE 31

Using Surrogate Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

21

MetaSEL’14

Real Benchmark GP-based benchmark RF-based benchmark

Neural Network

40h <200s <200s One optimization run: Whole comparison: 50d <1.5h <1.5h

slide-32
SLIDE 32
  • Extensive testing at early development stages
  • Fast comparison of different hyperparameter
  • ptimization methods
  • Metaoptimization of existing hyperparameter
  • ptimization methods

MetaSEL’14

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

23

Applications

slide-33
SLIDE 33

Conclusion

Can we construct cheap-to evaluate and realistic hyperparameter optimization benchmarks?

Yes, based on random forests and

Gaussian process regression models 

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

24

MetaSEL’14

slide-34
SLIDE 34

Conclusion

Can we construct cheap-to evaluate and realistic hyperparameter optimization benchmarks?

Yes, based on random forests and

Gaussian process regression models But, some work needs to be done for high dimensional benchmarks.

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

24

MetaSEL’14

slide-35
SLIDE 35

Albert-Ludwigs-Universität Freiburg

Thank you for your attention

This presentation was supported by an ECCAI travel award and the ECCAI sponsors

More information on hyperparameter optimization benchmarks can be found on automl.org/hpolib

slide-36
SLIDE 36

Regression models

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

26

MetaSEL’14

slide-37
SLIDE 37

Benchmarks

Surrogates for Hyperparameter Optimization Benchmarks – Eggensperger, Hutter, Hoos, and Leyton-Brown

27

MetaSEL’14