Class Structure Last time: Midterm This time: Fast Learning Next - - PowerPoint PPT Presentation

▶
class structure
SMART_READER_LITE
LIVE PREVIEW

Class Structure Last time: Midterm This time: Fast Learning Next - - PowerPoint PPT Presentation

Mar 01, 2023 216 likes •874 views

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

slide-1
SLIDE 1

Lecture 11: Fast Reinforcement Learning 1

Emma Brunskill

CS234 Reinforcement Learning

Winter 2019

1With many slides from or derived from David Silver, Examples new Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 1 / 65

slide-2
SLIDE 2

Class Structure

Last time: Midterm This time: Fast Learning Next time: Fast Learning

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 2 / 65

slide-3
SLIDE 3

Up Till Now

Discussed optimization, generalization, delayed consequences

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 3 / 65

slide-4
SLIDE 4

Teach Computers to Help Us

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 4 / 65

slide-5
SLIDE 5

Computational Efficiency and Sample Efficiency

Computational Efficiency Sample Efficiency

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 5 / 65

slide-6
SLIDE 6

Algorithms Seen So Far

How many steps did it take for DQN to learn a good policy for pong?

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 6 / 65

slide-7
SLIDE 7

Evaluation Criteria

How do we evaluate how ”good” an algorithm is? If converges? If converges to optimal policy? How quickly reaches optimal policy? Mistakes make along the way? Will introduce different measures to evaluate RL algorithms

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 7 / 65

slide-8
SLIDE 8

Settings, Frameworks & Approaches

Over next couple lectures will consider 2 settings, multiple frameworks, and approaches Settings: Bandits (single decisions), MDPs Frameworks: evaluation criteria for formally assessing the quality of a RL algorithm Approaches: Classes of algorithms for achieving particular evaluation criteria in a certain set Note: We will see that some approaches can achieve multiple frameworks in multiple settings

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 8 / 65

slide-9
SLIDE 9

Today

Setting: Introduction to multi-armed bandits Framework: Regret Approach: Optimism under uncertainty Framework: Bayesian regret Approach: Probability matching / Thompson sampling

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 9 / 65

slide-10
SLIDE 10

Multiarmed Bandits

Multi-armed bandit is a tuple of (A, R) A : known set of m actions (arms) Ra(r) = P[r | a] is an unknown probability distribution over rewards At each step t the agent selects an action at ∈ A The environment generates a reward rt ∼ Rat Goal: Maximize cumulative reward t

τ=1 rτ

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 10 / 65

slide-11
SLIDE 11

Regret

Action-value is the mean reward for action a Q(a) = E[r | a] Optimal value V ∗ V ∗ = Q(a∗) = max

a∈A Q(a)

Regret is the opportunity loss for one step lt = E[V ∗ − Q(at)] Total Regret is the total opportunity loss Lt = E[

t

  • τ=1

V ∗ − Q(aτ)] Maximize cumulative reward ⇐ ⇒ minimize total regret

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 11 / 65

slide-12
SLIDE 12

Evaluating Regret

Count Nt(a) is expected number of selections for action a Gap ∆a is the difference in value between action a and optimal action a∗, ∆i = V ∗ − Q(ai) Regret is a function of gaps and counts Lt = E t

  • τ=1

V ∗ − Q(aτ)

  • =
  • a∈A

E[Nt(a)](V ∗ − Q(a)) =

  • a∈A

E[Nt(a)]∆a A good algorithm ensures small counts for large gap, but gaps are not known

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 12 / 65

slide-13
SLIDE 13

Greedy Algorithm

We consider algorithms that estimate ˆ Qt(a) ≈ Q(a) Estimate the value of each action by Monte-Carlo evaluation ˆ Qt(a) = 1 Nt(a)

T

  • t=1

rt1(at = a) The greedy algorithm selects action with highest value a∗

t = arg max a∈A

ˆ Qt(a) Greedy can lock onto suboptimal action, forever

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 13 / 65

slide-14
SLIDE 14

ǫ-Greedy Algorithm

The ǫ-greedy algorithm proceeds as follows:

With probability 1 − ǫ select at = arg maxa∈A ˆ Qt(a) With probability ǫ select a random action

Always will be making a sub-optimal decision ǫ fraction of the time Already used this in prior homeworks

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 14 / 65

slide-15
SLIDE 15

Toy Example: Ways to Treat Broken Toes1

Consider deciding how to best treat patients with broken toes Imagine have 3 possible options: (1) surgery (2) buddy taping the broken toe with another toe, (3) do nothing Outcome measure / reward is binary variable: whether the toe has healed (+1) or not healed (0) after 6 weeks, as assessed by x-ray

1Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 15 / 65

slide-16
SLIDE 16

Toy Example: Ways to Treat Broken Toes1

Consider deciding how to best treat patients with broken toes Imagine have 3 common options: (1) surgery (2) surgical boot (3) buddy taping the broken toe with another toe Outcome measure is binary variable: whether the toe has healed (+1)

  • r not (0) after 6 weeks, as assessed by x-ray

Model as a multi-armed bandit with 3 arms, where each arm is a Bernoulli variable with an unknown parameter θi Check your understanding: what does a pull of an arm / taking an action correspond to? Why is it reasonable to model this as a multi-armed bandit instead of a Markov decision process?

1Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 16 / 65

slide-17
SLIDE 17

Toy Example: Ways to Treat Broken Toes1

Imagine true (unknown) Bernoulli reward parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

1Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 17 / 65

slide-18
SLIDE 18

Toy Example: Ways to Treat Broken Toes, Greedy1

Imagine true (unknown) Bernoulli reward parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

Greedy

1

Sample each arm once Take action a1 (r ∼Bernoulli(0.95)), get +1, ˆ Q(a1) = 1 Take action a2 (r ∼Bernoulli(0.90)), get +1, ˆ Q(a2) = 1 Take action a3 (r ∼Bernoulli(0.1)), get 0, ˆ Q(a3) = 0

2

What is the probability of greedy selecting each arm next? Assume ties are split uniformly.

1Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 18 / 65

slide-19
SLIDE 19

Toy Example: Ways to Treat Broken Toes, Optimism, Assessing Regret of Greedy

True (unknown) Bernoulli reward parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

Greedy Action Optimal Action Regret a1 a1 a2 a1 a3 a1 a1 a1 a2 a1 Will greedy ever select a3 again? If yes, why? If not, is this a problem?

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 19 / 65

slide-20
SLIDE 20

Toy Example: Ways to Treat Broken Toes, ǫ-Greedy1

Imagine true (unknown) Bernoulli reward parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

ǫ-greedy

1

Sample each arm once Take action a1 (r ∼Bernoulli(0.95)), get +1, ˆ Q(a1) = 1 Take action a2 (r ∼Bernoulli(0.90)), get +1, ˆ Q(a2) = 1 Take action a3 (r ∼Bernoulli(0.1)), get 0, ˆ Q(a3) = 0

2

Let ǫ = 0.1

3

What is the probability ǫ-greedy will pull each arm next? Assume ties are split uniformly.

1Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 20 / 65

slide-21
SLIDE 21

Toy Example: Ways to Treat Broken Toes, Optimism, Assessing Regret of Greedy

True (unknown) Bernoulli reward parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

UCB1 (Auer, Cesa-Bianchi, Fischer 2002) Action Optimal Action Regret a1 a1 a2 a1 a3 a1 a1 a1 a2 a1 Will ǫ-greedy ever select a3 again? If ǫ is fixed, how many times will each arm be selected?

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 21 / 65

slide-22
SLIDE 22

Check Your Understanding

Count Nt(a) is expected number of selections for action a Gap ∆a is the difference in value between action a and optimal action a∗, ∆i = V ∗ − Q(ai) Regret is a function of gaps and counts Lt = E t

  • τ=1

V ∗ − Q(aτ)

  • =
  • a∈A

E[Nt(a)](V ∗ − Q(a)) =

  • a∈A

E[Nt(a)]∆a A good algorithm ensures small counts for large gap, but gaps are not known Check your understanding: Does fixed ǫ = 0.1 greedy have large regret ?

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 22 / 65

slide-23
SLIDE 23

”Good”: Sublinear or below regret

Explore forever: have linear total regret Explore never: have linear total regret Is it possible to achieve sublinear regret?

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 23 / 65

slide-24
SLIDE 24

Types of Regret bounds

Problem independent: Bound how regret grows as a function of T, the total number of time steps the algorithm operates for Problem dependent: Bound regret as a function of the number of times we pull each arm and the gap between the reward for the pulled arm a∗

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 24 / 65

slide-25
SLIDE 25

Lower Bound

Use lower bound to determine how hard this problem is The performance of any algorithm is determined by similarity between

  • ptimal arm and other arms

Hard problems have similar looking arms with different means This is described formally by the gap ∆a and the similarity in distributions DKL(RaRa∗) Theorem (Lai and Robbins): Asymptotic total regret is at least logarithmic in number of steps lim

t→∞ Lt ≥ log t

  • a|∆a>0

∆a DKL(RaRa∗) Promising in that lower bound is sublinear

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 25 / 65

slide-26
SLIDE 26

Approach: Optimism in the Face of Uncertainty

Choose actions that that might have a high value Why? Two outcomes:

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 26 / 65

slide-27
SLIDE 27

Upper Confidence Bounds

Estimate an upper confidence Ut(a) for each action value, such that Q(a) ≤ Ut(a) with high probability This depends on the number of times Nt(a) action a has been selected Select action maximizing Upper Confidence Bound (UCB) at = arg max

a∈A[Ut(a)]

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 27 / 65

slide-28
SLIDE 28

Hoeffding’s Inequality

Theorem (Hoeffding’s Inequality): Let X1, . . . , Xn be i.i.d. random variables in [0, 1], and let ¯ Xn = 1

n

n

τ=1 Xτ be the sample mean. Then

P

  • E [X] > ¯

Xn + u

  • ≤ exp(−2nu2)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 28 / 65

slide-29
SLIDE 29

High Probability Regret Bound for UCB Multi-armed Bandit

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 29 / 65

slide-30
SLIDE 30

High Probability Regret Bound for UCB Multi-armed Bandit

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 30 / 65

slide-31
SLIDE 31

High Probability Regret Bound for UCB Multi-armed Bandit

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 31 / 65

slide-32
SLIDE 32

High Probability Regret Bound for UCB Multi-armed Bandit

Regret(UCB, T) =

T

  • t=1

Q(a∗) − Q(at)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 32 / 65

slide-33
SLIDE 33

UCB Bandit Regret

This leads to the UCB1 algorithm at = arg max

a∈A[ ˆ

Q(a) +

  • 2 log t

Nt(a) ] Theorem: The UCB algorithm achieves logarithmic asymptotic total regret lim

t→∞ Lt ≤ 8 log t

  • a|∆a>0

∆a

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 33 / 65

slide-34
SLIDE 34

Toy Example: Ways to Treat Broken Toes, Thompson Sampling1

True (unknown) parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

Optimism under uncertainty, UCB1 (Auer, Cesa-Bianchi, Fischer 2002)

1

Sample each arm once

1Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 34 / 65

slide-35
SLIDE 35

Toy Example: Ways to Treat Broken Toes, Optimism1

True (unknown) parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

UCB1 (Auer, Cesa-Bianchi, Fischer 2002)

1

Sample each arm once

Take action a1 (r ∼Bernoulli(0.95)), get +1, ˆ Q(a1) = 1 Take action a2 (r ∼Bernoulli(0.90)), get +1, ˆ Q(a2) = 1 Take action a3 (r ∼Bernoulli(0.1)), get 0, ˆ Q(a3) = 0

1Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 35 / 65

slide-36
SLIDE 36

Toy Example: Ways to Treat Broken Toes, Optimism1

True (unknown) parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

UCB1 (Auer, Cesa-Bianchi, Fischer 2002)

1

Sample each arm once Take action a1 (r ∼Bernoulli(0.95)), get +1, ˆ Q(a1) = 1 Take action a2 (r ∼Bernoulli(0.90)), get +1, ˆ Q(a2) = 1 Take action a3 (r ∼Bernoulli(0.1)), get 0, ˆ Q(a3) = 0

2

Set t = 3, Compute upper confidence bound on each action UCB(a) = ˆ Q(a) +

  • 2 log t

Nt(a)

1Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 36 / 65

slide-37
SLIDE 37

Toy Example: Ways to Treat Broken Toes, Optimism1

True (unknown) parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

UCB1 (Auer, Cesa-Bianchi, Fischer 2002)

1

Sample each arm once Take action a1 (r ∼Bernoulli(0.95)), get +1, ˆ Q(a1) = 1 Take action a2 (r ∼Bernoulli(0.90)), get +1, ˆ Q(a2) = 1 Take action a3 (r ∼Bernoulli(0.1)), get 0, ˆ Q(a3) = 0

2

Set t = 3, Compute upper confidence bound on each action UCB(a) = ˆ Q(a) +

  • 2 log t

Nt(a)

3

t = 3, Select action at = arg maxa UCB(a),

4

Observe reward 1

5

Compute upper confidence bound on each action

1Note:This is a made up example. This is not the actual expected efficacies of the Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 37 / 65

slide-38
SLIDE 38

Toy Example: Ways to Treat Broken Toes, Optimism1

True (unknown) parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

UCB1 (Auer, Cesa-Bianchi, Fischer 2002)

1

Sample each arm once Take action a1 (r ∼Bernoulli(0.95)), get +1, ˆ Q(a1) = 1 Take action a2 (r ∼Bernoulli(0.90)), get +1, ˆ Q(a2) = 1 Take action a3 (r ∼Bernoulli(0.1)), get 0, ˆ Q(a3) = 0

2

Set t = 3, Compute upper confidence bound on each action UCB(a) = ˆ Q(a) +

  • 2 log t

Nt(a)

3

t = t + 1, Select action at = arg maxa UCB(a),

4

Observe reward 1

5

Compute upper confidence bound on each action

1Note:This is a made up example. This is not the actual expected efficacies of the Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 38 / 65

slide-39
SLIDE 39

Toy Example: Ways to Treat Broken Toes, Optimism, Assessing Regret

True (unknown) parameters for each arm (action) are

surgery: Q(a1) = θ1 = .95 buddy taping: Q(a2) = θ2 = .9 doing nothing: Q(a3) = θ3 = .1

UCB1 (Auer, Cesa-Bianchi, Fischer 2002) Action Optimal Action Regret a1 a1 a2 a1 a3 a1 a1 a1 a2 a1

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 39 / 65

slide-40
SLIDE 40

Check Your Understanding

An alternative would be to always select the arm with the highest lower bound Why can this yield linear regret? Consider a two arm case for simplicity

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 40 / 65

slide-41
SLIDE 41

Bayesian Bandits

So far we have made no assumptions about the reward distribution R

Except bounds on rewards

Bayesian bandits exploit prior knowledge of rewards, p[R] They compute posterior distribution of rewards p[R | ht], where ht = (a1, r1, . . . , at−1, rt−1) Use posterior to guide exploration

Upper confidence bounds (Bayesian UCB) Probability matching (Thompson Sampling)

Better performance if prior knowledge is accurate

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 41 / 65

slide-42
SLIDE 42

Regret and Bayesian Regret

Frequentist regret assumes a true (unknown) set of parameters Regret(A, T; θ) =

T

  • t=1

E

  • Q(a∗) − Q(at) ≤

T

  • t=1

Ut(at) − Q(at)|θ

  • Bayesian regret assumes there is a prior over parameters

BayesRegret(A, T; θ) = Eθ∼pθ T

  • t=1

E

  • Q(a∗) − Q(at) ≤

T

  • t=1

Ut(at) − Q(at)|θ

  • *Note: Bayes regret and regret can be related using Markov

inequality

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 42 / 65

slide-43
SLIDE 43

Bayesian UCB Example: Independent Gaussians

Assume reward distribution is Gaussian, Ra(r) = N(r; µa, σ2

a)

Compute Gaussian posterior over µa and σ2

a (by Bayes law)

p[µa, σ2

a | ht] ∝ p[µa, σ2 a]

  • t|at=a

N(rt; µa, σ2

a)

Pick action that maximizes standard deviation of Q(a) at = arg max

a∈A µa + c

cσa

  • N(a)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 43 / 65

slide-44
SLIDE 44

Probability Matching

Assume have a parametric distribution over rewards for each arm Probability matching selects action a according to probability that a is the optimal action π(a | ht) = P[Q(a) > Q(a′), ∀a′ = a | ht] Probability matching is optimistic in the face of uncertainty

Uncertain actions have higher probability of being max

Can be difficult to compute analytically from posterior

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 44 / 65

slide-45
SLIDE 45

Thompson sampling implements probability matching

Thompson sampling: π(a | ht) = P[Q(a) > Q(a′), ∀a′ = a | ht] = ER|ht

  • 1(a = arg max

a∈A Q(a))

  • Emma Brunskill (CS234 Reinforcement Learning )

Lecture 11: Fast Reinforcement Learning 1 Winter 2019 45 / 65

slide-46
SLIDE 46

Thompson sampling implements probability matching

Thompson sampling: π(a | ht) = P[Q(a) > Q(a′), ∀a′ = a | ht] = ER|ht

  • 1(a = arg max

a∈A Q(a))

  • Use Bayes law to compute posterior distribution p[R | ht]

Sample a reward distribution R from posterior Compute action-value function Q(a) = E[Ra] Select action maximizing value on sample, at = arg maxa∈A Q(a) Update posterior

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 46 / 65

slide-47
SLIDE 47

Toy Example: Ways to Treat Broken Toes, Thompson Sampling

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose Beta(1,1) (Uniform)

1

Sample a Bernoulli parameter given current prior over each arm Beta(1,1), Beta(1,1), Beta(1,1):

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 47 / 65

slide-48
SLIDE 48

Toy Example: Ways to Treat Broken Toes, Thompson Sampling2

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose Beta(1,1)

1

Sample a Bernoulli parameter given current prior over each arm Beta(1,1), Beta(1,1), Beta(1,1): 0.3 0.5 0.6

2

Select a = arg maxa∈A Q(a) = arg maxainA θ(a) =

2Note:This is a made up example. This is not the actual expected efficacies of the

various treatment options for a broken toe

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 48 / 65

slide-49
SLIDE 49

Toy Example: Ways to Treat Broken Toes, Thompson Sampling

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose θi ∼Beta(1,1)

1

Per arm, sample a Bernoulli θ given prior: 0.3 0.5 0.6

2

Select at = arg maxa∈A Q(a) = arg maxainA θ(a) = 3

3

Observe the patient outcome’s outcome: 0

4

Update the posterior over the Q(at) = Q(a3) value for the arm pulled

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 49 / 65

slide-50
SLIDE 50

Toy Example: Ways to Treat Broken Toes, Thompson Sampling

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose θi ∼Beta(1,1)

1

Sample a Bernoulli parameter given current prior over each arm Beta(1,1), Beta(1,1), Beta(1,1): 0.3 0.5 0.6

2

Select at = arg maxa∈A Q(a) = arg maxainA θ(a) = 3

3

Observe the patient outcome’s outcome: 0

4

Update the posterior over the Q(at) = Q(a1) value for the arm pulled

Beta(c1, c2) is the conjugate distribution for Bernoulli If observe 1, c1 + 1 else if observe 0 c2 + 1

5

New posterior over Q value for arm pulled is:

6

New posterior p(Q(a3)) = p(θ(a3) = Beta(1, 2)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 50 / 65

slide-51
SLIDE 51

Toy Example: Ways to Treat Broken Toes, Thompson Sampling

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose θi ∼Beta(1,1)

1

Sample a Bernoulli parameter given current prior over each arm Beta(1,1), Beta(1,1), Beta(1,1): 0.3 0.5 0.6

2

Select at = arg maxa∈A Q(a) = arg maxainA θ(a) = 1

3

Observe the patient outcome’s outcome: 0

4

New posterior p(Q(a1)) = p(θ(a1) = Beta(1, 2)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 51 / 65

slide-52
SLIDE 52

Toy Example: Ways to Treat Broken Toes, Thompson Sampling

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose θi ∼Beta(1,1)

1

Sample a Bernoulli parameter given current prior over each arm Beta(1,1), Beta(1,1), Beta(1,2): 0.7, 0.5, 0.3

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 52 / 65

slide-53
SLIDE 53

Toy Example: Ways to Treat Broken Toes, Thompson Sampling

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose θi ∼Beta(1,1)

1

Sample a Bernoulli parameter given current prior over each arm Beta(1,1), Beta(1,1), Beta(1,2): 0.7, 0.5, 0.3

2

Select at = arg maxa∈A Q(a) = arg maxainA θ(a) = 1

3

Observe the patient outcome’s outcome: 1

4

New posterior p(Q(a1)) = p(θ(a1) = Beta(2, 1)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 53 / 65

slide-54
SLIDE 54

Toy Example: Ways to Treat Broken Toes, Thompson Sampling

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose θi ∼Beta(1,1)

1

Sample a Bernoulli parameter given current prior over each arm Beta(2,1), Beta(1,1), Beta(1,2): 0.71, 0.65, 0.1

2

Select at = arg maxa∈A Q(a) = arg maxainA θ(a) = 1

3

Observe the patient outcome’s outcome: 1

4

New posterior p(Q(a1)) = p(θ(a1) = Beta(3, 1)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 54 / 65

slide-55
SLIDE 55

Toy Example: Ways to Treat Broken Toes, Thompson Sampling

True (unknown) Bernoulli parameters for each arm/action

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Thompson sampling: Place a prior over each arm’s parameter. Here choose θi ∼Beta(1,1)

1

Sample a Bernoulli parameter given current prior over each arm Beta(2,1), Beta(1,1), Beta(1,2): 0.75, 0.45, 0.4

2

Select at = arg maxa∈A Q(a) = arg maxainA θ(a) = 1

3

Observe the patient outcome’s outcome: 1

4

New posterior p(Q(a1)) = p(θ(a1) = Beta(4, 1)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 55 / 65

slide-56
SLIDE 56

Toy Example: Ways to Treat Broken Toes, Thompson Sampling vs Optimism

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

How does the sequence of arm pulls compare in this example so far? Optimism TS Optimal Regret Optimism Regret TS a1 a3 a2 a1 a3 a1 a1 a1 a2 a1

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 56 / 65

slide-57
SLIDE 57

Toy Example: Ways to Treat Broken Toes, Thompson Sampling vs Optimism

Surgery: θ1 = .95 / Taping: θ2 = .9 / Nothing: θ3 = .1

Incurred regret? Optimism TS Optimal Regret Optimism Regret TS a1 a3 a1 a2 a1 a1 0.05 a3 a1 a1 0.85 a1 a1 a1 a2 a1 a1 0.05

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 57 / 65

slide-58
SLIDE 58

Thompson sampling implements probability matching

Thompson sampling(1929) achieves Lai and Robbins lower bound Bounds for optimism are tighter than for Thompson sampling But empirically Thompson sampling can be extremely effective

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 58 / 65

slide-59
SLIDE 59

Thompson Sampling for News Article Recommendation (Chapelle and Li, 2010)

Contextual bandit: input context which impacts reward of each arm, context sampled iid each step Arms = articles Reward = click (+1) on article (Q(a)=click through rate)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 59 / 65

slide-60
SLIDE 60

Bayesian Regret Bounds for Thompson Sampling

Regret(UCB,T) BayesRegret(TS, T) = Eθ∼pθ T

  • t=1

f ∗(a∗) − f ∗(at)

  • Posterior sampling has the same (ignoring constants) regret bounds

as UCB

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 60 / 65

slide-61
SLIDE 61

Optimistic Initialization

Simple and practical idea: initialize Q(a) to high value Update action value by incremental Monte-Carlo evaluation Starting with N(a) > 0 ˆ Qt(at) = ˆ Qt−1 + 1 Nt(at)(rt − ˆ Qt−1) Encourages systematic exploration early on But can still lock onto suboptimal action Depends on how high initialize Q Check your understanding: What is the downside to initializing Q too high?

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 61 / 65

slide-62
SLIDE 62

Greedy Bandit Algorithms and Optimistic Initialization

Greedy: Linear total regret Constant ǫ-greedy: Linear total regret Decaying ǫ-greedy: Sublinear regret but schedule for decaying ǫ requires knowledge of gaps, which are unknown Optimistic initialization: Sublinear regret if initialize values sufficiently optimistically, else linear regret Check your understanding: why does fixed ǫ-greedy have linear regret? (Do a proof sketch)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 62 / 65

slide-63
SLIDE 63

Consider Montezuma’s revenge

EB: move this to generalization and efficiency later on Bellemare et al. ”Unifying Count-Based Exploration and Intrinsic Motivation” Enormously better than standard DQN with ǫ-greedy approach Uses principle of optimism under uncertainty which we will see today

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 63 / 65

slide-64
SLIDE 64

Calculating UCB

Pick a probability p that true value exceeds UCB Now solve for Ut(a) exp(−2Nt(a)Ut(a)2) = p Ut(a) =

  • − log p

2Nt(a) Reduce p as we observe more rewards, e.q. p = t−4 Ensures we select optimal action as t → ∞ Ut(a) =

  • 2 log t

Nt(a)

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 64 / 65

slide-65
SLIDE 65

UCB1

This leads to the UCB1 algorithm at = arg max

a∈A[ ˆ

Q(a) +

  • 2 log t

Nt(a) ]

Emma Brunskill (CS234 Reinforcement Learning ) Lecture 11: Fast Reinforcement Learning 1 Winter 2019 65 / 65