Reinforcement Learning: Basic models and algorithms Optimal - - PowerPoint PPT Presentation

▶
reinforcement learning basic models and algorithms
SMART_READER_LITE
LIVE PREVIEW

Reinforcement Learning: Basic models and algorithms Optimal - - PowerPoint PPT Presentation

Mar 26, 2024 97 likes •407 views

Reinforcement Learning: Basic models and algorithms Optimal decisions, Part VII Christos Dimitrakakis Chalmers November 20, 2013 Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 1 / 28

slide-1
SLIDE 1

Reinforcement Learning: Basic models and algorithms

Optimal decisions, Part VII Christos Dimitrakakis

Chalmers

November 20, 2013

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 1 / 28

slide-2
SLIDE 2

Introduction

The reinforcement learning problem and MDPs

Markov decision processes

Description of environments Solutions to bandit problems

Algorithms for unknown MDPs.

Stochastic exact algorithms Stochastic estimation algorithms Online algorithms

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 2 / 28

slide-3
SLIDE 3

Introduction Bandit problems

The stochastic n-armed bandit problem

P = {Pi | i = 1, . . . , n} . rt | at = i ∼ Pi. Eπ Ut = Eπ

T

  • k=t

rk, a∗

t max {E(rt | at = i) | i = 1, . . . , n} .

P = {Pi(· | ω) | ω ∈ Ω} , rt | at = i, ω∗ = ω ∼ Pi(r | ω∗). (1.1) Eπ

ξ Ut = Eπ ξ T

  • k=t

rk. (1.2)

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 3 / 28

slide-4
SLIDE 4

Introduction Estimation and Robbins-Monro approximation

Algorithm 1 Robbins-Monro bandit algorithm

1: input Step-sizes (αt)t, initial estimates (µi,0)i, policy π. 2: for t = 1, . . . , T do 3:

Take action at = i with probability π(i | a1, . . . , at−1, r1, . . . , rt−1).

4:

Observe reward rt.

5:

µt,i = αi,trt + (1 − αi,t)µi,t−1 // estimation step

6:

µt,i = µj,t−1 for j = i.

7: end for 8: return µT

Definition 1

ǫ-greedy action selection (w.p. 1 − ǫ, select an apparently best action, otherwise a random action) ˆ π∗

ǫ (1 − ǫt)ˆ

π∗

t + ǫtUnif (A),

(1.3) ˆ π∗

t (i) = I

  • i ∈ ˆ

A∗

t

  • /| ˆ

A∗

t |,

ˆ A∗

t = arg max i∈A

µt,i (1.4)

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 4 / 28

slide-5
SLIDE 5

Introduction Estimation and Robbins-Monro approximation

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 0.01 0.1 0.5

Figure: ǫt = 0.1, α ∈ {0.01, 0.1, 0.5}.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 5 / 28

slide-6
SLIDE 6

Introduction Estimation and Robbins-Monro approximation

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 0.0 0.1 1.0

Figure: ǫt = ǫ, α = 0.1.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 6 / 28

slide-7
SLIDE 7

Introduction Estimation and Robbins-Monro approximation

Main idea of the algorithm

Estimate parameters Act according to the estimates

Requirements

Good estimation procedure. Balance estimation with getting rewards!

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 7 / 28

slide-8
SLIDE 8

Introduction The theory of the approximation

Consider the algorithm µt+1 = µt + αtzt+1. (1.5) Let ht = {µt, zt, αt, . . .} be the history of the algorithm.

Assumption 1

Assume a function f : Rn → R such that: (i) f (x) ≥ 0 for all x ∈ Rn. (ii) (Lipschitz derivative) f is continuously differentiable and ∃L > 0 such that: ∇f (x) − ∇f (y) ≤ L x − y , ∀ x, y ∈ Rn (iii) (Pseudo-gradient) ∃c > 0 such that: c ∇f (µt)2 ≤ −∇f (µt)⊤ E(zt+1 | ht), ∀ t. (iv) ∃K1, K2 > 0 such that E(zt+12 | ht) ≤ K1 + K2 ∇f (µt)2

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 8 / 28

slide-9
SLIDE 9

Introduction The theory of the approximation

Theorem 2

For the algorithm µt+1 = µt + αtzt+1, where αt ≥ 0 satisfy

∞

  • t=0

αt = ∞,

∞

  • t=0

α2

t < ∞,

(1.6) and under Assumption 1, with probability 1:

1 The sequence {f (µt)} converges. 2 limt→∞ ∇f (µt) = 0. 3 Every limit point µ∗ of µt satisfies ∇f (µ∗) = 0.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 8 / 28

slide-10
SLIDE 10

Introduction The theory of the approximation

A demonstration

  • 0.5

0.5 1 1.5 2 200 400 600 800 1000 µt t 1/t 1/√t t−3/2

Figure: Estimation of the expectation of xt ∼ N (0.5, 1) using use three step-size schedules.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 9 / 28

slide-11
SLIDE 11

Dynamic problems

Algorithm 2 Generic reinforcement learning algorithm

1: input Update-rule f : Θ × S2 × A × R → Θ, initial parameters θ0 ∈ Θ, policy

π : S × Θ → D (A).

2: for t = 1, . . . , T do 3:

at ∼ π(· | θt, st) // take action

4:

Observe reward rt+1, state st+1.

5:

θt+1 = f (θt, st, at, rt+1, st+1) // update estimate

6: end for

Questions

What should we estimate? What policy should we use?

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 10 / 28

slide-12
SLIDE 12

Dynamic problems

0.2 1

Figure: The chain task

s s1 s2 s3 s4 s5 V ∗(s) 6.672 7.111 7.689 8.449 9.449 Q∗(s, 1) 6.622 6.532 6.676 6.866 7.866 Q∗(s, 2) 6.672 7.111 7.689 8.449 9.449

Table: The chain task’s value function for γ = 0.95

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 11 / 28

slide-13
SLIDE 13

Dynamic problems Monte-Carlo policy evaluation and iteration

Algorithm 3 Stochastic policy evaluation

1: input Initial parameters v0, Markov policy π. 2: for s ∈ S do 3:

s1 = s.

4:

for k = 1, . . . , K do

5:

Run policy π for T steps.

6:

Observe utility Uk =

t rt.

7:

Update estimate vk+1(s) = vk(s) + αk(Uk − vk(s))

8:

end for

9: end for 10: return vK

For αk = 1/k and iterating over all S, this is the same as Monte-Carlo policy evaluation. Algorithm 4 Approximate policy iteration

1: input Initial parameters v0, inital Markov policy π0, stochastic estimator f . 2: for i = 1, . . . , N do 3:

Get estimate vi = f (vi−1, πi−1).

4:

Calculate new policy πi = arg maxπ L vi.

5: end for

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 12 / 28

slide-14
SLIDE 14

Dynamic problems Monte-Carlo policy evaluation and iteration

Monte Carlo update

Note that s1, . . . , sT contains sk, . . . , sT. Algorithm 5 Every-visit Monte-Carlo update

1: input Initial parameters vk, trajectory s1, . . . , sT, rewards r1, . . . , rT visit counts n. 2: for t = 1, . . . , T do 3:

Ut = T

t=1 rt.

4:

nt(st) = nt−1(st) + 1

5:

vt+1(st) = vt(s) + αnt(st)(st)(Ut − vt(st))

6:

nt(s) = nt−1(s), vt(s) = vt−1(s) ∀s = st.

7: end for 8: return vK

Example 3

Consider a two-state chain with P(st+1 = 1 | st = 0) = δ and P(st+1 = 1 | st = 1) = 1, and reward r(1) = 1, r(0) = 0. Then the every-visit estimate is biased.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 13 / 28

slide-15
SLIDE 15

Dynamic problems Monte-Carlo policy evaluation and iteration

Unbiased Monte-Carlo update

Algorithm 6 First-visit Monte-Carlo update

1: input Initial parameters vk, trajectory s1, . . . , sT, rewards r1, . . . , rT, visit counts n. 2: Let m ∈ N|S| be trajectory visit counts. 3: for t = 1, . . . , T do 4:

Ut = T

t=1 rt.

5:

nt(st) = nt−1(st) + 1

6:

mt(st) = mt−1(st) + 1

7:

vt+1(st) = vt(s) + αnt(st)(st)(Ut − vt(st)) if mt(st) = 1.

8:

nt(s) = nt−1(s), vt(s) = vt−1(s) otherwise

9: end for 10: return vK

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 14 / 28

slide-16
SLIDE 16

Dynamic problems Monte-Carlo policy evaluation and iteration

10-2 10-1 100 101 102 2000 4000 6000 8000 10000 iterations every first

vt − V π

Figure: Error as the number of iterations n increases, for first and every visit Monte Carlo estimation.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 15 / 28

slide-17
SLIDE 17

Dynamic problems Temporal difference methods

Temporal differences

The full stochastic update is of the form: vk+1(s) = vk(s) + α(Uk − vk(s)), Using the temporal difference error d(st, st+1) = v(st) − [r(st) + γv(st+1)], vk+1(s) = vk(s) + α

  • t

γtdt, dt d(st, st+1) (2.1) Stochastic, incremental, update: vt+1(s) = vt(s) + αγtdt. (2.2)

TD(λ)

Temporal-difference operator vn+1(i) = vni + τni, τn(i)

∞

  • t=0

Eπn,µ [(γλ)mdn(st, st+1) | s0 = i] . Stochastic update: vn+1(st) = vn(st) + α

∞

  • k=t

(γλ)k−tdk. (2.3)

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 16 / 28

slide-18
SLIDE 18

Dynamic problems Temporal difference methods

Algorithm 7 Online TD(λ)

1: input Initial parameters vk, trajectories (st, at, rt) 2: e0 = 0. 3: for t = 1, . . . , T do 4:

dt d(st, st+1) temporal difference

5:

et(st) = et−1(st) + 1 eligibility increase

6:

for s ∈ S do

7:

vt+1(st) = vt(s) + αtet(s)dt. update all eligible states

8:

end for

9:

et+1 = λet

10: end for 11: return vT

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 17 / 28

slide-19
SLIDE 19

Dynamic problems Value iteration methods

Algorithm 8 Simulation-based value iteration

1: Input µ, S. 2: Initialise st ∈ S, v0 ∈ V. 3: for t = 1, 2, . . . do 4:

s = st.

5:

πt(s) = arg maxa Pµ(s′|s, a)vt−1(s′)

6:

vt(s) = r(s) +

s′S Pµ(s′|s, πt(s))vt−1(s′)

7:

st+1 ∼ (1 − ǫ) · P(st+1 | st = a, πt, µ) + ǫ · Unif (S).

8: end for 9: Return πn, Vn.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 18 / 28

slide-20
SLIDE 20

Dynamic problems Value iteration methods

10-8 10-6 10-4 10-2 100 102 500 1000 1500 2000 error t 1.0 0.5 0.1 0.01 1-gamma

Figure: Simulation-based value iteration with v0 = 0, varying ǫt = 0.1.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 19 / 28

slide-21
SLIDE 21

Dynamic problems Value iteration methods

10-8 10-6 10-4 10-2 100 102 500 1000 1500 2000 error t 1.0 0.5 0.1 0.01 1-gamma

Figure: Simulation-based value iteration with v0 = 20 = 1/(1 − γ), varying ǫ.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 20 / 28

slide-22
SLIDE 22

Dynamic problems Value iteration methods

Algorithm 9 Q-learning

1: Input µ, S, ǫt, αt. 2: Initialise st ∈ S, q0 ∈ V. 3: for t = 1, 2, . . . do 4:

s = st.

5:

at ∼ ˆ π∗

ǫt (a | st, qt)

6:

st+1 ∼ P(st+1 | st = a, πt, µ).

7:

qt+1(st, at) = qt(st, at) − αt[r(s) + vt(st+1)], where vt(s) = maxa∈A qt(s, a).

8: end for 9: Return πn, Vn.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 21 / 28

slide-23
SLIDE 23

Dynamic problems Value iteration methods

10 20 30 40 50 60 70 200 400 600 800 1000 t x 10 1.0 0.5 0.1 0.05 0.01

qt − Q∗

Figure: Q-learning estimation error with v0 = 1/(1 − γ), ǫt = 1/nst , αt ∈ αn−2/3

st

.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 22 / 28

slide-24
SLIDE 24

Dynamic problems Value iteration methods

  • 500

500 1000 1500 2000 2500 3000 200 400 600 800 1000 t x 10 1.0 0.5 0.1 0.05 0.01

L

Figure: Q-learning estimation error with v0 = 1/(1 − γ), ǫt = 1/nst , αt ∈ αn−2/3

st

.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 23 / 28

slide-25
SLIDE 25

Dynamic problems Value iteration methods

Algorithm 10 Generalised stochastic value iteration

1: Input ˆ

µ0, S, ǫt, αt.

2: Initialise s1 ∈ S, q1 ∈ V. 3: for t = 1, 2, . . . do 4:

at ∼ ˆ π∗

ǫt (a | st, qt)

5:

Observe st+1, rt+1.

6:

ˆ µt = ˆ µt−1 | st, at, st+1, rt+1. // update MDP estimate.

7:

for s ∈ S, a ∈ A do

8:

With probability σt(s, a) do: qt+1(s, a) = qt(s, a) − αt

  • r(s) + γ
  • s′∈S

P(st+1 = s′ | st = s, at = a, ˆ µt)vt(s′)

  • 9:
  • therwise qt+1(s, a) = qt(s, a).

10:

end for

11: end for 12: Return πn, Vn.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 24 / 28

slide-26
SLIDE 26

Dynamic problems Value iteration methods

10 20 30 40 50 60 70 200 400 600 800 1000 t x 10 1.0 0.5 0.1 0.05 0.01

qt − Q∗

Figure: GSVI with Dirichlet model and single state-action update, with v0 = 1/(1 − γ), ǫt = 1/nst , αt ∈ αn−2/3

st

.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 25 / 28

slide-27
SLIDE 27

Dynamic problems Value iteration methods

  • 500

500 1000 1500 2000 2500 3000 200 400 600 800 1000 t x 10 1.0 0.5 0.1 0.05 0.01

L

Figure: Q-learning with Dirichlet model and single state-action update with v0 = 1/(1 − γ), ǫt = 1/nst , αt ∈ αn−2/3

st

.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 26 / 28

slide-28
SLIDE 28

Dynamic problems Value iteration methods

10 20 30 40 50 60 70 200 400 600 800 1000 t x 10 1.0 0.5 0.1 0.05 0.01

qt − Q∗

Figure: GSVI with Dirichlet model estimation and a uniform sweep over the state-space with v0 = 1/(1 − γ), ǫt = 1/nst , αt ∈ αn−2/3

st

.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 27 / 28

slide-29
SLIDE 29

Dynamic problems Value iteration methods

  • 500

500 1000 1500 2000 2500 3000 200 400 600 800 1000 t x 10 1.0 0.5 0.1 0.05 0.01

L

Figure: GSVI with Dirichlet estimation, and a uniform sweep over the state space, with v0 = 1/(1 − γ), ǫt = 1/nst , αt ∈ αn−2/3

st

.

Christos Dimitrakakis (Chalmers) Reinforcement Learning: Basic models and algorithms November 20, 2013 28 / 28