Self-Supervised Exploration via Disagreement Deepak Pathak* Dhiraj - - PowerPoint PPT Presentation

Self-Supervised Exploration via Disagreement

Deepak Pathak* UC Berkeley Dhiraj Gandhi* CMU Abhinav Gupta CMU, FAIR

* equal contribution

ICML 2019

Exploration – a major challenge!

• Mohamed et.al. “Variational information

maximisation for intrinsically motivated reinforcement learning”. NIPS, 2015.

• Houthooft et.al. “VIME: Variational information

maximizing exploration”. NIPS, 2016.

• Gregor et.al. “Variational intrinsic control”. ICLR

Workshop, 2017.

• Pathak et.al. “Curiosity-driven Exploration by Self-

supervised Exploration”. ICML 2017

• Ostrovski et.al. “Count-based exploration with

neural density models”. ICML, 2017.

• Burda*, Edwards*, Pathak* et.al. “Large-Scale

Study of Curiosity-driven Learning”. ICLR 2019

• Eysenbach et al. “Diversity is all you need: Learn

skills without a reward function”. ICLR 2019.

• Savinov et al. "Episodic curiosity through

reachability". ICLR 2019.

Exploration – a major challenge!

• Schmidhuber, Jurgen. “A possibility for

implementing curiosity and boredom in model building neural controllers”, 1991.

• Schmidhuber, Jurgen. “Formal theory of creativity,

fun, and intrinsic motivation (1990–2010)”, 2010.

• Oudeyer, P.-Y. and Kaplan, F. What is intrinsic

motivation? a typology of computational

• approaches. Frontiers in neurorobotics, 2009.
• Poupart et.al. “An analytic solution to discrete

bayesian reinforcement learning”. ICML, 2006.

• Lopes et.al. “Exploration in model-based

reinforcement learning by empirically estimating learning progress”. NIPS, 2012.

• Bellemare et.al. “Unifying count-based exploration

and intrinsic motivation”. NIPS, 2016.

Exploration – a major challenge!

• Schmidhuber, Jurgen. “A possibility for

implementing curiosity and boredom in model building neural controllers”, 1991.

• Schmidhuber, Jurgen. “Formal theory of creativity,

fun, and intrinsic motivation (1990–2010)”, 2010.

• Oudeyer, P.-Y. and Kaplan, F. What is intrinsic

motivation? a typology of computational

• approaches. Frontiers in neurorobotics, 2009.
• Poupart et.al. “An analytic solution to discrete

bayesian reinforcement learning”. ICML, 2006.

• Lopes et.al. “Exploration in model-based

reinforcement learning by empirically estimating learning progress”. NIPS, 2012.

• Bellemare et.al. “Unifying count-based exploration

and intrinsic motivation”. NIPS, 2016.

• Mohamed et.al. “Variational information

maximisation for intrinsically motivated reinforcement learning”. NIPS, 2015.

• Houthooft et.al. “VIME: Variational information

maximizing exploration”. NIPS, 2016.

• Gregor et.al. “Variational intrinsic control”. ICLR

Workshop, 2017.

• Pathak et.al. “Curiosity-driven Exploration by Self-

supervised Exploration”. ICML 2017

• Ostrovski et.al. “Count-based exploration with

neural density models”. ICML, 2017.

• Burda*, Edwards*, Pathak* et.al. “Large-Scale

Study of Curiosity-driven Learning”. ICLR 2019

• Eysenbach et al. “Diversity is all you need: Learn

skills without a reward function”. ICLR 2019.

• Savinov et al. "Episodic curiosity through

reachability". ICLR 2019.

Exploration – a major challenge!

• Schmidhuber, Jurgen. “A possibility for

implementing curiosity and boredom in model building neural controllers”, 1991.

• Schmidhuber, Jurgen. “Formal theory of creativity,

fun, and intrinsic motivation (1990–2010)”, 2010.

• Oudeyer, P.-Y. and Kaplan, F. What is intrinsic

motivation? a typology of computational

• approaches. Frontiers in neurorobotics, 2009.
• Poupart et.al. “An analytic solution to discrete

bayesian reinforcement learning”. ICML, 2006.

• Lopes et.al. “Exploration in model-based

reinforcement learning by empirically estimating learning progress”. NIPS, 2012.

• Bellemare et.al. “Unifying count-based exploration

and intrinsic motivation”. NIPS, 2016.

• Mohamed et.al. “Variational information

maximisation for intrinsically motivated reinforcement learning”. NIPS, 2015.

• Houthooft et.al. “VIME: Variational information

maximizing exploration”. NIPS, 2016.

• Gregor et.al. “Variational intrinsic control”. ICLR

Workshop, 2017.

• Pathak et.al. “Curiosity-driven Exploration by Self-

supervised Exploration”. ICML 2017

• Ostrovski et.al. “Count-based exploration with

neural density models”. ICML, 2017.

• Burda*, Edwards*, Pathak* et.al. “Large-Scale

Study of Curiosity-driven Learning”. ICLR 2019

• Eysenbach et al. “Diversity is all you need: Learn

skills without a reward function”. ICLR 2019.

• Savinov et al. "Episodic curiosity through

reachability". ICLR 2019.

Exploration – a major challenge!

• Schmidhuber, Jurgen. “A possibility for

implementing curiosity and boredom in model building neural controllers”, 1991.

• Schmidhuber, Jurgen. “Formal theory of creativity,

fun, and intrinsic motivation (1990–2010)”, 2010.

• Oudeyer, P.-Y. and Kaplan, F. What is intrinsic

motivation? a typology of computational

• approaches. Frontiers in neurorobotics, 2009.
• Poupart et.al. “An analytic solution to discrete

bayesian reinforcement learning”. ICML, 2006.

• Lopes et.al. “Exploration in model-based

reinforcement learning by empirically estimating learning progress”. NIPS, 2012.

• Bellemare et.al. “Unifying count-based exploration

and intrinsic motivation”. NIPS, 2016.

• Mohamed et.al. “Variational information

maximisation for intrinsically motivated reinforcement learning”. NIPS, 2015.

• Houthooft et.al. “VIME: Variational information

maximizing exploration”. NIPS, 2016.

• Gregor et.al. “Variational intrinsic control”. ICLR

Workshop, 2017.

• Pathak et.al. “Curiosity-driven Exploration by Self-

supervised Exploration”. ICML 2017

• Ostrovski et.al. “Count-based exploration with

neural density models”. ICML, 2017.

• Burda*, Edwards*, Pathak* et.al. “Large-Scale

Study of Curiosity-driven Learning”. ICLR 2019

• Eysenbach et al. “Diversity is all you need: Learn

skills without a reward function”. ICLR 2019.

• Savinov et al. "Episodic curiosity through

reachability". ICLR 2019.

S a m p l e I n e f f i c i e n t [ m i l l i

• n

s

• f

s a m p l e s ]

Simulation

Sample Inefficient

Real Robots Simulation

Sample Inefficient

Real Robots Simulation

Sample Inefficient “Stuck” in Stochastic Envs

Real Robots Simulation

Sample Inefficient “Stuck” in Stochastic Envs

Curiosity Exploration w/ Noisy TV & Remote

[Burda*, Edwards*, Pathak* et. al. ICLR’19] [Juliani et.al., ArXiv’19]

Why inefficient?

[Pathak et al. ICML, 2017]

current image xt

[Pathak et al. ICML, 2017]

current image xt policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

Prediction Model 𝑔(𝑦$, 𝑏$)

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt

Prediction Model 𝑔(𝑦$, 𝑏$)

action at

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt

Prediction Model 𝑔(𝑦$, 𝑏$)

action at predicted next image * 𝒚𝒖-𝟐

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt

Prediction Model 𝑔(𝑦$, 𝑏$)

Intrinsic Reward

action at predicted next image * 𝒚𝒖-𝟐

ri

t

𝑠$

0 =

2 𝑦$-3 − 𝑦$-3

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt

Prediction Model 𝑔(𝑦$, 𝑏$)

Intrinsic Reward

action at predicted next image * 𝒚𝒖-𝟐

ri

t

𝑠$

0 =

2 𝑦$-3 − 𝑦$-3

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt

Prediction Model 𝑔(𝑦$, 𝑏$)

Intrinsic Reward

action at predicted next image * 𝒚𝒖-𝟐

ri

t

𝑠$

0 =

2 𝑦$-3 − 𝑦$-3

Environment is “black-box” à hard optimization

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt

Prediction Model 𝑔(𝑦$, 𝑏$)

Intrinsic Reward

action at predicted next image * 𝒚𝒖-𝟐

ri

t

max

"

𝔽 9

$:3 ;

𝑠$

𝑠$

0 =

2 𝑦$-3 − 𝑦$-3

REINFORCE

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt

Prediction Model 𝑔(𝑦$, 𝑏$)

Intrinsic Reward

action at predicted next image * 𝒚𝒖-𝟐

ri

t

max

"

𝔽 9

$:3 ;

𝑠$

𝑠$

0 =

2 𝑦$-3 − 𝑦$-3

REINFORCE

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

[Pathak et al. ICML, 2017]

current image xt

Prediction Model 𝑔(𝑦$, 𝑏$)

Intrinsic Reward

action at predicted next image * 𝒚𝒖-𝟐

ri

t

max

"

𝔽 9

$:3 ;

𝑠$

𝑠$

0 =

2 𝑦$-3 − 𝑦$-3

REINFORCE

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

𝑦$-3 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

𝑠$

0 = 𝜏

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

𝑠$

0 = 𝜏

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

Intrinsic Reward

current image xt next image xt+1 action at policy network

𝜌" 𝑦$

𝑠$

0 = 𝜏

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

Intrinsic Reward Disagreement

Deterministic Environments

performs as well as state-of-the-art methods

Deterministic Environments

performs as well as state-of-the-art methods

Reward (not for training)

Deterministic Environments

performs as well as state-of-the-art methods

Reward (not for training) Number of Frames (in millions)

Deterministic Environments

performs as well as state-of-the-art methods

Reward (not for training) Number of Frames (in millions)

Stochastic Environments

Stochastic Environments

Every model’s goes to mean à variance drops à unstuck

Stochastic Environments

Every model’s goes to mean à variance drops à unstuck

Reward (not for training) Number of Frames (in millions)

: 3D Navigation

w/o TV Noisy TV w/ Remote w/o TV Noisy TV w/ Remote

Stochastic Environments

Every model’s goes to mean à variance drops à unstuck

Reward (not for training) Number of Frames (in millions)

: 3D Navigation

w/o TV Noisy TV w/ Remote w/o TV Noisy TV w/ Remote

current state xt next state xt+1 action at policy network

𝜌" 𝑦$

𝑠$

0 = 𝜏

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

Curiosity Reward Disagreement

Disagreement

current state xt next state xt+1 action at policy network

𝜌" 𝑦$

𝑠$

0 = 𝜏

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

Curiosity Reward

Disagreement

current state xt next state xt+1 action at policy network

𝜌" 𝑦$

𝑠$

0 = 𝜏

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

Curiosity Reward

Disagreement

current state xt next state xt+1 action at policy network

𝜌" 𝑦$

𝑠$

0 = 𝜏

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

Curiosity Reward

No dependency on the environment!

Disagreement

current state xt next state xt+1 action at policy network

𝜌" 𝑦$

𝑠$

0 = 𝜏

𝑦$-3 𝑛𝑗𝑜 𝑦$ 𝑏$ 𝑔

3

2 𝑦$-3

3

𝑦$-3 − 2 𝑦$-3

3

𝑦$ 𝑏$ 𝑔

<

2 𝑦$-3

<

𝑦$-3 − 2 𝑦$-3

<

𝑦$ 𝑏$ 𝑔

=

2 𝑦$-3

=

𝑦$-3 − 2 𝑦$-3

=

Curiosity Reward

No dependency on the environment!

Differentiable Exploration

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

min

"D,… ,"F 9 0:3 G

𝑔

"H 𝑦$, 𝜌 𝑦$; 𝜄K

− 𝑦$-3 <

Model Optimization

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

min

"D,… ,"F 9 0:3 G

𝑔

"H 𝑦$, 𝜌 𝑦$; 𝜄K

− 𝑦$-3 < max

"L

9

0:3 G

𝑔

"H 𝑦$, 𝜌 𝑦$; 𝜄K

− 1 𝑙 9

O:3 G

𝑔

"P 𝑦$, 𝜌 𝑦$; 𝜄K <

Model Optimization Policy Optimization

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Differentiable Exploration

Position Control: 1. Position 2. Direction 3. Gripper Angle 4. Gripper Distance

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Efficiency over REINFORCE

Object Interaction Rate Training Samples

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Efficiency over REINFORCE

Object Interaction Rate Training Samples

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Pushing skill Efficiency over REINFORCE

Object Interaction Rate Training Samples

Differentiable Exploration

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.

Pushing skill Picking skill Efficiency over REINFORCE

Object Interaction Rate Training Samples

Summary: Exploration via Disagreement

Summary: Exploration via Disagreement

• Similar to state-of-the-art in deterministic envs

(Atari games)

Summary: Exploration via Disagreement

• Similar to state-of-the-art in deterministic envs

(Atari games)

• Does not get stuck in stochastic scenarios

(Stochastic Atari; Unity-TV)

Summary: Exploration via Disagreement

• Similar to state-of-the-art in deterministic envs

(Atari games)

• Does not get stuck in stochastic scenarios

(Stochastic Atari; Unity-TV)

• Differentiable reformulation for real robots

(Sawyer Robot)

Code Available

https://pathak22.github.io/exploration-by-disagreement/

Thank you!

Poster # 39 (today)

Pathak*, Gandhi*, Gupta. “Self-Supervised Exploration via Disagreement“. ICML, 2019.