Bayesian Counterfactual Risk Minimization Ben London (blondon@) - - PowerPoint PPT Presentation

Bayesian Counterfactual Risk Minimization

International Conference on Machine Learning Long Beach, CA, June 11, 2019

Ted Sandler (sandler@) Amazon Music Ben London (blondon@) Amazon Music

Learning from Logged Data

Design/train new rec policy A/B test new policy vs

• ld policy

Launch new policy

if unsuccessful if successful

Pull log data

e.g., user i listened to item j

• Only observe outcomes from actions taken
• e.g., only get feedback on recommendations

Problem 1: Bandit Feedback

Here’s a station you might like … Alexa, play music

Problem 2: Bias

low support → who knows? high support → better estimate

• Logged data is biased
• Policy typically not uniform distribution
• User typically doesn’t see everything
• Bias affects inferences
• Self-fulfilling prophecies; “rich get richer”
• Miss key insights due to insufficient support

IPS Policy Optimization

• Use inverse propensity score (IPS) estimator

logged propensity pi = π0(ai | xi)

• IPS is an unbiased estimator of expected reward
• Caveat: logging policy must have full support

arg min

π

1 n

n

X

i=1

−ri π(ai | xi) pi

E

(x,ρ)∼D

E

a∼π(x)[ρ(x, a)] ≈ 1

n

n

X

i=1

ri π(ai | xi) pi

IPS Policy Optimization

• Use inverse propensity score (IPS) estimator

logged propensity pi = π0(ai | xi)

!"($|&) !((|&) !"($|&) !($|&)

• Problem: IPS has high variance

arg min

π

1 n

n

X

i=1

−ri π(ai | xi) pi

CRM Principle

• Counterfactual Risk Minimization (CRM) principle
• Motivated by PAC risk analysis
• Stochastic optimization of variance regularizer is tricky
• Policy optimization for exponential models (POEM) algorithm

[Swaminathan & Joachims, ICML 2015]

variance regularization

arg min

π

1 n

n

X

i=1

−ri π(ai | xi) pi

+ λ q ˆ Var(π, S)

Bayesian CRM Principle

• Bayesian Counterfactual Risk Minimization (CRM) principle
• Bayesian policy:
• Motivated by PAC-Bayes risk analysis
• Takeaway: posterior should stay close to the prior
• What should the prior be? How about the logging policy!

[London & Sandler, ICML 2019]

KL div. from prior to posterior

arg min

Q

1 n

n

X

i=1

−ri πQ(ai | xi) pi

+ λ Dkl(QkP)

πQ(a | x) = Prh∼Q{h(x) = a}, h : X → A

Application to Mixed Logits

• Mixed logit model
• Like a softmax with normally distributed parameters
• Risk bound motivates logging policy regularization

[London & Sandler, ICML 2019]

L2 distance to logging policy parameters

hw,γ(x) = arg maxa w · φ(x, a) + γa

γ ∼ Gumbel(0, 1)k

Dkl(QkP) = O(kµ µ0k2)

w ∼ N(µ, Σ)

πQ(a | x) = E(w,γ)∼Q [1{hw,γ(x) = a}] = Ew h

exp(w·f(x,a)) P

a0 exp(w·f(x,a0))

i

Contributions

• PAC Risk bounds for Bayesian policies
• Application to mixed logits
• Logging policy prior motivates new regularizer
• Two new learning objectives (one convex) that minimize bounds
• Experiments show proposed methods gain up to 76% more reward

than POEM, while also simpler/more efficient

Visit poster #113 in Pacific Ballroom