Logistic Regression Many slides attributable to: Prof. Mike Hughes - - PowerPoint PPT Presentation

Logistic Regression

2

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/

Many slides attributable to: Erik Sudderth (UCI) Finale Doshi-Velez (Harvard) James, Witten, Hastie, Tibshirani (ISL/ESL books)

• Prof. Mike Hughes

Objectives Today: Logistic Regression

Logistic Regression

• View as a probabilistic classifier
• Justification: minimize “log loss” is equivalent to

maximizing the likelihood of training set

• Computing log loss in numerically stable way
• Computing the gradient wrt parameters
• Training via gradient descent

3

Mike Hughes - Tufts COMP 135 - Fall 2020

Check-in Q1:

4

Mike Hughes - Tufts COMP 135 - Fall 2020

When training Logistic Regression, we minimize the log loss

• n the training set.

Can you provide 2 justifications for why this log loss

• bjective is sensible?

min

w,b N

X

n=1

log loss(yn, ˆ p(xn, w, b))

log loss(y, ˆ p) = −y log ˆ p − (1 − y) log(1 − ˆ p)

Check-in Q1:

5

Mike Hughes - Tufts COMP 135 - Fall 2020

When training Logistic Regression, we minimize the log loss

• n the training set.

Can you provide 2 justifications for why this log loss

• bjective is sensible?

1) Log loss is an upper bound of error rate. Minimizing log loss must reduce our (worst case) error rate. 2) Log loss = Binary Cross Entropy. Interpret as learning best probabilistic “encoding’ of the training data

min

w,b N

X

n=1

log loss(yn, ˆ p(xn, w, b))

log loss(y, ˆ p) = −y log ˆ p − (1 − y) log(1 − ˆ p)

What will we learn?

6

Mike Hughes - Tufts COMP 135 - Fall 2020

Supervised Learning Unsupervised Learning Reinforcement Learning

Data, Label Pairs Performance measure Task data x label y

{xn, yn}N

n=1

Training Prediction Evaluation

7

Mike Hughes - Tufts COMP 135 - Fall 2020

y

x2 x1

is a binary variable (red or blue)

Supervised Learning

binary classification

Unsupervised Learning Reinforcement Learning

Task: Binary Classification

>>> yproba_N2 = model.predict_proba(x_NF) >>> yproba1_N = model.predict_proba(x_NF)[:,1] >>> yproba1_N[:5] [0.143, 0.432, 0.523, 0.003, 0.994]

Probability Prediction

Goal: Predict probability of label given features

• Input:
• Output:

8

Mike Hughes - Tufts COMP 135 - Fall 2020

xi , [xi1, xi2, . . . xif . . . xiF ]

Entries can be real-valued, or

• ther numeric types (e.g. integer,

binary)

“features” “covariates” “attributes” “probability”

ˆ pi , p(Yi = 1|xi)

Value between 0 and 1 e.g. 0.001, 0.513, 0.987

Review: Logistic Regression

9

Mike Hughes - Tufts COMP 135 - Fall 2020

min

w,b N

X

n=1

log loss(yn, ˆ p(xn, w, b))

Parameters: Prediction: Training:

Find weights and bias that minimize (penalized) log loss

w = [w1, w2, . . . wf . . . wF ] b

weight vector bias scalar

p(yi = 1|xi) , sigmoid @

F

X

f=1

wfxif + b 1 A

ˆ p(xi, w, b) =

sigmoid(z) = 1 1 + e−z

Logistic Regression: Predicted Probas vs Binary Decisions

10

Mike Hughes - Tufts COMP 135 - Fall 2020

Predicted probability function is monotonically increasing in one direction That direction is perpendicular to the decision boundary Decision boundary is the set of x values where Pr(y=1|x) = 0.5 Decision boundary is a linear function of x

Check-in (warm up for lab)

Consider logistic regression classifier for 2D features What is the value (approximately)

• f w_1, w_2, and bias for each plot below?

11

Mike Hughes - Tufts COMP 135 - Fall 2020 Predicted proba of positive class x_1 x_2

Check-in Answered

Consider logistic regression classifier for 2D features What is the value (approximately)

• f w_1, w_2, and bias for each plot below?

12

Mike Hughes - Tufts COMP 135 - Fall 2020 Predicted proba of positive class x_1 x_2

13

Mike Hughes - Tufts COMP 135 - Fall 2020

Optimization Objective Why minimize log loss?

A probabilistic justification

Likelihood of labels under LR

14

Mike Hughes - Tufts COMP 135 - Fall 2020

We can write the probability for each possible outcome as:

p(Yi = 1|xi) = p(Yi = 0|xi) = 1 −

We can write the probability mass function of random variable Y as:

p(Yi = yi|xi) = ⇥ σ(wT xi + b) ⇤yi ⇥ 1 − σ(wT xi + b) ⇤1−yi

Interpret: p(y | x) is the “likelihood” of label y given input features x Goal: Fit model to make the training data as likely as possible

⇥ σ(wT xi + b) ⇥ σ(wT xi + b)

Maximizing likelihood

15

Mike Hughes - Tufts COMP 135 - Fall 2020

max

w,b N

Y

n=1

p(Yn = yn|xn, w, b)

Why might this be hard in practice? Hint: Think about datasets with 1000s of examples N

Maximizing log likelihood

16

Mike Hughes - Tufts COMP 135 - Fall 2020

The logarithm (with any base) is a monotonic transform

a > b implies log (a) > log (b)

Thus, the following are equivalent problems

w∗, b∗ = arg max

w,b N

Y

n=1

p(Yn = yn|xn, w, b) w∗, b∗ = arg max

w,b

log " N Y

n=1

p(Yn = yn|xn, w, b) #

Log likelihood for LR

17

Mike Hughes - Tufts COMP 135 - Fall 2020

We can write the probability mass function of Y as: Our training objective is to maximize log likelihood

p(Yi = yi|xi) = ⇥ σ(wT xi + b) ⇤yi ⇥ 1 − σ(wT xi + b) ⇤1−yi

Y

n=1

w∗, b∗ = arg max

w,b

log " N Y

n=1

p(Yn = yn|xn, w, b) #

In order to maximize likelihood, we can minimize negative log likelihood.

Two equivalent optimization problems:

18

Mike Hughes - Tufts COMP 135 - Fall 2020

w∗, b∗ = arg max

w,b N

X

n=1

log p(Yn = yn|xn, w, b) w∗, b∗ = arg min

w,b

−

N

X

n=1

log p(Yn = yn|xn, w, b)

Summary of “likelihood” justification for the way we train logistic regression

• We defined a probabilistic model for y given x
• We want to maximize probability of training data

under this model (“maximize likelihood”)

• We can show that another optimization problem

(”maximize log likelihood”) is easier numerically but produces the same optimal values for weights and bias

• Equivalent to minimizing -1 * log likelihood
• Turns out, minimizing log loss is precisely the

same thing as minimizing negative log likelihood

19

Mike Hughes - Tufts COMP 135 - Fall 2020

Computing the loss for Logistic Regression (LR) in a numerically stable way

20

Mike Hughes - Tufts COMP 135 - Fall 2020

Simplified notation

• Feature vector with first entry constant
• Weight vector (first entry is the “bias”)
• “Score” value s (real number, -inf to +inf)

21

Mike Hughes - Tufts COMP 135 - Fall 2020

w = [w0 w1 w2 . . . wF ]

sn = wT ˜ xn

Training Logistic Regression

22

Mike Hughes - Tufts COMP 135 - Fall 2020

min

w N

X

n=1

BCE(yn, σ(sn(w)))

σ(s) = sigmoid(s)

s

sn = wT ˜ xn

How can we evaluate this loss as a function of w reliably? How can we evaluate gradients of this loss with respect to w?

w = [w0 w1 w2 . . . wF ]

Simplifying per-example loss

23

Mike Hughes - Tufts COMP 135 - Fall 2020

scoreBCE(yn, sn) = BCE(yn, σ(sn)) = −yn log2 σ(sn) − (1 − yn) log2(1 − σ(sn)) = −yn log2 1 1 + e−sn − (1 − yn) log2 e−sn 1 + e−sn = ( log2(1 + e−sn) if yn = 1 log2(1 + esn) if yn = 0 = log2 ⇣ 1 + eflip(yn)sn⌘ = log2 ⇣ e0 + eflip(yn)sn⌘

sn = wT ˜ xn

Why care about numerical issues?

1) Avoid NaNs and other issues 2) Correctly rank bad solutions (avoid saturation) 3) Non-zero gradients everywhere (no flat regions)

24

Mike Hughes - Tufts COMP 135 - Fall 2020

logsumexp : the problem

25

Mike Hughes - Tufts COMP 135 - Fall 2020

See HW2

• n website

logsumexp : the solution

26

Mike Hughes - Tufts COMP 135 - Fall 2020

See HW2

• n website

Gradient of log loss wrt w

27

Mike Hughes - Tufts COMP 135 - Fall 2020

Simplifying yields: Per-example loss when y=0 Gradient w.r.t. weight on feature f: Use the chain rule

sn = wT ˜ xn

J(sn(w)) = log(1 + esn)

∂ ∂wf J(sn(w)) = ∂ ∂sn J(sn) · ∂ ∂wf sn(w)

∂ ∂wf J = (σ(sn(w)) − yn) ˜ xnf

Gradient of log loss wrt weights

Gradient of log loss wrt weights

28

Mike Hughes - Tufts COMP 135 - Fall 2020

sn = wT ˜ xn

∂ ∂wf J = (σ(sn(w)) − yn) xnf

Nice interpretation: ( PREDICTION - TRUE RESPONSE ) * FEATURE If feature is positive: Predicted probability is larger than truth means gradient is positive To improve our model (go downhill) we should make that weight smaller If feature is negative, we move weights in the opposite direction.

Gradient descent for Logistic Regression

29

Mike Hughes - Tufts COMP 135 - Fall 2020

Will gradient descent always find same solution?

30

Mike Hughes - Tufts COMP 135 - Fall 2020

Yes, if loss looks like this Not if multiple local minima exist

Log loss (aka BCE) is convex as a function of weight parameters

31

Mike Hughes - Tufts COMP 135 - Fall 2020

32

Mike Hughes - Tufts COMP 135 - Fall 2020

Logistic Regression

• View as a probabilistic classifier
• Justification: minimize “log loss” is equivalent to

maximizing the likelihood of training set

• Computing log loss in numerically stable way
• Computing the gradient
• Training via gradient descent

Lab: practice visual intuition for how weights and bias affect predicted probabilities

Objectives Today: Logistic Regression