IN5550: Neural Methods in Natural Language Processing Lecture 2 - - PowerPoint PPT Presentation

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning: from Linear Models to Neural Networks

Andrey Kutuzov, Vinit Ravishankar, Jeremy Barnes, Lilja Øvrelid, Stephan Oepen, & Erik Velldal

University of Oslo

21 January 2020

1

Contents

1

Introduction

2

Basics of supervised machine learning

3

Linear classifiers

4

Training as optimization

5

Limitations of linear models

6

Going deeply non-linear: multi-layered perceptrons

7

Next lecture on January 28

8

Before the next lecture

1

Introduction

I am Jeremy Barnes I will do the next two lectures, covering the following topics:

2

Introduction

I am Jeremy Barnes I will do the next two lectures, covering the following topics: ◮ a review of supervised learning (introducing notation)

2

Introduction

I am Jeremy Barnes I will do the next two lectures, covering the following topics: ◮ a review of supervised learning (introducing notation) ◮ Linear classifiers and simple feed-forward neural networks

2

Introduction

I am Jeremy Barnes I will do the next two lectures, covering the following topics: ◮ a review of supervised learning (introducing notation) ◮ Linear classifiers and simple feed-forward neural networks ◮ Multi-layer neural networks and training

2

Introduction

I am Jeremy Barnes I will do the next two lectures, covering the following topics: ◮ a review of supervised learning (introducing notation) ◮ Linear classifiers and simple feed-forward neural networks ◮ Multi-layer neural networks and training

2

Introduction

I am Jeremy Barnes I will do the next two lectures, covering the following topics: ◮ a review of supervised learning (introducing notation) ◮ Linear classifiers and simple feed-forward neural networks ◮ Multi-layer neural networks and training

2

Introduction

Technicalities ◮ Make sure to familiarize yourself with the course infrastructure.

3

Introduction

Technicalities ◮ Make sure to familiarize yourself with the course infrastructure. ◮ Check the course page for messages.

3

Introduction

Technicalities ◮ Make sure to familiarize yourself with the course infrastructure. ◮ Check the course page for messages. ◮ Test whether you can access https://github.uio.no/in5550/2020

◮ make sure to update your UiO github profile with your photo, and star the course repository :-)

3

Introduction

Technicalities ◮ Make sure to familiarize yourself with the course infrastructure. ◮ Check the course page for messages. ◮ Test whether you can access https://github.uio.no/in5550/2020

◮ make sure to update your UiO github profile with your photo, and star the course repository :-)

◮ Most of machine learning revolves around linear algebra.

3

Introduction

Technicalities ◮ Make sure to familiarize yourself with the course infrastructure. ◮ Check the course page for messages. ◮ Test whether you can access https://github.uio.no/in5550/2020

◮ make sure to update your UiO github profile with your photo, and star the course repository :-)

◮ Most of machine learning revolves around linear algebra. ◮ We created a LinAlg cheat sheet for this course.

3

Introduction

Technicalities ◮ Make sure to familiarize yourself with the course infrastructure. ◮ Check the course page for messages. ◮ Test whether you can access https://github.uio.no/in5550/2020

◮ make sure to update your UiO github profile with your photo, and star the course repository :-)

◮ Most of machine learning revolves around linear algebra. ◮ We created a LinAlg cheat sheet for this course.

◮ Linked from the course page, adapted for the notation of [Goldberg, 2017].

3

Contents

1

Introduction

2

Basics of supervised machine learning

3

Linear classifiers

4

Training as optimization

5

Limitations of linear models

6

Going deeply non-linear: multi-layered perceptrons

7

Next lecture on January 28

8

Before the next lecture

3

Basics of supervised machine learning

4

Basics of supervised machine learning

4

Basics of supervised machine learning

4

Basics of supervised machine learning

4

Basics of supervised machine learning

4

Basics of supervised machine learning

4

Basics of supervised machine learning

4

Basics of supervised machine learning

◮ Input 1: a training set of n training instances x1:n = x1, x2, . . . xn

5

Basics of supervised machine learning

◮ Input 1: a training set of n training instances x1:n = x1, x2, . . . xn

◮ for example, e-mail messages.

5

Basics of supervised machine learning

◮ Input 1: a training set of n training instances x1:n = x1, x2, . . . xn

◮ for example, e-mail messages.

◮ Input 2: corresponding ‘gold’ labels for these instances y1:n = y1, y2, . . . yn

5

Basics of supervised machine learning

◮ Input 1: a training set of n training instances x1:n = x1, x2, . . . xn

◮ for example, e-mail messages.

◮ Input 2: corresponding ‘gold’ labels for these instances y1:n = y1, y2, . . . yn

◮ for example, whether the message is spam (1) or not (0).

5

Basics of supervised machine learning

◮ Input 1: a training set of n training instances x1:n = x1, x2, . . . xn

◮ for example, e-mail messages.

◮ Input 2: corresponding ‘gold’ labels for these instances y1:n = y1, y2, . . . yn

◮ for example, whether the message is spam (1) or not (0).

◮ The trained models allow to make label predictions for unseen instances.

5

Basics of supervised machine learning

◮ Input 1: a training set of n training instances x1:n = x1, x2, . . . xn

◮ for example, e-mail messages.

◮ Input 2: corresponding ‘gold’ labels for these instances y1:n = y1, y2, . . . yn

◮ for example, whether the message is spam (1) or not (0).

◮ The trained models allow to make label predictions for unseen instances. ◮ Generally: some program for mapping instances to labels.

5

Basics of supervised machine learning

Recap on data split ◮ Recall: we want the model to make good predictions for unseen data.

6

Basics of supervised machine learning

Recap on data split ◮ Recall: we want the model to make good predictions for unseen data. ◮ It should not overfit to the seen data.

6

Basics of supervised machine learning

Recap on data split ◮ Recall: we want the model to make good predictions for unseen data. ◮ It should not overfit to the seen data.

6

Basics of supervised machine learning

Remember: we want models that generalize ◮ Thus, the datasets are usually split into:

• 1. train data;

7

Basics of supervised machine learning

Remember: we want models that generalize ◮ Thus, the datasets are usually split into:

• 1. train data;
• 2. validation/development data (optional);

7

Basics of supervised machine learning

Remember: we want models that generalize ◮ Thus, the datasets are usually split into:

• 1. train data;
• 2. validation/development data (optional);
• 3. test/held-out data.

7

Basics of supervised machine learning

◮ To recap, we want to find a function which makes good, generalizable predictions for our task.

8

Basics of supervised machine learning

◮ To recap, we want to find a function which makes good, generalizable predictions for our task. ◮ Searching among all possible functions is unfeasible.

8

Basics of supervised machine learning

◮ To recap, we want to find a function which makes good, generalizable predictions for our task. ◮ Searching among all possible functions is unfeasible. ◮ To cope with that, we choose an inductive bias...

8

Basics of supervised machine learning

◮ To recap, we want to find a function which makes good, generalizable predictions for our task. ◮ Searching among all possible functions is unfeasible. ◮ To cope with that, we choose an inductive bias... ◮ ...and set some hypothesis class...

8

Basics of supervised machine learning

◮ To recap, we want to find a function which makes good, generalizable predictions for our task. ◮ Searching among all possible functions is unfeasible. ◮ To cope with that, we choose an inductive bias... ◮ ...and set some hypothesis class... ◮ ...to search only within this class.

8

Basics of supervised machine learning

What do you think a good model would look like for this data?

9

Basics of supervised machine learning

What do you think a good model would look like for this data?

9

Basics of supervised machine learning

What do you think a good model would look like for this data?

9

Basics of supervised machine learning

What do you think a good model would look like for this data?

9

Basics of supervised machine learning

What do you think a good model would look like for this data?

9

Basics of supervised machine learning Linear functions: a popular hypothesis class

10

Contents

1

Introduction

2

Basics of supervised machine learning

3

Linear classifiers

4

Training as optimization

5

Limitations of linear models

6

Going deeply non-linear: multi-layered perceptrons

7

Next lecture on January 28

8

Before the next lecture

10

Linear classifiers

Simple linear function f (x; W , b) = x · W + b (1) ◮ Function input:

◮ feature vector x ∈ Rdin; ◮ each training instance is represented with din features; ◮ for example, some properties of the documents.

11

Linear classifiers

Simple linear function f (x; W , b) = x · W + b (1) ◮ Function input:

◮ feature vector x ∈ Rdin; ◮ each training instance is represented with din features; ◮ for example, some properties of the documents.

◮ Function parameters θ:

◮ matrix W ∈ Rdin×dout

◮ dout is the dimensionality of the desired prediction (number of classes)

◮ bias vector b ∈ Rdout

◮ bias ‘shifts’ the function output to some direction.

11

Linear classifiers

Training of a linear classifier f (x; W , b) = x · W + b θ = W , b ◮ Training is finding the optimal θ.

12

Linear classifiers

Training of a linear classifier f (x; W , b) = x · W + b θ = W , b ◮ Training is finding the optimal θ. ◮ ‘Optimal’ means ‘producing predictions ˆ y closest to the gold labels y

• n our n training instances’.

12

Linear classifiers

Training of a linear classifier f (x; W , b) = x · W + b θ = W , b ◮ Training is finding the optimal θ. ◮ ‘Optimal’ means ‘producing predictions ˆ y closest to the gold labels y

• n our n training instances’.

◮ Ideally, ˆ y = y

12

Linear classifiers

Here, training instances are represented with 2 features each (x = [x0, x1]) and labeled with 2 class labels (y = {black, red}):

13

Linear classifiers

Here, training instances are represented with 2 features each (x = [x0, x1]) and labeled with 2 class labels (y = {black, red}): ◮ Parameters of f (x; W , b) = x · W + b define the line (or hyperplane) separating the instances.

13

Linear classifiers

Here, training instances are represented with 2 features each (x = [x0, x1]) and labeled with 2 class labels (y = {black, red}): ◮ Parameters of f (x; W , b) = x · W + b define the line (or hyperplane) separating the instances. ◮ This decision boundary is actually our learned classifier.

13

Linear classifiers

Here, training instances are represented with 2 features each (x = [x0, x1]) and labeled with 2 class labels (y = {black, red}): ◮ Parameters of f (x; W , b) = x · W + b define the line (or hyperplane) separating the instances. ◮ This decision boundary is actually our learned classifier. ◮ NB: the dataset on the plot is linearly separable.

13

Linear classifiers

Here, training instances are represented with 2 features each (x = [x0, x1]) and labeled with 2 class labels (y = {black, red}): ◮ Parameters of f (x; W , b) = x · W + b define the line (or hyperplane) separating the instances. ◮ This decision boundary is actually our learned classifier. ◮ NB: the dataset on the plot is linearly separable. ◮ Question: lines with 3 values of b are shown. Which is the best?

13

Linear classifiers

How can we represent our data (X)?

14

Linear classifiers

How can we represent our data (X)? ◮ Imagine you have a review of a film and you want to know if the reviewer likes the film or hates it.

14

Linear classifiers

How can we represent our data (X)? ◮ Imagine you have a review of a film and you want to know if the reviewer likes the film or hates it. ◮ What are the simplest features that would help you decide this?

14

Linear classifiers

How can we represent our data (X)? ◮ Imagine you have a review of a film and you want to know if the reviewer likes the film or hates it. ◮ What are the simplest features that would help you decide this? ◮ Good, bad, great, terrible, etc.

14

Linear classifiers

How can we represent our data (X)? ◮ Imagine you have a review of a film and you want to know if the reviewer likes the film or hates it. ◮ What are the simplest features that would help you decide this? ◮ Good, bad, great, terrible, etc. ◮ Maybe actors’ names (Meryl Streep, Steven Segal)

14

Linear classifiers

How can we represent our data (X)? ◮ Imagine you have a review of a film and you want to know if the reviewer likes the film or hates it. ◮ What are the simplest features that would help you decide this? ◮ Good, bad, great, terrible, etc. ◮ Maybe actors’ names (Meryl Streep, Steven Segal) ◮ The simplest way to represent these words as features is a Bag-of-Words representation

14

Linear classifiers

Bag of words ◮ Each word from a pre-defined vocabulary D can be a separate feature.

15

Linear classifiers

Bag of words ◮ Each word from a pre-defined vocabulary D can be a separate feature. ◮ How many times the word a appears in the document i?

◮ or a binary flag {1, 0} of whether a appeared in i at all or not.

15

Linear classifiers

Bag of words ◮ Each word from a pre-defined vocabulary D can be a separate feature. ◮ How many times the word a appears in the document i?

◮ or a binary flag {1, 0} of whether a appeared in i at all or not.

◮ This schema is called ‘bag of words’ (BoW).

◮ for example, if we have 1000 words in the vocabulary: ◮ xi ∈ R1000

15

Linear classifiers

Bag of words ◮ Each word from a pre-defined vocabulary D can be a separate feature. ◮ How many times the word a appears in the document i?

◮ or a binary flag {1, 0} of whether a appeared in i at all or not.

◮ This schema is called ‘bag of words’ (BoW).

◮ for example, if we have 1000 words in the vocabulary: ◮ xi ∈ R1000 ◮ xi = [20, 16, 0, 10, 0, . . . , 3]

15

Linear classifiers

◮ Bag-of-Words feature vector of x can be interpreted as a sum of

• ne-hot vectors (o) for each token in it:

16

Linear classifiers

◮ Bag-of-Words feature vector of x can be interpreted as a sum of

• ne-hot vectors (o) for each token in it:

◮ D extracted from the text above contains 10 words (lowercased): {‘-’, ‘by’, ‘in’, ‘most’, ‘norway’, ‘road’, ‘the’, ‘tourists’, ‘troll’, ‘visited’}.

16

Linear classifiers

◮ Bag-of-Words feature vector of x can be interpreted as a sum of

• ne-hot vectors (o) for each token in it:

◮ D extracted from the text above contains 10 words (lowercased): {‘-’, ‘by’, ‘in’, ‘most’, ‘norway’, ‘road’, ‘the’, ‘tourists’, ‘troll’, ‘visited’}. ◮ o0 = [0, 0, 0, 0, 0, 0, 1, 0, 0, 0]

16

Linear classifiers

◮ Bag-of-Words feature vector of x can be interpreted as a sum of

• ne-hot vectors (o) for each token in it:

◮ D extracted from the text above contains 10 words (lowercased): {‘-’, ‘by’, ‘in’, ‘most’, ‘norway’, ‘road’, ‘the’, ‘tourists’, ‘troll’, ‘visited’}. ◮ o0 = [0, 0, 0, 0, 0, 0, 1, 0, 0, 0] ◮ o1 = [0, 0, 0, 0, 0, 0, 0, 0, 1, 0] ◮ etc...

16

Linear classifiers

◮ Bag-of-Words feature vector of x can be interpreted as a sum of

• ne-hot vectors (o) for each token in it:

◮ D extracted from the text above contains 10 words (lowercased): {‘-’, ‘by’, ‘in’, ‘most’, ‘norway’, ‘road’, ‘the’, ‘tourists’, ‘troll’, ‘visited’}. ◮ o0 = [0, 0, 0, 0, 0, 0, 1, 0, 0, 0] ◮ o1 = [0, 0, 0, 0, 0, 0, 0, 0, 1, 0] ◮ etc... ◮ i = [1, 1, 1, 1, 1, 2, 2, 1, 1, 1] (‘the’ and ‘road’ mentioned 2 times)

16

Linear classifiers

Can we interpret the different parts of a learned model as representations

• f the data?

17

Linear classifiers

Can we interpret the different parts of a learned model as representations

• f the data?

◮ Each of n instances (documents) is represented by a vector of features (x ∈ Rdin).

17

Linear classifiers

Can we interpret the different parts of a learned model as representations

• f the data?

◮ Each of n instances (documents) is represented by a vector of features (x ∈ Rdin). ◮ Inversely, each feature can be represented by a vector of instances (documents) it appears in (feature ∈ Rn).

17

Linear classifiers

Can we interpret the different parts of a learned model as representations

• f the data?

◮ Each of n instances (documents) is represented by a vector of features (x ∈ Rdin). ◮ Inversely, each feature can be represented by a vector of instances (documents) it appears in (feature ∈ Rn). ◮ Together these learned representations form a W matrix, part of θ.

◮ Thus, it contains data both about the instances and their features (more about this later).

17

Linear classifiers

Can we interpret the different parts of a learned model as representations

• f the data?

◮ Each of n instances (documents) is represented by a vector of features (x ∈ Rdin). ◮ Inversely, each feature can be represented by a vector of instances (documents) it appears in (feature ∈ Rn). ◮ Together these learned representations form a W matrix, part of θ.

◮ Thus, it contains data both about the instances and their features (more about this later).

◮ Feature engineering is deciding what features of the instances we will use during the training.

17

Linear classifiers

positive neutral negative great best terrible worst Segal the road

18

Linear classifiers

positive neutral negative great best terrible worst Segal the road

18

Linear classifiers

positive neutral negative great best terrible worst Segal the road

18

Linear classifiers

positive neutral negative great best terrible worst Segal the road

18

Linear classifiers Overview of Linear Models

19

Linear classifiers

f (x; W , b) = x · W + b Output of binary classification Binary decision (dout = 1):

20

Linear classifiers

f (x; W , b) = x · W + b Output of binary classification Binary decision (dout = 1): ◮ ‘Is this message spam or not?’

20

Linear classifiers

f (x; W , b) = x · W + b Output of binary classification Binary decision (dout = 1): ◮ ‘Is this message spam or not?’ ◮ W is a vector, b is a scalar.

20

Linear classifiers

f (x; W , b) = x · W + b Output of binary classification Binary decision (dout = 1): ◮ ‘Is this message spam or not?’ ◮ W is a vector, b is a scalar. ◮ The prediction ˆ y is also a scalar: either 1 (‘yes’) or −1 (‘no’).

20

Linear classifiers

f (x; W , b) = x · W + b Output of binary classification Binary decision (dout = 1): ◮ ‘Is this message spam or not?’ ◮ W is a vector, b is a scalar. ◮ The prediction ˆ y is also a scalar: either 1 (‘yes’) or −1 (‘no’). ◮ NB: the model can output any number, but we convert all negatives to −1 and all positives to 1 (sign function). θ = (W ∈ Rdin, b ∈ R1)

20

Linear classifiers

1 1 0 0 1 1 1 0.5 sign(1.5) = 1

21

Linear classifiers

f (x; W , b) = x · W + b Output of multi-class classification

22

Linear classifiers

f (x; W , b) = x · W + b Output of multi-class classification Multi-class decision (dout = k)

22

Linear classifiers

f (x; W , b) = x · W + b Output of multi-class classification Multi-class decision (dout = k) ◮ ‘Which of k candidates authored this text?’

22

Linear classifiers

f (x; W , b) = x · W + b Output of multi-class classification Multi-class decision (dout = k) ◮ ‘Which of k candidates authored this text?’ ◮ W is a matrix, b is a vector of k components.

22

Linear classifiers

f (x; W , b) = x · W + b Output of multi-class classification Multi-class decision (dout = k) ◮ ‘Which of k candidates authored this text?’ ◮ W is a matrix, b is a vector of k components. ◮ The prediction ˆ y is also a one-hot vector of k components.

22

Linear classifiers

f (x; W , b) = x · W + b Output of multi-class classification Multi-class decision (dout = k) ◮ ‘Which of k candidates authored this text?’ ◮ W is a matrix, b is a vector of k components. ◮ The prediction ˆ y is also a one-hot vector of k components. ◮ The component corresponding to the correct author has the value of 1,

• thers are zeros, for example:

ˆ y = [0, 0, 1, 0] (for k = 4) θ = (W ∈ Rdin×dout, b ∈ Rdout)

22

Linear classifiers

1 1 1 0 0 1 1 1 0 1 1 1 1 1 0 1 1 1 0 0 1 2 2 4 argmax( ) 3

23

Linear classifiers

Log-linear classification If we care about how confident is the classifier about each decision:

24

Linear classifiers

Log-linear classification If we care about how confident is the classifier about each decision: ◮ Map the predictions to the range of [0, 1]...

24

Linear classifiers

Log-linear classification If we care about how confident is the classifier about each decision: ◮ Map the predictions to the range of [0, 1]... ◮ ...by a squashing function, for example, sigmoid: ˆ y = σ(f (x)) = 1 1 + e−(f (x)) (2) ◮ The result is the probability of the prediction! σ(x)

24

Linear classifiers

◮ For multi-class cases, log-linear models produce probabilities for all classes, for example: ˆ y = [0.4, 0.1, 0.9, 0.5] (for k = 4)

25

Linear classifiers

◮ For multi-class cases, log-linear models produce probabilities for all classes, for example: ˆ y = [0.4, 0.1, 0.9, 0.5] (for k = 4) ◮ We choose the one with the highest score: ˆ y = arg max

i

ˆ y[i] = ˆ y[2] (3)

25

Linear classifiers

◮ For multi-class cases, log-linear models produce probabilities for all classes, for example: ˆ y = [0.4, 0.1, 0.9, 0.5] (for k = 4) ◮ We choose the one with the highest score: ˆ y = arg max

i

ˆ y[i] = ˆ y[2] (3) ◮ But often it is more convenient to transform scores into a probability distribution, using the softmax function: ˆ y = softmax(xW + b) (4) ˆ y[i] = e(xW +b)[i]

• j e(xW +b)[j]

(5)

25

Linear classifiers

◮ For multi-class cases, log-linear models produce probabilities for all classes, for example: ˆ y = [0.4, 0.1, 0.9, 0.5] (for k = 4) ◮ We choose the one with the highest score: ˆ y = arg max

i

ˆ y[i] = ˆ y[2] (3) ◮ But often it is more convenient to transform scores into a probability distribution, using the softmax function: ˆ y = softmax(xW + b) (4) ˆ y[i] = e(xW +b)[i]

• j e(xW +b)[j]

(5) ◮ ˆ y = softmax([0.4, 0.1, 0.9, 0.5]) = [0.22, 0.16, 0.37, 0.25]

◮ (all scores sum to 1)

25

Linear classifiers

1 1 1 0 0 1 1 1 0 1 1 1 1 1 0 1 1 1 0 0 1 2 2 4 .1 .1 .8 softmax

26

Linear classifiers Break

27

Contents

1

Introduction

2

Basics of supervised machine learning

3

Linear classifiers

4

Training as optimization

5

Limitations of linear models

6

Going deeply non-linear: multi-layered perceptrons

7

Next lecture on January 28

8

Before the next lecture

27

Training as optimization

◮ The goal of the training is to find the optimal values of parameters in θ.

28

Training as optimization

◮ The goal of the training is to find the optimal values of parameters in θ. ◮ Formally, it means to minimize the loss L(θ) on training or development dataset.

28

Training as optimization

◮ The goal of the training is to find the optimal values of parameters in θ. ◮ Formally, it means to minimize the loss L(θ) on training or development dataset. ◮ Conceptually, loss is a measure of how ‘far away’ the model predictions ˆ y are from gold labels y.

28

Training as optimization

◮ The goal of the training is to find the optimal values of parameters in θ. ◮ Formally, it means to minimize the loss L(θ) on training or development dataset. ◮ Conceptually, loss is a measure of how ‘far away’ the model predictions ˆ y are from gold labels y. ◮ Formally, it can be any function L(ˆ y, y) returning a scalar value:

◮ for example, L = (y − ˆ y)2 (square error)

28

Training as optimization

◮ The goal of the training is to find the optimal values of parameters in θ. ◮ Formally, it means to minimize the loss L(θ) on training or development dataset. ◮ Conceptually, loss is a measure of how ‘far away’ the model predictions ˆ y are from gold labels y. ◮ Formally, it can be any function L(ˆ y, y) returning a scalar value:

◮ for example, L = (y − ˆ y)2 (square error)

◮ It is averaged over all training instances and gives us estimation of the model ‘fitness’.

28

Training as optimization

◮ The goal of the training is to find the optimal values of parameters in θ. ◮ Formally, it means to minimize the loss L(θ) on training or development dataset. ◮ Conceptually, loss is a measure of how ‘far away’ the model predictions ˆ y are from gold labels y. ◮ Formally, it can be any function L(ˆ y, y) returning a scalar value:

◮ for example, L = (y − ˆ y)2 (square error)

◮ It is averaged over all training instances and gives us estimation of the model ‘fitness’. ◮ ˆ θ is the best set of parameters: ˆ θ = arg min

θ

L(θ) (6)

28

Training as optimization

How do you choose a loss function?

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression - mean absolute error, mean squared error, ...

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression - mean absolute error, mean squared error, ... ◮ classification

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression - mean absolute error, mean squared error, ... ◮ classification - hinge-loss, cross-entropy, ...

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression - mean absolute error, mean squared error, ... ◮ classification - hinge-loss, cross-entropy, ... ◮ ranking

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression - mean absolute error, mean squared error, ... ◮ classification - hinge-loss, cross-entropy, ... ◮ ranking - ranking loss, triplet loss

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression - mean absolute error, mean squared error, ... ◮ classification - hinge-loss, cross-entropy, ... ◮ ranking - ranking loss, triplet loss

• 2. A mix of theoretical and practical desires often determine your final

choice

◮ For classification, we often use some variant of...

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression - mean absolute error, mean squared error, ... ◮ classification - hinge-loss, cross-entropy, ... ◮ ranking - ranking loss, triplet loss

• 2. A mix of theoretical and practical desires often determine your final

choice

◮ For classification, we often use some variant of... ◮ ... hinge-loss (max margin)

29

Training as optimization

How do you choose a loss function?

• 1. It depends on your task...

◮ regression - mean absolute error, mean squared error, ... ◮ classification - hinge-loss, cross-entropy, ... ◮ ranking - ranking loss, triplet loss

• 2. A mix of theoretical and practical desires often determine your final

choice

◮ For classification, we often use some variant of... ◮ ... hinge-loss (max margin) ◮ ... cross-entropy loss

29

Training as optimization

Common loss functions

• 1. Hinge (binary): L(ˆ

y, y) = max(0, 1 − y · ˆ y)

30

Training as optimization

Common loss functions

• 1. Hinge (binary): L(ˆ

y, y) = max(0, 1 − y · ˆ y)

• 2. Hinge (multi-class): L(ˆ

y, y) = max(0, 1 − (ˆ y[t] − ˆ y[k]))

30

Training as optimization

Common loss functions

• 1. Hinge (binary): L(ˆ

y, y) = max(0, 1 − y · ˆ y)

• 2. Hinge (multi-class): L(ˆ

y, y) = max(0, 1 − (ˆ y[t] − ˆ y[k]))

• 3. Log loss: L(ˆ

y, y) = log(1 + exp(−(ˆ y[t] − ˆ y[k]))

30

Training as optimization

Common loss functions

• 1. Hinge (binary): L(ˆ

y, y) = max(0, 1 − y · ˆ y)

• 2. Hinge (multi-class): L(ˆ

y, y) = max(0, 1 − (ˆ y[t] − ˆ y[k]))

• 3. Log loss: L(ˆ

y, y) = log(1 + exp(−(ˆ y[t] − ˆ y[k]))

• 4. Binary cross-entropy (logistic loss):

L(ˆ y, y) = −y logˆ y − (1 − y)log(1 − ˆ y)

30

Training as optimization

Common loss functions

• 1. Hinge (binary): L(ˆ

y, y) = max(0, 1 − y · ˆ y)

• 2. Hinge (multi-class): L(ˆ

y, y) = max(0, 1 − (ˆ y[t] − ˆ y[k]))

• 3. Log loss: L(ˆ

y, y) = log(1 + exp(−(ˆ y[t] − ˆ y[k]))

• 4. Binary cross-entropy (logistic loss):

L(ˆ y, y) = −y logˆ y − (1 − y)log(1 − ˆ y)

• 5. Categorical cross-entropy (negative log-likelihood):

L(ˆ y, y) = −

i

y[i]log(ˆ y[i])

30

Training as optimization

Common loss functions

• 1. Hinge (binary): L(ˆ

y, y) = max(0, 1 − y · ˆ y)

• 2. Hinge (multi-class): L(ˆ

y, y) = max(0, 1 − (ˆ y[t] − ˆ y[k]))

• 3. Log loss: L(ˆ

y, y) = log(1 + exp(−(ˆ y[t] − ˆ y[k]))

• 4. Binary cross-entropy (logistic loss):

L(ˆ y, y) = −y logˆ y − (1 − y)log(1 − ˆ y)

• 5. Categorical cross-entropy (negative log-likelihood):

L(ˆ y, y) = −

i

y[i]log(ˆ y[i])

• 6. Ranking losses, etc, etc...

30

Training as optimization

Regularization ◮ Sometimes, so as not to overfit, we pose restrictions on the possible θ.

31

Training as optimization

Regularization ◮ Sometimes, so as not to overfit, we pose restrictions on the possible θ. ◮ We would like θ to be not only good in predictions, but also not too complex; it should be ‘lean’ and avoid large weights.

31

Training as optimization

Regularization ◮ Sometimes, so as not to overfit, we pose restrictions on the possible θ. ◮ We would like θ to be not only good in predictions, but also not too complex; it should be ‘lean’ and avoid large weights.

Why do you think this is?

31

Training as optimization

Regularization ◮ Sometimes, so as not to overfit, we pose restrictions on the possible θ. ◮ We would like θ to be not only good in predictions, but also not too complex; it should be ‘lean’ and avoid large weights.

Why do you think this is?

◮ We can live with some errors on the training data, if it gives more generalization power.

31

Training as optimization

Regularization ◮ Sometimes, so as not to overfit, we pose restrictions on the possible θ. ◮ We would like θ to be not only good in predictions, but also not too complex; it should be ‘lean’ and avoid large weights.

Why do you think this is?

◮ We can live with some errors on the training data, if it gives more generalization power. ◮ For that, we minimize both the loss and the regularization term R(θ): ˆ θ = arg min

θ

L(θ) + λR(θ) (7) ◮ The hyperparameter λ is regularization weight (how important is it).

31

Training as optimization

Regularization ◮ Sometimes, so as not to overfit, we pose restrictions on the possible θ. ◮ We would like θ to be not only good in predictions, but also not too complex; it should be ‘lean’ and avoid large weights.

Why do you think this is?

◮ We can live with some errors on the training data, if it gives more generalization power. ◮ For that, we minimize both the loss and the regularization term R(θ): ˆ θ = arg min

θ

L(θ) + λR(θ) (7) ◮ The hyperparameter λ is regularization weight (how important is it). ◮ Common regularization terms:

• 1. L2 norm (Gaussian prior or weight decay);
• 2. L1 norm (sparse prior or lasso)

31

Training as optimization Now we can measure model performance. How can we change our parameters θ to improve?

32

Training as optimization

• 1. We could just randomly change some of the parameters and see if the

result is better (hill climbing algorithm)...

• 2. or we could be smarter about it (gradient-based methods).

33

Training as optimization

Optimizing with gradient ◮ ˆ θ = arg min

θ

(L(θ) + λR(θ)) is an optimization problem.

34

Training as optimization

Optimizing with gradient ◮ ˆ θ = arg min

θ

(L(θ) + λR(θ)) is an optimization problem. ◮ Commonly solved using gradient methods:

• 1. compute the loss,

34

Training as optimization

Optimizing with gradient ◮ ˆ θ = arg min

θ

(L(θ) + λR(θ)) is an optimization problem. ◮ Commonly solved using gradient methods:

• 1. compute the loss,
• 2. compute gradient of θ parameters with respect to the loss,

34

Training as optimization

Optimizing with gradient ◮ ˆ θ = arg min

θ

(L(θ) + λR(θ)) is an optimization problem. ◮ Commonly solved using gradient methods:

• 1. compute the loss,
• 2. compute gradient of θ parameters with respect to the loss,
• 3. (gradient here is the collection of partial derivatives, one for each

parameter of θ)

34

Training as optimization

Optimizing with gradient ◮ ˆ θ = arg min

θ

(L(θ) + λR(θ)) is an optimization problem. ◮ Commonly solved using gradient methods:

• 1. compute the loss,
• 2. compute gradient of θ parameters with respect to the loss,
• 3. (gradient here is the collection of partial derivatives, one for each

parameter of θ)

• 4. move the parameters in the opposite direction (to decrease the loss),

34

Training as optimization

Optimizing with gradient ◮ ˆ θ = arg min

θ

(L(θ) + λR(θ)) is an optimization problem. ◮ Commonly solved using gradient methods:

• 1. compute the loss,
• 2. compute gradient of θ parameters with respect to the loss,
• 3. (gradient here is the collection of partial derivatives, one for each

parameter of θ)

• 4. move the parameters in the opposite direction (to decrease the loss),
• 5. repeat until the optimum is found (the derivative is 0) or until the

pre-defined number of iterations (epochs) is achieved.

34

Training as optimization

Optimizing with gradient ◮ ˆ θ = arg min

θ

(L(θ) + λR(θ)) is an optimization problem. ◮ Commonly solved using gradient methods:

• 1. compute the loss,
• 2. compute gradient of θ parameters with respect to the loss,
• 3. (gradient here is the collection of partial derivatives, one for each

parameter of θ)

• 4. move the parameters in the opposite direction (to decrease the loss),
• 5. repeat until the optimum is found (the derivative is 0) or until the

pre-defined number of iterations (epochs) is achieved.

Convexity ◮ Convex functions: a single optimum point. ◮ Non-convex functions: multiple optimum points.

34

Training as optimization

35

Training as optimization

Initial Parameters

35

Training as optimization

gradient

35

Training as optimization

update parameters

35

Training as optimization

global minimum

35

Training as optimization

Error surfaces of convex and not-convex functions: Convex function Non-convex function

36

Training as optimization

Error surfaces of convex and not-convex functions: Convex function Non-convex function ◮ Convex functions can be easily minimized with gradient methods, reaching the global optimum. ◮ With non-convex functions, optimization can end up in a local optimum.

36

Training as optimization

Error surfaces of convex and not-convex functions: Convex function Non-convex function ◮ Convex functions can be easily minimized with gradient methods, reaching the global optimum. ◮ With non-convex functions, optimization can end up in a local optimum. ◮ Linear and log-linear models as a rule have convex error functions.

36

Training as optimization

Stochastic gradient descent (SGD)

37

Training as optimization

Stochastic gradient descent (SGD) ◮ SGD samples one instance from the training set and computes the error

• f the gradient on it,

37

Training as optimization

Stochastic gradient descent (SGD) ◮ SGD samples one instance from the training set and computes the error

• f the gradient on it,

◮ then θ is updated in the opposite direction,

37

Training as optimization

Stochastic gradient descent (SGD) ◮ SGD samples one instance from the training set and computes the error

• f the gradient on it,

◮ then θ is updated in the opposite direction, ◮ the update is scaled by the learning rate n (can be decaying over the training time),

37

Training as optimization

Stochastic gradient descent (SGD) ◮ SGD samples one instance from the training set and computes the error

• f the gradient on it,

◮ then θ is updated in the opposite direction, ◮ the update is scaled by the learning rate n (can be decaying over the training time), ◮ repeat until convergence.

37

Training as optimization

Stochastic gradient descent (SGD) ◮ SGD samples one instance from the training set and computes the error

• f the gradient on it,

◮ then θ is updated in the opposite direction, ◮ the update is scaled by the learning rate n (can be decaying over the training time), ◮ repeat until convergence. Instead of one instance, batches can be used (more stable and computationally efficient).

37

Training as optimization

Other gradient-based optimizers: ◮ Momentum ◮ AdaGrad ◮ RMSProp ◮ Adam ◮ etc... All implemented in the libraries we are going to use: PyTorch, Scikit-Learn, TensorFlow, Keras, etc.

38

Contents

1

Introduction

2

Basics of supervised machine learning

3

Linear classifiers

4

Training as optimization

5

Limitations of linear models

6

Going deeply non-linear: multi-layered perceptrons

7

Next lecture on January 28

8

Before the next lecture

38

Limitations of linear models

◮ Linear classifiers are efficient and effective.

39

Limitations of linear models

◮ Linear classifiers are efficient and effective. ◮ Can be used on their own (often enough in practice)... ◮ ...or as building blocks for non-linear neural classifiers.

39

Limitations of linear models

◮ Linear classifiers are efficient and effective. ◮ Can be used on their own (often enough in practice)... ◮ ...or as building blocks for non-linear neural classifiers. ◮ Unfortunately, linear models can represent only linear relations in the data

39

Limitations of linear models

◮ Are there non-linear functions that linear models can’t deal with?

40

Limitations of linear models

◮ Are there non-linear functions that linear models can’t deal with? ◮ Yes, there are.

40

Limitations of linear models

◮ Are there non-linear functions that linear models can’t deal with? ◮ Yes, there are. ◮ One example is the XOR function:

40

Limitations of linear models

◮ Are there non-linear functions that linear models can’t deal with? ◮ Yes, there are. ◮ One example is the XOR function: It is clearly not linearly separable.

40

Limitations of linear models

Possible solutions ◮ We can transform the input so that it becomes linearly separable.

41

Limitations of linear models

Possible solutions ◮ We can transform the input so that it becomes linearly separable. ◮ Linear transformations will not be able to do this.

41

Limitations of linear models

Possible solutions ◮ We can transform the input so that it becomes linearly separable. ◮ Linear transformations will not be able to do this. ◮ We need non-linear transformations.

41

Limitations of linear models

Possible solutions ◮ We can transform the input so that it becomes linearly separable. ◮ Linear transformations will not be able to do this. ◮ We need non-linear transformations. φ(x1, x2) = [x1 × x2, x1 + x2] maps the instances to another representation and makes the XOR problem linearly separable:

41

Limitations of linear models

◮ But how to find the transformation function φ suitable for the task at hand?

42

Limitations of linear models

◮ But how to find the transformation function φ suitable for the task at hand? ◮ Often, this implies mapping instances to a higher-dimensional space, making it even more difficult to choose φ manually.

42

Limitations of linear models

◮ But how to find the transformation function φ suitable for the task at hand? ◮ Often, this implies mapping instances to a higher-dimensional space, making it even more difficult to choose φ manually. ◮ Support Vector Machines (SVM) classifiers handle this to some extent... [Cortes and Vapnik, 1995]

42

Limitations of linear models

◮ But how to find the transformation function φ suitable for the task at hand? ◮ Often, this implies mapping instances to a higher-dimensional space, making it even more difficult to choose φ manually. ◮ Support Vector Machines (SVM) classifiers handle this to some extent... [Cortes and Vapnik, 1995] ◮ ...but they scale linearly in time on the size of the training data (slow!).

42

Limitations of linear models

Training mapping functions ◮ Idea: leave it for the algorithm to train a suitable representation mapping function!

43

Limitations of linear models

Training mapping functions ◮ Idea: leave it for the algorithm to train a suitable representation mapping function! ˆ y = φ(x)W + b (8) φ(x) = g(xW ′ + b′) (9) ◮ ...where g is a non-linear activation function, and W ′, b′ are its trainable parameters.

43

Limitations of linear models

Training mapping functions ◮ Idea: leave it for the algorithm to train a suitable representation mapping function! ˆ y = φ(x)W + b (8) φ(x) = g(xW ′ + b′) (9) ◮ ...where g is a non-linear activation function, and W ′, b′ are its trainable parameters. ◮ The equation above defines a simple multi-layer perceptron (MLP): our first neural model.

43

Contents

1

Introduction

2

Basics of supervised machine learning

3

Linear classifiers

4

Training as optimization

5

Limitations of linear models

6

Going deeply non-linear: multi-layered perceptrons

7

Next lecture on January 28

8

Before the next lecture

43

Going deeply non-linear: multi-layered perceptrons

Perceptron with 2 hidden layers

44

Going deeply non-linear: multi-layered perceptrons

The nature of perceptrons ◮ Input data goes through successive transformations at each layer.

45

Going deeply non-linear: multi-layered perceptrons

The nature of perceptrons ◮ Input data goes through successive transformations at each layer. ◮ The transformations are linear, but followed with a non-linear activation at each hidden layer.

45

Going deeply non-linear: multi-layered perceptrons

The nature of perceptrons ◮ Input data goes through successive transformations at each layer. ◮ The transformations are linear, but followed with a non-linear activation at each hidden layer. ◮ At the last layer, the prediction ˆ y is produced.

45

Going deeply non-linear: multi-layered perceptrons

The nature of perceptrons ◮ Input data goes through successive transformations at each layer. ◮ The transformations are linear, but followed with a non-linear activation at each hidden layer. ◮ At the last layer, the prediction ˆ y is produced. ◮ Representation functions and the linear classifiers are trained simultaneously.

45

Going deeply non-linear: multi-layered perceptrons

The nature of perceptrons ◮ Input data goes through successive transformations at each layer. ◮ The transformations are linear, but followed with a non-linear activation at each hidden layer. ◮ At the last layer, the prediction ˆ y is produced. ◮ Representation functions and the linear classifiers are trained simultaneously. Important: neural networks with hidden layers can theoretically approximate any function [Leshno et al., 1993].

45

Going deeply non-linear: multi-layered perceptrons

Perceptron with 2 hidden layers

46

Contents

1

Introduction

2

Basics of supervised machine learning

3

Linear classifiers

4

Training as optimization

5

Limitations of linear models

6

Going deeply non-linear: multi-layered perceptrons

7

Next lecture on January 28

8

Before the next lecture

46

Next lecture on January 28

Training Deep Neural Networks ◮ More on multi-layer perceptrons and feed-forward neural networks. ◮ Is it really like brain? ◮ Common activation functions. ◮ Regularizing neural networks with dropout. ◮ Computation graphs.

47

Contents

1

Introduction

2

Basics of supervised machine learning

3

Linear classifiers

4

Training as optimization

5

Limitations of linear models

6

Going deeply non-linear: multi-layered perceptrons

7

Next lecture on January 28

8

Before the next lecture

47

Before the next lecture

Obligatory assignment ◮ The first obligatory assignment is already out! ◮ Due February 7. ◮ Look it up on the course page.

48

Before the next lecture

Obligatory assignment ◮ The first obligatory assignment is already out! ◮ Due February 7. ◮ Look it up on the course page. Group session ◮ Hands-on: using linear classifiers and MLPs in scikit-learn. ◮ Document classification with BOW features. ◮ Make sure to apply for Saga account!

48

Before the next lecture

Obligatory assignment ◮ The first obligatory assignment is already out! ◮ Due February 7. ◮ Look it up on the course page. Group session ◮ Hands-on: using linear classifiers and MLPs in scikit-learn. ◮ Document classification with BOW features. ◮ Make sure to apply for Saga account!

48

References I

Cortes, C. and Vapnik, V. (1995). Support-vector networks. In Machine Learning, pages 273–297. Goldberg, Y. (2017). Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1):1–309. Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861–867.

49