Traitement automatique des langues : Fondements et applications - - PowerPoint PPT Presentation

Traitement automatique des langues : Fondements et applications

Cours 11 : Neural networks (2) Tim Van de Cruys & Philippe Muller 2016—2017

Introduction

Machine learning for NLP

• Standard approach: linear model trained over high-dimensional

but very sparse feature vectors

• Recently: non-linear neural networks over dense input vectors

Neural Network Architectures

Feed-forward neural networks

• Best known, standard neural network approach
• Fully connected layers
• Can be used as drop-in replacement for typical NLP classifiers

Convolutional neural network

Introduction

• Type of feedforward neural network
• Certain layers are not fully connected but locally connected

(convolutional layers, pooling layers)

• same, local cues appear in different places in input (cfr. vision)

Convolutional neural network

Intuition

Convolutional neural network

Intuition

Convolutional neural network

Intuition

Convolutional neural network

Architecture

Convolutional neural network

Encoding sentences

How to represent variable number of features, e.g. words in a sentence, document?

• Continuous Bag of Words (CBOW): sum embedding vectors of

corresponding features

• no ordering info (”not good quite bad” = ”not bad quite good”)
• Convolutional layer
• ’Sliding window’ approach that takes local structure into account
• Combine individual windows to create vector of fixed size

Continuous bag of words

Variable number of features

• Feed-forward network assumes fixed dimensional input
• How to represent variable number of features, e.g. words in a

sentence, document?

• Continuous Bag of Words (CBOW): sum embedding vectors of

corresponding features

Convolutional neural network

Convolutional layer for NLP

• Goal: identify indicative local features (n-grams) in large

structure, combine them into fixed size vector

• Convolution: apply filter to each window (linear transformation

+ non-linear activation)

• Pooling: combine by taking maximum

Convolutional neural networks

Architecture for NLP

Neural Network Architectures

Recurrent (+ recursive) neural networks

• Handle structured data of arbitrary sizes
• Recurrent networks for sequences
• Recursive networks for trees

Recurrent neural network

Introduction

• CBOW: no ordering, no structure
• CNN: improvement, but mostly local patterns
• RNN: represent arbitrarily sized structured input as fixed-size

vectors, paying attention to structured properties

Recurrent neural network

Model

• x1: input layer (current word)
• a1: hidden layer of current timestep
• a0: hidden layer of previous timestep
• U, W and V: weights matrices
• f(·): element-wise activation function (sigmoid)
• g(·): softmax function to ensure probability distribution

a1 = f(Ux1 + Wa0) (1) y1 = g(Va1) (2)

Recurrent neural network

Graphical representation

Recurrent neural network

Training

• Consider recurrent neural network as very deep neural network

with shared parameters across computation

• Backpropagation through time
• What kind of supervision?
• Acceptor: based on final state
• Transducer: an output for each input (e.g. language modeling)
• Encoder-decoder: one RNN to encode sequence into vector

representation, another RNN to decode into sequence (e.g. machine translation)

Recurrent neural network

Training: graphical representation

Recurrent neural network

Multi-layer RNN

• multiple layers of RNNs
• input of next layer is output of RNN layer below it
• Empirically shown to work better

Recurrent neural network

Bi-directional RNN

• Input sequence both forward and backward to different RNNs
• Representation is concatenation of forward and backward state

(A & A’)

• Represent both history and future

Concrete RNN architectures

Simple RNN

Concrete RNN architectures

LSTM

• Long short term memory networks
• In practice, simple RNNs only able to remember narrow context

(vanishing gradient)

• LSTM: complex architecture able to capture long-term

dependencies

Concrete RNN architectures

LSTM

Concrete RNN architectures

LSTM

Concrete RNN architectures

LSTM

Concrete RNN architectures

LSTM

Concrete RNN architectures

LSTM

Concrete RNN architectures

LSTM

Concrete RNN architectures

GRU

• LSTM: effective, but complex, computationally expensive
• GRU: cheaper alternative that works well in practice

Concrete RNN architectures

GRU

• reset gate (r): how much information from previous hidden

state needs to be included (reset with current information?)

• upgate gate (z): controls updates to hidden state (how much

does hidden state need to be updated with current information?)

Recursive neural networks

Introduction

• Generalization of RNNs from sequences to (binary) trees
• Linear transformation + non-linear activation function applied

recursively throughout a tree

• Useful for parsing

Application

Image to caption generation

Application

Image to caption generation

Application

Neural machine translation

Application

Neural machine translation

Application

Neural dialogue generation (chatbot)

Software

• Tensorflow
• Python, C++
• http://www.tensorflow.org
• Theano
• Python
• http://deeplearning.net/software/theano/
• Keras
• Theano/tensorflow-based modular deep learning library
• Lasagne
• Theano-based deep learning library