Lecture 8: Autoencoder & DBM Princeton University COS 495 - - PowerPoint PPT Presentation

β–Ά
lecture 8 autoencoder dbm
SMART_READER_LITE
LIVE PREVIEW

Lecture 8: Autoencoder & DBM Princeton University COS 495 - - PowerPoint PPT Presentation

Sep 14, 2022 746 likes β€’1.04k views

Deep Learning Basics Lecture 8: Autoencoder & DBM Princeton University COS 495 Instructor: Yingyu Liang Autoencoder Autoencoder Neural networks trained to attempt to copy its input to its output Contain two parts: Encoder: map

slide-1
SLIDE 1

Deep Learning Basics Lecture 8: Autoencoder & DBM

Princeton University COS 495 Instructor: Yingyu Liang

slide-2
SLIDE 2

Autoencoder

slide-3
SLIDE 3

Autoencoder

  • Neural networks trained to attempt to copy its input to its output
  • Contain two parts:
  • Encoder: map the input to a hidden representation
  • Decoder: map the hidden representation to the output
slide-4
SLIDE 4

Autoencoder

β„Ž 𝑦 𝑠 Hidden representation (the code) Reconstruction Input

slide-5
SLIDE 5

Autoencoder

β„Ž 𝑦 𝑠 Decoder 𝑕(β‹…) Encoder 𝑔(β‹…) β„Ž = 𝑔 𝑦 , 𝑠 = 𝑕 β„Ž = 𝑕(𝑔 𝑦 )

slide-6
SLIDE 6

Why want to copy input to output

  • Not really care about copying
  • Interesting case: NOT able to copy exactly but strive to do so
  • Autoencoder forced to select which aspects to preserve and thus

hopefully can learn useful properties of the data

  • Historical note: goes back to (LeCun, 1987; Bourlard and Kamp, 1988;

Hinton and Zemel, 1994).

slide-7
SLIDE 7

Undercomplete autoencoder

  • Constrain the code to have smaller dimension than the input
  • Training: minimize a loss function

𝑀 𝑦, 𝑠 = 𝑀(𝑦, 𝑕 𝑔 𝑦 ) β„Ž 𝑦 𝑠

slide-8
SLIDE 8

Undercomplete autoencoder

  • Constrain the code to have smaller dimension than the input
  • Training: minimize a loss function

𝑀 𝑦, 𝑠 = 𝑀(𝑦, 𝑕 𝑔 𝑦 )

  • Special case: 𝑔, 𝑕 linear, 𝑀 mean square error
  • Reduces to Principal Component Analysis
slide-9
SLIDE 9

Undercomplete autoencoder

  • What about nonlinear encoder and decoder?
  • Capacity should not be too large
  • Suppose given data 𝑦1, 𝑦2, … , π‘¦π‘œ
  • Encoder maps 𝑦𝑗 to 𝑗
  • Decoder maps 𝑗 to 𝑦𝑗
  • One dim β„Ž suffices for perfect reconstruction
slide-10
SLIDE 10

Regularization

  • Typically NOT
  • Keeping the encoder/decoder shallow or
  • Using small code size
  • Regularized autoencoders: add regularization term that encourages

the model to have other properties

  • Sparsity of the representation (sparse autoencoder)
  • Robustness to noise or to missing inputs (denoising autoencoder)
  • Smallness of the derivative of the representation
slide-11
SLIDE 11

Sparse autoencoder

  • Constrain the code to have sparsity
  • Training: minimize a loss function

𝑀𝑆 = 𝑀(𝑦, 𝑕 𝑔 𝑦 ) + 𝑆(β„Ž) β„Ž 𝑦 𝑠

slide-12
SLIDE 12

Probabilistic view of regularizing β„Ž

  • Suppose we have a probabilistic model π‘ž(β„Ž, 𝑦)
  • MLE on 𝑦

log π‘ž(𝑦) = log ෍

β„Žβ€²

π‘ž(β„Žβ€², 𝑦)

  •  Hard to sum over β„Žβ€²
slide-13
SLIDE 13

Probabilistic view of regularizing β„Ž

  • Suppose we have a probabilistic model π‘ž(β„Ž, 𝑦)
  • MLE on 𝑦

max log π‘ž(𝑦) = max log ෍

β„Žβ€²

π‘ž(β„Žβ€², 𝑦)

  • Approximation: suppose β„Ž = 𝑔(𝑦) gives the most likely hidden

representation, and Οƒβ„Žβ€² π‘ž(β„Žβ€², 𝑦) can be approximated by π‘ž(β„Ž, 𝑦)

slide-14
SLIDE 14

Probabilistic view of regularizing β„Ž

  • Suppose we have a probabilistic model π‘ž(β„Ž, 𝑦)
  • Approximate MLE on 𝑦, β„Ž = 𝑔(𝑦)

max log π‘ž(β„Ž, 𝑦) = max log π‘ž(𝑦|β„Ž) + log π‘ž(β„Ž) Regularization Loss

slide-15
SLIDE 15

Sparse autoencoder

  • Constrain the code to have sparsity
  • Laplacian prior: π‘ž β„Ž =

πœ‡ 2 exp(βˆ’ πœ‡ 2 β„Ž 1)

  • Training: minimize a loss function

𝑀𝑆 = 𝑀(𝑦, 𝑕 𝑔 𝑦 ) + πœ‡ β„Ž 1

slide-16
SLIDE 16

Denoising autoencoder

  • Traditional autoencoder: encourage to learn 𝑕 𝑔 β‹…

to be identity

  • Denoising : minimize a loss function

𝑀 𝑦, 𝑠 = 𝑀(𝑦, 𝑕 𝑔 ΰ·€ 𝑦 ) where ΰ·€ 𝑦 is 𝑦 + π‘œπ‘π‘—π‘‘π‘“

slide-17
SLIDE 17

Boltzmann machine

slide-18
SLIDE 18

Boltzmann machine

  • Introduced by Ackley et al. (1985)
  • General β€œconnectionist” approach to learning arbitrary probability

distributions over binary vectors

  • Special case of energy model: π‘ž 𝑦 =

exp(βˆ’πΉ 𝑦 ) π‘Ž

slide-19
SLIDE 19

Boltzmann machine

  • Energy model:

π‘ž 𝑦 = exp(βˆ’πΉ 𝑦 ) π‘Ž

  • Boltzmann machine: special case of energy model with

𝐹 𝑦 = βˆ’π‘¦π‘ˆπ‘‰π‘¦ βˆ’ π‘π‘ˆπ‘¦ where 𝑉 is the weight matrix and 𝑐 is the bias parameter

slide-20
SLIDE 20

Boltzmann machine with latent variables

  • Some variables are not observed

𝑦 = 𝑦𝑀, π‘¦β„Ž , 𝑦𝑀 visible, π‘¦β„Ž hidden 𝐹 𝑦 = βˆ’π‘¦π‘€

π‘ˆπ‘†π‘¦π‘€ βˆ’ 𝑦𝑀 π‘ˆπ‘‹π‘¦β„Ž βˆ’ π‘¦β„Ž π‘ˆπ‘‡π‘¦β„Ž βˆ’ π‘π‘ˆπ‘¦π‘€ βˆ’ π‘‘π‘ˆπ‘¦β„Ž

  • Universal approximator of probability mass functions
slide-21
SLIDE 21

Maximum likelihood

  • Suppose we are given data π‘Œ = 𝑦𝑀

1, 𝑦𝑀 2, … , 𝑦𝑀 π‘œ

  • Maximum likelihood is to maximize

log π‘ž π‘Œ = ෍

𝑗

log π‘ž(𝑦𝑀

𝑗 )

where π‘ž 𝑦𝑀 = ෍

π‘¦β„Ž

π‘ž(𝑦𝑀, π‘¦β„Ž) = ෍

π‘¦β„Ž

1 π‘Ž exp(βˆ’πΉ(𝑦𝑀, π‘¦β„Ž))

  • π‘Ž = Οƒ exp(βˆ’πΉ(𝑦𝑀, π‘¦β„Ž)): partition function, difficult to compute
slide-22
SLIDE 22

Restricted Boltzmann machine

  • Invented under the name harmonium (Smolensky, 1986)
  • Popularized by Hinton and collaborators to Restricted Boltzmann

machine

slide-23
SLIDE 23

Restricted Boltzmann machine

  • Special case of Boltzmann machine with latent variables:

π‘ž 𝑀, β„Ž = exp(βˆ’πΉ 𝑀, β„Ž ) π‘Ž where the energy function is 𝐹 𝑀, β„Ž = βˆ’π‘€π‘ˆπ‘‹β„Ž βˆ’ π‘π‘ˆπ‘€ βˆ’ π‘‘π‘ˆβ„Ž with the weight matrix 𝑋 and the bias 𝑐, 𝑑

  • Partition function

π‘Ž = ෍

𝑀

෍

β„Ž

exp(βˆ’πΉ 𝑀, β„Ž )

slide-24
SLIDE 24

Restricted Boltzmann machine

Figure from Deep Learning, Goodfellow, Bengio and Courville

slide-25
SLIDE 25

Restricted Boltzmann machine

  • Conditional distribution is factorial

π‘ž β„Ž|𝑀 = π‘ž(𝑀, β„Ž) π‘ž(𝑀) = ΰ·‘

π‘˜

π‘ž(β„Žπ‘˜|𝑀) and π‘ž β„Žπ‘˜ = 1|𝑀 = 𝜏 𝑑

π‘˜ + π‘€π‘ˆπ‘‹ :,π‘˜

is logistic function

slide-26
SLIDE 26

Restricted Boltzmann machine

  • Similarly,

π‘ž 𝑀|β„Ž = π‘ž(𝑀, β„Ž) π‘ž(β„Ž) = ΰ·‘

𝑗

π‘ž(𝑀𝑗|β„Ž) and π‘ž 𝑀𝑗 = 1|β„Ž = 𝜏 𝑐𝑗 + 𝑋

𝑗,:β„Ž

is logistic function

slide-27
SLIDE 27

Deep Boltzmann machine

  • Special case of energy model. Take 3 hidden layers and ignore bias:

π‘ž 𝑀, β„Ž1, β„Ž2, β„Ž3 = exp(βˆ’πΉ 𝑀, β„Ž1, β„Ž2, β„Ž3 ) π‘Ž

  • Energy function

𝐹 𝑀, β„Ž1, β„Ž2, β„Ž3 = βˆ’π‘€π‘ˆπ‘‹1β„Ž1 βˆ’ (β„Ž1)π‘ˆπ‘‹2β„Ž2 βˆ’ (β„Ž2)π‘ˆπ‘‹3β„Ž3 with the weight matrices 𝑋1, 𝑋2, 𝑋3

  • Partition function

π‘Ž = ෍

𝑀,β„Ž1,β„Ž2,β„Ž3

exp(βˆ’πΉ 𝑀, β„Ž1, β„Ž2, β„Ž3 )

slide-28
SLIDE 28

Deep Boltzmann machine

Figure from Deep Learning, Goodfellow, Bengio and Courville