Advanced Machine Learning CS 7140 - Spring 2019 Lecture 20: - - PowerPoint PPT Presentation

Advanced Machine Learning

CS 7140 - Spring 2019

Lecture 20: Generative Adversarial Networks

Jan-Willem van de Meent Slide credits: Ian Goodfellow

Variational Autoencoders

784 (28 x 28) 256 Input Images Hidden Units 2-50 Encoding (random) Mean
 Std Dev 256 784 (28 x 28) Hidden Units Reconstructed
 Images

Variational Autoencoders

Assume prior:

Objective: Evidence Lower Bound (ELBO)

Reconstruction
 log-likelihood KL between approximate 
 posterior and prior Log marginal likelihood KL between approximate posterior and posterior

Implicit Generative Models

z x

• Can generate samples x ~ pG(x)
• Cannot evaluate density pG(x)
• Requirement: G(z ; θ) is differentiable

Maximum Likelihood

θ∗ = arg max

θ

Ex∼pdata log pmodel(x | θ)

Generative Adversarial Networks

x sampled from data Differentiable function D D(x) tries to be near 1 Input noise z Differentiable function G x sampled from model D D tries to make D(G(z)) near 0, G tries to make D(G(z)) near 1

Objective: Minimax Game

• Generator G(z) generates fake images.
• Discriminator D(x) classifies real images as 1 


and fake images as 0.

• Objective for G: Maximize probability that G fools D.
• Objective for D: Minimize probability that D is fooled.

max

G

min

D −Ex∼pDATA(x) [log D(x)] − Ez∼p(z) [log(1 − D(G(z)))]

Exercise

• Suppose that we could evaluate pDATA(x) and pG(x)
• Can you express D(x) in terms of pDATA(x) and pG(x)?

SOLUTION:

Learning Implicit Densities

z x

(Goodfellow et al, 2014)

Data: Discriminator: Implicit Density:

Generator:

Key Idea: Use classifier to 
 estimate ratio 


• f densities

Understanding the Loss

= JSD(pDATA(x) || pG(x)) log4

' 1 2Ex⇠pDATA(x) ï log pDATA(x) pDATA(x) + pG(x) ò ï ò

GAN loss minimizes the Jensen-Shannon divergence

ï ò 1 2Ex⇠pG(x) ï log pG(x) pDATA(x) + pG(x) ò

L(G, D) = 1 2Ex⇠pDATA(x) [log D(z)] 1 2Ez⇠p(z) [log(1 D(G(z)))]

Problem: Saturation

Question: What happens to the gradient w.r.t. θ 
 when the discriminator D(x) = 0

Problem: Saturation

Question: What happens to the gradient w.r.t. θ 
 when the discriminator D(x) = 0

Non-Saturating GAN Loss

• Use different loss for generator parameters θ
• Αvoid 0 gradient issue when D(x) = 0, 


(but loss no longer takes form of JS divergence)

L(φ) = −Ex∼pDATA(x) [log D(x;φ)] − Ez∼p(z) [log(1 − D(G(z;θ);φ))] L(θ) = Ez∼p(z) [log D(G(z;θ);φ)]

DCGAN Architecture

(Radford et al 2015) Most “deconvs” are batch normalized

LSUN Bedrooms

(Radford et al 2015)

Vector Space Arithmetic

• +

=

Man with Glasses Man Woman Woman with Glasses (Radford et al, 2015)

GANs vs VAEs

• VAEs learn a generative model and an inference model
• GANs learn a generator and a discriminator
• VAEs assume an explicit likelihood
• GANs assume an implicit likelihood
• VAEs do inference, but yield blurry images
• GANs don’t do inference, but yield crisper images

Next Frame Video Prediction

Ground Truth MSE Adversarial

(Lotter et al 2016)

Next Frame Video Prediction

(Mathieu et al. 2015)

Mean Squared Error Mean Absolute Error Adversarial

Is the Divergence Important?

x Probability Density

q∗ = argminqDKL(pq) p(x) q∗(x)

x Probability Density

q∗ = argminqDKL(qp) p(x) q∗(x)

(Goodfellow et al 2016) Variational Inference (and VAEs) Expectation Propagation (and GANs)

Using GANs for ML estimation

Generator loss minimizes reconstruction loss

Conclusion: GANs and VAEs not different due divergence

Challenges in Training GANS

Oscillation: Generator can “fool” discriminator by producing previously unseen garbage
 (for which discriminator output is unreliable) Mode Collapse: Generator can memorize a small subset of the data to which the discriminator attributes a high degree of realism

Mode Collapse

(Reed et al 2016)

this small bird has a pink breast and crown, and black primaries and secondaries. the flower has petals that are bright pinkish purple with white stigma this magnificent fellow is almost all black with a red crest, and white cheek patch. this white and yellow flower have thin white petals and a round yellow stamen this small bird has a pink breast and crown, and black primaries and secondaries. the flower has petals that are bright pinkish purple with white stigma this magnificent fellow is almost all black with a red crest, and white cheek patch. this white and yellow flower have thin white petals and a round yellow stamen

(Metz et al 2016)

max

G

min

D

L(G, D) 6= min

D max G

L(G, D)

Intuition: Generator learns to exploit weaknesses in discriminator

Minibatch Features

Idea: Discriminator looks at batch of images

• Add minibatch features that classify each example

by comparing it to other members of the minibatch (Salimans et al 2016)

• Nearest-neighbor style features detect if a minibatch

contains samples that are too similar to each other

Minibatch Features

Data (ImageNet) Generator [Salimans et al 2016]

Applications of GANs

What is in this Image?

(Yeh et al., 2016)

Generative Modeling Reveals a Face

(Yeh et al., 2016)

Image-to-Image Translation

(Liu et al., 2017) Day to Night

CycleGAN

(Zhu et al., 2017) Horse to Zebra

CycleGAN

(Zhu et al., 2017)

Zebras Horses

horse zebra zebra horse

Summer Winter

summer winter winter summer Photograph Van Gogh Cezanne Monet Ukiyo-e

Monet Photos

Monet photo photo Monet

CycleGAN

(Zhu et al., 2017)

Zebras Horses

horse zebra zebra horse

Summer Winter

summer winter winter summer Photograph Van Gogh Cezanne Monet Ukiyo-e

Monet Photos

Monet photo photo Monet

CycleGAN

(Zhu et al., 2017)

X Y G F DY DX

G

F ˆ Y X Y

• X

Y

• G

F ˆ X

(a) (b) (c)

cycle-consistency

loss

cycle-consistency loss

DY DX

ˆ y ˆ x

x y

Which of These are Real?

vue.ai

✓ ✓

Celebrities Who Have Never Existed

(Karras et al., 2017) “never before wholly perceived in reality”

Semi-supervised Learning

Input Real Hidden units Fake Input Real dog Hidden units Fake Real cat

L = Ex,y∼pDATA(x,y) [log p(y | x)] − Ez∼p(z) [log p(y = K + 1 | G(z))]

Semi-supervised Learning

MNIST: 100 training labels -> 80 test mistakes SVHN: 1,000 training labels -> 4.3% test error CIFAR-10: 4,000 labels -> 14.4% test error (Dai et al 2017)