Structured Prediction Problem Unstructured prediction Structured - - PDF document

▶
structured prediction problem
SMART_READER_LITE
LIVE PREVIEW

Structured Prediction Problem Unstructured prediction Structured - - PDF document

Oct 09, 2023 370 likes •635 views

Laplace Max-margin Markov Networks Recent Recent A Advances in dvances in L Learning earning SPARSE SPARSE Structured S tructured I I/ /O O M Model odels s: : models, algorithms, and applications models, algorithms, and

slide-1
SLIDE 1

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

1

Eric Xing Eric Xing

epxing@cs.cmu.edu Machine Learning Dept./Language Technology Inst./Computer Science Dept.

Carnegie Mellon University Carnegie Mellon University

1

Recent Recent A Advances in dvances in L Learning earning SPARSE SPARSE S Structured tructured I I/ /O O M Model

  • dels

s: : models, algorithms, and applications models, algorithms, and applications

8/6/2009 VLPR 2009 @ Beijing, China

2

Structured Prediction Problem

“Do you want sugar in it?”

⇒

<verb pron verb noun prep pron>

Unstructured prediction Structured prediction

Part of speech tagging Image segmentation

slide-2
SLIDE 2

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

3

Advantages:

1.Full probabilistic semantics 2.Straightforward Bayesian or direct regularization 3.Hidden structures or generative hierarchy

Classical Predictive Models

  • Inputs:

– a set of training samples , where

and

  • Outputs:

– a predictive function :

  • Examples:

– Support Vector Machines (SVM)

  • Max-margin learning

– Logistic Regression, Bayes classifiers

  • Max-likelihood estimation

Advantages:

1.Dual sparsity: few support vectors 2.Kernel tricks 3.Strong empirical results E.g.: 8/6/2009 VLPR 2009 @ Beijing, China

4

Structured Prediction Models

Conditional Random Fields

(CRFs) (Lafferty et al 2001)

–

Based on Logistic Regression

–

Max-likelihood estimation (point- estimate)

  • Max-margin Markov Networks

(M3Ns) (Taskar et al 2003)

– Based on SVM – Max-margin learning ( point-estimate)

Challenges:

  • SPARSE prediction model
  • Prior information of structures
  • Scalable to large-scale problems

(e.g., 104 input/output dimension) ACGTTTTACTGTACAATT

slide-3
SLIDE 3

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

5

Outline

Structured sparse regression

Graph-guided fused lasso: unlinked SNPs to trait networks (Kim and

Xing, PLoS Genetics)

Temporally-smoothed graph regression: learning time-varying

graphs (Ahmed and Xing, PNAS 2009, Kolar and Xing, under review, Annals of Statistics)

Maximum entropy discrimination Markov networks

– General Theorems (Zhu and Xing, JMLR submitted) – Gaussian MEDN: reduction to M3N (Zhu, Xing and Zhang, ICML 08) – Laplace MEDN: a sparse M3N (Zhu, Xing and Zhang, ICML 08) – Partially observed MEDN: (Zhu, Xing and Zhang, NIPS 08) – Max-margin/Max entropy topic model: (Zhu, Ahmed, and Xing, ICML 09) 8/6/2009 VLPR 2009 @ Beijing, China 6

Max-Margin Learning Paradigms

? SVM SVM

b r a c e

M3N MED MED M3N MED-MN = SMED + “Bayesian” M3N

Primal and Dual Sparse!

slide-4
SLIDE 4

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

7

Primal and Dual Problems of M3Ns

Primal problem:

Algorithms

–

Cutting plane

–

Sub-gradient

–

…

Dual problem:

  • Algorithms:

– SMO – Exponentiated gradient – …

  • So, M3N is dual sparse!

8/6/2009 VLPR 2009 @ Beijing, China

8

MLE versus max-margin learning

Likelihood-based estimation

–

Probabilistic (joint/conditional likelihood model)

–

Easy to perform Bayesian learning, and incorporate prior knowledge, latent structures, missing data

–

Bayesian regularization!!

  • Max-margin learning

– Non-probabilistic (concentrate on input-

  • utput mapping)

– Not obvious how to perform Bayesian learning or consider prior, and missing data – Sound theoretical guarantee with limited samples

  • Maximum Entropy Discrimination (MED) (Jaakkola, et al., 1999)

– Model averaging – The optimization problem (binary classification)

slide-5
SLIDE 5

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

9

Structured MaxEnt Discrimination (SMED): Feasible subspace of weight distribution: Average from distribution of M3Ns

MaxEnt Discrimination Markov Network

p

8/6/2009 VLPR 2009 @ Beijing, China

10 10

Solution to MaxEnDNet

Theorem 1:

– Posterior Distribution: – Dual Optimization Problem:

slide-6
SLIDE 6

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

11 11

Gaussian MaxEnDNet (reduction to M3N)

Theorem 2

– Assume

Posterior distribution: Dual optimization: Predictive rule:

Thus, MaxEnDNet subsumes M3Ns and admits all the

merits of max-margin learning

Furthermore, MaxEnDNet has at least three

advantages …

8/6/2009 VLPR 2009 @ Beijing, China

12 12

Three Advantages

An averaging Model: PAC-Bayesian prediction error

guarantee

Entropy regularization: Introducing useful biases

Standard Normal prior => reduction to standard M3N (we’ve seen it) Laplace prior => Posterior

shrinkage effects (sparse M3N)

Integrating Generative and Discriminative principles

Incorporate latent variables and structures (PoMEN) Semisupervised learning (with partially labeled data)

slide-7
SLIDE 7

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

13 13

I: Generalization Guarantee

MaxEntNet is an averaging model Theorem 3 (PAC-Bayes Bound)

If Then

8/6/2009 VLPR 2009 @ Beijing, China

14 14

II: Laplace MaxEnDNet (primal sparse M3N)

Laplace Prior: Corollary 4:

Under a Laplace MaxEnDNet, the posterior mean of

parameter vector w is:

The Gaussian MaxEnDNet and the regular M3N has no

such shrinkage

there, we have

slide-8
SLIDE 8

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

15 15

LapMEDN vs. L2 and L1 regularization

L1 and L2 norms KL norms

Corollary 5: LapMEDN corresponding to solving the

following primal optimization problem:

KL norm:

8/6/2009 VLPR 2009 @ Beijing, China

16 16

Variational Learning of LapMEDN

Exact dual function is hard to optimize Use the hierarchical representation, we get: We optimize an upper bound: Why is it easier?

– Alternating minimization leads to nicer optimization problems

slide-9
SLIDE 9

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

17 17

Experimental results on OCR datasets

We randomly construct OCR100, OCR150, OCR200, and

OCR250 for 10 fold CV.

8/6/2009 VLPR 2009 @ Beijing, China

18 18

Feature Selection

slide-10
SLIDE 10

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

19 19

Sensitivity to Regularization Constants

  • L1-CRFs are much sensitive to regularization constants;

the others are more stable

  • LapM3N is the most stable one

L1-CRF and L2-CRF:

  • 0.001, 0.01, 0.1, 1, 4, 9, 16

M3N and LapM3N:

  • 1, 4, 9, 16, 25, 36, 49, 64, 81

8/6/2009 VLPR 2009 @ Beijing, China

20 20

III: Latent Hierarchical MaxEnDNet

Web data extraction

Goal: Name, Image, Price,

Description, etc. Given Data Record Hierarchical labeling Advantages:

  • Computational efficiency
  • Long-range dependency
  • Joint extraction

{image} {name, price} {name} {price} {name} {price} {image} {name, price} {desc} {Head} {Tail} {Info Block} {Repeat block} {Note} {Note}

slide-11
SLIDE 11

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

21 21

Now we are given partially labeled data:

PoMEN: learning Prediction:

Partially Observed MaxEnDNet (PoMEN)

8/6/2009 VLPR 2009 @ Beijing, China

22 22

Alternating Minimization Alg.

Factorization assumption: Alternating minimization:

Step 1: keep fixed, optimize over Step 2: keep fixed, optimize over

  • Normal prior
  • M3N problem (QP)
  • Laplace prior
  • Laplace M3N problem (VB)

Equivalently reduced to an LP with a polynomial number of constraints

slide-12
SLIDE 12

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

23 23

Record-Level Evaluations

Overall performance:

Avg F1:

  • avg F1 over all attributes

Block instance accuracy:

  • % of records whose Name,

Image, and Price are correct

Attribute performance:

8/6/2009 VLPR 2009 @ Beijing, China

24 24

VI: Max-Margin/Max Entropy Topic Model – MED-LDA

(from images.google.cn)

slide-13
SLIDE 13

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

25 25

LDA: a generative story for documents

25 25

image, jpg, gif, file, color, file, images, files, format ground, wire, power, wiring, current, circuit,

Topic #1 Topic #2 Document #1: gif jpg image current file color images ground power file current format file formats circuit gif images Document #2: wire currents file format ground power image format wire circuit current wiring ground circuit images files… Mixture Weights Bayesian Approach Dirichlet Prior

LDA

Mixture Components

Bag-of-word representation of documents Each word is generated by ONE topic Each document is a random mixture over topics 8/6/2009 VLPR 2009 @ Beijing, China 26 26

LDA: Latent Dirichlet Allocation

26 26

Generative Procedure:

For each document d:

Sample a topic proportion For each word:

– Sample a topic – Sample a word

(Blei et al., 2003)

  • Joint Distribution:
  • Variational Inference with :
  • Minimize the variational bound to estimate parameters and infer the posterior distribution

exact inference intractable!

slide-14
SLIDE 14

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

27 27

Supervised Topic Model (sLDA)

27 27 LDA ignores documents’ side information (e.g., categories or rating

score), thus lead to suboptimal topic representation for supervised tasks

Supervised Topic Models handle such problems, e.g., sLDA (Blei &

McAuliffe, 2007) and DiscLDA (Simon et al., 2008) Generative Procedure (sLDA): For each document d:

Sample a topic proportion For each word:

– Sample a topic – Sample a word

Sample Joint distribution: Variational inference:

(Blei & McAuliffe, 2007)

8/6/2009 VLPR 2009 @ Beijing, China

28 28

The big picture

28 28

Max-Likelihood Estimation Max-Margin and Max- Likelihood sLDA MedLDA

How to integrate the max-margin principle into a

probabilistic latent variable model?

Traditional max-margin learning is not probabilistic But Max-Ent Discrimination (MED) learning solves the problem

slide-15
SLIDE 15

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

29 29

MedLDA Regression Model

29 29

Def:

is a hidden random variable Variational bound

Predictive Rule:

Generative Procedure (Bayesian sLDA): Sample a parameter For each document d:

  • Sample a topic proportion
  • For each word:

–

Sample a topic

–

Sample a word

  • Sample :

predictive accuracy model fitting

intractable

8/6/2009 VLPR 2009 @ Beijing, China

30 30

MedLDA Regression Model

30 30

Def:

  • is a hidden random variable

Variational bound

Predictive Rule:

Generative Procedure (Bayesian sLDA): Sample a parameter For each document d:

  • Sample a topic proportion
  • For each word:

–

Sample a topic

–

Sample a word

  • Sample :
slide-16
SLIDE 16

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

31 31 E-step: infer the posterior distribution of hidden r.v. M-step: estimate unknown parameters

  • Independence assumption:
  • Optimize L over :

— The first two terms are the same as in LDA — The third and fourth terms are similar to those of sLDA, but in expected

  • version. The variance matters!

— The last term is a regularizer. Only support vectors affect the topic proportions

  • Optimize L over other variables. See our paper for details!

Variational EM Alg.

31 31

8/6/2009 VLPR 2009 @ Beijing, China

32 32

MedLDA Classification Model

32 32

Normalization factor in GLM makes inference harder We use LDA as the underlying topic model Multiclass MedLDA Classification Model:

—Variational upper bound : —Expected margin constraints.

Predictive Rule:

predictive accuracy model fitting

slide-17
SLIDE 17

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

33 33

Variational EM Alg.

33 33

Independence assumption: Lagrangian function: Optimize L over

:

Optimize L over other variables. See the paper for

details

LDA MedLDA Only support vectors matter!

8/6/2009 VLPR 2009 @ Beijing, China

34 34

MedTM: a general framework

34 34

MedLDA can be generalized to arbitrary topic models:

– Unsupervised or supervised – Generative or undirected random fields (e.g., Harmoniums)

MED Topic Model (MedTM)：

  • : hidden r.v.s in the underlying topic model, e.g.,

in LDA

  • : parameters in predictive model, e.g., in sLDA
  • : parameters of the topic model, e.g., in LDA
  • : an variational upper bound of the log-likelihood
  • : a convex function over slack variables

model fitting predictive accuracy

slide-18
SLIDE 18

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

35 35

Experiments

35 35

Goal:

To qualitatively and quantitatively evaluate how the max-margin

estimates of MedLDA affect its topic discovering procedure

Data Sets：

20 Newsgroups (classification)

  • Documents from 20 categories
  • ~ 20,000 documents in each group
  • Remove stop word as listed in UMASS Mallet

Movie Review (regression)

  • 5006 documents, and 1.6M words
  • Dictionary: 5000 terms selected by tf-idf
  • Preprocessing to make the response approximately normal (Blei & McAuliffe, 2007)

8/6/2009 VLPR 2009 @ Beijing, China

36 36

Document Modeling

36 36 Data Set: 20 Newsgroups 110 topics + 2D embedding with t-SNE (var der Maaten & Hinton, 2008) MedLDA LDA

slide-19
SLIDE 19

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

37 37

Document Modeling (cont’)

Comp.graphics: comp.graphics politics.mideast

37 37

8/6/2009 VLPR 2009 @ Beijing, China

38 38

Classification

38 38

  • Data Set: 20Newsgroups

–

Binary classification: “alt.atheism” and “talk.religion.misc” (Simon et al., 2008)

–

Multiclass Classification: all the 20 categories

  • Models: DiscLDA, sLDA(Binary ONLY! Classification sLDA (Wang et al., 2009)),

LDA+SVM (baseline), MedLDA, MedLDA+SVM

  • Measure: Relative Improvement Ratio
slide-20
SLIDE 20

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

39 39

Regression

39 39 Data Set: Movie Review (Blei & McAuliffe, 2007) Models: MedLDA(partial), MedLDA(full), sLDA, LDA+SVR Measure: predictive R2 and per-word log-likelihood Sharp decrease in SVs

8/6/2009 VLPR 2009 @ Beijing, China

40 40

Summary

  • A general framework of MaxEnDNet for learning structured

input/output models

– Subsumes the standard M3Ns – Model averaging: PAC-Bayes theoretical error bound – Entropic regularization: sparse M3Ns – Generative + discriminative: latent variables, semi-supervised learning on

partially labeled data

  • Laplace MaxEnDNet: simultaneously primal and dual sparse

– Can perform as well as sparse models on synthetic data – Perform better on real data sets – More stable to regularization constants

  • PoMEN

– Provides an elegant approach to incorporate latent variables and structures

under max-margin framework

– Experimental results show the advantages of max-margin learning over

likelihood methods with latent variables

slide-21
SLIDE 21

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

41 41

Margin-based Learning Paradigms

Structured prediction Structured prediction Bayes learning Bayes learning 8/6/2009 VLPR 2009 @ Beijing, China

42 42

Acknowledgement

http://www.sailing.cs.cmu.edu/

slide-22
SLIDE 22

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

43 43

Thanks!

Reference:

8/6/2009 VLPR 2009 @ Beijing, China

44 44

Markov Chain Prior

slide-23
SLIDE 23

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

45 45

  • cj = cj-1 if

1) the distance between the two SNPs is small, or 2) the recombination rate between the two SNPs is small

Markov Chain Prior

8/6/2009 VLPR 2009 @ Beijing, China

46 46

Poisson process

  • : Recombination rate at jth SNP
  • : Distance between jth and (j-1)th SNP
  • : Transition probability matrix

Markov Chain Prior

slide-24
SLIDE 24

Laplace Max-margin Markov Networks

8/6/2009 VLPR 2009 @ Beijing, China

47 47

Variational Bayesian Learning (Cont’)

8/6/2009 VLPR 2009 @ Beijing, China

48 48

Experimental Results

Web data extraction:

Name, Image, Price, Description Methods:

  • Hierarchical CRFs, Hierarchical

M^3N

  • PoMEN, Partially observed HCRFs

Pages from 37 templates

  • Training: 185 (5/per template)

pages, or 1585 data records

  • Testing: 370 (10/per template)

pages, or 3391 data records Record-level Evaluation

  • Leaf nodes are labeled

Page-level Evaluation

  • Supervision Level 1:
  • Leaf nodes and data record nodes

are labeled

  • Supervision Level 2:
  • Level 1 + the nodes above data

record nodes