Heterogeneous Multi-output Gaussian Process Prediction Pablo - - PowerPoint PPT Presentation

Heterogeneous Multi-output Gaussian Process Prediction

Pablo Moreno-Muñoz Antonio Artés-Rodríguez Mauricio A. Álvarez 1 Poster #14

2

Problem

Motivation

Goal

Output observations are a mix of continuous, binary, categorical or discrete variables Multi-output Gaussian process models usually focus on all-regression or all-classification tasks Provide an extension of multi-output Gaussian processes for prediction in arbitrary heterogeneous datasets

2

Applications

Motivation

Intensive Care Units (ICU) Electronic Health Records Demographic, sociological or economic analysis of cities Functions with multiple outputs from different nature

Spatio-temporal Bayesian Optimization Medical

Flat Other

3 Heterogeneous Likelihoods

p(y|θ(x)) =

D

Y

d=1

p(yd|θd(x))

3 Heterogeneous Likelihoods

p(y|θ(x)) =

D

Y

d=1

p(yd|θd(x))

D

different statistical data types

3 Heterogeneous Likelihoods

p(y|θ(x)) =

D

Y

d=1

p(yd|θd(x))

f ∼ GP(0, k(·, ·))

Ber(y1|θ1(x))

Poisson(y2|θ2(x))

N(y3|θ3(x))

θ(x) = φ(f(x))

Example with D = 3

3 Heterogeneous Likelihoods

p(y|θ(x)) =

D

Y

d=1

p(yd|θd(x))

Ber(y1|θ1(x))

Poisson(y2|θ2(x))

N(y3|θ3(x))

Example with D = 3

θ1(x) = 1/(1 + exp(−f1(x)))

θ2(x) = exp(f2(x))

θ3(x) = f3(x)

Link functions

3 Heterogeneous Likelihoods

p(y|θ(x)) =

D

Y

d=1

p(yd|θd(x))

Example with D = 3

θ1(x) = 1/(1 + exp(−f1(x)))

θ2(x) = exp(f2(x))

θ3(x) = f3(x)

f(x) =

Q

X

q=1

aquq(x)

Linear combination

3 Heterogeneous Likelihoods

p(y|θ(x)) =

D

Y

d=1

p(yd|θd(x))

Example with D = 3

θ1(x) = 1/(1 + exp(−f1(x)))

θ2(x) = exp(f2(x))

θ3(x) = f3(x)

Linear combination

f(x) =

Q

X

q=1

aquq(x)

4 Multi-parameter priors

f1(x) =

Q

X

q=1

a1,quq(x)

f2(x) =

Q

X

q=1

a2,quq(x)

f3(x) =

Q

X

q=1

a3,quq(x)

Multi-output GP prior

uq(x) ∼ GP(0, kq(·, ·))

Independent latent functions

2 4 f1 f2 f3 3 5 ∼ MOGP 0,

Q

X

q=1

Bq ⊗ kq(·, ·) !

5 Scalable inference

with inducing inputs

Z = {zm}M

m=1

uq = [uq(z1), · · · , uq(zM)]>

Sparse MOGP approximation and variational inference

p(f, u|y, X) ≈ q(f, u) = p(f|u)q(u)

5 Scalable inference

with inducing inputs

Z = {zm}M

m=1

uq = [uq(z1), · · · , uq(zM)]>

Sparse MOGP approximation and variational inference

p(f, u|y, X) ≈ q(f, u) = p(f|u)q(u)

5 Scalable inference

with inducing inputs

Z = {zm}M

m=1

uq = [uq(z1), · · · , uq(zM)]>

Sparse MOGP approximation and variational inference

q(f, u) =

D

Y

d=1 Jd

Y

j=1

p(fd,j|u)

Q

Y

q=1

q(uq)

q(uq) = N(uq|µuq, Suq)

where

5 Scalable inference

Variational Lower Bound

L = ZZ p(f|u)q(u) log p(y|f)dfdu −

Q

X

q=1

KL(q(uq)||p(uq))

5 Scalable inference

Variational Lower Bound double expectation factorised divergences

L = ZZ p(f|u)q(u) log p(y|f)dfdu −

Q

X

q=1

KL(q(uq)||p(uq))

5 Scalable inference

Variational Lower Bound amenable for stochastic optimisation

N

X

n=1 D

X

d=1

Eq(fd(xn))[log p(yd|fd(xn))] −

Q

X

q=1

KL(q(uq||p(uq)))

5 Scalable inference

Variational Lower Bound Hyperparameter learning

{µuq}Q

q=1

{Luq}Q

q=1

Suq = LuqL>

uq

where

Variational parameters

Z

{Bq}Q

q=1

{γq}Q

q=1

inducing points linear combination kernel hyperparameters

VEM algorithm

N

X

n=1 D

X

d=1

Eq(fd(xn))[log p(yd|fd(xn))] −

Q

X

q=1

KL(q(uq||p(uq)))

6 Results

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −6 −4 −2 2 4 6

Real Input Real Output Output 1: Gaussian Regression

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1

Real Input Binary Output Output 2: Binary Classification

Multi-task toy experiment

Test prediction improves in the binary classification using training information from the regression task

6 Results

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −6 −4 −2 2 4 6

Real Input Real Output Output 1: Gaussian Regression

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1

Real Input Binary Output Output 2: Binary Classification

Multi-task toy experiment

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1

Real Input Binary Output Single Output: Binary Classification

Uncertainty is reduced in multi-output learning with respect to an independent training

6 Results

London House Price Data

• 0.51
• 0.34
• 0.17
• 0.0

0.16 0.33 51.29 51.37 51.45 51.53 51.61 51.69

Longitude Latitude Property Type Flat Other

• 0.51
• 0.34
• 0.17
• 0.0

0.16 0.33 51.29 51.37 51.45 51.53 51.61 51.69

Longitude Latitude Sale Price

79K£ 167K£ 351K£ 738K£ 1.5M£

Binary classification Heteroscedastic Gaussian Regression Heterogeneous multi-output GP modeling

6 Results

London House Price Data

• 0.51
• 0.34
• 0.17
• 0.0

0.16 0.33 51.29 51.37 51.45 51.53 51.61 51.69

Longitude Latitude Property Type Flat Other

• 0.51
• 0.34
• 0.17
• 0.0

0.16 0.33 51.29 51.37 51.45 51.53 51.61 51.69

Longitude Latitude Sale Price

79K£ 167K£ 351K£ 738K£ 1.5M£

• 0.51
• 0.34
• 0.17
• 0.0

0.16 0.33 51.29 51.37 51.45 51.53 51.61 51.69

Longitude Latitude Log-price Variance

0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4

• 0.51
• 0.34
• 0.17
• 0.0

0.16 0.33 51.29 51.37 51.45 51.53 51.61 51.69

Longitude Latitude Probability of Flat House

0.2 0.4 0.6 0.8 1

test prediction shared modes

Thanks for listening! Code at

github.com/pmorenoz/HetMOGP

Poster #14

See you at (today)

likelihood list = [Bernoulli(), Poisson(), Gaussian()]

likelihood list = [Categorical(), Gamma(), HetGaussian(), Gaussian()]

likelihood list = [Bernoulli(), Bernoulli(), Beta()]

easy syntax for arbitrary heterogeneous likelihoods

Python

7 Summary