[PPT] - A class of Laplacian multi-wavelets bases for high-dimensional data PowerPoint Presentation

SLIDE 1

MULTIVARIATE APPROXIMATION AND INTERPOLATION WITH APPLICATIONS 2013

A class of Laplacian multi-wavelets bases for high-dimensional data

Nir Sharon

Tel-Aviv University Joint work with Yoel Shkolnisky A part of PhD thesis under the supervision of Yoel Shkolnisky and Nira Dyn

September 26, 2013

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 1 / 24

SLIDE 2

Representing signals

1 2 3 4 5 6 7 −4 −3 −2 −1 1 2 3 4

1D signals – Fourier basis, wavelets, polynomials,. . .

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 2 / 24

SLIDE 3

Representing signals

1 2 3 4 5 6 7 −4 −3 −2 −1 1 2 3 4

1D signals – Fourier basis, wavelets, polynomials,. . . What to do in higher dimensions?

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 2 / 24

SLIDE 4

Representing signals

1 2 3 4 5 6 7 −4 −3 −2 −1 1 2 3 4

1D signals – Fourier basis, wavelets, polynomials,. . . What to do in higher dimensions? What to do for general data - images, documents, gene arrays, . . . ?

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 2 / 24

SLIDE 5

What is general data?

Let X = {xi}N

i=1, xi ∈ RD, be a set of N points with two requirements:

A B C E F D

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 3 / 24

SLIDE 6

What is general data?

Let X = {xi}N

i=1, xi ∈ RD, be a set of N points with two requirements:

1 The set X is associated with a kernel function K : RD × RD → R+,

and with the graph structure induced by K.

A B C E F D

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 3 / 24

SLIDE 7

What is general data?

Let X = {xi}N

i=1, xi ∈ RD, be a set of N points with two requirements:

1 The set X is associated with a kernel function K : RD × RD → R+,

and with the graph structure induced by K.

2 X has an associated tree structure – analog of a dydic partition.

A B C E F D

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 3 / 24

SLIDE 8

The goal

Find N functions {φn}N

n=1,

φn : X → R, such that φn, φm = δn,m. We use f , g =

x∈X

f (x)g(x), ∀f , g : X → R.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 4 / 24

SLIDE 9

The goal

Find N functions {φn}N

n=1,

φn : X → R, such that φn, φm = δn,m. We use f , g =

x∈X

f (x)g(x), ∀f , g : X → R. Further requirements

◮ The construction must be applicable in cases where D (the dimension

f each point in X) is very large.

◮ It should allow for a sparse representation of a large family of functions. ◮ It must have a fast and numerically stable algorithm. Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 4 / 24

SLIDE 10

Known solutions

Two known solutions for general data

◮ Haar basis – Piecewise constant functions ◮ Fourier basis – Eigenvectors of the (graph) Laplacian

Haar basis Fourier basis

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −0.4 −0.3 −0.2 −0.1 0.1 0.2 0.3 0.4 The eigenvectors of the graph Laplacian

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 5 / 24

SLIDE 11

Haar basis – general data

Haar-like on graphs (Gavish, Nadler, and Coifman)

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 6 / 24

SLIDE 12

Haar basis – general data

Haar-like on graphs (Gavish, Nadler, and Coifman) Pros

◮ Simple, fast. ◮ Applicable to high dimensional data.

Cons

◮ Poor representations of smooth functions. Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 6 / 24

SLIDE 13

Fourier basis

Eigenfunctions of the Laplacian, e.g., ϕ′′ = −λϕ.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 7 / 24

SLIDE 14

Fourier basis

Eigenfunctions of the Laplacian, e.g., ϕ′′ = −λϕ. How to generalize? – Eigenvectors of the “graph Laplacian”.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 7 / 24

SLIDE 15

Fourier basis

Eigenfunctions of the Laplacian, e.g., ϕ′′ = −λϕ. How to generalize? – Eigenvectors of the “graph Laplacian”. The graph Laplacian, in a nutshell:

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 7 / 24

SLIDE 16

Fourier basis

Eigenfunctions of the Laplacian, e.g., ϕ′′ = −λϕ. How to generalize? – Eigenvectors of the “graph Laplacian”. The graph Laplacian, in a nutshell:

1

For any set of points (in RD, on a manifold,. . . ), use kernel K to construct a graph Wi,j = K

xi − xj2/2ε
.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 7 / 24

SLIDE 17

Fourier basis

Eigenfunctions of the Laplacian, e.g., ϕ′′ = −λϕ. How to generalize? – Eigenvectors of the “graph Laplacian”. The graph Laplacian, in a nutshell:

1

For any set of points (in RD, on a manifold,. . . ), use kernel K to construct a graph Wi,j = K

xi − xj2/2ε
.

2

Normalize, e.g., L = I − B−1W , Bi,i =

N

j=1

Wi,j.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 7 / 24

SLIDE 18

Fourier basis

Eigenfunctions of the Laplacian, e.g., ϕ′′ = −λϕ. How to generalize? – Eigenvectors of the “graph Laplacian”. The graph Laplacian, in a nutshell:

1

For any set of points (in RD, on a manifold,. . . ), use kernel K to construct a graph Wi,j = K

xi − xj2/2ε
.

2

Normalize, e.g., L = I − B−1W , Bi,i =

N

j=1

Wi,j.

3

Compute the eigenvectors.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 7 / 24

SLIDE 19

Graph Laplacian basis

Graph Laplacian’s eigenvectors on meshes (Gabriel Peyr´ e) Pros

◮ Efficient representation for smooth functions. ◮ Applicable to high dimensional (almost arbitrary) data.

Cons

◮ Poor representation of non-smooth functions/rapidly changing

functions.

◮ Global basis functions. Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 8 / 24

SLIDE 20

Let’s construct a new family of bases

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 9 / 24

SLIDE 21

Let’s construct a new family of bases

Orthogonal

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 9 / 24

SLIDE 22

Let’s construct a new family of bases

Orthogonal Multi-scale – basis elements of varying support.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 9 / 24

SLIDE 23

Let’s construct a new family of bases

Orthogonal Multi-scale – basis elements of varying support. A family of bases parameterized by k – controls the localization of the basis elements. Extreme cases

◮ k = 1

= ⇒ Haar basis

◮ k = N =

⇒ Fourier basis

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 9 / 24

SLIDE 24

Let’s construct a new family of bases

Orthogonal Multi-scale – basis elements of varying support. A family of bases parameterized by k – controls the localization of the basis elements. Extreme cases

◮ k = 1

= ⇒ Haar basis

◮ k = N =

⇒ Fourier basis

Stable O(k2N log N + T(N, k) log(N)) algorithm, where T(N, k) is the complexity of computing k top eigenvectors. Usually N ≫ k.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 9 / 24

SLIDE 25

Let’s construct a new family of bases

Orthogonal Multi-scale – basis elements of varying support. A family of bases parameterized by k – controls the localization of the basis elements. Extreme cases

◮ k = 1

= ⇒ Haar basis

◮ k = N =

⇒ Fourier basis

Stable O(k2N log N + T(N, k) log(N)) algorithm, where T(N, k) is the complexity of computing k top eigenvectors. Usually N ≫ k. Building blocks: graph Laplacian and multi-resolution analysis.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 9 / 24

SLIDE 26

Construction overview

Two phases:

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 10 / 24

SLIDE 27

Construction overview

Two phases:

1 Define the vectors which span the approximation spaces

V0 ⊂ V1 ⊂ · · · ⊂ Vj, where Vj = RN, with N the number of data points.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 10 / 24

SLIDE 28

Construction overview

Two phases:

1 Define the vectors which span the approximation spaces

V0 ⊂ V1 ⊂ · · · ⊂ Vj, where Vj = RN, with N the number of data points.

2 Apply a fast orthogonalization process to obtain

Vj = V0 ⊕ W0 ⊕ W1 ⊕ · · · ⊕ Wj−1, with Wp⊥Vp, Wp ⊕ Vp = Vp+1 for p ≥ 0.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 10 / 24

SLIDE 29

Phase one – V0 ⊂ V1 ⊂ · · · ⊂ Vj,

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 11 / 24

SLIDE 30

Phase one – V0 ⊂ V1 ⊂ · · · ⊂ Vj,

Define the first approximation space V0 as V0 = span {first k eigenvectors of the (global) graph Laplacian}

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 11 / 24

SLIDE 31

Phase one – V0 ⊂ V1 ⊂ · · · ⊂ Vj,

Define the first approximation space V0 as V0 = span {first k eigenvectors of the (global) graph Laplacian} To construct Vj, j ≥ 1 we use

1

Restriction operator on Vj−1.

2

Local graph Laplacian and its first eigenvectors.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 11 / 24

SLIDE 32

Phase one – V0 ⊂ V1 ⊂ · · · ⊂ Vj,

Define the first approximation space V0 as V0 = span {first k eigenvectors of the (global) graph Laplacian} To construct Vj, j ≥ 1 we use

1

Restriction operator on Vj−1.

2

Local graph Laplacian and its first eigenvectors.

This construction is repeated until Vj satisfies dim(Vj) = N.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 11 / 24

SLIDE 33

Approximation spaces – an example

Constructing V1 = V1,0 + V1,1 Restriction Local eigenvectors

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 12 / 24

SLIDE 34

Approximation spaces – an example

Constructing V1 = V1,0 + V1,1 Restriction Local eigenvectors

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 12 / 24

SLIDE 35

A few additional remarks

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 13 / 24

SLIDE 36

A few additional remarks

1 Restriction operator and the tree ensure that the total number of

nonzeros in each level is kN (sparsity).

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 13 / 24

SLIDE 37

A few additional remarks

1 Restriction operator and the tree ensure that the total number of

nonzeros in each level is kN (sparsity).

2 Balanced tree means O(log N) levels (or tree depth). Therefore, we

can “pack” the nested spaces in a sparse matrix of O(kN log(N)) nonzeros.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 13 / 24

SLIDE 38

A few additional remarks

1 Restriction operator and the tree ensure that the total number of

nonzeros in each level is kN (sparsity).

2 Balanced tree means O(log N) levels (or tree depth). Therefore, we

can “pack” the nested spaces in a sparse matrix of O(kN log(N)) nonzeros.

3 We do not assume the tree is binary nor a complete tree. Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 13 / 24

SLIDE 39

Phase two

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 14 / 24

SLIDE 40

Phase two

1 Recall that

Vj ⊂ VJ, VJ ⊥ WJ = ⇒ Vj ⊥ WJ, 0 ≤ j < J.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 14 / 24

SLIDE 41

Phase two

1 Recall that

Vj ⊂ VJ, VJ ⊥ WJ = ⇒ Vj ⊥ WJ, 0 ≤ j < J.

2 Due to sparsity, every complement space Wj is calculated with

O(k2N) operations.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 14 / 24

SLIDE 42

Phase two

1 Recall that

Vj ⊂ VJ, VJ ⊥ WJ = ⇒ Vj ⊥ WJ, 0 ≤ j < J.

2 Due to sparsity, every complement space Wj is calculated with

O(k2N) operations.

3 Overall complexity for this phase is O(k2N log N). Usually N ≫ k. Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 14 / 24

SLIDE 43

Representing functions (synthetic data)

The 1D case: taking 128 equally spaced on [0, 1]. Compare the Haar (k = 1), Laplacian (k = N), and an intermediate case (1 < k < N)

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 15 / 24

SLIDE 44

Representing functions (synthetic data)

The function:

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

sin(4x)

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 15 / 24

SLIDE 45

Representing functions (synthetic data)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

sin(4x)

20 40 60 80 100 120 0.5 1 1.5 2 2.5

Number of used coefficients Error (L2 norm)

k=8 Haar (k=1) Laplacian (k=N)

L2 approximation error

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 15 / 24

SLIDE 46

Representing 1D functions – oscillatory function

The function:

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

sin

1

0.01+2x

Nir Sharon (Tel-Aviv University)

Laplacian multi-wavelets, MAIA 13 September 26, 2013 16 / 24

SLIDE 47

Representing 1D functions – oscillatory function

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

sin

1

0.01+2x

20

40 60 80 100 120 0.5 1 1.5 2 2.5 3 3.5

Number of used coefficients Error (L2 norm)

k=8 Haar (k=1) Laplacian (k=N)

L2 approximation error

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 16 / 24

SLIDE 48

Representing 1D functions – oscillatory function (cont.)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

sin

1

0.01+2x

20

40 60 80 100 120 0.5 1 1.5 2 2.5 3 3.5 Number of used coefficients Error (L2 norm)

k=8 k=16 k=32

L2 approximation error

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 17 / 24

SLIDE 49

Representing 1D functions – piecewise smooth function

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

sign(x − 1

2) sin(4x)

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 18 / 24

SLIDE 50

Representing 1D functions – piecewise smooth function

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

sign(x − 1

2) sin(4x)

20 40 60 80 100 120 0.5 1 1.5 2 2.5 3

Number of used coefficients Error (L2 norm)

k=8 Haar (k=1) Laplacian (k=N)

L2 approximation error

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 18 / 24

SLIDE 51

A smooth function on S2 ⊂ R3

A 3D case - 1000 data points distributed on the sphere. Compare between k = 1, 10, 50, 1000.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 19 / 24

SLIDE 52

A smooth function on S2 ⊂ R3

−0.5 0.5 −0.5 0.5 −0.5 0.5 1 1.1 1.2 1.3 1.4 1.5 1.6

(a) C(x) = cos(2x)

100 200 300 400 500 600 700 800 900 0.05 0.1 Number of used coefficients Relative L2 Error

Haar (k=1) k=10 k=50 Laplacian (k=N)

(b) L2 relative error

Figure: Representing a smooth function on the sphere.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 19 / 24

SLIDE 53

A rapidly changing function on S2 ⊂ R3

−0.5 0.5 −0.5 0.5 −0.5 0.5 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8

(a) R(x) = sin

(xTx0 + 0.2)−1

100 200 300 400 500 600 700 800 900 1000 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Number of used coefficients Relative L2 Error

Haar (k=1) k=10 k=50 Laplacian (k=N)

(b) L2 relative error

Figure: R oscillates rapidly in regions on the sphere where x close to be

rthogonal to x0.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 20 / 24

SLIDE 54

Compression of hyper spectral images

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 21 / 24

SLIDE 55

Compression of hyper spectral images

Remote-sensing platforms are often comprised of a cluster of different spectral range detectors or sensors to benefit from the spectral identification capabilities of each range.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 21 / 24

SLIDE 56

Compression of hyper spectral images

Remote-sensing platforms are often comprised of a cluster of different spectral range detectors or sensors to benefit from the spectral identification capabilities of each range. In this example, hyperspectral image of visible spectral region:

Figure: The 12 different wave length images given as the data.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 21 / 24

SLIDE 57

Compression of surface temperature

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 22 / 24

SLIDE 58

Compression of surface temperature

The surface temperature is derived form sensors in non-visible spectral region - long-wave infrared sensors. Measured in Kelvin.

20 40 60 80 100 120 140 160 180 200 20 40 60 80 100 120 140 160 180 200 280 290 300 310 320 330

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 22 / 24

SLIDE 59

Compression of surface temperature

The surface temperature is derived form sensors in non-visible spectral region - long-wave infrared sensors. Measured in Kelvin.

20 40 60 80 100 120 140 160 180 200 20 40 60 80 100 120 140 160 180 200 280 290 300 310 320 330

We compare the compression with two (non-adaptive) benchmarks: DCT and JPEG2000.

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 22 / 24

SLIDE 60

Compression results

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 23 / 24

SLIDE 61

Compression results

Using 200 coefficients, that is 0.5%:

50 100 150 200 20 40 60 80 100 120 140 160 180 200 280 290 300 310 320 330

(a) LMW

50 100 150 200 20 40 60 80 100 120 140 160 180 200 280 290 300 310 320 330

(b) DCT

50 100 150 200 20 40 60 80 100 120 140 160 180 200 80 100 120 140 160 180

(c) JPEG2000

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 23 / 24

SLIDE 62

Thank you !

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 24 / 24

SLIDE 63

Questions ?

Nir Sharon (Tel-Aviv University) Laplacian multi-wavelets, MAIA 13 September 26, 2013 24 / 24