SPECTRAL CLUSTERING OF LARGE NETWORKS A. Fender, N. Emad, S. - - PowerPoint PPT Presentation

• A. Fender, N. Emad, S. Petiton, M. Naumov

SPECTRAL CLUSTERING OF LARGE NETWORKS

May 8th, 2017

2

Agenda

Introduction Laplacian Modularity Conclusions

3

THE CLUSTERING PROBLEM

Example : detect relevant groups based on frequent co-purchasing

• n Amazon.com

Visualization: M. Bastian, S. Heymann, and M. Jacomy. “Gephi: An Open Source Software for exploring and manipulating networks” 2009 Data: V. Krebs. 2004

4

THE CLUSTERING PROBLEM

Pink Liberal Yellow Neutral Green Conservative

Data: V. Krebs. 2004 Visualization: M. Bastian, S. Heymann, and M. Jacomy. “Gephi: An Open Source Software for exploring and manipulating networks” 2009

5

CLUSTERING ALGORITHMS

• Spectral

Build a matrix, solve an eigenvalue problem, use eigenvectors for clustering

• Hierarchical / Agglomerative

Build a hierarchy (fine to coarse), partition coarse, propagate results back to fine level

• Local refinements

Switch one node at a time

A x x



fine coarse

6

Laplacian

7

RATIO CUT COST FUNCTION

Objective function: 𝐷𝑝𝑡𝑢 = 𝜖𝐹𝑗 𝑊

𝑗 𝑞 𝑗=1

where 𝜖𝐹𝑗 : # of edges cut and 𝑊

𝑗 : # of nodes in i-th partition

A compromise between small edge-cut and balanced partitions

𝐷𝑝𝑡𝑢 = 2 2 + 2 3 = 5 3

• M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

8

1 1 1 1 1 1 1 1 1 1

GRAPH LAPLACIAN

L = D − A

D: degree matrix A: adjacency matrix

1 3 3 2 1 1 -1

• 1 3 -1 -1
• 1 3 -1 -1
• 1 -1 2
• 1

1

= − 𝑀 𝐸 𝐵

1 2 3 4 5

• M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

9

GRAPH LAPLACIAN

For a vector x with elements that are 0 or 1: Number of edges cut Number of elements

1 2 3 4 5

|𝜖𝐹1| =

1 -1

• 1 3 -1 -1
• 1 3 -1 -1
• 1 -1 2
• 1

1

L

1 1

x

1 1

xT = 2 |𝑊1| = xT x = 2 V1 𝜖𝐹1

• M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

10

MINIMIZATION PROBLEM

min

xi xi TLxi

xi

Txi p i=1

where xi ∈ 0,1 n and xi ⊥ xj min

𝑊i 𝜖𝐹i

𝑊

i p i=1

1 2 3 4 5

V1 𝜖𝐹1 Next step Relax requirements on x, and let xi take real values

• M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

11

K-MEANS POINTS CLUSTERING

Points Centroids Lloyd’s Algorithm:

• Select centroids
• Compute distance of points to centroids
• Assign points to the closest centroid
• Recompute centroid position

p1 p2 pl … c1 ck

• M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016.

12

EDGE CUT MINIMIZATION PIPELINE

Eigensolver Clustering Points Clustering Graph Preprocessing

1.0 1.0 1.0 1.0 1.0

𝑦1

• 1.0
• 0.3

0.3 0.0 1.0

𝑦2

1 1

𝑦1

1 1 1

𝑦2

Laplacian

1

• 1
• 1

3

• 1
• 1
• 1

3

• 1
• 1
• 1
• 1

2

• 1

1

13

Balanced cut minimization Ground truth

SPECTRAL EDGE CUT MINIMIZATION

80% hit rate

14

Modularity

15

Measures the difference between how well vertices are assigned into clusters for the current graph G = (V,E) versus a random graph R = (V,F). G = (V,E) R = (V,F) 𝑅 = 1 2𝜕 (𝑥𝑗𝑘

𝑘

− 𝑤𝑗𝑤𝑘 2𝜕 )

𝑗

𝜀𝑑 𝑗 𝑑(𝑘)

for some assignment c(.) into clusters.

MODULARITY FUNCTION

… … … …

• A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017

16

MODULARITY MATRIX

Let matrix 𝐶 = 𝐵 − 1 2𝜕 𝑤𝑤𝑈 then modularity Q =

1 2𝜕 𝑈𝑠(𝑌𝑈𝐶𝑌)

where Tr(.) is the trace (sum of diagonal elements) and X = [𝑦1, … , 𝑦𝑞] is such that 𝑦𝑗𝑙 = 1 𝑗𝑔 𝑑 𝑗 = 𝑙 .

1 2 3 4 5

1 1 1 1 1 1 1 1 1 1

𝐵

1 3 3 2 1

𝑤

• A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017

17

MODULARITY MAXIMIZATION PIPELINE

Eigensolver Clustering Points Clustering Graph Preprocessing

• 0.5

0.0 0.0 0.6

• 0.5

𝑦4

• 0.6
• 0.4

0.4 0.0 0.6

𝑦5

1 1

𝑦1

1 1 1

𝑦2

Modularity

1 1 1 1 1 1 1 1 1 1

− 1 2𝜕 𝑤𝑤𝑈

18

SPECTRAL MODULARITY MAXIMIZATION

84% hit rate

Spectral Modularity maximization Ground truth

19

PROFILING

The sparse matrix vector multiplication takes 90% of the time in the eigensolver The eigensolver takes 90% of the time

20

MODULARITY VS. LAPLACIAN CLUSTERING

Nvidia Titan X (Pascal), Intel Core i7-3930K @3.2 GHz

Modularity  higher and steadier modularity score  3x speedup over Laplacian Laplacian  homogeneous cluster sizes

21

SPEEDUP AND QUALITY VS. AGGLOMERATIVE*

*: D. LaSalle and G. Karypis. “Multi- threaded Modularity Based Graph Clustering Using the Multilevel Paradigm” Parallel Distrib. Comput., Vol. 76, pp. 66- 80, 2015.

Speedup 3x over agglomerative* scheme Tradeoff Speed vs. quality

0.8s on network with 100 million edges on a single Titan X GPU

Nvidia Titan X (Pascal), Intel Core i7-3930K @3.2 GHz

22

Spectral Clustering in CUDA Toolkit 9.0 release of nvGRAPH

23

CUDA TOOLKIT 9.0

nvGRAPH API

nvgraphStatus_t nvgraphSpectralClustering ( nvgraphHandle_t handle, const nvgraphGraphDescr_t graph_descr, const size_t weight_index, const struct SpectralClusteringParameter *params, int *clustering, void *eig_vals, void *eig_vects); struct SpectralClusteringParameter { int n_clusters; int n_eig_vects; nvgraphSpectralClusteringType_t alg float evs_tolerance int evs_max_iter; float kmean_tolerance; … };

24

Conclusions

25

SPECTRAL CLUSTERING

• Software Framework
• similar for both
• Laplacian - Minimum balanced cut[1]
• Probably the most common metric, with balancing involved in the cost function
• Requires careful choice of eigensolver
• Modularity maximization [2]
• Widely used in analysis of social networks
• Faster to compute

[1] M. Naumov, T. Moon. “Parallel spectral graph partitioning.” Nvidia Technical Report, 2016. [2] A. Fender, N. Emad, S. Petiton, M. Naumov. “Parallel Modularity Clustering.” ICCS, 2017.

26

Thank you

H7129 - Accelerated Libraries Monday and Wednesday @ 4:00, Pod B