Bipartite Edge Prediction via Transductive Learning over Product - - PowerPoint PPT Presentation
Bipartite Edge Prediction via Transductive Learning over Product Graphs Bipartite Edge Prediction via Transductive Learning over Product Graphs Hanxiao Liu, Yiming Yang School of Computer Science, Carnegie Mellon University July 8, 2015 ICML
Bipartite Edge Prediction via Transductive Learning over Product Graphs
Hanxiao Liu, Yiming Yang
School of Computer Science, Carnegie Mellon University
July 8, 2015
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 1
Bipartite Edge Prediction via Transductive Learning over Product Graphs Problem Description
1 Problem Description 2 The Proposed Framework 3 Formulation
Product Graph Construction Graph-based Transductive Learning
4 Optimization 5 Experiment 6 Conclusion
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 2
Bipartite Edge Prediction via Transductive Learning over Product Graphs Problem Description
Many applications involve predicting the edges of a bipartite graph.
I II A B C ? ? ? ?
+5
1 Recommender System 2 Host-Pathogen Interaction 3 Question-Answering Mapping 4 Citation Network . . .
Bipartite Edge Prediction via Transductive Learning over Product Graphs Problem Description
Many applications involve predicting the edges of a bipartite graph.
I II A B C ? ? ? ?
+5
Graph G Graph H 1 Recommender System 2 Host-Pathogen Interaction 3 Question-Answering Mapping 4 Citation Network . . .
Sometimes, vertex sets on both sides are intrinsically structured.
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 4
Bipartite Edge Prediction via Transductive Learning over Product Graphs Problem Description
Many applications involve predicting the edges of a bipartite graph.
I II A B C ? ? ? ?
+5
Graph G Graph H 1 Recommender System 2 Host-Pathogen Interaction 3 Question-Answering Mapping 4 Citation Network . . .
Sometimes, vertex sets on both sides are intrinsically structured. Heterogeneous info: G + H + partial observations
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 5
Bipartite Edge Prediction via Transductive Learning over Product Graphs Problem Description
Many applications involve predicting the edges of a bipartite graph.
I II A B C ? ? ? ?
+5
Graph G Graph H 1 Recommender System 2 Host-Pathogen Interaction 3 Question-Answering Mapping 4 Citation Network . . .
Sometimes, vertex sets on both sides are intrinsically structured. Heterogeneous info: G + H + partial observations Combine them to make better edge predictions?
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 6
Bipartite Edge Prediction via Transductive Learning over Product Graphs The Proposed Framework
I II A B C ? ? ? ?
+5
Graph G Graph H
Transductive learning should be effective
1 Labeled edges (red) are highly sparse 2 Unlabeled edges (gray) are massively available
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 7
Bipartite Edge Prediction via Transductive Learning over Product Graphs The Proposed Framework
I II A B C ? ? ? ?
+5
Graph G Graph H
Transductive learning should be effective
1 Labeled edges (red) are highly sparse 2 Unlabeled edges (gray) are massively available
Assumption: similar edges should have similar labels
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 8
Bipartite Edge Prediction via Transductive Learning over Product Graphs The Proposed Framework
I II A B C ? ? ? ?
+5
Graph G Graph H
Transductive learning should be effective
1 Labeled edges (red) are highly sparse 2 Unlabeled edges (gray) are massively available
Assumption: similar edges should have similar labels Prerequisite: a similarity measure among the edges, i.e. a “Graph of Edges” (not directly provided)
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 9
Bipartite Edge Prediction via Transductive Learning over Product Graphs The Proposed Framework
I II A B C ? ? ? ?
+5
Graph G Graph H
Transductive learning should be effective
1 Labeled edges (red) are highly sparse 2 Unlabeled edges (gray) are massively available
Assumption: similar edges should have similar labels Prerequisite: a similarity measure among the edges, i.e. a “Graph of Edges” (not directly provided) Can be induced from G and H via Graph Product!
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 10
Bipartite Edge Prediction via Transductive Learning over Product Graphs The Proposed Framework
The “Graph of Edges” can be induced by taking the product of G and H In the product graph G ◦ H
Each Vertex ∼ edge (in the original bipartite graph) Each Edge ∼ edge-edge similarity
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 11
Bipartite Edge Prediction via Transductive Learning over Product Graphs The Proposed Framework
The “Graph of Edges” can be induced by taking the product of G and H In the product graph G ◦ H
Each Vertex ∼ edge (in the original bipartite graph) Each Edge ∼ edge-edge similarity
The adjacency matrix of the product graph is defined by “◦” (to be discussed later).
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 12
Bipartite Edge Prediction via Transductive Learning over Product Graphs The Proposed Framework
Problem Mapping Edge Prediction (Original Problem) Given G, H and labeled edges, predict the unlabeled edges
I II A B C ? ? ? ?
+5
Vertex Prediction (Equivalent Problem) Given G◦H and labeled vertices, predict the unlabeled vertices
(I, C) ? (I, A)
(I, B) ? (II, C) ? (II, A) ? (II, B) +5
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 13
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation
1 Problem Description 2 The Proposed Framework 3 Formulation
Product Graph Construction Graph-based Transductive Learning
4 Optimization 5 Experiment 6 Conclusion
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 14
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Product Graph Construction
1 Problem Description 2 The Proposed Framework 3 Formulation
Product Graph Construction Graph-based Transductive Learning
4 Optimization 5 Experiment 6 Conclusion
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 15
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Product Graph Construction
Q: When should vertex (i, j) ∼ (i′, j′) in the product graph? Tensor GP i ∼ i′ in G AND j ∼ j′ in H Cartesian GP
OR
Bipartite Edge Prediction via Transductive Learning over Product Graphs 16
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Product Graph Construction
Q: When should vertex (i, j) ∼ (i′, j′) in the product graph? Tensor GP i ∼ i′ in G AND j ∼ j′ in H Cartesian GP
OR
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 17
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Product Graph Construction
Q: When should vertex (i, j) ∼ (i′, j′) in the product graph? Tensor GP i ∼ i′ in G AND j ∼ j′ in H Cartesian GP
OR
To compute the adjacency matrices of PG G ◦T ensor H = G ⊗ H
Kronecker (a.k.a. Tensor) Product
G ◦Cartesian H = G ⊗ I + I ⊗ H = G ⊕ H
Kronecker Sum
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 18
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Product Graph Construction
Both GPs can be written in the form of spectral decomposition G ◦T ensor H =
(λi × µj)(ui ⊗ vj)(ui ⊗ vj)⊤ (1) G ◦Cartesian H =
(λi + µj)(ui ⊗ vj)(ui ⊗ vj)⊤ (2)
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 19
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Product Graph Construction
Both GPs can be written in the form of spectral decomposition G ◦T ensor H =
(λi × µj)
(ui ⊗ vj)(ui ⊗ vj)⊤ (1) G ◦Cartesian H =
(λi + µj)
(ui ⊗ vj)(ui ⊗ vj)⊤ (2) The interplay of graphs is captured by the interplay of their spectrum!
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 20
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Product Graph Construction
Both GPs can be written in the form of spectral decomposition G ◦T ensor H =
(λi × µj)
(ui ⊗ vj)(ui ⊗ vj)⊤ (1) G ◦Cartesian H =
(λi + µj)
(ui ⊗ vj)(ui ⊗ vj)⊤ (2) The interplay of graphs is captured by the interplay of their spectrum! Generalization: Spectral Graph Product G ◦ H
def
=
(λi ◦ µj)(ui ⊗ vj)(ui ⊗ vj)⊤ (3) where “◦” can be arbitrary binary operator (“×”, “+”, . . . )
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 21
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Product Graph Construction
Both GPs can be written in the form of spectral decomposition G ◦T ensor H =
(λi × µj)
(ui ⊗ vj)(ui ⊗ vj)⊤ (1) G ◦Cartesian H =
(λi + µj)
(ui ⊗ vj)(ui ⊗ vj)⊤ (2) The interplay of graphs is captured by the interplay of their spectrum! Generalization: Spectral Graph Product G ◦ H
def
=
(λi ◦ µj)(ui ⊗ vj)(ui ⊗ vj)⊤ (3) where “◦” can be arbitrary binary operator (“×”, “+”, . . . ) Commutative Property: G ◦ H and H ◦ G are isomorphic.
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 22
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Graph-based Transductive Learning
1 Problem Description 2 The Proposed Framework 3 Formulation
Product Graph Construction Graph-based Transductive Learning
4 Optimization 5 Experiment 6 Conclusion
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 23
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Graph-based Transductive Learning
With the product graph A
def
= G ◦ H constructed, we solve a standard graph-based transductive learning problem over A
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 24
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Graph-based Transductive Learning
With the product graph A
def
= G ◦ H constructed, we solve a standard graph-based transductive learning problem over A Learning Objective min
f
ℓ(f)
+ λf ⊤A−1f
(4) fi system-predicted value for vertex i in A ℓ(f) quantifies the gap between f and partially observed labels. λf ⊤A−1f quantifies the smoothness over graph Underlying assumption: f ∼ N (0, A)
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 25
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Graph-based Transductive Learning
The enhanced learning objective min
f
ℓ(f)
+ λf ⊤κ(A)−1f
(5) to incorporate a variety of graph transduction patterns:
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 26
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Graph-based Transductive Learning
The enhanced learning objective min
f
ℓ(f)
+ λf ⊤κ(A)−1f
(5) to incorporate a variety of graph transduction patterns: k-step Random Walk κ(A) = Ak Regularized Laplacian κ(A) = (ǫI − A)−1 = I + A + A2 + A3 + . . . Diffusion Process κ(A) = exp(A) ≡ I + A + 1
2!A2 + 1 3!A3 + · · ·
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 27
Bipartite Edge Prediction via Transductive Learning over Product Graphs Formulation Graph-based Transductive Learning
The enhanced learning objective min
f
ℓ(f)
+ λf ⊤κ(A)−1f
(5) to incorporate a variety of graph transduction patterns: k-step Random Walk κ(A) = Ak Regularized Laplacian κ(A) = (ǫI − A)−1 = I + A + A2 + A3 + . . . Diffusion Process κ(A) = exp(A) ≡ I + A + 1
2!A2 + 1 3!A3 + · · ·
All can be viewed as to transform the spectrum of A :=
i θiuiu⊤ i
Ak =
θk
i uiu⊤ i
(ǫI−A)−1 =
1 ǫ − θi uiu⊤
i
exp(A) =
eθiuiu⊤
i
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 28
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
1 Problem Description 2 The Proposed Framework 3 Formulation
Product Graph Construction Graph-based Transductive Learning
4 Optimization 5 Experiment 6 Conclusion
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 29
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Transductive Learning over Product Graph min
f
ℓ(f) + λ f ⊤κ(A)−1f
(6)
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 30
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Transductive Learning over Product Graph min
f
ℓ(f) + λ f ⊤κ(A)−1f
(6) Challenge: κ(A) = κ( G
) is a huge mn × mn matrix!
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 31
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Transductive Learning over Product Graph min
f
ℓ(f) + λ f ⊤κ(A)−1f
(6) Challenge: κ(A) = κ( G
) is a huge mn × mn matrix! Prohibitive to load it into memory Prohibitive to compute its inverse Even if κ(A)−1 is given, it is expensive to compute ∇r(f) naively
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 32
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Transductive Learning over Product Graph min
f
ℓ(f) + λ f ⊤κ(A)−1f
(6) Challenge: κ(A) = κ( G
) is a huge mn × mn matrix! Prohibitive to load it into memory No need to store κ(A) Prohibitive to compute its inverse Even if κ(A)−1 is given, it is expensive to compute ∇r(f) naively
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 33
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Transductive Learning over Product Graph min
f
ℓ(f) + λ f ⊤κ(A)−1f
(6) Challenge: κ(A) = κ( G
) is a huge mn × mn matrix! Prohibitive to load it into memory No need to store κ(A) Prohibitive to compute its inverse No need of matrix inverse Even if κ(A)−1 is given, it is expensive to compute ∇r(f) naively
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 34
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Transductive Learning over Product Graph min
f
ℓ(f) + λ f ⊤κ(A)−1f
(6) Challenge: κ(A) = κ( G
) is a huge mn × mn matrix! Prohibitive to load it into memory No need to store κ(A) Prohibitive to compute its inverse No need of matrix inverse Even if κ(A)−1 is given, it is expensive to compute ∇r(f) naively Can be performed much more efficiently
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 35
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Keys for complexity reduction
1 Instead of matrices—
κ only manipulates eigenvalues
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 36
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Keys for complexity reduction
1 Instead of matrices—
κ only manipulates eigenvalues
2 The “vec” trick:
Bottleneck: multiplication (X ⊗ Y )f
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 37
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Keys for complexity reduction
1 Instead of matrices—
κ only manipulates eigenvalues
2 The “vec” trick:
Bottleneck: multiplication (X ⊗ Y )f f = vec(F), where Fij
def
= system-predicted score for edge (i, j)
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 38
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Keys for complexity reduction
1 Instead of matrices—
κ only manipulates eigenvalues
2 The “vec” trick:
Bottleneck: multiplication (X ⊗ Y )f f = vec(F), where Fij
def
= system-predicted score for edge (i, j)
(X ⊗ Y )f
= (X ⊗ Y )vec(F) ≡ vec(XFY ⊤)
(7)
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 39
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Further speedup is possible by factorizing F into two low-rank matrices
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 40
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Further speedup is possible by factorizing F into two low-rank matrices The cost of each alternating gradient step is proportional to rank(F) · rank(Σ)
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 41
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Further speedup is possible by factorizing F into two low-rank matrices The cost of each alternating gradient step is proportional to rank(F) · rank(Σ) Σ: a “Characteristic Matrix” where Σij =
1 κ(λi◦µj)
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 42
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Further speedup is possible by factorizing F into two low-rank matrices The cost of each alternating gradient step is proportional to rank(F) · rank(Σ) Σ: a “Characteristic Matrix” where Σij =
1 κ(λi◦µj)
An interesting observation: rank(Σ) is usually a small constant!
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 43
Bipartite Edge Prediction via Transductive Learning over Product Graphs Optimization
Further speedup is possible by factorizing F into two low-rank matrices The cost of each alternating gradient step is proportional to rank(F) · rank(Σ) Σ: a “Characteristic Matrix” where Σij =
1 κ(λi◦µj)
An interesting observation: rank(Σ) is usually a small constant! Example: Diffusion process over the Cartesian PG Σ =
e−(λ1+µ1) . . . e−(λ1+µn) . . . ... . . . e−(λm+µ1) . . . e−(λm+µn)
=
e−λ1 . . . e−λm
. . . e−µn = ⇒ rank(Σ) = 1
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 44
Bipartite Edge Prediction via Transductive Learning over Product Graphs Experiment
1 Problem Description 2 The Proposed Framework 3 Formulation
Product Graph Construction Graph-based Transductive Learning
4 Optimization 5 Experiment 6 Conclusion
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 45
Bipartite Edge Prediction via Transductive Learning over Product Graphs Experiment
Datasets
Dataset G H Movielens-100K Users Movies Cora Publications Publications Courses Courses Prerequisite Courses
Baselines MC Matrix Completion. Ignores the info of G and H. TK Tensor Kernel. Implicitly construct PG, no transduction GRMC Graph Regularized Matrix Completion. Transduction over G and H, no PG constructed
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 46
Bipartite Edge Prediction via Transductive Learning over Product Graphs Experiment
Performance of several interesting combinations of ◦ and κ
Dataset Graph Transduction Graph Product MAP AUC ndcg@3 Courses Random Walk Tensor 0.488 0.827 0.461 Diffusion Cartesian 0.518 0.872 0.500 von-Neumann Tensor 0.472 0.861 0.449 von-Neumann Cartesian 0.366 0.531 0.359 Sigmoid Cartesian 0.443 0.617 0.431 Cora Random Walk Tensor 0.222 0.764 0.205 Diffusion Cartesian 0.256 0.884 0.232 von-Neumann Tensor 0.230 0.853 0.211 von-Neumann Cartesian 0.218 0.633 0.212 Sigmoid Cartesian 0.192 0.443 0.188 MovieLens Random Walk Tensor
Diffusion Cartesian
von-Neumann Tensor
von-Neumann Cartesian
Sigmoid Cartesian
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 47
Bipartite Edge Prediction via Transductive Learning over Product Graphs Experiment
Proposed method (Diff + Cartesian GP) v.s. Baselines
Dataset Method MAP AUC ndcg@3 Courses MC 0.319 0.758 0.294 GRMC 0.366 0.777 0.343 TK 0.449 0.810 0.446 Proposed 0.490 0.838 0.473 Cora MC 0.101 0.697 0.086 GRMC 0.115 0.702 0.101 TK 0.248 0.872 0.231 Proposed 0.268 0.894 0.243 MovieLens MC
GRMC
TK
Proposed
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 48
Bipartite Edge Prediction via Transductive Learning over Product Graphs Conclusion
1 Problem Description 2 The Proposed Framework 3 Formulation
Product Graph Construction Graph-based Transductive Learning
4 Optimization 5 Experiment 6 Conclusion
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 49
Bipartite Edge Prediction via Transductive Learning over Product Graphs Conclusion
Summary Problem Predicting the missing edges of a bipartite graph with graph-structured vertex sets on both sides. Contribution A novel approach via transductive learning over product graph, efficient algorithmic solution and good results.
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 50
Bipartite Edge Prediction via Transductive Learning over Product Graphs Conclusion
Summary Problem Predicting the missing edges of a bipartite graph with graph-structured vertex sets on both sides. Contribution A novel approach via transductive learning over product graph, efficient algorithmic solution and good results. On-going Work Extend to k Graphs (k > 2)
Bipartite Graph → k-partite Graph Edge → Hyperedge
Determine the “optimal” graph product for any given problem.
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 51
Bipartite Edge Prediction via Transductive Learning over Product Graphs Conclusion
hanxiaol@cs.cmu.edu
ICML 2015 Bipartite Edge Prediction via Transductive Learning over Product Graphs 52