http://cs246.stanford.edu ? ? x ? Machine Learning 2/12/20 - - PowerPoint PPT Presentation

β–Ά
http cs246 stanford edu
SMART_READER_LITE
LIVE PREVIEW

http://cs246.stanford.edu ? ? x ? Machine Learning 2/12/20 - - PowerPoint PPT Presentation

Aug 13, 2022 358 likes β€’923 views

Note to other teachers and users of these slides: We would be delighted if you found our material useful for giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. If you make use of a

slide-1
SLIDE 1

CS246: Mining Massive Datasets Jure Leskovec, Stanford University

http://cs246.stanford.edu

Note to other teachers and users of these slides: We would be delighted if you found our material useful for giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. If you make use of a significant portion of these slides in your own lecture, please include this message, or a link to our web site: http://www.mmds.org

slide-2
SLIDE 2

Machine Learning

Jure Leskovec, Stanford C246: Mining Massive Datasets 2

? ? ?

x

2/12/20

slide-3
SLIDE 3

? ? ? ? ?

Machine Learning

Node classification

Jure Leskovec, Stanford C246: Mining Massive Datasets 3 2/12/20

slide-4
SLIDE 4

Jure Leskovec, Stanford C246: Mining Massive Datasets 4

Classifying the function of proteins in the interactome

Image from: Ganapathiraju et al. 2016. Schizophrenia interactome with 504 novel protein–protein interactions. Nature.

2/12/20

slide-5
SLIDE 5

5

Raw Data Structured Data Learning Algorithm Model Downstream task Feature Engineering

Automatically learn the features

Β‘ (Supervised) Machine Learning Lifecycle

requires feature engineering every single time!

Jure Leskovec, Stanford C246: Mining Massive Datasets 2/12/20

slide-6
SLIDE 6

Goal: Efficient task-independent feature learning for machine learning in networks!

Jure Leskovec, Stanford C246: Mining Massive Datasets 6

vec node 𝑔: 𝑣 β†’ ℝ& ℝ&

Feature representation, embedding

u

2/12/20

slide-7
SLIDE 7
  • –

– –

17

Task: We map each node in a network to a point in a low-dimensional space

Β§ Distributed representation for nodes Β§ Similarity of embedding between nodes indicates their network similarity Β§ Encode network information and generate node representation

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 7

slide-8
SLIDE 8

2D embedding of nodes of the Zachary’s Karate Club network:

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 8

  • Zacharys Karate Network:

Image from: Perozzi et al. DeepWalk: Online Learning of Social Representations. KDD 2014.

slide-9
SLIDE 9

Β‘ Modern deep learning toolbox is designed

for simple sequences or grids

Β§ CNNs for fixed-size images/grids…. Β§ RNNs or word2vec for text/sequences…

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 9

slide-10
SLIDE 10

But networks are far more complex!

Β‘ Complex topographical structure (no spatial

locality like grids)

Β‘ No fixed node ordering or reference point Β‘ Often dynamic and have multimodal features.

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 10

vs vs.

Networks ks Im Imag ages es Te Text

slide-11
SLIDE 11
slide-12
SLIDE 12

Assume we have a graph G:

Β‘ V is the vertex set Β‘ A is the adjacency matrix (assume binary) Β‘ No node features or extra information is

used!

Jure Leskovec, Stanford C246: Mining Massive Datasets 12 2/12/20

slide-13
SLIDE 13

Β‘ Goal is to encode nodes so that similarity in

the embedding space (e.g., dot product) approximates similarity in the original network

Jure Leskovec, Stanford C246: Mining Massive Datasets 13 2/12/20

slide-14
SLIDE 14

Jure Leskovec, Stanford C246: Mining Massive Datasets 14

similarity(u, v) β‰ˆ z>

v zu

Go Goal: Ne Need t to d define!

2/12/20

in the original network Similarity of the embedding

slide-15
SLIDE 15

1.

Define an encoder (i.e., a mapping from nodes to embeddings)

2.

Define a node similarity function (i.e., a measure of similarity in the original network)

3.

Optimize the parameters of the encoder so that:

Jure Leskovec, Stanford C246: Mining Massive Datasets 15

similarity(u, v) β‰ˆ z>

v zu

2/12/20

in the original network Similarity of the embedding

slide-16
SLIDE 16

Β‘ Encoder maps each node to a low-

dimensional vector

Β‘ Similarity function specifies how relationships

in vector space map to relationships in the

  • riginal network

Jure Leskovec, Stanford C246: Mining Massive Datasets 16

enc(v) = zv

node in the input graph d-dimensional embedding Similarity of u and v in the original network dot product between node embeddings

similarity(u, v) β‰ˆ z>

v zu

2/12/20

slide-17
SLIDE 17

Β‘ Simplest encoding approach: encoder is just

an embedding-lookup

Jure Leskovec, Stanford C246: Mining Massive Datasets 17

Matrix, each column is 𝑒-dim node embedding [w [what w we l learn!] !] Indicator vector, all zeroes except for a β€œ1” at the position that corresponds to node 𝑀

enc(v) = Zv

Z ∈ RdΓ—|V| v ∈ I|V|

2/12/20

slide-18
SLIDE 18

Β‘ Simplest encoding approach: encoder is just

an embedding-lookup

Jure Leskovec, Stanford C246: Mining Massive Datasets 18

Z =

Dimension/size

  • f embeddings
  • ne column per node

embedding matrix embedding vector for a specific node

2/12/20

slide-19
SLIDE 19

Simplest encoding approach: encoder is just an embedding-lookup Each node is assigned a unique embedding vector Many methods: node2vec, DeepWalk, LINE

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 19

slide-20
SLIDE 20

Key choice of methods is how they define node similarity. E.g., should two nodes have similar embeddings if they…

Β‘ are connected? Β‘ share neighbors? Β‘ have similar β€œstructural roles”? Β‘ …?

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 20

slide-21
SLIDE 21

Material based on:

  • Perozzi et al. 2014. DeepWalk: Online Learning of Social Representations. KDD.
  • Grover et al. 2016. node2vec: Scalable Feature Learning for Networks. KDD.
slide-22
SLIDE 22

Jure Leskovec, Stanford C246: Mining Massive Datasets 22

Probability that 𝑣 and 𝑀 co-occur on a random walk over the network

z>

u zv β‰ˆ

2/12/20

𝑨0 … embedding of node 𝑣

slide-23
SLIDE 23

1.

Estimate probability of visiting node π’˜ on a random walk starting from node 𝒗 using some random walk strategy 𝑺

2.

Optimize embeddings to encode these random walk statistics:

Similarity (here: dot product=cos(πœ„)) encodes random walk β€œsimilarity”

Jure Leskovec, Stanford C246: Mining Massive Datasets 23 2/12/20

𝑨0 𝑨:

slide-24
SLIDE 24

1.

Expressivity: Flexible stochastic definition of node similarity that incorporates both local and higher-

  • rder neighborhood information

2.

Efficiency: Do not need to consider all node pairs when training; only need to consider pairs that co-occur on random walks

Jure Leskovec, Stanford C246: Mining Massive Datasets 24 2/12/20

slide-25
SLIDE 25

Β‘ Intuition: Find embedding of nodes in

𝑒-dimensional space so that node similarity is preserved

Β‘ Idea: Learn node embedding such that nearby

nodes are close together in the network

Β‘ Given a node 𝒗, how do we define nearby

nodes?

Β§ 𝑂< 𝑣 … neighbourhood of 𝑣 obtained by some strategy 𝑆

25 Jure Leskovec, Stanford C246: Mining Massive Datasets 2/12/20

slide-26
SLIDE 26

Β‘ Given 𝐻 = (π‘Š, 𝐹) Β‘ Our goal is to learn a mapping 𝑨: 𝑣 β†’ ℝ& Β‘ Log-likelihood objective:

max

F

G

0 ∈I

log P(𝑂M(𝑣)| 𝑨0)

Β§ where 𝑂<(𝑣) is neighborhood of node 𝑣

Β‘ Given node 𝑣, we want to learn feature

representations predictive of nodes in its neighborhood 𝑂M(𝑣)

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 26

slide-27
SLIDE 27

1.

Run short fixed-length random walks starting from each node on the graph using some strategy R

2.

For each node 𝑣 collect 𝑂<(𝑣), the multiset*

  • f nodes visited on random walks starting

from u

3.

Optimize embeddings according to: Given node 𝑣, predict its neighbors 𝑂M(𝑣) max

F

G

0 ∈I

log P(𝑂M(𝑣)| 𝑨0)

Jure Leskovec, Stanford C246: Mining Massive Datasets 27

*𝑂<(𝑣) can have repeat elements since nodes can be visited multiple times on random walks

2/12/20

slide-28
SLIDE 28

max

F

G

0 ∈I

log P(𝑂M(𝑣)| 𝑨0)

Β‘ Assumption: Conditional likelihood factorizes

  • ver the set of neighbors:

log P(𝑂M(𝑣)|𝑨0) = G

:∈PQ(0)

log P(z: | 𝑨0)

Β‘ Softmax parametrization:

P z: 𝑨0) =

STU(VWβ‹…FY) βˆ‘[∈\ STU(V]β‹…FY)

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 28

Why softmax? We want node 𝑀 to be most similar to node 𝑣 (out of all nodes π‘œ). Intuition: βˆ‘_ exp 𝑦_ β‰ˆ max

_

exp(𝑦_)

slide-29
SLIDE 29

Jure Leskovec, Stanford C246: Mining Massive Datasets 30

Putting it all together:

sum over all nodes 𝑣 sum over nodes 𝑀 seen on random walks starting from 𝑣 predicted probability of 𝑀 appearing in random walk starting from 𝑣

Optimizing random walk embeddings = Finding node embeddings π’œ that minimize L

L = X

u2V

X

v2NR(u)

βˆ’ log βœ“ exp(z>

u zv)

P

n2V exp(z> u zn)

β—†

2/12/20

slide-30
SLIDE 30

Jure Leskovec, Stanford C246: Mining Massive Datasets 31

But doing this naively is too expensive!!

Nested sum over nodes gives O(|V|2) complexity!

L = X

u2V

X

v2NR(u)

βˆ’ log βœ“ exp(z>

u zv)

P

n2V exp(z> u zn)

β—†

2/12/20

slide-31
SLIDE 31

Jure Leskovec, Stanford C246: Mining Massive Datasets 32

But doing this naively is too expensive!! The normalization term from the softmax is the culprit… can we approximate it?

L = X

u2V

X

v2NR(u)

βˆ’ log βœ“ exp(z>

u zv)

P

n2V exp(z> u zn)

β—†

2/12/20

slide-32
SLIDE 32

Jure Leskovec, Stanford C246: Mining Massive Datasets 33

sigmoid function

(makes each term a β€œprobability” between 0 and 1)

random distribution over all nodes

log βœ“ exp(z>

u zv)

P

n2V exp(z> u zn)

β—† β‰ˆ log(Οƒ(z>

u zv)) βˆ’ k

X

i=1

log(Οƒ(z>

u zni)), ni ∼ PV

2/12/20

Β‘ Solution: Negative sampling

Instead of normalizing w.r.t. all nodes, just normalize against 𝑙 random β€œnegative samples” π‘œ_

Why is the approximation valid? Technically, this is a different objective. But Negative Sampling is a form of Noise Contrastive Estimation (NCE) which approx. maximizes the log probability of softmax. New formulation corresponds to using a logistic regression (sigmoid func.) to distinguish the target node 𝑀 from nodes π‘œ_ sampled from background distribution 𝑄

:.

More at https://arxiv.org/pdf/1402.3722.pdf

slide-33
SLIDE 33

log βœ“ exp(z>

u zv)

P

n2V exp(z> u zn)

β—† β‰ˆ log(Οƒ(z>

u zv)) βˆ’ k

X

i=1

log(Οƒ(z>

u zni)), ni ∼ PV

Jure Leskovec, Stanford C246: Mining Massive Datasets 34

random distribution

  • ver all nodes

Β§ Sample 𝑙 negative nodes proportional to degree Β§ Two considerations for 𝑙 (# negative samples):

  • 1. Higher 𝑙 gives more robust estimates
  • 2. Higher 𝑙 corresponds to higher prior on negative events

In practice 𝑙 =5-20

2/12/20

slide-34
SLIDE 34

1.

Run short fixed-length random walks starting from each node on the graph using some strategy R.

2.

For each node u collect NR(u), the multiset of nodes visited on random walks starting from u

3.

Optimize embeddings using Stochastic Gradient Descent:

Jure Leskovec, Stanford C246: Mining Massive Datasets 35

We We can efficiently approximate this using ne negative sampling ng!

L = X

u∈V

X

v∈NR(u)

βˆ’ log(P(v|zu))

2/12/20

slide-35
SLIDE 35

Β‘ So far we have described how to optimize

embeddings given random walk statistics

Β‘ What strategies should we use to run these

random walks?

Β§ Simplest idea: Just run fixed-length, unbiased random walks starting from each node (i.e., DeepWalk from Perozzi et al., 2013).

Β§ The issue is that such notion of similarity is too constrained

Β§ How can we generalize this?

Jure Leskovec, Stanford C246: Mining Massive Datasets 36 2/12/20

slide-36
SLIDE 36

Β‘ Goal: Embed nodes with similar network

neighborhoods close in the feature space

Β‘ We frame this goal as prediction-task independent

maximum likelihood optimization problem

Β‘ Key observation: Flexible notion of network

neighborhood 𝑂<(𝑣) of node 𝑣 leads to rich node embeddings

Β‘ Develop biased 2nd order random walk 𝑆 to

generate network neighborhood 𝑂<(𝑣) of node 𝑣

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 37

slide-37
SLIDE 37

Idea: use flexible, biased random walks that can trade off between local and global views of the network (Grover and Leskovec, 2016).

Jure Leskovec, Stanford C246: Mining Massive Datasets 38

u s3 s2

s1

s4 s8 s9 s6 s7 s5

BFS DFS

2/12/20

slide-38
SLIDE 38

Two classic strategies to define a neighborhood 𝑂< 𝑣 of a given node 𝑣: Walk of length 3 (𝑂< 𝑣 of size 3):

Jure Leskovec, Stanford C246: Mining Massive Datasets 39

𝑂klm 𝑣 = { 𝑑p, 𝑑q, 𝑑r} 𝑂tlm 𝑣 = { 𝑑u, 𝑑v, 𝑑w} Local microscopic view Global macroscopic view

u s3 s2

s1

s4 s8 s9 s6 s7 s5

BFS DFS

2/12/20

slide-39
SLIDE 39

Biased fixed-length random walk 𝑺 that given a node 𝒗 generates neighborhood 𝑢𝑺 𝒗

Β‘ Two parameters:

Β§ Return parameter 𝒒:

Β§ Return back to the previous node

Β§ In-out parameter 𝒓:

Β§ Moving outwards (DFS) vs. spreading (BFS) Β§ Intuitively, π‘Ÿ is the β€œratio” of BFS vs. DFS

Jure Leskovec, Stanford C246: Mining Massive Datasets 40 2/12/20

slide-40
SLIDE 40

Biased 2nd-order random walks explore network neighborhoods:

Β§ Rnd. walk just traversed edge (𝑑p, π‘₯) and is now at π‘₯ Β§ Insight: Neighbors of π‘₯ can only be: Idea: Remember where that walk came from

Jure Leskovec, Stanford C246: Mining Massive Datasets 41

s1 s2 w s3 u

Back k to π’•πŸ Sam Same e distan ance ce to π’•πŸ Fa Farthe her fr from π’•πŸ

2/12/20

slide-41
SLIDE 41

Β‘ Walker came over edge (sp, w) and is at w.

Where to go next?

Β‘ π‘ž, π‘Ÿ model transition probabilities

Β§ π‘ž … return parameter Β§ π‘Ÿ … β€œwalk away” parameter

Jure Leskovec, Stanford C246: Mining Massive Datasets 42

1 1/π‘Ÿ 1/π‘ž

1/π‘ž, 1/π‘Ÿ, 1 are

unnormalized probabilities

s1 s2 w s3 u

2/12/20

s4

1/π‘Ÿ

slide-42
SLIDE 42

Β‘ Walker came over edge (sp, w) and is at w.

Where to go next?

Β§ BFS-like walk: Low value of π‘ž Β§ DFS-like walk: Low value of π‘Ÿ

𝑂<(𝑣) are the nodes visited by the biased walk

Jure Leskovec, Stanford C246: Mining Massive Datasets 43

w β†’

s1 s2 s3 s4 1/π‘ž 1 1/π‘Ÿ 1/π‘Ÿ

Unnormalized transition prob. segmented based

  • n distance from 𝑑!

2/12/20

  • Dist. (π’•πŸ, 𝒖)

1 2 2 1 1/π‘Ÿ 1/π‘ž

s1 s2 w s3 u s4

1/π‘Ÿ

Target 𝒖 Prob.

slide-43
SLIDE 43

Β‘ 1) Compute random walk probabilities Β‘ 2) Simulate 𝑠 random walks of length π‘š starting

from each node 𝑣

Β‘ 3) Optimize the node2vec objective using

Stochastic Gradient Descent Linear-time complexity. All 3 steps are individually parallelizable

Jure Leskovec, Stanford C246: Mining Massive Datasets 44 2/12/20

slide-44
SLIDE 44

Jure Leskovec, Stanford C246: Mining Massive Datasets 45

BFS: Micro-view of neighbourhood

u

DFS: Macro-view of neighbourhood

2/12/20

slide-45
SLIDE 45

Small network of interactions of characters in a novel:

Jure Leskovec, Stanford C246: Mining Massive Datasets 46

p=1, q=2

Microscopic view of the network neighbourhood

p=1, q=0.5

Macroscopic view of the network neighbourhood

2/12/20

slide-46
SLIDE 46

How does predictive performance change as we

Β‘ randomly remove a fraction of edges (left) Β‘ randomly add a fraction of edges (right)

Jure Leskovec, Stanford C246: Mining Massive Datasets 47

0.0 0.1 0.2 0.3 0.4 0.5 0.6

Fraction of missing edges

0.00 0.05 0.10 0.15 0.20

Macro-F1 score

0.0 0.1 0.2 0.3 0.4 0.5 0.6

Fraction of additional edges

0.00 0.05 0.10 0.15 0.20

Macro-F1 score

Predictive performance Predictive performance

2/12/20

slide-47
SLIDE 47

(not covered in detailed here but for your reference)

Β‘ Different kinds of biased random walks:

Β§ Based on node attributes (Dong et al., 2017). Β§ Based on a learned weights (Abu-El-Haija et al., 2017)

Β‘ Alternative optimization schemes:

Β§ Directly optimize based on 1-hop and 2-hop random walk probabilities (as in LINE from Tang et al. 2015).

Β‘ Network preprocessing techniques:

Β§ Run random walks on modified versions of the original network (e.g., Ribeiro et al. 2017’s struct2vec, Chen et al. 2016’s HARP).

Jure Leskovec, Stanford C246: Mining Massive Datasets 48 2/12/20

slide-48
SLIDE 48

Β‘ How to use embeddings π’œπ’‹ of nodes:

Β§ Clustering/community detection: Cluster nodes/points based on 𝑨_ Β§ Node classification: Predict label 𝑔(𝑨_) of node 𝑗 based on 𝑨_ Β§ Link prediction: Predict edge (𝑗, π‘˜) based on 𝑔(𝑨_, 𝑨

Ε )

Β§ Where we can: concatenate, avg, product, or take a difference between the embeddings:

Β§ Concatenate: 𝑔(𝑨_, 𝑨

Ε )= 𝑕([𝑨_, 𝑨 Ε ])

Β§ Hadamard: 𝑔(𝑨_, 𝑨

Ε )= 𝑕(𝑨_ βˆ— 𝑨 Ε ) (per coordinate product)

Β§ Sum/Avg: 𝑔(𝑨_, 𝑨

Ε )= 𝑕(𝑨_ + 𝑨 Ε )

Β§ Distance: 𝑔(𝑨_, 𝑨

Ε )= 𝑕(||𝑨_ βˆ’ 𝑨 Ε ||q)

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 49

slide-49
SLIDE 49

Β‘ Basic idea: Embed nodes so that similarities

in embedding space reflect node similarities in the original network.

Β‘ Different notions of node similarity:

Β§ Adjacency-based (i.e., similar if connected) Β§ Multi-hop similarity definitions. Β§ Random walk approaches (covered today)

Jure Leskovec, Stanford C246: Mining Massive Datasets 50 2/12/20

slide-50
SLIDE 50

Β‘ So what method should I use..? Β‘ No one method wins in all cases….

Β§ E.g., node2vec performs better on node classification while multi-hop methods perform better on link prediction (Goyal and Ferrara, 2017 survey)

Β‘ Random walk approaches are generally more

efficient

Β‘ In general: Must choose def’n of node

similarity that matches your application!

Jure Leskovec, Stanford C246: Mining Massive Datasets 51 2/12/20

slide-51
SLIDE 51
slide-52
SLIDE 52

Β‘ Tasks:

Β§ Classifying toxic vs. non-toxic molecules Β§ Identifying cancerogenic molecules Β§ Graph anomaly detection Β§ Classifying social networks

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 53

slide-53
SLIDE 53

‘ Goal: Want to embed an entire graph 𝐻

2/12/20 Jure Leskovec, Stanford C246: Mining Massive Datasets 54

π’œβ€˜

slide-54
SLIDE 54

Simple idea:

Β‘ Run a standard graph embedding

technique on the (sub)graph 𝐻

Β‘ Then just sum (or average) the node

embeddings in the (sub)graph 𝐻

Β‘ Used by Duvenaud et al., 2016 to classify

molecules based on their graph structure

Jure Leskovec, Stanford C246: Mining Massive Datasets 55

π‘¨β€˜ = G

:βˆˆβ€˜

𝑨:

2/12/20

slide-55
SLIDE 55

Β‘ Idea: Introduce a β€œvirtual node” to represent

the (sub)graph and run a standard graph embedding technique

Β‘ Proposed by Li et al., 2016 as a general

technique for subgraph embedding

Jure Leskovec, Stanford C246: Mining Massive Datasets 56 2/12/20