The Influence of Down-Sampling Strategies on SVD Word Embedding - - PowerPoint PPT Presentation

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 1

The Influence of Down-Sampling Strategies

• n SVD Word Embedding Stability

Johannes Hellrich, Bernd Kampe & Udo Hahn

Jena University Language & Information Engineering (JULIE) Lab Friedrich Schiller University Jena, Jena, Germany www.julielab.de

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 2

Typical Word Embeddings are Unstable

dog cat tiger

lots

• f

text corpus

dog cat tiger

random embeddings final embeddings random processing

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 3

Typical Word Embeddings are Unstable

dog cat tiger

lots

• f

text corpus

dog cat tiger

random embeddings final embeddings random processing

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 5

Measuring Stability

dog cat tiger dog cat tiger dog cat tiger

lots

• f

text corpus

j@n := 1 |A| X

a∈A

| T

m∈M msw(a, n, m)|

| S

m∈M msw(a, n, m)|

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 7

Why SVD Embeddings?

lots

• f

text corpus

dog cat tiger

final embeddings counting SVD

food roar dog 475 156 cat 823 492 tiger 51 19

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 8

Why SVD Embeddings?

lots

• f

text corpus

dog cat tiger

final embeddings counting & down-sampling SVD

food roar dog 0.02 0.01 cat 0.5 0.4 tiger 0.01 0.19

Replaced with association values in SVDPPMI (Levy et al., TACL 2015)

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 9

Why Down-Sampling?

• Avoids over-representing frequent words
• Closer context words are more salient than distant ones

à Increased Performance (Mikolov, NIPS 2013)

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 10

Down-Sampling Mechanism

Weighting

• GloVe
• New: SVDwPPMI

Probabilistic

• word2vec
• SVDPPMI

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 13

Experimental Design I/II

• Three Corpora:
• Corpus of Historical American English 2000s decade

(COHA; 28M tokens.)

• English News Crawl Corpus (NEWS; 550M tokens)
• Wikipedia (WIKI; 1.7G tokens)

à Other studies used mostly COHA-sized corpora!

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 14

Experimental Design II/II

• Train 10 models each with SGNS, GloVe, SVDPPMI

(none / prob. down-sampling), SVDwPPMI

• Evaluate intrinsically with four word similarity & two

analogy test sets

• Measure stability with j@10 for 1k most frequent

words

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 16

Stability Results

GloVe‘s high stability (Antoniak & Mimno, TACL 2018; Wendlandt et al., NAACL 2018) is true only for small corpora

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 17

Exemplary Accuracy Results

Wilcoxon rank-sum test shows SVDwPPMI and SGNS to be indistinguishable in accuracy over all test sets and corpora

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 18

Conclusion

• Typical word embeddings are unstable
• Down-sampling details greatly affect stability
• GloVe’s stability is worse than claimed in literature
• SVDwPPMI embeddings provide SGNS-like performance

and perfect stability

• See paper for additional results (and bootstrapping)

RepEval 2019 June 6th 2019, Minneapolis, USA Johannes Hellrich, Bernd Kampe & Udo Hahn Down-Sampling and SVD Word Embedding Stability 19

The Influence of Down-Sampling Strategies

• n SVD Word Embedding Stability

Johannes Hellrich, Bernd Kampe & Udo Hahn

Jena University Language & Information Engineering (JULIE) Lab Friedrich Schiller University Jena, Jena, Germany www.julielab.de