Sequence-Aware Factored Mixed Similarity Model for Next-Item - - PowerPoint PPT Presentation

Sequence-Aware Factored Mixed Similarity Model for Next-Item Recommendation

Liulan Zhong, Jing Lin, Weike Pan∗ and Zhong Ming∗

zhongliulan2017@email.szu.edu.cn, linjing2018@email.szu.edu.cn, panweike@szu.edu.cn, mingz@szu.edu.cn

National Engineering Laboratory for Big Data System Computing Technology, Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 1 / 30

Introduction

Problem Definition

Next-Item Recommendation Input: (u, Su), i.e., a sequence of items for each user u. Goal: Rank the unobserved items at user u’s next step by estimating the score ˆ ruj, j ∈ I\Iu to form the recommendation list.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 2 / 30

Introduction

Notations (1/2)

Table: Some notations and explanations.

n number of users m number of items u user ID, u ∈ {1, 2, . . . , n} i item ID, i ∈ {1, 2, . . . , m} U the whole set of users I the whole set of items P the whole set of observed (u, i) pairs A a sampled set of unobserved (u, i) pairs Iu a set of items that have been interacted by user u d ∈ R number of latent dimensions Vi·, Wi· ∈ R1×d item-specific latent feature vector w.r.t. item i bi ∈ R item bias γ learning rate αw, αv, βη, βv tradeoff parameters of regularization terms T iteration number

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 3 / 30

Introduction

Notations (2/2)

Table: Some notations and explanations.

Su a sequence of items, Su = {i1

u, i2 u, . . . , i|Su| u

} it

u

the tth item in Su ˆ ruit

u

predicted preference of user u to item it

u

sij predefined similarity between item i and item j λ tradeoff parameter in mixed similarity L the order of Markov chains ℓ the ℓth order of Markov chains, ℓ ∈ {1, 2, . . . , L} it-ℓ

u

the (t-ℓ)th item in Su η ∈ R1×L global weighting vector ηu ∈ R1×L personalized weighting vector w.r.t. user u

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 4 / 30

Background

Motivation

Previously proposed methods usually model the general representation and the sequential representation in two divided factorization components. Our proposed model can integrate the items’ general similarity and the items’ learnable sequential representations in a unified component. On the basis of Fossil [He et al., 2016], it considers the short-term sequential information via high-order Markov chains. The rationale behind the specific term ηℓ + ηu

ℓ is that each of the previous L

locations should contribute with different weights to the high-order smoothness, lacking the weight contribution from the latest specific items.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 5 / 30

Background

Fossil: Prediction Rule

On the basis of FISM [Kabbur et al., 2013], Fossil combines a similarity-based method and high-order Markov chains, the predicted function is as follows, ˆ ruit

u = bit u + ¯

U−it

u

u· V T it

u·,

(1) where ¯ U−it

u

u·

= 1

• |Iu\{it

u}|

• i′∈Iu\{it

u}

Wi′· +

L

• ℓ=1

(ηℓ + ηu

ℓ )Wii−ℓ

u

·,

(2) and ηu

ℓ controls the weight of user u’s preference and sequential

dynamics, while ηℓ is a global parameter shared by all the users.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 6 / 30

Background

Overall of Our Solution

Sequence-Aware Factored Mixed Similarity Model (S-FMSM) Our S-FMSM considers the weights of the specific history item it-ℓ

u

and its relative position in contributing to the target item it

u for sequence

modeling.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 7 / 30

Method

S-FMSM: Prediction Rule

The predicted preference of user u to item it

u:

ˆ ruit

u = bit u + ¯

U−it

u

u· V T it

u·,

(3) where

¯ U−it

u

u·

= 1

• |Iu\{it

u}|

• i′∈Iu\{it

u}

Wi′· +

L

• ℓ=1

(ηℓ + ηu

ℓ )((1 − λ) + λsit

uit-ℓ u )Wit-ℓ u

·.

(4)

Notes: sit

uit-ℓ u

is the cosine similarity between item it

u and item it-ℓ u . In fact,

what it captures is the weight of the history item it-ℓ

u

in contributing to the target item it

u.

The tradeoff parameter λ tuned among {0,0.2,0.4,0.6,0.8,1} adjusts the influence of sit

uit-ℓ u

in preference prediction. Notice that when λ = 0, it reduces to Fossil [He et al., 2016].

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 8 / 30

Method

S-FMSM: Objective Function

The objective function is as follows, min

Θ

• u∈U
• it

u∈Su,t=1

• j /

∈Iu

fuit

uj,

(5) where Θ = {Vi·, Wi·, bi, ηℓ, ηu

ℓ , i = 1, 2, . . . , m; u = 1, 2, . . . , n; ℓ = 1, 2, . . . , L}

and fuit

uj = − ln σ(ˆ

ruit

u − ˆ

ruj) + αv

2

• Vit

u·

• 2

+ αv

2

• Vj·
• 2 +

αw 2

• i′∈Iu ||Wi′·||2 + βv

2 b2 it

u + βv

2 b2 j + βη 2 ||ηℓ||2 + βη 2

• ηu

ℓ

• 2 is a tentative
• bjective function for a randomly sampled triple (u, it

u, j) via “first

positive (u, it

u) then negative j”.

Notes: Because the pairwise preference relaxes the assumption of the pointwise preference, we adopt a personalized pairwise ranking to keep the loss at a minimum.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 9 / 30

Method

Gradients (1/2)

The gradient of each parameter θ ∈ Θ, i.e., ∇θ =

∂(fuit

uj)

∂θ

, is computed as follows:

∇bit

u

= βv bit

u

+ (−1)σ(ˆ ruj − ˆ ruit

u

), (6) ∇bj = βv bj + σ(ˆ ruj − ˆ ruit

u

), (7) ∇Vit

u· = αv Vit u· + (−1)σ(ˆ

ruj − ˆ ruit

u

)[ 1

• |Iu\{it

u}|

• i′ ∈Iu\{it

u}

Wi′ · +

L

• ℓ=1

(ηℓ + ηu

ℓ)((1 − λ) + λsit uit-ℓ u

)Wit-ℓ

u ·],

(8) ∇Vj· = αv Vj· + σ(ˆ ruj − ˆ ruit

u

)[ 1

• |Iu|
• i′ ∈Iu

Wi′ · +

L

• ℓ=1

(ηℓ + ηu

ℓ)((1 − λ) + λsjit-ℓ u

)Wit-ℓ

u ·],

(9) Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 10 / 30

Method

Gradients (2/2)

∇ηℓ = βηηℓ + (−1)σ(ˆ ruj − ˆ ruit

u

)Wit-ℓ

u ·

[V T

it u·((1 − λ) + λsit uit-ℓ u

) − V T

j· ((1 − λ) + λsjit-ℓ u

)], ℓ = 1, . . . , L, (10) ∇ηu

ℓ

= βηηu

ℓ + (−1)σ(ˆ

ruj − ˆ ruit

u

)Wit-ℓ

u ·

[V T

it u·((1 − λ) + λsit uit-ℓ u

) − V T

j· ((1 − λ) + λsjit-ℓ u

)], ℓ = 1, . . . , L, (11) ∇Wi′ · = αw Wi′ · + (−1)σ(ˆ ruj − ˆ ruit

u

)[ 1

• |Iu\{it

u}|

Vit

u· −

1

• |Iu|

Vj·], i′ ∈ Iu\{it

u, it−1 u

, . . . , it−L

u

}, (12) ∇Wit

u· = αw Wit u· + (−1)σ(ˆ

ruj − ˆ ruit

u

)[− 1

• |Iu|

Vj·], (13) ∇Wit-ℓ

u ·

= αw Wit-ℓ

u · + (−1)σ(ˆ

ruj − ˆ ruit

u

)[(Vit

u·((1 − λ) + λsit uit-ℓ u

) −Vj·((1 − λ) + λsjit-ℓ

u

))(ηℓ + ηu

ℓ)], ℓ = 1 . . . , L.

(14) Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 11 / 30

Method

Update Rules

We have the update rule for each parameter, θ = θ − γ∇θ, (15) where γ > 0 is the learning rate, θ ∈ Θ, Θ = {Vi·, Wi·, bi, ηℓ, ηu

ℓ , i =

1, 2, . . . , m; u = 1, 2, . . . , n; ℓ = 1, 2, . . . , L}.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 12 / 30

Method

Algorithm

Algorithm 1 The algorithm of S-FMSM.

1: Initialize the model parameters. 2: for t = 1, · · · , T do 3:

for each (u, it

u) ∈ P in a random order do

4:

Randomly pick up an item j from I\Iu

5:

Calculate gradients according to Eqs.(6-14)

6:

Update the model parameters via Eq.(15)

7:

end for

8: end for

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 13 / 30

Experiments

Datasets (1/2)

We adopt two commonly used datasets in the experiments, including the MovieLens data, i.e., MovieLens 100K (ML100K), MovieLens 1M (ML1M), and the Amazon e-commerce data, i.e., Office Products (Office), Automotive (Auto), Video Games (Video), and Cell Phones & Accessories (Cell). We treat all the observed behaviors as positive feedback and preprocess each dataset as follows: We remove the records of the users who rate fewer than five times; We remove the records of the items that are rated fewer than five times; We sort all the records according to the timestamps and split each user’s sequence into three parts, i.e., the item(s) at the last step for test, the item(s) at the penultimate step for validation, and the remaining items for training.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 14 / 30

Experiments

Datasets (2/2)

Table: Statistics of the processed data used in the experiments, including the number of users (User #), the number of items (Item #), the number of records (Record #), and the density (Density). Dataset User # Item # Record # Density (%) ML100K 943 1, 349 99, 287 7.80 ML1M 6, 040 3, 416 999, 611 4.80 Office 16, 243 5, 526 97, 327 0.11 Video 30, 935 12, 111 260, 163 0.07 Auto 31, 877 9, 992 122, 009 0.04 Cell 67, 453 17, 969 346, 245 0.03

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 15 / 30

Experiments

Baselines

We compare our S-FMSM against five closely related baselines: rank based on the items’ popularities (PopRank); Bayesian personalized ranking (BPR) [Rendle et al., 2009]; factored item similarity model (FISM) [Kabbur et al., 2013]; factorizing personalized Markov chains (FPMC) [Rendle et al., 2010]; fusing similarity models with Markov chains (Fossil) [He et al., 2016]. From the table below, we can see that

• nly our S-FMSM incorporates the mixed similarity into the short-term

preference.

Table: Comparison among a popularity-based method, four factorization-based methods and our S-FMSM.

Property PopRank BPR FISM FPMC Fossil S-FMSM Personalized × √ √ √ √ √ Sequence-aware × × × √ √ √ Mixed Similarity × × × × × √ Pairwise ranking × √ × √ √ √

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 16 / 30

Experiments

Parameter Configurations (1/2)

For all the factorization-based methods, we fix the number of dimensions d = 20, the learning rate γ = 0.01, and we adopt stochastic gradient descent (SGD) algorithm to train all the factorization-based methods. For FISM, we randomly sample a set of negative items A with |A| = 3|P| following [Kabbur et al., 2013]. For BPR, FPMC, Fossil and our S-FMSM, we use the same sampling strategy, i.e., randomly selecting one negative sample each time, for fair comparison.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 17 / 30

Experiments

Parameter Configurations (2/2)

We use the following Strategies to choose the parameters of each factorization-based method on each dataset. We choose the tradeoff parameter of the regularization terms αv = αw = βv = βη from {0.1, 0.01, 0.001}, the order L from {1, 2, 3} and the iteration number T from {100, 500, 1000} via the NDCG@20 performance on the validation data. Moreover, the similarity tradeoff λ in our S-FMSM is chosen from {0, 0.2, 0.4, 0.6, 0.8, 1} on the validation data. For each validation data, we select the optimal parameters according to the averaged performance of NDCG@20 of three runs. With the

• ptimal parameter values, the final results on the test data are also

averaged values of three runs.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 18 / 30

Experiments

Evaluation Metrics

Precision@20 Recall@20 F1@20 NDCG@20 1-call@20

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 19 / 30

Experiments

Results (1/7)

Table: Recommendation performance of PopRank, Bayesian personalized ranking (BPR), factored item similarity model (FISM), factorizing personalized Markov chains (FPMC), fusing similarity models with Markov chains (Fossil), and our sequence-aware factored mixed similarity model (S-FMSM) on six real-world datasets. The best results are marked in bold.

Dataset Method Pre@20 Rec@20 F1@20 NDCG@20 1-call@20 ML100K PopRank 0.0153 0.1197 0.0247 0.0619 0.2174 BPR 0.0282±0.0004 0.1974±0.0049 0.0452±0.0007 0.1032±0.0012 0.3132±0.0043 FISM 0.0275±0.0005 0.1832±0.0070 0.0439±0.0009 0.0985±0.0025 0.2930±0.0080 FPMC 0.0273±0.0003 0.2292±0.0071 0.0452±0.0006 0.1147±0.0020 0.3574±0.0046 Fossil 0.0277±0.0004 0.2299±0.0004 0.0458±0.0006 0.1153±0.0020 0.3627±0.0018 S-FMSM 0.0286±0.0005 0.2322±0.0057 0.0471±0.0008 0.1220±0.0017 0.3669±0.0038 ML1M PopRank 0.0067 0.0678 0.0116 0.0313 0.1133 BPR 0.0113±0.0000 0.1102±0.0006 0.0196±0.0001 0.0518±0.0004 0.1827±0.0011 FISM 0.0100±0.0001 0.0999±0.0007 0.0174±0.0002 0.0451±0.0007 0.1648±0.0016 FPMC 0.0176±0.0002 0.1703±0.0026 0.0306±0.0004 0.0821±0.0003 0.2614±0.0039 Fossil 0.0192±0.0002 0.1870±0.0028 0.0334±0.0004 0.0879±0.0005 0.2865±0.0047 S-FMSM 0.0194±0.0003 0.1895±0.0019 0.0337±0.0004 0.0895±0.0015 0.2879±0.0036 Office PopRank 0.0003 0.0040 0.0005 0.0013 0.0051 BPR 0.0022±0.0001 0.0337±0.0013 0.0040±0.0002 0.0148±0.0007 0.0423±0.0017 FISM 0.0021±0.0001 0.0340±0.0004 0.0040±0.0000 0.0154±0.0003 0.0413±0.0004 FPMC 0.0020±0.0001 0.0297±0.0011 0.0037±0.0001 0.0133±0.0001 0.0353±0.0010 Fossil 0.0023±0.0000 0.0346±0.0007 0.0042±0.0001 0.0153±0.0004 0.0428±0.0006 S-FMSM 0.0025±0.0001 0.0398±0.0026 0.0047±0.0002 0.0175±0.0011 0.0479±0.0025

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 20 / 30

Experiments

Results (2/7)

Table: Recommendation performance of PopRank, Bayesian personalized ranking (BPR), factored item similarity model (FISM), factorizing personalized Markov chains (FPMC), fusing similarity models with Markov chains (Fossil), and our sequence-aware factored mixed similarity model (S-FMSM) on six real-world datasets. The best results are marked in bold.

Dataset Method Pre@20 Rec@20 F1@20 NDCG@20 1-call@20 Auto PopRank 0.0023 0.0362 0.0043 0.0173 0.0457 BPR 0.0030±0.0001 0.0448±0.0017 0.0056±0.0002 0.0196±0.0004 0.0576±0.0019 FISM 0.0029±0.0000 0.0435±0.0009 0.0054±0.0001 0.0188±0.0005 0.0560±0.0006 FPMC 0.0019±0.0000 0.0266±0.0003 0.0034±0.0001 0.0117±0.0002 0.0353±0.0007 Fossil 0.0032±0.0000 0.0467±0.0007 0.0059±0.0001 0.0209±0.0002 0.0612±0.0008 S-FMSM 0.0033±0.0000 0.0474±0.0004 0.0060±0.0000 0.0211±0.0002 0.0622±0.0004 Video PopRank 0.0024 0.0371 0.0044 0.0151 0.0467 BPR 0.0050±0.0000 0.0734±0.0006 0.0092±0.0001 0.0319±0.0003 0.0928±0.0008 FISM 0.0046±0.0000 0.0675±0.0005 0.0083±0.0001 0.0291±0.0003 0.0852±0.0006 FPMC 0.0055±0.0001 0.0850±0.0010 0.0101±0.0001 0.0376±0.0004 0.1026±0.0010 Fossil 0.0059±0.0000 0.0895±0.0002 0.0108±0.0001 0.0388±0.0001 0.1088±0.0005 S-FMSM 0.0059±0.0001 0.0890±0.0010 0.0108±0.0001 0.0398±0.0006 0.1089±0.0009 Cell PopRank 0.0027 0.0381 0.0050 0.0157 0.0532 BPR 0.0033±0.0000 0.0447±0.0005 0.0060±0.0001 0.0203±0.0009 0.0617±0.0007 FISM 0.0034±0.0001 0.0480±0.0008 0.0063±0.0001 0.0198±0.0004 0.0644±0.0012 FPMC 0.0031±0.0001 0.0435±0.0006 0.0057±0.0001 0.0201±0.0004 0.0575±0.0009 Fossil 0.0035±0.0001 0.0496±0.0013 0.0064±0.0002 0.0229±0.0003 0.0638±0.0017 S-FMSM 0.0040±0.0000 0.0568±0.0007 0.0074±0.0001 0.0265±0.0005 0.0737±0.0007

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 21 / 30

Experiments

Results (3/7)

Observations: Our model achieves the best performance on all the datasets except on Video where it is comparable with Fossil, which shows the effectiveness of our proposed mixed similarity model. PopRank is a basic method that ranks items according to their popularities, which provides poor results due to its non-personalization as expected.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 22 / 30

Experiments

Results (4/7)

Observations: For the non-sequential recommendation models, the pairwise approach BPR is better than the pointwise one FISM, which shows the advantage of the pairwise preference assumption. The sequential recommendation algorithms (i.e., FPMC, Fossil and S-FMSM) do not treat the user-interacted items as a bag of items, but rather as a sequence of items, which makes the models more powerful in terms of recommendation accuracy. . . .

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 23 / 30

Experiments

Results (5/7)

Figure: Recommendation performance (NDCG@20) of our S-FMSM and Fossil with different values of λ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1} on each dataset.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 24 / 30

Experiments

Results (6/7)

Figure: Recommendation performance (NDCG@20) of our S-FMSM and Fossil with different values of λ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1} on each dataset.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 25 / 30

Experiments

Results (7/7)

We choose the tradeoff parameters αv = αw = βv = βη ∈ {0.1, 0.01, 0.001}, L ∈ {1, 2, 3}, T ∈ {100, 500, 1000} with a fixed λ from {0,0.2,0.4,0.6,0.8,1} via the NDCG@20 performance on the validation data. Then, we study our model with the optimal parameters

• n each test data for three times and report the averaged results

above. Observations: With different values of the parameter λ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1}, the recommendation performance of our S-FMSM on the test data is better than that of Fossil on all the datasets. By adjusting the parameter λ of our S-FMSM, we can see that when λ is 1 (ML100K), 0.8 (ML1M), 0.8 (Office), 0.2 (Auto), 0.8 (Video) and 0.8 (Cell), it achieves the best performance. . . .

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 26 / 30

Conclusions and Future Work

Conclusions

We propose a novel sequence-aware recommendation method, i.e., sequence-aware factored mixed similarity model (S-FMSM), for a recently studied important task of next-item recommendation. Our model considers consine similarity between a recent item and the target item in order to better capture the short-term sequential

• effect. The main contribution is to study the mixed similarity in

sequential recommendation. We mainly focus on factorization-based methods and conduct extensive empirical studies in the context of several state-of-the-art factorization-based methods on six real-world datasets and find that our S-FMSM achieves very promising performance.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 27 / 30

Conclusions and Future Work

Future Work

For future works, we are interested in capturing more effect of users’ short-term preferences by designing personalized variable-length subsequences. Moreover, we are interested in leveraging some auxiliary information such as social context so as to improve the recommendation performance for inactive users. Finally, we may also apply the mixed similarity to deep learning-based methods.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 28 / 30

Thank You

Thank You!

We thank the support of National Natural Science Foundation of China Nos. 61872249, 61836005 and 61672358.

Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 29 / 30

References

References I

Santosh Kabbur, Xia Ning and George Karypis (2013) FISM: Factored Item Similarity Models for top-N Recommender Systems Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 659–667. Steffen Rendle, Christoph Freudenthaler, Zeno Gantner and Lars Schmidt-Thieme (2009) BPR: Bayesian Personalized Ranking from Implicit Feedback Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, 452–461. Steffen Rendle, Christoph Freudenthaler and Lars Schmidt-Thieme (2010) Factorizing Personalized Markov Chains for Next-basket Recommendation Proceedings of the 19th International Conference on World Wide Web, 811–820. Ruining He and Julian McAuley (2016) Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation Proceedings of the 2016 IEEE 16th International Conference on Data Mining, 191-200. Zhong, Lin, Pan and Ming (SZU) S-FMSM IEEE BigComp 2020 30 / 30