How to choose a recommender

Input Data

What is the input?

RePlay models differ by types of data they can process:

Collaborative use only user-item interaction logs.
Content-based use only user or item features.
Hybrid can use both log and features.
Are interactions explicit?

Our information can be either explicit e.g. ratings, or implicit e.g. view, number of play counts.
Some models transform any type of data to implicit (unary ratings).

Will there be new users?

Some models need to be completely retrained to give predictions for new users (not present in train data, but have interactions history) while others don’t.

Will there be new items?

The same goes for new items.

Algorithm	Data	Interactions	New Users	New Items
Popular Recommender	Collaborative	converted to unary ratings	+	-
Popular By Users	Collaborative	implicit feedback	-	-
Wilson Recommender	Collaborative	binary ratings	+	-
Random Recommender	Collaborative	converted to unary ratings	+	+
K-Nearest Neighbours	Collaborative	converted to unary ratings	+	-
Alternating Least Squares	Collaborative	implicit feedback	-	-
Neural Matrix Factorization	Collaborative	converted to unary ratings	-	-
SLIM	Collaborative	unary ratings, explicit feedback	+	-
ADMM SLIM	Collaborative	unary ratings, explicit feedback	+	-
Mult-VAE	Collaborative	converted to unary ratings	+	-
Word2Vec Recommender	Collaborative	converted to unary ratings	+	-
Association Rules	Collaborative	converted to unary ratings	+	-
Cluster Recommender	Hybrid	converted to unary ratings	+	-
LightFM Wrap	Hybrid	depends on loss	+	+
Implicit Wrap	Collaborative	depends on model	-	-
Two Stages Scenario	Hybrid	converted to unary ratings for second level	`*`	`*`

* - depends on base models.

Model requirements

Should recommendations be personalized?
Should cold users get recommendations? (without any interactions).
Should model recommend cold items? (that no one interacted with).
Should model be able to recommend unseen items?

Algorithm	Personalized	Cold Users	Cold Items	Recommends Unseen Items
Popular Recommender	-	+	-	+
Popular By Users	+	-	-	-
Wilson Recommender	-	+	-	+
Random Recommender	-	+	+	+
K-Nearest Neighbours	+	+	-	+
Alternating Least Squares	+	-	-	+
Neural Matrix Factorization	+	-	-	+
SLIM	+	-	-	+
ADMM SLIM	+	-	-	+
Mult-VAE	+	-	-	+
Word2Vec Recommender	+	-	-	+
Association Rules	+	-	-	+
Cluster Recommender	+	+	-	+
LightFM Wrap	+	+	+	+
Implicit Wrap	+	-	-	+
Two Stages Scenario	+	`*`	`*`	`*`

* - depends on base models.

More info on models.

Model Comparison

All metrics are calculated at $k=10$

MovieLens 1m

Model	HitRate	MAP	MRR	NDCG	Coverage	Surprisal	fit_pred_time
ALS (Implicit)	0.686	0.172	0.419	0.265	0.133	0.163	10.128
LightFM	0.699	0.167	0.431	0.263	0.151	0.168	10.833
SLIM	0.669	0.172	0.419	0.261	0.04	0.124	12.456
KNN	0.655	0.166	0.409	0.256	0.056	0.138	17.964
LightFM (w/ feats)	0.691	0.16	0.412	0.255	0.232	0.195	86.704
PopRec	0.645	0.157	0.39	0.244	0.034	0.118	12.3
MultVAE	0.64	0.151	0.396	0.238	0.031	0.123	26.977
NeuroMF	0.627	0.111	0.318	0.193	0.257	0.235	350.737
ADMM SLIM	0.591	0.084	0.304	0.159	0.367	0.237	77.647
Word2Vec	0.515	0.072	0.244	0.138	0.145	0.24	25.133
Wilson	0.414	0.045	0.181	0.092	0.017	0.262	10.034
RandomRec (popular)	0.382	0.028	0.142	0.069	0.654	0.318	6.827
RandomRec (uniform)	0.183	0.009	0.068	0.026	0.961	0.537	7.898
ALS (Explicit)	0.093	0.005	0.033	0.013	0.266	0.684	13.876