Leaderboard¶
MovieLens 100K¶
MovieLens 100K, official ua split, k=10, default hyper-parameters. Regenerate with python benchmarks/run.py run leaderboard --dataset movielens-100k.
| Model | NDCG@10 | P@10 | R@10 | hit rate | MAP | MRR | cat cov | user cov | mean pop | novelty |
|---|---|---|---|---|---|---|---|---|---|---|
| SLIM | 0.3024 | 0.2558 | 0.2558 | 0.9226 | 0.1621 | 0.6418 | 0.2226 | 1.0000 | 247.5139 | 8.65 |
| XSimGCL | 0.2931 | 0.2556 | 0.2556 | 0.9332 | 0.1540 | 0.6054 | 0.4458 | 1.0000 | 206.4415 | 9.05 |
| BPR | 0.2800 | 0.2467 | 0.2467 | 0.9226 | 0.1451 | 0.5782 | 0.3542 | 1.0000 | 228.5243 | 8.84 |
| EASE | 0.2767 | 0.2382 | 0.2382 | 0.9035 | 0.1444 | 0.5888 | 0.3030 | 1.0000 | 228.8724 | 8.79 |
| RP3Beta | 0.2762 | 0.2407 | 0.2407 | 0.9215 | 0.1414 | 0.5911 | 0.2417 | 1.0000 | 235.4201 | 8.78 |
| BM25+AutoTune | 0.2759 | 0.2333 | 0.2333 | 0.8929 | 0.1443 | 0.6049 | 0.1274 | 1.0000 | 282.8901 | 8.41 |
| BM25 | 0.2661 | 0.2292 | 0.2292 | 0.8918 | 0.1370 | 0.5788 | 0.1024 | 1.0000 | 293.0034 | 8.36 |
| ItemKNN | 0.2550 | 0.2200 | 0.2200 | 0.9173 | 0.1268 | 0.5624 | 0.2887 | 1.0000 | 217.0530 | 8.86 |
| SimpleX | 0.2467 | 0.2137 | 0.2137 | 0.8388 | 0.1267 | 0.5336 | 0.3315 | 1.0000 | 222.2856 | 8.96 |
| MostPopular | 0.1331 | 0.1215 | 0.1215 | 0.7306 | 0.0545 | 0.3218 | 0.0530 | 1.0000 | 371.7549 | 7.97 |
| ALS | 0.0422 | 0.0408 | 0.0408 | 0.3160 | 0.0144 | 0.1119 | 0.1464 | 1.0000 | 155.0729 | 10.09 |
Measured on more than one host, so compare timings only between rows that share one. SLIM, BPR, EASE, RP3Beta, BM25, ItemKNN, MostPopular, ALS: Apple M4 Pro (12 usable cores), macOS-26.7-arm64-arm-64bit-Mach-O, CPython 3.14.3, numpy 2.5.3, skrecsys 0.4.0, _core built in release mode. XSimGCL, SimpleX: Apple M4 Pro (12 usable cores), macOS-26.6.2-arm64-arm-64bit-Mach-O, CPython 3.14.3, numpy 2.5.3, skrecsys 0.4.0, _core built in release mode. BM25+AutoTune: Apple M4 Pro (12 usable cores), macOS-27.0-arm64-arm-64bit-Mach-O, CPython 3.14.3, numpy 2.5.3, skrecsys 0.5.0, _core built in release mode. The quality table above is deterministic and portable; the timings below are not comparable across machines or builds.
Wall clock per call. Each operation is sampled until it has spent 20s or reached its cap (1000 fits, 1000 recommend calls), after untimed warm-up calls (1 fit, 10 rank) so that no sample pays for a cold start; the samples columns say how many each row actually got, which is why a slow model shows fewer. The batch columns say what a single call processed: one fit covers the whole training split, and one recommend call ranks the entire catalog for every held-out user at once, so these are throughput numbers rather than single-request latency. Fit reports the spread a handful of samples can resolve; ranking is sampled often enough for nearest-rank quantiles, each of which is a call that really happened. Compare min across machines and watch max for the variance a run saw.
| Model | fit batch (interactions) | fit samples | fit min | fit median | fit max | rank batch (users x items) | rank samples | rank mean | rank median | rank q95 | rank q99 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SLIM | 90570 | 111 | 164 ms | 179 ms | 231 ms | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
| XSimGCL | 90570 | 3 | 457.64 s | 505.34 s | 621.14 s | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
| BPR | 90570 | 16 | 1.30 s | 1.31 s | 1.32 s | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
| EASE | 90570 | 710 | 26 ms | 28 ms | 33 ms | 943 x 1680 | 1000 | 3 ms | 3 ms | 3 ms | 4 ms |
| RP3Beta | 90570 | 1000 | 8 ms | 8 ms | 18 ms | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
| BM25+AutoTune | 90570 | 8 | 2.60 s | 2.63 s | 2.88 s | 943 x 1680 | 1000 | 7 ms | 7 ms | 8 ms | 9 ms |
| BM25 | 90570 | 1000 | 6 ms | 7 ms | 9 ms | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
| ItemKNN | 90570 | 1000 | 5 ms | 6 ms | 7 ms | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
| SimpleX | 90570 | 3 | 130.37 s | 134.37 s | 139.83 s | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
| MostPopular | 90570 | 1000 | 1 ms | 1 ms | 1 ms | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
| ALS | 90570 | 60 | 324 ms | 326 ms | 444 ms | 943 x 1680 | 1000 | 1 ms | 1 ms | 1 ms | 1 ms |
Every model uses its default hyper-parameters, so this ranks the library's baselines,
not the best each method can do. The one exception is BM25+AutoTune, which is
tuned on the training split. It shows how much of that gap a
search closes: 0.2661 to 0.2759 NDCG for BM25, at the cost of a 2.6 s fit. R@10 equals P@10 because the ua split holds out
exactly 10 items per user, and user cov is 1 by construction: recommend raises
rather than return a short list. The beyond-accuracy columns are the interesting ones —
MostPopular has the highest mean pop and the lowest novelty, and BM25 buys its
ranking score by covering a tenth of the catalog where EASE covers a third. SLIM
leads on every ranking metric while keeping a fifth of the catalog reachable, at a
similarity matrix a quarter as dense as RP3Beta's. BPR is the only model here
trained on pairwise preferences rather than fitted in closed form, and it buys the
second-best ranking with the widest catalog coverage of all; it is also the slowest to
fit, and the entry is seeded, which pins it to a single thread.
The second table times the same runs. Both operations are batched — one fit over the
whole training split, one recommend call scoring all 943 held-out users against all
1680 items the model saw — so these are throughput numbers, and the per-request latency
of a single user is a different measurement. Ranking separates the models by what they
have to touch rather than by how good they are. Every model goes through a kernel that
scores a query, drops what the user has seen and keeps the best ten before moving on, so
no score matrix is ever built: the neighbourhood models accumulate only the items a
query reaches, the factor models scan the catalog a tile of users at a time, and both
finish in about 1 ms. EASE is the slowest at 3 ms because its similarity matrix is
dense, so every item a user has seen adds a full catalog-wide row to their scores. The
max column is where a run's variance shows — RP3Beta fits in 8 ms at the median but
18 ms at its slowest sample. How many samples a row gets is the budget's doing, which is
why BPR at 1.3 s a fit gets sixteen and MostPopular gets its full thousand;
--budget and --repeat buy more.
Amazon Books¶
Amazon Books (amzn-books-l50), leave-one-out split, k=10, default hyper-parameters. Scored on a fixed random sample of 10,000 held-out users. Not run: EASE (a dense 674k x 674k item matrix does not fit in memory); SLIM (one elastic net per item over a 674k-item catalog does not finish); BM25+AutoTune (50 trials of 3-fold cross-validation over 7.4M interactions is hours of fitting); SimpleX (one training epoch of 7.4M interactions runs over an hour on CPU); XSimGCL (one training epoch of 7.4M interactions runs over an hour on CPU). Regenerate with python benchmarks/run.py run leaderboard --dataset amazon-books.
| Model | NDCG@10 | P@10 | R@10 | hit rate | MAP | MRR | cat cov | user cov | mean pop | novelty |
|---|---|---|---|---|---|---|---|---|---|---|
| BM25 | 0.0466 | 0.0078 | 0.0777 | 0.0777 | 0.0370 | 0.0370 | 0.1034 | 1.0000 | 52.9699 | 18.43 |
| ItemKNN | 0.0364 | 0.0060 | 0.0603 | 0.0603 | 0.0291 | 0.0291 | 0.1158 | 1.0000 | 63.6760 | 20.12 |
| RP3Beta | 0.0247 | 0.0045 | 0.0452 | 0.0452 | 0.0184 | 0.0184 | 0.1227 | 1.0000 | 32.6864 | 20.45 |
| BPR | 0.0075 | 0.0015 | 0.0151 | 0.0151 | 0.0052 | 0.0052 | 0.0160 | 1.0000 | 753.6300 | 14.20 |
| MostPopular | 0.0037 | 0.0008 | 0.0079 | 0.0079 | 0.0025 | 0.0025 | 0.0000 | 1.0000 | 4092.8595 | 10.99 |
| ALS | 0.0000 | 0.0000 | 0.0001 | 0.0001 | 0.0000 | 0.0000 | 0.0005 | 1.0000 | 132.8660 | 20.73 |
Measured on Apple M4 Pro (12 usable cores), macOS-26.7-arm64-arm-64bit-Mach-O, CPython 3.14.3, numpy 2.5.3, skrecsys 0.4.0, _core built in release mode. The quality table above is deterministic and portable; the timings below are not comparable across machines or builds.
Wall clock per call. Each operation is sampled until it has spent 60s or reached its cap (5 fits, 50 recommend calls), after untimed warm-up calls (1 fit, 2 rank) so that no sample pays for a cold start; the samples columns say how many each row actually got, which is why a slow model shows fewer. The batch columns say what a single call processed: one fit covers the whole training split, and one recommend call ranks the entire catalog for every held-out user at once, so these are throughput numbers rather than single-request latency. Fit reports the spread a handful of samples can resolve; ranking is sampled often enough for nearest-rank quantiles, each of which is a call that really happened. Compare min across machines and watch max for the variance a run saw.
| Model | fit batch (interactions) | fit samples | fit min | fit median | fit max | rank batch (users x items) | rank samples | rank mean | rank median | rank q95 | rank q99 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| BM25 | 7374280 | 5 | 566 ms | 600 ms | 604 ms | 10000 x 660940 | 50 | 22 ms | 22 ms | 27 ms | 34 ms |
| ItemKNN | 7374280 | 5 | 573 ms | 598 ms | 605 ms | 10000 x 660940 | 50 | 45 ms | 46 ms | 51 ms | 53 ms |
| RP3Beta | 7374280 | 5 | 1.63 s | 1.67 s | 1.70 s | 10000 x 660940 | 50 | 36 ms | 35 ms | 43 ms | 50 ms |
| BPR | 7374280 | 3 | 312.95 s | 313.07 s | 313.21 s | 10000 x 660940 | 16 | 3.81 s | 3.88 s | 3.94 s | 3.94 s |
| MostPopular | 7374280 | 5 | 132 ms | 133 ms | 137 ms | 10000 x 660940 | 50 | 595 ms | 595 ms | 610 ms | 627 ms |
| ALS | 7374280 | 3 | 64.91 s | 65.12 s | 65.19 s | 10000 x 660940 | 50 | 979 ms | 988 ms | 1.06 s | 1.08 s |