Sequential benchmark¶

A different question, and so a different table: each user's last interaction is held out, the model reads the 200 before it, and the held-out item is ranked against the whole catalog. This is the leave-one-out protocol the sequential-recommendation literature reports on, which is what skrecsys.nn.HSTU is built for and what the numbers in the HSTU and SASRec papers can be read against. It is not comparable with the leaderboard above, which splits interactions rather than predicting the next one.

uv run python benchmarks/run.py run sequential --dry-run   # say what is out of date
uv run python benchmarks/run.py run sequential             # measure it, re-render the table below

MovieLens 1M (ml-1m-l200), leave-one-out split, default hyper-parameters. Each user's last interaction is held out and ranked against the whole catalog, with the items they already saw removed. Not run: SimpleX (one fit of 1M interactions runs for hours on CPU); XSimGCL (one fit of 1M interactions runs for hours on CPU). Regenerate with python benchmarks/run.py run sequential --dataset movielens-1m.

Not yet measured: Mamba4Rec.

Measured on more than one host, so compare timings only between rows that share one. HSTU: Apple M4 Pro (12 usable cores), macOS-26.6.2-arm64-arm-64bit-Mach-O, CPython 3.14.3, numpy 2.5.3, skrecsys 0.4.0, _core built in release mode. EASE, SLIM, BPR, ItemKNN, BM25, RP3Beta, MostPopular, ALS: Apple M4 Pro (12 usable cores), macOS-26.7-arm64-arm-64bit-Mach-O, CPython 3.14.3, numpy 2.5.3, skrecsys 0.4.0, _core built in release mode. The sequential models ran with device="auto", which resolved to MPS on this host (torch 2.14.0); every other model is CPU only. BM25+AutoTune: Apple M4 Pro (12 usable cores), macOS-27.0-arm64-arm-64bit-Mach-O, CPython 3.14.3, numpy 2.5.3, skrecsys 0.5.0, _core built in release mode. The quality table above is deterministic and portable; the timings below are not comparable across machines or builds.

Wall clock per call. Each operation is sampled until it has spent 30s or reached its cap (1 fits, 5 recommend calls), after untimed warm-up calls (0 fit, 1 rank) so that no sample pays for a cold start; the samples columns say how many each row actually got, which is why a slow model shows fewer. The batch columns say what a single call processed: one fit covers the whole training split, and one recommend call ranks the entire catalog for every held-out user at once, so these are throughput numbers rather than single-request latency. Fit reports the spread a handful of samples can resolve; ranking is sampled often enough for nearest-rank quantiles, each of which is a call that really happened. Compare min across machines and watch max for the variance a run saw.

HSTU wins by a factor of three over the best non-sequential model here, which is the point of the table: on this split the answer depends on what a user watched most recently, and a model that reads their history as an unordered bag cannot use that. The ordering underneath is worth reading too — EASE and SLIM, the two models that solve for an item-item matrix in closed form, beat the neighbourhood models, and ALS at eight factors comes last, below even MostPopular.

For scale, the HSTU authors report HR@10 0.3097 / NDCG@10 0.1720 for HSTU on this dataset and 0.2853 / 0.1603 for SASRec. This implementation lands at 0.2871 / 0.1665: between the two, about 7% under published HSTU on HR@10 and 3% under on NDCG@10. That gap is the expected size and direction, because the relative attention bias here carries positions only — the published model also buckets the time gaps between interactions, which fit never sees. The result confirms that the block, the causal masking and the sampled softmax behave as published.

The cost is the other half of the story. One HSTU fit is 22 minutes against 33 ms for ItemKNN — some 40,000 times more compute for three times the hit rate, on a CPU and without the fused kernels the paper uses. Scoring, on the other hand, is a dot product against a precomputed state, so a fitted HSTU ranks the catalog for all 6,040 users in 42 ms, in the same range as the classical models (16–32 ms, and 83 ms for EASE).

Every score is an order of magnitude below the MovieLens table, and that is the dataset rather than the models: one held-out interaction per user over a 661k-item catalog, where MovieLens holds out ten out of 1682. R@10 equals the hit rate for the same reason — a single relevant item is either in the list or not — and P@10 is that hit rate over ten. For scale, the HSTU authors report HR@10 0.0416 / NDCG@10 0.0227 for HSTU and 0.0306 / 0.0164 for SASRec on this dataset. Those are close enough in protocol to be worth quoting — leave-one-out, full-corpus ranking, seen items filtered — but they were not produced by this harness, so compare them with the rows above with care: the candidate set here is the 660,940 items seen in training rather than the full 695,762-item space, and the rows are scored on a 10,000-user sample.

Ranking order changes completely. BM25 leads, the neighbourhood models take the top three places, and the two models fitted by gradient descent collapse: BPR scores a fifth of BM25 while recommending the head of the catalog (mean pop 754 against 53), and ALS at its default eight factors is indistinguishable from noise. Both defaults were chosen against a 1.7k-item catalog and neither has the capacity for this one; they are not evidence about the methods, only about the defaults. MostPopular is the reference baseline of a long-tailed catalog — its catalog coverage rounds to zero, because ten items out of 660,940 is what a global ranking can reach.

The timing table shows where sparsity pays. A catalog 400 times larger costs the neighbourhood models about a hundred times their MovieLens fit (0.6 s for BM25) because the item-item work follows co-occurrences rather than catalog size, and ranking 10,000 users against all 660,940 items takes 22–46 ms. The models that score every item for every user pay for the catalog instead: MostPopular inverts the usual order at 0.6 s per call, with nothing to fit and no structure to exploit, and the factor models rank in 1 s (ALS, eight factors) and 3.9 s (BPR, sixty-four). BPR is also the slowest to fit at 5 minutes, single-threaded because the entry is seeded.