Development¶

Native kernels (for example the libFM-style ALS solver) are written in Rust under rust/ and built by maturin as skrecsys._core, so development needs a Rust toolchain (rustup or your package manager). uv sync rebuilds the extension when Rust sources change.

The Rust side is a two-crate workspace. rust/kernels (skrecsys-kernels) holds the kernels and knows nothing about Python: its functions take borrowed slices and the sparse types in sparse.rs. rust/src/lib.rs is the skrecsys._core extension module, which validates the numpy arrays that come in, converts them and releases the GIL around a kernel call. The split is what lets the benchmarks link the kernels while the extension stays LTO'd; see [lib] crate-type below.

git clone https://github.com/mrk-andreev/skrecsys.git
cd skrecsys
uv sync
uv run pytest
uv run pytest -m benchmark  # quality benchmarks on real datasets (downloads data)
LIBFM_BIN=/path/to/libfm/bin/libFM uv run pytest -m benchmark -k libfm  # ALS vs libFM
uv run --group reference pytest -m benchmark -k implicit  # BM25 vs implicit
uv run pytest -m benchmark -k rectools  # EASE vs RecTools (needs network on first run)
uv run pytest -m benchmark -k dacrema  # RP3Beta and SLIMElasticNet vs their reference framework (needs network)
uv run pytest -m benchmark -k cornac   # BayesianPersonalizedRanking vs Cornac (needs network on first run)
uv run ruff check
uv run ty check
cargo test
cargo clippy --all-targets -- -D warnings
cargo bench -- --test  # run every benchmark once, without measuring

The website is built from the same README.md.j2 and docs/*.md.j2 sources as this README, one page per included fragment, and published to GitHub Pages by .github/workflows/docs.yml on every push to main. To preview it:

uv sync --group docs
uv run python scripts/build_site.py  # renders the pages into build/docs
uv run properdocs serve              # re-run the first command after editing a fragment

The release profile uses fat LTO and a single codegen unit. [lib] crate-type deliberately lists only cdylib: adding rlib back would make Cargo drop LTO silently, because it cannot link-time-optimize a unit that also emits an rlib. The kernels live in their own crate so that the benchmarks have an rlib to link against without the extension needing one; cargo build --release -v shows -C lto=fat on the cdylib and -C linker-plugin-lto on the kernels, so LTO still crosses the crate boundary.

Benchmarks¶

The tables in this README are generated, and so is the README itself: README.md.j2 is the source, with one docs/*.md.j2 fragment per section, and python scripts/render_readme.py fills in its tables, and the sentences around them, from the results stored in benchmarks/results. Edit the template, not README.md; a test fails when the two disagree.

What the benchmarks run is configuration rather than code. benchmarks/config/ holds one file per report — leaderboard.json, sequential.json, reranking.json, indexes.json — each listing its entries, a model or an index named by package and cls with its params, a version and a comment, and the datasets it runs on with the settings each is timed under; datasets.json defines the datasets they share, and may replace a dataset's own split with a splitter such as ColdStartSplit. The sequential and index reports take their models from the leaderboard's, so a model is defined once. A reranking entry names a builder in benchmarks/pipelines.py rather than a class, because its feature tables come from the dataset. Every result is keyed on a hash of everything its entry and its dataset say, which makes the workflow:

• Adding a model is adding an entry to leaderboard.json and running python benchmarks/run.py run leaderboard. That measures the new model, and nothing else.
• Editing a parameter or a setting re-runs exactly the results it touches, with no version to remember to bump.
• Bumping a version re-runs an entry after a change the config cannot see — a rewritten kernel, a fixed bug. That is why the quantized index is at v2.

status lists what is out of date. A missing result was never measured; a stale one was measured under a config that has since changed. A stale result is still reported, with a note under its table's caption, because an outdated measurement is still a measurement: it is only the run that treats it as work to do.

uv run python benchmarks/run.py status                    # fresh, stale and missing results
uv run python benchmarks/run.py run indexes --dry-run     # what a run would measure
uv run python benchmarks/run.py run indexes --dataset movielens-100k --index quantized-flat-8
uv run python benchmarks/run.py run leaderboard           # measure what is out of date, re-render README.md
uv run python benchmarks/run.py run leaderboard --dataset amazon-books --only ALS --force
uv run python benchmarks/run.py run leaderboard --no-store --set repeat=20 --set rank_repeat=5000
uv run python scripts/render_readme.py --check            # exit 1 if README.md is out of date

--no-store measures without keeping the result, which suits tighter timing runs like the one above; to change what the README reports, change the settings in benchmarks/config/leaderboard.json instead.

Each result records the host it was measured on. A run re-measures some rows and not others, so one table can hold rows from two machines; the caption under it then says which ran where, since a timing only compares against one taken on the same host.

rust/kernels/benches measures the kernels with criterion, grouped into sparse (the transpose, the parallel row builder, the accumulator), similarity (item-item kNN, cosine, RP3beta, pruning), ranking (dense and fused top-k selection), encode (id factorization, COO to CSR) and fit (EASE, SLIM, FM-ALS, BPR).

cargo bench                           # every group, roughly six minutes
cargo bench --bench similarity        # one group
cargo bench -- 'knn::'                # benchmarks whose id matches a regex
cargo bench -- --save-baseline main   # record the current numbers under a name
cargo bench -- --baseline main        # compare the working tree against that baseline

Criterion reports the change against the previous run -- or against a saved baseline, which is the reliable way to compare two commits -- and says whether it is statistically significant. The --save-baseline form on a clean checkout of main, then --baseline main on the branch, is the way to check a change that was meant to be faster. HTML reports with the distributions land in target/criterion/report/index.html.

Every input is generated from a fixed seed, and the rayon pool is pinned to four threads (or fewer on a smaller machine) so that two runs are comparable; SKRECSYS_BENCH_THREADS=n overrides it, and scaling numbers come from varying it deliberately. Comparisons are only meaningful between runs on the same machine, since the benchmarks inherit the release profile and the kernels are memory-bound.

python scripts/pgo.py rebuilds the extension with profile-guided optimization, training on a short leaderboard run. It is opt-in and not part of uv sync or the release workflow: an interleaved A/B on an Apple M4 Pro put the kernels within a few percent either way for a 3% smaller binary and roughly triple the build time, which is not a trade worth making by default. Tight numeric loops with predictable branches give PGO little to work with, but that result is specific to this workload and micro-architecture, so the script is there to re-measure with.

Micro-batching under load¶

python benchmarks/batching.py run answers when AsyncRecommender pays off. It serves one ItemKNNRecommender (20,000 users and items, 1,000,000 synthetic interactions) from a bare asyncio HTTP server, once with a recommend call per request and once through AsyncRecommender, both in the same one-thread executor, so batching is the only difference. The load comes from benchmarks/loadgen, a small Rust program, for two reasons. A Python generator would share the machine and the interpreter with the service it measures. And it is open-loop: it sends on a schedule whether or not the service keeps up, and charges each request from the moment it was due. A closed-loop generator slows down together with the service and so hides the queue that batching exists to drain.

These numbers are for this model on this machine, not for yours. How long one recommend call takes depends on the model, the catalog size, n_recommendations, filters such as exclude_interactions, and the hardware, and so do the loads at which batching starts to matter. Here a call takes about 0.19 ms, which puts the knee at 5,000 to 6,000 req/s; a model that takes 5 ms per user would hit its limit near 200 req/s, and a much cheaper one far above 6,000. As a rule of thumb, the direct service saturates near 1 / (time of one call), so measure that for your model first, then run the sweep with --rps values around it. Do not expect your break-even load, or the size of the gain, to match the table.

Measured on an Apple M4 Pro, 10 s per load after a 2 s warm-up, requests that took more than 5 s counted as failed:

What the table shows:

• Below the model's capacity, batching buys nothing and costs a little. At 500 req/s a request almost never finds another in flight, so each call carries one user and the only difference is the batcher's own overhead: a future and one more hop through the event loop. In the table it is about 0 to 0.5 ms at the median; a shorter rerun showed 0.6 ms at 500 req/s, so treat the low-load rows as a range, not a number.
• At the capacity of one call per request, it is the difference between serving and not. One user costs about 0.19 ms in recommend, so the direct service can do at most about 5,000 to 6,000 req/s on its one thread. It starts queueing at 6,000 (p95 89 ms against 2.8 ms batched) and falls over at 8,000, where requests wait seconds and most of the load times out.
• Batches grow with the load, and that is the point. Requests that arrive while a call runs share the next one, so the batch is 6.5 requests at 6,000 req/s and 14 at 16,000. A batch of 128 users costs about 1.5 ms, 0.011 ms per user against 0.19 ms alone, which is why the batched service holds a flat 1.7 to 1.8 ms median up to 16,000 req/s, the highest load measured, with the generator and the Python server taking the rest of the machine.

So batching pays when the offered load approaches what one recommend per request can serve. That depends on your model's execution time, and so does where this table's knee falls for you: the dearer a single call, the lower the load at which batching starts to matter. It costs a fraction of a millisecond when the load is low. The numbers come from one machine and one synthetic model, so read them as the shape of the curve and re-measure with your own: --users, --items, --interactions and --max-batch-size change the model and the cap, and --rps the loads. The p99 of a batched row can show a single slow batch, such as the 57.89 ms at 12,000 req/s; repeat a load before reading anything into one tail value.