From 13c09973001c386d9240ca08a6f307e0accb4ceb Mon Sep 17 00:00:00 2001 From: Evgeni Burovski Date: Fri, 6 Nov 2026 17:40:31 +0000 Subject: [PATCH] DOC: update the matmul-param README --- examples/matmul-params/README.md | 68 ++++++++++++++++++++++++++++++-- 1 file changed, 65 insertions(+), 3 deletions(-) diff --git a/examples/matmul-params/README.md b/examples/matmul-params/README.md index 5caacee..ac16c19 100644 --- a/examples/matmul-params/README.md +++ b/examples/matmul-params/README.md @@ -6,16 +6,21 @@ case, under [pyperf](https://pyperf.readthedocs.io). ## Run it -In an environment of your choice: +In a virtual environment. The workers run the first `python` on the `PATH`, so the +environment must be active, and must import pyperf, numpy and torch: ```sh +python -m venv .venv && . .venv/bin/activate pip install pyperf numpy -pip install torch --index-url https://download.pytorch.org/whl/cpu +pip install torch --index-url https://download.pytorch.org/whl/cpu # the CPU build cd examples/matmul-params export BENCHX_HOME=$PWD/out/home # optional: keep the result store here, not in ~ bx bench bench_matmul.py --out out ``` +The CPU build is a few hundred megabytes; the default wheel from PyPI also pulls in the CUDA libraries, +which this example does not use. (The timings below were taken with numpy 2.5.4 and torch 2.15.0, CPU.) + This uses pyperf's defaults (20 worker processes x 3 values, 0.1 s calibration) and takes several minutes, mostly importing torch in every worker. @@ -54,6 +59,61 @@ A regex is searched in `bench_matmul(10, 'torch')`, the function's name and the values (not their names). `--filter 'bench_matmul$'` is therefore refused, since `$` leaves no room for the arguments; the refusal says so. +## Comparing numpy and torch + +The two libraries are the two values of `xp`, so `bench_matmul` is also a +[UC-02](https://github.com/benchx-dev/benchx/blob/main/docs/use-cases/UC-02-variants.md) benchmark: the same function, run once with each +library, with the rounds interleaved. `n` is the other parameter, so each size is its own comparison and +`--vary xp` says which parameter is the one that differs. + +**With a loop of `bx bench` calls.** Each call runs one library, with `--filter`, as one side of a session; +`--round` and `--slot` place it, and `bx compare --run` compares the sides: + +```sh +for r in 0 1 2 3 4; do + bx bench bench_matmul.py --filter "'numpy'" --run-key mm --round $r --slot $((2*r)) --out out + bx bench bench_matmul.py --filter "'torch'" --run-key mm --round $r --slot $((2*r+1)) --out out +done +bx compare --run mm --profile variants --vary xp --baseline "'numpy'" +``` + +The result is one unit per size, with numpy as the baseline and torch as the contender: the effect is +torch's median time relative to numpy's, so `REGRESSED` means torch took longer, and the noise is taken +across the five rounds. The units are grouped by verdict. This is the output of the loop above with the +quick settings (`--processes 2 --values 2 --min-time 0.01`, added to both calls): + +``` +run mm profile variants baseline 'numpy' contender 'torch' + +REGRESSED (2) + bench_matmul [n=10] wall-time effect +17.19% noise 0.50% rounds=5 + bench_matmul [n=50] wall-time effect +87.08% noise 14.30% rounds=5 + +IMPROVED (1) + bench_matmul [n=100] wall-time effect -53.59% noise 1.24% rounds=5 +``` + +Here torch is slower than numpy at N=10 and N=50 and takes about half the time at N=100. The N=50 noise is high +with the quick settings, which take only a few values per case. `--baseline` is spelled as the +value is recorded, quotes included. + +One `bx bench bench_matmul.py` call is not enough. It measures all six cases in one order, with no round and no +slot: a grid of timings, not an alternation, so `bx compare --profile variants` refuses it +(`alternating-slots`: "a result has no slot…") and the message names the two ways to get an alternating run, +the loop above and the one-command form below. + +**In one command.** Measure mode runs the same loop, listing the cases once and writing one order per library +and round: + +```sh +bx compare . --profile variants --adapter pyperf --suite bench_matmul.py --vary xp --rounds 5 +``` + +It prints the same table, with the numbers of its own run (with the quick settings: +17.9%, +76.8% and +-52.7% for N=10, 50 and 100). Add the quick version's `--processes 2 --values 2 --min-time 0.01` to either form +for a rougher, faster comparison; the loop above takes about a minute and a half with them, mostly importing +torch in every worker. + ## How the parameters are declared ```python @@ -72,4 +132,6 @@ The whole body of `bench_matmul` is timed: the call to `operands(n, xp)` as well calibration or warmup call), not in every call. What every call still pays is the cache lookup, about 60 ns here: roughly 4% of the numpy time at N=10 and under 0.2% at N=100. The timings include it, for numpy and torch alike. The result array of `a @ b` is also -allocated inside the timed call. +allocated inside the timed call. In a comparison of the two libraries the lookup is a fixed cost +on both sides: it cancels in the difference but not in the percentage, so at N=10 the +percentage reads about 4% smaller than it is.