diff --git a/README.md b/README.md index 5c844c2..ef03278 100644 --- a/README.md +++ b/README.md @@ -9,4 +9,5 @@ Currently, the following examples are provided: - `examples/numpy-spin`: out-of-tree benchmarks for NumPy, using `spin` to build NumPy itself; - `examples/arrow`: Arrow-shaped examples with Google Benchmark and CMake: a walkthrough of both comparison profiles (`demo`) and one of the target provider contract (`archery-local`). +- `examples/pyperf-suite`: a suite of Python functions benchmarked with `bx bench`, results stored locally. diff --git a/examples/pyperf-suite/.gitignore b/examples/pyperf-suite/.gitignore new file mode 100644 index 0000000..ea93687 --- /dev/null +++ b/examples/pyperf-suite/.gitignore @@ -0,0 +1,2 @@ +out/ +__pycache__/ diff --git a/examples/pyperf-suite/README.md b/examples/pyperf-suite/README.md new file mode 100644 index 0000000..5181911 --- /dev/null +++ b/examples/pyperf-suite/README.md @@ -0,0 +1,154 @@ +# Example: benchmarking Python functions with `bx bench` + +A runnable example of a benchmark suite. `bench_suite.py` holds five small +functions; `bx bench` runs each one under [pyperf](https://pyperf.readthedocs.io), +writes benchx result files, delivers them to a local store, and prints a table. + +## Run it + +Install benchx in editable mode from a checkout (it reads its schemas from there), with +pyperf, then run the script from the root of this repository: + +```sh +git clone https://github.com/benchx-dev/benchx.git +python3 -m venv .venv && . .venv/bin/activate # Python >= 3.11, or any environment you like +pip install -e benchx/prototype pyperf +bash examples/pyperf-suite/run.sh +``` + +The script runs `bx` and `python` from the active environment, so benchx and pyperf must +be installed there; the workers use that same `python`. Extra arguments go to +`bx bench` in step 1 and win over the config file, e.g. `./run.sh --processes 20`. +It takes about 20 seconds. Nothing outside this directory changes: the local store is +`out/store.parquet` (`paths.store` in `benchx.conf.json`), and `out/` is deleted at the start +of each run. + +## What goes in + +| Input | Where | What it does | +|---|---|---| +| `.` | first argument | The target: this directory, a working tree. pyperf reads a bare path that way. | +| `benchx.conf.json` | found by walking up from where you run `bx`; its directory is the project root | The harness (`adapter.name`), the module (`adapter.suite`, `bench_suite.py`: each function named `bench_*` or `benchmark_*` is a case), and the precision: worker processes, values per process, calibration, warmup. Here: 5 processes x 3 values, calibrate to 0.05 s per value. | +| Function names | after the target | `bx bench . bench_sorted bench_join` runs only those, any function, whatever its name. `bx bench bench_suite.py bench_sorted` still works as a shorthand: the module's git checkout is the target and its path the suite. | +| Flags | command line | `--processes`, `--values`, `--min-time`, `--loops`, `--warmups` override the file, and so does `--suite`. | +| `-e NAME=VALUE` | command line | Sets an environment variable for the workers. The workers otherwise inherit your whole environment. | + +Precision is resolved as: **flags, then the config file, then pyperf's defaults** +(20 processes x 3 values, 0.1 s calibration, 1 warmup). A level you leave out of the +file takes pyperf's default. `bx config init` writes a `benchx.conf.json` that states +the defaults, as a template to edit, and `bx config show` says where each value comes from. The resolved values are written into the work order, +so a result records the precision that applied. A bad config file is reported before +anything runs. + +The benchmark functions are timed bodies: pyperf calls each one many times, and does +the calibration (how many calls make up one value) itself. Put setup at module level. + +## The config file + +`benchx.conf.json` is the example's project file. Its `adapter.precision` is the work-order +`precision` block; `adapter.name` and `adapter.suite` say which harness and which module: + +```json +{ + "schema_version": "benchx/config/0.1.0", + "adapter": { + "name": "pyperf", + "suite": "bench_suite.py", + "precision": { + "repetitions": { + "mode": "fixed", + "levels": [{"unit": "process", "n": 5}, {"unit": "value", "n": 3}] + }, + "calibration": {"mode": "adaptive", "minimum_sample_seconds": 0.05}, + "warmup": {"mode": "count", "n_warmup": 1} + } + }, + "paths": {"store": "out/store.parquet", "results": "out"} +} +``` + +- `repetitions.levels`: `process` is the number of worker processes, `value` the values + each one records. Leave a level out and it takes pyperf's default (20 and 3). +- `calibration`: `adaptive` with `minimum_sample_seconds` lets pyperf pick how many calls + make up one value; `{"mode": "fixed", "n_iterations": N}` fixes it. +- `warmup`: `{"mode": "count", "n_warmup": N}` discards N values per process, or `{"mode": "none"}`. +- Every key is optional; pyperf's defaults (20 x 3, adaptive 0.1 s, 1 warmup) fill the rest. + Run `bx config init` in your own project to get a file that states them all. + +## What `run.sh` does + +1. `bx bench .` runs all five cases with the config file's suite and precision. +2. `bx bench . bench_env_var -e EXAMPLE_N=100000 --processes 3 --values 2` + runs one case, sets a variable for the workers, and overrides the file's process count. +3. `bx bench . bench_join --run-key example-again` runs one case a second time + under a new run key, so its series gets a second point. +4. `ls -l` and `bx head` show the local store: one Parquet file, and its latest rows. +5. `bx series` lists the series in the store. +6. `bx history` lists the points of the `bench_join` series, one per run. The script fails + if there are not two, which is what CI relies on. +7. It lists the files written. + +## What comes out + +The table, for step 1 (your numbers will differ): + +``` +case median mean ± sd min .. max n status +----------------- -------- ------------------- -------------------- -- ------- +bench_dict_build 18.29 us 18.56 us ± 680.9 ns 17.71 us .. 19.83 us 15 success +bench_join 3.585 us 3.543 us ± 272.4 ns 3.236 us .. 3.997 us 15 success +``` + +`n` is processes x values (5 x 3 = 15). Each value is the time of one call, in seconds +underneath. A `[warning]` after the status means the result carries a quality warning, +such as `env-mismatch`. + +Files, under `out/all/` and `out/env/`: + +- `__wall-time.json`: one benchx result per case (schema `measurement-result`; + with `-e` the name also carries the percent-encoded parameters, as in + `example-env_bench_env_var_%7B%22EXAMPLE_N%22%3A%22100000%22%7D_wall-time.json`): + the observations grouped by worker process, the pyperf protocol that applied, the + environment, the revision of the tree, and warnings. +- `__load-average.json`: beside each case's result, the 1-minute load + average the runner read before the case ran, a condition measurement with the same + `attempt_key` (schema §4.4). `bx series` lists its series too; `--quantity wall-time` + leaves them out. +- `workorder-.json`: the work order that was run, and `orders/order.json`, the + same order as written for you to read. Results refer to it by hash. +- `artifacts//NNNN.native.json`: pyperf's own output; `.driver.py`: the generated + script that ran the function; `.worker-env.json`: the environment a worker really had; + `.stdout` and `.stderr`. + +The local store, `out/home/store.parquet`, holds the results. `run.sh` reads it back +(your numbers and fingerprints will differ): + +``` +$ bx series --quantity wall-time +d780c9564c72 local/be8372c04606 bench_dict_build wall-time [s] est=benchx/median/v1 +e6eaa02c8309 local/be8372c04606 bench_env_var wall-time [s] est=benchx/median/v1 +4be59ef6debc local/be8372c04606 bench_join wall-time [s] est=benchx/median/v1 +... +85a6aa9c6996 local/be8372c04606 bench_env_var wall-time [s] est=benchx/median/v1 + +$ bx history 4be59ef6debc +revision order unavailable without a revision graph; listed by measurement start +d83d37b69010 2026-10-13T11:04:28.656515Z 3.32107e-06 (server) example-all:bench_join:wall-time +d83d37b69010 2026-10-13T11:04:39.045277Z 3.76987e-06 (server) example-again:bench_join:wall-time +``` + +`bench_env_var` has two series because step 2 sets `EXAMPLE_N`, which changes the +environment the case ran in. `bench_join` has one series with two points, one for each +run key. You can also `bx ingest` more result files; comparing two runs is not part of +this step. + +## Notes + +- The tree measured is the git checkout containing the module (here, this repository), as + found, uncommitted changes included. The result records the revision and whether the tree + was dirty. Because that tree is the whole repository, an uncommitted change anywhere in it, + in a doc or another example as much as in `bench_suite.py`, makes the result `dirty` and + changes its tree id; commit or stash first for a clean result. Outside git, it records a hash of the directory's contents. `--tree DIR` picks + the tree. +- pyperf prints advice when the machine is noisy; it is in `.stderr`. On a laptop, + close other programs. `python -m pyperf system tune` (needs root) quiets the CPU further. diff --git a/examples/pyperf-suite/bench_suite.py b/examples/pyperf-suite/bench_suite.py new file mode 100644 index 0000000..cf6589d --- /dev/null +++ b/examples/pyperf-suite/bench_suite.py @@ -0,0 +1,39 @@ +"""Example benchmark suite: plain Python functions, one case each. + +`bx bench bench_suite.py` runs every function named bench_* or benchmark_*. +Name functions on the command line to run only those: + bx bench bench_suite.py bench_sorted bench_join + +Each function is the timed body; pyperf calls it many times. Do your setup +at module level (it runs once per worker process, outside the timing). +""" + +import os + +DATA = list(range(2000, 0, -1)) +WORDS = [f"word{i}" for i in range(500)] + + +def bench_sorted(): + sorted(DATA) + + +def bench_join(): + ",".join(WORDS) + + +def bench_dict_build(): + {w: len(w) for w in WORDS} + + +def bench_sum_squares(): + sum(i * i for i in range(1000)) + + +def bench_env_var(): + """Reads EXAMPLE_N from the environment, to show `-e NAME=VALUE` reaching the workers.""" + int(os.environ.get("EXAMPLE_N", "100")) ** 2 + + +def helper_not_a_case(): + """Not named bench_*, so only run if you name it explicitly.""" diff --git a/examples/pyperf-suite/benchx.conf.json b/examples/pyperf-suite/benchx.conf.json new file mode 100644 index 0000000..bc59544 --- /dev/null +++ b/examples/pyperf-suite/benchx.conf.json @@ -0,0 +1,13 @@ +{ + "schema_version": "benchx/config/0.1.0", + "adapter": { + "name": "pyperf", + "suite": "bench_suite.py", + "precision": { + "repetitions": { "mode": "fixed", "levels": [{ "unit": "process", "n": 5 }, { "unit": "value", "n": 3 }] }, + "calibration": { "mode": "adaptive", "minimum_sample_seconds": 0.05 }, + "warmup": { "mode": "count", "n_warmup": 1 } + } + }, + "paths": { "store": "out/store.parquet", "results": "out" } +} diff --git a/examples/pyperf-suite/run.sh b/examples/pyperf-suite/run.sh new file mode 100644 index 0000000..3c6ecb6 --- /dev/null +++ b/examples/pyperf-suite/run.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Run the example suite with `bx bench`. See README.md for what each step does. +# +# ./run.sh uses benchx.conf.json here (5 processes x 3 values) +# ./run.sh --processes 20 extra arguments go to `bx bench` and win over the file +# +# Run it in an activated environment that has benchx installed with pyperf: in editable +# mode, from a benchx checkout (`pip install -e /prototype pyperf`). +# Results go to ./out, and the local store to ./out/store.parquet (both set in +# benchx.conf.json), so nothing outside this directory changes. +set -euo pipefail +cd "$(dirname "$0")" +bx() { printf '\033[36m$ bx %s\033[0m\n' "$*" >&2; command bx "$@"; } + +command bx --help >/dev/null 2>&1 && python -c "import pyperf" 2>/dev/null || { + echo "missing dependencies: install benchx and pyperf into the active environment (see README.md)" >&2 + exit 1; } +rm -rf out; mkdir -p out + +echo "== 1. all bench_* functions, suite and precision from benchx.conf.json" +bx bench . --run-key example-all --out out/all "$@" + +echo; echo "== 2. one function, an environment variable set for the workers, flags over the file" +bx bench . bench_env_var -e EXAMPLE_N=100000 --processes 3 --values 2 \ + --run-key example-env --out out/env + +echo; echo "== 3. the same case again under a new run key, so its series gets a second point" +bx bench . bench_join --run-key example-again --out out/again + +echo; echo "== 4. what is in the local store: one Parquet file" +ls -l out/store.parquet +bx head -n 10 + +echo; echo "== 5. its series (bench_env_var has two: the -e setting changes the environment)" +bx series --quantity wall-time > out/series.txt +cat out/series.txt + +echo; echo "== 6. the history of bench_join: one point per run" +bx series --workload bench_join --quantity wall-time > out/join-series.txt +SERIES=$(awk 'NR == 1 {print $1}' out/join-series.txt) +[ -n "$SERIES" ] || { echo "no bench_join series in the store" >&2; exit 1; } +bx history "$SERIES" > out/history.txt +cat out/history.txt +[ "$(grep -c example- out/history.txt)" -eq 2 ] || { echo "expected two history points for $SERIES" >&2; exit 1; } + +echo; echo "== 7. what was written" +find out/all out/env out/again -type f | sort | sed 's/^/ /'