WIP: Collector imported build

README.md

@@ -7,4 +7,5 @@ represent best practices.
 Currently, the following examples are provided:
 
 - `examples/numpy-spin`: out-of-tree benchmarks for NumPy, using `spin` to build NumPy itself;
+- `examples/numpy-revisions`: comparing two revisions of NumPy, each built by `spin` in its own worktree, in one `bx compare`;
 

examples/numpy-revisions/.gitignore

@@ -0,0 +1,3 @@
+__pycache__/
+results/
+store.parquet

examples/numpy-revisions/LICENSE

@@ -0,0 +1,24 @@
+BSD 2-Clause License
+
+Copyright (c) 2026, Evgeni Burovski
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

examples/numpy-revisions/README.md

@@ -0,0 +1,129 @@
+# Demo BenchX user project: comparing two NumPy revisions
+
+Compare two revisions of NumPy in one `bx compare`. Each is built with `spin` in a
+worktree of its own, so both builds exist at once. benchx runs the benchmark on them
+alternately, round by round, in one session, and says whether the second differs from
+the first (the `revisions` profile).
+
+Compared to `numpy-spin`, which benchmarks one build at a time (the full list of
+differences is [below](#differences-from-numpy-spin)):
+
+- `bx compare` runs both sides itself, so there is no loop in the script;
+- the environment is activated once, for both sides, and there is **no `PYTHONPATH`**:
+  the provider reports the directory that holds each build's `numpy`, and pyperf puts
+  the target's directory first on `sys.path`, so each side imports its own build.
+
+## Files
+
+- `provider.py`: the target provider. For a revision of `$NUMPY` it makes a git
+  worktree under `$WORKTREES`, runs `spin build` in it, and describes the build.
+  `prepare` builds (nearly a no-op when nothing changed); `describe` builds nothing.
+- `numpy_facts.py`: how a numpy was built (BLAS, LAPACK, C compiler, SIMD). The
+  provider declares these facts, and the collector in `benchx.conf.json` checks that
+  the benchmark's process sees the same.
+- `benchx.conf.json`: the project file: pyperf and its precision, `OPENBLAS_NUM_THREADS=1`
+  on both sides, the `revisions` profile with 5 rounds, the provider, the collector.
+- `bench_linalg.py`: the benchmark, a matrix product.
+- `compare.sh`: the driver. `environment.yml`: an environment to build and run in.
+
+## Running it
+
+You need a git checkout of NumPy (it is never built in; the builds are in worktrees),
+and an environment with NumPy's build requirements, `spin`, pyperf and `bx`:
+
+```
+$ micromamba create -f environment.yml
+$ micromamba activate numpy-revisions
+$ pip install -e /path/to/benchx/prototype
+```
+
+The benchmark file is recorded as the benchmark's own source. If it is not committed, or
+has been edited, the comparison still runs and is labeled local-only (it says so before
+measuring, and `bx head` marks the results); commit this directory first for a comparison
+that is not. Then:
+
+```
+$ NUMPY=~/repos/numpy ./compare.sh v2.4.0 v2.4.3
+```
+
+The two revisions are any tags, branches or commits of `$NUMPY`, or `WORKSPACE`, which
+is the checkout in `$NUMPY` itself. That one is built in place and must have its
+submodules initialized. It may have uncommitted changes: the comparison is then labeled
+local-only, and says which side is not clean before measuring. A checkout git cannot hash,
+such as one with an unreadable untracked file, is refused. Local changes in `$NUMPY` do not
+matter for the other revisions, which are built in worktrees. The first run builds
+both (a few minutes each on a many-core machine); later runs reuse the worktrees,
+which are named after the revision and its commit, such as `v2.4.0-c5ab79c14c`. They
+stay under `$WORKTREES` (by default `$NUMPY-worktrees`); remove one with
+`git -C $NUMPY worktree remove`.
+
+## What you should see
+
+```
+run numpy-v2.4.0-vs-v2.4.3  profile revisions  baseline edd9d7c22ba2  contender 63d7b31efb1a
+  edd9d7c22ba2...: how built: {"blas":{"name":"openblas","version":"0.3.34"},"c_compiler":{"name":"gcc","version":"15.3.0"},...}
+  edd9d7c22ba2...: observed env: OPENBLAS_NUM_THREADS=1
+  63d7b31efb1a...: how built: {"blas":{"name":"openblas","version":"0.3.34"},"c_compiler":{"name":"gcc","version":"15.3.0"},...}
+  63d7b31efb1a...: observed env: OPENBLAS_NUM_THREADS=1
+
+NO CHANGE DETECTED (1)
+  bench_matmul                     wall-time  effect   +0.06%  noise   0.30%  rounds=5
+```
+
+followed by the stored results, `bx head --table`: five rows for each revision, which
+alternate. Both revisions send the matrix product to the same OpenBLAS, so "no change"
+is unsurprising. The point is what it was computed from: while this runs, the
+benchmark's processes import `v2.4.0-.../build-install/.../numpy` for the baseline and
+`v2.4.3-.../build-install/.../numpy` for the contender.
+
+## Differences from `numpy-spin`
+
+| | `numpy-spin` | `numpy-revisions` |
+|---|---|---|
+| **Goal** | Track a benchmark over a series of revisions (`track.sh`), or vary a setting for one build (`num_threads.sh`), and store each run. | Compare two revisions in one `bx compare` and get a verdict. |
+| **Command** | `bx bench @numpy ...`, once per revision or setting. | `bx compare @numpy=A @numpy=B`, with benchx running both sides. |
+| **Builds** | One build, in place in `$NUMPY/build-install`. `track.sh` checks out, rebuilds and runs each revision in turn, so two builds never exist at once. | One git worktree per revision under `$WORKTREES`, so both builds exist at once. |
+| **Provider** | Builds nothing and ignores the request. It describes the build in `$BUILD` and refuses it if it isn't the checkout's HEAD. | Handles `prepare` and `describe`. It reads the requested revision, creates the worktree, runs `spin build`, and takes a lock. It makes the same HEAD check. |
+| **Target path** | The build root, `build-install`. | The `site-packages` directory that holds `numpy`. |
+| **Getting numpy imported** | The script exports `PYTHONPATH=$BUILD/usr/lib/python3.12/site-packages/`. That works for one build, but it can't serve two. | No `PYTHONPATH`. The target's directory comes first on `sys.path`, and a stale exported one is harmless. |
+| **Collector facts** | `numpy_version`, the full `show_config`, `imported_from_target` and a summary. Fine for storing results, but the version fails `sides-agree` in a `revisions` comparison. | BLAS, LAPACK, compiler and SIMD, plus `imported_from_target`, so the two sides agree if both builds were configured alike. It imports from the working directory instead of relying on `PYTHONPATH`. |
+| **Thread setting** | `num_threads.sh` varies `OPENBLAS_NUM_THREADS` across runs, to see its effect. | Fixed at 1 for both sides, in `benchx.conf.json`. The config also sets the `revisions` profile and 5 rounds. |
+| **Output** | `bx head --table`: one run per revision. A later `bx compare` cannot pair them: it reads one run, which has to hold both sides, and `track.sh` makes a run per revision. | A verdict (`NO CHANGE DETECTED`, effect and noise) plus the table. |
+| **Extras** | The results and the store, in `results/` and `store.parquet`. | `environment.yml`, a `.gitignore` for the results and the store, `compare.sh`, and this README with expected output. |
+
+Shared: the `bench_linalg.py` suite (a copy), the LICENSE, the pyperf adapter and its
+precision settings, the project and source URI, and the provider-plus-collector
+structure with `verify: how_built` and `expect: imported_from_target`.
+
+## Notes
+
+- **The collector holds no numpy version, on purpose.** benchx matches a collector's
+  facts between the two sides of a `revisions` comparison, which may differ in nothing
+  but the revision. With the version among the facts (as in `numpy-spin`), comparing
+  two different versions fails with `sides-agree`. The facts here are the build configuration, so
+  the sides agree only if both builds were configured alike (same BLAS, same compiler,
+  same SIMD), which is also worth knowing before trusting a comparison.
+- **The collector checks where numpy is imported from, instead of which version.**
+  `imported_from_target` is true when the numpy that `numpy_facts.py` imports lives
+  under the target's directory, and `benchx.conf.json` expects it (`expect`, with
+  `on_fail: refuse`). Both sides print the same, so `sides-agree` has nothing to object
+  to, but each is computed against its own target, so the run is refused if a side would
+  import a numpy that is not its own build. With the `site-packages` directory as the
+  target that cannot happen here, because `provider.py` has checked that the package is
+  in it; the check is for a provider that reports the build root (as `numpy-spin` does,
+  with an exported `PYTHONPATH`) and for an environment with a numpy of its own on the
+  path. It emulates the benchmark's import, and the benchmark's own process is not
+  observed.
+- **The target is the `site-packages` directory, not `build-install`.** benchx's tests
+  (`test_build_import_path.py`) show on a toy package laid out like a `spin` build that,
+  with the build directory as the target and nothing exported, `import` fails in every
+  case, and that with one `PYTHONPATH` exported both sides import the build it names,
+  without the comparison noticing. Here an exported `PYTHONPATH` is harmless: a run with
+  one pointing at the baseline's build still imported each side's own numpy, because the
+  target's directory comes first.
+- **Python and the build must match.** `python` on the `PATH` runs the benchmark, so it
+  must be the Python `spin` built with, here 3.12; the environment provides both.
+- **One provider call at a time.** Initializing the submodules of two worktrees at once
+  fails on a lock in the shared `.git/config`, so `provider.py` takes a lock. `bx compare`
+  prepares its two sides one after the other anyway.
+- Tried with Python 3.12, `spin` 0.15, OpenBLAS 0.3.34, GCC 15.3 on Linux.

examples/numpy-revisions/bench_linalg.py

@@ -0,0 +1,8 @@
+import numpy as np
+
+N = 400
+a = np.eye(N)
+b = np.ones((N, N))
+
+def bench_matmul():
+    return np.matmul(a, b)

examples/numpy-revisions/benchx.conf.json

@@ -0,0 +1,62 @@
+{
+ "schema_version": "benchx/config/0.1.0",
+ "project": "numpy",
+ "source": {
+  "uri": "https://github.com/numpy/numpy.git"
+ },
+ "adapter": {
+  "name": "pyperf",
+  "quantities": [
+   "wall-time"
+  ],
+  "precision": {
+   "repetitions": {
+    "mode": "fixed",
+    "levels": [
+     {
+      "unit": "process",
+      "n": 20
+     },
+     {
+      "unit": "value",
+      "n": 3
+     }
+    ]
+   },
+   "calibration": {
+    "mode": "adaptive",
+    "minimum_sample_seconds": 0.1
+   },
+   "warmup": {
+    "mode": "count",
+    "n_warmup": 1
+   }
+  }
+ },
+ "environment_variables": {
+  "OPENBLAS_NUM_THREADS": "1"
+ },
+ "compare": {
+  "profile": "revisions",
+  "rounds": 5
+ },
+ "provider": {
+  "command": [
+   "python",
+   "provider.py"
+  ]
+ },
+ "collectors": {
+  "numpy-build": {
+   "command": [
+    "python",
+    "{root}/numpy_facts.py"
+   ],
+   "verify": "how_built",
+   "expect": {
+    "imported_from_target": true
+   },
+   "on_fail": "refuse"
+  }
+ }
+}

examples/numpy-revisions/compare.sh

@@ -0,0 +1,21 @@
+# compare two numpy revisions in one `bx compare`, each built by `spin` in a worktree of its own
+#
+# usage: ./compare.sh [BASELINE [CONTENDER]]       (tags, branches or commits of $NUMPY)
+#
+# The provider (provider.py) builds both revisions, and reports each build's
+# site-packages directory as the target, so each side of the comparison imports its own
+# numpy: there is no PYTHONPATH to set. The environment is activated once, for both.
+set -euo pipefail
+cd "$(dirname "$0")"
+
+export NUMPY=${NUMPY:-~/repos/numpy}                # a git checkout of numpy; never built in
+export WORKTREES=${WORKTREES:-$NUMPY-worktrees}     # where the provider makes one worktree per revision
+BASELINE=${1:-v2.4.0}
+CONTENDER=${2:-v2.4.3}
+
+# threads, rounds and the revisions profile are in benchx.conf.json
+bx compare "@numpy=$BASELINE" "@numpy=$CONTENDER" \
+    --suite "$PWD/bench_linalg.py" --run-key "numpy-$BASELINE-vs-$CONTENDER"
+
+# the stored results, one row per side
+bx head --table

examples/numpy-revisions/environment.yml

@@ -0,0 +1,26 @@
+# To use:
+#
+#   $ micromamba create -f environment.yml    # `conda env create` works too
+#   $ micromamba activate numpy-revisions
+#   $ pip install -e /path/to/benchx/prototype   # `bx`
+#
+# NumPy's own build requirements (its environment.yml, without the docs and lint
+# tools), plus what benchx needs: pyperf for the benchmark, pyarrow for the store.
+name: numpy-revisions
+channels:
+  - conda-forge
+dependencies:
+  - python=3.12 # the provider's builds and the benchmark must use the same Python
+  - cython>=3.0
+  - compilers
+  - openblas
+  - nomkl
+  - ninja
+  - pkg-config
+  - meson-python
+  - spin==0.15
+  - ccache
+  - git
+  - pyperf
+  - pyarrow
+  - pip

examples/numpy-revisions/numpy_facts.py

@@ -0,0 +1,62 @@
+"""Print how the numpy that this Python imports was built, as one JSON object.
+
+Both the provider (provider.py, which declares these facts as the target's how_built)
+and the collector (collectors in benchx.conf.json, which observes them in the workers'
+environment) run this script, so the two can only differ if the environment does.
+
+    blas, lapack   name and version of the library numpy was built against
+    c_compiler     name and version of the C compiler
+    simd_baseline, simd_found    the SIMD extensions numpy was built for, and detected
+    imported_from_target    whether the numpy imported here lives under the directory
+                   this runs in, the target's. An observation, not a fact about the build
+    summary        the same, for the results
+
+There is no numpy version here, on purpose. benchx matches a collector's facts between
+the two sides of a revisions comparison, which may differ in nothing but the revision,
+so facts that name the version would make every comparison fail. Here the sides agree
+only if both builds were configured alike, which is what a comparison of two revisions
+wants to know.
+
+`imported_from_target` checks which build is imported without naming it. It is true on
+both sides, so `sides-agree` has nothing to object to, but it is computed against each
+side's own target directory: it is false when the numpy that gets imported is not in the
+side's own build, such as another numpy that PYTHONPATH or the environment provides when
+the target is not the directory that holds the package. The collector's `expect` in
+benchx.conf.json then refuses the run. This emulates the benchmark's import (it puts the
+target first on sys.path, as pyperf does for the benchmark, after the suite's own
+directory); the benchmark's own process is not observed.
+"""
+
+import json
+import sys
+from pathlib import Path
+
+# The directory this runs in holds the package (benchx runs collectors in the target's
+# directory, as pyperf puts it first on sys.path for the benchmark), so import from there.
+target = Path.cwd().resolve()
+sys.path.insert(0, str(target))
+
+import numpy  # noqa: E402
+
+config = numpy.show_config(mode="dicts")
+dependencies = config.get("Build Dependencies", {})
+compilers = config.get("Compilers", {})
+simd = config.get("SIMD Extensions", {})
+
+
+def named(entry):
+    return {key: entry.get(key) for key in ("name", "version")}
+
+
+facts = {
+    "blas": named(dependencies.get("blas", {})),
+    "lapack": named(dependencies.get("lapack", {})),
+    "c_compiler": named(compilers.get("c", {})),
+    "simd_baseline": simd.get("baseline"),
+    "simd_found": simd.get("found"),
+}
+imported_from_target = Path(numpy.__file__).resolve().is_relative_to(target)
+if not imported_from_target:
+    print(f"numpy was imported from {numpy.__file__}, which is not under {target}", file=sys.stderr)
+print(json.dumps({**facts, "imported_from_target": imported_from_target,
+                  "summary": {**facts, "imported_from_target": imported_from_target}}, sort_keys=True))