From 3048efbf1b0f28e3ba7088feba38c2bacead7fab Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Tue, 27 Oct 2026 17:01:19 +0000 Subject: [PATCH 1/9] Plan: UC-04 on the design Compare two environments with separate `bx bench` calls joined by a run key. Place a run in a session with --label, --round and --slot, and describe each environment with the user's own collector, checked with verify/expect. Typed declared facts (--component, build) are deferred. Co-Authored-By: Claude Sonnet 5.5 --- Plan_UC04_on_the_design.md | 816 +++++++++++++++++++++++++++++++++++++ 1 file changed, 816 insertions(+) create mode 100644 Plan_UC04_on_the_design.md diff --git a/Plan_UC04_on_the_design.md b/Plan_UC04_on_the_design.md new file mode 100644 index 0000000..a63bc85 --- /dev/null +++ b/Plan_UC04_on_the_design.md @@ -0,0 +1,816 @@ +# Plan: UC-04 on the design + +Oct 27, 2026. + +This plan makes UC-04 work with what the design documents already +specify, and adds as little new machinery as possible. + +## Goal + +Make UC-04 work: compare two environments that differ in one dependency +(or build option), measured by separate `bx bench` calls in separate +environments, joined by a run key. + +## What UC-04 needs, and what is missing + +UC-04 holds the code, the commit and the machine fixed and varies the +environment: `numpy==1.26.4` against `numpy==2.0` under pandas, or two +builds of one source tree. Two interpreters cannot hold both numpy +versions, so no single invocation can do it. The user prepares both +environments, runs the benchmark in each, alternating round by round +under one session, and then compares. + +**What already works.** +- The comparator's `environments` profile pairs the sides' attempts by + round, and checks that the sides alternate by slot, that attempt counts + are equal, that everything but the varying coordinate agrees, and that + the sides differ. +- A label names each side (`provenance.labels`), and the round and slot + flow from the order into each result's `procedure`. +- `bx compare` in measure mode alternates two targets over rounds under + one run key. Both sides then run in the one environment `bx` was + started in, which fits build-option comparisons (the archery example) + but not a dependency that cannot share a process. + +**What is missing.** +1. **A separate runtime environment per side.** Three ways to provide it: + the runner enters each side's declared `activation`; the workbench + starts the runner inside each side's activated environment; or the + user's own script activates each environment and calls `bx` once per + order. This plan takes the third. It is UC-04's own sketch, and the + design makes the user responsible for preparing and entering the + environment, with the alternation a loop in the calling script + (`benchmark-environments.md` §2.2 principle 1, §5.7). benchx then + executes nothing it was not asked to. +2. **A way to place a run in a session (gap 1).** The sketch passes a + label, a session and a round on every call. `bx bench` takes + `--run-key` (the session) but no label, round or slot, and `bx run` + takes only an order file. The order builder that `bx compare` uses + already accepts all three, so the work is exposing them. +3. **A way to tell the sides apart (gap 2).** `sides-differ` refuses a + comparison in which nothing recorded differs. Two venvs over one + checkout record the same subject and the same environment: one + benchmark run from two identical venvs gave identical + `environment.system` and `subject`, and a numpy difference would be + invisible in the same way. The declared texts (`how_built`, + `activation`) travel in `provenance.info`, which is never identity. + The sides need a recorded, machine-readable dependency fact. +4. **The minimum as the estimator.** UC-04 compares the minimum timing + per environment. The comparator uses a paired median. This is shared + with other use cases, and the plan leaves it as a follow-up. + +Items 2 and 3 are the two gaps the work packages close (gap 1 and +gap 2). Item 1 is a choice the plan makes, and item 4 is deferred. + +## Design decisions + +Each decision follows the design documents. The reason is given with it. + +1. **Record a declared-vs-detected mismatch; never gate the run on it.** + - The detected value is recorded, the declaration stays in the work + order, and the result carries a warning (`benchmark-environments.md` + §3.4; `benchmark-result-schema.md` §4.2). + - A contradiction "is recorded as it is and never gates the run" + (`benchmark-environments.md` §2.2, principle 2). Gating is the + `verify` tier of an environment policy, per rule + (`runner-schema.md` §4), and the prototype defers policy. + - The prototype's collector check (`verify`, `expect`) gives + `collector-mismatch`. With `on_fail: warn` it records and the run + proceeds, which matches this decision. Its `refuse` default is a + documented departure (`prototype-design.md` §3), so the UC-04 + example uses `warn`. + - So there is no `on_mismatch` setting. + +2. **The user's collector describes the environment; benchx hard-codes no + environment kind.** + - The design's plugins are "read-only describers that know one build + system, package manager, or accelerator stack" + (`benchmark-environments.md` §3.3). A context collector is that + interface (`harness-adapter.md` §5.1): a command the user names, + which prints one JSON object of facts. + - Whatever extracts the facts is the user's: `conda list --json`, + `pixi list`, `pip freeze`, `numpy.show_config()`, `module list`, + `docker inspect`. benchx needs to know none of them. + - The facts land under `environment.system.` + (`harness-adapter.md` §5.1), which is in the `environments` axis, so + a difference makes the sides differ and shows in the comparison. + - A declaration can be checked against the collected facts: `verify: + how_built` and `expect` (see decision 1). So the need in gap 2 is + met with detection, a record in the results and a check, for any + environment kind, without detection built into benchx. + +3. **Collector output stays under `environment.system.`.** + - `harness-adapter.md` §5.1 puts collected facts there, with the + collector's name as their source. Moving one into the subject is a + placement rule (`benchmark-environments.md` §3.4), which is a + project decision, not something a collector's output keys decide. + +4. **The caller sets slot and round.** + - Alternation across interpreters "is a loop in the calling script + ... pass `round` and `slot` through" + (`benchmark-environments.md` §5.7). `round` is "set by the run + author" (`runner-schema.md` §3). benchx holds no queue (§5.3) and + wraps no measurement loop (§2.2, principle 4). + - The comparator already checks alternation, equal attempt counts and + that the sides differ (`benchmark-result-schema.md` §5.5), so wrong + numbering is caught there. + - So `bx bench` gets `--label`, `--round` and `--slot`, with no lock + and no slot counter. + +5. **Typed declared facts are deferred: `--component`, `--build-option`, + and the runner's `components` and `build`.** + - The design has a typed home for them. `components[]` and `build` are + work-order fields (`runner-schema.md` §3) that land in + `subject.components` and `subject.configuration`, and the prototype + refuses both (`DEFERRED` in `runner.py`). + - UC-04 does not need them. It needs the sides to differ in a recorded + fact, and a way to check a declaration against what is installed. + Decision 2 gives both for any environment kind. + - What they would add: the design's home for dependency versions + instead of `environment.system`; a declaration on each call, with + no provider or sidecar; no script for Python packages; and the + design's own warning names. What they cost: most of the new code + (a package manifest, a reconcile step, name-only resolution, new + flags) and two schema decisions. + - So they are a follow-up. See "Potential follow-ups". + +6. **No separate "verified but not identity" category.** + - Which facts split history is the identity policy's choice, not a + property of the fact (`benchmark-result-schema.md` §4.3; + `benchmark-environments.md` §3.4, last paragraph). + - So keeping a fact out of identity, as numpy-spin tracking needs, + is a policy question (see follow-ups), not a new kind of fact. + +7. **Per-side declared values come from the target description or the + setup file.** + - A collector's `verify: how_built` checks the collected facts + against the `how_built` object of the target description + (`prototype-design.md` §3). The check compares key by key from the + top level (`collectors._differences`), so the declared object + mirrors the structure the collector prints: a collector that prints + `{"summary": {"numpy": "2.0.2"}}` is checked with + `{"summary": {"numpy": "2.0.2"}}`. + - Each side has its own description: a provider configuration per + side (`bx bench @a`, `bx bench @b`), or a sidecar next to a build + directory (`system-decomposition.md` §3.4). `expect` lives in the + config, which both sides share, so it cannot differ per side. + - The setup file (`.benchx/setup.json`, `BENCHX_SETUP_FILE`) is for + declared facts when the party that prepares an environment is not + the one that runs benchmarks in it: in an `environments` run, "each + environment sets `BENCHX_SETUP_FILE` to its own file" + (`harness-adapter.md` §5.1). It is implemented. Its facts land + under `environment.system.setup`, in the axis, and are matched as a + whole because the prototype has no placement rules. + +## What the design already gives UC-04 + +Each item: the need, the design mechanism, the prototype today. + +- **Sides named by an environment label.** + Work order `labels` → `provenance.labels`. + Today: `bx compare --label` works; **`bx bench` has no `--label`**. +- **Interleaving over rounds, one session.** + `run_key`, `round`, `slot`, the caller's loop (§5.7). + Today: the fields flow to results; **`bx bench` cannot set them**. +- **A machine-readable varying coordinate.** + The `environments` profile axis: `subject.configuration`, + `subject.components`, `environment.system` (`compare.py`). + Today: implemented. +- **Environment facts, detected.** + The user's collector → `environment.system.`, in the axis. + Today: implemented (PR #71). +- **A check of a declaration against the collected facts.** + `verify: how_built` and `expect`; `collector-mismatch` with + `on_fail: warn`. + Today: implemented, as a documented departure from the design. +- **Declared per-environment facts.** + Setup file → `environment.system.setup` (`runner.py:177`), in the axis. + Today: implemented. Nobody has used it for this. +- **Typed pinned dependencies and declared build configuration.** + Work-order `components[]` and `build` + (`benchmark-environments.md` §8). + Today: refused (`DEFERRED`). Deferred by this plan. +- **Detected CMake options.** + The runner reads `CMakeCache.txt`. + Today: implemented (archery example). +- **Invariants: equal rounds, alternation, sides differ.** + Comparator, schema §5.5. + Today: implemented. + +**Consequence.** Gap 2 needs no new code. Either each venv sets +`BENCHX_SETUP_FILE` to a file that says `{"numpy": "2.0.0"}` (declared), +or the project names a collector that prints the dependency's version +(detected, and checkable). Both land in `environment.system`, which is in +the axis, so `sides-differ` should pass. This follows from reading the +code and has not been run, because `bx bench` cannot pass labels yet; the +tests cover it. + +## Design + +One rule from `benchmark-environments.md` §3.4 settles how a declaration +and a detection meet: + +> A coordinate holds the detected value where one exists. A declared +> value fills it only when nothing can be read back, with a warning. A +> disagreement keeps the detected value, keeps the declaration in the +> order, and adds a warning. No fact gates the run. + +Where each fact is declared, detected and stored: + +- **Dependency versions and other environment facts** (BLAS vendor, + install method, image digest). + Declared in the target description's `how_built` object, or the setup + file. Detected by the user's collector. Stored in + `environment.system.`, or `environment.system.setup` for the + setup file. +- **Build configuration of a CMake build directory.** + Detected by the runner's CMake reader. Stored in + `subject.configuration` (unchanged). +- **`how_built`, `activation`, `shell`, provider.** + Declared in the target description. Not detected. Stored in + `provenance.info`, never identity (unchanged). + +Warnings: `collector-mismatch` when a check fails and `on_fail` is +`warn`, and `collector-failed` when a collector cannot run. The design's +`requested-value-mismatch` and `build-config-unverified` come with the +typed-facts follow-up. + +## Work packages + +A and B are independent and give a working UC-04 with declared facts. C +adds detection and verification, with no new code. D is tests. + +### A. Place a run in a session (gap 1) + +- `cli.py`, `bx bench`: `--label NAME=VALUE` (repeatable), `--round N`, + `--slot N`. The order builder already takes `labels`, `round_` and + `slot` (`orders.py:30`), and so does the helper that builds a side's + order (`session.py:117`). `session.bench_target` and its call to + `_run_side` pass none of them today, although the compare loop does, + so thread them through from `_common`. No lock, no counter, no + `--interleave`. +- When a slot is given, write the order to `orders/order-.json`, + as `compare_targets` does. `bx bench` writes `orders/order.json` per + run key, so the loop's calls would otherwise overwrite each other's + order file. +- `--label` means different things in different commands: in `bx bench` + it is `NAME=VALUE` and names the side of one call; in measure-mode + `bx compare` it is `NAME=BASE,CONTENDER`; in read mode it is the name + alone. The `--help` text of each says so. +- `bx run` is unchanged. +- Refusals at the command line, before any order is built: + - `--label`, `--round` or `--slot` without an explicit `--run-key`. + The default run key is a timestamp per call, so two calls would + never share a run. + - `--label` or `--round` without `--slot`. The ingest key gets its + `:slot-N` suffix only when a slot is set (`runner.py:49`), so two + sides in one run would share an ingest key, and their differing + payloads would conflict at ingest. + - a duplicate label name. + + `--no-ingest` is allowed (nothing then joins the run, which is the + caller's choice). +- The comparator already refuses badly formed runs: non-alternating + slots, unequal attempts, sides that do not differ. Improve those + messages to say what to change ("run the sides in the same order each + round"). For `sides-differ`, the message says the sides differ only in + the label, and to describe the environment under test with a collector + or a setup file per environment. The whole package list is not hashed + into identity by default: that would start a new series for every + incidental package change. +- Show warnings in the comparison. The comparator collects the warning + codes (`collector-mismatch`, `collector-failed`) from each side's + results and lists them beside the facts that differ. The verdict does + not change, and the comparison is not marked `local_only`. + `comparator.md` §5 already lists quality caveats in the output, so + this needs no design change. +- The caller's loop, as the design says (§5.7), is documented in the + example. A helper that prints the next slot is a potential follow-up + (see below). + +### B. Declared per-environment facts (gap 2, declared half) + +- Document the setup-file route for `environments` runs, with a + two-venv example. No code. +- The example keeps each setup file outside the tree, or gitignores + `.benchx/`. An untracked file inside the tree changes the tree id and + marks the tree dirty (`harness-adapter.md` §5.1). + +### C. The collector route (gap 2, detected half) + +Documentation and an example. No new code. + +- The two-venv demo (see "Demo: two venvs, two numpy versions"): + `benchx.conf.json` names a collector, `deps`, whose command prints + `{"summary": {"numpy": ""}}`, read from the interpreter under + test. The loop of package A runs it once per side. +- Per-side checking: each side's provider configuration returns a + description whose `how_built` is an object that mirrors the + collector's output, `{"summary": {"numpy": "2.0.0"}}`. The collector + has `verify: how_built` and `on_fail: warn`. A fact that + disagrees gives `collector-mismatch` on every result, the run + proceeds, and the collected value is what is recorded. +- Document the contract: the command runs once per order, in the + environment the user entered, outside the timed region. The summary is + what lands in the results, so it should print the same value for the + same environment, with no paths, timestamps or unsorted lists + (`harness-adapter.md` §5.1, last paragraph). +- Document that a check's default `on_fail` is `refuse`, a departure + recorded in `prototype-design.md` §3 and `harness-adapter.md` §5.1. + The example sets `warn`. Leave the departure alone; revisit it when + the environment policy lands. +- The collector output's sha256 stays in identity (`collectors.py:104`); + that is a follow-up. +- Collectors run commands. `bx bench` and `bx compare` run the project's + own, and `bx run` needs `--allow-collectors`. + +### D. Tests + +Case, then expected result. + +- `bx bench --label env=a --round 0 --slot 0`, then b/0/1, a/1/2, b/1/3 + → the order carries the fields; results carry `provenance.labels`, + `procedure.round`, `procedure.slot`. +- `--label env=a` without `--slot` → refused, naming `--slot`. +- `--label`, `--round` or `--slot` without `--run-key` → refused. +- Slots a, a, b, b → fails `alternating-slots`. +- Unequal round counts → fails `equal-attempts`. +- Two sides, setup files differ in `numpy` → + `bx compare --run S --profile environments --label env --baseline a` + passes `sides-differ`. +- Same, setup files identical → fails `sides-differ`; the message names + where the sides can differ. +- A collector whose summary differs between the sides (`numpy`, or + `blas: mkl` against `blas: openblas`) → `sides-differ` passes, and the + comparison shows the difference under `environment.system.`. +- Same summary on both sides → fails `sides-differ`. +- `verify: how_built` with a declaration that matches the collected + summary → no warning on any result. +- `verify: how_built` with `on_fail: warn`, and a declared value that + disagrees → every result carries `collector-mismatch`, the run + proceeds, and the collected value is recorded. +- The same check with the default `on_fail` → the order is refused + before any case. +- Comparison where one side has `collector-mismatch` → the output lists + the warning beside the differing fact; the verdict and `local_only` + are unchanged. +- Plain `bx bench` → unchanged. +- End to end, two venvs differing only in a tiny pure-Python package → + verdicts computed; skipped without `uv`. +- CI runs the numpy-venvs demo (see the demo section). + +## Non-Python environments + +UC-04 §2 includes build systems, so the plan does not depend on any +one package manager or environment kind. The environment may be a venv, +conda, mamba, pixi, environment modules, Nix or a container. The user +prepares and enters it, and benchx hard-codes none of them: + +- **Any environment kind: a collector.** The user-side hook for + describing an environment is a context collector + (`harness-adapter.md` §5.1), named in `collectors` in + `benchx.conf.json`. It is a command that prints one JSON object of + facts, for example from `conda list --json`, `pixi list`, + `module list` or `docker inspect`. benchx never needs to know the + tool. + - It runs once per order, before the cases, in the harness's working + directory and environment, so under `conda run -n a` or + `pixi run -e a` it sees that environment. + - The full output is kept as an artifact. The results carry + `environment.system.`: the object's `summary` (if it has one) + and a sha256 of the full output. + - `environment.system` is in the `environments` axis, so a difference + in the summary, such as the BLAS flavour (`mkl` against `openblas`), + makes the sides differ and shows in the comparison. + - A collector that prints the whole sorted package list makes any + package difference count. That is the project's choice, and it + covers undeclared dependencies for projects that want it. + - A declaration can be checked against the collected facts + (`verify`, `expect`; see decision 7). + - Limits: the facts are not typed `components`. The sha256 of the full + output is in identity (see the follow-up on collector output), so + collectors should print the same value for the same environment. +- **Containers, libraries outside Python** (Boost, OpenSSL, BLAS). The + same collector, or a key in the setup file. A declared image digest in + the setup file lands under `environment.system.setup`, in the axis, so + two images differ there. +- **CMake.** Detection exists (`CMakeCache.txt` into + `subject.configuration`). A declared `build` is part of the + typed-facts follow-up (the archery README says the prototype refuses + it today). +- **Single-process comparison** (`bx compare @a @b`, as in + archery-local). One call runs both sides in the one environment `bx` + started in. A collector runs in that environment for both sides, and + cannot tell them apart. One setup file cannot differ between them + either (its default location is under the source directory, which both + sides share). So a dependency difference between the two sides of one + call has no route in this plan, apart from CMake-detected options. + Declaring `build.options` in each side's target description is the + route, and it belongs to the typed-facts follow-up. archery-local's + `deps-old` / `deps-new` case waits for it. + +## Design-doc changes + +Few documents change, because the plan uses mechanisms the design +already has. + +- **`UC-04-cross-env.md` is kept as it is.** Its spellings differ from + the prototype's. The map from one to the other lives next to the + two-venv example, in `examples/numpy-venvs/README.md` (see "Demo: two + venvs, two numpy versions"), which links to the use case. +- **`prototype-design.md`.** Record that `bx bench` takes `--label`, + `--round` and `--slot`, and describe the collector route for UC-04. + +`benchmark-environments.md`, `harness-adapter.md`, `runner-schema.md`, +`comparator.md`, `system-decomposition.md` and the result schema do not +change. + +## numpy-spin (benchx_examples PR #1) + +This section is informational. The plan makes no change to `benchx_user` +(the local checkout of `benchx_examples`) or to the numpy-spin demos. + +The PR has three demos. Only one is UC-04-shaped. + +- **Thread settings (`num_threads.sh`).** The same benchmark under three + run keys, with `OPENBLAS_NUM_THREADS` set to 10, to 1, and unset. It + runs no comparison. Left as it is; this plan does not change it. The + acceptance test for packages A and C is the two-venv demo below. +- **Build check (collector).** Gating a run on numpy's build facts. + Unchanged; works through PR #71's `verify` today. Not UC-04. +- **Revision tracking (`track.sh`).** UC-03-like history across + revisions. Separate follow-up. It needs the numpy version *out* of + identity, which is an identity-policy decision, not a fact-placement + one. + +**Threads.** `num_threads.sh` runs the benchmark three times, under +three run keys, and prints `bx head --table`. It sets no label, round or +slot, so nothing joins the three runs into a comparison. The variable is +set in the shell, not with `-e`, so it is the inherited environment of +the measured process and lands in `environment.system.process.environment` +(observed). A variable the runner sets would be a workload parameter +instead (`benchmark-result-schema.md` §4.2). With package A the same +three runs could be a UC-04 comparison, because that field is in the +`environments` axis. The schema's own view of a thread sweep is a +`variants` comparison within one suite, which the prototype does not have +(`compare.py` has `revisions` and `environments` only). The plan changes +none of this. Moving the variable to `protocol.threads` is for *enforced* +caps (`threads.max`, environment policy, deferred). It would not apply to +an inherited one. The flags of package A are additive, so the script +keeps working as it is. + +**Tracking.** Keeping the numpy version out of identity is the identity +policy's job, not a property of the fact (design decision 6). The +prototype has one fixed policy (`identity.POLICY_DATA`). Either the +follow-up adds a project-level identity policy (an ignore list in +`benchx.conf.json`), or the collector's sha256 leaves identity so that +numpy's version lives only in the collector's artifact, checked by +`verify`. The second option needs the sha256 change: while the sha256 +of the full output is in identity (`collectors.py:104`), any collector +fact that changes splits the series, `verify`-only or not. This is not +part of UC-04. Both options are deferred (see follow-ups). + +## Demo: two venvs, two numpy versions + +A separate demo, at `prototype/examples/numpy-venvs/`, beside +`pyperf-bench`. It does not touch the numpy-spin demos. It is the +acceptance test for packages A and C: UC-04 as its own sketch has it, +two environments that differ only in numpy, driven by a shell script. + +Files: +- `README.md`: what it shows, what to install, how to run it, what to + expect, and the map from UC-04's spellings to the prototype's (below). + numpy 1.26.4 has wheels only up to Python 3.12, so the demo asks for + Python 3.11 or 3.12. +- `versions.env`: `NUMPY_A=1.26.4` and `NUMPY_B=2.0.2`, read by the + script and by the provider, so the two never disagree. +- `benchx.conf.json`: project name, the pyperf adapter with suite + `bench_suite.py` and a small precision, `paths` into `out/`, a + `provider` (`./provider.sh`), and the collector: + `"collectors": {"deps": {"command": ["python", "{root}/describe_env.py"], + "verify": "how_built", "on_fail": "warn"}}`. +- `bench_suite.py`: three numpy functions, such as a matrix product, a + sort and a sum. +- `describe_env.py`: prints `{"summary": {"numpy": numpy.__version__}}` + and nothing else, with no paths or timestamps. +- `provider.sh`: answers `describe` and `prepare` with the same target + description and builds nothing. The target is this directory as a + working tree. `how_built` is an object that mirrors the collector's + output, `{"summary": {"numpy": ""}}`, with the version for + configuration `a` or `b` from `versions.env`. +- `run.sh`: the driver, below. +- `.gitignore`: `.venv-*/` and `out/`. Untracked venvs inside the tree + would mark it dirty, so the comparison would be `local_only`, and + slow the tree hashing. benchx leaves only the store, the results and + the `--out` paths out of the tree identity. +- `out/`: generated. + +The `python` in the collector command is found on `PATH`, so the script +enters each environment the way a user does, by sourcing the venv's +`activate`, in a subshell for each call. Each call then sees exactly one +environment, and the loop's own shell never holds one. Activation also +resets `VIRTUAL_ENV`, which matters under `uv run` in CI, where it points +at the outer environment. Activate scripts can trip `set -u`, so the +script turns it off around the `source`. benchx executes nothing for any +of this. + +`run.sh`: + +```bash +#!/usr/bin/env bash +# UC-04 with two venvs: the same code, numpy 1.26.4 against numpy 2.0.2. +set -euo pipefail +cd "$(dirname "$0")" +. ./versions.env +ROUNDS=${ROUNDS:-5} +rm -rf out .venv-a .venv-b; mkdir -p out + +# Run a command with one venv activated, in a subshell. +in_env() { local venv=".venv-$1"; shift + ( set +u; . "$venv/bin/activate"; set -u; "$@" ); } + +# 1. The user prepares both environments. benchx builds nothing. +for side in a b; do + v=NUMPY_${side^^} + uv venv -q --python "${PYTHON:-3.12}" ".venv-$side" + uv pip install -q --python ".venv-$side/bin/python" \ + "numpy==${!v}" pyperf -e ../.. +done + +# 2. The caller's loop: alternate the sides, round by round. +S=numpy-$(date +%s); slot=0 +for round in $(seq 0 $((ROUNDS - 1))); do + for side in a b; do + v=NUMPY_${side^^} + in_env "$side" bx bench "@$side" \ + --run-key "$S" --label "env=numpy-${!v}" \ + --round "$round" --slot "$slot" --out "out/$S" + slot=$((slot + 1)) + done +done + +# 3. Compare. The sides differ in the collected numpy version. +in_env a bx compare --run "$S" \ + --profile environments --label env --baseline "numpy-$NUMPY_A" \ + | tee out/compare.txt +grep -q "$NUMPY_B" out/compare.txt # the differing fact is shown + +# 4. The check. Declare version b, run in venv a: the collector disagrees. +in_env a bx bench @b --run-key "$S-mismatch" \ + --label "env=wrong" --round 0 --slot 0 --out "out/$S-mismatch" +grep -rq collector-mismatch "out/$S-mismatch" +``` + +What it should show. This is the expected behaviour, not an output +captured from a run: +- Step 3 prints the comparison for the two sides, with the differing + fact (`environment.system.deps.numpy`, `1.26.4` against `2.0.2`) and + a verdict per benchmark. +- Step 4 prints results that carry `collector-mismatch`. The run + proceeds, because the collector has `on_fail: warn`, and the recorded + value is the one the collector read. +- Dropping the `--label` difference, so that both sides use the same + venv, makes step 3 fail `sides-differ`, with the message that says to + describe the environment under test. + +The script exits nonzero unless the comparison shows the differing fact +and the mismatch run carries `collector-mismatch`, so it can run in CI. +It cannot run until package A has landed, because `bx bench` has no +`--label`, `--round` or `--slot` yet. The CI step below is added in the +same change as the demo, after package A. + +**Spelling map.** The demo's README carries the map from UC-04's +spellings (`docs/use-cases/UC-04-cross-env.md`, which is kept as +written) to the prototype's, and links to the use case. `run.sh` above +is written in the prototype's. The map: + +- `benchx new-session` (`SESSION=$(...)`) → no command. The session is + the run key, chosen by the caller: `S=numpy-$(date +%s)`. +- `--session-id "$SESSION"` → `--run-key "$S"`. It lands in + `provenance.run_key`. +- `benchx run run_benchmark.py` → `bx bench`. With a module and + functions, `bx bench MODULE.py [FUNCTION ...]`; with a prepared target, + `bx bench @CONFIG --suite NAME`. `bx run` takes an order file, not a + script. +- `--env-label numpy-1.26.4` → `--label env=numpy-1.26.4`. The field + `env_label` is `provenance.labels.env`. +- `--round "$round"` → `--round N` and `--slot K`. UC-04 uses one number + for two things. The prototype separates the round, which pairs one + attempt of each side (`procedure.round`), from the slot, the realized + position in the interleaved sequence (`procedure.slot`). UC-04's + `interleave_position` is the slot. The caller's loop keeps the slot + counter. +- `benchx compare-environments --session-id S --baseline A --variant B` + → `bx compare --run S --profile environments --label env + --baseline A`. The profile is UC-04's `cross-environment-comparison`. + There are exactly two sides, so the variant is the other one. +- `.venv-baseline/bin/benchx` → the `bx` of that environment, such as + `.venv-a/bin/bx`. +- The output table (`benchmark_name`, `env`, `time`, `vs. baseline`) → + the comparison output: verdict groups with the effect, the noise and + the rounds, and the facts that differ. +- UC-04's `session_id`, `env_label` and `interleave_position` fields → + `provenance.run_key`, `provenance.labels.env` and `procedure.slot`. + The schema adds no fields for a profile + (`benchmark-result-schema.md` §5.5). + +**CI.** The `prototype tests` workflow (`.github/workflows/tests.yml`) +already runs `examples/pyperf-bench/run.sh` and +`examples/archery-local/run.sh` after the tests, on a matrix of ubuntu +and macOS with Python 3.11 and 3.13. The plan adds a step for this demo: + +```yaml +- name: Run the numpy-venvs example + if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' + shell: bash + run: >- + ROUNDS=3 uv run --extra test --python ${{ matrix.python-version }} + bash examples/numpy-venvs/run.sh +``` + +- The demo makes its own venvs with `uv venv --python 3.12`, so the + matrix Python only runs the outer script. Running it in every matrix + cell would repeat the same work, so the step runs in one cell. +- The step needs network access, for the numpy wheels and pyperf, which + the runners have. `ROUNDS=3` is the fewest rounds that give a verdict + (`compare.min_rounds` defaults to 3), and keeps the run short. +- The workflow's `paths` filter already covers `prototype/**`. +- The venvs and `out/` are git-ignored. + +## Order of work + +Tests (package D) land with the package they test. + +1. A, with its tests from D: the flags, the refusals, the order file per + slot, the comparator's messages and warnings. Then B, with the + setup-file tests (the two sides differ, the two sides are identical). + UC-04 works, declared only. +2. C, with its tests from D: the collector tests, the two-venv demo and + its `run.sh`, and the CI step, all in one change. The demo needs the + flags of package A, so this follows step 1. UC-04 then works with + detection and a check. +3. The design-doc changes (`prototype-design.md`). + +## Potential follow-ups + +Not part of this plan. Build only if the need shows up in practice. + +- **Typed declared facts: `--component`, `--build-option`, and the + runner's `components` and `build`.** Deferred (decision 5). The flags + would fill the work order's `components[]` and `build.options` + (`runner-schema.md` §3), which land in `subject.components` and + `subject.configuration`. A name with no revision means "detect it". + - Why deferred: UC-04 needs the sides to differ in a recorded fact and + a check of a declaration. A collector with `verify` gives both, for + any environment kind. + - What it adds: the design's home for dependency versions; a + declaration on each call, with no provider or sidecar; no script for + Python packages; and the design's warning names with no gating. + - What it contains: + - The snapshot reads installed versions with `importlib.metadata` + (`benchmark-environments.md` §3.1, Runtime) and records the + inventory as a manifest artifact. That is tied to Python, not to + the environment manager, so it works the same in a venv, conda, + mamba or pixi. No manager's own metadata is read. + - A reconcile step for the rule in the Design section: both equal, + recorded; both differ, detected recorded with + `requested-value-mismatch`; only detected, recorded; only + declared, recorded with `build-config-unverified` (and a sibling, + `component-unverified`, for components, added to + `benchmark-environments.md` §3.4). + - A name-only component is resolved by `bx bench` when it builds the + order, because the work-order schema requires `revision` and + `bx bench` writes resolved orders. If nothing can be detected, the + request is refused with a message that says to give a revision. + This is an unsatisfiable request, refused visibly + (`benchmark-environments.md` §2.2, principle 2), not a gate on a + contradiction. A remote runner would have to fill the version in + itself and record the completed order as the artifact that results + cite (`runner-schema.md` §3.2). + - `source` takes a package URL (`pkg:pypi/numpy`, `type: pypi`) or a + repository URL; `revision` is the version of a released package + or a commit id. `runner-schema.md` §3 says so. + - The flags: when a flag and a config value name the same thing, the + flag wins (the project's usual order is flag, file, default). + - `build.options` in each side's target description. It is the route + for a single-process comparison, and for archery-local's + `deps-old` / `deps-new` case. + - An optional `placement` block in `benchx.conf.json` that maps a + collector name and key, or a setup-file key, to its destination + with the collector (or `declared`) as source + (`harness-adapter.md` §5.1 says placement rules exist; the + prototype has none). + - Cost: most of the code in the first version of this plan, and two + schema and documentation decisions. +- **Move the numpy-venvs demo to `benchx_examples`.** That repository + (checked out locally as `benchx_user/numpy-bench`) holds worked + examples of using benchx, such as `examples/numpy-spin`, and the + demo's own audience is a user, not the prototype's tests. Moving it + would change a few things: + - It installs benchx from a released version or a git ref, not with + `-e ../..`. + - Its README is listed in that repository's README. + - The CI step in this repository goes. That repository has no CI + workflow today, so it would need one to keep the demo running. + - `versions.env` and the provider move with it, and nothing in + `prototype/` depends on them. + It is deferred because the demo has to run against the prototype's + own code while package A is being built, and because it needs + `benchx_examples` to settle how it pins a benchx version. +- **`bx session next`.** A command that reads the store and prints the + next `slot` and `round` for a run key and label, for example + `bx session next --run-key S --label env=a`. It would help when the + two sides are not launched from one script: a run resumed after a + crash, or sides started from different shells or CI steps. It is left + out because the design makes the loop the caller's + (`benchmark-environments.md` §5.7) and `round` is set by the run + author (`runner-schema.md` §3). A helper that reads the store also + needs locking: two callers can race for the same slot, and a crashed + call leaves a gap that looks like a skipped slot. The comparator + already catches wrong numbering (`alternating-slots`, + `equal-attempts`). +- **A label-name check at bench time.** Calls in one run can use + different label names by mistake, for example `--label env=a` and + `--label environment=b`. Both calls succeed and ingest. Today the + comparator catches it, but only at compare time, with "a result has no + /provenance/labels/env", after all the measuring is done. Two steps, + smallest first: + - Make the comparator's message name the label keys it found in the + run ("found `env`, `environment`"). + - Have `bx bench` warn, not refuse, when its label names differ from + those already stored for the run key. This reads the store before + measuring, and cannot see `--no-ingest` runs or another store, so it + belongs with the session bookkeeping above. +- **A project identity policy.** The prototype has one fixed identity + policy (`identity.POLICY_DATA`), pinned by a test to its name, so a + project cannot say which facts must not split its series. The numpy + tracking demo needs this: numpy's own version is already the axis (the + subject revision), but the collector records it too, so each revision + becomes its own series. Options: an ignore list of JSON pointers in + `benchx.conf.json`, added to the built-in policy, with the identity + schema name changing with the list; or a separate file following + `identity-policy/0.1.0`, which is closer to the design. Editing the + policy starts new series for results already stored. Not UC-04: there + the dependency version is the thing being compared. +- **A package diff in the comparison.** The sides can differ in + packages nobody described. If they differ only there, `sides-differ` + fails. If they also differ in a described fact, the comparison passes + and attributes the whole difference to it: a hidden confound + (`benchmark-environments.md` §4, recommendation 1). The comparator + would list the packages that differ between the two sides' package + lists ("also differs: scipy 1.13 → 1.14") beside the described fact, + as a caveat, in the way it already carries differences in ignored + facts. The list would come from a collector's artifact, or from the + manifest of the typed-facts follow-up. + The catch: the list is an artifact file the result references, not + stored in the store (`comparator.md` §2.1 takes its inputs from the + store). The diff works only when the artifact files are reachable, for + example in the same results directory, and the comparison must say + when they were not. A name-to-version map in `provenance.info` would + be in the store, but adds hundreds of entries to every result. +- **Collector output in identity.** The runner stores each collector's + `summary` and a sha256 of its full output in + `environment.system.` (`collectors.py:104`), and the default + policy matches both. Any collector fact that changes splits the + series, even one used only for `verify`. Dropping the sha256 from + identity (the artifact already records it) would let such facts live + in the artifact alone. It is smaller than the policy item and removes + the numpy tracking split. It changes PR #71's design, so its author + should agree. +- **The minimum as the run-mode estimator.** UC-04 §7 compares the + minimum timing per environment. The comparator compares the median of + each result's observations per round, takes the mean of the relative + differences across rounds, and calls a change when the effect exceeds + `k` times its standard error (`compare.py`, `_unit`). On noisy + machines the two can disagree, so the comparator may say "no change + detected" where UC-04's example shows `+9.2%`. The plan keeps the + comparator as it is, and UC-04 §7 is left as written. An optional + `--estimator min` would take the minimum of the observations per + round instead. It needs a new method version, and + the comparison document's `estimator` field, hard-coded to the median, + would change. UC-02 wants the minimum too, and the choice of run-mode + method is open in `comparator.md` §7 question 1, so it is best settled + there. A table of one number per environment, as in UC-04's example, + is a smaller rendering-only addition. +- **Retiring the setup file.** If the typed-facts follow-up is built, + the setup file carries the same kind of fact through a second + channel. Retiring it would mean removing + `snapshot.setup_facts`, the `system["setup"]` assignment in + `runner.py`, `BENCHX_SETUP_FILE`, `test_setup_file`, and the + setup-file mentions in `prototype/README.md` and `prototype-design.md`. + `--component`, `--build-option` and the description's `build` would + replace it. The costs: the design keeps the setup file for the case + where the party that prepares an environment is not the one that runs + benchmarks (`harness-adapter.md` §5.1). Without it, the preparer's + facts reach benchx through a sidecar next to a build directory, or + through flags the caller's script passes, and two venvs over one + shared working tree have only the flags. It is a design change + (`harness-adapter.md` §5.1 and `benchmark-environments.md` §3.2). And + results already stored keep `environment.system.setup`, while new ones + would not, so those series would start fresh. +- **Peak memory.** UC-04 §6 lists peak memory (memray) as optional. No + adapter measures it yet, so it is out of scope here. + +## Open questions + +None. What was left open is under "Potential follow-ups". From 87037c6e1b8da201f043272cfdb8431d684e4449 Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Tue, 27 Oct 2026 17:04:31 +0000 Subject: [PATCH 2/9] bx bench: --label, --round and --slot place a call in a session Calls that share a --run-key become the sides of one session, so two environments can be measured by separate calls and compared with `bx compare --run`. The caller's loop keeps the slot counter. Refused before an order is built: any of the three without --run-key (the default key is a timestamp per call), --label or --round without --slot (the ingest key only gets its slot suffix when a slot is set, so two sides would collide), a malformed or repeated label, a negative number. With a slot, the order is written to orders/order-.json, as in bx compare, so the calls do not overwrite each other's order. Co-Authored-By: Claude Sonnet 5.5 --- prototype/benchx/cli.py | 36 ++++++++++++++++++++++++++++++ prototype/benchx/session.py | 14 +++++++----- prototype/tests/test_bench.py | 42 +++++++++++++++++++++++++++++++++++ 3 files changed, 87 insertions(+), 5 deletions(-) diff --git a/prototype/benchx/cli.py b/prototype/benchx/cli.py index 933c149..5d969fe 100644 --- a/prototype/benchx/cli.py +++ b/prototype/benchx/cli.py @@ -78,6 +78,31 @@ def _environment(items) -> dict: return env +def _session_position(args) -> tuple[dict, int | None, int | None]: + """Where `bx bench` sits in a session that other calls join through the same + `--run-key`: its `--label NAME=VALUE`s, `--round` and `--slot`. UsageError when + they could not join one.""" + labels = {} + for item in args.label or []: + name, eq, value = item.partition("=") + if not (name and eq and value): + raise UsageError(f"--label wants NAME=VALUE, not {item!r}") + if name in labels: + raise UsageError(f"--label {name} is given twice") + labels[name] = value + for flag, number in (("--round", args.round), ("--slot", args.slot)): + if number is not None and number < 0: + raise UsageError(f"{flag} wants a number from 0, not {number}") + placed = labels or args.round is not None or args.slot is not None + if placed and not args.run_key: + raise UsageError("--label, --round and --slot need an explicit --run-key: the default is a timestamp " + "per call, so two calls would never share a run") + if (labels or args.round is not None) and args.slot is None: + raise UsageError("--label and --round need --slot: without one, the sides of a run would share an " + "ingest key and their results would conflict") + return labels, args.round, args.slot + + def _bench_target(args) -> tuple[str, str | None, str | None]: """The target, the suite it forces (or None: --suite, then the file's), and the adapter it forces. With `--suite` the target is as given. Without it, the target @@ -155,6 +180,7 @@ def cmd_bench(args): raise UsageError("give case names or --filter, not both") target, forced_suite, forced_adapter = _bench_target(args) settings = _common(cfg, args, forced_adapter=forced_adapter, forced_suite=forced_suite) + settings["labels"], settings["round_"], settings["slot"] = _session_position(args) _needs_provider(cfg, target) except (UsageError, config_mod.ConfigError, bench_mod.SessionError) as e: print(f"bx bench: error: {e}", file=sys.stderr) @@ -493,6 +519,16 @@ def main(argv=None): p.add_argument("--case-timeout", type=float, help=f"seconds before one case is stopped (default: timeouts.case_s, else {orders.CASE_TIMEOUT_S:g}; " f"asv: {ASV_CASE_TIMEOUT_S}, all rounds, each also stopped at the benchmark's own timeout)") + p.add_argument("--label", action="append", metavar="NAME=VALUE", + help="name this call's side of a session, such as env=numpy-2.0 (repeatable); the calls of a " + "session share --run-key, and bx compare --run reads the label with --label NAME. " + "Needs --run-key and --slot") + p.add_argument("--round", type=int, + help="this call's round in a session: the sides' attempts are paired by it. " + "Needs --run-key and --slot") + p.add_argument("--slot", type=int, + help="this call's position in the session's alternation (0, 1, 2, ...), kept by the caller's loop. " + "Needs --run-key") p.add_argument("--no-ingest", action="store_true", help="only write result files; skip the local store") p.set_defaults(fn=cmd_bench) diff --git a/prototype/benchx/session.py b/prototype/benchx/session.py index 45cd90b..317cd42 100644 --- a/prototype/benchx/session.py +++ b/prototype/benchx/session.py @@ -177,9 +177,11 @@ def bench_target(target: str, *, suite: str, adapter: str, store=None, flags=Non environment_variables=None, filter=None, quantities=("wall-time",), project=None, run_key=None, out=None, source_uri=None, case_s=orders.CASE_TIMEOUT_S, order_s=orders.ORDER_TIMEOUT_S, progress=None, build=True, provider=None, config_path=None, results_dir=None, - collectors=None) -> dict: - """Run one target once: a comparison session of one order, with no round, - slot or comparison. Results are delivered to `store` unless it is None. + collectors=None, round_=None, slot=None, labels=None) -> dict: + """Run one target once: a comparison session of one order, with no comparison. + It has a round, a slot and labels when the caller places it in a session that + other calls join through the same run key. Results are delivered to `store` + unless it is None. `target` is `@CONFIG[=SOURCE_REF]` or a bare path, which the harness reads (see `_side`); `adapter`, `flags` and the rest are as for `compare_targets`. @@ -195,10 +197,12 @@ def bench_target(target: str, *, suite: str, adapter: str, store=None, flags=Non out = Path(out) if out else Path(results_dir or "results") / run_key.replace("/", "_") if progress: progress(f"precision: {harness.describe_precision(precision, config_path)}") - summary, outcome = _run_side(side, out / "orders" / "order.json", store, harness=harness, suite=suite, + order_file = "order.json" if slot is None else f"order-{slot:04d}.json" # one per slot, as compare_targets + summary, outcome = _run_side(side, out / "orders" / order_file, store, harness=harness, suite=suite, filter=filter, quantities=quantities, precision=precision, environment_variables=environment_variables, project=project, run_key=run_key, - source_uri=source_uri, case_s=case_s, order_s=order_s, collectors=collectors) + source_uri=source_uri, case_s=case_s, order_s=order_s, collectors=collectors, + round_=round_, slot=slot, labels=labels) return {**summary, "outcome": outcome, "precision": precision} diff --git a/prototype/tests/test_bench.py b/prototype/tests/test_bench.py index 6aa6d64..15e03c3 100644 --- a/prototype/tests/test_bench.py +++ b/prototype/tests/test_bench.py @@ -159,3 +159,45 @@ def test_source_uri_flag_overrides_the_remote(tree): "https://example.org/r.git", *FAST]) == cli.EXIT_OK order = json.loads((tree / "results" / "uri" / "orders" / "order.json").read_text()) assert order["target"]["source"]["uri"] == "https://example.org/r.git" + + +def test_bench_places_calls_in_a_session(tree): + """--label, --round and --slot make separate `bx bench` calls the sides of one session.""" + sides = [("a", 0), ("b", 0), ("a", 1), ("b", 1)] + for slot, (side, round_) in enumerate(sides): + assert cli.main(["bench", "mod.py", "bench_a", "--run-key", "S", "--label", f"env={side}", + "--round", str(round_), "--slot", str(slot), "--no-ingest", *FAST]) == cli.EXIT_OK + out = tree / "results" / "S" + assert sorted(p.name for p in (out / "orders").glob("*.json")) == [f"order-{n:04d}.json" for n in range(4)] + found = [] + for path in out.glob("*.json"): + document = json.loads(path.read_text()) + if path.name.startswith("workorder-") or document["measurement"].get("kind") == "condition": + continue + assert core.check(document, "measurement-result") == [] + found.append((document["procedure"]["slot"], document["procedure"]["round"], + document["provenance"]["labels"])) + assert sorted(found, key=lambda f: f[0]) == [(0, 0, {"env": "a"}), (1, 0, {"env": "b"}), + (2, 1, {"env": "a"}), (3, 1, {"env": "b"})] + + +@pytest.mark.parametrize("extra, message", [ + (["--label", "env=a", "--round", "0", "--slot", "0"], "explicit --run-key"), + (["--slot", "0"], "explicit --run-key"), + (["--run-key", "S", "--label", "env=a"], "need --slot"), + (["--run-key", "S", "--round", "0"], "need --slot"), + (["--run-key", "S", "--slot", "0", "--label", "env"], "NAME=VALUE"), + (["--run-key", "S", "--slot", "0", "--label", "env=a", "--label", "env=b"], "given twice"), + (["--run-key", "S", "--slot", "-1"], "from 0"), +]) +def test_session_flags_are_refused_before_anything_runs(extra, message, tree, capsys): + assert cli.main(["bench", "mod.py", "bench_a", "--no-ingest", *extra, *FAST]) == cli.EXIT_REFUSED + assert message in capsys.readouterr().err + assert not (tree / "results").exists() + + +def test_bench_without_session_flags_is_unchanged(tree): + assert cli.main(["bench", "mod.py", "bench_a", "--run-key", "plain", "--no-ingest", *FAST]) == cli.EXIT_OK + assert (tree / "results" / "plain" / "orders" / "order.json").exists() + order = json.loads((tree / "results" / "plain" / "orders" / "order.json").read_text()) + assert not {"round", "slot", "labels"} & set(order) From de402a8cbced490740576c5b2f2d3577ca72eb31 Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Tue, 27 Oct 2026 17:09:16 +0000 Subject: [PATCH 3/9] bx compare: say what differs, show warnings, and tell what to change The comparison now lists the facts that differ between the sides (unmatched_differences, which it already computed) and the warnings the sides' results carry (the schema's per-unit `caveats`), without changing a verdict. The refusals say what to change: the sides differ only in their label (describe the environment with a collector or a setup file), the sides do not alternate (same order every round), the attempt counts differ (same number of rounds), a call has no label or slot. tests/test_uc04.py runs UC-04 in process: separate `bx bench` calls with --run-key, --label, --round and --slot, the sides told apart by a context collector or by a setup file, then `bx compare --run`. Co-Authored-By: Claude Sonnet 5.5 --- prototype/benchx/compare.py | 36 ++++++++-- prototype/tests/test_uc04.py | 131 +++++++++++++++++++++++++++++++++++ 2 files changed, 162 insertions(+), 5 deletions(-) create mode 100644 prototype/tests/test_uc04.py diff --git a/prototype/benchx/compare.py b/prototype/benchx/compare.py index 9de9e64..b9b33a5 100644 --- a/prototype/benchx/compare.py +++ b/prototype/benchx/compare.py @@ -109,7 +109,8 @@ def compare(documents, *, run_key, profile, baseline, label=None, k=3.0, min_rou failed = doc["failed_invariants"] projected = [projection.apply(d, store_project(d)) for d in documents] if any(p.axis[0] is None for p in projected): - failed.append({"invariant": "sides", "detail": f"a result has no {varying}"}) + failed.append({"invariant": "sides", "detail": f"a result has no {varying}: give every bx bench call " + "--label NAME=VALUE, and name that label here"}) return doc # A side is a distinct axis value, named by its first part (a tree, a label). @@ -139,7 +140,9 @@ def compare(documents, *, run_key, profile, baseline, label=None, k=3.0, min_rou a, b = (json.loads(side) for side in (base, contender)) if all(x == y for path, x, y in zip(projection.axis, a, b) if not path.startswith("/provenance/")): failed.append({"invariant": "sides-differ", - "detail": f"the sides differ only in {varying}"}) + "detail": f"the sides differ only in their label ({varying}): nothing recorded about the " + "environment differs. Describe what differs with a context collector " + "(collectors in benchx.conf.json) or a setup file for each environment"}) if "one-tree-per-side" in check: trees = {side: {d["provenance"].get("subject_tree") for d, x in zip(documents, sides) if x == side} for side in (base, contender)} @@ -157,13 +160,16 @@ def compare(documents, *, run_key, profile, baseline, label=None, k=3.0, min_rou for d, s in zip(documents, sides): slot = d.get("procedure", {}).get("slot") if slot is None: - failed.append({"invariant": "alternating-slots", "detail": "a result has no slot"}) + failed.append({"invariant": "alternating-slots", + "detail": "a result has no slot: give every bx bench call --slot"}) break slots.setdefault(slot, set()).add(s) else: order = [slots[slot] for slot in sorted(slots)] if any(len(o) != 1 for o in order) or any(a == b for a, b in zip(order, order[1:])): - failed.append({"invariant": "alternating-slots", "detail": "sides do not alternate by procedure.slot"}) + failed.append({"invariant": "alternating-slots", + "detail": "sides do not alternate by procedure.slot: run the sides in the same " + "order every round (a, b, a, b, ...)"}) # Units: one workload variant and quantity each. Everything else the # projection matches must agree across both sides (schema §5.5). @@ -191,7 +197,8 @@ def compare(documents, *, run_key, profile, baseline, label=None, k=3.0, min_rou continue if "equal-attempts" in check and len(by_side[base]) != len(by_side[contender]): failed.append({"invariant": "equal-attempts", **where, - "detail": f"{len(by_side[base])} against {len(by_side[contender])}"}) + "detail": f"{len(by_side[base])} against {len(by_side[contender])}: run the same " + "number of rounds on both sides"}) continue rounds = {} for s in (base, contender): @@ -206,6 +213,8 @@ def compare(documents, *, run_key, profile, baseline, label=None, k=3.0, min_rou if base in r and contender in r] unit_doc = _unit(where, pairs, by_side, base, contender, k, min_rounds) unit_doc["unmatched_differences"] = _differences(projection, by_side[base], by_side[contender]) + if caveats := _caveats(by_side, base, contender): + unit_doc["caveats"] = caveats doc["units"].append(unit_doc) return doc @@ -247,6 +256,16 @@ def _declared(document) -> dict: return side +def _caveats(by_side, base, contender) -> list[str]: + """The warnings a side's results carry (quality.warnings), by side, for the reader.""" + found = set() + for role, side in (("baseline", base), ("contender", contender)): + for d in by_side[side]: + for w in d.get("quality", {}).get("warnings", []): + found.add(f"{role}: {w.get('code', w) if isinstance(w, dict) else w}") + return sorted(found) + + def _unit(where, pairs, by_side, base, contender, k, min_rounds): first = {s: by_side[s][0] for s in (base, contender)} quantity = first[base]["measurement"]["quantity"] @@ -310,6 +329,13 @@ def render(doc: dict) -> str: shown = side[key] if key != "observed_env" else ", ".join(f"{k}={v}" for k, v in side[key].items()) first, *rest = str(shown).splitlines() or [""] lines.append(f" {name}: {key.replace('_', ' ')}: {first}" + (f" (+{len(rest)} lines)" if rest else "")) + differs = sorted({p for u in doc["units"] for p in u.get("unmatched_differences", [])}) + if differs: + shown = ", ".join(_dotted(p) for p in differs[:6]) + lines.append(f" differs: {shown}" + (f" (+{len(differs) - 6} more)" if len(differs) > 6 else "")) + caveats = sorted({c for u in doc["units"] for c in u.get("caveats", [])}) + if caveats: + lines.append(f" caveats: {', '.join(caveats)}") for section in ("regressed", "improved", "changed", "no-change-detected", "indeterminate"): rows = [u for u in doc["units"] if u["verdict"] == section] if not rows: diff --git a/prototype/tests/test_uc04.py b/prototype/tests/test_uc04.py new file mode 100644 index 0000000..4c0fdd8 --- /dev/null +++ b/prototype/tests/test_uc04.py @@ -0,0 +1,131 @@ +"""UC-04: two environments measured by separate `bx bench` calls, then compared. + +The two environments are emulated in process: an environment variable stands for +what differs between two venvs, and a context collector (or a setup file) reads it, +as it would read a package version. The session flags are `bx bench`'s --run-key, +--label, --round and --slot; the comparison is `bx compare --run`. +""" + +import json +import sys +import textwrap + +import pytest + +pytest.importorskip("pyperf") + +from benchx import cli + +MODULE = "def bench_a():\n sum(range(100))\n" +FAST = ["--processes", "2", "--values", "2", "--loops", "20", "--no-project"] +COLLECT = textwrap.dedent(''' + import json, os + print(json.dumps({"summary": {"dep": os.environ.get("FAKE_DEP", "unset")}})) +''') + + +@pytest.fixture +def tree(tmp, monkeypatch): + path = tmp / "tree" + path.mkdir() + (path / "mod.py").write_text(MODULE) + monkeypatch.chdir(path) + return path + + +def write_config(tree, collector=None): + config = {"schema_version": "benchx/config/0.1.0", "adapter": {"name": "pyperf"}} + if collector is not None: + config["collectors"] = {"deps": {"command": [sys.executable, "-c", COLLECT], **collector}} + (tree / "benchx.conf.json").write_text(json.dumps(config)) + + +def call(side, round_, slot, *, dep=None, setup=None, monkeypatch, run_key="S"): + """One `bx bench` call, as one side of the session, in the environment `dep` or `setup` stand for.""" + if dep is not None: + monkeypatch.setenv("FAKE_DEP", dep) + if setup is not None: + monkeypatch.setenv("BENCHX_SETUP_FILE", str(setup)) + argv = ["bench", "mod.py", "bench_a", "--run-key", run_key, "--label", f"env={side}", + "--round", str(round_), "--slot", str(slot), *FAST] + assert cli.main(argv) == cli.EXIT_OK + + +def loop(sides, monkeypatch, rounds=3): + """The caller's loop: alternate the sides round by round, keeping the slot counter.""" + slot = 0 + for round_ in range(rounds): + for side, environment in sides.items(): + call(side, round_, slot, monkeypatch=monkeypatch, **environment) + slot += 1 + + +def compare(capsys, baseline="a", run_key="S"): + code = cli.main(["compare", "--run", run_key, "--profile", "environments", "--label", "env", + "--baseline", baseline]) + return code, capsys.readouterr().out + + +def test_a_collector_tells_the_sides_apart(tree, monkeypatch, capsys): + """The sides differ in what the collector reads, so the comparison goes ahead and says what differs.""" + write_config(tree, {}) + loop({"a": {"dep": "1.0"}, "b": {"dep": "2.0"}}, monkeypatch) + capsys.readouterr() + code, out = compare(capsys) + assert code == cli.EXIT_OK + assert "failed invariants" not in out + assert "differs:" in out and "coordinates.environment.system.deps" in out + assert "bench_a" in out + + +def test_a_setup_file_per_environment_tells_the_sides_apart(tree, tmp, monkeypatch, capsys): + write_config(tree) + files = {} + for side, version in (("a", "1.0"), ("b", "2.0")): + files[side] = tmp / f"setup-{side}.json" + files[side].write_text(json.dumps({"dep": version})) + loop({"a": {"setup": files["a"]}, "b": {"setup": files["b"]}}, monkeypatch) + capsys.readouterr() + code, out = compare(capsys) + assert code == cli.EXIT_OK + assert "failed invariants" not in out and "coordinates.environment.system.setup" in out + + +def test_identical_sides_are_refused_with_advice(tree, monkeypatch, capsys): + write_config(tree, {}) + call("a", 0, 0, dep="1.0", monkeypatch=monkeypatch) + call("b", 0, 1, dep="1.0", monkeypatch=monkeypatch) + capsys.readouterr() + _, out = compare(capsys) + assert "sides-differ" in out and "collector" in out and "setup file" in out + + +def test_sides_that_do_not_alternate_are_refused_with_advice(tree, monkeypatch, capsys): + write_config(tree, {}) + for slot, (side, round_, dep) in enumerate([("a", 0, "1.0"), ("a", 1, "1.0"), ("b", 0, "2.0"), ("b", 1, "2.0")]): + call(side, round_, slot, dep=dep, monkeypatch=monkeypatch) + capsys.readouterr() + _, out = compare(capsys) + assert "alternating-slots" in out and "same order every round" in out + + +def test_a_label_missing_on_a_call_is_refused_with_advice(tree, monkeypatch, capsys): + write_config(tree, {}) + call("a", 0, 0, dep="1.0", monkeypatch=monkeypatch) + capsys.readouterr() + cli.main(["compare", "--run", "S", "--profile", "environments", "--label", "env_name", "--baseline", "a"]) + out = capsys.readouterr() + assert "provenance.labels.env_name" in out.out + out.err + assert "--label NAME=VALUE" in out.out + out.err + + +def test_a_failed_check_is_shown_in_the_comparison_and_changes_nothing_else(tree, monkeypatch, capsys): + """expect, with on_fail: warn, flags the side that disagrees; the run goes ahead, the verdict is unchanged.""" + write_config(tree, {"expect": {"summary": {"dep": "1.0"}}, "on_fail": "warn"}) + loop({"a": {"dep": "1.0"}, "b": {"dep": "2.0"}}, monkeypatch) + capsys.readouterr() + code, out = compare(capsys) + assert code == cli.EXIT_OK + assert "caveats: contender: collector-mismatch" in out + assert "baseline: collector-mismatch" not in out + assert "failed invariants" not in out and "bench_a" in out From 9b2fdc42af2e9b17866cfaf2c4e0be46a18c5fe9 Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Tue, 27 Oct 2026 17:13:42 +0000 Subject: [PATCH 4/9] snapshot: do not exclude a path git already ignores from the tree id benchx leaves its own output (the store, the results directory, --out) out of the tree identity with exclude pathspecs. `git add -A` refuses an exclude pathspec that names a path the project's .gitignore already ignores, so the tree id came out unknown whenever the results directory was git-ignored, which is common. Such paths are now left out of the excludes: git leaves them out anyway. Co-Authored-By: Claude Sonnet 5.5 --- prototype/benchx/snapshot.py | 11 ++++++++++- prototype/tests/test_config_schema.py | 14 ++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/prototype/benchx/snapshot.py b/prototype/benchx/snapshot.py index ac8278d..8168de9 100644 --- a/prototype/benchx/snapshot.py +++ b/prototype/benchx/snapshot.py @@ -124,13 +124,22 @@ def _ignored_in(root: Path) -> list[str]: return [p.relative_to(root).as_posix() for p in _IGNORED if p != root and p.is_relative_to(root)] +def _git_ignores(top, rel) -> bool: + """Whether the repository's own ignore rules already leave `rel` out of `git add -A`. + Naming such a path in an exclude pathspec makes `git add` fail, and the tree id + with it, so a results directory the project git-ignores must not be excluded twice.""" + return subprocess.run(["git", "check-ignore", "-q", "--", rel], cwd=top, + capture_output=True).returncode == 0 + + def git_identity(path) -> dict | None: """Revision, dirty state, and working-tree id of a checkout (schema §4.1).""" top = _run(["git", "rev-parse", "--show-toplevel"], cwd=path) head = top and _run(["git", "rev-parse", "HEAD"], cwd=top) if not head: return None - excludes = [_PYCACHE_EXCLUDE, *(f":(top,exclude){rel}" for rel in _ignored_in(top))] + excludes = [_PYCACHE_EXCLUDE, *(f":(top,exclude){rel}" for rel in _ignored_in(top) + if not _git_ignores(top, rel))] status = _run(["git", "status", "--porcelain", "--untracked-files=all", "--", ".", *excludes], cwd=top) # A scratch index that does not exist yet, so the repository's own index # is untouched (schema §4.1). diff --git a/prototype/tests/test_config_schema.py b/prototype/tests/test_config_schema.py index 0169bc4..a1a1c64 100644 --- a/prototype/tests/test_config_schema.py +++ b/prototype/tests/test_config_schema.py @@ -100,3 +100,17 @@ def test_a_plain_directory_tree_id_skips_ignored_paths(tmp_path): assert snapshot.directory_identity(root)["tree"] != before snapshot.ignore(root / "results", root / "store.parquet") assert snapshot.directory_identity(root)["tree"] == before + + +def test_a_results_directory_the_project_git_ignores_still_gives_a_tree_id(checkout): + """Excluding a path that .gitignore already ignores used to make `git add` fail, so the tree id was unknown.""" + (checkout / ".gitignore").write_text("results/\n") + git(checkout, "add", "-A") + git(checkout, "commit", "-qm", "ignore results") + clean = snapshot.git_identity(checkout) + write_outputs(checkout) + snapshot.ignore(checkout / "results", checkout / "store.parquet") + again = snapshot.git_identity(checkout) + assert (again["dirty"], again["tree"]) == ("clean", clean["tree"]) + snapshot.ignore(checkout / "results") # the store file is untracked and not ignored: a real difference + assert snapshot.git_identity(checkout)["dirty"] == "dirty" From f67c3d6ebd1d2301902ca498388729cdf7348242 Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Tue, 27 Oct 2026 17:17:15 +0000 Subject: [PATCH 5/9] examples/numpy-venvs: UC-04 with two venvs and two numpy versions Two venvs hold different numpy versions, which no single process can, so run.sh drives separate `bx bench` calls that alternate round by round under one --run-key, with --label, --round and --slot, then compares them with `bx compare --run`. A context collector (describe_env.py) tells the sides apart: it prints the numpy its Python imports, which lands under environment.system.deps. The provider declares each side's numpy in how_built, and the collector's `verify: how_built` with `on_fail: warn` flags a side whose environment is not the declared one. The README carries the map from UC-04's spellings to the prototype's. The CI workflow runs the example on one matrix cell (it makes its own Python 3.12 venvs). prototype-design.md and the prototype README document the session flags and the collector route. The plan is updated to match the script as built. Co-Authored-By: Claude Sonnet 5.5 --- .github/workflows/tests.yml | 10 ++ Plan_UC04_on_the_design.md | 90 +++++---------- docs/design/prototype-design.md | 22 +++- prototype/README.md | 8 ++ prototype/examples/numpy-venvs/.gitignore | 2 + prototype/examples/numpy-venvs/README.md | 103 ++++++++++++++++++ prototype/examples/numpy-venvs/bench_suite.py | 20 ++++ .../examples/numpy-venvs/benchx.conf.json | 22 ++++ .../examples/numpy-venvs/describe_env.py | 12 ++ prototype/examples/numpy-venvs/provider.sh | 31 ++++++ prototype/examples/numpy-venvs/run.sh | 60 ++++++++++ prototype/examples/numpy-venvs/versions.env | 4 + 12 files changed, 319 insertions(+), 65 deletions(-) create mode 100644 prototype/examples/numpy-venvs/.gitignore create mode 100644 prototype/examples/numpy-venvs/README.md create mode 100644 prototype/examples/numpy-venvs/bench_suite.py create mode 100644 prototype/examples/numpy-venvs/benchx.conf.json create mode 100644 prototype/examples/numpy-venvs/describe_env.py create mode 100755 prototype/examples/numpy-venvs/provider.sh create mode 100755 prototype/examples/numpy-venvs/run.sh create mode 100644 prototype/examples/numpy-venvs/versions.env diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 2727c71..f9dcd4e 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -61,3 +61,13 @@ jobs: if: runner.os != 'Windows' shell: bash run: uv run --extra test --python ${{ matrix.python-version }} bash examples/archery-local/run.sh + + # Makes two venvs of its own with uv (Python 3.12, the newest with numpy 1.26.4 wheels), so + # the matrix Python only runs the outer script and one cell is enough. Needs network access. + # ROUNDS=3 is the fewest rounds that give a verdict. + - name: Run the numpy-venvs example + if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' + shell: bash + run: >- + ROUNDS=3 uv run --extra test --python ${{ matrix.python-version }} + bash examples/numpy-venvs/run.sh diff --git a/Plan_UC04_on_the_design.md b/Plan_UC04_on_the_design.md index a63bc85..e3608cb 100644 --- a/Plan_UC04_on_the_design.md +++ b/Plan_UC04_on_the_design.md @@ -514,70 +514,32 @@ at the outer environment. Activate scripts can trip `set -u`, so the script turns it off around the `source`. benchx executes nothing for any of this. -`run.sh`: - -```bash -#!/usr/bin/env bash -# UC-04 with two venvs: the same code, numpy 1.26.4 against numpy 2.0.2. -set -euo pipefail -cd "$(dirname "$0")" -. ./versions.env -ROUNDS=${ROUNDS:-5} -rm -rf out .venv-a .venv-b; mkdir -p out - -# Run a command with one venv activated, in a subshell. -in_env() { local venv=".venv-$1"; shift - ( set +u; . "$venv/bin/activate"; set -u; "$@" ); } - -# 1. The user prepares both environments. benchx builds nothing. -for side in a b; do - v=NUMPY_${side^^} - uv venv -q --python "${PYTHON:-3.12}" ".venv-$side" - uv pip install -q --python ".venv-$side/bin/python" \ - "numpy==${!v}" pyperf -e ../.. -done - -# 2. The caller's loop: alternate the sides, round by round. -S=numpy-$(date +%s); slot=0 -for round in $(seq 0 $((ROUNDS - 1))); do - for side in a b; do - v=NUMPY_${side^^} - in_env "$side" bx bench "@$side" \ - --run-key "$S" --label "env=numpy-${!v}" \ - --round "$round" --slot "$slot" --out "out/$S" - slot=$((slot + 1)) - done -done - -# 3. Compare. The sides differ in the collected numpy version. -in_env a bx compare --run "$S" \ - --profile environments --label env --baseline "numpy-$NUMPY_A" \ - | tee out/compare.txt -grep -q "$NUMPY_B" out/compare.txt # the differing fact is shown - -# 4. The check. Declare version b, run in venv a: the collector disagrees. -in_env a bx bench @b --run-key "$S-mismatch" \ - --label "env=wrong" --round 0 --slot 0 --out "out/$S-mismatch" -grep -rq collector-mismatch "out/$S-mismatch" -``` - -What it should show. This is the expected behaviour, not an output -captured from a run: -- Step 3 prints the comparison for the two sides, with the differing - fact (`environment.system.deps.numpy`, `1.26.4` against `2.0.2`) and - a verdict per benchmark. -- Step 4 prints results that carry `collector-mismatch`. The run - proceeds, because the collector has `on_fail: warn`, and the recorded - value is the one the collector read. -- Dropping the `--label` difference, so that both sides use the same - venv, makes step 3 fail `sides-differ`, with the message that says to - describe the environment under test. - -The script exits nonzero unless the comparison shows the differing fact -and the mismatch run carries `collector-mismatch`, so it can run in CI. -It cannot run until package A has landed, because `bx bench` has no -`--label`, `--round` or `--slot` yet. The CI step below is added in the -same change as the demo, after package A. +`run.sh` (the file is the reference) does, in order: +1. Creates `.venv-a` and `.venv-b`, each with its numpy, pyperf and + benchx (`-e ../..`), with `uv`, or with `python -m venv` and pip when + `uv` is absent. +2. Runs the caller's loop: for each round and side, `in_env "$side" bx + bench "@$side"` with `--run-key`, `--label env=numpy-`, + `--round`, `--slot` and `--out`. `in_env` sources the venv's + `activate` in a subshell, with `set -u` off around the `source`. +3. Compares: `bx compare --run S --profile environments --label env + --baseline numpy-`, kept in `out/compare.txt`. The script + exits nonzero unless the `differs:` line names `deps.numpy` and no + invariant failed. +4. Runs the check: it declares version `b` for a run in venv `a`, and + exits nonzero unless that run's results carry `collector-mismatch`. + +What the comparison shows. This was captured from a run with numpy 2.3.4 +and 2.4.3 on Python 3.14, without `uv`; the demo's defaults (1.26.4 and +2.0.2 on Python 3.12 through `uv`) were not run where it was built. +- A `differs:` line that names the facts that differ + (`coordinates.environment.system.deps.numpy` and `...deps.sha256`), + and for each side its declared `how built` object, which carries the + version. The comparison names what differs and does not print the + collected values, because `unmatched_differences` holds JSON Pointers. +- A verdict per benchmark. +- With both sides in the same venv, `sides-differ` fails, with the + message that says to describe the environment under test. **Spelling map.** The demo's README carries the map from UC-04's spellings (`docs/use-cases/UC-04-cross-env.md`, which is kept as diff --git a/docs/design/prototype-design.md b/docs/design/prototype-design.md index 02f1aa1..8c94023 100644 --- a/docs/design/prototype-design.md +++ b/docs/design/prototype-design.md @@ -39,6 +39,7 @@ bx bench TARGET --suite NAME [CASE …] # the single-target form of bx compar [--adapter …] [--filter REGEX] [--quantity NAMES] [-e NAME=VALUE ...] [precision flags, as for bx compare] [--project P | --no-project] [--run-key KEY] [--out DIR] [--no-build] [--source-uri URI] + [--label NAME=VALUE ... --round N --slot K] # one side of a session other calls join [--case-timeout S] [--no-ingest] bx bench MODULE.py [FUNCTION …] [--tree DIR] [...] # shorthand: the module's checkout, pyperf → result files → store → table @@ -51,13 +52,32 @@ target the same way, takes the same options, and goes through the same code: the side is resolved by `session._side` (a provider call for `@CONFIG`, else a bare path the harness reads; the adapter checks the suite before anything runs) and the order is built by `session._order`. A bench run is one order of a -comparison session with no round, no slot, and no comparison; it is run by the +comparison session with no comparison, and by default no round or slot; it is run by the same runner, delivered as `bx run` delivers (`--no-ingest`: files only), and summarized as a table of its cases. It is not `bx run`: that executes a finished order file and never resolves a target; `bx bench` turns options into an order (saved as `results/RUN_KEY/orders/order.json`, which `bx run` can replay) and then does what `bx run` does. +**A call as one side of a session (UC-04).** Environments that cannot share a process, such as two +venvs holding different numpy versions, are measured by separate `bx bench` calls that share a +`--run-key`; the caller's script is the loop (`benchmark-environments.md` §5.7). `--label NAME=VALUE` +names the call's side, `--round N` pairs its attempt with the other side's, and `--slot K` is its +position in the alternation, which the caller counts. Each call writes `orders/order-KKKK.json`. The +flags need an explicit `--run-key` (the default is a timestamp per call, so two calls would never +share a run), and `--label` and `--round` need `--slot` (the ingest key gets its `:slot-K` suffix only +when a slot is set, so two sides would otherwise share a key). `bx compare --run KEY --profile +environments --label NAME --baseline VALUE` then compares the sides, and refuses a run whose sides do +not alternate, have unequal attempt counts, or do not differ. + +What differs between the environments is recorded by the user's own commands, so benchx knows no +environment kind: a context collector (§3) prints facts about the environment it runs in, which land +under `environment.system.`, a part of the varying coordinate; a setup file does the same for +declared facts. A collector with `verify: how_built` and `on_fail: warn` checks the facts a target's +`how_built` object declares against what it found, and the comparison lists the warning beside the +facts that differ. `prototype/examples/numpy-venvs` is the worked example, with the map from UC-04's +spellings to these. + A bare path is read by the harness. For Google Benchmark it is `BUILD_DIR[:SOURCE_DIR]`, a `build` target. For pyperf it is a working tree, a `working_tree` target that is its own source, measured as found with its dirty diff --git a/prototype/README.md b/prototype/README.md index 1b58136..0b5ecab 100644 --- a/prototype/README.md +++ b/prototype/README.md @@ -27,6 +27,7 @@ bx bench TARGET --suite NAME [CASE …] # the single-target form of bx compar [--adapter …] [--filter REGEX] [--quantity …] [-e NAME=VALUE ...] [--processes N] [--values N] [--min-time S | --loops N] [--warmups N] [--repetitions N] [--project P] [--run-key KEY] [--out DIR] [--no-build] [--source-uri URI] + [--label NAME=VALUE ... --round N --slot K] # one side of a session other calls join [--case-timeout S] [--no-ingest] bx bench MODULE.py [FUNCTION …] [--tree DIR] [...] # shorthand: Python functions under pyperf → results → store → table @@ -88,6 +89,12 @@ functions are exact case names. `bx run` is the layer underneath: it executes a order, so `bx run results/KEY/orders/order.json` replays any bench run. `examples/pyperf-bench` is a runnable walk-through. +Environments that cannot share a process, such as two venvs with different numpy versions, are +measured by separate `bx bench` calls that share a `--run-key`; `--label NAME=VALUE`, `--round N` +and `--slot K` place each call in the session, and `bx compare --run KEY --profile environments +--label NAME --baseline VALUE` compares the sides ([UC-04](../docs/use-cases/UC-04-cross-env.md)). +`examples/numpy-venvs` is the worked example, with a shell script that drives it. + `bx bench PATH --adapter asv` runs an [asv](https://asv.readthedocs.io) benchmark suite unchanged, reading asv's benchmark format as a language and timing it with pyperf; neither asv nor asv_runner is used. PATH is a directory holding `asv.conf.json` (its @@ -240,6 +247,7 @@ examples/demo-suite a two-benchmark Google Benchmark suite for the demo examples/demo the demo's project root: its benchx.conf.json, and its persistent store examples/pyperf-bench Python functions under `bx bench` (own README) examples/archery-local an end-to-end, Arrow-local-shaped example with a target provider (own README) +examples/numpy-venvs UC-04 with two venvs and two numpy versions, driven by run.sh (own README) tests/ test_prototype.py: the success criteria and the runner and store rules, on a fake Google Benchmark binary; test_comparator.py: the comparator's invariants and method on hand-built results; diff --git a/prototype/examples/numpy-venvs/.gitignore b/prototype/examples/numpy-venvs/.gitignore new file mode 100644 index 0000000..388f608 --- /dev/null +++ b/prototype/examples/numpy-venvs/.gitignore @@ -0,0 +1,2 @@ +.venv-*/ +out/ diff --git a/prototype/examples/numpy-venvs/README.md b/prototype/examples/numpy-venvs/README.md new file mode 100644 index 0000000..1737e55 --- /dev/null +++ b/prototype/examples/numpy-venvs/README.md @@ -0,0 +1,103 @@ +# numpy-venvs: UC-04 with two virtual environments + +The same code, measured under two numpy versions, by separate `bx bench` calls that alternate +round by round, then compared. This is +[UC-04](../../../docs/use-cases/UC-04-cross-env.md), "fixed branch, comparing environments": +`numpy==1.26.4` and `numpy==2.0.2` cannot be imported in one process, so each side runs in its +own environment, and a shell script drives the loop. + +You prepare the environments and enter them; benchx builds nothing and activates nothing. + +## Run it + +You need `uv` (it fetches Python 3.12, the newest Python with numpy 1.26.4 wheels) and network +access the first time. Then, from this directory: + +```console +$ ./run.sh +``` + +`ROUNDS=3 ./run.sh` runs fewer rounds (3 is the fewest that give a verdict). Other versions: +`NUMPY_A=2.3.4 NUMPY_B=2.4.3 ./run.sh` (see `versions.env`). Without `uv`, `PYTHON=python3` uses +that Python for both venvs, which then needs wheels for both versions. Everything it creates is in +`.venv-a`, `.venv-b` and `out/`, all git-ignored. + +## What it does + +1. **Prepares two venvs**, `.venv-a` and `.venv-b`, each with its numpy, pyperf and benchx. +2. **Runs the caller's loop.** For each round, one `bx bench` call per side, each in its venv + (`run.sh` sources the venv's `activate` in a subshell). The calls share a `--run-key`; each says + which side it is (`--label env=numpy-1.26.4`), which round, and which slot of the alternation. +3. **Compares.** `bx compare --run S --profile environments --label env --baseline numpy-1.26.4`. + The sides differ because the collector read a different numpy in each. The comparison says + what differed (`differs: ...deps.numpy`), shows what each side declared, and gives a verdict per + benchmark: + + ``` + run numpy-1793121294 profile environments baseline numpy-2.3.4 contender numpy-2.4.3 + numpy-2.3.4: how built: {"summary":{"numpy":"2.3.4"}} + numpy-2.4.3: how built: {"summary":{"numpy":"2.4.3"}} + differs: coordinates.environment.system.deps.numpy, coordinates.environment.system.deps.sha256 + + NO CHANGE DETECTED (3) + bench_matmul wall-time effect -16.28% noise 7.43% rounds=3 + ... + ``` + + (The numbers vary from run to run; `[local-only]` appears while the demo's files are + uncommitted.) +4. **Checks a declaration.** It declares numpy 2.0.2 for a run in the venv that has 1.26.4. The + collector disagrees, so every result of that run carries `collector-mismatch`. The run goes + ahead, because the collector has `on_fail: warn`, and the recorded numpy is the one found. + +## How the sides are told apart + +- `describe_env.py` is a **context collector** (`harness-adapter.md` §5.1), named in + `benchx.conf.json`. benchx runs it once per call, in the environment the call runs in. It prints + `{"summary": {"numpy": ""}}`, and that lands in each result's `environment.system.deps`, + which the `environments` comparison treats as the varying coordinate. benchx knows nothing about + venvs, conda or pixi: any command that prints facts about an environment does the same. +- `provider.sh` is the **target provider**. For configuration `a` or `b` it declares, in + `how_built`, the numpy that side is meant to have. The collector has `verify: how_built`, so it + compares the declaration with what it found. The declared object has the same shape as the + collector's output, because the check compares key by key from the top. +- A check that fails refuses the run by default; `on_fail: warn` records a warning instead, which + is what the design asks for (a mismatch is recorded, not a gate). + +## UC-04's spellings, and the prototype's + +[UC-04](../../../docs/use-cases/UC-04-cross-env.md) is kept as written, and uses spellings the +prototype does not have. This is how each corresponds: + +- `benchx new-session` (`SESSION=$(...)`) → no command. The session is the run key, chosen by the + caller: `S=numpy-$(date +%s)`. +- `--session-id "$SESSION"` → `--run-key "$S"`. It lands in `provenance.run_key`. +- `benchx run run_benchmark.py` → `bx bench`. With a module and functions, + `bx bench MODULE.py [FUNCTION ...]`; with a prepared target, `bx bench @CONFIG --suite NAME`. + `bx run` takes an order file, not a script. +- `--env-label numpy-1.26.4` → `--label env=numpy-1.26.4`. The field `env_label` is + `provenance.labels.env`. +- `--round "$round"` → `--round N` and `--slot K`. UC-04 uses one number for two things. The + prototype separates the round, which pairs one attempt of each side (`procedure.round`), from + the slot, the realized position in the interleaved sequence (`procedure.slot`). UC-04's + `interleave_position` is the slot. The caller's loop keeps the slot counter. +- `benchx compare-environments --session-id S --baseline A --variant B` → + `bx compare --run S --profile environments --label env --baseline A`. The profile is UC-04's + `cross-environment-comparison`. There are exactly two sides, so the variant is the other one. +- `.venv-baseline/bin/benchx` → the `bx` of that environment, such as `.venv-a/bin/bx`. +- The output table (`benchmark_name`, `env`, `time`, `vs. baseline`) → the comparison output: + verdict groups with the effect, the noise and the rounds, and the facts that differ. +- UC-04's `session_id`, `env_label` and `interleave_position` fields → `provenance.run_key`, + `provenance.labels.env` and `procedure.slot`. The schema adds no fields for a profile + (`benchmark-result-schema.md` §5.5). + +## What is in this directory + +| File | Role | +|---|---| +| `run.sh` | the driver: venvs, the loop, the comparison, the check | +| `benchx.conf.json` | the project file: the adapter and suite, the provider, the `deps` collector | +| `provider.sh` | the target provider: declares each side's numpy in `how_built` | +| `describe_env.py` | the collector: prints the numpy this Python imports | +| `bench_suite.py` | three small numpy benchmarks | +| `versions.env` | the two versions, read by `run.sh` and `provider.sh` | diff --git a/prototype/examples/numpy-venvs/bench_suite.py b/prototype/examples/numpy-venvs/bench_suite.py new file mode 100644 index 0000000..866f15a --- /dev/null +++ b/prototype/examples/numpy-venvs/bench_suite.py @@ -0,0 +1,20 @@ +"""Three small numpy benchmarks. pyperf calls each bench_* function many times and times the +calls, so the arrays are made once, at module level.""" + +import numpy as np + +_rng = np.random.default_rng(0) +_square = _rng.standard_normal((200, 200)) +_vector = _rng.standard_normal(100_000) + + +def bench_matmul(): + _square @ _square + + +def bench_sort(): + np.sort(_vector) + + +def bench_sum(): + _vector.sum() diff --git a/prototype/examples/numpy-venvs/benchx.conf.json b/prototype/examples/numpy-venvs/benchx.conf.json new file mode 100644 index 0000000..be63add --- /dev/null +++ b/prototype/examples/numpy-venvs/benchx.conf.json @@ -0,0 +1,22 @@ +{ + "schema_version": "benchx/config/0.1.0", + "project": "numpy-venvs", + "adapter": { + "name": "pyperf", + "suite": "bench_suite.py", + "precision": { + "repetitions": { "mode": "fixed", "levels": [{ "unit": "process", "n": 3 }, { "unit": "value", "n": 3 }] }, + "calibration": { "mode": "adaptive", "minimum_sample_seconds": 0.02 }, + "warmup": { "mode": "count", "n_warmup": 1 } + } + }, + "provider": { "command": ["./provider.sh"] }, + "collectors": { + "deps": { + "command": ["python", "{root}/describe_env.py"], + "verify": "how_built", + "on_fail": "warn" + } + }, + "paths": { "store": "out/store.parquet", "results": "out" } +} diff --git a/prototype/examples/numpy-venvs/describe_env.py b/prototype/examples/numpy-venvs/describe_env.py new file mode 100644 index 0000000..7363bbb --- /dev/null +++ b/prototype/examples/numpy-venvs/describe_env.py @@ -0,0 +1,12 @@ +"""The collector: print what the numpy that this Python imports says about itself. + +benchx.conf.json runs it once per call, in the environment the call runs in, so it can only +differ between the two sides if their environments do. It prints the same text for the same +environment: no paths, no timestamps. +""" + +import json + +import numpy + +print(json.dumps({"summary": {"numpy": numpy.__version__}}, sort_keys=True)) diff --git a/prototype/examples/numpy-venvs/provider.sh b/prototype/examples/numpy-venvs/provider.sh new file mode 100755 index 0000000..4b86cb0 --- /dev/null +++ b/prototype/examples/numpy-venvs/provider.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# The target provider of this example (system-decomposition.md §3.4): one JSON request on stdin, +# one target description on stdout. It builds nothing, so `describe` and `prepare` answer alike. +# +# Configuration `a` or `b` names a side. The description declares, in how_built, the numpy the +# user means that side to have, in the shape describe_env.py prints. The collector in +# benchx.conf.json compares the two: a side whose environment is not the one it was declared +# with gets the warning collector-mismatch on every result. +set -euo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$HERE/versions.env" + +configuration=$(python3 -c 'import json, sys; print(json.load(sys.stdin).get("configuration") or "")') +case "$configuration" in + a) version=$NUMPY_A ;; + b) version=$NUMPY_B ;; + *) echo "provider: unknown configuration '$configuration' (a or b)" >&2; exit 2 ;; +esac +# The canonical URI this working tree's results are filed under, as in archery-local. +uri=https://example.org/numpy-venvs/demo.git + +HERE=$HERE URI=$uri VERSION=$version python3 - <<'PY' +import json, os +print(json.dumps({ + "schema_version": "benchx/target-description/0.1.0", + "target": {"kind": "working_tree", "path": os.environ["HERE"], + "source": {"uri": os.environ["URI"], "type": "git"}}, + "how_built": {"summary": {"numpy": os.environ["VERSION"]}}, + "provider": {"name": "provider.sh", "version": "1"}, +}, indent=1)) +PY diff --git a/prototype/examples/numpy-venvs/run.sh b/prototype/examples/numpy-venvs/run.sh new file mode 100755 index 0000000..84257dd --- /dev/null +++ b/prototype/examples/numpy-venvs/run.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# UC-04 with two venvs: the same code under two numpy versions, measured by separate `bx bench` +# calls that alternate round by round, then compared. See README.md for what each step shows. +# +# ./run.sh numpy 1.26.4 against 2.0.2, 5 rounds +# ROUNDS=3 ./run.sh fewer rounds (3 is the fewest that give a verdict) +# NUMPY_A=2.3.4 NUMPY_B=2.4.3 ./run.sh other versions (see versions.env) +# +# With `uv`, the venvs use Python $PYTHON (default 3.12: numpy 1.26.4 has no wheels beyond it). +# Without it, they use the python3 that $PYTHON names, which needs wheels for both versions. +# Everything it creates is here: .venv-a, .venv-b and out/ (git-ignored). benchx builds nothing. +set -euo pipefail +cd "$(dirname "$0")" +. ./versions.env +ROUNDS=${ROUNDS:-5} + +version() { case $1 in a) echo "$NUMPY_A" ;; b) echo "$NUMPY_B" ;; esac; } + +# Run a command with one venv activated, in a subshell. +in_env() { local venv=".venv-$1"; shift + ( set +u; . "$venv/bin/activate"; set -u; "$@" ); } + +say() { printf '\n\033[1m== %s\033[0m\n' "$*"; } +rm -rf out .venv-a .venv-b; mkdir -p out + +say "1. the user prepares both environments; benchx builds nothing" +for side in a b; do + if command -v uv >/dev/null; then + uv venv -q --python "${PYTHON:-3.12}" ".venv-$side" + uv pip install -q --python ".venv-$side/bin/python" "numpy==$(version $side)" pyperf -e ../.. + else + "${PYTHON:-python3}" -m venv ".venv-$side" + ".venv-$side/bin/pip" install -q "numpy==$(version $side)" pyperf -e ../.. + fi + echo " .venv-$side: numpy $(version $side)" +done + +say "2. the caller's loop: alternate the sides, round by round, one slot per call" +S=numpy-$(date +%s); slot=0 +for round in $(seq 0 $((ROUNDS - 1))); do + for side in a b; do + in_env "$side" bx bench "@$side" \ + --run-key "$S" --label "env=numpy-$(version $side)" \ + --round "$round" --slot "$slot" --out "out/$S" > /dev/null + echo " round $round, slot $slot: side $side" + slot=$((slot + 1)) + done +done + +say "3. compare: the sides differ in the numpy the collector read" +in_env a bx compare --run "$S" --profile environments --label env \ + --baseline "numpy-$NUMPY_A" | tee out/compare.txt +grep -q 'differs:.*deps\.numpy' out/compare.txt # the differing fact is shown +if grep -q 'failed invariants' out/compare.txt; then echo "the comparison was refused" >&2; exit 1; fi + +say "4. the check: declare numpy $NUMPY_B for a run in the venv that has $NUMPY_A" +in_env a bx bench @b --run-key "$S-mismatch" --label env=wrong \ + --round 0 --slot 0 --out "out/$S-mismatch" > /dev/null +grep -rq collector-mismatch "out/$S-mismatch" +echo " the results carry collector-mismatch; the run went ahead, with the numpy it found" diff --git a/prototype/examples/numpy-venvs/versions.env b/prototype/examples/numpy-venvs/versions.env new file mode 100644 index 0000000..78b5774 --- /dev/null +++ b/prototype/examples/numpy-venvs/versions.env @@ -0,0 +1,4 @@ +# The two numpy versions, read by run.sh and by provider.sh so that they never disagree. +# Override them to try others: NUMPY_A=2.3.4 NUMPY_B=2.4.3 ./run.sh +export NUMPY_A=${NUMPY_A:-1.26.4} +export NUMPY_B=${NUMPY_B:-2.0.2} From d4248f92c100873935eb1a7adfc8d8892cf5f2f0 Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Tue, 27 Oct 2026 17:42:34 +0000 Subject: [PATCH 6/9] examples/numpy-venvs: simpler script, fixed values, pin pyarrow below 25 run.sh assumes uv, fixes the numpy versions (1.26.4 and 2.0.2), the Python (3.12) and the rounds (3) in upper-case variables at the top, and sources each venv's activate directly: activating another venv replaces the one before, so there is no deactivate and no subshell. Checked with uv venvs under set -euo pipefail. provider.sh reads the versions from the environment run.sh exports, so versions.env is gone. Running the default pair for the first time showed that benchx cannot import pyarrow 25 under numpy 1.26.4 ("pyarrow requires NumPy 2.0 or newer"), and benchx imports pyarrow. The venvs install pyarrow<25 (up to 24 imports under numpy 1.26.4). The CI step loses its ROUNDS override. Co-Authored-By: Claude Sonnet 5.5 --- .github/workflows/tests.yml | 3 +- Plan_UC04_on_the_design.md | 53 +++++++------- prototype/examples/numpy-venvs/README.md | 23 +++--- prototype/examples/numpy-venvs/provider.sh | 2 +- prototype/examples/numpy-venvs/run.sh | 77 ++++++++++----------- prototype/examples/numpy-venvs/versions.env | 4 -- 6 files changed, 75 insertions(+), 87 deletions(-) delete mode 100644 prototype/examples/numpy-venvs/versions.env diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index f9dcd4e..8ff9f74 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -64,10 +64,9 @@ jobs: # Makes two venvs of its own with uv (Python 3.12, the newest with numpy 1.26.4 wheels), so # the matrix Python only runs the outer script and one cell is enough. Needs network access. - # ROUNDS=3 is the fewest rounds that give a verdict. - name: Run the numpy-venvs example if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' shell: bash run: >- - ROUNDS=3 uv run --extra test --python ${{ matrix.python-version }} + uv run --extra test --python ${{ matrix.python-version }} bash examples/numpy-venvs/run.sh diff --git a/Plan_UC04_on_the_design.md b/Plan_UC04_on_the_design.md index e3608cb..5a189fb 100644 --- a/Plan_UC04_on_the_design.md +++ b/Plan_UC04_on_the_design.md @@ -480,10 +480,8 @@ two environments that differ only in numpy, driven by a shell script. Files: - `README.md`: what it shows, what to install, how to run it, what to expect, and the map from UC-04's spellings to the prototype's (below). - numpy 1.26.4 has wheels only up to Python 3.12, so the demo asks for - Python 3.11 or 3.12. -- `versions.env`: `NUMPY_A=1.26.4` and `NUMPY_B=2.0.2`, read by the - script and by the provider, so the two never disagree. + The demo assumes `uv`. numpy 1.26.4 has wheels only up to Python + 3.12, so both venvs use Python 3.12. - `benchx.conf.json`: project name, the pyperf adapter with suite `bench_suite.py` and a small precision, `paths` into `out/`, a `provider` (`./provider.sh`), and the collector: @@ -497,7 +495,8 @@ Files: description and builds nothing. The target is this directory as a working tree. `how_built` is an object that mirrors the collector's output, `{"summary": {"numpy": ""}}`, with the version for - configuration `a` or `b` from `versions.env`. + configuration `a` or `b` from `NUMPY_A` and `NUMPY_B`, which `run.sh` + exports. - `run.sh`: the driver, below. - `.gitignore`: `.venv-*/` and `out/`. Untracked venvs inside the tree would mark it dirty, so the comparison would be `local_only`, and @@ -507,21 +506,23 @@ Files: The `python` in the collector command is found on `PATH`, so the script enters each environment the way a user does, by sourcing the venv's -`activate`, in a subshell for each call. Each call then sees exactly one -environment, and the loop's own shell never holds one. Activation also -resets `VIRTUAL_ENV`, which matters under `uv run` in CI, where it points -at the outer environment. Activate scripts can trip `set -u`, so the -script turns it off around the `source`. benchx executes nothing for any -of this. - -`run.sh` (the file is the reference) does, in order: -1. Creates `.venv-a` and `.venv-b`, each with its numpy, pyperf and - benchx (`-e ../..`), with `uv`, or with `python -m venv` and pip when - `uv` is absent. -2. Runs the caller's loop: for each round and side, `in_env "$side" bx - bench "@$side"` with `--run-key`, `--label env=numpy-`, - `--round`, `--slot` and `--out`. `in_env` sources the venv's - `activate` in a subshell, with `set -u` off around the `source`. +`activate`. Activating a venv replaces the one activated before, so +nothing is deactivated and no subshell is needed. This was checked with +`uv venv` environments under `set -euo pipefail`: the second `activate` +leaves only its own `bin` on `PATH` and resets `VIRTUAL_ENV`, which +matters under `uv run` in CI, where it points at the outer environment. +benchx executes nothing for any of this. + +`run.sh` (the file is the reference) sets its values at the top, in +upper case: `NUMPY_A=1.26.4`, `NUMPY_B=2.0.2`, `PYTHON_VERSION=3.12` and +`ROUNDS=3`, which is the fewest rounds that give a verdict +(`compare.min_rounds` defaults to 3). It then does, in order: +1. Creates `.venv-a` and `.venv-b` with `uv`, each with its numpy, + `pyarrow<25`, pyperf and benchx (`-e ../..`). benchx imports pyarrow, + and pyarrow 25 refuses to import under numpy 1.x. +2. Runs the caller's loop: for each round and side, it sources the + venv's `activate` and calls `bx bench "@$SIDE"` with `--run-key`, + `--label env=numpy-`, `--round`, `--slot` and `--out`. 3. Compares: `bx compare --run S --profile environments --label env --baseline numpy-`, kept in `out/compare.txt`. The script exits nonzero unless the `differs:` line names `deps.numpy` and no @@ -529,9 +530,8 @@ of this. 4. Runs the check: it declares version `b` for a run in venv `a`, and exits nonzero unless that run's results carry `collector-mismatch`. -What the comparison shows. This was captured from a run with numpy 2.3.4 -and 2.4.3 on Python 3.14, without `uv`; the demo's defaults (1.26.4 and -2.0.2 on Python 3.12 through `uv`) were not run where it was built. +What the comparison shows, from a run of the demo as it stands +(`uv`, Python 3.12, numpy 1.26.4 against 2.0.2): - A `differs:` line that names the facts that differ (`coordinates.environment.system.deps.numpy` and `...deps.sha256`), and for each side its declared `how built` object, which carries the @@ -586,7 +586,7 @@ and macOS with Python 3.11 and 3.13. The plan adds a step for this demo: if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' shell: bash run: >- - ROUNDS=3 uv run --extra test --python ${{ matrix.python-version }} + uv run --extra test --python ${{ matrix.python-version }} bash examples/numpy-venvs/run.sh ``` @@ -594,8 +594,7 @@ and macOS with Python 3.11 and 3.13. The plan adds a step for this demo: matrix Python only runs the outer script. Running it in every matrix cell would repeat the same work, so the step runs in one cell. - The step needs network access, for the numpy wheels and pyperf, which - the runners have. `ROUNDS=3` is the fewest rounds that give a verdict - (`compare.min_rounds` defaults to 3), and keeps the run short. + the runners have. - The workflow's `paths` filter already covers `prototype/**`. - The venvs and `out/` are git-ignored. @@ -674,7 +673,7 @@ Not part of this plan. Build only if the need shows up in practice. - Its README is listed in that repository's README. - The CI step in this repository goes. That repository has no CI workflow today, so it would need one to keep the demo running. - - `versions.env` and the provider move with it, and nothing in + - The provider and `run.sh` move with it, and nothing in `prototype/` depends on them. It is deferred because the demo has to run against the prototype's own code while package A is being built, and because it needs diff --git a/prototype/examples/numpy-venvs/README.md b/prototype/examples/numpy-venvs/README.md index 1737e55..2b85f85 100644 --- a/prototype/examples/numpy-venvs/README.md +++ b/prototype/examples/numpy-venvs/README.md @@ -10,24 +10,24 @@ You prepare the environments and enter them; benchx builds nothing and activates ## Run it -You need `uv` (it fetches Python 3.12, the newest Python with numpy 1.26.4 wheels) and network -access the first time. Then, from this directory: +You need [`uv`](https://docs.astral.sh/uv/) (it fetches Python 3.12, the newest Python with numpy +1.26.4 wheels) and network access the first time. Then, from this directory: ```console $ ./run.sh ``` -`ROUNDS=3 ./run.sh` runs fewer rounds (3 is the fewest that give a verdict). Other versions: -`NUMPY_A=2.3.4 NUMPY_B=2.4.3 ./run.sh` (see `versions.env`). Without `uv`, `PYTHON=python3` uses -that Python for both venvs, which then needs wheels for both versions. Everything it creates is in -`.venv-a`, `.venv-b` and `out/`, all git-ignored. +The versions (`NUMPY_A`, `NUMPY_B`), the Python and the number of rounds are set at the top of +`run.sh`. Everything it creates is in `.venv-a`, `.venv-b` and `out/`, all git-ignored. ## What it does -1. **Prepares two venvs**, `.venv-a` and `.venv-b`, each with its numpy, pyperf and benchx. -2. **Runs the caller's loop.** For each round, one `bx bench` call per side, each in its venv - (`run.sh` sources the venv's `activate` in a subshell). The calls share a `--run-key`; each says - which side it is (`--label env=numpy-1.26.4`), which round, and which slot of the alternation. +1. **Prepares two venvs**, `.venv-a` and `.venv-b`, each with its numpy, pyperf and benchx, and + `pyarrow<25`: benchx imports pyarrow, and pyarrow 25 refuses to import under numpy 1.x. +2. **Runs the caller's loop.** For each of the 3 rounds, one `bx bench` call per side, each in its + venv (`run.sh` sources the venv's `activate`; activating the other venv replaces it, so nothing + needs deactivating). The calls share a `--run-key`; each says which side it is + (`--label env=numpy-1.26.4`), which round, and which slot of the alternation. 3. **Compares.** `bx compare --run S --profile environments --label env --baseline numpy-1.26.4`. The sides differ because the collector read a different numpy in each. The comparison says what differed (`differs: ...deps.numpy`), shows what each side declared, and gives a verdict per @@ -97,7 +97,6 @@ prototype does not have. This is how each corresponds: |---|---| | `run.sh` | the driver: venvs, the loop, the comparison, the check | | `benchx.conf.json` | the project file: the adapter and suite, the provider, the `deps` collector | -| `provider.sh` | the target provider: declares each side's numpy in `how_built` | +| `provider.sh` | the target provider: declares each side's numpy in `how_built` (`run.sh` exports the versions) | | `describe_env.py` | the collector: prints the numpy this Python imports | | `bench_suite.py` | three small numpy benchmarks | -| `versions.env` | the two versions, read by `run.sh` and `provider.sh` | diff --git a/prototype/examples/numpy-venvs/provider.sh b/prototype/examples/numpy-venvs/provider.sh index 4b86cb0..013f464 100755 --- a/prototype/examples/numpy-venvs/provider.sh +++ b/prototype/examples/numpy-venvs/provider.sh @@ -8,7 +8,7 @@ # with gets the warning collector-mismatch on every result. set -euo pipefail HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) -. "$HERE/versions.env" +: "${NUMPY_A:?run.sh sets NUMPY_A and NUMPY_B}" "${NUMPY_B:?run.sh sets NUMPY_A and NUMPY_B}" configuration=$(python3 -c 'import json, sys; print(json.load(sys.stdin).get("configuration") or "")') case "$configuration" in diff --git a/prototype/examples/numpy-venvs/run.sh b/prototype/examples/numpy-venvs/run.sh index 84257dd..cb0b9bb 100755 --- a/prototype/examples/numpy-venvs/run.sh +++ b/prototype/examples/numpy-venvs/run.sh @@ -2,59 +2,54 @@ # UC-04 with two venvs: the same code under two numpy versions, measured by separate `bx bench` # calls that alternate round by round, then compared. See README.md for what each step shows. # -# ./run.sh numpy 1.26.4 against 2.0.2, 5 rounds -# ROUNDS=3 ./run.sh fewer rounds (3 is the fewest that give a verdict) -# NUMPY_A=2.3.4 NUMPY_B=2.4.3 ./run.sh other versions (see versions.env) -# -# With `uv`, the venvs use Python $PYTHON (default 3.12: numpy 1.26.4 has no wheels beyond it). -# Without it, they use the python3 that $PYTHON names, which needs wheels for both versions. -# Everything it creates is here: .venv-a, .venv-b and out/ (git-ignored). benchx builds nothing. +# Needs `uv`. Everything it creates is here: .venv-a, .venv-b and out/ (git-ignored). +# benchx builds nothing. set -euo pipefail cd "$(dirname "$0")" -. ./versions.env -ROUNDS=${ROUNDS:-5} - -version() { case $1 in a) echo "$NUMPY_A" ;; b) echo "$NUMPY_B" ;; esac; } -# Run a command with one venv activated, in a subshell. -in_env() { local venv=".venv-$1"; shift - ( set +u; . "$venv/bin/activate"; set -u; "$@" ); } +# provider.sh reads the numpy versions from the environment. numpy 1.26.4 has no wheels beyond +# Python 3.12, so both venvs use it. Three rounds are the fewest that give a verdict. +export NUMPY_A=1.26.4 NUMPY_B=2.0.2 +PYTHON_VERSION=3.12 +ROUNDS=3 +SIDES=("a $NUMPY_A" "b $NUMPY_B") +RUN_KEY=numpy-$(date +%s) +SLOT=0 -say() { printf '\n\033[1m== %s\033[0m\n' "$*"; } -rm -rf out .venv-a .venv-b; mkdir -p out +rm -rf out .venv-a .venv-b +mkdir -p out -say "1. the user prepares both environments; benchx builds nothing" -for side in a b; do - if command -v uv >/dev/null; then - uv venv -q --python "${PYTHON:-3.12}" ".venv-$side" - uv pip install -q --python ".venv-$side/bin/python" "numpy==$(version $side)" pyperf -e ../.. - else - "${PYTHON:-python3}" -m venv ".venv-$side" - ".venv-$side/bin/pip" install -q "numpy==$(version $side)" pyperf -e ../.. - fi - echo " .venv-$side: numpy $(version $side)" +echo "== 1. the user prepares both environments; benchx builds nothing" +for SIDE_VERSION in "${SIDES[@]}"; do + read -r SIDE VERSION <<<"$SIDE_VERSION" + uv venv -q --python "$PYTHON_VERSION" ".venv-$SIDE" + # benchx imports pyarrow, and pyarrow 25 refuses to import under numpy 1.x. + uv pip install -q --python ".venv-$SIDE/bin/python" "numpy==$VERSION" "pyarrow<25" pyperf -e ../.. + echo " .venv-$SIDE: numpy $VERSION" done -say "2. the caller's loop: alternate the sides, round by round, one slot per call" -S=numpy-$(date +%s); slot=0 -for round in $(seq 0 $((ROUNDS - 1))); do - for side in a b; do - in_env "$side" bx bench "@$side" \ - --run-key "$S" --label "env=numpy-$(version $side)" \ - --round "$round" --slot "$slot" --out "out/$S" > /dev/null - echo " round $round, slot $slot: side $side" - slot=$((slot + 1)) +echo "== 2. the caller's loop: alternate the sides, round by round, one slot per call" +for ROUND in $(seq 0 $((ROUNDS - 1))); do + for SIDE_VERSION in "${SIDES[@]}"; do + read -r SIDE VERSION <<<"$SIDE_VERSION" + # Activating a venv replaces the one activated before; no deactivate is needed. + source ".venv-$SIDE/bin/activate" + bx bench "@$SIDE" --run-key "$RUN_KEY" --label "env=numpy-$VERSION" \ + --round "$ROUND" --slot "$SLOT" --out "out/$RUN_KEY" > /dev/null + echo " round $ROUND, slot $SLOT: side $SIDE" + SLOT=$((SLOT + 1)) done done -say "3. compare: the sides differ in the numpy the collector read" -in_env a bx compare --run "$S" --profile environments --label env \ +echo "== 3. compare: the sides differ in the numpy the collector read" +bx compare --run "$RUN_KEY" --profile environments --label env \ --baseline "numpy-$NUMPY_A" | tee out/compare.txt grep -q 'differs:.*deps\.numpy' out/compare.txt # the differing fact is shown if grep -q 'failed invariants' out/compare.txt; then echo "the comparison was refused" >&2; exit 1; fi -say "4. the check: declare numpy $NUMPY_B for a run in the venv that has $NUMPY_A" -in_env a bx bench @b --run-key "$S-mismatch" --label env=wrong \ - --round 0 --slot 0 --out "out/$S-mismatch" > /dev/null -grep -rq collector-mismatch "out/$S-mismatch" +echo "== 4. the check: declare numpy $NUMPY_B for a run in the venv that has $NUMPY_A" +source .venv-a/bin/activate +bx bench @b --run-key "$RUN_KEY-mismatch" --label env=wrong \ + --round 0 --slot 0 --out "out/$RUN_KEY-mismatch" > /dev/null +grep -rq collector-mismatch "out/$RUN_KEY-mismatch" echo " the results carry collector-mismatch; the run went ahead, with the numpy it found" diff --git a/prototype/examples/numpy-venvs/versions.env b/prototype/examples/numpy-venvs/versions.env deleted file mode 100644 index 78b5774..0000000 --- a/prototype/examples/numpy-venvs/versions.env +++ /dev/null @@ -1,4 +0,0 @@ -# The two numpy versions, read by run.sh and by provider.sh so that they never disagree. -# Override them to try others: NUMPY_A=2.3.4 NUMPY_B=2.4.3 ./run.sh -export NUMPY_A=${NUMPY_A:-1.26.4} -export NUMPY_B=${NUMPY_B:-2.0.2} From c0da96a3d52d5a43f7dd2e3b37c475edb376ec33 Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Sun, 1 Nov 2026 11:41:15 +0000 Subject: [PATCH 7/9] simplify numpy-venvs/provider.sh --- prototype/examples/numpy-venvs/provider.sh | 29 ++++++++++++++-------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/prototype/examples/numpy-venvs/provider.sh b/prototype/examples/numpy-venvs/provider.sh index 013f464..b8ba11d 100755 --- a/prototype/examples/numpy-venvs/provider.sh +++ b/prototype/examples/numpy-venvs/provider.sh @@ -10,7 +10,9 @@ set -euo pipefail HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) : "${NUMPY_A:?run.sh sets NUMPY_A and NUMPY_B}" "${NUMPY_B:?run.sh sets NUMPY_A and NUMPY_B}" -configuration=$(python3 -c 'import json, sys; print(json.load(sys.stdin).get("configuration") or "")') +# The request is one flat JSON object (schemas/target-provider-request), so its configuration is the +# string after that key. An absent or null one leaves this empty, and is refused below. +configuration=$(sed -n 's/.*"configuration"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p') case "$configuration" in a) version=$NUMPY_A ;; b) version=$NUMPY_B ;; @@ -19,13 +21,18 @@ esac # The canonical URI this working tree's results are filed under, as in archery-local. uri=https://example.org/numpy-venvs/demo.git -HERE=$HERE URI=$uri VERSION=$version python3 - <<'PY' -import json, os -print(json.dumps({ - "schema_version": "benchx/target-description/0.1.0", - "target": {"kind": "working_tree", "path": os.environ["HERE"], - "source": {"uri": os.environ["URI"], "type": "git"}}, - "how_built": {"summary": {"numpy": os.environ["VERSION"]}}, - "provider": {"name": "provider.sh", "version": "1"}, -}, indent=1)) -PY +# A JSON string's body: the backslash and the quote escaped, which is all a path or a version needs. +json_string() { printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g'; } + +cat < Date: Sun, 1 Nov 2026 12:54:58 +0000 Subject: [PATCH 8/9] venv example: simplify provider.sh to provider.py --- prototype/examples/numpy-venvs/README.md | 4 +- .../examples/numpy-venvs/benchx.conf.json | 2 +- prototype/examples/numpy-venvs/provider.py | 45 +++++++++++++++++++ prototype/examples/numpy-venvs/provider.sh | 38 ---------------- prototype/examples/numpy-venvs/run.sh | 2 +- 5 files changed, 49 insertions(+), 42 deletions(-) create mode 100755 prototype/examples/numpy-venvs/provider.py delete mode 100755 prototype/examples/numpy-venvs/provider.sh diff --git a/prototype/examples/numpy-venvs/README.md b/prototype/examples/numpy-venvs/README.md index 2b85f85..2425bc7 100644 --- a/prototype/examples/numpy-venvs/README.md +++ b/prototype/examples/numpy-venvs/README.md @@ -57,7 +57,7 @@ The versions (`NUMPY_A`, `NUMPY_B`), the Python and the number of rounds are set `{"summary": {"numpy": ""}}`, and that lands in each result's `environment.system.deps`, which the `environments` comparison treats as the varying coordinate. benchx knows nothing about venvs, conda or pixi: any command that prints facts about an environment does the same. -- `provider.sh` is the **target provider**. For configuration `a` or `b` it declares, in +- `provider.py` is the **target provider**. For configuration `a` or `b` it declares, in `how_built`, the numpy that side is meant to have. The collector has `verify: how_built`, so it compares the declaration with what it found. The declared object has the same shape as the collector's output, because the check compares key by key from the top. @@ -97,6 +97,6 @@ prototype does not have. This is how each corresponds: |---|---| | `run.sh` | the driver: venvs, the loop, the comparison, the check | | `benchx.conf.json` | the project file: the adapter and suite, the provider, the `deps` collector | -| `provider.sh` | the target provider: declares each side's numpy in `how_built` (`run.sh` exports the versions) | +| `provider.py` | the target provider: declares each side's numpy in `how_built` (`run.sh` exports the versions) | | `describe_env.py` | the collector: prints the numpy this Python imports | | `bench_suite.py` | three small numpy benchmarks | diff --git a/prototype/examples/numpy-venvs/benchx.conf.json b/prototype/examples/numpy-venvs/benchx.conf.json index be63add..94146f8 100644 --- a/prototype/examples/numpy-venvs/benchx.conf.json +++ b/prototype/examples/numpy-venvs/benchx.conf.json @@ -10,7 +10,7 @@ "warmup": { "mode": "count", "n_warmup": 1 } } }, - "provider": { "command": ["./provider.sh"] }, + "provider": { "command": ["./provider.py"] }, "collectors": { "deps": { "command": ["python", "{root}/describe_env.py"], diff --git a/prototype/examples/numpy-venvs/provider.py b/prototype/examples/numpy-venvs/provider.py new file mode 100755 index 0000000..2fdbd5f --- /dev/null +++ b/prototype/examples/numpy-venvs/provider.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""The target provider of this example (system-decomposition.md §3.4): one JSON request on stdin, +one target description on stdout. It builds nothing, so `describe` and `prepare` answer alike. + +Configuration `a` or `b` names a side. The description declares, in how_built, the numpy the +user means that side to have, in the shape describe_env.py prints. The collector in +benchx.conf.json compares the two: a side whose environment is not the one it was declared +with gets the warning collector-mismatch on every result. +""" + +import json +import os +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +# The canonical URI this working tree's results are filed under, as in archery-local. +URI = "https://example.org/numpy-venvs/demo.git" + + +def main(): + versions = {"a": os.environ.get("NUMPY_A"), "b": os.environ.get("NUMPY_B")} + if not all(versions.values()): + sys.exit("provider: run.sh sets NUMPY_A and NUMPY_B") + + # An absent or null configuration is refused below, like an unknown one. + configuration = json.load(sys.stdin).get("configuration") + if configuration not in versions: + print(f"provider: unknown configuration {configuration!r} (a or b)", file=sys.stderr) + sys.exit(2) + + json.dump({ + "schema_version": "benchx/target-description/0.1.0", + "target": { + "kind": "working_tree", + "path": str(HERE), + "source": {"uri": URI, "type": "git"}, + }, + "how_built": {"summary": {"numpy": versions[configuration]}}, + "provider": {"name": "provider.py", "version": "1"}, + }, sys.stdout, indent=1) + print() + + +main() diff --git a/prototype/examples/numpy-venvs/provider.sh b/prototype/examples/numpy-venvs/provider.sh deleted file mode 100755 index b8ba11d..0000000 --- a/prototype/examples/numpy-venvs/provider.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env bash -# The target provider of this example (system-decomposition.md §3.4): one JSON request on stdin, -# one target description on stdout. It builds nothing, so `describe` and `prepare` answer alike. -# -# Configuration `a` or `b` names a side. The description declares, in how_built, the numpy the -# user means that side to have, in the shape describe_env.py prints. The collector in -# benchx.conf.json compares the two: a side whose environment is not the one it was declared -# with gets the warning collector-mismatch on every result. -set -euo pipefail -HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) -: "${NUMPY_A:?run.sh sets NUMPY_A and NUMPY_B}" "${NUMPY_B:?run.sh sets NUMPY_A and NUMPY_B}" - -# The request is one flat JSON object (schemas/target-provider-request), so its configuration is the -# string after that key. An absent or null one leaves this empty, and is refused below. -configuration=$(sed -n 's/.*"configuration"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p') -case "$configuration" in - a) version=$NUMPY_A ;; - b) version=$NUMPY_B ;; - *) echo "provider: unknown configuration '$configuration' (a or b)" >&2; exit 2 ;; -esac -# The canonical URI this working tree's results are filed under, as in archery-local. -uri=https://example.org/numpy-venvs/demo.git - -# A JSON string's body: the backslash and the quote escaped, which is all a path or a version needs. -json_string() { printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g'; } - -cat < Date: Sun, 1 Nov 2026 15:29:34 +0000 Subject: [PATCH 9/9] Remove the plan file for UC-04 --- Plan_UC04_on_the_design.md | 777 ------------------------------------- 1 file changed, 777 deletions(-) delete mode 100644 Plan_UC04_on_the_design.md diff --git a/Plan_UC04_on_the_design.md b/Plan_UC04_on_the_design.md deleted file mode 100644 index 5a189fb..0000000 --- a/Plan_UC04_on_the_design.md +++ /dev/null @@ -1,777 +0,0 @@ -# Plan: UC-04 on the design - -Oct 27, 2026. - -This plan makes UC-04 work with what the design documents already -specify, and adds as little new machinery as possible. - -## Goal - -Make UC-04 work: compare two environments that differ in one dependency -(or build option), measured by separate `bx bench` calls in separate -environments, joined by a run key. - -## What UC-04 needs, and what is missing - -UC-04 holds the code, the commit and the machine fixed and varies the -environment: `numpy==1.26.4` against `numpy==2.0` under pandas, or two -builds of one source tree. Two interpreters cannot hold both numpy -versions, so no single invocation can do it. The user prepares both -environments, runs the benchmark in each, alternating round by round -under one session, and then compares. - -**What already works.** -- The comparator's `environments` profile pairs the sides' attempts by - round, and checks that the sides alternate by slot, that attempt counts - are equal, that everything but the varying coordinate agrees, and that - the sides differ. -- A label names each side (`provenance.labels`), and the round and slot - flow from the order into each result's `procedure`. -- `bx compare` in measure mode alternates two targets over rounds under - one run key. Both sides then run in the one environment `bx` was - started in, which fits build-option comparisons (the archery example) - but not a dependency that cannot share a process. - -**What is missing.** -1. **A separate runtime environment per side.** Three ways to provide it: - the runner enters each side's declared `activation`; the workbench - starts the runner inside each side's activated environment; or the - user's own script activates each environment and calls `bx` once per - order. This plan takes the third. It is UC-04's own sketch, and the - design makes the user responsible for preparing and entering the - environment, with the alternation a loop in the calling script - (`benchmark-environments.md` §2.2 principle 1, §5.7). benchx then - executes nothing it was not asked to. -2. **A way to place a run in a session (gap 1).** The sketch passes a - label, a session and a round on every call. `bx bench` takes - `--run-key` (the session) but no label, round or slot, and `bx run` - takes only an order file. The order builder that `bx compare` uses - already accepts all three, so the work is exposing them. -3. **A way to tell the sides apart (gap 2).** `sides-differ` refuses a - comparison in which nothing recorded differs. Two venvs over one - checkout record the same subject and the same environment: one - benchmark run from two identical venvs gave identical - `environment.system` and `subject`, and a numpy difference would be - invisible in the same way. The declared texts (`how_built`, - `activation`) travel in `provenance.info`, which is never identity. - The sides need a recorded, machine-readable dependency fact. -4. **The minimum as the estimator.** UC-04 compares the minimum timing - per environment. The comparator uses a paired median. This is shared - with other use cases, and the plan leaves it as a follow-up. - -Items 2 and 3 are the two gaps the work packages close (gap 1 and -gap 2). Item 1 is a choice the plan makes, and item 4 is deferred. - -## Design decisions - -Each decision follows the design documents. The reason is given with it. - -1. **Record a declared-vs-detected mismatch; never gate the run on it.** - - The detected value is recorded, the declaration stays in the work - order, and the result carries a warning (`benchmark-environments.md` - §3.4; `benchmark-result-schema.md` §4.2). - - A contradiction "is recorded as it is and never gates the run" - (`benchmark-environments.md` §2.2, principle 2). Gating is the - `verify` tier of an environment policy, per rule - (`runner-schema.md` §4), and the prototype defers policy. - - The prototype's collector check (`verify`, `expect`) gives - `collector-mismatch`. With `on_fail: warn` it records and the run - proceeds, which matches this decision. Its `refuse` default is a - documented departure (`prototype-design.md` §3), so the UC-04 - example uses `warn`. - - So there is no `on_mismatch` setting. - -2. **The user's collector describes the environment; benchx hard-codes no - environment kind.** - - The design's plugins are "read-only describers that know one build - system, package manager, or accelerator stack" - (`benchmark-environments.md` §3.3). A context collector is that - interface (`harness-adapter.md` §5.1): a command the user names, - which prints one JSON object of facts. - - Whatever extracts the facts is the user's: `conda list --json`, - `pixi list`, `pip freeze`, `numpy.show_config()`, `module list`, - `docker inspect`. benchx needs to know none of them. - - The facts land under `environment.system.` - (`harness-adapter.md` §5.1), which is in the `environments` axis, so - a difference makes the sides differ and shows in the comparison. - - A declaration can be checked against the collected facts: `verify: - how_built` and `expect` (see decision 1). So the need in gap 2 is - met with detection, a record in the results and a check, for any - environment kind, without detection built into benchx. - -3. **Collector output stays under `environment.system.`.** - - `harness-adapter.md` §5.1 puts collected facts there, with the - collector's name as their source. Moving one into the subject is a - placement rule (`benchmark-environments.md` §3.4), which is a - project decision, not something a collector's output keys decide. - -4. **The caller sets slot and round.** - - Alternation across interpreters "is a loop in the calling script - ... pass `round` and `slot` through" - (`benchmark-environments.md` §5.7). `round` is "set by the run - author" (`runner-schema.md` §3). benchx holds no queue (§5.3) and - wraps no measurement loop (§2.2, principle 4). - - The comparator already checks alternation, equal attempt counts and - that the sides differ (`benchmark-result-schema.md` §5.5), so wrong - numbering is caught there. - - So `bx bench` gets `--label`, `--round` and `--slot`, with no lock - and no slot counter. - -5. **Typed declared facts are deferred: `--component`, `--build-option`, - and the runner's `components` and `build`.** - - The design has a typed home for them. `components[]` and `build` are - work-order fields (`runner-schema.md` §3) that land in - `subject.components` and `subject.configuration`, and the prototype - refuses both (`DEFERRED` in `runner.py`). - - UC-04 does not need them. It needs the sides to differ in a recorded - fact, and a way to check a declaration against what is installed. - Decision 2 gives both for any environment kind. - - What they would add: the design's home for dependency versions - instead of `environment.system`; a declaration on each call, with - no provider or sidecar; no script for Python packages; and the - design's own warning names. What they cost: most of the new code - (a package manifest, a reconcile step, name-only resolution, new - flags) and two schema decisions. - - So they are a follow-up. See "Potential follow-ups". - -6. **No separate "verified but not identity" category.** - - Which facts split history is the identity policy's choice, not a - property of the fact (`benchmark-result-schema.md` §4.3; - `benchmark-environments.md` §3.4, last paragraph). - - So keeping a fact out of identity, as numpy-spin tracking needs, - is a policy question (see follow-ups), not a new kind of fact. - -7. **Per-side declared values come from the target description or the - setup file.** - - A collector's `verify: how_built` checks the collected facts - against the `how_built` object of the target description - (`prototype-design.md` §3). The check compares key by key from the - top level (`collectors._differences`), so the declared object - mirrors the structure the collector prints: a collector that prints - `{"summary": {"numpy": "2.0.2"}}` is checked with - `{"summary": {"numpy": "2.0.2"}}`. - - Each side has its own description: a provider configuration per - side (`bx bench @a`, `bx bench @b`), or a sidecar next to a build - directory (`system-decomposition.md` §3.4). `expect` lives in the - config, which both sides share, so it cannot differ per side. - - The setup file (`.benchx/setup.json`, `BENCHX_SETUP_FILE`) is for - declared facts when the party that prepares an environment is not - the one that runs benchmarks in it: in an `environments` run, "each - environment sets `BENCHX_SETUP_FILE` to its own file" - (`harness-adapter.md` §5.1). It is implemented. Its facts land - under `environment.system.setup`, in the axis, and are matched as a - whole because the prototype has no placement rules. - -## What the design already gives UC-04 - -Each item: the need, the design mechanism, the prototype today. - -- **Sides named by an environment label.** - Work order `labels` → `provenance.labels`. - Today: `bx compare --label` works; **`bx bench` has no `--label`**. -- **Interleaving over rounds, one session.** - `run_key`, `round`, `slot`, the caller's loop (§5.7). - Today: the fields flow to results; **`bx bench` cannot set them**. -- **A machine-readable varying coordinate.** - The `environments` profile axis: `subject.configuration`, - `subject.components`, `environment.system` (`compare.py`). - Today: implemented. -- **Environment facts, detected.** - The user's collector → `environment.system.`, in the axis. - Today: implemented (PR #71). -- **A check of a declaration against the collected facts.** - `verify: how_built` and `expect`; `collector-mismatch` with - `on_fail: warn`. - Today: implemented, as a documented departure from the design. -- **Declared per-environment facts.** - Setup file → `environment.system.setup` (`runner.py:177`), in the axis. - Today: implemented. Nobody has used it for this. -- **Typed pinned dependencies and declared build configuration.** - Work-order `components[]` and `build` - (`benchmark-environments.md` §8). - Today: refused (`DEFERRED`). Deferred by this plan. -- **Detected CMake options.** - The runner reads `CMakeCache.txt`. - Today: implemented (archery example). -- **Invariants: equal rounds, alternation, sides differ.** - Comparator, schema §5.5. - Today: implemented. - -**Consequence.** Gap 2 needs no new code. Either each venv sets -`BENCHX_SETUP_FILE` to a file that says `{"numpy": "2.0.0"}` (declared), -or the project names a collector that prints the dependency's version -(detected, and checkable). Both land in `environment.system`, which is in -the axis, so `sides-differ` should pass. This follows from reading the -code and has not been run, because `bx bench` cannot pass labels yet; the -tests cover it. - -## Design - -One rule from `benchmark-environments.md` §3.4 settles how a declaration -and a detection meet: - -> A coordinate holds the detected value where one exists. A declared -> value fills it only when nothing can be read back, with a warning. A -> disagreement keeps the detected value, keeps the declaration in the -> order, and adds a warning. No fact gates the run. - -Where each fact is declared, detected and stored: - -- **Dependency versions and other environment facts** (BLAS vendor, - install method, image digest). - Declared in the target description's `how_built` object, or the setup - file. Detected by the user's collector. Stored in - `environment.system.`, or `environment.system.setup` for the - setup file. -- **Build configuration of a CMake build directory.** - Detected by the runner's CMake reader. Stored in - `subject.configuration` (unchanged). -- **`how_built`, `activation`, `shell`, provider.** - Declared in the target description. Not detected. Stored in - `provenance.info`, never identity (unchanged). - -Warnings: `collector-mismatch` when a check fails and `on_fail` is -`warn`, and `collector-failed` when a collector cannot run. The design's -`requested-value-mismatch` and `build-config-unverified` come with the -typed-facts follow-up. - -## Work packages - -A and B are independent and give a working UC-04 with declared facts. C -adds detection and verification, with no new code. D is tests. - -### A. Place a run in a session (gap 1) - -- `cli.py`, `bx bench`: `--label NAME=VALUE` (repeatable), `--round N`, - `--slot N`. The order builder already takes `labels`, `round_` and - `slot` (`orders.py:30`), and so does the helper that builds a side's - order (`session.py:117`). `session.bench_target` and its call to - `_run_side` pass none of them today, although the compare loop does, - so thread them through from `_common`. No lock, no counter, no - `--interleave`. -- When a slot is given, write the order to `orders/order-.json`, - as `compare_targets` does. `bx bench` writes `orders/order.json` per - run key, so the loop's calls would otherwise overwrite each other's - order file. -- `--label` means different things in different commands: in `bx bench` - it is `NAME=VALUE` and names the side of one call; in measure-mode - `bx compare` it is `NAME=BASE,CONTENDER`; in read mode it is the name - alone. The `--help` text of each says so. -- `bx run` is unchanged. -- Refusals at the command line, before any order is built: - - `--label`, `--round` or `--slot` without an explicit `--run-key`. - The default run key is a timestamp per call, so two calls would - never share a run. - - `--label` or `--round` without `--slot`. The ingest key gets its - `:slot-N` suffix only when a slot is set (`runner.py:49`), so two - sides in one run would share an ingest key, and their differing - payloads would conflict at ingest. - - a duplicate label name. - - `--no-ingest` is allowed (nothing then joins the run, which is the - caller's choice). -- The comparator already refuses badly formed runs: non-alternating - slots, unequal attempts, sides that do not differ. Improve those - messages to say what to change ("run the sides in the same order each - round"). For `sides-differ`, the message says the sides differ only in - the label, and to describe the environment under test with a collector - or a setup file per environment. The whole package list is not hashed - into identity by default: that would start a new series for every - incidental package change. -- Show warnings in the comparison. The comparator collects the warning - codes (`collector-mismatch`, `collector-failed`) from each side's - results and lists them beside the facts that differ. The verdict does - not change, and the comparison is not marked `local_only`. - `comparator.md` §5 already lists quality caveats in the output, so - this needs no design change. -- The caller's loop, as the design says (§5.7), is documented in the - example. A helper that prints the next slot is a potential follow-up - (see below). - -### B. Declared per-environment facts (gap 2, declared half) - -- Document the setup-file route for `environments` runs, with a - two-venv example. No code. -- The example keeps each setup file outside the tree, or gitignores - `.benchx/`. An untracked file inside the tree changes the tree id and - marks the tree dirty (`harness-adapter.md` §5.1). - -### C. The collector route (gap 2, detected half) - -Documentation and an example. No new code. - -- The two-venv demo (see "Demo: two venvs, two numpy versions"): - `benchx.conf.json` names a collector, `deps`, whose command prints - `{"summary": {"numpy": ""}}`, read from the interpreter under - test. The loop of package A runs it once per side. -- Per-side checking: each side's provider configuration returns a - description whose `how_built` is an object that mirrors the - collector's output, `{"summary": {"numpy": "2.0.0"}}`. The collector - has `verify: how_built` and `on_fail: warn`. A fact that - disagrees gives `collector-mismatch` on every result, the run - proceeds, and the collected value is what is recorded. -- Document the contract: the command runs once per order, in the - environment the user entered, outside the timed region. The summary is - what lands in the results, so it should print the same value for the - same environment, with no paths, timestamps or unsorted lists - (`harness-adapter.md` §5.1, last paragraph). -- Document that a check's default `on_fail` is `refuse`, a departure - recorded in `prototype-design.md` §3 and `harness-adapter.md` §5.1. - The example sets `warn`. Leave the departure alone; revisit it when - the environment policy lands. -- The collector output's sha256 stays in identity (`collectors.py:104`); - that is a follow-up. -- Collectors run commands. `bx bench` and `bx compare` run the project's - own, and `bx run` needs `--allow-collectors`. - -### D. Tests - -Case, then expected result. - -- `bx bench --label env=a --round 0 --slot 0`, then b/0/1, a/1/2, b/1/3 - → the order carries the fields; results carry `provenance.labels`, - `procedure.round`, `procedure.slot`. -- `--label env=a` without `--slot` → refused, naming `--slot`. -- `--label`, `--round` or `--slot` without `--run-key` → refused. -- Slots a, a, b, b → fails `alternating-slots`. -- Unequal round counts → fails `equal-attempts`. -- Two sides, setup files differ in `numpy` → - `bx compare --run S --profile environments --label env --baseline a` - passes `sides-differ`. -- Same, setup files identical → fails `sides-differ`; the message names - where the sides can differ. -- A collector whose summary differs between the sides (`numpy`, or - `blas: mkl` against `blas: openblas`) → `sides-differ` passes, and the - comparison shows the difference under `environment.system.`. -- Same summary on both sides → fails `sides-differ`. -- `verify: how_built` with a declaration that matches the collected - summary → no warning on any result. -- `verify: how_built` with `on_fail: warn`, and a declared value that - disagrees → every result carries `collector-mismatch`, the run - proceeds, and the collected value is recorded. -- The same check with the default `on_fail` → the order is refused - before any case. -- Comparison where one side has `collector-mismatch` → the output lists - the warning beside the differing fact; the verdict and `local_only` - are unchanged. -- Plain `bx bench` → unchanged. -- End to end, two venvs differing only in a tiny pure-Python package → - verdicts computed; skipped without `uv`. -- CI runs the numpy-venvs demo (see the demo section). - -## Non-Python environments - -UC-04 §2 includes build systems, so the plan does not depend on any -one package manager or environment kind. The environment may be a venv, -conda, mamba, pixi, environment modules, Nix or a container. The user -prepares and enters it, and benchx hard-codes none of them: - -- **Any environment kind: a collector.** The user-side hook for - describing an environment is a context collector - (`harness-adapter.md` §5.1), named in `collectors` in - `benchx.conf.json`. It is a command that prints one JSON object of - facts, for example from `conda list --json`, `pixi list`, - `module list` or `docker inspect`. benchx never needs to know the - tool. - - It runs once per order, before the cases, in the harness's working - directory and environment, so under `conda run -n a` or - `pixi run -e a` it sees that environment. - - The full output is kept as an artifact. The results carry - `environment.system.`: the object's `summary` (if it has one) - and a sha256 of the full output. - - `environment.system` is in the `environments` axis, so a difference - in the summary, such as the BLAS flavour (`mkl` against `openblas`), - makes the sides differ and shows in the comparison. - - A collector that prints the whole sorted package list makes any - package difference count. That is the project's choice, and it - covers undeclared dependencies for projects that want it. - - A declaration can be checked against the collected facts - (`verify`, `expect`; see decision 7). - - Limits: the facts are not typed `components`. The sha256 of the full - output is in identity (see the follow-up on collector output), so - collectors should print the same value for the same environment. -- **Containers, libraries outside Python** (Boost, OpenSSL, BLAS). The - same collector, or a key in the setup file. A declared image digest in - the setup file lands under `environment.system.setup`, in the axis, so - two images differ there. -- **CMake.** Detection exists (`CMakeCache.txt` into - `subject.configuration`). A declared `build` is part of the - typed-facts follow-up (the archery README says the prototype refuses - it today). -- **Single-process comparison** (`bx compare @a @b`, as in - archery-local). One call runs both sides in the one environment `bx` - started in. A collector runs in that environment for both sides, and - cannot tell them apart. One setup file cannot differ between them - either (its default location is under the source directory, which both - sides share). So a dependency difference between the two sides of one - call has no route in this plan, apart from CMake-detected options. - Declaring `build.options` in each side's target description is the - route, and it belongs to the typed-facts follow-up. archery-local's - `deps-old` / `deps-new` case waits for it. - -## Design-doc changes - -Few documents change, because the plan uses mechanisms the design -already has. - -- **`UC-04-cross-env.md` is kept as it is.** Its spellings differ from - the prototype's. The map from one to the other lives next to the - two-venv example, in `examples/numpy-venvs/README.md` (see "Demo: two - venvs, two numpy versions"), which links to the use case. -- **`prototype-design.md`.** Record that `bx bench` takes `--label`, - `--round` and `--slot`, and describe the collector route for UC-04. - -`benchmark-environments.md`, `harness-adapter.md`, `runner-schema.md`, -`comparator.md`, `system-decomposition.md` and the result schema do not -change. - -## numpy-spin (benchx_examples PR #1) - -This section is informational. The plan makes no change to `benchx_user` -(the local checkout of `benchx_examples`) or to the numpy-spin demos. - -The PR has three demos. Only one is UC-04-shaped. - -- **Thread settings (`num_threads.sh`).** The same benchmark under three - run keys, with `OPENBLAS_NUM_THREADS` set to 10, to 1, and unset. It - runs no comparison. Left as it is; this plan does not change it. The - acceptance test for packages A and C is the two-venv demo below. -- **Build check (collector).** Gating a run on numpy's build facts. - Unchanged; works through PR #71's `verify` today. Not UC-04. -- **Revision tracking (`track.sh`).** UC-03-like history across - revisions. Separate follow-up. It needs the numpy version *out* of - identity, which is an identity-policy decision, not a fact-placement - one. - -**Threads.** `num_threads.sh` runs the benchmark three times, under -three run keys, and prints `bx head --table`. It sets no label, round or -slot, so nothing joins the three runs into a comparison. The variable is -set in the shell, not with `-e`, so it is the inherited environment of -the measured process and lands in `environment.system.process.environment` -(observed). A variable the runner sets would be a workload parameter -instead (`benchmark-result-schema.md` §4.2). With package A the same -three runs could be a UC-04 comparison, because that field is in the -`environments` axis. The schema's own view of a thread sweep is a -`variants` comparison within one suite, which the prototype does not have -(`compare.py` has `revisions` and `environments` only). The plan changes -none of this. Moving the variable to `protocol.threads` is for *enforced* -caps (`threads.max`, environment policy, deferred). It would not apply to -an inherited one. The flags of package A are additive, so the script -keeps working as it is. - -**Tracking.** Keeping the numpy version out of identity is the identity -policy's job, not a property of the fact (design decision 6). The -prototype has one fixed policy (`identity.POLICY_DATA`). Either the -follow-up adds a project-level identity policy (an ignore list in -`benchx.conf.json`), or the collector's sha256 leaves identity so that -numpy's version lives only in the collector's artifact, checked by -`verify`. The second option needs the sha256 change: while the sha256 -of the full output is in identity (`collectors.py:104`), any collector -fact that changes splits the series, `verify`-only or not. This is not -part of UC-04. Both options are deferred (see follow-ups). - -## Demo: two venvs, two numpy versions - -A separate demo, at `prototype/examples/numpy-venvs/`, beside -`pyperf-bench`. It does not touch the numpy-spin demos. It is the -acceptance test for packages A and C: UC-04 as its own sketch has it, -two environments that differ only in numpy, driven by a shell script. - -Files: -- `README.md`: what it shows, what to install, how to run it, what to - expect, and the map from UC-04's spellings to the prototype's (below). - The demo assumes `uv`. numpy 1.26.4 has wheels only up to Python - 3.12, so both venvs use Python 3.12. -- `benchx.conf.json`: project name, the pyperf adapter with suite - `bench_suite.py` and a small precision, `paths` into `out/`, a - `provider` (`./provider.sh`), and the collector: - `"collectors": {"deps": {"command": ["python", "{root}/describe_env.py"], - "verify": "how_built", "on_fail": "warn"}}`. -- `bench_suite.py`: three numpy functions, such as a matrix product, a - sort and a sum. -- `describe_env.py`: prints `{"summary": {"numpy": numpy.__version__}}` - and nothing else, with no paths or timestamps. -- `provider.sh`: answers `describe` and `prepare` with the same target - description and builds nothing. The target is this directory as a - working tree. `how_built` is an object that mirrors the collector's - output, `{"summary": {"numpy": ""}}`, with the version for - configuration `a` or `b` from `NUMPY_A` and `NUMPY_B`, which `run.sh` - exports. -- `run.sh`: the driver, below. -- `.gitignore`: `.venv-*/` and `out/`. Untracked venvs inside the tree - would mark it dirty, so the comparison would be `local_only`, and - slow the tree hashing. benchx leaves only the store, the results and - the `--out` paths out of the tree identity. -- `out/`: generated. - -The `python` in the collector command is found on `PATH`, so the script -enters each environment the way a user does, by sourcing the venv's -`activate`. Activating a venv replaces the one activated before, so -nothing is deactivated and no subshell is needed. This was checked with -`uv venv` environments under `set -euo pipefail`: the second `activate` -leaves only its own `bin` on `PATH` and resets `VIRTUAL_ENV`, which -matters under `uv run` in CI, where it points at the outer environment. -benchx executes nothing for any of this. - -`run.sh` (the file is the reference) sets its values at the top, in -upper case: `NUMPY_A=1.26.4`, `NUMPY_B=2.0.2`, `PYTHON_VERSION=3.12` and -`ROUNDS=3`, which is the fewest rounds that give a verdict -(`compare.min_rounds` defaults to 3). It then does, in order: -1. Creates `.venv-a` and `.venv-b` with `uv`, each with its numpy, - `pyarrow<25`, pyperf and benchx (`-e ../..`). benchx imports pyarrow, - and pyarrow 25 refuses to import under numpy 1.x. -2. Runs the caller's loop: for each round and side, it sources the - venv's `activate` and calls `bx bench "@$SIDE"` with `--run-key`, - `--label env=numpy-`, `--round`, `--slot` and `--out`. -3. Compares: `bx compare --run S --profile environments --label env - --baseline numpy-`, kept in `out/compare.txt`. The script - exits nonzero unless the `differs:` line names `deps.numpy` and no - invariant failed. -4. Runs the check: it declares version `b` for a run in venv `a`, and - exits nonzero unless that run's results carry `collector-mismatch`. - -What the comparison shows, from a run of the demo as it stands -(`uv`, Python 3.12, numpy 1.26.4 against 2.0.2): -- A `differs:` line that names the facts that differ - (`coordinates.environment.system.deps.numpy` and `...deps.sha256`), - and for each side its declared `how built` object, which carries the - version. The comparison names what differs and does not print the - collected values, because `unmatched_differences` holds JSON Pointers. -- A verdict per benchmark. -- With both sides in the same venv, `sides-differ` fails, with the - message that says to describe the environment under test. - -**Spelling map.** The demo's README carries the map from UC-04's -spellings (`docs/use-cases/UC-04-cross-env.md`, which is kept as -written) to the prototype's, and links to the use case. `run.sh` above -is written in the prototype's. The map: - -- `benchx new-session` (`SESSION=$(...)`) → no command. The session is - the run key, chosen by the caller: `S=numpy-$(date +%s)`. -- `--session-id "$SESSION"` → `--run-key "$S"`. It lands in - `provenance.run_key`. -- `benchx run run_benchmark.py` → `bx bench`. With a module and - functions, `bx bench MODULE.py [FUNCTION ...]`; with a prepared target, - `bx bench @CONFIG --suite NAME`. `bx run` takes an order file, not a - script. -- `--env-label numpy-1.26.4` → `--label env=numpy-1.26.4`. The field - `env_label` is `provenance.labels.env`. -- `--round "$round"` → `--round N` and `--slot K`. UC-04 uses one number - for two things. The prototype separates the round, which pairs one - attempt of each side (`procedure.round`), from the slot, the realized - position in the interleaved sequence (`procedure.slot`). UC-04's - `interleave_position` is the slot. The caller's loop keeps the slot - counter. -- `benchx compare-environments --session-id S --baseline A --variant B` - → `bx compare --run S --profile environments --label env - --baseline A`. The profile is UC-04's `cross-environment-comparison`. - There are exactly two sides, so the variant is the other one. -- `.venv-baseline/bin/benchx` → the `bx` of that environment, such as - `.venv-a/bin/bx`. -- The output table (`benchmark_name`, `env`, `time`, `vs. baseline`) → - the comparison output: verdict groups with the effect, the noise and - the rounds, and the facts that differ. -- UC-04's `session_id`, `env_label` and `interleave_position` fields → - `provenance.run_key`, `provenance.labels.env` and `procedure.slot`. - The schema adds no fields for a profile - (`benchmark-result-schema.md` §5.5). - -**CI.** The `prototype tests` workflow (`.github/workflows/tests.yml`) -already runs `examples/pyperf-bench/run.sh` and -`examples/archery-local/run.sh` after the tests, on a matrix of ubuntu -and macOS with Python 3.11 and 3.13. The plan adds a step for this demo: - -```yaml -- name: Run the numpy-venvs example - if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' - shell: bash - run: >- - uv run --extra test --python ${{ matrix.python-version }} - bash examples/numpy-venvs/run.sh -``` - -- The demo makes its own venvs with `uv venv --python 3.12`, so the - matrix Python only runs the outer script. Running it in every matrix - cell would repeat the same work, so the step runs in one cell. -- The step needs network access, for the numpy wheels and pyperf, which - the runners have. -- The workflow's `paths` filter already covers `prototype/**`. -- The venvs and `out/` are git-ignored. - -## Order of work - -Tests (package D) land with the package they test. - -1. A, with its tests from D: the flags, the refusals, the order file per - slot, the comparator's messages and warnings. Then B, with the - setup-file tests (the two sides differ, the two sides are identical). - UC-04 works, declared only. -2. C, with its tests from D: the collector tests, the two-venv demo and - its `run.sh`, and the CI step, all in one change. The demo needs the - flags of package A, so this follows step 1. UC-04 then works with - detection and a check. -3. The design-doc changes (`prototype-design.md`). - -## Potential follow-ups - -Not part of this plan. Build only if the need shows up in practice. - -- **Typed declared facts: `--component`, `--build-option`, and the - runner's `components` and `build`.** Deferred (decision 5). The flags - would fill the work order's `components[]` and `build.options` - (`runner-schema.md` §3), which land in `subject.components` and - `subject.configuration`. A name with no revision means "detect it". - - Why deferred: UC-04 needs the sides to differ in a recorded fact and - a check of a declaration. A collector with `verify` gives both, for - any environment kind. - - What it adds: the design's home for dependency versions; a - declaration on each call, with no provider or sidecar; no script for - Python packages; and the design's warning names with no gating. - - What it contains: - - The snapshot reads installed versions with `importlib.metadata` - (`benchmark-environments.md` §3.1, Runtime) and records the - inventory as a manifest artifact. That is tied to Python, not to - the environment manager, so it works the same in a venv, conda, - mamba or pixi. No manager's own metadata is read. - - A reconcile step for the rule in the Design section: both equal, - recorded; both differ, detected recorded with - `requested-value-mismatch`; only detected, recorded; only - declared, recorded with `build-config-unverified` (and a sibling, - `component-unverified`, for components, added to - `benchmark-environments.md` §3.4). - - A name-only component is resolved by `bx bench` when it builds the - order, because the work-order schema requires `revision` and - `bx bench` writes resolved orders. If nothing can be detected, the - request is refused with a message that says to give a revision. - This is an unsatisfiable request, refused visibly - (`benchmark-environments.md` §2.2, principle 2), not a gate on a - contradiction. A remote runner would have to fill the version in - itself and record the completed order as the artifact that results - cite (`runner-schema.md` §3.2). - - `source` takes a package URL (`pkg:pypi/numpy`, `type: pypi`) or a - repository URL; `revision` is the version of a released package - or a commit id. `runner-schema.md` §3 says so. - - The flags: when a flag and a config value name the same thing, the - flag wins (the project's usual order is flag, file, default). - - `build.options` in each side's target description. It is the route - for a single-process comparison, and for archery-local's - `deps-old` / `deps-new` case. - - An optional `placement` block in `benchx.conf.json` that maps a - collector name and key, or a setup-file key, to its destination - with the collector (or `declared`) as source - (`harness-adapter.md` §5.1 says placement rules exist; the - prototype has none). - - Cost: most of the code in the first version of this plan, and two - schema and documentation decisions. -- **Move the numpy-venvs demo to `benchx_examples`.** That repository - (checked out locally as `benchx_user/numpy-bench`) holds worked - examples of using benchx, such as `examples/numpy-spin`, and the - demo's own audience is a user, not the prototype's tests. Moving it - would change a few things: - - It installs benchx from a released version or a git ref, not with - `-e ../..`. - - Its README is listed in that repository's README. - - The CI step in this repository goes. That repository has no CI - workflow today, so it would need one to keep the demo running. - - The provider and `run.sh` move with it, and nothing in - `prototype/` depends on them. - It is deferred because the demo has to run against the prototype's - own code while package A is being built, and because it needs - `benchx_examples` to settle how it pins a benchx version. -- **`bx session next`.** A command that reads the store and prints the - next `slot` and `round` for a run key and label, for example - `bx session next --run-key S --label env=a`. It would help when the - two sides are not launched from one script: a run resumed after a - crash, or sides started from different shells or CI steps. It is left - out because the design makes the loop the caller's - (`benchmark-environments.md` §5.7) and `round` is set by the run - author (`runner-schema.md` §3). A helper that reads the store also - needs locking: two callers can race for the same slot, and a crashed - call leaves a gap that looks like a skipped slot. The comparator - already catches wrong numbering (`alternating-slots`, - `equal-attempts`). -- **A label-name check at bench time.** Calls in one run can use - different label names by mistake, for example `--label env=a` and - `--label environment=b`. Both calls succeed and ingest. Today the - comparator catches it, but only at compare time, with "a result has no - /provenance/labels/env", after all the measuring is done. Two steps, - smallest first: - - Make the comparator's message name the label keys it found in the - run ("found `env`, `environment`"). - - Have `bx bench` warn, not refuse, when its label names differ from - those already stored for the run key. This reads the store before - measuring, and cannot see `--no-ingest` runs or another store, so it - belongs with the session bookkeeping above. -- **A project identity policy.** The prototype has one fixed identity - policy (`identity.POLICY_DATA`), pinned by a test to its name, so a - project cannot say which facts must not split its series. The numpy - tracking demo needs this: numpy's own version is already the axis (the - subject revision), but the collector records it too, so each revision - becomes its own series. Options: an ignore list of JSON pointers in - `benchx.conf.json`, added to the built-in policy, with the identity - schema name changing with the list; or a separate file following - `identity-policy/0.1.0`, which is closer to the design. Editing the - policy starts new series for results already stored. Not UC-04: there - the dependency version is the thing being compared. -- **A package diff in the comparison.** The sides can differ in - packages nobody described. If they differ only there, `sides-differ` - fails. If they also differ in a described fact, the comparison passes - and attributes the whole difference to it: a hidden confound - (`benchmark-environments.md` §4, recommendation 1). The comparator - would list the packages that differ between the two sides' package - lists ("also differs: scipy 1.13 → 1.14") beside the described fact, - as a caveat, in the way it already carries differences in ignored - facts. The list would come from a collector's artifact, or from the - manifest of the typed-facts follow-up. - The catch: the list is an artifact file the result references, not - stored in the store (`comparator.md` §2.1 takes its inputs from the - store). The diff works only when the artifact files are reachable, for - example in the same results directory, and the comparison must say - when they were not. A name-to-version map in `provenance.info` would - be in the store, but adds hundreds of entries to every result. -- **Collector output in identity.** The runner stores each collector's - `summary` and a sha256 of its full output in - `environment.system.` (`collectors.py:104`), and the default - policy matches both. Any collector fact that changes splits the - series, even one used only for `verify`. Dropping the sha256 from - identity (the artifact already records it) would let such facts live - in the artifact alone. It is smaller than the policy item and removes - the numpy tracking split. It changes PR #71's design, so its author - should agree. -- **The minimum as the run-mode estimator.** UC-04 §7 compares the - minimum timing per environment. The comparator compares the median of - each result's observations per round, takes the mean of the relative - differences across rounds, and calls a change when the effect exceeds - `k` times its standard error (`compare.py`, `_unit`). On noisy - machines the two can disagree, so the comparator may say "no change - detected" where UC-04's example shows `+9.2%`. The plan keeps the - comparator as it is, and UC-04 §7 is left as written. An optional - `--estimator min` would take the minimum of the observations per - round instead. It needs a new method version, and - the comparison document's `estimator` field, hard-coded to the median, - would change. UC-02 wants the minimum too, and the choice of run-mode - method is open in `comparator.md` §7 question 1, so it is best settled - there. A table of one number per environment, as in UC-04's example, - is a smaller rendering-only addition. -- **Retiring the setup file.** If the typed-facts follow-up is built, - the setup file carries the same kind of fact through a second - channel. Retiring it would mean removing - `snapshot.setup_facts`, the `system["setup"]` assignment in - `runner.py`, `BENCHX_SETUP_FILE`, `test_setup_file`, and the - setup-file mentions in `prototype/README.md` and `prototype-design.md`. - `--component`, `--build-option` and the description's `build` would - replace it. The costs: the design keeps the setup file for the case - where the party that prepares an environment is not the one that runs - benchmarks (`harness-adapter.md` §5.1). Without it, the preparer's - facts reach benchx through a sidecar next to a build directory, or - through flags the caller's script passes, and two venvs over one - shared working tree have only the flags. It is a design change - (`harness-adapter.md` §5.1 and `benchmark-environments.md` §3.2). And - results already stored keep `environment.system.setup`, while new ones - would not, so those series would start fresh. -- **Peak memory.** UC-04 §6 lists peak memory (memray) as optional. No - adapter measures it yet, so it is out of scope here. - -## Open questions - -None. What was left open is under "Potential follow-ups".