ENH: implement UC-02 comparisons over variants

docs/benchx.md

@@ -48,9 +48,13 @@ bx compare BASELINE CONTENDER --profile revisions|environments
            [--project P] [--label NAME=BASE,CONTENDER]
            [--run-key KEY] [--out DIR] [--no-build] [--k 3] [--json]
 
+# measure mode, one target: alternate the two values of a benchmark parameter
+bx compare TARGET --profile variants --adapter pyperf|asv --suite NAME --rounds R
+           [--vary NAME] [--baseline VALUE] [--filter REGEX] [...]  # the options above, except --label
+
 # read mode: compare results already measured
-bx compare --run KEY --profile revisions|environments --baseline VALUE
-           [--label NAME] [--results DIR] [--k 3] [--json]
+bx compare --run KEY --profile revisions|environments|variants --baseline VALUE
+           [--label NAME] [--vary NAME] [--results DIR] [--k 3] [--json]
 
 # low-level command: run a handwritten work oreder
 bx run <order.json> [--out DIR] [--no-ingest]
@@ -84,8 +88,8 @@ pyperf and for Google Benchmark. The store and the results directory are left ou
 measured tree's identity when they sit inside it. `.benchx/setup.json` (declared
 environment facts) is a separate file, written by whoever prepares an environment.
 
-Each target is `BUILD_DIR[:SOURCE_DIR]` or `@CONFIG[=SOURCE_REF]`. Two targets
-select measure mode and `--run` selects read mode; mixing them is an error.
+Each target is `BUILD_DIR[:SOURCE_DIR]` or `@CONFIG[=SOURCE_REF]`. Two targets (one for
+`--profile variants`) select measure mode and `--run` selects read mode; mixing them is an error.
 
 `@CONFIG` asks the project's **target provider**, the command named by `provider.command`
 in `benchx.conf.json` (it runs with the project root as its working directory). The provider speaks the contract of
@@ -131,6 +135,39 @@ and `--slot K` place each call in the session, and `bx compare --run KEY --profi
 `examples/numpy-venvs` in [benchx_examples](https://github.com/benchx-dev/benchx_examples)
 is the worked example, with a shell script that drives it.
 
+Two functions of one benchmark ([UC-02](https://benchx-dev.github.io/benchx/use-cases/UC-02-variants/)) are the two values of
+one parameter, declared as pyperf and asv benchmarks declare theirs. Build the data once, outside the
+function (at module level, or behind `functools.cache`), because pyperf times the whole body:
+
+```python
+IMPLS = {"baseline": pandas_native_truncate, "variant": pandas_via_narwhals_truncate}
+
+def bench_truncate(role):
+    IMPLS[role](data())
+
+bench_truncate.params = ["baseline", "variant"]
+bench_truncate.param_names = ["role"]
+```
+
+One command alternates the two values over the rounds and compares them (`adapter.name` in
+`benchx.conf.json` makes `--adapter` optional):
+
+```console
+$ bx compare . --profile variants --adapter pyperf --suite bench_suite.py --rounds 5
+```
+
+The values are recorded as `repr()` strings. The baseline is the first declared value, and
+`--baseline "'variant'"` swaps the sides (spelled as recorded, quotes included). `--vary NAME`
+names the parameter and is needed only when the selected benchmarks have more than one: every
+combination of the others is then its own unit (`bench_truncate [size=1000]`), and `--filter`
+narrows the benchmarks. A benchmark without the parameter runs on neither side, and a failed
+attempt on either side makes its unit indeterminate (`side-failed`). The verdict is the
+comparator's noise-aware one; there is no tolerance flag yet. As for UC-04, the loop can be the
+caller's: one `bx bench` call per value (a case name such as `"bench_truncate('baseline')"`, or
+`--filter "'baseline'"`) with `--run-key`, `--round` and `--slot`, then `bx compare --run KEY
+--profile variants --baseline "'baseline'"`. One `bx bench` call that runs both values is not an
+alternating run, and the comparison says so.
+
 `bx bench PATH --adapter asv` runs an [asv](https://asv.readthedocs.io) benchmark suite
 unchanged, reading asv's benchmark format as a language and timing it with pyperf;
 neither asv nor asv_runner is used. PATH is a directory holding `asv.conf.json` (its
@@ -199,9 +236,11 @@ every result is pyperf. Results also record the installed project's version
   are not clean), writes one order per side and
   round, alternates them over the rounds, delivers every result to the local
   store, and hands the run to the comparator. It builds nothing: both build
-  directories must already exist.
-- **The comparator** (`bx compare --run`) runs in run mode for the `revisions` and
-  `environments` profiles, each a projection built in code from the identity
+  directories must already exist. For `variants` it has one target: it lists
+  the cases once, splits them by the two values of one parameter, and writes
+  one order per value and round.
+- **The comparator** (`bx compare --run`) runs in run mode for the `revisions`,
+  `environments` and `variants` profiles, each a projection built in code from the identity
   policy (`benchx/compare.py`): the axis tells the sides apart, everything the
   policy matches must agree, and it checks the profile's invariants (`revisions` also
   requires one tree id per side and, for a benchmark recorded apart from the
@@ -210,7 +249,9 @@ every result is pyperf. Results also record the installed project's version
   sides by round, and applies `benchx/paired-relative/v1` (effect = mean
   relative paired difference, noise = its standard error, change when
   |effect| > *k* × noise). `environments` names its sides by the label
-  `--label` gives. `--results` compares result files through a throwaway
+  `--label` gives, and `variants` by the two values of one workload parameter
+  (`--vary`); under `variants` a failed attempt on either side fails its unit
+  (`side-failed`). `--results` compares result files through a throwaway
   store: the ad hoc path. Condition measurements (`measurement.kind:
   condition`) are not compared, and deterministic quantities are deferred.
 
@@ -224,9 +265,18 @@ $ pip install benchx
 
 ```console
 $ uv venv && uv pip install -e '.[test]'   # editable; a plain `uv pip install .` works too, the schemas ship in the package
-$ .venv/bin/python -m pytest tests     # 344 tests, fake Google Benchmark binary, no compiler
+$ .venv/bin/python -m pytest tests     # 470 tests, fake Google Benchmark binary, no compiler
 ```
 
+`tests/test_e2e_numpy_revisions.py` is an end-to-end test that builds NumPy with
+`spin` and compares two tags in one `bx compare`. It is skipped unless
+`BENCHX_E2E_NUMPY` (a numpy git checkout) and `BENCHX_E2E_EXAMPLE` (the
+`numpy-revisions` example directory) are set and `spin` is on the PATH; it also
+needs the project-file entries of PR #65 and PR #71. `BENCHX_E2E_WORKTREES`
+keeps the builds between runs, `BENCHX_E2E_REVISIONS` picks the two tags,
+`BENCHX_E2E_SPIN_EXAMPLE` (the `numpy-spin` example directory) adds a single-run
+scenario, and the test's docstring has the rest.
+
 A regular (non-editable) install ships the test suite too, so an installed copy can test itself:
 `pip install 'benchx[test]'` followed by `python -m pytest --pyargs benchx`.
 

docs/design/benchmark-result-schema.md

@@ -497,16 +497,16 @@ Results carry facts; a **profile** is a comparator-time check that a set of resu
 | Profile | Varies | Must agree | Invariants checked over the run |
 |---|---|---|---|
 | `single-run` ([UC-01](../use-cases/UC-01-short-slug.md)) | workload parameters | protocol | none beyond attempt integrity |
-| `variants` ([UC-02](../use-cases/UC-02-variants.md)) | one or more workload parameters chosen by the caller, `parameters.role` by default | subject tree id, subject configuration, environment, protocol, every other workload parameter | equal attempt counts per side; sides alternate by `procedure.slot` |
+| `variants` ([UC-02](../use-cases/UC-02-variants.md)) | one or more workload parameters chosen by the caller | subject tree id, subject configuration, environment, protocol, every other workload parameter | equal attempt counts per side; sides alternate by `procedure.slot`; a separately recorded benchmark has one `benchmark_tree` across both sides and all rounds, invariant `one-benchmark-tree` |
 | `revisions` ([UC-03](../use-cases/UC-03-compare.md)) | subject tree id | workload variant, subject configuration, environment, protocol | equal attempt counts per side; sides alternate by `procedure.slot`; each side has exactly one tree id, so a side whose dirty state is `unknown` and has none fails, and the two differ; a separately recorded benchmark has one `benchmark_tree` across both sides and all rounds, invariant `one-benchmark-tree` |
 | `environments` ([UC-04](../use-cases/UC-04-cross-env.md)) | one pinned component, subject configuration, or `environment.system` field, named by `labels` | subject revision, workload variant, host environment, protocol | exactly two label values whose reported coordinates differ; equal attempt counts per label; sides alternate by `procedure.slot` |
 | `cross-machine` (template example, pending a use case) | environment identity | subject tree id, workload variant, subject configuration | each side references an anchor result from the same run via `provenance.references` |
 
-**Sides are chosen at comparison time.** The coordinate a profile varies is an argument to the comparator, not a declaration by the producer. Workload parameters are always recorded, so any parameter or set of parameters can be the varying one: `stable` for a sort-algorithm study, `fast_path` for a fast-path deficit, `{role, size}` for a two-factor design. Each distinct assignment of the chosen parameters is one side, and every other coordinate must agree. `role` is only the conventional name that UC-02's `--baseline`/`--variant` flags write; nothing in the schema privileges it. A comparison that was not planned when the run was made is therefore the same operation as one that was, provided the run contains the sides; what a run did not measure, the comparator cannot invent, and a planned run is what guarantees the interleaving invariant.
+**Sides are chosen at comparison time.** The coordinate a profile varies is an argument to the comparator, not a declaration by the producer. Workload parameters are always recorded, so any parameter or set of parameters can be the varying one: `stable` for a sort-algorithm study, `fast_path` for a fast-path deficit, `{role, size}` for a two-factor design. Each distinct assignment of the chosen parameters is one side, and every other coordinate must agree. A comparison has two sides (`comparator.md` §1), so the chosen parameters must take exactly two joint assignments in the run. `role` is only the conventional name that UC-02's `--baseline`/`--variant` flags write; nothing in the schema privileges it. A comparison that was not planned when the run was made is therefore the same operation as one that was, provided the run contains the sides; what a run did not measure, the comparator cannot invent, and a planned run is what guarantees the interleaving invariant.
 
 A comparison in which any side is not `clean` is **local-only** under the profiles that allow it (`single-run`, `variants`, `revisions`, `environments`): the comparator produces it, labels it so, names the sides that are not clean, and never promotes it into tracked history. The tree id is what makes each side internally consistent even though neither may have a commit. Under `revisions` this is the comparison of an uncommitted change against HEAD, or against another revision, that the one-off and Arrow local stories want (UC-03 §10). What `revisions` still requires is that every side can be identified: a side whose dirty state is `unknown` has no tree id (§4.1), so it cannot be told apart from the other side or checked for a change during the run, and it fails the invariant that each side has exactly one tree id, so its units get no verdict.
 
-A benchmark is **recorded separately** from the subject when its suite lives outside the subject's checkout (`runner-schema.md` §3.3). Its `benchmark_tree` is then the suite file's own id and differs from the `subject_tree`; this includes a suite in another checkout of the same repository, which has the same source URI but its own file id. A benchmark inside the subject's checkout has `benchmark_tree` equal to `subject_tree` and the subject's own source and revision as `benchmark.source` and `benchmark.revision`; a result whose two trees are equal but whose `benchmark.source` differs from its `source` is malformed. Under `revisions` a separately recorded benchmark must be one file on both sides and in every round (invariant `one-benchmark-tree`): UC-03 requires the sides to agree on everything under code identity, and an edit or commit of the suite during the run is otherwise invisible, because the identity policy maps exact benchmark revisions to declared versions (§4.1) and `benchmark_dirty` only compares the file with its own HEAD. Benchmarks inside the subject's checkout differ between the sides exactly as the subject trees do and are not checked. A result without a `benchmark_tree`, such as a thin result, is outside the rule.
+A benchmark is **recorded separately** from the subject when its suite lives outside the subject's checkout (`runner-schema.md` §3.3). Its `benchmark_tree` is then the suite file's own id and differs from the `subject_tree`; this includes a suite in another checkout of the same repository, which has the same source URI but its own file id. A benchmark inside the subject's checkout has `benchmark_tree` equal to `subject_tree` and the subject's own source and revision as `benchmark.source` and `benchmark.revision`; a result whose two trees are equal but whose `benchmark.source` differs from its `source` is malformed. Under `revisions` and `variants` a separately recorded benchmark must be one file on both sides and in every round (invariant `one-benchmark-tree`): the suite is part of the code identity, which UC-03 requires the sides to agree on and UC-02 holds controlled (UC-02 §5), and an edit or commit of the suite during the run is otherwise invisible, because the identity policy maps exact benchmark revisions to declared versions (§4.1) and `benchmark_dirty` only compares the file with its own HEAD. Benchmarks inside the subject's checkout differ between the sides exactly as the subject trees do under `revisions` and are not checked; under `variants` an edit of one changes the subject tree, which the sides must share (`sides-agree`). A result without a `benchmark_tree`, such as a thin result, is outside the rule.
 
 Thin results (§4.1) may claim `single-run`, `variants`, and `revisions` within their own run; the other profiles require a full environment identity.
 

docs/design/comparator.md

@@ -111,6 +111,15 @@ Before computing a verdict, the comparator checks, in both modes, that:
 - the evidence is enough for the configured method, or the method declares a
   provisional rule that applies.
 
+In run mode the status rule belongs to the profile. Under `variants` it is
+strict (UC-02 §10): a unit with an attempt of status `partial`, `censored` or
+`error`, on either side, is indeterminate with reason `side-failed`, whatever
+the other rounds show. Otherwise a variant that times out in the slow rounds is
+judged on the fast ones. The other profiles drop the failed attempt's pair and
+keep the verdict from the rest, and a `skipped` attempt is dropped in every
+profile. The schema's §5.5 table is not touched, since this is a status rule,
+not an invariant.
+
 In history mode it additionally checks that:
 
 - the target and every reference point have the same series fingerprint;
@@ -124,9 +133,9 @@ In history mode it additionally checks that:
 In run mode it additionally checks the profile's invariants over the run (schema
 §5.5): the two sides differ only in the varying coordinate, attempt counts per
 side are equal, and sides alternate by `procedure.slot`; under `revisions`,
-that each side has exactly one tree id and the two differ, and that a
-separately recorded benchmark has one `benchmark_tree` across both sides and
-all rounds (`one-benchmark-tree`, schema §5.5). A side that is not clean does
+that each side has exactly one tree id and the two differ; under `revisions`
+and `variants`, that a separately recorded benchmark has one `benchmark_tree`
+across both sides and all rounds (`one-benchmark-tree`, schema §5.5). A side that is not clean does
 not fail the run; it makes the comparison local-only (UC-03 §10). A run that
 fails an invariant is not compared: the comparison document records the failed invariant
 and contains no verdicts for the units it affects.