Repository navigation
ENH: implement UC-02 comparisons over variants - #96
Merged
Merged
Conversation
ev-br
force-pushed
the
uc-02-variants
branch
4 times, most recently
from
November 5, 2026 16:15
fa04be8 to
c2054ef
Compare
The rest of this commit (allow dirty trees in local comparisons) is already in main as benchx-dev#89; only these two documentation hunks were missing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A script that gives each `bx bench` call its own directory inside the tree (`--out out/NAME`) changed the tree between calls: call 2 saw call 1's directory as an untracked file and recorded another subject_tree, so the comparison failed `sides-agree` (variants, environments) or `one-tree-per-side` (revisions). git_identity now excludes `**/out/**`, only when the repository tracks nothing under an `out/` (a blanket exclusion would give a clean checkout a tree id that differs from HEAD's). directory_identity skips `out`. UC-02 plan section 5.0, step 1 of its commit order. Applies to every profile and stands on its own. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Move two pieces out of the code that used them, so that the next change can reuse them instead of copying them: - compare: `_run_documents`, the measurands of one run in a fixed order (and the refusal of a run with none), which `compare()` built inline; - projection: `workload_label`, the `name [k=v, ...]` text `bx bench` prints for a case, which `bench.row_of` built inline. No change in behavior for any valid result: the same results are selected in the same order, and the same labels are printed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A comparison whose sides are the two values of one workload parameter, in one revision and one environment. The parameter is the profile's axis, added per comparison (`parameter=`, a token, so the document's `varying` stays one); everything else must agree, the tree and the suite included. - A unit is named by what both sides share: `_workload` drops the varied parameter, and `render` prints the rest with `workload_label` (`bench_truncate [size=1000]`). - A failed attempt (partial, censored, error) on either side makes its unit indeterminate with reason `side-failed`, so a variant that times out in the slow rounds is not judged on the fast ones (UC-02 §10). `skipped` stays an ordinary exclusion, and the other profiles keep their rule. - `alternating-slots` names the ways to get an alternating run. - `infer_parameter` gives read mode the run's only parameter. UC-02 plan section 5.1 and 5.5 (read mode), step 2 of its commit order. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…LUE]
Read mode for the variants profile: compare a stored run whose results
alternate two values of one workload parameter, such as one `bx bench`
call per value with --run-key, --round and --slot.
- `--vary NAME` names the parameter. Without it the run's only parameter
is used and said so on stderr; several are refused, naming them.
Refused with the other profiles, and `--label` is refused with variants.
- `--baseline` is the recorded value, quotes included ('baseline').
- The profile list comes from the comparator: `session.PROFILES`, the
`--profile` choices, and the config schema's `compare.profile` enum
(a test keeps the three equal).
- Measure mode for variants is refused until its own step.
test_uc02.py runs the loop through the CLI on a pyperf module and an asv
benchmark directory: one parameter, a second parameter with --vary, a
single bench call (alternating-slots), a directory per call inside an
untracked-.gitignore tree, and an uncommitted edit.
UC-02 plan sections 5.3 (read mode), 5.4 and 5.5, step 3 of its commit
order.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ilter text Move three pieces out of the code that used them, so that the next change can reuse them for a session of one target instead of copying them: - session: `_refuse_unknown_tree` and `_note_local_only`, which `compare_targets` built inline (the refusal of a side with no working-tree id, UC-03 §10, and the "will be local-only" notes); - benchx_params: `_search_text`, the `name(p0, p1)` text `expand` searches a case filter in. No change in behavior: same messages, same conditions, same order. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
UC-02 in one command: alternate the two values of one workload parameter
of the benchmark over --rounds rounds, deliver the results, and compare.
bx compare . --profile variants --suite bench_suite.py --rounds 5
`session.compare_variants` lists the cases once, with the user's --filter,
and finds the parameter (--vary, else the only one the selected
benchmarks and the `-e` variables give; several are refused, naming who
declares each). Each round runs one order per value: that value's cases
only, picked by `benchx_params.search_name` through the filter the runner
already understands, baseline at slot 2r and the other at 2r + 1.
Benchmarks without the parameter run on neither side and are reported.
Refused before anything runs: no parameter; a parameter the selected
benchmarks do not all take the same two values of (so `--filter` can pick
two of three); a --baseline that is not one of them, which lists the
values as they must be typed; a tree with no working-tree id; Google
Benchmark, which has no parameters. A dirty tree is noted before
measuring and the comparison is local-only, as for revisions.
`--baseline` is allowed in measure mode for variants only, and `bx compare`
takes one target for variants.
UC-02 plan section 5.2, the rest of 5.3 and 5.5, step 4 of its commit
order.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… READMEs - prototype-scope: variants is in the prototype, for one workload parameter per comparison (several at once stay deferred); the one-target form of `bx compare`. - prototype-design: the commands, the loop of `bx bench` calls with --filter or a case name, `bx compare TARGET --profile variants` (measure mode, its refusals before running), the profile row, which parameter is varied, the unit labels, and the side-failed rule. - benchmark-result-schema 5.5: the variants row loses "role by default" and gains one-benchmark-tree; a comparison has two sides, so the chosen parameters take exactly two joint assignments; the one-benchmark-tree paragraph covers revisions and variants. - comparator 3: one-benchmark-tree under revisions and variants; the strict status rule of variants (side-failed), not an invariant. - prototype/README: the benchmark, the command and the loop. The examples/matmul-params README changes (install stanzas, "Comparing numpy and torch") are not in this commit: the example moved to benchx_examples. UC-02 plan section 7, step 5 of its commit order. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Units were listed in the order of their canonical JSON, a string order: the sizes of a parametrized benchmark came out as n=10, n=100, n=50. Sort by workload name, then parameter name and value, then quantity, through `projection.axis_order`, so numbers inside names and values sort as people read them (n=10, n=50, n=100; bench_2 before bench_10). This changes the order of `units` in every comparison document and in the rendered table, for every profile. Which units there are, and what they say, is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MaanasArora
approved these changes
Nov 7, 2026
MaanasArora
left a comment
MaanasArora
left a comment
Collaborator
There was a problem hiding this comment.
Thanks @ev-br, looks great to me. Most of this is downstream from the design, and I have a few nits but I'll try to make issues for them. In it goes!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UC-02 compares two functions of one benchmark in one revision and environment, with the rounds interleaved.
This PR implements the
variantsprofile which enables the UC-02 comparisons.The PR is stacked on top of #89 (for dirty trees and comparators)and #94 (for parametrization), thus the patch size is actually much smaller than it looks. But will need a rebase once the two stacks land.The following commit, 46c4c0e, has all updates to the docs, design clarifications.
There's a worked example in the
matmul-paramREADME, benchx-dev/benchx_examples#12.The example compares
numpyandtorch, so the baseline vs contender language of UC-02 may or may not be best. Am open to tweaking of the way it reports the results!