Repository navigation
Add selector/method framework to comparator to merge modes - #113
MaanasArora wants to merge 5 commits into
Conversation
Ports the variants profile onto the run-profile selector: the `parameter` argument and `infer_parameter`, units named without the varied parameter, natural unit order, workload labels in the table, and the variants advice on alternating-slots (now carried by the selection, since that check is the method's). UC-02's strict status rule (side-failed) becomes a selector fact on the evidence, `no_verdict`: every method returns indeterminate with that reason and the framework refuses one that does not, so the rule holds for paired-relative and mann-whitney alike. UC-02's tests are updated to comparison-document 0.2.0 field paths only. Against main's comparator on 30,010 fuzzed runs (10,000 of them variants), every difference is one of the three deliberate changes of the framework (duplicate-points, layout, no pairing of results without a round). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ev-br
left a comment
ev-br
left a comment
There was a problem hiding this comment.
This looks very reasonable and very cool on a quick (and fairly superficial!) read-through. I'm certainly very happy to roll with this version and will poke around a bit more. My poking around is absolutely non-blocking, thus if you prefer to land this and iterate in a follow-up or several, please do!
| One series throughout: NumPy `sort`, float64, 10⁶ elements, wall time. **The values | ||
| are constructed to show the mechanism, not measured.** Every number below was | ||
| computed from these values by test code that returns in a follow-up change. | ||
| All three use *k* = 3. |
There was a problem hiding this comment.
Are these example available somewhere public?
|
thanks! one comment (spotted with claude): A negative k crashes bx compare (methods/paired_relative.py:271). The config schema accepts any number for k, but the details schema requires threshold = k*noise >= 0. So --k -1 gets through configuration and then raises jsonschema.ValidationError in compare.py:106, which the CLI doesn't catch, and the user sees a traceback. Fix: add minimum: 0 to k. |
Replaces the history/run mode separation with a new comparator framework that specifies a selector (how to gather points to compare) and a method (how to statistically compute the comparison). This uses series for the comparison unit, so it reuses projections. The key idea is that the method being selectable allows specifying different algorithms down the line; this could be a bit more harness-specific later too.
The comparison document schema has changed to account for these parameters, so JSON outputs change. There are also a few small behavior changes, mostly in the round bookkeeping: duplicate results are refused (not overwritten), runs with inconsistent rounds don't get verdicts, and results without a round aren't paired.
I tried prototyping a history mode with Claude (not here) and it was additive. There's a prototype (in tests) for a different method using the Mann-Whitney U-Test.
Mostly drafted by Claude and still working to make sure everything is right, but I already iterated a lot, so would appreciate reviews if it looks right to others. cc @ev-br @rok, thanks!
(There is a slightly outdated plan before the last commit to aid review.)