Repository navigation
UC-03 : allow dirty trees for local-only runs - #83
Merged
Merged
Conversation
A `revisions` comparison may now have a dirty side. It is labeled local-only, names the sides that are not clean, and is never promoted into tracked history, as the `single-run`, `variants` and `environments` profiles already do. - UC-03: §10 allows dirty trees and lists what still fails (a side with an unknown dirty state, a tree that changes during the run, a separately recorded suite that is not one file throughout); §4 and §8 follow; §12 question 3 is resolved. - Result schema: §5.5 drops the `clean-tree` invariant from `revisions`, adds `one-benchmark-tree`, and defines a benchmark recorded separately from the subject; §4.1 says the runner adds the dirty warnings. - Comparator §3 and system decomposition §3.4 and §6 follow. This resolves the inconsistency between schema §4.1, which allows comparing an uncommitted change against HEAD, and §5.5, which did not. Allowing dirty trees by default is subject to review. The prototype docs (prototype-scope, prototype-design, prototype/README) still describe the old refusal and change together with the code. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Collaborator
Author
|
WDYT @MaanasArora ? |
Collaborator
Author
|
The implementation is in this branch https://github.com/ev-br/benchx/tree/dirty-trees-code. It's stacked on top of gh-81, so will need sorting out the "stack" and then a rebase. |
MaanasArora
reviewed
Oct 29, 2026
MaanasArora
left a comment
MaanasArora
left a comment
Collaborator
There was a problem hiding this comment.
Thanks, I read through the design changes and skimmed the plan! I think it makes a lot of sense to allow dirty trees for revisions - I've been bit by the same thing with asv indeed. This PR does minimal plumbing which makes it even more appealing!
I didn't find any issues so far, so no objections to merging and iterating.
ev-br
force-pushed
the
dirty-trees-docs
branch
from
October 29, 2026 12:10
2fd01fb to
df46a82
Compare
Collaborator
Author
|
Thanks for the review @MaanasArora ! I merged the design doc change as approved, and here's a follow-up with an actual implementation: #89 |
ev-br
pushed a commit
to ev-br/benchx
that referenced
this pull request
Oct 29, 2026
This is a follow-up PR: - benchx-dev#83 adds the design docs - this PR is branched off benchx-dev#81, and needs a rebase once that one lands
ev-br
pushed a commit
to ev-br/benchx
that referenced
this pull request
Nov 1, 2026
This is a follow-up PR: - benchx-dev#83 adds the design docs - this PR is branched off benchx-dev#81, and needs a rebase once that one lands
ev-br
pushed a commit
to ev-br/benchx
that referenced
this pull request
Nov 1, 2026
This is a follow-up PR: - benchx-dev#83 adds the design docs - this PR is branched off benchx-dev#81, and needs a rebase once that one lands
ev-br
pushed a commit
to ev-br/benchx
that referenced
this pull request
Nov 3, 2026
This is a follow-up PR: - benchx-dev#83 adds the design docs - this PR is branched off benchx-dev#81, and needs a rebase once that one lands
ev-br
pushed a commit
to ev-br/benchx
that referenced
this pull request
Nov 5, 2026
This is a follow-up PR: - benchx-dev#83 adds the design docs - this PR is branched off benchx-dev#81, and needs a rebase once that one lands
ev-br
pushed a commit
to ev-br/benchx
that referenced
this pull request
Nov 5, 2026
This is a follow-up PR: - benchx-dev#83 adds the design docs - this PR is branched off benchx-dev#81, and needs a rebase once that one lands
ev-br
pushed a commit
to ev-br/benchx
that referenced
this pull request
Nov 5, 2026
This is a follow-up PR: - benchx-dev#83 adds the design docs - this PR is branched off benchx-dev#81, and needs a rebase once that one lands
ev-br
pushed a commit
to ev-br/benchx
that referenced
this pull request
Nov 5, 2026
This is a follow-up PR: - benchx-dev#83 adds the design docs - this PR is branched off benchx-dev#81, and needs a rebase once that one lands
ev-br
added a commit
that referenced
this pull request
Nov 6, 2026
* ENH: allow dirty trees in local comparisons This is a follow-up PR: - #83 adds the design docs - this PR is branched off #81, and needs a rebase once that one lands * TST: ignore dirty-tree warnings in the setup_scope tests The two asv tests that came in with setup_scope assert the exact quality warnings of a result. With dirty trees allowed in local comparisons, a result also carries the dirty-* warnings the test earns by running in a tree with uncommitted files, so compare through warnings_of(), as the collector and pyperf tests do. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
asv has an unpleasant UI cut of ignoring uncommitted changes. UC-03 has an open question on whether benchx comparisons should allow dirty trees or refuse. For one-off/local-only uses, allowing them feels useful: if I work on a PR and want to check the current variant against the baseline, do I strictly need to commit? The UC-03 workflow sketch is
$spin bench --compare main $BENCHMARKSo this PR works towards allowing this: detect a dirty tree, label it as such, mark the comparison as local-only.
This is a design doc only change, plus a plan for implementation. If we agree with the doc edits, I'll remove the plan and submit a follow-up PR with an implementation.