UC-03 : allow dirty trees for local-only runs

docs/design/benchmark-result-schema.md

@@ -108,7 +108,7 @@ In metrology, the quantity intended for a workload variant at a revision is the
 
 `source` and `revision` serve both subject and benchmark code. A revision key has no intrinsic order, and the revision graph is not a core table: history queries take ancestry from the source repository or from an auxiliary revision catalog keyed by `(source_id, revision_key)`. When neither is available, ordering falls back to the `native_order` or `native_time` a producer reported with the revision, as migrated LNT or rustc-perf results do; that yields a linear order but not ancestry, so ancestor-based baselines still need a graph. Without any of these, results stay queryable but revision order is unavailable. Parent hashes are not copied into results.
 
-A `revision` row is keyed by the clean VCS identifier. Whether the measured checkout had uncommitted changes is a fact about the result: `subject_dirty` and `benchmark_dirty` are required tri-state flags, `clean`, `dirty`, or `unknown`, in provenance. A clean and a dirty result at the same commit therefore share a revision row. `unknown` is distinct from `clean` so that a comparator can tell what is known to be clean. The flag, the working-tree id below, and a quality warning exist so that a detector or comparator can filter or annotate dirty results trivially; what it does with them is its own policy. The failure mode to avoid is the one asv has, where uncommitted changes are silently ignored and a result is charted as if it were the commit. Comparators may compare dirty sides within one run, as a contributor does with an uncommitted change against HEAD; such a comparison is *local-only* and never promotes into tracked history (§5.5).
+A `revision` row is keyed by the clean VCS identifier. Whether the measured checkout had uncommitted changes is a fact about the result: `subject_dirty` and `benchmark_dirty` are required tri-state flags, `clean`, `dirty`, or `unknown`, in provenance. A clean and a dirty result at the same commit therefore share a revision row. `unknown` is distinct from `clean` so that a comparator can tell what is known to be clean. The flag, the working-tree id below, and a quality warning exist so that a detector or comparator can filter or annotate dirty results trivially; what it does with them is its own policy. The runner adds the quality warning `dirty-subject-tree` or `dirty-benchmark-tree` to a result whose matching flag is `dirty`. The failure mode to avoid is the one asv has, where uncommitted changes are silently ignored and a result is charted as if it were the commit. Comparators may compare dirty sides within one run, as a contributor does with an uncommitted change against HEAD; such a comparison is *local-only* and never promotes into tracked history (§5.5).
 
 **Working-tree identity.** A result also records `subject_tree` and `benchmark_tree`: the content hash of the checkout that was measured, independent of its commit. For git this is the tree object id of the working tree, the same id a commit made from this checkout on this machine would carry. It is obtained by staging everything into a scratch index that does not yet exist, so the repository's own index is untouched, and writing a tree:
 
@@ -498,13 +498,17 @@ Results carry facts; a **profile** is a comparator-time check that a set of resu
 |---|---|---|---|
 | `single-run` ([UC-01](../use-cases/UC-01-short-slug.md)) | workload parameters | protocol | none beyond attempt integrity |
 | `variants` ([UC-02](../use-cases/UC-02-variants.md)) | one or more workload parameters chosen by the caller, `parameters.role` by default | subject tree id, subject configuration, environment, protocol, every other workload parameter | equal attempt counts per side; sides alternate by `procedure.slot` |
-| `revisions` ([UC-03](../use-cases/UC-03-compare.md)) | subject tree id | workload variant, subject configuration, environment, protocol | equal attempt counts per side; sides alternate by `procedure.slot`; each side has exactly one tree id and the two differ; every result of both sides is `clean` (`subject_dirty` and `benchmark_dirty`; `dirty` and `unknown` both fail), invariant `clean-tree` |
+| `revisions` ([UC-03](../use-cases/UC-03-compare.md)) | subject tree id | workload variant, subject configuration, environment, protocol | equal attempt counts per side; sides alternate by `procedure.slot`; each side has exactly one tree id, so a side whose dirty state is `unknown` and has none fails, and the two differ; a separately recorded benchmark has one `benchmark_tree` across both sides and all rounds, invariant `one-benchmark-tree` |
 | `environments` ([UC-04](../use-cases/UC-04-cross-env.md)) | one pinned component, subject configuration, or `environment.system` field, named by `labels` | subject revision, workload variant, host environment, protocol | exactly two label values whose reported coordinates differ; equal attempt counts per label; sides alternate by `procedure.slot` |
 | `cross-machine` (template example, pending a use case) | environment identity | subject tree id, workload variant, subject configuration | each side references an anchor result from the same run via `provenance.references` |
 
 **Sides are chosen at comparison time.** The coordinate a profile varies is an argument to the comparator, not a declaration by the producer. Workload parameters are always recorded, so any parameter or set of parameters can be the varying one: `stable` for a sort-algorithm study, `fast_path` for a fast-path deficit, `{role, size}` for a two-factor design. Each distinct assignment of the chosen parameters is one side, and every other coordinate must agree. `role` is only the conventional name that UC-02's `--baseline`/`--variant` flags write; nothing in the schema privileges it. A comparison that was not planned when the run was made is therefore the same operation as one that was, provided the run contains the sides; what a run did not measure, the comparator cannot invent, and a planned run is what guarantees the interleaving invariant.
 
-A comparison in which any side is not `clean` is **local-only** under the profiles that allow it (`single-run`, `variants`, `environments`): the comparator produces it, labels it so, and never promotes it into tracked history. The tree id is what makes each side internally consistent even though neither may have a commit. The `revisions` profile does not allow it: UC-03 §10 rejects dirty trees, because the comparison is invalid when neither revision can be validated, so a dirty or unknown side fails the `clean-tree` invariant and its units get no verdict. Whether `revisions` should instead allow a local-only comparison of an uncommitted change against HEAD, as the one-off and Arrow local stories want, is open (UC-03 §12 question 3). Thin results (§4.1) may claim `single-run`, `variants`, and `revisions` within their own run; the other profiles require a full environment identity.
+A comparison in which any side is not `clean` is **local-only** under the profiles that allow it (`single-run`, `variants`, `revisions`, `environments`): the comparator produces it, labels it so, names the sides that are not clean, and never promotes it into tracked history. The tree id is what makes each side internally consistent even though neither may have a commit. Under `revisions` this is the comparison of an uncommitted change against HEAD, or against another revision, that the one-off and Arrow local stories want (UC-03 §10). What `revisions` still requires is that every side can be identified: a side whose dirty state is `unknown` has no tree id (§4.1), so it cannot be told apart from the other side or checked for a change during the run, and it fails the invariant that each side has exactly one tree id, so its units get no verdict.
+
+A benchmark is **recorded separately** from the subject when its suite lives outside the subject's checkout (`runner-schema.md` §3.3). Its `benchmark_tree` is then the suite file's own id and differs from the `subject_tree`; this includes a suite in another checkout of the same repository, which has the same source URI but its own file id. A benchmark inside the subject's checkout has `benchmark_tree` equal to `subject_tree` and the subject's own source and revision as `benchmark.source` and `benchmark.revision`; a result whose two trees are equal but whose `benchmark.source` differs from its `source` is malformed. Under `revisions` a separately recorded benchmark must be one file on both sides and in every round (invariant `one-benchmark-tree`): UC-03 requires the sides to agree on everything under code identity, and an edit or commit of the suite during the run is otherwise invisible, because the identity policy maps exact benchmark revisions to declared versions (§4.1) and `benchmark_dirty` only compares the file with its own HEAD. Benchmarks inside the subject's checkout differ between the sides exactly as the subject trees do and are not checked. A result without a `benchmark_tree`, such as a thin result, is outside the rule.
+
+Thin results (§4.1) may claim `single-run`, `variants`, and `revisions` within their own run; the other profiles require a full environment identity.
 
 A revision may hold several attempts with identical coordinates, since every attempt is its own point, so viewers and comparators must expect more than one result per coordinates and revision. A comparison document therefore lists the `(producer, ingest_key)` of every result it consumed. That is also what lets a saved baseline be reused by reference instead of re-measured (Arrow local story).
 

docs/design/comparator.md

@@ -124,8 +124,11 @@ In history mode it additionally checks that:
 In run mode it additionally checks the profile's invariants over the run (schema
 §5.5): the two sides differ only in the varying coordinate, attempt counts per
 side are equal, and sides alternate by `procedure.slot`; under `revisions`,
-that every result of both sides has a `clean` subject and benchmark tree
-(`clean-tree`, UC-03 §10). A run that fails an invariant is not compared: the comparison document records the failed invariant
+that each side has exactly one tree id and the two differ, and that a
+separately recorded benchmark has one `benchmark_tree` across both sides and
+all rounds (`one-benchmark-tree`, schema §5.5). A side that is not clean does
+not fail the run; it makes the comparison local-only (UC-03 §10). A run that
+fails an invariant is not compared: the comparison document records the failed invariant
 and contains no verdicts for the units it affects.
 
 Excluded points and their reasons remain visible in the output. Changes in

docs/design/system-decomposition.md

@@ -121,10 +121,11 @@ harness did not report.
   comparison target, so expensive baselines are measured once. A replayed
   baseline suits unpaired comparisons; a paired, interleaved comparison
   (UC-03) needs both sides measured fresh in one session.
-- Allows a single-revision run on a dirty tree, recorded as dirty (UC-01); a
-  two-revision comparison refuses a dirty tree (UC-03 §10), before running, by a
-  `tree.clean` verify rule where an environment policy applies, and again in
-  the comparator by the `revisions` profile's `clean-tree` invariant.
+- Allows a single-revision run on a dirty tree, recorded as dirty (UC-01),
+  and a two-revision comparison on dirty trees, labeled local-only (UC-03
+  §10); it says so before measuring. It refuses a side whose working-tree id
+  is unknown, before running, and the comparator fails the same side through the
+  `revisions` profile's invariant that each side has exactly one tree id.
 - Refuses, or loudly warns about, invalid comparisons: different machine,
   different build configuration, unknown provenance.
 - Supports the iteration loop: scoping by filter, quick low-precision runs
@@ -353,10 +354,10 @@ Not components of this system:
 3. Is the workbench a distinct deliverable or a thin skin over runner +
    comparator? Treated here as a skin with its own UX obligations, plus the
    session loop and target-provider calls of §3.4.
-4. Dirty trees in comparisons. UC-03 rejects them; a single-revision run
-   accepts them. Should `--compare` also allow a dirty tree as a local-only
+4. *Resolved:* dirty trees in comparisons. UC-03 allows them as a local-only
    comparison, keyed on working-tree ids (`benchmark-result-schema.md` §5.5),
-   so a contributor can compare uncommitted work without committing first?
+   so a contributor can compare uncommitted work without committing first; a
+   tree whose id is unknown is still refused.
 5. *Partly resolved:* the target-provider contract is in §3.4: its input is
    a request, its output a target description, a build failure is a nonzero
    exit reported by the caller, and `describe` reports without building. Still

docs/use-cases/UC-03-compare.md

@@ -80,8 +80,9 @@ work order per side per round, alternating the sides under one `run_key`, runs
 the comparator with the `revisions` profile, and returns the comparison
 document for `spin` to render (`system-decomposition.md` §3.4). Environment
 variables such as thread caps go in the work order's environment policy, not
-in `spin`'s shell. Plain `spin bench` (UC-01) is the same path with one target
-and no comparison.
+in `spin`'s shell. If the current checkout has uncommitted changes, the
+comparison still runs and is labeled local-only (§10). Plain `spin bench`
+(UC-01) is the same path with one target and no comparison.
 
 ---
 
@@ -138,8 +139,9 @@ possible ensures reliability.
   - **interleave position**: must be stored for each measurement.
     However, see open question 1.
   - **ratio, ci_low, ci_high, classification**
-- **Comparability key:** records must *differ* in the code version (the revision)
-  and *agree* on everything under code identity.
+- **Comparability key:** records must *differ* in the code version (the
+  revision, or the working-tree id when the code is uncommitted) and *agree*
+  on everything under code identity.
 - **Validation invariants:** all records in a comparison share all of
   `project_id`, `name`, `parameters`, `environment`.
   The `environment` is compared by its hash.
@@ -155,12 +157,22 @@ possible ensures reliability.
 
 - Missing data: if the process is stopped before all interleaved runs are complete,
   the series cannot claim the profile.
-- Reject dirty trees - the comparison is invalid if there is no way to validate either
-  revision. The schema should make `dirty` non-nullable in this profile: the
-  `revisions` profile's `clean-tree` invariant fails a dirty or unknown side.
-  The comparison's environment policy carries a `tree.clean` verify rule with
-  `refuse`, so the run is refused before any measurement. This applies to comparisons only: a single-revision run (UC-01)
-  may be dirty and is recorded as dirty.
+- Dirty trees are allowed in a comparison, which then becomes local-only. A
+  side may be a checkout with uncommitted changes, such as a contributor's
+  current branch against `main`. The comparison is labeled local-only, names
+  the sides that are not clean, says so before measuring, and is never
+  promoted into tracked history (`benchmark-result-schema.md` §5.5). A
+  single-revision run (UC-01) may be dirty too, and is recorded as dirty. What
+  still fails the comparison, with no verdict for the units it affects:
+  - a side whose dirty state is `unknown`. It has no working-tree id, so the
+    sides cannot be told apart and a change during the run cannot be seen;
+    the `revisions` profile requires exactly one tree id per side;
+  - a tree that changes during the run (a side with more than one tree id);
+  - a benchmark suite recorded apart from the subject (outside its checkout)
+    that is not the same file on both sides and in every round. A suite
+    inside the subject's checkout differs between the sides exactly as the
+    subject does, and is not checked. This is the run-level form of §8: the
+    sides agree on everything under code identity.
 - Build failure of either revision happens in the target provider, before any
   work order exists, and is reported to the caller; there is no result to
   compare.
@@ -175,7 +187,7 @@ possible ensures reliability.
 1. For storing interleaving positions / other data, do we create pair
    identities under the protocol?
 2. For the future CI path, do we need a shared store for validation?
-3. Should `--compare` allow a dirty tree as a local-only comparison keyed on
-   working-tree ids, so a contributor can compare uncommitted work on the
-   current branch without committing? This use case rejects it (§10); the
-   schema would permit it (`benchmark-result-schema.md` §5.5).
+3. *Resolved:* `--compare` allows a dirty tree as a local-only comparison
+   keyed on working-tree ids, so a contributor can compare uncommitted work on
+   the current branch without committing (§10;
+   `benchmark-result-schema.md` §5.5).