feat(evals): add blinded paired experiment

.agents/plans/02-eval-engineering/evidence/phase-7-architecture.md

@@ -0,0 +1,168 @@
+# Phase 7 architecture synthesis
+
+## Grounding
+
+The legacy benchmark shuffles independent Flow and ordinary jobs, keeps results in
+memory, and writes marginal summaries. The v2 boundary already owns paired cells,
+whole-pair reserve validation, opaque product evidence, exact provenance, atomic
+attempt storage, and descriptive opaque win counts. Phase 7 adds an experiment
+layer and adapts the runner; it does not create another report or persistence
+system.
+
+## Arena
+
+- Terra supplied the strongest runner adaptation and rejection rules.
+- Luna supplied the selected pure `experiment.ts` base and macro task-stratified
+  estimand.
+- One candidate exceeded the time box and is recorded as a dropout.
+- The independent judge scored the inspectable base 16/25 and required a salted
+  commitment, exact report binding, coherent token semantics, and corrected
+  finalization order.
+
+## Usage
+
+The runner constructs and persists one frozen plan before starting either arm.
+Each primary `(case, repetition)` block has two opaque tokens. A private allocation
+maps them to candidate and ordinary baseline. Preallocated reserve blocks repeat
+the same `(case, repetition)` and token mapping.
+
+```ts
+const experiment = createPairedPlan({
+  cases,
+  model,
+  repetitions,
+  reservePairsPerBlock,
+  randomizationSeed,
+  allocationSeed,
+  commitmentNonce,
+  budget,
+});
+
+const report = await store.finalize({
+  reportId,
+  completion,
+  allocationCommitmentSha256: experiment.allocationCommitmentSha256,
+});
+const masked = freezeMaskedAnalysis({ report, scans, frozenAt });
+await store.writeMaskedAnalysis(masked);
+const revealed = revealPairedAnalysis({ report, masked, secret, revealedAt });
+await store.writeAllocation(revealed.allocation);
+```
+
+The order is mandatory. A crash before the masked write publishes no allocation.
+A crash after the masked write leaves a valid opaque analysis and no directional
+claim. Only a reveal that binds the exact plan, report, masked record, commitment,
+nonce, and token map can produce candidate-minus-baseline results.
+
+## Selected shape
+
+`evals/experiment.ts` owns the pure contracts.
+
+```ts
+type Arm = "candidate" | "baseline";
+
+type BlockAllocation = {
+  readonly blockId: string;
+  readonly caseId: string;
+  readonly caseVersion: number;
+  readonly repetition: number;
+  readonly tokens: readonly [string, string];
+  readonly tokenToArm: Readonly<Record<string, Arm>>;
+};
+
+type AllocationSecret = {
+  readonly schemaVersion: 1;
+  readonly planSha256: string;
+  readonly nonce: string;
+  readonly blocks: readonly BlockAllocation[];
+};
+
+type MaskedAnalysisRecord = {
+  readonly schemaVersion: 1;
+  readonly reportId: string;
+  readonly planSha256: string;
+  readonly reportSha256: string;
+  readonly allocationCommitmentSha256: string;
+  readonly analysisPolicySha256: string;
+  readonly observations: readonly MaskedPairObservation[];
+  readonly completePairs: number;
+  readonly unresolvedPairs: number;
+  readonly ties: number;
+  readonly opaqueEstimate: number | null;
+  readonly interval95: readonly [number, number] | null;
+  readonly scannerSha256: string;
+  readonly scannerPassed: boolean;
+  readonly treatmentBlinding: "flow-tool-presence-visible";
+  readonly frozenAt: string;
+  readonly sha256: string;
+};
+
+type AllocationRecord = {
+  readonly schemaVersion: 1;
+  readonly reportId: string;
+  readonly planSha256: string;
+  readonly reportSha256: string;
+  readonly maskedAnalysisSha256: string;
+  readonly allocationCommitmentSha256: string;
+  readonly nonce: string;
+  readonly blocks: readonly BlockAllocation[];
+  readonly revealedAt: string;
+};
+```
+
+Pure entry points are `createPairedPlan`, `scanPairedTranscript`,
+`freezeMaskedAnalysis`, `revealPairedAnalysis`, and
+`taskStratifiedPairedBootstrap`. Strict Zod schemas validate masked and allocation
+records at the boundary. Domain-separated canonical hashes omit only a record's
+own `sha256` field.
+
+Tokens are unique per primary block and allocation is independently seeded within
+each block. A reserve for that block reuses its case, repetition, tokens, and
+mapping. This avoids the rejected contradiction between two global tokens and
+per-block random assignment. The masked record freezes canonical per-block
+outcomes without candidate or baseline labels. Directional bootstrap happens only
+after reveal.
+
+The estimand is the equal-weight macro average of each task's mean paired hidden-
+correctness difference. Bootstrap resamples complete pairs within each task,
+preserves stratum size, treats ties as zero, and uses the policy seed and a
+versioned fixed replicate count. Power metadata uses a documented conservative
+paired-difference bound. Insufficient planned power, unresolved pairs, scanner
+findings, incomplete completion, an invalid policy digest, or an interval wider
+than twice the minimum detectable effect prevents a directional claim.
+
+## Runner and storage
+
+`evals/benchmark-run.ts` becomes a thin v2 shell. It packs and inspects the
+candidate once, runs both cells in every started block, grades hidden correctness
+symmetrically, persists redacted transcripts and exact attempt provenance, and
+activates only a complete preallocated reserve block after host, provider, or
+evaluator failure. A scored product failure is never retried. Scanner findings are
+immutable stop-gate evidence, not a reason to rerun until a clean transcript
+appears.
+
+`evals/report-store.ts` adds immutable masked-analysis and allocation writes using
+its existing write-once primitive. It verifies that a masked record exists and its
+hash matches before accepting an allocation record. The underlying report remains
+opaque and exact; the allocation file is a later reveal artifact.
+
+## Rejections
+
+- No marginal `summarizeBenchmark()` result is used for a paired claim.
+- No unsalted two-choice commitment is accepted.
+- No candidate or baseline label enters the plan, transcript, or masked record.
+- No allocation is written before exact report finalization and durable masked
+  analysis.
+- No single-arm retry or silent removal of an incomplete block is allowed.
+- No scanner failure is repaired by replacement sampling.
+- No claim of full treatment blinding is made because Flow tool presence is
+  observable.
+
+## Verification contract
+
+Focused tests cover deterministic planning, per-block arm permutation, commitment
+tampering, report and masked hash drift, reveal ordering, known paired effects,
+bootstrap reproducibility, power metadata, ties, unresolved pairs, reserve
+replacement, budget stops, scanner labels and ground-truth paths, and the honest
+tool-presence limitation. The runner must emit one strict v2 low-budget pilot and
+freeze its masked record before allocation reveal.

.agents/plans/02-eval-engineering/evidence/phase-7-pilot.json

@@ -0,0 +1,42 @@
+{
+  "schemaVersion": 1,
+  "reportId": "paired-1787646912516",
+  "reportFileSha256": "sha256:e1f8ccbd1b471a50cb3be25b6f91131bee9cd240b7d2cfe2b504926da340f4b2",
+  "maskedFileSha256": "sha256:c29c4c5a28a96047e82f63e90e44d96922a8c484972678e9d995e0063c093896",
+  "allocationFileSha256": "sha256:b49ca9f54104717789103d58e972a8c59c4169db34135f1fc9ef1111415bfc39",
+  "planSha256": "sha256:50d5f8424556855306d8bf164d7ceace24d103a0a18ad922ace0441cc95c2e93",
+  "analysisVersionSha256": "sha256:675b036d0fac423ef2d5841ffbd4b7a45220976a2aa874263e404a095122ff9b",
+  "evaluatorGraderBundleSha256": "sha256:08c6738971f109872bde820ae36473ddb51486b578c70ebde9ef5fcd3b5fa8f3",
+  "reportSha256": "sha256:20e85b23348cf035479eda7b7f38d6b4c918d55aa7566c0aabc3bfa836c944a9",
+  "maskedAnalysisSha256": "sha256:b9816dcba780be71450509649248d764a919f1b844f9702ab69736fd5cfdfdf2",
+  "allocationCommitmentSha256": "sha256:933cfed41c233ad726ad2677878701c7c4107e7a6195116d94c483ba5ada1147",
+  "scannerVersionSha256": "sha256:3b295814e87ea22fa0d83ddbdc837ec08849f49108d390deb4f62b6f6aa62ddc",
+  "scannerSha256": "sha256:ac32a46cf7944347eda6900871be05262c3f87c6e1332c6eae9778f19003bd41",
+  "completionStatus": "complete",
+  "completionCause": "fixed-target",
+  "attempts": 2,
+  "completePairs": 1,
+  "unresolvedPairs": 0,
+  "candidateCorrect": true,
+  "baselineCorrect": true,
+  "candidateMinusBaseline": 0,
+  "interval95": [0, 0],
+  "ties": 1,
+  "scannerPassed": true,
+  "scanCount": 2,
+  "transcriptCount": 2,
+  "maskedBeforeAllocation": true,
+  "maxUsd": 1,
+  "costUsd": 0.24507220000000007,
+  "outputTokens": 4464,
+  "artifactTarballSha256": "sha256:c89f7363248ccc3e3f69728c1aa42044a25938cd533e470d9f73ef08bc64ad24",
+  "claim": "inconclusive",
+  "claimEligible": false,
+  "gateReasons": ["power-insufficient"],
+  "plannedPairs": 1,
+  "requiredPairs": 265,
+  "treatmentBlinding": "flow-tool-presence-visible",
+  "strictReportParsePassed": true,
+  "semanticMaskedValidationPassed": true,
+  "scanTranscriptMultisetMatched": true
+}

.agents/plans/02-eval-engineering/evidence/phase-7-review.md

@@ -0,0 +1,37 @@
+# Phase 7 Interrogate review and pilot
+
+Phase 7 replaces the marginal benchmark summary with one frozen v2 paired
+experiment. Per-block arm allocation is independently seeded and hidden behind a
+nonce-backed commitment. Product outcomes are immutable, only non-product failures
+activate a preallocated whole reserve pair, and task-stratified bootstrap gives
+each task equal weight.
+
+Architect and Arena selected a pure experiment boundary over extending the legacy
+summary. The independent judge required four corrections before implementation:
+salted commitments, exact report and masked hash binding, per-block token
+semantics, and the order finalized report, durable masked analysis, allocation
+reveal, directional estimate.
+
+Interrogate found and fixed scanner-to-transcript binding, fabricated masked
+observations, final-link durability, startup cleanup, plain reserved-label
+detection, rejected request delivery, failure-origin attribution, and a permissive
+pure reveal seam. The masked record now carries every
+versioned scan and exact transcript SHA-256. `ReportStore` recomputes masked
+semantics from the finalized report and fsyncs the final directory entry before
+removing its recovery link. The evaluator digest binds benchmark cases, hidden
+graders, runner, experiment, power contract, report schema, and pairing logic.
+
+The accepted packed-host pilot ran one complete pair under a $1 ceiling. Both the
+candidate and ordinary baseline passed the hidden executable grader. The scan was
+clean, no reserve was activated, and masked analysis was durable before allocation
+reveal. Observed cost was $0.2450722 for 4,464 output tokens.
+
+The tie supports no product-value claim. The conservative power contract requires
+265 pairs for the preregistered 0.2 minimum detectable effect and 0.8 target power;
+the pilot planned one. `claimEligible` is therefore false and the directional
+decision is `inconclusive`. Flow tool presence remains visibly unblinded and is
+recorded as a limitation.
+
+Earlier Phase 7 campaigns are diagnostic only and excluded from evidence. The
+final full repository gate passes, and the final multi-model recheck has no
+unresolved blocker.

.agents/plans/02-eval-engineering/phase-7-paired-comparison.md

@@ -35,3 +35,12 @@ report states that the model cannot be blinded to Flow tool presence.
 
 Stop gate. No claim from incomplete pairs, an invalid analysis policy, scanner
 failures, or intervals wider than the preregistered decision bound.
+
+## Outcome
+
+Implemented and verified. The accepted one-pair packed-host pilot produced a
+strict v2 report, transcript-bound clean scans, a durable masked record before
+allocation reveal, and no unresolved pair. Both arms passed, so the descriptive
+effect is zero. The result remains `inconclusive` because one planned pair misses
+the 265-pair power requirement. See `evidence/phase-7-architecture.md`,
+`evidence/phase-7-review.md`, and `evidence/phase-7-pilot.json`.

.audit/eval-engineering.tsv

@@ -49,3 +49,8 @@ ts	phase	decision	why	evidence	result
 2026-08-25T07:12:18Z	phase-6	fixed truth, labels, durable scoring, archive recovery, source binding, and controls	reviewer calibration cannot trust prose, partial fixture hashes, mutable labels, or evaluator-created drift	evals/reviewer-cases.ts; evals/reviewer-assignment.ts; evals/reviewer-run.ts; evals/reviewer-calibration.ts	VERIFIED focused integration and strict parser gates green
 2026-08-25T07:12:18Z	phase-6	ran the final paid packed-host reviewer pilot	the phase needs real defect and clean evidence without manager selection or repair confounding	.agents/plans/02-eval-engineering/evidence/phase-6-pilot.json	VERIFIED 1 detection, 0 false positives, 2 submissions, advisory only, $0.1480602
 2026-08-25T07:12:18Z	phase-6	ran Deslop, four-model Interrogate, and the full repository gate	the phase must remain reviewable, regression-free, and make no unsupported promotion claim	bun run check; .agents/plans/02-eval-engineering/evidence/phase-6-review.md	VERIFIED 483 pass, 1 skip, 0 fail; no unresolved blocker
+2026-08-25T08:28:46Z	phase-7	selected the paired experiment contract through Architect and Arena	marginal summaries cannot prove paired effects and unsalted two-arm hashes do not blind allocation	.agents/plans/02-eval-engineering/evidence/phase-7-architecture.md	VERIFIED nonce-backed per-block allocation and finalize-mask-reveal order
+2026-08-25T08:28:46Z	phase-7	implemented atomic paired execution and controlled unblinding	whole-pair replacement, exact provenance, scanner binding, power, and bootstrap must be reusable pure checks	evals/experiment.ts; evals/experiment-power.ts; evals/benchmark-run.ts; evals/report-store.ts	VERIFIED focused paired, store, budget, scanner, and reveal tests green
+2026-08-25T08:28:46Z	phase-7	fixed multi-model Interrogate findings	count-only scans, self-hashed fabricated masks, early recovery-link removal, startup leaks, and permissive reveal weakened evidence integrity	.agents/plans/02-eval-engineering/evidence/phase-7-review.md	VERIFIED exact scan multiset, semantic mask recomputation, durable link order, and strict reveal
+2026-08-25T08:28:46Z	phase-7	ran the accepted bounded paired pilot	the runner needs real packed-host evidence while refusing an underpowered product claim	.agents/plans/02-eval-engineering/evidence/phase-7-pilot.json	VERIFIED 1 tie, 0 unresolved, exact clean scans, $0.2450722 of $1, INCONCLUSIVE on power
+2026-08-25T08:28:46Z	phase-7	ran Deslop and the full repository gate	the phase must finish reviewable and regression-free	bun run check	VERIFIED 496 pass, 1 skip, 0 fail