Derive eval decisions from atomic evidence

.agents/plans/02-eval-engineering/architecture.md

@@ -299,6 +299,12 @@ declare function analyzePairs(report: ValidatedReport, masked: MaskedAnalysisRec
 };
 ```
 
+Phase 2 implements the pure release decision without a canary input. Phase 9 adds
+the canary gate at the release-alignment boundary. Phase 2 paired analysis returns
+only opaque complete, incomplete, tie, and arm-win counts. Phase 7 adds masked
+allocation, directional risk difference, and bootstrap intervals. Release sample
+and pass floors apply per represented scheduled provider within each required case.
+
 Parsing and provenance validation happen once at the JSON boundary. Internal
 analysis trusts `ValidatedReport`. CLI, filesystem, host, and provider concerns
 stay in thin shells.

.agents/plans/02-eval-engineering/evidence/phase-2-architecture.md

@@ -0,0 +1,63 @@
+# Phase 2 architecture synthesis
+
+## Grounding
+
+The legacy qualifier reads report summaries and stays unchanged until Phase 5.
+Phase 2 consumes only `ValidatedReport` and `ValidatedCaseCatalog`. Frozen plan
+cells define coverage. Atomic product outcomes define scored rows. Caller-supplied
+provenance remains independent of the report.
+
+## Arena
+
+- `gpt-5.6-terra` produced the selected small pure-analysis base.
+- `gpt-5.6-luna` made paired artifact comparison explicit but exposed a premature
+  directional paired estimate.
+- `gpt-5.5` provided the strongest requested-versus-actual model contract but
+  treated every failed product row as an unconditional release failure.
+- `gpt-5.4` did not complete a candidate and is recorded as a dropout.
+- An independent `gpt-5.4` judge selected Terra with 21 of 25 points. Luna and
+  `gpt-5.5` each scored 17 of 25.
+
+## Selected shape
+
+`evals/analysis.ts` owns four pure entry points.
+
+- `deriveReleaseDecision` derives a three-valued release decision from required
+  catalog policy, frozen cells, atomic attempts, and exact release provenance.
+- `compareExpectedProvenance` compares exact release artifacts or the unordered
+  allowed artifact set for paired evidence. Requested and actual models are
+  separate expectations. An unobserved actual model needs an explicit exception.
+- `analyzeReviewer` derives fixed-label detection and false-positive counts,
+  rates, unsubmitted counts, and Wilson 95 percent intervals.
+- `analyzePairs` derives eligible, complete, incomplete, tie, and opaque-arm win
+  counts. It cannot express candidate direction, risk difference, or an interval.
+
+Release policy is evaluated per required case and per represented scheduled
+provider. Every represented provider must reach the case sample and rate floor.
+The distinct-provider count must reach `minProviders`. Product failures contribute
+to the measured rate. False completion, unsubmitted review, exact-provenance drift,
+an unplanned required case, or a sufficiently sampled rate below its floor is
+`NOT VERIFIED`. Stops, missing attempts, unscored outcomes, and insufficient
+provider or sample evidence are `INCONCLUSIVE`. Hard failures take precedence.
+The scheduled `routeProvider` is the coverage denominator. Actual observed model
+identity is compared separately as provenance.
+
+## Grafts and rejections
+
+The selected base takes explicit release and paired provenance variants from
+`gpt-5.5` and unordered paired artifact membership from Luna. It rejects a
+report-wide artifact exception, pooled provider rates, individual product-failure
+vetoes, first-arm risk differences, and nullable placeholder inputs for later
+phases.
+
+Canary input remains Phase 9. Allocation commitments, masked analysis,
+candidate-minus-baseline direction, and bootstrap intervals remain Phase 7.
+Phase 3 computes expected provenance; Phase 2 only compares a supplied value.
+
+## Verification contract
+
+Synthetic fixtures must cross the Phase 1 parser before analysis. A table-driven
+monotonicity check removes or downgrades every required row and proves the result
+cannot remain `VERIFIED`. Release tests live in `tests/atomic-analysis.test.ts` and
+reviewer and paired tests live in `tests/advisory-analysis.test.ts`. The legacy
+qualifier tests remain unchanged and every file stays below 1,000 lines.

.agents/plans/02-eval-engineering/evidence/phase-2-review.md

@@ -0,0 +1,51 @@
+# Phase 2 Interrogate review
+
+## Intent
+
+Phase 2 must derive release, reviewer, and paired analysis from validated atomic
+rows without cutting over the legacy qualifier, CLI, or workflows. Required case
+policy and caller-supplied provenance must determine the release verdict.
+Reviewer and paired outputs remain advisory.
+
+## Reviewers
+
+- `gpt-5.6-terra` reviewed the first implementation and corrected diff.
+- `gpt-5.6-luna` reviewed the first implementation and corrected diff.
+- `gpt-5.5` reviewed the first implementation and corrected diff.
+- `gpt-5.4` independently judged the architecture and reviewed the corrected diff.
+
+## Acted on
+
+- Release provenance, stop gaps, false completion, and unsubmitted review checks
+  now consider required cells only. Report-only failures cannot change a release
+  verdict.
+- Provider attribution uses the frozen scheduled model and falls back to the
+  required atomic actor when the plan carries no model.
+- Expected attempt and actor role sets are unique and bidirectional. Extra,
+  missing, duplicated, or mismatched provenance fails closed.
+- Reviewer analysis exposes incomplete assignments. Reviewer and paired entry
+  points return zero metrics for incompatible campaign kinds.
+- Pair analysis counts primary and activated reserve blocks only. Incomplete
+  blocks cannot donate a winner.
+- The monotonicity test now deletes an attempt while preserving the original
+  frozen plan. The parser and analyzer together prevent a downgraded row from
+  remaining `VERIFIED`.
+- The v2 release and advisory suites are separate so every changed TypeScript
+  file remains below 1,000 lines.
+
+## Lead judgment
+
+- Scheduled `routeProvider` remains the provider-coverage denominator. Actual
+  identity is a separate exact provenance claim. A gateway may observe a different
+  downstream model without changing which scheduled route the experiment covered.
+- Public `compareExpectedProvenance` compares a complete report. The release
+  decision uses the same primitive over required cells only. This preserves a
+  reusable full-report audit without allowing report-only rows to gate release.
+- Product failures remain rate inputs. Only a sufficiently sampled rate below its
+  policy floor is `NOT VERIFIED`.
+
+## Verdict
+
+`VERIFIED`. All four rechecks reported no unresolved blocker after lead judgment.
+The focused gate passes 59 tests. The full repository gate passes 448 tests, one
+intentional live-host skip, and zero failures.

.agents/plans/02-eval-engineering/phase-2-atomic-qualification.md

@@ -11,7 +11,7 @@ attempts without cutting over any CLI or workflow.
 
 - `evals/analysis.ts`. Add pure release, reviewer, and paired derivation over
   `ValidatedReport`, including external expected-provenance comparison.
-- `tests/release-qualification.test.ts`. Cover missing cells, scored escalations,
+- `tests/atomic-analysis.test.ts`. Cover missing cells, scored escalations,
   immutable product failures, evaluator and infrastructure stops, completion
   causes, reviewer truth, pair ties, and mismatched expected provenance.
 
@@ -20,9 +20,24 @@ attempts without cutting over any CLI or workflow.
 `ReleaseDecision = Verdict + DecisionReason[] + derived counts`; every verdict
 can become an immutable decision record.
 
+## Settled implementation contract
+
+- Release floors apply to every represented scheduled provider within a required
+  case, plus the case-level distinct-provider floor.
+- Product failures contribute to rates. A sufficiently sampled rate below policy
+  is `NOT VERIFIED`; insufficient evidence is `INCONCLUSIVE`.
+- Expected provenance separates exact release evidence from paired candidate and
+  baseline artifacts, and separates requested from actual model identity.
+- Reviewer Wilson intervals are descriptive only. Promotion remains Phase 6.
+- Paired analysis remains opaque. Direction, allocation, bootstrap, and intervals
+  remain Phase 7. Canary gating remains Phase 9.
+
+See [the Architect and Arena synthesis](evidence/phase-2-architecture.md).
+
 ## Verification
 
-Static. `bun test tests/eval-report.test.ts tests/release-qualification.test.ts`.
+Static. `bun test tests/eval-report.test.ts tests/atomic-analysis.test.ts
+tests/advisory-analysis.test.ts tests/release-qualification.test.ts`.
 
 Runtime. The malformed report shape must be rejected before analysis. Complete
 synthetic matrices exercise all three verdicts. Legacy report behavior is recorded

.audit/eval-engineering.tsv

@@ -26,3 +26,8 @@ ts	phase	decision	why	evidence	result
 2026-08-24T23:49:00Z	phase-1	implemented the strict report and catalog boundary	atomic attempts must fail closed on malformed structure and inconsistent plans	evals/report.ts; evals/catalog.ts; tests/eval-report.test.ts	VERIFIED 19 focused cases
 2026-08-24T23:49:00Z	phase-1	fixed the multi-model Interrogate findings	budgets, terminal state, pair retries, evidence coherence, canonical JSON, and immutable validated values were trust boundaries	.agents/plans/02-eval-engineering/evidence/phase-1-review.md	VERIFIED no unresolved blocker
 2026-08-24T23:49:00Z	phase-1	ran Deslop and the whole repository gate	the phase must finish in a reviewable state without monolithic growth or product regressions	bun run check	VERIFIED 430 pass, 1 skip, 0 fail
+2026-08-25T00:08:36Z	phase-2	started from merged Phase 1 main	atomic analysis must build on the exact verified report boundary	git status on codex/eval-phase-2 at b6f3000	VERIFIED clean baseline, 430 pass, 1 skip, 0 fail
+2026-08-25T00:08:36Z	phase-2	settled atomic analysis through Architect and Arena	canary timing, actor identity, provider floors, and paired direction were underspecified	.agents/plans/02-eval-engineering/evidence/phase-2-architecture.md	VERIFIED Terra base with independent judge and two grafts
+2026-08-25T03:43:39Z	phase-2	recovered the implementation after a delegate usage-limit stop	the partial module and tests were inspectable, so the lead could finish without discarding verified work	evals/analysis.ts; tests/atomic-analysis.test.ts; tests/advisory-analysis.test.ts	VERIFIED focused gate green
+2026-08-25T03:43:39Z	phase-2	fixed the multi-model Interrogate findings	report-only scoping, provider fallback, exact provenance sets, reviewer incompleteness, analysis-kind guards, and reserve eligibility affected decision integrity	.agents/plans/02-eval-engineering/evidence/phase-2-review.md	VERIFIED no unresolved blocker
+2026-08-25T03:43:39Z	phase-2	ran Deslop and the whole repository gate	the analysis and tests must remain small, pure, and regression-free	bun run check	VERIFIED 448 pass, 1 skip, 0 fail