feat(evals): add stable report trends

.agents/plans/02-eval-engineering/evidence/phase-8-review.md

@@ -0,0 +1,33 @@
+# Phase 8 Interrogate review and trend proof
+
+Phase 8 adds structured evidence cards and a canonical compatibility key for
+longitudinal comparisons. Cards render planned, attempted, missing, product,
+failed-product, operational-failure, and unscored counts with Wilson intervals,
+completion state, distinct artifacts, evaluator digests, and reviewer or paired
+projections.
+
+The compatibility key treats candidate artifact identity and evaluator source
+commit as the intended treatment axis. It freezes normalized case/version/
+repetition/schedule/model cells, analysis/stopping/abort/budget policy, catalog
+oracle and release semantics, evaluator case/policy/grader digests, host
+configuration, requested actors, and delivered instruction hashes. Reports must
+be complete. Per-case and aggregate deltas are emitted only when the key matches;
+a longitudinal chain breaks at the first incompatible adjacent report.
+
+Strictly parsed fixtures prove that equal semantics with different artifact and
+evaluator source commits compare. Separate mutations to case version, analysis
+version, oracle, evaluator grader digest, host configuration, requested model, and
+instruction digest all refuse comparison.
+
+Benchmark coverage grows from three to five cases. The two new tasks require
+coordinated changes across multiple source files. All five cases carry versioned
+public/withheld contamination notes and at least two executable known-bad
+mutations. The hidden graders reject all twelve mutations and pristine fixtures,
+while known-good
+implementations pass. Model-facing prompts contain no evaluation, arm, oracle, or
+grader labels.
+
+All benchmark cases remain report-only. No historical report is backfilled and no
+new release regression threshold is claimed. The full repository gate passes 509
+tests with one intentional live-smoke skip. The final four-model review found no
+unresolved blocker.

.agents/plans/02-eval-engineering/phase-8-promotion-trends.md

@@ -31,3 +31,12 @@ change one compatibility-key field and prove comparison refuses.
 
 Stop gate. Legacy reports are never backfilled. A case becomes a release regression
 only after its calibration predicate is recorded.
+
+## Outcome
+
+Implemented and verified. Five benchmark cases now carry versioned contamination
+notes and twelve executable mutation controls. Structured cards expose missing and
+failed evidence. Strict parsed-report comparisons allow different candidate
+artifacts while refusing case, policy, oracle, evaluator, host, actor, or
+instruction drift. Every case remains report-only and no legacy report is
+backfilled. See `evidence/phase-8-review.md`.

.audit/eval-engineering.tsv

@@ -54,3 +54,7 @@ ts	phase	decision	why	evidence	result
 2026-08-25T08:28:46Z	phase-7	fixed multi-model Interrogate findings	count-only scans, self-hashed fabricated masks, early recovery-link removal, startup leaks, and permissive reveal weakened evidence integrity	.agents/plans/02-eval-engineering/evidence/phase-7-review.md	VERIFIED exact scan multiset, semantic mask recomputation, durable link order, and strict reveal
 2026-08-25T08:28:46Z	phase-7	ran the accepted bounded paired pilot	the runner needs real packed-host evidence while refusing an underpowered product claim	.agents/plans/02-eval-engineering/evidence/phase-7-pilot.json	VERIFIED 1 tie, 0 unresolved, exact clean scans, $0.2450722 of $1, INCONCLUSIVE on power
 2026-08-25T08:28:46Z	phase-7	ran Deslop and the full repository gate	the phase must finish reviewable and regression-free	bun run check	VERIFIED 496 pass, 1 skip, 0 fail
+2026-08-25T08:55:41Z	phase-8	added stable-semantics report cards and longitudinal comparison	artifact changes are the treatment axis; case, policy, oracle, evaluator, host, actor, and instruction changes invalidate a trend	evals/report-render.ts; tests/report-render.test.ts	VERIFIED parsed-report compatibility and drift matrix green
+2026-08-25T08:55:41Z	phase-8	expanded mutation-tested hidden coverage	new tasks need executable controls and explicit contamination boundaries before producing useful evidence	evals/benchmarks.ts; tests/benchmark-reporting.test.ts	VERIFIED 5 cases, 12 rejected mutations, known-good implementations pass
+2026-08-25T08:55:41Z	phase-8	kept coverage promotion closed	uncalibrated cases cannot silently become release regressions	evals/benchmark-run.ts catalog policy; .agents/plans/02-eval-engineering/evidence/phase-8-review.md	VERIFIED every benchmark case remains report-only; no legacy backfill
+2026-08-25T08:55:41Z	phase-8	ran Deslop, four-model Interrogate, and full repository gate	trend and coverage changes must remain reviewable and regression-free	bun run check	VERIFIED 509 pass, 1 skip, 0 fail; no unresolved blocker

evals/benchmark.ts

@@ -7,12 +7,30 @@ export type BenchmarkGrade = {
 	readonly issues: readonly string[];
 };
 
+export type BenchmarkContaminationNotes = {
+	readonly schemaVersion: 1;
+	readonly public: readonly string[];
+	readonly withheld: readonly string[];
+};
+
+export type BenchmarkKnownBadMutation = {
+	readonly id: string;
+	readonly fileOverrides: Readonly<Record<string, string>>;
+};
+
+export type BenchmarkOracleMetadata = {
+	readonly schemaVersion: 1;
+	readonly contamination: BenchmarkContaminationNotes;
+	readonly knownBadMutations: readonly BenchmarkKnownBadMutation[];
+};
+
 /** One task whose result can be graded without trusting model-written tests. */
 export type BenchmarkCase = {
 	readonly id: string;
 	readonly description: string;
 	readonly files: Readonly<Record<string, string>>;
 	readonly prompt: string;
+	readonly oracle: BenchmarkOracleMetadata;
 	readonly grade: (project: string) => Promise<BenchmarkGrade>;
 };