feat(evals): add reviewer calibration campaign

.agents/plans/02-eval-engineering/evidence/phase-6-pilot.json

@@ -0,0 +1,34 @@
+{
+  "schemaVersion": 1,
+  "reportId": "flow-reviewer-2026-08-25T07-08-21-624Z",
+  "reportSha256": "sha256:f5840f53260ca7e7e031c94bcaea80895e901d07beeef942930aaa752e2a6afc",
+  "planSha256": "sha256:75ac17c6ffdf7de1f8fb6c93b605a9e415a0a4995ea1dfd1b4b4ecf6a03e37b6",
+  "completionStatus": "complete",
+  "completionCause": "fixed-target",
+  "attemptCount": 2,
+  "submitted": 2,
+  "truePositives": 1,
+  "falseNegatives": 0,
+  "falsePositives": 0,
+  "trueNegatives": 1,
+  "detectionRate": 1,
+  "detectionInterval95": [0.20654931437723745, 1],
+  "falsePositiveRate": 0,
+  "falsePositiveInterval95": [0, 0.7934506856227626],
+  "outputTokens": 1747,
+  "costUsd": 0.14806019999999998,
+  "artifactTarballSha256": "sha256:c89f7363248ccc3e3f69728c1aa42044a25938cd533e470d9f73ef08bc64ad24",
+  "defectTranscriptSha256": "sha256:c09cd249220683b6b2ad3fb51d17ce7058c2c96637152de2758270ae6238bbfc",
+  "cleanTranscriptSha256": "sha256:fbd7065faaf18f421c4cd8e17b6077492fcec6158293c1b2309b5d4d4b834e11",
+  "reviewerProviderModelObserved": true,
+  "reviewerFullIdentityObserved": false,
+  "strictParsePassed": true,
+  "promotion": "advisory",
+  "promotionReasons": [
+    "pilot-only",
+    "sample-floor-missed",
+    "detection-bound-missed",
+    "false-positive-bound-missed",
+    "full-reviewer-identity-unavailable"
+  ]
+}

.agents/plans/02-eval-engineering/evidence/phase-6-review.md

@@ -0,0 +1,39 @@
+# Phase 6 Interrogate review and pilot
+
+Phase 6 measures the reviewer without a manager model selecting, repairing, or
+submitting the work. The evaluator seeds an approved plan, real JUnit-backed
+validation, and a pending assignment through Flow's application services. The
+normal `/flow-review` command creates the reviewer child. Scoring reads only the
+durable assignment result from active or archived Flow state.
+
+Architect and Arena rejected a direct-agent shortcut because it bypassed child
+lineage and production dispatch semantics. The selected design reuses the real
+review command and existing reviewer identity observation. Requested identity
+stays separate from host-observed provider and model fields; missing family,
+gateway, and revision fields remain explicitly unobserved.
+
+The adversarial pass fixed four evidence defects before the accepted pilot:
+
+- fixed truth now hashes the complete fixture, and promotion labels must exactly
+  match the versioned immutable registry;
+- unsubmitted outcomes enter neither rate denominator, and any unsubmitted or
+  non-fixed-target campaign is ineligible for promotion;
+- the runner preserves user escalation and scores a verdict only when Flow
+  durably accepted the reviewer submission;
+- completed sessions are read from the archive, and OpenCode's lazy package setup
+  finishes before source binding;
+- both controls run the declared JUnit case, include baseline inventory, and use a
+  safe-integer requirement. The planted defect passes visible validation but still
+  violates the approved behavior.
+
+The accepted paid pilot produced two submitted outcomes. The reviewer detected the
+planted defect and passed the clean control. Cost was $0.1480602 for 1,747 output
+tokens. Strict v2 parsing and reviewer analysis passed. The apparent 100% rates are
+descriptive only: one case per truth class gives a detection lower bound of 0.2065
+and a false-positive upper bound of 0.7935. Full reviewer identity is unavailable
+on OpenCode 1.18.6. Promotion therefore remains advisory and release policy remains
+report-only.
+
+Earlier interrupted and invalid campaigns are excluded. They exposed archive,
+source-binding, and control-definition defects and support no performance claim.
+The final four-model recheck found no unresolved blocker.

.agents/plans/02-eval-engineering/phase-6-reviewer-calibration.md

@@ -34,3 +34,12 @@ measurement, not calibration by itself.
 Stop gate. Release promotion needs a preregistered human-labelled set, at least two
 raters per case, Krippendorff alpha at or above 0.8, and confidence bounds that meet
 the recorded detection and false-positive thresholds.
+
+## Outcome
+
+Implemented and verified. The packed-host pilot exercised two real reviewer child
+assignments with durable submissions and exact fixed labels. It detected the
+planted defect and passed the clean control, but remains advisory because the
+sample is below the preregistered floor, confidence bounds miss promotion
+thresholds, and the pinned host cannot expose a full reviewer identity. See
+`evidence/phase-6-review.md` and `evidence/phase-6-pilot.json`.

.audit/eval-engineering.tsv

@@ -45,3 +45,7 @@ ts	phase	decision	why	evidence	result
 2026-08-25T04:54:04Z	phase-5	fixed the multi-model Interrogate findings	workflow contracts, record handoff, full artifact identity, catalog hashing, per-attempt host config, and required policies affected release integrity	.agents/plans/02-eval-engineering/evidence/phase-5-review.md	VERIFIED no unresolved blocker
 2026-08-25T04:54:04Z	phase-5	completed the throughput checkpoint	the vertical slice must reduce maintenance load before new evidence families	.agents/plans/02-eval-engineering/evidence/phase-5-review.md	VERIFIED new scenario touches scenario, policy, and one test; legacy path has no authority
 2026-08-25T04:54:04Z	phase-5	ran Deslop, workflow lint, and the full repository gate	the cutover must land atomically and regression-free	actionlint; bun run check	VERIFIED 473 pass, 1 skip, 0 fail
+2026-08-25T07:12:18Z	phase-6	selected the production reviewer path through Architect and Arena	direct agent prompts bypass reviewer-child lineage and production dispatch semantics	.agents/plans/02-eval-engineering/evidence/phase-6-review.md	VERIFIED normal /flow-review child over evaluator-seeded durable assignment
+2026-08-25T07:12:18Z	phase-6	fixed truth, labels, durable scoring, archive recovery, source binding, and controls	reviewer calibration cannot trust prose, partial fixture hashes, mutable labels, or evaluator-created drift	evals/reviewer-cases.ts; evals/reviewer-assignment.ts; evals/reviewer-run.ts; evals/reviewer-calibration.ts	VERIFIED focused integration and strict parser gates green
+2026-08-25T07:12:18Z	phase-6	ran the final paid packed-host reviewer pilot	the phase needs real defect and clean evidence without manager selection or repair confounding	.agents/plans/02-eval-engineering/evidence/phase-6-pilot.json	VERIFIED 1 detection, 0 false positives, 2 submissions, advisory only, $0.1480602
+2026-08-25T07:12:18Z	phase-6	ran Deslop, four-model Interrogate, and the full repository gate	the phase must remain reviewable, regression-free, and make no unsupported promotion claim	bun run check; .agents/plans/02-eval-engineering/evidence/phase-6-review.md	VERIFIED 483 pass, 1 skip, 0 fail; no unresolved blocker

evals/canonical-json.ts

@@ -46,7 +46,10 @@ export function canonicalJson(value: unknown): string {
 	throw new Error("Canonical JSON requires JSON values.");
 }
 
-export function canonicalSha256(domain: string, value: unknown): string {
+export function canonicalSha256(
+	domain: string,
+	value: unknown,
+): `sha256:${string}` {
 	return `sha256:${createHash("sha256")
 		.update(`${domain}\u0000`)
 		.update(canonicalJson(value))

evals/harness.ts

@@ -982,6 +982,8 @@ export class EvalHost {
 		packageCache: string;
 		opencodeVersion: string;
 		files: Readonly<Record<string, string>>;
+		/** Pins the hidden reviewer child independently from the command model. */
+		reviewerModel?: string;
 		/** False creates the paired benchmark's ordinary OpenCode control host. */
 		withFlow?: boolean;
 	}): Promise<EvalHost> {
@@ -1055,6 +1057,9 @@ export class EvalHost {
 				cwd: project,
 				env: {
 					...process.env,
+					...(options.reviewerModel
+						? { OPENCODE_FLOW_REVIEWER_MODEL: options.reviewerModel }
+						: {}),
 					HOME: childHome,
 					XDG_CACHE_HOME: childCache,
 					XDG_CONFIG_HOME: join(childHome, ".config"),

evals/reviewer-assignment.ts

@@ -0,0 +1,255 @@
+import type { ReviewFinding, Session } from "../src/domain/session.js";
+import { observeAssertions } from "../src/domain/test-results.js";
+import { normalizeEvidencePlatform } from "../src/domain/validation.js";
+import { createFileSessionRepository } from "../src/infrastructure/fs/session-repository.js";
+import {
+	flowPlanApprove,
+	flowPlanSave,
+	flowReviewStart,
+	flowRunStart,
+} from "../src/infrastructure/fs/workspace-flow-service.js";
+import {
+	persistWorkspaceValidation,
+	prepareWorkspaceValidation,
+	readWorkspaceTestReport,
+} from "../src/infrastructure/fs/workspace-validation.js";
+import { canonicalSha256 } from "./canonical-json.js";
+import {
+	assertReviewerCaseTruth,
+	type ReviewerCase,
+} from "./reviewer-cases.js";
+
+const FEATURE_ID = "review-target";
+const RESULTS_PATH = ".flow/reviewer-results.xml";
+const VALIDATION_COMMAND = `bun test --reporter=junit --reporter-outfile=${RESULTS_PATH}`;
+
+export type SeededReviewerAssignment = {
+	readonly flowSessionId: string;
+	readonly featureId: string;
+	readonly runId: string;
+	readonly assignmentId: string;
+};
+
+export type DurableReviewerSubmission =
+	| { readonly kind: "unsubmitted" }
+	| {
+			readonly kind: "submitted";
+			readonly verdict: "passed" | "failed";
+			readonly findings: readonly ReviewFinding[];
+	  };
+
+function requireFlowSuccess(response: {
+	readonly status: "ok" | "error";
+	readonly summary: string;
+}): void {
+	if (response.status === "error") throw new Error(response.summary);
+}
+
+async function currentSession(workspace: string): Promise<Session> {
+	const session = await createFileSessionRepository(workspace).read();
+	if (!session) throw new Error("Reviewer assignment setup lost Flow state.");
+	return session;
+}
+
+async function verifySeedValidation(workspace: string): Promise<{
+	readonly outputDigest: `sha256:${string}`;
+	readonly report: string;
+}> {
+	const process = Bun.spawn(
+		["bun", "test", "--reporter=junit", `--reporter-outfile=${RESULTS_PATH}`],
+		{
+			cwd: workspace,
+			stdout: "pipe",
+			stderr: "pipe",
+		},
+	);
+	const [exitCode, stdout, stderr] = await Promise.all([
+		process.exited,
+		new Response(process.stdout).text(),
+		new Response(process.stderr).text(),
+	]);
+	if (exitCode !== 0) {
+		throw new Error(`Reviewer seed validation failed: ${stderr || stdout}`);
+	}
+	const report = await readWorkspaceTestReport(workspace, RESULTS_PATH);
+	if (!report)
+		throw new Error("Reviewer seed validation wrote no JUnit report.");
+	return {
+		outputDigest: canonicalSha256("flow-reviewer-seed-validation-v1", {
+			command: VALIDATION_COMMAND,
+			exitCode,
+			stdout,
+			stderr,
+			report: report.text,
+		}),
+		report: report.text,
+	};
+}
+
+/** Creates a real pending Flow assignment without involving a manager model. */
+export async function seedReviewerAssignment(input: {
+	readonly workspace: string;
+	readonly fixture: ReviewerCase;
+}): Promise<SeededReviewerAssignment> {
+	assertReviewerCaseTruth(input.fixture);
+	const operationSuffix = `${input.fixture.caseId}-v${input.fixture.caseVersion}`;
+	requireFlowSuccess(
+		await flowPlanSave(input.workspace, {
+			request: {
+				operationId: `reviewer-plan-save-${operationSuffix}`,
+				expectedRevision: 0,
+				goal: "Verify the current value implementation.",
+				plan: {
+					summary: "Verify the value implementation.",
+					overview:
+						"Inspect the implementation independently before completion.",
+					requirements: [
+						"For every safe-integer input below Number.MAX_SAFE_INTEGER, value returns input plus one.",
+					],
+					decisions: [
+						"Use the exported value function as the public boundary.",
+					],
+					evidence: [
+						{
+							scope: "gate",
+							requirement: "Behavioral correctness",
+							environment: "isolated evaluator workspace",
+							command: VALIDATION_COMMAND,
+							platform: normalizeEvidencePlatform(process.platform),
+							assertions: ["value returns the next safe integer"],
+						},
+					],
+					features: [
+						{
+							id: FEATURE_ID,
+							kind: "inspect",
+							title: "Review the value implementation",
+							summary:
+								"Verify the value function against the approved requirement.",
+							targets: Object.keys(input.fixture.files).sort(),
+							validation: [VALIDATION_COMMAND],
+							dependsOn: [],
+						},
+					],
+				},
+			},
+		}),
+	);
+	requireFlowSuccess(
+		await flowPlanApprove(input.workspace, {
+			request: {
+				operationId: `reviewer-plan-approve-${operationSuffix}`,
+				expectedRevision: (await currentSession(input.workspace)).revision,
+			},
+		}),
+	);
+	requireFlowSuccess(
+		await flowRunStart(input.workspace, {
+			request: {
+				operationId: `reviewer-run-start-${operationSuffix}`,
+				expectedRevision: (await currentSession(input.workspace)).revision,
+				featureId: FEATURE_ID,
+			},
+		}),
+	);
+	const prepared = await prepareWorkspaceValidation(input.workspace, {
+		expectedRevision: (await currentSession(input.workspace)).revision,
+		featureId: FEATURE_ID,
+		command: VALIDATION_COMMAND,
+		scope: "broad",
+		resultsPath: RESULTS_PATH,
+	});
+	const validation = await verifySeedValidation(input.workspace);
+	await persistWorkspaceValidation(input.workspace, {
+		...prepared,
+		captureId: `reviewer-validation-${operationSuffix}`,
+		exitCode: 0,
+		outputDigest: validation.outputDigest,
+		outputComplete: true,
+		hostPlatform: normalizeEvidencePlatform(process.platform),
+		observedAssertions: observeAssertions(
+			prepared.assertions,
+			validation.report,
+		),
+	});
+	requireFlowSuccess(
+		await flowReviewStart(input.workspace, {
+			request: {
+				operationId: `reviewer-review-start-${operationSuffix}`,
+				expectedRevision: (await currentSession(input.workspace)).revision,
+				featureId: FEATURE_ID,
+				artifactsChanged: Object.keys(input.fixture.files)
+					.sort()
+					.map((path) => ({ path })),
+				packet: {
+					summary:
+						"Verify the value function against the approved requirement. Baseline inventory: src/value.ts and src/value.test.ts are tracked regular non-executable files; there are no source deletions, renames, generated artifacts, symlinks, or file-mode changes. Host-owned .flow and .opencode files are outside the source change.",
+					riskLenses: ["functional correctness", "boundary behavior"],
+				},
+			},
+		}),
+	);
+	const session = await currentSession(input.workspace);
+	const run = session.runs.find(
+		(candidate) =>
+			candidate.featureId === FEATURE_ID && candidate.state === "active",
+	);
+	const assignment = run?.reviews.find((review) => review.result === null);
+	if (!run || !assignment) {
+		throw new Error(
+			"Reviewer assignment setup did not persist a pending review.",
+		);
+	}
+	return {
+		flowSessionId: session.id,
+		featureId: FEATURE_ID,
+		runId: run.id,
+		assignmentId: assignment.id,
+	};
+}
+
+export function durableReviewerSubmission(input: {
+	readonly session: Session | null;
+	readonly seed: SeededReviewerAssignment;
+}): DurableReviewerSubmission {
+	const run = input.session?.runs.find(
+		(candidate) => candidate.id === input.seed.runId,
+	);
+	const assignment = run?.reviews.find(
+		(candidate) => candidate.id === input.seed.assignmentId,
+	);
+	if (!assignment) {
+		throw new Error(
+			"The seeded reviewer assignment disappeared from Flow state.",
+		);
+	}
+	if (
+		assignment.result === null ||
+		assignment.result.terminalDisposition !== "submitted"
+	) {
+		return { kind: "unsubmitted" };
+	}
+	return {
+		kind: "submitted",
+		verdict: assignment.result.verdict,
+		findings: assignment.result.findings,
+	};
+}
+
+export async function readDurableReviewerSubmission(input: {
+	readonly workspace: string;
+	readonly seed: SeededReviewerAssignment;
+}): Promise<DurableReviewerSubmission> {
+	const repository = createFileSessionRepository(input.workspace);
+	const active = await repository.read();
+	if (active?.runs.some((run) => run.id === input.seed.runId)) {
+		return durableReviewerSubmission({ session: active, seed: input.seed });
+	}
+	const archived = await repository.transact((transaction) =>
+		transaction.loadArchive(input.seed.flowSessionId),
+	);
+	return durableReviewerSubmission({
+		session: archived,
+		seed: input.seed,
+	});
+}