evidence: cohort-split the benchmark, and withdraw the baseline claim that did not survive a fair comparison

[aryamthecodebreaker]

What this corrects

Three of twelve held-out tasks name their expected fixing file in the task text. They remain valid tests of explicit-file-mention handling, but they are no longer counted as generalization evidence.

Held-out cohort	Cases	Top-1	Top-3	Top-5
Task did not name the file	9	4/9 (44.4%)	5/9 (55.6%)	6/9 (66.7%)
Task named the file	3	3/3	3/3	3/3
Pooled, previously published	12	7/12	8/12	9/12

The cohort flag is derived during evaluation from the same task text the ranker reads; it is never hand-maintained in dataset JSON.

Fair baseline comparison

All arms rank one shared scanRepo() result per case. Literal retrieval and BM25 are reported under raw, source, and code candidate policies; the strongest recorded policy is used instead of handicapping a baseline with documentation FixMap itself deprioritizes.

On the nine held-out tasks that did not name the file:

Arm	Top-1	Top-3	Top-5
Literal retrieval over code files	2/9	4/9	6/9
BM25 over code files	4/9	5/9	9/9
FixMap	4/9	5/9	6/9

FixMap does not beat BM25-over-code on this unseen cohort. Top-1 and Top-3 are exact ties; at Top-5 BM25 wins three cases FixMap misses.

Review hardening

Codex review found and fixed four release-blocking inconsistencies before merge:

• evaluate-baseline.mjs --check-recorded previously did nothing; it now rejects missing or drifted artifacts.
• The site previously hard-coded which candidate policy was strongest; it now reads the recorded selection and fails if the arm is absent.
• The README graphic, benchmark docs, held-out/regression suite docs, launch kit, and growth decision log previously retained misleading pooled promotional copy. They now agree with the cohort and baseline evidence.
• The first Ubuntu evidence run correctly rejected Windows-specific checkout file counts. Those environment diagnostics are now omitted from recorded artifacts; rankings and hit outcomes remain gated and unchanged.

A dated, citable narrative is in docs/releases/2026-08-04-benchmark-self-audit.md.

Verification

After merging the independent security fix #450 into this branch:

• clean npm ci — 0 vulnerabilities
• full npm run ci — 430 tests, audit, lint, builds, Action bundle drift, smokes, evaluation, and scan benchmark all pass
• held-out and regression ranking records reproduce byte-for-byte
• held-out and regression baseline records reproduce byte-for-byte
• a deliberate held-out baseline mutation is rejected with exit code 1, then restored to its original SHA-256
• production Next.js build prerenders all routes, including /evidence
• generated benchmark SVG is derived from the corrected recorded artifacts

Not in scope

This PR does not change the shipped ranker, add replacement held-out cases, release or deploy a version, or begin ranking optimization. The nine-case interval remains wide; expanding the mechanically selected cohort is follow-up evidence work.

🤖 Initial implementation generated with Claude Code; independently reviewed and hardened by Codex.

[aryamthecodebreaker]

@aryamthecodebreaker — requesting review before this lands. Two things worth a reviewer's attention specifically:

1. The headline reversed inside this PR. Commit 66d9167 reported FixMap beating BM25 at p = 0.004–0.016. That was wrong: the baselines ranked over the raw scan while FixMap ranked its own gated candidate set, so the baselines returned README.md and CONTRIBUTING.md first. Commit 0ca1294 fixes it and the result flips to a tie at Top-1/Top-3 and a loss at Top-5 on held-out. Both commits are kept rather than squashed so the error and its correction are inspectable.

The specific thing to check: is code = isSource && !isTest && kind === "code" a fair stand-in for FixMap's own candidate policy? It is an approximation built from public RepoFile fields, not the real gate in rank.ts (which also drops lockfiles, excluded paths, generated output, and benchmarks/). If it is too generous to the baseline, the tie is overstated; if too strict, the loss is understated. This single choice drives the whole conclusion.

2. CI is red for an unrelated reason. npm audit --omit=dev --audit-level=high fails on fast-uri and ip-address advisories published after main last ran green. This branch touches no manifest or lockfile and the failure reproduces on unmodified 2e19cba. It needs its own dependency bump — this PR should not be the one to fix it, but it also cannot merge until something does.

Datasets are unchanged, pooled rates still reproduce (--check-recorded exits 0 on both suites), and no release, deploy, or publish is included.