Proposed evaluation set: 23 labeled issues from this repo, cohort-split (78% top-1 with paths named, 43% without)

[aryamthecodebreaker]

Not a bug report. This proposes a new evaluation case set and reports what FixMap scores on it today.

Issues #451–#473 were filed against this repository in a single session, each with a fix site identified while investigating it. That makes 23 labeled cases on FixMap's own codebase that appear in no existing suite — benchmarks/cases.json has 8 and benchmarks/external/dataset.json has 16, neither containing any of these.

They were also written before any of this measurement, so nothing about the ranking influenced how they were worded.

Measured today, at 8ecf5f1 (v0.8.7)

Ranked with rankContextFiles(repo, { issueText }, 5) against a local scan of this repository. Both arms score the identical 23 cases; only the input text differs.

	full issue (title + body)	title only
top-1	78%	43%
top-3	96%	83%
top-5	100%	91%

The bodies of these issues cite file paths and line ranges (repo-scan.ts:501-512, and so on), which places the full-issue arm squarely in the mentioned cohort. The title-only arm removes every path mention and is the closer analogue of the unmentioned cohort.

The 35-point top-1 gap reproduces the cohort effect established in #449 on a completely independent case set — which is the useful part. It is not a new finding so much as an independent confirmation that the split is real and large.

Per-case movement, full → title-only

  #451  T1 -> T3   packages/cli/src/cli-runner.ts
  #455  T1 -> T3   packages/cli/src/cli-runner.ts
  #459  T3 -> T1   packages/action/src/issue-source.ts
  #461  T1 -> --   scripts/evaluate.mjs
  #462  T1 -> T3   .github/workflows/publish.yml
  #464  T1 -> T3   packages/core/src/repo-scan.ts
  #465  T5 -> T3   packages/core/src/verify.ts
  #466  T1 -> T3   packages/core/src/rank.ts
  #468  T3 -> T5   packages/cli/src/cli-runner.ts
  #470  T1 -> T5   packages/core/src/repo-scan.ts
  #471  T1 -> --   packages/core/src/report.ts
  #472  T1 -> T3   packages/core/src/repo-scan.ts

Title-only top-5 misses: #461 (scripts/evaluate.mjs) and #471 (packages/core/src/report.ts).

#459 improving without its body is worth a look — the body discusses both issue-source.ts and the CLI's repository-source.ts, and the CLI file outranks the correct Action file when both are described.

Proposed labels

451 packages/cli/src/cli-runner.ts        463 apps/web/app/demo.tsx
452 packages/core/src/repo-scan.ts        464 packages/core/src/repo-scan.ts
453 packages/core/src/repo-scan.ts        465 packages/core/src/verify.ts
454 packages/core/src/verify.ts           466 packages/core/src/rank.ts
455 packages/cli/src/cli-runner.ts        467 packages/core/src/signals.ts
456 packages/core/src/verify.ts           468 packages/cli/src/cli-runner.ts
457 packages/action/src/runner.ts         469 packages/core/src/repo-scan.ts
458 packages/action/src/runner.ts,        470 packages/core/src/repo-scan.ts
    packages/action/src/github.ts         471 packages/core/src/report.ts
459 packages/action/src/issue-source.ts   472 packages/core/src/repo-scan.ts
460 packages/cli/src/cli-runner.ts        473 packages/core/src/repo-scan.ts
461 scripts/evaluate.mjs
462 .github/workflows/publish.yml

Limitations, stated plainly

• The labels are self-assigned. I identified each fix site while writing the issue; none has been confirmed by an actual fix. A real fixing commit is stronger evidence, and these should be re-labelled from the diffs once the issues are addressed. Until then this set is weaker than benchmarks/external, which is anchored to merged fixing PRs.
• Single repository. All 23 cases are FixMap ranking FixMap, so the result says nothing about generalization and should not be pooled with the cross-repository suites.
• Not adversarial. These are issues I wrote, in my own vocabulary, about code I had just read. Real reporters describe symptoms, not mechanisms.
• Small n. 23 cases; a single case moves the rate by ~4 points, and no confidence intervals are computed here. The existing suites report intervals95; this should too before any number from it is published.

Given those limitations I would not put these figures on the website. The set's value is as a regression fixture on the repository's own code — cheap to re-run, and the title-only arm gives a hard cohort that the current 8-case cases.json lacks.

The measurement scripts are throwaway; if this is worth adopting I would expect the cases folded into benchmarks/ in the existing format rather than kept as a separate harness.

Related: #466 notes that neither existing suite contains a task mentioning an HTTP status code or deployment vocabulary, which is a separate coverage gap in the same area.