feat(statistics): add repository-clustered paired inference

.claude/skills/agent-eval/SKILL.md

@@ -145,6 +145,7 @@ Extend, don't fork — see §"Extend, don't duplicate."
 | `HeldOutGate` | `held-out-gate.ts` | Paired-delta + overfit-gap gate. Three rejection codes: `few_runs`, `negative_delta`, `overfit_gap`. Use before promoting an optimizer's top-1. Pairs with `promotion-gate.ts` (bootstrap CI for "is this real?") — use both. |
 | `RunRecord` | `run-record.ts` | Typed run schema. `validateRunRecord` throws on missing fields and on bare model aliases — record the snapshot (`claude-sonnet-4-6@2025-04-15`). |
 | `pairedBootstrap`, `pairedWilcoxon`, `bhAdjust` | `paired-stats.ts` | Stats primitives. Pass `seed` to `pairedBootstrap` when the result feeds a CI / promotion decision. |
+| `clusteredPairedBinary`, `holm` | `clustered-paired-binary.ts`, `statistics.ts` | Repository- or subject-clustered paired binary inference plus strong family-wise multiple-comparison adjustment. The clustered result preserves original paired/unpaired rows, resamples whole clusters for a task-weighted interval, and sign-flips whole-cluster totals for the same estimand with a required deterministic seed. |
 | `runCanaries` | `canary.ts` | Silent fallback (constant confidence), calibration drift (KS), distribution shift (chi-square). Returns a report; doesn't fail tests — wire it to a notification. |
 | `summaryTable`, `paretoChart`, `gainHistogram` | `summary-report.ts` | A/B reporting. `summaryTable` emits markdown with bootstrap CIs + paired Wilcoxon p (BH-adjusted) + Cohen's d. The other two return vega-lite-friendly specs. |
 | `researchReport` | `summary-report.ts` | Async, launch-decision-grade artifact: paired-evidence-only verdicts (`promote` / `hold` / `equivalent` / `reject` / `needs_more_data`), ROPE, Pr(Δ>0), per-candidate MDE via `pairedMde`, SHA-256 `runFingerprint`, optional `preregistrationHash`, embedded methodology. See [`docs/research-report-methodology.md`](../../../docs/research-report-methodology.md). |

.evolve/critical-audit/2026-07-09T22-35-41Z/findings.jsonl

@@ -0,0 +1 @@
+{"severity":"LOW","status":"accepted-risk","file":"src/clustered-paired-binary.ts","line":134,"issue":"The implementation is covered by exact synthetic cases and adversarial counterexamples but has no checked-in parity vector from an independent statistics package.","action":"Before a paper claim depends on this function, add one offline golden vector produced by an independent implementation and pin every reported statistic.","verification":"The new test should compare pair counts, both effect estimates, the interval, and the randomization p-value against the frozen external vector."}

.evolve/critical-audit/2026-07-09T22-35-41Z/manifest.json

@@ -0,0 +1,29 @@
+{
+  "scope": [
+    "src/clustered-paired-binary.ts",
+    "src/statistics.ts",
+    "src/index.ts",
+    "tests/clustered-paired-binary.test.ts",
+    ".claude/skills/agent-eval/SKILL.md"
+  ],
+  "base": "39e8002748f43aa9b440805e7c73000d8aba1a19",
+  "head": "abe1d89a11fe1b40b75d0e4d96ccf68dfa2403b1",
+  "project_type": "typescript-node",
+  "flags": ["--diff-only"],
+  "score": "8/10",
+  "verdict": "APPROVE",
+  "findings_count_by_severity": {
+    "CRITICAL": 0,
+    "HIGH": 0,
+    "MEDIUM": 0,
+    "LOW": 1
+  },
+  "verification": [
+    "pnpm test: 270 files, 2752 passed, 2 skipped",
+    "pnpm typecheck",
+    "pnpm build",
+    "pnpm verify:package",
+    "pnpm lint: 0 errors, 4 pre-existing warnings",
+    "git diff --check"
+  ]
+}

.evolve/critical-audit/2026-07-09T22-35-41Z/summary.md

@@ -0,0 +1,20 @@
+# Critical audit: clustered paired binary inference
+
+The implementation is correct for the reviewed contract, the previous untrustworthy audit records are gone, and the exact source commit audited here is `abe1d89a11fe1b40b75d0e4d96ccf68dfa2403b1`.
+
+## Finding
+
+- LOW — `src/clustered-paired-binary.ts:134`: no checked-in parity vector from an independent statistics package.
+  Action: add one frozen external comparison before this function supports a paper claim.
+  Verification: pin every output dimension against the independent vector.
+
+## Checks
+
+- `pnpm test`: 2,752 passed and 2 skipped across 270 files.
+- `pnpm typecheck`: passed.
+- `pnpm build`: passed.
+- `pnpm verify:package`: passed.
+- `pnpm lint`: zero errors and four pre-existing warnings.
+- `git diff --check`: passed.
+
+APPROVE — no critical, high, or incident-causing medium finding remains.

.evolve/skill-runs.jsonl

@@ -15,3 +15,5 @@
 {"ts":"2026-06-01","skill":"pursue","generation":6,"goal":"empirical proof of substrate lift","outcome":"HONEST-NEGATIVE: mechanism proven e2e on AppWorld d3 (real backend), but NO significant prompt-opt lift on a competent baseline; lift needs a weak-baseline config","files":["src/campaign/presets/compare-drivers.ts","examples/benchmarks/gsm8k/compare-drivers.ts","examples/benchmarks/appworld/run-bench.ts","examples/benchmarks/appworld/repl_agent.py"],"branch":"pursue/empirical-proof","cost":"~$3.50 live"}
 {"skill":"/research","ts":"2026-06-05T13:21:01Z","project":"agent-eval-belief-roadmap","target":"belief-state-code-agent-corpus","operatorPrompt":"","durationMin":null,"verdict":"SHIP","dispatchedTo":"stop","operatorOverride":null,"transcriptPath":null,"traceDir":null}
 {"skill":"critical-audit","ts":"2026-06-21T12:57:05Z","project":"agent-eval","target":"PR #264 review comment 4762012574","verdict":"APPROVE_AFTER_FIXES","score":"8/10","findings":"1 medium, 9 low; 8 fixed, 1 not_actionable, 1 deferred","files":[".gitignore","CHANGELOG.md","src/agent-profile.ts","src/agent-profile.test.ts","src/contract/define-agent-eval.ts","src/contract/index.ts","tests/contract-define-agent-eval.test.ts"]}
+{"skill":"critical-audit","ts":"2026-07-09T21:54:37Z","project":"agent-eval","target":"clustered paired binary inference for repository-disjoint coding benchmarks","verdict":"APPROVE_AFTER_FIXES","score":"8/10","findings":"3 medium found and resolved: estimand mismatch, one-cluster interval, insufficient bootstrap draws","files":["src/clustered-paired-binary.ts","src/statistics.ts","src/index.ts","tests/clustered-paired-binary.test.ts",".claude/skills/agent-eval/SKILL.md"]}
+{"skill":"critical-audit","ts":"2026-07-09T22:35:41Z","project":"agent-eval","target":"clustered paired binary inference at abe1d89a11fe1b40b75d0e4d96ccf68dfa2403b1","verdict":"APPROVE","score":"8/10","findings":"1 low accepted risk: independent-package parity vector remains before paper use","files":["src/clustered-paired-binary.ts","src/statistics.ts","src/index.ts","tests/clustered-paired-binary.test.ts",".claude/skills/agent-eval/SKILL.md"]}

CHANGELOG.md

@@ -4,6 +4,15 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
+## [0.111.0] — 2026-07-09 — repository-clustered paired inference
+
+### Added
+
+- Added `clusteredPairedBinary`, which pairs binary outcomes by work item, exposes every unmatched row, resamples whole repositories for a task-weighted confidence interval, and tests the same effect with whole-repository sign flips.
+- Added Holm step-down adjustment for strong family-wise error control across benchmark arms.
+
+The cluster interval is unavailable below two repositories, and consumers must reject unmatched rows before promotion.
+
 ## [0.110.1] — 2026-07-09 — proposer portfolio export
 
 ### Added

clients/python/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agent-eval-rpc"
-version = "0.110.1"
+version = "0.111.0"
 description = "Python RPC client for @tangle-network/agent-eval — judge content against rubrics over HTTP or stdio RPC. Eval logic runs in the Node runtime; this package is a thin wire client."
 readme = "README.md"
 requires-python = ">=3.10"

clients/python/src/agent_eval_rpc/__init__.py

@@ -58,7 +58,7 @@
 try:
     __version__ = version("agent-eval-rpc")
 except PackageNotFoundError:
-    __version__ = "0.110.1"
+    __version__ = "0.111.0"
 
 __all__ = [
     "Client",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.110.1",
+  "version": "0.111.0",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {