[igerber]

Summary

• Two blinded, pre-registered 44-review campaigns on the tools/reviewer-eval harness frozen by Pre-register the gpt-5.6-sol -> gpt-6-astra Codex reviewer campaign (PR-1: harness re-pin, no model swap) #849:
  • Campaign 1 (B = gpt-5.6-sol vs E = gpt-6-astra, both @ xhigh, k=2, frozen at the Pre-register the gpt-5.6-sol -> gpt-6-astra Codex reviewer campaign (PR-1: harness re-pin, no model swap) #849 merge 225a4afb): mechanical DEFINITIVE NO-GO on one regression (s4-cs-unbalanced-nan-outcome: B caught 2/2, E partial 2/2). Accepted by the user; no astra swap. tools/reviewer-eval/campaigns/2026-10-gpt6-VERDICT.md.
  • Campaign 2 (B vs F = gpt-6.1-sol @ xhigh, k=2, fresh control sample, same rule re-issued for F, frozen at a local commit anchored by the ledger + refs/campaign/B-F/frozen): mechanical DEFINITIVE PARITY: zero rule-defined regressions, zero downgrades, 0 vs 0 false positives, two favourable catch-status changes against one UNSTABLE-to-MISSED judgment flag, 11 vs 10 caught repeat-cells, lower latency (mean 181 s vs 213 s), half the input price. The user accepted PARITY as GO. tools/reviewer-eval/campaigns/2026-10-gpt61sol-VERDICT.md.
  • Both verdicts ship with their grading records (campaigns/2026-10-gpt6*-grading/: three blinded grader tables, verifier rulings, reconciled grades, the audit hit list with command lines hashed, both verdict outputs), machine paths redacted.
• Production swap, in lockstep (enforced by tests/test_openai_review.py::TestWorkflowCodexActionContract): CI workflow model: gpt-6.1-sol, openai_review.DEFAULT_MODEL, plan-review CODEX_MODEL (carried over from the code-review campaign, not re-run through plan-review-eval; TODO.md row), plan-review CODEX_TIMEOUT_S raised 2400 -> 3600 to equal the harness CALL_TIMEOUT_S (test pins the equality), prose pins in CLAUDE.md / SKILL.md / the local-review command doc.
• Harness: arm F registered with the contrast (B, F) (REGISTERED_CONTRASTS); campaign 1's rule archived as campaigns/2026-10-gpt6-DECISION_RULE.md; run --configs defaults to the last registered shape; F takes role=control (B kept as previous production); the candidate-exposure gate exempts the control arm so bare smoke keeps working after a swap; PRICING gains gpt-6.1-sol; the DEFERRED gpt-6-sol cost-fallback row is retired (codex-cli 0.162.1's ChatGPT-account login accepts the gpt-6 Sol tier; campaign 2 was that evaluation).
• This PR's own CI review is the end-to-end canary of gpt-6.1-sol on codex-action + CLI 0.162.1; a rejection by the action is to be surfaced, never worked around.

Methodology references (required if estimator / math changes)

• Method name(s): N/A - no methodology changes (review infrastructure and evaluation records only)
• Paper / source link(s): N/A
• Any intentional deviations from the source (and why): None

Validation

• Tests added/updated: tests/test_evals_runtime.py (B,F shape, exposure refusals, control-arm exemption, control id), tests/test_reviewer_verdict.py (inverted and double-candidate contrast refusals), tests/test_openai_review.py (production pin, gpt-6.1-sol pricing row), tests/test_plan_review_skill.py (CODEX_MODEL pin, CODEX_TIMEOUT_S == harness CALL_TIMEOUT_S). 900 passed on 3.14 for the touched suites; 451 passed + 1 pre-existing skip on the 3.9 leg.
• Backtest / simulation / notebook evidence (if applicable): the two campaign verdict files and grading records under tools/reviewer-eval/campaigns/.

Security / privacy

• Confirm no secrets/PII in this PR: Yes (committed grading records are path-redacted; grep -rnE '/Users/|/private/|/var/folders/|/tmp/|~/|\$HOME' tools/reviewer-eval/campaigns/ is empty; audit command lines are stored as sha256 only)

Changelog

• changelog.d/ fragment added (or N/A - no user-visible change): Yes (changelog.d/20261017-reviewer-eval-gpt61sol-prereg.md, Internal)

[github-actions]

Overall assessment

✅ Looks good — no unmitigated P0 or P1 findings identified.

Executive summary

• Changes affect review infrastructure; no causal estimator, weighting, variance, or inference implementation changes.
• Production model pins, control selection, and timeout propagation are consistent.
• Both campaign verdicts reproduce from committed grades and corpus metadata; all 16 recorded file hashes match.
• Plan-review revalidation is properly tracked. Pytest was unavailable locally.

Methodology

No findings. The diff does not change methods governed by docs/methodology/REGISTRY.md; estimator paper cross-checks are therefore not applicable.

Code Quality

No findings. Read-only checks passed for ten exposure-gate configurations and three invalid contrast combinations. Promoting F preserves production smoke runs while retaining restrictions on E.

Performance

No findings. The longer plan-review timeout remains bounded and agrees with the harness ceiling. Committed latency figures support the reported improvement; live performance was not rerun.

Maintainability

No findings. Contract tests cover alignment among the workflow, local reviewer, plan reviewer, and production control arm.

Tech Debt

• P3 — Informational: plan-review validation is tracked.
  Location: TODO.md:L89; .claude/skills/plan-review/codex_review.py:L22-L34.
  Impact: The new model’s performance on plan review remains unmeasured; this limitation is explicitly disclosed.
  Concrete fix: Complete the tracked campaign: update CLI pins, add the new model arm, and compare against the previous baseline. No merge-blocking action required.

Security

No findings. Scoped scans found no secret patterns or unredacted machine paths in the added campaign records.

Documentation/Tests

No additional findings. The required changelog fragment is present, and the archived decision rule matches its original Git blob.

Local validation covered verdict consistency, hashes, gates, defaults, and pin alignment. Pytest could not run because it is not installed. Official pricing and live Codex availability could not be independently verified with network access unavailable.