feat(evals): add blinded paired experiment - #38
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d140f1d03
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| maxUsd: options.maxUsd, | ||
| unknownCostPolicy: "stop", | ||
| maxOutputTokens: 200_000, | ||
| maxWallClockMs: 3_600_000, |
There was a problem hiding this comment.
For any successful run large enough to clear the fixed power gate, the runner needs at least 265 complete pairs (530 sequential arm attempts), while each successful runPrompt/runCommand waits through the harness's default 25-second quiet period. That alone requires over 3.6 hours, so this hard-coded one-hour ceiling necessarily stops the campaign before it becomes claim-eligible. Derive the wall-clock allowance from the planned pair count or make it configurable.
Useful? React with 👍 / 👎.
| const scansValid = input.scans.every( | ||
| (scan) => | ||
| TranscriptScanSchema.safeParse(scan).success && | ||
| scan.sha256 === scanSha256(scan), |
There was a problem hiding this comment.
When scan records are reconstructed from persisted or external JSON, this validation accepts any well-shaped record whose hash merely authenticates its own fields; it does not require the current scanner version or enforce that passed agrees with findings. A caller can therefore supply a matching transcript digest with passed: true and even nonempty findings, causing a contaminated transcript to remain claim-eligible. Validate the scanner version and pass/finding invariant, or rescan the persisted transcript bytes at this boundary.
Useful? React with 👍 / 👎.
| if ( | ||
| record.reportId !== report.reportId || | ||
| record.planSha256 !== report.plan.planSha256 || | ||
| record.reportSha256 !== pairedReportSha256(report) || | ||
| record.allocationCommitmentSha256 !== report.allocationCommitmentSha256 || | ||
| record.allocationCommitmentSha256 !== allocationCommitmentSha256(secret) |
There was a problem hiding this comment.
When an exported ReportStore.writeAllocation() caller supplies an allocation directly instead of obtaining it from revealPairedAnalysis, these checks prove only that the secret matches the previously committed hash. They do not verify that its blocks and tokens match the plan or that each attempt's ordinary/plugin artifact matches the revealed arm, even though validateRevealBindings performs those checks elsewhere. This permits the durable store to publish an allocation that contradicts the finalized report, so the store should invoke the same reveal-binding validation before writing.
Useful? React with 👍 / 👎.
2 participants
Summary
Verification
Stop gate
The pilot is advisory and makes no product-value claim. Both arms tied, Flow tool presence is visibly unblinded, and 1 planned pair misses the conservative 265-pair power requirement. The decision is INCONCLUSIVE.