feat(evals): add reviewer calibration campaign - #37
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5c6bc4faa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ...(analysis.defectCases < input.minimumCasesPerTruth || | ||
| analysis.cleanCases < input.minimumCasesPerTruth |
There was a problem hiding this comment.
When a plan schedules multiple repetitions of the same case/version, analysis.defectCases and analysis.cleanCases count every submitted attempt, so repeated runs of one defect and one clean fixture can satisfy minimumCasesPerTruth and artificially narrow the Wilson intervals. This can promote a reviewer without the preregistered number of independently human-labelled cases; enforce the floor using distinct case/version keys rather than observation counts.
Useful? React with 👍 / 👎.
| observedReviewerModels.every((observed) => | ||
| input.reviewerModels.some( | ||
| (expected) => | ||
| canonicalSha256("flow-reviewer-model-v1", observed) === | ||
| canonicalSha256("flow-reviewer-model-v1", expected), |
There was a problem hiding this comment.
When reviewerModels contains the observed model plus any additional untested model, this one-way subset check still passes and the promotion record claims calibration for every supplied model. This permits an unobserved model or family to inherit another model's results; compare the distinct observed and declared model sets in both directions before issuing the record.
Useful? React with 👍 / 👎.
| caseCatalogSha256: canonicalSha256( | ||
| "flow-reviewer-calibration-catalog-v1", | ||
| input.catalog, |
There was a problem hiding this comment.
When the caller supplies a different valid catalog from the one used to parse report, promotion can still succeed and this records the unrelated catalog's hash. ValidatedReport does not retain its parsing catalog, while the preceding checks compare the plan only with labels and the fixture registry, so the promotion record can misstate its frozen policy input; verify the supplied catalog against the report's planned cases and policies or preserve its digest at parsing time.
Useful? React with 👍 / 👎.
2 participants
Summary
Verification
Stop gate
Reviewer evidence remains report-only. The two-case pilot misses preregistered sample and confidence thresholds, and OpenCode 1.18.6 does not expose full reviewer identity. No release or product improvement claim is made.