feat(evals): add stable report trends - #39
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bbdc68a45
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| async grade(project) { | ||
| return hiddenBunCheck( | ||
| project, | ||
| 'import { formatCents, orderTotal, renderOrderSummary, summarizeOrders } from "./src/index.ts"; const lines = [{ id: "A", unitCents: 125, quantity: 2 }, { id: "A", unitCents: 50, quantity: 1 }, { id: "B", unitCents: 201, quantity: 1 }]; const value = summarizeOrders(lines); if (orderTotal(lines[0]) !== 250 || formatCents(501) !== "501 cents" || JSON.stringify(value) !== JSON.stringify({ lineCount: 3, orderCount: 2, totalCents: 501, averageOrderCents: 250 }) || renderOrderSummary(lines) !== "2 orders / 501 cents") process.exit(1); if (JSON.stringify(summarizeOrders([])) !== JSON.stringify({ lineCount: 0, orderCount: 0, totalCents: 0, averageOrderCents: 0 })) process.exit(1);', |
There was a problem hiding this comment.
When a correct implementation returns the required summary properties in a different insertion order, JSON.stringify(value) differs from this literal even though every field has the expected value, causing the hidden grader to record a false failure and corrupt benchmark results. The markdown-link-report grader repeats the same top-level object comparison; compare the individual fields or use a structural deep-equality assertion in both graders.
Useful? React with 👍 / 👎.
| actors: sortedByCanonical( | ||
| attempt.actors.map((actor) => ({ | ||
| role: actor.role, | ||
| requestedModel: actor.requestedModel, | ||
| })), |
There was a problem hiding this comment.
When the same requested route resolves to a different actual model or revision between runs, these rows remain identical because only requestedModel is retained, so compareReports declares the reports compatible and can attribute a model-driven outcome change to the artifact. Since each attempt records actualModel specifically as provenance, include that identity in the actor semantics (or reject comparisons when it is unavailable).
Useful? React with 👍 / 👎.
2 participants
Summary
Verification
Stop gate
All benchmark cases remain report-only. No legacy report is backfilled and no new release regression threshold is claimed.