chore: refresh codex-gpt-5.6 prometheus-metrics runs at medium reasoning - #160
Open
barryroodt wants to merge 2 commits into
Open
chore: refresh codex-gpt-5.6 prometheus-metrics runs at medium reasoning#160barryroodt wants to merge 2 commits into
barryroodt wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Collaborator
|
>the two cannot share a matrix |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the
describes each experiment the same way on every one of its runsfailure ineval-results.test.ts, which fails onmaintoday.#142 moved GPT-5.6 Sol to
reasoningEffort: 'medium', but thedeploy-database-001-prometheus-metricsruns forcodex-gpt-5.6andcodex-gpt-5.6-no-skillswere never re-run, so each experiment carried 18 runs atmediumand 1 atlow. SincegetExperimentDisplaytakes the first matching run, that disagreement can flip a model's label depending on sort order.Refreshed both pairs through
eval-refresh.yml(runs 30806941644 + 30807365508), one experiment at a time because the harness ANDs--experimentwith--experiment-suite, so the two cannot share a matrix.Only the two target rows changed, 190 rows in and 190 out. Scores held at 3/3 for both, so this is metadata and transcript churn rather than a scoring change.
pnpm --filter @supabase-evals/web testgoes 70 passed + 1 failed to 71 passed.Worth noting this comes back on any future
reasoningEffortchange that skips a re-run. Making the label read from the live experiment config instead of the first run would close it off for good, though that is a bigger call about what a historical run should report.