Repository navigation
feat(evals): background hill-climb that opens draft PRs for skill edits beating the baseline #12035
Description
Refs: #6405
Outcome
#6405's title asks for a background hill-climb: "propose skill edits, keep only those that beat the baseline over enough runs, open each as a draft PR". No issue tracks it, and the lane that runs suites (.github/workflows/eval-browser-dispatch.yml) only measures. This slice builds a scheduled or dispatched run that proposes one skill edit, measures it against the current skill, and opens a draft PR only when the edit wins beyond run-to-run variance.
The design is open, so this slice starts with a short plan comment on #6405 covering the run count, the win margin, which suites are eligible, and the authority limits below. It is labeled work-class: structural to match the parent.
Done when
- A plan comment on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 names the trigger, the eligible suites, the run count and win margin with their
Basis:, and the per-run--max-cost-usd. - A dispatched run on one suite either opens one draft PR whose body links the with-versus-without result for the edit and for the baseline, or exits with no PR and says why in the job summary.
- The model step holds no merge right, no
ci-statuswrite, and nochecks: writeorworkflowspermission, and it pushes only to a lane branch, never tomain(ADR 0049,docs/adr/0049-run-ci-lanes-on-github-hosted-runners-under-trigger-and-token-hardening.md). - Eval transcripts that reach the model step are framed as data per
docs/contracts/untrusted-content/README.md. - A case whose health line reads
skill loaded: nois excluded from the win test.
Out of scope
- Auto-merging any hill-climb PR.
- Case repair and the health line, which are their own slices of feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.
Activity
github-actions commentedon Nov 1, 2026
Proposed work class: work-class: structural
Basis: automated intake triage; a maintainer confirms or changes it.
Size: Large
Complexity: High
Uncertainty: Knowable
Flags: none
Blocked by: none
I read this as a new scheduled or dispatched eval run, tracked under #6405. It would propose one skill edit and measure it against the current skill. It would open a draft PR only when the edit beats the baseline by more than run-to-run variance. The issue makes the first step a plan comment that fixes the run count, win margin, eligible suites and authority limits. That plan is part of the work, so I'm not treating it as an open question. The done-when items are specific. I found no open duplicate. Because the work touches CI permissions and eval transcripts that reach a model step, it likely needs splitting into smaller changes.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 1, 2026
Not admitted to the issue lane: Size unset is not one of small, medium; Uncertainty unset is not one of known, knowable. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).