feat(evals): run plugin eval suites in a CI lane and hill-climb in the background

Problem

Every claude plugin eval run in this repository is started by hand in the operator's terminal and paid for there. No workflow under .github/workflows/ runs a plugin eval suite; pr-require-checks.yml only lints skill-creator evals/evals.json files and fixtures. Sessions cannot run the suites either: a worktree-isolated session refuses any command containing eval (#5696, #6374), and a background run dies with its launcher (#6333). So a change to a skill that has a suite (performance, improvement, source-control, code-tidying, education, discovery, planning, evals) merges without its with-versus-without delta unless someone remembers to run it.

Evidence from one session (2026-10-04)

These are the failure modes an automated lane has to handle, not only the reason to build one.

All runs were manual. The session's worktree-isolation guard refused claude plugin eval, so every run was pasted into the user's terminal and paid for there.
Several suites at first measured nothing.
- The skill never loaded in some cases: Bash was denied during the fanout skill's shell preprocessing, so the with-arm ran without the skill body.
- The skill-fired grader misread slash-invoked runs.
- Firing varied widely between runs: implement-dispatch fired 6/12, then 0/3, then 6/6.
Judge graders needed calibration. One judge scored 89%, then 80% agreement, and was replaced by regex graders.
Confirming runs found real effects. Four changes each moved a case from 0/3 to 3/3.

A lane that reports a delta without checking that the skill loaded, that the grader reads the run shape correctly, and that the run count is enough for the observed variance will post confident numbers that mean nothing.

Can it run in Actions today

Yes, as a plain workflow step. https://code.claude.com/docs/en/plugin-evals#run-evals-in-ci (fetched 2026-10-04) documents a CI invocation with --trust-plugin, --json, --threshold, pinned --model and --judge-model, --no-publish and --max-cost-usd, and says a runner needs a Claude Code install and credentials in the environment. Each case runs as its own non-interactive session with only the granted tools, so no --permission-mode applies to the eval step itself. Cases that grant Bash, Write or Edit need the OS sandbox: on Linux, bubblewrap and socat (Grant tools). Whether those run on a GitHub-hosted Ubuntu runner (unprivileged user namespaces) is not verified.

The auto-fix and hill-climb parts would be model steps, which https://code.claude.com/docs/en/github-actions runs through claude-code-action with a prompt and claude_args. Those fall under ADR 0049.

Our runner notes: /evals:plugin-eval CI section and its reference/ci.md (exit-code parser, partial and skippedPaidGraders handling, the "currently unavailable" server-side switch). For spend, see https://code.claude.com/docs/en/costs and https://platform.claude.com/docs/en/about-claude/pricing; this issue states no figures.

Proposed lane

Trigger: pull_request on same-repository branches that touch plugins/<p>/skills/**, plugins/<p>/agents/** or plugins/<p>/evals/**, for plugins with a suite. Skip drafts, matching the other lanes. A workflow_dispatch input for a full sweep, and later a schedule for judge-graded cases.
Scope: per changed plugin, only that plugin's suite. No cross-plugin runs on a PR.
Authority it must not have:
- No merge, no ci-status write, no checks: write or workflows permission in any model step (ADR 0049 rule 3).
- Any fix, new case, or skill edit is pushed as a draft PR or a commit to a lane branch, never to the PR's branch without the author's opt-in and never to main.
- PR titles, bodies, comments and eval transcripts reaching a model step are framed as data per docs/conventions/untrusted-content/README.md; trusted-actor and same-repo gates per ADR 0049 rules 1 and 2; the org kill switch per rule 4.

Smallest first slice

On a PR that touches plugins/<p>/skills, agents or evals:

Run /evals:validate (validate-cases.py) on that plugin's suite. No model call.
Run that plugin's suite with deterministic graders only, --max-cost-usd set, both models pinned, --json.
Post the with-versus-without table as an advisory check or job summary, gated through the reference/ci.md parser so a partial, skippedPaidGraders, errored or delta-less case reads NOT COMPARABLE instead of a score.
Add one health line per case: did the skill load in the with-arm (from the trace), and how many runs fired.

No auto-fix, no new cases, no hill-climb in this slice. Those come after the advisory numbers have been read on real PRs for a while.

Later slices (not in scope yet)

Author or repair cases for skills whose suite measured nothing (skill not loaded, grader misreading the run shape), as draft PRs.
Judge calibration: run a judge against its must-pass and must-fail samples and refuse to use it below a set agreement.
Background hill-climb on a schedule: propose skill edits, keep only those that beat the baseline over enough runs, open each as a draft PR.

Open questions

Sandbox on hosted runners. Do bubblewrap and socat work on ubuntu-latest, or do Bash-granting cases need a container or --ablation none read-only suites only? Settled by one dispatch run of a Bash-granting case.
Spend control. Which credential (API key vs subscription token, per the github-actions page), what --max-cost-usd per PR, and whether a label opts a PR in rather than running on every push. Needs the operator's decision.
Flakes. With firing at 6/12 then 0/3 on the same skill, what run count makes a PR delta trustworthy, and does the lane report an interval instead of a point? /evals:plugin-eval reading-results guidance is the starting point.
Judge calibration gating. Should judge-graded cases stay out of the PR lane until each judge passes its calibration samples (feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879 lists known gaps)?
Server-side switch. plugin eval is currently unavailable can fire with no local fix, so the lane must stay advisory and never become a required check.

Worktree-isolated sessions refuse claude plugin eval (guard reads 'eval' as shell eval) #5696, docs(evals): document that plugin eval refuses a plugin inside a repo with many linked worktrees #6374, docs(evals): plugin-eval runs started in the background die when a headless launcher exits #6333: why sessions cannot run suites today.
feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879: judge calibration gaps and converting other plugins' evals.
ADR 0049 (docs/adr/0049-run-ci-lanes-on-github-hosted-runners-under-trigger-and-token-hardening.md): hosted-runner lane hardening.

Proposed work class: work-class: structural
Basis: automated intake triage; the lanes App applied this class.

Size: Large
Complexity: High
Uncertainty: Knowable
Flags: none
Blocked by: none

Decision recorded.

Question: No CI lane runs claude plugin eval. The question is which credential and spend control it uses. Options: (a) linked-org API key per ADR 0062, opt-in by label, fixed --max-cost-usd; (b) the org OAuth token on every qualifying push; (c) workflow_dispatch only until #6962 answers.

Owner's answer: (c), on the role's subscription token, with deterministic graders and --max-cost-usd. Add a label trigger once #6962 answers. [quote truncated; the full decision is at the link]

#6405 (comment)

This decision settles the credential and spend question. The sandbox, flake and server-side-switch questions are answered in the earlier re-triage comment. The issue reads as a Large umbrella: a generic eval lane in CI, with the first slice done and later slices (case repair, judge calibration, hill-climb) explicitly out of scope. Per the re-triage, it stays open as a tracker and closes when its last slice closes. No duplicates found.

_{Automated triage, applied by the lanes App.}

The credential question here changes: Max and Team plans now carry monthly API credits in the linked Console org (help article 17154008). Credits cover API-key traffic only; runs started by the Claude Code GitHub Action are excluded even with -p. Whether a self-written claude -p or claude plugin eval step with an org key draws credits is the probe in #6962. The credential choice for CI lanes is tracked on the capacity map #6958 (#6968), and ADR 0037 is being amended to allow the operator's own org keys (#6961).

Decision recorded.

Question: No CI lane runs claude plugin eval. The question is which credential and spend control it uses. Options: (a) linked-org API key per ADR 0062, opt-in by label, fixed --max-cost-usd; (b) the org OAuth token on every qualifying push; (c) workflow_dispatch only until #6962 answers.

Owner's answer: (c), on the role's subscription token, with deterministic graders and --max-cost-usd. Add a label trigger once #6962 answers. The 2026-10-16 amendment of ADR 0062 makes a linked-org key a fallback after a subscription limit, not the lane's main credential, and says that for CI the fallback waits on probe #6962, which is still open. The docs say any normal credential works and that Bash-granting suites on Linux need bubblewrap and socat. Basis: docs/adr/0062-*.md:96-104; https://code.claude.com/docs/en/plugin-evals ("on Linux, install bubblewrap and socat first"; "credentials in the environment such as ANTHROPIC_API_KEY").

The owner accepted this recommendation as item D10 of a 21-decision batch in an attended backlog-shaping session on 2026-10-18. The batch question: "Accept all 21 batch 7 decisions, with my change to D13 (#6008 goes to the lane, but you merge its PR because it edits AGENTS.md)? I post a 'Decision recorded.' comment on each issue." The owner answered: "Accept, with my D13 change (Recommended)".

Posted by AI from the owner's answer in that session.

Split into slices by the owner's HITL session (2026-10-24, C12 and Q93): this waited only on a gate the owner removed.

feat(ci): validate, gate and summarize dispatched plugin eval runs #9704: extend eval-browser-dispatch.yml with validate, deterministic-only runs, NOT COMPARABLE gating and a per-case health line (lane).
test(ci): dispatch the generalized plugin eval workflow once and check its summary #9713: dispatch it once and check the summary (needs-human: a paid run a lane cannot perform), blocked by feat(ci): validate, gate and summarize dispatched plugin eval runs #9704.
feat(ci): opt-in PR label trigger for the changed plugin's eval suite #9705: opt-in PR label trigger (lane), blocked by feat(ci): validate, gate and summarize dispatched plugin eval runs #9704 and test(ci): dispatch the generalized plugin eval workflow once and check its summary #9713. Probe: does a self-written claude -p step in GitHub Actions draw the linked org's API credits? #6962 is closed, which was this step's stated wait.

The later slices (case repair, judge calibration, hill-climb) stay out of scope as the body says. Related: #6649.

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

The work-resolve-issue lane stopped on this issue (split-required): the issue's Size is too large for one run, and intake's split (owner Q93) files its sub-issues.
Its Size is too large for one run: intake splits it into sub-issues (owner Q93).
It removed agent-ready and status: ready, so the hourly triage sweep sends it back through intake triage, and it waits on no decision from a person.

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41212699964

Cordoned under the After zero milestone, still open. Issue Triage session reading, at main (fetched 2026-11-01).

The three sub-issues (#9704, #9705, #9713) shipped the first slice's lane, but two parts of this issue are not done:

The background hill-climb in the title, and case repair, both listed under "Later slices (not in scope yet)". Neither has its own issue. Judge calibration shipped under feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879.
Slice item 4, one health line per case, is built (.github/scripts/plugin-eval/summarize.mjs:142) but not shown on a real run. The one dispatch, https://github.com/melodic-software/claude-code-plugins/actions/runs/41055400082, reported a FAIL or NOT COMPARABLE result (its summary step exits 1 for either, summarize.mjs:22-23), and the health lines live in the job summary, which was not read.

Admission's refusal ("every sub-issue is closed, but it is still open") is correct here: work no sub-issue covers remains. Before this leaves After zero, the next pass should read that run's job summary and split the hill-climb and case repair into - [ ] items or sub-issues.

Host run, post-merge. This dispatched one eval run on main after #12084 merged. The job log now shows the health lines, and the with-arm health line reads skill loaded: yes.

Dispatch: gh workflow run eval-browser-dispatch.yml --ref main -f target=plugins/evals -f case=pe-ci-pin-full-ids -f runs=1 -f deterministic-only=true, as run 41353703294. It concluded success.

Health lines from the Summarize the results job log, verbatim:

Health, with-arm runs:
- pe-ci-pin-full-ids: skill loaded: yes; fired in 1 of 1 runs with a kept trace (1 runs)

Against test(ci): show the plugin eval health line reading skill loaded on a dispatched run #12032's done-when:
1. A dispatched run's health line is readable by an agent: met, from the job log (ci(plugin-eval): write each suite's summary to the job log and results artifact (#12032) #12084).
2. It reads skill loaded: yes on a dispatched run: met.
3. Fix summarize.mjs trace resolution if a line still reads unknown: not triggered, because no line reads unknown.

This answers #6405's first-slice item 4: the line that read skill loaded: unknown on run 41055400082 now reads yes.

Target labels: add none; remove host-run.

Case survey for #12043. Seven deterministic dispatches on main (improvement, discovery, performance, evals, education, planning, source-control p*) gave 88 health lines. None reads skill loaded: no. 59 read "fired in 0":

48 slash-invoked and 3 agent-dispatch cases read 0 because summarize.mjs counted only Skill tool calls. The health-line fix is on branch host-run/12043-eval-skill-load; its PR follows.
4 are negative controls that should not fire: evals control-deepeval-geval, control-no-trigger and control-promptfoo-json, and performance climb-routes-eval-score-to-hillclimb.
4 natural-language cases did not fire: evals de-criteria-first (also 0 of 3 locally), and source-control pull-request-create-refs-writes-remaining, pull-request-dirty-worktree-stashes-before-checkout and pull-request-merge-holds-on-unclassified-ci (0 of 1 each, not yet repeated).

Not surveyed yet: code-tidying, source-control outside p*, and planning past the cost ceiling.

This was generated by AI.

Navigation Menu

Repository navigation

feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405

Problem

Evidence from one session (2026-10-04)

Can it run in Actions today

Proposed lane

Smallest first slice

Later slices (not in scope yet)

Open questions

Related

18 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405

Description

Problem

Evidence from one session (2026-10-04)

Can it run in Actions today

Proposed lane

Smallest first slice

Later slices (not in scope yet)

Open questions

Related

Activity

github-actions commented on Oct 5, 2026

kyle-sexton commented on Oct 11, 2026

kyle-sexton commented on Oct 18, 2026

kyle-sexton commented on Oct 24, 2026

github-actions commented on Oct 25, 2026

18 remaining items

melodic-automation-lanes commented on Oct 31, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions