feat(evals): repair or retire eval cases whose with-arm never loaded the skill

Blocked by: #12032

Outcome

#6405's "Later slices" list "author or repair cases for skills whose suite measured nothing (skill not loaded, grader misreading the run shape)". A case whose with-arm never loads the skill reports a delta that measures nothing. Once the health line answers on dispatched runs, each case it reports as skill loaded: no, or as firing in 0 runs, is either repaired so that the skill loads, or removed with a reason.

Done when

A comment on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 lists every case that a dispatched run's health line (.github/scripts/plugin-eval/summarize.mjs:140-142) reports as skill loaded: no, or as firing in 0 runs, and links the run.
For each listed case, a PR either repairs it so that the next dispatch reads skill loaded: yes, or removes it with the reason in the PR body. A repair might be a missing tool grant, a prompt that does not trigger the skill, or a grader that misreads a slash-invoked run.
Each repaired case passes /evals:validate (validate-cases.py) on its suite with no FAIL.
A re-dispatch after the PRs merge shows no listed case still reading no. It is linked on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.

Out of scope

Judge calibration (shipped under feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879).
The background hill-climb, which is its own slice of feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #12032

Understood: once the plugin eval health line reports on dispatched runs, list every case showing skill loaded: no or firing in 0 runs, then repair or remove each with a reason, validate repaired suites, and re-dispatch to confirm. The issue states it is blocked by #12032, which is still open. Related but not duplicate: #12000-range suite-specific issues such as the vision-case one (run-e2e never fires) cover single suites, while this one covers the fleet-wide sweep.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane yet: blocker #12032 is not closed by a listed human or a merge.

No description provided.

The work-resolve-issue lane stopped partway through this issue and needs a person.

Reason: security

Matched security word: token. A match holds the stop whether or not the text names a risk, so read the word in context below.

Why, in the run's own words:

host-only: the work starts from a list of cases that dispatched eval runs report as 'skill loaded: no' or as firing in 0 runs. Building that list means hand-dispatching .github/workflows/eval-browser-dispatch.yml (workflow_dispatch only, one plugin directory per run, paid model calls on the role's subscription token) for each plugin's suite, then reading each job log, which needs a sign-in. This lane can do neither, and it cannot post the required comment on ＃6405 or re-dispatch after the repairs merge. The one dispatched run since the blocker ＃12032 closed, 41353703294, reports one case, pe-ci-pin-full-ids: skill loaded: yes; fired in 1 of 1 runs. That leaves nothing to repair until someone runs the other suites.

What the run says it did not do:

Done-when 1: the ＃6405 comment listing every case that reads 'skill loaded: no' or fires in 0 runs. This needs hand-dispatched eval runs for each plugin suite, the ability to read their sign-in-gated logs, and a way to post on ＃6405. This lane has none of these.
Done-when 2 and 3: repairing or removing each listed case and passing validate-cases.py. Not started, because the case list from done-when 1 does not exist yet.
Done-when 4: a re-dispatch after the repairs merge. This needs a host run.
Tidy, review, verify and checks steps: skipped because the run stopped before any edit and the working tree is unchanged.

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41354400109

To retry, remove needs-human and add agent-ready again.

Claimed by a host-run session. This host is surveying the dispatched eval health lines (eval-browser-dispatch.yml, deterministic-only, runs 1, one plugin per run) for the suites #6405 names, then repairing or retiring each case that reads skill loaded: no or fires in 0 runs, on branch host-run/12043-eval-skill-load. A record follows here when the survey ends.

This was generated by AI.

Navigation Menu

Repository navigation

feat(evals): repair or retire eval cases whose with-arm never loaded the skill #12043

Outcome

Done when

Out of scope

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

feat(evals): repair or retire eval cases whose with-arm never loaded the skill #12043

Description

Outcome

Done when

Out of scope

Activity

github-actions commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions