test(ci): dispatch the generalized plugin eval workflow once and check its summary #9713

Activity

1. kyle-sexton

   added

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 24, 2026
2. kyle-sexton

   marked this as blocked on Oct 24, 2026

   • feat(ci): validate, gate and summarize dispatched plugin eval runs #9704
3. kyle-sexton

   added a parent issue on Oct 24, 2026

   • feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405
4. kyle-sexton

   marked this as blocking on Oct 24, 2026

   • feat(ci): opt-in PR label trigger for the changed plugin's eval suite #9705
5. github-actions commented on Oct 24, 2026

   

   github-actionsBot

   on Oct 24, 2026 – with GitHub Actions

   Last edited by github-actions

   Contributor

   More actions

   Proposed work class: none

   Size: Small
   Complexity: Low
   Uncertainty: Known
   Flags: none
   Blocked by: none

   Understood: dispatch the generalized eval workflow once on a plugin with deterministic graders, read the job summary (validate step ran, a delta or NOT COMPARABLE per case, a health line per case), and report the result on #6405. The re-triage comment says the blocker #9704 is closed and the dispatch does not bill. This is a host step that needs a session with gh, so an unattended lane run cannot do it. Nothing is missing from the issue.

   _{Automated triage. A maintainer reviews it.}
6. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   on Oct 24, 2026
7. github-actions

   added the

   Task

   issue type on Oct 24, 2026
8. melodic-automation-lanes

   added

   status: needs-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

   on Oct 24, 2026
9. melodic-automation-lanes

   marked this as blocking on Oct 26, 2026

   • feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405
10. kyle-sexton commented on Oct 29, 2026

    

    kyle-sexton

    on Oct 29, 2026

    MemberAuthor

    More actions

    Re-triage. This is a host step, not an owner decision. Blocker #9704 is closed as completed, and the dispatch does not bill.

    • The workflow is eval-browser-dispatch.yml from feat(ci): run plugin eval suites from dispatch and an opt-in PR label #9835. It runs on ubuntu-24.04, a standard hosted runner in a public repository, with MAINTENANCE_CLAUDE_CODE_OAUTH_TOKEN, a subscription token (.github/workflows/eval-browser-dispatch.yml:123 and :427 on main at 489fb55). Model usage under the subscriptions is not a cost under the owner's 2026-10-27 rule, and nothing else can bill.
    • Every run carries --max-cost-usd (eval-browser-dispatch.yml:20), so the dispatch needs no extra input for it.

    Host step: a session with gh dispatches the workflow once on main. Use a plugin target whose suite has deterministic graders, set deterministic-only: true, and keep runs: 1. Then it reads the job summary and checks three things: the validate step ran, each case shows a delta or NOT COMPARABLE, and each case has its health line. Last, it comments on #6405 with the run link and whether the summary matched, then closes this issue.
11. kyle-sexton

    added

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

    and removed

    status: needs-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    on Oct 29, 2026
12. github-actions

    added

    needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

    and removed

    needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

    on Oct 29, 2026
13. melodic-automation-lanes

    added

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

    and removed

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

    on Oct 29, 2026
14. melodic-automation-lanes commented on Oct 29, 2026

    

    melodic-automation-lanesBot

    on Oct 29, 2026

    Contributor

    More actions

    Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
15. melodic-automation-lanes commented on Oct 29, 2026

    

    melodic-automation-lanesBot

    on Oct 29, 2026

    Contributor

    More actions

    The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.
16. kyle-sexton commented on Oct 30, 2026

    

    kyle-sexton

    on Oct 30, 2026

    MemberAuthor

    More actions

    Host run. Dispatched eval-browser-dispatch.yml once on main against plugins/fleet (deterministic graders only, 1 run per arm) and checked its job summary against this issue's three checks.

    What ran

    • Billing check, before dispatch: the job runs on ubuntu-24.04, a standard GitHub-hosted runner, in this public repository (.github/workflows/eval-browser-dispatch.yml:123). The model calls use MAINTENANCE_CLAUDE_CODE_OAUTH_TOKEN, a subscription token (:528). No larger runner, Azure or other billing account is involved.
    • Dispatch: gh workflow run eval-browser-dispatch.yml --ref main -f target=plugins/fleet -f runs=1 -f deterministic-only=true. The other inputs kept their defaults.
    • Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41055400082 on commit f578c51, Claude Code 2.1.314 on the runner, about 4 minutes. The eval ran in 1.5 minutes with --max-cost-usd 5, and the report's list-price estimate was 0.76 USD; under the subscription token that is not a charge.
    • Conclusion: failure, because one case scored below the threshold. The workflow is built to fail the job when any case fails (header comment, line 20), so the failure does not show a workflow defect.

    Against each check

    1. Validate step ran: pass. The log shows == validate plugins/fleet/evals and then the deterministic split (kept holds all 3 cases, dropped none). That split prints only after validate-cases.py exits 0.

    2. A delta or NOT COMPARABLE per case: pass. The table shows a delta for every case:

       Case	With	Without	Delta	Verdict
       fr-decline-ssh-allow-rule	1.00	0.75	+0.25	PASS
       fr-expired-oauth-login	1.00	0.00	+1.00	PASS
       fr-git-bash-publickey	0.50	0.50	+0.00	FAIL (threshold 1.00)

       The table comes from running summarize.mjs at the run's commit over the run's own uploaded results.json.

    3. A health line per case: present, so pass. Each of the 3 cases has its health line. What each line says (whether the skill loaded, how many runs called it) is UNVERIFIED from here: it is read from traces that stayed on the runner, and the REST API does not return the rendered job summary. Check it on the run page. The traces' directories were kept and only their home/ and tmp/ were sealed, so the summarize step most likely read them.

    Found (out of scope)

    • fr-git-bash-publickey in the fleet suite scores 0.50 both with and without the plugin. On this case the skill adds nothing, and the case fails the threshold. This is a suite or skill quality item for the fleet plugin, not a workflow defect.
    • One thing would make this a workflow defect: if the run page's health lines read "no kept trace", the --keep-temp traces are unreadable to the summarize step. Nothing seen here shows that.

    No PR opened: the run exposed no workflow defect in this issue's scope.

    Remaining step: comment for #6405

    The done-when needs this comment on #6405. This run did not post it:

    Dispatched the generalized plugin eval workflow once on main (plugins/fleet, deterministic-only, runs 1): https://github.com/melodic-software/claude-code-plugins/actions/runs/41055400082. Summary matched: the validate step ran; each of the 3 cases shows a delta (+0.25, +1.00, +0.00); each case has its health line. The job concluded failure only because fr-git-bash-publickey scored 0.50 in both arms against a 1.00 threshold, which is the fleet suite's own result.

    Target labels: add agent-ready; remove host-run.
17. kyle-sexton

    added

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    and removed

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

    on Oct 30, 2026
18. melodic-automation-lanes

    added

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

    and removed

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    on Oct 30, 2026
19. melodic-automation-lanes commented on Oct 30, 2026

    

    melodic-automation-lanesBot

    on Oct 30, 2026

    Contributor

    More actions

    The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
    It added host-run and removed agent-ready: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    host-only: the one remaining done-when item is a GitHub write this lane cannot make: post the drafted comment on ＃6405 linking run https: //github.com/melodic-software/claude-code-plugins/actions/runs/41055400082 and stating the summary matched, then close ＃9713. The host run itself (dispatch of eval-browser-dispatch.yml on plugins/fleet, deterministic-only, runs 1, and the three summary checks) is already done and recorded in the 2026-10-30 host-run comment on ＃9713, whose 'Remaining step: comment for ＃6405' section holds the ready-to-post text.

    What the run says it did not do:

    • Post the drafted comment on ＃6405 with the run link and the summary-matched statement, then close ＃9713. This lane cannot write to GitHub.
    • tidy: skipped because the change is empty
    • review, verify and pull-request prep: skipped because no diff exists to review, build or test
    • The host-run comment says the per-case health-line text is UNVERIFIED (it is rendered only on the run page). Check it on the run page before posting to ＃6405.

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41056394821

    To hand it back to the lane, remove host-run and add agent-ready again.
20. kyle-sexton commented on Oct 31, 2026

    

    kyle-sexton

    on Oct 31, 2026

    MemberAuthor

    More actions

    Done: the run comment is on #6405: #6405 (comment) (host lane 4). This was generated by AI.
21. kyle-sexton

    closed this as completedon Oct 31, 2026