feat(ci): validate, gate and summarize dispatched plugin eval runs #9704

Activity

1. kyle-sexton

   marked this as blocking on Oct 24, 2026

   • test(ci): dispatch the generalized plugin eval workflow once and check its summary #9713
2. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 24, 2026
3. kyle-sexton

   added a parent issue on Oct 24, 2026

   • feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405
4. kyle-sexton

   marked this as blocking on Oct 24, 2026

   • feat(ci): opt-in PR label trigger for the changed plugin's eval suite #9705
5. github-actions commented on Oct 24, 2026

   

   github-actionsBot

   on Oct 24, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Known
   Flags: none

   Understood: extend the existing dispatched eval workflow so it validates the suite before any model call, offers a deterministic-graders-only input, gates JSON results so partial, skippedPaidGraders, errored or delta-less cases read NOT COMPARABLE, and writes a with-versus-without summary with a per-case health line. The ADR 0049 hardening stays as it is. The scope and done-when items are clear. I found no likely duplicates; this is a slice of #6405.

   _{Automated triage, applied by the lanes App.}
6. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 24, 2026
7. github-actions

   added the

   Feature

   issue type on Oct 24, 2026
8. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   and removed

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   on Oct 24, 2026
9. melodic-automation-lanes commented on Oct 24, 2026

   

   melodic-automation-lanesBot

   on Oct 24, 2026

   Contributor

   More actions

   The work-resolve-issue lane stopped partway through this issue and needs a person.

   Why, in the run's own words:

   The run reported done, but apply-issue-patch did not commit its patch (patch-protected-path): the patch changes a path the issue lane may not push: a workflow, a CODEOWNERS file, submodule config or a settings file. The run's log names the path or check.

   Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40112622739

   To retry, remove needs-human and add agent-ready again.
10. kyle-sexton

    marked this as blocking on Oct 26, 2026

    • Concise skills rollout: mcp-tools (2 skills, 2 model-invoked) #6585
11. 21 remaining items

    Load more
12. kyle-sexton

    marked this as blocking 14 issues on Oct 27, 2026

    • Concise skills rollout: attribution (2 skills, 1 model-invoked) #6599
    • Concise skills rollout: codebase-health (2 skills, 1 model-invoked) #6600
    • Concise skills rollout: computer-use (2 skills, 1 model-invoked) #6601
    • Concise skills rollout: coupling (1 skills, 1 model-invoked) #6602
    • Concise skills rollout: debugging (1 skills, 1 model-invoked) #6603
    • Concise skills rollout: domain-driven-design (1 skills, 1 model-invoked) #6604
    • Concise skills rollout: dometrain (3 skills, 1 model-invoked) #6605
    • Concise skills rollout: firecrawl (3 skills, 1 model-invoked) #6606
    • Concise skills rollout: gaming (2 skills, 1 model-invoked) #6608
    • Concise skills rollout: naming (1 skills, 1 model-invoked) #6611
    • Concise skills rollout: playgrounds (1 skills, 1 model-invoked) #6612
    • Concise skills rollout: repo-fleet-hygiene (4 skills, 1 model-invoked) #6613
    • Concise skills rollout: tdd (1 skills, 1 model-invoked) #6615
    • Concise skills rollout: x (1 skills, 1 model-invoked) #6618
13. kyle-sexton

    closed this as completedin #9835on Oct 28, 2026