Repository navigation
feat(ci): opt-in PR label trigger for the changed plugin's eval suite #9705
Description
Part of #6405. Add the PR label trigger the owner's 2026-10-18 decision on #6405 deferred until #6962 answered (#6962 is closed).
Scope
- On a same-repository, non-draft PR that carries an opt-in label and touches
plugins/<p>/skills/**,plugins/<p>/agents/**orplugins/<p>/evals/**, run only the changed plugin's suite through the generalized eval workflow, deterministic graders,--max-cost-usdset. - Advisory only: never a required check, never a
ci-statusinput, no write token in any model step (ADR 0049). - Credential: the role's subscription token, as the dispatch job uses. Probe: does a self-written claude -p step in GitHub Actions draw the linked org's API credits? #6962 showed a linked-org key draws credits for
claude -p; itsclaude plugin evalhalf is unprobed, so do not switch credentials here. - Create the label if it does not exist and document it where the other lane labels are documented.
Done when
-
actionlintexits 0 on every changed workflow file. - A test of the path filter shows a PR touching
plugins/testing/evals/xselectsplugins/testingonly, and one touching onlydocs/selects nothing. -
scripts/check-affected.sh --uncommittedpasses.
Activity
github-actions commentedon Oct 24, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Understood: add an opt-in, advisory PR label that runs only the changed plugin's eval suite (deterministic graders, cost-capped, subscription token, no write token) for same-repo non-draft PRs touching skills, agents or evals. It also needs the label created and documented. The scope and done-when items are clear and checkable. I found no likely duplicates beyond the parent #6405.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 24, 2026
Not admitted to the issue lane yet: blocker #9704 is not closed as completed by a listed human or a merge; blocker #9713 is not closed as completed by a listed human or a merge.