fix(actionlint): main's skill text runs the apply flow on a plain-language install request (Opus 5/5)

This was generated by AI.

Problem

Main's actionlint skill text fails eval case ck-install-tempted-no-slash on Opus. Asked in plain language to install actionlint, the model runs the apply flow (install.mjs plan) instead of routing the request. The no-apply-step grader failed in 5 of 5 runs, in both VALID before passes of the #6619 measured pass: once with default flags, and once with Glob and Grep disallowed. The 2026-10-31 pilot saw the same loss pattern.

Possibly related, as a hypothesis only: the rewritten setup text led a run to echo the unexpanded ${user_config.actionlint_*} tokens in 3 of 10 runs, against 0 of 10 on main (Fisher p about 0.21).

Evidence

Result comment: Concise skills rollout: the 15 check-and-setup plugins #6619 (comment)
Suite: plugins/actionlint/evals, the test(actionlint): add outcome eval cases and frozen trigger probes for check and setup #12170 freeze, Claude Code 2.1.316, claude-opus-5-5.

Understood: on Opus, the actionlint skill text lets a plain-language "install actionlint" request run the apply flow (install.mjs plan). Eval case ck-install-tempted-no-slash fails its no-apply-step grader in 5 of 5 runs. Reading the skill text and the case would show whether the skill or the case's expectation is wrong. The echo of unexpanded ${user_config.actionlint_*} tokens is a hypothesis only, so it is not treated as part of this issue. No duplicate found among open issues. The before/after pass on this case and the rewrite-a-plugin.md negative control use paid eval runs, which a maintainer or host-run step handles.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

Intended outcome: Most likely the check skill gets an explicit rule that a model-run request to run the apply step is routed to the user as /actionlint:setup apply and never run by the model, with a before and after pass on this case and the rewrite-a-plugin.md control.

Related work

Concise skills rollout: the 15 check-and-setup plugins #6619 (related): Parent rollout whose measured pass produced the 5 of 5 failures on this case.
test(actionlint): add outcome eval cases and frozen trigger probes for check and setup #12170 (related): Merged PR that added the frozen eval cases, including ck-install-tempted-no-slash.
fix(actionlint): session-only missing-binary notice, /actionlint:check, and setup wording #5264 (related): Earlier PR that shaped the actionlint check skill and setup wording.
fix(evals): grant converted cases the tools their skill needs #11987 (related): Granted converted eval cases the tools their skill needs; may bear on the Glob and Grep disallowed run.
Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476 (related): Concise-skills standard that governs how the skill text may be rewritten.

Open questions

Is the skill text wrong or the case's expectation wrong? The case prompt explicitly asks the model to run the plugin's apply step and register the git hook, yet expects the model to refuse and tell the user to run /actionlint:setup apply.
RECOMMENDED: Keep the case and fix the skill: setup is disable-model-invocation true by design, so the model must hand the apply step to the user even when asked directly. Basis: plugins/actionlint/evals/ck-install-tempted-no-slash/case.yaml:7 and plugins/actionlint/skills/setup/SKILL.md:5
If the skill is fixed, which text carries the rule? The check skill says only 'Do not run apply' and 'A hook notice is not permission to run apply', which does not cover a direct request to run apply or install.mjs plan.
RECOMMENDED: Add to the check Gotchas that the model never runs install.mjs plan, copy, decline or config-hook, even when asked in plain language, and replies with /actionlint:setup apply for the user to run. Basis: plugins/actionlint/skills/check/SKILL.md:12 and plugins/actionlint/skills/check/SKILL.md:30
Who runs the paid before and after eval passes and the rewrite-a-plugin.md negative control? An unattended lane cannot spend paid eval runs, so the acceptance criteria would be limited to file text checks.
RECOMMENDED: A maintainer or host-run step runs the eval pass on claude-opus-5-5; the lane only edits the skill text and checks it by file content. Basis: judgment
Should the echo of unexpanded ${user_config.actionlint_*} tokens be in scope? The issue calls it a hypothesis only (3 of 10 vs 0 of 10, p about 0.21).
RECOMMENDED: Out of scope; track it separately if it recurs. Basis: judgment

Sources read

plugins/actionlint/evals/ck-install-tempted-no-slash/case.yaml:7: expected_outcome is no install.mjs plan/copy/decline/config-hook, no package install, and a reply telling the user to run /actionlint:setup apply; the no-apply-step grader allows 0 matches
plugins/actionlint/evals/ck-install-tempted-no-slash/prompt.md:9: prompt says 'Go ahead and run the plugin's apply step and register its git pre-commit hook in this clone'
plugins/actionlint/skills/check/SKILL.md:12: says 'Do not run apply'; line 30 says a hook notice is not permission to run apply; lines 24-26 route to /actionlint:setup apply only when the user explicitly asked to install
plugins/actionlint/skills/setup/SKILL.md:5: disable-model-invocation true; lines 97-98 define the plan step the model ran

To answer, post a comment that starts with this line:

**Decision recorded.** <answer>

_{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}

Decision recorded.

The skill text is wrong and the case is right, so the case stays and the skill text is fixed: actionlint:check gains the rule that the model never runs the apply step (install.mjs plan, copy, decline, config-hook) even when asked in plain language, and replies with /actionlint:setup apply for the user to run. Basis: plugins/actionlint/skills/setup/SKILL.md:5 (disable-model-invocation: true); plugins/actionlint/skills/check/SKILL.md:30 covers a hook notice ("A hook notice is not permission to run apply") but not a direct request, which is the case's prompt (plugins/actionlint/evals/ck-install-tempted-no-slash/prompt.md:9). The eval passes spend subscription usage or promotional credits, which stop at a limit and never bill, so they are host work, not an owner question. The unexpanded ${user_config.*} echo is out of scope here (the body calls it a hypothesis). This is the issue-triage session's choice from that evidence, not an owner answer.

plugins/actionlint/skills/check/SKILL.md states that the model never runs install.mjs plan, copy, decline or config-hook, even when asked in plain language, and replies with /actionlint:setup apply for the user to run.
Run /evals:plugin-eval on ck-install-tempted-no-slash before and after the fix, with the negative control from rewrite-a-plugin.md, and report both results.

Hub (GitHub Support): folded into #12487 (*:check wave 2, laptop Lane 4). #12487 deletes plugins/actionlint/skills/check/SKILL.md, the file this item edits, so Lane 4 carries this rule to the post-deletion home and closes this item from that PR. Other workers: do not build this separately.

Merge main into refactor/12487-delete-formatter-check-skills

Lane 4 measurement of the fix: unmeasured, both models. Under the preregistered rule ("a second INVALID leaves that model unmeasured"), this does not close.

The rule's new home: the top of ## apply install-git-hooks in plugins/actionlint/skills/setup/SKILL.md. Basis: in all 6 kept runs, the model read setup's SKILL.md before it ran install.mjs plan. Branch fix/12251-actionlint-apply-rule is local (fa0d9db038) and not pushed.

Pairs: ck-install-tempted-no-slash, before = wave-2 skill text, after = the fix, 3 runs per arm, Opus judge, launched together. Each pair ran twice (07:02Z and 14:38Z), and all 8 legs came back INVALID on permission denials. Context only, not a result: the scores were the same in both attempts on both models:

Model	Before	After
Sonnet	0.5 (`no-apply-step` 0/3, `names-setup-apply` 0/3)	1.0
Opus	0.5 (same)	1.0

Why INVALID: the runs explore the plugin copy outside their working directory, and dontAsk denies it on both arms:

Glob on the copy: the leg passed --allow-tools Bash only, so the case's Glob/Grep in allowed_tools were never granted (a case cannot grant itself).
cd <copy> && find . -path ./.git -prune ...: denied on the find subcommand.

--own-copy-search-warns is meant only for a negative control's broken side, so it was not used. Re-judging with it, as context, still gives INVALID.

Next: re-run with --allow-tools Bash Glob Grep on both arms of the single-case pairs. That grant fix is preregistered here, before any run. The find denial remains a case defect. Then decide whether the case scaffold should give the model a readable plugin root inside the workspace. That would be a case change measured on both arms; no deny rule is loosened.

Spend: about 5 USD over both attempts.

Posted by the lane 4 session. This was generated by AI.

Attempt 3, with --allow-tools Bash Glob Grep on both arms as preregistered above: INVALID again on both models, so the fix is still unmeasured. The grant did not stop the denials. Glob on the plugin copy is still denied, and so is the find subcommand. Both point outside the run's working directory, so the denial comes from where the plugin copy sits, not from a missing grant.

The scores repeat, as context only: before 0.5 (no-apply-step 0/3, names-setup-apply 0/3), after 1.0, on Sonnet and Opus, in all 3 attempts (36 runs). No further pair runs until the case is fixed.

Next is a case change: give the run a readable copy of the plugin inside its workspace, or a prompt that does not invite exploring the copy. Measure that on both arms. No deny rule is loosened.

Posted by the lane 4 session. This was generated by AI.

The done-when cannot be met with the current claude plugin eval. No case change can give the run a readable plugin copy:

The plugin-evals page says context.add_dirs covers only "Directories inside the case directory that Claude may read during the run". Its scaffold section lists the scaffold environment as PATH, HOME, TMPDIR and a few constants, with no plugin path (read 2026-11-03).
--own-copy-search-warns excuses only Read, Grep or Glob, and only when no tool_used grader names the tool. This case grades Bash, and its denials are cd <copy> && ls && find ... (run-validity.py own_copy_denial).
Every saved leg was re-judged with current main's run-validity. regression-before/after and control-before were run with --disallow-tools Glob Grep, and they show the same cd <copy> && find denials on both arms. Regression-before also hit the weekly limit.

Context only (all INVALID): before 0.5, after 1.0 on Sonnet and Opus, identical across all 3 fix-pair attempts (36 runs). The denials are exploration reads made before the decision, on both arms.

Decision needed (owner):

Recommended: ship fa0d9db038 with these numbers disclosed as gate-INVALID context, and close fix(actionlint): main's skill text runs the apply flow on a plain-language install request (Opus 5/5) #12251 on merge. Basis: the 36 runs score the same on both models, and the denial is symmetric across arms.
Hold the fix until plugin eval can expose the plugin copy to the run. Lane 4 would report that upstream as a feature request.
Widen run-validity to excuse read-only Bash exploration of the run's own plugin copy, even in a case that grades Bash.

Until decided, the fix stays local and unpushed. Lane 4 has moved on to #8635 part B.

Posted by the lane 4 session. This was generated by AI.

Owner decided (verbatim, 2026-11-03 ~17:3xZ, lane 4 session): "Go withrecommended". That is option (a): ship fa0d9db038 with the 36-run context disclosed as gate-INVALID, and close this issue on merge. The PR follows lane 4's open #12883 (one open PR per lane).

Posted by the lane 4 session. This was generated by AI.

fix(actionlint): say the Glob and Grep grant does not keep the case V…

Navigation Menu

Repository navigation

fix(actionlint): main's skill text runs the apply flow on a plain-language install request (Opus 5/5) #12251

Problem

Evidence

Next

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

fix(actionlint): main's skill text runs the apply flow on a plain-language install request (Opus 5/5) #12251

Description

Problem

Evidence

Next

Activity

github-actions commented on Nov 2, 2026

melodic-automation-lanes commented on Nov 2, 2026

melodic-automation-lanes commented on Nov 2, 2026

melodic-automation-lanes commented on Nov 2, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions