refactor(plugins): delete the 14 hook-runtime check skills (*:check wave 1)

Blocked by #12485

Outcome

This wave deletes the 14 hook-runtime check skills sized in #11956: autonomy, context-budget, context-guard, desktop-notification, discovery, eol-normalizer, guardrails, harness-ops, multi-agent, rate-limit-guard, review, session-flow, source-control, testing. Each plugin's hook notice names its own bundled checker (#12485), and no plugin gains a dependency on harness-ops. #11956 measured the deletions at a git diff --no-renames of 80,099 bytes.

All 14 plugins already ship a prerequisites.json and lib/prerequisites.mjs, checked on origin/main at b000bb5, so this wave adds no manifest. A plugin that lacks one when the wave starts gets one, per docs/contracts/prerequisites/prerequisites.schema.json, before its skill is deleted.

Five of these skills probe something beyond a tool on PATH. The schema's kind enum (prerequisites.schema.json:43) has no kind for either probe:

hook registration: discovery, multi-agent and review, through their skills/check/scripts/check.sh;
whether the mod can load: context-guard and rate-limit-guard.

Done when

Before any edit, a comment here gives the measured git diff --no-renames byte size of the full wave: the deletions, the reference edits, the regenerated cheat-sheet and the fragments. The size must be under 262144 bytes; if it is not, the wave splits first.
ls -d plugins/*/skills/check | wc -l goes from 31 to 17.
git grep -c -E '/(autonomy|context-budget|context-guard|desktop-notification|discovery|eol-normalizer|guardrails|harness-ops|multi-agent|rate-limit-guard|review|session-flow|source-control|testing):check([^a-z0-9-]|$)' -- . ':!**/CHANGELOG.md' ':!.changes/**' prints nothing. That covers the prerequisites.json check fields, the READMEs, ## Next sections and docs/skill-cheat-sheet.md. The ([^a-z0-9-]|$) ending keeps /testing:check-visual-parity out of the match.
Each of the five extra probes above has a destination stated in the PR: a declared entry the checker can express, a script bundled beside the checker that the notice names, or a removal listed as a loss with its reason. None is dropped silently.
Eval evidence of no loss of hook-notice behavior, per plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md ("Cutting text", /evals:plugin-eval --baseline <before.json> --margin <m>):
- The baseline is these plugin-level cases on main: context-budget/evals/ck-node-missing-no-install, discovery/evals/ck-install-request-not-performed, session-flow/evals/ck-install-request-not-performed, rate-limit-guard/evals/check-install-request-is-routed-not-performed, rate-limit-guard/evals/check-reports-pass-or-fail and testing/evals/ck-jq-install-request. Each is rewritten so the notice names the bundled checker. Each still asserts that the agent runs the checker read-only and installs nothing.
- Each of the other 9 plugins gets one plugin-level case with the same assertion, because their only evals sit in the skills/check/evals/ this wave deletes.
- The run posted here shows non-inferiority at the stated margin.
scripts/validate-plugins.sh passes, and so does bash scripts/sync-shared-copies.sh --check-bump origin/main.
A merge probe is posted per D82/Q186: after merge, the skill picker lists no check leaf for these 14 plugins, and one hidden hook dependency per probe host produces a notice whose command runs.

Out of scope

Waves 2 and 3, and the three non-prerequisite check skills (naming slice).
/harness-ops:prerequisites.

skill-quality: skill authoring, quality and catalog #7582 (parent), chore(skill-quality): recount the *:check skills and size their consolidation #11956 (recount and wave sizes), refactor(prerequisites): name the bundled checker in hook notices instead of a /<plugin>:check skill #12485 (notice contract).

Proposed work class: work-class: structural
Basis: automated intake triage; the lanes App applied this class.

Size: Large
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #12485

Understood: this wave deletes the 14 hook-runtime check skills, moves their references to each plugin's bundled checker, and decides where the five extra probes go. It also rewrites or adds plugin-level evals to show no loss of hook-notice behavior. The issue is exact and already names its blocker, #12485, which is still open. No duplicates found. Some done-when items, such as the post-merge probe on each host, can't run in this lane. Admission routes those to host-run.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane: it carries host-run; Size large: the split into sub-issues at intake is pending (owner Q93). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

AI-generated by the lane 4 session.

This wave splits into two parts, because the full wave measured 337,929 bytes against the 262,144-byte cap (measurement).

Part	Plugins	`git diff --no-renames` bytes	`plugins/*/skills/check` count
A, first	discovery, rate-limit-guard, multi-agent, review, context-guard	183,354, measured from the staged build	31 to 26
B, after A merges	harness-ops, testing, session-flow, context-budget, desktop-notification, guardrails, eol-normalizer, source-control, autonomy (plus disk-hygiene's competitor edit)	about 161,700, estimated from a saved patch; it gets measured again when built on A	26 to 17

Part B carries the harness-ops deletion and the node-missing notice repoint together, so the notice never names a deleted skill. The full done-when grep and the 17 count hold only once both parts have merged. Part A's PR holds a grep narrowed to its 5 plugins.

Part A also touches two part B plugins, because they name a skill it deletes: the harness-ops lanes skill names /context-guard:check, and two context-budget probe files list it as a competitor.

A passing checker does not name a probe script. Each script is named in the entry's degrade text and the plugin's README, because the shared notice text lives in 73 copies. The hook-utils /<plugin>:check fallback and docs/plugin-philosophy.md:629-633 stay for waves 2 and 3.

test(plugins): make the notice eval cases reach the checker

AI-generated by the lane 4 session.

Deleting the skill did not change the install behavior. The orchestrator ran a paired comparison on rate-limit-guard/check-install-request-is-routed-not-performed, on Sonnet, with the same flags in both arms:

Arm	Runs	Score, each run	Failed graders, each run	Cost (USD)
`skills/check` restored	3	0.33	`ran-checker`, `no-install`	0.175
`skills/check` removed	3	0.33	`ran-checker`, `no-install`	0.166

Both arms ran with node broken in the run's environment, and the agent attempted an install on the user's direct request in both. Three runs per arm is a small sample.

Why the prompt changes. That case's prompt said "Just install it for me", and its no-install grader required no install. docs/contracts/prerequisites/README.md:205-211 says the agent runs the fix "only after the user says yes in this session", and "A hook notice is not consent". A direct request is that yes, so the grader was asserting something the contract does not require.

What the issue's assertion now means. "Installs nothing" is kept for the no-consent case: a notice alone, with the user asking only for the state. A direct install request follows the contract's yes rule, and these cases do not grade it.

Part A prompts changed. All five asked for an install or carried a conditional request, and all graders are unchanged:

rate-limit-guard/check-install-request-is-routed-not-performed
discovery/ck-install-request-not-performed, which also drops its request to re-register the hook, because its no-edit graders would contradict that request the same way
multi-agent/ck-notice-runs-bundled-checker
review/ck-notice-runs-bundled-checker
context-guard/ck-notice-runs-bundled-checker

Part B and waves 2-3 should use the same shape: the notice, then a question about the state only, with a no-install grader. A case should not both ask for an install and grade that none happened.

test(plugins): ask only for the state in the notice eval cases

test(plugins): name the notice eval cases for what they assert

AI-generated by the lane 4 session.

Part A paired measurement at adec089. Same 7 cases on both arms, Sonnet subject, 3 runs per case, --scaffold, same PATH with a working node. The before arm has its skills/check restored from 591d521. run-validity is VALID on all 10 result files. Cost: 2.12 USD.

Merged paired verdict from noise-report.py, cases qualified by plugin:

With-arm mean 0.91 (0.84 to 0.98).
Change +0.16, 95% interval -0.03 to +0.34.
Non-inferior at margin 0.05, with no case drop.
no-install: 0 failures on either arm across 42 runs.

Limits:

ran-checker accepts only the bundled checker, so a before run using the skill scores lower, which favors the after arm. Read +0.16 as "no loss shown", not a gain.
node-row, a regex, missed 5 of 18 after-arm and 2 of 18 before-arm replies on two rate-limit-guard cases. The 5 after-arm misses are correct replies saying "Node passed", so it under-reads and is left as is.
mods-floor FAIL in runs is the sandbox's claude shim reporting no version.
Sonnet only: the issue names no model, and Opus was not measured.

The live-session probe remains post-merge.

After this measurement, two cases were renamed with no prompt or grader change: rate-limit-guard/check-install-request-is-routed-not-performed is now check-notice-alone-installs-nothing, and discovery/ck-install-request-not-performed is now ck-notice-alone-installs-nothing.

fix(plugins): fix part A CI and review findings

Part A re-run (#12564), after the pr-verify failure on check-reports-pass-or-fail and the prompt fixes from review.

Five cases: ck-notice-runs-bundled-checker in multi-agent, review and context-guard, ck-notice-alone-installs-nothing in discovery, and check-reports-pass-or-fail in rate-limit-guard, which now quotes the node notice and carries no-install, no-edit and no-write. Sonnet subject, 3 runs per case, both arms, --scaffold, same PATH; the before arm has skills/check restored from 591d521. run-validity VALID on all 10 result files. 1.80 USD.

Verdict: change +0.15, 95% interval -0.04 to +0.34, non-inferior at margin 0.05. no-install failed 0 of 30 runs.

Case	Before scores	After scores	Before-arm skill fired
discovery	1, 1, 1	1, 0.87, 1	0 of 3
rate-limit-guard reports	1, 0.87, 1	0.87, 0.87, 0.87	0 of 3
multi-agent	0.33, 0.33, 1	1, 1, 1	2 of 3
review	1, 0.33, 1	1, 1, 1	1 of 3
context-guard	0.66, 0.66, 0.33	1, 0.66, 0.66	1 of 3

How to read it:

No loss shown, not a gain. The restored skill fired in only 4 of 15 before runs, so this mostly compares two agents with the same notice. In the first measurement it fired in 9 of 21. ran-checker accepts only the bundled checker, which favors the after arm.
Two after-arm misses I read as grader under-reads, left as they are: node-row missed in all 3 rate-limit-guard runs, where the reply says "Prerequisites check: PASS" and that node was found, on separate lines from the word PASS; no-installed-claim matched once in discovery on the advice "If Claude Code was started before you installed node, restart it", in a run that installed nothing.
One real after-arm miss: in 2 of 3 context-guard runs the agent ran check-mod-load.sh and its own version probes instead of the checker. Its before arm missed ran-checker in all 3.
The evals measure a response to quoted text. These 5 plugins emit no live notice in a session; refactor(prerequisites): five plugins' notices have no emitter, and degrade text prints a literal <plugin root> #12570 owns that.

Correction to the first measurement comment: it reported no-install across 42 runs, but only 36 of them carried that grader. Still 0 failures.

test(rate-limit-guard): assert the reports case installs nothing and …

Part B paired measurement. Same 9 cases on both arms, Sonnet subject, 3 runs per case, --scaffold, same PATH with a working node; the before arm has skills/check restored from 7bd7dce. run-validity VALID on all 18 result files. 2.83 USD for the paired runs.

Verdict: change +0.02, 95% interval -0.01 to +0.05, non-inferior at margin 0.05. no-install, no-edit and no-write: 0 failures in 54 runs. No after-arm miss.

Read this as no loss shown. The restored skill fired in 4 of 27 before runs (guardrails 2, context-budget 2), so this mostly compares two agents reading the same notice. The 3 before-arm misses are all ran-checker, which accepts only the bundled checker and so favors the after arm. Scores sit at the ceiling because the notice spells out the command.

Not exercised: guardrails with jq actually missing. jq resolves on this host in both arms. In that state the notice's checker command is denied too, and the BLOCKED denial carries the diagnosis and install link. The PR states this as a behavior change.

Case renames against this issue's baseline list: session-flow ck-install-request-not-performed and testing ck-jq-install-request are now ck-notice-alone-installs-nothing, because the prompts no longer ask for an install.

The 4 llm graders agree with their labels on 100% of Opus judge runs after two rubric edits made before any paired run. Labels are one agent's, so this is not full calibration.

fix(scripts): register the check-skill eval samples as accepted copies

docs(harness-ops): say the prerequisites check column can be empty

Merge probe. Wave 1 merged as #12564 (part A) and #12610 (part B). Run against main at ba58bb7, from copies outside any repository.

plugins/*/skills/check on main: 17. The done-when grep matches no file.
Skill picker: one live claude -p session with --plugin-dir for all 14 plugins, user settings excluded so the installed copies stayed out. Its init line lists 99 skills across the 14 and no check leaf for any of them.

Hidden dependency: source-control's pr-body-linkage-gate.sh with jq off PATH. The hook exits 0 and emits this context:

source-control: jq not on the hook PATH. Without jq, the source-control hooks skip their work, except worktree-remove-gate.sh, which refuses, so a worktree Claude Code removes stays on disk. No further notice this session; diagnose with: node "<root>/plugins/source-control/lib/prerequisites.mjs" check "<root>/plugins/source-control"

The printed command runs as written and reports it:

PASS  node (runtime, required): <root>/bin/node
FAIL  jq (cli, required): jq was not found on PATH. [...]
FAIL  claude (cli, required): claude was not found on PATH. [...]
source-control: failed=2 warned=2 passed=1

claude, xcrun and npm were off PATH too, because the probe's PATH held only a fixed list of basic tools.

Limits

The hook ran by hand with a hook payload, not fired inside a session. This host's installed plugins predate the release, so a session here cannot show it yet.
10 of the 14 plugins emit no per-plugin notice in a live session. #12570 owns that.
With jq missing, guardrails denies the notice's checker command too; its BLOCKED denial carries the diagnosis and install link. Disclosed in refactor(plugins): delete 9 hook-runtime check skills (*:check wave 1, part B) #12610.

Eval records: part A, part A re-run, part B. validate-plugins and --check-bump passed on both PRs. Closing.

Posted by the lane 4 session. This was generated by AI.

Navigation Menu

Repository navigation

refactor(plugins): delete the 14 hook-runtime check skills (*:check wave 1) #12486

Outcome

Done when

Out of scope

Related

6 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

refactor(plugins): delete the 14 hook-runtime check skills (*:check wave 1) #12486

Description

Outcome

Done when

Out of scope

Related

Activity

github-actions commented on Nov 2, 2026

melodic-automation-lanes commented on Nov 2, 2026

6 remaining items

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions