Repository navigation
Open
Copy link
Labels
priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.Refactors, migrations, contract changes; cross-cutting and hard to reverse.
Description
Refs #7582
Outcome
The three code-metrics:audit-* leaves (coverage, duplication, suppressions) and the audit router stay as they are on main. Each leaf description opens with a first clause no sibling shares and is trimmed to the repository's description standard, with no loss in triggering.
Scope is option 3, chosen by the owner on 2026-11-03 (decision: #12491 (comment)), amended on 2026-11-04 to three leaves plus the router, since PR 12803 had already folded size, complexity and type-debt into the router. Owner, 2026-11-04: "Go with recomemndations".
Done when
- No two
plugins/code-metrics/skills/audit-*/SKILL.mddescriptions share a first clause, and/skill-quality:checkpasses on each. - Each changed description passes a two-labeler
/docs-hygiene:compress comparemeaning diff before any paid run. -
plugins/code-metrics/probesexists (the plugin has none today), written perplugins/skill-quality/probes/template.mdby an author who never sees the rewrite. - Triggering is re-measured on those probes before and after (
plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, "Cutting text", steps 3 and 4): no lost positive and no new false fire. A description that fails keeps its main text. -
ls -d plugins/code-metrics/skills/audit-* | wc -lis 3,plugins/code-metrics/skills/auditexists, andscripts/validate-plugins.shpasses.
Out of scope
- Skill bodies, and the
architecture:map-*descriptions (refactor(architecture): give the nine map-* skills distinct, trimmed descriptions #12490).
Related
- skill-quality: skill authoring, quality and catalog #7582 (parent), refactor(architecture): give the nine map-* skills distinct, trimmed descriptions #12490 (same scope for
architecture:map-*), refactor(plugins): give the three non-prerequisite check skills distinct names so no check leaf remains #12489 (owner: no rename for Cursor), perf(session-flow): run the description trim pilot and measure skill visibility #11954 (trimmed descriptions held triggering).
Metadata
Metadata
Assignees
Labels
priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.Refactors, migrations, contract changes; cross-cutting and hard to reverse.
Type
Fields
Work status
agent-ready
Activity
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; a maintainer confirms or changes it.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: keep the six
code-metrics:audit-*skills as separate leaves. Give each description a distinct first clause, trimmed to the repo standard, with no triggering loss. This includes creatingplugins/code-metrics/probesand measuring triggering before and after.The owner's decision comment (#issuecomment-6618125708) chose this option-3 scope. The rewritten body matches it, so no open questions remain.
Some done-when items, such as paid probe runs and the two-labeler compare, may need a host or an attended run. Admission handles routing for those.
No duplicates found. #12490 covers the same work for
architecture:map-*and is separate.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 2, 2026
The latest triage verdict holds nothing: the issue is at
status: ready.melodic-automation-lanes commentedon Nov 2, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Consolidate the six code-metrics audit-* skills into one code-metrics:audit router skill with actions per D107, delivered in at least two waves each under 262144 bytes.
Dependencies
Related work
Open questions
RECOMMENDED: Option 1: one code-metrics:audit router skill with actions/ per D107, confirmed against the #11954 visibility measurement and each skill's eval baseline in the plan. Basis: judgment
RECOMMENDED: The plan comment is this issue's first step; moves land in follow-up wave issues filed from it, so the remaining done-when items belong to those issues. Basis: judgment
RECOMMENDED: Remove them at the end of the final wave and repoint all references in the same wave; the done-when count then equals 1 for the router. Basis: judgment
Sources read
To answer, post a comment that starts with this line:
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.37 remaining items
kyle-sexton commentedon Nov 3, 2026
Decision recorded. Owner answer, given in the lane 2 session on 2026-11-03: "Go with recommendations."
The recommendation it answers (lane 2, Q2): re-scope both consolidations to option 3. Keep the leaf skills; give each a distinct first description clause and a description trimmed to the repository's description standard; measure triggering for no loss per
plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md. The merge into one router skill (option 1) and its waves are dropped.Basis: the only written reason for consolidating was picker fit in Cursor and Claude Code. The owner ruled Cursor out ("I wouldn't change ours if it's just to accommodate cursor", #12489). Claude Code always lists every skill name and drops only descriptions on budget overflow, which users tune with
skillListingBudgetFractionandskillOverrides(Skill descriptions are cut short, read 2026-11-03). One merged description would also have to carry every view's triggers under a 1,536-character cap. #11954 showed trimmed descriptions hold triggering (0.979 to 0.984).Next on this issue: its body's done-when items still describe the consolidation; they are rewritten to the option-3 scope before work starts.
[-]refactor(code-metrics): consolidate the six audit-* skills (plan sized waves first)[/-][+]refactor(code-metrics): give the six audit-* skills distinct, trimmed descriptions[/+]kyle-sexton commentedon Nov 3, 2026
Body and title rewritten to the option-3 scope recorded above. Dropped: the wave plan, the patch-cap sizing and the "Blocked by #12486" line (#12486 is closed, and keeping the leaves no longer depends on it). Added: write
plugins/code-metrics/probesfirst, since the plugin has none.kyle-sexton commentedon Nov 4, 2026
kyle-sexton commentedon Nov 4, 2026
Preregistration, before any paid run.
Setup fact, and a done-when this cannot meet. At base
cb26b84af99only threeaudit-*leaves exist:audit-coverage,audit-duplicationandaudit-suppressions.audit-size,audit-complexityandaudit-type-debtbecame actions of thecode-metrics:auditrouter in #12803, merged before the option 3 decision. Sols -d plugins/code-metrics/skills/audit-* | wc -lreads 3, not 6. Restoring the leaves is outside this run's scope. This run measures the three leaves and leaves the router's text unchanged; the count comes back to the owner as a question in the hand-back.Probes. Frozen at commit
259db6b1712onrefactor/12491-code-metrics-audit-descriptions, before any rewrite existed. A fresh agent with no tools wrote them from before descriptions andplugins/skill-quality/probes/template.mdalone: 18 per skill, 9 should-trigger (at least 6 contesting a named rival, none using the skill's own name words) and 9 near-misses. Competitors:code-metrics:auditin all three, plusverification:measure,testing:plan,testing:audit,mutation-testing:audit(coverage),docs-hygiene:extract-ssot,verification:measure(duplication),toolchain:ratchet,toolchain:lint(suppressions). No probe changes after this comment. No second probe round.Rewrites. A second fresh agent with no tools wrote them, without seeing any probe. Each is under 419 characters (coverage 400, duplication 380, suppressions 410), with no first clause shared.
/skill-quality:check: 6 passed, 0 failed./docs-hygiene:compress compare, two labelers, intended-cuts file: bothVERDICT: PASSin round 1, no SEMANTIC LOSS or AMBIGUITY rows. Text is fixed from here.Arms. All are copies under
~/.ev/c12491/armsfromgit archive 259db6b1712of the six plugins the cases load, outside any git repository, each with its own emitted cases.before: as committed.after: all three rewrites.ccov,cdup,csup:beforewith one skill'sdescription:line replaced by"Helper for repository maintenance tasks.".Run. Per skill, its before, after and control launch together on that skill's 18 cases, one skill at a time.
claude plugin eval <arm> --eval-dir probe-evals/code-metrics --case 'code-metrics-<skill>-*' --runs 3 -j 2 --ablation none --keep-temp --model opus --disallow-tools Read Glob Grep Agent --trust-plugin --no-publish --threshold 0 --max-cost-usd 25, throughdocker-links.py run, withTMPDIRunder~/.evof at most 24 characters and one directory per arm. Claude Code 2.1.318. Launch waits while load is 24 or more, memory is under 16G, or more than 5 evals run. That is 54 runs per arm, 486 in total, and at most 6 runs of this work live at once.Ceiling. 25 USD per arm, 225 USD across the 9 arm runs. Basis: the highest measured figure to hand, about 0.38 USD per run on a worktree suite, times 54 is 20.5 USD, plus headroom. #13011's measured figure for this case shape (about 99 USD over 822 runs, 0.12 per run) gives an expected total near 58 USD. These are list-price estimates against plan usage, and nothing is billed.
Smoke, which does not count. One should-trigger case, one run,
beforearm, to prove loading and read one run's cost. Its result enters no decision. A setup defect it finds is fixed before the first counted run, and the fix is posted here first.Validity, per arm result. All of these:
run-validity.py <json> --runs 3printsverdict: VALID(its exit code alone is not read, since a total failure can exit 0);partialis false;run-validity.pyalready reads as a warning;run-validity.pyreports a should-trigger miss as a measured rate, in every arm, so a miss never invalidates a run.Control fire miss. It is the detection signal and never a validity failure, whether no skill fired or a rival fired. A miss in a run carrying a non-null error is not a miss: it is that run's error, under validity above.
Denials, in every arm alike, before arm included. A denied call other than Skill that
run-validity.pyreads as a warning does not invalidate the run, and the run's fire bit counts. A denial it reads as a FAIL makes that arm INVALID, which is a named setup defect under the re-run rule. No case is dropped from a comparison, on any side.Detect, per skill, control against before. On the 9 should-trigger cases, either:
noise-report.py <control.json> --baseline <before.json>reports acase dropon one of them, and a fresh before and control pair, launched together, 3 runs each on each dropped case, reports a drop on the same case.Near-miss cases do not decide detection. The rule comes from
plugins/skill-quality/reference/invocation-probes.md, "Description controls".Decide, per skill, after against before. Every clause is checked and reported before a verdict.
case dropfromnoise-report.py <after.json> --baseline <before.json>, where a drop is re-run once on a fresh before and after pair, launched together, 3 runs each, and counts only if it recurs on the same case.compare verdictreadsnon-inferior at margin 0.05.Outcomes.
Re-run. One per skill, of all three of its arms launched together, and only for a named setup defect: a usage-limit, auth, crash or sandbox-start error, a ceiling hit, a validity FAIL that names a denial or a missing plugin. The defect is posted here before the re-run. The reproduction pairs in Detect and (c) are part of those rules and are not this re-run.
Disclosed now. The
afterarm holds all three rewrites together. If only some ship, a shipped skill was measured beside a sibling's rewrite that did not ship; no further run follows from that.kyle-sexton commentedon Nov 4, 2026
Named setup defect, before the one re-run for
audit-coverage. Under the preregistration's re-run rule: itsbeforeandafterarms readverdict: INVALID, with 30 and 32 of 54 runs endingFailed to authenticate: OAuth token revoked. Theccovarm launched with them finished VALID, and so did all six arms of the two skills launched afterward, so this looks like a credential change partway through those arms; the cause is otherwise UNVERIFIED. All threeaudit-coveragearms now re-run together with the same flags. The first attempt's results do not count, and no rule changes. This is that skill's one re-run.kyle-sexton commentedon Nov 4, 2026
Result: no control detected a loss, so all three leaves keep their before text. Decided by the preregistration, outcome "control does not detect". The probes stay committed.
The after text held on all of (a) to (d) for every skill; (e) is what fails. Every Skill call on a should-trigger case named the target, under the generic description too, so no rival took a probe.
Validity. All 9 counted arm results read
verdict: VALID, with 54 rows,partialfalse and no run error. The warnings aremodels(no judge is recorded, and these cases have none) andskill fired(near-miss cases carry no fire grader). Every init line listed each plugin the case names, and every trace showsclaude-opus-5-5. Claude Code 2.1.318.The one re-run.
audit-coverage's first attempt lost its before and after arms toOAuth token revoked, posted here before the re-run. The re-run is what the table reports. In that first attempt the control arm alone was valid, at 27/27 fired.Cost. About 53 USD list-price estimate over 649 runs, smoke and the failed attempt included. This is plan usage, and nothing is billed.
Done-when.
/skill-quality:check: PASS on each.plugins/code-metrics/probes: PASS, written blind.ls -d plugins/code-metrics/skills/audit-* | wc -l: FAIL, reads 3, not 6, since #12803 folded size, complexity and type-debt into the router before the option 3 decision.Not done here: the router's text and the six-leaf count both wait on the owner's call between restoring the leaves and amending this issue. The rewrites are kept out of the branch.
chore(code-metrics): add the changelog fragment for the audit probes
chore(code-metrics): mark the probes fragment as no release
kyle-sexton commentedon Nov 4, 2026
Preregistration, name-and-description control, before any paid run.
Question. The generic-description control fired 81 of 81 here and 72 of 72 in #13050, so it may be unable to fail while the skill's name stays in the listing. This run tests a control that also changes the name, then reports whether that control can detect a loss.
This run changes no shipped text. Any change to the certification rule in
plugins/skill-quality/reference/invocation-probes.md, or to a shipped description, is a separate step.Base.
origin/mainatce2bc9d09c2, Claude Code 2.1.318,--model opus. Scoring scripts are that commit'srun-validity.pyandnoise-report.py, extracted under~/.ev/c12491n/tools.Arms. Copies under
~/.ev/c12491n/arms, outside any git repository, fromgit archiveof the six plugins the cases load.before: as committed, cases emitted bymeasure-invocation.sh emit-plugin-eval.ncov,ndup,nsup:beforewith that one skill's folder renamed tohelper-a, which shares no word with the original, and itsdescription:line replaced by the earlier control's"Helper for repository maintenance tasks.".Setup fact. These skills have no
name:line, so the folder rename is the whole rename. The skills page gives thenamefield as "Defaults to the directory name", and a plugin skill's command as "Frontmatternameor the directory name, namespaced by plugin" (skills). Noname:line is added, because that would be a second change.How a fire is scored under the rename. Cases are not re-emitted, because re-emitting would rename them and
noise-report.pypairs by case name. Each control holds a copy ofbefore's cases, and only the skill's 9graders/skill-fired.mdchange, from(?:code-metrics:)?<skill>to(?:code-metrics:)?helper-a.prompt.mdkeeps its text. Itsdescriptionandexpected_outcomefrontmatter still name the old skill. No prompt body names any of the three leaves (checked).Checked with
diff -rbefore this comment. Each control differs frombeforeby the folder rename, onedescription:line and the 9 grader lines, and nothing else.Scope. The 9 should-trigger cases only, since near-misses did not decide detection. Probes are the frozen ones, unchanged since
259db6b1712.Run. Per skill, its
beforeand control launch together, one skill at a time, so at most 4 runs of this work are live.python3 -I <repo>/plugins/evals/skills/plugin-eval/scripts/docker-links.py run -- env TMPDIR=<dir> timeout 3000 claude plugin eval <arm> --eval-dir probe-evals/code-metrics --case 'code-metrics-<skill>-pos-*' --runs 3 -j 2 --ablation none --keep-temp --model opus --disallow-tools Read Glob Grep Agent --trust-plugin --no-publish --threshold 0 --max-cost-usd 5 --json <result><dir>is~/.ev/c12491n/bor~/.ev/c12491n/c, 24 characters expanded. A launch waits while load is 24 or more, memory is under 16G, or more than 5 evals run. That is 27 runs per arm, 162 in total.Ceiling. 5 USD per arm, 30 USD across the 6 arm runs. Basis: the earlier counted results measured 0.052 to 0.073 USD per should-trigger run, 0.082 at most, so 2.2 USD per arm at that maximum. Expected total is about 11 USD. A control miss's cost has not been measured, which is what the rest of the ceiling covers. These are list-price estimates against plan usage, and nothing is billed.
Smoke, which does not count. One run of
code-metrics-audit-coverage-pos-train-01inncov. It shows that the init line listscode-metrics:helper-aand notcode-metrics:audit-coverage. Its score enters no decision. A setup defect it finds is fixed before the first counted run, and the fix is posted here first.Validity, per arm result. All of these:
run-validity.py <json> --runs 3printsverdict: VALID(its exit code alone is not read);partialis false;run-validity.pyreads as a warning;code-metrics:helper-aand not the old name.Fire miss, in either arm. It is the measured quantity and never a validity failure, whether no skill fired or a rival fired. A miss in a run carrying a non-null error is not a miss: it is that run's error, under validity above.
run-validity.pygives a max-turns error with no graded Skill call no exemption, so such a run makes that arm INVALID.beforemisses stay in its mean, and no case is dropped on either side.Denials, in either arm. A denied call other than Skill that
run-validity.pyreads as a warning does not invalidate the run, and the run's fire bit counts. A denial it reads as a FAIL makes that arm INVALID, which is a named setup defect under the re-run rule. Its rules give a should-fire miss with a denial no probe exemption, so a miss with a denial fails unless it scored the same as every denial-free run of its case. No flag is added to change that reading.Detect, per skill, control against
before, the earlier rule unchanged, and only on a VALID pair. On the 9 should-trigger cases, either:noise-report.py <control.json> --baseline <before.json> --margin 0.05 --threshold 0reports acase drop, and a fresh before and control pair, launched together, 3 runs each on each dropped case, shows a drop on the same case.noise-report.pyprints no case-drop line under 3 paired cases, so the reproduction applies its rule directly to the pair's case scores: a fall of a third or more.Description-only control: cited, not re-run. It fired 81 of 81 with mean fall 0.000, under the same version and model and with all 3 arms read
VALIDby the script of that run.git diff 259db6b1712 ce2bc9d09c2anddiff -rshow no skill or probe change in these six plugins since then: five READMEs and one hook test differ, and the emitted cases are identical. Its sandboxes are gone, so today'srun-validity.pycannot re-read its traces. The recorded verdict stands as recorded.Outcomes.
Limits, stated now.
code-metrics:helper-a, so the plugin's name is still beside the renamed skill.code-metrics:auditrouter's listing description ends "Coverage and CRAP: /code-metrics:audit-coverage.", and it stays, because everything else is identical. Inncova run may call that old name, which no longer exists. That scores as a miss under the copied grader, and the result reports its count separately. No sibling's listing description namesaudit-duplicationoraudit-suppressions;code-metrics:principleshas only the wildcard "/code-metrics:audit-* skills".Re-run. One per skill, of its pair launched together, and only for a named setup defect: a usage-limit, auth, crash or sandbox-start error, a ceiling hit, a validity FAIL that names a denial or a missing plugin. The defect is posted here before the re-run. The reproduction pair in Detect is part of that rule and is not this re-run.
kyle-sexton commentedon Nov 4, 2026
Restart note, before any counted result was read.
I stopped the first counted launch about a minute in, during its
audit-coveragepair. My own shell wrapper had a 590-second limit, and three pairs at about 180 seconds each could have reached it and killed the third pair. This is not one of the named defects; the stop was mine.The stopped CLIs wrote result files, which I moved aside unread. They enter no decision. So that no discretion enters, I count this as
audit-coverage's one re-run. No rule changes. The pairs now launch one at a time in separate commands, with the same flags. Smoke result:code-metrics:helper-ais on the init line, the old name is not, and the run took no denial.kyle-sexton commentedon Nov 4, 2026
Result: the name-and-description control detected a loss on all three skills, so outcome (a) holds on each. Decided by the preregistration. This run changes no shipped text. Any change to the certification rule or a shipped description is a separate step.
What this shows. On these probes the name carries the trigger: with the name kept and the description generic, every run fired; with both changed, none did. So on these three skills the description-only control cannot certify a cut: it passed with a generic description, so on these probes it cannot fail. The probes themselves can fail, so outcome (b) does not hold.
Compare line, identical for all three controls.
with-arm change (this minus baseline), paired over 9 cases: -1.00, 95% interval -1.00 to -1.00;compare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -1.00 is below -0.05), and 9 case(s) dropped by a third or more. The mean fall alone meets the detect rule, so no reproduction pair ran.Validity. All 6 arm results read
verdict: VALID, with 27 rows,partialfalse, no run error and no denial. The warnings aremodelsin all 6 (no judge is recorded, and these cases have none) and, in the 3 controls,skill firedas the measured rate. Every init line lists each plugin the case names, and every trace showsclaude-opus-5-5. Claude Code 2.1.318. Every control init line listscode-metrics:helper-aand not the old name; everybeforeinit line lists the old name.What the control runs called (descriptive, decides nothing). In
helper-a's place:helper-acode-metrics:audit6code-review4,code-metrics:audit3toolchain:ratchet5,code-metrics:audit3Spot-read of 9 miss runs, three per control, from the kept traces: each attempted the request and reported that it could not do it, because the session has no read or shell tool and no listed skill fits. All 27 runs of each
beforecalled only the target skill.Limits on reading this.
/code-metrics:audit-coverage. No run called it, and 25 of 27 answers name it, the ones sampled as unavailable, so that skill's misses were reached with the old name in view. The duplication and suppressions controls have no such pointer in the listing and fell the same: no answer in either names the old name. In the suppressions control, the 5 runs that loadedtoolchain:ratchetsaw the old name in its body, and none called it.helper-awas visible and was not chosen: 11 of 27 duplication answers and 2 of 27 coverage answers name it, 0 of 27 suppressions answers, and no run called it.architecture:map-*skills of #13050 were not run under this control, and carrying the finding to them is UNVERIFIED.Deviations.
name:line, so the folder rename was the rename, as the preregistration stated.costUsdandpartialwere read: bothpartialtrue,interrupted. It counts asaudit-coverage's one re-run. No named setup defect came up, and no other re-run happened.Cost. About 11.8 USD list-price estimate: 11.2 over the 162 counted runs, 0.1 for the smoke run, 0.5 for the stopped launch. Each arm cost 1.47 to 2.13 USD, under its 5 USD ceiling. This is plan usage, and nothing is billed.