Skip to content

refactor(code-metrics): give the six audit-* skills distinct, trimmed descriptions #12491

Open
6 of 6 issues completed

Description

@kyle-sexton

Refs #7582

Outcome

The three code-metrics:audit-* leaves (coverage, duplication, suppressions) and the audit router stay as they are on main. Each leaf description opens with a first clause no sibling shares and is trimmed to the repository's description standard, with no loss in triggering.

Scope is option 3, chosen by the owner on 2026-11-03 (decision: #12491 (comment)), amended on 2026-11-04 to three leaves plus the router, since PR 12803 had already folded size, complexity and type-debt into the router. Owner, 2026-11-04: "Go with recomemndations".

Done when

  • No two plugins/code-metrics/skills/audit-*/SKILL.md descriptions share a first clause, and /skill-quality:check passes on each.
  • Each changed description passes a two-labeler /docs-hygiene:compress compare meaning diff before any paid run.
  • plugins/code-metrics/probes exists (the plugin has none today), written per plugins/skill-quality/probes/template.md by an author who never sees the rewrite.
  • Triggering is re-measured on those probes before and after (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, "Cutting text", steps 3 and 4): no lost positive and no new false fire. A description that fails keeps its main text.
  • ls -d plugins/code-metrics/skills/audit-* | wc -l is 3, plugins/code-metrics/skills/audit exists, and scripts/validate-plugins.sh passes.

Out of scope

Related

Activity

  1. added
    priority: mediumReal value, no hard deadline; normal backlog flow.
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 2, 2026
  2. github-actions commented on Nov 2, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; a maintainer confirms or changes it.

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: keep the six code-metrics:audit-* skills as separate leaves. Give each description a distinct first clause, trimmed to the repo standard, with no triggering loss. This includes creating plugins/code-metrics/probes and measuring triggering before and after.

    The owner's decision comment (#issuecomment-6618125708) chose this option-3 scope. The rewritten body matches it, so no open questions remain.

    Some done-when items, such as paid probe runs and the two-labeler compare, may need a host or an attended run. Admission handles routing for those.

    No duplicates found. #12490 covers the same work for architecture:map-* and is separate.

    Automated triage. A maintainer reviews it.

  3. added
    status: needs-decisionAwaiting a human or maintainer judgment call.
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 2, 2026
  4. melodic-automation-lanes commented on Nov 2, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: the issue is at status: ready.

  5. removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 2, 2026
  6. melodic-automation-lanes commented on Nov 2, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Consolidate the six code-metrics audit-* skills into one code-metrics:audit router skill with actions per D107, delivered in at least two waves each under 262144 bytes.

    Dependencies

    • #​12486: Blocked by it; the *:check consolidation goes first per #​7582 order.

    Related work

    Open questions

    1. Which consolidation target: one code-metrics:audit router skill with six actions, two skills split by data source, or keep six leaves with distinct first clauses?
      RECOMMENDED: Option 1: one code-metrics:audit router skill with actions/ per D107, confirmed against the #​11954 visibility measurement and each skill's eval baseline in the plan. Basis: judgment
    2. Is the first deliverable only the sized wave plan comment, with the skill moves filed as later issues, or must this issue also complete the moves?
      RECOMMENDED: The plan comment is this issue's first step; moves land in follow-up wave issues filed from it, so the remaining done-when items belong to those issues. Basis: judgment
    3. Should the old audit-* skill names remain as aliases or redirects during the transition, or be removed outright?
      RECOMMENDED: Remove them at the end of the final wave and repoint all references in the same wave; the done-when count then equals 1 for the router. Basis: judgment

    Sources read

    • /home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/code-metrics/skills: six audit-* directories exist beside principles and setup; I did not measure bytes or read their contents.

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  7. 37 remaining items

  8. kyle-sexton commented on Nov 3, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. Owner answer, given in the lane 2 session on 2026-11-03: "Go with recommendations."

    The recommendation it answers (lane 2, Q2): re-scope both consolidations to option 3. Keep the leaf skills; give each a distinct first description clause and a description trimmed to the repository's description standard; measure triggering for no loss per plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md. The merge into one router skill (option 1) and its waves are dropped.

    Basis: the only written reason for consolidating was picker fit in Cursor and Claude Code. The owner ruled Cursor out ("I wouldn't change ours if it's just to accommodate cursor", #12489). Claude Code always lists every skill name and drops only descriptions on budget overflow, which users tune with skillListingBudgetFraction and skillOverrides (Skill descriptions are cut short, read 2026-11-03). One merged description would also have to carry every view's triggers under a 1,536-character cap. #11954 showed trimmed descriptions hold triggering (0.979 to 0.984).

    Next on this issue: its body's done-when items still describe the consolidation; they are rewritten to the option-3 scope before work starts.

  9. changed the title [-]refactor(code-metrics): consolidate the six audit-* skills (plan sized waves first)[/-] [+]refactor(code-metrics): give the six audit-* skills distinct, trimmed descriptions[/+] on Nov 3, 2026
  10. kyle-sexton commented on Nov 3, 2026

    @kyle-sexton
    MemberAuthor

    Body and title rewritten to the option-3 scope recorded above. Dropped: the wave plan, the patch-cap sizing and the "Blocked by #12486" line (#12486 is closed, and keeping the leaves no longer depends on it). Added: write plugins/code-metrics/probes first, since the plugin has none.

  11. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 4, 2026
  12. self-assigned this
    on Nov 4, 2026
  13. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor
    No description provided.
  14. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Preregistration, before any paid run.

    Setup fact, and a done-when this cannot meet. At base cb26b84af99 only three audit-* leaves exist: audit-coverage, audit-duplication and audit-suppressions. audit-size, audit-complexity and audit-type-debt became actions of the code-metrics:audit router in #12803, merged before the option 3 decision. So ls -d plugins/code-metrics/skills/audit-* | wc -l reads 3, not 6. Restoring the leaves is outside this run's scope. This run measures the three leaves and leaves the router's text unchanged; the count comes back to the owner as a question in the hand-back.

    Probes. Frozen at commit 259db6b1712 on refactor/12491-code-metrics-audit-descriptions, before any rewrite existed. A fresh agent with no tools wrote them from before descriptions and plugins/skill-quality/probes/template.md alone: 18 per skill, 9 should-trigger (at least 6 contesting a named rival, none using the skill's own name words) and 9 near-misses. Competitors: code-metrics:audit in all three, plus verification:measure, testing:plan, testing:audit, mutation-testing:audit (coverage), docs-hygiene:extract-ssot, verification:measure (duplication), toolchain:ratchet, toolchain:lint (suppressions). No probe changes after this comment. No second probe round.

    Rewrites. A second fresh agent with no tools wrote them, without seeing any probe. Each is under 419 characters (coverage 400, duplication 380, suppressions 410), with no first clause shared. /skill-quality:check: 6 passed, 0 failed. /docs-hygiene:compress compare, two labelers, intended-cuts file: both VERDICT: PASS in round 1, no SEMANTIC LOSS or AMBIGUITY rows. Text is fixed from here.

    Arms. All are copies under ~/.ev/c12491/arms from git archive 259db6b1712 of the six plugins the cases load, outside any git repository, each with its own emitted cases.

    • before: as committed.
    • after: all three rewrites.
    • ccov, cdup, csup: before with one skill's description: line replaced by "Helper for repository maintenance tasks.".

    Run. Per skill, its before, after and control launch together on that skill's 18 cases, one skill at a time. claude plugin eval <arm> --eval-dir probe-evals/code-metrics --case 'code-metrics-<skill>-*' --runs 3 -j 2 --ablation none --keep-temp --model opus --disallow-tools Read Glob Grep Agent --trust-plugin --no-publish --threshold 0 --max-cost-usd 25, through docker-links.py run, with TMPDIR under ~/.ev of at most 24 characters and one directory per arm. Claude Code 2.1.318. Launch waits while load is 24 or more, memory is under 16G, or more than 5 evals run. That is 54 runs per arm, 486 in total, and at most 6 runs of this work live at once.

    Ceiling. 25 USD per arm, 225 USD across the 9 arm runs. Basis: the highest measured figure to hand, about 0.38 USD per run on a worktree suite, times 54 is 20.5 USD, plus headroom. #13011's measured figure for this case shape (about 99 USD over 822 runs, 0.12 per run) gives an expected total near 58 USD. These are list-price estimates against plan usage, and nothing is billed.

    Smoke, which does not count. One should-trigger case, one run, before arm, to prove loading and read one run's cost. Its result enters no decision. A setup defect it finds is fixed before the first counted run, and the fix is posted here first.

    Validity, per arm result. All of these:

    • run-validity.py <json> --runs 3 prints verdict: VALID (its exit code alone is not read, since a total failure can exit 0);
    • partial is false;
    • 54 rows;
    • no non-null run error, except a max-turns error after the graded Skill call, which run-validity.py already reads as a warning;
    • each arm's kept init lines list every plugin the case names, and the served model is Opus.

    run-validity.py reports a should-trigger miss as a measured rate, in every arm, so a miss never invalidates a run.

    Control fire miss. It is the detection signal and never a validity failure, whether no skill fired or a rival fired. A miss in a run carrying a non-null error is not a miss: it is that run's error, under validity above.

    Denials, in every arm alike, before arm included. A denied call other than Skill that run-validity.py reads as a warning does not invalidate the run, and the run's fire bit counts. A denial it reads as a FAIL makes that arm INVALID, which is a named setup defect under the re-run rule. No case is dropped from a comparison, on any side.

    Detect, per skill, control against before. On the 9 should-trigger cases, either:

    • the mean case score falls by 0.15 or more; or
    • noise-report.py <control.json> --baseline <before.json> reports a case drop on one of them, and a fresh before and control pair, launched together, 3 runs each on each dropped case, reports a drop on the same case.

    Near-miss cases do not decide detection. The rule comes from plugins/skill-quality/reference/invocation-probes.md, "Description controls".

    Decide, per skill, after against before. Every clause is checked and reported before a verdict.

    • (a) Trigger rate on the should-trigger cases: after ≥ before − 0.05.
    • (b) False-trigger rate on the near-misses: after ≤ before + 0.05.
    • (c) No lost positive and no new false fire: no case drop from noise-report.py <after.json> --baseline <before.json>, where a drop is re-run once on a fresh before and after pair, launched together, 3 runs each, and counts only if it recurs on the same case.
    • (d) That noise report's compare verdict reads non-inferior at margin 0.05.
    • (e) The control detects.

    Outcomes.

    • Valid, control detects, (a) to (d) hold: the new text ships.
    • Valid, control detects, any of (a) to (d) fails: the before text ships.
    • Valid, control does not detect: unmeasurable, the before text ships, probes stay committed. This is a likely result: #13050 saw no control drop on four skills, and the listing keeps each skill's name under a generic description.
    • Any of the skill's arms still invalid after the one re-run: that skill's run does not count, and the before text ships.

    Re-run. One per skill, of all three of its arms launched together, and only for a named setup defect: a usage-limit, auth, crash or sandbox-start error, a ceiling hit, a validity FAIL that names a denial or a missing plugin. The defect is posted here before the re-run. The reproduction pairs in Detect and (c) are part of those rules and are not this re-run.

    Disclosed now. The after arm holds all three rewrites together. If only some ship, a shipped skill was measured beside a sibling's rewrite that did not ship; no further run follows from that.

  15. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Named setup defect, before the one re-run for audit-coverage. Under the preregistration's re-run rule: its before and after arms read verdict: INVALID, with 30 and 32 of 54 runs ending Failed to authenticate: OAuth token revoked. The ccov arm launched with them finished VALID, and so did all six arms of the two skills launched afterward, so this looks like a credential change partway through those arms; the cause is otherwise UNVERIFIED. All three audit-coverage arms now re-run together with the same flags. The first attempt's results do not count, and no rule changes. This is that skill's one re-run.

  16. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Result: no control detected a loss, so all three leaves keep their before text. Decided by the preregistration, outcome "control does not detect". The probes stay committed.

    skill before chars shipped should-trigger fired, before / after / control false fires, before / after / control mean fall, control control case drops after vs before
    audit-coverage (re-run) 474 before text 27/27, 27/27, 27/27 0/27, 0/27, 0/27 0.000 0 non-inferior at 0.05, no drop
    audit-duplication 434 before text 27/27, 27/27, 27/27 1/27, 0/27, 0/27 0.000 0 non-inferior at 0.05, no drop
    audit-suppressions 898 before text 27/27, 27/27, 27/27 0/27, 0/27, 0/27 0.000 0 non-inferior at 0.05, no drop

    The after text held on all of (a) to (d) for every skill; (e) is what fails. Every Skill call on a should-trigger case named the target, under the generic description too, so no rival took a probe.

    Validity. All 9 counted arm results read verdict: VALID, with 54 rows, partial false and no run error. The warnings are models (no judge is recorded, and these cases have none) and skill fired (near-miss cases carry no fire grader). Every init line listed each plugin the case names, and every trace shows claude-opus-5-5. Claude Code 2.1.318.

    The one re-run. audit-coverage's first attempt lost its before and after arms to OAuth token revoked, posted here before the re-run. The re-run is what the table reports. In that first attempt the control arm alone was valid, at 27/27 fired.

    Cost. About 53 USD list-price estimate over 649 runs, smoke and the failed attempt included. This is plan usage, and nothing is billed.

    Done-when.

    • First clauses: PASS on the before text, none shared. /skill-quality:check: PASS on each.
    • Meaning diff: PASS, two labelers, on the rewrites that did not ship.
    • plugins/code-metrics/probes: PASS, written blind.
    • Triggering: no lost positive and no new false fire. The cuts are unmeasurable on these probes, so none ships and all three stay over the 419-character warn line.
    • ls -d plugins/code-metrics/skills/audit-* | wc -l: FAIL, reads 3, not 6, since #12803 folded size, complexity and type-debt into the router before the option 3 decision.

    Not done here: the router's text and the six-leaf count both wait on the owner's call between restoring the leaves and amending this issue. The rewrites are kept out of the branch.

  17. removed their assignment
    on Nov 4, 2026
  18. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Preregistration, name-and-description control, before any paid run.

    Question. The generic-description control fired 81 of 81 here and 72 of 72 in #13050, so it may be unable to fail while the skill's name stays in the listing. This run tests a control that also changes the name, then reports whether that control can detect a loss.

    This run changes no shipped text. Any change to the certification rule in plugins/skill-quality/reference/invocation-probes.md, or to a shipped description, is a separate step.

    Base. origin/main at ce2bc9d09c2, Claude Code 2.1.318, --model opus. Scoring scripts are that commit's run-validity.py and noise-report.py, extracted under ~/.ev/c12491n/tools.

    Arms. Copies under ~/.ev/c12491n/arms, outside any git repository, from git archive of the six plugins the cases load.

    • before: as committed, cases emitted by measure-invocation.sh emit-plugin-eval.
    • ncov, ndup, nsup: before with that one skill's folder renamed to helper-a, which shares no word with the original, and its description: line replaced by the earlier control's "Helper for repository maintenance tasks.".

    Setup fact. These skills have no name: line, so the folder rename is the whole rename. The skills page gives the name field as "Defaults to the directory name", and a plugin skill's command as "Frontmatter name or the directory name, namespaced by plugin" (skills). No name: line is added, because that would be a second change.

    How a fire is scored under the rename. Cases are not re-emitted, because re-emitting would rename them and noise-report.py pairs by case name. Each control holds a copy of before's cases, and only the skill's 9 graders/skill-fired.md change, from (?:code-metrics:)?<skill> to (?:code-metrics:)?helper-a. prompt.md keeps its text. Its description and expected_outcome frontmatter still name the old skill. No prompt body names any of the three leaves (checked).

    Checked with diff -r before this comment. Each control differs from before by the folder rename, one description: line and the 9 grader lines, and nothing else.

    Scope. The 9 should-trigger cases only, since near-misses did not decide detection. Probes are the frozen ones, unchanged since 259db6b1712.

    Run. Per skill, its before and control launch together, one skill at a time, so at most 4 runs of this work are live.

    python3 -I <repo>/plugins/evals/skills/plugin-eval/scripts/docker-links.py run -- env TMPDIR=<dir> timeout 3000 claude plugin eval <arm> --eval-dir probe-evals/code-metrics --case 'code-metrics-<skill>-pos-*' --runs 3 -j 2 --ablation none --keep-temp --model opus --disallow-tools Read Glob Grep Agent --trust-plugin --no-publish --threshold 0 --max-cost-usd 5 --json <result>

    <dir> is ~/.ev/c12491n/b or ~/.ev/c12491n/c, 24 characters expanded. A launch waits while load is 24 or more, memory is under 16G, or more than 5 evals run. That is 27 runs per arm, 162 in total.

    Ceiling. 5 USD per arm, 30 USD across the 6 arm runs. Basis: the earlier counted results measured 0.052 to 0.073 USD per should-trigger run, 0.082 at most, so 2.2 USD per arm at that maximum. Expected total is about 11 USD. A control miss's cost has not been measured, which is what the rest of the ceiling covers. These are list-price estimates against plan usage, and nothing is billed.

    Smoke, which does not count. One run of code-metrics-audit-coverage-pos-train-01 in ncov. It shows that the init line lists code-metrics:helper-a and not code-metrics:audit-coverage. Its score enters no decision. A setup defect it finds is fixed before the first counted run, and the fix is posted here first.

    Validity, per arm result. All of these:

    • run-validity.py <json> --runs 3 prints verdict: VALID (its exit code alone is not read);
    • partial is false;
    • 27 rows;
    • no non-null run error, except a max-turns error after the graded Skill call, which run-validity.py reads as a warning;
    • each arm's kept init lines list every plugin the case names, and the served model is Opus;
    • each control's kept init lines list code-metrics:helper-a and not the old name.

    Fire miss, in either arm. It is the measured quantity and never a validity failure, whether no skill fired or a rival fired. A miss in a run carrying a non-null error is not a miss: it is that run's error, under validity above. run-validity.py gives a max-turns error with no graded Skill call no exemption, so such a run makes that arm INVALID. before misses stay in its mean, and no case is dropped on either side.

    Denials, in either arm. A denied call other than Skill that run-validity.py reads as a warning does not invalidate the run, and the run's fire bit counts. A denial it reads as a FAIL makes that arm INVALID, which is a named setup defect under the re-run rule. Its rules give a should-fire miss with a denial no probe exemption, so a miss with a denial fails unless it scored the same as every denial-free run of its case. No flag is added to change that reading.

    Detect, per skill, control against before, the earlier rule unchanged, and only on a VALID pair. On the 9 should-trigger cases, either:

    • the mean case score falls by 0.15 or more; or
    • noise-report.py <control.json> --baseline <before.json> --margin 0.05 --threshold 0 reports a case drop, and a fresh before and control pair, launched together, 3 runs each on each dropped case, shows a drop on the same case.

    noise-report.py prints no case-drop line under 3 paired cases, so the reproduction applies its rule directly to the pair's case scores: a fall of a third or more.

    Description-only control: cited, not re-run. It fired 81 of 81 with mean fall 0.000, under the same version and model and with all 3 arms read VALID by the script of that run. git diff 259db6b1712 ce2bc9d09c2 and diff -r show no skill or probe change in these six plugins since then: five READMEs and one hook test differ, and the emitted cases are identical. Its sandboxes are gone, so today's run-validity.py cannot re-read its traces. The recorded verdict stands as recorded.

    Outcomes.

    • (a) The control detects on a skill. That shows the name carries the trigger, since the same probes fired 27 of 27 with the name kept and the description generic, and it tells us the description-only control cannot certify a cut on that skill. Reported per skill.
    • (b) All three pairs VALID and the control detects on no skill. That shows the probes themselves cannot fail.
    • Any pair still INVALID after the re-run: that skill's pair does not count. If no other skill detects, neither (a) nor (b) holds, and the result says so.

    Limits, stated now.

    • The namespace stays: each control lists code-metrics:helper-a, so the plugin's name is still beside the renamed skill.
    • The code-metrics:audit router's listing description ends "Coverage and CRAP: /code-metrics:audit-coverage.", and it stays, because everything else is identical. In ncov a run may call that old name, which no longer exists. That scores as a miss under the copied grader, and the result reports its count separately. No sibling's listing description names audit-duplication or audit-suppressions; code-metrics:principles has only the wildcard "/code-metrics:audit-* skills".
    • The result also reports which skill each should-trigger run called. That is descriptive and decides nothing.

    Re-run. One per skill, of its pair launched together, and only for a named setup defect: a usage-limit, auth, crash or sandbox-start error, a ceiling hit, a validity FAIL that names a denial or a missing plugin. The defect is posted here before the re-run. The reproduction pair in Detect is part of that rule and is not this re-run.

  19. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Restart note, before any counted result was read.

    I stopped the first counted launch about a minute in, during its audit-coverage pair. My own shell wrapper had a 590-second limit, and three pairs at about 180 seconds each could have reached it and killed the third pair. This is not one of the named defects; the stop was mine.

    The stopped CLIs wrote result files, which I moved aside unread. They enter no decision. So that no discretion enters, I count this as audit-coverage's one re-run. No rule changes. The pairs now launch one at a time in separate commands, with the same flags. Smoke result: code-metrics:helper-a is on the init line, the old name is not, and the run took no denial.

  20. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Result: the name-and-description control detected a loss on all three skills, so outcome (a) holds on each. Decided by the preregistration. This run changes no shipped text. Any change to the certification rule or a shipped description is a separate step.

    skill should-trigger fired, before description-only control, earlier name-and-description control mean fall case drops before verdict control verdict
    audit-coverage 27/27 27/27 0/27 1.000 9 of 9 VALID VALID
    audit-duplication 27/27 27/27 0/27 1.000 9 of 9 VALID VALID
    audit-suppressions 27/27 27/27 0/27 1.000 9 of 9 VALID VALID

    What this shows. On these probes the name carries the trigger: with the name kept and the description generic, every run fired; with both changed, none did. So on these three skills the description-only control cannot certify a cut: it passed with a generic description, so on these probes it cannot fail. The probes themselves can fail, so outcome (b) does not hold.

    Compare line, identical for all three controls. with-arm change (this minus baseline), paired over 9 cases: -1.00, 95% interval -1.00 to -1.00; compare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -1.00 is below -0.05), and 9 case(s) dropped by a third or more. The mean fall alone meets the detect rule, so no reproduction pair ran.

    Validity. All 6 arm results read verdict: VALID, with 27 rows, partial false, no run error and no denial. The warnings are models in all 6 (no judge is recorded, and these cases have none) and, in the 3 controls, skill fired as the measured rate. Every init line lists each plugin the case names, and every trace shows claude-opus-5-5. Claude Code 2.1.318. Every control init line lists code-metrics:helper-a and not the old name; every before init line lists the old name.

    What the control runs called (descriptive, decides nothing). In helper-a's place:

    control Skill calls runs with no Skill call calls to helper-a calls to the old name
    audit-coverage code-metrics:audit 6 21 0 0
    audit-duplication code-review 4, code-metrics:audit 3 21 0 0
    audit-suppressions toolchain:ratchet 5, code-metrics:audit 3 19 0 0

    Spot-read of 9 miss runs, three per control, from the kept traces: each attempted the request and reported that it could not do it, because the session has no read or shell tool and no listed skill fits. All 27 runs of each before called only the target skill.

    Limits on reading this.

    • The control is close to removing the skill from view, and the fall is all or nothing. It shows these probes can fail. It says nothing about whether they could detect a smaller loss, such as a weaker description under the real name.
    • No arm kept the description and changed the name, so this does not show whether the description could carry the trigger alone. The name is sufficient here, which is why description cuts are unmeasurable, but it is not shown to be the only carrier.
    • In the coverage control, the router's description still names /code-metrics:audit-coverage. No run called it, and 25 of 27 answers name it, the ones sampled as unavailable, so that skill's misses were reached with the old name in view. The duplication and suppressions controls have no such pointer in the listing and fell the same: no answer in either names the old name. In the suppressions control, the 5 runs that loaded toolchain:ratchet saw the old name in its body, and none called it.
    • helper-a was visible and was not chosen: 11 of 27 duplication answers and 2 of 27 coverage answers name it, 0 of 27 suppressions answers, and no run called it.
    • This covers three skills on Opus 5.5. The architecture:map-* skills of #13050 were not run under this control, and carrying the finding to them is UNVERIFIED.

    Deviations.

    • These skills carry no name: line, so the folder rename was the rename, as the preregistration stated.
    • The first counted launch was stopped by me, as the restart note says. Its files stayed unread until the counted runs ended, then only their costUsd and partial were read: both partial true, interrupted. It counts as audit-coverage's one re-run. No named setup defect came up, and no other re-run happened.

    Cost. About 11.8 USD list-price estimate: 11.2 over the 162 counted runs, 0.1 for the smoke run, 0.5 for the stopped launch. Each arm cost 1.47 to 2.13 USD, under its 5 USD ceiling. This is plan usage, and nothing is billed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

    Type

    Fields

    Work status

    agent-ready

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions