Repository navigation
Concise skills rollout: naming (1 skills, 1 model-invoked) #6611
Description
Part of #6476. Rewrite naming's skills to the skill criteria in one draft PR, following the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). The runbook and the rewrite protocol it cites decide what ships; this issue holds only what is specific to the plugin.
Path: description-only, measured by trigger probes (owner decision on #6476, 2026-10-18: the 1-skill tail takes the description-only path). The body is not rewritten.
The plugin today
As of origin/main on 2026-10-24:
- Skills:
name-it-better(model-invoked). - Outcome eval case directories under the plugin: 0. Probe files under
probes/: 0.
Done when
-
git diff origin/main -- plugins/naming/skillschanges no line outside thedescriptionkey ofplugins/naming/skills/name-it-better/SKILL.md. -
plugins/naming/probes/name-it-better.jsonexists in the PR. - The PR body shows
/skill-quality:check measure-invocationtrigger scores for the before and the after description on the same probes. - The after description scores at least the before score, or the PR ships the before description and says so.
-
/skill-quality:check plugins/namingon the PR head reports no FAIL that the run on main does not. - The plugin's version or changelog entry is in the PR, per the plugin's release mode.
Eval spend and review
- Eval passes within prepaid API credit are pre-approved (owner decision Q46, 2026-10-24), at the
--max-cost-usd 5per-pass ceiling set on #6476 on 2026-10-18. - No person gates the PR: the owner's 2026-10-24 decision (C12) leaves only an intent mismatch during the build or a security issue at review as stops.
Generated by Claude Code
Metadata
Metadata
Assignees
Labels
Type
Fields
No fields configured for Task.
Activity
github-actions commentedon Oct 7, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: rewrite only the
descriptionof thename-it-betterskill in thenamingplugin, add a trigger-probe file, compare before/after trigger scores, and add the version or changelog entry. The runbook and the done-when list are specific, and eval spend is already pre-approved. Nothing is missing, and I found no likely duplicates.Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 24, 2026
Released by the owner's HITL session (2026-10-24, C12): it waited only on a removed gate. Intake triage re-runs on it to set its fields, then admission takes it.
github-actions commentedon Oct 25, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
3 remaining items
kyle-sexton commentedon Oct 27, 2026
This was generated by AI.
Parked for capacity, not for the owner. This rollout's remaining work is the measured eval passes of the batch-1 protocol, which the issue lane cannot run: it cannot start
claudeorclaude plugin eval(see the lane's escalation on #6585, https://github.com/melodic-software/claude-code-plugins/actions/runs/40186335064). It waits on the plugin eval dispatch work in #9704, now a blocked-by edge, so admission holds it until #9704 closes.Under the owner's 2026-10-26 stop rule, a person is needed only for a change in intent or vision, an implementation finding that changes the intent or primary design, or a high or critical security issue; host, attended and eval runs are agent work. So
needs-humancomes off, and the issue moves tohost-runwithstatus: confirmed, which keeps the issue lane from taking it when #9704 closes: #9704 adds an eval dispatch workflow, not eval capability for the issue lane.kyle-sexton commentedon Oct 31, 2026
Host run. Description-only trigger screen for
naming:name-it-better: before and after copies launched together, frozen probes.What ran
a781adf89a, modelsclaude-opus-5-5andclaude-sonnet-5-5,--ablation none,-j 2, cases frommeasure-invocation.sh emit-plugin-eval, 20 probes (10 positive, 10 negative), digest49ea5128d1ae.--runs 1screen, then--runs 3, where after must score at least before on every model (the done-when), or the before description ships.claude plugin eval <copy> --eval-dir trigger --model <model> --ablation none --runs N -j 2 --keep-temp --max-cost-usd 5, each in a capped scope./home/kyle/worktrees/host-run-tail-eval/out/<label>-before.jsonand-after.json, labelss1-naming-opus,s1-naming-sonnet,s3-naming-sonnet. Rule:PREREG.mdbeside them. Frozen probes:/home/kyle/worktrees/host-run-tail-eval/probes/naming/name-it-better.json.Done-when
/skill-quality:check, no new FAIL: not covered: no PR head.Scores (hits on both polarities, over probes x runs)
Result
No improvement. The after description scored below the before on sonnet (one negative probe fired in 1 of 3 runs), so under the done-when the before description stays. The gap is one run in 60, which is inside the protocol's 5-point margin; the done-when is stricter than that margin. The opus gate was stopped because it could not change the result. Remaining for this issue: the done-when still lists the probe file, which is not committed, because this run opened PRs only for shipped changes.
Caveats
run-validity.pyhas no trigger mode: a missed positive prints INVALID onskill fired, and an ungranted-toolNo such tool availableerror prints a denial. Both were read per an addendum written before any--runs 3pass, the same for both versions. That addendum fails a pass on a denial of a granted tool, and every other row had to pass.Target labels: add none; remove none.
kyle-sexton commentedon Nov 1, 2026
kyle-sexton commentedon Nov 1, 2026
Progress (lane 3): #12002 merged as e3da95f.
naming:name-it-betterare on main.hands-back-choicepasses a "which ... do/would you" or "which ... to you?" question in two forms: when "which" opens a sentence, or when no comma or semicolon separates it from the ask. A relative-clause "which" in a sentence that applied a rename still fails.names-runners-upno longer needs a prefix.states-blast-radiusis tightened.git rev-parse c339f34953:plugins/naming/evals=1afdcd492c46a804ed37993c19419991c9b543e6.Lease released.
kyle-sexton commentedon Nov 1, 2026
kyle-sexton commentedon Nov 2, 2026
Reopened (lane 3): the #12186 description cut failed a retroactive negative control, so it is being reverted.
Owner rule (2026-11-02): before a cut ships, the suite must score a deliberately broken copy clearly below the before version, or the cut is unmeasurable. #12186 merged before the rule existed, so lane 3 ran the control afterwards.
Setup: the shipped 624-character description against
description: "Helps with naming.", on Sonnet 5.5. Both versions were launched together, and all 12 passes were VALID.The noise report called no routing case drop. The trigger suite's one case drop (neg-val-03, a single false fire out of 3) did not reproduce in a preregistered confirmation run, which was quiet 3/3 on both versions.
FAIL. The suite cannot tell the description from a generic one, so round 1's tie is no evidence the cut is safe.
Likely cause (UNVERIFIED): with only this plugin loaded, the skill and plugin names route naming asks by themselves and nothing competes. A control that can bite needs competing skills such as
docs-hygiene:rename-referencesin the session, whichplugin evalcannot load. A headlessclaude -psession with those plugins enabled can.Next: a revert PR restores the 650-character description. This issue stays open for a round with a competitor-loaded routing suite. Cost of the control: 9.32 USD (list-price estimate, the sum of
costUsd).kyle-sexton commentedon Nov 2, 2026
This was generated by AI.
Negative control with competitor plugins loaded (#12249): PASS
Part: naming
name-it-betterdescription (#6611).Copies.
main:git archiveof origin/mainplugins/at cf60764 (#12300 merged).broken:the same, with only the skill's
description:line replaced by"Helper for repository maintenance tasks.".diff -r main brokenshows that one line only, beforeand after emission.
Suite. The 20 frozen probes in
probes/name-it-better.json(10 should-trigger, 10 should-not),emitted per copy with the merged
measure-invocation.sh emit-plugin-eval. Every case'splugins:line names naming and its competitors' plugins: docs-hygiene, docs-naming, domain-driven-design. The
emitted cases are byte-identical in both copies. Main and broken were launched together on
claude-sonnet-5-5 (Claude Code 2.1.316), each with
--runs 3 --ablation none --disallow-tools Read Glob Grep Agent -j 2. The rule was preregisteredbefore any paid run.
Loading. All 60 with-arm init lines on each side list exactly naming, docs-hygiene, docs-naming
and domain-driven-design, with no
plugin_errors. They also list two Claude Code built-ins(
cc-plugin-agents-md,cc-plugin-plugin-authoring), the same on both sides. run-validity'splugins loadedcheck passes on both passes.Noise report, broken against main (20 cases, margin 0.05):
with-arm change (this minus baseline), paired over 20 cases: -0.32, 95% interval -0.51 to -0.12case drop: case naming-name-it-better-pos-train-01 with-arm fell from 1.00 to 0.00, a third or more; name its cause and prove it before shippingcase drop: case naming-name-it-better-pos-train-03 with-arm fell from 1.00 to 0.33, a third or more; name its cause and prove it before shippingcase drop: case naming-name-it-better-pos-train-04 with-arm fell from 1.00 to 0.00, a third or more; name its cause and prove it before shippingcase drop: case naming-name-it-better-pos-val-01 with-arm fell from 1.00 to 0.00, a third or more; name its cause and prove it before shippingcase drop: case naming-name-it-better-pos-val-02 with-arm fell from 1.00 to 0.33, a third or more; name its cause and prove it before shippingcase drop: case naming-name-it-better-pos-val-03 with-arm fell from 1.00 to 0.00, a third or more; name its cause and prove it before shippingcase drop: case naming-name-it-better-pos-val-04 with-arm fell from 1.00 to 0.00, a third or more; name its cause and prove it before shippingcompare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -0.51 is below -0.05), and 7 case(s) dropped by a third or moreSeven of the ten should-trigger probes dropped, five of them in all three runs. In the missed runs
the traces show no denial or run error: the broken copy answered the naming request directly,
without the skill. Three positives still fired 3 of 3 (
pos-train-02,pos-train-05,pos-val-05), and every should-not probe stayed quiet on both versions.Result: PASS, on the rule's mean criterion: the with-arm mean fell by 0.32, past the 0.15
threshold. That criterion decides on its own, so no confirmation pair was needed for the case drops.
With competitors loaded, the suite tells the shipped description from a generic one, so the #6611
description cut can be measured with this suite. The single-plugin control could not do that (24/24
fired on both versions).
Cost: 5.79 USD (list-price estimate, sum of
costUsdover 2 passes).Next for this issue: the suite can now tell a description from a generic one, so a description cut is measurable. A rewrite ships only on a before/after pass of these probes with the same competitors loaded, with no case drop.