Concise skills rollout: debugging (1 skills, 1 model-invoked)

Part of #6476. Rewrite debugging's skills to the skill criteria in one draft PR, following the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). The runbook and the rewrite protocol it cites decide what ships; this issue holds only what is specific to the plugin.

The plugin today

As of origin/main on 2026-10-24:

Skills: analyze-profile, debug (model-invoked).
Outcome eval case directories under the plugin: 15. Probe files under probes/: 1.

Done when

/skill-quality:check plugins/debugging on the PR head reports no FAIL that the run on main does not.
/evals:validate passes on plugins/debugging's eval suite at the PR head.
Each model-invoked skill has a probe file: plugins/debugging/probes/analyze-profile.json, plugins/debugging/probes/debug.json.
The PR body holds a per-skill decision table, description and body decided separately, with the before and after numbers behind each row.
The PR body names every skill or case left unmeasured, with the reason.
The plugin's version or changelog entry is in the PR, per the plugin's release mode.

Eval spend and review

Eval passes within prepaid API credit are pre-approved (owner decision Q46, 2026-10-24), at the --max-cost-usd 5 per-pass ceiling set on #6476 on 2026-10-18.
No person gates the PR: the owner's 2026-10-24 decision (C12) leaves only an intent mismatch during the build or a security issue at review as stops.

Generated by Claude Code

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none

Understood: rewrite the two skills in the debugging plugin (analyze-profile, debug) to the skill criteria in one draft PR, following the per-plugin runbook. The PR needs probe files, a passing eval validation, a per-skill decision table, and a version or changelog entry. The issue already states the eval spend approval and the review rules, so nothing is missing. It is part of #6476. I found no duplicates among the open issues.

_{Automated triage, applied by the lanes App.}

Released by the owner's HITL session (2026-10-24, C12): it waited only on a removed gate. Intake triage re-runs on it to set its fields, then admission takes it.

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

Claimed by the Backlog HITL host session; work in progress.

Released: this row belongs to laptop Lane 2/3; the Backlog HITL session has stopped work on it.

test(debugging): keep only the frozen debug probes; defer the body re…

This was generated by AI.

Progress: #11849 merged (543eb6c). It adds the missing frozen trigger probe file plugins/debugging/probes/debug.json (20 queries, 10 positive and 10 negative). No skill text changed.

The concise debug body rewrite that #11849 first carried was not shipped. It is kept in commit 352a71b on branch host-run/6603-debugging-concise for the rewrite to start from. Its measured passes ran on Opus only: 3 runs, --ablation none, every result VALID. Words went from 5292 to 5091, and the with-plugin mean from 0.83 to 0.87 over 11 cases, with no case dropping. Per case, before → after:

multiple-falsifiable-hypotheses 0.33 → 0.67
no-blind-retry-as-fix 0.83 → 0.83
no-correct-seam-is-a-finding 1.00 → 1.00
no-loop-possible-is-the-deliverable 0.78 → 0.78
observed-failure-builds-loop-first 1.00 → 1.00
offers-built-post-mortem-view 0.33 → 0.33
cleanup-gate-removes-tagged-instrumentation 0.86 → 0.91
debug-hands-artifact-to-analyze-profile 1.00 → 1.00
the three red-run cases 1.00 → 1.00

It did not meet the rewrite protocol, which measures per model the skill targets and ships on non-inferiority within 5 points (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:212-222). No Sonnet run was made, and the read-only group's noise interval reached the margin (+0.06 [-0.05, +0.16]).

This issue stays open. The rewrite and its measured passes wait on the eval validity fixes in #11618 and #11628.

No description provided.

Host run. #6603 measured a concise rewrite of the debug skill body on Opus 5.5 and Sonnet 5.5. It held on Sonnet but showed a case drop on Opus that restoring the related text did not close. No text ships, no PR was opened, and the issue can close with no text change.

Result

Words shipped: 0. plugins/debugging/skills/debug/SKILL.md stays at 5292 words (367 lines).
Words tried:
- The body went from 5292 to 5154 words (138 cut, 2.6%; 367 to 361 lines).
- The cuts were judgment prose only: the git-context rationale, Purpose, Scope, the running-example lead-in, "Adapting to your environment", the secrets intro line, the loop-recursion hazard, cold-vs-warm timing, and the outcome-verifier bullet.
- Contract sections were left byte-identical: the --unattended handling, the Emit checklist, Boundary and Routing, and the "Optional view" paragraph.
Not rewritten (before text ships): both descriptions, the analyze-profile body, and every spoke. The frontmatter was byte-identical, so no trigger pass was run. A 16-query held-out set (8 should-trigger, 8 should-not) was written blind and stays unspent for a later description round.
No-loss verdict: fail, on Opus.
- Sonnet: rules held.
- Opus: multiple-falsifiable-hypotheses fell from 0.83 to 0.50, which meets the preregistered one-third drop.
- Restoring the three cut hunks that bear on that case (git context, Purpose, Scope) did not close the gap: a fresh pair read 0.83 before and 0.33 after.
- By the preregistered rule, the before body ships.

What ran

Claude Code 2.1.316, claude plugin eval on before and after copies outside the repository.
Flags: --trust-plugin --keep-temp --judge-model opus --threshold 0.8 --runs 3 -j 2 --scaffold --max-cost-usd 5 --no-publish, default ablation (with and without arms).
Each case ran as its own pair, before and after launched together. --allow-tools granted exactly the gated tools each case lists: Bash, Edit and Write for cleanup-gate and the three red-run cases; Bash only for debug-hands-artifact; none for the rest.
Cases: the 11 that load the debug body. The 4 analyze-profile-only cases were not run, since that text did not change.
run-validity.py (with the fix(evals): let trigger-measurement probe runs pass run-validity #11618, fix(skill-quality): stop invocation-probe runs failing validity on denied searches outside the workspace #11624 and test(visualization): fix the frozen suite's graders and re-measure present and visualize with VALID runs #11628 fixes) ran on every pass.
Before the first after-version run:
- Judge calibration: 7 of 9 llm graders agreed at 100%. hypothesis-and-grep judges the trace, which a calibration run cannot reproduce, so it stays uncalibrated. ranked-falsifiable-grounded's must-pass sample was not reproduced.
- A one-run smoke pass, whose failures went to a fresh classifier.
- One grader widening (below), then the suite was frozen.
Meaning diff: two fresh docs-hygiene:compare-labeler agents both returned PASS, with one shared ambiguity, which was restored. A recheck returned PASS with 0 ambiguity.
Every reading rule was written into the preregistration before the passes it governs ran.

Pair set	Gate	Use
Opus, 11 cases	22 of 22 passes VALID	read
Sonnet, 10 cases	20 of 20 passes VALID	read
Sonnet, offers-built-post-mortem-view	INVALID on both versions, same denial (a Glob of the plugin root)	unmeasured
Opus, multiple-falsifiable restore pair	VALID / VALID	decides: before ships

Mean with-arm scores, before to after (3 runs each):

Case	Opus	Sonnet
`cleanup-gate-removes-tagged-instrumentation`	0.86 to 0.90	0.62 to 0.62
`debug-hands-artifact-to-analyze-profile`	1.00 to 1.00	1.00 to 1.00
`multiple-falsifiable-hypotheses`	0.83 to 0.50 (restore pair: 0.83 to 0.33)	0.00 to 0.33
`no-blind-retry-as-fix`	0.83 to 1.00	1.00 to 1.00
`no-correct-seam-is-a-finding`	0.83 to 1.00	1.00 to 1.00
`no-loop-possible-is-the-deliverable`	0.78 to 0.67	0.89 to 0.89
`observed-failure-builds-loop-first`	0.83 to 1.00	1.00 to 1.00
`offers-built-post-mortem-view`	0.33 to 0.33	not measured
`red-run-before-suggested-fix`	1.00 to 1.00	1.00 to 1.00
`red-run-crash-on-empty-team`	1.00 to 1.00	1.00 to 1.00
`red-run-wrong-discount-total`	1.00 to 1.00	1.00 to 1.00
Mean over read cases	0.85 to 0.86	0.85 to 0.88

Why it fails

One case drops on Opus, and the restore did not bring it back.
- The aggregate mean holds on both models (Opus +0.01, Sonnet +0.03, inside the 5-point margin).
- The case rule does not hold: multiple-falsifiable-hypotheses fell by a third on Opus. Restoring the cut text that bears on it left a 0.50 gap.
- Over 6 runs per version, before read 1, 1, 1, 1, 0.5, 0.5 and the two after texts read 1, 0.5, 0.5, 0.5, 0, 0.
- The Phase 3 text the case checks was never edited, so the cause is not identified. It could be noise on a defective case (next section) or an effect of the remaining cuts. The protocol keeps the before text when no restore closes the gap.
The meaning diff did not predict it. Three labelers found no semantic loss in the final text.

Findings for the next round

Two case defects make debug cases noisy. The workspace in both is empty.
- multiple-falsifiable-hypotheses says "I already have a failing test", but the workspace holds no code, test or git history. Replies that decline to invent causes fail ranked-falsifiable-grounded.
- offers-built-post-mortem-view asks for the view of a finished post-mortem that does not exist, and the user has already asked for the view, so there is nothing to offer. It scores 0.33 on both versions and both models.
- Fix both cases (seed the code, test and history, or the post-mortem) before measuring another round.
- In the empty workspace, ranked-falsifiable-grounded passed a before reply whose ranked table of generic exporter causes had no repo grounding: there were no commits to cite. So whether this case passes depends on whether the reply happens to include a generic table, not on grounding. That fits the drop being noise on this case, but it was not tested.
Grader fix, not landed. no-loop-possible-is-the-deliverable/says-no-loop rejected "couldn't build a reproduction loop", a reply that meets the case. The widening (could(n't| not), curly apostrophes, plus a must-pass sample) was proved on the observed reply and all must-fail samples and used in this run. Nothing ships, so it stays off main. Land it with the case fixes.
hypothesis-and-grep cannot be judge-calibrated as written, because its focus is the trace and a calibration run has no tool calls. Calibrate it from labeled real traces, or split it into a reply grader and tool_used graders.
Sonnet's offers-built-post-mortem-view with-arm Globs the plugin root, and the sandbox denies it on both versions. This is unrelated to the rewrite.

Spend

52.61 USD list-price estimate, subscription login:

2.50 judge calibration
5.66 smoke
28.52 Opus pairs
14.64 Sonnet pairs
1.29 restore pair

That is 48 pass invocations, run 18:15 to 18:59 UTC.

Remaining

The rewritten text, the restore variant and the preregistration with its Run log are kept off-repo on the lane-3 host.
Branch feat/6603-concise-debugging carries no commits and was not pushed.
A next round would need two things:
- Fix the two case defects first and land the grader widening.
- Re-cut only the hunks with no behavioral load, and decide on fresh pairs.

The issue can close with no text change.

Target labels: add none; remove none.

Closing on the measured result above: the rewrite failed the no-loss rule on Opus, so the current text stays.

Reopening: the measured result above rests on one Opus case, multiple-falsifiable-hypotheses, and the same pass found that case defective. It claims an existing failing test in an empty workspace. offers-built-post-mortem-view is defective too, and says-no-loop needs the widening proven in that pass. Next: land those suite fixes, then run a second measured round on the fixed suite. The first round's result stays on record above.

revert(debugging): keep main's debug body; the cut failed its negativ…

Host run (lane 3): the round-2 debug body cut measured with no loss but failed the new negative control, so main's body stays.

Round 2 (before vs after, launched together): Opus 0.851 → 0.829, Sonnet 0.780 → 0.859, every case read, no case fell by a third.

Negative control (owner rule, 2026-11-02)

The broken copy is the shortened body with every line under its six ## Phase N sections deleted (361 → 138 lines; headings, frontmatter and other sections kept). It ran on Sonnet 5.5 against a fresh pass of main's body launched with it, on the same 11 outcome cases, judge opus, threshold 0.8, 5 runs.

With-arm mean over the 10 read cases: 0.790 before vs 0.684 control, a drop of 0.106 against the 0.15 required.
The noise report, over the 10 cases merged per side, called one case drop: no-loop-possible-is-the-deliverable, 1.00 → 0.60. It is a grader artifact. All 5 control replies say "no reproduction loop can be built" and do what the case asks, but the says-no-loop grader matches only the active wording ("can't build a loop").
debug-hands-artifact-to-analyze-profile was INVALID twice on the control side and is not counted.

FAIL: the suite cannot tell the body from a copy with its phases removed, so it does not show the cut is safe.

What lands

#12208 now carries only the states-predictions grader fix: table-column predictions are credited, and a backtracking stall on long tables is fixed. The body change is reverted there. This issue stays open for a round with outcome cases that depend on the phases.

Note: earlier lane-3 no-loss checks compared one case per results file, and noise-report.py cannot call a case drop below 3 cases. The control above merged the cases first.

Cost: 56.01 USD for round 2 and 16.59 USD for the control (list-price estimates, the sum of costUsd).

Closed under the owner's Q2 decision, which stops the concise-rewrite series: #6476 (comment). The debug body cut failed its negative control, so main's body stays; the eval-case fixes already merged stay.

Navigation Menu

Repository navigation

Concise skills rollout: debugging (1 skills, 1 model-invoked) #6603

The plugin today

Done when

Eval spend and review

9 remaining items

Result

What ran

Why it fails

Findings for the next round

Spend

Remaining

Negative control (owner rule, 2026-11-02)

What lands

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Concise skills rollout: debugging (1 skills, 1 model-invoked) #6603

Description

The plugin today

Done when

Eval spend and review

Activity

github-actions commented on Oct 7, 2026

kyle-sexton commented on Oct 24, 2026

github-actions commented on Oct 25, 2026

9 remaining items

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

Result

What ran

Why it fails

Findings for the next round

Spend

Remaining

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

Negative control (owner rule, 2026-11-02)

What lands

kyle-sexton commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions