Concise skills rollout: playgrounds (1 skills, 1 model-invoked)

Part of #6476. Rewrite playgrounds's skills to the skill criteria in one draft PR, following the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). The runbook and the rewrite protocol it cites decide what ships; this issue holds only what is specific to the plugin.

Path: description-only, measured by trigger probes (owner decision on #6476, 2026-10-18: the 1-skill tail takes the description-only path). The body is not rewritten.

The plugin today

As of origin/main on 2026-10-24:

Skills: use (model-invoked).
Outcome eval case directories under the plugin: 0. Probe files under probes/: 0.

Done when

git diff origin/main -- plugins/playgrounds/skills changes no line outside the description key of plugins/playgrounds/skills/use/SKILL.md.
plugins/playgrounds/probes/use.json exists in the PR.
The PR body shows /skill-quality:check measure-invocation trigger scores for the before and the after description on the same probes.
The after description scores at least the before score, or the PR ships the before description and says so.
/skill-quality:check plugins/playgrounds on the PR head reports no FAIL that the run on main does not.
The plugin's version or changelog entry is in the PR, per the plugin's release mode.

Eval spend and review

Eval passes within prepaid API credit are pre-approved (owner decision Q46, 2026-10-24), at the --max-cost-usd 5 per-pass ceiling set on #6476 on 2026-10-18.
No person gates the PR: the owner's 2026-10-24 decision (C12) leaves only an intent mismatch during the build or a security issue at review as stops.

Generated by Claude Code

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none

Understood: rewrite only the description of the playgrounds:use skill, add probes/use.json, and show before/after trigger scores. Also add the version or changelog entry. The issue has no open questions, and the runbook and owner decisions it cites set the approach. I found no likely duplicates.

_{Automated triage, applied by the lanes App.}

This was generated by AI during triage.

Held: this one-skill plugin is in the description-only tail the owner set on #6476 (2026-10-18), measured by paid trigger probes in the attended eval session, not by the lane.

Released by the owner's HITL session (2026-10-24, C12): it waited only on a removed gate. Intake triage re-runs on it to set its fields, then admission takes it.

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

The work-resolve-issue lane stopped partway through this issue and needs a person.

Why, in the run's own words:

Two of the issue's Done-when items need measurements this lane cannot run. (1) The PR body must show /skill-quality:check measure-invocation trigger scores for the before and after descriptions on the same probes. (2) /skill-quality:check plugins/playgrounds must be compared between the PR head and main. Both need plugins/skill-quality/scripts/measure-invocation.sh (and the check skill), and on the description-only path a live plugin-eval or claude -p run, because plugins/skill-quality/reference/invocation-probes.md says the listing-overlap floor is not a model-graded rate. This lane loads only the work-items plugin and runs only the named wrapper scripts, so neither measurement can run here. The 2026-10-19 triage comment also routes this measurement to the attended eval session. Without the scores, the issue's keep-or-replace rule for the description cannot be applied, so I made no edits instead of shipping an unmeasured description or an unvalidated probe file.

What the run says it did not do:

Writing plugins/playgrounds/probes/use.json: not done, because measure-invocation.sh validate cannot run here to check its schema, polarity and copy-span rules, and the probes are only useful together with the measurement
Before/after trigger scores via /skill-quality:check measure-invocation and a plugin-eval run: the skill-quality plugin and its scripts cannot run in this lane
/skill-quality:check plugins/playgrounds on main and on the PR head: cannot run in this lane
Release record (changelog fragment) for playgrounds: no plugin change was made, so none is owed yet
Fresh-context review of the diff: skipped because there is no diff

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40195212155

To retry, remove needs-human and add agent-ready again.

This was generated by AI.

Parked for capacity, not for the owner. This rollout's remaining work is the measured eval passes of the batch-1 protocol, which the issue lane cannot run: it cannot start claude or claude plugin eval (see the lane's escalation on #6585, https://github.com/melodic-software/claude-code-plugins/actions/runs/40186335064). It waits on the plugin eval dispatch work in #9704, now a blocked-by edge, so admission holds it until #9704 closes.

Under the owner's 2026-10-26 stop rule, a person is needed only for a change in intent or vision, an implementation finding that changes the intent or primary design, or a high or critical security issue; host, attended and eval runs are agent work. So needs-human comes off, and the issue moves to host-run with status: confirmed, which keeps the issue lane from taking it when #9704 closes: #9704 adds an eval dispatch workflow, not eval capability for the issue lane.

Host run. Description-only trigger screen for playgrounds:use: before and after copies launched together, frozen probes.

What ran

Claude Code 2.1.316, base a781adf89a, models claude-opus-5-5 and claude-sonnet-5-5, --ablation none, -j 2, cases from measure-invocation.sh emit-plugin-eval, 20 probes (10 positive, 10 negative), digest 79ba59f7d95a.
Description length: 972 to 839 characters (13.7%). Probes were written by a fresh agent from the before description. The rewriter was barred from reading probes and evals, and kept every quoted trigger phrase.
Rule written before any after-version result: --runs 1 screen, then --runs 3, where after must score at least before on every model (the done-when), or the before description ships.
Meaning diff: two labelers, union of losses. PASS from both.
Command in short form: claude plugin eval <copy> --eval-dir trigger --model <model> --ablation none --runs N -j 2 --keep-temp --max-cost-usd 5, each in a capped scope.
Result files: /home/kyle/worktrees/host-run-tail-eval/out/<label>-before.json and -after.json, labels s1-playgrounds-opus, s1-playgrounds-sonnet. Rule: PREREG.md beside them. Frozen probes: /home/kyle/worktrees/host-run-tail-eval/probes/playgrounds/use.json.

Done-when

Description-only diff: not covered, no PR. The run copies differed only in the description.
Probe file in the PR: not met: frozen outside the repository, no PR.
Before and after scores in the PR body: not covered, no PR. They are in this record.
After at least before, or the PR ships the before description: not covered: no readable pass yet.
/skill-quality:check, no new FAIL: not covered: no PR head.
Changelog entry: not met, no PR.

Scores (hits on both polarities, over probes x runs)

Pass	Model	Before	After
screen 1 run	opus	19/19, unreadable	19/19, unreadable
screen 1 run	sonnet	20/20, unreadable	20/20, unreadable

Result

Unchanged on main. Neither model produced a readable pass, so nothing here supports a change. Remaining: rerun the screen under a rule set in advance for denials the skill's own presence check causes, then the gate, then the PR with the probe file.

Caveats

run-validity.py has no trigger mode: a missed positive prints INVALID on skill fired, and an ungranted-tool No such tool available error prints a denial. Both were read per an addendum written before any --runs 3 pass, the same for both versions. That addendum fails a pass on a denial of a granted tool, and every other row had to pass.
The eval loads one plugin, so these numbers show no rival skills from other plugins. A score at the ceiling cannot show an improvement.
No separate held-out set ran: the rewriter never saw these probes and cut no quoted phrase.
The first pass was void: the session listed only built-in plugins because the manifest depends on playground@claude-plugins-official, and the CLI printed no warning, so positives scored 0/10 in both versions. Dropping the dependencies key from both run copies fixed it (10/10). The upstream skill is therefore absent from these numbers.
neg-val-01 timed out at 300s in both versions on opus and is excluded from both, per the written rule.
Opus before and both sonnet versions logged a Glob denial, a granted tool, in positive runs looking for the upstream plugin. The written rule fails those passes, and opus after also fails run errors, so each score is marked unreadable.

Target labels: add none; remove none.

No description provided.

Progress (lane 3): #12036 merged as b6f80cf.

Outcome eval cases and a frozen trigger-probe file for playgrounds:use are on main.
Grader fixes made in review:
- no-workaround no longer treats a dot inside a URL or path as the end of a sentence.
- A negation now covers a workaround only when it comes before it.
- no-playground-skill also rejects a bare {"skill":"use"} call.
Frozen suite digest: git rev-parse 6c583909da:plugins/playgrounds/evals = 2d76993c3774ceb6f7c6b2fbda81906f23378615.
This issue stays open for its measured before/after pass, which is the concise rewrite. The validity defects that held that pass back (fix(evals): let trigger-measurement probe runs pass run-validity #11618, fix(skill-quality): stop invocation-probe runs failing validity on denied searches outside the workspace #11624, test(visualization): fix the frozen suite's graders and re-measure present and visualize with VALID runs #11628) are closed.

Lease released.

No description provided.

Host run (lane 3): measured description-only pass for playgrounds:use. Nothing ships; the before description stays on main.

What ran

Claude Code 2.1.316, base fd0c36f751, models claude-opus-5-5 and claude-sonnet-5-5, --ablation none (with-arm only), before and after copies launched together per pass, -j 1 each.
Frozen suite: plugins/playgrounds/evals tree 2d76993c37 (test(playgrounds): add outcome eval cases and frozen trigger probes for use #12036). Run-copy digest re-checked before every pass.
Description 972 to 907 characters. A fresh rewriter worked on a copy outside the repository, barred from every eval and probe file. It made six edits: it cut "in one step" and "field-tested", tightened two connective phrases, and dropped "owns that" from both exclusions. Every quoted trigger phrase and both exclusion owners were kept.
Meaning diff: two fresh /docs-hygiene:compress compare labelers, both PASS, with 0 SEMANTIC LOSS and 0 AMBIGUITY.
Held-out trigger set: 16 queries (8 positive, 8 negative), written blind from a neutral statement of purpose by a print session with skills and tools disabled. They were emitted with measure-invocation.sh emit-plugin-eval --runs 3 and run with --disallow-tools Read Glob Grep Agent.
Routing cases:
- use-routes-natural-explorer (must route; Write granted, run under the docker link swap)
- use-boundary-visual-form (must not route)
Each ran 5 times per version.
Not run: the three slash-invoked cases. A /playgrounds:use prompt never reaches description routing, and the body is byte-identical.
The manifest dependencies key was dropped from both run copies, per the 10-31 screen.
A host-safety probe ran before the Write grant and passed: Windows drives listed empty, Windows binaries were unreachable, WSL_INTEROP was unset, and no canary write landed.
The decision rule was preregistered before the first paid run. The after description ships only if every rule holds on both models: held-out trigger at or above before with no lost trigger, both routing cases VALID, routing within 1 run, mean within 0.05, no case drop, and no new must-not failure.

Scores (VALID passes only)

Pass	Model	Before	After
Held-out trigger: should-trigger fired	Opus	24/24	24/24
Held-out trigger: should-not quiet	Opus	24/24	24/24
Held-out trigger: should-trigger fired	Sonnet	21/24	21/24
Held-out trigger: should-not quiet	Sonnet	23/24	24/24
`use-boundary-visual-form` (mean, 5 runs)	Opus	1.000	1.000
`use-boundary-visual-form` (mean, 5 runs)	Sonnet	1.000	1.000
`use-routes-natural-explorer` (mean, 5 runs)	Opus	unmeasured	unmeasured
`use-routes-natural-explorer` (mean, 5 runs)	Sonnet	1.000	1.000

The noise report calls the held-out trigger result non-inferior at a 0.05 margin on both models (lower ends +0.00 on Opus and -0.02 on Sonnet). Every case pair is "n too small to call", and no case drop line appeared.
Sonnet's only missed positive is the same query in both versions, 0/3 each. It asks for a clickable architecture map of an inherited repository, and every reply says the repository has to be read first. The one before-only false fire was a prototyping near-miss ("mock up three layouts ... in the real code"). The after version stayed quiet on it in all 3 runs.
On Sonnet, the routing case routed 5/5 in both versions, with 0 no-write failures.

Result

The before description ships, so main is unchanged. Rule 2a failed on Opus: use-routes-natural-explorer was never VALID on the after side.

In the first pair, before was VALID and after was INVALID (1 denial).
The preregistered terminal paired rerun gave the same result: before VALID, after INVALID (1 denial).
Each denial was the body's Step 1 presence check searching the host's ~/.claude/plugins for the upstream install (Glob, then Grep), made after the graded Skill call.
run-validity.py counts a with-arm host-home denial as a FAIL unless every run in both arms was denied the same call.
No number from an INVALID pass was read.

Every other deciding rule that was read held on both models. The part is due a second round, so this issue stays open.

Done-when

Description-only diff: no PR, because nothing ships. The run copies differed only in line 2.
plugins/playgrounds/probes/use.json exists: met on main (test(playgrounds): add outcome eval cases and frozen trigger probes for use #12036).
Before and after trigger scores: in this record (held-out set). The frozen probe set was not model-run this round. Its role was report-only and could not change the result. The free lexical score on it was before 1.0 trigger / 0.71 false (train) and 1.0 / 1.0 (validation), and after 1.0 / 0.43 and 1.0 / 0.33. That score is a tripwire, not the verdict.
After at least before, or ship before: before ships. The deciding routing case went unmeasured on Opus.
/skill-quality:check: no PR head. On main: PASS, 0 errors, 1 warning (description 972 > 419).
Changelog entry: none owed, because no plugin change was made.

For round 2

Harness defect: use-routes-natural-explorer grants Glob and Grep, and the presence check sometimes searches the host's ~/.claude.
- Pooled over both models, presence-check denials hit 6 of 15 after-side routing runs and 1 of 15 before-side runs. The two FAILs were both Opus after-side.
- On Sonnet, the denials scored as warnings (1 denied run before, 4 after), and every denied run scored the same as the denial-free runs.
- The body that holds the presence check is identical in both versions. That the shorter description makes Step 1 search the host more often is an open hypothesis, not a finding: on Opus alone (about 1 denied run in 5 per attempt) the pattern cannot be told apart from chance.
Fix: prefer a run-validity.py carve-out for a host-home presence probe made after the Skill call, and count presence-check calls per version from the traces. A blanket --disallow-tools Glob Grep would hide the difference above. Either way, apply the rule from the first pair.
The Opus routing case's before side scored 0.800 in the first pair and 1.000 in the rerun on identical text, so 5 runs per version is a noisy deciding case.
Measure the same after text again (six edits, two-labeler PASS) with a fresh held-out set. This round's set is spent.
Behavior defect on main, for triage: in the first Opus pair, main's text (the before side, a VALID pass) routed and then hand-wrote the HTML page into the run directory in 3 of 5 runs (no-write failed). That contradicts the body's "never builds a playground itself". The before rerun did not repeat it.

Cost

12.59 USD list-price estimate (sum of costUsd) over 15 passes, including the probe and the INVALID passes. Every pass stayed under the 5 USD per-pass ceiling.

Target labels: add none; remove none.

Navigation Menu

Repository navigation

Concise skills rollout: playgrounds (1 skills, 1 model-invoked) #6612

The plugin today

Done when

Eval spend and review

1 remaining item

What ran

Done-when

Scores (hits on both polarities, over probes x runs)

Result

Caveats

What ran

Scores (VALID passes only)

Result

Done-when

For round 2

Cost

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Concise skills rollout: playgrounds (1 skills, 1 model-invoked) #6612

Description

The plugin today

Done when

Eval spend and review

Activity

github-actions commented on Oct 7, 2026

kyle-sexton commented on Oct 19, 2026

kyle-sexton commented on Oct 24, 2026

github-actions commented on Oct 25, 2026

1 remaining item

melodic-automation-lanes commented on Oct 25, 2026

kyle-sexton commented on Oct 27, 2026

kyle-sexton commented on Oct 31, 2026

What ran

Done-when

Scores (hits on both polarities, over probes x runs)

Result

Caveats

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

What ran

Scores (VALID passes only)

Result

Done-when

For round 2

Cost

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions