docs(skills): ship the five eval-gated #6916 drift fixes after a no-loss measurement

Slice of #6916.

Problem

The repo-mode /multi-agent:audit-defaults sweep (#6916) confirmed five skill lines whose fix changes what the skill does, or cuts its body. plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md ("Rewriting an existing skill") ships such a change only with eval evidence of no loss. The sweep's fix pull requests run no claude plugin eval, so these were left out of them.

Findings

plugins/planning/skills/draft-goal-condition/SKILL.md:69 restates where the ultracode keyword works. Upstream: workflows#where-the-keyword-works. Fix: cut to a pointer.
plugins/planning/skills/draft-goal-condition/SKILL.md:76 restates plan availability and the xhigh condition. Upstream: model-config#when-ultracode-is-available. Fix: cut to a pointer. Findings 1 and 2 steer a recommendation (they stop the skill routing to a dead ultracode form), and its evals/evals.json has no workflow case, so a case comes first.
plugins/discovery/skills/setup/SKILL.md:61-63 reports a stale unset fork-mode default. Correcting it changes what setup reports. Upstream heading: sub-agents, "Turn fork mode off".
plugins/docs-hygiene/skills/audit-derivability/SKILL.md: the spot-test agent choice. Allowing an omitClaudeMd custom agent would change behavior.
plugins/docs-hygiene/skills/write-for-agents/SKILL.md:139: a skill-body cut.

Line numbers are at origin/main on 2026-11-01. Re-read each against current main first, since #11771, #11774 and #11627 touched these files.

Done when

Each of the four skills has an eval case that exercises the changed line (adding a case where none exists).
Each change is measured before and after on the subscription login, with VALID runs per run-validity.py, and ships only where the comparison shows no loss.
A change that shows a loss is recorded here and left out.
Each shipped pointer carries an as-of date and recheck trigger per .claude/rules/skill-bodies-state-current-rules.md.

I understood this as a slice of #6916. It asks for eval cases on four skills, a before and after measurement of each of the five drift fixes, and shipping only the fixes that show no loss. Each shipped pointer needs an as-of date and a recheck trigger. A fix that loses ground is recorded on the issue and left out.

Nothing is missing. The scope and done-when items are exact. Some steps need the subscription login, so an unattended run may not be able to do them. I found no duplicate among the open issues.

_{Automated triage, applied by the lanes App.}

No description provided.

The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
It added host-run and removed agent-ready: an agent run on a host builds it, and it waits on no decision from a person.

In the run's words:

host-only: the issue's done-when needs each change measured before and after with claude plugin eval on the subscription login, with VALID runs per plugins/evals/skills/plugin-eval/scripts/run-validity.py, and ships only changes that show no loss. This CI lane can run only its named wrapper scripts, so it cannot call claude plugin eval, and it has no subscription login. Shipping the five drift fixes without that measurement is the exact thing plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md ("Rewriting an existing skill") forbids. A host session with the subscription login has to run the measurement.

What the run says it did not do:

Eval cases for draft-goal-condition (ultracode workflow case), discovery:setup, docs-hygiene:audit-derivability and docs-hygiene:write-for-agents: not added, because they only count with before/after claude plugin eval runs that this lane cannot make
Before/after claude plugin eval measurement on the subscription login, with run-validity.py VALID runs, for each of the five findings: needs a host run
The five skill-line fixes (draft-goal-condition:69 and :76, discovery setup:61-63, audit-derivability spot-test agent, write-for-agents:139) and their as-of/recheck pointers: not made, because they ship only after a no-loss measurement
Tidy, review, verify and checks pipeline steps: skipped, with no change to tidy or review

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41271956996

To hand it back to the lane, remove host-run and add agent-ready again.

Claimed by the hitl-backlog host run: building eval cases and before/after claude plugin eval measurements for the five eval-gated drift fixes on branch host-run/11823-eval-gated-drift-fixes.

Host run. Before/after claude plugin eval measurements ran for three of the five eval-gated #6916 drift fixes: the two draft-goal-condition ultracode cuts and the discovery:setup fork default.

What ran (Claude Code 2.1.316, subscription login):

New eval cases: plugins/planning/evals/dg-workflow-keyword-unattended (a Pro-plan cron claude -p run that wants a workflow fan-out) and plugins/discovery/evals/su-fork-default-print-mode (the unset fork default in -p runs). Both use deterministic graders and pass validate-cases.py with no FAIL.
claude plugin eval <plugin copy> --case <case> --ablation none --runs 3 -j 2 --keep-temp, each in a capped systemd-run scope with a short TMPDIR. Before = origin/main skill; after = edited skill. The plugins ran from copies outside the git checkout, because the CLI refuses a plugin whose git directory "registers more linked worktrees than can be screened".
run-validity.py: VALID for all four runs. Warnings: models on all four; skill fired on both setup runs, which invoke the skill by slash command.
noise-report.py --baseline: n too small to call (one case each), so the decision rests on the per-run scores and spot-read traces.

Results against the done-when:

Eval case per skill: added for draft-goal-condition and discovery:setup. Not added for audit-derivability or write-for-agents.
Finding 3 (setup fork default): before 0, 1, 1; after 1, 1, 1. The failed before run gave the right answer ("ON by default"), but on a line that also quoted the skill's stale "off", which the grader rejects. No loss, so it ships as a pointer to sub-agents#turn-fork-mode-on-or-off with an as-of date of 2026-11-01 and a recheck trigger.
Findings 1 and 2 (draft-goal-condition): before 1, 1, 1; after 0.67, 0.67, 0.67. pro-config-row went from 3/3 to 0/3. The eval sandbox denied network to code.claude.com, so the skill could not read its new pointers and declined to answer the Pro /config question. That is a loss, so the cut was reverted and left out. Only the case ships.
Finding 4 (audit-derivability, omitClaudeMd): not run, because of the time box. Upstream supports the change: sub-agents "What loads at startup" says an omitClaudeMd subagent loads only managed policy files. The page's other line, that CLAUDE.md loads "even when ... omitClaudeMd", is about claude --agent, where the field is ignored.
Finding 5 (write-for-agents cut): not run, because of the time box.

PR: #12272 (Refs: #11823, with ## Remaining on #11823). Opened as a draft through /source-control:pull-request create after origin/main was merged in, then marked ready. Before the flip, the verify gate (scripts/check-affected.sh --tests) passed on head bc4384cfb2: 14 passed, 2 skipped. The review lanes raised 5 advisory findings: grader false passes and false fails in both cases, plus a restated /subtask version. All 5 are fixed, replied to and resolved. The two graders were tightened after the measurement runs, so the new patterns were re-applied offline to the kept traces. Setup scores 1,1,1 before and 1,1,1 after. The draft-goal keyword grader scores 1,1,1 before and 1,0,1 after, the after side being the reverted cut. Neither decision changes. Final head aaad7abac5: ci-status success; claude-review and claude-security-review report 0 findings; 0 unresolved threads. Not armed or merged.

Target labels: add none; remove none.

This was generated by AI.

Sliced from #12272's "Remaining on #11823" into sub-issues, all host-run (each needs paid claude plugin eval before/after runs on the subscription login, which a CI issue lane cannot make):

docs(planning): point draft-goal-condition's ultracode lines at the docs without losing behavior offline #12357: findings 1 and 2 (draft-goal-condition ultracode and plan-availability restatements).
feat(docs-hygiene): let audit-derivability's spot test run on an omitClaudeMd agent, measured no-loss #12358: finding 4 (audit-derivability spot-test agent, omitClaudeMd).
docs(docs-hygiene): cut write-for-agents' 'Ask for outcomes, not thinking' paragraph, measured no-loss #12359: finding 5 (write-for-agents "Ask for outcomes, not thinking" paragraph).

Finding 3 shipped in #12272. This issue stays open for the other remaining item: the three behavior changes from #12106 that the #6916 close-out routes here.

Claimed by lane 3: the remaining item, the three #12106 behavior changes in harness-config:audit-instructions (criteria.md inert-set entry for opus-4-8, the per-message effort carve-out, and the SKILL.md bytes-per-token budget). Plan: an eval case per changed line, before/after claude plugin eval with a negative control, then a Refs PR.

Done-when holds for the remaining item (the three #12106 behavior changes in harness-config:audit-instructions). #12926 carried Refs: #11823, so this issue is closed by hand with the record.

Change	Result	Evidence
`criteria.md` I8: `opus-4-8` out of the inert set	Shipped in #12926	Case `ai-throttle-target-scope`, Opus, 3 runs per arm, all VALID: before 0.67, after 1.00, negative control 0.33
`criteria.md` per-message effort carve-out	No change needed	The issue's proposed anchor `#per-message-effort-beta` does not exist; the live page's heading id is `change-effort-mid-conversation-beta`, which criteria.md already cites. The Haiku 5.5 `thinking: disabled` exclusion is already its own bullet.
`SKILL.md` bytes-per-token 3.5 → 2.7	Left out: unmeasurable	The control could still find the old value in `CHANGELOG.md`, so the suite cannot tell a broken copy from a working one (`rewrite-a-plugin.md`). Follow-up with a sound control: #12879.

Pointers carry an as-of date and recheck trigger per .claude/rules/skill-bodies-state-current-rules.md; a review round on #12926 removed one restated upstream rationale. Both Claude review lanes and Codex reported no open findings on the merged head. Spend 6.93 USD list across 33 runs.

Navigation Menu

Repository navigation

docs(skills): ship the five eval-gated #6916 drift fixes after a no-loss measurement #11823

Problem

Findings

Done when

Related

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

docs(skills): ship the five eval-gated #6916 drift fixes after a no-loss measurement #11823

Description

Problem

Findings

Done when

Related

Activity

github-actions commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions