feat(docs-hygiene): add a grounding rule to /docs-hygiene:write-for-humans

Outcome

/docs-hygiene:write-for-humans grounds each concept before a sentence relies on it. The #12025 probe showed the gap: with the skill loaded and fired, the wfh-grounding-readme case failed the terms-grounded grader in 3 of 3 runs, the same as without the plugin (Claude Code 2.1.316).

Done when

The skill states the rule: introduce each concept before a sentence relies on it, or name it as knowledge the reader is assumed to have.
Over 10 runs on main's rule, wfh-grounding-readme's with-plugin terms-grounded pass rate is at least 0.6 and at least 0.3 above the without-plugin rate, and wfh-grounding-howto still passes, recorded with run-validity.py VALID. (Owner decision 2026-11-02 ~19:2xZ, replacing "3 of 3 runs": at n=3 the measured changes could not be told apart.)

Out of scope

Other write-for-humans criteria.

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Small
Complexity: Low
Uncertainty: Knowable
Flags: none
Blocked by: none

Understood: #12098 added the grounding rule to the skill, so done-when 1 is met. Done-when 2 is still open. The latest probe shows wfh-grounding-readme failing terms-grounded 0/3 in the with-plugin arm, while the how-to case passes. The likely cause is that the rule only names project-specific terms, which is unverified. The remaining work is to widen the rule's scope, or require a definition at or before first use, then re-probe. The re-probe may need a run outside this lane. No likely duplicates found.

_{Automated triage, applied by the lanes App.}

No description provided.

The lanes App's pull request #12098 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.

Probe on main f7dde4f (Claude Code 2.1.316, VALID): the rule from #12098 is in the skill (SKILL.md:51), but wfh-grounding-readme still fails terms-grounded 0/3 in the with-plugin arm; how-to passes 3/3. Every with-arm README leaves collector undefined, and uses checkpoint one sentence before defining it. The likely cause is that the rule names only "project-specific" terms (UNVERIFIED). Next step: widen the rule's scope, or require the definition at or before first use, then re-probe. Done-when 1 is met; done-when 2 is not.

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

Not admitted to the issue lane: merged pull request #12098 names it with Refs: after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is **Decision recorded.** saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

Claimed by a Backlog HITL host-run worker: changing the grounding rule in /docs-hygiene:write-for-humans so a term is defined at or before its first use, then re-probing wfh-grounding-readme. Branch host-run/12085-wfh-grounding-first-use. Time box 90 min.

Host run. Reworded the grounding rule's test in /docs-hygiene:write-for-humans and re-probed the wfh-grounding-* eval twice. Verdict: improved, not done. The README with-plugin arm went from 0/3 to 2/3 in both probes, and the done-when needs 3/3.

What changed (branch host-run/12085-wfh-grounding-first-use)

plugins/docs-hygiene/skills/write-for-humans/SKILL.md:51: the rule title is unchanged, so it still matches done-when 1 word for word. The test column now says three things:
- read as someone new to the system;
- a common word that names one specific thing ("the broker", "the vault") counts as a term to introduce;
- the definition goes in the first sentence that uses the term or an earlier one, never the next, because notes often define a term after using it.
The example column gains a negative case: "Run stash export … A vault is a folder…" puts the definition one sentence late. The example words avoid the eval's own terms.
.changes/docs-hygiene/host-run-12085-wfh-grounding-first-use-5d81c3a2.md, bump: patch.
The stumble evidence is in the commit messages, not the skill body. It is the test(docs-hygiene): run the write-for-humans grounding eval and post its probe result #12025 probe on main f7dde4f0e4: VALID, with-arm 0/3. Every with-arm README left collector undefined and used checkpoint one sentence before defining it.

What ran

Claude Code 2.1.316 on the subscription login. Model claude-opus-5-5, pinned with --model (12 of 12 traces in each probe). The judge was the CLI default, 3 votes per verdict.
Each probe ran from a git archive of the committed head, extracted outside any git repository. Each was one suite in a capped claude.slice scope with a short on-disk TMPDIR. The running-suite count was 0 at each launch.
The command matches the test(docs-hygiene): run the write-for-humans grounding eval and post its probe result #12025 baseline: claude plugin eval <copy>/docs-hygiene --case 'wfh-grounding-*' --ablation with-without -j 2 --model claude-opus-5-5 --disallow-tools Agent Task Bash --max-cost-usd 20 --trust-plugin --no-publish --keep-temp, at each case's runs: 3.
Ceiling --max-cost-usd 20 per probe. The CLI reported $1.39 and $1.41 notional spend, with no partial result.
The case files and graders were not changed.

Scores (terms-grounded grader, passes over runs; n = 12 runs per probe)

Probe	Head	README with	README without	How-to with	How-to without	`run-validity.py`
Baseline (#12025, main `f7dde4f0e4`)	old rule	0/3	0/3	3/3	3/3	VALID
1	`2ffea86411` (reworded test)	2/3	0/3	3/3	3/3	VALID
2	`94a81df5da` (plus a post-draft first-use check)	2/3	1/3	3/3	3/3	VALID

Validity warnings in both probes: the judge model is not recorded; the how-to without-arm is at the 1.00 ceiling, so it is left out of the delta; and there are split votes. Probe 1 split on README with-arm run 3 (FAIL PASS PASS, a pass) and without-arm run 1. Probe 2 split on README with-arm run 1 (PASS FAIL FAIL, a fail) and without-arm run 1.
The skill fired in 6 of 6 with-arm runs in each probe.
Hand check of the failing with-arm READMEs. Both make the same slip in the opening paragraph:
- probe 1: "sends each new line to the collector. The collector is the central log service…";
- probe 2: "sends each new line to a collector. The collector is the log service…".
checkpoint was defined at or before first use in the probe 1 failure. In probe 2, a heading used checkpoint before the body defined it.
The post-draft check (probe 2) did not move the with-arm score. It was reverted in 225851286b, so the branch tree equals probe 1's head 2ffea86411. A standing line that does not change behavior does not ship.
I stopped after the one planned revision. Rerunning the same wording until a probe hits 3/3 would only select a lucky sample.

Done when (#12085)

Met: the skill states the rule. The title at SKILL.md:51 is unchanged ("Introduce each concept before a sentence relies on it, or name it as assumed knowledge"), and the test now requires the definition at or before first use.
Unmet: wfh-grounding-readme passes terms-grounded in the with-plugin arm in 2 of 3 runs, not 3 of 3 (probe 1, VALID; probe 2 repeats 2/3). wfh-grounding-howto still passes 3/3, and run-validity.py gave VALID.

Options for the remaining gap (decided from evidence, no owner decision needed)

Land the reworded test as is, with Refs: #12085. It moves the README with-arm from 0/3 to 2/3, and the how-to case does not regress. Recommended. Basis: probes 1 and 2 above.
Try a different lever next. Candidates are an opening-paragraph-specific example in the rule's example column, or a larger runs count to measure the rate more tightly than n=3. Either is a new probe, not a rerun of this wording.
Treat 2/3 as the model's floor for this case and revise done-when 2 to a rate threshold. That changes the issue's contract, so the coordinator or owner decides it.

The skill-creator case grounds-each-concept-before-relying-on-it (skills/write-for-humans/evals/evals.json) is not run by claude plugin eval and was not probed.

Branch: host-run/12085-wfh-grounding-first-use, from main fd0c36f751, head 225851286b. Commits: 2ffea86411 (reworded test), 94a81df5da (first-use check), 225851286b (revert of the check). scripts/check-affected.sh --tests gave 14 passed, 2 skipped on 2ffea86411 and on 94a81df5da. The head tree is identical to 2ffea86411. Pushed. No PR is open.

Target labels: add none; remove none.

Taken over by lane 1 (melo-lap-001, hub assignment 2026-11-02 16:5xZ); the 00:13Z Backlog HITL claim's 90-min box ended with the 02:45Z report. Plan: one new lever from option 2 there (an opening-paragraph example in the rule's example column), probed on wfh-grounding-* with the same command and n=3, VALID required. If README with-arm reaches 3/3 and how-to holds, ship with Closes; otherwise ship option 1 (the reworded test, 0/3 -> 2/3) with Refs: #12085 and record the gap.

Lane 1 probe of the next change (option 2 in the 02:45Z report): did not ship.

State going in. Option 1, the reworded test, is already on main through fix(docs-hygiene): define a term at or before its first use in write-for-humans #12199 (squash 8ad11e5). The probe branch started from main 57477a9. Main's SKILL.md, both prompts and both graders are identical to the 2/3 baseline tree 2ffea86411.
The change was one example in the rule's example column about the opening sentence: "beacon posts each build result to the relay, the bot that forwards messages to team chat", not "…to the relay. The relay is the bot…". It is commit 27648d0, reverted in 99a2413.
The probe ran once, preregistered before launch, with the same command as the 02:45Z report: Claude Code 2.1.316, claude-opus-5-5, --ablation with-without, runs 3. run-validity.py returned VALID.

Case	With plugin	Without plugin
wfh-grounding-readme	0/3 (baseline on main: 2/3)	1/3
wfh-grounding-howto	3/3	3/3

Hand check. All three failing with-plugin READMEs make the slip this example targets, in the opening paragraph: "sends every line to a collector. A collector is the HTTP service…". So the example did not move the behavior, and it does not ship.
Measured so far on the README case, with plugin, at n=3: 0/3 on the old rule, 2/3 and 2/3 on the reworded rule (on main now), 2/3 with a post-draft check that was reverted, and 0/3 with this example. The without-plugin arm ranged from 0/3 to 1/3. At n=3 these counts cannot separate one change from another, so done-when 2 ("3 of 3 runs") cannot be shown reliably.
Done when. Item 1 is met on main. Item 2 is unmet. Whether to change item 2 to a rate threshold measured at a larger n is an owner decision, and lane 1 has raised it there.

Lane 1: the revised done-when 2 was measured once on main 971e449 (preregistered, VALID, 10 runs, Claude Code 2.1.316, claude-opus-5-5). It is not met.

Case	With plugin	Without plugin
wfh-grounding-readme	5/10 (0.5)	2/10 (0.2)
wfh-grounding-howto	10/10	10/10

With-plugin rate of at least 0.6: fails (0.5).
Margin over without-plugin of at least 0.3: met (0.3).
How-to still passes: met.
run-validity.py: VALID.

What the run shows:

Judge noise. The judge split its votes on 5 runs. It passed runs 3 and 4 but failed run 6, although all three open with the same sentence shape ("…to a collector. The collector is the HTTP service…"). Two of the 5 passes were split votes. README judge agreement was 15/20, against 20/20 on how-to.
The skill's slips. Runs 2, 8 and 10 never define "collector", and run 5 defines it one sentence late.

Result: the rule now on main (#12199) lifts the README case by 0.3 over no plugin, but it does not reach 0.6. Two levers have been tried and neither moved the opening-sentence slip. Lane 1 releases its claim. Next lever for whoever picks this up: make the terms-grounded judge deterministic or better calibrated before trying another wording change, so that the rate can be measured at all. Data: 40 kept traces, on the lane 1 host.

Navigation Menu

Repository navigation

feat(docs-hygiene): add a grounding rule to /docs-hygiene:write-for-humans #12085

Outcome

Done when

Out of scope

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

feat(docs-hygiene): add a grounding rule to /docs-hygiene:write-for-humans #12085

Description

Outcome

Done when

Out of scope

Activity

github-actions commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions