Repository navigation
test(docs-hygiene): run the write-for-humans grounding eval and post its probe result #12025
Description
Parent #7591's done-when reads "The write-for-humans eval slice carries a Probe result: comment", and Q526 ships a grounding rule in /docs-hygiene:write-for-humans only if an eval shows the gap; the lane cannot run evals (comment 6556369883) and only a host with the claude CLI can, so this separate slice runs the cases slice S2 adds. The eval passes are model usage under the Claude subscriptions or the accounts' promotional API credits, which the owner's standing direction section 3 calls "Not cost, so spend it freely" (docs/contracts/loop-lane/README.md:351); each pass still sets a --max-cost-usd ceiling.
Done when
-
claude plugin evalruns thewfh-grounding-*cases with and without the docs-hygiene plugin, at each case'sruns:count. Each pass sets a--max-cost-usdceiling. - A comment on this issue whose first line starts with
Probe result:gives:- the CLI version;
- the ceiling used;
- per case and arm, the grounding grader's pass count over its runs;
- the verdict: gap shown when the with-plugin arm fails the grounding grader in at least one run of any case, otherwise no gap.
- The probe result is also committed: a PR adds the
Probe result:verdict line to eachwfh-grounding-*case'sdescriptionincase.yaml, adds a.changes/docs-hygiene/*.mdfragment withbump: none, and closes this issue. - When the gap is shown, an issue to add the rule to
/docs-hygiene:write-for-humansis filed as a sub-issue of domain-driven-design: glossary layout, pressure-test observe, write-for-humans eval #7591, withParent: #7591in its body, and is linked in the comment. When no gap is shown, the comment says the rule does not ship.
Out of scope
- Writing the grounding rule in this slice.
Parent: #7591
Activity
kyle-sexton commentedon Nov 1, 2026
Claim: Backlog HITL host-run worker is running this probe on branch host-run/12025-wfh-grounding-probe (stacked on #12034). The Probe result comment follows from the coordinator.
kyle-sexton commentedon Nov 1, 2026
Probe result: Host run. Ran the
/docs-hygiene:write-for-humansgrounding eval (wfh-grounding-readme,wfh-grounding-howto) with and without the docs-hygiene plugin. Verdict: gap shown.What ran
claude-opus-5-5pinned with--model(12 of 12 traces). The judge was the CLI default (haiku), 3 votes per verdict.03bf0daa986712f193618021ead0090eff99b5bf. If test(docs-hygiene): add eval cases for the write-for-humans grounding gap #12034's case files change before it merges, this probe is stale and must re-run./evals:validate(validate-cases.py) on the suite before and after the edit: no FAIL. The only warnings were judge calibration for thesamples/answers.claude.slicescope with a short TMPDIR:claude plugin eval <copy>/docs-hygiene --case 'wfh-grounding-*' --ablation with-without -j 2 --model claude-opus-5-5 --disallow-tools Agent Task Bash --max-cost-usd 20 --trust-plugin --no-publish --keep-temp. Each case's ownruns: 3was used.--max-cost-usd 20. The CLI reported $1.45 notional spend, with no partial result.a git directory of the plugin registers more linked worktrees than can be screened. The second run used an rsync copy of the plugin in a directory outside any git repository, as/evals:plugin-evaldescribes.run-validity.py --runs 3gave VALID: complete, no run errors, no permission denials, 12 of 12 traces resolved, and the skill fired in 6 of 6 with-arm runs. Warnings:Scores (terms-grounded grader, passes over runs)
wfh-grounding-readmewfh-grounding-howtocheckpoint,collectorandtail set. The result records votes only, not which term failed. Under the issue's rule (a with-arm grounding fail in at least one run of any case), the gap is shown.Done when
claude plugin evalran bothwfh-grounding-*cases with and without the plugin at each case'sruns: 3, under a--max-cost-usd 20ceiling.Probe result:comment, with the CLI version, the ceiling, per-case and per-arm pass counts, and the verdict.21132fbe15adds theProbe result:verdict line to each case'sdescriptionand adds.changes/docs-hygiene/host-run-12025-wfh-grounding-probe-ce98e335.mdwithbump: none. The commit message carriesCloses #12025./docs-hygiene:write-for-humans) is not filed yet. This worker may not file issues, so the coordinator files it and adds its link to this comment.Branch:
host-run/12025-wfh-grounding-probe, head21132fbe15b54fa9c4086f341a8aae176a178822. It is stacked on #12034's commit03bf0daa98, because the cases exist only there.scripts/check-affected.sh --testson the committed head gave 13 passed and 3 skipped. Pushed. No PR is open.Target labels: add none; remove host-run.
Follow-up filed: #12085 (sub-issue of #7591).
github-actions commentedon Nov 1, 2026
melodic-automation-lanes commentedon Nov 1, 2026
The work-resolve-issue lane stopped on this issue (
host-only): the run reports the work needs a host run the lane cannot make.It added
host-runand removedagent-ready: an agent run on a host builds it, and it waits on no decision from a person.In the run's words:
What the run says it did not do:
claude plugin evalon the merged wfh-grounding-* cases with and without docs-hygiene, under a --max-cost-usd ceiling: this lane has noclaudeCLI and cannot run evalsRun: https://github.com/melodic-software/claude-code-plugins/actions/runs/41350324634
To hand it back to the lane, remove
host-runand addagent-readyagain.github-actions commentedon Nov 1, 2026
Intake triage did not triage this issue after 2 dispatches, so the sweep no longer dispatches it and labeled it needs-triage for a maintainer. Triage it by hand, and read the runs for why intake did not:
github-actions commentedon Nov 1, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: this slice runs the write-for-humans grounding eval with and without the plugin, posts a
Probe result:comment, and commits the verdict line to the case descriptions. The latest comment reports the host run finished. It gives the verdict "gap still shown", with the README case at 0/3 with the plugin. The branch is pushed, but no PR is open yet, so the remaining work is opening that PR. The follow-up for the rule is tracked in #12085. I found no duplicates.Automated triage, applied by the lanes App.
kyle-sexton commentedon Nov 1, 2026
Probe result: Host run. Re-ran the
/docs-hygiene:write-for-humansgrounding eval (wfh-grounding-readme,wfh-grounding-howto) with and without the docs-hygiene plugin on currentmain, which now carries the grounding rule. Verdict: gap still shown. The README case fails the terms-grounded grader in 3 of 3 with-plugin runs.What ran
claude-opus-5-5, pinned with--model(12 of 12 traces). The judge was the CLI default, 3 votes per verdict.git archiveoforigin/mainatf7dde4f0e40a88d2153cade94904be555a100b5d, extracted to a directory outside any git repository. The last docs-hygiene change on that head is24c6e71e07(feat(docs-hygiene): add a grounding rule to write-for-humans #12098), which adds the rule "Introduce each concept before a sentence relies on it, or name it as assumed knowledge" toplugins/docs-hygiene/skills/write-for-humans/SKILL.md:51. The cases and the reworded terms-grounded grader came from test(docs-hygiene): add eval cases for the write-for-humans grounding gap #12034 (a948531806)./evals:validate(validate-cases.py) on the suite gave no FAIL. The only warnings were judge calibration for thesamples/answers, and Bash, Edit and Write grants on unrelated cases.claude.slicescope with a short TMPDIR on disk:claude plugin eval <copy>/docs-hygiene --case 'wfh-grounding-*' --ablation with-without -j 2 --model claude-opus-5-5 --disallow-tools Agent Task Bash --max-cost-usd 20 --trust-plugin --no-publish --keep-temp. Each case's ownruns: 3was used. Started when one other suite was running (count 1, under the cap of 2).--max-cost-usd 20. The CLI reported $1.44 notional spend, 234 s wall time, no partial result. The CLI exits 1 when a case fails, as it did here.run-validity.py --runs 3gave VALID: complete, no run errors, no permission denials, 12 of 12 traces resolved, and the skill fired in 6 of 6 with-arm runs. Warnings:Scores (terms-grounded grader, passes over runs)
wfh-grounding-readmewfh-grounding-howtotail setappears in none of them.collectoris used in the first paragraph ("sends each new line to your log collector", "to the log collector", "to the collector"). No README says what the collector is or names it as assumed knowledge.checkpointis used one sentence before its definition ("loglet records a checkpoint. A checkpoint is the byte offset…"). The grader requires the definition at or before the first sentence that relies on the term.SKILL.md:51). The model may treat "collector" as general vocabulary, outside that scope. A fix for feat(docs-hygiene): add a grounding rule to /docs-hygiene:write-for-humans #12085 has two possible levers. Widening the rule's scope would reachcollector. Requiring the definition at or before the first use would reachcheckpoint. A re-probe with the judge's reasoning, or a per-term grader, would show which term fails.Done when, #12085 (sub-issue of #7591)
plugins/docs-hygiene/skills/write-for-humans/SKILL.md:51onmain, from feat(docs-hygiene): add a grounding rule to write-for-humans #12098).wfh-grounding-readmedoes not pass terms-grounded in the with-plugin arm (0 of 3).wfh-grounding-howtostill passes (3 of 3).run-validity.pygives VALID.Done when, #12025
claude plugin evalran bothwfh-grounding-*cases with and without the plugin atruns: 3, under--max-cost-usd 20.Probe result:and gives the CLI version, the ceiling, per-case and per-arm pass counts, and the verdict.032f4597ecadds theProbe result:verdict line (gap shown, Claude Code 2.1.316, mainf7dde4f0e4) to both case descriptions. It also adds.changes/docs-hygiene/host-run-12025-wfh-grounding-probe-main-7c1e4a09.mdwithbump: none. The commit message carriesCloses #12025.Branch:
host-run/12025-wfh-grounding-probe-main, frommainatf7dde4f0e4, head032f4597ec6355cc7cabf5c4ef538b6d0fdf0eba.scripts/check-affected.sh --testson the committed head gave 13 passed and 3 skipped. Pushed. No PR is open.Target labels: add none; remove host-run.
Follow-up for the rule: #12085 stays open. The rule shipped in #12098, but the README with-arm still fails (0/3).