Skip to content

test(docs-hygiene): add eval cases for the write-for-humans grounding gap #12023

Description

@kyle-sexton

Parent #7591 decides that "/docs-hygiene:write-for-humans grounding rule only if an eval shows the gap" (Q526), and its done-when needs the write-for-humans eval slice to carry a Probe result: comment, but that slice was never filed (comment 6556369883); this slice authors the eval cases and slice S3 runs them on a host. The rule under test, from the accepted Q526 list, is "introduce each concept before a sentence relies on it, or name it as assumed reader knowledge", and plugins/docs-hygiene/skills/write-for-humans/ has no such rule on main.

Done when

  • plugins/docs-hygiene/evals/ gains at least two wfh-grounding-* cases. Each case has case.yaml, prompt.md, graders/skill-fired.md (same form as wfh-tutorial-is-reference/graders/skill-fired.md) and a grounding grader.
  • Each grader has a samples/<grader-name>.json with at least one pass and one fail entry (same shape as wfh-tutorial-is-reference/samples/skill-fired.json).
  • Each prompt asks for a human-facing document (README, design doc or how-to) from source material that uses at least one domain term before it defines it.
  • The grounding grader is type: llm. Its text says PASS only when every such term is introduced before a sentence relies on it or is named as assumed reader knowledge, and FAIL otherwise.
  • Each case.yaml sets runs: 3 or more.
  • The diff edits neither plugins/docs-hygiene/skills/write-for-humans/SKILL.md nor reference/sentence-rules.md: the rule ships only if S3 shows the gap.
  • A .changes/docs-hygiene/*.md fragment with bump: none describes the new cases.
  • The PR's CI is green.

Out of scope

  • Writing the grounding rule.
  • Running the eval (slice S3).

Parent: #7591

Activity

  1. added theissue type on Nov 1, 2026
  2. added
    priority: mediumReal value, no hard deadline; normal backlog flow.
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    on Nov 1, 2026
  3. github-actions commented on Nov 1, 2026

    @github-actions
    Contributor
    No description provided.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions