Repository navigation
feat(docs-hygiene): let audit-derivability's spot test run on an omitClaudeMd agent, measured no-loss #12358
Closed as not planned
Description
This was generated by AI.
Parent
Sliced from #11823 (finding 4; see #12272's "Remaining on #11823").
Problem
plugins/docs-hygiene/skills/audit-derivability/SKILL.md picks the agent its spot test runs on (:76, also :152 and :162 on main, 2026-11-02; today Explore or Plan). The sweep's proposal is to allow a custom agent that sets omitClaudeMd, so the spot test runs without the repository's CLAUDE.md. That changes what the skill does, so it ships only on eval evidence of no loss. Upstream: sub-agents#what-loads-at-startup. A subagent whose definition sets omitClaudeMd loads only the managed policy files. The field is ignored when the agent runs as the main session with claude --agent.
Done when
- An eval case exercises the spot-test agent choice (add one if none exists), with a negative control that drops clearly below the before text.
- Before and after are measured on VALID runs. The change ships only on no loss; a loss is recorded here.
- Any pointer carries an as-of date and a recheck trigger.
Activity
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
I read this as a slice of #11823. It would let audit-derivability's spot test run on a custom agent that sets omitClaudeMd, so the repository's CLAUDE.md isn't loaded. The change ships only if before/after eval runs on valid passes show no loss, with a negative control that drops clearly below the before text. Any upstream pointer would carry an as-of date and a recheck trigger.
The done-when items are specific enough to start from. Paid or repeated eval passes may need a host run, and admission routes those separately. I found no duplicate among the open issues. #11823 is the parent, not a duplicate.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 2, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 2, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 2, 2026
Claimed by lane 1 (melo-lap-001, hub assignment 2026-11-02 16:5xZ). Plan: eval case for the spot-test agent choice with a negative control, before/after on VALID runs; ships only on no loss.
test(docs-hygiene): drop the loads-at-launch hint from the spot-test …
kyle-sexton commentedon Nov 2, 2026
Lane 1 result: kept with reason. The agent-choice change does not ship, and audit-derivability keeps its current text.
Measured 2026-11-02 18:41-18:45Z on Opus 5.5 (Claude Code 2.1.316). The
ad-*suite (3 cases, including the newad-launch-loaded-spot-test-agent) ran in three versions, 3 runs each, launched together. All three versions passedrun-validity.py(VALID).case drop. With the agent-choice text gone, Opus 5.5 still dispatchedExplorefor the spot test in all 3 runs. So the suite cannot measure this part, and the preregistered rule keeps the before text.docs-hygiene:derivability-spot-tester, which confirms that a plugin agent withomitClaudeMdresolves in the eval sandbox under its scoped name.subagent_type, and no negative-control run read the case files.Re-measure trigger: a case where the model's default agent choice is wrong (a target that Explore suits poorly, or a prompt that pulls toward general-purpose), or a model release that stops defaulting to Explore. Branch
feat/12358-derivability-omit-claudemd(headb2b52a9a4c) holds the case, the agent and the run record. It has no PR, because the skill change is unmeasured.