planning: plan_stress_test per mode, settled by the devils-advocate eval #12478

Activity

1. kyle-sexton

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

   on Nov 2, 2026
2. kyle-sexton

   added a parent issue on Nov 2, 2026

   • planning: interview restate, wayfind absorb, stress test per mode #7586
3. github-actions commented on Nov 2, 2026

   

   github-actionsBot

   on Nov 2, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Knowable
   Flags: none
   Blocked by: none

   Understood: make plan_stress_test configurable per mode (plan, proposal, incumbent), resolved through the convention home from #7571. The default then changes only after a devils-advocate eval (12 plans, 3 arms, 3 runs) is recorded in a Probe result: comment on this issue. The done-when items are clear. The eval run itself is host-run work. #7571 doesn't appear among the open issues, so I recorded no blocker. No likely duplicates found.

   _{Automated triage, applied by the lanes App.}
4. github-actions

   added the

   Task

   issue type on Nov 2, 2026
5. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   on Nov 2, 2026
6. melodic-automation-lanes

   set

   Work status

   to

   ready

   on Nov 2, 2026
7. melodic-automation-lanes

   removed

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   on Nov 2, 2026
8. melodic-automation-lanes commented on Nov 2, 2026

   

   melodic-automation-lanesBot

   on Nov 2, 2026

   Contributor

   More actions

   Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
9. melodic-automation-lanes commented on Nov 2, 2026

   

   melodic-automation-lanesBot

   on Nov 2, 2026

   Contributor

   More actions

   The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.
10. melodic-automation-lanes

    changed

    Work status

    to

    confirmed

    on Nov 2, 2026
11. kyle-sexton commented on Nov 3, 2026

    

    kyle-sexton

    on Nov 3, 2026

    Last edited by kyle-sexton

    MemberAuthor

    More actions

    Probe result: plan_stress_test keeps auto in every mode. The negative control did not detect a loss, so this suite cannot measure what /planning:devils-advocate adds, and the pre-registered rule keeps the default.

    Design (pre-registered in #12835 before any paid run; arms by owner decision 2026-11-03)

    • Suite 56f217b92f: 12 drafts, 4 per mode (plan, proposal, incumbent). Each has 3 seeded defects and one sound-part must-not-flag check, all graded by regex on last_message.
    • Arms: A reviewer (plan-reviewer only), B devils-advocate (devils-advocate in the draft's mode only), C both. The negative control is B with the devils-advocate body replaced by a body with no procedure and no output format.
    • 3 runs per case per arm, 144 runs in all, claude-opus-5-5 on Claude Code 2.1.317. plan-reviewer keeps its opus pin, as the smoke traces showed.

    Validity: A and the control read VALID. B and C first read INVALID on 3 denials, each the run searching its own plugin copy, which sits outside the sandbox but is readable in a real install. An addendum written before any score was read applies --own-copy-search-warns to every file (all four then VALID) and excludes the 6 runs with such a denial in every arm (B 2, C 1, control 3). Each affected case keeps 2 runs. Every remaining run conformed to its arm (arm-report.py).

    Variance (median per-case range across runs; scores step by 0.25; above 0.25 fails)

    mode	A	B	C	control
    plan	0.12	0.12	0.00	0.25
    proposal	0.00	0.00	0.00	0.00
    incumbent	0.12	0.12	0.12	0.12

    Negative control: not detected. Control against B, paired over 12 cases, changed by −0.05 (95% interval −0.09 to −0.01), with no case drop. Detection needed a case drop or a fall of 0.15 or more. Leaving out the 3 cases whose control runs carried an own-copy search gives the same result (9 cases, −0.05). With the control undetected, every mode keeps auto under the pre-registered rule.

    Scores, reported only; the rule above decides. Means are over cases, with 95% intervals.

    mode	A mean	C mean	C over A	A against C (margin 0.15)	B against C (margin 0.05)
    plan	0.92	0.92	−0.00 (−0.07 to +0.07)	non-inferior	non-inferior (+0.02, −0.02 to +0.06)
    proposal	0.98	1.00	+0.02 (−0.02 to +0.06)	non-inferior	non-inferior (+0.00)
    incumbent	0.96	0.94	−0.02 (−0.06 to +0.02)	non-inferior	not shown non-inferior (+0.02, −0.06 to +0.10)

    Every arm scored 0.92 or higher, so the suite sits near its ceiling. That is the likely reason the control could not detect a loss: plan-reviewer alone, or a gutted devils-advocate, already finds most seeded defects. A suite that could separate these arms needs harder defects or more cases (plugins/evals/skills/methodology/reference/local-decisions.md, "Repeat count").

    Rules not reached: rule 1 (C beats A by +0.15) is not met in any mode. Rule 2 (A non-inferior to C, so reviewer) would hold in every mode. It is not applied, because the control showed this suite cannot see a loss of the skill.

    Limits

    • The own-copy searches occur only in arms that run devils-advocate: B 2, C 1, control 3, A 0. They were the run looking up its own skill and config in the plugin copy (for example a Glob of skills/devils-advocate/** and a Grep of reference/config.md). The skill also links ../../context/problem-framing.md (SKILL.md:137). A real install can read those files. In this eval the plugin copy sits outside the sandbox, so devils-advocate may have run handicapped in every run that uses it, not only the 6 runs that showed a denial. The result therefore tilts against devils-advocate. Read it as "not measurable by this suite", never as "devils-advocate adds nothing". A suite that measures it needs the plugin copy readable from inside the sandbox.
    • sound-part misses (A/B/C/control: plan 2/3/3/5, proposal 0/0/0/0, incumbent 0/1/2/3) can be inflated. In a spot-read, st-plan-3 arm C run 3 failed sound-part on a real, different finding (migration 003's DEFAULT now() makes every subscription due), which the grader's migration…next_renewal_at alternative also matches. No mode reaches the "3 or more" reporting line for C over A.
    • The spot-read covered one failing run, not one passing and one failing per mode as pre-registered.
    • The result holds for claude-opus-5-5 only.

    Hand-off phrasing (Q188): devils-advocate's dispatch conformed in every remaining B, C and control run (arm-report.py printed no non-conforming run), which bears on #11959's open note about a hand-off deep in a long skill body.
12. kyle-sexton

    added a commit that references this issue on Nov 3, 2026

    docs(planning): cite the plan_stress_test probe result for the auto d…

    ...

    This issue will close once commit 4462137 is merged into the 'main' branch.

    4462137
13. kyle-sexton

    closed this as completedin #12835on Nov 3, 2026