Skip to content

feat(verification): place the explain-the-number and rerunnable-evidence principle parts in measure, confirm, performance verify and plugin-eval #12149

Description

@kyle-sexton

Parent #7589 decides that pstack's principles are placed "as reasoned sentences in the phase-skill step that applies them" (D58, Q174, Q175). The owner approved this 2026-10-15 (poteto-review decision rows 58 and 105). The wording rule the owner accepted (Q175) is that each placed part states the reason and the test that shows it applies, rather than a bare rule, and names the standard or discipline it comes from.

This slice places these parts from Lauren Tan's explain-the-number and prove-it-works principles:

  • 7c: an untuned arm, left on default settings, is an alternative explanation for a measured gap. /performance:verify (plugins/performance/skills/verify/SKILL.md:119-124) names build modes and flags, but not an untuned arm.
  • 7e: code-metric and quick performance compares state what limits the number and what was ruled out, or hand off to /performance:verify. /verification:measure (plugins/verification/skills/measure/context/performance.md:24) has neither line.
  • 7f: before trusting an eval result, check that every run did the task, and that the gap holds across trials and models. /evals:plugin-eval's "Reading the delta" section (plugins/evals/skills/plugin-eval/SKILL.md:275) covers validity and trial noise, not a second model.
  • 18e: script the check and keep output a reviewer can rerun. /verification:confirm's evidence manifest says to "cite a ## Reproduction block" (plugins/verification/skills/confirm/SKILL.md:218), not that the check be scripted.

Done when

  • plugins/performance/skills/verify/SKILL.md's ruled-out list names an untuned arm (one left on default settings while the other was tuned) as an explanation to rule out, with its test.
  • /verification:measure's compare output carries a Limiter: line and a Ruled out: line for code-metric and quick performance compares, or hands the compare to /performance:verify, and says which. The reason is that a number with no known limit cannot be trusted.
  • plugins/evals/skills/plugin-eval/SKILL.md's reading step checks that every counted run did the task, and says that a gap claimed for more than the model it ran on needs a run on a second model.
  • /verification:confirm's evidence step says that a check worth citing is scripted, with its command and output kept so a reviewer can rerun it, with the reason.
  • Each added sentence states its reason and its test.
  • .changes/performance/*.md, .changes/verification/*.md and .changes/evals/*.md fragments with bump: patch describe the additions.
  • The PR's CI is green.

Parent: #7589

Activity

  1. added
    priority: mediumReal value, no hard deadline; normal backlog flow.
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Nov 1, 2026
  2. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    on Nov 1, 2026
  3. github-actions commented on Nov 1, 2026

    @github-actions
    Contributor
    No description provided.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Fields

    No fields configured for Feature.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions