Repository navigation
fix(evals): let run-validity gate a base side whose unfired runs are the measured change #11979
Description
Problem
plugins/evals/skills/plugin-eval/scripts/run-validity.py marks a result file INVALID when a should-trigger case does not fire in a with-arm run. On the base side of a change that makes a skill fire, not firing is the baseline being measured, so the base file can never gate VALID however clean its runs are.
Observed on #11895's round 5: shape-and-depth-nested-radius fired design 0/3 on base (answer grader 3/3) and 3/3 on head (3/3). Every base run had no error and no denial, yet run-validity.py --runs 3 reported verdict: INVALID (... the skill did not fire in 5 with-arm runs of a should-trigger case). The owner had to rule that the case be judged on runs with no error or denial, counting unfired runs as data.
Done when
-
run-validity.pycan gate a base side whose unfired runs are the measured quantity (for example, an operator flag naming the cases or the arm where firing is under test), without weakening the skill-fired check elsewhere. - A test fails on main and passes on the branch: a base file whose only defect is unfired runs of a named case gates VALID with the flag and INVALID without it.
-
/evals:plugin-evalnames the flag where it covers trigger-change comparisons.
Activity
github-actions commentedon Nov 1, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: run-validity.py marks a base-side result INVALID when a should-trigger case doesn't fire. On a change that makes a skill fire, that non-firing is the baseline being measured. You want an operator flag that lets the base side gate VALID without weakening the skill-fired check elsewhere. You also want a test for it and a mention in /evals:plugin-eval. The done-when items are exact, and I found no likely duplicates among the open issues.
Automated triage, applied by the lanes App.
github-actions commentedon Nov 1, 2026