fix(evals): run-validity passes a run whose plugin under test never loaded #12190

Activity

1. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Nov 2, 2026
2. github-actions commented on Nov 2, 2026

   

   github-actionsBot

   on Nov 2, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Small
   Complexity: Low
   Uncertainty: Known
   Flags: none
   Blocked by: none

   Understood: run-validity.py passes a run even when the plugin under test never loaded. The trace init line showed plugin_errors (dependency-unsatisfied) and the plugin was missing from the loaded list. The fix is a new check that marks such a with-arm run INVALID, with a reason naming the plugin and the error. It comes with fixture tests that fail before the fix.

   The issue text and the existing refinement comment give a clear done-when, so nothing is missing. Related, but not duplicates: #8635, #12043, #12191 (the playgrounds suite issue).

   _{Automated triage, applied by the lanes App.}
3. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Nov 2, 2026
4. github-actions

   added the

   Bug

   issue type on Nov 2, 2026
5. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   and removed

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   on Nov 2, 2026
6. melodic-automation-lanes commented on Nov 2, 2026

   

   melodic-automation-lanesBot

   on Nov 2, 2026

   Contributor

   More actions

   Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
7. melodic-automation-lanes commented on Nov 2, 2026

   

   melodic-automation-lanesBot

   on Nov 2, 2026

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   The latest triage verdict holds nothing: the issue is at status: ready.
8. melodic-automation-lanes

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   and removed

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   on Nov 2, 2026
9. melodic-automation-lanes commented on Nov 2, 2026

   

   melodic-automation-lanesBot

   on Nov 2, 2026

   Contributor

   More actions

   Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

   Intended outcome: run-validity.py fails a run whose with-arm trace init line reports a plugin_errors entry for a suite plugin or omits a suite plugin from its loaded plugins, with a reason naming the plugin and error.

   Acceptance criteria

   • run-validity.py has a new check (for example 'plugins loaded') that reads each with-arm run's trace init line, the (system, init) line already matched by TRACE_INIT at plugins/evals/skills/plugin-eval/scripts/run-validity.py:155.
   • The check identifies each suite plugin from suite.plugins[].path (read at run-validity.py:125), taking its name from the plugin's .claude-plugin/plugin.json.
   • A with-arm run whose init line plugin_errors has an entry for a suite plugin makes the check FAIL, with a detail line naming the run, the plugin and the error text (for example dependency-unsatisfied).
   • A with-arm run whose init line loaded plugins list lacks a suite plugin makes the check FAIL, with a detail line naming the run and the missing plugin.
   • A FAIL from the check adds a reason to the final INVALID output, so main() prints INVALID and exits with the invalid status.
   • A run with no trace, or no init line or plugins field, is reported as UNCHECKED or warned like check_probe_tools, not passed silently.
   • A suite with no plugins, and a run where every suite plugin is loaded with no plugin_errors, still pass the check.
   • A fixture trace in test_run_validity.py with plugin_errors dependency-unsatisfied and the plugin missing from loaded plugins makes the new test assert INVALID and the reason naming the plugin and error; this test fails on the pre-fix script.
   • A second test covers a plugin missing from the loaded list with no plugin_errors entry, and a third covers the passing case.
   • The script's header docstring lists the new check next to 'session started', and run-validity.test.sh and test_run_validity.py pass.

   External state: none

   Related work

   • feat(evals): convert other plugins' evals.json files to claude plugin eval suites #8635 (related): conversion smoke runs rely on this check; the playgrounds cases found the gap
   • feat(evals): repair or retire eval cases whose with-arm never loaded the skill #12043 (related): repairs or retires eval cases whose with-arm never loaded the skill; a different failure from the plugin never loading
   • fix(evals): run invocation probes without subagents and name the run line in run-validity #11901 (related): merged change to run-validity that touched the run line and init handling
   • fix(evals): fail eval runs that never started or had no sandbox #10077 (related): merged change that added the 'session started' check this extends

   Sources read

   • plugins/evals/skills/plugin-eval/scripts/run-validity.py:125: SUITE_PLUGINS/PLUGIN_PATH read suite.plugins[].path, with no name or plugin_errors handling
   • plugins/evals/skills/plugin-eval/scripts/run-validity.py:155: TRACE_INIT is the (system, init) trace line
   • plugins/evals/skills/plugin-eval/scripts/run-validity.py:683: check_started judges only 0-turn runs and sandbox failures, nothing about plugin loading
   • plugins/evals/skills/plugin-eval/scripts/run-validity.py:1222: check_probe_tools shows how to read the init line and report UNCHECKED or WARN
   • plugins/evals/skills/plugin-eval/scripts/run-validity.py:1486: main() assembles the checks, and a new check is added in the later list
   • plugins/evals/skills/plugin-eval/scripts/test_run_validity.py:760: existing tests set data["suite"]["plugins"] = [{"path": ...}] for fixtures
   • repository grep for plugin_errors: no matches, so no check exists yet

   _{Automated refinement by the lanes App. Intake triages the issue again with it.}
10. github-actions

    removed

    needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

    on Nov 2, 2026
11. melodic-automation-lanes

    added

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

    admission-deferredAdmitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.Admitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    and removed

    admission-deferredAdmitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.Admitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.

    on Nov 2, 2026
12. github-actions commented on Nov 2, 2026

    

    github-actionsBot

    on Nov 2, 2026 – with GitHub Actions

    Last edited by github-actions

    Contributor

    More actions

    No description provided.
13. kyle-sexton

    closed this as completedin #12206on Nov 2, 2026