fix(evals): run-validity's session-started check fails runs whose Bash calls exit non-zero with output #11290

Activity

1. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 30, 2026
2. github-actions commented on Oct 30, 2026

   

   github-actionsBot

   on Oct 30, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Small
   Complexity: Low
   Uncertainty: Known
   Flags: none
   Blocked by: none

   Understood: run-validity.py's "session started" check treats a run as having no working shell when every Bash call returns an error, even when those calls produced real output (for example ls on a missing path). You want it to flag only a shell that never ran: sandbox failure, or an empty result. You also want a stay-quiet test for non-zero exits that come with output. The scope is clear and the refinement comment lists acceptance criteria. I found no duplicate. #10504 is related and touches the same file.

   _{Automated triage, applied by the lanes App.}
3. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 30, 2026
4. github-actions

   added the

   Bug

   issue type on Oct 30, 2026
5. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   and removed

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   on Oct 30, 2026
6. melodic-automation-lanes commented on Oct 30, 2026

   

   melodic-automation-lanesBot

   on Oct 30, 2026

   Contributor

   More actions

   Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
7. melodic-automation-lanes commented on Oct 30, 2026

   

   melodic-automation-lanesBot

   on Oct 30, 2026

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   The latest triage verdict holds nothing: the issue is at status: ready.
8. melodic-automation-lanes

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   and removed

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   on Oct 30, 2026
9. melodic-automation-lanes commented on Oct 30, 2026

   

   melodic-automation-lanesBot

   on Oct 30, 2026

   Contributor

   More actions

   Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

   Intended outcome: run-validity.py's session-started check stops flagging runs whose Bash calls exit non-zero with real output, and flags only a shell that never ran (sandbox failure text or empty results).

   Acceptance criteria

   • In plugins/evals/skills/plugin-eval/scripts/run-validity.py, shell_failure no longer adds a failed Bash result to failed_ids when its content has non-empty output other than the sandbox failure text.
   • shell_failure still returns 'the shell sandbox failed to start' for a Bash result containing SANDBOX_FAILURE_TEXT.
   • shell_failure still returns 'all N Bash calls failed' when at least MIN_BASH_CALLS Bash calls each returned an is_error result with empty content.
   • Denied Bash calls (is_denial) still count toward neither side, as at run-validity.py:367.
   • The module docstring at run-validity.py:16-18 and the comment at :115-116 describe the new rule: sandbox failure, empty result or known sandbox error text, not a non-zero exit with output.
   • test_run_validity.py has a new stay-quiet test where every Bash call is is_error true with output (e.g. 'Exit code 1' plus a listing, 2 and 3 calls) and 'check session started: PASS' is printed.
   • The existing test test_a_run_whose_every_bash_call_failed_is_invalid_whatever_the_wording (test_run_validity.py:345) is rewritten to use empty-content error results and still expects INVALID with 'all 2 Bash calls failed'.
   • The sandbox-failure test (test_run_validity.py:371) and the denied-calls test (:356) still pass unchanged.
   • python3 -I plugins/evals/skills/plugin-eval/scripts/test_run_validity.py exits 0, and run-validity.test.sh passes if it covers this check.

   External state: none

   Related work

   • fix(evals): fail an eval run whose every Bash call failed #10096 (related): Merged PR that added the all-Bash-calls-failed rule this issue narrows.
   • fix(evals): fail eval runs that never started or had no sandbox #10077 (related): Merged PR that added the sandbox-start failure check in the same function.
   • fix(evals): run scaffold scripts in plugin-eval and fail validity when a case was never scaffolded #10504 (related): Open issue also changing run-validity checks in the evals plugin; may touch the same file.

   Sources read

   • plugins/evals/skills/plugin-eval/scripts/run-validity.py:352-375: shell_failure counts every is_error Bash result (except denials) as failed and returns 'all N Bash calls failed' when all of at least 2 failed, regardless of output.
   • plugins/evals/skills/plugin-eval/scripts/run-validity.py:113-117: SANDBOX_FAILURE_TEXT and MIN_BASH_CALLS=2 definitions and comments.
   • plugins/evals/skills/plugin-eval/scripts/run-validity.py:397-423: check_started uses shell_failure to fail the 'session started' check.
   • plugins/evals/skills/plugin-eval/scripts/test_run_validity.py:325-369: bash_trace helper and a test that currently expects INVALID for error results that carry output ('Exit code 1\nREADME.md'), which conflicts with the fix and must be changed.

   _{Automated refinement by the lanes App. Intake triages the issue again with it.}
10. github-actions

    removed

    needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

    on Oct 30, 2026
11. melodic-automation-lanes

    added

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

    admission-deferredAdmitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.Admitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    and removed

    admission-deferredAdmitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.Admitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.

    on Oct 30, 2026
12. kyle-sexton

    added a parent issue on Oct 30, 2026

    • test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148
13. melodic-automation-lanes

    closed this as completedin #11297on Oct 30, 2026