Raven x - #240
Open
TongLi31 wants to merge 7 commits into
Open
Conversation
…ONPATH Motivation: trajectory analysis of 177 raven failures on SWE-bench Verified found the dominant framework-attributable failure (47/60 cases) was the agent declaring completion after ad-hoc self-checks, because the bundled runtime's PYTHONPATH leaked into exec child processes and broke the workspace project's own test tooling (bundle packages shadowing a much older interpreter). - raven/__main__.py: drop raven's own install dir from PYTHONPATH at startup so exec children never inherit bundle packages; sys.path of the raven process itself is already resolved and unaffected. - raven/agent/loop/main.py: opt-in completion gate (env RAVEN_REQUIRE_REAL_TEST_EVIDENCE): a final text answer is only accepted after the session has run the project's own test runner at least once; bounded to 2 nudges so the gate itself can never loop. Evaluated on a 136-task SWE-bench Verified subset: 67 resolved vs 56 baseline (net +11); the gate fired in 13 tasks with no runaway loops. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
Motivation: after wave-1 (environment isolation + completion gate), the remaining SWE-bench Verified failures cluster into generalizable methodology gaps, not benchmark specifics: over-wide edits that break neighboring behavior (10 of 34 remaining target failures show passed-before/failing-after tests), fixes that cover only the literal example and miss sibling code paths (16 of 34), and correct fixes reverted because an existing test encoded the old behavior. Add a compact, domain-general discipline block to the identity segment so every harness benefits (not only the SWE eval prompt): root-cause-first, requirement text over stale tests, smallest complete change applied to all sibling paths, verify with the project's own tooling, treat passed-before/failing-after as a self-introduced regression, review the full diff before declaring completion. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…licts A comprehension test by an independent agent (given only the block, no context) rated the first version 6/10 and found two defects: - the stale-test rule (Understand) and the regression rule (Verify) gave opposite default verdicts for the same observable event with no classification criterion; now the criterion is explicit (stale = asserts the exact old behavior the task asks to change; everything else is a self-introduced regression) and the two rules cross-reference each other. - 'smallest change' conflicted with 'fix ALL occurrences'; now scoped as smallest change per fix site, applied to every site sharing the flaw. Also restructured from flat compound bullets into Understand / Implement / Verify / Before-declaring-done phases with short imperative rules, matching the prompt style that stronger harnesses use. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…l result persistence Wave-4 completion-guardrail refinement, opt-in and layered on the wave-1 evidence gate (RAVEN_REQUIRE_REAL_TEST_EVIDENCE): - Evidence ledger in the agent loop: parse each real-test run's output as red/green (conservative regexes; unparseable output never triggers anything) and track non-doc file edits made after it. - RAVEN_GATE_STALE: when the model declares completion with untested edits, append ONE fact-stating reminder asking it to re-run or justify; doc-only edits exempt. - RAVEN_GATE_RED: when the model declares completion right after a failing run, append ONE inquiry with an explicit exit (a test asserting the old behavior the task changes is stale and may remain failing) so a correct judgment is never forced into iteration. - Persist oversized tool results as head 12k + tail 4k instead of head-only 16k: test runners put the pass/fail summary at the end, so the old truncation cut exactly the evidence later turns need. Both reminders fire at most once per turn, state only verifiable facts, and leave the decision to the model. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…ratch paths Three detector fixes surfaced by the 50-task variant exploration (6 of 12 v1-gate fires were false positives, one inducing a 163-turn re-verify loop): - Strip ANSI color sequences before matching test output: color codes glue onto adjacent digits (ESC[32m201 passed) and defeat word-boundary matching. - Recognize sympy's native runner invoked as a module (runtests without .py, sympy.doctest) and its output banners (test process starts, tests finished). - Do not mark evidence stale for edits under /tmp, /var/tmp, or /dev/shm: scratch files are not part of the deliverable (2 of 3 stale fires were /tmp reproducer scripts). - Gate nudge now says NOT to install packages and points at bundled runners (4 sessions responded to the nudge with pip install pytest, violating task constraints and adding skip/fail noise from pytest-8-on-old-repo). Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…g self-written checks The old wording (a check you wrote yourself is NOT verification) is a dead-end instruction in projects that have no test suite. Reword as an evidence ranking: project tests first; if none cover the change, write one following project conventions and run it through a real runner; ad-hoc self-written checks are the weakest evidence because they re-encode the same assumptions as the change. The completion claim must state what evidence backs it.
Root-level change description for the raven-x branch: the runtime PYTHONPATH leak fix, the opt-in test-evidence guardrails (completion gate, stale-evidence reminder, red-test inquiry with a semantic exit), the engineering-discipline identity block, and the evaluation results backing each change. Written for reviewers (human or agent) landing on this branch. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type
Verification
Risk
Related Issues