Add the free-form code arm and the repair-clause ablation switch - #104
Open
rajeeja wants to merge 1 commit into
Open
Add the free-form code arm and the repair-clause ablation switch#104rajeeja wants to merge 1 commit into
rajeeja wants to merge 1 commit into
Conversation
The multi-turn eval could measure how agents behave with typed tools but had no way to answer the obvious objection: why not just let the model write Python? This adds that arm. It hands the model one run_python tool in a namespace with uxarray imported and the fixture paths bound, then scores it with the same score_run as the typed arm so the two are directly comparable. Tasks, fixtures, models, and turn budget are held fixed; only the action surface changes. Two scores are specific to code execution. silent_wrong marks a run that terminated normally and stated a number that was wrong, which is the failure the typed surface exists to prevent. guardrail_respected records whether the model noticed that the unlabeled wind field has no units or directional metadata; free-form code will compute a curl from it without complaint because nothing in the library requires the caller to have established that two arrays are vector components. Also adds EVAL_ERROR_MODE, which strips the repair clause from an error without changing anything else, so the value of naming a repair can be measured rather than assumed, and the front-door repair text and handle diagnostics the ablation toggles. The interpreter arm is opt-in, runs only against synthetic fixtures in a temporary state directory, and is never imported by the default eval run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the code-execution arm to the multi-turn eval so the typed surface can be compared against the obvious alternative of letting the model write Python, and adds
EVAL_ERROR_MODEso the value of naming a repair in an error can be measured instead of assumed.Both arms share tasks, fixtures, models, turn budget, and
score_run, so only the action surface differs. Measured over five common models, task completion was 16/25 typed against 10/25 free-form, and the unlabeled-field guardrail held 4/5 against 1/5 — free-form code has nowhere to put a precondition, so it computes a curl from a wind field with no units or directional metadata.The interpreter arm is opt-in, runs only against synthetic fixtures in a temp state dir, and is not imported by the default eval run. 640 tests pass and pre-commit is clean.