feat(verifiers): give verifiers the run's conversation and trigger firing logs

[sarah-chitty-1]

What

Give verifiers the run's conversation, including the user sim's turns, and the trigger firing logs, in a general way.

Today env_outcome_verifier gets only the MCP URL. A task with triggers or a user sim can't grade "noticed the change", "asked before acting" or "didn't leak the answer" without pulling those records itself.

• run_context.load_run_context builds a RunContext from what the run recorded:
  • The stored conversation, with every user-side turn: the prompt, then each user-sim or human message. Each user message is marked with the agent triggers that wrote it (fired) and the user sim's extra fields.
  • Each env's gateway firing log, and the user agent's own firing log.
  • Anything it couldn't read is listed in unavailable, so a missing log isn't mistaken for an empty one.
• as_of_turn(k) cuts it the way feat(rubrics_verifier): grade a multi-turn run as of turn k #222 grades "as of turn k": messages up to turn k's reply, and events from before turn k+1's message.
• rubrics_verifier: a new run_context_for_judge: ["conversation", "trigger_log"] adds Conversation and Trigger firings sections to the judge prompt, cut at grade_up_to_turn. Size is capped: 4000 characters per message, the latest 200 events per log.
• env_outcome_verifier: a script whose verify names a run parameter is called as verify(mcp_url, run=...). Every other script, one with an unrelated second parameter included, is called exactly as before. prompt_id picks which prompt, defaulting to the run's last one.

Worth a look

1. Opt-in only: the run context reaches a verify that names run, and the judge only when run_context_for_judge is set. Nothing changes for existing tasks or scripts.
2. Multi-prompt runs: the env firing log comes from the capture prompt_agent stores at the end of a prompt. The latest capture is the env's whole log, so in a task with several prompts it can include events from later prompts. It's cut by turn, not by prompt.
3. Not covered: agent_prompt_response_verifier doesn't get the run context. Should it get criteria like "trigger X fired"?
4. feat(rubrics_verifier): grade a multi-turn run as of turn k #222's rule: feat(rubrics_verifier): grade a multi-turn run as of turn k #222 refuses grade_up_to_turn with use_trajectory=False. With the conversation shown to the judge, that could be allowed; the rule is left unchanged here.
5. When the user agent's log counts as missing: prompt_agent only logs a warning when it can't read the user agent's firing log back, so the loader infers that a log was expected. It was if the step recorded a trigger firing or the prompt's user agent had registered triggers. If the conversation can't be read, any registered agent trigger counts.
6. A turn cut without the conversation: when as_of_turn can't find when turn k+1 began, it withholds every firing log and marks each one unavailable, so an early grade never sees later events.
7. Follow-up: once "who changed what" (feat(gateway): name the role and call behind each change to an env's state #240) lands, add an "env_changes" part.

Tests

• Unit: tst/unit/task_step/verifiers/test_run_context.py, 26 new tests against the real local conversation and object stores: the loader, the turn cut, missing records (a user-agent log that wasn't read back, and a turn cut with no conversation to cut by), the judge prompt sections, size caps and option validation, and which scripts get run (a run parameter does; one, two-with-another-name don't).
• Integration: test_verifier_run_context_local.py runs a real two-turn conversation, with the echo agent as both solver and user sim, and reads it back through load_run_context.
• Results: unit suite 8623 passed, 20 skipped; ruff, plugin-api (no break against the merge base) and uv lock --check pass; the local integration test passes. The branch still merges cleanly with current main.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; the previous requirement is satisfied and no new blocking issue was found.

Summary

Gives verifiers optional access to the recorded conversation and trigger firing logs.

• Verifiers can read a run’s conversation and trigger records.
• Outcome scripts can inspect the run behind a prompt.
• Rubric judges can see selected conversation and trigger logs.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Conversation[Stored conversation] --> Context[RunContext]
  Logs[Saved trigger logs] --> Context
  Context --> Script[Script that requests run]
  Context --> Cut[Optional turn cutoff]
  Cut --> Render[Selected sections and size limits]
  Render --> Judge[Rubric judge]

Loading

_{Reviews (3) · Last reviewed commit: "review: the judge's run-context limits a..." · Reviewed by Greptile}