feat(verifiers): read coding-agent CLI trajectories in the judge action log

[kaihackney-scale]

Why

The direct (non-agent) judge's action log (compaction_type: screenshot, trajectory_text: action_log) only parsed OTel spans. Trajectories from coding-agent CLIs (stream-json rows: system / assistant / user / result) and from OpenCode (part events) produced an empty transcript. A direct judge on those runs therefore graded without seeing a single tool call. The fixed per-call cuts (110 chars of arguments, 160 of result) are also too small for MCP tool traffic.

What

• transcript_from_raw detects the format:
  • OTel spans: existing behaviour, moved unchanged into _transcript_from_otel.
  • stream-json rows: tool_use blocks are paired with their tool_result by id; is_error results become {"error": ...}; assistant text blocks become messages; the closing result row is skipped (the judge already gets the final response).
  • OpenCode events: tool parts (input, then output or error) and text parts; a part re-emitted as its state advances keeps its first position and its latest state; the random mcp_<hex>_ tool prefix is dropped.
  • In the two new parsers, JSON-in-text tool results are decoded (up to 3 levels), because a result like {"result": "{\"id\": ...}"} otherwise renders as mostly escape characters.
• action_log takes max_args_chars / max_result_chars. TrajectoryFilter gains action_log_max_args_chars / action_log_max_result_chars (validated like action_log_max_lines, serialized in to_dict / from_dict). Defaults stay 110 / 160, so existing tasks render exactly as before. rubrics_verifier passes them through.
• Criterion matching and the regex knobs keep using rendered_action at the default cut, as before.
• Transcript records the format it was read from (source). When trajectory_text=action_log and the trajectory is JSON in no format the action log reads, the verifier raises instead of handing the judge an empty log. Text that is not JSON at all stays a failed run, as before.

Testing

• New tst/unit/task_step/verifiers/judge_utils/test_frame_selection_cli_formats.py covers stream-json pairing (errors, images, orphan results), OpenCode parts (status updates, error, prefix), nested-JSON decoding, the new cuts, and that unrecognised lists still give an empty transcript. Its 4 behaviour tests fail on main and pass here.
• test_rubrics_verifier_per_criterion.py: the new filter fields round-trip and are validated, and a stream-json trajectory reaches the direct judge's prompt as an action log at the configured cuts.
• The guard: an unrecognised format raises before any judge call, and an empty trajectory ([]) still reaches the judge. With the guard disabled, its test fails.
• An OpenCode error whose reason is in output keeps the reason, and a [] stream-json result stays apart from no result. Each test fails with its fix reverted.
• tst/unit/task_step/verifiers/: 359 passed locally. The full suite is left to CI: the local environment lacks some newer dependencies.

Why it matters

On a sample of real MCP-agent runs, a direct judge reading this action log agreed with the agent judge on about 92% of criteria, at a fraction of the latency.

No task's configuration changes in this PR.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; both previous findings are fixed and no new blocking issues were found.

Summary

The PR adds coding-agent CLI and OpenCode tool calls to the direct judge’s action log.

• Direct judges can read coding-agent tool histories.
• Tasks can choose how much tool input and output the action log shows.

No new actionable issues were found. Checks were based on code and tests; tests were not run during this review.

_{Reviews (2) · Last reviewed commit: "fix(verifiers): keep OpenCode error reas..." · Reviewed by Greptile}