Skip to content

Fix cache-corrupted token metrics in multi-turn Claude Code eval - #509

Open
prernakakkar-google wants to merge 6 commits into
mainfrom
cache-vertex
Open

Fix cache-corrupted token metrics in multi-turn Claude Code eval#509
prernakakkar-google wants to merge 6 commits into
mainfrom
cache-vertex

Conversation

@prernakakkar-google

@prernakakkar-google prernakakkar-google commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Problem
Multi-turn claude_code evaluations drove each turn with a fresh claude -p --resume --fork-session subprocess. Prompt-cache continuity does not survive across separate claude -p invocations, so every turn after the first re-paid a cold cache_creation cost instead of reading from cache. This inflated cache_creation_input_tokens and corrupted the token metrics the eval exists to measure (tokens_processed, effective_billed_tokens). (known issue)

Fix
Drive an entire scenario through one persistent claude process using Claude Code's streaming-input mode (-p --input-format stream-json --output-format stream-json --verbose): each turn is a newline-delimited user message on stdin, and the process stays alive across turns, so session context and the prompt cache persist. Closing stdin ends the session.

Introduces a small per-scenario session abstraction (AgentCliSession) so the change is backward-compatible:

  1. Default _PerTurnSession reproduces the old --resume-per-turn behavior exactly → gemini_cli / codex_cli / agy_cli are unchanged.
  2. ClaudeCodeGenerator overrides start_session() with a persistent streaming session (background stderr drain to avoid pipe deadlock, per-turn watchdog timeout, clean stdin-EOF shutdown, error path when the process dies before a result). Reuses the existing _parse_stream_json unchanged.

Also included (related: Vertex cache accuracy)
Default Claude Vertex calls to the global endpoint (vertex_region: "global"); regional pinning worsens cache-hit determinism and is unsupported past Sonnet 4.6.

https://paste.googleplex.com/newui/?id=5943794385158144

Comment thread evalbench/generators/models/claude_code.py Fixed
Comment thread evalbench/generators/models/claude_code.py Fixed
Comment thread evalbench/generators/models/claude_code.py Fixed
Comment thread evalbench/generators/models/claude_code.py Fixed
@prernakakkar-google prernakakkar-google changed the title Cache vertex Fix cache-corrupted token metrics in multi-turn Claude Code eval Jul 21, 2026
prernakakkar-google and others added 4 commits July 21, 2026 14:10
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
@prernakakkar-google

Copy link
Copy Markdown
Collaborator Author

/gcbrun

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants