perf(cache): keep the prompt-cache prefix stable and request/report the cache#20
Open
santhreal wants to merge 2 commits into
Open
perf(cache): keep the prompt-cache prefix stable and request/report the cache#20santhreal wants to merge 2 commits into
santhreal wants to merge 2 commits into
Conversation
The position-0 system message is the prompt-cache prefix, but it was rebuilt each turn to include the prompt-focused repo map (ranked by the current user message). That made the prefix change on every repo-related turn, invalidating the whole cached prefix -- system, tools, and all prior turns -- for both hosted prompt caching and local KV/prefix reuse. Build the system message from a prompt-independent stable prompt and fold the focused repo map into the current user turn instead, below the cache boundary. Durable history keeps the user's raw prompt; one-shot/eval callers are unchanged.
Now that the position-0 system prefix is stable across turns, make the client actually use the hosted prompt cache instead of merely being able to resend the same prefix. - Send a stable prompt_cache_key on every agent-loop request, derived from model + the cacheable system prefix (session_cache_key). Hosted providers use it to co-locate a session on one warm cache shard; local backends omit it. - Deserialize prompt_tokens_details.cached_tokens from usage, aggregate it into RunResult.cached_input_tokens, and surface "<n> cached" in the per-turn footer when the provider reports a hit. Tests: session_cache_key stability/scoping, prompt_cache_key wire serialization, and footer cached-token rendering.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The interactive loop rebuilt the position-0 system message every turn to embed a prompt-ranked repo map (
Focused map for <user prompt>), whose content changes with each user prompt. Since prompt caching — OpenAI's hosted auto-cache and local llama.cpp/Ollama/vLLM KV reuse alike — matches on the longest shared token prefix, a system message that changes every turn invalidated the entire cached prefix (system + tools + prior turns) on every repo-related turn. The harness even re-runswarmupwhenever the system-prompt fingerprint changes, so the churn was already costing real requests.Fix, two parts:
Keep the prefix stable.
run_user_turnnow builds msg[0] from a newrender_stable_system_prompt(base prompt + static.small-harness/prompt.mdonly, no focused map), and folds the prompt-focused map into the current user message (below the cache boundary) in the request copy only — so the map still reaches the model for the turn it was built for, durable history keeps the raw prompt, and the cached prefix survives across turns. One-shot/eval/status/compact callers are unchanged.Actually use and report the cache. Send a stable
prompt_cache_key(derived from model + the cacheable system prefix) on hosted requests so a session co-locates on one warm cache shard; omitted for local backends. Read backprompt_tokens_details.cached_tokens, aggregate it intoRunResult, and show<n> cachedin the per-turn footer when the provider reports a hit — for both the chat-completions and Codex Responses paths.Tests: prefix stability (full prompt varies by input while the stable prompt carries no focused map), context folds into the last user message not the system prefix,
session_cache_keystability/scoping,prompt_cache_keywire serialization, footer cached-token rendering, and Codexresponse.completedcached-token parsing.cargo fmt/clippy -D warningsclean.