fix: Hermes agent cache token-mismatch#581
Open
vestigecounty wants to merge 1 commit into
Open
Conversation
A thinking model that emits tool calls (or a final answer) leaves its hidden <think> reasoning in the live KV, but the next request re-renders that turn with an empty <think></think> because clients replay the tool-call bytes without the reasoning. The live checkpoint therefore diverged from the next prompt at the first reasoning token, so every follow-up turn missed the cache (memory_miss_reason token-mismatch, cached_tokens 0) and re-prefilled the whole conversation. Remember a reasoning-stripped visible key for the live frontier after a tool-call turn (both the exact-DSML-replay and canonicalize paths) and after a tools-enabled thinking final answer, mirroring the existing tool-less thinking_live mechanism. thinking_live_visible_prefix_prompt then reuses the verbatim live KV while tokenizing only the new suffix. The final-answer path deliberately does not gate on prompt_preserves_reasoning: that flag is really "history uses tools" (always true for the agent workload) and would disable the path entirely. Reuse stays gated by an exact byte-prefix match, so a client that genuinely replays reasoning simply no-ops. Adds server unit tests covering the tool-call and final-answer reasoning-stripped roundtrips and the build_tool_checkpoint_suffix empty-reasoning contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes some of the continuous cache misses when using with the Hermes agent.
NOT fixed:
auxiliary.title_generation.enabled: falsedelegation.max_concurrent_children: 1,delegation.max_async_children: 1Reuse live KV across thinking tool-call turns (thinking-visible)
A thinking model that emits tool calls (or a final answer) leaves its hidden reasoning in the live KV, but the next request re-renders that turn with an empty because clients replay the tool-call bytes without the reasoning. The live checkpoint therefore diverged from the next prompt at the first reasoning token, so every follow-up turn missed the cache (memory_miss_reason token-mismatch, cached_tokens 0) and re-prefilled the whole conversation.
Remember a reasoning-stripped visible key for the live frontier after a tool-call turn (both the exact-DSML-replay and canonicalize paths) and after a tools-enabled thinking final answer, mirroring the existing tool-less thinking_live mechanism. thinking_live_visible_prefix_prompt then reuses the verbatim live KV while tokenizing only the new suffix.
The final-answer path deliberately does not gate on prompt_preserves_reasoning: that flag is really "history uses tools" (always true for the agent workload) and would disable the path entirely. Reuse stays gated by an exact byte-prefix match, so a client that genuinely replays reasoning simply no-ops.
Adds server unit tests covering the tool-call and final-answer reasoning-stripped roundtrips and the build_tool_checkpoint_suffix empty-reasoning contract.