Skip to content

Match a cached llama.cpp context only to its live session - #423

Merged
mattt merged 1 commit into
mainfrom
mattt/fix-stream-usage-parity
Nov 18, 2026
Merged

mattt merged 1 commit into
mainfrom
mattt/fix-stream-usage-parity

Conversation

@mattt

@mattt mattt commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

This PR fixes a case where a new session could reuse the KV cache of a session that was already freed. LlamaLanguageModel keeps one llama.cpp context for a session, keyed by the session's ObjectIdentifier. When a session is freed, the next one can get the same address, so it matched the freed session's context and reported the reused prompt prefix as cached input tokens on its first request.

The cached context now holds its session weakly and matches only that session while it's alive, as the MLX backend's KV store already does. A new session's first request decodes its full prompt and reports no cached tokens.

@mattt
mattt merged commit c4c150b into main Nov 18, 2026
14 checks passed
@mattt
mattt deleted the mattt/fix-stream-usage-parity branch November 18, 2026 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant