Repository navigation
server: context checkpoints ride with a slot save/restore — a restored slot is warm again - #2
Merged
joelteply merged 7 commits intoSep 13, 2026
Conversation
… a restored slot is warm again The slot state file carries KV + tokens; the context checkpoints (recurrent/SWA rollback points) lived only in slot.prompt and were cleared by prompt.clear() on restore. On a hybrid-cache model the next request can reuse a prefix only by rolling back to a checkpoint at or before its divergence — and even an identical prompt rolls back one token ([TAG_PROMPT_LOGITS]) — so a restored slot always hit do_reset: "restore ok" followed by a full re-prefill (measured 2026-09-04: n_restored 11298 → cache_n 0; the second identical request 11291). SLOT_SAVE now writes `<file>.ckpt` beside the state (magic, version, n; per checkpoint n_tokens/pos_min/pos_max + the three state blobs). SLOT_RESTORE reads it back into slot.prompt.checkpoints; a missing or malformed sidecar leaves the pre-fork behaviour (valid restore, no checkpoints) with a warning, never an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
Author
|
A/B on the M5 (Ornith-1.5-35B-A3B Q4_K_M, Metal,
Server log: |
…mat-truncation on the CI (cpu) x64 job Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…truncation) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… whole restore round-trip (4096 tokens) — fast hosted runners ended the 512-token gen before the assertion Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… story' in a few tokens on hosted runners, which read as a binding failure Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…erve both slots busy before the restores queue Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…restore returns beside its OWN slot's end, not the long slot's; skips when a runner cannot separate the slots Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
merged commit Sep 13, 2026
965d38a
into
feat/props-weight-residency
13 of 32 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
/slots/N?action=restorebrought back KV + tokens butprompt.clear()dropped the context checkpoints. On a hybrid/recurrent cache the next request can reuse a prefix only by rolling back to a checkpoint at or before its divergence (even an identical prompt rolls back one token,[TAG_PROMPT_LOGITS]), so a restored slot always hitdo_reset: "restore ok" then a full re-prefill. Measured on Ornith-1.5-35B-A3B, 2026-09-04: n_restored 11298 → cache_n 0; the second identical request 11291. This is what made Continuum's KV context-switch (page to disk on eviction, restore on return) cost a full prefill on every return.Change (server-context.cpp only, +73)
SLOT_SAVEwrites<file>.ckptbeside the state file: magic/version/count, then per checkpointn_tokens, pos_min, pos_maxand the three state blobs (data_tgt,data_dft,data_spec).SLOT_RESTOREreads it back intoslot.prompt.checkpoints. A missing or malformed sidecar leaves exactly the pre-change behaviour (valid restore, no checkpoints) with a warning — never an error.Acceptance (Continuum side)
serving/cache-probe --roundtrip: prime → save → restore → same prompt must reportcache_n ≈ n_restoredon the FIRST request after the restore (today: 0).🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo