Skip to content

server: context checkpoints ride with a slot save/restore — a restored slot is warm again - #2

Merged
joelteply merged 7 commits into
feat/props-weight-residencyfrom
continuum/kv-checkpoints-survive-slot-save
Sep 13, 2026
Merged

joelteply merged 7 commits into
feat/props-weight-residencyfrom
continuum/kv-checkpoints-survive-slot-save

Conversation

@joelteply

Copy link
Copy Markdown

Why

/slots/N?action=restore brought back KV + tokens but prompt.clear() dropped the context checkpoints. On a hybrid/recurrent cache the next request can reuse a prefix only by rolling back to a checkpoint at or before its divergence (even an identical prompt rolls back one token, [TAG_PROMPT_LOGITS]), so a restored slot always hit do_reset: "restore ok" then a full re-prefill. Measured on Ornith-1.5-35B-A3B, 2026-09-04: n_restored 11298 → cache_n 0; the second identical request 11291. This is what made Continuum's KV context-switch (page to disk on eviction, restore on return) cost a full prefill on every return.

Change (server-context.cpp only, +73)

  • SLOT_SAVE writes <file>.ckpt beside the state file: magic/version/count, then per checkpoint n_tokens, pos_min, pos_max and the three state blobs (data_tgt, data_dft, data_spec).
  • SLOT_RESTORE reads it back into slot.prompt.checkpoints. A missing or malformed sidecar leaves exactly the pre-change behaviour (valid restore, no checkpoints) with a warning — never an error.

Acceptance (Continuum side)

serving/cache-probe --roundtrip: prime → save → restore → same prompt must report cache_n ≈ n_restored on the FIRST request after the restore (today: 0).

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… a restored slot is warm again

The slot state file carries KV + tokens; the context checkpoints (recurrent/SWA rollback
points) lived only in slot.prompt and were cleared by prompt.clear() on restore. On a
hybrid-cache model the next request can reuse a prefix only by rolling back to a checkpoint
at or before its divergence — and even an identical prompt rolls back one token
([TAG_PROMPT_LOGITS]) — so a restored slot always hit do_reset: "restore ok" followed by a
full re-prefill (measured 2026-09-04: n_restored 11298 → cache_n 0; the second identical
request 11291).

SLOT_SAVE now writes `<file>.ckpt` beside the state (magic, version, n; per checkpoint
n_tokens/pos_min/pos_max + the three state blobs). SLOT_RESTORE reads it back into
slot.prompt.checkpoints; a missing or malformed sidecar leaves the pre-fork behaviour
(valid restore, no checkpoints) with a warning, never an error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply

Copy link
Copy Markdown
Author

A/B on the M5 (Ornith-1.5-35B-A3B Q4_K_M, Metal, --cache-ram 0 so the disk restore is what is measured; 1321-token prompt; prime → save → clobber the slot with another prompt → restore → same prompt):

build first request after restore second
installed b10757 (pin 76dfd64) prompt_n 1321, cache_n 0, 4342 ms cache_n 1317, 172 ms
this PR prompt_n 4, cache_n 1317, 163 ms cache_n 1317, 162 ms

Server log: saved 3 context checkpoints beside the slot state / restored 3 context checkpoints from the sidecar. Sidecar 197 MB for 3 checkpoints on this hybrid model (~65 MB each: the recurrent state), so checkpoint spacing decides the sidecar size — follow-up in Continuum.

…mat-truncation on the CI (cpu) x64 job

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@github-actions github-actions Bot added the ggml label Sep 13, 2026
joelteply and others added 5 commits September 13, 2026 13:43
…truncation)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… whole restore round-trip (4096 tokens) — fast hosted runners ended the 512-token gen before the assertion

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… story' in a few tokens on hosted runners, which read as a binding failure

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…erve both slots busy before the restores queue

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…restore returns beside its OWN slot's end, not the long slot's; skips when a runner cannot separate the slots

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply
joelteply merged commit 965d38a into feat/props-weight-residency Sep 13, 2026
13 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant