Repository navigation
Conversation
…ache_type: q8_0 default, q4_0, f16) Continuum card 34d574fc: a dream step must fit BESIDE serving rather than borrow from it. Kimi's smallest whole window on the 5090 needs 5,104,337,280 B against 3,894,411,264 B governed free. Part of that is the training context, whose Q8_0 K/V for the whole window stacks beside serving (:1020-1023 records ~0.6 GB at 16,896). It was hard-coded Q8_0. - context_cache_type, validated before the claim like every field; an unknown type is refused by name. Default q8_0, so a request without it is unchanged. - The cache holds only constants (the context before a chunk); a chunk's own K/V stay F32 in the graph, as before. - The run's state reports context_cache_type beside context_mib, so a measured row says what it measured. A refused preflight already reports graph_mib and context_mib, so a measurement needs no training. Test (test-chat --train-control): the default is q8_0; an unknown type is refused; a q4_0 run reports its type and a smaller context_mib than the q8_0 baseline at the same window. Not built locally; CI is the first build.
…t moment Codex root on #59: a refused preflight's receipt must name the requested context_cache_type, recompute and shape beside its separate memory terms; the type was only written at the final state. The starting state now records context_cache_type, chunk, recompute and exact (window was already there), so a refusal at context creation, a late chunk refusal and a finished run all say what they measured. The redundant final-state assignment is removed.
joelteply
added a commit
that referenced
this pull request
Oct 13, 2026
…graph needed and what the device allowed then Codex root on #59 / continuum 34d574fc: 9c8cb1ae failed after 1,955 windows with 'the training graph did not fit in device memory at window 45056'. That's a memory refusal, but not which chunk or why. Two explanations fit and read differently: (1) the deepest chunk: a chunk's graph grows with its depth in the window (attention to everything cached before it, gradient taps over the horizon), and the deepest one appears only on the longest example; (2) a falling allowance: each chunk's allowance is the device's free memory AT THAT MOMENT (capped by the budget, minus slack), and her live turns allocate beside training. - ggml_opt remembers, for the device that refused, the bytes the graph needed and the bytes it was allowed (ggml_opt_last_refusal_bytes). Kept apart from , which stays empty for a memory refusal so the exact walk's horizon-halving retry is unchanged. - llama_context remembers the refused chunk's position and length (llama_opt_last_refusal), reset per run. - On a no-fit end, /train state reports refused_chunk_pos, refused_chunk_tokens, refused_need_mib and refused_allowance_mib beside graph_mib (the allocator's own peak, which already covers every allocation, not a sample). Not testable on CPU CI (the budget gate applies to non-CPU devices); the next 5090 refusal is its receipt.
…graph needed and what the device allowed then Codex root on #59 / continuum 34d574fc: 9c8cb1ae failed after 1,955 windows with 'the training graph did not fit in device memory at window 45056'. That's a memory refusal, but not which chunk or why. Two explanations fit and read differently: (1) the deepest chunk: a chunk's graph grows with its depth in the window (attention to everything cached before it, gradient taps over the horizon), and the deepest one appears only on the longest example; (2) a falling allowance: each chunk's allowance is the device's free memory AT THAT MOMENT (capped by the budget, minus slack), and her live turns allocate beside training. - ggml_opt remembers, for the device that refused, the bytes the graph needed and the bytes it was allowed (ggml_opt_last_refusal_bytes). Kept apart from the refusal string, which stays empty for a memory refusal so the exact walk's horizon-halving retry is unchanged. - llama_context remembers the refused chunk's position and length (llama_opt_last_refusal), reset per run. - On a no-fit end, /train state reports refused_chunk_pos, refused_chunk_tokens, refused_need_mib and refused_allowance_mib beside graph_mib (the allocator's own peak, which already covers every allocation, not a sample). Not testable on CPU CI (the budget gate applies to non-CPU devices); the next 5090 refusal is its receipt.
joelteply
force-pushed
the
34d574fc/training-context-cache-type
branch
from
October 13, 2026 05:23
4742581 to
830a379
Compare
Author
|
Closing unmerged. This extends /train as a job in a SECOND context. The design (docs/architecture/ONE-RESIDENT-MODEL-PATIENT-DOCTOR-DREAM.md, S4) is a DREAM stage in the server loop, one optimizer micro-step between update_slots decode batches on the resident context, and Joel restated it on 2026-10-13. I built on the second context without reading that doc first; this PR refines the shape the design deletes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Continuum card 34d574fc: a dream step has to fit beside serving, never by borrowing from it. Her live slot and cache are never touched (constraints from Fable, approach agreed with Codex root).
Kimi's smallest whole window on the 5090 (core 323dc8f91) needs 5,104,337,280 B against 3,894,411,264 B governed free. Part of that is the training context: its K/V for the whole window stacks beside serving, and the comment at
server-train.cpp~:1020 records ~0.6 GB at a 16,896-token window. Its type was hard-coded Q8_0.What
context_cache_type, one ofq8_0(default),q4_0orf16, for the training context's K/V.context_cache_typebesidecontext_mib, so a measured row says what it measured. A refused preflight already reportsgraph_mibandcontext_mib, so measuring a shape needs no training run.Next (the receipt that decides the approach)
Whoever holds the hardware lease (Codex root on the 5090, Fable on the M5 27B) measures, at Kimi's smallest whole window:
{q8_0, q4_0} x {recompute off, on}, at chunk 64;If no cell fits, that receipt sends us to the borrow design, with Fable's a-d as its spec.
Test
test-chat --train-controlchecks three things:context_mibthan the q8_0 baseline at the same window.Not built locally; CI is the first build.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE