Repository navigation
Conversation
…s K/V format resolved and recorded (continuum card 6edd495c, slice 1) - One owner for whether a K/V format can hold the model: llama_kv_cache_types_check. llama_init_from_model refuses by it, and a caller choosing a format asks it first instead of reading a null context, which also means out of memory. - One cache-type parser in common (common_kv_cache_type_from_str / _types / _types_str). arg.cpp's file-static copy is deleted. - common_context_params_for_training holds the training mode's only differences from serving: no rollback slots, chunk = batch = ubatch, a 2-sequence unified cache, flash attention, every output, no embeddings. server-train's hand overrides are deleted. - The training K/V format is RESOLVED, not hard-coded: requested (cache_type_k/v on /train), else serving's, else the default Q8_0. Each counts only where training has a measured run with it (F16 c424b00, Q8_0 a8a235d). Only a requested format is refused, by name. Serving at Q4 still trains, at Q8_0, as today. - The receipt records training_cache {type_k, type_v, source} beside serving_cache. Where it changes behavior: the core's KvCachePlan serves Q8_0 on CUDA and Metal (kv_cache_plan.rs:160), so the 5090 and the M5 still train at Q8_0, unchanged. CPU and Vulkan serving is F16, so those nodes now train at F16 instead of Q8_0: twice the training K/V bytes, charged to the same lease (budget minus context bytes), and no quantized dequant on a CPU. Not measured yet; the receipt is a /train beside responsive serving. Compiled: clang++ -fsyntax-only on the four changed TUs (IntelMac, 2026-10-13 ~17:00Z). Not built or run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
joelteply
marked this pull request as draft
October 13, 2026 19:14
Author
|
Closing unmerged. This extends /train as a job in a SECOND context. The design (docs/architecture/ONE-RESIDENT-MODEL-PATIENT-DOCTOR-DREAM.md, S4) is a DREAM stage in the server loop, one optimizer micro-step between update_slots decode batches on the resident context, and Joel restated it on 2026-10-13. I built on the second context without reading that doc first; this PR refines the shape the design deletes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
llama_init_from_model refuses by it, and a caller choosing a format asks it first
instead of reading a null context, which also means out of memory.
arg.cpp's file-static copy is deleted.
serving: no rollback slots, chunk = batch = ubatch, a 2-sequence unified cache,
flash attention, every output, no embeddings. server-train's hand overrides are deleted.
/train), else serving's, else the default Q8_0. Each counts only where training has a
measured run with it (F16 c424b00, Q8_0 a8a235d). Only a requested format is
refused, by name. Serving at Q4 still trains, at Q8_0, as today.
Where it changes behavior: the core's KvCachePlan serves Q8_0 on CUDA and Metal
(kv_cache_plan.rs:160), so the 5090 and the M5 still train at Q8_0, unchanged. CPU and
Vulkan serving is F16, so those nodes now train at F16 instead of Q8_0: twice the
training K/V bytes, charged to the same lease (budget minus context bytes), and no
quantized dequant on a CPU. Not measured yet; the receipt is a /train beside
responsive serving.
Compiled: clang++ -fsyntax-only on the four changed TUs (IntelMac, 2026-10-13 ~17:00Z).
Not built or run.
Supersedes #59's context_cache_type parser (a copy of kv_cache_type_from_str); #59 should be closed in favour of this.
Not in this slice: shared physical-pool admission (slice 2, after the Metal unified-memory type prerequisite with its MoE-cache measurement), and the chat-input mismatch (serving reviewer).
Receipt needed before adoption: a /train beside responsive serving, on the 5090 or M5 (Q8_0 path, unchanged) and on a CPU node (F16 path, changed), with training_cache and serving_cache from the receipt, plus context_mib, graph_mib, SHA and time.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE