Skip to content

train: one training-context derivation; K/V format resolved and recorded (6edd495c slice 1) - #60

Closed
joelteply wants to merge 1 commit into
masterfrom
6edd495c/shared-admission
Closed

joelteply wants to merge 1 commit into
masterfrom
6edd495c/shared-admission

Conversation

@joelteply

Copy link
Copy Markdown
  • One owner for whether a K/V format can hold the model: llama_kv_cache_types_check.
    llama_init_from_model refuses by it, and a caller choosing a format asks it first
    instead of reading a null context, which also means out of memory.
  • One cache-type parser in common (common_kv_cache_type_from_str / _types / _types_str).
    arg.cpp's file-static copy is deleted.
  • common_context_params_for_training holds the training mode's only differences from
    serving: no rollback slots, chunk = batch = ubatch, a 2-sequence unified cache,
    flash attention, every output, no embeddings. server-train's hand overrides are deleted.
  • The training K/V format is RESOLVED, not hard-coded: requested (cache_type_k/v on
    /train), else serving's, else the default Q8_0. Each counts only where training has a
    measured run with it (F16 c424b00, Q8_0 a8a235d). Only a requested format is
    refused, by name. Serving at Q4 still trains, at Q8_0, as today.
  • The receipt records training_cache {type_k, type_v, source} beside serving_cache.

Where it changes behavior: the core's KvCachePlan serves Q8_0 on CUDA and Metal
(kv_cache_plan.rs:160), so the 5090 and the M5 still train at Q8_0, unchanged. CPU and
Vulkan serving is F16, so those nodes now train at F16 instead of Q8_0: twice the
training K/V bytes, charged to the same lease (budget minus context bytes), and no
quantized dequant on a CPU. Not measured yet; the receipt is a /train beside
responsive serving.

Compiled: clang++ -fsyntax-only on the four changed TUs (IntelMac, 2026-10-13 ~17:00Z).
Not built or run.

Supersedes #59's context_cache_type parser (a copy of kv_cache_type_from_str); #59 should be closed in favour of this.

Not in this slice: shared physical-pool admission (slice 2, after the Metal unified-memory type prerequisite with its MoE-cache measurement), and the chat-input mismatch (serving reviewer).

Receipt needed before adoption: a /train beside responsive serving, on the 5090 or M5 (Q8_0 path, unchanged) and on a CPU node (F16 path, changed), with training_cache and serving_cache from the receipt, plus context_mib, graph_mib, SHA and time.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

…s K/V format resolved and recorded (continuum card 6edd495c, slice 1)

- One owner for whether a K/V format can hold the model: llama_kv_cache_types_check.
  llama_init_from_model refuses by it, and a caller choosing a format asks it first
  instead of reading a null context, which also means out of memory.
- One cache-type parser in common (common_kv_cache_type_from_str / _types / _types_str).
  arg.cpp's file-static copy is deleted.
- common_context_params_for_training holds the training mode's only differences from
  serving: no rollback slots, chunk = batch = ubatch, a 2-sequence unified cache,
  flash attention, every output, no embeddings. server-train's hand overrides are deleted.
- The training K/V format is RESOLVED, not hard-coded: requested (cache_type_k/v on
  /train), else serving's, else the default Q8_0. Each counts only where training has a
  measured run with it (F16 c424b00, Q8_0 a8a235d). Only a requested format is
  refused, by name. Serving at Q4 still trains, at Q8_0, as today.
- The receipt records training_cache {type_k, type_v, source} beside serving_cache.

Where it changes behavior: the core's KvCachePlan serves Q8_0 on CUDA and Metal
(kv_cache_plan.rs:160), so the 5090 and the M5 still train at Q8_0, unchanged. CPU and
Vulkan serving is F16, so those nodes now train at F16 instead of Q8_0: twice the
training K/V bytes, charged to the same lease (budget minus context bytes), and no
quantized dequant on a CPU. Not measured yet; the receipt is a /train beside
responsive serving.

Compiled: clang++ -fsyntax-only on the four changed TUs (IntelMac, 2026-10-13 ~17:00Z).
Not built or run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply
joelteply marked this pull request as draft October 13, 2026 19:14
@joelteply

Copy link
Copy Markdown
Author

Closing unmerged. This extends /train as a job in a SECOND context. The design (docs/architecture/ONE-RESIDENT-MODEL-PATIENT-DOCTOR-DREAM.md, S4) is a DREAM stage in the server loop, one optimizer micro-step between update_slots decode batches on the resident context, and Joel restated it on 2026-10-13. I built on the second context without reading that doc first; this PR refines the shape the design deletes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant