train in place: labels ride the batch, the reply learns on the slot that served it (POST /learn) by joelteply · Pull Request #65 · CambrianTech/llama.cpp

joelteply · 2026-10-15T06:56:37Z

Training in place on the serving lane: a reply learns on the slot that served it, in the serving context, with no second context and no second copy of her window. One line that supersedes #61, #62 and #64 (all included), plus the Metal training ops (fable/metal-train-ops) and Cormac's CPU attention backward (cormac/cpu-attn-back).

What changes

Labels ride the batch (llama_batch.label): a labelled ubatch is a training step inside decode(); the trainable layer (the shadow) applies to labelled rows only, by row selection; a batch without labels is the serving graph bit for bit.
Frozen served prefix: a reply trains given everything before it as served; the exact walk's backward covers the reply only, and its K/V gradient is sized to the reply.
Memory: cross-entropy over one target index per row (no one-hot), the head computed a row slice at a time, the attention backward as one op (ggml_flash_attn_back: CPU, Metal tiled on simdgroup matrices; CUDA in cuda: implement shared FLASH_ATTN_BACK with direct quantized cache reads #63), the acc_or_set leaf fix, no host logits for loss-only rows.
llama_opt_learn trains one reply on a live sequence and restores it; refusals precede any change.
POST /learn (with --train-dir): the served conversation plus the reply or correction, checked against her slot's prompt token by token, her recurrent state from a checkpoint one token before the prompt end, one optimizer step per call, a 10% duty cycle while other slots work, her slot returned to the whole served prompt.
/train is unchanged: the core keeps using it until it moves to /learn.

Measured (M5, Metal, 2026-10-14/15)

test-walk-exact on qwen2.5-0.5b and qwen3.5-4b hybrid: every exact row 1.0000/1.0000; the cache after training byte-identical to her stable decode; learning on a live sequence equal to the walk's exact step (1.0000/1.0000) in a unified cache and in a stream-per-slot f16 flash cache, the neighbour byte-identical.
4B chunk graph at window 8192, chunk 512, recompute: 2,417 -> 1,287 MiB.
FLASH_ATTN_BACK Metal: 196 -> 67 ms per call at a training shape; CPU (Cormac, i7-8850H): 2,195.5 -> 316.2 ms; Metal tiled vs CPU 5/5 at NMSE 1e-5.
test-backend-ops MTL0: FLASH_ATTN_BACK 5/5, OUT_PROD 414/414, GET_ROWS_BACK 4/4, PAD 30/30, SET_ROWS 172/172, CROSS_ENTROPY_LOSS 5/5, RMS_NORM_BACK 20/20, SOFT_MAX_BACK 24/24.
llama-server, 4B, 2 slots: a 17-token correction learned 4 times, loss 2.860 -> 1.111, ~300 ms per learn; the served conversation + correction, loss 1.573 -> 0.474; an unserved conversation refused at the first differing token; the other slot's per-token p50/p95 46.8/73.3 -> 46.1/73.8 ms with learning on (her live lane busy on the same GPU: paired comparison).

Measured after deploy (owner Fable)

On her lane (Qwen3.8-27B, 2 slots, q8_0 cache, served window): the memory peak of a learning step against the serving reservation, the other slot's p50/p95 with learning on vs off, and the loss falling on a correction. The core does not call /learn on her lane until these are posted (agreed with Cormac).

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…uns in a context with rollback slots Serving keeps recurrent rollback slots for drafts (n_rs_seq > 0); the exact walk refused any context that had them, which is what forced a second, rollback-free training context beside serving. llm_graph_context::rs_keep() is now the one rule every family's state write and every snapshot-producing op reads (delta-net, mamba, lfm2, bailing; each decided it separately before): serving with rollback keeps n_rs_seq + 1 states, a training graph keeps one, the live state in group 0, which is the same bytes in both layouts. The walk now refuses only a sequence with an undo PENDING (llama_memory_recurrent::rollback_pending), and refuses by name a model whose state write does not report its exit to the walk, where the gradient across chunks was silently dropped before. Measured on the M5 (Metal), base 1782249: - serving unchanged: 19 recurrent/hybrid archs x n_rs_seq {0, 4}, random weights, 100-token prefill + 4 single decodes + (where rollback is live: qwen35, qwen35moe, deepseek4, nemotron_h, nemotron_h_moe, bailingmoe3, lfm2, lfm2moe) a 2-token undo and redo: logits and full sequence state byte-identical to the base, 38 of 38 rows. - training: test-walk-exact on qwen3.5-4b Q4_K_M, the exact walk in a context with n_rs_seq 4 vs 0: cosine 1.0000, norm ratio 1.0000; every other check in the file unchanged (exit 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… so training stops falling back to the CPU A LoRA training epoch on the M5 spent most of its time on the CPU: Metal's OUT_PROD refused (1) any ne[2]/ne[3] above 1, which is every batched backward with grouped heads (the 4B's delta-net backward), and (2) any src0 width not a multiple of 16, which is every LoRA parameter gradient at rank 8. The 16-multiple rule only matters for the dequantizing copy of a quantized src0; it now applies only there. The 2D path becomes a per-slice helper, and the op loops over dims 2-3 with the CPU backend's broadcast rule (dst per src0 = ne2/ne02, ne3/ne03), reusing one slice of scratch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ed copy-and-transpose path is deleted OUT_PROD was composed from a dequantizing copy, two transposing copies, a mul_mm and an add per row block, over scratch reserved after dst. Every pass wrote device memory and read it back, so the backward matmul ran at about half the forward's rate. kernels/out_prod.metal is mul_mm with the contraction along the rows of both operands. Each thread dequantizes 16 consecutive elements of one weight row in the weight's own block order and stages them transposed in threadgroup memory. The gradient is read with its own strides, so both layouts work, and staging is float. Batching and broadcast come from the grid. Deleted with it: the row blocks, OUT_PROD_BLOCK_BYTES, the scratch after dst and its get_alloc_size hook, and the synthetic-view helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… so a LoRA training graph stays on the GPU The ops a Metal training graph still sent to the CPU after OUT_PROD, each refused by supports_op: - PAD with left padding. The kernel now reads src[i - lp] per dim, through the source's element stride (a training graph pads a view). - RMS_NORM_BACK and SOFT_MAX_BACK on strided rows (a per-head norm). Rows are now addressed through each tensor's dim 1-3 strides, and each row stays contiguous. - GET_ROWS_BACK (no kernel). This is the gradient of the output-row gather. One threadgroup per dst row sums, in id order, the source rows whose id is that row: no atomics, and the same summation order as the CPU reference. Tests: left-pad cases per dim on contiguous, non-contiguous and permuted sources, and an RMS_NORM_BACK over row views of a twice-wide tensor. Not in this change: ADD with a transposed src1 (a strided read in the hot binary kernel; it measured about 1% of a 4B epoch). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…o every ubatch P1 step 1 (continuum ggml-org#5032 section 9.6): training is inference plus extras that switch on and off, so the switch is in the batch. llama_batch gains `label`: the target token at that position, -1 for none, and NULL for an inference batch, which is unchanged. llama_batch_init allocates it filled with -1. The allocator carries labels into each ubatch the same way it carries the output flags, counts them (get_n_labels), and refuses by name a labelled position that is not an output, since the loss reads that position's logits row. Next (step 2): decode() runs a ubatch with labels through ggml_opt in the same loop, and llama_context::opt_walk::train_chunk, the second copy of decode's batch loop, is deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…k's copy of the batch loop is gone P1 step 2 (continuum ggml-org#5032 section 9.6). process_ubatch has one switch: a ubatch whose rows carry labels, on a context with a trainable adapter, is built fresh and runs through ggml_opt (llama_context::opt_ubatch: the loss over the labelled rows, the backward, the optimizer step at a period's end, and the active walk's taps). Every other ubatch runs graph_compute as before. opt_walk::train_chunk, which repeated decode's batch loop, now fills a chunk's labels and calls llama_decode. - cparams.training joins the graph reuse key. It was missing, so a serving decode with the same shape as a training ubatch could reuse the training graph. - Every batch view (server, speculative, mtmd, batched-bench, perplexity, parallel) carries its slice of the labels. - test-walk-exact: the device-refusal case is skipped on a CPU-only run, because the budget caps GPU memory only (ggml_opt_alloc). Its old skip test (graph size == 0) never fired: the CPU graph is measured in host buffers. Measured (fork bdd3cd5 + 32b8657 + this, M5, --device none, qwen3.5-4b hybrid Q4_K_M, 2026-10-14T04:46Z-06:01Z): every exact row is 1.0000 through the new path (exact walk vs one graph, checkpointed state, f16 and q8_0 caches, context + reply, recompute, rollback slots). The plain walk's rows (0.8376, 0.6710) are close to the pre-P1 Metal values (0.8378, 0.6714); the difference is backend, not path. There is no pre-P1 CPU run to compare against. Still owed: the Metal run, for the device-refusal case (the 4B, P1 vs bdd3cd5, in a quiet window). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ed run is restored P1 step 3a (continuum ggml-org#5032 section 9.6). The objective is each labelled run (a reply) given everything before it as served: only labelled rows see the shadow (the adapter being trained), so the context before a run carries no gradient and is never trained. The exact walk's backward now covers the reply only, not the window. - The shadow is added on the labelled rows by row selection (llm_graph_context::shadow_add): those rows of the base and of the shadow's term are gathered, added, and written into a copy of the base, so every other row is the stable graph's byte for byte. A ubatch with no label never applies the shadow; the label count is in the graph reuse key. (A 0/1 mask multiply was rejected in review: inf or NaN times zero is NaN, and x + 0 turns -0 into +0.) - SET_ROWS gains the destination's gradient in ggml backward (the result's gradient except at the replaced rows), which row selection needs. - The walk goes run by run: the context before a run decodes under her stable set; the run decodes under the shadow, trains (exact: in reverse, with the K/V and recurrent-state adjoints inside the run only; plain: a step per chunk), and is then restored, its cells and recurrent state decoded again under her stable set. One checkpoint helper and one rewind serve both walks; the plain walk's scratch sequence (n_seq_max >= 2) is gone. - decode_span no longer toggles cparams.training: process_ubatch sets it per ubatch (one of the four forward differences in section 9.6 removed). - The batch allocator refuses, before any mutation, a labelled row that is not an output or has more than one sequence, and a sequence whose labelled rows are not one run ending its rows in the batch. A training ubatch whose output rows are not exactly its labelled rows fails by name: the loss is their mean. - Tests: test-walk-plan for the per-run marks; test-walk-exact case 8, the cache after either walk byte-identical to a fresh stable decode; test-backend-ops SET_ROWS with grad=1. Measured: in the follow-up receipts (CPU and Metal, this SHA). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ot the window P1 step 3b (continuum ggml-org#5032 section 9.6). Under the served-prefix objective only a run's own positions carry K/V gradient, so the host accumulator is sized to the longest run and rebased to each run's start (llama_walk_acc::start_run, cparams.walk_acc_base). Before, it was sized to the window: at her served window on a 27B that alone was gigabytes of host memory. - The in-place (flash) path taps the window's cells but keeps only the run's rows before the chunk: ggml_opt_accumulate_grad_rows adds rows [row0, row0 + n) of a tensor's gradient into a caller's span; the rest stays on the device for its consumers. What later chunks put on a chunk's own rows reaches it through the surrogate, now one helper for both attention paths (build_walk_surrogate). - The host budget's refusal names the reply, which is what the K/V gradient grows with. Measured (this commit, M5 Metal, test-walk-exact, 2026-10-14): qwen2.5-0.5b 12:56:43-12:57:41Z and qwen3.5-4b hybrid 12:57:41-13:01:53Z, both OK; every exact row 1.0000/1.0000; the cache after either walk byte-identical to her stable decode; device refusal 3 retries, 4 refusals, 0 ubatch failures. Owed: the 27B at her served window with a 2k reply, accumulator bytes and memory peak measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…s no vocabulary-wide one-hot P1 (continuum ggml-org#5032 section 9.6, the precision and memory rows). A training step's labels were a one-hot F32 tensor the shape of the logits: [vocab x rows], 485 MiB at a 248k vocabulary and a 512-row chunk, beside the logits themselves. - CROSS_ENTROPY_LOSS and its backward accept I32 labels, one target per row, -1 for a row that carries no loss (ggml_cross_entropy_labels_ok names the two forms). CPU, Metal and CUDA kernels; SYCL refuses the I32 form until it has one. - ggml-opt: GGML_OPT_LOSS_TYPE_CROSS_ENTROPY_TARGETS (appended to the enum) builds an I32 labels input and counts accuracy against it directly. llama trains with it and writes each output row's target token. - GGML_OPT_TOP=N lists a training graph's N largest tensors at allocation: where its memory goes, measured. - test-backend-ops: CROSS_ENTROPY_LOSS with I32 targets (masked rows included), forward and finite-difference gradient. test_out_prod no longer crashes gradient mode (it built its weight in a null context), which had stopped every grad run. Measured (M5 Metal, 2026-10-14): test-backend-ops CROSS_ENTROPY_LOSS test and grad OK on CPU and MTL0 ([10,5,4,3] and [30000,2,1,1]); test-walk-exact OK on qwen2.5-0.5b (13:17-13:18Z) and qwen3.5-4b hybrid (13:18-13:23Z), every exact row 1.0000/1.0000 and the cache after either walk byte-identical to her stable decode. qwen3.5-4b chunk graph at window 8192, chunk 512, recompute (13:23-13:24Z): 2,417 -> 2,108 MiB on the flash path, 2,265 -> 2,037 MiB explicit. CUDA: not run here; owed on the 5090. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

P1 (continuum ggml-org#5032 section 9.6, the memory rows). The logits of a chunk and their gradient were the two largest tensors of its graph, [vocab x rows] F32 each (485 MiB apiece on the 4B at a 248k vocabulary and 512 rows), and their size does not shrink with the model. - ggml_opt_prepare_alloc_head: given the hidden state and the outputs (a row-wise function of it: MUL_MAT, MUL, ADD, SCALE, as a language model's head with its adapters and scales is), the head leaves the forward graph and is built again for each slice of rows; the loss is the mean over every row, each slice weighted by its share. Under recompute only one slice's logits and their gradient exist at a time. A head with any other node is not sliced. - llama trains with it, OPT_HEAD_SLICE_BYTES (64 MiB) of logits per slice: 67 rows at a 248k vocabulary. - ggml-opt rebuilt a per-step graph's loss into its predecessor's tensors when it chained the slices, and kept pred/ncorrect from the previous graph when a graph had none: both reset per graph now (the second read a freed tensor). Measured (M5 Metal, 2026-10-14): test-walk-exact OK on qwen2.5-0.5b (13:45:52-13:46:41Z) and qwen3.5-4b hybrid (13:46:41-13:52:10Z): every exact row 1.0000/1.0000, the cache after either walk byte-identical to her stable decode, the forward loss identical to the unsliced head (0.5B chunk 512: 4.122972 before and after). qwen3.5-4b chunk graph at window 8192, chunk 512, recompute (13:52:10-13:53:28Z): 2,108 -> 1,623 MiB on the flash path, 2,037 -> 1,552 explicit; the largest tensors are now the flash-attention backward's score tiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… and Metal P1 (continuum ggml-org#5032 section 9.6, the memory rows). FLASH_ATTN_EXT's backward was composed from generic ops over 4096-position tiles, three passes, keeping [tile x rows x heads] F32 score tensors (128 MiB each on the 4B at chunk 512) and adding up to 96 graph nodes per tile per layer, so it grew with the window. - ggml_flash_attn_back replaces the dead stub of the same name (it aborted: "TODO: adapt to ggml_flash_attn_ext"): q, k, v, mask in flash_attn_ext's layouts, d the output's gradient; one F32 result holding dQ, dK, dV (ggml_flash_attn_back_offset) and each query row's log-sum-exp and D. Scores are recomputed, never kept. - CPU: the reference (a pass over query rows, then over KV positions with the GQA group summed). - Metal: kernel_flash_attn_back_q / _kv for F32, F16 and q8_0 caches, one simdgroup per row; dq = scale (sum p dp k - D sum p k), so two sweeps over the window instead of three. - FLASH_ATTN_EXT's backward builds the op when the forward's back tile is GGML_FA_BACK_NATIVE; llama selects it when every GPU it computes on has the kernel (probed at the model's attention shape and the cache's types in opt_init), else the composed tiles (CUDA until its kernel). - test-backend-ops: FLASH_ATTN_BACK each backend against the CPU (F16, F32, q8_0, GQA, head 256, odd rows); FLASH_ATTN_EXT gradients through either backward, by finite difference. Measured (M5, 2026-10-14): test-backend-ops FLASH_ATTN_BACK 4/4 on MTL0 against CPU. Finite differences: the native and composed backward pass the direct check and sit at the same level on the SUM-wrapped one (1e-4 to 7e-4 against a 1e-4 bound, varying with the random init, for both): that check does not separate them; the walk test below does. test-walk-exact (native selected) OK on qwen2.5-0.5b (14:04:36-14:05:28Z) and qwen3.5-4b hybrid (14:05:28-14:11:14Z): every exact row 1.0000/1.0000, the cache byte-identical after either walk; the flash path against the F32 explicit gradient went from 0.9999/0.9995 to 1.0000/1.0000 on the 0.5B at an f16 cache. qwen3.5-4b chunk graph, window 8192, chunk 512, recompute (14:11:14-14:12:43Z): 1,623 -> 1,306 MiB on the flash path. SLOWER for now: the 4B walk test took 346 s against 329 s with the composed backward (the kernel is dot products per row, not simdgroup matrices); the matrix version is next. CUDA: no kernel yet (the composed path runs there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

Assisted-by: OpenAI Codex

…owners Assisted-by: OpenAI Codex

Assisted-by: OpenAI Codex

Share row projection and bounded head replay between direct decode and the walk. Keep shadow scatter coordinates explicit and preserve replayed operator precision. Cover sparse outputs and head adapters against the independent optimizer objective. Assisted-by: OpenAI Codex

PR64 (999d682) built the zero tensor of ggml_acc_or_set with ggml_fill over a new tensor, to remove 0 * inf = NaN. A tensor made by ggml_new_tensor is a graph leaf, and leaves stay allocated for the whole graph, so every such zero (cache-window-sized ones included) stayed resident. The zero is now a one-element filled leaf repeated to src's shape: the same NaN-free value, freed after its use. Measured (M5 Metal, qwen3.5-4b hybrid, WALK_MEASURE window 8192, chunk 512, recompute, 2026-10-14): chunk graph 2,118.6 MiB at PR64's head -> 1,286.6 MiB (the base 88c5c1e measures 1,305.6); explicit path 2,324.8 -> 1,551.8. Bisected: 88c5c1e + PR64's ggml.c alone 2,118.6; reverting only this hunk 1,305.6. test-walk-exact OK on qwen2.5-0.5b (22:25:13-22:26:04Z) and the 4B (22:26:04-22:36:01Z): every exact row 1.0000/1.0000, cache byte-identical after the walk, 4B chunk graph 666.8 MiB with recompute (PR64 head: 904.1). test-opt-dynamic-accum OK; test-backend-ops grad SET_ROWS and FLASH_ATTN_EXT on CPU: 0 failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… dO scaled for half tiles FLASH_ATTN_BACK's Metal kernel was one simdgroup per row with no reuse: 196 ms at a training shape (head 256, 16 heads over 4 KV heads, 256 rows, 2048 window, f16 cache) where the forward takes 2.5 ms. Now three passes (FlashAttention-2 style; the per-row kernel stays for K and V heads of different sizes): - stats (query blocks of 8): each row's log-sum-exp and D in one sweep, online; - dkv (KV blocks of 16): dK += dS^T Q and dV += P^T dO over the KV head's whole group; - dq (query blocks of 8): dQ += dS K. S and dP are 8x8 simdgroup tiles of half inputs accumulated in F32; staging steps rows with threads across columns (no division per element); partial blocks write through threadgroup memory. dO is scaled by the power of two that brings its largest |value| to [4, 8) (ggml_flash_attn_back_scale, recorded in the result after D by every backend): training gradients sit below half's normal range, and unscaled the half tiles cost the gradient its precision (4B against the F32 gradient at an f16 cache: 0.9943/0.9946 unscaled, 1.0000/0.9998 scaled). test-backend-ops: a perf case beside the forward at the same shape, with the backward's FLOPs. Measured (M5 Metal, 2026-10-14): FLASH_ATTN_BACK 4/4 against the CPU (f16, f32, q8_0, GQA, head 256, 300 positions x 37 rows). Perf at the shape above: 196 ms (per row) -> 67 ms per call, 321 GFLOPS (the forward: 2.5 ms, 3.45 TFLOPS): faster, not yet near the forward; the next work. 4B training measure at window 8192 (both resident slots busy, confounded): 77.8 s against 78 s for the composed backward, chunk graph 1,286.6 MiB against its 1,623. test-walk-exact OK on qwen2.5-0.5b and the 4B hybrid (23:02:52-23:08:04Z): every exact row 1.0000/1.0000, against the F32 gradient back to the composed path's values, cache byte-identical after the walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… threadgroup memory Cormac's review of 8ec4f9d: the tiled path's threadgroup memory is 26,176 bytes at head 256 but 50,752 at head 512, over Apple's 32 KiB, and the only guard was D == Dv && D % 32 == 0. The route now also requires the computed size to fit the device's maximum; otherwise the per-row path runs. test-backend-ops gains a head-512 case, which takes the per-row path. Measured (M5 Metal, 2026-10-15): FLASH_ATTN_BACK 5/5 against the CPU, the head-512 case included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ng context P2 (continuum ggml-org#5032 section 9.6). The engine half of "the server batches learning with serving": a reply trains on the sequence that served it, in the serving context, with no second context and no second copy of her window. - llama_opt_learn(ctx, seq, n_cached, tokens, labels, n, result): the sequence holds her context exactly as served ([0, n_cached)); attention cells past it are dropped, and a recurrent part must already end there (restore the served-prefix checkpoint first: a recurrent state cannot be cut back), refused by name otherwise, as is a pending rollback. The reply decodes with labels (the shadow on labelled rows only, PR64's physical-row path), then is restored under her stable set from the recurrent snapshot. Her context is never decoded again; other sequences are not touched. One batch is one exact step; a longer reply steps per batch. - The one-ubatch-per-batch rule moves from llama_opt_init into the walk (a named refusal there), so a serving context with -b > -ub can learn. - test-walk-exact case 9: two sequences in one context, her reply learned in place on sequence 0. Measured (M5 Metal, 2026-10-15): test-walk-exact OK. qwen2.5-0.5b: learned in place vs the walk's exact step cosine 1.0000, norm ratio 0.9997; her sequence byte-identical to a stable decode of the reply; the neighbour byte-identical. qwen3.5-4b hybrid (05:35:29-05:44:47Z): cosine 1.0000, norm ratio 1.0000, her sequence and the neighbour byte-identical; every other row unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…re touching the cache Cormac's review of 6fc7bc6: null tokens or labels crashed in the batch fill, and a reply past the sequence's context failed inside the training decode (and then the restore) after her reply cells were already dropped. Both are refused by name before any change. Each call also clears the failure state a previous refusal left, so a refusal never refuses the next call. test-walk-exact case 9 gains the refusals: null labels, past the context, a prefix the sequence does not hold; each refused, her sequence byte-identical after all three. Measured (M5 Metal, qwen2.5-0.5b, 2026-10-15): the three refused with her sequence byte-identical; the following live learn unchanged (cosine 1.0000, norm ratio 0.9997 against the walk's exact step; her cache and the neighbour byte-identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…); older GPUs take the per-row path Raised by Cormac's lane H on IntelMac: the tiled route checked head sizes and threadgroup memory but not has_simdgroup_mm. Measured (M5 Metal, 2026-10-15): FLASH_ATTN_BACK 5/5 against the CPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… serving context P2, server half (continuum ggml-org#5032 section 9.6). A learning turn is a task on the slot loop (SERVER_TASK_TYPE_LEARN), run between that slot's turns: no second context, no copy of her window. - POST /learn {id_slot, n_prefix, reply | reply_tokens, lora_id, lr} (only with --train-dir). The served prompt is what the slot holds; her recurrent state comes back from a checkpoint one token before the prompt's end, which a learning lane keeps (the prompt checkpoint offsets gain 1 when --train-dir is set); the last prompt token is re-decoded labelled with the reply's first token; then the slot returns to her whole served prompt (checkpoint + that token decoded again), so her next turn reuses it. The shadow joins her adapter set for the step only, at scale 1; llama_opt_init runs once (AdamW, the request's learning rate), one shadow per server run for now. - llama_opt_learn: trains to the last labelled row (an unlabelled row after the run is refused in one batch), takes exactly ONE optimizer step per call (manual period end at the ubatch that completes its labelled rows, each ubatch weighted by its share), and reserves no host logits for rows that feed only the loss (opt_outputs_internal; the walk too): a 512-row chunk at a 248k vocabulary was 485 MiB of host RAM, and the serving context's output cap tripped on it. - KV training admission: a context with a stream per slot may train with flash attention when the batch touches one stream (was: refused whenever the cache had several streams, which is how a serving lane runs). - ggml-opt: a caller-driven step's loss is recorded unscaled by opt_period, so the result reports it right (was 4x at -b 2048 -ub 512). - test-walk-exact case 9 runs on a unified F32 cache with explicit attention AND a stream per slot with an f16 cache and flash attention. Measured (M5 Metal, 2026-10-15): test-walk-exact case 9 on qwen2.5-0.5b: unified 1.0000/0.9997, streams + flash + f16 1.0000/1.0000 against the walk's exact step, her sequence and the neighbour byte-identical, the three refusals refused with her sequence unchanged. test-opt-dynamic-accum OK. llama-server, qwen3.5-4b hybrid, 2 slots, -fa on, --train-dir: a served turn, then POST /learn of a 17-token correction four times: loss 2.860 -> 2.525 -> 1.769 -> 1.111, 2.3-2.6 s per learn (slow: GET_ROWS_BACK and OUT_PROD are not on Metal in this branch; the Metal training-op branches merge next). Serving after learning unchanged on her slot (the shadow applies to labelled rows only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ed row backward, GET_ROWS_BACK on Metal) into the P2 line So a learning step stays on the GPU. Conflicts were adjacent additions (FLASH_ATTN_BACK beside GET_ROWS_BACK), both kept. Measured (M5 Metal, 2026-10-15): test-backend-ops MTL0 against the CPU: FLASH_ATTN_BACK 5/5, OUT_PROD 414/414, GET_ROWS_BACK 4/4, PAD 30/30, SET_ROWS 172/172, CROSS_ENTROPY_LOSS 5/5, RMS_NORM_BACK 20/20, SOFT_MAX_BACK 24/24. llama-server qwen3.5-4b, POST /learn of a 17-token correction: 2.3-2.6 s -> 298-410 ms per learn, losses unchanged (2.860, 2.525, 1.769, 1.111). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

Cormac's review of 018ab27 (P1-b): the re-decode of the last prompt token ran with the learn set (shadow at scale 1). The shadow was already off for a batch without labels (llama-graph.cpp lora_on: n_shadow_rows is the ubatch's label count, llama-context.cpp graph_params), so the restored state was hers; her slot's own set is now applied before the restore so this does not depend on that gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

… works Measured on the M5 (qwen3.5-4b hybrid, 2 slots, the 4B test server; her live lane on the same GPU was busy throughout, so compare the pairs, not the runs): learns run back to back took the other slot from p50 218 / p95 450 ms per token to 686 / 2,567. Learning must barely touch the flow. While any other slot is processing, a learn waits out a rest of t*(1/LEARN_DUTY - 1) after a learn of t (LEARN_DUTY = 0.10); with every other slot idle it runs at once. A deferred learn is bound to her slot (task.id_slot), so it comes back when a slot frees: learning lands in the gaps. Measured after (06:13:32-06:13:53Z): the other slot p50 46.8 -> 46.1, p95 73.3 -> 73.8 ms per token with learning on; the two learns waited for the gap (p50 10.4 s wall) instead of the GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…lock, on SIMD dot products FLASH_ATTN_BACK's CPU kernel converted every f16 K and V row again for each query row (twice in pass 1), accumulated its dot products as scalar doubles, and in pass 2 regathered the strided query and output-gradient rows for every KV position. It is the reference other backends are checked against and the backward on machines with no usable GPU. Now pass 1 works blocks of 8 query rows of one head and pass 2 blocks of 16 KV positions of one KV head: each K and V row is converted once per block, each query and dO row gathered once per block, the dot products and accumulations run on ggml_vec_dot_f32 / ggml_vec_mad_f32, and D stays a double. Scratch per thread: max(8*(M + D + Dv), 16*(D + Dv)) + D + Dv floats (ops.h), sized by the work planner (ggml-cpu.c). Measured (IntelMac i7-8850H, 12 threads, test-backend-ops perf, 2026-10-15): at head 256, 16 heads over 4 KV heads, 256 rows, 2048 window, f16 cache: 2,195.5 -> 316.2 ms/call (9.78 -> 67.92 GFLOPS); the forward at the same shape is 51.6 ms. Exactness: the old and new kernels on identical seeded inputs (causal mask, GQA, f16 cache, dO at 1e-3) agree to float rounding on every part of the result: NMSE dq <= 1.97e-13, dk <= 1.49e-13, dv <= 1.00e-13, lse <= 6.6e-16, D <= 4.7e-14, scale identical, at (128,8,2,512,64), (64,4,2,300,37) and (256,16,4,2048,256). test-backend-ops grad: the native backward's MUL case OK; its SUM finite-difference case misses 1e-4 on both the old and new kernels (3/5 and 4/5 runs, random inputs), and goes to inf with dV negated, so it reaches this kernel. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

…/p2-learn-live

…l holds its prompt Learning usually comes after review, when her slot may have served other turns since: n_prefix alone said only that the slot held that many tokens, not that they were this turn's. /learn now also takes `messages` (the conversation she was served, its last message her reply or the correction): rendered through the same chat template serving framed it with, the served prompt is everything before the reply, and her slot must hold exactly those tokens, else the learn is refused with the first differing position. One renderer: /train's render_example becomes train_render, shared by both routes, and server_trainer exposes its serving renderer. Measured (M5 Metal, qwen3.5-4b hybrid test server, 2026-10-15): a turn served through /v1/chat/completions (30 prompt tokens), then /learn with the same messages plus a corrected assistant reply, three times: loss 1.573 -> 1.085 -> 0.474, 632-687 ms each. A conversation her slot never served: refused, "slot 0 no longer holds this turn's prompt (it differs at token 23 of 30): serve the prompt again, then learn". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

joelteply · 2026-10-15T06:59:57Z

Held as a draft (Fable, 2026-10-15). The engine here is kept, but the trigger is wrong: Joel says training is continual from her own engrams and actions, not called by an agent after a review. The redesign has learning ride her next turn's prefill, with labels chosen by her own signals. The design is out for Astra's check in fable-astra and cambriantech. Do not merge until that lands.

ggml_cross_entropy_loss_weighted(logits, targets, weights): row i contributes w_i * -log softmax(logits_i)[t_i] / rows and its gradient scales by w_i, so how much a row counts is the caller's to say (her relevance on what she perceived), not only whether. A zero weight skips the row's softmax entirely. The weights are F32, one per row: 4 bytes a row beside a vocabulary-wide logits row. CPU and Metal implement it; CUDA refuses the weighted form in supports_op (the scheduler runs it on the CPU) until its kernel takes the weights. Measured (M5, 2026-10-15): test-backend-ops CROSS_ENTROPY_LOSS on MTL0 7/7, gradients (finite difference, weighted rows including zeros) OK on MTL0 and CPU at [10,5,4,3] and [30000,2,1,1]; full grad mode 19815/19815 on both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ws only The walk reads the K/V gradient of the run's positions only ([g0, n_win)); everything before g0 is her served prefix, a constant. The attention backward computed dK/dV for the whole window anyway and the graph discarded all but the run's rows. - ggml_flash_attn_back(..., kv0): dK/dV for the key rows [kv0, M) only, row j at j - kv0 (op_params[1]); pass 1 (dQ, lse, D) still over every key. CPU and Metal (tiled and per-row). - ggml_flash_attn_ext_set_grad_sinks(a, k_tap, k_cur, v_tap, v_cur): an attention whose K/V gradient goes to the run's rows (the tap of the cached rows before the chunk, then the chunk's own K/V) instead of the window it reads; its backward takes kv0 = M - (tap + chunk rows) and adds each part to its sink. The composed backward (devices without the one op) skips the tiles before kv0 for dK/dV the same way. - The in-place training graph uses the sinks, so no window-sized K/V gradient exists: the tap is now the run's cached rows as F32 (as on the explicit path), and ggml_opt_accumulate_grad_rows, which tapped rows of a window-sized gradient, is deleted with its only caller. Measured (M5 Metal, 2026-10-15): - test-walk-exact, qwen2.5-0.5b and qwen3.5-4b hybrid: every exact row cosine 1.0000 (f16 and q8_0 caches, context + reply, recompute, rollback slots, learning on a live sequence); caches after the walk byte-identical to her stable decode. - test-backend-ops MTL0 vs CPU: FLASH_ATTN_BACK 8/8 (kv0 = M-37 unaligned, q8_0 at 203, per-row head 512 at 100); grad mode 19818/19818. - FLASH_ATTN_BACK at head 256, 16/4 heads, 256 rows, window 2048: 66.0 ms per call at kv0 = 0, 45.7 ms at kv0 = 1792 (the reply's keys). Pass 1 over every key is what remains. - Graph peak, 4B, window 8192, chunk 512, recompute: 1,084.8 MiB, unchanged: the per-layer backward results were never alive together, so at this window the head slices set the peak. The long-window peak is measured next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…nce per block of dst rows, on simd_gemm The LoRA backward's dx = out_prod(W, dy^T) through a frozen quantized base weight is the CPU training step's largest term. The kernel dequantized every src0 row again for EVERY dst row (n times over) and accumulated with one scalar-weighted mad per row. Now each task owns a block of 32 dst rows of one (i2,i3) plane, dequantizes src0 in tiles of 32 rows once per block, gathers the matching src1 values, and accumulates dst[BN x ne0] += S1[BN x KT] * W[KT x ne0] on simd_gemm (the forward attention's GEMM). Non-contiguous dst rows keep the mad loop. Scratch per thread: KT*ne0 + BN*KT floats (ops.h), sized by the work planner. Measured (IntelMac i7-8850H, 12 threads, test-backend-ops perf, q4_K, 2026-10-15): the 1.5B LoRA backward shapes at 1024 tokens, trans_b: m=1536 k=1536 410.9 -> 34.7 ms/call (11.8x); m=1536 k=8960 2,463.2 -> 194.3 ms (12.7x); m=8960 k=1536 494.4 ms (its baseline run was cut). Exactness: old and new kernels on identical seeded q4_K inputs give BIT-IDENTICAL results (NMSE 0, max abs 0) at 1536x1024x1536, 1536x1024x8960, 8960x1024x1536 (transposed b) and 1536x300x1536 (contiguous b): the same FMAs in the same k order, from a dequantized tile instead of a fresh row. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

…e per-row f16 copy is deleted ggml_compute_forward_out_prod_f16_f32 was the quantized kernel's old per-row loop with ggml_fp16_to_fp32_row in place of the dequantizer. F16's to_float trait is that function, so f16 now routes through the blocked kernel (renamed out_prod_converted_f32: it converts any src0 row type through its to_float trait), and the duplicate is gone. Measured (IntelMac i7-8850H, 12 threads, same seeded inputs, best of 2-3, 2026-10-15): f16 1536x1024x1536 transposed b 208.9 -> 23.3 ms (9.0x); 1536x1024x8960 1,276.6 -> 145.5 ms (8.8x); 1536x300x1536 contiguous b 64.8 -> 9.8 ms (6.6x). BIT-IDENTICAL to the deleted kernel on all three (cmp), and the q4_K path's output is unchanged by the rename. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

Three window-sized holds in a training graph, found by measuring the 4B at 8k and 32k: - Metal reserved a window-sized F16 copy of K/V inside every FLASH_ATTN_EXT's output, at any K type, "to avoid graph reallocations". Only a quantized K is ever dequantized (and only at 32+ rows), so a K the kernel never dequantizes now reserves nothing; a quantized K keeps the reservation, so decode and prefill graphs still match. - A training graph keeps each attention's output for the backward, so a quantized cache's F16 copy lived per layer: training now dequantizes through a cast node of its own, and the native backward reads the cast's quantized source, so the copy is not kept for the backward. - Recompute never rebuilt a node that read a buffer the forward writes in place (the cache), so every layer's cast stayed alive from the forward to that layer's backward. A clone runs after the forward, so it reads such a buffer as the forward left it: the same value unless a write comes AFTER the original read. Only reads followed by a write are kept now. GGML_OPT_TOP also lists the graph's input leaves, each tensor's sources, and the graph's total. Measured (M5 Metal, qwen3.5-4b hybrid, chunk 512 at the window's end, recompute; graph peak MiB; loss identical before and after, 2.904847 / 2.904831 / 2.905061 at 32k): window explicit F32 flash f16 flash q8_0 (her lane's cache) 8192 1,546 -> 1,220 1,085 -> 557 1,096 -> 550 32768 4,608 -> 3,540 2,020 -> 581 1,888 -> 615 test-walk-exact on qwen2.5-0.5b and qwen3.5-4b: every exact row cosine 1.0000, recompute vs without 1.0000, caches byte-identical; test-backend-ops MTL0 FLASH_ATTN_EXT 4809/4809, grad 19818/19818. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…p2-learn-live

…forward did not compute The recompute rule now rebuilds a read whose buffer is written only before it. The LAST in-place write (a view-result op, not a view) into a persistent buffer (a cache, a state) was therefore no longer kept, and its clone, whose view source resolves to the live buffer, would apply the write a second time in the backward. Such a write is a side effect, like SET_ROWS and a non-cast CPY: returned unchanged (Cormac's review of continuum#5057). Measured (M5 Metal, 2026-10-15): test-walk-exact on qwen2.5-0.5b (8 exact rows) and qwen3.5-4b (11) all cosine 1.0000, recompute vs without 1.0000; the 4B graph at window 32768 unchanged at 3,540.3 / 581.4 / 615.0 MiB (explicit / f16 / q8_0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

joelteply · 2026-10-15T10:02:28Z

Receipt correction (Fable, 2026-10-15). The e77b823 commit message says CROSS_ENTROPY_LOSS gradients are 'full grad mode 19815/19815 on both' (MTL0 and CPU). On the CPU that was a run filtered with -o CROSS_ENTROPY_LOSS: the weighted and unweighted cross-entropy cases pass, but it is NOT a full CPU grad suite. A full CPU grad run aborts at REGLU, which has no backward (Cormac, card 397cefc6), so no full CPU grad receipt exists today. The MTL0 counts in this PR came from the same filtered invocations, so read every 'N/N grad' in these commits as 'the named op's cases pass', not as a full-suite result.

…ble range, no params on ops with no backward (partial, card 397cefc6) `test-backend-ops grad -b CPU` (full suite) never finished: it aborted at the first case that marks params on an op ggml_compute_backward has no case for, so no summary was ever printed and every later grad case went unchecked on the CPU. - test_unary's EXPM1 grad cases use [-5, 5]: over [-150, 150] exp overflows f32 (both sides inf, the "SUM nonfinite" failures), and over [-10, 10] the summed loss is too coarse in f32 for the finite difference to see the small gradients (MAA ~1). The EXPM1 backward itself is correct (direct check: grad = exp(x) exactly at -2, 0, 0.5, 3). - test_glu_split marks params only for SWIGLU (the only glu with a backward); test_swiglu_oai and test_pool1d mark none (no backward). Still aborts later, at IM2COL_3D: the remaining cases want one structural fix (ggml says which ops have a backward and the grad harness skips the rest), not one guard per test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

…growing with the window (#5057) * engine: learn in place on the serving lane; the training graph stops growing with the window Bumps core/vendor/llama.cpp 178224903 -> cde8ffea3 (CambrianTech/llama.cpp#65, fable/p2-learn-live). What her lane gains: - llama_opt_learn / POST /learn: a reply learns on the slot that served it, exact against the walk, her cache byte-identical after; one optimizer step per call; a 10% duty cycle while another slot works. The core does not call it (the trigger is being redesigned per Joel, 2026-10-15: learning from her own perception, weighted by her relevance); /train is unchanged. - The training graph no longer grows with the window (three window-sized holds removed: a Metal F16 K/V copy reserved in every attention's output, a quantized cache's dequant kept per layer, recompute refusing to rebuild cache reads); K/V gradients only for the run's rows; a per-row weighted cross-entropy; Cormac's CPU attention backward (6.9x) and CPU OUT_PROD (9-13x). ## Measured (M5 Metal, 2026-10-15, fork cde8ffea3) - qwen3.5-4b hybrid training graph peak, chunk 512, recompute, q8_0 cache: 8k 1,096 -> 550 MiB, 32k 1,888 -> 615 MiB; f16 cache 32k 2,020 -> 581 MiB; loss identical. - test-walk-exact (qwen2.5-0.5b, qwen3.5-4b): every exact row cosine 1.0000, recompute vs not 1.0000, caches byte-identical; test-backend-ops MTL0 grad 19818/19818. - 4B test server, 2 slots: the other slot's per-token p50/p95 46.8/73.3 -> 46.1/73.8 ms with learning on; a served conversation + correction learned, loss 1.573 -> 0.474. ## Measured after deploy (owner Fable, before anything calls /learn on her lane) On her 27B lane at her served window: serving per-token p50/p95 vs the previous engine; a /train-or-/learn step's graph peak against the serving reservation; the other slot's p50/p95 with learning on vs off. Posted to this PR and continuum-dev. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo * engine: pin cde8ffea3 + Cormac's recompute guard (5b8500ee2) in both the gitlink and LLAMA_CPP_PIN The source-hygiene vendor_pin check reads core/vendor/LLAMA_CPP_PIN beside the submodule (Cormac's review: it still named 178224903). The fork commit adds the guard his review asked for: recompute never clones an in-place write into a buffer the forward did not compute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

… through this node"; the grad suite reports instead of aborting (card 397cefc6) `test-backend-ops grad -b CPU` never finished: the first case that marked params on an op ggml_compute_backward has no case for (REGLU, then SWIGLU_OAI, POOL_1D, IM2COL_3D, an in-place SET, ...) aborted the whole run, so no summary ever printed and every later case went unchecked. Upstream (ggml-org master) lacks those backwards too. - ggml_node_has_backward(tensor): the op (and unary or glu op) has a backward case and the node is not an in-place write other than the re-shaping views. ggml_build_backward_expand and ggml_compute_backward both assert it, so the list and the switch cannot disagree silently: a missing case or a missing list entry fails the grad suite loudly. - test_case::eval_grad walks the forward graph and reports a param-reachable node without one as "not supported (no backward for OP)", the way supports_op already reports forward ops. The per-test param guards of c010269 are reverted: this owner replaces them. - test_unary's EXPM1 grad follows the smooth ops' convention (test_sin): eps 0.2, grad_precise, inputs in [-2, 2], tolerance 1e-3. Its grad_eps was 15 (meant for ABS/SGN/RELU over +-150), which no central difference of exp survives, and inputs to 150 overflowed f32 (the "SUM nonfinite" failures). The EXPM1 backward is correct: a direct check gives grad = exp(x) exactly, 1024 elements on 12 threads. Measured (IntelMac CPU, 2026-10-15): the full suite now COMPLETES with a summary (exit 1 instead of 134). 19673/19818 pass; 234 cases report "no backward for ..." instead of aborting. The 145 that fail are tolerance misses (MAA 1.2e-4 to 7e-4): 74 reported as SUM, 68 MUL_MAT and 3 MUL_MAT_HADAMARD. 66 of the MUL_MAT are f16 src0, and they fail the same way at 0135932 (67 there), before the CPU OUT_PROD change, so they are not that kernel's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

…e/p2-learn-live

…hat/completions uses Measured on her lane (the M5, Qwen3.8-27B, 2026-10-15): every /learn with messages was refused, "slot 1 holds 27439 prompt tokens; the served prefix is 27475 and the reply must not be empty": /learn rendered the conversation through /train's renderer, which ignores chat_template_kwargs, and Qwen3.8's thinking template renders a different prompt without enable_thinking=false (and drops the reply span). It passed on the 4B only because that template renders the same either way. The served prompt is now every message but the last through oaicompat_chat_params_parse and tokenize_input_prompts, the request's own template arguments included: the exact tokens /v1/chat/completions served. The reply is what the model would generate after it: the last message's text and the end of the turn. /learn no longer uses /train's renderer, so its accessor and the exported train_render are removed (train_render is static again). Measured (M5, qwen2.5-0.5b served with Qwen3.8-27B's own chat template from her lane's /props, enable_thinking false): a 1,708-token served conversation + correction, three learns: loss 2.092 -> 1.093 -> 0.316, 221-242 ms each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…lies it would mislearn Cormac's review of continuum#5061: - P3: serving_prompt_tokens(vocab, mctx, prompt, files, chat) is now the one place a parsed request becomes the tokens it is served as (mtmd for a chat request with a projector, else tokenize_input_prompts). handle_completions_impl and /learn both call it, so "the slot holds what serving put there" is true by construction. - P2: a reply is learned as its text then the end of the turn. A reply whose substance is elsewhere is refused by name until it renders through the template: tool_calls (a coding citizen's commonest turn; it would have taught ending the turn instead of calling the tool), reasoning_content, or no text. A conversation with media is refused too (text prompts only). Measured (M5, qwen2.5-0.5b with Qwen3.8-27B's chat template): learning unchanged, loss 2.092 -> 1.093 -> 0.316 (226-255 ms); a tool-call reply, a reasoning reply and an empty reply each refused with 400 and the reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

joelteply · 2026-10-15T18:46:18Z

Upstream candidate 1: metal : reserve the FA F16 K/V scratch only for a K that can be dequantized

Branch: CambrianTech/llama.cpp fable/metal-fa-kv-f16-reserve @ 3b433d9, one commit on ggml-org/llama.cpp master 620d222.
File: ggml/src/ggml-metal/ggml-metal-ops.cpp (+20 / -14).

What and why

FLASH_ATTN_EXT on Metal reserves an F16 copy of K and V (ggml_metal_op_flash_attn_ext_extra_kv_f16) at every batch size,
"to avoid graph reallocations", because whether the kernel dequantizes K depends on the batch. But it only ever dequantizes a
quantized K (Q4_0/Q4_1/Q5_0/Q5_1/Q8_0; use_kv_f16). For an F16/BF16/F32 K the reservation is never written or read at any
batch size. The change reserves it only for a dequantizable K (one helper, kv_dequantizable, now shared by use_kv_f16 and the
reservation), so the decode/prefill no-reallocation property is unchanged.

Safety: the scratch is written only under use_kv_f16 (implies dequantizable). The index buffer after it is offset by the same
extra_kv_f16 in both the allocation size and the encoder, so it stays consistent.

Measured (Apple M5 Pro, Metal, 2026-10-15; qwen3.5-4b Q4_K_M; upstream master vs the change)

test-backend-ops -o FLASH_ATTN_EXT -b MTL0 (vs CPU): 4984/4984 pass with the change.
llama-bench -fa 1 -p 512 -n 32, MTL0 compute buffer (max), from -v:
- f16 cache, depth 32768: 537.5 -> 537.5 MiB (no change: other buffers dominate)
- f16 cache, depth 131072: 711.8 -> 633.5 MiB (-78.3 MiB)
- q8_0 cache, depth 32768: 537.8 -> 537.8 MiB (unchanged, as intended)
Speed, f16 cache, depth 131072, three alternations (base first): pp512 349.1 vs 342.4 t/s mean, tg32 17.9 vs 18.9 t/s mean:
no difference beyond noise. CAVEAT: measured on a GPU shared with a live 27B serving lane (two earlier fa-first pairs read
much slower and did not reproduce); a quiet-machine rerun is requested.
Larger effect in training graphs (llama.cpp's opt path keeps attention outputs for the backward): measured in our fork.

How to reproduce

cmake -B build -DGGML_METAL=ON -DLLAMA_BUILD_TESTS=ON && cmake --build build --target test-backend-ops llama-bench
./build/bin/test-backend-ops test -o FLASH_ATTN_EXT -b MTL0
./build/bin/llama-bench -m <model> -fa 1 -ctk f16 -ctv f16 -p 512 -n 32 -d 131072 -r 3 -v | grep "MTL0 compute buffer"

…label_weight) How much a token counts, not only whether: a mind's relevance on what it perceived scales its surprise. llama_batch gains a trailing `float * label_weight` (NULL: every labelled position counts 1; >= 0 and finite; read for labelled positions only). llama_batch_init allocates it at 1; the ubatch split carries it; every batch view that slices `label` slices it too (parallel, batched-bench, perplexity, mtmd's media batch), so a view never drops a weight. ggml-opt: ggml_opt_set_next_label_weights(opt_ctx, weighted) asks the next graph for a weighted cross-entropy (ggml_cross_entropy_loss_weighted per head slice, over ggml_opt_label_weights); an unweighted batch keeps the unweighted loss, so CUDA, which has no weighted kernel yet, trains an unweighted batch on the GPU as before. llama sets it per ubatch from label_weight. Measured (M5, CPU quick check, qwen2.5-0.5b, test-walk-exact WALK_MIXED_ONLY, F32 and flash q8_0): weights 1,1 vs an unweighted batch 1.0000/1.0000; weights 2,2 vs twice the step 1.0000/1.0000; weights 1,0 vs half of row 0's own step 1.0000/1.0000; the mixed-decode checks still OK. Build has zero warnings. Metal run owed (held while the M5's deploy can relaunch its lane). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…e slot is released On the M5's 27B (one slot, busy almost always) a separate POST /learn lost the race for the slot 12 times in 12, 2026-10-15: another request took it between the turn and the learn. A completion request may now carry "learn": {"reply": text, "lora_id", "lr"}: when the turn finishes, its reply is learned on that same slot before release (no request can take it in between), and the outcome is in the final response's "learn" field. One learner: learn_on_slot(slot, request, t0) holds everything from the adapter checks to the slot's restore to its served prompt; the SERVER_TASK_TYPE_LEARN case keeps only what is a task's (finding the slot, deferring while it serves, the duty cycle) and calls it, and both finish paths (plain and speculative) call learn_if_asked before send_final_response and release. The learn types are named (learn_request, learn_outcome) and shared by the task, the result and the completion params. The server's batch view slices label_weight with label. Measured (M5, Metal, qwen2.5-0.5b with Qwen3.8-27B's chat template, ONE slot, a second client taking it continuously): 4 learn-carrying turns, 4 succeeded, loss 2.178 -> 1.368 -> 0.282 -> 0.092, 889-962 ms each, 5 competing requests served on the same slot meanwhile; a learn with an empty reply refused (400). Full build: zero warnings. Found, NOT caused by this (card owed): training on a CPU-only server (-ngl 0) crashes in ggml_backend_tensor_copy (a null copy function) inside llama_opt_learn's decode, through POST /learn as well; every earlier learn receipt ran on Metal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ed in the final response, not omitted Cormac's review question on 9a10993: every way a turn ends either learns or says why. Both finish paths (plain, speculative) learn before release; a cancelled turn has no reader; an errored turn returns its error; the empty-prompt final response now carries learn.success=false with the reason instead of omitting the field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…thropic conversion keeps "learn" Cormac's review of continuum#5082: the learn outcome was serialized only by /completion, /v1/completions and /v1/chat/completions without streaming, while continuum's chat requests stream. It is now on the final chunk or event of every format: chat stream (the last chunk, beside timings), responses and responses stream (response.completed), Anthropic and Anthropic stream (message_delta, beside usage). Testing every format also found the Anthropic request conversion copied a fixed list of fields without "learn", so a learn there never ran; it now carries it. tools/server/tests/learn-every-format.py asserts exactly one successful learn outcome for each of the 10 endpoint x stream combinations. Measured (M5, Metal, qwen2.5-0.5b with Qwen3.8's template, one slot): all 10 OK (before the conversion fix: 8 OK, /v1/messages plain and stream carried none). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…ed copy dropped its import line) Measured: run against the server as documented in its header: ALL FORMATS CARRY THE LEARN OUTCOME, exit 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

…nuum card 5b497852) /train refused a server that may hold base weights in a repacked CPU buffer; learn_on_slot (POST /learn and a learn on a completion) did not. Measured 2026-10-15 at cc2e05f: on x86 at -ngl 0 the learn failed in the backward ("no backend can run" OUT_PROD 'grad for ffn_swiglu-23' on blk.23.ffn_down.weight); on an M5 at -ngl 0 the scheduler copied the repacked weight to Metal and crashed in ggml_backend_tensor_copy (CPU_REPACK get_tensor/cpy_tensor are NULL). With --no-repack the learn succeeds. The guard is lifted to one function, train_repack_refusal, which /train and learn_on_slot both call; the learn returns learn.success=false with the reason and the turn is still served. Its comment no longer claims "no crash". Test (tools/server/tests/learn-refuses-repacked.py), IntelMac i7-8850H, CPU build, Qwen2.5-0.5B Q4_K_M, 23:40:58Z: repacked -> turn answered, learn refused with the --no-repack remedy, server alive; --no-repack -> learn success, loss 1.6083. The pre-change binary fails the test (its failure text names no remedy, 23:35:32Z). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

joelteply and others added 29 commits October 13, 2026 15:38

llama: connect labelled decode to physical cache row gradients

999d682

Assisted-by: OpenAI Codex

llama: keep serving indices and compose training admission in memory …

87a0f17

…owners Assisted-by: OpenAI Codex

test: compare multi-sequence label steps and isolate recompute

44abe56

Assisted-by: OpenAI Codex

ggml: recount graph uses after checkpoint rewriting

07af30f

Assisted-by: OpenAI Codex

Merge remote-tracking branch 'origin/cormac/cpu-attn-back' into fable…

264f54b

…/p2-learn-live

github-actions Bot added the server label Oct 15, 2026

github-actions Bot added SYCL CUDA testing Apple Metal model mtmd labels Oct 15, 2026

joelteply marked this pull request as draft October 15, 2026 06:59

joelteply and others added 6 commits October 15, 2026 02:04

Merge remote-tracking branch 'origin/cormac/cpu-out-prod' into fable/…

cde8ffe

…p2-learn-live

joelteply mentioned this pull request Oct 15, 2026

engine: learn in place on the serving lane; the training graph stops growing with the window CambrianTech/continuum#5057

Merged

joelteply and others added 4 commits October 15, 2026 06:33

Merge remote-tracking branch 'origin/cormac/grad-suite-fix' into fabl…

0fbbae2

…e/p2-learn-live

joelteply and others added 6 commits October 15, 2026 15:47

Navigation Menu

Repository navigation

train in place: labels ride the batch, the reply learns on the slot that served it (POST /learn) - #65

train in place: labels ride the batch, the reply learns on the slot that served it (POST /learn)#65
joelteply wants to merge 47 commits into
masterfrom
fable/p2-learn-live

joelteply commented Oct 15, 2026

Uh oh!

joelteply commented Oct 15, 2026

Uh oh!

joelteply commented Oct 15, 2026

Uh oh!

joelteply commented Oct 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

joelteply commented Oct 15, 2026

What changes

Measured (M5, Metal, 2026-10-14/15)

Measured after deploy (owner Fable)

Uh oh!

joelteply commented Oct 15, 2026

Uh oh!

joelteply commented Oct 15, 2026

Uh oh!

joelteply commented Oct 15, 2026

Upstream candidate 1: metal : reserve the FA F16 K/V scratch only for a K that can be dequantized

What and why

Measured (Apple M5 Pro, Metal, 2026-10-15; qwen3.5-4b Q4_K_M; upstream master vs the change)

How to reproduce

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant