train: walk the window — decode context into the cache, train each reply chunk against it

[joelteply]

What it is: the training walk (Fable's design). A window's context is decoded into the KV cache, and each reply chunk is trained against the cached prefix, which it reads as constants. The chunks are then re-decoded, so the next chunk sees them. A training graph therefore holds one chunk, not the whole window. That's what lets Kimi's 63k-token windows train without truncating anything, which Joel's rule forbids.

Contents:

• The walk itself, in opt_epoch_iter.
• A snapshot of the recurrent state around each chunk (Fable), so a reply can span chunks on a hybrid.
• Cache constants at F16 or q8_0. Training graphs build explicit attention, and the serving context keeps flash attention.
• A yield before every decode piece and every chunk, so serving waits at most one chunk.
• /train takes a chunk parameter (multiple of 256; interim default 512).

Measured on the 5090, rebased onto dc1a379:

Run	Losses (train / eval per epoch)	Windows	Longest window	Graph
1.5B, chunk 256	0.680/0.621, 0.422/0.455, 0.325/0.483 (eval identical before and after the rebase)	504	124 ms	692 MiB
1.5B, long examples	0.442	98	182 ms	1,249 MiB
0.8B hybrid, recompute off	0.702/0.496, 0.341/0.290	338	429 ms	718 MiB
0.8B hybrid, recompute on	0.701/0.496, 0.342/0.293	338	1,011 ms	408 MiB

Why it's a draft: the gradient is truncated. Cached prefixes are stop-gradient constants, so a chunk's loss doesn't reach the earlier context's activations. Before a gene ships from this, Fable wants an exact reverse pass with horizon H. The test plan: H = 0, 1k, 4k and all, on the 1.5B parity set, accepted when H = all matches the full-window loss (0.222). Also open: the qwen35 recompute gap (card 9355b90c). The two questions are separate.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc

[joelteply]

Rebased onto 2d4d63b (#41 and #42). #42's backend-failure check now also covers the walk's chunk step. Git had merged it in with a bare return; inside a lambda that returns bool, so I fixed it to return false, the same exit a refused graph takes. Rerun on the 5090, 1.5B, chunk 256: eval 0.6211 / 0.4545 / 0.4830, identical to before.

[joelteply]

Approve at 539c75d (Fable). This is the walk as designed: unlabelled runs decoded into the cache; her reply trained in chunks against the cached prefix as constants; each chunk then decoded into the cache under the adapter as it now is; nothing past last_label computed; the yield point is the chunk, not the window. The recurrent snapshot via the scratch sequence (n_seq_max = 2, unified) is right, as are flash attention for the decodes and explicit attention for the training graphs (!cparams.training in build_attn_mha), so V is cached untransposed and can be quantized. The early sched_reserve return under opt_ctx closes the freed-scheduler trap.

Two notes, both about Joel's 'align bit depth to inference' (tonight). Fine to fix on top in #47 or a follow-up:

1. server-train.cpp hardcodes type_k = type_v = GGML_TYPE_Q8_0, overriding whatever common_context_params_to_llama(params_base) took from the serving context. Training should read the cache type serving runs at, so drop the override and inherit it. train: the exact walk, one optimizer step per window on the gradient of its whole loss (opt-in) #47's measurement says both served types are safe (f16 0.9986, q8_0 0.9978).
2. The comment above that override still says 'F16: the cache holds only CONSTANTS…' while the code sets Q8_0. With (1) it becomes 'the serving context's cache type'.