training: fix no KV cache

This commit is contained in:
Xuan Son Nguyen
2026-08-16 21:06:08 +02:00
parent 4df29be4f4
commit d2fd589f51
5 changed files with 122 additions and 15 deletions
+2
View File
@@ -6,6 +6,8 @@ Finetuning of Stories 260K and LLaMA 3.2 1b seems to work with 24 GB of memory.
**For CPU training, compile llama.cpp without any additional backends such as CUDA.**
**For CUDA training, use the maximum number of GPU layers.**
Gradients cannot flow through the KV cache, so training attends to the tokens of the current ubatch instead. This requires `-c`, `-b` and `-ub` to be equal. Flash attention is disabled during training because `FLASH_ATTN_EXT` has no backward pass.
Proof of concept:
``` sh