* Add benchmark-only synthetic speculative acceptance to llama-server and llama-cli
* Address review comments
* Address review comments
* Add some comments in the code
* hexagon: use non-host bufs by default and make the backend fully async
* hex-hb: remove optional hostbuf support and fix async copy
* hex-unary: relax supported unary check
* hex-bufs: use same get_alignment for host bufs
* snapdragon: bump android_platform to 34
* hex-rows: super hacky get/set rows for q8_0
* hex-get-rows: fix q8_0
* hex-get-rows: supprot for f16 and cleanup for q8_0
* hex-get-rows: generic macros and specialized thread funcs
* hex-get-rows: add DMA pipeline, vtcm_layout and kernel params
* hex-set-rows: fix q8_0 support, add dma and tracing
* hex-tests: override nmse threshold for HTP of Q8_0 quants
* hex-fa: add support for Q8_0 with inplace dequantizers
* hex-get-rows: simplify type dispatch
* hex-rows: simplify GET/SET_ROWS DMA pipeline
* hex-async: add events, set/get-tensor-async and rest of the async api support
* hex-repack: use slice instead of expert in repack functions
* hex-cpy: update event/async-cpy logging
* hex-set-rows: optimize smaller tensors
* hex-geglu: fix perf regression with larger tensors
* hex-get-rows: add missing header
* hex-set-rows: add missing header
* hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default
* hexagon: do not reject ops with non-heaxon buffers
* hex-get-rows: apply >=32 restriction only for q8_0
* hex-res: bump vtcm acquire timeout to 10 seconds
* hex-bufs: add support for cloning buffers between sessions to speed up tensor copies
* hex-async: rework event recording and batch flushing and integrate with meta backend
* hex-bufs: improved handling of repacked tensors
* hex-repack: handle get_tensor_2d offsets
* hex-dev: add support for devices with multiple NPUs
* hex-sync: add support for sync tokens to synchronize npu devices for async splits
* hex-mmap: cleanup mmap calls and add a retry for robustness
* hex-sync: add failsafe if sync wait gets stuck
* hex-sync: use sync_seq to check for completed events
* hex-sync: rotate tokens for extra robustness
* hex-devs: add supprot for legacy device names for now
* hex-bufs: add support for auto-cloning buffers from diff sessions
* hex-fusion: simplify and optimize htp-opnode fusion handling
* hex-sync: override opnode name so that it shows up in the profiles
* hex-trace: update scripts to handle multiple devices
* hex-sync: bump the size of the opbatch queue and number of sync tokens
* hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma
* hex-sync: add graph-flush threshold to avoid single op batches
* hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops
* hex-bufs: introduce tensor->extra and shadow_bufs for repacking
* hex-l2: flush tiny tensors inline
* hex-sync: use explicit l2flush for sync tokens
* hex-extra: track weight flags via tensor extra
* hex-fence: rename sync to fence
* hex-repack: proper handling of set-tensor-2d in the shadow_buf
* hex-trace: remove obsolete opstage mask that we used for profiling
* hex-env: remove obsolete use_hmx variable
* hexagon: new unified run.py and build.py and updated docs
* snapdragon: update run script to auto-escapt test-backend-op -p argument
* hex-scripts: fix trailing spaces
* hex-scripts: fix flake8 warnings
* snapdragon: cleanup dst lib/bin dirs before copying new build
* hex-ops: add support for allreduce
* hex-ar: improved allreduce with dma pipeline
* hex-ar: align macros
* hex-ar: consistent use of fence_seq
* hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback
* hex-ar: add proper synchronize handling for ALLREDUCE
* hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold
* hex-ar: bump block size to improve dma efficiency
* hex-ar: fused ALLREDUCE+ADD
* hex-ar: cleaner fence buffer management
* hex-ar: futher allreduce tweaking to remove race conditions
* hex-ar: add simple solver and remove non-dma kernels
* hex-ar: add row-broadcast to fuse with bias ADD
* hex-fence: pass seq numbers via op_params
* hex-ar: allow for both entry/exit seq for completing entry wait
* hex-ar: align macros
* hex-ar: do not refetch broadcast row
* hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things
* hex-fusion: fix incorrect MUL_MAT reordering
* hex-mm: make fused 2x and 3x matmuls more generic
* hex-fusion: move tensor fusion tagging to graph_compute
* hexagon: make sure to copy tensor->extra by value
* hex-get-rows: fix offset calc with row-chunking
* hex-repack: get_tensor_2d fixes for non-zero offsets
* snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default
* hex-devices: use legacy device nameing by default to ease the transition
* hex-devices: hardcode CDSP domain IDs for current devices for now
* hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values
* hex-optrace: more robust handling of the fence events
The arch was skipped with a note guessing that the hyper-connection keys
never reached the synthesised file. They did. The suite builds a model, then
saves and reloads it, and llama_model_saver did not re-emit those keys, so
the failure was in the roundtrip leg rather than the first load. Three gaps,
all in shared code and all additive:
- add_kv_from_model wrote no hyper-connection, compress-ratio or PLE keys.
The PLE group only means anything whole, so it is written or omitted
together; the rest follow the file's existing style of writing every key
unconditionally, since an architecture that does not read one is
unaffected by a zero.
- the saver had no uint64 path at all, which the PLE hash constants need.
- add_tensors_from_model enumerates model-level tensors by hand and was
missing per_layer_tok_embd and the three final-mixer tensors.
Two smaller fixes on the qwen4exp side, both found by running the test:
- build_qsa_top_k divided by the compression ratio before asserting it was
non-zero, so a file without the key crashed instead of reporting.
- a layer with no compression ratio now falls back to dense attention,
which is what the model computes below the budget anyway. The test then
has to write a ratio to reach QSA at all, and an indexer key length no
narrower than n_rot, since the indexer ropes with the main attention's
rotary width.
Full suite: 126 archs, qwen4exp at 0.00e+00 with roundtrip OK. The tiny
fixture is unchanged, max logit delta 0.0 against the pre-QSA dense run.
The old note guessed that the hyper-connection keys never reach the file.
They do: dumping the gguf_context handed to llama_model_init_from_user
shows both among its 67 KVs, and the loader still reports one missing.
build_rs writes into the state tensor in place, zeroing one row and copying the
carried-over states, so calling it twice for the same layer let the second call
clobber the first write-back. The PLE layer is also a delta-net layer, so that
is exactly what happened: both convolutions gathered the same row. They now
share a single gather per layer.
The earlier claim that the conv state was carried correctly was tested on a
fixture whose conv weights are zero, where the branch contributes nothing and
chunking matches trivially. Re-running with non-zero conv weights showed the
divergence, growing with the number of ubatch boundaries: 97.1% top-1 at one
boundary down to 90.2% at seven. With the shared gather it is bit-identical to
the single-shot run at every chunk size tried, 512, 128 and 64, with a maximum
logprob deviation of exactly zero over 1023 positions. The delta-net-only model
stays bit-identical too, so nothing regressed there.
Also derive the delta-net conv channel count the way load_arch_tensors sizes
wqkv instead of from ssm_d_inner. The two agree for this model, but n_embd_r()
only bounds the row and the convolution has to match the tensor feeding it.
test-llama-archs previously aborted on this architecture and took every later
architecture with it. qwen4exp is marked MoE-only, given the hyper-connection
keys and an ssm_d_inner consistent with its tensor derivation, and skipped for
now: the hyper-connection keys written by get_gguf_ctx are not reaching the
synthesised file, which needs a separate look. The suite completes again, 124
architectures at 0.00e+00.
* metal: WIP chunked SSD SSM_SCAN kernels for multi-token prefill
* metal: drop scalar SSD path; MMA + sequential tail
* drop WIP ssm scan test noise
* remove state_from_dst and rename CS and NSG constants
* remove unrelated added whitespace padding
* added clarity to mma_tokens calculation
* added clarity to use_mma bool checks
* added comments to metal ssd op constants for clarity
* reserve K tokens for sequential kernel rollback snapshots
* reset concurrency between mma and seq tail
* remove print args no longer used
* fixed comment to no longer point to specific line
* add FC_SSM_SCAN so seq path skips token offlset unless it's mma tail
* added changes to new ssm.metal for rebase after ggml-metal.metal refactor
* specialize ssm_scan tail with a template instead of a function constant
---------
Co-authored-by: dpantaleoni <dominikpantaleoni@gmail.com>
Co-authored-by: forforever73 <690105611@qq.com>
* metal : per-device tuned (Q, NE) for flash-attn vec (#25750)
* rebase Q-generic FA vec body from 01dc93607 (#23114)
* add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133)
* add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback
* add FA vec (Q,NE) perf sweep
* fill tuning result
* fold family table into a per-family representative SKU
* refactor tuning result format
* extend FA vec tuning to quantized KV caches
* sync fa vec tuner bucketing with runtime, use pointwise tuning regret
* update tuned table
* format and cleanup
* prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active
* add device id -> token lookup for the offline tuning tool
* add ggml-metal-tuning skeleton
* add op-agnostic perf cell + median timing for the tuner
* add FA-vec graph build + tensor init to the tuner
* tools : add FA-vec (Q,NE) sweep, compression and table emit
* cool down and re-measure the dirty window on thermal drift
* test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice
* tools : document the Metal tuner, point the table comment at it
* abort on unknown KV type, single-source fa_vec_legal_ne
* cleanup
* honor -o in the FA vec (Q,NE) slice
* retune FA-vec (Q, NE) under a pointwise no-harm gate
* cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* DSV4: sm tensor
* set coarser granularity for head splits
* fix dspark
* add model saving for dsv4 + allow dflash to return on specific device
* add comment about dsv4 seq_rm
* simplify
* add shared expert delayed allreduce
* remove special test for dsv4
* DeepseekV4: fix rollback with multi-seq
* fix model loading
* make pending rollback single use
* only clear cache for seq_id for full load
* add assert for compress ratio
* make graph topology static
* pass true instead of flags in clear_compressed
* cont : clean-up + TODOs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Resolve the TODO in test_flash_attn_ext: the branch that creates V as a
sub-view of K (MLA-based models) was hardcoded for the 576/512 head shapes.
Add a v_is_view_of_k test case parameter (default false) and select the
sub-view branch on it; the existing 576/512 (DeepSeek MLA) cases now pass it
explicitly, so the test coverage is unchanged.
Also add more V-is-sub-view-of-K cases: the 320/256 (Mistral4 MLA) and
192/128 head shapes, and full views with equal head sizes (128/128 F16,
64/64 q8_0).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a
static K=32 tile to the matmul2d op on every iteration. On the last, partial
K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the
tensor, and the op reads those out-of-bounds elements (undefined behavior per
the MSL specification, section 2.22.2). Depending on stale memory contents,
this corrupted the result or produced NaN.
Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both
operand tensor views to the remaining valid K range (min(32, K - loop_k)) per
iteration, so the op reads exactly the valid K range on every iteration
(mirroring the tail handling of the MPP matmul2d examples). On K-aligned
inputs the clamp degenerates to the full 32-wide tile: the only difference
from the static-K op is that the dynamic-K op derives K from the operand
extents and edge-checks the tile against the tensor extents (a handful of
integer ops per iteration).
Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise
the unaligned K path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal: dequantize q8_0 KV to f16 before flash attention
Add a preprocessing pass for GGML_OP_FLASH_ATTN_EXT on the Metal backend:
when the KV cache is quantized (Q8_0 for now), dequantize K and V into a
contiguous F16 scratch buffer and run the existing F16 flash attention
kernels on it, instead of the in-kernel dequantization path.
- new kernel kernel_flash_attn_ext_dequant_to_f16<block_t, QK, deq_t4x4>:
one thread per quant block (K then V), stride-aware so permuted KV is
supported; instantiated for Q8_0 (extending to Q4_0/Q4_1/Q5_0/Q5_1 is
one instantiation + one gate case)
- the gate is type-only: dequantize whenever the KV is quantized,
regardless of head sizes, GQA ratio or n_kv; the attention kernels
themselves are untouched
- the F16 copies live in the op's own scratch allocation
(ggml_metal_op_flash_attn_ext_extra_dequant_f16); the KV pad kernel
reads the dequantized buffers when the path is active
- the FA pipeline getters gain a use_f16_kv flag selecting the existing
f16 kernels and contiguous strides
- ref: https://github.com/ggml-org/llama.cpp/pull/25556
Verification (M2 Ultra):
- test-backend-ops test -o FLASH_ATTN_EXT: 4798/4798 pass, including the
new q8_0 eval cases (decode/prompt, permuted, sinks+ALiBi+softcap,
kv=113 pad path, kv=16384)
- llama-perplexity on Qwen2.5-0.5B with -ctk q8_0 -ctv q8_0 matches the
f16 KV reference (PPL 1.0008 vs 1.0008)
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : launch the FA KV dequant kernel separately for K and V
Simplify kernel_flash_attn_ext_dequant_to_f16: it now dequantizes a single
tensor (its own ne/nb and dst) with no is_v branching, and the op dispatches
it twice with the same pipeline - once for K and once for V. The kargs
struct shrinks to a single ne/nb set plus nblocks.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : dequantize q4_0, q4_1, q5_0 and q5_1 KV to f16 before flash attention
The dequant pass now covers all quantized KV types supported by the Metal
flash attention kernels. The dequant kernel, kargs, scratch allocation and
dispatch are type-generic, so each type is one kernel instantiation plus one
gate case.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : skip the redundant V dequant when V is a view of K
In MLA-based models, the V of the FA op is a view of K (the first ne20
elements of each K row); the dequantized V is then a view of the dequantized
K, so skip the second dequant dispatch, do not reserve the V scratch region,
and let the pad and attention kernels read V from the K F16 buffer with K's
strides. The detection follows the CUDA backend:
V->view_src && (V->view_src == K || (V->view_src == K->view_src && V->view_offs == K->view_offs))
Also fix the FA pipeline getters: ns10/ns20 are function constants baked into
the kernels and must be the actual K/V row widths as seen by the kernel. The
dispatch now passes them explicitly (nb11_attn/nb10_attn, nb21_attn/nb20_attn)
instead of the getters assuming contiguous F16 KV (ns20 = dv), which was wrong
when V is read from K with K's row pitch (e.g. 576 vs 512).
New test cases: 576/512 q8_0 (MLA shape, V is a view of K) at kv=113 (KV pad),
nb=1 (vec) and nb=64 (non-vec).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* test : remove backend-specific wording from test-backend-ops comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : avoid backend mentions in test-backend-ops comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : rename the FA dequant_f16 identifiers to kv_f16
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : clean-up
* cont : remove TODO
* vulkan : dequant q8_0 KV once in coopmat1
Assisted-by: Claude (Opus 4.8)
* vulkan : fall back instead of aborting when FA scratch exceeds maxStorageBufferRange
* vulkan : require KV-cache layout in FA dequant path
Assisted-by: Claude (Opus 4.8)
* vulkan : skip FA dequant path on coopmat2
Assisted-by: Claude (Opus 4.8)
* tests : add contiguously-allocated quant K/V FA tests
Assisted-by: Claude (Opus 4.8)
* vulkan : trim comments
* vulkan : tighten permutation checks for FA path
* vulkan : set prealloc_x_need_sync after the FA dispatch
* vulkan : exclude Intel Xe1 from FA dequant path
* feat(convert): Add conversion for GraniteSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(llama): Add granite_swa support
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add conversion infra for rope_pattern array
NOTE: There is other work also targeting this, so this may be
removed depending on merge order.
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(conversion): Fix SWA pattern logic and support for non-rope layers
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add support for GraniteMoeSWA
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add llama_hparams::has_rope and arch constants
NOTE: This shadows the work done for Granite Speech
https://github.com/ggml-org/llama.cpp/pull/25107
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add support for per-layer rope determination
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Fix failing flake8 for extra newlines
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver
Branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Load MoE params as optional
Branch: GraniteSWAForCausalLM
AI-usage: draft (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Handle MoE params in conversion
branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Remove unnecessary newline
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Remove unnecessary tensor additions to GRANITE architecture
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Correctly handle naming for ffn gate inp
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Always default hparams.rope_pattern to 1s
This isn't strictly necessary, but it will allow other models to rely on
hparams.has_rope(il) without needting to prepopulate.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Move to has_rope for all granite model architectures
Now that we have a proper hparam for this, it's better to use it and not
require a hacky fallback in the hparam method itself.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: No hacky rope_finetuned fallback in has_rope
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fully remove rope hparam filling in granitemoe
There are no granitemoe models that use NoPE (it's not actually used in the
layer building below), so this was just dead code.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Save out rope_pattern in model-saver
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Set hparams.rope_finetuned for round trip
Since the value is _read_ from rope_finetuned, we need to persist it when
the model is saved with the saver.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Code review cleanup
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* refactor: Keep gate/up fused for MoE path
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Skip GRANITE_SWA in model saver
https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651
Keeping is_swa_impl in the saver can break other models.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* add sliding window pattern for model in test
* style: Fix indentation
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fix \r\n
Thanks Claude!
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Keep shared expert fused
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: More indentation fixes
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* add params
* cpu kernel
* metal kernel
* add test backend ops
* gate other backends
* ggml: (cuda) support ggml_rope_set_offset (#27121)
* rm cuda supports_op guard, fix webgpu clang-format
* ggml: support ggml_rope_set_offset on vulkan (#27344)
* ggml: support ggml_rope_set_offset on vulkan
* remove inplace optimization
* vulkan: tiled transpose for 0<->2 permuted CONT
-ggml_vk_get_cpy_pipeline only routed to the tiled shared-memory transpose
shader when dim1 was the innermost dimension, i.e. ggml_transpose (a 0<->1
swap). A 0<->2 swap -- ggml_cont(ggml_permute(x, 2, 1, 0, 3)) -- fell back to
the generic per-element strided copy, whose source reads stride by ne0*ne1
elements: one cache line per lane.
-DeepSeek-V4's lightning indexer performs exactly that permute on a
[n_kv, n_tokens, n_head] tensor. On Vulkan/RADV gfx1151 it ran at ~1-9 GB/s of
a ~200 GB/s part and accounted for 43% of total prefill time.
-Add copy_transpose_02.comp, mirroring copy_transpose.comp but tiling over dst
dims (0, 2) with dims 1 and 3 as the batch, so reads walk src dim2 and writes
walk dst dim0 -- both contiguous. The selection condition additionally requires
a non-contiguous source and a contiguous destination so it cannot take cases
the contiguous-copy shader already handles.
-test-backend-ops only exercised ggml_transpose for CONT, so the strided path
was untested. Add test_cont_permute covering (2,1,0,3), (1,2,0,3) and (0,2,1,3)
over f32/f16 at tile-aligned, tile-unaligned and large shapes. The large shapes
are in the eval set rather than only in perf because perf mode does not verify
results.
-Measured on gfx1151, ne=[n_kv,64,64,1], perm=(2,1,0,3), f32:
n_kv=1024: 9.08 -> 579.85 GB/s
n_kv=1280: 20.03 -> 153.71 GB/s
n_kv=2048: 7.11 -> 91.68 GB/s
n_kv=2304: 16.24 -> 86.49 GB/s
-The ~2.2x penalty previously seen at power-of-two n_kv (destination-stride
aliasing) is gone. End to end, DeepSeek-V4-Flash IQ3_XXS prefill on a 9k-token
prompt goes from 56.33 t/s to 103.74 t/s (+84%).
-Note: at n_tokens=512 a single slow-path dispatch takes ~273 ms and looping it
in perf mode can trip the GPU watchdog, so the perf cases use n_tokens=64.
* tests: fold test_cont_permute into test_cont, add L2-exceeding perf shapes
Review feedback: test_cont gains a permute parameter ({0,0,0,0} = none),
matching test_mul_mat's pattern, and the separate struct is gone. Perf
adds [n_kv, 512, 64, 1] variants (~0.5 GB per run) that exceed GPU L2,
since the 64-token shapes fit in cache on large parts and read above
memory bandwidth.
* tests: trim perf-case comment to the two-line summary
* vulkan: trim comments on the 0<->2 transpose path
Drop the shader file header, the read/write block comments and the
rationale prose in the CONT test cases. Keep the tile-shape and
bank-conflict notes and the permute parameter documentation.
---------
Co-authored-by: Kevin Hopper <no-reply@maestro.press>
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
* Adding support for bailingmoe3
* Adds speculative decoding support
* Make BailingMoE3 safe gate metadata optional
* bailingmoe3: apply trained SwiGLU clamps
* common: fix Bailing V3 tool argument parsing
* llama-model-saver, instantiate float vector metadata writer
* bailingmoe3: support Q-LoRA (Ling-3.0-tiny)
Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current
implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256
and routes Q through a LoRA bottleneck instead:
q_a_proj -> q_a_layernorm -> q_b_proj
Conversion therefore failed with:
ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight'
Add the missing path, mirroring the existing deepseek2 MLA implementation:
* constants.py - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3
* tensor_mapping.py - map model.layers.{bid}.attention.q_{a,b}_proj and
q_a_layernorm
* conversion - emit attention.q_lora_rank when the config has it
* bailingmoe3.cpp - read n_lora_q; create the Q-LoRA tensors and build Q
through the bottleneck when q_lora_rank > 0
Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no
q_lora_rank, the converter only emits the key when present, hparams.n_lora_q
defaults to 0, and get_key(..., required=false) leaves the target untouched when
the key is absent - so flash keeps taking the existing direct-Q branch.
The LoRA path produces the same shape as the direct projection, so the
nope/rope split, RoPE application and wk_b absorption downstream are unchanged.
* small mtp change
* bailingmoe3: support separate MTP GGUF and Q-LoRA MTP
* gguf: remove duplicate add_kda_gate_lower_bound definition
---------
Co-authored-by: bloomer <bloomer@booper.brushtail.me>
Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com>
* model: add Kimi-K3 text model
Hybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five
things that architecture does not have:
1. cross-layer residual attention (attn_res_block_size)
2. latent MoE (routed experts run at n_expert_latent)
3. situ activation (replaces SwiGLU everywhere)
4. MLA output gate (sigmoid gate before o_proj)
5. full-rank KDA gate (single ssm_g instead of ssm_g_a/ssm_g_b)
K3's text_config reports KimiLinearForCausalLM - the older 48B architecture -
so get_model_architecture routes on the top-level name instead.
The KDA decay gate has two forms, selected by linear_attn_config's
gate_lower_bound. It is not a clamp: when set it swaps the activation entirely
(fla/ops/kda/gate.py), from -exp(A_log)*softplus(x) to
lower_bound*sigmoid(exp(A_log)*x). K3 sets it to -5.0; kimi-linear leaves it
unset, so that path is unchanged.
Cross-layer residuals reuse ggml_dsv4_hc_pre for the weighted sum. That op is
CPU + CUDA only, so Metal/Vulkan will fall back per-node until those kernels
exist.
The routed experts ship as compressed-tensors "mxfp4-pack-quantized". That is
bit-compatible with ggml's MXFP4 - same E2M1 code assignment, same E8M0 scale
byte, only the nibble positions within a block differ - so they are repacked
rather than dequantized, losslessly and without a ~5.5 TB bf16 round-trip.
The repack is built lazily because gguf_writer holds every added tensor until
the final write. DeepSeek-V4 was already doing the identical bit-shuffling, so
it now shares the helper.
Verified against Moonshot's own code path (transformers + fla's Triton KDA
kernels) on a tiny model exercising every K3-specific feature. Final-position
logits vs the fp32 reference: 6.7e-05 rel / corr 1.00000000 for both the
chunked and the recurrent delta-net path. MXFP4 blocks dequantize to the source
weights with 0.0e+00 error.
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* model: fix ty errors in the Kimi-K3 converter
- `_res_parts` buffers (kind, tensor) pairs, not bare tensors
- `get_tensors` must return an Iterator, matching ModelBase
- LazyBase's `func` takes one argument, so pass the expert loaders through
`args` instead of the closure
- borrowing KimiLinearModel.set_vocab from an unrelated TextModel is
deliberate and safe, but not expressible in the signature
No behaviour change: the MXFP4 repack still dequantizes to the source weights
with 0.0e+00 error and end-to-end logits are unchanged (8.386e-03 rel,
corr 0.99996630).
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update conversion/kimi_k3.py
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
* Increase LLAMA_MAX_EXPERTS from 512 to 1024
* tests : support for Kimi K3 in archs test
* chat : add Kimi K3 chat format (reasoning, content, typed tool calls)
K3's assistant output is an XTML-ish tagged format built by the template's
open_tag/close_tag macros. Two properties break generic parsing:
1. The generation prompt ends with open_tag('think'), so the completion
starts inside the think section with no opening marker in the output
(thinking_forced_open).
2. Only <|open|>/<|close|>/<|sep|>/<|end_of_msg|> are special tokens; tag
names ("think", "response", "message") are ordinary text tokens.
Adds common_chat_params_init_kimi_k3 (PEG_NATIVE) with detection on the
marker trio, reasoning extraction, response unwrapping, and tool-call
parsing of the tools/call/argument tag structure with argument types
taken from the tool schema. Includes the K3 chat template fixture and 9
test-chat cases derived from real generations of the full 2.8T model.
Verified end-to-end against Kimi-K3-Q2_K (GrEarl/Kimi-K3-GGUF) on 8x B200:
content, reasoning_content, streaming deltas, and tool_calls all correct;
finish_reason stop/tool_calls as appropriate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chat : add message_delimiters for Kimi K3
Per-role message-start markers for token-level span splitting. User and
assistant messages carry only the role attribute, so their full opener
(through <|sep|>) is used; system and tool messages continue with more
attributes (type=/tool=/index=), so those delimiters stop after the
role's closing quote. Verified against the K3 tiktoken vocabulary that
the closing quote is always a standalone token across all attribute
variants, so the token-level prefix match stays exact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: apply nits from @ngxson and text fixes from @danielhanchen
* tests : added missing hyperparameters and tensors for Kimi K3 in test-llama-archs
* chore : move overly verbose header file comments to Kimi K3 source file
* tests : re-enabled KIMI_K3 in test-llama-archs for WebGPU backend
* model-saver : emit kda_gate_lower_bound for Kimi K3
Quick fix. The Kimi K3 loader reads kda_gate_lower_bound and gates a graph branch on it (it scales the KDA gate when the bound is above -INFINITY), but the model
saver never wrote the key, so a save->load roundtrip silently dropped it back to the -INFINITY default and changed the model's output. The real K3 config sets gate_lower_bound = -5.0.
I propose to emit it from the saver, and set it to -5.0 in the test-llama-archs K3 case so the roundtrip check exercises it (the roundtrip fails without the saver line).
* Refactor conditional for model architecture check
* tests : re-enabled (again) KIMI_K3 and MINIMAX_M3 in test-llama-archs for WebGPU backend
* fix code comments
* add template on conversion
* move repack_mxfp4_blocks to model base
* nits
* add_value_length
* optimize res_stack construction
* nits
---------
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Deepankar Singh <singh.deepankar39@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Caleb DeLeeuw <caleb.deleeuw@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* llama : support for MiniMax-Text-01 model
* chore : renames to match the other MiniMax models
* model : add logits mask as MiniMax-Text-01 embeddings tensor has zero-valued embeddings for tokens >= 200032 that produce zero logits disrupting the token sampling process
* llama : replace hardcoded conditions with hparams.is_recr()
* model : used build_rs() for recurrent state management
* chore : code cleanup
* model : optimized MiniMax-Text-01 by removing the state tranpose operations
* chore : removed unnecessary ggml_cont() in MiniMax-Text-01 implementation
* llama : add generic logits mask graph input
* model : permuted diag_decay dimensions to avoid doing it inside MiniMax-Text-01 graph
* chore : code cleanup
* chore : code cleanup
* model : use token positions when calculating MiniMax-Text-01 decay tensors
* convert : add support for MiniMaxM1ForCausalLM as it seems to be the same as MiniMaxText01ForCausalLM
* chat : add jinja template for MiniMax-M1
Co-authored-by: QscQ <qscqesze@gmail.com>
* chore : code cleanup
* tests : MINIMAX_01-related fixes
* chore : silence Python lint errors
* vocab : remove unnecessary vocab type
* convert : update MiniMaxText01Model conversion to use yield when modifying tensors
* convert : suppress tokens with zero-valued embeddings during MiniMax-Text-01 conversion
* llama : removed logits mask - no longer necessary as token suppression is used instead
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : override non-working built-in chat template during conversion
* tests : skip arch MINIMAX_01 tests for WebGPU backend (it breaks again)
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: QscQ <qscqesze@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* chat: add reasoning_effort to common_chat_templates_inputs
Store OpenAI Chat Completions reasoning_effort and make it
available to jinja templates (with model specific translations
where required).
Assisted-by: llama.cpp:Muse-Glimmer-30B
* server : fixup reading reasoning effort from body
server_chat_convert_responses_to_chatcmpl already handles conversion of
Responses API reasoning.effort to reasoning_effort
* chat : expose reasoning effort
Assisted-by: Claude Opus 5
* chat : add reasoning_effort to generation_params
Assisted-by: Claude Opus 5
* chat : move reasoning_effort next to enable_thinking
Assisted-by: Claude Opus 5
* cont : mirror preserve_reasoning
* cont : pass context through analyze function
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
* Initial changes for Recurrent state rollback for nemotron for cpu and cuda
* Removing CPU RS rollback. Will enable it in subsequent PRs
* addition of test case
* Removing assert and calling runtime API to check if op is supported
* removing extra API and updating the call sites for K
* replace static cuda detection to runtime fused_op api
* address review comments and fallback when SSM rollback not supprted
* Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda
* removing memory manipulation as rs rollback is now supported in CPU
* removing the static probe which is not needed now
* correcting the format
* address review comments
* enabling test for all the backends, unsupported backends will fallback to CPU
* Apply suggestions from code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported
* Support K > 1 in ssm_scan for all backends
* Fix CI Issues
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Scrub developer-specific /home/<user>/ paths from example docs and test
fixtures so they don't leak into the tree.
- examples/test-cmake/README.md: /home/danbev/... -> /path/to/llama.cpp/...
- tests/test-chat.cpp: /home/jarvis/... -> /home/user/... (input and
expected string kept identical so the parser test still passes)
Co-authored-by: Jim Wu <ywu@xilinx.com>
* metal: add TQ2_0 support
Add support for the GGML_TYPE_TQ2_0 (ternary, 2 bits per element) type in
the Metal backend.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* cont : optimize mul_mv kernel
- float ops over integer ops
- precalculate sums
- hoist coef out of the inner loop
- contiguous y loads
llama.cpp:DeepSeek-v4-Flash-0731
* cmake : introduce semantic versioning (wip)
This commit introduces semantic versioning to llama.cpp.
* squash! cmake : introduce semantic versioning (wip)
* cmake : update test-cmake README notes [no ci]
* include libmtmd in output so show its semversioned
* ci : add make-release workflow
* ci : fix build number check in build-cmake-pkg.yml
* examples : remove trailing whitespace
* ci : abort if upstream ggml version does not exist
* ci : extract step contents into scripts
* ci : add GGML_NATIVE=OFF to ubuntu job
* examples : remove CI build information from test-cmake [no ci]
This commit removes the nightly/release information that I added
previously to keep this focused only on using building and installing
llama.cpp with cmake and being able to quickly verify changes or
troubleshoot issues.
* ci : merge scripts into single script
* remove -dev-build_number support
This commit removes the incremental build number (versioning) support
that I added. This was incorrect and we should only use the semver for
the version. Releases will be tag a nightly build and package
maintainers/managers that build from source can use the tag and it is
therefor important that the correct version is reported. So a
nightly-build will report the semver without the build number. The build
number and commit as availble via cmake and test-cmake has been updated
to include an example of using them:
```console
$ ./build.sh
[test-cmake] version: 0.1.0, build: 10360 (08c69e381)
...
```
Refs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969
* docs: add initial release.md documentation
* cmake : clean-up and add LLAMA_BUILD_IS_DEV option
* ci : remove version input from make-release job
* ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml
Refs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145
* docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci]
* ci : add TODO to winget workflow [no ci]
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* gguf : harden loader against malformed tensor dims and metadata types
* gguf: address review on malformed-metadata hardening
- report the expected vs. actual type when general.alignment is not u32
- use ggml_nelements() > 0 for the zero-element guard and keep the
representability checks visually aligned
- add test-gguf cases for a wrong-typed alignment key and a zero-dim
tensor (both used to crash: assert-abort and SIGFPE respectively)
Ran tests/test-gguf: 164/164 pass. Used an AI assistant to help draft
these edits; reviewed and verified by me.
* cont : less comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan: TQ2_0 (ternary) support — dequant + dedicated mul_mat_vec + matmul via dequant_funcs
First Vulkan ternary type in ggml. Correctness: OM-125m TQ2_0 vs F16 top-12
logprobs identical to 4 decimals fully offloaded (float dequant path, no Q8_K
activation quant). Speed at 125m ~= F16 (overhead-bound at this scale); the
bandwidth win targets larger BitNet SKUs. MMQ/int-dot path intentionally not
wired yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: enable TQ2_0 in backend-ops type lists
Vulkan now implements TQ2_0 (dequant, mul_mat_vec, mul_mm, get_rows); backends
without support skip via not-supported as usual. TQ1_0 stays disabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Michael Trabalka <michael.trabalka@sqv.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* chat : fix muse-glimmer swallowing a trailing tool call into content
Muse Glimmer routinely answers the user and calls a tool in a single
generation. The template terminates a message with <|eom|> when more
messages follow in the same turn and <|eot|> only at the end of the turn,
so the answer is closed by <|eom|> and the call opens a fresh header:
<prose><|eom|><|start|>assistant to=<tool><|message|><atem:function_calls>...
The final-message rule read content with until("<|eot|>"), which assumed the
user-facing message is always last. There is no <|eot|> before the call, so
content ran to the end of the turn, absorbed the markup, and no tool_calls
were emitted - the tool never ran. On a tau2-bench telecom run this hit 43
turns across 19 of 114 tasks.
Stop the answer at <|eom|> and parse what follows as tool calls.
Adds models/templates/muse-glimmer.jinja and four parser tests: a plain
answer, the <|eom|> junction, markup quoted in an answer staying content,
and tool markup inside the to=self channel staying reasoning.
* address comment
* test-backend-sampler: skip multi_output_sampling_chain on HIP
The new multi_output_sampling_chain test uses top_k, whose backend probs
path needs CUB (unavailable on HIP), so sampled_probs is null and the test
aborts. Add it to the existing HIP skip list alongside the other TOP_K tests.
* ci: keep gpu-rocm logs in a per-run dir keyed by GitHub run id
The self-hosted gpu-rocm runner can't upload logs to Azure blob (egress
firewalled), so a run's logs were wiped by the next run. Write each run's
logs to $OUT/run-<run_id>-<attempt>/ so an Actions run URL maps to its logs.
* test-backend-sampler: also skip multi_output_cpu on HIP
Like the other TOP_K-based subtests, multi_output_cpu's backend sampler
never initializes on HIP (no CUB TOP_K), so it aborts. Add it to the skip list.
---------
Co-authored-by: Jim Wu <ywu@xilinx.com>
* Enable backend sampling with token speculation
* Clamp the mask sum before converting it into the sampled index
* Add a numeric context parameter declaring the maximum outputs one sequence
* More fixes
* Don't reuse memory for output views.
* Match dist between CPU and GPU
* Fix CPU and backend sampling mismatches
* Simpify some of the changes
* Fix tests on Vulkan
* More test fixes
* Rebase changes
* Rebase and address review comments
* Address review comments
* Address review comments
* Update src/llama-sampler.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a
valid input, but the CUDA and Metal roll kernels index by ne alone and
never read the nb strides. A non-contiguous src therefore produced
silently wrong results. Neither backend declared a contiguity
requirement in supports_op, so the scheduler did not fall back to the
CPU implementation, which does handle strides correctly.
Add the requirement to both backends, matching the existing
GGML_OP_ROPE guard, and add a permuted test_roll case.
Xuan Son Nguyen
Gaurav Garg
Max Krasnyansky
Daniel Han
Dominik Pantaleoni
YiChen Lv
fairydreaming
Aman Gupta
Georgi Gerganov
Aldehir Rojas
Nathanw1014
Gabe Goodhart
Kevin Hopper
Masashi Yoshimura
Jeff Bolz
Thiago Padilha
BlackFoil
Toby
Piotr Wilkin (ilintar)
0
Andy Williams
lnigam
Jim Wu
Titaniumtown
Daniel Bevenius
HarrisonSec
michaeltrabalka-tech
ruanslv
Rafail Giavrimis
0
Yash Raj Pandey