* Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (#28184)
This reverts commit c845263f8b.
* tests : fix unused variable M in test-backend-ops
* tests: fix trailing space error and isolate kronecker tests for sycl backend only
define two new environment variables to better understand how much
memory is being allocated, and when. This has been invaluable in
inproving the --fit algorithm, and is likely to be useful when debugging
other memory-related issues.
`-lv 4` will be required to enable the following:
GGML_SYCL_MEMTRACE=1 will show per-site memory usage, updated whenever
it increases by more than 64MiB.
GGML_SYCL_MEMTRACE=2 will show every allocation and deallocation.
To change the default 64MiB threshold for reporting memory usage increases, use
GGML_SYCL_MEMTRACE_STEP.
A sample log line:
[SYCL-MEMTRACE] device memory query (dev): total 59493 MiB, free 4494, in use 54998; allocated 0 (buffers 0 + scratch 0), peak 0 MiB
* ui : fix MCP image attachments not displayed in tool block (#25789)
Fixes regression from #25450 where ChatMessageAgenticContent passed
message.extra instead of section.toolResultExtras to tool blocks,
leaving tool images invisible. Also fixes TOOL_RESULT_JSON_OPEN_REGEX
which misclassified "[Attachment saved: ...]" as JSON.
Fixes#25789
Assisted-by: Muse Spark
* Addressed PR comments: 1.- Removed ·?? mesage?extra· as it has no case left to cover 2.- Added ·[\· to cover the case of ·[[1, 2], [3, 4]]· case suggested in the PR comment 3.- Added unit test for covering up this regex case
* ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) - Addressed lint error on regex (redundant \)
* opencl: add extended elementwise unary ops (sgn, step, elu, hardswish, hardsigmoid, floor, ceil, round, trunc)
Adds nine GGML_UNARY_OP_* elementwise ops that were falling back to CPU on the
OpenCL backend, following the same variant shape as the existing ABS op: f32,
f32_4 (vec4), f16, f16_4 (vec4), and stride-addressed f32_nc / f16_nc for
non-contiguous inputs. New kernels/unary_ext.cl (macro-generated), a shared
ggml_cl_unary_ext dispatch helper mirroring ggml_cl_abs, the supports_op cases,
and the compute-forward cases.
Values are computed in float (the f16 variants read/write half and convert), so
the conditional ops (step, elu) match the CPU reference; the vec4 forms use
select() for the branch.
Validated with test-backend-ops on Adreno 840 and 850 (E17): all nine ops pass
every case including the vec4 and non-contiguous variants (8/8 or 14/14).
* opencl: dispatch a contiguous f32 copy over the whole device
kernel_cpy_f32_f32 maps one workgroup to each (i01,i02,i03) row and strides the
row across that workgroup's lanes, and the host launches ne01*MIN(64,ne00) work
items. A tensor with few long rows therefore runs on a single workgroup. The
mamba2 and gated-delta-net recurrent state cache is one row of 524288 floats,
copied once per layer per graph, and lands on 64 work items.
When both sides are contiguous the copy is a linear move, so dispatch it over
the whole device: one work item per float4. Gated on ggml_is_contiguous for both
tensors and equal element counts, so copies already spread over many rows keep
the existing path. The kernel is created optionally, so a driver that rejects it
falls back rather than aborting.
vload4/vstore4 rather than a float4 cast: they require only the scalar type's
alignment, and these buffers carry an arbitrary 4-byte view offset.
CPY, DUP and CONT are 217/217 on Adreno 840 and 740 with the path enabled and
disabled. GGML_OPENCL_CPY_FLAT=0 forces the old kernel.
* opencl: support all easy-copy types in CONCAT
CONCAT was F32-only. Extend it to every "easy-copy" type -- any non-quantized
type with a block size of 1 and an element size of 1, 2, 4 or 8 bytes, i.e.
f16/bf16/i8/i16/i32/i64 as well as f32.
The kernels are keyed by element SIZE rather than by type, which is what CUDA
already does for the same op: one kernel per byte width (b1/b2/b4/b8) plus the
packed b4 fast path, instead of one per ggml type. supports_op gates on the
same property, so a new type of a supported width is picked up with no further
work.
Validated with test-backend-ops on Adreno 840 / A8X and X2-90 / X2E.
TOKEN/TOKEN_NOT early return: operator[] pre-inserts an empty candidates
vector at line 1123, setting cache_target. The TOKEN/TOKEN_NOT branch
returns without writing to *cache_target, leaving an empty vector in the
map. Subsequent lookups return that empty vector, reporting "reject
nothing" for what should be a non-empty reject set. Fix: write to
cache_target before the early return.
Pointer aliasing in candidate key: candidate lists were hashed as raw
struct bytes, which includes the code_points pointer value. glibc can
reuse heap addresses across calls, so the same pointer value can point
to different content, causing false cache hits. Fix: hash actual content
(index, id, partial_utf8 fields, and the decoded code_points sequence)
rather than raw bytes including the pointer.
* model: add Tencent Hy 4 (hy_v4) preview architecture support
Adds support for the Tencent Hy 4 model (Hugging Face architecture
HYV4ForCausalLM, GGUF arch hy_v4):
Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry
Register hy_v4 GGUF constants, arch enum, and writer support
Implement the hy-v4 model graph, hparams, vocab and context changes
Register the new arch in llama-arch and models registry
Extend arch tests to cover hy_v4
Assisted by Claude Opus 5
* Update convert_hf_to_gguf_update.py
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* Update conversion/base.py
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* convert : move hy_v4 entry to the same place as in convert_hf_to_gguf_update.py
* model : apply changes related to n_ff_exp becoming per-layer in Hy4-preview
* n_layer_all
---------
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
This commit adds a cmake version configuration file to replace the
current compile definition solution for the version.
The motivation for this change is that I made a mistake and did not take
into consideration that the compile definition means that this will
become a compiler flag for all sources in the target. This means that
when a version update happens that will recompile all sources in the
target even if they have not changed.
Refs: https://github.com/ggml-org/llama.cpp/pull/28278
TOKEN/TOKEN_NOT early return: operator[] pre-inserts an empty candidates
vector at line 1123, setting cache_target. The TOKEN/TOKEN_NOT branch
returns without writing to *cache_target, leaving an empty vector in the
map. Subsequent lookups return that empty vector, reporting "reject
nothing" for what should be a non-empty reject set. Fix: write to
cache_target before the early return.
Pointer aliasing in candidate key: candidate lists were hashed as raw
struct bytes, which includes the code_points pointer value. glibc can
reuse heap addresses across calls, so the same pointer value can point
to different content, causing false cache hits. Fix: hash actual content
(index, id, partial_utf8 fields, and the decoded code_points sequence)
rather than raw bytes including the pointer.
Use std::error_code overloads of fs::current_path() and
fs::directory_iterator in ggml_backend_load_best() so an
inaccessible search path (WebDAV mount, removed CWD) is
skipped instead of terminating the process with an uncaught
filesystem_error.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Let llama_print_build_info write to a caller-provided FILE* instead of
hardcoding stderr. The parameter defaults to stderr so existing callers
keep their current behavior.
The version command in llama-app now passes stdout, so plain version
output goes to stdout where users expect it.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* src : add n_expert_used_max function
With Commit c61b98b875 ("model: add
NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)") it
is now possible for each layer to have a specific number of experts but
there are a few checks that need to be updated to handle this upon model
loading. For example:
```console
llama_model_load: error loading model: model has expert layers but no expert layers are used
```
And later:
```console
/llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed
```
This commit adds the n_expert_used_max function so that these checks
can use it.
Refs: https://github.com/ggml-org/llama.cpp/pull/25444#issuecomment-5524976031
* src : use hparams.n_expert_used_max in llama_model_base::load_hparams
* src : use 0 as initial value for n_expert_used_max
Fuse RMS_NORM+MUL+ADD and ADD+ADD under GGML_SYCL_ENABLE_FUSION.
ADD+ADD uses the same binbcast indexing and type matrix as standalone
add() (f32, f16, f16/f32, i32, i16, bf16, including broadcast and
non-contiguous). Unsupported combinations fall back to two add() launches.
* opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations
* opencl: guard q4_K/q6_K tiled_ns convert-kernel registration for non-Adreno build
* opencl: gate q4_K MUL_MAT+GLU fusion dispatch to Adreno
* opencl: require the noshuffle weight layout in the q4_K GLU fusion gate
* opencl: do not take the vectorized f16 mrow GEMV path on an unaligned row stride
* opencl: pass the new get_scale_min_k4 stride argument at the row-major call sites
* opencl: enable the q4_K split-K decode GEMV only where it is measured to win
* opencl: record the X1-85 split-K datapoint (neutral, exclusion confirmed)
* opencl: restrict the tiled lm_head/embed GEMV default to X2E/A8X
* opencl: fix q4_K variant kernels to read the transposed scales layout
* opencl: keep the flat-GEMV large-m escape opt-in
* opencl: guard the o4 GEMV store against the rounded-up dispatch tail
* opencl: restore the tiled q4_K/q6_K layout on tensor read-back
* opencl: split-K for the q8_0 decode GEMV at small M
* opencl: keep the q6_K noshuffle correctness escape ahead of the opt-in gate
* server : use pytest-xdist for server tests
This commit adds pytest-xdist to the server tests. This is pytest
plugin that distributes test execution across multiple CPU cores.
Assisted-by: pi:llama.cpp/qwen3.8-27B
Refs: https://github.com/ggml-org/llama.cpp/pull/26734#issuecomment-5220707042
* remove server_base_port and BASE_PORT
* use worksteal and pytest builting tmp_path
* metal : support n_kv_max sparse mask hint in flash attention vec kernel
- add kernel_flash_attn_ext_vec_idx: compacts finite mask entries into
a per-row index list (Hillis-Steele scan, one threadgroup per row)
- extend vec FA kernel with optional sparse index gathering (FC slot 5)
- add host-side gate: sparse path when n_kv_max > 0, mask present,
supported head sizes / KV types, n_kv_max <= 4096
- new buffer region extra_idx for the index list
- pipeline getter extended with has_sparse param
- add test cases: head sizes, quant types, nb>1, nr23 variants,
sinks, ALiBi, softcap, permute, v_view_of_k, no-mask fallback
Note: multi-row (nb*nr23[1] > 1) cases still failing - rid mapping
in the store phase needs revisiting for the sparse path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : fix sparse flash attention row addressing
- kernel_flash_attn_ext_vec_idx: mask param is half* but nb31 is a byte
stride, so the per-row mask offset was scaled by 2x; cast to char*
before applying the byte strides
- kernel_flash_attn_ext_vec: sparse pidx param is char* so the per-row
element offset was under-scaled by sizeof(int); scale it by sizeof(int)
to get the correct byte offset
- fixes the multi-row (nb*nr23[1] > 1) sparse flash attention failures
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* cont : use sparse vec FA for prefill
* metal : single-pass flash attention sparse index compaction
The idx kernel previously read the mask row twice: once to count the finite
entries (for the prefix scan) and again to recover their positions. Since the
kernel is memory-bound, this doubled the mask traffic.
Keep the finite positions in a per-thread register array during the count
pass and write them out directly, avoiding the second mask read. A dense
mask with more than NLOCAL finite entries in a slice falls back to re-reading
the mask to write the remaining positions.
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* tests : add perf cases for sparse flash attention prefill
Measure the sparse vec FA kernel across KV sizes, n_kv_max hints and batch
sizes. Run with:
./build/bin/test-backend-ops -b MTL0 -o FLASH_ATTN_EXT -p "n_kv_max=[1-9]" perf
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* qwen4 : enable sparse attention
* cont : adjust nsg
* cont : sync test-backend-ops
* cont : disable Qwen4 for now
* cont : clean-up + tests
* mtmd : mark context as const in more methods
Mark `mtmd_context` as `const` in:
- mtmd_bitmap_init_lazy
- mtmd_tokenize
- mtmd_tokenize_from_parts
- mtmd_helper_support_video
- mtmd_helper_bitmap_init_from_file
- mtmd_helper_bitmap_init_from_buf
- mtmd_helper_video_init
- mtmd_helper_video_init_from_buf
- mtmd_helper_model_can_chat
The tokenization functions in particular are useful to have marked
`const`, as that allows more easily telling the compiler that we can
safely tokenize from multiple threads (`mtmd_tokenize` is already
documented as thread-safe, this just reifies that in the signature).
* mtmd : mark tokenization input pointer as const
Mark the `bitmaps` and `parts` pointers in `mtmd_tokenize` and
`mtmd_tokenize_from_parts` as `const`. This allows more easily calling
these with immutable arrays / vectors.
* mtmd : mark llama_context as const in mtmd_helper_model_can_chat
Concedo
Jingxin (Philip) Li
Nick Farrell
Isaac
nachobh
Hongqiang Wang
yzyyzyhhh
Reithan
yhz5613813
Sigbjørn Skjæret
Tom Tan
HongHuang
Georgi Gerganov
Xuan-Son Nguyen
Niklas Wenzel
Daniel Bevenius
Evan Huus
Adrien Gallouët
Alessandro de Oliveira Faria (A.K.A.CABELO)
Ravi Panchumarthy
Aaron Teo
Frosty40
Ozymandias_EBON
Wagner Bruna
Sergey Sklyarov
kbenkhaled
Max Krasnyansky
Todor Boinovski
Mads Marquart