* vulkan: TQ2_0 (ternary) support — dequant + dedicated mul_mat_vec + matmul via dequant_funcs
First Vulkan ternary type in ggml. Correctness: OM-125m TQ2_0 vs F16 top-12
logprobs identical to 4 decimals fully offloaded (float dequant path, no Q8_K
activation quant). Speed at 125m ~= F16 (overhead-bound at this scale); the
bandwidth win targets larger BitNet SKUs. MMQ/int-dot path intentionally not
wired yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: enable TQ2_0 in backend-ops type lists
Vulkan now implements TQ2_0 (dequant, mul_mat_vec, mul_mm, get_rows); backends
without support skip via not-supported as usual. TQ1_0 stays disabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Michael Trabalka <michael.trabalka@sqv.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: handle nested global_head_dim in Gemma4 config
Gemma-4 E4B models have global_head_dim inside text_config
rather than at the top level. Add fallback to support both layouts.
* fix: add fallback for global_head_dim to support per_layer_config format
* fix: read head_dim only from full_attention layers in per_layer_config and num_global_key_value_heads compatibility
* fix: added fallback for num_global_key_value_heads
* fix: read per_layer_config from root hparams
* fix: delete unused text_config
* cleanup and fixes
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* chat : fix muse-glimmer swallowing a trailing tool call into content
Muse Glimmer routinely answers the user and calls a tool in a single
generation. The template terminates a message with <|eom|> when more
messages follow in the same turn and <|eot|> only at the end of the turn,
so the answer is closed by <|eom|> and the call opens a fresh header:
<prose><|eom|><|start|>assistant to=<tool><|message|><atem:function_calls>...
The final-message rule read content with until("<|eot|>"), which assumed the
user-facing message is always last. There is no <|eot|> before the call, so
content ran to the end of the turn, absorbed the markup, and no tool_calls
were emitted - the tool never ran. On a tau2-bench telecom run this hit 43
turns across 19 of 114 tasks.
Stop the answer at <|eom|> and parse what follows as tool calls.
Adds models/templates/muse-glimmer.jinja and four parser tests: a plain
answer, the <|eom|> junction, markup quoted in an answer staying content,
and tool markup inside the to=self channel staying reasoning.
* address comment
* sd: generation progress fixes
The preview callback is not called if preview images are not enabled,
so when a preview image wasn't requested, the step count wouldn't be
updated. So move the update to the progress callback. Additionally,
adjust the total step count when the progress call reports a lower
total (e.g. for img2img).
Also remove the preview reset from inside the callback, since it
often caused a preview miss, depending on when the next preview
request arrived.
* sd: fix image preview behavior for VAE encoding / tiling
The progress callback is also called for VAE encoding and decoding,
receiving the number of tiles as step count, so there is no simple
way to detect the diffusion beginning. So we set up the first preview
callback to detect it, and transition to the decoding phase when
we reach the last step.
* adapt the api
* text model ok
* working impl, need verify and clean up
* mtmd: build the pocket-tts transposed convolutions as GEMM + col2im
ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample
was built as one convolution and one concat per channel, which floods
the graph with small nodes and makes kernel launches dominate the
decoder.
Fold both cases into the column form the seanet decoder already needs:
the general case reshapes the kernel to [IC, K * OC] and matmuls it
with the input, the depthwise case batches a matmul over the channels
so a step scales its own kernel. A single col2im_1d then scatter-adds
the columns back to the signal, with the same shape as before, so the
overlap-add tail, the streaming state and the bias are untouched.
Generation time per frame drops by 80% on CUDA and by 50% on CPU. The
output matches the previous implementation sample for sample, with a
correlation of 0.999994 and identical frame counts.
* flow_temp + frames_after_eos
* chunking
* mtmd: carry the remaining pocket-tts per-pack settings
The language packs also tune the end-of-speech padding and the padding
of short prompts, next to the temperature already carried in the
mmproj: french_24l asks for 8 tail frames instead of the guessed 3,
english_2026-01 asks for short prompts to be padded with spaces.
Write both in the mmproj as clip.gen.audio.frames_after_eos and
clip.gen.audio.pad_short_text, keyed on the pack in the conversion
script like the temperature. The loader keeps them optional, so a
mmproj without them behaves as before. Map semicolons to commas for
every pack instead, the reference only asks for it on three of them and
it costs nothing elsewhere.
Existing mmproj files must be converted again to carry the two keys.
On a long french text the port now lands within 2% of the reference:
22.96s against 23.44s, with the same peak level and the same amount of
silence.
* clip.gen.audio.model_variant
* clean up code comments
* nit: drop the dead flow_temp hparam, the pack table holds the default
* update docs
* address security problems
* less invasive base.py
* lint
* add mtmd_gen_inp_default
* add docs
* rm gen_flow_temp
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Most of the old ones have been resolved (yay) but the recent refactor of mmq paramters has caused some symbol names to change,
leaving a couple of non-ignored failures
This commit updates the python script that runs the original model to
generate embeddings for the causal model, to use save_output_data which
stores the token ids and the prompt in addition to logits.
The motivation for this is that the embedding logits verification will
fail as it expects these files (-prompt.txt and -tokens.bin) to exist.
With the changes in this commit the causal-verify-embeddings target
works again.
* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used
* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* set mmap_support to false on OpenCL backend
* fix order of load modes
* use -1 for auto
* resolve load mode auto earlier to correctly pick gpu host or cpu memory
* add load mode auto to llama-bench
* bump virtgpu api version, regenerate docs
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* test-backend-sampler: skip multi_output_sampling_chain on HIP
The new multi_output_sampling_chain test uses top_k, whose backend probs
path needs CUB (unavailable on HIP), so sampled_probs is null and the test
aborts. Add it to the existing HIP skip list alongside the other TOP_K tests.
* ci: keep gpu-rocm logs in a per-run dir keyed by GitHub run id
The self-hosted gpu-rocm runner can't upload logs to Azure blob (egress
firewalled), so a run's logs were wiped by the next run. Write each run's
logs to $OUT/run-<run_id>-<attempt>/ so an Actions run URL maps to its logs.
* test-backend-sampler: also skip multi_output_cpu on HIP
Like the other TOP_K-based subtests, multi_output_cpu's backend sampler
never initializes on HIP (no CUB TOP_K), so it aborts. Add it to the skip list.
---------
Co-authored-by: Jim Wu <ywu@xilinx.com>
* model : fix SWA not being enabled for EXAONE 4.5
load_arch_hparams tests `hparams.n_layer() == 64` before
LLM_KV_NEXTN_PREDICT_LAYERS has been read. n_layer() returns
n_layer_all - n_layer_nextn and n_layer_nextn defaults to 0, so a GGUF
carrying the MTP head (block_count=65, nextn=1) evaluates to 65 and the
whole SWA block is skipped. The model type switch further down in the
same function reads 64, because by then the key has been loaded.
n_swa is still filled in by the unconditional get_key below the block, so
llama_model_n_swa() reports 4096 and the logs look correct while only
swa_type stays LLAMA_SWA_TYPE_NONE.
This affects the official LGAI-EXAONE GGUF release as well. EXAONE 4.0 has
no MTP head, so block_count is 64 there and the check matches.
* model-loader : skip TENSOR_SKIP tensors in the metadata-only path
create_tensor asserts on a null buffer type when building from metadata
alone, but buft_for_tensor returns null by design for tensors marked
TENSOR_SKIP, which is how architectures with nextn/MTP layers mark theirs.
Those models cannot be constructed by llama_model_init_from_user at all.
The file-backed path below already returns nullptr for the same tensors, so
callers see the same thing either way.
* tests : cover exaone4 hparams ordering
Builds a synthetic exaone4 model with the layout the shipped EXAONE 4.5
GGUFs use (block_count 65 + nextn 1). The swa_type check is the one that
catches the ordering bug; the n_layer_nextn and n_layer() checks only tell
a broken fixture apart from a real regression.
Fails before the ordering fix with "swa_type is not STANDARD", passes after.
* Revert "tests : cover exaone4 hparams ordering"
This reverts commit d2f3bafeee591ad691396b2708de4baef3aaf602.
* Revert "model-loader : skip TENSOR_SKIP tensors in the metadata-only path"
This reverts commit aecb9bc0c7896b52afbc43921a1f572aa7b5e53c.
* test new flash_attn test
* rebase and fix to disable subgrou matrices when max_kv_tile == 0
* delete log output
* Add i32 support to cpy and enables the all ops test
* restore the non target ci tests
* comment out of TODO of build-cpu.yml
* fix format
* Switch ROCm from 7.2.1 to 7.14
ROCm 7.14 is the first production release using TheRock build system.
It can be installed using multi-arch deliverables from wheels, debs,
rpms, tarballs or runfiles.
Adjust ROCm targets for Linux and Windows to use this instead.
* ci: switch all other Windows ROCm jobs to ROCm 7.14 wheels
Move the shared windows-setup-rocm composite action from the HIP SDK PRO
Edition installer to the multi-arch ROCm wheels (rocm[libraries,devel]).
The wheel-install logic that previously lived inline in release.yml is now
in the shared action, and both build-cache.yml and release.yml call it.
Also migrate the build-cuda-windows.yml hip job to the same wheel-based
layout (cache path/key, rocm-sdk environment setup, llvm/bin compiler
paths) so it keeps working after the action's contract changed; drop its
now-unused ROCm 7.2.1 rocWMMA download and stale include path.
* Enable backend sampling with token speculation
* Clamp the mask sum before converting it into the sampled index
* Add a numeric context parameter declaring the maximum outputs one sequence
* More fixes
* Don't reuse memory for output views.
* Match dist between CPU and GPU
* Fix CPU and backend sampling mismatches
* Simpify some of the changes
* Fix tests on Vulkan
* More test fixes
* Rebase changes
* Rebase and address review comments
* Address review comments
* Address review comments
* Update src/llama-sampler.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a
valid input, but the CUDA and Metal roll kernels index by ne alone and
never read the nb strides. A non-contiguous src therefore produced
silently wrong results. Neither backend declared a contiguity
requirement in supports_op, so the scheduler did not fall back to the
CPU implementation, which does handle strides correctly.
Add the requirement to both backends, matching the existing
GGML_OP_ROPE guard, and add a permuted test_roll case.
* ui: split the markdown rendering setting per surface
User content and thinking get their own toggle again, so turning off
markdown for a message leaves reasoning blocks formatted. Both default
to markdown. A stored renderContentAsRawText unfolds onto the user key
and is dropped from the config.
File mentions render as badges in the raw text path too, through a
narrow pass over [name](file://path) that leaves everything else
untouched.
* ui: let the rich chat input scroll past its max height
The contenteditable renderer caps its height with max-height but had no
overflow rule, so a long buffer overflowed into the input area wrapper
and got clipped by its overflow-hidden, leaving no way to reach the
bottom of the message. The textarea renderer scrolls natively and was
never affected.
* ui: apply the new lint and format config
* ui: move the render keys unfolding into the migration service
Address review from @allozaur: the settings store no longer rewrites
persisted config on load, the raw text toggle now unfolds onto the
per-surface render keys in migration.service.ts, next to the other
config migrations. The mention scanner flag and the directory path
suffix become named constants.
* server: add an ssh transport to the tools runtime
--tools-runtime ssh:<target> runs the built-in tools on a remote host,
where target is whatever ssh already resolves, a user@host or a config
alias, so no credentials live in llama.cpp.
Only build_argv and upload differ from the docker transport: the remote
shell re-parses the command line, so the argv travels through
shell_quote_join, and files go over scp with the same quoting on the
remote path. Authentication is key-based and the host key must already
be trusted, since the tools run without a console and any prompt would
hang them.
The target is validated before use. The spec can reach us from the
x-tool-runtime header, and a leading dash would turn it into an ssh
option, which is enough to run a command back on the host.
Nothing is created and nothing is reclaimed, so an ssh spec goes
straight to the tool call instead of through the container runtime.
Note that this is remoting rather than isolation: the tools can do
whatever the target account can do, and the isolation is whatever runs
them on the far side.
* server: support podman in the tools runtime
docker and podman expose the same run, exec, cp and inspect verbs with the
same argument order, so a single implementation drives both and the engine
is carried by the spec prefix: podman:<image> and podman-container:<id> sit
next to the docker forms.
tools_io_docker becomes tools_io_container and the runtime spawner becomes
server_tools_container_runtime, both holding the client binary chosen at
parse time. A single parse_container_runtime() resolves every spec, so
adding another engine is one string in the table.
make_tools_io() now rejects the spawning forms. The spec also reaches it
from the x-tool-runtime header, which is client controlled, and only the
runtime that owns a container is allowed to create one: a tool call can
attach to a running container, nothing more.
* ./build/bin/llama-gen-docs
* server: simplify the tools runtime and drop the file copy step
A server_tools_runtime base with one virtual spec() replaces the
container runtime and the bare spec string that ssh needed next to it,
so server_tools is back to a single pointer and neither setup nor the
handler tests which of the two is set.
write_file used to spill its content into a temporary file on the host
and copy it in, because run_subprocess had no way to feed a child. It
now takes an optional stdin payload and creates the parent directory
and the file in a single round trip through a shell in the isolate.
That removes the upload virtual and both implementations: no more
container cp or scp, no second binary on the host, no sftp subsystem on
the target, no predictable temporary in a shared tmp, and none of the
content reaching an argv the remote shell re-parses. It also fixes
write_file over ssh, which never worked: scp speaks sftp and takes the
remote path literally, so quoting it kept the quotes in the file name.
Writing the payload before reading the output relies on the child
draining stdin as it goes, which holds for cat, its only user today.
* ./build/bin/llama-gen-docs
* server: harden the tools runtime against argv injection and a stdin stall
Validate the container id from x-tool-runtime and --tools-runtime the
same way the ssh target already is, so an id shaped like an option
(docker-container:--privileged) is rejected before it reaches the
engine's exec command line instead of running against a hardened
container. Feed the child's stdin after the watchdog is armed, so a
transport that stalls mid-write is terminated at the deadline rather
than blocking the request forever.
Cover both guards and fix the unknown-scheme test, which used ssh: as
its example and now names a real runtime.
* tests: exercise the tools runtime tests on podman as well as docker
Follow-up #26507. The container runtime drives docker and podman
through one implementation, so parametrize the availability helper,
the container fixture and the attach test on the engine, and cover
both engine prefixes in the container id injection test. Each engine
skips on its own when it is not installed.
The spawn cleanup test stays docker only: it recovers the spawned id
from the container hostname, which docker sets to the short id and
podman rootless does not guarantee. Podman keeps its coverage through
the attach path.
* server: release the container handle before respawning
Follow-up #26507. create() writes over the handle it is given, so a
respawn after the container died on its own leaked the pipes and the
process handle of the previous one.
* server: trim the tools runtime comments
* server: read tool output as raw bytes and harden the runtime on Windows
The stdout pipe is read with read() instead of fgets(), so a chunk
can hold any byte, including NUL, and still streams as soon as data
is available. Past the size cap the pipe keeps draining so the child
never blocks on a full pipe. Both pipe fds are forced to binary mode
on Windows, where the CRT defaults them to text mode and translates
line endings in both directions. Stdin is now always closed after
the feed: the child reads a deterministic EOF, and the Windows
docker and ssh clients stop outliving their command on a stdin pipe
that never closes.
The attach form of --tools-runtime has no lifecycle to own, so it
becomes a static target validated once at startup. This removes the
subprocess that ran on every tool call and
serialized calls behind a mutex; a stopped container now surfaces
the engine's own error at exec time.
The cidfile path is passed as UTF-8, matching the encoding the
subprocess layer expects for the CreateProcessW command line, so
the spawn form works from a non-ASCII Windows profile.
The SIGPIPE note in server.cpp now names the tools runtime children
as well as the MCP ones.
* clean up comments
* less pollute global scope
* nits
* tests: name the container image after both engines
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Hongqiang Wang
Georgi Gerganov
michaeltrabalka-tech
Oğuzhan Akkaya
Wang Zhiyu
lhez
ruanslv
Sigbjørn Skjæret
Rafail Giavrimis
0
Aldehir Rojas
Bartowski
Niklas Wenzel
Concedo
ynankani
lnigam
Wagner Bruna
Xuan-Son Nguyen
Tom Tan
uvos
Daniel Bevenius
Ruben Ortlam
Jim Wu
Junmo Kim
Masashi Yoshimura
Gaurav Garg
Mario Limonciello
Hitesh Chopra
Yash Raj Pandey
Pascal