* speculative : add common_speculative_n_max helper function
Extract the speculative max-draft-size logic from server_n_outputs_max
into a reusable common_speculative_n_max() function in common/speculative.
Assisted-by: llama.cpp:local pi
* cont : draft context always has n_parallel outputs
* llama : log n_outputs_max
* speculative : remove draft-simple auto-enable
* ci : enable server tests on PRs
* nix: add nix-nodejs facilities to build Web UI
Build the Web UI locally using standard Nix systems for building NodeJS
packages.
- Create derivation for the web UI
- npm dependencies are imported via buildNodeModules. Does not require
setting any shasum.
- Copy build artifacts to the correct folders.
- Prevents having to download from huggingface.co
Fixes#23067
* nix: simplify webui derivation using LLAMA_UI_OUT_DIR
- Move npm build to installPhase with LLAMA_UI_OUT_DIR=$out to write
output directly to the Nix store
- Copy built assets to tools/ui/dist (source tree) instead of
build/tools/ui/dist so CMake's copy_src_dist() finds them
* opencl: add general q5_0 support
* opencl: add general q5_1 support
* opencl: support non-uniform workgrp size
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* llama: save more VRAM by reserving n_outputs == n_seqs when possible
* add n_outputs_per_seq
* move n_outputs_max to server-context
* change ubatch to batch everywhere
Drops the hardcoded f32 GLU kernels in favor of a single template. We now load/store in the native tensor type (half or float) to save memory bandwidth, but keep the actual ALU compute in float to avoid exploding math in geglu/swiglu. Also opened up the dispatch gate to allow f16 inputs.
* vulkan: don't hold the device mutex while compiling pipelines
We need to hold a lock while we traverse all pipelines and lazily initialize
them, but we don't need to hold it while the pipeline is being compiled. And
it doesn't need to be the same lock as the device mutex. We call load_shaders
each time a pipeline is needed, so we only need to compile that one pipeline
(and, for example, don't want to end up compiling a pipeline that another
thread should be compiling).
* remove 'needed'
Q2_K/Q3_K/Q6_K do much better when using MMVQ on Intel BMG even
though they're only 2-byte aligned, and Q3_K still wins on
NVIDIA as well.
mesa isn't all that great at coalescing back-to-back loads from
alternating arrays, so we force it instead. Further, we can do
subtraction directly on a full int32_t rather than an i8vec4
with bit twiddling because the high bit is always free to start.
On Intel BMG on mesa, the switch to MMVQ provides an immediate
~57% perf increase in tg128 for unsloth/Qwen3.5-9B-GGUF:Q3_K and
~78% perf increase in tg128 for unsloth/Qwen3.5-9B-GGUF:Q6_K.
The futher switch to block loads leads to a ~24% perf increase in
tg128 for unsloth/Qwen3.5-9B-GGUF:Q3_K and a ~48% perf increase in
tg128 for unsloth/Qwen3.5-9B-GGUF:Q6_K.
Finally, Xe2 wins on MMVQ even for small k, so we take the NVIDIA
override for K quants on Xe2 as well.
Fixes: https://github.com/ggml-org/llama.cpp/pull/23927#discussion_r3332213086
The cpu-x64-high-perf job was missing the Linux label in its runs-on
specification, causing the runner to not be discovered. All other
self-hosted Linux jobs include this label.
Assisted-by: llama.cpp:local pi
* add to support Q1_0, NVFP4, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ1_S, IQ1_M, IQ3_S, IQ4_NL, IQ4_XS, I32, MXFP4, Q2_K, Q3_K, Q5_K, and Q6_K in GET_ROWS OP
* correct the link
* remove redundant apple job
openvino gpu and cpu test can share the same build and machine
Update build-rpc.yml
Update build-openvino.yml
cpu any doesnt make sense as we have an arm job already, so do high perf on both x86 and arm
remove duplicate x86 vulkan
combine backend sampling
Update server.yml
run server on arm as windows is x86
* emdawn on one machine only
* fix openvino, remove cpu tag as we dont have many x64 machines with that tag
* webui: add custom CSS injection via config
register a customCSS setting in the Developer section under Custom JSON,
syncable so it rides the existing ui-config pass through. inject the value
into a single style element in the head, reactive on the setting. lets an
operator theme a prebuilt binary through --ui-config without rebuilding,
and lets a user set it from the settings panel.
* ui: address review from @niutech and @allozaur, rename custom JSON key and CSS field
* ui: address review from @allozaur, move custom CSS injection to a style tag in svelte:head
* ui: inject custom CSS through a svelte action instead of a bound element
move the textContent write into a use: action on the head style node.
the action is the idiomatic way to touch a node, so the no-dom-manipulating
lint rule is satisfied without a disable. value stays text through
textContent, never parsed as HTML.
* Update tools/ui/src/lib/constants/settings-keys.ts
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* ui: address review from @allozaur, rename custom config key to customJson with migration
rename the custom config key to customJson across the type, the chat
request builder, the settings save check and the custom tools reader,
keeping the custom API param name unchanged. add a non destructive
migration that copies the legacy custom key to customJson at startup.
only render the head style tag when custom CSS is set.
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* Support `-fa auto` in llama-bench
Make the default value of `-ngl` -1, similar to other tools.
Update README with latest usage and examples
* Address review comments
* ci : disable libcommon build from xcframework
* ocd : fix name
* ci : ios-xcode change to macos-26
* cont : pin xcode
* cont : pin xcode to minor version
Georgi Gerganov
Christian Hoener zu Siederdissen
shaofeiqi
Adrien Gallouët
Aman Gupta
Shrivas Shankar
Jeff Bolz
Winston Ma
o7si
Concedo
Johannes Gäßler
Junwon Hwang
Matt Corallo
Aldehir Rojas
Neo Zhang
Eve
Eric Zhang
Vladislav
Ruben Ortlam
Wagner Bruna
Pascal
Gaurav Garg
lhez
Xuan-Son Nguyen