* fit: also take into account n_streams
* server: make the draft context follow the target context
With a non-unified KV cache the target context now holds n_ctx_train
tokens per sequence, while the draft context was still created with
n_ctx = 0 and fell back to n_ctx_train / n_streams per sequence. A slot
filled beyond that point makes the draft batch fail to decode, and the
server answers 500 on the request.
The draft context now takes its size from the target context, so both
hold the same number of tokens per sequence. Contexts that share their
cells with the target no longer need the kv_size override.
The memory reserved for the draft model before fitting is measured at
the largest context the target can take, since the draft context grows
with the target and a fixed byte margin cannot express that.
* fit: take an optional second model into account
Illustrates the alternative discussed on the draft context fix. The
memory of a draft or MTP context is currently handed to the fit as a
fixed byte margin, which cannot express a memory that grows with the
context the fit is still deciding on.
common_fit_params now takes an optional second model that shares the
devices of the main one. Its context follows the main context and its
memory is measured again whenever that context changes, so the reduce
path stays exact instead of conservative. A model that cannot be
measured on its own, such as a shared cell MTP context, is skipped with
a warning and the main model is fitted alone.
This drops the reservation block in the server, which no longer has to
probe the trained context size of the target to guess an upper bound.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* Revert "ci : disable ubuntu-rocm (#26969)"
This reverts commit 9558fa44c9.
* ci: set ccache compiler_check=content for ROCm build
The ROCm toolchain is pip-installed fresh on every run, so the clang binary's
mtime changes each time. With ccache's default compiler_check=mtime that
invalidates the whole cache and warm builds only reached ~70% hits. Hash the
compiler contents instead so the cache survives toolchain reinstalls.
* Update ccache size to 1GB
We're waivering with so many architectures built, we need a bigger
ccache limit.
* merge fix
---------
Co-authored-by: Jim Wu <ywu@xilinx.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ci : add older, min and dry-run options to ccache-clear
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : add note about not wrapping lines in PR descriptions
[no ci]
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* refactor: add new module for backend-specific code
* replace ifdefs on rwkv_v3
* replace ifdefs on llama_v2
* replace ifdefs on llama_v3
* replace ifdefs on gpt2_v3
* replace ifdefs on gptj_v3
* replace ifdefs on mpt_v3
* replace ifdefs on neox_v3
* replace ifdefs on whisper
* adjust build for kcpp_backend
Mostly a:
sed 's/ gpttype_adapter\([^ ]*\)\.o / gpttype_adapter_default.o kcpp_backend\1.o /g' Makefile
and include variant objects for kcpp_backend.
* opencl: fold the gpt-oss MoE bias adds into swiglu_oai
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_GLU=0.
* opencl: fold the MoE down-projection bias into the combine
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_COMBINE=0.
The 'Create release' step had no id, so steps.create_release.outputs.id
resolved to an empty string in the 'Upload nightly-tag.txt' step. The
uploadReleaseAsset call then hit /releases//assets and failed with HTTP
404 (Unhandled error: HttpError), e.g. run 32513839499.
Add id: create_release to the step; the action already exposes the id
output.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : run ccache-clear as the last step of release jobs
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* update disabled job too to force rebase
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Similar to ggml/scripts/release.sh: validates repo state, creates a
release candidate branch (llama-rc-vX.Y.Z), bumps LLAMA_VERSION_* in
CMakeLists.txt and commits the version bump.
Usage: ./scripts/release.sh [major|minor|patch] [--dry-run]
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui : rework the settings registry into ordered raw-data sections
SETTINGS_REGISTRY becomes an ordered SettingsSectionEntry[] array; the
array order is the sidebar display order. Section titles, color mode
options and title radio options are declared inline in their section or
entry. Entries gain showInUi; MCP servers, the system-message toggle and
the title LLM flag become hidden entries of their own section.
Derived values (config defaults, help info, chat sections, numeric field
lists, syncable parameters) are still derived here; they move to their
actual consumers in follow-up commits.
* ui : extract settings localStorage persistence into SettingsService
Stateless load/save of the settings config and user-override keys, plus the
legacy theme key migration. Business logic (default merging, mobile
sendOnEnter default, applying the migrated theme) stays in the store.
* ui : move the settings exit route into ROUTES
SETTINGS_FALLBACK_EXIT_ROUTE is just a route, so it lives with the other
routes as ROUTES.SETTINGS_EXIT.
* ui : derive the syncable parameter list in the parameter sync service
The syncable parameter mapping is only consumed by the sync service, so
derive it there from the registry instead of exporting it from the
constants file.
* ui : restore isPrivate for API key masking
* ui : clean up settings registry and router fetch guard
Drop the per-entry section field (duplicates the parent slug and is
never read) and guard the router model fetch on fields?.length so the
Tools/Import-Export pages with empty fields are excluded again.
Assisted-by: pi
* ui : merge sampling and penalties settings into one section
Assisted-by: pi
As agreed in ggml discussion #1579, the official semver releases now
include a nightly-tag.txt asset containing the tag of the corresponding
nightly release (e.g. b10485). The Web UI assets are published to the
HF bucket under the nightly tag, so this makes them discoverable for
each official release.
- make-release-desc.sh: expose the resolved nightly tag as a
nightly_tag output
- make-release.yml: create nightly-tag.txt from that tag, upload it
as a release asset (skipped on dry-run), mention it in the release
body and in the dry-run summary
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add a reordered Q2_K MMVQ kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q2_K ESIMD kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Resolve the TODO in test_flash_attn_ext: the branch that creates V as a
sub-view of K (MLA-based models) was hardcoded for the 576/512 head shapes.
Add a v_is_view_of_k test case parameter (default false) and select the
sub-view branch on it; the existing 576/512 (DeepSeek MLA) cases now pass it
explicitly, so the test coverage is unchanged.
Also add more V-is-sub-view-of-K cases: the 320/256 (Mistral4 MLA) and
192/128 head shapes, and full views with equal head sizes (128/128 F16,
64/64 q8_0).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q5_K ESIMD kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Remove redundant copyright notice
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* opencl: keep the vocab-scale K-quant lm_head on the CPU on the Adreno A7X
* opencl: revise comments
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Concedo
Xuan-Son Nguyen
Mario Limonciello
Tiwei Bie
Georgi Gerganov
Kartik Sirohi
Shahir BIn Zulfiker
Sigbjørn Skjæret
Wagner Bruna
Hongqiang Wang
Niklas Wenzel
Aleksander Grygier
Charles Xu
Todd Malsbary
HumerousGorgon
Ian Faust
Neo Zhang
Chris Danis