* sd: sync with master-801-9cfe2af
* sd: sync with master-802-e92e86f
* sd: sync with master-805-e31a86c
* sd: sync with master-810-db99efd
* sd: sync with master-812-ea7f0c8
* sd: minimax-h3 support
In autoswap mode, a POST to /v1/completions or /v1/chat/completions
carrying a `model` name that is not an entry in the admin dir set
`model_switch_pass = True` before checking the whitelist. No swap was
performed, but the flag suppressed the request-type dispatch below it,
so the text model was never loaded on demand.
The same requests without a `model` field, and every other model type
(stt/tts/embed/music/image), skip that branch entirely and load fine --
which is why only chat was affected, and why sending one model-less
request worked around it. It also recurs after --adminunloadtimeout
fires, since the "nomodel" state is recovered from by that same
dispatch.
Only set the flag on the path that actually issues the reload.
The reasoning budget derived from reasoning_effort never applied to Mistral
models. gpttype_adapter.cpp picks the think delimiters from a switch on the
model architecture, and mistral3 has no case, so it falls back to <think> /
</think>. Those are not vocabulary tokens for Ministral-3, so TokenizeString
returns more than one token each, the expected_start/end_tokens guard clears
all three vectors, and apply_reasoning_budget() returns at its first if.
The parameter is accepted, converted and passed down to the sampler, then
dropped on a size check, with nothing logged.
Adding the mistral3 case arms the budget. [THINK] and [/THINK] are single
vocabulary tokens (ids 34 and 35 on Ministral-3), so the size guard passes.
The thinkformats entry is a separate fix for a separate defect: without it the
thinking block was never split out, so it leaked into content with its [THINK]
marker still in it, instead of going to reasoning_content.
Measured on Ministral-3-14B-Reasoning-2512 (IQ4_XS, ctx 8192, --jinja), 5 real
prompts x 3 samples per cell, max_tokens 3000 (so a 750-token budget at "low"):
reasoning_effort thinking words before thinking words after
none 311 - 2314 7 (the forced-close phrase)
low 340 - 2255 521 - 574
Forced closes: 0/15 before, 14/15 after at "low" and 15/15 at "none". Three
samples per cell because this model's variance at temperature 0.7 spans a
factor of 4 on an identical payload — a single sample per cell cannot tell an
effect from noise.
No regression on a non-reasoning mistral3 model: Ministral-3-8B-Instruct with
reasoning_effort "low" returns finish_reason "stop", a normal answer and zero
forced closes, since apply_reasoning_budget() bails out when the start marker
never appears.
This commit contains a suggestion to reduce some code duplication in
common_speculative_init when adding the enabled speculative decoding
configurations.
No tests were added but the existing server tests still passes with this
change:
```console
$ ./tests.sh unit/test_speculative.py -v -x
```
The Python GGUF reader lacked two guards the C++ loader has:
- n_dims read as uint32 with no GGML_MAX_DIMS bound -> crafted file with
huge n_dims triggers oversized memmap read / OOM.
- np.prod(dims) on uint64 wraps silently -> a crafted dims triple can
overflow to a tiny element count, passing an undersized read through.
Add a GGML_MAX_DIMS check and compute the element count with Python ints.
Fixes#25378
bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.
Applies the same fix to chatglm.py.
* sycl: parallelize the non-contiguous concat kernel
Launch geometry only: the non-contiguous concat kernel launched a single-lane
work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one.
SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`.
llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV),
on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)
* sycl: cap non-contiguous concat block at ne0
* sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)
* Revert "sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)"
This reverts commit 2709909e790d298cc0746950caff16ad8ecf9d6f.
* sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches
Extends the oneDNN SDPA path (PR #25222) to handle non-F16 KV caches by
dequantizing or converting K/V to dense FP16 on-device before feeding
them into the SDPA graph. The fused systolic kernel then runs identically
to the native FP16 path.
Supported KV types:
- Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl
- F32: cont_to_f16_sycl<float>
- BF16 and IQ types are excluded (no conversion kernel available)
Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only).
F16 KV runs at any length (existing behavior).
Also includes the stream sync fix (stream->wait_and_throw() unconditional,
PR #25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are
always dequantized to separate buffers).
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* common/chat: update DeepSeek V4 templates
Align the DeepSeek V4 templates with the official encoders while keeping parser behavior out of this change.
- Default drop_thinking for DeepSeek V4 history so prior thinking is omitted unless preserve_reasoning is requested or tools are present.
- Add structured output response-format instructions to the V4 templates and pass the schema into template rendering.
- Add a separate Flash 0731 template for the updated high and max reasoning effort mapping.
- Cover reasoning effort, drop_thinking, structured output prompts, preserved reasoning, continuations, and empty tool arguments in template rendering tests.
Official references:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.pyhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py
Assisted-by: Codex
* Fix deepseek v4 0731 template selection
* remove unneeded lower normalization
* Fix DSML parser to consume the tool call separator
* address aldehir requests
* address aldehir comment
* tests: add model resolution test on synthetic repo listings
Include download.cpp and arg.cpp inside a namespace with hf_cache
monkey patched to serve hardcoded listings, so the resolution and the
model handler assembly are tested end-to-end through the real CLI
parsing, without modifying the tested code and without network access.
Covers the primary, shard, mmproj, sidecar and preset resolution on
layouts mimicking real vendor conventions, replays every case on
permutations of the listing to assert determinism, and asserts the
final wired paths for the spec type auto-selection, the -md precedence
and the fallback suppression.
* tests: keep model resolution checks active and let the handler resolve
Replace assert with a REQUIRE macro alive in Release builds, key the
fake hf_cache by repo id so the real handler init resolves every plan
itself, check the exact shard sets, restrict the permutation exception
to the order dependent picks, and cover dflash and eagle3.
* tests: fix model resolution build on fatal warnings CI and Windows
The namespaced copy of the sources leaves many static functions unused
in this TU, exempt it from the unused warnings. Pre-include the
windows headers so arg.cpp does not pull them inside the namespace.
Declare the renamed copies of the download.h functions, verbatim from
the header and renamed in sync by the macros, so missing declaration
and missing prototype warnings are satisfied on every toolchain.
* tests: fix winsock inclusion order for the model resolution test
WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does
not redefine the socket types afterwards.
* tests: link cpp-httplib to the model resolution test
The test compiles its own copy of download.cpp, which calls httplib
directly, and the private link of llama-common does not propagate the
symbols under lld-link.
* common_http_client
* common: finish the http client wrapper
Add the virtual Head, Get and Post methods and the passthrough
setters to the common_http_client skeleton, move follow_location
into the constructor, expose the underlying client for the ranged
pull path, and rename the missed common_http_client_init call sites.
* tests: rewrite model resolution on the http client stub
Replace the namespace inclusion of the sources by a plain TU: the
common_http_client factory returns a stub serving hardcoded HF API
responses, so the real hf_cache parsing, resolution and CLI handler
run against synthetic listings in an isolated cache directory.
Failures print the named case, the reordering and the actual versus
expected values, the assembly cases use the full command line as
context, and the empty result cases are checked once to keep the
logs short.
* tests: fix the model resolution on Windows and the builds without TLS
Assert the exact expected paths composed like the cache does instead of
suffix matching on forward slashes, set the environment portably, and
serve the stub through an http endpoint so the builds rejecting the
https scheme still reach it. Pause the log so the negative cases can be
replayed on every reordering.
* tests: make the model resolution failures self explanatory
Resume the paused log before the failure report so the CI shows why
the tested code bailed, and format the stub oids portably.
* common: hold the http client factory behind exported functions
The factory was an inline variable, and the Windows shared builds
export functions but not data, so the executable and the DLL each had
their own instance: the stub installed by the test was invisible to
the library, which reached for the real endpoint and resolved nothing.
Route the creation through functions compiled into the library and
format the stub oids portably.
* common: add the http client factory source missed in the previous commit
* common: typedef the http client factory callback
Address review from @ngxson
* tests: serve the model resolution repos over the loopback
Replace the client stub by a real httplib server bound to the
loopback, so no C++ object crosses the module boundary anymore and
the library exercises its own client and transport end to end. The
debug shared build on Windows crashed inside the stubbed path.
* common: add portable common_get_env and common_set_env helpers
Address review from @ngxson
* common: drop the http client factory left without a caller
The loopback server made the stub substitution unnecessary, the client
init builds the real client directly again.
* common: read the model endpoint through the env helpers
* nit: drop the stub leftovers from the model resolution test
* common: align common_set_env and isolate the test cache per run
The POSIX branch now behaves like _putenv_s, so the helper has a single
contract on every platform, and common_get_env already reads an unset and
an empty variable alike.
The model resolution test keys its cache directory on the loopback port,
where two concurrent runs on the same machine used to share one directory
and the initial cleanup of either wiped the other.
* tests: move the model resolution server into main
* tests: support the DSpark sidecar resolution
* common: revert the http client to the plain httplib client
address review from @ngxson
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: add get_info tool
* fix --rpc in docs
* server: harden get_info probe result handling
Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.
* server: simplify get_info result handling
Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* add a direct size condition for `large` weights; the original
dimension condition is insufficient -- q6_K lm_head for gemma-4 E2B
has [1536, 262144], which is big enough to slowdown gemv_noshuffle but
does not satisfy the dimension condition (ne0 >= 2048)
* sampling: enhance penalty handling in common_sampler_init
- Set default value for penalty_last_n based on model context if not specified.
- Ensure penalty_last_n and n_prev are non-negative.
- Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties.
- Implement backend initialization and application logic for penalties, including frequency and presence adjustments.
* tests: add backend penalties sampling tests and utility functions
- Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness.
- Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties.
- Added `test_backend_penalties_sampling` to validate backend penalties with various configurations.
- Enhanced the test suite for better coverage of penalty handling in sampling.
* sampling: add support for top-k penalties in backend sampling
* sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN.
* sampling: enhance penalty comparison tests with masking penalties logic
* add comments on padding
* sampling: add comments on modifications
* add the unit test to cover masked-out token as -INF
* validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values
* refactor: test functions to share logic and be less verbose
* add test to cover case where previously penalized token is not part of candidates
* remove comments
* remove redundant penalty_last_n initialization and validation in common_sampler_init
* add support for penalties in sampler chain with configurable positions
* add validation for penalty parameters and enhance tests for non-finite values
* add context parameter to common_sampler_init and set default for penalty_last_n
* add llama_n_ctx parameter to common_sampler_init for improved sampler initialization
* replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor
* add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64
* add test for backend penalties sampling after top-p with large history window
* remove as unused
* add is_disabled method, tensor logits reshape, add rest review suggestions
* clarify comment
* CUDA: Fix data-races when reusing block_reduce
block_reduce currently doesn't resync after reading from SMEM, causing
potential data-races when reusing SMEM for multiple reductions.
One may consider simply always adding this in block_reduce, but this
comes at a potential perf cost
* double-buffering for single-row softmax
* double-buffering for norm as well
* Add comment
* Add explanatory comment to block_reduce
* Specify need for + do memory barrier only in multi-warp scenario
* Implement review-suggestion from @gaugarg-nv
* llama : MTP support for DeepSeek V3.2
* model : no need to include MTP layers during DeepSeek V3.2 model type discovery
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Concedo
Wagner Bruna
Tai An
Julien BODIN
Georgi Gerganov
Niklas Wenzel
Daniel Bevenius
hcl
Angel Galindo
Pranav Uttarkar
Sigbjørn Skjæret
Caleb DeLeeuw
Eve
Titaniumtown
Ozymandias_EBON
Thiago Padilha
Alessandro de Oliveira Faria (A.K.A.CABELO)
jacekpoplawski
Pascal
Xuan-Son Nguyen
AgoraPete
Hongqiang Wang
timkhronos
fairydreaming
Konrad Moren
Oliver Simons
Masashi Yoshimura