* doc: document MCP stdio servers and CORS defaults in the server README
The MCP arguments were listed but nothing explained what an MCP server
is or how to declare one. Cover the stdio transport, the config keys,
the tool naming, and add a POSIX shell echo server as a minimal
example.
Also document the CORS behavior: the default reflected origin, the
switch to localhost once tools are enabled, and the recommended setting
per deployment.
* doc: drop the inline MCP shell example from the server README
The example parsed JSON-RPC by hand and sat in a page people copy paste
from, into servers spawned with the privileges of llama-server. Point to
the specification instead.
Link the pull request that introduced the feature, and keep a short
mcp.json snippet so the table of configuration keys has a declaration to
refer to.
Add a "Create and push git tag" step to the release job, right before
the "Create release" step. The tag is created with git tag and pushed
with the deploy key already configured by the Clone step, instead of
relying on the Releases API (action-create-release) to create it as a
side effect.
The tag is lightweight, matching all existing b<number> release tags.
The step is idempotent: if the tag already exists (e.g. on a re-run),
creation and push are skipped.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui: move get_datetime tool to frontend
* clarify docs
* server: drop the now unused ctime include
strftime() and gmtime_r() were the only users, both went away with the
get_datetime tool. Also make the renderer's catch inert: the browser
executor always emits JSON, so a non-JSON result is no longer a date to
display.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ci : parallelize platform builds in build-xcframework.sh
The ios-xcode release job builds 7 platform/simulator configurations
sequentially, each with -j $(nproc). Run them with at most 3 concurrent
builds (the release runner has 3 cores), splitting the cores between the
builds (-j 1 each on the runner), so total CPU pressure is unchanged while
the build phase runs about 2.3x faster.
- convert the 7 build blocks into functions (flags unchanged)
- add a run_builds_parallel pool: 3-slot sliding window, per-build logs,
dumps the failing log and aborts on error (background job failures do
not trigger set -e)
- queue the 2-arch builds first so the slower builds occupy the slots early
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : add guideline for comments
* cont : disable 5/7 builds
* ci : make build-xcframework.sh builds configurable via CLI args
The script now takes an optional list of builds to run
(ios-sim ios-device macos visionos visionos-sim tvos-sim tvos-device);
with no arguments it builds all of them, as before. The per-build
lists for the build pool, framework setup, static library combining
and xcframework creation are now driven by a single build_spec
lookup instead of four hardcoded (partially commented-out) lists.
release.yml builds only macos and ios-device to cut the build time.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* dspark: support speculators-format checkpoints (SpecForge exports)
Speculators-format DSpark drafts (e.g. SpecForge exports for the
Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in
three ways:
- the config nests the backbone hparams under transformer_layer_config
and gives the extract layers as aux_hidden_state_layer_ids
- the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus
token, not a prediction slot. Written as dflash.bonus_anchor; such
drafts build the block and read the mask positions exactly like
DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence
sampling comes from DSpark
- the draft output vocab may be reduced (draft_vocab_size < vocab_size)
with a d2t remap table. The converter expands lm_head/markov_w2 back
to the full vocab and synthesizes an lm_head bias of -1e9 on the rows
the draft cannot produce, so the runtime needs no d2t remapping. Such
drafts ship their own (now optional) token_embd/output tensors instead
of sharing the target's
Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical
with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the speculators draft class into DSparkModel
One class now covers every DSpark variant. What used to pick the class is
a single flag, because the arch name turns out to be the only thing that
separates the two families: SpecForge also exports a flat schema that
carries no speculators_* fields yet still uses the 1+N bonus-anchor block,
so keying on those fields would silently mis-read its drafts.
Also rename i0 to i_first_pred in the draft read loop and the Markov head,
and give the head a real bonus_anchor bool instead of testing i0 > 0.
Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts
produces byte-identical GGUFs. The one behaviour change is that the
markov_head_type check now also covers the DeepSpec checkpoints, which
previously skipped it.
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Opus 5
* dspark: address review comments
- rename bonus_anchor to sample_from_anchor (GGUF key and code), matching
the checkpoint config field; absent key still means anchor-first
- rework the reduced draft vocab to match EAGLE3: d2t is written as I64
absolute target ids and the logits are scattered at runtime, instead of
expanding lm_head/markov_w2 and synthesizing an output bias at conversion
- move the t2d skip to modify_tensors, like EAGLE3
- drop _is_specforge: the arch name only picks the sample_from_anchor
default, embed/lm_head sharing is decided by the draft vocab size
- deduplicate the tok_embd create_tensor left behind by the rebase
Verified with the RedHat gemma-4-31b speculator draft: greedy output is
byte-identical with and without the draft; acceptance 0.26 (n_max 7).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the sample_from_anchor read into the block_size block
* dspark: fix flake8 continuation indent
* clean up
* dspark: key the sample_from_anchor default off the export format
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: drop t2d in filter_tensors
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: map model.lm_head instead of bypassing the dflash prefix
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
---------
Co-authored-by: desovo7 <942845546@qq.com>
Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>
* ci : allow make-release to target a specific commit
The make-release workflow now accepts an optional 'commit' input. When
set, that commit is checked out and the release checks verify that it
belongs to the branch selected in the "Run workflow" dialog and is not
older than 3 days from the branch tip. The check is part of
make-release-checks.sh (driven by the RELEASE_BRANCH env), so it follows
the same dry-run semantics as the other checks.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : scan latest 100 relase workflow runs
* cont : do not check manually for release.yml success
* Adding support for bailingmoe3
* Adds speculative decoding support
* Make BailingMoE3 safe gate metadata optional
* bailingmoe3: apply trained SwiGLU clamps
* common: fix Bailing V3 tool argument parsing
* llama-model-saver, instantiate float vector metadata writer
* bailingmoe3: support Q-LoRA (Ling-3.0-tiny)
Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current
implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256
and routes Q through a LoRA bottleneck instead:
q_a_proj -> q_a_layernorm -> q_b_proj
Conversion therefore failed with:
ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight'
Add the missing path, mirroring the existing deepseek2 MLA implementation:
* constants.py - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3
* tensor_mapping.py - map model.layers.{bid}.attention.q_{a,b}_proj and
q_a_layernorm
* conversion - emit attention.q_lora_rank when the config has it
* bailingmoe3.cpp - read n_lora_q; create the Q-LoRA tensors and build Q
through the bottleneck when q_lora_rank > 0
Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no
q_lora_rank, the converter only emits the key when present, hparams.n_lora_q
defaults to 0, and get_key(..., required=false) leaves the target untouched when
the key is absent - so flash keeps taking the existing direct-Q branch.
The LoRA path produces the same shape as the direct projection, so the
nope/rope split, RoPE application and wk_b absorption downstream are unchanged.
* small mtp change
* bailingmoe3: support separate MTP GGUF and Q-LoRA MTP
* gguf: remove duplicate add_kda_gate_lower_bound definition
---------
Co-authored-by: bloomer <bloomer@booper.brushtail.me>
Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com>
The log output does not append a newline, so the warning ran into the
next line printed on stdout, corrupting the benchmark table header.
Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Adjusts the thread/block count to be proportional to the size
of the quant, reducing under/over subscription.
Largest perf improvement is the q4_0 -> f32 path, with, on
a Arc 70, throughput goes from 20.21 GB/s to 158.19 GB/s
The rest of the quants are flat in performance uplift.
* ui: mask API Key field in settings and error splash to stop browser autofill
* ui: set autocomplete=new-password on private fields
The password input type makes browsers offer to save the API key
in the password manager and autofill saved site credentials into
the field. The new-password autocomplete value disables both.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* model: add Kimi-K3 text model
Hybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five
things that architecture does not have:
1. cross-layer residual attention (attn_res_block_size)
2. latent MoE (routed experts run at n_expert_latent)
3. situ activation (replaces SwiGLU everywhere)
4. MLA output gate (sigmoid gate before o_proj)
5. full-rank KDA gate (single ssm_g instead of ssm_g_a/ssm_g_b)
K3's text_config reports KimiLinearForCausalLM - the older 48B architecture -
so get_model_architecture routes on the top-level name instead.
The KDA decay gate has two forms, selected by linear_attn_config's
gate_lower_bound. It is not a clamp: when set it swaps the activation entirely
(fla/ops/kda/gate.py), from -exp(A_log)*softplus(x) to
lower_bound*sigmoid(exp(A_log)*x). K3 sets it to -5.0; kimi-linear leaves it
unset, so that path is unchanged.
Cross-layer residuals reuse ggml_dsv4_hc_pre for the weighted sum. That op is
CPU + CUDA only, so Metal/Vulkan will fall back per-node until those kernels
exist.
The routed experts ship as compressed-tensors "mxfp4-pack-quantized". That is
bit-compatible with ggml's MXFP4 - same E2M1 code assignment, same E8M0 scale
byte, only the nibble positions within a block differ - so they are repacked
rather than dequantized, losslessly and without a ~5.5 TB bf16 round-trip.
The repack is built lazily because gguf_writer holds every added tensor until
the final write. DeepSeek-V4 was already doing the identical bit-shuffling, so
it now shares the helper.
Verified against Moonshot's own code path (transformers + fla's Triton KDA
kernels) on a tiny model exercising every K3-specific feature. Final-position
logits vs the fp32 reference: 6.7e-05 rel / corr 1.00000000 for both the
chunked and the recurrent delta-net path. MXFP4 blocks dequantize to the source
weights with 0.0e+00 error.
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* model: fix ty errors in the Kimi-K3 converter
- `_res_parts` buffers (kind, tensor) pairs, not bare tensors
- `get_tensors` must return an Iterator, matching ModelBase
- LazyBase's `func` takes one argument, so pass the expert loaders through
`args` instead of the closure
- borrowing KimiLinearModel.set_vocab from an unrelated TextModel is
deliberate and safe, but not expressible in the signature
No behaviour change: the MXFP4 repack still dequantizes to the source weights
with 0.0e+00 error and end-to-end logits are unchanged (8.386e-03 rel,
corr 0.99996630).
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update conversion/kimi_k3.py
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
* Increase LLAMA_MAX_EXPERTS from 512 to 1024
* tests : support for Kimi K3 in archs test
* chat : add Kimi K3 chat format (reasoning, content, typed tool calls)
K3's assistant output is an XTML-ish tagged format built by the template's
open_tag/close_tag macros. Two properties break generic parsing:
1. The generation prompt ends with open_tag('think'), so the completion
starts inside the think section with no opening marker in the output
(thinking_forced_open).
2. Only <|open|>/<|close|>/<|sep|>/<|end_of_msg|> are special tokens; tag
names ("think", "response", "message") are ordinary text tokens.
Adds common_chat_params_init_kimi_k3 (PEG_NATIVE) with detection on the
marker trio, reasoning extraction, response unwrapping, and tool-call
parsing of the tools/call/argument tag structure with argument types
taken from the tool schema. Includes the K3 chat template fixture and 9
test-chat cases derived from real generations of the full 2.8T model.
Verified end-to-end against Kimi-K3-Q2_K (GrEarl/Kimi-K3-GGUF) on 8x B200:
content, reasoning_content, streaming deltas, and tool_calls all correct;
finish_reason stop/tool_calls as appropriate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chat : add message_delimiters for Kimi K3
Per-role message-start markers for token-level span splitting. User and
assistant messages carry only the role attribute, so their full opener
(through <|sep|>) is used; system and tool messages continue with more
attributes (type=/tool=/index=), so those delimiters stop after the
role's closing quote. Verified against the K3 tiktoken vocabulary that
the closing quote is always a standalone token across all attribute
variants, so the token-level prefix match stays exact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: apply nits from @ngxson and text fixes from @danielhanchen
* tests : added missing hyperparameters and tensors for Kimi K3 in test-llama-archs
* chore : move overly verbose header file comments to Kimi K3 source file
* tests : re-enabled KIMI_K3 in test-llama-archs for WebGPU backend
* model-saver : emit kda_gate_lower_bound for Kimi K3
Quick fix. The Kimi K3 loader reads kda_gate_lower_bound and gates a graph branch on it (it scales the KDA gate when the bound is above -INFINITY), but the model
saver never wrote the key, so a save->load roundtrip silently dropped it back to the -INFINITY default and changed the model's output. The real K3 config sets gate_lower_bound = -5.0.
I propose to emit it from the saver, and set it to -5.0 in the test-llama-archs K3 case so the roundtrip check exercises it (the roundtrip fails without the saver line).
* Refactor conditional for model architecture check
* tests : re-enabled (again) KIMI_K3 and MINIMAX_M3 in test-llama-archs for WebGPU backend
* fix code comments
* add template on conversion
* move repack_mxfp4_blocks to model base
* nits
* add_value_length
* optimize res_stack construction
* nits
---------
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Deepankar Singh <singh.deepankar39@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Caleb DeLeeuw <caleb.deleeuw@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common: support --models-dir loading MTP assistant models
* common: preset: check for MTP models with strict prefix
* common: preset: Take advantage of PR #27005
* handle other draft types
* drop eagle3
* clean up
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Replace the deprecated --mmap, --no-mmap, --mlock, and --direct-io flags with
the unified --load-mode argument across scripts, examples, and documentation.
Internal warning message and env var docs updated accordingly.
Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
* llama : support for MiniMax-Text-01 model
* chore : renames to match the other MiniMax models
* model : add logits mask as MiniMax-Text-01 embeddings tensor has zero-valued embeddings for tokens >= 200032 that produce zero logits disrupting the token sampling process
* llama : replace hardcoded conditions with hparams.is_recr()
* model : used build_rs() for recurrent state management
* chore : code cleanup
* model : optimized MiniMax-Text-01 by removing the state tranpose operations
* chore : removed unnecessary ggml_cont() in MiniMax-Text-01 implementation
* llama : add generic logits mask graph input
* model : permuted diag_decay dimensions to avoid doing it inside MiniMax-Text-01 graph
* chore : code cleanup
* chore : code cleanup
* model : use token positions when calculating MiniMax-Text-01 decay tensors
* convert : add support for MiniMaxM1ForCausalLM as it seems to be the same as MiniMaxText01ForCausalLM
* chat : add jinja template for MiniMax-M1
Co-authored-by: QscQ <qscqesze@gmail.com>
* chore : code cleanup
* tests : MINIMAX_01-related fixes
* chore : silence Python lint errors
* vocab : remove unnecessary vocab type
* convert : update MiniMaxText01Model conversion to use yield when modifying tensors
* convert : suppress tokens with zero-valued embeddings during MiniMax-Text-01 conversion
* llama : removed logits mask - no longer necessary as token suppression is used instead
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : override non-working built-in chat template during conversion
* tests : skip arch MINIMAX_01 tests for WebGPU backend (it breaks again)
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: QscQ <qscqesze@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* chat: add reasoning_effort to common_chat_templates_inputs
Store OpenAI Chat Completions reasoning_effort and make it
available to jinja templates (with model specific translations
where required).
Assisted-by: llama.cpp:Muse-Glimmer-30B
* server : fixup reading reasoning effort from body
server_chat_convert_responses_to_chatcmpl already handles conversion of
Responses API reasoning.effort to reasoning_effort
* chat : expose reasoning effort
Assisted-by: Claude Opus 5
* chat : add reasoning_effort to generation_params
Assisted-by: Claude Opus 5
* chat : move reasoning_effort next to enable_thinking
Assisted-by: Claude Opus 5
* cont : mirror preserve_reasoning
* cont : pass context through analyze function
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
* Initial changes for Recurrent state rollback for nemotron for cpu and cuda
* Removing CPU RS rollback. Will enable it in subsequent PRs
* addition of test case
* Removing assert and calling runtime API to check if op is supported
* removing extra API and updating the call sites for K
* replace static cuda detection to runtime fused_op api
* address review comments and fallback when SSM rollback not supprted
* Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda
* removing memory manipulation as rs rollback is now supported in CPU
* removing the static probe which is not needed now
* correcting the format
* address review comments
* enabling test for all the backends, unsupported backends will fallback to CPU
* Apply suggestions from code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported
* Support K > 1 in ssm_scan for all backends
* Fix CI Issues
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Xuan Son Nguyen
Pascal
Sigbjørn Skjæret
Tarek Dakhran
Mario Limonciello
Georgi Gerganov
王金旭
Toby
Fathi Boudra
Titaniumtown
Neo Zhang
Daniel Bevenius
fairydreaming
Oğuzhan Akkaya
Cristiano Pinto
Gautam0507
Piotr Wilkin (ilintar)
Alessandro de Oliveira Faria (A.K.A.CABELO)
Eric Zhang
Jiang, Fish
Masato Nakasaka
Hemanth Battu
0
Andy Williams
lnigam