Commit Graph

10514 Commits

Author SHA1 Message Date
Gabe Goodhart 7221e24f57 model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)
* feat(convert): Add conversion for GraniteSWAForCausalLM

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(llama): Add granite_swa support

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(conversion): Add conversion infra for rope_pattern array

NOTE: There is other work also targeting this, so this may be
removed depending on merge order.

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix(conversion): Fix SWA pattern logic and support for non-rope layers

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(conversion): Add support for GraniteMoeSWA

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Add llama_hparams::has_rope and arch constants

NOTE: This shadows the work done for Granite Speech
https://github.com/ggml-org/llama.cpp/pull/25107

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Add support for per-layer rope determination

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Fix failing flake8 for extra newlines

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver

Branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Load MoE params as optional

Branch: GraniteSWAForCausalLM
AI-usage: draft (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Handle MoE params in conversion

branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Remove unnecessary newline

AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Remove unnecessary tensor additions to GRANITE architecture

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Correctly handle naming for ffn gate inp

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Always default hparams.rope_pattern to 1s

This isn't strictly necessary, but it will allow other models to rely on
hparams.has_rope(il) without needting to prepopulate.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Move to has_rope for all granite model architectures

Now that we have a proper hparam for this, it's better to use it and not
require a hacky fallback in the hparam method itself.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: No hacky rope_finetuned fallback in has_rope

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Fully remove rope hparam filling in granitemoe

There are no granitemoe models that use NoPE (it's not actually used in the
layer building below), so this was just dead code.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Save out rope_pattern in model-saver

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Set hparams.rope_finetuned for round trip

Since the value is _read_ from rope_finetuned, we need to persist it when
the model is saved with the saver.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Code review cleanup

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* refactor: Keep gate/up fused for MoE path

Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Skip GRANITE_SWA in model saver

https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651

Keeping is_swa_impl in the saver can break other models.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* add sliding window pattern for model in test

* style: Fix indentation

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Fix \r\n

Thanks Claude!

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Keep shared expert fused

Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: More indentation fixes

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-08-19 16:53:31 +02:00
Titaniumtown 6cc504a2e9 sycl: report zero devices instead of aborting when the host has none (#27291)
Prevents crash when not even performing SYCL compute, for instance
when trying to run `llama-quantize`.
2026-08-19 07:46:01 -07:00
Sigbjørn Skjæret 01ac3ad761 ci : add release attestation url (#27389) 2026-08-19 16:40:47 +02:00
Sigbjørn Skjæret 2e92ecd024 models : remove duplicate metadata load (#27378) 2026-08-19 16:05:50 +02:00
Sigbjørn Skjæret 645ca2834b ci : re-enable release dependency for sycl (#27385) 2026-08-19 16:44:16 +03:00
Xuan-Son Nguyen fe8156f789 ggml: add ggml_rope_set_offset (+ metal support) (#27120)
* add params

* cpu kernel

* metal kernel

* add test backend ops

* gate other backends

* ggml: (cuda) support ggml_rope_set_offset (#27121)

* rm cuda supports_op guard, fix webgpu clang-format

* ggml: support ggml_rope_set_offset on vulkan (#27344)

* ggml: support ggml_rope_set_offset on vulkan

* remove inplace optimization
2026-08-19 14:04:57 +02:00
Pascal 77acca437f ui: read persisted settings before the API key probe (#27365)
The route loads run ahead of the root layout script, so validateApiKey
read the settings store while it still held factory defaults and probed
/props without the stored key. initStores() now hands the same startup
promise to every caller and the chat loads await it before probing.

The one-time admin baseline no longer overwrites a key the user has
already set: on a first visit the config carries factory values only, so
a diverging key comes from the user and wins.
2026-08-19 14:02:08 +02:00
Xuan-Son Nguyen 95c409c136 mtmd: add mtmd_bitmap_set_mergeable (#27348) 2026-08-19 13:48:22 +02:00
Georgi Gerganov 8ef78e644f metal : dequantize q8_0 using packed types (#27370) 2026-08-19 14:10:05 +03:00
Xuan-Son Nguyen ee4c505a4f server: add dedup-cache-models preset option (#27346) 2026-08-19 11:04:26 +02:00
Kevin Hopper 98d1e92c21 vulkan: tiled transpose for 0<->2 permuted CONT (#26585)
* vulkan: tiled transpose for 0<->2 permuted CONT

-ggml_vk_get_cpy_pipeline only routed to the tiled shared-memory transpose
shader when dim1 was the innermost dimension, i.e. ggml_transpose (a 0<->1
swap). A 0<->2 swap -- ggml_cont(ggml_permute(x, 2, 1, 0, 3)) -- fell back to
the generic per-element strided copy, whose source reads stride by ne0*ne1
elements: one cache line per lane.

-DeepSeek-V4's lightning indexer performs exactly that permute on a
[n_kv, n_tokens, n_head] tensor. On Vulkan/RADV gfx1151 it ran at ~1-9 GB/s of
a ~200 GB/s part and accounted for 43% of total prefill time.

-Add copy_transpose_02.comp, mirroring copy_transpose.comp but tiling over dst
dims (0, 2) with dims 1 and 3 as the batch, so reads walk src dim2 and writes
walk dst dim0 -- both contiguous. The selection condition additionally requires
a non-contiguous source and a contiguous destination so it cannot take cases
the contiguous-copy shader already handles.

-test-backend-ops only exercised ggml_transpose for CONT, so the strided path
was untested. Add test_cont_permute covering (2,1,0,3), (1,2,0,3) and (0,2,1,3)
over f32/f16 at tile-aligned, tile-unaligned and large shapes. The large shapes
are in the eval set rather than only in perf because perf mode does not verify
results.

-Measured on gfx1151, ne=[n_kv,64,64,1], perm=(2,1,0,3), f32:

  n_kv=1024:   9.08 ->  579.85 GB/s
  n_kv=1280:  20.03 ->  153.71 GB/s
  n_kv=2048:   7.11 ->   91.68 GB/s
  n_kv=2304:  16.24 ->   86.49 GB/s

-The ~2.2x penalty previously seen at power-of-two n_kv (destination-stride
aliasing) is gone. End to end, DeepSeek-V4-Flash IQ3_XXS prefill on a 9k-token
prompt goes from 56.33 t/s to 103.74 t/s (+84%).

-Note: at n_tokens=512 a single slow-path dispatch takes ~273 ms and looping it
in perf mode can trip the GPU watchdog, so the perf cases use n_tokens=64.

* tests: fold test_cont_permute into test_cont, add L2-exceeding perf shapes

Review feedback: test_cont gains a permute parameter ({0,0,0,0} = none),
matching test_mul_mat's pattern, and the separate struct is gone. Perf
adds [n_kv, 512, 64, 1] variants (~0.5 GB per run) that exceed GPU L2,
since the 64-token shapes fit in cache on large parts and read above
memory bandwidth.

* tests: trim perf-case comment to the two-line summary

* vulkan: trim comments on the 0<->2 transpose path

Drop the shader file header, the read/write block comments and the
rationale prose in the CONT test cases. Keep the tile-shape and
bank-conflict notes and the permute parameter documentation.

---------

Co-authored-by: Kevin Hopper <no-reply@maestro.press>
2026-08-19 10:20:21 +02:00
Masashi Yoshimura 5112b9738b ggml-webgpu: add mulmat with overlapping src0/src1 (e.g., for minimax-01) (#27321) 2026-08-19 16:29:33 +09:00
Jeremie Miller 0adcc3bb57 ci : add attestation for signed release artifacts (#25933) 2026-08-19 10:23:52 +03:00
Jeff Bolz 79fe799194 tests: skip test-unicode build on win32/BUILD_SHARED_LIBS (#27336) 2026-08-19 08:50:50 +02:00
Oğuzhan Akkaya 0329fcdac8 gguf-py : add size guards to GGUFReader (#27188)
* gguf-py : add size guards to GGUFReader

Guard kv_count, tensor_count, string length, and array length
against crafted values that cause unbounded allocation or hangs.

Assisted-by: opencode

* gguf : validate tensor data section fits within file

When no_alloc=true, gguf_init_from_reader accepted files where the
tensor data section (computed from header claims) exceeded the remaining
file size. This allowed crafted GGUF files to pass validation while
having insufficient data, leading to OOB reads when the loader later
mapped tensor data from the file.

Assisted-by: opencode

* gguf-py : move size limits into gguf_reader.py

Per review feedback, the limits are not part of gguf.h but are
arbitrary limits defined in gguf.cpp, so define them locally in
the reader instead of exporting them from constants.

Assisted-by: opencode

* remove gguf.ccp changes

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-08-19 09:35:27 +03:00
Xuan-Son Nguyen 6d05498314 server: (cosmetic) do not print cmd_child_to_router messages [no release] (#27347) 2026-08-19 01:45:56 +02:00
Hongqiang Wang 60addddf3c opencl: fix WAR race in the generic FA tile kernels when the WG spans subgroups (#26434) 2026-08-18 13:57:24 -07:00
Saba Fallah 9731ad3f29 mtmd: deepseek-ocr SAM ggml_conv_2d with the im2col kept in F32 (#26727) 2026-08-18 19:05:19 +02:00
Aman Gupta af5172627d RPC: populate use_count to enable fusion inside backends (#27142)
* RPC: populate use_count to enable fusion inside backends

* bump minor version
2026-08-18 21:08:57 +05:30
Aleksander Grygier 3dc7285b4f ui: Services consolidation refactor (#27239)
* ui: Move stream lookup and replay fetches into ChatService

chatStore called fetch() directly for /v1/streams/lookup and the
/v1/stream replay. These now live next to the other stream-session
methods in ChatService, so services stay the only API I/O layer.

* ui: Move /models/sse feed reader into ModelsService

ModelsService.watchModelEvents owns the byte stream, reconnect loop
and SSE record parsing; modelsStore keeps only event routing and
state.

* ui: Extract conversation import/export into ConversationTransferService

The JSONL session format, ZIP archiving and browser downloads are
pure I/O with no store state, so they move out of
conversationsStore. The store keeps the DB orchestration
(bulkExportConversations, downloadConversation,
importConversationsData) and delegates the format work.

* ui: Consolidate active model resolution into modelsStore.activeModelId

The same resolution chain was duplicated in useChatScreenActiveModel,
ChatForm, ChatFormActionModels and contextStatsStore, with slight
drift in the single-model fallback. The canonical getter now lives in
modelsStore, and the shared last-assistant-model lookup moved to
utils as getConversationModel.

* ui: Initialize stores explicitly via initStores()

Store constructors and module-level side effects ran migrations and
localStorage reads in import order. Migrations rename and rewrite
localStorage keys, so a settings load racing ahead of them could
clobber migrated values. initStores() is called once from the root
layout and runs migrations first, then the stores that read
localStorage, then the conversations DB load.

* refactor: Constants for stream query params
2026-08-18 16:39:32 +02:00
Aleksander Grygier fdf4c64604 ui: Stores consolidation refactor (#27238)
* ui: Remove dead code from stores

- persisted() helper was exported but never used
- messageUpdateCallback / registerMessageUpdateCallback were never wired up
- conversationsStore.initialize() alias, single caller moved to init()

* ui: Merge device, theme and viewport into a single deviceStore

All three are reactive browser-environment signals, now exposed as one
class store: deviceStore.isMobile, deviceStore.isIOSDevice / isIOSSafari
/ isWKWebView / isStandalone and deviceStore.systemTheme.isDark. The
systemTheme name disambiguates the OS preference from the user theme
preference in settingsStore. Drops the unused viewport export (only
isMobile was consumed).

* ui: Merge build info into version store

One VersionStore class with build (llama.cpp build number from
build.json) and frontend (PWA version from _app/version.json),
matching the class pattern of the other stores.

* ui: Colocate context gauge popup state with its components

The gauge popup state is local UI state shared only by the
ChatFormContextGauge subtree, so it lives next to its consumers
instead of the app-scope stores barrel.
2026-08-18 16:37:26 +02:00
Niklas Wenzel 04b569142d common: share thread pools when n_threads differ (#27138) 2026-08-18 16:23:43 +02:00
Ed Addario 0596704284 quant : Optimise memory usage by evicting weights after processing each layer (#22877)
* Evict weights from memory after processing each layer

* Revert changes

* Move unmap to libllama

* Unmap weights offloaded to backend

* Change member's constness

* Remove unmap weights offloaded to backend
2026-08-18 16:22:32 +02:00
Titaniumtown 0882c7bc89 sycl: honor GGML_HINT_SRC0_IS_HADAMARD (#27298)
Kernel is a port of `ggml-cuda/fwht.cu`

(us/run, median):
```
m x n x k           GEMM      FWHT   speedup
64 x    1 x  64     10.20     2.93     3.48x
64 x 2048 x  64     10.75     2.71     3.97x
128 x    1 x 128    10.33     2.88     3.59x
128 x   32 x 128     9.20     2.77     3.33x
128 x 2048 x 128    16.46     2.76     5.95x
256 x    1 x 256    10.19     2.77     3.68x
256 x 2048 x 256    16.69     3.41     4.89x
512 x 2048 x 512    54.16    12.89     4.20x
```
2026-08-18 21:21:25 +08:00
Thiago Padilha afd439df1f unicode : include '~' in collapsed symbol class (#26972)
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.

Assisted-by: Codex
2026-08-18 15:15:22 +02:00
Georgi Gerganov 169e4a7ff2 readme : update status badges + regen AUTHORS (#27317)
* readme : update status badges

* authors : regen
2026-08-18 14:35:04 +03:00
Zijun Yu 9d77fa1725 ci : Update OpenVINO to 2026.3, skip nemotron-h rollback test (#27292)
* update to ov-2026.3, update device drivers

* ci: skip nemotron-h rollback test on OpenVINO

The OpenVINO backend does not support SSM_SCAN, so the Nemotron-H recurrent state rollback graph is split and cannot preserve the recurrent cache output shape. Keep the test enabled for other backends and retain the qwen35 OpenVINO rollback coverage.

---------

Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
2026-08-18 12:02:22 +02:00
shivamkumard-ctrl c0296022f3 ci: add Windows ARM64 CUDA support to the manual workflow (#27300)
- Add a CUDA 13.4 ARM64 matrix entry.
- Build only ggml-cuda for x64 and ARM64.
2026-08-18 11:55:45 +02:00
BlackFoil 7acdbb1f19 mtmd: fix LFM2 image tiling threshold (#27057)
* mtmd: fix LFM2 image tiling threshold

* refactor testing

* fix

* fix on windows

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-18 11:11:19 +02:00
Georgi Gerganov 1511ce3bc3 sync : ggml 2026-08-18 11:30:03 +03:00
Georgi Gerganov da786dc23e ggml : bump version to 0.20.2 (ggml/1589) 2026-08-18 11:30:03 +03:00
Georgi Gerganov 27e345b574 build : fix xcframework + cmake clean-up (#27304)
* xcframework : fix build

* mtmd : remove unused include path

* vendor : use vendor::hash alias target in cmake

CMake reserves "::" in target names for imported/alias targets, so the real
target keeps the name vendor-hash and a vendor::hash ALIAS target is added.
Consumers (mtmd, llama-gguf-hash) now link against the namespaced alias.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* vendor : add cmake targets for all vendored libs with vendor:: aliases

Add INTERFACE targets for the header-only vendor libs (miniaudio, nlohmann,
sheredom, stb) and ALIAS targets named vendor::<lib> for all of them,
including cpp-httplib and hash. Each exposes the vendor/ root so includes
are namespaced, e.g. <nlohmann/json.hpp>.

Consolidate the per-lib add_subdirectory calls into a single
add_subdirectory(vendor), keeping the cpp-httplib gate on LLAMA_BUILD_COMMON.
Consumers (llama-common, mtmd) now link the aliases instead of relying on
raw vendor/ include paths.

hash: consumers now include via "hash/hash.h"; the vendor/hash dir is kept
as a PRIVATE include so the synced upstream sources compile unmodified.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* readme : use foo/bar names in acknowledgements

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* ocd : fix valign
2026-08-18 11:16:51 +03:00
Georgi Gerganov 8b86400975 ci : create pre-release with change log and nightly link in make-release (#27302)
* ci : create pre-release with change log and nightly link in make-release

After pushing the tag, create a pre-release using
ggml-org/action-create-release. The release description is generated by
scripts/make-release-desc.sh: the change log between the current and
previous version (one line per commit), a link to the corresponding
nightly build when it exists, and a note that semantic versioning is
still work in progress.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* cmake : bump version to 0.1.2

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* ci : find the nightly tag by commit in make-release-desc.sh

The nightly release is guaranteed by the release checks to point at HEAD,
so instead of reconstructing its name (commit count, branch, hash) just
pick the b* tag pointing at HEAD. This also drops the RELEASE_BRANCH env
var from the workflow.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* ci : resolve the release commit from the version tag in make-release-desc.sh

The change log and nightly lookup now use the commit the version tag
points at (HEAD when the tag does not exist), instead of always HEAD.
This makes the script usable locally for older versions, e.g.
./scripts/make-release-desc.sh v0.1.1. The tag is resolved to a SHA
first, since --points-at does not peel annotated tags.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* ci : normalize the version argument in make-release-desc.sh

Accept the version with or without the leading v (0.1.1 == v0.1.1) and
reject anything else, instead of silently treating a bare version as a
non-existent tag name.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* cont : clean-up
2026-08-18 10:14:41 +03:00
ynankani 25ae3a9b33 CUDA: MMVQ nwarps=8 for bs=1 for dense models on DGX Spark (#26843)
* CUDA: MMVQ nwarps=8 for bs=1 for dense models on DGX Spark

Signed-off-by: ynankani <ynankani@nvidia.com>

* skip moe experts and allow others based on k geometry (allow only small idle tail)

Signed-off-by: ynankani <ynankani@nvidia.com>

* rename MMVQ DGX Spark params to GB10 and fix MSVC constexpr lambda capture

Signed-off-by: ynankani <ynankani@nvidia.com>

---------

Signed-off-by: ynankani <ynankani@nvidia.com>
2026-08-18 09:45:53 +05:30
Aleksander Grygier 01818e4956 ui: enforce alphabetical enum member ordering (#27272) 2026-08-17 23:52:00 +02:00
Aleksander Grygier 0021a77de0 ui: Refactor Built-In Tools naming (Server/Browser) (#27271)
* server: rename built-in tools to server tools

* ui: rename built-in tools to server/browser tools
2026-08-17 22:23:22 +02:00
Eve 058df671b2 ci: more optimizations (#26983)
* replace rpc job with cpu

* remove vulkan cache

* move windows to build vulkan
2026-08-17 19:59:40 +00:00
Pascal 087f94d82e doc: document MCP stdio servers and CORS defaults in the server README [no release] [no ci] (#26847)
* doc: document MCP stdio servers and CORS defaults in the server README

The MCP arguments were listed but nothing explained what an MCP server
is or how to declare one. Cover the stdio transport, the config keys,
the tool naming, and add a POSIX shell echo server as a minimal
example.

Also document the CORS behavior: the default reflected origin, the
switch to localhost once tools are enabled, and the recommended setting
per deployment.

* doc: drop the inline MCP shell example from the server README

The example parsed JSON-RPC by hand and sat in a page people copy paste
from, into servers spawned with the privileges of llama-server. Point to
the specification instead.

Link the pull request that introduced the feature, and keep a short
mcp.json snippet so the table of configuration keys has a declaration to
refer to.
2026-08-17 21:32:15 +02:00
Xuan-Son Nguyen 533b18257b server: save processed mtmd chunks as placeholder (#27278) 2026-08-17 21:29:50 +02:00
Xuan-Son Nguyen ed1c3a20f5 mtmd: use sha256 for input hashing (#27274)
* mtmd: use sha256 for input hashing

* void conflict with boringssl
2026-08-17 21:20:12 +02:00
Sigbjørn Skjæret d8df12ebc4 vocab : support integer tokenizer scores (#27260) 2026-08-17 20:13:15 +02:00
Tarek Dakhran b75ecd1971 mtmd : skip thumbnail for non-tiled LFM2 images (#27246)
* mtmd : skip thumbnail for non-tiled LFM2 images

Matches HF behavior.

* Update tools/mtmd/mtmd-image.cpp

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
2026-08-17 19:41:32 +02:00
Mario Limonciello 60eeeb6082 cuda : skip UMA override for HIP builds (#27083)
AMD APUs report accurate memory via hipMemGetInfo. Using
MemAvailable over-promises on small-carveout systems.

fixes #18159
2026-08-17 22:05:36 +05:30
Xuan-Son Nguyen 39be55c97e vendor: move hash to vendor (#27262)
* vendor: move hash to vendor

* group hashes into one single static lib
2026-08-17 18:21:02 +02:00
Georgi Gerganov 34af94cd9a ci : push release tag explicitly in release.yml (#27261)
Add a "Create and push git tag" step to the release job, right before
the "Create release" step. The tag is created with git tag and pushed
with the deploy key already configured by the Clone step, instead of
relying on the Releases API (action-create-release) to create it as a
side effect.

The tag is lightweight, matching all existing b<number> release tags.
The step is idempotent: if the tag already exists (e.g. on a re-run),
creation and push are skipped.

Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-08-17 16:14:13 +03:00
Xuan-Son Nguyen 666f8898a2 ui: move get_datetime tool to frontend (#27255)
* ui: move get_datetime tool to frontend

* clarify docs

* server: drop the now unused ctime include

strftime() and gmtime_r() were the only users, both went away with the
get_datetime tool. Also make the renderer's catch inert: the browser
executor always emits JSON, so a non-JSON result is no longer a date to
display.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-08-17 14:33:55 +02:00
Georgi Gerganov 805984d676 ci : reduce builds in build-xcframework.sh (#27252)
* ci : parallelize platform builds in build-xcframework.sh

The ios-xcode release job builds 7 platform/simulator configurations
sequentially, each with -j $(nproc). Run them with at most 3 concurrent
builds (the release runner has 3 cores), splitting the cores between the
builds (-j 1 each on the runner), so total CPU pressure is unchanged while
the build phase runs about 2.3x faster.

- convert the 7 build blocks into functions (flags unchanged)
- add a run_builds_parallel pool: 3-slot sliding window, per-build logs,
  dumps the failing log and aborts on error (background job failures do
  not trigger set -e)
- queue the 2-arch builds first so the slower builds occupy the slots early

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* pi : add guideline for comments

* cont : disable 5/7 builds

* ci : make build-xcframework.sh builds configurable via CLI args

The script now takes an optional list of builds to run
(ios-sim ios-device macos visionos visionos-sim tvos-sim tvos-device);
with no arguments it builds all of them, as before. The per-build
lists for the build pool, framework setup, static library combining
and xcframework creation are now driven by a single build_spec
lookup instead of four hardcoded (partially commented-out) lists.

release.yml builds only macos and ios-device to cut the build time.

Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-08-17 14:53:21 +03:00
王金旭 9cd719af21 model: support speculators-format checkpoints for DSpark (#26275)
* dspark: support speculators-format checkpoints (SpecForge exports)

Speculators-format DSpark drafts (e.g. SpecForge exports for the
Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in
three ways:

- the config nests the backbone hparams under transformer_layer_config
  and gives the extract layers as aux_hidden_state_layer_ids
- the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus
  token, not a prediction slot. Written as dflash.bonus_anchor; such
  drafts build the block and read the mask positions exactly like
  DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence
  sampling comes from DSpark
- the draft output vocab may be reduced (draft_vocab_size < vocab_size)
  with a d2t remap table. The converter expands lm_head/markov_w2 back
  to the full vocab and synthesizes an lm_head bias of -1e9 on the rows
  the draft cannot produce, so the runtime needs no d2t remapping. Such
  drafts ship their own (now optional) token_embd/output tensors instead
  of sharing the target's

Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical
with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6).

Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5

* dspark: fold the speculators draft class into DSparkModel

One class now covers every DSpark variant. What used to pick the class is
a single flag, because the arch name turns out to be the only thing that
separates the two families: SpecForge also exports a flat schema that
carries no speculators_* fields yet still uses the 1+N bonus-anchor block,
so keying on those fields would silently mis-read its drafts.

Also rename i0 to i_first_pred in the draft read loop and the Markov head,
and give the head a real bonus_anchor bool instead of testing i0 > 0.

Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts
produces byte-identical GGUFs. The one behaviour change is that the
markov_head_type check now also covers the DeepSpec checkpoints, which
previously skipped it.

Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Opus 5

* dspark: address review comments

- rename bonus_anchor to sample_from_anchor (GGUF key and code), matching
  the checkpoint config field; absent key still means anchor-first
- rework the reduced draft vocab to match EAGLE3: d2t is written as I64
  absolute target ids and the logits are scattered at runtime, instead of
  expanding lm_head/markov_w2 and synthesizing an output bias at conversion
- move the t2d skip to modify_tensors, like EAGLE3
- drop _is_specforge: the arch name only picks the sample_from_anchor
  default, embed/lm_head sharing is decided by the draft vocab size
- deduplicate the tok_embd create_tensor left behind by the rebase

Verified with the RedHat gemma-4-31b speculator draft: greedy output is
byte-identical with and without the draft; acceptance 0.26 (n_max 7).

Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5

* dspark: fold the sample_from_anchor read into the block_size block

* dspark: fix flake8 continuation indent

* clean up

* dspark: key the sample_from_anchor default off the export format

  Co-authored-by: desovo7 <942845546@qq.com>
  Assisted-by: Claude Fable

* dspark: drop t2d in filter_tensors

  Co-authored-by: desovo7 <942845546@qq.com>
  Assisted-by: Claude Fable

* dspark: map model.lm_head instead of bypassing the dflash prefix

  Co-authored-by: desovo7 <942845546@qq.com>
  Assisted-by: Claude Fable

---------

Co-authored-by: desovo7 <942845546@qq.com>
Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>
2026-08-17 13:51:06 +02:00
Xuan-Son Nguyen 7077abbe14 ui: add browser get_info tool (#27251)
* ui: add browser get_info tool

* format

* nits

* Update tools/ui/src/lib/stores/tools.svelte.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
2026-08-17 13:14:33 +02:00
Georgi Gerganov d83f72d463 ci : restore release.yml check during make-release.yml (#27247)
* ci : restore release.yml check during make-release.yml

* cont : bump version to v0.1.1
2026-08-17 13:10:56 +03:00