* feat(convert): Add conversion for GraniteSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(llama): Add granite_swa support
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add conversion infra for rope_pattern array
NOTE: There is other work also targeting this, so this may be
removed depending on merge order.
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(conversion): Fix SWA pattern logic and support for non-rope layers
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add support for GraniteMoeSWA
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add llama_hparams::has_rope and arch constants
NOTE: This shadows the work done for Granite Speech
https://github.com/ggml-org/llama.cpp/pull/25107
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add support for per-layer rope determination
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Fix failing flake8 for extra newlines
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver
Branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Load MoE params as optional
Branch: GraniteSWAForCausalLM
AI-usage: draft (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Handle MoE params in conversion
branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Remove unnecessary newline
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Remove unnecessary tensor additions to GRANITE architecture
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Correctly handle naming for ffn gate inp
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Always default hparams.rope_pattern to 1s
This isn't strictly necessary, but it will allow other models to rely on
hparams.has_rope(il) without needting to prepopulate.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Move to has_rope for all granite model architectures
Now that we have a proper hparam for this, it's better to use it and not
require a hacky fallback in the hparam method itself.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: No hacky rope_finetuned fallback in has_rope
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fully remove rope hparam filling in granitemoe
There are no granitemoe models that use NoPE (it's not actually used in the
layer building below), so this was just dead code.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Save out rope_pattern in model-saver
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Set hparams.rope_finetuned for round trip
Since the value is _read_ from rope_finetuned, we need to persist it when
the model is saved with the saver.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Code review cleanup
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* refactor: Keep gate/up fused for MoE path
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Skip GRANITE_SWA in model saver
https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651
Keeping is_swa_impl in the saver can break other models.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* add sliding window pattern for model in test
* style: Fix indentation
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fix \r\n
Thanks Claude!
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Keep shared expert fused
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: More indentation fixes
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* add params
* cpu kernel
* metal kernel
* add test backend ops
* gate other backends
* ggml: (cuda) support ggml_rope_set_offset (#27121)
* rm cuda supports_op guard, fix webgpu clang-format
* ggml: support ggml_rope_set_offset on vulkan (#27344)
* ggml: support ggml_rope_set_offset on vulkan
* remove inplace optimization
The route loads run ahead of the root layout script, so validateApiKey
read the settings store while it still held factory defaults and probed
/props without the stored key. initStores() now hands the same startup
promise to every caller and the chat loads await it before probing.
The one-time admin baseline no longer overwrites a key the user has
already set: on a first visit the config carries factory values only, so
a diverging key comes from the user and wins.
* vulkan: tiled transpose for 0<->2 permuted CONT
-ggml_vk_get_cpy_pipeline only routed to the tiled shared-memory transpose
shader when dim1 was the innermost dimension, i.e. ggml_transpose (a 0<->1
swap). A 0<->2 swap -- ggml_cont(ggml_permute(x, 2, 1, 0, 3)) -- fell back to
the generic per-element strided copy, whose source reads stride by ne0*ne1
elements: one cache line per lane.
-DeepSeek-V4's lightning indexer performs exactly that permute on a
[n_kv, n_tokens, n_head] tensor. On Vulkan/RADV gfx1151 it ran at ~1-9 GB/s of
a ~200 GB/s part and accounted for 43% of total prefill time.
-Add copy_transpose_02.comp, mirroring copy_transpose.comp but tiling over dst
dims (0, 2) with dims 1 and 3 as the batch, so reads walk src dim2 and writes
walk dst dim0 -- both contiguous. The selection condition additionally requires
a non-contiguous source and a contiguous destination so it cannot take cases
the contiguous-copy shader already handles.
-test-backend-ops only exercised ggml_transpose for CONT, so the strided path
was untested. Add test_cont_permute covering (2,1,0,3), (1,2,0,3) and (0,2,1,3)
over f32/f16 at tile-aligned, tile-unaligned and large shapes. The large shapes
are in the eval set rather than only in perf because perf mode does not verify
results.
-Measured on gfx1151, ne=[n_kv,64,64,1], perm=(2,1,0,3), f32:
n_kv=1024: 9.08 -> 579.85 GB/s
n_kv=1280: 20.03 -> 153.71 GB/s
n_kv=2048: 7.11 -> 91.68 GB/s
n_kv=2304: 16.24 -> 86.49 GB/s
-The ~2.2x penalty previously seen at power-of-two n_kv (destination-stride
aliasing) is gone. End to end, DeepSeek-V4-Flash IQ3_XXS prefill on a 9k-token
prompt goes from 56.33 t/s to 103.74 t/s (+84%).
-Note: at n_tokens=512 a single slow-path dispatch takes ~273 ms and looping it
in perf mode can trip the GPU watchdog, so the perf cases use n_tokens=64.
* tests: fold test_cont_permute into test_cont, add L2-exceeding perf shapes
Review feedback: test_cont gains a permute parameter ({0,0,0,0} = none),
matching test_mul_mat's pattern, and the separate struct is gone. Perf
adds [n_kv, 512, 64, 1] variants (~0.5 GB per run) that exceed GPU L2,
since the 64-token shapes fit in cache on large parts and read above
memory bandwidth.
* tests: trim perf-case comment to the two-line summary
* vulkan: trim comments on the 0<->2 transpose path
Drop the shader file header, the read/write block comments and the
rationale prose in the CONT test cases. Keep the tile-shape and
bank-conflict notes and the permute parameter documentation.
---------
Co-authored-by: Kevin Hopper <no-reply@maestro.press>
* gguf-py : add size guards to GGUFReader
Guard kv_count, tensor_count, string length, and array length
against crafted values that cause unbounded allocation or hangs.
Assisted-by: opencode
* gguf : validate tensor data section fits within file
When no_alloc=true, gguf_init_from_reader accepted files where the
tensor data section (computed from header claims) exceeded the remaining
file size. This allowed crafted GGUF files to pass validation while
having insufficient data, leading to OOB reads when the loader later
mapped tensor data from the file.
Assisted-by: opencode
* gguf-py : move size limits into gguf_reader.py
Per review feedback, the limits are not part of gguf.h but are
arbitrary limits defined in gguf.cpp, so define them locally in
the reader instead of exporting them from constants.
Assisted-by: opencode
* remove gguf.ccp changes
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: Move stream lookup and replay fetches into ChatService
chatStore called fetch() directly for /v1/streams/lookup and the
/v1/stream replay. These now live next to the other stream-session
methods in ChatService, so services stay the only API I/O layer.
* ui: Move /models/sse feed reader into ModelsService
ModelsService.watchModelEvents owns the byte stream, reconnect loop
and SSE record parsing; modelsStore keeps only event routing and
state.
* ui: Extract conversation import/export into ConversationTransferService
The JSONL session format, ZIP archiving and browser downloads are
pure I/O with no store state, so they move out of
conversationsStore. The store keeps the DB orchestration
(bulkExportConversations, downloadConversation,
importConversationsData) and delegates the format work.
* ui: Consolidate active model resolution into modelsStore.activeModelId
The same resolution chain was duplicated in useChatScreenActiveModel,
ChatForm, ChatFormActionModels and contextStatsStore, with slight
drift in the single-model fallback. The canonical getter now lives in
modelsStore, and the shared last-assistant-model lookup moved to
utils as getConversationModel.
* ui: Initialize stores explicitly via initStores()
Store constructors and module-level side effects ran migrations and
localStorage reads in import order. Migrations rename and rewrite
localStorage keys, so a settings load racing ahead of them could
clobber migrated values. initStores() is called once from the root
layout and runs migrations first, then the stores that read
localStorage, then the conversations DB load.
* refactor: Constants for stream query params
* ui: Remove dead code from stores
- persisted() helper was exported but never used
- messageUpdateCallback / registerMessageUpdateCallback were never wired up
- conversationsStore.initialize() alias, single caller moved to init()
* ui: Merge device, theme and viewport into a single deviceStore
All three are reactive browser-environment signals, now exposed as one
class store: deviceStore.isMobile, deviceStore.isIOSDevice / isIOSSafari
/ isWKWebView / isStandalone and deviceStore.systemTheme.isDark. The
systemTheme name disambiguates the OS preference from the user theme
preference in settingsStore. Drops the unused viewport export (only
isMobile was consumed).
* ui: Merge build info into version store
One VersionStore class with build (llama.cpp build number from
build.json) and frontend (PWA version from _app/version.json),
matching the class pattern of the other stores.
* ui: Colocate context gauge popup state with its components
The gauge popup state is local UI state shared only by the
ChatFormContextGauge subtree, so it lives next to its consumers
instead of the app-scope stores barrel.
Kernel is a port of `ggml-cuda/fwht.cu`
(us/run, median):
```
m x n x k GEMM FWHT speedup
64 x 1 x 64 10.20 2.93 3.48x
64 x 2048 x 64 10.75 2.71 3.97x
128 x 1 x 128 10.33 2.88 3.59x
128 x 32 x 128 9.20 2.77 3.33x
128 x 2048 x 128 16.46 2.76 5.95x
256 x 1 x 256 10.19 2.77 3.68x
256 x 2048 x 256 16.69 3.41 4.89x
512 x 2048 x 512 54.16 12.89 4.20x
```
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
* update to ov-2026.3, update device drivers
* ci: skip nemotron-h rollback test on OpenVINO
The OpenVINO backend does not support SSM_SCAN, so the Nemotron-H recurrent state rollback graph is split and cannot preserve the recurrent cache output shape. Keep the test enabled for other backends and retain the qwen35 OpenVINO rollback coverage.
---------
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
* xcframework : fix build
* mtmd : remove unused include path
* vendor : use vendor::hash alias target in cmake
CMake reserves "::" in target names for imported/alias targets, so the real
target keeps the name vendor-hash and a vendor::hash ALIAS target is added.
Consumers (mtmd, llama-gguf-hash) now link against the namespaced alias.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* vendor : add cmake targets for all vendored libs with vendor:: aliases
Add INTERFACE targets for the header-only vendor libs (miniaudio, nlohmann,
sheredom, stb) and ALIAS targets named vendor::<lib> for all of them,
including cpp-httplib and hash. Each exposes the vendor/ root so includes
are namespaced, e.g. <nlohmann/json.hpp>.
Consolidate the per-lib add_subdirectory calls into a single
add_subdirectory(vendor), keeping the cpp-httplib gate on LLAMA_BUILD_COMMON.
Consumers (llama-common, mtmd) now link the aliases instead of relying on
raw vendor/ include paths.
hash: consumers now include via "hash/hash.h"; the vendor/hash dir is kept
as a PRIVATE include so the synced upstream sources compile unmodified.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* readme : use foo/bar names in acknowledgements
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ocd : fix valign
* ci : create pre-release with change log and nightly link in make-release
After pushing the tag, create a pre-release using
ggml-org/action-create-release. The release description is generated by
scripts/make-release-desc.sh: the change log between the current and
previous version (one line per commit), a link to the corresponding
nightly build when it exists, and a note that semantic versioning is
still work in progress.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cmake : bump version to 0.1.2
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : find the nightly tag by commit in make-release-desc.sh
The nightly release is guaranteed by the release checks to point at HEAD,
so instead of reconstructing its name (commit count, branch, hash) just
pick the b* tag pointing at HEAD. This also drops the RELEASE_BRANCH env
var from the workflow.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : resolve the release commit from the version tag in make-release-desc.sh
The change log and nightly lookup now use the commit the version tag
points at (HEAD when the tag does not exist), instead of always HEAD.
This makes the script usable locally for older versions, e.g.
./scripts/make-release-desc.sh v0.1.1. The tag is resolved to a SHA
first, since --points-at does not peel annotated tags.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : normalize the version argument in make-release-desc.sh
Accept the version with or without the leading v (0.1.1 == v0.1.1) and
reject anything else, instead of silently treating a bare version as a
non-existent tag name.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : clean-up
* CUDA: MMVQ nwarps=8 for bs=1 for dense models on DGX Spark
Signed-off-by: ynankani <ynankani@nvidia.com>
* skip moe experts and allow others based on k geometry (allow only small idle tail)
Signed-off-by: ynankani <ynankani@nvidia.com>
* rename MMVQ DGX Spark params to GB10 and fix MSVC constexpr lambda capture
Signed-off-by: ynankani <ynankani@nvidia.com>
---------
Signed-off-by: ynankani <ynankani@nvidia.com>
* doc: document MCP stdio servers and CORS defaults in the server README
The MCP arguments were listed but nothing explained what an MCP server
is or how to declare one. Cover the stdio transport, the config keys,
the tool naming, and add a POSIX shell echo server as a minimal
example.
Also document the CORS behavior: the default reflected origin, the
switch to localhost once tools are enabled, and the recommended setting
per deployment.
* doc: drop the inline MCP shell example from the server README
The example parsed JSON-RPC by hand and sat in a page people copy paste
from, into servers spawned with the privileges of llama-server. Point to
the specification instead.
Link the pull request that introduced the feature, and keep a short
mcp.json snippet so the table of configuration keys has a declaration to
refer to.
Add a "Create and push git tag" step to the release job, right before
the "Create release" step. The tag is created with git tag and pushed
with the deploy key already configured by the Clone step, instead of
relying on the Releases API (action-create-release) to create it as a
side effect.
The tag is lightweight, matching all existing b<number> release tags.
The step is idempotent: if the tag already exists (e.g. on a re-run),
creation and push are skipped.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui: move get_datetime tool to frontend
* clarify docs
* server: drop the now unused ctime include
strftime() and gmtime_r() were the only users, both went away with the
get_datetime tool. Also make the renderer's catch inert: the browser
executor always emits JSON, so a non-JSON result is no longer a date to
display.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ci : parallelize platform builds in build-xcframework.sh
The ios-xcode release job builds 7 platform/simulator configurations
sequentially, each with -j $(nproc). Run them with at most 3 concurrent
builds (the release runner has 3 cores), splitting the cores between the
builds (-j 1 each on the runner), so total CPU pressure is unchanged while
the build phase runs about 2.3x faster.
- convert the 7 build blocks into functions (flags unchanged)
- add a run_builds_parallel pool: 3-slot sliding window, per-build logs,
dumps the failing log and aborts on error (background job failures do
not trigger set -e)
- queue the 2-arch builds first so the slower builds occupy the slots early
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : add guideline for comments
* cont : disable 5/7 builds
* ci : make build-xcframework.sh builds configurable via CLI args
The script now takes an optional list of builds to run
(ios-sim ios-device macos visionos visionos-sim tvos-sim tvos-device);
with no arguments it builds all of them, as before. The per-build
lists for the build pool, framework setup, static library combining
and xcframework creation are now driven by a single build_spec
lookup instead of four hardcoded (partially commented-out) lists.
release.yml builds only macos and ios-device to cut the build time.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* dspark: support speculators-format checkpoints (SpecForge exports)
Speculators-format DSpark drafts (e.g. SpecForge exports for the
Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in
three ways:
- the config nests the backbone hparams under transformer_layer_config
and gives the extract layers as aux_hidden_state_layer_ids
- the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus
token, not a prediction slot. Written as dflash.bonus_anchor; such
drafts build the block and read the mask positions exactly like
DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence
sampling comes from DSpark
- the draft output vocab may be reduced (draft_vocab_size < vocab_size)
with a d2t remap table. The converter expands lm_head/markov_w2 back
to the full vocab and synthesizes an lm_head bias of -1e9 on the rows
the draft cannot produce, so the runtime needs no d2t remapping. Such
drafts ship their own (now optional) token_embd/output tensors instead
of sharing the target's
Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical
with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the speculators draft class into DSparkModel
One class now covers every DSpark variant. What used to pick the class is
a single flag, because the arch name turns out to be the only thing that
separates the two families: SpecForge also exports a flat schema that
carries no speculators_* fields yet still uses the 1+N bonus-anchor block,
so keying on those fields would silently mis-read its drafts.
Also rename i0 to i_first_pred in the draft read loop and the Markov head,
and give the head a real bonus_anchor bool instead of testing i0 > 0.
Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts
produces byte-identical GGUFs. The one behaviour change is that the
markov_head_type check now also covers the DeepSpec checkpoints, which
previously skipped it.
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Opus 5
* dspark: address review comments
- rename bonus_anchor to sample_from_anchor (GGUF key and code), matching
the checkpoint config field; absent key still means anchor-first
- rework the reduced draft vocab to match EAGLE3: d2t is written as I64
absolute target ids and the logits are scattered at runtime, instead of
expanding lm_head/markov_w2 and synthesizing an output bias at conversion
- move the t2d skip to modify_tensors, like EAGLE3
- drop _is_specforge: the arch name only picks the sample_from_anchor
default, embed/lm_head sharing is decided by the draft vocab size
- deduplicate the tok_embd create_tensor left behind by the rebase
Verified with the RedHat gemma-4-31b speculator draft: greedy output is
byte-identical with and without the draft; acceptance 0.26 (n_max 7).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the sample_from_anchor read into the block_size block
* dspark: fix flake8 continuation indent
* clean up
* dspark: key the sample_from_anchor default off the export format
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: drop t2d in filter_tensors
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: map model.lm_head instead of bypassing the dflash prefix
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
---------
Co-authored-by: desovo7 <942845546@qq.com>
Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>