73cc7d9287
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/s390x.Dockerfile
# .devops/vulkan.Dockerfile
# .devops/zendnn.Dockerfile
# .dockerignore
# .pi/gg/SYSTEM.md
# ggml/CMakeLists.txt
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# scripts/sync-ggml.last
# scripts/sync_vendor.py
# tools/cli/README.md
# tools/cli/cli.cpp
# tools/mtmd/clip.cpp
# tools/server/README.md
2026-06-20 17:16:19 +08:00
84b8856295
Merge commit '32eddaf2ea8dd5d499dee9655592a89b91bfde9d' into concedo_experimental
...
# Conflicts:
# docs/multimodal.md
# docs/preset.md
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hex-dma.h
# ggml/src/ggml-hexagon/htp/hex-utils.h
# ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# ggml/src/ggml-hexagon/htp/hmx-queue.c
# ggml/src/ggml-hexagon/htp/hmx-queue.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# scripts/snapdragon/ggml-hexagon-profile.py
# scripts/ui-assets.cmake
# tools/export-lora/README.md
# tools/server/CMakeLists.txt
2026-06-20 11:18:24 +08:00
2b686a9120
server: refactor child --> router communication ( #24821 )
...
* server: refactor child --> router communication
* fix wakeup case
* add docs
* improve update_status()
* nits
2026-06-20 01:02:26 +02:00
4b48a53b6c
server : optimize get_token_probabilities ( #24796 )
...
Use std::partial_sort to order only the requested top-n tokens instead
of the full vocabulary
logprobs sort: vocab=128000 n_top=0 iters=100
full sort: 8555.6 us/op
partial sort: 704.3 us/op
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2026-06-19 23:26:54 +02:00
175147e8f6
server: remove all internal mentions about "webui" ( #24817 )
2026-06-19 22:12:46 +02:00
fabde3bf51
arg: Add comment line support to --api-key-file ( #23168 )
2026-06-19 17:33:54 +02:00
8c2d6f6475
server: add --agent arg, remove redundant webui naming compat ( #24801 )
...
* server: add --agent arg, remove redundant webui naming compat
* corrent env
* fix the test
* llama-gen-docs
* nits: wordings
2026-06-19 16:06:13 +02:00
b14e3fb90c
spec: support eagle3 for qwen3.5 & 3.6 ( #24593 )
...
* spec: support qwen3.5 & 3.6 eagle3 draft
* eagle3: Add deferred boundary checkpoints restore support for hybrid models
* apply suggestions
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* spec: adapt to API change
* spec: fix naming
* cont : add TODO
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-19 13:08:50 +03:00
159d093a43
server: fix non-bound n_discard value (ctx shifting) ( #24786 )
...
* server: fix non-bound n_discard value
* Update tools/server/server-context.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-19 10:53:44 +02:00
80452d65b9
server : consolidate slot selection into get_available_slot ( #24755 )
...
Absorb get_slot_by_id logic into get_available_slot so slot selection
is handled by a single function call. When a specific slot id is
requested, the LCP similarity check still runs to enable proper
prompt cache updates.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
2026-06-19 09:22:34 +03:00
40f3aafc45
server: add "X-Accel-Buffering": "no" header to streaming endpoints ( #24774 )
...
* server: add "X-Accel-Buffering": "no" header to streaming endpoints
This header tells Nginx (as a reverse proxy) to NOT buffer responses. (only affects streaming endpoints)
Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
2026-06-18 22:01:24 +02:00
fe7c8b2414
server: (router) fix stopping_thread potentially hang ( #24728 )
...
* server: (router) fix stopping_thread potentially hang
* fix windows build
2026-06-18 15:41:09 +02:00
e1efd0991d
server: add "schema" and validation ( #24150 )
...
* wip
* working
* correct some limits
* add field name to error message
2026-06-18 15:40:58 +02:00
6591c33667
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/release.yml
# app/llama.cpp
# common/download.cpp
# docs/backend/SYCL.md
# docs/ops.md
# docs/ops/SYCL.csv
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/backend.hpp
# ggml/src/ggml-sycl/common.cpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/convert.cpp
# ggml/src/ggml-sycl/dequantize.hpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/dpct/helper.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/outprod.cpp
# ggml/src/ggml-sycl/vecdotq.hpp
# tools/server/README.md
2026-06-18 21:00:52 +08:00
08023072ef
server : add last-5-seconds generation speed display ( #24291 )
...
* server : add last-5-seconds generation speed display
* cont : clean-up
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-18 14:02:20 +02:00
10786217e9
server : return HTTP 400 on invalid grammar ( #24144 ) ( #24154 )
...
Throw on grammar parse failure so the server returns HTTP 400
instead of silently dropping the constraint.
Add a regression test for the invalid-grammar response.
Fixes #24144
2026-06-18 12:49:14 +02:00
552258c535
server: (router) rework -hf preset repo ( #24739 )
...
* server: temporary remove HF remote preset
* rework remove preset.ini support
* rm unused get_remote_preset_whitelist()
* print warning
* add docs
* rm stray file
2026-06-18 12:45:23 +02:00
968c43891a
server: fix router args not being forwarded to child instances ( #24760 )
2026-06-18 12:15:46 +02:00
4b4d13ae72
server: (router) add model management API ( #23976 )
...
* wip
* server: (router) add SSE realtime updates API
* nits
* wip
* add download API
* add download api
* update docs
* add delete endpoint
* fix std::terminate
* fix crash
* fix 2
* add tests
* nits
2026-06-17 18:04:58 +02:00
382ce55fb7
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/intel.Dockerfile
# README.md
# docs/backend/SYCL.md
# docs/install.md
# docs/ops.md
# docs/ops/SYCL.csv
# ggml/src/ggml-sycl/binbcast.cpp
# ggml/src/ggml-sycl/concat.cpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/element_wise.cpp
# ggml/src/ggml-sycl/element_wise.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/mmvq.hpp
# tests/peg-parser/test-gbnf-generation.cpp
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tools/llama-bench/llama-bench.cpp
2026-06-16 17:55:04 +08:00
635b65ad7a
spec: add spec metrics mean acceptance length and acceptance rate per position ( #24536 )
...
* spec: add spec metrics mean acceptance length and acceptance per pos
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix as suggestions
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-16 10:23:09 +03:00
e3a74b2990
bench : add --offline ( #24511 )
...
* bench : add --offline
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
* Add default
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2026-06-16 08:26:05 +02:00
e3cab403bf
mtmd : add post-decode callback ( #24645 )
...
Assisted-by: pi:llama.cpp/Qwen3.6-27B
2026-06-15 16:02:05 +03:00
b809994360
can build llama server now
2026-06-14 15:06:37 +08:00
ea0351c71a
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/release.yml
# .github/workflows/ui-build-self-hosted.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui-publish.yml
# .github/workflows/ui-self-hosted.yml
# .github/workflows/ui.yml
# .gitignore
# README.md
# docs/ops.md
# docs/ops/Vulkan.csv
# ggml/CMakeLists.txt
# scripts/sync-ggml.last
# scripts/sync_vendor.py
# scripts/ui-assets.cmake
# tests/test-jinja.cpp
# tests/test-llama-archs.cpp
2026-06-14 11:26:16 +08:00
ef5a724ee4
allow building server
2026-06-14 10:31:14 +08:00
e8067a8b36
ui: build-time gzip compression ( #24571 )
...
* ui: keep original file name and path
* fix nocache
* ui: build-time gzip compression
2026-06-13 16:57:27 +02:00
597b6672e8
ui: keep original file name and path ( #24568 )
...
* ui: keep original file name and path
* fix nocache
2026-06-13 14:31:41 +02:00
57fe1f07c3
server: clean up static assets handling ( #24550 )
...
* server: clean up static assets handling
* nits
* simplify file name handling, use static file name everywhere
* cmake/ui : bundle UI assets in an archive
* ui : run prettier on post-build.js
---------
Co-authored-by: Alde Rojas <hello@alde.dev >
2026-06-13 11:51:20 +02:00
d8a24ccee2
fit : wrap llama_device_memory_data ( #24522 )
2026-06-13 08:09:52 +03:00
e37abd6b5f
mtmd: add batching API ( #24384 )
...
* mtmd: add batching API
* wip
* first working version (gemma4v)
* add arg
* nits
* wire up support_batch()
* fix 0.0 output embd
* fix audio
* nits
* refactor a bit
* nits
* fix non-batching case
* fix comment
2026-06-13 00:10:29 +02:00
ebc10770ac
server : fix reasoning budget WebUI precedence over model.ini ( #24517 )
...
When reasoning-budget is set in model.ini, the per-request
thinking_budget_tokens from the WebUI was ignored because the
model.ini value took unconditional precedence.
Swap the precedence so the WebUI per-request value is checked
first, with the model.ini value serving as a fallback default.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
2026-06-12 17:59:56 +03:00
f7ca93d12c
ui: PWA support ( #23871 )
...
* feat: Add basic PWA support and service worker for offline caching
* feat: Vite PWA implementation WIP
* feat: Improve PWA icons generation
* feat: Add PWA workbox to server routes
* feat: Include `version.json` in static assets
* feat: Add HTTP cache headers for PWA static assets
* feat: Update app name for `apple-mobile-web-app-title`
* feat: Implement PWA versioning and automatic update detection
* chore: Update `.gitignore` files
* feat: Splash Screens
* feat: Add dark mode favicon support
* refactor: Cleanup
* fix: Use dark logo for dark splash screens
* refactor: Simplify favicons SVG code
* fix: Adjust caching and polling for reliable service worker updates
* fix: Add missing favicon entry
* fix: Align PWA service worker configuration with SvelteKit build structure
* fix: Replace hashed bundle paths with versioned static paths
* test: Add PWA tests
* ci: Add build output for unit tests
* refactor: Cleanup
* fix: Server build & release versioning
* chore: Update package-lock.json
* chore: Increase PWA cache size
* chore: Update packages
* feat: Update favicons
* refactor: Post-merge fix
* feat: support explicit build version for PWA cache busting
* fix: CI
* feat: Improve PWA Refresh Alert UI
* feat: Add toggleable build version display
* refactor: Cleanup
* feat: Add version mismatch detection and manual app reload
* refactor: replace dynamic imports with static
* refactor: Cleanup
* feat: Add safe space for `pwa-<size>.png` rendered icons
* fix: use relative paths for PWA assets to support base path deployment
* feat: add PWA mode detection via URL query parameter
* feat: Use ?cache=true for SW-cached PWA assets
* refactor: Build process cleanup
* refactor: Decouple PWA versioning and remove ?cache=true workaround
* chore: Update README logo
* feat: Include PWA Assets generation in build script
* refactor: `usePwa` hook for core layout
* fix: Relativize base vite plugin
* fix: remove unnecessary backslash escapes in test regexes
* test: update static asset paths for API Key test
* refactor: Move SvelteKit PWA Options config to constants
* ui: fix update notification never appearing
Keep the PWA hook object intact instead of destructuring needRefreshByStorage,
which freezes the reactive getter. Also exclude loading.html from PWA
precache to prevent 404 errors and broken SW installation.
2026-06-12 15:53:26 +02:00
be1750a3b8
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cuda.Dockerfile
# .github/workflows/build-sycl.yml
# .github/workflows/release.yml
# ggml/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-opencl/kernels/gated_delta_net.cl
# ggml/src/ggml-sycl/gated_delta_net.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/gated_delta_net.wgsl
# scripts/sync-ggml.last
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
2026-06-12 15:39:57 +08:00
18ef86ecec
server: skip unused log lines on router mode ( #24463 )
2026-06-11 11:36:35 +02:00
db94854ff5
server : skip checkpoints beyond pos_next ( #24411 )
...
* server : skip checkpoints beyond pos_next
* cont : update comment + TODO + ref
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-11 10:18:12 +03:00
afc8de2c6b
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/vulkan.Dockerfile
# .devops/zendnn.Dockerfile
# .github/workflows/build-webgpu.yml
# .github/workflows/release.yml
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/binary.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/concat.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/scale.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/unary.wgsl
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-mtmd-c-api.c
# tools/cli/cli.cpp
# tools/mtmd/CMakeLists.txt
# tools/server/README.md
2026-06-10 17:21:05 +08:00
dd2f966f4b
Merge commit 'c2b1518fd4834bdd255a8ad9639738de3fb7d4ef' into concedo_experimental
...
# Conflicts:
# .devops/intel.Dockerfile
# ggml/CMakeLists.txt
# scripts/sync-ggml.last
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
# tools/cli/cli.cpp
2026-06-10 00:56:44 +08:00
cd49b19016
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/docker.yml
# tests/test-chat.cpp
# tools/mtmd/clip-impl.h
# tools/server/README.md
2026-06-10 00:36:26 +08:00
1e912561dd
server: log prompts to directory ( #22031 )
...
* server: log prompts to directory
Add `--log-prompts-dir` to write each prompt to a separate text file in
the specified directory.
* Apply suggestion from @ngxson
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com >
2026-06-09 12:09:07 +02:00
961e9a3e46
server : do not clear slots without unified KV cache ( #24190 )
...
* Always export idle slots to RAM
Without this, a slot's VRAM cache may not be written to RAM. If this
slot happens to be busy then later on, this triggers needless
preprocessing in another slot.
* cont : clean-up
---------
Co-authored-by: Christoph Weiss <weiss@wsoptics.de >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-09 10:45:16 +03:00
42a0afd594
server : do not parse when flushing http headers ( #24281 )
2026-06-08 13:32:41 -05:00
8f83d6c271
mtmd : add video input support ( #24269 )
...
* wip
* ok: lazy bitmap API
* remember to free lazy text
* wip
* add mtmd_helper_video
* support video input on server (base64 input)
* add MTMD_VIDEO config
* add timestamp
* update CLI
* cli: allow auto-completion for video
* add --video arg
* fix build
* update docs
* rename as suggested
2026-06-08 14:40:12 +03:00
9e3b928fd8
common : relax sampler name matching ( #23744 )
...
* common : relax sampler name matching
Currently, in some cases, the alternative names for samplers (like
`top-k` and `min-p` instead of the canonical `top_k` and `min_p`) are
not always recognized by the `common_sampler_types_from_names` function
in `common/sampling.cpp`.
This PR changes the signature of this function to remove the `bool
allow_alt_names` flag, and removes all occurences of the flag from call
sites. Therefore, the function will now always match all known names.
I also changed the logic of the function to unconditionally check the
provided sampler names against both the canonical and alternative names,
and to be case-insensitive.
This fixes an issue I was seeing wherein samplers specified in the
`llama-server` UI were not recognized as valid when the alternative
names were used.
* add more alt names
* cont. fix
* cast to unsigned char for correctness
* common : unify sampler name mapping
* annotate canonical vs. alt sampler name mappings per @CISC
* Update common/sampling.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* common : auto-generate sampler name aliases per @ngxson
* use merged map for matching
* use `.merge` instead of iterating
* nit: simplify comment
* nit: use insert everywhere, not index assignment
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-06-07 22:48:11 +02:00
04eb4c446d
llama : add Gemma4 MTP ( #23398 )
2026-06-07 20:50:54 +08:00
202aaef40b
Merge commit '6effcecd0bf3cb2209999cecfa297ed4d8523b5a' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-msys.yml
# .pi/gg/SYSTEM.md
# common/CMakeLists.txt
# examples/speculative-simple/speculative-simple.cpp
# ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# scripts/ui-assets.cmake
# src/llama-model.cpp
# tools/cli/cli.cpp
# tools/imatrix/imatrix.cpp
# tools/quantize/quantize.cpp
# tools/server/README.md
# tools/ui/tests/stories/a11y/ChatScreenForm.a11y.stories.svelte
2026-06-07 17:10:58 +08:00
f5c6ae1827
mtmd, server: add "placeholder bitmap" for counting tokens , add */input_tokens API ( #23913 )
...
* mtmd: add "placeholder bitmap" for counting tokens w/o preprocessing
* fast path skip preproc for placeholder
* fix build
* correct the api
* add server endpoint + tests
* add object name
* update docs
* add proxy handling
* fix build
* fix audio input path
* use is_placeholder in process_mtmd_prompt()
* nits
* nits (2)
* docs: clarify chat/completions/input_tokens is not official
* fix merge problem
2026-06-06 11:06:51 +02:00
9c955c48b0
Fix link to available UI settings ( #24169 )
...
The current link is to a non-existent file. I had a look at the repo, spotted the file containing the UI configuration key and updated the link
2026-06-05 14:39:32 +02:00
7c158fbb4a
server : disable on-device spec checkpoints ( #24108 )
2026-06-04 19:30:59 +03:00
af3bcde675
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# AGENTS.md
# build-xcframework.sh
2026-06-04 23:44:05 +08:00
Concedo
Xuan-Son Nguyen
Adrien Gallouët
Mikolaj Kucharski
Ruixiang Wang
Georgi Gerganov
Reguna
Aarni Koskela
Anuj Attri
Aleksander Grygier
Aldehir Rojas
jacekpoplawski
fiesh
ddh0
Aman Gupta
Mario