b1f3a6e5db
llama: automatically set parameters not set by the user in such a way that maximizes GPU utilization ( #16653 )
...
* llama: automatically fit args to free memory
llama-fit-params tool
* fix CI
* hints for bug reports, ensure no reallocation
* fix segfault with Vulkan
* add llama-fit-params to CI
* fix CI
* fix CI
* fix CI
* minor adjustments
* fix assignment of 1 dense layer
* fix logger not being reset on model load failure
* remove --n-gpu-layer hint on model load failure
* fix llama-fit-params verbosity
* fix edge case
* fix typo [no ci]
2025-12-15 09:24:59 +01:00
0759b09c90
graph: add f_attn_temp_offset ( #18025 )
2025-12-14 13:05:59 +01:00
609a2d0268
models : fix YaRN regression + consolidate logic ( #18006 )
...
* models : fix YaRN regression + consolidate logic
* cont : fix the fix
* cont : remove header
* cont : add header
2025-12-14 08:34:56 +02:00
5266379bca
llama_context: synchronize before reallocating output buffer ( #17974 )
2025-12-13 09:19:51 -06:00
7bed317f53
models : fix the attn_factor for mistral3 graphs + improve consistency ( #17945 )
...
* models : fix the attn_factor for mistral3 graphs
* cont : rework attn_factor correction logic
* cont : make deepseek2 consistent
* cont : add TODO
* cont : special-case DSv2
* cont : revert Mistral 3 Large changes
* cont : fix DS2 to use the original attn_factor
* cont : minor comments
2025-12-12 17:12:40 +02:00
34d243bf3c
Merge commit 'b6777218198888ed4b18d839e805f887562a9be4' into concedo_experimental
...
# Conflicts:
# CONTRIBUTING.md
# common/chat.cpp
# docs/ops.md
# docs/ops/CPU.csv
# docs/ops/CUDA.csv
# docs/ops/OpenCL.csv
# ggml/src/ggml-cann/aclnn_ops.cpp
# ggml/src/ggml-cann/common.h
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-sycl/softmax.cpp
# grammars/README.md
# src/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-grammar-integration.cpp
# tests/test-grammar-parser.cpp
# tests/test-llama-grammar.cpp
# tools/mtmd/CMakeLists.txt
2025-12-11 23:33:19 +08:00
278e45becf
Merge commit '2fa51c19b028180b35d316e9ed06f5f0f7ada2c1' into concedo_experimental
...
# Conflicts:
# .github/actions/windows-setup-cuda/action.yml
# .github/workflows/build-linux-cross.yml
# .github/workflows/release.yml
# README.md
# docs/build-riscv64-spacemit.md
# examples/model-conversion/logits.cpp
# ggml/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# models/templates/Kimi-K2-Instruct.jinja
# models/templates/Kimi-K2-Thinking.jinja
# tests/test-chat.cpp
# tools/server/README.md
2025-12-11 23:04:48 +08:00
fd0d0cab03
move pipeline parallelism to a --pipelineparallel launch flag
2025-12-11 21:03:41 +08:00
d9f8f60618
batch : fix sequence id ownership ( #17915 )
...
* batch : fix sequence id ownage
* cont : reduce allocations
2025-12-11 14:29:47 +02:00
4dff236a52
ggml : remove GGML_KQ_MASK_PAD constant ( #17910 )
...
* ggml : remove GGML_KQ_MASK_PAD constant
* cont : remove comment
2025-12-10 20:53:16 +02:00
b677721819
model : Qwen3-Next-80B-A3B has 48 layers ( #17898 )
...
* model : Qwen3-Next-80B-A3B has 48 layers
* model : Add 80B-A3B type name
2025-12-10 15:22:40 +01:00
63908b631a
cmake: fix Mach-O current version number ( #17877 )
...
PR #17091 set the VERSION of various libraries to 0.0.abcd, where abcd
is the LLAMA_BUILD_NUMBER. That build number is too large to fit in the
Mach-O 'current version' field's 'micro' part, which only goes up to
255. This just sets the Mach-O current version to 0 to get it building
properly again.
Fixes #17258 .
2025-12-09 13:17:41 +02:00
42b12b5608
model : nit, DeepSeek V1 MoE is 16B and GigaChat is 20B ( #12652 )
...
* nit, DeepSeek V1 MoE is 16B
* base type on n_ff_exp instead
2025-12-09 12:15:06 +01:00
e39502e74b
llama : add token matching support to llama-grammar ( #17816 )
...
* llama : add token support to llama-grammar
* fix inverse token comment
* refactor trigger_patterns to replay tokens instead of the entire string
* add token documentation
* fix test-llama-grammar
* improve test cases for tokens
2025-12-09 00:32:57 -06:00
1d2a1ab73d
model : support Rnj-1 ( #17811 )
...
* add support for rnj1
* refactor gemma3 to support rnj-1
* address review comments
2025-12-09 04:49:03 +01:00
c8554b66e0
graph : use fill instead of scale_bias in grouped expert selection ( #17867 )
...
* use fill instead of scale_bias in grouped expert selection
* do not explicitly use _inplace
2025-12-08 21:29:59 +01:00
e4e9c4329c
Make graph_max_nodes vary by ubatch size ( #17794 )
...
* Make graph_max_nodes vary by ubatch size for models where chunking might explode the graph
* Update src/llama-context.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* Add missing const
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-08 14:32:41 +01:00
4d3726278b
model: add llama 4 scaling for mistral-large (deepseek arch) ( #17744 )
2025-12-07 22:29:54 +01:00
17c0c8d55d
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# README.md
# docs/backend/zDNN.md
# docs/build.md
# docs/ops.md
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-rpc/ggml-rpc.cpp
# ggml/src/ggml-sycl/convert.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# src/llama-quant.cpp
# tests/test-backend-ops.cpp
# tools/llama-bench/llama-bench.cpp
# tools/server/README.md
2025-12-07 16:48:38 +08:00
7c5d271d6c
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build.yml
# .github/workflows/release.yml
# .github/workflows/winget.yml
# CMakeLists.txt
# CODEOWNERS
# CONTRIBUTING.md
# cmake/build-info.cmake
# docs/ops.md
# docs/ops/BLAS.csv
# docs/ops/Metal.csv
# examples/CMakeLists.txt
# examples/save-load-state/save-load-state.cpp
# examples/simple-cmake-pkg/README.md
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-rpc/ggml-rpc.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py
# src/llama-quant.cpp
# tests/test-backend-ops.cpp
# tools/server/CMakeLists.txt
2025-12-07 16:37:32 +08:00
444f00b0ec
llama : remove quantization sanity check ( #17788 )
...
* llama : remove quantization sanity check
This commit removes the quantization sanity check for attention layers.
The motivation for this is that there are model that are hybrid models
that have recurrent layers, experts layers, and attention layers. For
these models the current check fails as the experts layers are not
taking into account. After consideration, it was decided that this check
is not strictly necessary, and can be removed to allow for more flexible
model architectures.
* llama : remove unused pruned_attention_w and is_clip_model vars
2025-12-06 12:26:20 +01:00
1be97831e4
fix: prevent segfault in tokenizer on highly repetitive input ( #17786 )
...
Add nosubs|optimize flags to std::regex constructors to prevent
catastrophic backtracking when processing prompts with repeated
identical characters (e.g., 'A' * 10000).
The nosubs flag disables subgroup capture, significantly reducing
memory usage and backtracking on uniform token sequences
2025-12-05 13:52:23 +02:00
a67ef0f47f
llama : fix sanity checks during quantization ( #17721 )
2025-12-04 10:33:42 +02:00
03cec02a3d
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build.yml
# .github/workflows/release.yml
# .github/workflows/winget.yml
# CODEOWNERS
# README.md
# ci/run.sh
# docs/build.md
# docs/ops.md
# docs/ops/Vulkan.csv
# ggml/CMakeLists.txt
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# scripts/sync_vendor.py
# src/CMakeLists.txt
# tests/test-json-schema-to-grammar.cpp
# tests/test-quantize-stats.cpp
# tools/server/CMakeLists.txt
# tools/server/README.md
2025-12-03 18:56:31 +08:00
83269df91b
Merge commit '649495c9d915a284aeec5bca5d0efaa6d1bc7c87' into concedo_experimental
...
# Conflicts:
# CONTRIBUTING.md
# SECURITY.md
# docs/backend/SYCL.md
# examples/sycl/run-llama2.sh
# examples/sycl/run-llama3.sh
# examples/sycl/win-run-llama2.bat
# examples/sycl/win-run-llama3.bat
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/cpy.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# tests/test-backend-ops.cpp
# tests/test-json-schema-to-grammar.cpp
# tools/server/CMakeLists.txt
2025-12-03 18:43:46 +08:00
37adc9c6ba
ggml, llama : use defaulted constructors/destructors ( #17649 )
2025-12-03 07:12:18 +01:00
f3a9674ae8
llama : fix signed comparison warning on FreeBSD ( #17497 )
...
This ensures correct RLIM_INFINITY handling and compatibility on all platforms (32/64-bit).
warning: comparison of integers of different signs: 'rlim_t' (aka 'long') and 'size_t' (aka 'unsigned long') [-Wsign-compare]
488 | if (suggest && (lock_limit.rlim_max > lock_limit.rlim_cur + size)) {
| ~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2025-12-02 12:05:38 +01:00
746f9ee889
Override SSM_A op for Qwen3 Next to reduce splits ( #17587 )
...
* Override SSM_A op for Qwen3 Next to reduce splits
* New tensor mapping SSM_A_NOSCAN for SSM_A used outside of OP_SSM_SCAN context.
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-02 00:43:13 +01:00
00c361fe53
fix: llama arch implementation ( #17665 )
2025-12-01 21:21:13 +01:00
cd3c118908
model: support Ministral3 ( #17644 )
...
* conversion script
* support ministral 3
* maybe this is better?
* add TODO for rope_yarn_log_mul
* better ppl (tested on 14B-Instruct)
* Add Ministral3 support to Mistral format
* improve arch handling
* add sizes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* nits
---------
Co-authored-by: Julien Denize <julien.denize@mistral.ai >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-01 12:26:52 +01:00
6eea666912
llama-graph: avoid expand_forward for fusion ( #17633 )
2025-12-01 11:12:48 +02:00
bf5efcf86d
Merge commit 'd82b7a7c1d73c0674698d9601b1bbb0200933f29' into concedo_experimental
...
# Conflicts:
# ci/run.sh
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cuda/common.cuh
# tests/CMakeLists.txt
2025-11-30 15:43:11 +08:00
e072b2052e
ggml : add GGML_SCHED_NO_REALLOC option to disable reallocations in ggml_backend_sched ( #17276 )
...
* ggml : add GGML_SCHED_NO_REALLOC option to disable reallocations in ggml_backend_sched
Enabled in ggml-ci for testing.
* llama : update worst-case graph for unified cache
* ci : disable op offload in some tests
* fix spelling
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-28 17:33:23 +02:00
0ccb298087
Merge commit 'ddf9f94389a614ce005347f1c3f60ce477df1be1' into concedo_experimental
...
# Conflicts:
# examples/model-conversion/scripts/causal/run-converted-model.sh
# examples/model-conversion/scripts/causal/run-org-model.py
# src/CMakeLists.txt
# src/llama-quant.cpp
# tools/server/README.md
2025-11-28 23:27:50 +08:00
ff55414c42
model : Qwen3 Next ( #16095 )
...
* Qwen3 Next - cleaned up version
* Whitespaces and stuff
* Correct minor errors
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* Misc. fixes.
* Clean up code, add missing hybrid qualifier
* Did someone transpose the SOLVE_TRI result matrix? Perhaps...
* Whitespace
* Proper tensors for cb calls
* Use llama-graph.h vertical alignment
* BROKEN: chunking
* Set new tensors as inputs.
* Proper chunk logic
* It's the circle of life...
* More shenanigans for n_seq > 1
* Nail in the coffin?
* Fix Windows build
* Eh, one fails on Windows, the other fails on Mac... just use general capture.
* quant : cleanup
* model : cleanup
* qwen3 : cleanup
* cont : cleanup
* cont : cleanup
* ggml : revert change
* qwen3 : cleanup
* cont : cleanup
* Readd cmath
* qwen3 : fix typo
* Update convert_hf_to_gguf.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* Usual suspects
* fix my bad suggestion
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-28 12:02:56 +01:00
eda4a312cb
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/vulkan.Dockerfile
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-sycl/common.hpp
# tests/test-backend-ops.cpp
# tools/server/README.md
2025-11-28 13:22:02 +08:00
c386114922
arch : add description about LLM_TENSOR_INFOS ( #17550 )
2025-11-27 16:34:13 +02:00
6783b11fb0
models : fix LFM2 tensors ( #17548 )
2025-11-27 16:04:29 +02:00
4497096cb0
Merge commit '3e18dba9fd599ace0c3f2bc94bd74e5398d2efb6' into concedo_experimental
...
# Conflicts:
# CODEOWNERS
# ggml/src/ggml-cann/aclnn_ops.cpp
# ggml/src/ggml-cann/aclnn_ops.h
# ggml/src/ggml-cann/common.h
# ggml/src/ggml-cann/ggml-cann.cpp
# scripts/sync_vendor.py
# tests/test-backend-ops.cpp
2025-11-27 00:07:37 +08:00
583cb83416
ggml : add ggml_top_k ( #17365 )
...
* ggml : add ggml_top_k
* cont : add ggml_argsort_top_k
* metal : add top_k support
* ggml : cleanup
* tests : add virtual err() function for test_case
* ggml : add comments
2025-11-25 15:31:43 +02:00
724763fdec
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/vulkan.Dockerfile
# .github/workflows/build.yml
# .github/workflows/server.yml
# common/common.cpp
# examples/batched/README.md
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/arch-fallback.h
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/sync-ggml.last
# src/CMakeLists.txt
# tests/test-backend-ops.cpp
# tools/server/CMakeLists.txt
2025-11-25 16:38:07 +08:00
877566d512
llama: introduce support for model-embedded sampling parameters ( #17120 )
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Successful in 1m12s
Python check requirements.txt / check-requirements (push) Failing after 6s
Python Type-Check / pyright type-check (push) Successful in 37s
2025-11-25 09:56:07 +08:00
134e6940ca
llama : skip output reordering for single token batches ( #17466 )
...
This commit adds a check to skip the output reordering logic when
n_outputs == 1. With a single output token, the data is trivially
sorted and the reordering code is currently doing unnecessary work
(resetting and rebuilding output_ids to the same values).
The motivation for this change is improved code clarity and avoiding
confusion when debugging. While the performance impact is probably
negligible, this unnecessary work happens on every decode call in
llama-server when processing batches with single-token outputs.
2025-11-24 21:06:17 +01:00
4902eebe33
models : Added support for RND1 Diffusion Language Model ( #17433 )
...
* Converted RND1 model to GGUF weights
* RND1 llama.cpp support v1
* RND1 llama.cpp support v2 non causal bug
* RND1 llama.cpp support v3 doccumentation
* RND1 llama.cpp support v4 clean code
* linting issues
* RND1 pr fixes v1
* RND1 pr fixes v2
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* Diffusion documentation edits
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-11-24 14:16:56 +08:00
5248838a05
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .github/workflows/build.yml
# .github/workflows/release.yml
# .gitignore
# README.md
# common/CMakeLists.txt
# docs/ops.md
# docs/ops/Vulkan.csv
# examples/eval-callback/eval-callback.cpp
# ggml/src/ggml-cann/aclnn_ops.cpp
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/arch/x86/repack.cpp
# ggml/src/ggml-cpu/kleidiai/kernels.cpp
# scripts/sync-ggml.last
# src/llama-grammar.cpp
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tools/server/CMakeLists.txt
2025-11-22 18:26:13 +08:00
23bc779a6e
model : detect GigaChat3-10-A1.8B as deepseek lite ( #17420 )
...
* Detect GigaChat3-10-A1.8B as deepseek lite
Hardcodes checking number of layers to detect if lite version of deepseek.
* Add commnent identifying deepseek lite variants
deepseek lite variants include DeepSeek-V2-Lite, GigaChat3-10B-A1.8B
2025-11-21 14:51:38 +01:00
054a45c3d3
grammar: fix regression caused by #17381 ( #17412 )
...
* grammar: fix regression caused by #17381
* more readable
2025-11-20 18:35:10 +01:00
92c0b387a9
grammar : fix integer overflow ( #17381 )
...
* Fix DoS / integer overflow
* Remove optional, use INT64_MAX instead as placeholder value (it's technically -1, so it fits :)
* White space
* Actually, since it's unsigned, use UINT64_MAX
2025-11-20 14:47:04 +02:00
196f5083ef
common : more accurate sampling timing ( #17382 )
...
* common : more accurate sampling timing
* eval-callback : minor fixes
* cont : add time_meas impl
* cont : fix log msg [no ci]
* cont : fix multiple definitions of time_meas
* llama-cli : exclude chat template init from time measurement
* cont : print percentage of unaccounted time
* cont : do not reset timings
2025-11-20 13:40:10 +02:00
a045492088
vocab : call reserve() for building plamo-2-translate suffix ( #17343 )
...
Test 'Q4_K_M' quantization on https://huggingface.co/pfnet/plamo-2-translate
The 'suffix_to_score' size is 193510, it needs 19 memory allocation with final
capacity 262144 to hold the value, if not preserve the memory.
Signed-off-by: Haiyue Wang <haiyuewa@163.com >
2025-11-18 18:58:22 +01:00
Johannes Gäßler
Xuan-Son Nguyen
Georgi Gerganov
Jeff Bolz
Concedo
Eric Zhang
Rhys-T
Sigbjørn Skjæret
Aldehir Rojas
philip-essential
Piotr Wilkin (ilintar)
Daniel Bevenius
Pascal
Herman Semenoff
Adrien Gallouët
Gilad S.
Aman Gupta
Diego Devesa
Aaron Teo
william pan
ubergarm
Haiyue Wang