86af848153
server: (docs) remove mention about extra_args ( #18262 )
2025-12-22 12:22:01 +01:00
147a521636
tool/ex/tests: consistently free ctx, then model ( #18168 )
2025-12-22 11:00:37 +01:00
7fad4dc0ad
fixed ordering of gpu overhead detection
2025-12-22 17:39:05 +08:00
7c82cad72c
support ovis, added taehv wan embed, fixed compile error (+1 squashed commits)
...
Squashed commits:
[ab71f6d33] support ovis, added taehv wan embed
2025-12-22 17:08:09 +08:00
44ce1a80b3
sd: sync to master-431-23fce0b ( #1893 )
...
* sd: sync to master-427-78e15bd
* add kl_optimal to the available schedulers list
* more robust workaround to avoid stb linkage issues
* sd: sync to master-431-23fce0b
* add TAEHV support and disable TAE if the model isn't found
2025-12-22 15:07:09 +08:00
27c53099f4
adjust scaler checks
2025-12-22 11:50:15 +08:00
a0e4b8c18a
text for maingpu
2025-12-22 11:07:18 +08:00
e1f15b454f
vulkan: Implement set_tensor_async and the event interfaces ( #18047 )
...
The goal is to enable the async loading code paths in
llama_model_loader::load_all_data, originally from #7896 . This works and the
loads themselves are faster, but with host visible vidmem I think the cost of
allocating/mapping vidmem moves and becomes more expensive, and I don't see a
benefit by default. But with GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 I do see a
significant improvement in model loading time.
2025-12-21 21:52:09 +01:00
0e1ccf15c7
llama: fix RPC for -fit on ( #18233 )
2025-12-21 19:33:08 +01:00
5e25ddebff
move copilot instructions to AGENTS.md ( #18259 )
...
* move copilot --> agents.md
* agents: add disclose AI usage
* refine
2025-12-21 19:09:21 +01:00
db4634b9a4
testing new workaround for corrupt scaling
2025-12-21 22:54:40 +08:00
4b899b19dc
fixed save state a bit better
2025-12-21 22:24:13 +08:00
b51e3592ba
revert all tk experiments
2025-12-21 21:10:36 +08:00
80a0269dbe
improve snapshotting for rnn
2025-12-21 21:07:31 +08:00
d577187875
update sdui
2025-12-21 20:35:19 +08:00
fd05c51cec
vulkan: fix im2col overflowing maxworkgroupcount ( #18180 )
2025-12-21 10:32:58 +01:00
b365c3ff01
vulkan/cuda: fix topk_moe with exp_probs_b ( #18071 )
...
I updated test_topk_moe to more closely match llm_graph_context::build_moe_ffn
and added coverage for exp_probs_b and some other missing combinations. This
exposed a bug in both CUDA and Vulkan backends where they were assuming the
input to argsort and the input to get_rows are the same. I'd like to optimize
this graph in another change, but for now just get it functional.
CUDA also had a bug where it got n_experts from the wrong place, leading to
GGML_ASSERT failures in some of the new tests.
2025-12-21 10:27:34 +01:00
cb64222b0c
vulkan: support GGML_UNARY_OP_XIELU ( #18062 )
2025-12-21 10:17:58 +01:00
6eb7081860
vulkan: in graph_optimize, try to group ADD operations ( #18060 )
...
I saw the adds not staying together in the new nemotron 3 nano model.
2025-12-21 10:05:08 +01:00
4117ae5557
Vulkan: some improvement on mul_mat_iq2_xs ( #18031 )
...
* Some improvement on mul_mat_iq2_xs
Refactor calculations for db values and grid data to optimize performance and reduce redundancy.
* Fix trailing whitespace
2025-12-21 09:59:52 +01:00
65e96a2464
docs : fix links in parsing.md ( #18245 )
...
This commit corrects the links in the parsing.md which currently result
in 404 errors.
2025-12-21 09:35:40 +01:00
b4bdc26d64
try tk 8.6.13 instead
2025-12-21 16:17:03 +08:00
6128a91d5a
trying somethning else (+1 squashed commits)
...
Squashed commits:
[bf497e5cf] trying somethning else
2025-12-21 15:38:07 +08:00
fedd529fdc
autofit counts overheads
2025-12-21 14:31:08 +08:00
edfc961ff8
transplanted tk
2025-12-21 13:33:45 +08:00
8b066d9765
don't crash workgroup size
2025-12-21 13:22:34 +08:00
9496bbb808
common : reorganize includes to prioritize vendored deps ( #18222 )
2025-12-20 21:43:21 -06:00
0c7e1d91ea
try a transplanted tk (+1 squashed commits)
...
Squashed commits:
[1eb87e4d1] try a transplanted tk (+1 squashed commits)
Squashed commits:
[094d1566a] try a transplanted tk
2025-12-21 11:31:32 +08:00
ddcb75dd8a
server: add auto-sleep after N seconds of idle ( #18228 )
...
* implement sleeping at queue level
* implement server-context suspend
* add test
* add docs
* optimization: add fast path
* make sure to free llama_init
* nits
* fix use-after-free
* allow /models to be accessed during sleeping, fix use-after-free
* don't allow accessing /models during sleep, it is not thread-safe
* fix data race on accessing props and model_meta
* small clean up
* trailing whitespace
* rm outdated comments
2025-12-21 02:24:42 +01:00
52ab19df63
tests: Avoid floating point precision false positives in SUM ( #17471 )
...
* tests: Avoid floating point precision false positives in SUM
* also apply to test_mean
2025-12-20 13:46:46 -06:00
5182dd64cd
test-backend-ops: improve msvc build time ( #18209 )
2025-12-20 13:45:45 -06:00
10b4f82d44
Added comments explaining thread block size selection logic based on row count and column size, derived from historical commit context ( #18212 )
2025-12-20 19:28:57 +08:00
408616adbd
server : [easy] fix per round speculative decode logging ( #18211 )
...
Currently we always log 0, as we clear slot.drafted before.
To reproduce:
Run llama-server with devstral-2 as main model and devstral-2-small as
md, and verbose logging:
```
% ./build/bin/llama-server -v \
-m ~/llms/Devstral-2-123B-Instruct-2512-UD-Q6_K_XL-00001-of-00003.gguf \
-md ~/llms/Devstral-Small-2-24B-Instruct-2512-UD-Q2_K_XL.gguf \
-c 8192 2> /tmp/llama.cpp.debug
Check the log:
slot update_slots: id 3 | task 0 | accepted 11/0 draft tokens, new
n_tokens = 741
slot update_slots: id 3 | task 0 | accepted 4/0 draft tokens, new
n_tokens = 746
slot update_slots: id 3 | task 0 | accepted 16/0 draft tokens, new
n_tokens = 763
slot update_slots: id 3 | task 0 | accepted 11/0 draft tokens, new
n_tokens = 775
slot update_slots: id 3 | task 0 | accepted 2/0 draft tokens, new
n_tokens = 778
slot update_slots: id 3 | task 0 | accepted 4/0 draft tokens, new
n_tokens = 783
slot update_slots: id 3 | task 0 | accepted 8/0 draft tokens, new
n_tokens = 792
slot update_slots: id 3 | task 0 | accepted 2/0 draft tokens, new
n_tokens = 795
slot update_slots: id 3 | task 0 | accepted 1/0 draft tokens, new
n_tokens = 797
slot update_slots: id 3 | task 0 | accepted 1/0 draft tokens, new
n_tokens = 799
slot update_slots: id 3 | task 0 | accepted 0/0 draft tokens, new
n_tokens = 800
slot update_slots: id 3 | task 0 | accepted 2/0 draft tokens, new
n_tokens = 803
slot update_slots: id 3 | task 0 | accepted 1/0 draft tokens, new
n_tokens = 805
slot update_slots: id 3 | task 0 | accepted 6/0 draft tokens, new
n_tokens = 812
slot update_slots: id 3 | task 0 | accepted 3/0 draft tokens, new
n_tokens = 816
```
After the fix, get correct per round logging:
```
slot update_slots: id 3 | task 0 | accepted 7/8 draft tokens, new
n_tokens = 654
slot update_slots: id 3 | task 0 | accepted 1/2 draft tokens, new
n_tokens = 656
slot update_slots: id 3 | task 0 | accepted 2/16 draft tokens, new
n_tokens = 659
slot update_slots: id 3 | task 0 | accepted 1/16 draft tokens, new
n_tokens = 661
slot update_slots: id 3 | task 0 | accepted 2/16 draft tokens, new
n_tokens = 664
slot update_slots: id 3 | task 0 | accepted 16/16 draft tokens, new
n_tokens = 681
slot update_slots: id 3 | task 0 | accepted 16/16 draft tokens, new
n_tokens = 698
slot update_slots: id 3 | task 0 | accepted 3/4 draft tokens, new
n_tokens = 702
slot update_slots: id 3 | task 0 | accepted 5/12 draft tokens, new
n_tokens = 708
slot update_slots: id 3 | task 0 | accepted 16/16 draft tokens, new
n_tokens = 725
slot update_slots: id 3 | task 0 | accepted 1/1 draft tokens, new
n_tokens = 727
slot update_slots: id 3 | task 0 | accepted 8/16 draft tokens, new
n_tokens = 736
```
2025-12-20 10:57:40 +01:00
9e39a1e6a9
server: support load model on startup, support preset-only options ( #18206 )
...
* server: support autoload model, support preset-only options
* add docs
* load-on-startup
* fix
* Update common/arg.cpp
Co-authored-by: Pascal <admin@serveurperso.com >
---------
Co-authored-by: Pascal <admin@serveurperso.com >
2025-12-20 09:25:27 +01:00
d69db26b44
fix stb multiple impl
2025-12-20 12:05:50 +08:00
17b4b888d0
revert changes for now, we'll do it again next time
2025-12-20 11:02:34 +08:00
c406b9f33e
another font check (+1 squashed commits)
...
Squashed commits:
[6da9493ec] another font check
2025-12-20 09:49:29 +08:00
7304640f72
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build.yml
# .github/workflows/release.yml
# docs/android.md
# docs/backend/hexagon/CMakeUserPresets.json
# examples/llama.android/app/src/main/res/layout/activity_main.xml
# examples/llama.android/app/src/main/res/layout/item_message_assistant.xml
# examples/llama.android/app/src/main/res/layout/item_message_user.xml
# examples/model-conversion/scripts/causal/run-org-model.py
# examples/model-conversion/scripts/utils/common.py
# ggml/CMakeLists.txt
# ggml/src/ggml-hexagon/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# tests/test-arg-parser.cpp
# tools/server/README.md
2025-12-20 09:32:06 +08:00
714ab0682e
Revert "Revert "llama : Async DirectIO model loading on Linux ( #18012 )""
...
This reverts commit a45fc5ee88
2025-12-20 09:25:10 +08:00
710d88687b
try a more modern way of fixing font since xft is dead
2025-12-20 09:24:30 +08:00
74e05131e9
ci : remove non-windows zip artifacts ( #18201 )
...
* remove non-windows zip artifacts
* add cuda dll links
2025-12-19 22:29:46 +01:00
f74747d886
ci : only save ccache on master ( #18207 )
2025-12-19 22:29:37 +01:00
ce734a8a2f
ggml-hexagon: Implement true Q8_0 quantization on Hexagon NPU for more accurate mixed-precision matmul operations ( #17977 )
...
* feat: implement real Q8_0
* feat: adding cmake option for configuring FP32 quantize group size
* typo: set() shall be used
---------
Co-authored-by: ngdxzy <zhenyu_xu@uri.edu >
2025-12-19 09:42:28 -08:00
14931a826e
arg: fix order to use short form before long form ( #18196 )
...
* arg: fix order to use short form before long form
* arg: update doc
* arg: update test-arg-parser
* arg: address review feedback from ngxson
simplified to check first.length() <= last.length() only
fixed: --sampler-seq, --rerank, --draft ordering
note: middle positions in 3+ arg sets are not verified
* arg: update doc
2025-12-19 18:01:56 +01:00
9458e08346
fixed https://github.com/LostRuins/koboldcpp/issues/1892
2025-12-19 22:52:39 +08:00
f99ef53d2a
llama : Changing off_t to size_t for Windows ( #18204 )
2025-12-19 16:42:46 +02:00
9ea153c14c
try a more modern way of fixing font since xft is dead
2025-12-19 21:50:17 +08:00
9ea6a3fa62
add download page
2025-12-19 19:59:12 +08:00
cc0a04343e
server: friendlier error msg when ctx < input ( #18174 )
...
* llama-server: friendlier error msg when ctx < input
This PR adds formatted strings to the server's send_error function
* llama-server: use string_format inline
* fix test
2025-12-19 12:10:00 +01:00
98c1c7a7bf
presets: refactor, allow cascade presets from different sources, add global section ( #18169 )
...
* presets: refactor, allow cascade presets from different sources
* update docs
* fix neg arg handling
* fix empty mmproj
* also filter out server-controlled args before to_ini()
* skip loading custom_models if not specified
* fix unset_reserved_args
* fix crash on windows
2025-12-19 12:08:20 +01:00
Xuan-Son Nguyen
Johannes Gäßler
Concedo
Wagner Bruna
Jeff Bolz
lovedheart
Daniel Bevenius
Aldehir Rojas
Aadeshveer Singh
Oleksandr Kuvshynov
Sigbjørn Skjæret
Alfred
Pascal
Julius Tischbein
Aman Gupta