8e23a087e7
updated readme, memory detection prints
2025-04-08 20:23:52 +08:00
11c4e7c2c4
automatic memory detection for vulkan
2025-04-07 22:56:12 +08:00
a3f7de7142
fixed outetts docs
2025-04-07 21:31:43 +08:00
6e42e673c6
attempt to fall back to system glslc
2025-04-07 00:33:52 +08:00
5edbacdd0e
fix tools (+3 squashed commit)
...
Squashed commit:
[95a489ee] fix tools build
[1d3d3451] add accelerate
[2837705c
2025-04-06 21:30:48 +08:00
11f993ca10
added flag to adjust max request size
2025-04-06 00:13:00 +08:00
8415cac7ac
add vk shaders source (+1 squashed commits)
...
Squashed commits:
[45359f49] add vk shaders source
2025-04-05 22:45:18 +08:00
c92b0cf1a4
early merge https://github.com/ggml-org/llama.cpp/pull/12767 to test
2025-04-05 18:48:57 +08:00
65cd25d3a1
some data sanitization
2025-04-05 18:42:26 +08:00
06149f6a6c
updated lite
2025-04-05 18:14:35 +08:00
93a226d9e4
added prefix for llava, reverted system role in template as it degreaded gemma3. truncated debug logs
2025-04-05 18:06:41 +08:00
b3143384b4
larger warmup batch
2025-04-05 10:57:04 +08:00
59c02aa1a6
embeddings model colab
2025-04-05 10:30:47 +08:00
34ddd874fe
try containerized ci (+3 squashed commit)
...
Squashed commit:
[f0600744fe11073c0c7f8a1d
2025-04-05 01:51:03 +08:00
3105eeec93
added queuing for sdui
2025-04-04 18:42:32 +08:00
57e12b73af
try containerized ci (+1 squashed commits)
...
Squashed commits:
[fc53c200] try containerized ci (+1 squashed commits)
Squashed commits:
[4b48b0d5] try containerized ci
2025-04-04 17:19:27 +08:00
4e740311fe
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# ci/run.sh
# docs/backend/SYCL.md
# docs/build.md
# ggml/src/ggml-vulkan/CMakeLists.txt
# ggml/src/ggml-vulkan/vulkan-shaders/CMakeLists.txt
# tests/test-chat-template.cpp
2025-04-04 15:07:47 +08:00
c48a4a73d4
try fix file open
2025-04-04 14:38:17 +08:00
43e9b049d6
another silly bug silly silly silly (tavern)
2025-04-04 14:16:42 +08:00
74d4f5b041
vulkan: Hybrid waitForFences/getFenceStatus to reduce fence latency ( #12630 )
...
There seems to be a bubble waking up from waitForFences, which costs a few
percent performance and also increased variance in performance. This change
inserts an "almost_ready" fence when the graph is about 80% complete and we
waitForFences for the almost_ready fence and then spin (with _mm_pauses) waiting
for the final fence to be signaled.
2025-04-04 07:54:35 +02:00
35e592eb30
vulkan: set cmake minimum and project name in vulkan-shaders ( #12744 )
2025-04-04 07:53:20 +02:00
7d7b1bafa7
opencl: update doc for OpenCL ( #12702 )
...
* opencl: add OpenCL to build.md
* opencl: remove fixed issue/TODO
* opencl: add link to OPENCL.md
* opencl: update doc - refine tools requirement for Windows 11 arm64
2025-04-03 22:18:17 -07:00
c262beddf2
CUDA: Prefer vector flash decoding kernel for Gemma models ( #12738 )
...
* Prefer vector flash decoding kernel for Gemma models
Vector flash decoding kernel was not being picked for models with head dimension 256. Gemma models are in this category.
Removing this limit improves e2e performance by upto 12% in gen phase throughput for Gemm models.
* Update ggml/src/ggml-cuda/fattn.cu
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2025-04-03 18:20:29 +02:00
5dd5d1ab00
vocab : use string_view::find() to avoid unnecessary looking up beyond the fragment range ( #12706 )
2025-04-03 18:32:54 +03:00
1c059995e0
vulkan: Fix missing cmake logic for dot product extension ( #12721 )
2025-04-03 10:08:26 -05:00
59b7796b96
binops does not need clblast anymore
2025-04-03 23:06:19 +08:00
47768b2780
update lite
2025-04-03 21:05:39 +08:00
8c74520586
added NO_VULKAN_EXTENSIONS flag to disable dp4a and coopmat if needed
2025-04-03 20:51:17 +08:00
07a96d63fa
try to ensure correct file extension
2025-04-03 20:13:53 +08:00
2004644b7a
ci : add env variable in ggml-ci and document the same in SYCL.md ( #12736 )
2025-04-03 15:12:39 +03:00
5f696e88e0
sync : minja (inclusionAI/Ling) and update tests ( #12699 )
...
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2025-04-03 13:51:35 +02:00
6e086bd309
fixed savedatafile bug, try remove unneeded old clblast code path
2025-04-03 19:11:27 +08:00
103d60ed2c
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# common/common.cpp
# examples/batched-bench/batched-bench.cpp
# examples/batched/batched.cpp
# examples/export-lora/export-lora.cpp
# examples/gritlm/gritlm.cpp
# examples/parallel/parallel.cpp
# examples/passkey/passkey.cpp
# examples/speculative-simple/speculative-simple.cpp
# examples/speculative/speculative.cpp
# ggml/src/ggml-cann/CMakeLists.txt
# ggml/src/ggml-cann/acl_tensor.cpp
# ggml/src/ggml-cann/acl_tensor.h
# ggml/src/ggml-cann/aclnn_ops.cpp
# ggml/src/ggml-cann/aclnn_ops.h
# ggml/src/ggml-vulkan/CMakeLists.txt
# tests/test-arg-parser.cpp
# tests/test-backend-ops.cpp
2025-04-03 18:57:49 +08:00
193c3e03a6
fix MUSA compiler warning ( #12704 )
...
* fix MUSA compiler warning
* replace (void) with GGML_UNUSED
2025-04-03 09:32:55 +02:00
65cfe136a0
CANN: Support operator SIN COS ARGMAX ( #12709 )
...
* [CANN]support sin cos argmax
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
* [CANN]codestyle adjustment
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
* [CANN]Remove redundant code
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
---------
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
Co-authored-by: noemotiovon <noemotiovon@gmail.com >
2025-04-03 15:18:08 +08:00
3f9da22c2b
Simplify and improve CUDA graphs through use of indirect copy pointers ( #9017 )
...
* CUDA: Simplify and improve CUDA graphs through use of indirect copy pointers
Previously there was complexity in the CUDA graphs implementation due
frequently changing parameters to copy kernels associated with K and V
cache pointers. This patch simplifies by using indirection to avoid
such parameters frequently changing, avoiding the need for frequent
graph updates.
Fixes #12152
* Addressed comments
* fix HIP builds
* properly sync to stream
* removed ggml_cuda_cpy_fn_ptrs
* move stream sync before free
* guard to only use indirection with graphs
* style fixes
* check for errors
---------
Co-authored-by: slaren <slarengh@gmail.com >
2025-04-03 03:31:15 +02:00
2a0dc97e56
CANN: Fix failed test cases ( #12708 )
...
* CANN: Fix memory waste in aclnn_tensor
* CANN: fix backend ops fail
* CANN: fix acl_tensor memory alloc.
* CANN: format
* CANN: remove trailing whitespace
2025-04-03 08:49:51 +08:00
97a20c012b
opencl: use max_alloc_size in backend ctx instead of querying again ( #12705 )
2025-04-02 17:01:42 -07:00
f01bd02376
vulkan: Implement split_k for coopmat2 flash attention. ( #12627 )
...
When using group query attention, we have one workgroup per KV batch and this
can be very few workgroups (e.g. just 8 in some models). Enable split_k to
spread the work across SMs. This helps a lot when the KV cache is large.
2025-04-02 14:25:08 -05:00
6f3bd38640
cmake: remove caching from vulkan coopmat checks ( #12719 )
2025-04-02 14:56:26 -03:00
be0a0f8cae
vulkan: Implement grouped query attention in the coopmat2 FA shader ( #12559 )
...
When adjacent batches of Q share the same batches of K/V, batch them into
the same workgroup. For example, when:
dst(128,32,1,1) = FA(q(128,1,32,1), k(128,16640,8,1), v(128,16640,8,1))
previously we would run 32 workgroups computing 1 result each, now we will
run 8 workgroups computing 4 results each.
This doesn't directly translate to better performance (at least when you have
>=32 SMs), but in a subsequent change I'll enable split_k which will scale much
better with 4x fewer workgroups.
2025-04-02 19:40:32 +02:00
92e3006bb6
Vulkan: Fix mmq int dot float cache size ( #12722 )
2025-04-02 19:12:30 +02:00
833e2b7409
model : print tensor size during load ( #12711 )
...
* model : print tensor size during load
* cont : fix units MB -> MiB
Co-authored-by: Diego Devesa <slarengh@gmail.com >
---------
Co-authored-by: Diego Devesa <slarengh@gmail.com >
2025-04-02 16:38:54 +03:00
e0e912f49b
llama : add option to override model tensor buffers ( #11397 )
...
* llama : add option to override tensor buffers
* ggml : fix possible underflow in ggml_nbytes
2025-04-02 14:52:01 +02:00
a10b36c91a
llama : refactor kv cache guard ( #12695 )
...
* llama : refactor kv cache guard
ggml-ci
* cont : fix comment [no ci]
* llama : fix kv_cache restore logic
ggml-ci
* context : simplify kv cache updates
ggml-ci
* cont : better name [no ci]
* llama : fix llama_decode return code when could not find KV slot
ggml-ci
* context : change log err -> warn [no ci]
* kv-cache : add comment + warning
2025-04-02 14:32:59 +03:00
7f1003be44
warning for max tokens being too high
2025-04-02 18:58:38 +08:00
83a88bd6af
vocab : BailingMoE : change possessive quantifiers to greedy ( #12677 )
2025-04-02 11:21:48 +02:00
42eb248f46
common : remove json.hpp from common.cpp ( #12697 )
...
* common : remove json.hpp from common.cpp
* fix comment
2025-04-02 09:58:34 +02:00
9bacd6b374
[CANN] get_rows and dup optimization ( #12671 )
...
* [CANN]get_rows and dup optimization.
Co-authored-by: hipudding <huafengchun@gmail.com >
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
* [CANN]GET_ROWS and CPY/DUP optimization
Co-authored-by: hipudding <huafengchun@gmail.com >
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
* [CANN]code style adjustment
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
* [CANN]code style adjustment
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
* [CANN]code style adjustment
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
* [CANN]code style adjustment
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
---------
Signed-off-by: noemotiovon <noemotiovon@gmail.com >
Co-authored-by: noemotiovon <noemotiovon@gmail.com >
Co-authored-by: hipudding <huafengchun@gmail.com >
2025-04-02 15:22:13 +08:00
669311365c
fixed gemma system prompt
2025-04-02 13:58:51 +08:00
Concedo
Jeff Bolz
lhez
Gaurav Garg
yumeyao
Atharva Dubey
R0CKSTAR
a3sh
Chenguang Li
Alan Gray
hipudding
bandoti
0cc4m
Georgi Gerganov
Diego Devesa
Sigbjørn Skjæret
Xuan-Son Nguyen