b3298fa47a
metal : refactor mat-vec code ( #12569 )
...
* metal : refactor mat-vec code
ggml-ci
* metal : rename all_sum -> sum_all
ggml-ci
* metal : fix comments [no ci]
* metal : fix nr constant [no ci]
* metal : mv q6_K support nr0 > 1
ggml-ci
* metal : reduce register pressure
ggml-ci
* metal : fix typo [no ci]
* metal : reduce register pressure
ggml-ci
2025-03-26 21:38:38 +02:00
2447ad8a98
upgrade to llguidance 0.7.10 ( #12576 )
2025-03-26 11:06:09 -07:00
02082f1519
clip: Fix llama-llava-clip-quantize-cli quantization error under CUDA backend ( #12566 )
...
* [Fix] Compiling clip-quantize-cli and running it in a CUDA environment will cause ggml_fp16_to_fp32 to report an error when trying to access video memory. You need to switch to the CPU backend to run quantize.
After the fix, it will automatically run in the CPU backend and will no longer be bound to CUDA.
* [Fix]Roll back the signature and implementation of clip_model_load, and change the call in clip_model_quantize to clip_init.
2025-03-26 15:06:04 +01:00
4bf675a83d
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# ci/README.md
# docs/build.md
# examples/run/run.cpp
2025-03-26 21:22:57 +08:00
b4a8a5a278
Added CLI chat mode
...
minor cli fixes (+1 squashed commits)
Squashed commits:
[60af39a9] Added CLI chat mode
2025-03-26 21:01:58 +08:00
df4d20cd53
convert : fix squeeze for ssm_conv tensors ( #12573 )
...
* convert : fix squeeze for ssm_conv tensors
* convert : match ssm_conv tensors by type
---------
Co-authored-by: Francis Couture-Harpin <git@compilade.net >
2025-03-26 08:21:05 -04:00
5ed38b6852
ggml : fix MUL_MAT_ID repack with Q8_K ( #12544 )
...
* ggml : fix MUL_MAT_ID repack with Q8_K
ggml-ci
* ggml : improve repack templates
ggml-ci
2025-03-26 13:02:00 +02:00
75e7902789
add localtunnel fallback (+1 squashed commits)
...
Squashed commits:
[ff0a63f6] add localtunnel fallback
2025-03-26 17:35:59 +08:00
fd7855f8f5
doc: [MUSA] minor changes ( #12583 )
...
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2025-03-26 09:09:48 +02:00
53af4dba42
convert: fix Mistral3/Gemma3 model hparams init ( #12571 )
...
* Fix Mistral3/Gemma3 model hparams init
* set positional args correctly
* use existing hparams if passed
2025-03-25 23:03:10 +01:00
ef19c71769
run: de-duplicate fmt and format functions and optimize ( #11596 )
2025-03-25 18:46:11 +01:00
ea358369cc
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# ci/README.md
# ci/run.sh
# docs/backend/CUDA-FEDORA.md
# docs/build.md
# docs/install.md
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cuda/common.cuh
# tests/test-backend-ops.cpp
2025-03-26 00:18:01 +08:00
2bdf1dacff
embeddings done
2025-03-25 22:41:46 +08:00
053b3f9aae
ggml-cpu : update KleidiAI to v1.5.0 ( #12568 )
...
ggml-cpu : bug fix related to KleidiAI LHS packing
Signed-off-by: Dan Johansson <dan.johansson@arm.com >
2025-03-25 13:10:18 +02:00
e2f560175a
SYCL: disable Q4_0 reorder optimization ( #12560 )
...
ggml-ci
2025-03-25 18:40:18 +08:00
36ee06dd2d
docs : add build instructions for KleidiAI ( #12563 )
...
Signed-off-by: Dan Johansson <dan.johansson@arm.com >
2025-03-25 11:35:20 +02:00
3cd3a39532
ci: [MUSA] add CI and update doc ( #12562 )
...
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2025-03-25 09:45:08 +02:00
2d77d88e70
context : fix worst-case reserve outputs ( #12545 )
...
ggml-ci
2025-03-25 09:19:23 +02:00
c95fa362b3
ci: [SYCL] ggml-ci Use main GPU and enable sysman ( #12547 )
2025-03-24 19:35:38 +02:00
2b65ae3029
opencl: simplify kernel embedding logic in cmakefile ( #12503 )
...
Co-authored-by: Max Krasnyansky <quic_maxk@quicinc.com >
2025-03-24 09:20:47 -07:00
82f2654049
wip embeddings model
2025-03-25 00:18:02 +08:00
48d7021c61
CI: fix SYCL build ( #12546 )
2025-03-24 14:58:32 +02:00
3361e2deba
docs: update: improve the Fedoa CUDA guide ( #12536 )
...
* docs: update fedora-cuda guide
- Rename and place into Backend Folder.
- Update Host-Supplied Packages.
- Expand Recommended Users Section.
* docs: improve the flow of CUDA-FEDORA.md
2025-03-24 11:02:26 +00:00
00d53800e0
llama-vocab : add SuperBPE pre-tokenizer ( #12532 )
2025-03-24 11:47:24 +01:00
7ea75035b6
CUDA: Fix clang warnings ( #12540 )
...
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2025-03-24 11:28:34 +01:00
c54f6b7988
mmap : skip resource limit checks on AIX ( #12541 )
2025-03-24 12:17:10 +02:00
3992fb79cc
wip adding embeddings support
2025-03-24 18:01:23 +08:00
9b169a4d4e
vulkan: fix mul_mat_vec failure in backend tests ( #12529 )
...
The OOB calculation could be wrong if the last iteration was during one of
the unrolled loops. Adjust the unrolling counts to avoid this. Add a couple
new backend tests that hit this failure on NVIDIA GPUs.
2025-03-24 07:56:17 +01:00
77f9c6bbe5
server : Add verbose output to OAI compatible chat endpoint. ( #12246 )
...
Add verbose output to server_task_result_cmpl_final::to_json_oaicompat_chat_stream, making it conform with server_task_result_cmpl_final::to_json_oaicompat_chat, as well as the other to_json methods.
2025-03-23 19:30:26 +01:00
b1641ee4a2
allow quant K without quant V but with a warning (+1 squashed commits)
...
Squashed commits:
[45408dd9] allow quant K without quant V but with a warning
2025-03-23 22:56:02 +08:00
18b663d8e4
install : add macports ( #12518 )
...
MacPorts section added
2025-03-23 10:21:48 +02:00
fbdfefe74e
llama : gemma3 : use output tensor if it exists in model weight ( #12506 )
...
* llama : gemma3 : use output tensor if it exists in model weight
* also add to the llm_tensor_names
2025-03-22 23:28:19 +01:00
ba932dfb50
ggml : fix quantized cpy op ( #12310 )
...
* ggml : fix quantized cpy op
ggml-ci
* tests : add cpy tests for all types
ggml-ci
* tests : add BF16 copy tests
ggml-ci
* tests : fix loop for same-type copy
ggml-ci
* tests : add option to permute the dst tensor
ggml-ci
2025-03-22 16:23:26 +02:00
a20a29ddeb
tool calling improved, auto now works
2025-03-22 17:44:55 +08:00
fac63a3d78
musa: refine compute capability ( #12493 )
...
* musa: refine compute capability
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* Address review comments
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
---------
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2025-03-22 10:11:37 +01:00
eddfb43850
vulkan: Optimize mul_mat_vec p021 and nc shaders ( #12505 )
...
* tests: add mul_mat perf/functional tests for p021/nc vulkan shaders
* vulkan: Optimize mul_mat_vec p021 and nc shaders.
These shaders are used in attention calculations, and when the KV cache grows
large they start to dominate the run time. For the nc shader (which is called
with large 'k' dimension), use unrolling and vector loads. For the p021 shader
(which is called with large 'm' and small 'k' dimensions), take advantage of
grouped query attention to reuse loads from the A matrix for the whole group,
and reduce the number of workgroups (too much overhead from tiny dispatches).
Using subgroupAdd in the p021 shader also helps, use that conditionally.
2025-03-22 09:40:11 +01:00
350427dc3a
adjust subprocess timeouts
2025-03-22 11:10:01 +08:00
e31da5861a
1435: add timeout for vulkaninfo ( #1436 )
...
* Fix the Colab PR
* 1435: add timeout for vulkaninfo
There's a bug in vulkaninfo where it can hang, and this will prevent
koboldcpp from starting. This adds a 5 second timeout
* restoring colab.ipynb
* Formatting
---------
Co-authored-by: henk717 <henk@henk.tech >
2025-03-22 11:01:22 +08:00
27bd7b95f5
Merge branch 'upstream' into concedo_experimental
2025-03-22 09:42:33 +08:00
ef90d2b7b1
Revert "Revert "vulkan: matmul dequantization improvements ( #12015 )""
...
This reverts commit 26b71e33b1
2025-03-22 09:41:57 +08:00
4375415b4a
Vulkan: RTE rounding for cpy to quant ( #12480 )
...
* Vulkan: RTE rounding for cpy to quant
Co-Authored-By: Jeff Bolz <jbolz@nvidia.com >
* remove trailing whitespace
* avoid duplicating pipeline_cpy_f32_quant
* fix copypasting issue
* remove duplicated code
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com >
2025-03-21 20:34:50 +01:00
30c42ef5cb
vulkan: workaround for AMD Windows driver 16 bit unpack8 bug ( #12472 )
2025-03-21 20:27:47 +01:00
ae670dbe0e
no repacking for avx2 for kcpp because it breaks 4_0_4_4 quants
2025-03-22 00:33:27 +08:00
7030ebf401
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# docs/backend/SYCL.md
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp
# ggml/src/ggml-sycl/CMakeLists.txt
# tests/test-backend-ops.cpp
2025-03-22 00:32:42 +08:00
c1e58419c7
support for voice cloning is done (+2 squashed commit)
...
Squashed commit:
[e7301628] support for voice cloning is done
[1653c576] wip adding voice cloning
2025-03-21 22:28:59 +08:00
af04481e6b
model : do not repack if a GPU device is present ( #12498 )
...
ggml-ci
2025-03-21 16:14:29 +02:00
960e726077
chore : cleanup llama_model_loader::TENSOR_ usage ( #12492 )
2025-03-21 10:21:36 +01:00
ea1518e839
llama-tts : avoid crashes related to bad model file paths ( #12482 )
2025-03-21 11:12:45 +02:00
1aa87ee53d
[SYCL] Fix build on Windows when ccache enabled ( #9954 ) ( #9976 )
...
* [SYCL] Fix build on Windows when ccache enabled (#9954 )
* take effect only on windows and force it to icl
---------
Co-authored-by: Romain Biessy <romain.biessy@codeplay.com >
2025-03-21 14:58:47 +08:00
9ffcc9e374
sycl: cleanup oneDNN related code ( #12097 )
2025-03-21 10:15:56 +08:00
Georgi Gerganov
Michał Moskal
Ivy233
Concedo
R0CKSTAR
Sigbjørn Skjæret
Eric Curtin
Dan Johansson
Akarshan Biswas
lhez
Tei Home
compilade
Prajwal B Mehendarkar
Jeff Bolz
Marius Gerdes
Lars Sonchocky-Helldorf
Xuan-Son Nguyen
InconsolableCellist
stduhpf
Eve
marcoStocchi
蕭澧邦
Svetlozar Georgiev