Public/koboldcpp
1
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-09-18 08:44:57 +02:00
Code Issues Packages Projects Releases Wiki Activity
8,352 Commits 3 Branches 254 Tags
0272ecf2d514b42111eaaafd1e68eaa7d5225dff
Commit Graph

8352 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi Gerganov 029c693fdc sync : ggml 
ggml-ci
 2025-03-27 10:09:29 +02:00
Georgi Gerganov 771d84371c scripts : update sync + fix cmake merge 
ggml-ci
 2025-03-27 10:09:29 +02:00
Georgi Gerganov df0665a483 sync : ggml 
ggml-ci
 2025-03-27 09:04:38 +02:00
Georgi Gerganov 0306aad1ca cmake : sync/merge PowerPC build commands (#0) 2025-03-27 09:04:38 +02:00
amritahs-ibm c7b43ab608 llamafile : ppc64le MMA implementation for Q4_0. (#12489) 
This change upstreams llamafile's cpu matrix
multiplication kernels for ppc64le ISA using MMA
builtins. This patch handles matrix multiplication
between quantised datatypes, block_q4_0 and
block_q8_0.

This change results in 5% - 50% improvement
in total speed(ie all tokens/total time), across
various batch sizes.

The patch is tested with Meta-Lllama-3-8B,
Mistral-7B, Llama-2-7B-chat-hf models on a
IBM POWER10 machine.

Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
 2025-03-27 08:51:47 +02:00
xctan 24feaec057 ggml : riscv: add 128-bit RVV support (#12530) 
* ggml : add 128-bit RVV support

* ggml : revert to old RVV 256+ q2_K, q3_K, q4_K, q6_K impl

* remove trailing whitespaces

* restructure vector length selection code
 2025-03-27 08:38:34 +02:00
Georgi Gerganov f28bc4c286 llama : make loras compatible with repacking (#12593) 
* llama : make loras compatible with repacking

ggml-ci

* cont : simplify

ggml-ci

* cont : add TODO [no ci]
 2025-03-27 08:24:10 +02:00
Concedo 6a709be50a replace deprecated 2025-03-27 10:27:20 +08:00
Akarshan Biswas f17a3bb4e8 SYCL: implement memset ggml backend buffer interface (#12580) 
* SYCL: implement memset ggml backend buffer interface

* use GGML_ABORT macro

* Do not wait for all queues to finish for memset operation
 2025-03-27 09:46:00 +08:00
Slobodan Josic bd40678df7 HIP: Add support for RDNA4 targets (#12372) 2025-03-26 23:46:30 +01:00
Georgi Gerganov b3298fa47a metal : refactor mat-vec code (#12569) 
* metal : refactor mat-vec code

ggml-ci

* metal : rename all_sum -> sum_all

ggml-ci

* metal : fix comments [no ci]

* metal : fix nr constant [no ci]

* metal : mv q6_K support nr0 > 1

ggml-ci

* metal : reduce register pressure

ggml-ci

* metal : fix typo [no ci]

* metal : reduce register pressure

ggml-ci
 2025-03-26 21:38:38 +02:00
Michał Moskal 2447ad8a98 upgrade to llguidance 0.7.10 (#12576) 2025-03-26 11:06:09 -07:00
Ivy233 02082f1519 clip: Fix llama-llava-clip-quantize-cli quantization error under CUDA backend (#12566) 
* [Fix] Compiling clip-quantize-cli and running it in a CUDA environment will cause ggml_fp16_to_fp32 to report an error when trying to access video memory. You need to switch to the CPU backend to run quantize.
After the fix, it will automatically run in the CPU backend and will no longer be bound to CUDA.

* [Fix]Roll back the signature and implementation of clip_model_load, and change the call in clip_model_quantize to clip_init.
 2025-03-26 15:06:04 +01:00
Concedo 4bf675a83d Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ci/README.md
#	docs/build.md
#	examples/run/run.cpp
 2025-03-26 21:22:57 +08:00
Concedo b4a8a5a278 Added CLI chat mode 
minor cli fixes (+1 squashed commits)

Squashed commits:

[60af39a9] Added CLI chat mode
 2025-03-26 21:01:58 +08:00
Georgi Gerganov df4d20cd53 convert : fix squeeze for ssm_conv tensors (#12573) 
* convert : fix squeeze for ssm_conv tensors

* convert : match ssm_conv tensors by type

---------

Co-authored-by: Francis Couture-Harpin <git@compilade.net>
 2025-03-26 08:21:05 -04:00
Georgi Gerganov 5ed38b6852 ggml : fix MUL_MAT_ID repack with Q8_K (#12544) 
* ggml : fix MUL_MAT_ID repack with Q8_K

ggml-ci

* ggml : improve repack templates

ggml-ci
 2025-03-26 13:02:00 +02:00
Concedo 75e7902789 add localtunnel fallback (+1 squashed commits) 
Squashed commits:

[ff0a63f6] add localtunnel fallback
 2025-03-26 17:35:59 +08:00
R0CKSTAR fd7855f8f5 doc: [MUSA] minor changes (#12583) 
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
 2025-03-26 09:09:48 +02:00
Sigbjørn Skjæret 53af4dba42 convert: fix Mistral3/Gemma3 model hparams init (#12571) 
* Fix Mistral3/Gemma3 model hparams init

* set positional args correctly

* use existing hparams if passed
 2025-03-25 23:03:10 +01:00
Eric Curtin ef19c71769 run: de-duplicate fmt and format functions and optimize (#11596) 2025-03-25 18:46:11 +01:00
Concedo ea358369cc Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ci/README.md
#	ci/run.sh
#	docs/backend/CUDA-FEDORA.md
#	docs/build.md
#	docs/install.md
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-cuda/common.cuh
#	tests/test-backend-ops.cpp
 2025-03-26 00:18:01 +08:00
Concedo 2bdf1dacff embeddings done 2025-03-25 22:41:46 +08:00
Dan Johansson 053b3f9aae ggml-cpu : update KleidiAI to v1.5.0 (#12568) 
ggml-cpu : bug fix related to KleidiAI LHS packing

Signed-off-by: Dan Johansson <dan.johansson@arm.com>
 2025-03-25 13:10:18 +02:00
Akarshan Biswas e2f560175a SYCL: disable Q4_0 reorder optimization (#12560) 
ggml-ci
 2025-03-25 18:40:18 +08:00
Dan Johansson 36ee06dd2d docs : add build instructions for KleidiAI (#12563) 
Signed-off-by: Dan Johansson <dan.johansson@arm.com>
 2025-03-25 11:35:20 +02:00
R0CKSTAR 3cd3a39532 ci: [MUSA] add CI and update doc (#12562) 
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
 2025-03-25 09:45:08 +02:00
Georgi Gerganov 2d77d88e70 context : fix worst-case reserve outputs (#12545) 
ggml-ci
 2025-03-25 09:19:23 +02:00
Akarshan Biswas c95fa362b3 ci: [SYCL] ggml-ci Use main GPU and enable sysman (#12547) 2025-03-24 19:35:38 +02:00
lhez 2b65ae3029 opencl: simplify kernel embedding logic in cmakefile (#12503) 
Co-authored-by: Max Krasnyansky <quic_maxk@quicinc.com>
 2025-03-24 09:20:47 -07:00
Concedo 82f2654049 wip embeddings model 2025-03-25 00:18:02 +08:00
Akarshan Biswas 48d7021c61 CI: fix SYCL build (#12546) 2025-03-24 14:58:32 +02:00
Tei Home 3361e2deba docs: update: improve the Fedoa CUDA guide (#12536) 
* docs: update fedora-cuda guide

- Rename and place into Backend Folder.
- Update Host-Supplied Packages.
- Expand Recommended Users Section.

* docs: improve the flow of CUDA-FEDORA.md
 2025-03-24 11:02:26 +00:00
compilade 00d53800e0 llama-vocab : add SuperBPE pre-tokenizer (#12532) 2025-03-24 11:47:24 +01:00
R0CKSTAR 7ea75035b6 CUDA: Fix clang warnings (#12540) 
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
 2025-03-24 11:28:34 +01:00
Prajwal B Mehendarkar c54f6b7988 mmap : skip resource limit checks on AIX (#12541) 2025-03-24 12:17:10 +02:00
Concedo 3992fb79cc wip adding embeddings support 2025-03-24 18:01:23 +08:00
Jeff Bolz 9b169a4d4e vulkan: fix mul_mat_vec failure in backend tests (#12529) 
The OOB calculation could be wrong if the last iteration was during one of
the unrolled loops. Adjust the unrolling counts to avoid this. Add a couple
new backend tests that hit this failure on NVIDIA GPUs.
 2025-03-24 07:56:17 +01:00
Marius Gerdes 77f9c6bbe5 server : Add verbose output to OAI compatible chat endpoint. (#12246) 
Add verbose output to server_task_result_cmpl_final::to_json_oaicompat_chat_stream, making it conform with server_task_result_cmpl_final::to_json_oaicompat_chat, as well as the other to_json methods.
 2025-03-23 19:30:26 +01:00
Concedo b1641ee4a2 allow quant K without quant V but with a warning (+1 squashed commits) 
Squashed commits:

[45408dd9] allow quant K without quant V but with a warning
 2025-03-23 22:56:02 +08:00
Lars Sonchocky-Helldorf 18b663d8e4 install : add macports (#12518) 
MacPorts section added
 2025-03-23 10:21:48 +02:00
Xuan-Son Nguyen fbdfefe74e llama : gemma3 : use output tensor if it exists in model weight (#12506) 
* llama : gemma3 : use output tensor if it exists in model weight

* also add to the llm_tensor_names
 2025-03-22 23:28:19 +01:00
Georgi Gerganov ba932dfb50 ggml : fix quantized cpy op (#12310) 
* ggml : fix quantized cpy op

ggml-ci

* tests : add cpy tests for all types

ggml-ci

* tests : add BF16 copy tests

ggml-ci

* tests : fix loop for same-type copy

ggml-ci

* tests : add option to permute the dst tensor

ggml-ci
 2025-03-22 16:23:26 +02:00
Concedo a20a29ddeb tool calling improved, auto now works 2025-03-22 17:44:55 +08:00
R0CKSTAR fac63a3d78 musa: refine compute capability (#12493) 
* musa: refine compute capability

Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

* Address review comments

Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

---------

Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
 2025-03-22 10:11:37 +01:00
Jeff Bolz eddfb43850 vulkan: Optimize mul_mat_vec p021 and nc shaders (#12505) 
* tests: add mul_mat perf/functional tests for p021/nc vulkan shaders

* vulkan: Optimize mul_mat_vec p021 and nc shaders.

These shaders are used in attention calculations, and when the KV cache grows
large they start to dominate the run time. For the nc shader (which is called
with large 'k' dimension), use unrolling and vector loads. For the p021 shader
(which is called with large 'm' and small 'k' dimensions), take advantage of
grouped query attention to reuse loads from the A matrix for the whole group,
and reduce the number of workgroups (too much overhead from tiny dispatches).

Using subgroupAdd in the p021 shader also helps, use that conditionally.
 2025-03-22 09:40:11 +01:00
Concedo 350427dc3a adjust subprocess timeouts 2025-03-22 11:10:01 +08:00
InconsolableCellist e31da5861a 1435: add timeout for vulkaninfo (#1436) 
* Fix the Colab PR

* 1435: add timeout for vulkaninfo

There's a bug in vulkaninfo where it can hang, and this will prevent
koboldcpp from starting. This adds a 5 second timeout

* restoring colab.ipynb

* Formatting

---------

Co-authored-by: henk717 <henk@henk.tech>
 2025-03-22 11:01:22 +08:00
Concedo 27bd7b95f5 Merge branch 'upstream' into concedo_experimental 2025-03-22 09:42:33 +08:00
Concedo ef90d2b7b1 Revert "Revert "vulkan: matmul dequantization improvements (#12015)"" 
This reverts commit 26b71e33b1.
 2025-03-22 09:41:57 +08:00