Public/koboldcpp
1
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-09-19 09:15:18 +02:00
Code Issues Packages Projects Releases Wiki Activity
8,898 Commits 3 Branches 254 Tags
8192cd6747f2a1a07e29df9972f3cd78a0c8e178
Commit Graph

723 Commits

Author SHA1 Message Date
Diego Devesa 6adc3c3ebc llama : add thread safety test (#14035) 
* llama : add thread safety test

* llamafile : remove global state

* llama : better LLAMA_SPLIT_MODE_NONE logic

when main_gpu < 0 GPU devices are not used

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-06-16 08:11:43 -07:00
Đinh Trọng Huy ad590be98c model : add NeoBERT (#14164) 
* convert neobert model to gguf

* add inference graph

* fix flake8 lint

* followed reviewer suggestions

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* follow reviewers suggestions

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* override NeoBERT feed-forward length

---------

Co-authored-by: dinhhuy <huy.dinh@brains-tech.co.jp>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-06-16 14:53:41 +02:00
Georgi Gerganov d3e64b9f49 llama : rework embeddings logic (#14208) 
* llama : rework embeddings logic

ggml-ci

* cont : fix rerank

ggml-ci

* cont : engrish [no ci]

* cont : fix rerank

ggml-ci

* server : support both embeddings and completions with single model

ggml-ci

* cont : avoid embeddings_org

ggml-ci
 2025-06-16 14:14:00 +03:00
Bartowski d7da8dc83a model : Add support for Arcee AI's upcoming AFM model (#14185) 
* Add Arcee AFM support

* Add draft update code

* Fix linter and update URL, may still not be final

* Update src/llama-model.cpp

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>

* Remote accidental blank line

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
 2025-06-16 01:04:06 +02:00
Ed Addario 30e5b01de2 quantize : change int to unsigned int for KV overrides (#14197) 2025-06-15 18:53:45 +02:00
Georgi Gerganov 5fce5f948d kv-cache : fix use-after-move of defrag info (#14189) 
ggml-ci
 2025-06-15 10:52:11 +03:00
Mikko Juola 9ae4143bc6 model : add dots.llm1 architecture support (#14044) (#14118) 
Adds:

* Dots1Model to convert_hf_to_gguf.py

* Computation graph code to llama-model.cpp

* Chat template to llama-chat.cpp to detect this model's template.

---

The model is called "dots.llm1" (I decided to shorten it to dots1 or
DOTS1 in the code generally) architecture.

The only models that exist as of writing of this commit that follow this
architecture are "dots.llm1.inst" and "dots.llm1.base" from here:

* https://huggingface.co/rednote-hilab/dots.llm1.inst

* https://huggingface.co/rednote-hilab/dots.llm1.base

The model architecture is a combination of Qwen and Deepseek parts, as
seen here:

https://github.com/huggingface/transformers/blob/ffe12627b4e84489d2ab91dd0ec00614855edc79/src/transformers/models/dots1/modular_dots1.py
 2025-06-15 09:52:06 +02:00
Georgi Gerganov c311ac664d cparams : rename LLAMA_MAX_PARALLEL_SEQUENCES to LLAMA_MAX_SEQ (#14188) 
ggml-ci
 2025-06-15 10:08:58 +03:00
Georgi Gerganov b9912ac570 batch : auto-gen positions + verify multi-sequence input (#14177) 
* batch : verify multi-sequence input batches

ggml-ci

* cont : auto-gen positions + verify multi-seq input

ggml-ci

* cont : first print debug info, then perform validation

ggml-ci

* cont : fix position auto-gen + add comments

ggml-ci
 2025-06-15 09:18:37 +03:00
Concedo 5f9e96e82d Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/intel.Dockerfile
#	CMakeLists.txt
#	README.md
#	common/CMakeLists.txt
#	docs/multimodal.md
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-metal/CMakeLists.txt
#	ggml/src/ggml-sycl/CMakeLists.txt
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/cpy.cpp
#	ggml/src/ggml-sycl/gemm.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	src/llama-context.cpp
 2025-06-14 09:05:45 +08:00
Concedo 69e4a32ca2 Merge commit 'd4e0d95cf581f50c9a21d06eaecae2dd580076bd' into concedo_experimental 
# Conflicts:
#	.github/workflows/build.yml
#	common/CMakeLists.txt
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-rpc/ggml-rpc.cpp
#	scripts/sync-ggml.last
#	tests/CMakeLists.txt
 2025-06-14 01:58:53 +08:00
Georgi Gerganov fb85a288d7 vocab : fix build (#14175) 
ggml-ci
 2025-06-13 20:03:05 +03:00
Guy Goldenberg 3cfbbdb44e Merge commit from fork 
* vocab : prevent integer overflow during load

* Add static cast and GGML_ABORT

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-06-13 19:20:25 +03:00
Georgi Gerganov 80709b70a2 batch : add LLAMA_BATCH_DEBUG environment variable (#14172) 
* batch : add LLAMA_BATCH_DEBUG environment variable

ggml-ci

* cont : improve seq_id display
 2025-06-13 18:35:00 +03:00
Concedo 4204f111f7 Merge commit '8f47e25f56e9792093b7497c68e9f80bab82ed19' into concedo_experimental 
# Conflicts:
#	.github/labeler.yml
#	.github/workflows/build-linux-cross.yml
#	docs/backend/CANN.md
#	examples/batched.swift/Sources/main.swift
#	examples/embedding/embedding.cpp
#	examples/gritlm/gritlm.cpp
#	examples/llama.android/llama/src/main/cpp/llama-android.cpp
#	examples/llama.swiftui/llama.cpp.swift/LibLlama.swift
#	examples/lookahead/lookahead.cpp
#	examples/lookup/lookup.cpp
#	examples/parallel/parallel.cpp
#	examples/passkey/passkey.cpp
#	examples/retrieval/retrieval.cpp
#	examples/save-load-state/save-load-state.cpp
#	examples/simple-chat/simple-chat.cpp
#	examples/speculative-simple/speculative-simple.cpp
#	examples/speculative/speculative.cpp
#	ggml/src/ggml-cann/common.h
#	ggml/src/ggml-cann/ggml-cann.cpp
#	ggml/src/ggml-sycl/convert.cpp
#	ggml/src/ggml-sycl/cpy.cpp
#	ggml/src/ggml-sycl/dequantize.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	ggml/src/ggml-sycl/vecdotq.hpp
#	tools/batched-bench/batched-bench.cpp
#	tools/cvector-generator/cvector-generator.cpp
#	tools/imatrix/imatrix.cpp
#	tools/llama-bench/llama-bench.cpp
#	tools/perplexity/perplexity.cpp
#	tools/run/run.cpp
 2025-06-13 22:05:03 +08:00
Georgi Gerganov 60c666347b batch : rework llama_batch_allocr (#14153) 
* batch : rework llama_batch_allocr

ggml-ci

* cont : move validation inside class

ggml-ci

* cont : move output counting to class

ggml-ci

* cont : minor

ggml-ci

* batch : add TODOs

ggml-ci
 2025-06-13 13:47:55 +03:00
Đinh Trọng Huy d714dadb57 pooling : make cls_b and cls_out_b optional (#14165) 
Co-authored-by: dinhhuy <huy.dinh@brains-tech.co.jp>
 2025-06-13 11:34:08 +03:00
Georgi Gerganov c33fe8b8c4 vocab : prevent heap overflow when vocab is too small (#14145) 
ggml-ci
 2025-06-13 08:03:54 +03:00
Georgi Gerganov f6e1a7aa87 context : simplify output counting logic during decode (#14142) 
* batch : remove logits_all flag

ggml-ci

* context : simplify output counting logic during decode

ggml-ci

* cont : fix comments
 2025-06-12 11:50:01 +03:00
Georgi Gerganov c3ee46fab4 batch : remove logits_all flag (#14141) 
ggml-ci
 2025-06-12 11:49:26 +03:00
Georgi Gerganov 9596506965 kv-cache : fix split_equal handling in unified implementation (#14130) 
ggml-ci
 2025-06-12 10:02:15 +03:00
compilade a20b2b05bc context : round n_tokens to next multiple of n_seqs when reserving (#14140) 
This fixes RWKV inference which otherwise failed
when the worst case ubatch.n_seq_tokens rounded to 0.
 2025-06-12 02:56:04 -04:00
Georgi Gerganov 89a184fa71 kv-cache : relax SWA masking condition (#14119) 
ggml-ci
 2025-06-11 16:48:45 +03:00
Georgi Gerganov 7ae2932116 kv-cache : add LLAMA_KV_CACHE_DEBUG environment variable (#14121) 2025-06-11 12:52:45 +03:00
compilade dad5c44398 kv-cache : avoid modifying recurrent cells when setting inputs (#13834) 
* kv-cache : avoid modifying recurrent cells when setting inputs

* kv-cache : remove inp_s_mask

It was replaced with equivalent and simpler functionality
with rs_z (the first zeroed state) and the already-existing inp_s_copy.

* kv-cache : fix non-consecutive token pos warning for recurrent models

The problem was apparently caused by how the tail cells were swapped.

* graph : simplify logic for recurrent state copies

* kv-cache : use cell without src refs for rs_z in recurrent cache

* llama-graph : fix recurrent state copy

The `state_copy` shuffle assumes everything is moved at once,
which is not true when `states_extra` is copied back to the cache
before copying the range of states between `head` and `head + n_seqs`.
This is only a problem if any of the cells in [`head`, `head + n_seqs`)
have an `src` in [`head + n_seqs`, `head + n_kv`),
which does happen when `n_ubatch > 1` in the `llama-parallel` example.

Changing the order of the operations avoids the potential overwrite
before use, although when copies are avoided (like with Mamba2),
this will require further changes.

* llama-graph : rename n_state to state_size in build_recurrent_state

This naming should reduce confusion between the state size
and the number of states.
 2025-06-10 18:20:14 -04:00
Sigbjørn Skjæret 3678b838bb llama : support GEGLU for jina-bert-v2 (#14090) 2025-06-10 18:02:08 +02:00
Georgi Gerganov 40cbf571c9 kv-cache : fix shift and defrag logic (#14081) 
* kv-cache : fix shift

ggml-ci

* cont : reset shift[i]

ggml-ci

* cont : fix defrag erasing cells that didn't move

ggml-ci
 2025-06-09 23:04:35 +03:00
Concedo 7d8aa31f1f fixed embeddings, added new parameter to limit max embeddings context 2025-06-10 01:11:55 +08:00
Georgi Gerganov 201b31dc2e graph : fix geglu (#14077) 
ggml-ci
 2025-06-09 17:17:31 +03:00
Đinh Trọng Huy 91a8ee6a6f add geglu activation function (#14074) 
Co-authored-by: dinhhuy <huy.dinh@brains-tech.co.jp>
 2025-06-09 05:15:31 +01:00
Sigbjørn Skjæret 0974ad7a7c llama : fix llama_model_chat_template with template name (LLM_KV with suffix) (#14050) 2025-06-07 14:13:12 +02:00
Georgi Gerganov 745aa5319b llama : deprecate llama_kv_self_ API (#14030) 
* llama : deprecate llama_kv_self_ API

ggml-ci

* llama : allow llama_memory_(nullptr)

ggml-ci

* memory : add flag for optional data clear in llama_memory_clear

ggml-ci
 2025-06-06 14:11:15 +03:00
Georgi Gerganov 487a5e0401 context : fix SWA-related warning for multiple sequences (#14045) 2025-06-06 13:29:18 +03:00
Concedo d18938fc70 fixed build 2025-06-06 18:05:44 +08:00
Concedo d33c88b1f4 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	README.md
#	ci/run.sh
#	examples/embedding/embedding.cpp
#	ggml/CMakeLists.txt
#	ggml/src/CMakeLists.txt
#	src/CMakeLists.txt
 2025-06-06 17:56:51 +08:00
Sigbjørn Skjæret d17a809ef0 llama : support multiple classifier outputs and labels (#13940) 2025-06-06 09:03:25 +02:00
Georgi Gerganov 7f37b6cf1e memory : migrate from llama_kv_cache to more generic llama_memory (#14006) 
* memory : merge llama_kv_cache into llama_memory + new `llama_memory` API

ggml-ci

* context : fix casts

ggml-ci
 2025-06-05 15:29:22 +03:00
Diego Devesa 3a077146a4 llama : allow using mmap without PrefetchVirtualMemory, apply GGML_WIN_VER to llama.cpp sources (#14013) 2025-06-05 11:57:42 +02:00
Sigbjørn Skjæret 9f47fa5792 vocab : warn about missing mask token (#14022) 2025-06-05 09:29:18 +02:00
Georgi Gerganov 9e31bec4fd context : fix pos_min initialization upon error decode (#14008) 
ggml-ci
 2025-06-05 09:06:29 +03:00
Concedo bc89b465a8 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/build.yml
#	.github/workflows/release.yml
#	.github/workflows/server.yml
#	README.md
#	docs/build.md
#	docs/install.md
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	ggml/src/ggml-sycl/vecdotq.hpp
#	tests/test-backend-ops.cpp
#	tests/test-chat.cpp
 2025-06-05 11:03:34 +08:00
Georgi Gerganov 3e63a58ef7 kv-cache : refactor the update/defrag mechanism (#13988) 
* kv-cache : refactor update mechanism

ggml-ci

* memory : improve status handling

* defrag : reset head + add comments

ggml-ci

* cont : minor fixes

ggml-ci
 2025-06-04 18:58:20 +03:00
Xuan-Son Nguyen 3ac67535c8 llama-graph : use ggml_repeat_4d (#13998) 2025-06-04 10:11:26 +02:00
Georgi Gerganov e0e806f52e kv-cache : fix unified::seq_rm to work with seq_id < 0 (#13985) 
ggml-ci
 2025-06-04 09:50:32 +03:00
Georgi Gerganov 5582c49c39 gemma : more consistent attention scaling for v2 and v3 (#13951) 
* gemma : fix attn scale for 27B

* cont : apply scale before attn

* cont : consistent attention scaling
 2025-06-02 20:54:26 +03:00
Concedo b42b618897 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	README.md
#	examples/parallel/parallel.cpp
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-blas/CMakeLists.txt
#	ggml/src/ggml-sycl/CMakeLists.txt
#	ggml/src/gguf.cpp
#	scripts/sync-ggml.last
#	tests/test-gguf.cpp
 2025-06-02 23:26:43 +08:00
Concedo 5e667659ec Merge commit '0fc16b42e8793364918e830c234c1f3caec29dae' into concedo_experimental 
# Conflicts:
#	src/CMakeLists.txt
#	src/llama-kv-cache.cpp
 2025-06-02 23:14:23 +08:00
Concedo 6ce85c54d6 not working correctly 2025-06-02 22:12:10 +08:00
Sigbjørn Skjæret 5e1c3aed40 convert : fix nomic-bert-moe mask token (#13757) 2025-06-01 18:07:21 +02:00
Georgi Gerganov 0fc16b42e8 kv-cache : split implementation in separate sources (#13920) 
ggml-ci
 2025-06-01 11:39:27 +03:00