31ac5834fe
llama : keep track of all EOG tokens in the vocab ( #9609 )
...
ggml-ci
2024-09-24 10:16:06 +03:00
cea1486ecf
log : add CONT level for continuing previous log entry ( #9610 )
2024-09-24 10:15:35 +03:00
0aa15011e3
server : add newline after chat example ( #9616 )
2024-09-24 09:04:39 +03:00
b0f27361f3
sampling : avoid expensive softmax during greedy sampling ( #9605 )
...
* sampling : avoid expensive softmax during greedy sampling
ggml-ci
* speculative : fix default RNG seed + set sparams.n_probs
* Update tests/test-sampling.cpp
Co-authored-by: slaren <slarengh@gmail.com >
* sampling : add clarifying comment [no ci]
---------
Co-authored-by: slaren <slarengh@gmail.com >
2024-09-24 09:03:17 +03:00
c087b6f11d
threads: fix msvc build without openmp ( #9615 )
...
We're missing atomic_thread_fence() in MSVC builds when openmp is disabled.
2024-09-23 21:18:48 -07:00
116efee0ee
cuda: add q8_0->f32 cpy operation ( #9571 )
...
llama: enable K-shift for quantized KV cache
It will fail on unsupported backends or quant types.
2024-09-24 02:14:24 +02:00
0b3bf966f4
server : add --no-context-shift option ( #9607 )
...
* server : add --no-context-shift option
* small fix
* Update examples/server/tests/features/embeddings.feature
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* tests : minor fix
* revert usage of GGML_ASSERT
* update server documentation
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-09-23 22:23:54 +02:00
f0c7b5edf8
threads: improve ggml_barrier scaling with large number of threads ( #9598 )
...
Make sure n_barrier and n_barrier_passed do not share the cache line to avoid cache line bouncing.
This optimization shows performance improvements even for n_threads <= 8 cases.
Resurect TSAN (Thread Sanitizer) check so that we can avoid doing expensive read-modify-write
in the normal case and just use thread-fence as originally intended.
---
Here is the original description and suggestions from Willy Tarreau :
There's currently some false sharing between n_barrier and
n_barrier_passed that is amplified in ggml_barrier() by the fact that
all threads need to increment n_barrier when entering, while all
previous threads continue to read n_barrier_passed, waiting for the last
one to release them all. The side effect is that all these readers are
slowing down all new threads by making the cache line bounce back and
forth between readers and writers.
Just placing them in two distinct cache lines is sufficient to boost
the performance by 21% on a 80-core ARM server compared to the
no-openmp version, and by 3% compared to the openmp version.
Note that the variables could have been spread apart in the structure
as well, but it doesn't seem that the size of this threadpool struct is
critical so here we're simply aligning them.
Finally, the same issue was present when leaving the barrier since all
threads had to update the n_barrier_passed counter, though only one
would add a non-zero value. This alone is responsible for half of the
cost due to undesired serialization.
It might be possible that using a small array of n_barrier counters
could make things even faster on many-core systems, but it would likely
complicate the logic needed to detect the last thread.
Co-authored-by: Willy Tarreau <w@1wt.eu >
2024-09-23 11:42:43 -07:00
1d48e98e4f
readme : add programmable prompt engine language CLI ( #9599 )
2024-09-23 18:58:17 +03:00
f3979df762
flake.lock: Update ( #9586 )
...
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/4f807e8940284ad7925ebd0a0993d2a1791acb2f?narHash=sha256-IiA3jfbR7K/B5%2B9byVi9BZGWTD4VSbWe8VLpp9B/iYk%3D' (2024-09-11)
→ 'github:NixOS/nixpkgs/c04d5652cfa9742b1d519688f65d1bbccea9eb7e?narHash=sha256-PmUr/2GQGvFTIJ6/Tvsins7Q43KTMvMFhvG6oaYK%2BWk%3D' (2024-09-19)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-09-23 08:43:40 -07:00
1e7b9299c6
ggml : AVX512 gemm for Q4_0_8_8 ( #9532 )
...
* AVX512 version of ggml_gemm_q4_0_8x8_q8_0
* Remove zero vector parameter passing
* Rename functions and rearrange order of macros
* Edit commments
* style : minor adjustments
* Update x to start from 0
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-09-23 17:06:38 +03:00
37f8c7b4c9
perplexity : remove extra new lines after chunks ( #9596 )
2024-09-23 11:28:02 +03:00
bf9c1013ac
metal : use F32 prec for K*Q in vec FA ( #9595 )
...
ggml-ci
2024-09-23 11:27:47 +03:00
e62e9789cd
Revert "[SYCL] fallback mmvq ( #9088 )" ( #9579 )
...
This reverts commit 50addec9a5
2024-09-23 11:28:06 +08:00
c35e586ea5
musa: enable building fat binaries, enable unified memory, and disable Flash Attention on QY1 (MTT S80) ( #9526 )
...
* mtgpu: add mp_21 support
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* mtgpu: disable flash attention on qy1 (MTT S80); disable q3_k and mul_mat_batched_cublas
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* mtgpu: enable unified memory
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* mtgpu: map cublasOperation_t to mublasOperation_t (sync code to latest)
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
---------
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2024-09-22 16:55:49 +02:00
912c331d3d
Fix merge error in #9454 ( #9589 )
...
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
2024-09-22 15:26:50 +02:00
a5b57b08ce
CUDA: enable Gemma FA for HIP/Pascal ( #9581 )
2024-09-22 09:34:52 +02:00
5a4bc89c8d
quiet mode on perf endpoint
2024-09-22 13:03:02 +08:00
ecd5d6b65b
llama: remove redundant loop when constructing ubatch ( #9574 )
2024-09-22 04:30:34 +02:00
2a63caaa69
RWKV v6: RWKV_WKV op CUDA implementation ( #9454 )
...
* ggml: CUDA unary op EXP
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* ggml: rwkv_wkv op CUDA impl
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
---------
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
2024-09-22 04:29:12 +02:00
c38d1ecc8d
update templates, fix rwkv
2024-09-22 01:32:12 +08:00
d4d19f6dbe
Fixing Mistral Templates ( #1131 )
...
* Update and rename Mistral.json to Mistral V1.json
* Create Mistral V2 & V3
* Rename Mistral V2 & V3 to Mistral V2 & V3.json
* Create Mistral V3-Tekken.json
2024-09-22 01:30:17 +08:00
d09770cae7
ggml-alloc : fix list of allocated tensors with GGML_ALLOCATOR_DEBUG ( #9573 )
2024-09-21 14:24:23 +02:00
229108f877
fixed incorrect auto gpu settings, fixed clblast not working
2024-09-21 17:59:52 +08:00
004a35b16d
add mutually exclusive group
2024-09-21 15:46:55 +08:00
cd1a52a29e
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build.yml
# scripts/sync-ggml.last
# tests/test-backend-ops.cpp
# tests/test-grad0.cpp
2024-09-21 11:23:54 +08:00
4b6a12e9c0
allow overriding kcpps values with explicit args
2024-09-21 11:00:10 +08:00
41f477879f
Update CUDA graph on scale change plus clear nodes/params ( #9550 )
...
* Avoid using saved CUDA graph if scale changes and reset nodes/params on update
Fixes https://github.com/ggerganov/llama.cpp/issues/9451
* clear before resize
2024-09-21 02:41:07 +02:00
e948a7da7a
CI: Provide prebuilt windows binary for hip ( #9467 )
2024-09-21 02:39:41 +02:00
63351143b2
quantize : improve type name parsing ( #9570 )
...
quantize : do not ignore invalid types in arg parsing
quantize : ignore case of type and ftype arguments
2024-09-20 20:55:36 +02:00
d13edb17ed
ggml : fix builds ( #0 )
...
ggml-ci
2024-09-20 21:15:05 +03:00
27609c49b9
ggml : fix trailing whitespace ( #0 )
...
ggml-ci
2024-09-20 21:15:05 +03:00
4301535326
sync : ggml
...
ggml-ci
2024-09-20 21:15:05 +03:00
424c5d00a9
ggml/examples: add backend support for numerical optimization (ggml/949)
...
* CUDA eval works
* stochastic gradient descent op
* Adam except decay
* CUDA CROSS_ENTROPY_LOSS_BACK
* CUDA mnist-fc training works
* backend CLI arg
* refactor gguf load
* remove sched from opt_step_adam
* implement l1 regularization (weight decay)
* extra call to add optimizer
* initialize gradients with ggml_graph_reset
* gradient accumulation
* increment iter per eval instead of epoch
* adjust backend interfaces
* fix ggml_graph_reset without backend
* fix ggml graph export/import
* fixup
* rename
* revert ggml_opt changes
* more general CUDA repeat_back
* update documentation, fix CNN
* validation split
* add clarifying comment
* optimize PyTorch training
* adjust buffer size, thread count
* fix 0.0f validation split
* Update examples/mnist/mnist-common.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix gradient accumulation
* tensor flag for accumulators -> tensor hash set
* Update include/ggml.h
Co-authored-by: slaren <slarengh@gmail.com >
* Update tests/test-backend-ops.cpp
Co-authored-by: slaren <slarengh@gmail.com >
* Update tests/test-backend-ops.cpp
Co-authored-by: slaren <slarengh@gmail.com >
* fix test prints
* Update src/ggml-backend.c
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* better CUDA support for noncontiguous out_prod
* add comment
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
Co-authored-by: slaren <slarengh@gmail.com >
2024-09-20 21:15:05 +03:00
a6809c6a2e
examples : add null threadpool args where needed (ggml/0)
...
ggml-ci
2024-09-20 21:15:05 +03:00
5cb12f6839
CUDA: fix sum.cu compilation for CUDA < 11.7 ( #9562 )
2024-09-20 18:35:35 +02:00
55a249d222
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# examples/imatrix/imatrix.cpp
# examples/infill/infill.cpp
# examples/perplexity/perplexity.cpp
2024-09-20 18:03:45 +08:00
68aee56498
updated lite, allow GUI to import launcher args and config files with showgui
2024-09-20 17:52:52 +08:00
d39e26741f
examples : flush log upon ctrl+c ( #9559 )
2024-09-20 11:46:56 +03:00
722ec1eb51
perplexity : do not escape input data by default ( #9548 )
2024-09-20 09:38:10 +03:00
6026da52d6
server : clean-up completed tasks from waiting list ( #9531 )
...
ggml-ci
2024-09-19 12:44:53 +03:00
eca0fab44e
imatrix : disable prompt escape by default ( #9543 )
2024-09-19 10:58:14 +03:00
29625c3d2e
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build.yml
# .github/workflows/server.yml
# CMakeLists.txt
# Makefile
# README.md
# ci/run.sh
# common/CMakeLists.txt
# common/common.cpp
# docs/backend/SYCL.md
# examples/embedding/embedding.cpp
# examples/imatrix/imatrix.cpp
# examples/infill/infill.cpp
# examples/llama-bench/llama-bench.cpp
# examples/main/README.md
# examples/parallel/parallel.cpp
# examples/perplexity/perplexity.cpp
# examples/server/CMakeLists.txt
# examples/server/README.md
# examples/server/bench/README.md
# examples/server/tests/README.md
# examples/speculative/speculative.cpp
# flake.lock
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# grammars/README.md
# scripts/compare-commits.sh
# scripts/compare-llama-bench.py
# tests/CMakeLists.txt
2024-09-19 14:53:57 +08:00
0b0e456202
try fix compile issues
2024-09-19 13:56:19 +08:00
64c6af3195
ggml : fix n_threads_cur initialization with one thread ( #9538 )
...
* ggml : fix n_threads_cur initialization with one thread
* Update ggml/src/ggml.c
---------
Co-authored-by: Max Krasnyansky <quic_maxk@quicinc.com >
2024-09-18 10:13:08 -07:00
0d2f22e45c
scripts : verify py deps at the start of compare ( #9520 )
2024-09-18 18:34:32 +03:00
6443ddd985
llama : use reserve/emplace_back in sampler_sample ( #9534 )
...
This commit updates the llama_sampler_sample function to use reserve and
emplace_back for the vector of llama_token_data structs.
The motivation for this change is to avoid the creation of n_vocab
default-constructed llama_token_data structs which are then
immediately overwritten.
2024-09-18 14:42:36 +03:00
8a308354f6
server : match OAI structured output response ( #9527 )
2024-09-18 09:50:34 +03:00
f799155ab8
server : fix OpenSSL build (remove obsolete LOG_INFO) ( #9529 )
2024-09-18 09:28:20 +03:00
faf67b3de4
[SYCL]set context default value to avoid memory issue, update guide ( #9476 )
...
* set context default to avoid memory issue, update guide
* Update docs/backend/SYCL.md
Co-authored-by: Meng, Hengyu <hengyu.meng@intel.com >
---------
Co-authored-by: arthw <14088817+arthw@users.noreply.github.com >
Co-authored-by: Meng, Hengyu <hengyu.meng@intel.com >
2024-09-18 08:30:31 +08:00
Georgi Gerganov
StrangeBytesDev
Max Krasnyansky
Ivan
Xuan Son Nguyen
Riceball LEE
Srihari-mcw
Akarshan Biswas
R0CKSTAR
Molly Sophia
Johannes Gäßler
Concedo
Shankar
pandora
slaren
agray3
Huang Qi
Sigbjørn Skjæret
Daniel Bevenius
Vinesh Janarthanan
Eric Zhang
Neo Zhang Jianyu