27bd7b95f5
Merge branch 'upstream' into concedo_experimental
2025-03-22 09:42:33 +08:00
ef90d2b7b1
Revert "Revert "vulkan: matmul dequantization improvements ( #12015 )""
...
This reverts commit 26b71e33b1
2025-03-22 09:41:57 +08:00
4375415b4a
Vulkan: RTE rounding for cpy to quant ( #12480 )
...
* Vulkan: RTE rounding for cpy to quant
Co-Authored-By: Jeff Bolz <jbolz@nvidia.com >
* remove trailing whitespace
* avoid duplicating pipeline_cpy_f32_quant
* fix copypasting issue
* remove duplicated code
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com >
2025-03-21 20:34:50 +01:00
30c42ef5cb
vulkan: workaround for AMD Windows driver 16 bit unpack8 bug ( #12472 )
2025-03-21 20:27:47 +01:00
ae670dbe0e
no repacking for avx2 for kcpp because it breaks 4_0_4_4 quants
2025-03-22 00:33:27 +08:00
7030ebf401
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# docs/backend/SYCL.md
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp
# ggml/src/ggml-sycl/CMakeLists.txt
# tests/test-backend-ops.cpp
2025-03-22 00:32:42 +08:00
c1e58419c7
support for voice cloning is done (+2 squashed commit)
...
Squashed commit:
[e7301628] support for voice cloning is done
[1653c576] wip adding voice cloning
2025-03-21 22:28:59 +08:00
af04481e6b
model : do not repack if a GPU device is present ( #12498 )
...
ggml-ci
2025-03-21 16:14:29 +02:00
960e726077
chore : cleanup llama_model_loader::TENSOR_ usage ( #12492 )
2025-03-21 10:21:36 +01:00
ea1518e839
llama-tts : avoid crashes related to bad model file paths ( #12482 )
2025-03-21 11:12:45 +02:00
1aa87ee53d
[SYCL] Fix build on Windows when ccache enabled ( #9954 ) ( #9976 )
...
* [SYCL] Fix build on Windows when ccache enabled (#9954 )
* take effect only on windows and force it to icl
---------
Co-authored-by: Romain Biessy <romain.biessy@codeplay.com >
2025-03-21 14:58:47 +08:00
9ffcc9e374
sycl: cleanup oneDNN related code ( #12097 )
2025-03-21 10:15:56 +08:00
e04643063b
webui : Prevent rerendering on textarea input ( #12299 )
...
* webui: Make textarea uncontrolled to eliminate devastating lag
* Update index.html.gz
* use signal-style implementation
* rm console log
* no duplicated savedInitValue set
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co >
2025-03-20 15:57:43 +01:00
a66d0f7743
safeguard for bad vision input
2025-03-20 22:20:38 +08:00
9d5efd68b6
dont print full base64 for images
2025-03-20 21:20:04 +08:00
dbb3a4739e
llama : make Qwen2MoE QKV bias optional ( #12477 )
2025-03-20 12:49:59 +01:00
3d82dbcbce
ggml : block interleaving support for Q4_K quantization for x86 AVX2 architecture ( #12332 )
...
* Add block interleaving support for Q4_K quantization
* Remove whitespaces and fix CI/CD issues
* Update pointer of bsums from int16_t to const int16_t
* Add vector version of quantize_q8_K_4x8 function
* Update code formatting based on review comments
2025-03-20 13:35:34 +02:00
732b5fbf5e
convert : avoid calls to tokenizer.added_tokens_decoder ( #12473 )
...
tokenizer.added_tokens_decoder returns a fresh dict every time relatively slowly (~0.04s on average) which results in massive slowdowns when we have a huge number of added tokens
2025-03-20 08:36:37 +02:00
568013d0cd
context : clear sets containing encoder output sequence ids before storing new values ( #12470 )
...
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com >
2025-03-19 21:01:57 +01:00
517b5ddbf0
CUDA: Improve flash decoding kernel GPU occupancy for BS=1 case ( #12183 )
...
- Find out active blocks per SM using cudaOccupancyMaxActiveBlocksPerMultiprocessor API. Use this value to determine the optimal parallel_blocks value.
- Prefer vector flash attention kernels over MMA kernel for BS=1
Fixes Issue: #12182
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2025-03-19 20:52:06 +01:00
a9b59288e2
vulkan: optimize iq1 coopmat2 dequant functions ( #12427 )
2025-03-19 19:56:23 +01:00
143b611274
updated workflows
2025-03-19 21:56:35 +08:00
9910f3abe0
remove precompiled vulkan shaders from repo. They will now have to be recreated in vulkan-shaders-gen from scratch at runtime, which is auto handled by the makefile for windows and linux.
2025-03-19 21:51:16 +08:00
0c90d2ebcf
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build.yml
# CMakeLists.txt
# cmake/common.cmake
# docs/backend/SYCL.md
# examples/main/README.md
# examples/speculative/speculative.cpp
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-musa/CMakeLists.txt
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-vulkan/vulkan-shaders/CMakeLists.txt
# tests/test-backend-ops.cpp
2025-03-19 19:27:11 +08:00
0fd8487b14
Fix visionOS build and add CI ( #12415 )
...
* ci: add visionOS build workflow
Add a new GitHub Actions workflow for building on visionOS with CMake and Xcode.
* ggml: Define _DARWIN_C_SOURCE for visionOS to fix missing u_xxx typedefs
* ci: remove define hacks for u_xxx system types
---------
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com >
2025-03-19 11:15:23 +01:00
108e53c2f1
llama : add support for GPT2, Bloom and CodeShell tied word embeddings ( #12456 )
...
* Add support for GPT2, Bloom and CodeShell tied word embeddings
* Deduplicate tied word embeddings weights
* Workaround for incorrect weight map
It appears transformer.wte.weight is in the weight map even though the weights are not there, remove it if output weights are encountered first.
* check++
* fatfingers--
2025-03-19 09:08:49 +01:00
a686171ea7
convert : Support chat_template.json ( #12460 )
2025-03-19 08:58:13 +01:00
c446b2edd2
vulkan: Submit once enough matmul work has been recorded ( #12406 )
...
I've been seeing significantly worse performance for tg with flash attention
enabled vs disabled, and it seems to be related to the submit heuristic.
Change the heuristic to check how many bytes worth of weight matrix are
used and flush every 100MB, and ramp up after the first few submits.
This seems to resolve the issue, and also increases perf for non-FA a bit.
2025-03-19 08:26:26 +01:00
d84635b1b0
opencl: improve profiling ( #12442 )
...
* opencl: more profiling timing
* opencl: generate trace for profiling
* opencl: reduce profiling overhead
* Populate profiling timing info at the end rather than after each
kernel run
* opencl: fix for chrome tracing
2025-03-18 12:54:55 -07:00
75422e8bc4
graph : normalize Q, K, V shapes + sync cross attention ( #12449 )
...
* graph : normalize Q, K, V shapes and add comments
ggml-ci
* context : synchronize before getting cross attention data
* model : fix command-r attention norm check
2025-03-18 21:35:19 +02:00
bb115d2bf7
musa: override warp_size of musa device to 32 ( #12445 )
...
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2025-03-18 19:28:26 +01:00
29fff308c7
llama : support converting Mistral Small text-only ( #12450 )
2025-03-18 19:16:19 +01:00
c6af2161b2
speculative : fix seg fault in certain cases ( #12454 )
2025-03-18 19:35:11 +02:00
99aa304fb9
llama : add support for EXAONE tied word embeddings ( #12451 )
2025-03-18 17:24:33 +01:00
8551c44d84
context : always use non-causal attention for encoder graphs ( #12447 )
...
* context : always use non-causal attention for encoder graphs
ggml-ci
* context : move the change to llama_context::encode()
ggml-ci
2025-03-18 13:05:49 +02:00
35cae5ba05
SYCL: using graphs is configurable by environment variable and compile option ( #12371 )
...
* alberto changes
* enable sycl graphs by env variable
* fixed compilation warnings in ggml-sycl.cpp
* renamed graph variables
* fix markdown in docs/backend/SYCL.md
Co-authored-by: Romain Biessy <romain.biessy@codeplay.com >
* fix markdown in docs/backend/SYCL.md again
* compiling graphs by default, renamed graph_enable to graph_disable
---------
Co-authored-by: Romain Biessy <romain.biessy@codeplay.com >
2025-03-18 11:16:31 +01:00
810e0af3f5
server : fix warmup draft cache type ( #12446 )
...
ggml-ci
2025-03-18 12:05:42 +02:00
eba92d64c3
cmake : fix PowerPC build ( #12241 )
...
Closes #12240
2025-03-18 11:37:33 +02:00
d9a14523bb
ggml : add SVE support for q6_K_q8_K ( #12361 )
2025-03-18 10:14:39 +02:00
fd123cfead
Vulkan: Default to 1GB allocations instead of 4GB to avoid fragmentation and driver issues ( #12434 )
2025-03-18 07:21:40 +01:00
a53f7f7b88
fixed compilation warnings in ggml-sycl ( #12424 )
2025-03-18 08:51:25 +08:00
7dfad387e3
llama: Add support for RWKV v7 architecture ( #12412 )
...
* ggml: Add op l2_norm
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* ggml: Add op rwkv_wkv7
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* llama: Add support for RWKV7 and ARWKV7 models
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* llama: fix inference with RWKV6Qwen2
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* llama: add more (a)rwkv7 variants in size
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* Apply code-format changes
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* fix MUSA build
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* llama: fix shape error with rwkv using llama-parallel
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
---------
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
2025-03-18 07:27:50 +08:00
60c902926c
docs : bring llama-cli conversation/template docs up-to-date ( #12426 )
2025-03-17 21:14:32 +01:00
b1b132efcb
cuda : enable CUDA Graph on CUDA Toolkit < 12.x ( #12394 )
...
* Enable CUDA Graph on CTK < 12.x
`cudaGraphExecUpdate` API was changed on 12.x. For this reason CUDA graph support was disabled on older CUDA toolkit. This change enables CUDA support in CTK version < 12.x by using older API if CTK < 12.x.
* Fix compilation errors with MUSA
* Disable CUDA Graph for MUSA
2025-03-17 20:25:13 +02:00
01e8f2138b
ggml-vulkan: remove unused find_program(glslc) ( #12416 )
...
It's already found by FindVulkan.cmake in the parent CMakeLists
2025-03-17 13:35:43 -03:00
484a8ab513
vulkan: Add N/2 and N/4 optimized paths in coopmat2 shader ( #12312 )
2025-03-17 09:26:18 -05:00
ddaa8d5a38
fixed saving path for savedata
2025-03-17 22:19:52 +08:00
0cfd8d23cb
handle symlinks (+1 squashed commits)
...
Squashed commits:
[fb8477b9] fixed makefile (+4 squashed commit)
Squashed commit:
[4a245bba] fixed a makefile issue
[d68eba69] alias usehipblas to usecublas
[a9ab0a7c] dynamic rocwmma selection
[fefe17c7] revert rocwmma
2025-03-17 21:03:30 +08:00
cf2270e4d3
vulkan: subgroup size tuning ( #12087 )
...
* vulkan: subgroup size test
* Vulkan: Add device architecture enum and logic to recognize AMD generations
* vulkan: use new architecture logic to specify subgroup size
* Initial vulkan subgroup size tuning for RDNA3
* vulkan: commonize RDNA subgroup tuning
* vulkan: override subgroup size if required_subgroup_size = 0
* vulkan: disable warp 32 for RDNA3
* vulkan: fine tuned RDNA1 subgroup sizes
* vulkan: adjusted subgroup size map
* vulkan: fixed RDNA2 subgroup map
---------
Co-authored-by: 0cc4m <picard12@live.de >
2025-03-17 12:42:33 +01:00
f07690c930
vulkan: use fp32 in coopmat2 q4_k dequant function ( #12309 )
2025-03-17 10:43:35 +01:00
Concedo
stduhpf
Eve
Georgi Gerganov
Sigbjørn Skjæret
marcoStocchi
蕭澧邦
Svetlozar Georgiev
Woof Dog
Srihari-mcw
Bartowski
fairydreaming
Gaurav Garg
Jeff Bolz
Guus Waals
lhez
R0CKSTAR
Xuan-Son Nguyen
Łukasz Ślusarczyk
Prajwal B Mehendarkar
fj-y-saito
0cc4m
Molly Sophia
Daniele