ad30a3cc99
Merge commit '7d701b59296bc6f5ba504d5a4cddaf416c449a15' into concedo_experimental
...
# Conflicts:
# .github/workflows/python-type-check.yml
# examples/pydantic_models_to_grammar.py
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-arith.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.h
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/conv2d.cl
# ggml/src/ggml-opencl/kernels/conv2d_f16_f32.cl
# scripts/jinja/jinja-tester.py
# scripts/snapdragon/ggml-hexagon-profile.py
# scripts/tool_bench.py
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-recurrent-state-rollback.cpp
2026-09-09 10:19:55 +08:00
5a6caa05fc
ggml : update ggml_prec specification ( #26675 )
...
* ggml : update ggml_prec specification
[no ci]
* cont : add GGML_PREC_BF16
* cont : rework API
* cont : use new API
* cont : swap arg order
* cont : support for MUL_MAT_ID
* cont : fix accidental remove of "break;"
* cont : return bools, add doc TAG_GGML_PREC, clean-up
* cont : add search tag
* cont : ws
2026-09-08 09:06:24 +03:00
f64ab79adf
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-apple.yml
# .github/workflows/build-cpu.yml
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/build-sycl.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/build-wasm.yml
# .github/workflows/build-webgpu.yml
# .github/workflows/hip-quality-check.yml
# .github/workflows/server.yml
# CMakeLists.txt
# docs/backend/SYCL.md
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/concat.cl
# ggml/src/ggml-opencl/kernels/cpy.cl
# ggml/src/ggml-sycl/common.cpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/fattn-buffers.cpp
# ggml/src/ggml-sycl/fwht.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# scripts/sync-ggml.last
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
2026-09-05 12:09:30 +08:00
64a155d242
sync : ggml ( #28379 )
...
* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0)
cont https://github.com/ggml-org/llama.cpp/pull/27960
* ggml : bump version to 0.23.0 (ggml/1618)
* sync : ggml
2026-09-04 14:39:19 +03:00
2e04e6509d
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-android.yml
# .github/workflows/build-apple.yml
# .github/workflows/build-cpu.yml
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/build-cuda-windows.yml
# .github/workflows/build-msys.yml
# .github/workflows/build-opencl.yml
# .github/workflows/build-openvino.yml
# .github/workflows/build-riscv.yml
# .github/workflows/build-sanitize.yml
# .github/workflows/build-sycl.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/build-wasm.yml
# .github/workflows/build-webgpu.yml
# .github/workflows/copilot-setup-steps.yml
# .github/workflows/hip-quality-check.yml
# .github/workflows/release.yml
# .github/workflows/server.yml
# CODEOWNERS
# build-xcframework.sh
# docs/android.md
# docs/backend/SYCL.md
# docs/build.md
# ggml/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-opnode.h
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.c
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-hexagon/htp/set-rows-ops.c
# ggml/src/ggml-metal/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32.cl
# ggml/src/ggml-sycl/ggml-sycl.cpp
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
# tests/test-save-load-state.cpp
# tools/mtmd/CMakeLists.txt
2026-09-02 23:24:09 +08:00
8e93a9773b
CUDA + ggml: add sparse-fa for DSV4/GLM ( #27970 )
2026-09-02 17:27:37 +03:00
c22d6e5c35
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# ggml/src/ggml-cann/aclnn_ops.cpp
# ggml/src/ggml-cann/aclnn_ops.h
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-et/et-kernels/src/glu_f32.c
# ggml/src/ggml-et/ggml-et-cpu-compare.cpp
# ggml/src/ggml-et/ggml-et-ops.cpp
# ggml/src/ggml-et/ggml-et.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/act-ops.c
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/glu.cl
# ggml/src/ggml-openvino/openvino/op/glu_swiglu.cpp
# ggml/src/ggml-openvino/openvino/op_table.cpp
# ggml/src/ggml-openvino/openvino/op_table.h
# ggml/src/ggml-rpc/CMakeLists.txt
# ggml/src/ggml-sycl/element_wise.cpp
# ggml/src/ggml-sycl/element_wise.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/glu.wgsl
# scripts/sync_vendor.py
# tests/test-backend-ops.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/llama-bench/README.md
# tools/llama-bench/llama-bench.cpp
# tools/rpc/CMakeLists.txt
# tools/server/README.md
2026-08-31 18:39:16 +08:00
0190529ec4
ggml: add SWIGLU_CLAMP ( #27930 )
...
* ggml: add SWIGLU_CLAMP
* add vulkan shader
2026-08-30 23:00:02 +08:00
73f56d105b
ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC ( #27960 )
...
some backends (Metal, SYCL, WebGPU) require additional memory for
fleeting data for certain ops, which is reflected in their
get_alloc_size implementations.
add ggml_backend_op_alloc_size_may_expand() to the backend utils,
listing these ops, and assert in ggml_backend_buft_get_alloc_size
that a backend expanding the alloc size of a compute op only does so
for ops listed in the helper.
use the helper in the RPC backend to decide whether to query the
remote server for the actual alloc size, instead of a hardcoded list.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-08-30 09:17:47 +03:00
ae60e1d6e0
Merge commit 'deae5ee133a3c4c56fbd46c17c8c2103af3bd643' into concedo_experimental
...
# Conflicts:
# .devops/openvino.Dockerfile
# .github/workflows/build-apple.yml
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/docker.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# .github/workflows/ui-build-self-hosted.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui-publish.yml
# .github/workflows/ui-self-hosted.yml
# .github/workflows/ui.yml
# CMakeLists.txt
# docs/backend/snapdragon/CMakeUserPresets.json
# docs/backend/snapdragon/README.md
# docs/backend/snapdragon/developer.md
# docs/backend/snapdragon/linux.md
# docs/backend/snapdragon/windows.md
# docs/ops.md
# docs/ops/Vulkan.csv
# ggml/cmake/ggml-config.cmake.in
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/kleidiai/kernels.cpp
# ggml/src/ggml-cpu/kleidiai/kernels.h
# ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-opnode.h
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/act-ops.c
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/dma-queue.h
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.c
# ggml/src/ggml-hexagon/htp/hex-utils.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/htp-tensor.c
# ggml/src/ggml-hexagon/htp/htp-tensor.h
# ggml/src/ggml-hexagon/htp/hvx-arith.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-hexagon/htp/set-rows-ops.c
# ggml/src/ggml-rpc/CMakeLists.txt
# scripts/snapdragon/ggml-hexagon-profile.py
# scripts/snapdragon/ggml-hexagon-trace.py
# tests/test-backend-ops.cpp
# tools/cli/README.md
# tools/llama-bench/llama-bench.cpp
# tools/rpc/README.md
# tools/server/README.md
# tools/ui/tests/stories/a11y/ChatScreenForm.a11y.stories.svelte
2026-08-28 19:31:29 +08:00
d0132a680a
rpc : implement event and async backend APIs ( #18626 )
...
* rpc : implement event and async backend APIs
* cache responses from RPC_CMD_GET_ALLOC_SIZE
2026-08-26 17:34:46 +03:00
4447017602
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/actions/ccache-clear/action.yml
# .github/workflows/build-apple.yml
# .github/workflows/build-cpu.yml
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/build-opencl.yml
# .github/workflows/build-openvino.yml
# .github/workflows/build-sycl.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/build-wasm.yml
# .github/workflows/build-webgpu.yml
# .github/workflows/hip-quality-check.yml
# .github/workflows/server.yml
# CONTRIBUTING.md
# README.md
# ci/run.sh
# common/CMakeLists.txt
# common/chat.cpp
# docs/autoparser.md
# ggml/src/ggml-webgpu/wgsl-shaders/flash_attn.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_tile.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_vec_split.wgsl
# scripts/sync_vendor.py
# tests/CMakeLists.txt
# tests/peg-parser/test-json-serialization.cpp
# tests/peg-parser/tests.h
# tests/test-chat-auto-parser.cpp
# tests/test-chat-peg-parser.cpp
# tests/test-chat-template.cpp
# tests/test-chat.cpp
# tests/test-grammar-integration.cpp
# tests/test-jinja.cpp
# tests/test-json-schema-to-grammar.cpp
# tests/test-llama-archs.cpp
# tests/test-model-resolution.cpp
# tests/test-recurrent-state-rollback.cpp
# tools/CMakeLists.txt
2026-08-25 20:44:34 +08:00
6036c635e2
ggml : fix ggml_clamp ( #27644 )
...
* ggml : fix ggml_clamp
* cont : update ggml-alloc
2026-08-24 10:43:04 +03:00
8c732ca2cc
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/openvino.Dockerfile
# .github/actions/windows-setup-cuda/action.yml
# .github/workflows/build-cache.yml
# .github/workflows/build-cpu.yml
# .github/workflows/build-cuda-windows.yml
# .github/workflows/build-openvino.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/docker.yml
# .github/workflows/make-release.yml
# .github/workflows/release.yml
# AUTHORS
# CMakeLists.txt
# README.md
# build-xcframework.sh
# ci/run.sh
# common/CMakeLists.txt
# docs/backend/OPENVINO.md
# examples/gguf-hash/CMakeLists.txt
# examples/gguf-hash/gguf-hash.cpp
# ggml/CMakeLists.txt
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-et/ggml-et.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/flash-attn-ops.h
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/flash_attn_f16.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32.cl
# ggml/src/ggml-opencl/kernels/moe_sort_by_expert.cl
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/common_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_reg_tile.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_subgroup_matrix.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_acc.tmpl
# scripts/sync-ggml.last
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
# tools/mtmd/CMakeLists.txt
# tools/mtmd/mtmd-helper.cpp
# tools/perplexity/perplexity.cpp
# tools/server/README.md
# tools/ui/src/lib/hooks/use-tools-panel.svelte.ts
# vendor/hash/CMakeLists.txt
2026-08-21 23:27:27 +08:00
fe8156f789
ggml: add ggml_rope_set_offset (+ metal support) ( #27120 )
...
* add params
* cpu kernel
* metal kernel
* add test backend ops
* gate other backends
* ggml: (cuda) support ggml_rope_set_offset (#27121 )
* rm cuda supports_op guard, fix webgpu clang-format
* ggml: support ggml_rope_set_offset on vulkan (#27344 )
* ggml: support ggml_rope_set_offset on vulkan
* remove inplace optimization
2026-08-19 14:04:57 +02:00
af5172627d
RPC: populate use_count to enable fusion inside backends ( #27142 )
...
* RPC: populate use_count to enable fusion inside backends
* bump minor version
2026-08-18 21:08:57 +05:30
82e15c08bf
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# docs/backend/SYCL.md
# docs/backend/snapdragon/developer.md
# examples/convert-llama2c-to-ggml/convert-llama2c-to-ggml.cpp
# examples/sycl/run-llama2.sh
# examples/sycl/start-svr.sh
# examples/sycl/test.sh
# examples/sycl/win-run-llama2.bat
# examples/sycl/win-start-svr.bat
# examples/sycl/win-test.bat
# ggml/CMakeLists.txt
# ggml/src/ggml-et/et-kernels/src/ssm_scan_f32.c
# ggml/src/ggml-et/ggml-et-ops.cpp
# ggml/src/ggml-et/ggml-et-ops.h
# ggml/src/ggml-sycl/ssm_scan.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/ssm_scan.wgsl
# scripts/bench-models.sh
# scripts/snapdragon/adb/run-bench.sh
# scripts/snapdragon/adb/run-cli.sh
# scripts/snapdragon/adb/run-completion.sh
# scripts/snapdragon/adb/run-mtmd.sh
# scripts/snapdragon/windows/run-bench.ps1
# scripts/snapdragon/windows/run-cli.ps1
# scripts/snapdragon/windows/run-completion.ps1
# scripts/snapdragon/windows/run-mtmd.ps1
# scripts/sync-ggml.last
# scripts/sync_vendor.py
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-jinja.cpp
# tests/test-llama-archs.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/llama-bench/README.md
# tools/server/README.md
2026-08-15 22:50:04 +08:00
1692f9e50b
ggml : recurrent state rollback for ggml_ssm_scan ( #26623 )
...
* Initial changes for Recurrent state rollback for nemotron for cpu and cuda
* Removing CPU RS rollback. Will enable it in subsequent PRs
* addition of test case
* Removing assert and calling runtime API to check if op is supported
* removing extra API and updating the call sites for K
* replace static cuda detection to runtime fused_op api
* address review comments and fallback when SSM rollback not supprted
* Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda
* removing memory manipulation as rs rollback is now supported in CPU
* removing the static probe which is not needed now
* correcting the format
* address review comments
* enabling test for all the backends, unsupported backends will fallback to CPU
* Apply suggestions from code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported
* Support K > 1 in ssm_scan for all backends
* Fix CI Issues
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com >
2026-08-14 17:20:40 +03:00
dcfe29317e
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/actions/windows-setup-rocm/action.yml
# .github/workflows/build-cache.yml
# .github/workflows/build-cuda-windows.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/server.yml
# ci/run.sh
# docs/speculative.md
# examples/lookup/lookup.cpp
# examples/speculative-simple/speculative-simple.cpp
# examples/speculative/speculative.cpp
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-et/ggml-et.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_q4_0.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_q8_0.cl
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-virtgpu/backend/backend-dispatched-device.cpp
# ggml/src/ggml-virtgpu/backend/shared/api_remoting.h
# ggml/src/ggml-virtgpu/ggml-backend-buffer-type.cpp
# ggml/src/ggml-virtgpu/ggml-backend-device.cpp
# ggml/src/ggml-virtgpu/virtgpu-forward-device.cpp
# ggml/src/ggml-virtgpu/virtgpu-forward.gen.h
# ggml/src/ggml-vulkan/ggml-vulkan.cpp
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/cpy.wgsl
# ggml/src/ggml-zdnn/ggml-zdnn.cpp
# ggml/src/ggml-zendnn/ggml-zendnn.cpp
# tests/peg-parser/test-json-parser.cpp
# tests/test-arg-parser.cpp
# tests/test-backend-sampler.cpp
# tests/test-sampling.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/llama-bench/llama-bench.cpp
# tools/server/README.md
2026-08-11 15:13:15 +08:00
153d324bcf
llama: add default load-mode auto, which avoids mmap on iGPUs ( #26081 )
...
* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used
* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com >
* set mmap_support to false on OpenCL backend
* fix order of load modes
* use -1 for auto
* resolve load mode auto earlier to correctly pick gpu host or cpu memory
* add load mode auto to llama-bench
* bump virtgpu api version, regenerate docs
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com >
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-08-11 09:20:46 +03:00
8a16f96307
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-apple.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/release.yml
# SECURITY.md
# build-xcframework.sh
# ci/run.sh
# docs/development/HOWTO-add-model.md
# examples/model-conversion/scripts/causal/convert-model.sh
# examples/model-conversion/scripts/embedding/convert-model.sh
# scripts/sync_vendor.py
# scripts/ui-assets.cmake
# tests/test-arg-parser.cpp
# tests/test-backend-sampler.cpp
# tests/test-grammar-parser.cpp
# tests/test-llama-archs.cpp
# tests/test-sampling.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/mtmd/CMakeLists.txt
# tools/mtmd/mtmd.h
# tools/mtmd/tests/test-deepseek-ocr.py
# tools/server/README.md
# tools/tts/CMakeLists.txt
# tools/tts/convert_pt_to_hf.py
2026-08-07 20:46:56 +08:00
c8e03ce812
mtmd/ggml: add ggml_build_forward_order ( #26649 )
...
* ggml: add ggml_build_forward_order
ggml_build_forward_expand marks the tensor and all its ancestors for
compute, so using it as a pure ordering hint (keeping q, k and v
together) defeats ggml_build_forward_select: the unselected branch is
forced to run with inputs that were never uploaded. In the mtmd audio
graph this makes GEN_WAV calls execute the GEN_CODE branch with a
stale inp_code0, hitting the get_rows bound assert on CPU.
Add ggml_build_forward_order, which inserts nodes without the compute
flag; the flag is restored when the branch is actually selected.
Switch the q/k/v hints in clip_graph::build_attn to it.
* nit: reduce comments (AGENTS.md)
2026-08-06 00:47:59 +02:00
98926f27c1
Merge commit '11b068d06605288ce7917534b46d52b47823dc13' into concedo_experimental
...
# Conflicts:
# CONTRIBUTING.md
# docs/backend/SYCL.md
# docs/install.md
# docs/speculative.md
# ggml/src/ggml-hip/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/element_wise.cpp
# ggml/src/ggml-sycl/fattn-onednn.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/glu.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/ssm_scan.wgsl
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tools/cli/README.md
# tools/llama-bench/llama-bench.cpp
# tools/mtmd/CMakeLists.txt
# tools/server/README.md
2026-07-31 17:06:19 +08:00
f5b9bd39b5
RPC: add tensor_memset ( #25912 )
2026-07-29 15:04:30 +08:00
0a0b88a5c0
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# docs/backend/OPENCL.md
# ggml/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/kleidiai/kernels.cpp
# ggml/src/ggml-cpu/kleidiai/kernels.h
# ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-drv.cpp
# ggml/src/ggml-hexagon/htp-drv.h
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/act-ops.c
# ggml/src/ggml-hexagon/htp/argsort-ops.c
# ggml/src/ggml-hexagon/htp/binary-ops.c
# ggml/src/ggml-hexagon/htp/cumsum-ops.c
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hex-dma.h
# ggml/src/ggml-hexagon/htp/hex-profile.h
# ggml/src/ggml-hexagon/htp/hex-utils.h
# ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/hmx-queue.c
# ggml/src/ggml-hexagon/htp/hmx-queue.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/gemm_moe_mxfp4_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q4_0_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q4_k_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q6_k_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_k_f32.cl
# ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_k_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32.cl
# ggml/src/ggml-opencl/kernels/mul_mv_q4_k_f32_flat.cl
# ggml/src/ggml-opencl/kernels/mul_mv_q5_k_f32_flat.cl
# ggml/src/ggml-sycl/dmmv.cpp
# scripts/snapdragon/ggml-hexagon-profile.py
# scripts/sync-ggml.last
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
# tests/test-recurrent-state-rollback.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
2026-07-18 12:26:15 +08:00
0dc74e332e
DeepseekV4: Add fused hyper-connection ops ( #25585 )
...
* dsv4 hc-ops
* add missing files;
* add cparams
* update rpc version
* address review comments
* address review comments
2026-07-17 00:33:33 +08:00
f15bd60901
kleidiai: Add SME vs SME2 distinction in kernel dispatch ( #25478 )
...
The current integration treats SME as a single capability (CPU_FEATURE_SME)
with no distinction between SME(v1) and SME2. The kernels dispatched under
CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect
on SME(v1)-only hardware.
We introduce build-time and runtime distinction between SME and SME2, and
wire SME(v1) and SME2 kernels based on actual hardware support.
2026-07-16 08:57:04 -07:00
9001369da0
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/kleidiai/kernels.cpp
# ggml/src/ggml-cpu/kleidiai/kernels.h
# ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
# ggml/src/ggml-cpu/ops.cpp
# ggml/src/ggml-cuda/mmq.cuh
# ggml/src/ggml-hexagon/htp/hmx-queue.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_iq4_nl_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q1_0_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32_spec.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_1_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_0_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_1_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_k_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q6_k_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q8_0_f32.cl
# ggml/src/ggml-opencl/kernels/mul_mv_f16_f16.cl
# ggml/src/ggml-opencl/kernels/mul_mv_f16_f32.cl
# ggml/src/ggml-opencl/kernels/mul_mv_f16_f32_1row.cl
# ggml/src/ggml-sycl/fattn-vec.hpp
# tests/test-backend-ops.cpp
# tests/test-chat-auto-parser.cpp
# tests/test-export-graph-ops.cpp
# tests/test-jinja.cpp
# tests/test-llama-archs.cpp
# tools/tokenize/tokenize.cpp
2026-07-15 15:39:31 +08:00
a7312ae94f
ggml : add a set of functions for checking contiguity of inner tensor dimensions ( #25650 )
...
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com >
2026-07-14 14:37:52 +02:00
fa21872f97
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# docs/backend/SYCL.md
# ggml/src/ggml-cpu/ggml-cpu.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_mxfp4_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q4_k_f32_ns.cl
# ggml/src/ggml-sycl/backend.hpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# scripts/sync_vendor.py
# tests/CMakeLists.txt
# tests/test-alloc.cpp
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-gguf.cpp
# tests/test-save-load-state.cpp
2026-07-13 20:53:37 +08:00
ad8d821991
gguf : add tensor shape accessor ( #24405 )
...
* gguf : add tensor shape accessors
* gguf : return tensor shape as const int64_t *
* gguf : remove n_dims accessor, keep only gguf_get_tensor_ne
2026-07-13 13:55:15 +03:00
00f5442cc4
ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer ( #24231 )
...
* ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer
* ggml : remove scale parameters from lightning indexer OP, add f16 mask parameter
* tests : add GGML_OP_LIGHTNING_INDEXER tests
* ggml : bump RPC version
* chore : check if lightning indexer input tensors are not transposed
* tests : count flops instead of bandwidth in lightning indexer test
* chore : add missing const
* chore : whitespace
* ggml : renamed variables in CPU lightning indexer implementation
* ggml : fix lightning indexer mask broadcasting
* tests : tests for lightning indexer mask broadcasting
* chore : whitespace
* llama : use GGML_OP_LIGHTNING_INDEXER in DeepSeek V3.2 and DeepSeek V4 models
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com >
2026-07-11 11:39:07 +02:00
f57cd915a9
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/ui-publish.yml
# CODEOWNERS
# docs/ops.md
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/argsort-ops.c
# scripts/sync-ggml.last
# tools/mtmd/tests/test-deepseek-ocr.py
2026-07-11 11:36:59 +08:00
0f0245161e
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# ggml/src/ggml-opencl/ggml-opencl.cpp
# tests/test-backend-ops.cpp
# tests/test-quantize-fns.cpp
# tools/server/README.md
# tools/ui/src/lib/constants/settings-registry.ts
2026-07-11 09:20:40 +08:00
fc6e197fdd
Merge commit '33ca0dcb9d78c7c3a3b543db4c5fc9182abfe519' into concedo_experimental
...
# Conflicts:
# docs/backend/SYCL.md
# docs/build.md
# docs/ops.md
# docs/ops/SYCL.csv
# ggml/src/ggml-cuda/ggml-cuda.cu
# ggml/src/ggml-hip/CMakeLists.txt
# ggml/src/ggml-opencl/fa_tune.h
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/flash_attn_f16.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_q4_0.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_q8_0.cl
# ggml/src/ggml-opencl/kernels/mul_mv_f16_f32_l4.cl
# ggml/src/ggml-sycl/backend.hpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/cpy.cpp
# ggml/src/ggml-sycl/cpy.hpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/element_wise.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/presets.hpp
# src/llama-graph.cpp
# src/llama-kv-cache.cpp
2026-07-11 08:46:16 +08:00
082b326fc7
ggml-et: Initial ET backend ( #24179 )
...
* ggml-et: Add performance logging
* ggml-et: Quants helpers
* ggml-et: Add MUL_MAT kernel
* ggml-et: Add ROPE kernel
* ggml-et: Add RMS_NORM kernel
* ggml-et: Add GLU kernel
* ggml-et: Add SOFT_MAX kernel
* ggml-et: Add GET_ROWS kernel
* ggml-et: Add CONT kernel
* ggml-et: Add SET_ROWS kernel
* ggml-et: Add MUL_MAT_ID kernel
* ggml-et: Build et kernels as part of ggml
* ggml-et: Embed kernels with fs fallback
* ggml-et: Build fixes
* ggml-et: Add MUL_MAT F32xF32 op
* ggml_et: Add MUL_MAT_ID op
* ggml-et: Disable offloading for debug
* ggml-et: Refactor out block ops
* ggml-et: ggml backend API changes
* ggml-et: Add RESHAPE/TRANSPOSE to supported
* ggml-et: Add CONT_F16
* ggml-et: Add supported ops doc
* gglm-et: Initial doc
* ggml-et: Remove runtime import hacks
We can now import the runtime by a simple find_package(), so we
can cleanup the CMakeLists.txt.
* ggml-et: Fix GET_ROWS kernel
Fix lost batch dimension.
Also clean vibe-comments.
* ggml-et: Fix SET_ROWS kernel
Remove incorrect broadcasting guard.
* ggml-et: Use custom instruction for fp32->fp16
* ggml-et: Vectorize set_rows fp32->fp16
* ggml-et: Fix ROPE kernel (yarn)
ggml-et: fix et_logf
WIP: Fix ramp
WIP: fix ROPE!
* ggml-et: Better sinf
* ggml-et: Fix SOFT_MAX
Add `max_bias` and `sink` support.
* ggml-et: Fix CONT
Reorder from contiguous write to read with atomic stores.
* ggml-et: Fix elmap kernel
Remainder handlin
* ggml-et: Fix MUL_MAT MUL_MAT_ID remainders
* ggml-et: Fix ET-SOC reference
* ggml-et: Fix embed kernels scripts for old python
This allows GGML-ET to build on pre-3.8 python.
* Add sysemu support with compile time flag `-DGGML_ET_SYSEMU=ON` (#6 )
* Example using ET-Soc-1 emulator configuration
Example usage:
```bash
cmake -B build -DGGML_CUDA=OFF -DGGML_ET=ON -DLLAMA_CURL=OFF -DGGML_CCACHE=ON
cmake --build build --config Release -j $(nproc)
time ./build/bin/test-backend-ops
./build/bin/llama-server \
--model Qwen3-0.6B-Q8_0.gguf \
--alias Qwen3-0.6B-Q8_0 \
-fa 0 \
--ctx-size 1024 \
--no-warmup \
--host 127.0.0.1 \
--port 8080
```
* build: proper dep tracking for kernels
* support host using MOLD linker
* initial multi core GET_ROW F32 implementation
* vectorized q8 dequant
* wip: cland warning clenaups and initial logging refactor
* wip: message default message cleanup
* chore: message cleanups
* cmake cleanup
* migrate to use platform provided functions
* cmake back into subdir
* support et_print() in kernels
* fix: repair kernel building
* perf: operations run async by default
* debug: proper kernel dep tracking and error detection on kenrel launch
* fix: kernel binary dep tracking and fixing get_rows_f32 erroring
* perf: back to doing async kernel runs by default
* perf: vectorize and parallel device memset
* merge matmul work
* misc: align allocation and enable all offload
* misc: delete deadcode and respect memory limits
* fix: repair tensor debug print
* fix: loosen RMS_NORM op percision
* feat: Q4_0 GET_ROWS
* perf: FP32 MUL_MAT using TensorFMA
* update limitations
* perf: redue L1 load in compute_block_dot_product_q8_0
* feat: save kernel mapping (name to id) when profiling is enabled
* chore: memops cleanup
* perf: parallelize softmax by rows
* perf: vectorize 2nd phase of softmax
* perf: ban GET_ROWS from offloaded
* perf: vectorize and non-atomic for eltwise ops and sub support
* perf: vectorize normal rope
* perf: glu runs in parallel
* merge: manually merge saqib's work on kernel fixes
* perf: more vectorized RoPE
* perf: parallelize mul_mat_id
* perf: parallelize set_rows_f32
* perf: vectorize softmax
* feat: support kernel fusion and fuse RMS_NORM + MUL
* fix: mostly resolve test-backend-ops failure in SOFT_MAX and ROPE
* fix: bump max rope dims for gemma
* feat: GeGLU and SCALE support to fully offload Gemma
* perf: faster device memset
* feat: get_rows supporting Q4_K and avoid cont cache coherent issues
* better F32 MM
* feat: NORM for ET backend
* feat: SQR for ET backend
* feat: UNARY on ET
* feat: el_map support broadcasting for ET
* feat: SUM_ROWS in ET backend
* feat: more ops in ET backend
* feat: WKV* operators in ET backend
* perf: parallelize operators across cacheline instead of row
* perf: parallelize get_rows on cacheline
* wip: baseline FlashAttention for ET backend
* wip: enough FA and CPY f32->f16 to run llama 3.1 fully offloaded with FA on
* feat: f16 x f16 -> f32 MM using matrix engine
* wip: f16 FlashAttention using matrix engine
* wip: clean up
* feat: barriers
* perf: optimize FA_F16 in ET
* perf: vectorize pack_k_for_transpose16
* perf: prefetch next loop matrix tile
* perf: FlashAttention 2nd MM uses TensorFMA and optimizations
* cleanup: flashattention reorg
* perf: optimizations and fixes
* feat: L2SCP API and make FlashAttention support DV = 256 for gemma
* perf: parallelize norms beyond single row
* feat: GATED_DELTA_NET support and relaxed L2_NORM requirment
* feat: loosen RMS_NORM, NORM, ROPE contingous req too
* feat: repeat supports brocasting on dim 0 and loosen cont check
* feat: FILL and DIAG operator
* feat: loosen UNARY support chcek
* feat: TRI support
* feat: SOLVE_TRI support
* feat: basic SET support
* feat: loosen CONT req
* perf: fp16_to_fp32 use ASM
* feat: IMROPE support
* feat: PAD support
* feat: global barrier
* fix: view must live on the same backend as backing tensor
* feat: relax CONCAT in ET backend
* feat: dead simple CUMSUM implementation
* feat: basic SSM_CONV support
* feat: loosen CONCAT req
* feat: relax GATED_DELTA_NET and add SET support proper
* cleanup: cleanup LCM math
* feat: SWIGLU single input
* feat: SSM_SCAN support
* feat: el_map supports non aligned tensors in best effort
* feat: basic GROUP_NORM support
* feat: loosen MUL_MAT capablities slightly
* feat: loosen MUL_MAT and GET_ROWS and add IM2COL
* feat: special case for softmax 1x1x1x1
* feat: loosen SOFT_MAX req in ET backend
* fix: el_map unaligned acse fixes
* perf: optimize zero_acc_vec in flash_attn_ext_f16_me
* perf: use hart 1 for packing in MM and FA for FP16
* feat: kernel semaphore
* perf: better instruction sequence in FlashAttention
* fix: gated_delta_net with proper masking
* perf: better parallelization for GATED_DELTA_NET
* perf: parallelize SSM_CONV over nr
* perf: vectorize SSM_CONV
* perf: optimize MUL_MAT for q8
* feat: support Gemma 4
* fix: support multi-device
* feat: broader GLU support
* feat: unary ops supports view
* fix: repair fp16 MM using matrix engine
* perf: handle large N GEMV better
* perf: better q8_0 MM
* perf: better set_rows
* add back deleted files
* fix: repair after merge
* feat: POC version of uberkernel
* feat: RMS_NORM in uberkernel
* feat: add more kernels into usage
* chore: clean up uberkernel compilation
* perf: faster flash attention
* perf: opt flash attention for large seq length
* feat: loosen op bounds. clamp and mean support
* perf: vectorize ssm_scan
* perf: slightly faster FA
* perf: FlashAttention parallel MM and load
* perf: fuse Q8 MM and ADD
* feat: basic conv kernel for ET
* softMAx_test
* set_rows_f32
* get_rows and cont
* testing
* set_rows_exp
* Junk addition
* Narrowing the issue
* Update flash_attn_ext_f16_me.c
Focusing FA_ext_f16_me
* test
* Eviction updated
* Detailed cache eviction debug
* mulmat
* removeal of `BUILD_FOR_UBERKERNEL` flag
* cleaning...
* fix: balance FCC0 count
* feat: implement mul_mat and mul_mat_id for Q4_0 type
* optimize uberkernel plan upload
* add mul_mat q4 into uberkernel
* enable gating flush to just uberkernel
* update docs for ET
* update op support for ET
* et-backend: optimize Q4_0 and Q8_0 mul_mat_id row accumulations
* et-backend: specialize mul_mat_id kernels for Q4_0 and Q8_0
* et-backend: fix RoPE YaRN corr_dim formula and handle degenerate inputs
* test-backend-ops: add DeepSeek-V2-Lite RoPE test coverage
* et-backend: add Q4_0 mul_mat matrix-engine kernel using TensorFMA32
* et-backend: vectorize Q4_0 matrix-engine dequantization
* et-backend: support hybrid matrix/vector engine execution for Q4_0 mul_mat tail
* et-backend: run partial-N tiles on matrix engine for Q4_0 mul_mat
* et-backend: route Q4_0 mul_mat N < 53 to vecdot for better prefill latency
* Update uberkernel.c
* Update unary_f32.c
* gemma 4
* bisect gemma4: enable scale_f32 only
* bisect gemma4: +rms_norm_f32
* bisect gemma4: +rms_norm_mul_f32
* bisect gemma4: disable rms_norm_mul_f32 -- BREAKS OUTPUT
* bisect gemma4: +rope_f32 (skip rms_norm_mul)
* bisect gemma4: +el_map_f32
* bisect gemma4: +softmax_f32
* bisect gemma4: +get_rows_f32
* bisect gemma4: +glu_f32
* bisect gemma4: +mul_mat_f32 +mul_mat_f32_matrix_engine
* bisect gemma4: +mul_mat_f16 +mul_mat_f16_matrix_engine
* bisect gemma4: +mul_mat_Q8_0 +mul_mat_Q4_0
* bisect gemma4: +flash_attn_ext_f32 +flash_attn_ext_f16_me
* bisect gemma4: +mul_mat_id_f32
* bisect gemma4: +sum_rows_f32
* bisect gemma4: +cont_f16
* bisect gemma4: +fill_f32
* bisect gemma4: +unary_f32 (all ops re-enabled except rms_norm_mul)
* Update rms_norm_mul_f32.c
* bisect2 gemma4 n64: +scale_f32 only
* bisect2 gemma4 n64: +rms_norm_f32 +rope_f32
* bisect2 gemma4 n64: +rms_norm_mul_f32 (with ET_UBERKERNEL eviction fix)
* bisect2 gemma4 n64: +el_map +get_rows +glu +softmax (skip rms_norm_mul)
* bisect2 gemma4 n64: all ops enabled except rms_norm_mul
* bisect2 n64: test unary+cont+fill+sum_rows (no mul_mat/flash_attn)
* bisect2 n64: +mul_mat_f32 +mul_mat_f32_matrix_engine
* bisect2 n64: +mul_mat_f16 +mul_mat_f16_matrix_engine
* bisect2 n64: +mul_mat_Q8_0 +mul_mat_Q4_0
* bisect2 n64: +mul_mat_Q8_0 only (disable Q4_0)
* bisect2 n64: +mul_mat_Q4_0 only (Q8_0 breaks)
* bisect2 n64: +mul_mat_id +flash_attn_ext (skip Q8_0)
* run-3: matmul + rms_norm_mul
* run-4
* Revert "run-4"
* run5
* changes after cleanup
* cleanup before upstream
* restrict changes into ET backend
* move kernel embedding from Python to CMake
* move uberkernel gen into CMake
* apply clang format
* update CMake style
* update to match C and C++ style
* use source ggml and quant headers instead of ET's
* MROPE support
* absorb view ops into same branch as none
* fix bad rebase
* add marty1885 to codeowners
* oops
* remove redundant newline
* fix CI editor warnings
---------
Co-authored-by: Vidas <vidas@nuolat.lt >
Co-authored-by: Gianluca Guida <glguida@tlbflush.org >
Co-authored-by: Gianluca Guida <gianluca@nekko.ai >
Co-authored-by: ubergarm <leimgrub@gmail.com >
Co-authored-by: SaqibAkram-10xE <saqib.akram@10xengineers.ai >
Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai >
2026-07-10 12:38:34 +08:00
bec4772f6a
Add Q2_0 quantization: type definition and CPU backend ( #24448 )
2026-07-07 12:05:47 -07:00
74976e1aef
CUDA: remove -sm row, refactor cuBLAS ( #24216 )
...
* CUDA: remove -sm row, refactor cuBLAS
* fix CDNA + BF16 logic
* fix bad return
* fix src0 strides, contiguous requirements
* fix GGML_CUDA_FORCE_CUBLAS
* fix casts to BF16
2026-07-06 20:04:53 +02:00
e9fb3b3fc0
sycl : support --split-mode tensor ( #24152 )
...
* Sycl tp stage1 (#1 )
* SYCL: tensor parallelism (--split-mode tensor) for dual-GPU
Adds the comm_init/comm_free/comm_allreduce_tensor trio that the
meta-backend queries via get_proc_address to enable backend-specific
all-reduce, mirroring the pattern used by ggml-cuda.cu.
For N=2 (the common dual-GPU case) implements a degenerate ring
all-reduce with two size-branched paths:
* Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel
chained via depends_on(memcpy_event). 4 SYCL submissions/call.
* Large (nelem >= 32768): BF16-compressed. Each device compresses
FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's
inbox (HALF the PCIe bytes), then decompresses + adds into the
local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved
-- wins for any tensor where PCIe dominates kernel time.
Threshold and BF16 path pattern mirror the CUDA NCCL allreduce.
Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes
(matches both path layouts: FP32 nelem floats; BF16 outbox+inbox =
2 * nelem uint16_t each). Single alloc+free per device keeps the
SYCL pool's strict-LIFO invariant trivial.
Initial impl handles N=2 FP32 contiguous tensors. Other cases return
false, causing the meta-backend to use its generic butterfly fallback.
Per-call sync is intentionally omitted. SYCL in-order queue semantics
ensure that the meta-backend's next compute on the same per-device
queue waits for our final ADD, and the next allreduce's first op on
the same persistent buffer waits via the same queue. Only comm_free
does an explicit final wait.
OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process
in communicator_impl.hpp:47 (condition devices.size() == 1), which is
incompatible with llama.cpp's single-process multi-GPU model.
Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 +
DPC++ nightly):
Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16:
pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%)
tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%)
Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE):
pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%)
tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%)
Qwen3-4B Q4_K_M:
pp64 = 984.51 t/s, tg16 = 49.29 t/s
Llama-3.3-70B in SYCL TP now comfortably beats production layer mode
on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on
both — the BF16 path is what unlocks the many-medium-allreduces
prefill pattern.
Build/CMake: no changes. No new dependencies. ~210 lines added across
ggml-sycl.h and ggml-sycl.cpp.
* Fix comments
* documentation update to address PR feedback
* Bring over my device-to-device memcpy chagnes
* move the dev2dev_memcpy calls to the upstream 7-parameter variety
* Fix a typo and remove a trailing whitespace
2026-06-25 08:35:21 +03:00
be1750a3b8
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cuda.Dockerfile
# .github/workflows/build-sycl.yml
# .github/workflows/release.yml
# ggml/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-opencl/kernels/gated_delta_net.cl
# ggml/src/ggml-sycl/gated_delta_net.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/gated_delta_net.wgsl
# scripts/sync-ggml.last
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
2026-06-12 15:39:57 +08:00
e95dae18d6
Remove padding and multiple D2D copies for MTP ( #24086 )
...
* Make ggml_gated_delta_net take only the initial recurrent state (D, 1, n_seqs) and passes the snapshot count K as an op parameter instead of inferring it from state->ne[1].
Remove the padding hack and copy all emitted snapshots into the recurrent cache with a single strided ggml_cpy
* Make GDN changes in all backends. Address review comments.
* Fix CI build errors
2026-06-10 23:21:16 +05:30
afc8de2c6b
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/vulkan.Dockerfile
# .devops/zendnn.Dockerfile
# .github/workflows/build-webgpu.yml
# .github/workflows/release.yml
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/binary.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/concat.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/scale.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/unary.wgsl
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-mtmd-c-api.c
# tools/cli/cli.cpp
# tools/mtmd/CMakeLists.txt
# tools/server/README.md
2026-06-10 17:21:05 +08:00
26021699bc
ggml : add GGML_OP_COL2IM_1D ( #24206 )
...
* cpu: add GGML_OP_COL2IM_1D
Add the overlap-add (scatter-add) step of a 1D transposed convolution.
A ConvTranspose1d factorizes as a GEMM followed by col2im: a weight
pre-permuted to [IC, K*OC] is contracted against the [IC, T_in] input
with mul_mat to produce a column matrix [K*OC, T_in], and col2im_1d
scatters those columns back into the [T_out, OC] signal, with
T_out = (T_in - 1)*s0 + K - 2*p0.
Keeping the contraction as a plain mul_mat leaves the heavy work on the
optimized (and quantizable) matmul kernels, so col2im_1d only does the
cheap overlap-add.
CPU uses a gather formulation parallelized over output channels,
supporting F32, F16 and BF16 with an F32 accumulator.
* tests: add backend coverage for GGML_OP_COL2IM_1D
Add test_col2im_1d next to the conv_transpose_1d cases, covering F32,
F16 and BF16 across eight geometries: the canonical kernel = 2*stride
DAC upsampling shape, overlap, no overlap, cropping (p0 = 1 and
p0 = stride/2), kernel < stride with zeroed gaps, kernel not a
multiple of stride, and a single column unfold.
Perf mode gets three real vocoder stage shapes reporting memory
bandwidth. max_nmse_err relaxes to 5e-4 for F16 and BF16.
* cpu: harden GGML_OP_COL2IM_1D
ggml_col2im_1d validates s0, oc, p0 and input contiguity at graph
build time, before the oc division, protecting every backend at once.
The kernel asserts the contiguity its flat indexing assumes and its
doc states the full output length including the crop term.
The kernel parallelizes over the time axis: the split stays balanced
down to OC = 1, where the previous channel split was single threaded.
Values are bit identical on the three real vocoder chains, two out of
three improve.
* tests: extend the GGML_OP_COL2IM_1D grid
The eval grid grows to eleven geometries: OC = 1 (mono output stage),
K = 1 with stride > 1 (sparse scatter, every gap position zeroed) and
a crop down to T_out = 2 where all the gather bounds act at once.
* tests: add col2im_1d equivalence test
tests/test-col2im-1d.cpp proves mul_mat + col2im_1d matches the
native ggml_conv_transpose_1d on the CPU backend, F32 bit exact, F16
and BF16 through casts of the column matrix. test-backend-ops cannot
cover this for a CPU only op since the CPU backend is its own
reference there.
* rpc: bump protocol patch version for GGML_OP_COL2IM_1D
GGML_OP_COUNT goes from 96 to 97 with the new op, which trips the
static_assert in ggml-rpc.h. Bump RPC_PROTO_PATCH_VERSION since the
op is appended and no existing op code shifts.
2026-06-09 12:01:37 +03:00
153214c66a
Merge commit '5aba5364d959756c43eccc45596731e7b4eb3aa4' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-apple.yml
# .github/workflows/build-cpu.yml
# .github/workflows/build-openvino.yml
# .github/workflows/build-rpc.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/build-webgpu.yml
# .github/workflows/server.yml
# SECURITY.md
# docs/backend/SYCL.md
# docs/backend/ZenDNN.md
# docs/build.md
# docs/ops.md
# docs/ops/SYCL.csv
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/convert.cpp
# ggml/src/ggml-sycl/dequantize.hpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/fattn-common.hpp
# ggml/src/ggml-sycl/getrows.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/quants.hpp
# ggml/src/ggml-sycl/vecdotq.hpp
# tests/test-reasoning-budget.cpp
# tools/llama-bench/README.md
# tools/llama-bench/llama-bench.cpp
# tools/mtmd/CMakeLists.txt
2026-06-02 18:30:40 +08:00
8e6fff84de
TP: quantized KV cache support ( #23792 )
...
* TP: quantized KV cache support
* fix partial view
* remove overly strict assert
2026-06-01 12:30:10 +02:00
9fbc0cd7dc
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-and-test-snapdragon.yml
# .github/workflows/build-android.yml
# .github/workflows/build-cache.yml
# .github/workflows/build-cann.yml
# .github/workflows/build-cross.yml
# .github/workflows/build-openvino.yml
# .github/workflows/build-riscv.yml
# .github/workflows/build-sanitize.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/build-sycl.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/build.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# ci/run.sh
# docs/backend/SYCL.md
# docs/backend/snapdragon/README.md
# ggml/CMakeLists.txt
# ggml/src/ggml-cuda/ggml-cuda.cu
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.c
# ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-utils.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/set-rows-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/common_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_acc.tmpl
# scripts/sync-ggml.last
# tests/gguf-model-data.cpp
# tests/test-backend-ops.cpp
2026-05-27 00:26:08 +08:00
9204f78926
Merge commit 'ae251b5ff2634108822e0f8bb20ca4cd5c2c5dcc' into concedo_experimental
...
# Conflicts:
# .github/actions/linux-setup-spacemit/action.yml
# .github/actions/unarchive-tar/action.yml
# .github/workflows/build-android.yml
# .github/workflows/build-cmake-pkg.yml
# .github/workflows/build-cross.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/build.yml
# .github/workflows/check-vendor.yml
# .github/workflows/code-style.yml
# .github/workflows/editorconfig.yml
# .github/workflows/pre-tokenizer-hashes.yml
# .github/workflows/python-check-requirements.yml
# .github/workflows/python-lint.yml
# .github/workflows/python-type-check.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui.yml
# .github/workflows/update-ops-docs.yml
# ci/run.sh
# docs/build-riscv64-spacemit.md
# examples/convert_legacy_llama.py
# ggml/cmake/ggml-config.cmake.in
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/sync_vendor.py
# tests/test-chat-auto-parser.cpp
# tests/test-chat.cpp
# tests/test-gguf.cpp
# tools/cli/README.md
# tools/perplexity/perplexity.cpp
# tools/server/README.md
2026-05-26 22:01:57 +08:00
b251f74f49
ggml.h: correct ggml_silu_back arg docstring (a=dy, b=x) (ggml/1500)
2026-05-25 12:38:01 +03:00
66efd13375
ggml: gguf_init_from_callback and gguf_init_from_buffer ( #22341 )
...
* ggml: implement `gguf_init_from_buffer`
* test: `gguf_init_from_buffer`
* fix: memory breakdown for a model loaded with `no_alloc` from a file is consistent with being loaded from a buffer
* fix: use `GGML_UNUSED`
Co-authored-by: Copilot <copilot@github.com >
* fix: remove `total_size` from `gguf_reader`
* fix: file offset calculation, rename `offset` to `data_offset`
Co-authored-by: Copilot <copilot@github.com >
* refactor: extract model loader bug fixes to another PR
* feat: add `gguf_init_from_callback`
* fix: always require a max expected size
* fix: change `gguf_reader_callback_t`'s `output` type to `void *`, change `max_expected_size` and offsets to `uint64_t`
* fix: harden against offset overflow in buffer read
* fix: remove seek behavior from the callback
* feat: `max_chunk_read == 0` means `SIZE_MAX`
* fix: seeking in a gguf file with no tensors
---------
Co-authored-by: Copilot <copilot@github.com >
2026-05-25 11:33:29 +02:00
fff63b5108
TP: fix entirely zero-sized slices per device ( #23525 )
2026-05-24 08:19:33 +02:00
Concedo
Georgi Gerganov
Aman Gupta
Radoslav Gerganov
Xuan-Son Nguyen
lnigam
Ruben Ortlam
Pascal
Rajendra Matcha
fairydreaming
QuintinShaw
Martin Chang
Pasha Khosravi
Johannes Gäßler
David Spruill
Gaurav Garg
Ori Pekelman
Gilad S.