Public/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-08-30 00:51:19 +02:00
Code Issues Packages Projects Releases Wiki Activity

opencl: use a better matmul path on two Adreno GPU generations (#27640)

Browse Source 
* opencl: default the Adreno xmem F16xF32 GEMM on for X2E

kernel_mul_mm_f16_f32_l4_lm is the slowest matmul this backend has on Adreno: on
the X2-90 it runs the gpt-oss-20b attention projections at roughly a quarter of
what the tuned dense q4_0 GEMM reaches on the same device. That matters for any
model whose non-expert weights stay f16 -- the stock gpt-oss-20b release is
exactly that, and its prefill spends 40.8% of GPU time in that one kernel. The
xmem route already existed but was left opt-in, so nobody hit it.

Worth about 25% prefill on gpt-oss-20b on an Adreno X2-90. Gated to X2E: the
Adreno 840 measures neutral. Decode is untouched -- the dispatch gate needs
N >= 16. It is worth nothing on the q8attn variant, whose attention weights
already take the dp4a dense GEMM.

The env var was presence-tested before, so =0 previously enabled it; it is now
atoi()'d. MUL_MAT 963 OK / 0 FAIL on both arms.

* opencl: bypass the tiled f32 GEMM on the Adreno A7X

The A7X (E031.41) compiler executes kernel_mul_mm_f32_f32_l4_lm at roughly a
tenth of what the same silicon reaches in its own f16 and q4_K kernels. It
allocates 488 B/WI of private memory against 304 for the same source on the
following generation, i.e. the older register allocator spills in the K-loop.
Models with per-layer F32 projection pairs kept F32 by quantization policy land
on this kernel twice per layer, and it dominates their prefill on that part.

Route batched f32xf32 (ne11 > 8) around the tiled path on the A7X and let it
fall through to the per-row f32 kernel, which that compiler handles fine; small
batches keep the tiled path. Weights stay GPU-resident, so decode placement is
untouched -- declining the op in supports_op instead was measured first and
rejected, because the per-layer CPU round-trips cost more decode than the
prefill it gained.

Worth about 9% prefill on gemma-3n-E4B on an Adreno 740, with MUL_MAT counts
identical on and off. No other generation is affected. Override with
GGML_OPENCL_A7X_F32_LM_BYPASS=0.

* opencl: enable xmem GEMM for adreno by default

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
This commit is contained in:
Hongqiang Wang
2026-08-29 10:46:27 -07:00
committed by GitHub
parent 3173a56471
commit c841aeeb8b
1 changed files with 17 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-opencl/ggml-opencl.cpp
+17 -4
View File
@@ -6059,9 +6059,13 @@ static ggml_backend_opencl_context * ggml_cl_init(ggml_backend_dev_t dev) {
}
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
// determine whether to use Adreno xmem GEMM
backend_ctx->adreno_xmem_gemm_enabled = getenv("GGML_OPENCL_ADRENO_XMEM_GEMM") != nullptr &&
backend_ctx->gpu_family == GPU_FAMILY::ADRENO;
// Adreno xmem F16xF32 GEMM, default on adreno, opt out with GGML_OPENCL_ADRENO_XMEM_GEMM=0.
// This helps models with f16 attention weights, e.g., gpt-oss-20b-f16
{
const char * xmem_env = getenv("GGML_OPENCL_ADRENO_XMEM_GEMM");
backend_ctx->adreno_xmem_gemm_enabled = backend_ctx->gpu_family == GPU_FAMILY::ADRENO &&
(xmem_env ? atoi(xmem_env) != 0 : true);
}
#endif
// determine whether to use large buffer for Adreno
@@ -19534,9 +19538,18 @@ static void ggml_cl_mul_mat(ggml_backend_t backend, const ggml_tensor * src0, co
// GEMM using local memory
// Current BK = 16, so ne00 % 16 == 0
//
// Certain A7X compiler (E031.41) executes kernel_mul_mm_f32_f32_l4_lm poorly;
// matrices with ne11 <= 8 appears OK.
// Fallback to the MV style kernels for A7x and ne11 > 8.
// Override with GGML_OPENCL_A7X_F32_LM_BYPASS=0.
static const char * a7x_f32lm_env = getenv("GGML_OPENCL_A7X_F32_LM_BYPASS");
static const bool a7x_f32lm_bypass = (a7x_f32lm_env == nullptr || a7x_f32lm_env[0] != '0');
if (src1t == GGML_TYPE_F32 &&
ne00 % 16 == 0 &&
ne11 > 1) {
ne11 > 1 &&
!(a7x_f32lm_bypass && src0t == GGML_TYPE_F32 && ne11 > 8 &&
backend_ctx->adreno_gen == ADRENO_GPU_GEN::A7X)) {
switch(src0t) {
case GGML_TYPE_F32: {
kernel = backend_ctx->kernel_mul_mm_f32_f32_l4_lm;