From fccf7166fb4c797567cf30d795828106031127b7 Mon Sep 17 00:00:00 2001 From: WenqiangJia2026 Date: Wed, 16 Sep 2026 15:55:02 +0800 Subject: [PATCH] HIP: broaden MoE ncols_opt tile heuristic on RDNA3.5 architecture (#28935) It's found the MoE ncols_opt tile heuristic needs to be broadened to include the RDNA3.5 architecture. The code change is implemented in ggml/src/ggml-cuda/mmq.cu and just change the GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3 in the condition. The dense dispatch logic remains unchanged. The Test machine configuration we used is AMD Radeon 8060S, gfx1151 (RDNA3.5), 20 CU, wave32 + AMD Ryzen AI MAX+ 388, 8C/16T, 23.79 GB RAM we complete the Correctness verification and performance evaluation as follows: test-backend-ops test -b ROCm0 -o MUL_MAT -p type_a= test-backend-ops test -b ROCm0 -o MUL_MAT_ID -p type_a= all pass: MUL_MAT 64/64, 29/29, 48/48, 14/14; MUL_MAT_ID 84/84, 3/3, 74/74, 3/3 Performance result on target machine: LFM2.5-8B-A1B-UD-Q4_K_M (Q4_K MoE) +16.198% [+12.704, +19.799] 8/8 Qwen1.5-MoE-A2.7B-Q2_K (Q2_K MoE) +6.189% [ +5.245, +7.141] 8/8 pooled (16 pairs) +11.081% [ +7.972, +14.279] 16/16 Token generation (tg128) is unchanged on the Q4_K MoE model and +2.188% [+0.905, +3.488] on the Q2_K one. --- ggml/src/ggml-cuda/mmq.cu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ggml/src/ggml-cuda/mmq.cu b/ggml/src/ggml-cuda/mmq.cu index 9b6038adf..b13b34ee9 100644 --- a/ggml/src/ggml-cuda/mmq.cu +++ b/ggml/src/ggml-cuda/mmq.cu @@ -247,7 +247,7 @@ void ggml_cuda_mul_mat_q( // Each expert only sees ne12*n_expert_used/ne02 tokens on average. // On RDNA3 and RDNA4 it is faster to pick the tile size against this value instead of ne12. int64_t ncols_opt = ne12; - if (GGML_CUDA_CC_IS_RDNA3_0(cc) || GGML_CUDA_CC_IS_RDNA4(cc)) { + if (GGML_CUDA_CC_IS_RDNA3(cc) || GGML_CUDA_CC_IS_RDNA4(cc)) { ncols_opt = (ne12*n_expert_used + ne02 - 1) / ne02; }