ggml-cuda: pad GEMM weight rows to avoid cache-set aliasing

[roberteg16]

Summary

When a GEMM weight's row size in bytes is a multiple of 2048, consecutive rows can collide in the same last-level cache sets, throttling the weight-memory-bound matmuls. This pads such a weight's row stride by one 128-byte cache line, breaking the aliasing. It is enabled by default and can be disabled with GGML_CUDA_NO_PAD_WEIGHTS. It is a pure memory-layout change: the same values are stored at a padded stride and every consuming kernel is fed that stride, so results are unchanged within floating-point tolerance. The change lives in the generic CUDA/HIP backend and is not tied to any GPU family; how much it helps depends on the device's cache geometry.

What changed

• ggml.h: add GGML_TENSOR_FLAG_PAD_ROWS, a backend hint marking a GEMM weight whose row stride may be padded.
• llama-model-loader.cpp: tag weights whose intended op is GGML_OP_MUL_MAT / GGML_OP_MUL_MAT_ID (via llm_tensor_info_for), and upload their rows with a strided 2D copy so packed on-disk data lands in the padded layout.
• ggml-cuda.cu: reserve the padded allocation and set the padded nb[1..3] in the buffer allocator/initializer; make the cuBLAS path pass the strided weight through un-repacked and use lda = nb[1] / type_size; skip quantized weights, whose block size a 128-byte pad would misalign.
• common.cuh: add ggml_cuda_is_contiguous_rows, shared by the cuBLAS and vector-kernel paths.
• mmvf.cu: honor the padded row stride in the split-path callback.

Scope

Only float tensors used as a GEMM src0 are padded (attention q/k/v/o, ffn gate/up/down, output); embeddings (get_rows), norms, and biases are never touched. Quantized weights are explicitly excluded because a 128-byte pad is not a multiple of their block size (144/176/210 bytes, etc.) and would misalign the block-indexed matmul kernels. The feature is enabled by default; setting GGML_CUDA_NO_PAD_WEIGHTS restores the previous byte-for-byte layout.

Performance (Radeon 8060S, -fa 1, f16 KV; on = default, off = GGML_CUDA_NO_PAD_WEIGHTS)

Models are F16/BF16 because the optimization only affects float weights whose row size in bytes is a multiple of 2048; it has no effect on Q4_K_M (or any other k-quant) models, whose block layout can never form a 2048-byte row, so those configurations run completely unchanged. Since the benchmark dashboard runs its llama.cpp models as Q4_K_M, the numbers below use F16/BF16 builds to exercise the paths this change touches. Qwen3-1.7B and Qwen3-8B are F16 builds of dashboard model families; Llama-3.2-1B, Gemma-3-1B, and Phi-4-mini (BF16) add architecture coverage including the bf16 GEMM path; OLMoE-1B-7B exercises the MoE mul_mat_id path; and Gemma-3-1B doubles as a no-op control whose K dimensions never hit the alias.

Prefill speedup (prompt tok/s, on vs off) across shapes:

shape	Qwen3-1.7B	Llama-3.2-1B	Phi-4-mini BF16	Gemma-3-1B
pp128	+128%	+169%	+62%	+4%
pp256	+121%	+162%	+38%	+5%
pp512	+61%	+120%	+25%	+3%
pp1024	+53%	+113%	+21%	-6%
pp2048	+52%	+109%	+25%	-4%
pp4096	+48%	+106%	+21%	+2%
pp8192	+39%	+92%	+17%	+1%

Larger dense model (Qwen3-8B F16): pp512 +44%, pp2048 +37%, pp4096 +35%; TTFT (pp4096 @ depth 4096) +22% faster; decode +5%.

MoE (OLMoE-1B-7B F16, exercises mul_mat_id): pp512 +78%, pp2048 +75%.

TTFT (pp4096 prefill latency, on faster than off) at increasing context depth:

depth	Qwen3-1.7B	Llama-3.2-1B
0	+34%	+52%
4096	+23%	+44%
8192	+17%	+38%

Decode (tg128 tok/s, on vs off): Qwen3-1.7B +11% (depth 0) / +9% (depth 4096); Llama-3.2-1B +9% at both.

Gemma-3-1B is flat because its K dimensions (n_embd 1152, n_ff 6912) are not multiples of 1024, so no F16 row hits the 2048-byte alias and almost nothing is padded; it is a control confirming the change is a no-op when the aliasing condition is not met.

Correctness

Perplexity is identical with padding on (default) vs off: Qwen3-1.7B 1.0043 == 1.0043 and Qwen3-8B 1.0024 == 1.0024 and Llama-3.2-1B 1.0162 == 1.0162 (dense, mul_mat), and OLMoE-1B-7B 1.0093 == 1.0093 (MoE, mul_mat_id). Deterministic greedy output matches across qwen3, gemma3, and phi3 (bf16); Llama occasionally flips a single near-tie token, which is the expected effect of cuBLAS selecting a different tiling for the new leading dimension, not a correctness regression. Quantized (Q4_K) models are a verified no-op.

Test plan

• Builds with -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release.
• mul_mat correctness: llama-perplexity on dense F16 models is identical on (default) vs GGML_CUDA_NO_PAD_WEIGHTS (Qwen3-1.7B, Llama-3.2-1B).
• mul_mat_id correctness: llama-perplexity on an F16 MoE is identical on vs off (OLMoE-1B-7B).
• llama-bench prefill/TTFT/decode show the gain on F16/BF16 models; a Q4_K model is unchanged.

[mgehre-amd]

I guess this needs some gfx11 qualification as this is not generally applicable to all CUDA GPUs