[VK] `linear_dq8ca_q4gsw` silently produces wrong results when `M > K/2` — int8 input-sums scratch buffer is undersized

[xuyanwen2012]

🐛 Describe the bug

Context: I was trying to run LLaMA 3.1 8B and LLaMA 3.2 1/3B in 8da4w quantization under the stable release 1.3 (fresh clone, no inputs from me at all).
I ran it at 2048 prefill and 1024 decode and found linear_dq8ca_q4gsw silently produces wrong results on all hardware, not just one.

Minimal Reproduction

Add one case to backends/vulkan/test/custom_ops/test_q4gsw_linear.cpp, in generate_quantized_linear_test_cases():

{256, 128, 128, 32}, // M=256 > K=128; all dims < kRefDimSizeLimit so the reference runs

Build Instructions

Note that backends/vulkan/test/custom_ops is a standalone CMake project (project(prototyping_shaders) + find_package(executorch CONFIG REQUIRED)); it is not built by -DEXECUTORCH_BUILD_TESTS=ON, so it must be configured separately against the installed runtime.

1. Build runtime with Vulkan:

cmake . -Bcmake-out-vk --preset linux \
-DCMAKE_INSTALL_PREFIX=cmake-out-vk \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_PAL_DEFAULT=posix \
-DEXECUTORCH_BUILD_VULKAN=ON

cmake --build cmake-out-vk -j$(nproc) --target install --config Release

2. Build and run the custom_ops test project:

cmake -S backends/vulkan/test/custom_ops -B cmake-out-vk-tests \
-DCMAKE_PREFIX_PATH=$PWD/cmake-out-vk \
-DCMAKE_BUILD_TYPE=Release

cmake --build cmake-out-vk-tests -j$(nproc) --target test_q4gsw_linear

./cmake-out-vk-tests/test_q4gsw_linear

Expected Result

The framework's own reference check catches the mismatch and aborts (exit 134 / SIGABRT):

Mismatch at element 0: computed=1.189116, reference=0.183432, diff=1.006, 
abs_tolerance=0.002, rel_tolerance=0.001

Correctness validation FAILED for test 
ACCU f32 [256,128]x[128,128] g32 Tex(WP) [linear_dq8ca_q4gsw] no_bias

terminate called after throwing an instance of 'std::runtime_error'

Versions

Collecting environment information...
PyTorch version: 2.12.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Rocky Linux 10.2 (Red Quartz) (x86_64)
GCC version: (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4)
Clang version: 21.1.8 (RESF 21.1.8-1.el10)
CMake version: version 3.31.10
Libc version: glibc-2.39

Python version: 3.12.13 (main, Jul 10 2026, 00:00:00) [GCC 14.3.1 20251022 (Red Hat 14.3.1-4)] (64-bit runtime)
Python platform: Linux-6.12.0-211.39.1.el10_2.x86_64-x86_64-with-glibc2.39
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Caching allocator config: N/A

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7940HS w/ Radeon 780M Graphics
CPU family: 25
Model: 116
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 1
Frequency boost: enabled
CPU(s) scaling MHz: 58%
CPU max MHz: 5263.0610
CPU min MHz: 402.7860
BogoMIPS: 7985.16
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca flush_l1d amd_lbr_pmc_freeze
Virtualization: AMD-V
L1d cache: 256 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 8 MiB (8 instances)
L3 cache: 16 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Old microcode: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsa: Mitigation; Clear CPU buffers
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

Versions of relevant libraries:
[pip3] executorch==1.4.0a0+39c0df6
[pip3] flake8==6.1.0
[pip3] flake8-breakpoint==1.1.0
[pip3] flake8-bugbear==24.4.26
[pip3] flake8-comprehensions==3.14.0
[pip3] flake8-plugin-utils==1.3.3
[pip3] flake8-pyi==23.5.0
[pip3] mypy==1.14.1
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.5.0
[pip3] pytorch_tokenizers==1.3.0
[pip3] torch==2.12.0+cpu
[pip3] torchao==0.17.0+git02105d46c
[conda] Could not collect

[xuyanwen2012]

AI debugger came back with a root cause:

The scratch buffer holding per-row, per-group int8 activation sums in the dynamically-quantized

Vulkan linear path (linear_dq8ca_q4gsw) is allocated with **both the wrong dimension and the

wrong element type**. It is sized by K in the output's floating-point dtype, but the shaders

index it by M as ivec4 of int32.

The allocation is large enough only while sizeof(out_dtype) * K >= 4 * M. Past that bound the

shaders read out of bounds, the activation zero-point correction term silently drops out, and the

op returns plausible-looking but numerically wrong output. There is no crash, no validation

error, and no throughput anomaly.

This affects any long-prompt prefill using 8da4w / dq8ca group-quantized weights on the Vulkan

backend. For a force_fp16 Llama 3.2 1B, any prompt longer than 1024 tokens is silently wrong.

Root cause

backends/vulkan/runtime/graph/ops/impl/QuantizedLinear.cpp (:661 on release/1.3, :836 on main):

TmpTensor int_input_sums(

&graph,

{num_groups, K}, // <-- sized by K; shaders index by M

graph.dtype_of(output), // <-- fp dtype; shaders read int32

utils::kBuffer,

utils::kWidthPacked);

Both shaders that touch this buffer declare it as an ivec4 array of int32 and index it by rows:

// glsl/quantize_and_pack_4h4w_with_group_sums.glsl:33 (declaration)

${layout_declare_tensor(B, "w", "t_int8_input_sums", "int", "buffer", is_scalar_array=False)}

// :120 (write)

t_int8_input_sums[group_idx * M4 + m4] = shared_sums[group_offset][0];

  

// glsl/linear_int8_input_sums_load.glslh:20 (read)

return t_int8_input_sums[quant_group_idx * M4 + m4];

with M4 = div_up_4(M). So:

• allocated = sizeof(out_dtype) * num_groups * K bytes

• required = 16 * num_groups * div_up_4(M) = 4 * num_groups * M bytes

The two mistakes partially cancel, which is why the op works at all for small M.

The consumer is glsl/linear_fp_output_tile_int8_int4_compute.glslh:74-76:

ivec4 accum_adjusted = accum.data[m][n4] -

(weight_sums.data[n4] + 8 * group_size) * input_zp_m +

mul_8(group_size * input_zp_m - input_sum_m);

When input_sum_m reads back as 0 (out-of-bounds), the activation zero-point correction is

silently wrong for every affected row.

[SS-JIA]

Fix will be ready soon.