GPU RepeatInterleaveF32 kernel launch segfaults (SIGSEGV, null-pointer CUDA call) -- crashes any GQA model on GPU

[dndungu]

Summary

compute.(*GPUEngine).RepeatInterleave -> gpuapi.(*CUDAKernels).RepeatInterleaveF32 -> the purego/cgo kernel call crashes the whole process with SIGSEGV: segmentation violation, PC=0x0 -- a null-pointer CUDA kernel launch, not a recoverable Go error. Reproduced on ztensor v1.19.2 on an NVIDIA GB10 (aarch64).

This is reachable from zerfoo's layers/attention/grouped_query_attention.go via its "fused RepeatInterleave" decode optimization (added 2026-03-30, zerfoo@641a119b), whenever numQueryHeads != numKeyValueHeads -- true for essentially every modern GQA architecture (Llama, Mistral, Qwen, Gemma) -- and SDPA runs without a flash-decode shortcut. Because it's a hard SIGSEGV rather than an error return, zerfoo's own fallback path (Reshape -> Repeat -> Reshape) is never reached; the type-assertion gate around the fused call cannot protect against a crash inside the call itself.

Repro

zerfoo tests/parity.TestGPUParity_GQA reproduces it directly:

dModel, nQHeads, nKVHeads := 32, 4, 2
// ... construct GroupedQueryAttention with these head counts, WithNoRoPE ...
gpuOut, err := gpuGQA.Forward(ctx, gpuInput) // <-- SIGSEGV here

Full stack trace (zerfoo main, GB10):

SIGSEGV: segmentation violation
PC=0x0 m=0 sigcode=1 addr=0x0
signal arrived during cgo execution

goroutine ... [syscall]:
runtime.cgocall(0x91e130, ...)
	runtime/cgocall.go:167
github.com/zerfoo/ztensor/internal/cuda._Cfunc_ccall_wrapper(0x0, ...)
	_cgo_gotypes.go:93
github.com/zerfoo/ztensor/internal/cuda.ccall(...)
	internal/cuda/purego_linux_arm64_cgo.go:64
github.com/zerfoo/ztensor/internal/cuda.Ccall(...)
	internal/cuda/purego.go:238
github.com/zerfoo/ztensor/internal/cuda/kernels.RepeatInterleaveF32(0xf06420000900, 0xf06420000a00, 0x1, 0x2, 0x2, 0x8, 0x2, 0x21abf240)
	internal/cuda/kernels/fused_repeat_interleave_purego.go:23
github.com/zerfoo/ztensor/internal/gpuapi.(*CUDAKernels).RepeatInterleaveF32(...)
	internal/gpuapi/cuda_kernels.go:218
github.com/zerfoo/ztensor/compute.(*GPUEngine[...]).RepeatInterleave(...)
	compute/gpu_engine_memory.go:666

The first argument to the underlying ccall (0x0) looks like a null kernel handle or device pointer -- consistent with the call site at gpu_engine_memory.go:641/666 passing an empty/nil dst (the RepeatInterleave signature accepts a variadic optional dst ...*tensor.TensorNumeric[U]), suggesting the kernel path may not correctly allocate/handle the no-dst-provided case before launching.

Environment

• ztensor v1.19.2
• NVIDIA GB10, aarch64, CUDA 13.0, driver 580.159.03
• purego/dlopen kernel binding (no explicit -tags cuda build)

Current mitigation (in zerfoo, not a real fix)

zerfoo has disabled the fused path unconditionally (fusedRepeatInterleaveEnabled = false in layers/attention/grouped_query_attention.go) and falls back to the slower Reshape -> Repeat -> Reshape sequence, which does not crash and produces numerically correct output (verified maxDiff=7.6e-6 against CPU). This trades away the fused-kernel perf win until this is fixed upstream.

Ask

Root-cause the null-pointer launch in RepeatInterleaveF32's purego/cgo path (internal/cuda/kernels/fused_repeat_interleave_purego.go:23 and internal/gpuapi/cuda_kernels.go:218), likely around how the dst argument is handled when the caller doesn't pre-allocate one. Once fixed and released, zerfoo can bump its ztensor dependency and re-enable the fused path.