Eval bug: Segfault at slot initialization with CUDA on SM75 (Turing) — ngl > 0

[Bino5150]

Name and Version

./llama-server --version
version: 9867 (e727109)
built with GNU 14.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

./llama-server --model model.gguf --port 8081 -ngl 32 -c 8192 --flash-attn on -ctk turbo4 -ctv turbo3 -t 6

Problem description & steps to reproduce

CPU: Intel i7
GPU: NVIDIA Quadro T1000 Max-Q (SM75 / Turing 4GB)
CUDA: 12.8

Server segfaults immediately after initializing slots when any CUDA layers are offloaded (-ngl > 0) on SM75 (Turing) hardware. CPU-only (-ngl 0) works correctly. Reproducible on both MTP and non-MTP model.

Segfaults at initializing slots with -ngl > 0
Warning: fused Gated Delta Net (chunked) not supported, set to disabled
CPU-only (-ngl 0) works fine
Reproducible on both MTP and non-MTP model
Occurs with and without --spec-type mtp
Occurs with and without --no-warmup
fused Gated Delta Net disabled warning appears consistently before crash
CPU-only run (-ngl 0) loads and serves correctly
Model: Qwopus3.5-4B-v3-MTP Q5_K_M GGUF

First Bad Commit

No response

Relevant log output

Logs

Here's the last log lines before the SegFault:
W sched_reserve: layer 0 is assigned to device CPU but the fused Gated Delta Net tensor is assigned to device CUDA0
W sched_reserve: fused Gated Delta Net (chunked) not supported, set to disabled
I srv load_model: initializing slots, n_slots = 4
[segfault]

[Anbeeld]

Could you please try these variants and report which ones crash?

# 1. CUDA offload, but no turbo KV / forced FA
./llama-server --model model.gguf --port 8081 -ngl 1 -c 8192 --flash-attn off -t 6

# 2. Same, but one slot only
./llama-server --model model.gguf --port 8081 -ngl 1 -c 8192 --flash-attn off --parallel 1 -t 6

# 3. Original-style settings, but one slot
./llama-server --model model.gguf --port 8081 -ngl 32 -c 8192 --parallel 1 --flash-attn on -ctk turbo4 -ctv turbo3 -t 6

# 4. Try full offload instead of mixed CPU/CUDA placement
./llama-server --model model.gguf --port 8081 -ngl 999 -c 8192 --flash-attn on -ctk turbo4 -ctv turbo3 -t 6

Also attach a backtrace from:

CUDA_LAUNCH_BLOCKING=1 gdb --args ./llama-server --model model.gguf --port 8081 -ngl 32 -c 8192 --flash-attn on -ctk turbo4 -ctv turbo3 -t 6
run
bt full

[Bino5150]

#1 (-ngl 1, no flash attn, no turbo KV) and #2 (-ngl 1, no flash attn, --parallel 1) loaded. #3 SegFault at warmup. Backtrace: null ptr in ggml_compute_forward_flash_attn_ext called from ggml_graph_compute_thread
The sched_reserve warnings about Gated Delta Net tensor CPU/CUDA conflict appearing just before the crash. Skipped #4 since #3 failed.

Backtrace:
#0 0x0000000000000000 in ?? ()
#1 ggml_compute_forward_flash_attn_ext ← crash here
#2 ggml_graph_compute_thread

Right before the crash:
W sched_reserve: layer 0 is assigned to device CPU but the fused Gated Delta Net tensor is assigned to device CUDA0
W sched_reserve: fused Gated Delta Net (chunked) not supported, set to disabled