Bug: lunch moss-transcribe-cpp-0.9b success but do a request failed

[yourchanges]

LocalAI version:
4.7.1

Environment, CPU architecture, OS, and Version:

[root@aiserver2 localai]# nvidia-smi
Thu Jul 16 18:10:50 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080        Off |   00000000:04:00.0 Off |                  N/A |
|  0%   35C    P2            109W /  320W |    1109MiB /  20480MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3080        Off |   00000000:05:00.0 Off |                  N/A |
|  0%   33C    P8             22W /  320W |    2631MiB /  20480MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 3080        Off |   00000000:08:00.0 Off |                  N/A |
|  0%   34C    P8             17W /  320W |       4MiB /  20480MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 3080        Off |   00000000:09:00.0 Off |                  N/A |
|  0%   32C    P8             12W /  320W |       4MiB /  20480MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA GeForce RTX 3080        Off |   00000000:85:00.0 Off |                  N/A |
|  0%   32C    P8             13W /  320W |       4MiB /  20480MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA GeForce RTX 3080        Off |   00000000:86:00.0 Off |                  N/A |
|  0%   31C    P8             16W /  320W |       4MiB /  20480MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA GeForce RTX 3080        Off |   00000000:89:00.0 Off |                  N/A |
|  0%   29C    P8             11W /  320W |       4MiB /  20480MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA GeForce RTX 3080        Off |   00000000:8A:00.0 Off |                  N/A |
|  0%   32C    P8             14W /  320W |       4MiB /  20480MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
[root@aiserver2 localai]# uname -an
Linux aiserver2 5.14.0-611.24.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jan 23 11:42:43 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
lunch the moss-transcribe-cpp-0.9b via localai webui, and do a request ,then it fails

To Reproduce
curl -X POST http://aidev2:8080/v1/audio/transcriptions
-F "file=@./output.wav"
-F "model=moss-transcribe-cpp-0.9b"
-F "response_format=text"
{"error":{"code":500,"message":"rpc error: code = Unavailable desc = error reading from server: EOF","type":""}}

Expected behavior
success

Logs

18:09:59.320
stdout
Using lib/ld.so
18:09:59.403
stderr
[moss-transcribe-cpp] ABI=1
18:09:59.403
stderr
2026/07/16 18:09:59 gRPC Server listening at 127.0.0.1:31623
18:10:02.185
stderr
ggml_cuda_init: found 8 CUDA devices (Total VRAM: 160439 MiB):
18:10:02.186
stderr
  Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 20054 MiB
18:10:02.188
stderr
  Device 1: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 20054 MiB
18:10:02.190
stderr
  Device 2: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 20054 MiB
18:10:02.192
stderr
  Device 3: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 20054 MiB
18:10:02.194
stderr
  Device 4: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 20054 MiB
18:10:02.196
stderr
  Device 5: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 20054 MiB
18:10:02.198
stderr
  Device 6: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 20054 MiB
18:10:02.201
stderr
  Device 7: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 20054 MiB
18:10:02.222
stderr
[mt I] backend: CUDA0
18:10:02.753
stderr
[mt I] loaded /home/localai/models/moss-transcribe-cpp/moss-transcribe-q5_k.gguf: 684 tensors, 47 kv (backend=CUDA0)
18:10:49.317
stderr
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
18:11:21.981
stderr
[mt I] Tokenizer: loaded 151672 tokens, 151387 merges, 17 special
18:11:21.982
stderr
/LocalAI/backend/go/moss-transcribe-cpp/sources/moss-transcribe.cpp/third_party/ggml/src/ggml-cuda/getrows.cu:226: ggml_cuda_get_rows_switch_src0_type: unsupported src0 type: q5_K
18:11:21.983
stderr
18:11:22.041
stderr
gdb: symbol lookup error: /home/localai/backends/cuda12-moss-transcribe-cpp/lib/libc.so.6: undefined symbol: __tunable_is_initialized, version GLIBC_PRIVATE
18:11:22.043
stderr
SIGABRT: abort
18:11:22.043
stderr
PC=0x14782049eb2c m=0 sigcode=18446744073709551610
18:11:22.043
stderr
signal arrived during cgo execution

Additional context

[localai-bot]

Thanks for the detailed report - root cause found and a fix is up.

Root cause: moss-transcribe-q5_k.gguf stores the token embedding table (token_embd.weight) as q5_K. ggml's CUDA backend implements the GET_ROWS op only for F32/F16/BF16/I32 and the classic quants (q4_0/q4_1/q5_0/q5_1/q8_0) - no K-quants. The moss-transcribe.cpp engine runs its graphs on a single backend with no CPU fallback scheduler, so on CUDA the token embedding lookup for the very first request hits GGML_ABORT in getrows.cu ("unsupported src0 type: q5_K"). That kills the backend process, which is the rpc error: ... EOF you saw. CPU-only runs of the same model are unaffected, and it is not specific to your setup - any CUDA (or other GPU) run of the q5_k/q4_k/q6_k GGUFs would crash the same way.

Fix: engine-side, in localai-org/moss-transcribe.cpp#2. The embed lookup now asks ggml_backend_supports_op first and, when the backend cannot run GET_ROWS for the tensor type, dequantizes the needed rows on the host with the reference dequant (bit-identical results). Verified on a CUDA 13 box: the current pinned build reproduces your exact abort with moss-transcribe-q5_k.gguf, the fixed build transcribes it fine and the transcript is byte-identical to the CPU run. A new regression test covers all shipped quant types on both CPU and CUDA.

Once that PR merges, LocalAI's moss-transcribe-cpp backend pin will be bumped and the next backend image rebuild will pick it up. In the meantime, a workaround on current images is to force CPU for this model (set the env MTD_DEVICE=cpu for the backend) - the model is fast on CPU as well.