Eval bug: low DFlash tps/AR for multi-GPU setups

[Anbeeld]

Name and Version

v0.3.0

Operating systems

Linux

GGML backends

CUDA

Hardware

Multi-GPU

Models

Qwen 3.6, Gemma 4

Problem description & steps to reproduce

Multiple reports:
#38 (comment)
#28 (comment)

First Bad Commit

No response

Relevant log output

Provided in linked reports

[Anbeeld]

@nikitabalakin @maxim-ts
Updated v0.3.0 with possible fix for multi-GPU acceptance, please try it.

[nikitabalakin]

kotokin@kotokin-x570ud:~/beellama.cpp/build/bin$ ./llama-server -m /home/kotokin/model/google_gemma-4-31B-it-Q6_K.gguf -ngl 999 -sm tensor -c 65536 --jinja -np 1 --ctx-checkpoints 4 --kv-unified -fa on --spec-type dflash --spec-draft-model /home/kotokin/model/gemma4-31b-it-dflash-Q6_K.gguf --spec-dflash-cross-ctx 1024 --spec-draft-device cuda0,cuda1,cuda2

Now, I get the usual speed, as when using tensor. 37 t/s

https://drive.google.com/file/d/1DEiXIHe1G2iG_3wWIHBxgfJ97YTCwhmu/view?usp=sharing

[maxim-ts]

This commend was originally added to #28. Adding it here for better visibility.

Used same command line used as base one:

./llama-server.exe `
  -m "C:\Models\unsloth\Qwen3.6-27B-Q5_K_S.gguf" `
  --spec-draft-model "C:\DFlash\Qwen3.6-27B-DFlash-Q4_K_M.gguf" `
  --spec-type dflash `
  --spec-dflash-cross-ctx 1024 `
  --host 0.0.0.0 `
  --port 8033 `
  -np 1 `
  --kv-unified `
  -ngl all `
  --spec-draft-ngl all `
  -b 2048 -ub 512 `
  --ctx-size 102400 `
  --cache-type-k q5_0 --cache-type-v q4_1 `
  --flash-attn on `
  --cache-ram 0 `
  --jinja `
  --no-mmap --mlock `
  --no-host `
  --reasoning on `
  --chat-template-kwargs '{\"preserve_thinking\":true}' `
  --temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0

I was using same 'double linked' prompt.

Built using latest commit from v0.3.0 branch ( specifically spiritbuun@3706706 ).

DFlash, --spec-draft-device CUDA0 = 108 tg/s, 30% acceptance
DFlash, --spec-draft-device not set = 95.3 tg/s, 31% acceptance
MTP, --spec-draft-device CUDA0 = 30.8 tg/s, 57% acceptance
MTP, --spec-draft-device not set = 33.7 tg/s, 53.9% acceptance
MTP with llama.cpp b9339 = 66 tg/s, 82% acceptance (all same settings, except KV set to q4_0/q4_0).
No spec decoding at all = 34.4 tg/s, GARBAGE

My major concern is DFlash acceptance with multi-gpu set up.
I was expecting to receive average acceptance of 50%, just as @Anbeeld had on average on single 3090. But currently having median acceptance of 30% with same prompt and same cmd params.

It is worth noticing that somehow acceptance is lower for MTP as well in beellama (53% vs 82% in llama.cpp). So maybe there is a shared root cause with DFlash.