Skip to content

perf(qwen35): accelerate Ampere prefill GDN#524

Draft
dusterbloom wants to merge 2 commits into
Luce-Org:mainfrom
dusterbloom:perf/qwen35-ampere-prefill
Draft

perf(qwen35): accelerate Ampere prefill GDN#524
dusterbloom wants to merge 2 commits into
Luce-Org:mainfrom
dusterbloom:perf/qwen35-ampere-prefill

Conversation

@dusterbloom

@dusterbloom dusterbloom commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • keep Q/K at their compact 16-head shape for fused Qwen3.5 Gated DeltaNet instead of materializing two 16-to-48 repeats
  • select the existing grouped-column GDN kernel automatically for NVIDIA Ampere prefill launches of at least 512 tokens while retaining the classic decode/partial-chunk path
  • preserve the existing force/disable diagnostics and the opt-in chunked fallback's required Q/K repeats
  • add pure selector, graph-structure, and focused CUDA numerical parity tests

Dependency

This is stacked on #518 (perf(cuda): reduce MMQ stream-k overhead). The branch is current main (0e002364) plus #518 (506cd972) plus this commit (3c0be1f). Once #518 merges, this PR's visible diff should collapse to the GDN/QK change only.

Performance

Matched Q4_K_M / F16-F16 KV prefill on one RTX 3090, CUDA 12.6, one slot, prompt caches disabled, one generated token, default ubatch 512, and no GDN/ubatch environment overrides. Each value pools ten measured samples from an A-B-B-A run with one warmup and five measurements per phase.

Prompt depth Lucebox stack llama.cpp Delta
1K 1341.47 tok/s 1278.72 tok/s +4.91%
8K 1393.73 tok/s 1359.53 tok/s +2.52%
16K 1332.63 tok/s 1306.86 tok/s +1.97%

The deltas are approximately 7.0x, 7.3x, and 5.5x their combined standard errors. The exact same prompt payload and model artifact were used, but the runtimes report a two-token accounting difference (988/986, 8156/8154, and 16349/16347), so this is not presented as bit-identical tokenizer accounting. The raw result's AutoLuce content ID is measurement-792d2c28b337ed544dc8c1e3e3bd699c65d8c2b4f182813dfac272c1359fb047.

Correctness

  • gdn_grouped_policy: verifies the 512-token Ampere threshold and override precedence
  • qwen35_gdn_broadcast: verifies compact Q/K reaches fused GDN and the chunked fallback retains exactly two repeats
  • gdn_grouped_cuda: compares forced classic and grouped output/final state on separate graphs with freshly restored state at 1, 221, 477, 512, and 768 tokens using the real 16-Q/K-head / 48-value-head shape; all NMSE values are approximately 1e-14, below the 1e-6 gate

Verification

cmake --build /tmp/lucebox-prefill-gdn-q4-build --target dflash_server test_gdn_grouped_policy test_qwen35_gdn_broadcast test_gdn_grouped_cuda -j4
ctest --test-dir /tmp/lucebox-prefill-gdn-q4-build -R '^(gdn_grouped_policy|qwen35_gdn_broadcast|gdn_grouped_cuda)$' --output-on-failure

All three tests pass on SM86.

Review in cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant