perf(qwen35): accelerate Ampere prefill GDN#524
Draft
dusterbloom wants to merge 2 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dependency
This is stacked on #518 (
perf(cuda): reduce MMQ stream-k overhead). The branch is currentmain(0e002364) plus #518 (506cd972) plus this commit (3c0be1f). Once #518 merges, this PR's visible diff should collapse to the GDN/QK change only.Performance
Matched Q4_K_M / F16-F16 KV prefill on one RTX 3090, CUDA 12.6, one slot, prompt caches disabled, one generated token, default ubatch 512, and no GDN/ubatch environment overrides. Each value pools ten measured samples from an A-B-B-A run with one warmup and five measurements per phase.
The deltas are approximately 7.0x, 7.3x, and 5.5x their combined standard errors. The exact same prompt payload and model artifact were used, but the runtimes report a two-token accounting difference (988/986, 8156/8154, and 16349/16347), so this is not presented as bit-identical tokenizer accounting. The raw result's AutoLuce content ID is
measurement-792d2c28b337ed544dc8c1e3e3bd699c65d8c2b4f182813dfac272c1359fb047.Correctness
gdn_grouped_policy: verifies the 512-token Ampere threshold and override precedenceqwen35_gdn_broadcast: verifies compact Q/K reaches fused GDN and the chunked fallback retains exactly two repeatsgdn_grouped_cuda: compares forced classic and grouped output/final state on separate graphs with freshly restored state at 1, 221, 477, 512, and 768 tokens using the real 16-Q/K-head / 48-value-head shape; all NMSE values are approximately1e-14, below the1e-6gateVerification
All three tests pass on SM86.