Repository navigation
[shardformer] Add explicit attention backends and packed varlen support - #6508
Conversation
9790b3b to
cd2aff5
Compare
cd2aff5 to
81c2cfb
Compare
e1fa37e to
81c2cfb
Compare
There was a problem hiding this comment.
Thanks, the strict explicit-backend contract and the FP64-referenced test matrix look good to me. The comments fall into three groups.
A. Hard-coded parts worth loosening for other devices
This PR does not need to add any new device backend. The aim is only that a future backend can be registered, rather than wired into several if-chains and lookup tables.
-
Backend names and the CUDA check are hard-coded (
attn.pyL138–L161,kernel_loader.pyL145–L160). The name check, theautoorder,q.device.type != "cuda", and the two name tables inFlashAttentionBackendLoaderwould all need editing for each new backend, and an unknown name hits a bareKeyError. Suggestion: keep one table,name -> (device types, capability check, extension class), and derive theautoorder fromq.device.type(CUDA: fa3 → fa2 → sdpa; other devices: their registered backends, then sdpa). Behaviour on CUDA stays the same. -
attention_varlensyncs on every call and only takes device tensors (attn.pyL263–L271; the SDPA path reads them again atflash_attention_sdpa_cuda.pyL106).prefix.tolist()is a device→host sync per layer per step. FA2/FA3 varlen themselves do not need it, since the max lengths are Python ints.- Other device kernels, such as the Ascend varlen API (
actual_seq_qlen), take host-side lists, and packers usually already hold the lengths on the host.
Suggestion: also accept
Sequence[int]. Validate it on the host and convert it once to device int32 for FA2/FA3. For tensors, add avalidate: bool = Trueswitch. Zero-length samples (rejected at L271) are worth a second look too: some packers legitimately emit them, so either filter them inside or document the rejection. -
State the bool mask convention as a contract.
True= may attend is the right choice. Please write it down as the rule every backend must follow, in the docstring and the ShardFormer docs. The legacy path is inconsistent here: CUDA SDPA readsTrueas attend, while the NPU extension calls.bool()and passes the mask to a kernel whereTruemeans masked. A future device backend should convert internally.
B. Possible bugs
- FA3 return value (
flash_attention_dao_v3_cuda.pyL41 and L63). At the pinned revision,flash_attn_func/flash_attn_varlen_funcreturn onlyout, but earlierhopperbuilds return(out, softmax_lse). With such a build, the dense path fails at.transposeand the varlen path returns a tuple. Sinceis_availableonly checksfind_spec("flash_attn_3"), that build would still be selected. Suggest unwrapping a tuple, or checking the version or signature. lower_rightwithsq > sk(attn.pyL203–L209). The firstsq - skquery rows see no key at all. FA2/FA3 return 0 for those rows, while SDPA with an all-False bool row may return NaN depending on the kernel, as the legacy path itself notes at L531. Becauseautocan pick different backends on different machines, the same call could give different outputs and NaN gradients. Could you either reject this case in_validate_causal, or define the output for empty rows and add a test? I could not tell whether the unequal-length cases includesq > sk.- ROCm passes the SM90 check (
attn.pyL161). ROCm builds also use device type"cuda", andget_device_capabilityreturns the gfx major, e.g.(9, 4)on MI300. That passes both the FA3 and the FA2 checks. Suggest gating ontorch.version.hip is None, or folding this into the table in item 1.
C. CI / validation
- Batch 4f fails by design when
flash_attn_3is missing. Please confirm the CI image has it, or document that 4f is manual-only on hosts that do. - The current head has not been re-run on GPU (as the description says), and
CPU unit testsis currently red, which the description attributes to an upstream flake. A link to a passing re-run would help once there is one.
|
Thanks @JazyJiang, I addressed your feedback in 1d1eef68.
|
|
The remaining |
Issue
Summary
The existing
ColoAttentionentry uses legacy mask-based dispatch and does not provide an explicit FA3/FA2/SDPA contract with a common packed-varlen entry. This PR supplies that shared core for model adapters.Model integration, 3D RoPE and NPU/Cosmos3 branch integration are separate work.
Compatibility
attention_varlenare opt-in. Calls withoutbackendretain the legacy dispatch entry. Existing training scripts do not need to adopt the new API.Validation
Current head:
81c2cfb0a6858bcbe5e64cbcae72ec12fe28be04, one feature commit.The final cleanup only removes the unused
kargument from private_select_backendand the corresponding arguments at two production and four test callsites. AST comparison againstcd2aff5fconfirms that these are the only changes; calculations, validation and numerical judging are unchanged.cd2aff5f3632b9f2dcd4e0957b5083b021314973GPU acceptance has not been rerun after this argument-only cleanup. The prior H20 evidence is retained under its original SHA, not relabeled as a new-head GPU run. Current GitHub checks are shown in the PR Checks tab; they are not claimed all green.
The GPU matrix covers FP16/BF16, D32/64/128, MHA/GQA, dense/packed and unequal sequences, causal alignment, SDPA masks, packed sample isolation, and D128 sequences of 512/2048. Outputs and dQ/dK/dV use independent FP64 truth and the unchanged team judging defaults. Actual backend calls, FA2/FA3 CUDA profiler traces and dense compilation are checked.
The previous three short-causal failures came from the low-precision reference backward recipe. The corrected independent recipe passes FP64 gradcheck; five-seed calibration accepts 30 correct cases and detects 60 injected scale/causal errors. This did not change production calculations, FP64 truth or judging tolerances.
Environment
sha256:ae8ff34bf340a692803edca6e0f38a576fc1a2b206c01b4ef6f7c2a40dfd8c10; distribution release not recorded in the inspected acceptance metadatalibcuda.so.595.71.05nvcc --versionoutput was not retained in the inspected metadataReproduction
Run from the repository root in an isolated environment with the dependencies above. GPU commands require one allocated GPU and the pinned FA2/FA3 builds.
The Attention suite does not use
FAST_TESTto reduce its matrix. The formal GPU run additionally used the archived evidence plugin/wrapper to save reports and traces; its exact commands are ingpu/run.json.Internal evidence:
/mnt/beegfs/ColossalAI/shared-results/numerics/2F/attention/20261110-cd2aff5f-084103Z/. It contains JUnit, per-case JSON, three-way traces, profiler output, source/image/build manifests and hashes. Historical failures and subsequent runs are retained.Known limitations
test_unsynchronized_parameter_detectedduring worker exit, not in Attention numerical comparison. A separate diagnosis reproduced SIGABRT on the upstream baseline in 3/5 runs. A cleanup candidate still hung in additional minimal experiments and is not included here.Follow-up issues, implementation owners and completion dates for the deferred items are not assigned in this PR.
Risks and rollback
The main risks are optional-extension build/ABI compatibility and performance differences from backend selection. Explicit unsupported or broken backends fail rather than silently falling back. Revert the complete PR change to restore the previous entry and adapters; no checkpoint or data conversion is needed.