Skip to content

[shardformer] Add explicit attention backends and packed varlen support - #6508

Merged
yuxuan-lou merged 2 commits into
hpcaitech:mainfrom
b1ack-j:feat/shared-attention-backends
Nov 18, 2026
Merged

yuxuan-lou merged 2 commits into
hpcaitech:mainfrom
b1ack-j:feat/shared-attention-backends

Conversation

@b1ack-j

@b1ack-j b1ack-j commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Issue

Summary

The existing ColoAttention entry uses legacy mask-based dispatch and does not provide an explicit FA3/FA2/SDPA contract with a common packed-varlen entry. This PR supplies that shared core for model adapters.

  • Add explicit backend selection, packed variable-length attention, GQA and causal alignment for unequal Q/K lengths.
  • Propagate selected-backend import and execution failures; never silently replace a requested FA2/FA3 call with SDPA.
  • Preserve FP32 additive-mask precision and exclude cuDNN SDPA for half-precision inputs with that mask, addressing incorrect results observed with Torch 2.13.
  • Add regression/numerical tests, extend existing test runners and update the English/Chinese ShardFormer documentation.

Model integration, 3D RoPE and NPU/Cosmos3 branch integration are separate work.

Compatibility

  • New backend arguments and attention_varlen are opt-in. Calls without backend retain the legacy dispatch entry. Existing training scripts do not need to adopt the new API.
  • Model parameters and checkpoint formats are unchanged. There is no data migration.
  • FA2/FA3 remain optional. Explicit FA2 requires version >=2.1 and D<=192; this FA3 adapter requires a compatible SM90 build, FP16/BF16 and zero dropout. Its pinned source is 54b044c8.
  • Excluding cuDNN for the affected FP32-mask combination can affect performance; it has not been benchmarked. Packed SDPA uses per-sample calls and is a correctness path, not a fused varlen kernel.
  • No new C++/CUDA kernel implementation, mandatory package dependency, shared image change, version bump, PyPI release or DockerHub publication is included.

Validation

Current head: 81c2cfb0a6858bcbe5e64cbcae72ec12fe28be04, one feature commit.

The final cleanup only removes the unused k argument from private _select_backend and the corresponding arguments at two production and four test callsites. AST comparison against cd2aff5f confirms that these are the only changes; calculations, validation and numerical judging are unchanged.

Validation Source Result
Attention CPU regression after cleanup current head 52 passed, 0 failed, 0 skipped; 550 GPU cases deselected; exit 0
Black 24.10.0 and diff whitespace checks current head passed
Full single-H20 numerical acceptance cd2aff5f3632b9f2dcd4e0957b5083b021314973 550 passed, 0 failed, 0 skipped; exit 0
Strict FA2 / FA3 / SDPA subsets same prior head 179 / 179 / 192 passed; no failures or skips; exit 0; overlap with the full run
Personal Linux full CPU batch same prior head 208 passed, 585 device/version skips; exit 0

GPU acceptance has not been rerun after this argument-only cleanup. The prior H20 evidence is retained under its original SHA, not relabeled as a new-head GPU run. Current GitHub checks are shown in the PR Checks tab; they are not claimed all green.

The GPU matrix covers FP16/BF16, D32/64/128, MHA/GQA, dense/packed and unequal sequences, causal alignment, SDPA masks, packed sample isolation, and D128 sequences of 512/2048. Outputs and dQ/dK/dV use independent FP64 truth and the unchanged team judging defaults. Actual backend calls, FA2/FA3 CUDA profiler traces and dense compilation are checked.

The previous three short-causal failures came from the low-precision reference backward recipe. The corrected independent recipe passes FP64 gradcheck; five-seed calibration accepts 30 correct cases and detects 60 injected scale/causal errors. This did not change production calculations, FP64 truth or judging tolerances.

Environment

Item Recorded environment
GPU acceptance OS / image Linux; image sha256:ae8ff34bf340a692803edca6e0f38a576fc1a2b206c01b4ef6f7c2a40dfd8c10; distribution release not recorded in the inspected acceptance metadata
Python 3.10.12
GPU 1 x NVIDIA H20, SM90
NVIDIA driver Acceptance process loaded libcuda.so.595.71.05
PyTorch / CUDA runtime 2.13.0+cu130 / 13.0
CUDA Toolkit / nvcc 13.0 toolchain; complete nvcc --version output was not retained in the inspected metadata
Transformers / Diffusers 5.17.0 / 0.36.0
FlashAttention FA2 2.8.4; fixed-source FA3 3.0.1; build flags and wheel hashes in the evidence manifest
Current cleanup CPU check macOS, Python 3.12.13, Torch 2.5.1, pytest 8.3.5; no GPU
Prior full CPU batch Linux, Python 3.10.12, Torch 2.5.1+cpu

Reproduction

Run from the repository root in an isolated environment with the dependencies above. GPU commands require one allocated GPU and the pinned FA2/FA3 builds.

COLOSSALAI_ACCELERATOR=cpu python -m pytest tests/test_shardformer/test_layer/test_attention_backends.py -k 'not gpu' -q
COLOSSALAI_TEST_ATTENTION_BACKEND=fa2 python -m pytest tests/test_shardformer/test_layer/test_attention_backends.py -k 'gpu and fa2'
COLOSSALAI_TEST_ATTENTION_BACKEND=fa3 python -m pytest tests/test_shardformer/test_layer/test_attention_backends.py -k 'gpu and fa3'
COLOSSALAI_TEST_ATTENTION_BACKEND=sdpa python -m pytest tests/test_shardformer/test_layer/test_attention_backends.py -k 'gpu and sdpa'
COLOSSALAI_TEST_ATTENTION_BACKEND=all python -m pytest tests/test_shardformer/test_layer/test_attention_backends.py -k gpu

The Attention suite does not use FAST_TEST to reduce its matrix. The formal GPU run additionally used the archived evidence plugin/wrapper to save reports and traces; its exact commands are in gpu/run.json.

Internal evidence: /mnt/beegfs/ColossalAI/shared-results/numerics/2F/attention/20261110-cd2aff5f-084103Z/. It contains JUnit, per-case JSON, three-way traces, profiler output, source/image/build manifests and hashes. Historical failures and subsequent runs are retained.

Known limitations

  • Dense single-token coverage is deferred: the complete 72-case family (previously 69 failed, 3 passed) was removed. This reduces acceptance scope and does not fix the zero-gradient-budget boundary. Packed cases containing single-token samples remain.
  • The prior GitHub CPU check failed in test_unsynchronized_parameter_detected during worker exit, not in Attention numerical comparison. A separate diagnosis reproduced SIGABRT on the upstream baseline in 3/5 runs. A cleanup candidate still hung in additional minimal experiments and is not included here.
  • The prior fork GPU check was skipped; personal H20 evidence does not turn that check green. Dynamic packed fullgraph, D192/256, other hardware and performance tuning were not accepted in this run.

Follow-up issues, implementation owners and completion dates for the deferred items are not assigned in this PR.

Risks and rollback

The main risks are optional-extension build/ABI compatibility and performance differences from backend selection. Explicit unsupported or broken backends fail rather than silently falling back. Revert the complete PR change to restore the previous entry and adapters; no checkpoint or data conversion is needed.

@b1ack-j
b1ack-j requested a review from a team as a code owner November 4, 2026 07:48
@b1ack-j
b1ack-j force-pushed the feat/shared-attention-backends branch 3 times, most recently from 9790b3b to cd2aff5 Compare November 10, 2026 03:22
@b1ack-j
b1ack-j force-pushed the feat/shared-attention-backends branch from cd2aff5 to 81c2cfb Compare November 10, 2026 11:27
@b1ack-j
b1ack-j force-pushed the feat/shared-attention-backends branch 2 times, most recently from e1fa37e to 81c2cfb Compare November 11, 2026 05:06

@JazyJiang JazyJiang left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the strict explicit-backend contract and the FP64-referenced test matrix look good to me. The comments fall into three groups.

A. Hard-coded parts worth loosening for other devices

This PR does not need to add any new device backend. The aim is only that a future backend can be registered, rather than wired into several if-chains and lookup tables.

  1. Backend names and the CUDA check are hard-coded (attn.py L138–L161, kernel_loader.py L145–L160). The name check, the auto order, q.device.type != "cuda", and the two name tables in FlashAttentionBackendLoader would all need editing for each new backend, and an unknown name hits a bare KeyError. Suggestion: keep one table, name -> (device types, capability check, extension class), and derive the auto order from q.device.type (CUDA: fa3 → fa2 → sdpa; other devices: their registered backends, then sdpa). Behaviour on CUDA stays the same.

  2. attention_varlen syncs on every call and only takes device tensors (attn.py L263–L271; the SDPA path reads them again at flash_attention_sdpa_cuda.py L106).

    • prefix.tolist() is a device→host sync per layer per step. FA2/FA3 varlen themselves do not need it, since the max lengths are Python ints.
    • Other device kernels, such as the Ascend varlen API (actual_seq_qlen), take host-side lists, and packers usually already hold the lengths on the host.

    Suggestion: also accept Sequence[int]. Validate it on the host and convert it once to device int32 for FA2/FA3. For tensors, add a validate: bool = True switch. Zero-length samples (rejected at L271) are worth a second look too: some packers legitimately emit them, so either filter them inside or document the rejection.

  3. State the bool mask convention as a contract. True = may attend is the right choice. Please write it down as the rule every backend must follow, in the docstring and the ShardFormer docs. The legacy path is inconsistent here: CUDA SDPA reads True as attend, while the NPU extension calls .bool() and passes the mask to a kernel where True means masked. A future device backend should convert internally.

B. Possible bugs

  1. FA3 return value (flash_attention_dao_v3_cuda.py L41 and L63). At the pinned revision, flash_attn_func / flash_attn_varlen_func return only out, but earlier hopper builds return (out, softmax_lse). With such a build, the dense path fails at .transpose and the varlen path returns a tuple. Since is_available only checks find_spec("flash_attn_3"), that build would still be selected. Suggest unwrapping a tuple, or checking the version or signature.
  2. lower_right with sq > sk (attn.py L203–L209). The first sq - sk query rows see no key at all. FA2/FA3 return 0 for those rows, while SDPA with an all-False bool row may return NaN depending on the kernel, as the legacy path itself notes at L531. Because auto can pick different backends on different machines, the same call could give different outputs and NaN gradients. Could you either reject this case in _validate_causal, or define the output for empty rows and add a test? I could not tell whether the unequal-length cases include sq > sk.
  3. ROCm passes the SM90 check (attn.py L161). ROCm builds also use device type "cuda", and get_device_capability returns the gfx major, e.g. (9, 4) on MI300. That passes both the FA3 and the FA2 checks. Suggest gating on torch.version.hip is None, or folding this into the table in item 1.

C. CI / validation

  1. Batch 4f fails by design when flash_attn_3 is missing. Please confirm the CI image has it, or document that 4f is manual-only on hosts that do.
  2. The current head has not been re-run on GPU (as the description says), and CPU unit tests is currently red, which the description attributes to an upstream flake. A link to a passing re-run would help once there is one.

@b1ack-j

b1ack-j commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @JazyJiang, I addressed your feedback in 1d1eef68.

  • 1–3: Backend selection now uses one registry. Varlen accepts host offsets and can skip the host read for prevalidated FA2/FA3 inputs.
  • 4–6: The FA3 adapter handles both return formats. Tests now check zero output and zero query gradient for fully masked rows. The NVIDIA adapters explicitly exclude ROCm.
  • 7–8: I documented 4f as manual and ran the tests directly on H20. 568 GPU tests passed, including 185 strict FA3 cases. The PR’s Fork-safe CPU check is still red. I reproduced the Gloo shutdown failure on upstream main, and the full CPU check passed with a separate fix.
    Could you take another look when you have a chance?

@b1ack-j
b1ack-j requested a review from JazyJiang November 18, 2026 04:47

@JazyJiang JazyJiang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1d1eef6, LGTM.

@JazyJiang

Copy link
Copy Markdown
Collaborator

The remaining CPU unit tests failure here (SIGABRT at worker exit in tests/test_numerics) is handled separately in #6566, which includes the teardown fix from your investigation run and applies it to every numerical worker. Nothing needs to change in this PR for it.

@yuxuan-lou
yuxuan-lou merged commit d3916b5 into hpcaitech:main Nov 18, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants