[TIRx][CUDA] Version-gate CUDA 12.8 tensor-map enums and fix registry-test lock leak - #20154
Open
spectrometerHBH wants to merge 2 commits into
Open
[TIRx][CUDA] Version-gate CUDA 12.8 tensor-map enums and fix registry-test lock leak#20154spectrometerHBH wants to merge 2 commits into
spectrometerHBH wants to merge 2 commits into
Conversation
…fdef CU_TENSOR_MAP_SWIZZLE_128B_ATOM_32B and friends are enumerators, not macros, so every #ifdef guard around them was always false and the features they gate were unconditionally rejected -- requesting the 128B_ATOM_32B swizzle failed host validation with "Unsupported swizzle enum value: 4" even on CUDA 13.2. Use CUDA_VERSION >= 12080 checks for the CUDA 12.8 additions and drop the dead guards around the base-enum TFLOAT32/FLOAT32_FTZ members, which exist wherever the tensor-map API does.
… MegaMoE Release the per-device flocks in a nested finally so a poisoned CUDA context (empty_cache raising after a kernel fault) cannot leak a held lock: pytest keeps the failure's traceback frame alive, so the leaked lock's fd never closes and the worker deadlocks against itself when its next test reopens the same lock file. Skip sm100_fp8_fp4_mega_moe here: its dedicated multi-process scheduler validates physical-device assignments that reject any process already owning a CUDA context, which every process in this suite does (the free-memory probe alone creates one).
This was referenced Aug 18, 2026
tlopex
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent fixes found while running the TIRx kernel registry correctness suite on sm100.
Version-gate the CUDA 12.8 tensor-map enums.
CU_TENSOR_MAP_SWIZZLE_128B_ATOM_32Band the other CUDA 12.8 additions are enumerators, not macros, so every#ifdefguard around them incuda_device_api.ccwas always false: the swizzle/dtype variants they gate were unconditionally rejected, and requesting a128B_ATOM_32Bswizzle failed host validation withUnsupported swizzle enum value: 4even on CUDA 13.2. The guards are nowCUDA_VERSION >= 12080checks, and the dead guards around the base-enumTFLOAT32/FLOAT32_FTZmembers (present wherever the tensor-map API exists) are dropped.Harden the registry-test GPU locking. The per-device flock release ran after
torch.cuda.empty_cache()in the samefinally; when a kernel fault poisons the CUDA context,empty_cache()raises, and pytest's saved traceback keeps the frame (and the lock file descriptors) alive — the held flock never releases, and the same worker deadlocks against itself when its next test reopens the lock file. The release now sits in a nestedfinally. Also skipsm100_fp8_fp4_mega_moein this suite: its dedicated multi-process scheduler validates physical-device assignments that reject any process already owning a CUDA context, which every process in this suite has by the time the free-memory probe has run.Verified on sm100: the two
gdn_cp_prefill_sm100configs that requested the128B_ATOM_32Bswizzle now pass, the MegaMoE configs skip, and the previously deadlocking registry suite completes.