A rank-local planning exception can leave another rank blocked in the next WORLD reduction instead of failing the operation together. This is a separate failure-handling gap discovered while qualifying #848/#870; it is not an observed CUDA OOM.
A bounded two-rank CPU/Gloo reproducer executes the original TrainerRank selector, search and estimator methods from PR #900 head 2b84bc90d08c0457687b46e264a67b49b4b88dae (runtime SHA256 5df3a1138923faa6705862037cd067d4b8b27cbdb9ef99c1094116efeb237d23). Both ranks finish the checkpoint-prerequisite barrier. Rank 0 then raises its original ValueError in the pure local packing estimator; rank 1 proceeds to _all_ranks_true and waits alone in WORLD MIN until the configured 15-second timeout. The origin stays alive until its peer records the timeout, so process teardown does not manufacture the failure.
The post-cache-release path completes 14 matched collectives before the mismatch (peer timeout 15.007 seconds). The same exception with no recovery also reproduces it (15.005 seconds; no release, first-use permission unconsumed, cost/work ledgers zero). This establishes the weakness in the tested current source, not that #900 introduced it. Nine healthy asymmetric recovery cases retain 84 matched reductions. Original error identity/cause/context are checked, and all test children/groups are closed.
The fix must coordinate the pure local planning outcome after collective prerequisites and before the next planner collective. A catch around the entire search followed by an outcome reduction is too late. Preserve legal unavailable estimates, local original exception chains, current plan math and recovery accounting. Audit equivalent materialization/pricing boundaries; avoid reductions per local group because group counts can differ across ranks.
Qualification limits: real Gloo reductions and CPU tokens; CUDA counters/release, timing/model callbacks, and checkpoint loading are explicit facades. This is not NCCL, full-model, GPU-memory, or communicator-repair qualification.
Owner: Schulman; Peirce implements, Faraday independently reviews. Local correction is in progress, with no shared runtime change or merge yet. Runtime scope should remain art.trainer_rank, with no public API or art.megatron changes.
Durable evidence:
/var/tmp/art870-cooperating-recovery-20260916-physical-_frpqj6e/manifest.json (8d0d8e43), source, rank traces, ordinary control and cleanup./var/tmp/art870-cooperating-recovery-review-20260916-forward/manifest.json (bbe3183b), independent verification./var/tmp/art870-planning-exchange-proposal-20260916-physical-veyf2nl8/manifest.json (68ab0a7c), proposed correction boundaries.
A rank-local planning exception can leave another rank blocked in the next WORLD reduction instead of failing the operation together. This is a separate failure-handling gap discovered while qualifying #848/#870; it is not an observed CUDA OOM.
A bounded two-rank CPU/Gloo reproducer executes the original TrainerRank selector, search and estimator methods from PR #900 head
2b84bc90d08c0457687b46e264a67b49b4b88dae(runtime SHA2565df3a1138923faa6705862037cd067d4b8b27cbdb9ef99c1094116efeb237d23). Both ranks finish the checkpoint-prerequisite barrier. Rank 0 then raises its original ValueError in the pure local packing estimator; rank 1 proceeds to_all_ranks_trueand waits alone in WORLD MIN until the configured 15-second timeout. The origin stays alive until its peer records the timeout, so process teardown does not manufacture the failure.The post-cache-release path completes 14 matched collectives before the mismatch (peer timeout 15.007 seconds). The same exception with no recovery also reproduces it (15.005 seconds; no release, first-use permission unconsumed, cost/work ledgers zero). This establishes the weakness in the tested current source, not that #900 introduced it. Nine healthy asymmetric recovery cases retain 84 matched reductions. Original error identity/cause/context are checked, and all test children/groups are closed.
The fix must coordinate the pure local planning outcome after collective prerequisites and before the next planner collective. A catch around the entire search followed by an outcome reduction is too late. Preserve legal unavailable estimates, local original exception chains, current plan math and recovery accounting. Audit equivalent materialization/pricing boundaries; avoid reductions per local group because group counts can differ across ranks.
Qualification limits: real Gloo reductions and CPU tokens; CUDA counters/release, timing/model callbacks, and checkpoint loading are explicit facades. This is not NCCL, full-model, GPU-memory, or communicator-repair qualification.
Owner: Schulman; Peirce implements, Faraday independently reviews. Local correction is in progress, with no shared runtime change or merge yet. Runtime scope should remain
art.trainer_rank, with no public API orart.megatronchanges.Durable evidence:
/var/tmp/art870-cooperating-recovery-20260916-physical-_frpqj6e/manifest.json(8d0d8e43), source, rank traces, ordinary control and cleanup./var/tmp/art870-cooperating-recovery-review-20260916-forward/manifest.json(bbe3183b), independent verification./var/tmp/art870-planning-exchange-proposal-20260916-physical-veyf2nl8/manifest.json(68ab0a7c), proposed correction boundaries.