Repository navigation
[test] numerical alignment, benchmark: release the process group of every spawned worker - #6566
Merged
Merged
Conversation
…mark worker The data-parallel and resume scenarios already synchronize on success and release their group through destroy_worker_process_group, also on errors. The other workers still return with the group alive, or, in the benchmark, destroy it directly right after all_gather_object and not at all when training raises. On the torch 2.5.1 CPU check that can end in SIGABRT at interpreter exit (pytorch/pytorch#179238) or in a hang inside destroy_process_group, which holds the GIL before torch 2.8 (pytorch/pytorch#154976). Move that pattern into worker_process_group (and a @worker decorator) next to destroy_worker_process_group, keep the two scenarios' behaviour unchanged, and use it for the acceptance worker, the 14 independent-impl test workers, the token-loss and Gemma 4 checkpoint test workers, and the benchmark worker. The README example uses it too.
JazyJiang
force-pushed
the
fix/numerics-worker-teardown
branch
from
November 18, 2026 10:20
50dfc31 to
2deeb69
Compare
…umbers from their arguments
yuxuan-lou
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. Related issues
CPU unit testsfailures on several PRs, all at worker exit, not in the checks themselves:process 1 terminated with signal SIGABRTintests/test_numerics(test_micro_batch_without_tokens,test_unsynchronized_parameter_detected);tests/test_benchmark/test_benchmark_core.py::test_cpu_run_end_to_endand intest_micro_batch_without_tokens, both seen on earlier pushes of this PR.scenarios/_distributed.py. This PR applies the same teardown to the remaining workers.ProcessGroupis destroyed with the GIL held, fixed in torch 2.8).2. Background
A worker started by
colossalai.testing.spawnowns a process group. On the torch 2.5.1 CPU check (Gloo) there are two ways to fail on the way out:destroy_process_group()directly: before torch 2.8 the genericProcessGroupholder destroys its Gloo backend with the GIL held, while a Gloo thread may still need the GIL to release the Python tensors of the last collective. Each waits for the other and the worker hangs.destroy_worker_process_group()from #6507 avoids both: it keeps the Gloo backend's own Python holder, which releases the GIL in its destructor, until the generic group is gone. The data-parallel and resume scenarios use it, after a barrier on success. Six other workers do not:acceptance._workertest_numerics_independent_impl.py_without_token_lossintest_numerics_data_parallel.py_amp_checkpoint_workerintest_gemma4_checkpoint_regressions.pymeasure._workerall_gather_object; leaves the group alive when training raises3. Changes
tests/kit/numerics/scenarios/_distributed.py:worker_process_group(rank, world_size, port)holds feat(shardformer): add HF5 adapters for LLaDA and Ouro #6507's pattern in one place: launch ColossalAI; on success pass a barrier; on success or error release the group withdestroy_worker_process_group(); an error propagates unchanged.@workerapplies it to aspawntarget and keeps the target picklable.scenarios/data_parallel.py,scenarios/resume.py: use it. Behaviour is unchanged, and feat(shardformer): add HF5 adapters for LLaDA and Ouro #6507'stest_worker_releases_process_grouppasses as before.tests/kit/numerics/README.md: the worker example uses@workerinstead of a barecolossalai.launch.tests/kit/benchmark/README.md(second commit, from [test] Benchmark README: batch functions take their size and random numbers from their arguments #6568): under "Adding a model", a model'sbatch_fnbuilds the batch at the size it is given, because the maximum search changesmicro_batchandseq_lenbetween calls, and takes its random numbers fromgeneratorrather than the global random state, so that each rank trains different data and a MoE router does not route identically on every rank. The existing batch functions already follow this.4. User and compatibility impact
colossalaipackage, APIs, checkpoints, training behavior, performance: none. Onlytests/changes.5. Test environment
CPU unit testsjob's environment rebuilt asfork_pr_checks.ymlinstalls it: Ubuntu 22.04.5 LTS, Python 3.10.12, torch 2.5.1+cpu,requirements/requirements.txt,OMP_NUM_THREADS=2,FAST_TEST=1, pinned to 4 cores.torch2.13-cu130-tf5.17.0-peft0.21.2-fla0.6.0-20261111-163212, 2× NVIDIA H20, torch 2.13.0+cu130.tests/kit/benchmark/README.md.6. Test commands and results
this torch can recompute on the meta device), 1164 s.Teardown under CPU pressure. The hang does not show on an idle 208-core host, so it was reproduced on 2 cores shared with 3 busy loops, torch 2.5.1+cpu. A minimal reproducer spawns 2 Gloo ranks, runs a few
all_reduceand oneall_gather_objecton Python-owned tensors, and leaves in one of the ways below; a run is a hang after 120 s, when every process dumps its stacks throughfaulthandler.destroy_process_group()barrier(), thendestroy_process_group()barrier(), thendestroy_worker_process_group()(this PR, #6507)The two
destroymodes ran in the same windows as the last row, so the load did reproduce the hang there. In every hang the stack of the main thread ends indestroy_process_group(), next to a thread with no Python frame. Under the same load,test_micro_batch_without_tokenswith a barrier and a directdestroy_process_group()hung 2 times in 30 runs.No
xfail,--ignoreor manual reruns.7. Known issues
destroy_worker_process_group(). It relies on how torch 2.5 bindsProcessGroupGloo; torch 2.8 and later remove the problem. The GPU jobs already use torch 2.13.8. Risks and rollback
tests/depends on it.9. Documentation and release impact
tests/kit/numerics/README.md: worker example updated.tests/kit/benchmark/README.md: how a model's batch function uses its size and random-number arguments.