Skip to content

[test] numerical alignment, benchmark: release the process group of every spawned worker - #6566

Merged
yuxuan-lou merged 2 commits into
mainfrom
fix/numerics-worker-teardown
Nov 18, 2026
Merged

yuxuan-lou merged 2 commits into
mainfrom
fix/numerics-worker-teardown

Conversation

@JazyJiang

@JazyJiang JazyJiang commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

1. Related issues

2. Background

A worker started by colossalai.testing.spawn owns a process group. On the torch 2.5.1 CPU check (Gloo) there are two ways to fail on the way out:

  • Return with the group alive: Gloo's threads are still running at interpreter exit and the process can abort with SIGABRT.
  • Call destroy_process_group() directly: before torch 2.8 the generic ProcessGroup holder destroys its Gloo backend with the GIL held, while a Gloo thread may still need the GIL to release the Python tensors of the last collective. Each waits for the other and the worker hangs.

destroy_worker_process_group() from #6507 avoids both: it keeps the Gloo backend's own Python holder, which releases the GIL in its destructor, until the generic group is gone. The data-parallel and resume scenarios use it, after a barrier on success. Six other workers do not:

Worker Before this PR
acceptance._worker returns with the group alive
14 workers in test_numerics_independent_impl.py return with the group alive
_without_token_loss in test_numerics_data_parallel.py returns with the group alive
_amp_checkpoint_worker in test_gemma4_checkpoint_regressions.py returns with the group alive
benchmark measure._worker destroys directly after all_gather_object; leaves the group alive when training raises

3. Changes

  • tests/kit/numerics/scenarios/_distributed.py: worker_process_group(rank, world_size, port) holds feat(shardformer): add HF5 adapters for LLaDA and Ouro #6507's pattern in one place: launch ColossalAI; on success pass a barrier; on success or error release the group with destroy_worker_process_group(); an error propagates unchanged. @worker applies it to a spawn target and keeps the target picklable.
  • scenarios/data_parallel.py, scenarios/resume.py: use it. Behaviour is unchanged, and feat(shardformer): add HF5 adapters for LLaDA and Ouro #6507's test_worker_releases_process_group passes as before.
  • All workers in the table above use it.
  • tests/kit/numerics/README.md: the worker example uses @worker instead of a bare colossalai.launch.
  • tests/kit/benchmark/README.md (second commit, from [test] Benchmark README: batch functions take their size and random numbers from their arguments #6568): under "Adding a model", a model's batch_fn builds the batch at the size it is given, because the maximum search changes micro_batch and seq_len between calls, and takes its random numbers from generator rather than the global random state, so that each rank trains different data and a MoE router does not route identically on every rank. The existing batch functions already follow this.

4. User and compatibility impact

  • colossalai package, APIs, checkpoints, training behavior, performance: none. Only tests/ changes.
  • Numerical results and verdicts, benchmark measurements: unchanged. The barrier runs after the trace or record is written and outside every timed step.

5. Test environment

  • CPU, the CPU unit tests job's environment rebuilt as fork_pr_checks.yml installs it: Ubuntu 22.04.5 LTS, Python 3.10.12, torch 2.5.1+cpu, requirements/requirements.txt, OMP_NUM_THREADS=2, FAST_TEST=1, pinned to 4 cores.
  • GPU: CI image torch2.13-cu130-tf5.17.0-peft0.21.2-fla0.6.0-20261111-163212, 2× NVIDIA H20, torch 2.13.0+cu130.
  • ColossalAI: this branch on main d3916b5. The results below are for 2deeb69; the second commit, 03f25d8, changes only tests/kit/benchmark/README.md.

6. Test commands and results

CUDA_VISIBLE_DEVICES= FAST_TEST=1 python -m pytest -q tests/test_numerics/test_numerics_acceptance.py tests/test_numerics/test_numerics_core.py tests/test_numerics/test_numerics_data_parallel.py tests/test_numerics/test_numerics_independent_impl.py tests/test_numerics/test_numerics_resume.py tests/test_numerics/test_gemma4_checkpoint_regressions.py tests/test_benchmark
  • torch 2.5.1+cpu: 110 passed, 0 failed, 59 skipped, 504 s. Every skip needs a device.
CUDA_VISIBLE_DEVICES=<two GPUs> python -m pytest -q -rs tests/test_numerics tests/test_benchmark
  • torch 2.13 (CI image): 168 passed, 0 failed, 1 skipped (this torch can recompute on the meta device), 1164 s.

Teardown under CPU pressure. The hang does not show on an idle 208-core host, so it was reproduced on 2 cores shared with 3 busy loops, torch 2.5.1+cpu. A minimal reproducer spawns 2 Gloo ranks, runs a few all_reduce and one all_gather_object on Python-owned tensors, and leaves in one of the ways below; a run is a hang after 120 s, when every process dumps its stacks through faulthandler.

Exit Runs Hang SIGABRT
destroy_process_group() 120 6 0
barrier(), then destroy_process_group() 30 5 0
barrier(), then destroy_worker_process_group() (this PR, #6507) 200 0 0

The two destroy modes ran in the same windows as the last row, so the load did reproduce the hang there. In every hang the stack of the main thread ends in destroy_process_group(), next to a thread with no Python frame. Under the same load, test_micro_batch_without_tokens with a barrier and a direct destroy_process_group() hung 2 times in 30 runs.

No xfail, --ignore or manual reruns.

7. Known issues

8. Risks and rollback

  • Risk: a worker whose peers diverge before the end now waits at the final barrier. A diverging worker already fails inside its collectives before reaching it, and a failing worker skips the barrier.
  • Rollback: revert this PR; nothing outside tests/ depends on it.

9. Documentation and release impact

  • tests/kit/numerics/README.md: worker example updated.
  • tests/kit/benchmark/README.md: how a model's batch function uses its size and random-number arguments.
  • Installation, API and Docker documentation, changelog, version, PyPI, Docker images: unchanged.

@JazyJiang
JazyJiang requested a review from a team as a code owner November 18, 2026 07:02
@JazyJiang JazyJiang changed the title [test] numerical alignment: destroy worker process groups to fix the intermittent SIGABRT [test] numerical alignment, benchmark: destroy worker process groups to fix an intermittent SIGABRT and hang Nov 18, 2026
…mark worker

The data-parallel and resume scenarios already synchronize on success and
release their group through destroy_worker_process_group, also on errors.
The other workers still return with the group alive, or, in the benchmark,
destroy it directly right after all_gather_object and not at all when
training raises. On the torch 2.5.1 CPU check that can end in SIGABRT at
interpreter exit (pytorch/pytorch#179238) or in a hang inside
destroy_process_group, which holds the GIL before torch 2.8
(pytorch/pytorch#154976).

Move that pattern into worker_process_group (and a @worker decorator) next
to destroy_worker_process_group, keep the two scenarios' behaviour
unchanged, and use it for the acceptance worker, the 14 independent-impl
test workers, the token-loss and Gemma 4 checkpoint test workers, and the
benchmark worker. The README example uses it too.
@JazyJiang
JazyJiang force-pushed the fix/numerics-worker-teardown branch from 50dfc31 to 2deeb69 Compare November 18, 2026 10:20
@JazyJiang JazyJiang changed the title [test] numerical alignment, benchmark: destroy worker process groups to fix an intermittent SIGABRT and hang [test] numerical alignment, benchmark: release the process group of every spawned worker Nov 18, 2026
@yuxuan-lou
yuxuan-lou merged commit d5579c5 into main Nov 18, 2026
8 checks passed
@yuxuan-lou
yuxuan-lou deleted the fix/numerics-worker-teardown branch November 18, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants