Repository navigation
Conversation
The TP/DP groups in deepspeed.utils.groups are process-wide and the first engine creates them. _init_tp_mesh_device returned early when they were set, so a second engine with another autotp_size silently ran on the first engine's groups. An engine without AutoTP and without its own mpu took the earlier AutoTP engine's data parallel group and failed later in initialize. Check the requested sizes against the existing groups and raise a clear error in both cases. Engines with the same layout keep sharing the groups. This is stage 1 of deepspeedai#9081. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
|
Thanks @jinyouzhi. First, a correction to my #9081 comment that your run makes: I read the engine without AutoTP as coming up on the TP2 engine's ranks, but on master it fails earlier, in I ran this on CPU (macOS, gloo, 4 ranks, torch 2.8.0):
One small suggestion: |
An engine without an mpu inherits groups.mpu. After an engine that passed its own mpu, that is the user's mpu, not the cached AutoTP groups, so the guard raised for a valid sequence (AutoTP TP2, then an explicit TP1 mpu, then an engine without an mpu). Run the check only when groups.mpu is the groups module, and test that sequence. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a721054f2b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@Achyuthan-S thanks for running it on macOS/gloo. I added Tested on CPU (Linux, gloo, 4 ranks): |
…mesh A mesh replaces neither groups.mpu nor the model parallel size read from it, so after an AutoTP engine such an engine failed with an AttributeError on its missing mpu. Also match the issue number in the size mismatch test. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
|
@sfc-gh-truwase ready to review. Could you have a look? Thansk |
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…engine Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Stage 1 of #9081: the guard (option 3), which @LiRunGuo and @Achyuthan-S agreed should come first.
Problem
The TP/DP groups in
deepspeed.utils.groupsare process-wide, and the first engine creates them. When a second engine is built in the same process:autotp_size:_init_tp_mesh_devicereturns early because_TENSOR_MODEL_PARALLEL_GROUPis already set. An engine that asks for TP4 after a TP2 engine silently becomes a TP2 model (reproducer in [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081).mpu(pointed out by @Achyuthan-S on [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081): this engine never reaches_init_tp_mesh_device. The earlier AutoTP engine leftgroups.mpu = groups, so the new engine takes the TP2 engine's data parallel group, which covers only part of the ranks. On master it then fails in_configure_distributed_modelwithAttributeError: 'NoneType' object has no attribute 'get_model_parallel_rank'.Change
groups._check_existing_tp_groups()compares the requested TP size with the existing groups. It raises aRuntimeErrorthat names both layouts and points to [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081. It is a no-op when no TP groups exist yet._init_tp_mesh_devicecalls it before its early return. This covers the engine path,tp_model_init(init_utils.py) andTpTrainingManager(tp_manager.py).DeepSpeedEngine.__init__also performs a preflight before publishinggroups.mesh_device, applying deterministic settings or configuring AutoEP/AutoTP. It checks the requested AutoTP size, or size 1 for an engine without AutoTP and without its ownmpuwhengroups.mpuis still thegroupsmodule set by an earlier AutoTP engine. This includes data/sequence mesh-backed engines: a mesh does not replacegroups.mpu.groups.mesh_devicefirst, so without this check the AutoTP engine would silently use the mesh engine's DP group.mpu, a later non-AutoTP engine inherits that one and the stale-AutoTP check does not run. Engines with the same layout keep sharing the groups, as before.The preflight avoids a rejected engine overwriting the existing engine's global mesh before the caller catches the layout error. This is not a general rollback mechanism for other initialization failures.
Not covered
These are left for stage 2 of #9081, as the RFC lays out:
mpureplacesgroups.mpu. Every later reader ofgroupsthen sees the newest engine's groups. [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 relies on this: it passes anmputo every engine. Making this an error would break that approach, and it is not a misconfiguration of the new engine.mpuafter one with a usermpu(the DP case @LiRunGuo reported on [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081). The second engine takes the first one's DP group. Megatron-style code may rely on passing thempuonce ("For subsequent use, just query groups"), so a check here would be a behavior change. I would rather fix it with per-engine groups than turn it into an error.initialize. The new engine then builds its own groups, so the check does not fire.Duplicate check
I ran
gh pr list --search "9081 in:body", and searched for_init_tp_mesh_deviceand AutoTP guard PRs. The only related open PR, #8801, changes thebwc.pyfallbacks and is unrelated. On #9081 nobody else has taken the guard. @Achyuthan-S plans to move thestage3.pyuniversal-checkpoint reader in a later stage.Tests
Latest preflight regression (
b66fba642)Hardware/software: Linux host with 2 x AMD EPYC 9575F CPUs; 4 CPU processes, gloo, Python 3.12.3, torch 2.14.0.dev20260804+xpu running with
DS_ACCELERATOR=cpu. No GPUs were used for this follow-up.DS_ACCELERATOR=cpu LOCAL_SIZE=4 PYTHONPATH=tests:. OMP_NUM_THREADS=2 \ python -m pytest --forked \ tests/unit/v1/autotp/test_autotp_multiple_models.py \ tests/unit/utils/test_groups.py -q --color=no # 17 passed in 65.18sScoped pre-commit and
git diff --checkalso pass.The mesh regression now covers a TP2 engine followed by either a non-AutoTP engine or a TP4 engine with a data/sequence mesh. Both cases fail before this follow-up because the rejected engine changes
groups.mesh_device. After the fix, the mesh remains unchanged and the original engine can complete an SGD update matching an independent dense reference. The test explicitly disables gradient clipping for that comparison. Same-layout sharing and explicit-mpuinheritance continue to pass.Latest compatibility matrix: ZeRO stage 0, FP32, AutoTP 1/2/4, no offload, CPU/gloo. This follow-up has not been rerun on GPU. The guard is initialization-only; no throughput, peak-memory or convergence improvement is claimed.
Mesh-before-AutoTP follow-up (
4b5cc6d8a)test_autotp_engine_after_mesh_raisesbuilds a data/sequence mesh engine and then a TP2 engine; it fails without the new check.test_autotp_multiple_models.pycases pass.pytest tests/unit/v1/autotp/test_autotp_multiple_models.py tests/unit/utils/test_groups.py-> 18 passed.Earlier XPU validation (before the latest preflight follow-up)
Hardware: 4 x Intel Arc Pro B70 (XPU, oneCCL), torch 2.14.0.dev20260804+xpu, on upstream master 5dc44e8.
skip_on_device()on XPU, plus H100-only tests.TestGroupCacheTeardown::test_nonmember_destroy_preserves_cached_groups: a worker was killed by signal 6 during teardown. That test does not reach the changed code; it passed when rerun alone, and in the full run on 1d13628.mpu. The latter initially raised; 1d13628 fixes that. The original two rejection cases fail on master.Earlier compatibility matrix: ZeRO stage 0, bf16 off, AutoTP 1/2/4, no offload, on XPU. Not run on CUDA.
This PR was prepared with AI assistance; the tests above were run on the hardware listed.