Skip to content

Raise when a later engine asks for another tensor parallel layout (#9081 stage 1) - #9088

Open
jinyouzhi wants to merge 7 commits into
deepspeedai:masterfrom
jinyouzhi:per-engine-groups-guard
Open

jinyouzhi wants to merge 7 commits into
deepspeedai:masterfrom
jinyouzhi:per-engine-groups-guard

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Stage 1 of #9081: the guard (option 3), which @LiRunGuo and @Achyuthan-S agreed should come first.

Problem

The TP/DP groups in deepspeed.utils.groups are process-wide, and the first engine creates them. When a second engine is built in the same process:

Change

  • groups._check_existing_tp_groups() compares the requested TP size with the existing groups. It raises a RuntimeError that names both layouts and points to [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081. It is a no-op when no TP groups exist yet.
  • _init_tp_mesh_device calls it before its early return. This covers the engine path, tp_model_init (init_utils.py) and TpTrainingManager (tp_manager.py).
  • DeepSpeedEngine.__init__ also performs a preflight before publishing groups.mesh_device, applying deterministic settings or configuring AutoEP/AutoTP. It checks the requested AutoTP size, or size 1 for an engine without AutoTP and without its own mpu when groups.mpu is still the groups module set by an earlier AutoTP engine. This includes data/sequence mesh-backed engines: a mesh does not replace groups.mpu.
  • An AutoTP engine created after a data/sequence mesh engine is also rejected in the preflight. The groups getters read groups.mesh_device first, so without this check the AutoTP engine would silently use the mesh engine's DP group.
  • If an engine in between passed its own mpu, a later non-AutoTP engine inherits that one and the stale-AutoTP check does not run. Engines with the same layout keep sharing the groups, as before.
  • The unused DP-size comparison was removed from the guard; production callers only request a TP size.
  • The AutoTP tutorial lists this as limitation 5.

The preflight avoids a rejected engine overwriting the existing engine's global mesh before the caller catches the layout error. This is not a general rollback mechanism for other initialization failures.

Not covered

These are left for stage 2 of #9081, as the RFC lays out:

Duplicate check

I ran gh pr list --search "9081 in:body", and searched for _init_tp_mesh_device and AutoTP guard PRs. The only related open PR, #8801, changes the bwc.py fallbacks and is unrelated. On #9081 nobody else has taken the guard. @Achyuthan-S plans to move the stage3.py universal-checkpoint reader in a later stage.

Tests

Latest preflight regression (b66fba642)

Hardware/software: Linux host with 2 x AMD EPYC 9575F CPUs; 4 CPU processes, gloo, Python 3.12.3, torch 2.14.0.dev20260804+xpu running with DS_ACCELERATOR=cpu. No GPUs were used for this follow-up.

DS_ACCELERATOR=cpu LOCAL_SIZE=4 PYTHONPATH=tests:. OMP_NUM_THREADS=2 \
python -m pytest --forked \
  tests/unit/v1/autotp/test_autotp_multiple_models.py \
  tests/unit/utils/test_groups.py -q --color=no
# 17 passed in 65.18s

Scoped pre-commit and git diff --check also pass.

The mesh regression now covers a TP2 engine followed by either a non-AutoTP engine or a TP4 engine with a data/sequence mesh. Both cases fail before this follow-up because the rejected engine changes groups.mesh_device. After the fix, the mesh remains unchanged and the original engine can complete an SGD update matching an independent dense reference. The test explicitly disables gradient clipping for that comparison. Same-layout sharing and explicit-mpu inheritance continue to pass.

Latest compatibility matrix: ZeRO stage 0, FP32, AutoTP 1/2/4, no offload, CPU/gloo. This follow-up has not been rerun on GPU. The guard is initialization-only; no throughput, peak-memory or convergence improvement is claimed.

Mesh-before-AutoTP follow-up (4b5cc6d8a)

test_autotp_engine_after_mesh_raises builds a data/sequence mesh engine and then a TP2 engine; it fails without the new check.

  • 4 x Intel Arc Pro B70 (XPU, oneCCL): test_autotp_multiple_models.py cases pass.
  • 4 x NVIDIA L20 (CUDA, NCCL): pytest tests/unit/v1/autotp/test_autotp_multiple_models.py tests/unit/utils/test_groups.py -> 18 passed.

Earlier XPU validation (before the latest preflight follow-up)

Hardware: 4 x Intel Arc Pro B70 (XPU, oneCCL), torch 2.14.0.dev20260804+xpu, on upstream master 5dc44e8.

CUDA_VISIBLE_DEVICES=0,1,2,3 pytest -q tests/unit/v1/autotp/test_autotp_multiple_models.py tests/unit/utils/test_groups.py
15 passed

pytest -q tests/unit/v1/autotp tests/unit/utils/test_groups.py tests/unit/v1/moe/test_autoep_autotp_runtime.py
38 passed, 84 skipped (1d13628)
  • The skips are the AutoTP tests that call skip_on_device() on XPU, plus H100-only tests.
  • On the first commit, one run failed in TestGroupCacheTeardown::test_nonmember_destroy_preserves_cached_groups: a worker was killed by signal 6 during teardown. That test does not reach the changed code; it passed when rerun alone, and in the full run on 1d13628.
  • The original engine cases cover TP2 sharing, TP2-to-TP4 rejection, TP2-to-non-AutoTP rejection, and inheritance of an intervening explicit TP1 mpu. The latter initially raised; 1d13628 fixes that. The original two rejection cases fail on master.

Earlier compatibility matrix: ZeRO stage 0, bf16 off, AutoTP 1/2/4, no offload, on XPU. Not run on CUDA.

This PR was prepared with AI assistance; the tests above were run on the hardware listed.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 15, 2026
The TP/DP groups in deepspeed.utils.groups are process-wide and the first
engine creates them. _init_tp_mesh_device returned early when they were set,
so a second engine with another autotp_size silently ran on the first
engine's groups. An engine without AutoTP and without its own mpu took the
earlier AutoTP engine's data parallel group and failed later in initialize.

Check the requested sizes against the existing groups and raise a clear
error in both cases. Engines with the same layout keep sharing the groups.
This is stage 1 of deepspeedai#9081.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks @jinyouzhi. First, a correction to my #9081 comment that your run makes: I read the engine without AutoTP as coming up on the TP2 engine's ranks, but on master it fails earlier, in _configure_distributed_model, as you found. Either way, initialize is the right place to stop it.

I ran this on CPU (macOS, gloo, 4 ranks, torch 2.8.0):

  • test_autotp_multiple_models.py and test_groups.py: 14 passed.
  • On 5dc44e8 without the change, the TP1 case fails with the same AttributeError, and a TP4 engine built after a TP2 engine comes up on the TP2 groups (mpu TP size 2, dp_world_size 2), as the description says.
  • The tests that build AutoTP engines on the affine paths, test_zero_affine.py (including the TP2 × DP2 storage maps) and test_autotp_uc_checkpoint.py: 63 passed. The same 9 fail with and without the change on this machine, for setup reasons: fused_adam needs ninja, and a tree that isn't installed reports version 0.0.0, which the ZeRO-1 checkpoint loader refuses.

One small suggestion: pytest.raises(RuntimeError, match="9081") in test_different_tp_size_raises, so that a RuntimeError from somewhere else in initialize can't pass it.

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 16, 2026
An engine without an mpu inherits groups.mpu. After an engine that passed
its own mpu, that is the user's mpu, not the cached AutoTP groups, so the
guard raised for a valid sequence (AutoTP TP2, then an explicit TP1 mpu,
then an engine without an mpu). Run the check only when groups.mpu is the
groups module, and test that sequence.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@jinyouzhi
jinyouzhi marked this pull request as ready for review November 17, 2026 15:55
@gpu-test-labeler gpu-test-labeler Bot removed the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T16:01:16.147776Z a721054 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a721054f2b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread deepspeed/runtime/engine.py Resolved
@gpu-test-labeler gpu-test-labeler Bot removed the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026
@jinyouzhi

jinyouzhi commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S thanks for running it on macOS/gloo. I added match="9081" to test_different_tp_size_raises in 59ddc50, which also addresses Codex's point about engines with a data/sequence parallel mesh.

Tested on CPU (Linux, gloo, 4 ranks): test_autotp_multiple_models.py + test_groups.py, 16 passed.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026
…mesh

A mesh replaces neither groups.mpu nor the model parallel size read from it, so after an AutoTP engine such an engine failed with an AttributeError on its missing mpu. Also match the issue number in the size mismatch test.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 18, 2026
@jinyouzhi

Copy link
Copy Markdown
Contributor Author

@sfc-gh-truwase ready to review. Could you have a look? Thansk

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 18, 2026
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 18, 2026
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…engine

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants