Skip to content

Give an engine with another tensor parallel layout groups of its own (#9081 stage 2) - #9120

Draft
jinyouzhi wants to merge 20 commits into
deepspeedai:masterfrom
jinyouzhi:engine-own-groups
Draft

jinyouzhi wants to merge 20 commits into
deepspeedai:masterfrom
jinyouzhi:engine-own-groups

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Stage 2 of #9081. Depends on #9088 (stage 1). The commits up to #9088's head are #9088's; the rest are new.

With #9088, a second deepspeed.initialize whose tensor parallel layout differs from an earlier engine's raises. With this PR, every engine gets its own groups object, and an engine with another layout gets TP/DP groups of its own.

Design

  • Every AutoTP engine's engine.mpu is its own groups.TensorParallelGroups: an object with the mpu interface (get_tensor_model_parallel_group/world_size/rank/src_rank, get_model_parallel_*, get_data_parallel_*, get_data_parallel_group_ranks, and a pipeline size of 1).
    • Its batch sizes are computed as for an engine without an mpu, so they count sequence_parallel_size too.
    • When the engine's layout matches the process-wide groups, the object holds those same groups (process_wide=True), so no new communicator is created. The first engine creates the process-wide groups, groups.get_* keep describing it, and groups.mpu stays groups for the code that still reads it.
    • Otherwise the object holds new groups (process_wide=False) and is not published to groups.mpu.
    • When groups.mpu is an mpu passed to an earlier engine, an AutoTP engine without one does not replace it: it holds groups of its own (process_wide=False), reusing the process-wide TP groups if they have its layout. Before, it published its groups over that mpu, so the earlier engine's code that still queries groups (gradient clipping among it) ran on the AutoTP engine's DP group.
  • An engine without an mpu no longer inherits an earlier engine's groups.mpu (reported on [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081: after an engine with an explicit mpu over half of the ranks, an engine over all ranks took that mpu's DP group). If an earlier engine has published an mpu, the new engine gets TP1 groups of its own (DP over every rank). The first engine without an mpu is unchanged (engine.mpu is None).
    • An engine whose config sets sequence_parallel_size > 1 without a mesh keeps inheriting groups.mpu when that provides sequence parallelism (an earlier engine's Ulysses mpu), the only source of its sequence parallel groups; otherwise it gets TP1 groups too. With TP1 groups, ZeRO-3 failed the batch size check and the other stages ran without sequence parallelism.
    • These TP1 groups are plain data parallelism over every rank: BF16 Muon accepts them as dense DP, and the hybrid engine builds its inference TP groups (inference_tp_size) as it does without an mpu.
  • An engine can share another engine's groups by passing them: deepspeed.initialize(..., mpu=other_engine.mpu). The tensor parallel size must match, and it cannot be combined with mesh_param. Before this PR an AutoTP engine ignored the mpu it was given.
    • The batch sizes are computed as for an AutoTP engine without an mpu (over every rank), so passing the groups does not change train_batch_size or the gradient accumulation steps.
    • An AutoEP engine can be passed the process-wide groups this way; groups of another layout still raise for AutoEP.
  • groups._get_or_create_tp_groups(tp) makes the choice: it initializes the process-wide groups if there are none, and returns a TensorParallelGroups over them if the layout matches or over new groups otherwise.
  • The engine reads its groups from self.mpu in these places:
    • AutoTP sharding (tp_group, and mp_size in training mode);
    • the DP and sequence-DP groups and sizes;
    • the dataloader check;
    • get_*_rank;
    • non-expert gradient reduction;
    • the checkpoint DP rank and broadcast source;
    • the checkpoint writer's DP layout (checkpoint.writer), which read groups.mpu. The writer still gets the expert groups from groups.
  • ZeRO 1/2/3 take the groups through mpu as before. Stage 3 also uses them when loading a universal checkpoint.
  • In groups.py, group creation is split out from writing the globals (_new_tp_groups, _build_tp_groups). The existing _init_tp_* paths behave as before.
  • The choice is made at the start of DeepSpeedEngine.__init__, where Raise when a later engine asks for another tensor parallel layout (#9081 stage 1) #9088 checks the layout. An engine that will run on groups other than the process-wide ones is checked there for what is not supported yet, before it changes any process-wide state, the model, or creates groups.
  • clip_grad_norm_ (ZeRO-0 FP32 gradient clipping) averages the norm over the engine's own DP group.

Behavior changes

  • An engine without an mpu, created after an engine that published one (an AutoTP engine or an explicit mpu), runs on TP1 groups over every rank instead of the earlier engine's groups. Code that passes an mpu once and relies on later engines picking it up must pass it to each engine. A data/sequence parallel mesh engine, and an engine whose config sets sequence_parallel_size > 1, keep the old behavior when groups.mpu provides sequence parallelism.
  • Such an engine cannot use MoE, AutoEP, or the DeepCompile autotp pass yet (see below).
  • An AutoTP engine created after an engine with an explicit mpu also runs on groups of its own, so it cannot use mesh_param, MoE, AutoEP, or the DeepCompile autotp pass yet either.
  • A data/sequence parallel mesh engine created after an AutoTP engine or an engine with an explicit mpu raises, also when the tensor parallel size matches. Before, it replaced the data parallel group that groups returns for the earlier engine.
  • A model already sharded by AutoTP, passed to an engine with another tensor parallel size, raises. Before, its layers kept the groups they were sharded with, and at TP1 the engine's initial broadcast overwrote the other ranks' shards, which the first engine still uses. The same tensor parallel size is still accepted.
  • A model already sharded by AutoTP is also found inside a wrapper module, and so are AutoTP layers passed without the model they were sharded in.
  • tp_model_init with tp_size > 1, or with tp_size 1 after an earlier engine, no longer creates the process-wide groups itself; the engine creates them, or groups of its own when an earlier engine has another tensor parallel size. A tp_group passed to tp_model_init must have the ranks of the engine's TP group, which the layers now shard over. All ranks agree on the check, so the ranks whose group matches do not hang.
  • autotp_storage_affine_maps takes its default TP group from the model's AutoTP layers, since an engine with groups of its own does not publish them.
  • ZeRO-3 uses the process-wide hpZ group only when hpZ is enabled for the engine; before, a later engine with hpZ disabled used the group an earlier engine built for its layout.
  • The hybrid engine is now given the mesh from mesh_param (or data_parallel_size/sequence_parallel_size), which it ignored, so a mesh after an AutoTP engine raises for it too.
  • Creating those TP1 groups is collective, so every rank must create the same engines in the same order, as for any engine.
  • With a single AutoTP engine, checkpoint.writer failed on master with AttributeError: module 'deepspeed.utils.groups' has no attribute 'get_pipeline_model_parallel_world_size', because groups.mpu was the groups module. The writer now reads the engine's TensorParallelGroups, so this works.

Not covered yet (raises with a link to #9081)

An engine that runs on groups other than the process-wide ones (its own, or shared from such an engine) cannot yet use:

  • mesh_param;
  • AutoEP;
  • the DeepCompile autotp pass;
  • MoE layers, including AutoEP layers already in the model.

These still read the process-wide groups. A mesh engine created after an AutoTP engine or an explicit mpu, and an AutoTP engine created after a mesh engine, raise. Following the roadmap, MoE/AutoEP/compile are stage 4, and the remaining checkpoint helpers (the zero_affine default replica group of parameters outside a ZeRO optimizer) are stage 3.

Left for later stages of #9081:

#2989 and #3592 were looked at as possibly related. Both fix nested zero.Init and dynamic classes, and neither involves process groups.

Tests

tests/unit/v1/autotp/test_autotp_multiple_models.py:

  • A same-layout engine has its own object over the process-wide groups. A TP1 or TP4 engine after a TP2 engine gets its own groups with the right ranks, and the TP2 engine and groups keep TP2.
  • An engine without an mpu after an engine with an explicit mpu over half of the ranks runs with DP over every rank, while groups keeps describing the explicit mpu; also after an AutoTP engine and an explicit mpu. The first engine without an mpu is unchanged. After a Ulysses SP2 engine, an engine without an mpu with sequence_parallel_size: 2 keeps SP2 under ZeRO-1 and ZeRO-3; after a TP2 engine or an explicit mpu over half of the ranks, such an engine gets TP1 groups with DP over every rank.
  • A TP2 engine after an explicit mpu leaves groups.mpu to it, holds groups of its own, and trains a step; after a TP2 engine and an explicit mpu, it holds the first engine's groups without publishing them.
  • After a TP2 engine, a BF16 Muon engine (ZeRO-1, FP32 accumulation) initializes, and a hybrid engine with inference_tp_size=2 gets inference TP groups of size 2.
  • [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074's order, DP4 -> DP2 -> DP4: the engines over every rank get DP4, and a further engine passed mpu=world.mpu shares its groups, trains a step, and raises for MoE. A TP4 engine passed another TP4 engine's mpu shares its groups; a TP2 engine passed it raises.
  • mesh after AutoTP (TP1/2/4), mesh after an explicit mpu (TP1/2), AutoTP after mesh, and MoE with groups of its own raise. The MoE case raises before the model is moved or any group is created.
  • A TP1 or TP4 engine with checkpoint.writer after a TP2 engine gets a writer config for its own DP size and rank (the TP1 case used to hang listing the ranks of the world group). A TP2 engine with MoE layers and checkpoint.writer builds its writer.
  • A TP1 or TP4 engine given a model sharded at TP2 raises, and the TP2 engine keeps its groups; a TP2 engine given it is accepted. The same holds for the model inside nn.Sequential, and for a sharded layer of it wrapped alone. After a TP2 engine, a model through tp_model_init(tp_size=4) gets TP4 groups of its own, and one through tp_model_init(tp_size=1) gets DP over every rank; a tp_model_init group with other ranks than the engine's TP group raises, also when only some ranks' groups differ. After an explicit mpu, the storage maps of a TP2 engine's model are built over its own TP group.
  • After a ZeRO-3 engine with hpZ, a ZeRO-3 TP2 engine without it has no secondary partitions. AutoEP layers built by an earlier engine raise in an engine with groups of its own. A hybrid engine with mesh_param after a TP2 engine raises.
  • An engine passed another engine's groups keeps the same train_batch_size as that engine, with train_batch_size and gradient_accumulation_steps both set.
  • An AutoEP+TP2 engine passed a TP2 engine's mpu shares its groups and trains a step.
  • Training parity: TP(2,4) and TP(4,2) engines in one process, each matching an unsharded model over 4 steps with gradient clipping, under ZeRO 0/1/2/3. Each DP rank gets its own part of the batch, and the reference trains on the whole batch, so a wrong DP group changes the result.

Each new test fails without the change it covers (checked by reverting it).

Results (latest commit):

  • 4×B70: all tests of TestAutoTPEnginesWithDifferentLayouts (41), the parity test (8), and the tp_model_init tests of test_autotp_training.py (5, skipped on XPU by default, run without the skip) pass under torchrun. Under pytest, some tests intermittently hang in the harness's XCCL init barrier before any test code runs; run alone, each test passes.
  • 4×L20: test_autotp_multiple_models.py + test_groups.py 30 passed on the previous commit (before sharing via mpu). On the wider AutoTP set, the failures are the FusedAdam JIT build on that machine, and TestGPTBigCodeMultiQueryTraining hangs identically on this PR's earlier head.

The TP/DP groups in deepspeed.utils.groups are process-wide and the first
engine creates them. _init_tp_mesh_device returned early when they were set,
so a second engine with another autotp_size silently ran on the first
engine's groups. An engine without AutoTP and without its own mpu took the
earlier AutoTP engine's data parallel group and failed later in initialize.

Check the requested sizes against the existing groups and raise a clear
error in both cases. Engines with the same layout keep sharing the groups.
This is stage 1 of deepspeedai#9081.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
An engine without an mpu inherits groups.mpu. After an engine that passed
its own mpu, that is the user's mpu, not the cached AutoTP groups, so the
guard raised for a valid sequence (AutoTP TP2, then an explicit TP1 mpu,
then an engine without an mpu). Run the check only when groups.mpu is the
groups module, and test that sequence.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…mesh

A mesh replaces neither groups.mpu nor the model parallel size read from it, so after an AutoTP engine such an engine failed with an AttributeError on its missing mpu. Also match the issue number in the size mismatch test.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 18, 2026
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 18, 2026
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…eepspeedai#9081 stage 2)

The first engine, and any later engine with the same layout, keep using the
process-wide groups. An engine whose TP/DP layout differs now gets a
TensorParallelGroups of its own instead of raising, and AutoTP, the engine,
ZeRO and checkpointing read the groups from it. groups.mpu keeps describing
the first engine.

Not covered yet, and still raising: mesh_param, AutoEP, the DeepCompile
autotp pass and MoE layers on an engine with groups of its own.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…engine

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

# Conflicts:
#	deepspeed/runtime/engine.py
#	tests/unit/v1/autotp/test_autotp_multiple_models.py
…rank its own batch in the parity test

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026
An engine created without an mpu no longer inherits an earlier engine's
mpu (deepspeedai#9081). Engines whose layout matches the process-wide groups still
share the same underlying process groups.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026
An AutoTP engine used to ignore the mpu it was given. Passing another
engine's TensorParallelGroups now makes the engine run on those groups,
which must have its tensor parallel size. An engine sharing groups other
than the process-wide ones gets the same checks as one with groups of its
own.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…gine's groups

Passing another AutoTP engine's groups as mpu no longer changes the batch
size arithmetic, AutoEP accepts the process-wide groups passed that way, and
the checkpoint writer still finds the expert groups of an engine that holds
its own groups.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
… as dense data parallelism

An AutoTP engine created after an engine with an explicit mpu replaced that
mpu in groups.mpu, so the earlier engine's code that still queries groups ran
on the AutoTP engine's data parallel group. Such an engine now holds groups of
its own instead.

An engine's own TP1 groups are data parallelism over every rank: BF16 Muon
accepts them, and the hybrid engine builds its inference tensor parallel
groups as it does without an mpu.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…d the ranks of TP1 data parallel groups

TensorParallelGroups.get_data_parallel_group_ranks used get_all_ranks_from_group, which never ends for the world group,
the data parallel group of an engine's TP1 groups, so a TP1 engine with a checkpoint writer hung.

A data/sequence parallel mesh engine built after an engine that set groups.mpu replaced the data parallel group the
module getters return for that engine; an AutoTP engine with a matching tensor parallel size bypassed the deepspeedai#9081 check.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…ngine create the groups for tp_model_init

A model already sharded by AutoTP, passed to an engine with another tensor parallel size, kept its layers on the
groups it was sharded with; at TP1 the engine's initial broadcast overwrote the other ranks' shards, which the
first engine still uses.

tp_model_init created process-wide groups of its size before the engine could create groups of its own, so it
failed after an engine with another tensor parallel size.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…id meshes against the engine's groups

- A model wrapping one already sharded at another tensor parallel size raises, like the sharded model itself.
- A tp_group passed to tp_model_init must have the ranks of the engine's TP group; the layers shard over the
  engine's group, which its data parallel broadcasts and reductions assume.
- ZeRO-3 uses the hpZ group only when hpZ is enabled, instead of one built by an earlier engine for its layout.
- Prebuilt AutoEP layers are rejected with groups of the engine's own, like MoE layers.
- The hybrid engine is given the mesh, so a mesh after an AutoTP engine raises instead of being ignored.
- tp_model_init with tp_size 1 does not create process-wide TP1 groups after an earlier engine.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…ence parallelism

An engine without an mpu after a Ulysses engine got TP1 groups, which ignored its
sequence_parallel_size: ZeRO-3 failed the batch size check and the other stages
ran without sequence parallelism.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
… require a sequence parallel mpu to inherit

- Inherit groups.mpu for sequence_parallel_size > 1 only when it provides
  sequence parallelism; after an AutoTP engine the engine crashed, and after
  an explicit mpu it took its data parallel group.
- Raise for AutoTP layers sharded at another tensor parallel size without
  the model they were sharded in, whose shards the broadcast overwrote.
- Reduce the tp_model_init group check over all ranks, so the ranks whose
  group matches do not hang.
- Default the storage map TP group to the group the layers were sharded over.
- Count sequence_parallel_size in the batch sizes of an engine's TP groups,
  as without an mpu.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant