Repository navigation
Conversation
The TP/DP groups in deepspeed.utils.groups are process-wide and the first engine creates them. _init_tp_mesh_device returned early when they were set, so a second engine with another autotp_size silently ran on the first engine's groups. An engine without AutoTP and without its own mpu took the earlier AutoTP engine's data parallel group and failed later in initialize. Check the requested sizes against the existing groups and raise a clear error in both cases. Engines with the same layout keep sharing the groups. This is stage 1 of deepspeedai#9081. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
An engine without an mpu inherits groups.mpu. After an engine that passed its own mpu, that is the user's mpu, not the cached AutoTP groups, so the guard raised for a valid sequence (AutoTP TP2, then an explicit TP1 mpu, then an engine without an mpu). Run the check only when groups.mpu is the groups module, and test that sequence. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…mesh A mesh replaces neither groups.mpu nor the model parallel size read from it, so after an AutoTP engine such an engine failed with an AttributeError on its missing mpu. Also match the issue number in the size mismatch test. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…eepspeedai#9081 stage 2) The first engine, and any later engine with the same layout, keep using the process-wide groups. An engine whose TP/DP layout differs now gets a TensorParallelGroups of its own instead of raising, and AutoTP, the engine, ZeRO and checkpointing read the groups from it. groups.mpu keeps describing the first engine. Not covered yet, and still raising: mesh_param, AutoEP, the DeepCompile autotp pass and MoE layers on an engine with groups of its own. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…engine Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> # Conflicts: # deepspeed/runtime/engine.py # tests/unit/v1/autotp/test_autotp_multiple_models.py
…rank its own batch in the parity test Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
An engine created without an mpu no longer inherits an earlier engine's mpu (deepspeedai#9081). Engines whose layout matches the process-wide groups still share the same underlying process groups. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
An AutoTP engine used to ignore the mpu it was given. Passing another engine's TensorParallelGroups now makes the engine run on those groups, which must have its tensor parallel size. An engine sharing groups other than the process-wide ones gets the same checks as one with groups of its own. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…gine's groups Passing another AutoTP engine's groups as mpu no longer changes the batch size arithmetic, AutoEP accepts the process-wide groups passed that way, and the checkpoint writer still finds the expert groups of an engine that holds its own groups. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
… as dense data parallelism An AutoTP engine created after an engine with an explicit mpu replaced that mpu in groups.mpu, so the earlier engine's code that still queries groups ran on the AutoTP engine's data parallel group. Such an engine now holds groups of its own instead. An engine's own TP1 groups are data parallelism over every rank: BF16 Muon accepts them, and the hybrid engine builds its inference tensor parallel groups as it does without an mpu. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…d the ranks of TP1 data parallel groups TensorParallelGroups.get_data_parallel_group_ranks used get_all_ranks_from_group, which never ends for the world group, the data parallel group of an engine's TP1 groups, so a TP1 engine with a checkpoint writer hung. A data/sequence parallel mesh engine built after an engine that set groups.mpu replaced the data parallel group the module getters return for that engine; an AutoTP engine with a matching tensor parallel size bypassed the deepspeedai#9081 check. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…ngine create the groups for tp_model_init A model already sharded by AutoTP, passed to an engine with another tensor parallel size, kept its layers on the groups it was sharded with; at TP1 the engine's initial broadcast overwrote the other ranks' shards, which the first engine still uses. tp_model_init created process-wide groups of its size before the engine could create groups of its own, so it failed after an engine with another tensor parallel size. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…id meshes against the engine's groups - A model wrapping one already sharded at another tensor parallel size raises, like the sharded model itself. - A tp_group passed to tp_model_init must have the ranks of the engine's TP group; the layers shard over the engine's group, which its data parallel broadcasts and reductions assume. - ZeRO-3 uses the hpZ group only when hpZ is enabled, instead of one built by an earlier engine for its layout. - Prebuilt AutoEP layers are rejected with groups of the engine's own, like MoE layers. - The hybrid engine is given the mesh, so a mesh after an AutoTP engine raises instead of being ignored. - tp_model_init with tp_size 1 does not create process-wide TP1 groups after an earlier engine. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…ence parallelism An engine without an mpu after a Ulysses engine got TP1 groups, which ignored its sequence_parallel_size: ZeRO-3 failed the batch size check and the other stages ran without sequence parallelism. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
… require a sequence parallel mpu to inherit - Inherit groups.mpu for sequence_parallel_size > 1 only when it provides sequence parallelism; after an AutoTP engine the engine crashed, and after an explicit mpu it took its data parallel group. - Raise for AutoTP layers sharded at another tensor parallel size without the model they were sharded in, whose shards the broadcast overwrote. - Reduce the tp_model_init group check over all ranks, so the ranks whose group matches do not hang. - Default the storage map TP group to the group the layers were sharded over. - Count sequence_parallel_size in the batch sizes of an engine's TP groups, as without an mpu. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 2 of #9081. Depends on #9088 (stage 1). The commits up to #9088's head are #9088's; the rest are new.
With #9088, a second
deepspeed.initializewhose tensor parallel layout differs from an earlier engine's raises. With this PR, every engine gets its own groups object, and an engine with another layout gets TP/DP groups of its own.Design
engine.mpuis its owngroups.TensorParallelGroups: an object with the mpu interface (get_tensor_model_parallel_group/world_size/rank/src_rank,get_model_parallel_*,get_data_parallel_*,get_data_parallel_group_ranks, and a pipeline size of 1).mpu, so they countsequence_parallel_sizetoo.process_wide=True), so no new communicator is created. The first engine creates the process-wide groups,groups.get_*keep describing it, andgroups.mpustaysgroupsfor the code that still reads it.process_wide=False) and is not published togroups.mpu.groups.mpuis anmpupassed to an earlier engine, an AutoTP engine without one does not replace it: it holds groups of its own (process_wide=False), reusing the process-wide TP groups if they have its layout. Before, it published its groups over thatmpu, so the earlier engine's code that still queriesgroups(gradient clipping among it) ran on the AutoTP engine's DP group.mpuno longer inherits an earlier engine'sgroups.mpu(reported on [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081: after an engine with an explicitmpuover half of the ranks, an engine over all ranks took thatmpu's DP group). If an earlier engine has published anmpu, the new engine gets TP1 groups of its own (DP over every rank). The first engine without anmpuis unchanged (engine.mpu is None).sequence_parallel_size > 1without a mesh keeps inheritinggroups.mpuwhen that provides sequence parallelism (an earlier engine's Ulyssesmpu), the only source of its sequence parallel groups; otherwise it gets TP1 groups too. With TP1 groups, ZeRO-3 failed the batch size check and the other stages ran without sequence parallelism.inference_tp_size) as it does without anmpu.deepspeed.initialize(..., mpu=other_engine.mpu). The tensor parallel size must match, and it cannot be combined withmesh_param. Before this PR an AutoTP engine ignored thempuit was given.mpu(over every rank), so passing the groups does not changetrain_batch_sizeor the gradient accumulation steps.groups._get_or_create_tp_groups(tp)makes the choice: it initializes the process-wide groups if there are none, and returns aTensorParallelGroupsover them if the layout matches or over new groups otherwise.self.mpuin these places:tp_group, andmp_sizein training mode);get_*_rank;checkpoint.writer), which readgroups.mpu. The writer still gets the expert groups fromgroups.mpuas before. Stage 3 also uses them when loading a universal checkpoint.groups.py, group creation is split out from writing the globals (_new_tp_groups,_build_tp_groups). The existing_init_tp_*paths behave as before.DeepSpeedEngine.__init__, where Raise when a later engine asks for another tensor parallel layout (#9081 stage 1) #9088 checks the layout. An engine that will run on groups other than the process-wide ones is checked there for what is not supported yet, before it changes any process-wide state, the model, or creates groups.clip_grad_norm_(ZeRO-0 FP32 gradient clipping) averages the norm over the engine's own DP group.Behavior changes
mpu, created after an engine that published one (an AutoTP engine or an explicitmpu), runs on TP1 groups over every rank instead of the earlier engine's groups. Code that passes anmpuonce and relies on later engines picking it up must pass it to each engine. A data/sequence parallel mesh engine, and an engine whose config setssequence_parallel_size > 1, keep the old behavior whengroups.mpuprovides sequence parallelism.autotppass yet (see below).mpualso runs on groups of its own, so it cannot usemesh_param, MoE, AutoEP, or the DeepCompileautotppass yet either.mpuraises, also when the tensor parallel size matches. Before, it replaced the data parallel group thatgroupsreturns for the earlier engine.tp_model_initwithtp_size > 1, or withtp_size1 after an earlier engine, no longer creates the process-wide groups itself; the engine creates them, or groups of its own when an earlier engine has another tensor parallel size. Atp_grouppassed totp_model_initmust have the ranks of the engine's TP group, which the layers now shard over. All ranks agree on the check, so the ranks whose group matches do not hang.autotp_storage_affine_mapstakes its default TP group from the model's AutoTP layers, since an engine with groups of its own does not publish them.mesh_param(ordata_parallel_size/sequence_parallel_size), which it ignored, so a mesh after an AutoTP engine raises for it too.checkpoint.writerfailed on master withAttributeError: module 'deepspeed.utils.groups' has no attribute 'get_pipeline_model_parallel_world_size', becausegroups.mpuwas thegroupsmodule. The writer now reads the engine'sTensorParallelGroups, so this works.Not covered yet (raises with a link to #9081)
An engine that runs on groups other than the process-wide ones (its own, or shared from such an engine) cannot yet use:
mesh_param;autotppass;These still read the process-wide groups. A mesh engine created after an AutoTP engine or an explicit
mpu, and an AutoTP engine created after a mesh engine, raise. Following the roadmap, MoE/AutoEP/compile are stage 4, and the remaining checkpoint helpers (thezero_affinedefault replica group of parameters outside a ZeRO optimizer) are stage 3.Left for later stages of #9081:
mpu. Destroying groups needs an owner and a reference count; reusing them automatically by layout belongs with that.Zero3Repartitiondoes by hand), and engines over a changed set of ranks.#2989 and #3592 were looked at as possibly related. Both fix nested
zero.Initand dynamic classes, and neither involves process groups.Tests
tests/unit/v1/autotp/test_autotp_multiple_models.py:groupskeep TP2.mpuafter an engine with an explicitmpuover half of the ranks runs with DP over every rank, whilegroupskeeps describing the explicitmpu; also after an AutoTP engine and an explicitmpu. The first engine without anmpuis unchanged. After a Ulysses SP2 engine, an engine without anmpuwithsequence_parallel_size: 2keeps SP2 under ZeRO-1 and ZeRO-3; after a TP2 engine or an explicitmpuover half of the ranks, such an engine gets TP1 groups with DP over every rank.mpuleavesgroups.mputo it, holds groups of its own, and trains a step; after a TP2 engine and an explicitmpu, it holds the first engine's groups without publishing them.inference_tp_size=2gets inference TP groups of size 2.mpu=world.mpushares its groups, trains a step, and raises for MoE. A TP4 engine passed another TP4 engine'smpushares its groups; a TP2 engine passed it raises.mpu(TP1/2), AutoTP after mesh, and MoE with groups of its own raise. The MoE case raises before the model is moved or any group is created.checkpoint.writerafter a TP2 engine gets a writer config for its own DP size and rank (the TP1 case used to hang listing the ranks of the world group). A TP2 engine with MoE layers andcheckpoint.writerbuilds its writer.nn.Sequential, and for a sharded layer of it wrapped alone. After a TP2 engine, a model throughtp_model_init(tp_size=4)gets TP4 groups of its own, and one throughtp_model_init(tp_size=1)gets DP over every rank; atp_model_initgroup with other ranks than the engine's TP group raises, also when only some ranks' groups differ. After an explicitmpu, the storage maps of a TP2 engine's model are built over its own TP group.mesh_paramafter a TP2 engine raises.train_batch_sizeas that engine, withtrain_batch_sizeandgradient_accumulation_stepsboth set.mpushares its groups and trains a step.Each new test fails without the change it covers (checked by reverting it).
Results (latest commit):
TestAutoTPEnginesWithDifferentLayouts(41), the parity test (8), and thetp_model_inittests oftest_autotp_training.py(5, skipped on XPU by default, run without the skip) pass under torchrun. Under pytest, some tests intermittently hang in the harness's XCCL init barrier before any test code runs; run alone, each test passes.test_autotp_multiple_models.py+test_groups.py30 passed on the previous commit (before sharing viampu). On the wider AutoTP set, the failures are the FusedAdam JIT build on that machine, andTestGPTBigCodeMultiQueryTraininghangs identically on this PR's earlier head.