Skip to content

[rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move - #9074

Draft
LiRunGuo wants to merge 9 commits into
deepspeedai:masterfrom
LiRunGuo:in-memory-relayout
Draft

LiRunGuo wants to merge 9 commits into
deepspeedai:masterfrom
LiRunGuo:in-memory-relayout

Conversation

@LiRunGuo

@LiRunGuo LiRunGuo commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Status: draft for discussion, not for merge as it is. Part of RFC #9026, which has not been accepted yet. I'm opening it so that the two-stage relay, the pipelined execution and the training-state move can be discussed on code that runs, next to #9045. The split I'd propose for upstream is at the end; each piece would be its own PR after the RFC is settled.

This PR does not propose a new rollout engine. The NativeRollout in the third commit is only the destination its tests write into; it is not registered with build_rollout and stays out of upstream. For the hybrid engine, the re-shard at the training/rollout boundary (ZeRO-3 partitions into the inference layout of HybridEngineRollout) is #9045's AffineWeightSync, which takes maps only and can be called from HybridEngine.eval(). What is here are the mechanisms beyond a fixed trainer on one node: the trainer's own layout changing with its optimizer state, ZeRO-3 over a subset of the ranks (#9076), and moves across nodes.

What it is. Moving model state between two sharded layouts in GPU memory, without a checkpoint and without gathering a whole parameter on any rank:

  • weights from a ZeRO-3 trainer into a tensor-parallel inference copy on other ranks, or from one inference layout to another (other ranks, another TP degree);
  • a ZeRO-3 trainer's fp32 master weights, optimizer moments and step counters over other ranks (for example DP4 → DP2 → DP4 inside the same processes), in two forms: from one live engine to another (Zero3StateMove), or inside one engine that changes its data-parallel ranks in place (Zero3Repartition, commits 6 to 8).

The eight commits

  1. [zero] ZeRO-3's rank-agreement checks and barriers run over the partition group. 31 lines added and 19 removed in five core files (stage3.py, partition_parameters.py, partitioned_param_coordinator.py, parameter_offload.py, zero/utils.py).
    • A ZeRO-3 engine whose data-parallel group is a subset of the world hung or failed its trace checks: the checks broadcast from global rank 0 over the world, and the barriers waited for every rank.
    • With the default group the same ranks take part as before: assert_ints_same_as_other_ranks(ints, group=None) compares over the world, and an engine's default dp_process_group holds every rank.
  2. [rollout] relayout.py, affine_plan.py, zero3_state.py (new files; no existing file changes).
    • RelayoutPlan: both sides are declared per rank as affine pieces. Source and destination ranks may be disjoint, equal or overlapping, and a weight may have several holders. Planning runs identically on every rank from the gathered descriptions; execution is one all_to_all_single per bucket of weights and stage.
    • topology="relay": a block that crosses nodes is fetched once per node and copied to the node's other ranks in a second stage. spread=True has the replicas of a node fetch bands of each large block and exchange them, so the traffic uses every NIC of the node. Over buckets the stages are pipelined: stage one of a bucket travels with stage two of the bucket before.
    • planner="affine" (affine_plan.py): the same moves planned by plan_transfer. A relay is two plans through a relay map whose shards are bands of the destinations themselves, so there is no relay buffer. Two cases still fall back, per weight, to the module's own rectangle planner: holders in two layouts (waiting for Describe a parameter held in several layouts as one affine map #9052) and partial targets (RFC [RFC] Two-stage relay for cross-node weight sync, priced by a calibrated cost model #9026, open question 1).
    • Zero3StateMove(src_engine, dst_engine): moves the training state through zero_optimizer_storage_affine_maps ([AffineIR] Affine storage maps for live ZeRO parameters and optimizer states #8885). release_training_state / restore_training_state give back and re-allocate the inactive engine's memory.
  3. [rollout] native/ and the tests. A small tensor-parallel decoder whose weights are views of one buffer per rank, written in place, so CUDA graphs captured over one version of the weights stay valid for the next. NativeRollout is the destination the tests move weights into. RolloutBatch gains an optional response_logprobs.
  4. [rollout] NativeRollout leaves build_rollout, which is as on master again. The tests construct NativeRollout directly.
  5. [rollout] Zero3StateMove aligns the destination's micro-step counter to the destination's accumulation steps. It copied the source's counter, so a destination that accumulates over more micro-batches than the source started mid-window: after one step of a source with one micro-batch per step, a destination with two stepped on its first micro-batch alone.
  6. [rollout] Zero3Repartition (in zero3_state.py, and a new test file). One ZeRO-3 engine changes its data-parallel ranks at a step boundary, with no second engine.
    • It re-points the engine's references to its data-parallel group. It allocates buffers of the new partition sizes (one per dtype, as ZeRO-3 holds the low-precision and gradient partitions) and points the existing tensor objects at them, so the optimizer state, the hooks and the parameter coordinator stay valid.
    • It recomputes the cached sizes and the gradient accumulation steps (the global batch is kept). It moves the fp32 master partitions and the moments with a RelayoutPlan, cached per source layout and optimizer state kinds, and refreshes the low-precision partitions from the fp32 master. The ranks that leave release their state; they keep the engine and can come back.
    • A rank never holds a second engine's partitions: the peak on a DP2 rank in the table below is 7.9 GiB instead of 9.4.
  7. [rollout] Zero3Repartition keeps the parameter fetch trace when every new member has a complete one.
    • The sixth commit restarted the trace at every change, so the first step after each change ran without prefetching.
    • On the setup below, in one job: that step took 1.00–1.04 s instead of 0.82–0.83 s on two ranks, and 2.35 s instead of 1.49–1.51 s on four. After the commit, it takes as long as with two engines.
    • A restart is still needed when a member has not stepped in the engine yet. Its trace is compared with the others' in a collective that ranks with a complete trace do not join (the case noted in [BUG] ZeRO-3 ids come from process-wide counters, so ranks that built different models fail the rank-agreement checks #9098).
  8. [rollout] Zero3Repartition with parameters kept in fp32, and its preconditions checked on every rank.
    • With data_types.keep_in_fp32_modules (Keep the parameters a model names for fp32 in fp32 under ZeRO-3 #8843), the kept parameters form sub-groups of their own whose fp32 master is the weight partition itself. The sixth commit pointed these masters with ZeRO-3's _point_aliased_fp32_partitions_at_lp_buffer, which slices the training dtype's buffer only, so a kept parameter's master would view the bf16 buffer. Each aliased master now views its own sub-group's new partition.
    • The parameters' process group, hpZ and Muon were checked after the first collective and only on the new layout's members, so the ranks that leave waited in the next collective instead of failing. They are now checked with the other preconditions, on every rank. One case this turns from a hang into an error: a model built under zero.Init with an mpu but no data_parallel_group is partitioned over the world, not over the engine's group.

How it relates to open work

Validation

One node with 4× RTX 3090 (24 GB), driver 595.91.07, torch 2.13.0+cu130, NCCL 2.29.7. Commands from tests/, each as python -m pytest -q -rfEs <files>. The rows marked head ran on the head commit, on another node with the same GPUs and software (driver 595.58.03). The others ran on the sixth commit; the seventh and eighth change only Zero3Repartition and its test.

Tests Result
New: unit/runtime/rollout/test_zero3_repartition.py (4 ranks), head 6 passed
New: unit/runtime/rollout/test_relayout.py, test_native_decoder.py, test_native_rollout.py 168 passed
test_relayout.py, test_native_rollout.py, head 162 passed
All of unit/runtime/rollout 268 passed, 1 skipped, 1 failed (below)
Affine IR: unit/checkpoint/test_affine_shard_map.py, test_affine_transfer_plan.py, test_affine_transfer_executor.py, test_affine_transfer_executor_external_group.py, test_affine_replica_policy.py, test_zero_affine.py 350 passed, 1 skipped (CPU-only test)
ZeRO, for commit 1: unit/v1/zero/test_zero.py, unit/v1/zero/test_zero_init_pg_contradiction.py, unit/runtime/zero/test_zero_context.py 136 passed, 1 skipped (marked "not working" on master)
  • The failure and the skip are both in test_hybrid_engine_rollout.py. They need the transformer_inference extension, which does not build in this environment: its CUDA toolkit has no cuda_profiler_api.h. With that header taken from the machine's CUDA 13.0 install, the file passes (72 passed). Neither test touches this change.
  • ZeRO-3 over a subgroup of the world (commit 1) is exercised by TestNativeRolloutApart and TestTrainerSwitch in test_native_rollout.py.
    • TestTrainerSwitch also compares training across two trainer layouts with a plain-torch AdamW reference.
    • TestTrainerSwitchAccumulation moves the state into an engine that accumulates two micro-batches per step. It fails without the fifth commit (its first micro-batch is taken as a boundary).
  • test_zero3_repartition.py, main test. One engine trains on four ranks, shrinks to two, trains, grows back to four, trains, and shrinks again with the cached plan.
    • Every micro-batch is the whole batch on every rank, so each layout computes the same gradient exactly.
    • After every phase, the fp32 master and the Adam moments equal bit for bit those of a plain engine that stays on four ranks. This holds for the rectangle and affine planners, in fp32 and bf16.
  • test_zero3_repartition.py, second test. The engine shrinks before its first step, then grows back ranks that never stepped (the trace case of commit 7).
  • test_zero3_repartition.py, third test (commit 8). A tiny Qwen3 is built under zero.Init with a logit scale kept in fp32 beside its bf16 weights. It shrinks from four ranks to two, trains, grows back and trains, against an engine that trains on four ranks throughout.
    • The moves keep the state bit for bit, and after every phase the state equals the other engine's. The partitions keep both dtypes, and the kept parameter's forward value is its fp32 master.
    • On the seventh commit, the new members raise in the install (the master's slice runs past the end of the bf16 buffer) and the other ranks wait.
  • Breaking the code fails the tests. I broke Zero3Repartition in eleven ways, one at a time, and each break makes these tests fail or hang (head). The breaks: moving only the fp32 master; keeping the old gradient positions; keeping the old partition sizes; keeping stale step counters on a rank that comes back; not aligning the micro-step counter; never restarting the trace; always restarting it; leaving the state kinds out of the plan cache key; one low-precision buffer in the training dtype for every parameter; one gradient buffer for every parameter; ZeRO-3's alias helper instead of the per-sub-group alias (the code before commit 8).

End to end, a ZeRO-3 trainer goes DP4 → DP2 → DP4 inside the same four processes (Qwen3-0.6B, bf16, Adam; 310 parameters; 6.66 GiB of fp32 master weights and two moments). Both paths ran, three rounds each per planner, with a training step after every move (sixth commit):

Path Planner DP4 → DP2 (s) DP2 → DP4 (s) Peak on a DP2 rank, DP4 → DP2 / DP2 → DP4 (GiB)
two engines (Zero3StateMove) rectangle 0.58 0.92 9.41 / 9.41
two engines (Zero3StateMove) affine (plan_transfer) 0.56 0.93 9.41 / 9.41
one engine (Zero3Repartition) rectangle 0.57 0.93 7.92 / 7.37
one engine (Zero3Repartition) affine (plan_transfer) 0.57 0.94 7.92 / 7.37
  • Every move is bitwise equal (6 of 6 per path and planner), by a layout-independent digest of every parameter's master weight and moments, taken before and after each move. The step counters are carried. With the rectangle planner on both paths, one flipped bit in one moment is caught and named, and the next check is clean.
  • Times are medians of 3 on RTX 3090s, which exchange data over PCIe, so they say little about faster links. Each plan is made once per direction (rectangle 0.06 s, affine 0.45 s) and reused.
  • Memory. On both paths a DP2 rank holds 3.17 GiB before the move and 5.39 GiB after. The peak covers both layouts plus the receive buffers; it is 1.5 to 2 GiB lower with one engine.
  • The two-engine path builds its second engine in 2.1 s inside the running processes. That engine's first step takes 2.6 s, since it records its fetch trace; with one engine, the first step on two ranks takes 0.9 s.
  • First step after a move, measured in a separate job that ran the sixth and seventh commits alternately, twice each:
    • two engines: 0.83–0.85 s on two ranks and 1.48–1.51 s on four, at both commits;
    • one engine at the sixth commit: 1.00–1.04 s and 2.35–2.37 s;
    • one engine at the seventh commit: 0.82–0.83 s and 1.49–1.51 s.
    • In that job, steps and moves over all four GPUs were slower than in the run above on the same machine (a four-rank step took 1.5 s against 0.7 s, even before any move). Steps on two ranks took the same time in both. So only comparisons within the job hold.
  • In an RL loop on the same machine (Qwen3-0.6B, generation and training sharing the GPUs, the trainer moving between all four GPUs and two), on the sixth commit:
    • one re-partitioned engine ends in the same training state as two engines with the state moved between them: the sums of the fp32 master weights and of both moments agree to the last digit;
    • the loop takes as long, within the variation between launches;
    • it needs 1.5 GiB less on the training GPUs.
  • The benchmark scripts are not part of this PR.

What I could not run, and known limits

  • ZeRO-3 only, bf16 with fp32 master weights, Adam / AdamW, no parameter or optimizer offload, no NVMe. ZeRO-1/2, fp16 loss scaling, pipeline parallelism and MoE / expert parallelism are not covered.
  • CUDA with NCCL only. Not run on CPU (gloo) or on another accelerator. The new code goes through get_accelerator() and deepspeed.comm, but the native decoder's graph capture is CUDA-specific.
  • After Keep the parameters a model names for fp32 in fp32 under ZeRO-3 #8843 a ZeRO-3 model may hold partitions of two dtypes (keep_in_fp32_modules). Zero3Repartition keeps one buffer per dtype, as ZeRO-3 does; the third test covers one parameter kept in fp32 beside bf16 weights. NativeRollout assumes one dtype.
  • Zero3Repartition changes ranks only among the processes that built the engine. It does not support offloaded state, frozen parameters (e.g. a LoRA base), hpZ, MiCS, sequence parallelism or quantized weights. No test saves a checkpoint after a change.
  • Zero3Repartition checks its preconditions before its first collective, on every rank. An error after that point on some ranks only (in the install) still leaves the other ranks waiting.
  • Zero3Repartition sets ZeRO-3 internals directly: the group references, including the parameter coordinator's private process group, and the cached partition sizes. A change to those internals needs a change here. [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081's per-engine groups would remove the group part.
  • LoRA / PEFT students are refused by NativeRollout.bind (NotImplementedError).
  • On both paths, both layouts are resident on a rank during a state move, so a rank needs room for its old and new partitions at once.
  • Open issue on two nodes, below this code. With a 32B model on 2 nodes × 4 H100, the bitwise check after a cross-node move failed in 2 of 162 checked moves.
    • An element-wise dump shows blocks on a 128 KiB grid holding the bytes of another block of the same all_to_all_single message. Replaying the same plans on CPU is exact for all 707 weights.
    • A stress test of the bare collective reproduces the same signature at the message sizes of these moves: torch.distributed.all_to_all_single with self-describing data, no DeepSpeed, no model.
      • It found 6 misplaced 128 KiB blocks in 370 TB between two nodes.
      • It happened with NCCL's default settings (1 in 121 TB) as well as with NCCL_IB_QPS_PER_CONNECTION=4 or NCCL_NCHANNELS_PER_NET_PEER=8.
      • There were none in 159 TB in an earlier run whose messages all stayed at or below 256 MiB per pair.
    • So the cause is below this executor (torch 2.13.0, NCCL 2.29.7, IB on that cluster); which component it is has not been established. The digest after each move is what detects it.
    • An 8B model's digests were clean in all 39 two-node configurations I ran.
  • No documentation page and no tutorial yet. They belong with the pieces that go upstream (the weight path would extend affine-transfer.rst from [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045).

Size, and the split I'd propose. 2,709 new lines under deepspeed/ and 1,476 in tests is too much for one review. For upstream I would not send it in this form:

  1. Commit 1 as its own small PR (it is useful to any ZeRO-3 engine over a subgroup): now [zero] Run ZeRO-3's rank-agreement checks and barriers over the partition group #9076.
  2. The training-state change rewritten on the IR alone (plan_transfers + execute_transfers, as [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045 does for weights), with DP4 → DP2 → DP4 as its test.
  3. The relay as an addition to RFC [RFC] Two-stage relay for cross-node weight sync, priced by a calibrated cost model #9026: two plan_transfers calls through a relay map. The planner needs nothing new; the open questions are where the relay-map constructor lives and how the choice meets the cost model of [AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) #8966 / [AffineIR] Analyze affine transfer batches before execution (Plan analysis & calibration Part-1) #8967.
  4. The pipelined execution after [checkpoint] Expose the affine transfer schedule as schedule_transfers #9027.
  5. The rectangle planner stays out of upstream. It goes away once Describe a parameter held in several layouts as one affine map #9052 and partial targets cover its two remaining cases.
  6. NativeRollout stays out; the destination upstream is feat(rollout): add AutoTP rollout engine with in-memory weight sync #9023's engine through [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045's sync.

Three questions for the RFC:

AI assistance: AI assistance was used. The code, tests and this description were drafted with an AI coding agent (Claude Code). The submitter reviewed every changed line and ran the tests above.

Maintenance contact: @LiRunGuo. cc @Achyuthan-S @0z5a @jinyouzhi

…tion group

A ZeRO-3 engine whose data-parallel group is a subset of the world (two
trainer layouts living in the same processes, or a trainer next to
inference-only ranks) hung or failed its trace checks: the checks broadcast
from global rank 0 over the world and the barriers waited for every rank.
They now run over the parameters' partition group; with the default group
nothing changes.

Signed-off-by: RunguoLi <li19107254665@gmail.com>
…age relay, pipelined execution, ZeRO-3 training-state move

relayout.py: RelayoutPlan moves weights between two sharded layouts in GPU
memory. Both sides are declared per rank as affine pieces; the source and
destination ranks may be disjoint, equal or overlapping, and a weight may
have several holders (the trainer's ZeRO-3 partitions, surviving shards of
the previous layout, replicas).
- Planning runs identically on every rank. Holders are preferred in the
  order: the destination rank itself, its node, the caller's layout order,
  the least loaded.
- topology="relay": a block that crosses nodes is fetched once per node and
  copied to the node's other ranks in a second stage; spread=True has the
  replicas of a node fetch bands of each large block and exchange them.
- Execution: one all_to_all_single per bucket of weights and stage; over
  buckets the stages are pipelined (stage one of a bucket travels with stage
  two of the bucket before).

affine_plan.py: the same moves planned through the affine IR's plan_transfer
(planner="affine"), a relay being two plans through a relay map whose
shards are bands of the destinations themselves. Weights it does not express
yet (holders in two layouts, partial targets) fall back to the rectangle
planner per weight.

zero3_state.py: Zero3StateMove moves a ZeRO-3 engine's fp32 master
partitions, optimizer moments and step counters to another ZeRO-3 engine of
the same model over other ranks, from the optimizer storage maps;
release_training_state / restore_training_state give back and re-allocate
the inactive engine's memory.

Tests come with the next commit (they use the native rollout's layouts).

Signed-off-by: RunguoLi <li19107254665@gmail.com>
…a; tests of the relayout and the state move

native/: a small tensor-parallel decoder whose weights are views of one
buffer per rank, allocated when the layout is created and written in place,
so CUDA graphs captured over the first version of the weights stay valid for
later ones. NativeRollout syncs it from a ZeRO-3 trainer and changes its
ranks or tensor-parallel degree through RelayoutPlan; a replica whose ranks
and degree are unchanged keeps its buffer, KV cache and graphs.
build_rollout takes engine="native"; RolloutBatch gains an optional
response_logprobs.

Tests: test_relayout.py (both planners: ZeRO-3 to tensor parallel, degree
changes, a lost rank refilled, two nodes flat / relay / spread, pipelined
buckets, the planners' inter-node bytes compared, fallbacks, contradictory
maps), test_native_decoder.py, test_native_rollout.py (sync, re-placement,
the trainer switch with Zero3StateMove).

Signed-off-by: RunguoLi <li19107254665@gmail.com>
@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks @LiRunGuo, this is great to see running end to end. A DP4 → DP2 → DP4 state move in 0.6–0.8 s with every move digest-checked is the comparison the RFC needed against the checkpoint path: the same Qwen3-0.6B ZeRO-3 changes through a universal checkpoint took me 46 s (DP4 → DP2) and 51 s (DP2 → DP4) on 4× A100, most of it the relaunch.

To your two questions, as the RFC's author:

The split: +1.

The relay-map formulation: yes. Bands of the destinations as relay shards is the alternative already recorded under open question 1 in #9026, from your earlier comment: no partial targets and no relay buffer. On the execute_transfers path it still needs #8809 to accept the stage where a relay rank's source lies inside its own target (_validate_aliases). The RFC lists relay_map and the pipelined execution under @0z5a. @0z5a, would you take this formulation as the constructor's definition, or keep relay_map(source_map, relay_locations, min_split_bytes) and add bands as an option?

The choice. I'll take what is a flag here: whether a weight goes direct, through a relay, or through a relay with spread, and the thresholds (min_relay and min_split, both 65,536 elements now), chosen by the calibrated model in §3 as the cost-aware policy on #8936. Your relay becomes one of the candidates the policy prices, and your InfiniBand runs are its first check.

Before and after. I can run your state move on the same 4× A100 node as my checkpoint-path numbers, with the same model and changes, so there is one comparison on one machine. I'll update #9026's body with the relay-map formulation and the split once @0z5a has weighed in.

@tohtana tohtana added the needs-gpu-test GPU testing is recommended to validate the changes in this PR. label Nov 13, 2026
@jinyouzhi

Copy link
Copy Markdown
Contributor

@LiRunGuo, thanks for the split at the end of the description. A note on step 2 (the training-state move rewritten on plan_transfers + execute_transfers).

The description says no open PR covers a live move of the optimizer state. deepspeedai/DeepSpeedExamples#1021 (open since 10-22) already does this on the IR, beyond ZeRO-3:

  • this comment moves bf16 params, the fp32 master and the Adam moments through the shipped zero_affine / autotp_storage_affine_maps(..., optimizer_states=True) maps, mid-training, across:
    • training world size: Z3x4 → Z3x2 → Z3x4, and 4 → 8 → 4;
    • ZeRO stage: Z3 ↔ Z2, Z3 ↔ Z1, and stage and DP changed together (for example Z2x8 → Z3x4 → Z1x8);
    • AutoTP size: TP2 ↔ TP4 under ZeRO-1/2/3, and up to TP8 combined with a stage change;
    • AutoEP size: EP2 ↔ EP4 ↔ EP8 under ZeRO-1/2/3.
  • this comment adds ZeRO-3 with aliased masters (ZeRO-3: reuse the weight partition as the master copy when it already has the master dtype #8842).
  • Every switch is checked against an oracle that shares no code with the IR: safe_get_full_* / safe_set_* plus a plain all-gather over the TP or EP group. The 60-step loss curves and the final logits are compared bitwise between the two paths.
  • Hardware: 8× NVIDIA H20.

Suggestion. Since step 2 is a rewrite anyway, could it target this wider matrix rather than ZeRO-3 only? The IR already describes every one of these layouts. What differs per case is:

  • which maps the engine exposes;
  • what has to be refreshed after the move (low-precision partitions, optimizer scalars, counters).

Zero3StateMove would become one case of a move between two engines. The scripts in #1021 (z3x.py, tpep.py) can be turned into unit tests, with the gather path as the oracle. I'm glad to help with that part, or to take the ZeRO-1/2 and AutoTP cases on top of your ZeRO-3 piece, whichever you prefer.

One gap the TP cases expose. A second engine with another TP size overwrites the process-global groups in deepspeed/utils/groups.py (_TENSOR_MODEL_PARALLEL_GROUP, _DATA_PARALLEL_GROUP, mesh_device, mpu). #1021 works around this by saving and restoring them per engine, which is not something core should do. #9076 fixes the same class of problem for ZeRO-3's checks and barriers; for TP and EP it would need each engine to hold its own groups. I think that is worth agreeing on in #9026 before step 2 goes beyond DP changes.

This comment was drafted with AI assistance and checked by me against #9074's description and the #1021 comments.

@LiRunGuo

Copy link
Copy Markdown
Contributor Author

Thanks @Achyuthan-S.

One machine, both paths. Thanks for the offer. The benchmark script is not in the PR and I'm not publishing it yet, so I can't hand you the run for now. I have the same run queued on a 4× A100 node and will post it here when it finishes; that is another machine than yours, so it puts the same GPU model next to your checkpoint numbers, not one machine. @jinyouzhi's run on #9026 already gives one pair on one machine, measured independently (4× L20, ZeRO-1 DP4 → ZeRO-3 DP4: 0.62 s in the running processes against 27.4 s through a universal checkpoint). The difference to this PR is the check: sums there, a bitwise digest here.

Per-weight planning. Right: affine_plan.py calls plan_transfer once per weight with its own selector. The selector keeps its load across the weights of a move, so the balance is over the batch, but it is not #8936's policy interface. Item 2 of the split moves the state move onto plan_transfers. The order the selector implements today is the target itself, then a holder on the target's node, then the caller's order, then the least loaded. That is local_first_least_loaded with two keys before the load, so a policy that expresses it needs each location's node, or the link class from #8966.

_validate_aliases. Agreed that this is what the relay map needs from #8809. In stage two a relay rank's source is a band inside its own destination shard. The check compares whole buffer ranges, so it refuses the batch, although no element is both read and written: the band onto itself is the same memory, and the other bands the rank receives lie outside it. The narrowest change I see is to drop a local segment whose source and target are the same memory with the same strides, and to check overlap on the regions the remaining segments read and write. relayout.py's executor relies on the same property today. @0z5a, your call.

The choice. Good, it is yours. Data for it, on 2 nodes × 4 H100 with Qwen3-8B (a ZeRO-3 trainer on node 0, replicas on node 1): I ran 30 configurations with both planners: three syncs from the trainer (into TP4, TP2 × 2 and TP1 × 4) and seven transitions (four layout changes, read from the old replica or from both sources), each direct, relayed, and relayed with spread.

  • Every move passed a per-replica digest (270 checks).
  • The two planners send the same inter-node bytes in all 30 configurations, and their medians agree within 1% in 22.
  • Sync from the trainer, relayed with spread: 0.176 / 0.186 / 0.213 s for TP4 / TP2 × 2 / TP1 × 4 with plan_transfer, 0.177 / 0.187 / 0.213 s with the rectangle planner.

Both planners run on this PR's executor, not on #8809. I can post the per-configuration table (bytes inside and across nodes, times) on #9026 as points for §3.

The two-layout fallback. In those runs 217 of 399 weights planned from both sources fell back to the rectangle planner. I'll rerun that case on #9052, and on #9070 for the column case, and report how many each removes.

#9076 now carries its regression test and is marked ready for review.

@tohtana tohtana removed the needs-gpu-test GPU testing is recommended to validate the changes in this PR. label Nov 14, 2026
NativeRollout is the destination the relayout and state-move tests write into, not a rollout engine proposed for
users. build_rollout is as on master again; the tests construct NativeRollout directly.

Signed-off-by: RunguoLi <li19107254665@gmail.com>
@LiRunGuo

Copy link
Copy Markdown
Contributor Author

The 4× A100 run finished: A100 80GB PCIe, one node, the same DP4 → DP2 → DP4 move inside the same four processes, six rounds per planner, a training step after every move. The node's other four GPUs were running other jobs at the time, so the spread is wide.

Model (fp32 master + Adam moments) Planner DP4 → DP2 (s) DP2 → DP4 (s) Plan, once (s) Peak on a DP2 rank (GiB)
Qwen3-0.6B (6.66 GiB) rectangle 0.79 (0.68–1.00) 1.96 (1.39–2.40) 0.05 9.41
Qwen3-0.6B affine (plan_transfer) 0.68 (0.59–1.19) 1.65 (1.53–2.49) 0.39 9.41
Qwen3-1.7B (19.2 GiB) rectangle 2.81 (1.97–3.56) 4.76 (4.12–5.78) 0.05 21.98
Qwen3-1.7B affine (plan_transfer) 1.75 (1.69–2.84) 4.23 (3.95–5.88) 0.38 21.98

Medians of 6, with min and max.

  • 48 of 48 moves are bitwise equal by the layout-independent digest. The flipped-bit test was caught for both models.
  • The DP2 engine is built in 2.3 s (0.6B) and 4.5 s (1.7B) inside the running processes.
  • Next to your 46 s (DP4 → DP2) and 51 s (DP2 → DP4) through a universal checkpoint on 4× A100 for the same 0.6B change, this is 0.7–2 s. These are different nodes, so read it as an order of magnitude rather than a ratio.
  • One thing I can't explain yet: DP2 → DP4 takes 2–2.5× as long as DP4 → DP2 here, while on the RTX 3090s the two were close (0.65 and 0.80 s). I haven't looked at this node's PCIe topology.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks @LiRunGuo, the A100 run is the comparison I was after, and 48 of 48 bitwise with the flipped bit caught for both models is reassuring.

To read it next to mine: of my 46 s (DP4 → DP2) and 51 s (DP2 → DP4), save + convert + load is about 15 s and 18 s. The rest is the relaunch, about 32 s and 33 s, and 23 s of that is building the model: on that cluster, importing transformers from a venv on the NFS home directory takes 21–25 s by itself. As @jinyouzhi's 10 s relaunch on #9026 shows, that part depends on the environment, so the part that carries over between machines is the 15–18 s through the checkpoint.

The policy's inputs. Right, a ReplicaRequest can carry processes and bytes but not where a process sits. #8936 lets a policy be any callable object, so the cost-aware policy will take the topology when it is built: each process's node, or the link classes from #8966's calibration, both data every rank shares. Your selector's order (the target itself, then its node, then the least loaded) is one ordering it can express, and the planner needs nothing new. The CSV on #9026 has what the fit needs; I'll fit it there.

The two-layout fallback. #9052 has the Codex fixes. Now that #9011 has merged I'm rebasing it onto master (the only conflict is that both add a helper at the same spot in affine_geometry.py and import it). I'll then put it on top of #9070 on a branch and link it on #9070, so you can run the column cases on both together.

Per-engine groups. Agreed with @jinyouzhi that a move across TP or EP sizes inside the same processes needs each engine to hold its own groups. I'll list #9081 in #9026 as what state moves beyond DP changes depend on.

…to its accumulation steps

The move copied micro_steps from the source. A destination that accumulates over more micro-batches than the
source then started mid-window: after one step of a source with one micro-batch per step, a destination with two
stepped on its first micro-batch alone. The counter is now rounded up to a multiple of the destination's
accumulation steps (the engine's boundary is (micro_steps + 1) % gradient_accumulation_steps == 0).

Signed-off-by: RunguoLi <li19107254665@gmail.com>
… state over other data-parallel ranks in place

Zero3StateMove moves a trainer's state between two engines, one per layout, built in advance. Zero3Repartition keeps
one engine and changes its data-parallel ranks at a step boundary: it re-points the engine's references to its
data-parallel group, allocates buffers of the new partition sizes (one per dtype, as ZeRO-3 holds the low-precision
and gradient partitions) and points the existing tensor objects at them (so
the optimizer state, the hooks and the parameter coordinator stay valid), recomputes the cached sizes and the
gradient accumulation steps (the global batch is kept), moves the fp32 master partitions and the optimizer's moments
with a RelayoutPlan from the old storage maps to the new ones, refreshes the low-precision partitions from the fp32
master, and releases the state of the ranks that leave. A rank that leaves keeps the engine and can come back. Every
new member restarts its parameter fetch trace, so that ranks which never stepped in the engine record the next step
together with the others. The plan is cached per source layout and optimizer state kinds (an optimizer has no
moments before its first step).

Supported: same-process changes of a ZeRO-3 engine whose parameters are all trained. Not supported: offloaded state,
frozen parameters, hpZ, MiCS, sequence parallelism, quantized weights, and a process that joins after the engine was
built.

Tests (tests/unit/runtime/rollout/test_zero3_repartition.py, 4 ranks): one engine trains on four ranks, shrinks to
two, trains, grows back to four, trains, and shrinks again with the cached plan; every micro-batch is the whole batch
on every rank, so each layout computes the same gradient exactly, and after every phase the fp32 master and Adam
moments equal bit for bit those of a plain engine that stays on four ranks (rect and affine planners, fp32 and bf16).
A second test shrinks before the first step, grows back ranks that never stepped, and shrinks again with the plan
made before the first step. Not covered by a test: a model with keep_in_fp32_modules (partitions in two dtypes).

Signed-off-by: RunguoLi <li19107254665@gmail.com>
… member has one

The parameter coordinator's fetch trace does not depend on the partition sizes. The re-partition restarted it on
every new member at every change, so the first training step after each change ran without prefetching (on four RTX
3090s with Qwen3-0.6B: 1.01 s instead of 0.82 s on two ranks, 0.94 s instead of 0.69 s on four). It is needed only
when a member has no complete trace (it has not stepped in this engine yet), since that member would compare its trace
with the others' in a collective the ranks with a complete trace do not join. The members now exchange whether their
trace is complete with the information they already gather, and all of them restart only if one has none.
`trace_restarted` says which happened. Tests: after a shrink of ranks that all stepped, the trace stays complete;
a grow that brings back ranks that never stepped restarts it.

Signed-off-by: RunguoLi <li19107254665@gmail.com>
…s on every rank

With data_types.keep_in_fp32_modules, ZeRO-3 puts the kept parameters in sub-groups of their own whose fp32 master is
the weight partition itself. The install pointed these masters with ZeRO-3's
_point_aliased_fp32_partitions_at_lp_buffer, which slices the training dtype's buffer only: a kept parameter's master
would view the bf16 buffer (past its end in the test). Each aliased master now views its own sub-group's new
partition and is again that sub-group's flat, as ZeRO-3 builds it.

The parameters' process group, hpZ and Muon were checked after the first collective and only on the new layout's
members, so the ranks that leave waited in the next collective instead of failing (zero.Init partitions over its
data_parallel_group, the world when none is given, also when given an mpu). They are now checked with the other
preconditions, on every rank, and raised on all of them together.

Test: test_parameters_kept_in_fp32 keeps a logit scale in fp32 beside the bf16 weights of a tiny Qwen3 built under
zero.Init, shrinks the engine from four ranks to two, trains, grows it back and trains, against an engine that trains
on four ranks throughout. The moves keep the state bit for bit, the state equals the other engine's after every phase,
the partitions keep both dtypes, and the kept parameter's forward value is its fp32 master. Without the fix the new
members fail in the install.

Signed-off-by: RunguoLi <li19107254665@gmail.com>
… partitions again

Upstream deepspeedai#8981 zeroes, at each accumulation boundary, the gradient
partitions that the window did not write. It finds them from a list of
the parameters with data in this rank's partition
(_params_with_grad_partition), which ZeRO-3 makes once. The list depends
on the layout, and a re-partition kept it: a parameter that had only
padding on a rank before and has data there after was left out, and when
it got no gradient in a window, the rank applied its last gradient again.
With gradient clipping (on by default), that gradient also enters the
global norm, so every parameter's update changes. The install now resets
the list (when the optimizer has one), and ZeRO-3 makes it again at the
next boundary.

test_unused_parameter_after_a_change: a two-element parameter, with data
on ranks 0 and 1 of four, moves to ranks [2, 3] (data on both), which
train a step that uses it and one that does not; the engine must match
one that stays on four ranks. On this branch merged with master the test
fails without the reset and passes with it; on the branch's own base,
before deepspeedai#8981, it passes either way.

Signed-off-by: RunguoLi <li19107254665@gmail.com>
banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Nov 20, 2026
…ai#9093)

For open question 1 in deepspeedai#9026, as @LiRunGuo proposed in
deepspeedai#9074 (comment).

In stage two of a relay, a relay rank holds its band as a view of its
own destination. It sends the band to its peers and receives the other
bands around it, so no element is both read and written.
`_validate_aliases` compares whole buffers, though, so
`execute_transfers` refuses the batch.

**Change.**
- The whole-buffer check stays as the fast path. When no target buffer
shares storage with a source or another target, nothing changes and
nothing extra runs.
- Otherwise, once the schedule is built, `_validate_aliases` collects
the regions this process reads (sends and local-copy sources) and writes
(receives and local-copy targets).
- A local copy whose source and target are the same memory at the same
strides moves nothing. It is dropped, and its region counts as read, so
another write into it is still refused.
- No write may overlap a read or another write. Regions are compared by
the bytes they span, so strided regions that interleave without sharing
an element are still refused. A band that is a view of its holder's
destination is one contiguous run of it, so the relay case doesn't hit
that.
- The check runs before the agreement, as before, so a refusal on any
rank stops every rank before data moves.
- `schedule_transfers` doesn't see buffers, so it still counts an
in-place copy as a local copy, and its docstring says so.
`TransferStats.local_copy_bytes` does not count it.
- Conjugate and negative views are refused in `_flat`, on the process
that holds one, before the agreement. Otherwise a lazy view of a
destination would pass as in place and be dropped (raised by the Codex
review, @LiRunGuo and @jinyouzhi). Point-to-point transport does not
carry the lazy bit, on master either. With gloo on CPU, a conj shard
fails inside the send after the agreement while its peer waits, and a
neg-view target is never written. The docs say to pass such views
resolved.
- Region extents use `address_range`, as `_validate_buffers` does.

**Tests** (14 new, in `test_affine_transfer_executor.py`):
- `TestB2RelayBandInPlace`, at 2 and 4 processes: every rank's band is a
view of its own full copy. Each rank completes its copy exactly, copies
nothing locally, and sends and receives N − 1 bands. With the view moved
to a peer's rows, every rank refuses before data moves, and the cause
names the overlap.
- In one process: a band kept in place next to a band copied in.
Refusals, each matched to its message, for swapped bands, for a band
kept in place that another request writes into, and for two requests
writing the same rows.
- `TestB2LazyViewsRefused`, at 2 processes: a conj or neg view of the
destination, and one on either side of a cross-process segment. Every
process refuses, the holder's cause names the view, and nothing is
written. On master (gloo, CPU) the destination cases fail on the old
message, the cross-process conj cases hang, and the neg-target case
completes with nothing written.
- On master the 3 acceptance cases are refused. If an in-place copy
counted as neither read nor written, the kept-band refusal would fail.

On CPU (gloo, up to 4 processes), `test_affine_*.py`,
`test_autoep_affine*.py`, `test_zero_affine.py` and
`v1/moe/test_autoep_checkpoint.py` pass: 970 passed, 4 skipped (956 on
master daf175b). yapf and flake8 at the pre-commit pins are clean.

**How it was found.** By reading `_validate_aliases` against the
stage-two shape of Runguo's relay on the affine IR (deepspeedai#9026, Q1): each
relay rank's band is a view of its own destination, so the whole-buffer
comparison refuses the batch although no element is both read and
written. The conj/neg part came out of review, and the cross-process
behaviour on master from probing each side and view on CPU.

**Production impact (estimate).** Today a relay built on
`execute_transfers` cannot run its second stage at all; with this change
it runs, and copies nothing for the band it already holds. Batches whose
buffers do not share storage see no change and pay nothing extra.
Conj/neg views need a complex tensor viewed through `.conj()` or an
explicit neg view, so we expect them to be rare in checkpoints; where
they did occur across processes, master hung or left the target
unwritten with gloo on CPU, and they are now refused up front.

---------

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants