Skip to content

[checkpoint] Fused copies, symmetric-memory window and staging release for prepared affine transfers - #9118

Draft
jinyouzhi wants to merge 9 commits into
deepspeedai:masterfrom
jinyouzhi:affine-fused-window
Draft

jinyouzhi wants to merge 9 commits into
deepspeedai:masterfrom
jinyouzhi:affine-fused-window

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #9094, which is itself rebased on #9093 (RFC #9026, step after "prepare once, run per step"). Until both merge, only the last four commits belong to this PR:

  • [checkpoint] Fused region copies for prepared affine transfers
  • [comm] Expose symmetric-memory rendezvous and peer tensors
  • [checkpoint] Copy window shards in place and release staging between runs
  • [checkpoint] Check window copies for aliases and keep 16-byte elements off the tile kernel

As agreed with @0z5a on #9026, the fused copy and the window transport share one PR and one copy-table format. copy_ stays the default everywhere.

What changes

Fused copies (opt-in copier=)

  • prepare_transfers(..., copier=fused_copies) replaces each round's pack, unpack and local-copy lists with a single Triton kernel launch (copy_tiles). The kernel reads a table of addresses, strides and extents for each region.
  • sequential_copies (one copy_ per view) is the default and keeps current behaviour.
  • fused_copies falls back to copy_ for views it cannot tile, for elements that are not 1, 2, 4 or 8 bytes (complex128), for host tensors, and when Triton is unavailable.
  • The kernel uses only generic Triton ops. The same code runs on CUDA and on XPU (Triton XPU backend). Triton is imported lazily, so CPU workers never touch a device driver.
  • The copier is built inside _agree. If building it fails, every process refuses the batch.

Symmetric-memory window (opt-in window=)

  • deepspeed.comm gains symm_mem_rendezvous and symm_mem_peer_tensor, so torch.distributed stays behind deepspeed/comm.
  • Tensors placed in a TransferWindow are copied directly into or out of the peer's window, pull or push, and skip the staged rounds.
  • Window copies use the same copier as the staged lists.
  • TransferStats.mapped_bytes reports how many bytes went through the window.
  • The window layout is part of the agreed fingerprint, so mismatched placements refuse the batch on every process.
  • Window copies take part in the region-level alias check from [checkpoint] Check aliases on the regions a transfer moves #9093: a pulled or pushed target counts as a write, a pulled or pushed source as a read on the process that owns it.
  • Every process must pass the same window, or none. A window passed on only some processes, or a failure while building it, hangs instead of refusing; the docstring and the docs say so.
  • Available on execute_transfer, execute_transfers and prepare_transfers.

Staging lifecycle

  • PreparedTransfer.release_staging() frees the staging arenas, round views and copier tables between runs, for example during training steps.
  • A process may release on its own; the next run() re-stages them. With check=True, a failure while re-staging refuses on every process.

Why this is not a duplicate

#9094 adds the prepared handle with copy_ only. This PR is the follow-up proposed in #9026 §5: the fused copy, the window transport and release_staging(). No other open PR touches affine_transfer_executor.py beyond #9094.

Tests

pytest tests/unit/checkpoint/test_affine_transfer_executor.py
Platform Result
CPU, gloo (DS_ACCELERATOR=cpu), current head 94 passed, 27 skipped (window and Triton tests need a GPU)
4× Intel Arc Pro B70, xe driver (kernel 6.19.10), torch 2.14.0.dev20260804+xpu, triton 3.8.0 (xpu), xccl, current head all pass at 2 and 4 processes
4× NVIDIA L20, torch 2.14.1+cu130, before the rebase 88 passed

New tests:

  • every copy list goes through the copier;
  • fused copies match copy_ for every region shape (strided, transposed, misaligned, 0-d, empty);
  • window transfers for each placement × copier, prepared and one-shot;
  • schedules leave window copies out of the rounds;
  • a released handle stages again on its next run (2 and 4 processes);
  • fused copies for complex64 and complex128;
  • a window region written on one process while a peer reads it is refused (pull and push, prepared and one-shot);
  • one process may release on its own; a process that cannot stage again stops every process with check=True.

Performance

Workload: Qwen3-4B, 398 bf16 tensors. 2-D weights move from a row split to a column split over N processes, and 1-D weights move between shifted even splits. Each cell is the median of 5 warm PreparedTransfer.run() calls in ms (lower is better); copy_ vs fused_copies.

8× Arc Pro B70 (cards 8–15, xccl):

N staging copy_ fused Δ
2 8 MiB 162.6 171.6 +5.5%
2 64 MiB 123.7 124.2 +0.5%
4 8 MiB 164.9 165.0 0.0%
4 64 MiB 137.6 136.7 −0.7%
8 8 MiB 103.2 99.6 −3.6%
8 64 MiB 109.0 104.7 −4.0%

4× L20:

N staging copy_ fused Δ
2 64 MiB 103.5 102.1 −1.4%
4 8 MiB 84.4 81.5 −3.4%
4 64 MiB 86.1 80.4 −6.6%
  • The fused copy helps more as N grows, because each round has more regions. At N=2 the runs are dominated by communication, and on B70 the 8 MiB case is slightly slower.
  • prepare_transfers with fused_copies costs an extra 50–75 ms once, to build the tables and compile the kernel. This is why copy_ stays the default.
  • Window transport, run() median at 64 MiB staging:
Platform N staged window pull window push
L20 4 83.6 83.8 79.2
B70 2 123.7 90.5 98.3

Compatibility

  • Only the affine checkpoint transfer path changes, and every new feature is opt-in. ZeRO stages, offload and the engine are untouched.
  • Tested with bf16 only. Multi-node runs and the window across nodes were not tested. The window needs torch.distributed._symmetric_memory.

AI assistance was used for this change.

0z5a and others added 4 commits November 18, 2026 16:49
_validate_aliases refused a batch whenever a target buffer overlapped a
source buffer. A relay rank in stage two holds its band as a view of its
own destination: it sends the band to its peers and receives the other
bands around it, so no element is both read and written, but the
buffers overlap and the batch was refused.

Keep the whole-buffer check as the fast path. When it finds an overlap,
check the regions this process's schedule reads and writes instead. A
local copy whose source and target are the same memory at the same
strides moves nothing: drop it and count its region as read, so another
write into it is still refused. Regions are compared by the bytes they
span, so strided regions that interleave are refused as before.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
The region check took a conjugate or negative view of a destination for
the destination itself: same address, same strides, so the copy was
dropped as in place and the batch returned without moving anything.
Master refused that batch only because the view shares the
destination's storage.

The lazy bit is not safe across processes either, on master too.
Point-to-point transport does not carry it. With gloo on CPU, a
conjugate shard fails inside the send on its own process after the
preflight has agreed, and the peer waits for it; a negative-view target
is never written. Refuse both views in the preflight, on the process
that holds one, so every process refuses before data moves.

Region extents now come from address_range, as in _validate_buffers.

schedule_transfers sees no buffers, so it still counts a copy dropped
as in place as a local copy; its docstring now says so.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
A weight sync moves the same batch every step, and every
execute_transfers call repeats validation, the alias check, scheduling,
the digest agreement, arena allocation and region views before the
first byte moves.

prepare_transfers does that work once and returns a PreparedTransfer
whose run only moves data. The handle keeps the arenas that deepspeedai#9027's
schedule reports, and for each round one list of (destination, source)
views: the packs before a send and the unpacks after a receive, plus
one list of local copies. execute_transfers is now prepare_transfers
followed by one run, with the same collectives as before. Python's
cyclic collector is paused while the views are built: tens of
thousands stay alive together, and the collections their allocation
triggered made a one-shot call at 8 processes on CPU about a third
slower than master.

A buffer that moves after prepare, as reload_states moves offloaded
ZeRO-3 partitions, would leave those views writing into the old storage. So
run checks by default: each process compares its buffers (the held
tensors, or requests rebuilt from the live state) with what it
prepared, and one small all-gather refuses on every process, before
data moves, if a buffer moved on any of them or the processes are not
running the same handle for the same time.

The shape of prepare_transfers and the buffer layout check follow
#8 (a8c34e3); this is its staged half, without the
window transport.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Jin, Youzhi <youzhi.jin@intel.com>
…they run

Two handles prepared for the same batch over different buffers had the
same schedule digest, so run's check passed when each process ran a
different one, and the peers exchanged payloads of different batches.
Each process now counts its prepare_transfers calls and adds the count
to the prepare all-gather. A handle's identity is a hash of the digest
and the gathered counts, the same on every process, and the check
compares that instead of the digest. The scheme is jinyouzhi's, from
the review.

execute_transfers no longer goes through a handle. It runs the same
preflight and agreement, then builds each round's views when the move
reaches that round, as it did before prepare_transfers, so building
them overlaps the previous round's transfer. On L20 GPUs, building them
all first made a one-shot call 10-16 ms slower than master.

The docs and the handle's docstring now say that run() checks only the
tensors the handle holds: a view such as zero_param_storage_shard
returns keeps the old storage when reload_states re-points the
partition, so such handles need run(requests) rebuilt from the live
state. They also say that a handle keeps its buffers allocated until it
is dropped.

The prepared-run test no longer reads the handle's private rounds, and
the garbage-collector test refuses through a too-small budget instead
of patching a private helper.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@0z5a

0z5a commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Thanks @jinyouzhi. This keeps both things we agreed on #9026: one copy-table format for arena and window copies, and copy_ as the default everywhere. I read all three commits. I couldn't run the Triton or window paths, because I have no GPU here, so points 1 and 2 come from reading the code. On CPU (gloo, up to 4 processes) I ran the 9 files test_affine_*.py, test_autoep_affine*.py, test_zero_affine.py and v1/moe/test_autoep_checkpoint.py on 57c37d5: 779 passed and 15 skipped. That is 768/4 on #9094 plus your 11 new tests, with the 11 window and Triton tests skipped. test_affine_transfer_executor.py alone gives your 77 passed and 11 skipped. The pytest line in the body also names tests/unit/checkpoint/test_affine_transfer.py, which doesn't exist, so as written that command stops at "file or directory not found".

1. 16-byte elements in fused_copies. copy_tiles uses int8/int16/int32 for 1-, 2- and 4-byte elements, and int64 for anything else. The table counts columns and strides in elements, so a complex128 region (16 bytes) is copied 8 bytes per element: half its bytes, at the wrong offsets, and nothing raises. _TileTable doesn't send that case to fallback, and test_fused_copies_match_copy_for_every_region_shape covers int8/bf16/fp32/fp64 only. Sending element sizes outside {1, 2, 4, 8} to fallback and adding complex64/complex128 to that test would cover it.

2. Rebasing onto #9093. #9093 goes in first, and #9094 is rebased on it. I have that rebase ready and will push it once #9093 merges. Two things there change what this PR can rely on:

  • _flat refuses conjugate and negative views. The tile kernel and the window copies both move raw bits. On the current base, a neg view of a window shard is contiguous, so _window_offsets publishes it and the peer pulls it without its sign, with either copier. That is the cross-process case [checkpoint] Check aliases on the regions a transfer moves #9093 fixes for the staged path. After the rebase the holder refuses it in the preflight, so nothing else is needed for it.

  • _validate_aliases becomes region-level. It builds its reads and writes from the lane: local copies, send rounds and receive rounds. _lanes leaves window copies out, so after the rebase the check can't see them:

    • on the owner, a source region a peer pulls is not counted as a read, and a target region a peer pushes into is not counted as a write;
    • on the copier, the region it pulls into or pushes from is not counted either.

    Today the buffer-level check refuses any such overlap (along with the relay band). After the rebase it won't. Window copies run on the copier while the owner is still in its rounds, so an overlap would race rather than fail. I'd add every window copy that touches a process to that process's regions, using the same _mapped_direction call as the lanes: pulled targets and pushed targets as writes, pulled sources and pushed sources as reads. A 2-process test with a window source region that overlaps a region written on the same process would pin this down.

Conflicts to expect: _agree now takes copies = _validate_aliases(requests, lane, element_size) if aliased else lane.local_copies and keeps them on _Agreement.copies, _local_copy_views takes copies instead of the lane, and _buffer_layout includes is_conj()/is_neg().

Non-blocking.

  • test_ranks_disagreeing_about_the_window_refuse_everywhere covers two windows with different directions. Passing window= on some processes and None on others hangs instead of refusing, because all_gather_object runs only where a window was passed. A failed symm_mem.empty or rendezvous on one process (for example out of memory) also leaves the others waiting inside TransferWindow.__init__. The docs say every process must pass the same window, so a sentence saying that a mismatch hangs may be enough for now.
  • release_staging says every process must release before the same run. _restage is local, though, and the only cross-process step is the failure bit in _check, so a process that releases on its own looks fine to me. If there is a reason for the rule, a short comment would help. The check=True failure path, where one process can't re-stage and every process raises with nothing written, has no test yet. Patching _restage to raise on the last process would cover it.

A prepared handle decides at prepare which local copies are dropped as in place, so a run must refuse a
buffer set that would change that decision. The layout it compares now carries each buffer's conjugate
and negative bits next to its address, shape and strides, and the docs say that check=False cannot see
such a change.

The relay-band and lazy-view tests run through prepare_transfers as well, and a new test rebuilds the
band at run time (cloned, as a negative view, or as a copy where the handle kept a view) and expects
every process to refuse with the destination untouched.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@jinyouzhi

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Rebased onto the current #9094 (on #9093) and addressed each point in 95e799d:

  1. 16-byte elements. fused_copies now sends any element size outside {1, 2, 4, 8} to copy_, and the kernel asserts ELEMENT == 8 in the last branch. The fused test covers complex64 and complex128; complex128 fails without the fix.
  2. Window copies and region aliases. _validate_aliases also takes the window operations of this process: a pulled/pushed target counts as a write, a pulled/pushed source as a read. New 2-process test: each rank's source shard and a target the peer writes are the same window tensor; every process refuses with "a source and a target overlap" and the shard is unchanged (pull/push × prepared/one-shot). The four cases do not raise without the fix.
  3. Partial window / construction failure. Documented in TransferWindow and in the docs: pass the same window on every process or none.
  4. release_staging. Relaxed the docstring: a process may release on its own; use check=True if re-allocation can fail. Added a test for that.
  5. check=True restage failure. New test: the last rank fails to re-stage, every process raises "could not allocate the released staging", targets are unchanged, and the next run succeeds.
  6. PR body. Fixed the pytest line.

CPU (gloo): 94 passed, 27 skipped. 4× B70 (xccl): all pass at 2 and 4 processes.

prepare_transfers takes a copier that turns each round's pack/unpack list
and the local copy list into one callable. sequential_copies (default)
keeps one copy_ per region; fused_copies builds a per-alignment tile
table at prepare time and moves the list with one Triton launch per
class on CUDA/XPU, falling back to copy_ elsewhere.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
symm_mem_rendezvous allocates a buffer every member of a group can address
and exchanges its handles; symm_mem_peer_tensor maps one peer's buffer into
this process. On backends whose peer views carry no device index (XPU),
the view is rebuilt from the mapped address.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…runs

TransferWindow is a symmetric-memory pool every process of the group can address. An operation between two processes whose source or target lies in a window is copied in place by one of them (the target pulls or the source pushes) with no message, arena or budget; the rest of the batch keeps the staged protocol. Ranks agree on which shards lie in a window as part of the schedule digest, and a run with window copies ends with a group barrier.

Window copies are (destination, source) pairs like a round's packs and unpacks, so the same copier builds them: fused_copies covers copies whose source or destination lies in a peer's mapped window.

PreparedTransfer.release_staging() frees the arenas between runs; the next run allocates them again, and a process that cannot refuses on every process when run checks.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…s off the tile kernel

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants