Skip to content

Describe a parameter held in several layouts as one affine map - #9052

Open
Achyuthan-S wants to merge 3 commits into
deepspeedai:masterfrom
Achyuthan-S:union-affine-maps
Open

Achyuthan-S wants to merge 3 commits into
deepspeedai:masterfrom
Achyuthan-S:union-affine-maps

Conversation

@Achyuthan-S

@Achyuthan-S Achyuthan-S commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Refs #9026 (open question 8), #9045.

Why

A rollout replica that changes placement can often keep a tensor-parallel shard that is still current, while the trainer's ZeRO-3 partitions hold the whole parameter. plan_transfer can read a block from any of its holders only if the source map's blocks are equal or disjoint and each block lists every holder. A map that simply lists both layouts' pieces breaks both rules, since a partition can start or end inside a row of the shard. So today the planner refuses it, and #9045 falls back to its own planner for these weights.

What

  • union_affine_maps(maps) in deepspeed/checkpoint/affine.py returns one map of a parameter that several layouts hold at once.
    • Each piece is cut at the edges of every block held over it, in any layout (replicas within one layout may store a region whole or in parts), and each cut piece lists the holders from every layout.
    • A cut keeps the piece's strides and moves its offsets, so every location still stores exactly what it did.
    • Layouts use disjoint locations, one per buffer. TransferEndpoints already lets several locations map to one process.
    • More than two layouts are merged pairwise.
    • Blocks are taken in order of their first row, and each is compared only with the parts that still reach into its rows; a piece is cut against only the parts that start in its rows. That keeps the work close to linear in the number of partitions (see Cost).
    • Layouts that contradict each other raise InvalidMapError, and a piece that is not an axis-aligned block raises UnsupportedLayoutError: the planner's types for the same problems, so a caller that falls back on AffineTransferError catches these too.
  • subtract moves unchanged from affine_transfer._subtract into affine_geometry, the module [AffineIR] Move shared affine index and box geometry into affine_geometry #9010 added for shared box arithmetic. The planner imports it under its old name.
  • Spec §10.8 and a section in affine-transfer.rst, after the replica-policy section from [checkpoint] Batch replica selection for affine transfer plans #8936.

Scope

Cuts along rows are planned today. A ZeRO-3 block of several rows cut at a column edge keeps its row-major strides and its partition's offset, so it is not dense in its flat shard, and plan_transfer refuses it. @0z5a's #9070 makes the planner accept that case (see #9026). The union already describes it exactly, and the tests check that at the map level.

Cost

@jinyouzhi measured the first version growing with the square of the number of ZeRO-3 partitions. union_affine_maps on a (4096, 4096) parameter's ZeRO-3 partitions plus another layout, macOS arm64 CPU (before = the previous version):

Other layout 200 partitions 1000 partitions
8-way row split 1.93 s → 0.03 s 48.0 s → 0.18 s
8-way column split 11.0 s → 0.26 s 1.3 s (34k pieces)
whole replica 1.91 s → 0.03 s 0.18 s

A column split is the slowest case left, because each of its blocks spans every row.

Tests

test_affine_transfer_plan.py:

  • A surviving row shard plus ZeRO-3 partitions, in 4 configurations (uneven TP sizes, one or two surviving ranks, 1 to 5 partitions), planned into a row-split and a column-split target. Checked:
    • each element's holders are exactly the union of both layouts' holders;
    • every location extracts what its own layout says;
    • every plan reproduces extract exactly;
    • the surviving shard is read.
  • Three layouts merged in all six orders: the same holders, and exact plans.
  • A column cut: holders and exactness at the map level.
  • A layout whose replicas store a block whole and in two halves, merged with ZeRO-3 partitions: holders, and an exact plan into a column-split target.
  • Refusals, with the planner's error types: no maps, a location in both layouts, different shapes, a transposed piece that is not a block, and a row-major run that wraps into the next row.

test_affine_transfer_executor.py: on 2 ranks, one process holds a ZeRO-3 partition and a surviving row shard under two locations. The plan runs through execute_transfer, and every target shard equals extract.

Not merging the holders, dropping the part of a block that a later block doesn't overlap, or leaving the destination offset unmoved makes 7, 8 and 7 of these 9 tests fail. Setting a part aside one row too early, or ending the search for the parts under a piece one row short, makes 5 of them fail.

Validation

Rebased on master 97aa3c1, after #9039 renamed the geometry imports. The only conflicts were those imports and affine_geometry.__all__; subtract now sits with master's unaliased imports. macOS arm64 CPU, Python 3.9.6, torch 2.8.0, gloo (DS_ACCELERATOR=cpu LOCAL_SIZE=4):

  • All nine affine test files: 933 passed (master 97aa3c1: 924 passed).
  • yapf 0.43.0 and flake8 5.0.4 pass on the changed files.
  • Against the previous version, on 6,000 random unions of two or three layouts (ZeRO-3 partitions, row and column splits, replicas, any subset of surviving shards): identical pieces, the same holders for every element, the same shard at every location, and the same refusals.

The runs below are from before this rebase and the cost change.

Earlier, rebased on master 892ed83, which includes #9011. The code and test changes are the same as before the rebase; the only conflicts were additions next to #9011's (subtract next to boxes_disjoint in affine_geometry.py, and the imports). macOS arm64 CPU, Python 3.9.6, torch 2.8.0, gloo (DS_ACCELERATOR=cpu LOCAL_SIZE=4):

  • All eight affine test files: 715 passed (master 892ed83: 706 passed).
  • yapf and flake8 pass on the files the rebase touched.

On master 988b2e3, before that rebase (with #9010 and #8936):

  • All eight affine test files, then including [checkpoint] Batch replica selection for affine transfer plans #8936's test_affine_replica_policy.py: 412 passed.
  • The four union configurations above, planned through [checkpoint] Batch replica selection for affine transfer plans #8936's plan_transfers with local_first_least_loaded and with first_holder into both targets: all 16 plans reproduce extract exactly.
  • Against the previous commit (110ff4b, before the two fixes from review), on random layouts: 300 unions with a row-split surviving shard, 300 with a column-split one, and 300 of three or four layouts. Every element has the same holders, and every plan into a row- or column-split target has the same outcome as before: exact, except 82 of the column-split cases, which both commits refuse (the column cut under Scope).
  • pre-commit passes on the changed files.

Before the rebase, on f9fe4df with #9010 applied:

  • All seven affine test files: 346 passed.
  • With [AffineIR] Check affine holders and source regions without pairwise comparisons #9011 on top: 648 passed. The only conflicts were adjacent additions: import lines and appended definitions.
  • 300 random two-layout cases (uneven TP sizes, any subset of surviving ranks, 1 to 6 partitions):
    • holders and map-level exactness hold in every case;
    • cuts along rows: 137 of 137 planned and exact;
    • cuts along columns: 79 planned and exact, and 84 refused by the current planner.
  • 300 random unions of three or four layouts, merged in random order: all planned and exact.

On GPU, also before the rebase: 4× A100-SXM4-40GB (NVLink), driver 595.71.05, torch 2.12.1+cu130, NCCL 2.29.7, Python 3.12.11, with pytest --forked over six of the affine test files (all but test_autoep_affine_universal.py):

  • master (8a6b53f): 317 passed, 1 skipped;
  • this branch: 325 passed, 1 skipped, so the 8 new tests also pass over NCCL, including the executor test.
  • The one skip is the external-group test that runs only on the CPU accelerator.

cc @0z5a @LiRunGuo @jinyouzhi

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T20:08:36.647260Z 110ff4b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 110ff4b8b3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread deepspeed/checkpoint/affine.py Outdated
Comment on lines +1040 to +1043
if len(piece.shape) == len(logical_shape) and all(
size <= 1 or stride == expected
for size, stride, expected in zip(piece.shape, piece.source_strides, strides)):
origin = _coordinate(piece.source_offset, strides, logical_shape)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject blocks that cross logical-axis bounds

When an otherwise executable piece wraps across a logical-axis boundary (for example shape=(1, 4), source_offset=0, and source_strides=(3, 1) for a (2, 3) parameter), matching row-major strides and a valid starting coordinate make this branch classify it as axis-aligned even though origin + shape exceeds the logical shape. _cut then models offset 3 as nonexistent coordinate (0, 3) rather than (1, 0), and the returned union advertises different locations for the two real copies; both validate_coverage() and plan_transfer() reject it. Check that LogicalBox(origin, piece.shape) is contained in the logical parameter before accepting it.

Useful? React with 👍 / 👎.

@Achyuthan-S Achyuthan-S Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed: for a (2, 3) parameter the (1, 4) run from offset 0 was cut as the box (0, 0)+(1, 4), and the union came back with a piece that validate_coverage and plan_transfer then rejected. _block_of now refuses a block that doesn't fit inside the parameter, the same way it refuses other pieces that aren't blocks, and test_a_union_refuses_layouts_it_cannot_merge covers it. Fixed in e97f115 (45652c6 before the rebase onto master).

Comment thread deepspeed/checkpoint/affine.py Outdated
Comment on lines +1058 to +1061
for other, holders in blocks:
overlap = block.intersection(other)
if not overlap.is_empty:
parts.append((overlap, piece.locations | holders))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Refine replica boundaries within each input map

When one valid input has replicas with different piece boundaries—for example location 0 stores one (2, 3) box while location 1 stores two (1, 3) boxes in swapped shard rows, all declaring {0, 1}—_blocks_held contains a full box plus its halves. This loop appends overlaps against the original block rather than a common disjoint refinement, so unioning a third full-copy layout gives that location the full box and both halves; affine_map.validate() then raises because 12 elements are assigned to a 6-element shard. Include boundaries within each input map when building the disjoint refinement before emitting pieces.

Useful? React with 👍 / 👎.

@Achyuthan-S Achyuthan-S Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed: that input made the union raise "Rank 10 holds a shard of 6 elements but its pieces account for 12". Every piece is now cut into the disjoint refinement of all the blocks both layouts hold, including one layout's own replica edges, so each part appears once and lists every holder. test_a_union_cuts_replicas_that_split_a_block_differently covers it. On 900 random layouts the holders match the previous commit for every element, and every plan is exact or refused exactly as before. Fixed in e97f115 (45652c6 before the rebase onto master).

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 14, 2026
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 16, 2026

@jinyouzhi jinyouzhi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cutting looks right to me, and checking holders and extract against each layout separately is a good oracle. The new tests pass locally (CPU). Two things:

1. Union cost grows quadratically with the number of partitions. _disjoint_blocks intersects every block with every part found so far, and _cut intersects every piece with every part. A (4096, 4096) parameter, ZeRO-3 partitions plus an 8-way row split:

partitions pieces union_affine_maps
25 160 0.02 s
50 308 0.08 s
100 604 0.29 s
200 1196 1.12 s

At 200 partitions, about 2/3 of the time is in _cut and 1/3 in _disjoint_blocks, nearly all of it in LogicalBox.intersection (545k calls). Extrapolating, 1000 partitions is about 28 s for one parameter; a run at that size had not finished after 15 minutes. With a few hundred parameters per model and a data-parallel size in the hundreds, the union alone would take minutes, and it isn't covered by PlanBudget. Since the parts are already sorted, an interval index on axis 0 (bisect on the row range, then intersect only the candidates) should make both loops close to linear for the row/flat layouts this targets. A timing check at a few hundred partitions would also be worth adding to the PR description.

script
import time
from deepspeed.checkpoint.affine import contiguous_split_map, nest_affine_maps, replicated_map, union_affine_maps
from deepspeed.checkpoint.zero_affine import zero_storage_affine_map

shape = (4096, 4096)
numel = shape[0] * shape[1]
for parts in (25, 50, 100, 200):
    zero = nest_affine_maps(replicated_map(shape, 1),
                            {0: zero_storage_affine_map(numel, list(range(100000, 100000 + parts)), -(-numel // parts))})
    shard = contiguous_split_map(shape, [512] * 8, 0)
    start = time.time()
    union = union_affine_maps([zero, shard])
    print(parts, sum(len(p) for p in union.pieces_by_rank.values()), f"{time.time() - start:.2f}s")

2. Error types. A piece that isn't an axis-aligned block raises NotImplementedError, and mismatched inputs raise ValueError, while the planner reports the same kinds of problems as UnsupportedLayoutError / InvalidMapError (both AffineTransferError). A caller such as #9045 that falls back to its own planner on AffineTransferError would not catch these. Using the planner's error types would keep that one except. Minor; #9045 doesn't call this yet.

@delock
delock self-requested a review November 18, 2026 15:05
A rollout replica can keep a tensor-parallel shard that is still current while the trainer's ZeRO-3 partitions hold the whole parameter. union_affine_maps cuts each layout's pieces at the edges of the blocks the other layouts hold and lists every holder, so plan_transfer can read either copy. Move the planner's box subtraction into affine_geometry so both use one copy.

Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
Replicas within one layout may store a region whole or in parts, so cutting a piece only at the other layout's blocks could place the same elements twice in one shard. Cut every piece into the disjoint refinement of all the blocks both layouts hold, each part listing every holder. A row-major run longer than what is left of its row is not a block, so refuse it instead of cutting it as one.

Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
…'s errors

A union compared every block with every part found so far, and cut every
piece against every part, so its time grew with the square of the number
of ZeRO-3 partitions: about 48 s at 1000 partitions of a 4096 x 4096
parameter. The blocks come in order of their first row, so a part that
ends above a block is final, and each block now meets only the parts that
reach into its rows. The parts refine every block, so a piece is cut
against only the parts that start in its rows. The pieces are the same.

Layouts that contradict each other now raise InvalidMapError, and a piece
that is not a block UnsupportedLayoutError, so a caller that falls back
on AffineTransferError also catches a union that cannot be formed.

Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
@Achyuthan-S

Copy link
Copy Markdown
Contributor Author

Thanks @jinyouzhi, both points were right, and your script made the first one easy to reproduce.

1. Union cost. Fixed in 4a6bd3d.

  • The blocks were already sorted by their first row. So _disjoint_blocks now sets a part aside once it ends above the current block's first row, since it can't meet that block or any later one. Each block is compared only with the parts that still reach into its rows.
  • The finished parts refine every block a layout holds, so the parts covering a piece are exactly those that start in its rows. _cut finds them with a bisect on their first rows, as you suggested.

union_affine_maps on a (4096, 4096) parameter's ZeRO-3 partitions plus another layout (macOS arm64, CPU):

Other layout Partitions Before After
8-way row split (your script) 200 1.93 s 0.03 s
1000 48.0 s 0.18 s
8-way column split 200 11.0 s 0.26 s
1000 not run 1.3 s
whole replica 200 1.91 s 0.03 s
1000 not run 0.18 s

The column split is the slowest case left, because each of its blocks spans every row. At 1000 partitions the union has 34k pieces.

I compared the new code against the previous commit on 6,000 random unions of two or three layouts: ZeRO-3 partitions, row and column splits and replicas, with any subset of shards surviving. Both give identical pieces, the same holders for every element and the same shard at every location, and they refuse the same inputs. The timing is in the description now.

2. Error types. Agreed. Now:

  • layouts that contradict each other raise InvalidMapError;
  • a piece that isn't a block raises UnsupportedLayoutError;
  • a union that fails its own validation, which takes an inconsistent input map, raises InvalidMapError from the original error.

Both subclass ValueError, so except ValueError still works. An empty list still raises a plain ValueError, since that's a caller mistake rather than a layout. test_a_union_refuses_layouts_it_cannot_merge checks the new types.

I also rebased onto master 97aa3c1, after #9039. The only conflicts were the geometry imports: subtract now sits with master's unaliased imports. The affine test files give 933 passed, against 924 on master.

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants