Skip to content

[AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) - #8966

Merged
delock merged 13 commits into
deepspeedai:masterfrom
jinyouzhi:affine-link-calibration
Nov 17, 2026
Merged

delock merged 13 commits into
deepspeedai:masterfrom
jinyouzhi:affine-link-calibration

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Calibrate affine transfer links per path

Stacked on #8809 only for its docs page (affine-transfer.rst); the module itself needs nothing beyond master. Until #8809 merges, please review only the commits after the #8809 merge.

Why

A replica policy or cost model choosing between plans (#8252, #8935/#8936) needs link costs measured on the machine it runs on. Executor measurements show that round count and receiver sharing dominate sync time, so per-pair numbers are not enough on their own; this PR also measures 1→N and N→1.

What

deepspeed/checkpoint/affine_transfer_calibration.py:

  • calibrate_links(sizes, repeats, warmup, path_of, fan_width, fan_peers, group): collective over group (the default group unless given). Returns the same raw samples on every process, with no aggregation, for:

    • local copy;
    • peer (each ordered pair alone);
    • fan_out (1→N);
    • fan_in (N→1, receiver contention).

    Sizes run from 4 KiB to 256 MiB by default.

  • Each message is labelled by path_of(from_process, to_process). default_path_of gives local/peer; node_path_of(n) gives intra_node/inter_node; any callable works.

  • fit_links(calibration): per-path alpha + bytes / bandwidth, fitted to the per-size medians; peer_pairs per ordered pair; fans keyed by (root, peers), with send/receive giving each root's aggregate rate.

Weight transfers are one-way (a training process sends, a rollout process receives), so no pattern has a process send and receive at once.

It measures only. It does not plan, select replicas, or change the planner or executor.

Example (one node, 8 devices, default sizes)

pattern B70 xccl: alpha (µs) / GB/s L20 nccl (PCIe): alpha (µs) / GB/s H20 nccl (NVLink): alpha (µs) / GB/s
local 73 / 359 16 / 1534* 9 / 1881
peer 301 / 28.2 72 / 25.3 54 / 275
fan_out (7 peers) 531 / 3.4 141 / 3.6 88 / 49.6
fan_in (7 peers) 670 / 2.6 56 / 3.8 46 / 54.5
run time 88 s 34 s 13 s

* poor linear fit (max relative error 0.77); small copies are cache-resident.

Multi-node (2 × 8 H20, RoCE 8 × 200 Gb/s per node, node_path_of(8), 16 processes)

pattern alpha (µs) / GB/s max rel. error
peer, intra_node 57 / 53.8 0.07
peer, inter_node 97 / 16.6 (pairs 14.0–24.5) 0.50
fan_in, 7 intra-node peers: root receive 94.6 GB/s
fan_out, 8 cross-node peers: root send 62.1 GB/s median (33–78 over roots)
run time 72 s (+111 s for cross-node fans of width 1/4/8)
  • The same NVLink pair measured 275 GB/s in the 8-process group and 54 GB/s in the 16-process group over RoCE (105 GB/s with NCCL over TCP), so calibrate on the group and communication settings the transfer uses.

Tests

tests/unit/checkpoint/test_affine_transfer_calibration.py:

  • fit on synthetic data with known link costs, per-pair and per-peer-set fits;
  • schema and argument checks;
  • 4-rank runs checking completeness, sample counts, and fan path direction with an asymmetric path_of.

10 passed on 8× B70 (xccl). pytest tests/unit/checkpoint/ -k affine: 344 passed, 1 skipped on 8× H20 with torch 2.13.0+cu129 / NCCL 2.29.7 and with torch 2.9.1+cu129 / NCCL 2.29.7. Mutation check: flipping the fan_in label direction fails a test. Cross-node runs used NCCL 2.29.7 on both nodes (torch 2.13.0 and 2.9.1).

AI assistance (GitHub Copilot) was used for the multi-node runs and this description; I reviewed the changes and the results.

Follow-ups

@jinyouzhi jinyouzhi changed the title Calibrate affine transfer links per path, including fan-out and fan-in [AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) Oct 31, 2026
@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks, this is the input the cost model needs. I'll build the cost-aware policy on #8936 from fit_links here and analyze_transfers in #8967, rather than measuring separately. Two things the receiver side of the model needs from the fan-in data:

  1. Fan-in at several widths. The policy caps how many senders feed one receiver at once, so it needs the cost as a function of the number of concurrent senders, not only at world - 1. Could fan_width take a list, say (1, 2, 4, world - 1), with the fan fits keyed by width? The fans are cheap next to the pair sweep.
  2. The root's time, fitted. For a fan-in, the finish time is when the root has all k messages: root_seconds against k * bytes. That is the receiver's ingest bandwidth, and it currently stays in the raw result. A fitted entry beside the per-peer one would save every consumer from refitting it.

Your B70 numbers already show why the cap matters. If I read the per-link fit right, seven senders give about 7 × 2.6 ≈ 18 GB/s into one receiver, which is less than one pair alone (28.2 GB/s).

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S Yes, you're very observant—I agree. Based on these experiments, we can refine some of our policies.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks! If it's easier, I can send both as a small commit to your branch: fan_width accepting a list, with the fan fits keyed by width, and a fitted entry for the root's time. One refinement since: @0z5a measured flat incast on GH200 over Slingshot, so the policy should read the receive cap from the fit at each width rather than assume one.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S Sure, I have added you as my fork's collabration or I also can cherry-pick a commit provided by you. Thanks

@0z5a 0z5a left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the last commit (3cfa4153), as asked. The measurement design is careful: the barrier and synchronize brackets keep collectives out of timed regions, the pair sweep isolates each link, and the fans separate sender and receiver sharing. On CPU (gloo, 4 ranks), the calibration tests pass together with #8967's and the executor's (62).

A few thoughts, mostly about feeding a cost model directly:

  • Per-process send and receive capacity. A model of the form t = max over processes of max(sent / send_capacity, received / receive_capacity) needs each process's aggregate send and receive capacity. fit_links gives per-path link costs; the fan fits use the peers' times, and the roots' times, which carry the aggregate, stay only in the raw result. Computing bytes * len(peers) / root_seconds from the raw fan entries is a few lines. On CPU it gives a per-root figure for each direction. Having fit_links fit those per process (or per root) as send and receive would let #8967's ProcessReport.sent_bytes / received_bytes be priced directly.
  • Fan peers. The peers are root + 1 … root + fan_width, so with fan_width below the processes per node every fan stays within a node. If receive capacity differs across nodes, a way to choose the peers (a fan_peers(root) hook, or a stride) would let the fans cover it.
  • Defaults at scale. With the default fan_width = world - 1, each process can allocate (1 + world) * max(sizes), about 16 GiB at 64 ranks and 256 MiB, and the pair sweep is O(world²) barriers per size. The docstring says so, but a default capped at the processes per node, or a check against free memory, might save someone a surprise.
  • Process group. Calibration always uses the default group, while the executor runs on whatever group TransferEndpoints names. An optional group argument would let it measure the group the executor will actually use, e.g. an external trainer–rollout group, with the same rank numbering the analysis report uses.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S @0z5a Thanks both. I agree with the direction: a list of fan widths and a fitted root time (per-process send / receive). Before we key the fits by width alone, here is fan-in data at four widths on 8x B70 (PCIe, xccl, 256 MiB, median of 10). It shows that on a PCIe tree, which senders feed a receiver matters more than how many.

Topology. Four PCIe switches, each with two cards behind one Gen5 x16 root port, and two switches per socket: {0,1} {2,3} on socket 0 and {4,5} {6,7} on socket 1 (assuming ranks follow PCI address order). GH200 over Slingshot is a Clos, so every sender set looks alike there, which fits your flat incast.

Pair bandwidth, GB/s (row = sender, column = receiver):

from\to   0     1     2     3  |  4     5     6     7
0         -    8.3   7.9   8.3 | 27.3  27.5  27.6  27.8
1        6.0    -    5.5   3.6 | 27.4  27.5  27.4  27.5
2       53.9  53.7    -   53.6 | 14.5  14.6  14.6  14.6
3       53.6  53.5  53.8    -  | 14.6  14.6  14.6  14.6
4       27.4  27.3  27.4  27.3 |  -   14.4  14.3  14.4
5       27.3  27.3  27.4  27.4 | 25.2   -   25.3  25.3
6       27.6  27.7  27.0  27.3 |  8.8   8.7   -    8.6
7       27.5  27.8  27.3  27.4 | 42.8  42.9  43.0   -

These are stable across runs. Cross-socket traffic sits near 27 GB/s. Within a socket, bandwidth depends on the sender and the direction (2→0 is 54, 0→2 is 7.9, on the same switches), which the tree alone does not predict.

Fan-in root aggregate (k * bytes / root_seconds), GB/s, by width:

width min ~ max over roots spread
1 7.0 ~ 53.9 7.7x
2 12.6 ~ 55.2 4.4x
4 18.7 ~ 37.3 2.0x
7 16.4 ~ 23.6 1.4x

Three cases explain most of it:

  • 0 <- [1, 2] = 12.6: the root waits for the slowest sender (1→0 at 6.0), so the aggregate is about 2 × 6.
  • 1 <- [2, 3] = 55.2: two 54 GB/s senders reach only about 55 together, so a receive-side limit binds.
  • 3 <- [4, 5, 6, 7] = 27.9: four 27 GB/s senders together still get about 28, because the shared bottleneck is the cross-socket link, not the receiver.

So the finish time looks like max(slowest sender's bytes / pair bandwidth, bytes through a shared link / that link's capacity). Both terms depend on the sender set. A single fit per width would average 12.6 and 55.2 at width 2, and a single receive capacity per process would miss the shared cross-socket link.

Proposal for this PR:

  1. Width list and root fit: keep both as you describe. They are right for Clos and other uniform fabrics.
  2. Per-pair fit: add an optional per-pair entry in fit_links, keyed by (from, to) and fitted from the existing pair sweep, so no new measurement. It captures sender and direction asymmetry, which a path-class fit hides (the 28 GB/s peer fit here is the median of 3.6–54).
  3. Choosing fan peers: add a fan_peers(root, width) hook (0z5a's suggestion). The fan fits are keyed by the peer set, or by the peers' path classes, not by width alone. That way a fan can target "same switch", "same socket" or "cross socket" senders and measure the shared-link capacity directly. The raw entries already record peers.
  4. Docs: state that width-keyed fits assume a uniform fabric, and include this table as the counterexample.

On 0z5a's other points:

  • I'll add the optional group argument, with ranks numbered as in the analysis report.
  • I'll cap the default fan_width at the processes per node and document the memory needed.

@Achyuthan-S Please push your width-list and root-fit commit. I'll add the per-pair fit and the fan_peers hook on top so they don't conflict. Raw JSON for all four widths is available if you want to check the fits.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Pushed the width-list and root-fit commit:

  • fan_width takes one width or a list. Each fan sample records its width, and the result lists fan_widths.
  • fit_links keys the fan fits by width, and adds send and receive: per width, a fit per process of the root's own time against all the bytes it moved. That is the aggregate capacity @0z5a described, so [AffineIR] Analyze affine transfer batches before execution (Plan analysis & calibration Part-1) #8967's sent_bytes / received_bytes can be priced against it directly.
  • Docs and tests are updated: a synthetic test that recovers per-process capacities at two widths, and the 4-rank tests cover a list of widths. On CPU (gloo, 4 ranks), the calibration, executor and planner tests pass together (180).

Your B70 table is a great counterexample. It fits the same form once every shared resource is a term, not only the endpoints: a fan finishes when the busiest resource it crosses does, i.e. the max over each sender's link, the receiver, and any shared link such as your cross-socket hop, of the bytes through it over its capacity. Your three cases are one of each:

  • 0 <- [1, 2] is bound by the 1→0 link;
  • 1 <- [2, 3] by the receiver;
  • 3 <- [4..7] by the cross-socket link.

So your per-pair fit and fan_peers are what calibrate those resources. I kept the width as the only key here so your peer-set keys can replace it without a conflict.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S @0z5a Pushed ba8b39f (on top of Achyuthan's commit, with #8809's 11fe945 merged in).

Updates

  • Fan fits and send/receive are keyed by (root, tuple(sorted(peers))) instead of width, as Achyuthan anticipated; width is len(peers), and the width list and root fit stay.
  • fits['peer_pairs'][(src, dst)]: one fit per direction.
  • fan_peers(root, width): choose which peers form each fan.
  • group: calibrate inside a process group; ranks match [AffineIR] Analyze affine transfer batches before execution (Plan analysis & calibration Part-1) #8967's report.
  • Default width is capped at processes-per-node minus one, same-host peers first; docs note the buffer memory.
  • Docs: uniform-fabric note, the B70 table, and the busiest-resource model.

On the B70 counterexample: send/receive are the completion rate of a fan with those peers, bound by the busiest resource, not the root's capacity. Per-resource capacities are left to the #8936 policy, built from peer_pairs plus targeted fan_peers fans.

Validation: 8× L20 (driver 580.173.02, torch 2.14.1+cu130): calibration tests on 4 GPUs, 10 passed; external-group test on CPU passed 20 of 20 runs; pre-commit passed. Multi-node not validated.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks, this is the right split. Keying by peer set captures exactly what the B70 table showed, and I agree on the semantics: send/receive are the completion rate of a given fan, and the per-resource capacities belong in the policy. I'll derive them in the #8936 policy from peer_pairs plus targeted fan_peers fans, and check the result against your B70 table and 0z5a's GH200 and B300 runs.

One small thing for consumers: the fits now have tuple keys (peer_pairs[(src, dst)], fans by (root, peers)), so json.dumps(fits) raises. The raw calibration still serializes, so caching that and refitting on load works. A line in the docstring saying so would save someone the surprise.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

Good catch. I added a note to the fit_links() docstring: the returned fits contain tuple keys and are not directly JSON-serializable; callers should persist the raw calibrate_links() result and call fit_links() again after loading. The doc update is in my local worktree and will be included in the next signed-off PR update.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks for the JSON note and the H20 multi-node numbers. The same NVLink pair measuring 275 GB/s in one group and 54 GB/s in another is a useful warning, and I'll make "calibrate in the transfer's own group" explicit on the policy side.

One concern with dropping the concurrent pattern: some transfers do have processes sending and receiving at once.

  • Colocated resharding: on the same GPUs, ZeRO-3 → AutoTP, each GPU sends its partition and receives its TP shard.
  • Stage 2 of a relay: the engines on the target node all-gather, so each one sends and receives.
  • Pipelined relay: the relay GPUs receive stage 1 while they send stage 2.
  • Moves between trainers of different DP sizes.

@0z5a's B300 runs show what this costs: 618 GB/s per direction duplex, against 664 GB/s one-way. On GH200 the overlap held because the NIC's receive and the NVLink send are separate resources. So the policy needs a duplex number per path.

The shift pattern's labelling was the problem, not the measurement, and a simpler pattern would do: for example, each ordered pair exchanging both ways at once, labelled by the pair's path in each direction. If you'd rather keep this PR one-way, a line in the docs saying duplex isn't measured, plus a follow-up, works too.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S Thanks. I had assumed weight_sync in an RL loop is always one-way, so I missed the cases where a process sends and receives at the same time. Colocated resharding, relay stage 2 and the pipelined relay are all real ones, and the B300 duplex vs. one-way gap shows the policy needs this number. I'll bring the concurrent pattern back in this PR and add the duplex.

I agree the labelling was the problem. The old shift pattern made it unclear which participant each sample described.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S Pushed d00ba30, following your suggestion:

  • peer both ways: peer samples now carry both_ways. A two-way sample has a pair send to each other at once while every other process is idle, and gives one sample per direction, labelled with that direction's path. peer / peer_pairs stay one-way. peer_both_ways / peer_both_ways_pairs hold the two-way fits.
  • relay (the old shift pattern, relabelled): every process receives from p - shift while sending to p + shift. Each sample is one process's own time, labelled with its from_process / to_process and both receive_path and send_path. fit_links keys relay by (receive_path, send_path), so nothing is filed under a path it only half used.

H20, one node, 4 GPUs over NVLink, 256 MiB, median of 10, GB/s per direction: one-way 331–335, both ways 276–287 (0.83–0.86×), relay 282–287 (≈0.85×). Calibration tests pass on 4 GPUs (12 passed). Cross-node relay is not measured yet.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks @jinyouzhi, d00ba30 covers all four cases. This is how the #8936 policy will read it:

  • colocated resharding and the all-gather stage of a relay: peer_both_ways / peer_both_ways_pairs, per direction;
  • the relay GPUs of a pipelined relay: relay[(inter_node, intra_node)], once a cross-node relay is measured;
  • one-way moves stay on peer / peer_pairs.

Your H20 numbers (both ways at 0.83–0.86× one-way) and @0z5a's B300 (618 vs 664 GB/s, 0.93×) show the same thing: the cost is real and depends on the machine, so it has to be calibrated. On CPU (gloo, 4 ranks) the calibration tests pass at d00ba30 (12 passed).

One small thing: the raw format gained both_ways and relay, but CALIBRATION_SCHEMA is still 1. A raw calibration saved before this commit, such as the earlier B70/H20 JSON, passes the schema check, and then fit_links raises KeyError: 'both_ways'. Since the docstring now recommends caching the raw result and refitting on load, either bump the schema to 2 or read old samples with sample.get('both_ways', False) and calibration.get('relay', []). Either works for me.

When you get a multi-node slot, the (inter_node, intra_node) relay is the number the pipelined relay needs. No rush.

@0z5a

0z5a commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Re-read up to d00ba303. One fix needed, plus one wording point.

  • The external-group executor test fails on this head. It asserts that calibration['peer'] holds exactly (0, 1) and (1, 0). Since d00ba303 the list also carries the two both_ways samples, so the worker's assertion fails on CPU (gloo), and the CPU lane will hit it. The calibration tests themselves pass (12). Keeping the assertion to one-way samples fixes it (passes here):

        assert sorted((sample['from_process'], sample['to_process']) for sample in calibration['peer']
                      if not sample['both_ways']) == [(0, 1), (1, 0)]
  • peer_both_ways is a pair time, not a per-direction time. Both directions of a two-way sample get the same time, the max over the two endpoints, so peer_both_ways_pairs[(a, b)] always equals peer_both_ways_pairs[(b, a)]. On an asymmetric tree like your B70 table (2→0 at 54 GB/s, 0→2 at 7.9 GB/s), the fast direction inherits the slow one's time. The fit_links docstring calls it the per-direction cost while the reverse direction is busy. Could it say instead that it is the pair's completion time, so the duplex loss should be read against the slower of the two one-way fits?

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Agreed with @0z5a on both points. For the #8936 policy, I'll read peer_both_ways as the pair's completion time and compare it with the slower of the two one-way fits, rather than per direction as I wrote above.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

Thanks @0z5a, both fixed in 57a47ff: the external-group assertion keeps to one-way samples, and the docs now describe peer_both_ways as the pair's completion time, to be read against the slower one-way fit.

@jinyouzhi
jinyouzhi force-pushed the affine-link-calibration branch from 57a47ff to 851ea68 Compare November 11, 2026 10:11
@jinyouzhi
jinyouzhi marked this pull request as ready for review November 11, 2026 10:28
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T10:36:21.628251Z 851ea68 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Thanks @jinyouzhi, d465334 covers both Codex points, and @0z5a's two are in as well. One from my earlier comment is still open at d465334: CALIBRATION_SCHEMA is still 1, while fit_links now reads sample['both_ways'] and calibration['relay'] directly. A raw calibration saved before d00ba30 still passes the schema check and then raises KeyError: 'both_ways', and the docstring now recommends caching the raw result and refitting on load. Bumping the schema to 2, or reading old samples with .get('both_ways', False) and .get('relay', []), fixes it. LGTM otherwise.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S Fixed in f678c9d: CALIBRATION_SCHEMA is now 2, so a raw calibration saved before two-way and relay samples is refused with a clear error rather than misread. A test feeds fit_links such a legacy result.

fit_links reads peer samples' both_ways and the raw relay list. A raw
calibration saved before those existed still claimed schema 1, so it
passed the schema check and then raised KeyError. Schema 2 makes
fit_links refuse it with a clear error instead.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026
banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Nov 15, 2026
deepspeedai#9027)

## Why

A replica policy (deepspeedai#8936) or cost model (deepspeedai#8252) needs to know what a
batch will cost before running it: messages per peer, rounds, and
staging memory. deepspeedai#8967 builds that report today by importing the
executor's private `_packed_rounds` and `build_operations`. It then
re-derives the in-place decision and the arena sizing, so its numbers
would drift silently whenever the executor's staging changes. It asked
for a public API instead.

## What

`schedule_transfers(plans, endpoints, budget, element_size,
process=None)` in `affine_transfer_executor` returns, for one process or
for every process the endpoints name, the `ProcessSchedule` that
`execute_transfers` would run. It uses no buffers and no communication:
- the rounds per peer and direction (`ScheduledRound`: elements, chunks,
and whether the round is sent or received in place);
- the staging arena each peer direction allocates;
- the local copies;
- the round count that `TransferStats.rounds` reports.

`execute_transfers` now builds its rounds and arenas from the same
private computation, so the report and the run cannot disagree. The
in-place decision is made from geometry rather than from the buffer.
`_region` is an `as_strided` view of a packed shard, so it is contiguous
exactly when `(chunk_shape, strides)` is row-major packed, ignoring
extent-1 axes.

Unchanged:
- `plan_digest`, the batch fingerprint and `TransferStats`, so mixed
versions still agree on the schedule;
- `build_operations` and `_packed_rounds`, which keep their names and
signatures until deepspeedai#8967 switches.

## Tests

- `test_schedule_is_what_the_executor_runs` (2 and 4 ranks; mixed packed
and strided regions with 16-element messages) checks:
- rounds, sent, received and local-copy bytes against the executor's
`TransferStats`;
  - that both sides of every message agree;
  - the arena sizes;
  - that the batch exercises both the in-place and the staged path.
- `test_packed_regions_are_exactly_the_contiguous_views` compares the
geometric rule with `Tensor.is_contiguous` over 2000 random shapes and
strides, including extent-1 axes.
- A single-process case covers local-only batches and the argument
checks.
- Forcing every round to be staged, or dropping the extent-1 exception,
makes these tests fail.

On macOS arm64 CPU (Python 3.12, torch 2.15.0):
- The executor, external-group, planner and shard-map suites pass (263).
- All other affine checkpoint tests pass, except the four
`TestAffineUniversalCheckpointResume` cases, which fail the same way on
master in this environment (uninstalled package version).
- With deepspeedai#8967 merged on top, its analysis tests pass unchanged (59
together with the executor suite).
- pre-commit is clean.

This merges cleanly with deepspeedai#8936, deepspeedai#9010 and deepspeedai#9011. With deepspeedai#8966 and deepspeedai#8967
the only conflict is that both append a section to
`affine-transfer.rst`.

---------

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
@delock

delock commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Hi @jinyouzhi can you resolve merge conflict? Thanks!

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 16, 2026
@delock
delock enabled auto-merge November 16, 2026 09:11
@delock
delock added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@delock
delock added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@delock
delock added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
…tion

# Conflicts:
#	docs/code-docs/source/affine-transfer.rst
@delock

delock commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@jinyouzhi can you take a look whether these modal failures in merge queue test relates to your PR? Thanks!
[aws:test:tail] FAILED tests/unit/v1/zero/test_zero3_nvme_param_prefetch.py::TestZero3NVMeParamPrefetch::test_matches_cpu_offload[none]
[aws:test:tail] FAILED tests/unit/v1/zero/test_zero3_nvme_param_prefetch.py::TestZero3NVMeParamPrefetch::test_matches_cpu_offload[reentrant]
[aws:test:tail] FAILED tests/unit/v1/zero/test_zero3_nvme_param_prefetch.py::TestZero3NVMeParamPrefetch::test_matches_cpu_offload[non_reentrant]
[aws:test:tail] FAILED tests/unit/v1/zero/test_zero3_nvme_offload.py::TestZero3NVMeOffload::test_param_and_optimizer[True]
[aws:test:tail] FAILED tests/unit/v1/zero/test_zero3_nvme_offload.py::TestZero3NVMeOffload::test_param
[aws:test:tail] FAILED tests/unit/v1/zero/test_zero3_nvme_offload.py::TestZero3NVMeOffload::test_param_and_optimizer[False]

@Achyuthan-S

Copy link
Copy Markdown
Contributor

@delock I don't think these come from this PR. The same six NVMe tests fail on master itself, in the nightly at e35282b (#9095), and in the merge-queue run for #9078, which only changes op_builder/dc.py. All of the failing runs fell back to AWS, where these tests run. On Modal they are skipped. Details are in #9095.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

Thanks @delock @Achyuthan-S I checked yesterday, it looks like not PR self. And just now another failure in merge queue.

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 17, 2026
@delock

delock commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@Achyuthan-S @jinyouzhi thanks for the feedback. The failed issue is fixed in #9099 . We can restart merging.

@delock
delock added this pull request to the merge queue Nov 17, 2026
Merged via the queue into deepspeedai:master with commit 4466172 Nov 17, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants