Repository navigation
[AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) - #8966
Conversation
|
Thanks, this is the input the cost model needs. I'll build the cost-aware policy on #8936 from
Your B70 numbers already show why the cap matters. If I read the per-link fit right, seven senders give about 7 × 2.6 ≈ 18 GB/s into one receiver, which is less than one pair alone (28.2 GB/s). |
|
@Achyuthan-S Yes, you're very observant—I agree. Based on these experiments, we can refine some of our policies. |
|
Thanks! If it's easier, I can send both as a small commit to your branch: |
|
@Achyuthan-S Sure, I have added you as my fork's collabration or I also can cherry-pick a commit provided by you. Thanks |
0z5a
left a comment
There was a problem hiding this comment.
Read the last commit (3cfa4153), as asked. The measurement design is careful: the barrier and synchronize brackets keep collectives out of timed regions, the pair sweep isolates each link, and the fans separate sender and receiver sharing. On CPU (gloo, 4 ranks), the calibration tests pass together with #8967's and the executor's (62).
A few thoughts, mostly about feeding a cost model directly:
- Per-process send and receive capacity. A model of the form t = max over processes of max(sent / send_capacity, received / receive_capacity) needs each process's aggregate send and receive capacity.
fit_linksgives per-path link costs; the fan fits use the peers' times, and the roots' times, which carry the aggregate, stay only in the raw result. Computingbytes * len(peers) / root_secondsfrom the raw fan entries is a few lines. On CPU it gives a per-root figure for each direction. Havingfit_linksfit those per process (or per root) assendandreceivewould let #8967'sProcessReport.sent_bytes/received_bytesbe priced directly. - Fan peers. The peers are
root + 1 … root + fan_width, so withfan_widthbelow the processes per node every fan stays within a node. If receive capacity differs across nodes, a way to choose the peers (afan_peers(root)hook, or a stride) would let the fans cover it. - Defaults at scale. With the default
fan_width = world - 1, each process can allocate(1 + world) * max(sizes), about 16 GiB at 64 ranks and 256 MiB, and the pair sweep is O(world²) barriers per size. The docstring says so, but a default capped at the processes per node, or a check against free memory, might save someone a surprise. - Process group. Calibration always uses the default group, while the executor runs on whatever group
TransferEndpointsnames. An optionalgroupargument would let it measure the group the executor will actually use, e.g. an external trainer–rollout group, with the same rank numbering the analysis report uses.
|
@Achyuthan-S @0z5a Thanks both. I agree with the direction: a list of fan widths and a fitted root time (per-process Topology. Four PCIe switches, each with two cards behind one Gen5 x16 root port, and two switches per socket: {0,1} {2,3} on socket 0 and {4,5} {6,7} on socket 1 (assuming ranks follow PCI address order). GH200 over Slingshot is a Clos, so every sender set looks alike there, which fits your flat incast. Pair bandwidth, GB/s (row = sender, column = receiver): These are stable across runs. Cross-socket traffic sits near 27 GB/s. Within a socket, bandwidth depends on the sender and the direction (2→0 is 54, 0→2 is 7.9, on the same switches), which the tree alone does not predict. Fan-in root aggregate (
Three cases explain most of it:
So the finish time looks like Proposal for this PR:
On 0z5a's other points:
@Achyuthan-S Please push your width-list and root-fit commit. I'll add the per-pair fit and the |
|
Pushed the width-list and root-fit commit:
Your B70 table is a great counterexample. It fits the same form once every shared resource is a term, not only the endpoints: a fan finishes when the busiest resource it crosses does, i.e. the max over each sender's link, the receiver, and any shared link such as your cross-socket hop, of the bytes through it over its capacity. Your three cases are one of each:
So your per-pair fit and |
|
@Achyuthan-S @0z5a Pushed ba8b39f (on top of Achyuthan's commit, with #8809's 11fe945 merged in). Updates
On the B70 counterexample: Validation: 8× L20 (driver 580.173.02, torch 2.14.1+cu130): calibration tests on 4 GPUs, 10 passed; external-group test on CPU passed 20 of 20 runs; pre-commit passed. Multi-node not validated. |
|
Thanks, this is the right split. Keying by peer set captures exactly what the B70 table showed, and I agree on the semantics: One small thing for consumers: the fits now have tuple keys ( |
|
Good catch. I added a note to the |
|
Thanks for the JSON note and the H20 multi-node numbers. The same NVLink pair measuring 275 GB/s in one group and 54 GB/s in another is a useful warning, and I'll make "calibrate in the transfer's own group" explicit on the policy side. One concern with dropping the concurrent pattern: some transfers do have processes sending and receiving at once.
@0z5a's B300 runs show what this costs: 618 GB/s per direction duplex, against 664 GB/s one-way. On GH200 the overlap held because the NIC's receive and the NVLink send are separate resources. So the policy needs a duplex number per path. The shift pattern's labelling was the problem, not the measurement, and a simpler pattern would do: for example, each ordered pair exchanging both ways at once, labelled by the pair's path in each direction. If you'd rather keep this PR one-way, a line in the docs saying duplex isn't measured, plus a follow-up, works too. |
|
@Achyuthan-S Thanks. I had assumed I agree the labelling was the problem. The old shift pattern made it unclear which participant each sample described. |
|
@Achyuthan-S Pushed d00ba30, following your suggestion:
H20, one node, 4 GPUs over NVLink, 256 MiB, median of 10, GB/s per direction: one-way 331–335, both ways 276–287 (0.83–0.86×), relay 282–287 (≈0.85×). Calibration tests pass on 4 GPUs (12 passed). Cross-node relay is not measured yet. |
|
Thanks @jinyouzhi, d00ba30 covers all four cases. This is how the #8936 policy will read it:
Your H20 numbers (both ways at 0.83–0.86× one-way) and @0z5a's B300 (618 vs 664 GB/s, 0.93×) show the same thing: the cost is real and depends on the machine, so it has to be calibrated. On CPU (gloo, 4 ranks) the calibration tests pass at d00ba30 (12 passed). One small thing: the raw format gained When you get a multi-node slot, the |
|
Re-read up to
|
57a47ff to
851ea68
Compare
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Thanks @jinyouzhi, d465334 covers both Codex points, and @0z5a's two are in as well. One from my earlier comment is still open at d465334: |
|
@Achyuthan-S Fixed in f678c9d: |
fit_links reads peer samples' both_ways and the raw relay list. A raw calibration saved before those existed still claimed schema 1, so it passed the schema check and then raised KeyError. Schema 2 makes fit_links refuse it with a clear error instead. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
deepspeedai#9027) ## Why A replica policy (deepspeedai#8936) or cost model (deepspeedai#8252) needs to know what a batch will cost before running it: messages per peer, rounds, and staging memory. deepspeedai#8967 builds that report today by importing the executor's private `_packed_rounds` and `build_operations`. It then re-derives the in-place decision and the arena sizing, so its numbers would drift silently whenever the executor's staging changes. It asked for a public API instead. ## What `schedule_transfers(plans, endpoints, budget, element_size, process=None)` in `affine_transfer_executor` returns, for one process or for every process the endpoints name, the `ProcessSchedule` that `execute_transfers` would run. It uses no buffers and no communication: - the rounds per peer and direction (`ScheduledRound`: elements, chunks, and whether the round is sent or received in place); - the staging arena each peer direction allocates; - the local copies; - the round count that `TransferStats.rounds` reports. `execute_transfers` now builds its rounds and arenas from the same private computation, so the report and the run cannot disagree. The in-place decision is made from geometry rather than from the buffer. `_region` is an `as_strided` view of a packed shard, so it is contiguous exactly when `(chunk_shape, strides)` is row-major packed, ignoring extent-1 axes. Unchanged: - `plan_digest`, the batch fingerprint and `TransferStats`, so mixed versions still agree on the schedule; - `build_operations` and `_packed_rounds`, which keep their names and signatures until deepspeedai#8967 switches. ## Tests - `test_schedule_is_what_the_executor_runs` (2 and 4 ranks; mixed packed and strided regions with 16-element messages) checks: - rounds, sent, received and local-copy bytes against the executor's `TransferStats`; - that both sides of every message agree; - the arena sizes; - that the batch exercises both the in-place and the staged path. - `test_packed_regions_are_exactly_the_contiguous_views` compares the geometric rule with `Tensor.is_contiguous` over 2000 random shapes and strides, including extent-1 axes. - A single-process case covers local-only batches and the argument checks. - Forcing every round to be staged, or dropping the extent-1 exception, makes these tests fail. On macOS arm64 CPU (Python 3.12, torch 2.15.0): - The executor, external-group, planner and shard-map suites pass (263). - All other affine checkpoint tests pass, except the four `TestAffineUniversalCheckpointResume` cases, which fail the same way on master in this environment (uninstalled package version). - With deepspeedai#8967 merged on top, its analysis tests pass unchanged (59 together with the executor suite). - pre-commit is clean. This merges cleanly with deepspeedai#8936, deepspeedai#9010 and deepspeedai#9011. With deepspeedai#8966 and deepspeedai#8967 the only conflict is that both append a section to `affine-transfer.rst`. --------- Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
|
Hi @jinyouzhi can you resolve merge conflict? Thanks! |
…tion # Conflicts: # docs/code-docs/source/affine-transfer.rst
|
@jinyouzhi can you take a look whether these modal failures in merge queue test relates to your PR? Thanks! |
|
@delock I don't think these come from this PR. The same six NVMe tests fail on master itself, in the nightly at e35282b (#9095), and in the merge-queue run for #9078, which only changes |
|
Thanks @delock @Achyuthan-S I checked yesterday, it looks like not PR self. And just now another failure in merge queue. |
|
@Achyuthan-S @jinyouzhi thanks for the feedback. The failed issue is fixed in #9099 . We can restart merging. |
4466172
Calibrate affine transfer links per path
Stacked on #8809 only for its docs page (
affine-transfer.rst); the module itself needs nothing beyond master. Until #8809 merges, please review only the commits after the #8809 merge.Why
A replica policy or cost model choosing between plans (#8252, #8935/#8936) needs link costs measured on the machine it runs on. Executor measurements show that round count and receiver sharing dominate sync time, so per-pair numbers are not enough on their own; this PR also measures 1→N and N→1.
What
deepspeed/checkpoint/affine_transfer_calibration.py:calibrate_links(sizes, repeats, warmup, path_of, fan_width, fan_peers, group): collective overgroup(the default group unless given). Returns the same raw samples on every process, with no aggregation, for:localcopy;peer(each ordered pair alone);fan_out(1→N);fan_in(N→1, receiver contention).Sizes run from 4 KiB to 256 MiB by default.
Each message is labelled by
path_of(from_process, to_process).default_path_ofgiveslocal/peer;node_path_of(n)givesintra_node/inter_node; any callable works.fit_links(calibration): per-pathalpha + bytes / bandwidth, fitted to the per-size medians;peer_pairsper ordered pair; fans keyed by(root, peers), withsend/receivegiving each root's aggregate rate.Weight transfers are one-way (a training process sends, a rollout process receives), so no pattern has a process send and receive at once.
It measures only. It does not plan, select replicas, or change the planner or executor.
Example (one node, 8 devices, default sizes)
* poor linear fit (max relative error 0.77); small copies are cache-resident.
Multi-node (2 × 8 H20, RoCE 8 × 200 Gb/s per node,
node_path_of(8), 16 processes)Tests
tests/unit/checkpoint/test_affine_transfer_calibration.py:path_of.10 passed on 8× B70 (xccl).
pytest tests/unit/checkpoint/ -k affine: 344 passed, 1 skipped on 8× H20 with torch 2.13.0+cu129 / NCCL 2.29.7 and with torch 2.9.1+cu129 / NCCL 2.29.7. Mutation check: flipping the fan_in label direction fails a test. Cross-node runs used NCCL 2.29.7 on both nodes (torch 2.13.0 and 2.9.1).AI assistance (GitHub Copilot) was used for the multi-node runs and this description; I reviewed the changes and the results.
Follow-ups