Repository navigation
[checkpoint] Batch replica selection for affine transfer plans - #8936
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24881a444e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return _emit(prepared, choices, budget) | ||
|
|
||
|
|
||
| def plan_transfers(pairs: Mapping[str, Tuple[ParamAffineMap, ParamAffineMap]], |
There was a problem hiding this comment.
Document the new batch planning API
For downstream users adopting plan_transfers, this patch exposes the entry point in __all__ plus a new callback/split protocol and default selection behavior, but adds no user documentation; the existing AutoEP docs only explain plan_expert_transfer and the old per-request replica_selector, so there is no supported reference for the process mappings, element sizes, split constraints, or policy answers. Add the required documentation alongside the feature.
AGENTS.md reference: AGENTS.md:L71-L71
Useful? React with 👍 / 👎.
| if type(part.source_location) is not int or part.source_location not in holders: | ||
| raise InvalidReplicaPolicyError(f'{where}: location {part.source_location!r} is not a validated ' | ||
| f'holder {sorted(holders)}') | ||
| origin, shape = tuple(part.logical_origin), tuple(part.shape) |
There was a problem hiding this comment.
Validate part fields before tuple conversion
When a custom policy returns a ReplicaPart whose logical_origin or shape is non-iterable (for example None), these tuple(...) calls raise a raw TypeError. The function's new contract says every invalid answer raises InvalidReplicaPolicyError; callers that catch AffineTransferError to reject a policy or fall back will instead abort, so convert and validate these fields under the policy-error path.
Useful? React with 👍 / 👎.
| parts = _resolve(request, item, answer, min_split_bytes, element_size) | ||
| choices[name][index] = parts | ||
| parts_per_name[name] += len(parts) | ||
| if parts_per_name[name] > budget.max_pair_checks: |
There was a problem hiding this comment.
Enforce split budgets before materializing parts
When a custom policy accidentally returns a very large split for one request, _resolve materializes every LogicalBox and sorts the full answer before this cumulative check runs. Consequently even a tiny max_pair_checks does not bound planner work or memory, so a malformed policy can cause a large allocation or OOM before it is rejected; enforce the remaining per-parameter allowance while or before resolving the answer.
Useful? React with 👍 / 👎.
Achyuthan-S
left a comment
There was a problem hiding this comment.
I reviewed this from the policy side, since the cost-aware policy will sit on this interface. It has what that policy needs:
- Each request carries its bytes, the target process and every candidate's process.
- The whole sync arrives in one call, in a fixed order, so a policy can balance load over all of it, including the single-holder load it can't move.
The topology and the calibration (#8966's fit_links) can live inside the policy, so I agree the process_node mapping can stay out of the planner. The refactor keeps plan_transfer's selection and emission as they were, and local_first_least_loaded does what its docstring says.
Checked on CPU:
- At 24881a4:
test_affine_replica_policy.py,test_affine_transfer_plan.pyandtest_affine_shard_map.pypass (251). - Merged with current master (after #8809, #8885 and #8975), the merge is clean, and the planner, executor, AutoEP and ZeRO-affine tests pass together (583 passed, 8 skipped).
A few small things:
- Codex's first P2 is real: a
ReplicaPartwithlogical_origin=Noneraises a rawTypeError(I checked), notInvalidReplicaPolicyError. The second is minor: the policy's answer is already in memory by then, so checkinglen(answer)against the remainingmax_pair_checksallowance before_resolveonly saves the planner's own work on a malformed split. It's still cheap to add. - Docs: #8809 has landed, so the
plan_transferssection can go intoaffine-transfer.rstnow. - A note for policy authors: every rank has to return the same answers. #8809's schedule digest already refuses mismatched plans ("ranks disagree about the transfer schedule"), so this fails loudly. Still, one line in the docstring saying that cost inputs should come from data every rank shares, such as
fit_linkson the gathered calibration, would help. - The test docstring cites "Theorem 3 of the proof draft". In the PDF I posted it's Theorem 4, "Any consistent plan is exact". Citing it by name would survive renumbering.
LGTM from the policy side. The first part of 1 is worth doing before merge, and the rest is up to you.
Add plan_transfers, which plans every parameter of one sync and hands a replica policy every request at once, as agreed in deepspeedai#8935. The policy returns a holder location per request, or splits a request along axis 0 across validated holders. The planner checks every answer before it emits a segment: one answer per request, only validated holders, and parts that tile the request exactly. So a policy decides where data is read from, never what arrives. Requests name map locations and, when the caller passes the location-to-process mappings, the processes behind them, so a policy can tell a local copy from a transfer. element_size turns elements into bytes and min_split_bytes bounds every part of a split. Policies live in affine_replica_policy.py: first_holder (plan_transfer's default), local_first_least_loaded (the plan_transfers default) and per_request, which runs an existing replica_selector as a batch policy. plan_transfer itself is unchanged; it now shares its validation and emission steps with plan_transfers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
A ReplicaPart whose logical_origin or shape is not iterable is now refused with InvalidReplicaPolicyError instead of raising a raw TypeError, and an answer with more parts than the remaining max_pair_checks allowance is refused on its length before _resolve expands it. The pair budget now raises InvalidReplicaPolicyError as well, so every refused answer uses the error the planner documents. Document plan_transfers, the request and answer protocol, the process mappings, element sizes (one entry per parameter), split constraints and the built-in policies in affine-transfer.rst, and tell policy authors that answers must agree across ranks, so cost inputs must come from data every rank shares. Cite the deepspeedai#8252 theorem "Any consistent plan is exact" by name instead of by number. Tests: the rejected-answer cases include non-iterable part fields, test_the_pair_budget_is_checked_before_the_parts_are_read pins that the planner decides from len(answer) without reading the parts, and both budget tests pin the policy error, including parts accumulated across requests. All five fail on the previous planner. On macOS arm64 CPU (Python 3.11, torch 2.15.0), test_affine_replica_policy.py passes (65 tests), the four affine checkpoint suites pass together (306), and the AutoEP unit suites pass (215 passed, 8 skipped). pre-commit (yapf, flake8, codespell, license) is clean. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
24881a4 to
773ddfd
Compare
|
@Achyuthan-S thanks — all four are in
Validation on macOS arm64 CPU (Python 3.11, torch 2.15.0): 65 tests in the new file, 306 across the four affine checkpoint suites, and 215 passed / 8 skipped for the AutoEP suites; pre-commit (yapf, flake8, codespell, license) is clean. No GPU path is touched. |
|
Re-checked 773ddfd: all four are in.
I merged the branch with current master: the merge is clean, and the planner, replica-policy, shard-map, ZeRO-affine, AutoEP-affine and executor suites (including the external-group test) pass together on CPU, with 383 passed. LGTM from the policy side. The cost-aware policy will build directly on this interface. |
…omparisons (deepspeedai#9011) **Stacked on deepspeedai#9010** (only the last two commits are new here; I will rebase after deepspeedai#9010 merges). `_validate_holders` compared every piece with every other piece, `_sources` compared every source region with every other region, and `_shards_tile` compared each rank's pieces pairwise by shard address. Planning time therefore grew quadratically with the number of pieces, and a map with 1024 pieces already exceeded the default `max_pair_checks`. When no rank repeats a box and distinct boxes do not overlap, a rank covers a box exactly when it holds that very box, so the holders of each box can be read from a table. This PR adds `boxes_disjoint` to `affine_geometry`. It sweeps along the axis whose box origins vary most and compares each box only with the boxes still open where it starts. The same sweep proves that a rank's pieces tile its shard when they are disjoint by address and fill it (suggested by 0z5a in review). Maps that fail the fast checks fall back to the original pairwise walk, which names the offending piece, so error messages are unchanged, and maps that pass no longer hit the per-rank `len(held)**2` budget. The sweep counts its comparisons against `max_pair_checks` too (suggested by Achyuthan-S in review). A disjoint layout can keep every box open for the whole sweep: for example, h full-height one-column boxes plus 2h one-element boxes beside them. When the sweep runs out of budget it reports "not shown disjoint". It compares at most n·(n−1)/2 pairs, so running out implies n² > `max_pair_checks`, and the pairwise fallback refuses at once. With 4,095 such pieces, planning went from 19.2 s to a budget refusal in 0.6 s. Planning time (8 ranks with interleaved rows, re-sharded to 8 row shards, 64 columns; ms, single CPU process): | pieces | before | holders/sources only | + shard addresses (this PR) | |---|---|---|---| | 256 | 211 | 15 | 7 | | 512 | 827 | 46 | 15 | | 1024 | 3278 (refused by default budget) | 156 | 33 | | 2048 | 12926 | 569 | 76 | | 4096 | 50469 | 2227 | 235 | Tests: - Randomized oracle over 200 seeds: plans and errors match the pairwise walk, including replicas split at different points, overlapping pieces and wrong holder lists. - `boxes_disjoint` is checked against an all-pairs reference over 100 seeds. - Mutation check: the tests catch dropped intersections and wrong holder sets. - A rank holding 1024 pieces plans under the default `max_pair_checks` (refused before this change). - The always-open layout above plans correctly under the default budget and is refused with `max_pair_checks` just below its sweep cost. Without the sweep budget, that test plans instead of refusing. - `tests/unit/checkpoint -k affine` passes on 8× NVIDIA L20 (645 passed). At 12663c3 on CPU (`DS_ACCELERATOR=cpu`, 4 ranks): 642 passed, 5 skipped. The remaining near-quadratic steps, the source × target join and `_covers_union`, overlap with deepspeedai#8936 and will follow after it lands; segment merging in `_coalesce` has its own `max_merge_checks` budget and will follow separately. --------- Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
deepspeedai#9027) ## Why A replica policy (deepspeedai#8936) or cost model (deepspeedai#8252) needs to know what a batch will cost before running it: messages per peer, rounds, and staging memory. deepspeedai#8967 builds that report today by importing the executor's private `_packed_rounds` and `build_operations`. It then re-derives the in-place decision and the arena sizing, so its numbers would drift silently whenever the executor's staging changes. It asked for a public API instead. ## What `schedule_transfers(plans, endpoints, budget, element_size, process=None)` in `affine_transfer_executor` returns, for one process or for every process the endpoints name, the `ProcessSchedule` that `execute_transfers` would run. It uses no buffers and no communication: - the rounds per peer and direction (`ScheduledRound`: elements, chunks, and whether the round is sent or received in place); - the staging arena each peer direction allocates; - the local copies; - the round count that `TransferStats.rounds` reports. `execute_transfers` now builds its rounds and arenas from the same private computation, so the report and the run cannot disagree. The in-place decision is made from geometry rather than from the buffer. `_region` is an `as_strided` view of a packed shard, so it is contiguous exactly when `(chunk_shape, strides)` is row-major packed, ignoring extent-1 axes. Unchanged: - `plan_digest`, the batch fingerprint and `TransferStats`, so mixed versions still agree on the schedule; - `build_operations` and `_packed_rounds`, which keep their names and signatures until deepspeedai#8967 switches. ## Tests - `test_schedule_is_what_the_executor_runs` (2 and 4 ranks; mixed packed and strided regions with 16-element messages) checks: - rounds, sent, received and local-copy bytes against the executor's `TransferStats`; - that both sides of every message agree; - the arena sizes; - that the batch exercises both the in-place and the staged path. - `test_packed_regions_are_exactly_the_contiguous_views` compares the geometric rule with `Tensor.is_contiguous` over 2000 random shapes and strides, including extent-1 axes. - A single-process case covers local-only batches and the argument checks. - Forcing every round to be staged, or dropping the extent-1 exception, makes these tests fail. On macOS arm64 CPU (Python 3.12, torch 2.15.0): - The executor, external-group, planner and shard-map suites pass (263). - All other affine checkpoint tests pass, except the four `TestAffineUniversalCheckpointResume` cases, which fail the same way on master in this environment (uninstalled package version). - With deepspeedai#8967 merged on top, its analysis tests pass unchanged (59 together with the executor suite). - pre-commit is clean. This merges cleanly with deepspeedai#8936, deepspeedai#9010 and deepspeedai#9011. With deepspeedai#8966 and deepspeedai#8967 the only conflict is that both append a section to `affine-transfer.rst`. --------- Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
…calibration Part-2) (deepspeedai#8966) ## Calibrate affine transfer links per path Stacked on deepspeedai#8809 only for its docs page (`affine-transfer.rst`); the module itself needs nothing beyond master. Until deepspeedai#8809 merges, please review only the commits after the deepspeedai#8809 merge. ### Why A replica policy or cost model choosing between plans (deepspeedai#8252, deepspeedai#8935/deepspeedai#8936) needs link costs measured on the machine it runs on. Executor measurements show that round count and receiver sharing dominate sync time, so per-pair numbers are not enough on their own; this PR also measures 1→N and N→1. ### What `deepspeed/checkpoint/affine_transfer_calibration.py`: - `calibrate_links(sizes, repeats, warmup, path_of, fan_width, fan_peers, group)`: collective over `group` (the default group unless given). Returns the same raw samples on every process, with no aggregation, for: - `local` copy; - `peer` (each ordered pair alone); - `fan_out` (1→N); - `fan_in` (N→1, receiver contention). Sizes run from 4 KiB to 256 MiB by default. - Each message is labelled by `path_of(from_process, to_process)`. `default_path_of` gives `local`/`peer`; `node_path_of(n)` gives `intra_node`/`inter_node`; any callable works. - `fit_links(calibration)`: per-path `alpha + bytes / bandwidth`, fitted to the per-size medians; `peer_pairs` per ordered pair; fans keyed by `(root, peers)`, with `send`/`receive` giving each root's aggregate rate. Weight transfers are one-way (a training process sends, a rollout process receives), so no pattern has a process send and receive at once. It measures only. It does not plan, select replicas, or change the planner or executor. ### Example (one node, 8 devices, default sizes) | pattern | B70 xccl: alpha (µs) / GB/s | L20 nccl (PCIe): alpha (µs) / GB/s | H20 nccl (NVLink): alpha (µs) / GB/s | |---|---|---|---| | local | 73 / 359 | 16 / 1534* | 9 / 1881 | | peer | 301 / 28.2 | 72 / 25.3 | 54 / 275 | | fan_out (7 peers) | 531 / 3.4 | 141 / 3.6 | 88 / 49.6 | | fan_in (7 peers) | 670 / 2.6 | 56 / 3.8 | 46 / 54.5 | | run time | 88 s | 34 s | 13 s | \* poor linear fit (max relative error 0.77); small copies are cache-resident. ### Multi-node (2 × 8 H20, RoCE 8 × 200 Gb/s per node, `node_path_of(8)`, 16 processes) | pattern | alpha (µs) / GB/s | max rel. error | |---|---|---| | peer, intra_node | 57 / 53.8 | 0.07 | | peer, inter_node | 97 / 16.6 (pairs 14.0–24.5) | 0.50 | | fan_in, 7 intra-node peers: root receive | 94.6 GB/s | | | fan_out, 8 cross-node peers: root send | 62.1 GB/s median (33–78 over roots) | | | run time | 72 s (+111 s for cross-node fans of width 1/4/8) | | - The same NVLink pair measured 275 GB/s in the 8-process group and 54 GB/s in the 16-process group over RoCE (105 GB/s with NCCL over TCP), so calibrate on the group and communication settings the transfer uses. ### Tests `tests/unit/checkpoint/test_affine_transfer_calibration.py`: - fit on synthetic data with known link costs, per-pair and per-peer-set fits; - schema and argument checks; - 4-rank runs checking completeness, sample counts, and fan path direction with an asymmetric `path_of`. 10 passed on 8× B70 (xccl). `pytest tests/unit/checkpoint/ -k affine`: 344 passed, 1 skipped on 8× H20 with torch 2.13.0+cu129 / NCCL 2.29.7 and with torch 2.9.1+cu129 / NCCL 2.29.7. Mutation check: flipping the fan_in label direction fails a test. Cross-node runs used NCCL 2.29.7 on both nodes (torch 2.13.0 and 2.9.1). AI assistance (GitHub Copilot) was used for the multi-node runs and this description; I reviewed the changes and the results. ### Follow-ups - Sampled pairs for large worlds. - Joining the samples to a plan's messages (analysis PR, stacked on this one and deepspeedai#8809). --------- Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com> Signed-off-by: iLeGend <824040212@qq.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
Implements the planner side of #8935. It does not close the issue: the cost-aware policies are the policy layer under #8252.
What it adds
plan_transfers(pairs, limits, *, source_processes, target_processes, element_size, min_split_bytes, replica_policy)indeepspeed/checkpoint/affine_transfer.py. It plans every parameter of one sync and calls one replica policy with every request at once.ReplicaRequestandReplicaPart.ReplicaParts that cut the region along axis 0.InvalidReplicaPolicyErrorotherwise:TypeError;min_split_bytes, every part of a split is at least that large;PlanBudget.max_pair_checks, checked from the answer's length before the parts are expanded.deepspeed/checkpoint/affine_replica_policy.pyholds three policies:first_holder: whatplan_transferdoes by default.local_first_least_loaded: theplan_transfersdefault. It takes a holder in the target's own process first, then the sending process with the fewest bytes assigned so far in this batch. Ties go to the lowest location, and it never splits.per_request(selector): runs an existingreplica_selectoras a batch policy.Review follow-up (commit
773ddfd3)ReplicaPart.logical_origin/shapefields are refused asInvalidReplicaPolicyError; the pair budget now uses that error too, so every refused answer uses the documented class._resolveexpands it.plan_transfers, the request/answer protocol, the process mappings,element_size(one entry per parameter) and the split constraints are documented indocs/code-docs/source/affine-transfer.rst, the page from [checkpoint] Execute affine transfer plans with bounded peer staging #8809.fit_linkson the calibration from [AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) #8966), never from a rank-local measurement; the executor's schedule digest refuses a mismatch.What it does not change
plan_transferbehaves exactly as before; it shares its validation and emission steps withplan_transfers. AutoEP'splan_expert_transferand its docs are untouched.Decisions from #8935
TransferEndpoints, so the planner does not import the executor.min_split_bytescomes from my last comment on #8935 and has not been agreed yet. It is optional and off by default. Theprocess_nodemapping proposed there is deferred.Tests
tests/unit/checkpoint/test_affine_replica_policy.py, CPU only:extract. This is the check that matches the "Any consistent plan is exact" theorem of the correctness draft for Define affine IR for checkpointing #8252.first_holderandper_requestreproduceplan_transfer's plans exactly.Noneand strings;min_split_bytes, and more parts than the budget allows, including parts accumulated across requests.With
DS_ACCELERATOR=cpuon macOS arm64 (Python 3.11, torch 2.15.0), the new file passes (65 tests) together withtest_affine_transfer_plan.py,test_autoep_affine.pyandtest_affine_shard_map.py(306 passed), andtests/unit/v1/moe/test_autoep_unit.py(which coversplan_expert_transfer) together withtest_autoep_affine_universal.pypass too (215 passed, 8 skipped). yapf 0.43.0, flake8 5.0.4, codespell, the license check and the other pre-commit hooks are clean. The five rejection/budget tests added here fail on the previous planner and pass now.How it fits with other PRs
plan_transferstill shares_prepare/_emitwithplan_transfers.affine_geometry.pymove can land before or after this; I'll rebase in whichever order suits you.fit_links) that cost-aware policies will read; the planner does not consume it.Scope, compatibility and AI assistance
Only the CPU planning interface, its tests and its documentation change: no training-loop, GPU, ZeRO, offload or precision path is touched, so no compatibility-matrix cell or on-device serving check is affected, and none was run. AI assistance was used: the initial implementation was co-authored with Claude Opus 5.5, this review follow-up was prepared with an AI coding agent, and the submitter reviewed every changed line and ran the tests above.
cc @Achyuthan-S @delock