Skip to content

[checkpoint] Batch replica selection for affine transfer plans - #8936

Merged
delock merged 3 commits into
deepspeedai:masterfrom
0z5a:replica-batch-policy-20261028
Nov 12, 2026
Merged

delock merged 3 commits into
deepspeedai:masterfrom
0z5a:replica-batch-policy-20261028

Conversation

@0z5a

@0z5a 0z5a commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Implements the planner side of #8935. It does not close the issue: the cost-aware policies are the policy layer under #8252.

What it adds

  • plan_transfers(pairs, limits, *, source_processes, target_processes, element_size, min_split_bytes, replica_policy) in deepspeed/checkpoint/affine_transfer.py. It plans every parameter of one sync and calls one replica policy with every request at once.
  • ReplicaRequest and ReplicaPart.
    • A request names the parameter, the target location, the region and its validated holders. When the caller passes the location→process mappings, it also names the processes behind them.
    • Regions with a single holder are included, so a policy sees load it cannot move.
    • Requests arrive in a fixed order: parameter, then target location, then origin.
  • An answer is either a holder location, or ReplicaParts that cut the region along axis 0.
  • The planner checks every answer before it emits any segment, and raises InvalidReplicaPolicyError otherwise:
    • one answer per request;
    • only validated holders;
    • parts that tile the request exactly along axis 0;
    • part fields that are not iterable coordinates, rather than a raw TypeError;
    • with min_split_bytes, every part of a split is at least that large;
    • the parts of one parameter stay within PlanBudget.max_pair_checks, checked from the answer's length before the parts are expanded.
  • deepspeed/checkpoint/affine_replica_policy.py holds three policies:
    • first_holder: what plan_transfer does by default.
    • local_first_least_loaded: the plan_transfers default. It takes a holder in the target's own process first, then the sending process with the fewest bytes assigned so far in this batch. Ties go to the lowest location, and it never splits.
    • per_request(selector): runs an existing replica_selector as a batch policy.

Review follow-up (commit 773ddfd3)

What it does not change
plan_transfer behaves exactly as before; it shares its validation and emission steps with plan_transfers. AutoEP's plan_expert_transfer and its docs are untouched.

Decisions from #8935

  • One sync per batch.
  • Only axis-0 splits for now. Other split shapes are refused, so widening later is an additive change.
  • Plain location→process mappings rather than TransferEndpoints, so the planner does not import the executor.
  • Policies live in their own module.
  • The per-request adapter stays.

min_split_bytes comes from my last comment on #8935 and has not been agreed yet. It is optional and off by default. The process_node mapping proposed there is deferred.

Tests
tests/unit/checkpoint/test_affine_replica_policy.py, CPU only:

  • Random policies over 24 seeds, including random splits across holders, on five parameters: whole replicas, a partly replicated fused block, a vector and a scalar. Every plan writes each target element exactly once and matches extract. This is the check that matches the "Any consistent plan is exact" theorem of the correctness draft for Define affine IR for checkpointing #8252.
  • first_holder and per_request reproduce plan_transfer's plans exactly.
  • The default policy:
    • copies locally when it can;
    • balances load, counting single-holder load;
    • never assumes a local copy without mappings.
  • Refused answers:
    • holders that are not candidates;
    • bools, floats, None and strings;
    • gaps, overlaps and parts outside the region;
    • cuts along axis 1, empty parts and parts with the wrong number of axes;
    • non-iterable part fields;
    • the wrong number of answers;
    • splits under min_split_bytes, and more parts than the budget allows, including parts accumulated across requests.
  • The maps are validated before the policy runs.
  • The pair budget is decided from the answer's length before the planner reads its parts.

With DS_ACCELERATOR=cpu on macOS arm64 (Python 3.11, torch 2.15.0), the new file passes (65 tests) together with test_affine_transfer_plan.py, test_autoep_affine.py and test_affine_shard_map.py (306 passed), and tests/unit/v1/moe/test_autoep_unit.py (which covers plan_expert_transfer) together with test_autoep_affine_universal.py pass too (215 passed, 8 skipped). yapf 0.43.0, flake8 5.0.4, codespell, the license check and the other pre-commit hooks are clean. The five rejection/budget tests added here fail on the previous planner and pass now.

How it fits with other PRs

Scope, compatibility and AI assistance
Only the CPU planning interface, its tests and its documentation change: no training-loop, GPU, ZeRO, offload or precision path is touched, so no compatibility-matrix cell or on-device serving check is affected, and none was run. AI assistance was used: the initial implementation was co-authored with Claude Opus 5.5, this review follow-up was prepared with an AI coding agent, and the submitter reviewed every changed line and ran the tests above.

cc @Achyuthan-S @delock

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-05T09:59:49.703718Z 24881a4 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24881a444e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

return _emit(prepared, choices, budget)


def plan_transfers(pairs: Mapping[str, Tuple[ParamAffineMap, ParamAffineMap]],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Document the new batch planning API

For downstream users adopting plan_transfers, this patch exposes the entry point in __all__ plus a new callback/split protocol and default selection behavior, but adds no user documentation; the existing AutoEP docs only explain plan_expert_transfer and the old per-request replica_selector, so there is no supported reference for the process mappings, element sizes, split constraints, or policy answers. Add the required documentation alongside the feature.

AGENTS.md reference: AGENTS.md:L71-L71

Useful? React with 👍 / 👎.

Comment thread deepspeed/checkpoint/affine_transfer.py Outdated
if type(part.source_location) is not int or part.source_location not in holders:
raise InvalidReplicaPolicyError(f'{where}: location {part.source_location!r} is not a validated '
f'holder {sorted(holders)}')
origin, shape = tuple(part.logical_origin), tuple(part.shape)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate part fields before tuple conversion

When a custom policy returns a ReplicaPart whose logical_origin or shape is non-iterable (for example None), these tuple(...) calls raise a raw TypeError. The function's new contract says every invalid answer raises InvalidReplicaPolicyError; callers that catch AffineTransferError to reject a policy or fall back will instead abort, so convert and validate these fields under the policy-error path.

Useful? React with 👍 / 👎.

Comment thread deepspeed/checkpoint/affine_transfer.py Outdated
Comment on lines +664 to +667
parts = _resolve(request, item, answer, min_split_bytes, element_size)
choices[name][index] = parts
parts_per_name[name] += len(parts)
if parts_per_name[name] > budget.max_pair_checks:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce split budgets before materializing parts

When a custom policy accidentally returns a very large split for one request, _resolve materializes every LogicalBox and sorts the full answer before this cumulative check runs. Consequently even a tiny max_pair_checks does not bound planner work or memory, so a malformed policy can cause a large allocation or OOM before it is rejected; enforce the remaining per-parameter allowance while or before resolving the answer.

Useful? React with 👍 / 👎.

@Achyuthan-S Achyuthan-S left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this from the policy side, since the cost-aware policy will sit on this interface. It has what that policy needs:

  • Each request carries its bytes, the target process and every candidate's process.
  • The whole sync arrives in one call, in a fixed order, so a policy can balance load over all of it, including the single-holder load it can't move.

The topology and the calibration (#8966's fit_links) can live inside the policy, so I agree the process_node mapping can stay out of the planner. The refactor keeps plan_transfer's selection and emission as they were, and local_first_least_loaded does what its docstring says.

Checked on CPU:

  • At 24881a4: test_affine_replica_policy.py, test_affine_transfer_plan.py and test_affine_shard_map.py pass (251).
  • Merged with current master (after #8809, #8885 and #8975), the merge is clean, and the planner, executor, AutoEP and ZeRO-affine tests pass together (583 passed, 8 skipped).

A few small things:

  1. Codex's first P2 is real: a ReplicaPart with logical_origin=None raises a raw TypeError (I checked), not InvalidReplicaPolicyError. The second is minor: the policy's answer is already in memory by then, so checking len(answer) against the remaining max_pair_checks allowance before _resolve only saves the planner's own work on a malformed split. It's still cheap to add.
  2. Docs: #8809 has landed, so the plan_transfers section can go into affine-transfer.rst now.
  3. A note for policy authors: every rank has to return the same answers. #8809's schedule digest already refuses mismatched plans ("ranks disagree about the transfer schedule"), so this fails loudly. Still, one line in the docstring saying that cost inputs should come from data every rank shares, such as fit_links on the gathered calibration, would help.
  4. The test docstring cites "Theorem 3 of the proof draft". In the PDF I posted it's Theorem 4, "Any consistent plan is exact". Citing it by name would survive renumbering.

LGTM from the policy side. The first part of 1 is worth doing before merge, and the rest is up to you.

0z5a and others added 2 commits November 5, 2026 19:59
Add plan_transfers, which plans every parameter of one sync and hands a
replica policy every request at once, as agreed in deepspeedai#8935. The policy
returns a holder location per request, or splits a request along axis 0
across validated holders. The planner checks every answer before it
emits a segment: one answer per request, only validated holders, and
parts that tile the request exactly. So a policy decides where data is
read from, never what arrives.

Requests name map locations and, when the caller passes the
location-to-process mappings, the processes behind them, so a policy can
tell a local copy from a transfer. element_size turns elements into bytes
and min_split_bytes bounds every part of a split.

Policies live in affine_replica_policy.py: first_holder (plan_transfer's
default), local_first_least_loaded (the plan_transfers default) and
per_request, which runs an existing replica_selector as a batch policy.
plan_transfer itself is unchanged; it now shares its validation and
emission steps with plan_transfers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
A ReplicaPart whose logical_origin or shape is not iterable is now refused
with InvalidReplicaPolicyError instead of raising a raw TypeError, and an
answer with more parts than the remaining max_pair_checks allowance is
refused on its length before _resolve expands it. The pair budget now raises
InvalidReplicaPolicyError as well, so every refused answer uses the error the
planner documents.

Document plan_transfers, the request and answer protocol, the process
mappings, element sizes (one entry per parameter), split constraints and the
built-in policies in affine-transfer.rst, and tell policy authors that answers
must agree across ranks, so cost inputs must come from data every rank shares.
Cite the deepspeedai#8252 theorem "Any consistent plan is exact" by name instead of by
number.

Tests: the rejected-answer cases include non-iterable part fields,
test_the_pair_budget_is_checked_before_the_parts_are_read pins that the planner
decides from len(answer) without reading the parts, and both budget tests pin
the policy error, including parts accumulated across requests. All five fail on
the previous planner. On macOS arm64 CPU (Python 3.11, torch 2.15.0),
test_affine_replica_policy.py passes (65 tests), the four affine checkpoint
suites pass together (306), and the AutoEP unit suites pass (215 passed,
8 skipped). pre-commit (yapf, flake8, codespell, license) is clean.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@0z5a
0z5a force-pushed the replica-batch-policy-20261028 branch from 24881a4 to 773ddfd Compare November 5, 2026 19:09
@0z5a

0z5a commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@Achyuthan-S thanks — all four are in 773ddfd3 (branch rebased on current master, after #8809, #8885 and #8975):

  1. A ReplicaPart whose logical_origin or shape is not iterable now raises InvalidReplicaPolicyError; the pair budget uses the same class, so every refused answer uses the documented one. The budget is decided from len(answer) before _resolve expands the parts, and it still accumulates across requests. The tests pin both (test_the_pair_budget_is_checked_before_the_parts_are_read, test_parts_count_against_the_budget_across_requests); the five rejection/budget cases fail on the previous planner.
  2. plan_transfers is documented in docs/code-docs/source/affine-transfer.rst: the request/answer protocol, source_processes/target_processes, element_size (one entry per parameter), the axis-0 split and min_split_bytes constraints, the policy error, and the three built-in policies.
  3. The plan_transfers and affine_replica_policy docstrings now say answers must agree on every rank, so cost inputs must come from data every rank shares (for example fit_links on the gathered calibration, [AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) #8966), and quote the executor's "ranks disagree about the transfer schedule (batch schedule)".
  4. The test docstring cites "Any consistent plan is exact" by name.

Validation on macOS arm64 CPU (Python 3.11, torch 2.15.0): 65 tests in the new file, 306 across the four affine checkpoint suites, and 215 passed / 8 skipped for the AutoEP suites; pre-commit (yapf, flake8, codespell, license) is clean. No GPU path is touched.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

Re-checked 773ddfd: all four are in.

  • A ReplicaPart with a non-iterable logical_origin or shape now raises InvalidReplicaPolicyError, since the tuple(...) conversion is guarded. The pair budget is decided from len(answer) before _resolve, and the budget tests pass.
  • plan_transfers is documented in affine-transfer.rst, the docstrings say answers must agree on every rank, and the test cites "Any consistent plan is exact" by name.

I merged the branch with current master: the merge is clean, and the planner, replica-policy, shard-map, ZeRO-affine, AutoEP-affine and executor suites (including the external-group test) pass together on CPU, with 383 passed. LGTM from the policy side. The cost-aware policy will build directly on this interface.

@delock
delock enabled auto-merge November 12, 2026 03:55
@delock
delock added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@delock
delock added this pull request to the merge queue Nov 12, 2026
Merged via the queue into deepspeedai:master with commit 2d02194 Nov 12, 2026
18 checks passed
LiRunGuo added a commit to LiRunGuo/DeepSpeed that referenced this pull request Nov 12, 2026
); shards given at each sync follow offload/reload

Signed-off-by: RunguoLi <li19107254665@gmail.com>
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026
banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Nov 14, 2026
…omparisons (deepspeedai#9011)

**Stacked on deepspeedai#9010** (only the last two commits are new here; I will
rebase after deepspeedai#9010 merges).

`_validate_holders` compared every piece with every other piece,
`_sources` compared every source region with every other region, and
`_shards_tile` compared each rank's pieces pairwise by shard address.
Planning time therefore grew quadratically with the number of pieces,
and a map with 1024 pieces already exceeded the default
`max_pair_checks`.

When no rank repeats a box and distinct boxes do not overlap, a rank
covers a box exactly when it holds that very box, so the holders of each
box can be read from a table. This PR adds `boxes_disjoint` to
`affine_geometry`. It sweeps along the axis whose box origins vary most
and compares each box only with the boxes still open where it starts.
The same sweep proves that a rank's pieces tile its shard when they are
disjoint by address and fill it (suggested by 0z5a in review). Maps that
fail the fast checks fall back to the original pairwise walk, which
names the offending piece, so error messages are unchanged, and maps
that pass no longer hit the per-rank `len(held)**2` budget.

The sweep counts its comparisons against `max_pair_checks` too
(suggested by Achyuthan-S in review). A disjoint layout can keep every
box open for the whole sweep: for example, h full-height one-column
boxes plus 2h one-element boxes beside them. When the sweep runs out of
budget it reports "not shown disjoint". It compares at most n·(n−1)/2
pairs, so running out implies n² > `max_pair_checks`, and the pairwise
fallback refuses at once. With 4,095 such pieces, planning went from
19.2 s to a budget refusal in 0.6 s.

Planning time (8 ranks with interleaved rows, re-sharded to 8 row
shards, 64 columns; ms, single CPU process):

| pieces | before | holders/sources only | + shard addresses (this PR) |
|---|---|---|---|
| 256 | 211 | 15 | 7 |
| 512 | 827 | 46 | 15 |
| 1024 | 3278 (refused by default budget) | 156 | 33 |
| 2048 | 12926 | 569 | 76 |
| 4096 | 50469 | 2227 | 235 |

Tests:
- Randomized oracle over 200 seeds: plans and errors match the pairwise
walk, including replicas split at different points, overlapping pieces
and wrong holder lists.
- `boxes_disjoint` is checked against an all-pairs reference over 100
seeds.
- Mutation check: the tests catch dropped intersections and wrong holder
sets.
- A rank holding 1024 pieces plans under the default `max_pair_checks`
(refused before this change).
- The always-open layout above plans correctly under the default budget
and is refused with `max_pair_checks` just below its sweep cost. Without
the sweep budget, that test plans instead of refusing.
- `tests/unit/checkpoint -k affine` passes on 8× NVIDIA L20 (645
passed). At 12663c3 on CPU (`DS_ACCELERATOR=cpu`, 4 ranks): 642 passed,
5 skipped.

The remaining near-quadratic steps, the source × target join and
`_covers_union`, overlap with deepspeedai#8936 and will follow after it lands;
segment merging in `_coalesce` has its own `max_merge_checks` budget and
will follow separately.

---------

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Nov 15, 2026
deepspeedai#9027)

## Why

A replica policy (deepspeedai#8936) or cost model (deepspeedai#8252) needs to know what a
batch will cost before running it: messages per peer, rounds, and
staging memory. deepspeedai#8967 builds that report today by importing the
executor's private `_packed_rounds` and `build_operations`. It then
re-derives the in-place decision and the arena sizing, so its numbers
would drift silently whenever the executor's staging changes. It asked
for a public API instead.

## What

`schedule_transfers(plans, endpoints, budget, element_size,
process=None)` in `affine_transfer_executor` returns, for one process or
for every process the endpoints name, the `ProcessSchedule` that
`execute_transfers` would run. It uses no buffers and no communication:
- the rounds per peer and direction (`ScheduledRound`: elements, chunks,
and whether the round is sent or received in place);
- the staging arena each peer direction allocates;
- the local copies;
- the round count that `TransferStats.rounds` reports.

`execute_transfers` now builds its rounds and arenas from the same
private computation, so the report and the run cannot disagree. The
in-place decision is made from geometry rather than from the buffer.
`_region` is an `as_strided` view of a packed shard, so it is contiguous
exactly when `(chunk_shape, strides)` is row-major packed, ignoring
extent-1 axes.

Unchanged:
- `plan_digest`, the batch fingerprint and `TransferStats`, so mixed
versions still agree on the schedule;
- `build_operations` and `_packed_rounds`, which keep their names and
signatures until deepspeedai#8967 switches.

## Tests

- `test_schedule_is_what_the_executor_runs` (2 and 4 ranks; mixed packed
and strided regions with 16-element messages) checks:
- rounds, sent, received and local-copy bytes against the executor's
`TransferStats`;
  - that both sides of every message agree;
  - the arena sizes;
  - that the batch exercises both the in-place and the staged path.
- `test_packed_regions_are_exactly_the_contiguous_views` compares the
geometric rule with `Tensor.is_contiguous` over 2000 random shapes and
strides, including extent-1 axes.
- A single-process case covers local-only batches and the argument
checks.
- Forcing every round to be staged, or dropping the extent-1 exception,
makes these tests fail.

On macOS arm64 CPU (Python 3.12, torch 2.15.0):
- The executor, external-group, planner and shard-map suites pass (263).
- All other affine checkpoint tests pass, except the four
`TestAffineUniversalCheckpointResume` cases, which fail the same way on
master in this environment (uninstalled package version).
- With deepspeedai#8967 merged on top, its analysis tests pass unchanged (59
together with the executor suite).
- pre-commit is clean.

This merges cleanly with deepspeedai#8936, deepspeedai#9010 and deepspeedai#9011. With deepspeedai#8966 and deepspeedai#8967
the only conflict is that both append a section to
`affine-transfer.rst`.

---------

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
jinyouzhi added a commit to jinyouzhi/DeepSpeed that referenced this pull request Nov 17, 2026
…calibration Part-2) (deepspeedai#8966)

## Calibrate affine transfer links per path

Stacked on deepspeedai#8809 only for its docs page (`affine-transfer.rst`); the
module itself needs nothing beyond master. Until deepspeedai#8809 merges, please
review only the commits after the deepspeedai#8809 merge.

### Why
A replica policy or cost model choosing between plans (deepspeedai#8252,
deepspeedai#8935/deepspeedai#8936) needs link costs measured on the machine it runs on.
Executor measurements show that round count and receiver sharing
dominate sync time, so per-pair numbers are not enough on their own;
this PR also measures 1→N and N→1.

### What
`deepspeed/checkpoint/affine_transfer_calibration.py`:
- `calibrate_links(sizes, repeats, warmup, path_of, fan_width,
fan_peers, group)`: collective over `group` (the default group unless
given). Returns the same raw samples on every process, with no
aggregation, for:
  - `local` copy;
  - `peer` (each ordered pair alone);
  - `fan_out` (1→N);
  - `fan_in` (N→1, receiver contention).
  
  Sizes run from 4 KiB to 256 MiB by default.
- Each message is labelled by `path_of(from_process, to_process)`.
`default_path_of` gives `local`/`peer`; `node_path_of(n)` gives
`intra_node`/`inter_node`; any callable works.
- `fit_links(calibration)`: per-path `alpha + bytes / bandwidth`, fitted
to the per-size medians; `peer_pairs` per ordered pair; fans keyed by
`(root, peers)`, with `send`/`receive` giving each root's aggregate
rate.

Weight transfers are one-way (a training process sends, a rollout
process receives), so no pattern has a process send and receive at once.

It measures only. It does not plan, select replicas, or change the
planner or executor.

### Example (one node, 8 devices, default sizes)
| pattern | B70 xccl: alpha (µs) / GB/s | L20 nccl (PCIe): alpha (µs) /
GB/s | H20 nccl (NVLink): alpha (µs) / GB/s |
|---|---|---|---|
| local | 73 / 359 | 16 / 1534* | 9 / 1881 |
| peer | 301 / 28.2 | 72 / 25.3 | 54 / 275 |
| fan_out (7 peers) | 531 / 3.4 | 141 / 3.6 | 88 / 49.6 |
| fan_in (7 peers) | 670 / 2.6 | 56 / 3.8 | 46 / 54.5 |
| run time | 88 s | 34 s | 13 s |

\* poor linear fit (max relative error 0.77); small copies are
cache-resident.

### Multi-node (2 × 8 H20, RoCE 8 × 200 Gb/s per node,
`node_path_of(8)`, 16 processes)
| pattern | alpha (µs) / GB/s | max rel. error |
|---|---|---|
| peer, intra_node | 57 / 53.8 | 0.07 |
| peer, inter_node | 97 / 16.6 (pairs 14.0–24.5) | 0.50 |
| fan_in, 7 intra-node peers: root receive | 94.6 GB/s | |
| fan_out, 8 cross-node peers: root send | 62.1 GB/s median (33–78 over
roots) | |
| run time | 72 s (+111 s for cross-node fans of width 1/4/8) | |

- The same NVLink pair measured 275 GB/s in the 8-process group and 54
GB/s in the 16-process group over RoCE (105 GB/s with NCCL over TCP), so
calibrate on the group and communication settings the transfer uses.

### Tests
`tests/unit/checkpoint/test_affine_transfer_calibration.py`:
- fit on synthetic data with known link costs, per-pair and per-peer-set
fits;
- schema and argument checks;
- 4-rank runs checking completeness, sample counts, and fan path
direction with an asymmetric `path_of`.

10 passed on 8× B70 (xccl). `pytest tests/unit/checkpoint/ -k affine`:
344 passed, 1 skipped on 8× H20 with torch 2.13.0+cu129 / NCCL 2.29.7
and with torch 2.9.1+cu129 / NCCL 2.29.7. Mutation check: flipping the
fan_in label direction fails a test. Cross-node runs used NCCL 2.29.7 on
both nodes (torch 2.13.0 and 2.9.1).

AI assistance (GitHub Copilot) was used for the multi-node runs and this
description; I reviewed the changes and the results.

### Follow-ups
- Sampled pairs for large worlds.
- Joining the samples to a plan's messages (analysis PR, stacked on this
one and deepspeedai#8809).

---------

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
Signed-off-by: iLeGend <824040212@qq.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants