ep(bench): add mscclpp high-throughput (rank-major) backend to the unified Python EP benchmark - #860
Open
Qinghua Zhou (seagater) wants to merge 57 commits into
Open
Conversation
…on APIs ep_bench_unified.py drives both mscclpp.ep.MoECommunicator and the nccl.ep (nccl4py) dispatch/combine APIs in one process through a single shared paired-benchmark loop, so the two are timed identically and emit the same '=== Summary (Low Latency) ===' block as mscclpp_ep_bench.cu. Bootstrap is MPI (mpi4py) shared by both backends. --kernel-timing adds a pure device-time pass via an in-process CUPTI Activity collector (cupti_kernel_timer.cpp), which captures the cooperative-launch LL kernels that torch.profiler mis-handles. The launch command lives in src/ext/ep/README.md under 'Unified in-process benchmark'.
…fied_ep_bench_ll_python # Conflicts: # test/python/ep/cupti_kernel_timer.cpp
ep_bench_unified.py now covers the mscclpp-vs-NCCL-EP Python API comparison in-process, so the standalone Python MoECommunicator backend (ep_bench_ll.py) launched via torchrun is redundant. - Delete test/python/ep/ep_bench_ll.py. - run_ep_bench.py: remove the 'mscclpp' (Python) --ep-lib choice, build_mscclpp_cmd, and its --mscclpp-bench/--python/--conda-*/--torch-profiler args + dead validation. --ep-lib both/all now = mscclpp-cpp + nccl-ep (all kept as alias of both). - Update README / CMakeLists / bench comments to reference ep_bench_unified.py.
Clearer naming alongside run_ep_bench.py (the C++/mpirun driver): this is the in-process Python-API driver comparing mscclpp EP vs NCCL-EP. Updated README and cross-references.
- Drop the removed get_low_latency_rdma_size_hint import (num_rdma_bytes=0). - Docstring: clarify mscclpp LL supports both CUDA-IPC (NVLink) and RDMA/IB. Within a shared node / NVLink-MNNVL domain no HCA list or fabric-IPC env is needed (IPC path); cross-domain peers use RDMA/IB with MSCCLPP_HCA_DEVICES. - Add a multi-node (shared MNNVL fabric) launch note: HPCX OMPI4 + libmpi.so.40 preload + NCCL_MNNVL_ENABLE=1.
…python.py docstring
…azhou/unified_ep_bench_ll_python # Conflicts: # src/ext/ep/README.md
…, --cupti-inproc alias)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…ndant NCCL-EP complete() The paired dispatch->combine timed loop no longer calls stream.synchronize() or comm.Barrier() between launches, so the two kernels pipeline back-to-back on the stream (CUDA events / CUPTI still bound each kernel). NCCL-EP dispatch/combine also drop the redundant ep_handle.complete() (full-mode DispatchConfig/CombineConfig make it a no-op). Docstring updated to match.
…o qinghuazhou/unified_ep_bench_ll_python # Conflicts: # test/python/ep/ep_bench_ll.py # test/python/ep/run_ep_bench.py
… (default) Adds a torch.profiler (kineto) kernel-timing path over the paired dispatch->combine loop, with a per-iteration L2 flush and a GPU-side torch NCCL all_reduce barrier (cuda._sleep + dist.all_reduce) that aligns ranks on-device -- the DeepEP bench_kineto methodology. This removes the Python-driver inter-rank recv-spin skew that inflated the CUPTI/host averages, so the reported avg matches the C++/DeepEP numbers for both dispatch and combine (no representative-min workaround needed). Controlled via env: EP_KERNEL_TIMER=kineto (default; cupti falls back to the in-process CUPTI collector) and EP_KINETO_BARRIER=nccl (default; mpi uses the host barrier). A torch.distributed NCCL group is initialized alongside MPI (MPI supplies the rendezvous).
# Conflicts: # python/mscclpp/ep/README.md # python/mscclpp/ep/low_latency.py # python/mscclpp/ep/types.py # src/ext/ep/README.md # src/ext/ep/include/api.cuh # src/ext/ep/low_latency/combine.cu # src/ext/ep/low_latency/dispatch.cu # src/ext/ep/moe_runtime.cc # test/python/ep/ep_bench_ll.py # test/python/ep/run_ep_bench.py # test/python/ep/test_low_latency_multirank.py
- Integrate DeepEP V2 (deep_ep.ElasticBuffer LOW_LATENCY) as a first-class backend via setup_deepep (plain per-expert dispatch/combine, scope-matched to dispatch_impl+copy / combine_impl+reduce). - Fix DeepEP combine recv-spin skew in the kineto timer: EP_KINETO_SEPARATE (default on) profiles dispatch and combine in separate barrier-aligned passes (mirrors DeepEP bench_kineto), collapsing per-rank skew without a mid-pipeline barrier (which crashes DeepEP multi-node, Cuda 719). - Replace --backend both with --backend all (nccl, mscclpp, deepep).
…azhou/unified_ep_bench_ll_python # Conflicts: # test/python/ep/ep_bench_ll.py
- Integrate FlashInfer trtllm_moe_alltoall (MoeAlltoAll over MNNVL) as a first-class backend via setup_flashinfer, reusing the shared routing inputs (topk_idx -> int32 expert ids, x -> BF16 hidden payload). - MoeAlltoAll is stateful (dispatch->combine phase order); use natural dispatch/combine pairing with a per-op MPI barrier so ranks enter each kernel aligned (FlashInfer in-kernel peer-readiness spin otherwise deadlocks multi-node). Force EP_KINETO_SEPARATE=0 for this backend so the timer replays the pair in order. - Make the kineto kernel-name matcher case-insensitive so FlashInfers moeA2ADispatchKernel/moeA2ACombineKernel match the dispatch/combine buckets. - Add env-gated faulthandler dump (EP_FAULTHANDLER_SECS) for hang diagnosis. - --backend choices now include flashinfer; all runs nccl, mscclpp, deepep, flashinfer. Verified 1/2/4-node (ep4/8/16): dispatch 19/27/35, combine 30/35/39 us -- matches the standalone FlashInfer bench.
- Add setup_mscclpp_ht: MoECommunicator with mode=MoEMode.HIGH_THROUGHPUT (TOKEN_MAJOR, GB200 TMA), following the cached-dispatch flow from test_intranode_multirank.py (uncached dispatch records routing layout on the handle; timed loop replays cached dispatch via previous_handle + combine). - Register as --backend mscclpp-ht (opt-in; not part of --backend all). - MSCCLPP_EP_NUM_SMS (default 20) sets HT SM count. - Verified 1-node (ep4) e256 t128 d7168 k8 BF16: dispatch 60.6, combine 87.5 us (vs LL 22.9/28.9 -- HT is throughput-optimized, LL wins at low token counts).
Make the mscclpp cuda-graph path capture dispatch+combine into a SINGLE combined CUDA graph (dispatch_fn replays the pair, combine_fn is a no-op), matching the NCCL-EP / DeepEP single-graph path and how a real serving stack replays a fused MoE step. Measured on GB200 (rank-major, 1/2/4 nodes): per-kernel kineto times are unchanged vs the previous two-graph capture (dispatch ~16/23/31 us, combine ~15/21/26 us). Only the host per-phase split changes -- the host combine timer folds into dispatch (~2.4 us combine) -- and end-to-end Total(D+C) host latency is marginally lower (one graph launch instead of two).
Adopt feature/ep modular ep-bench structure (ep_bench_common + per-backend ep_bench_*.py, thin run_ep_bench_python.py, --ep-layout, all-backend cuda-graph) and re-add the high-throughput contribution as a new module test/python/ep/ep_bench_mscclpp_ht.py (setup_mscclpp_ht). The mscclpp-ht backend is wired back into run_ep_bench_python.py: import, --backend choices, and the _SETUP registry. The NCCL-EP JIT CLI knobs removed on feature/ep are dropped (the launch command sets NCCL_EP_JIT_* via env).
The HT (HIGH_THROUGHPUT) EP backend previously supported only TOKEN_MAJOR.
Add DispatchLayout.RANK_MAJOR, which places each dispatched token at a fixed
[num_ranks, max_tokens_per_rank, hidden] slot (grouped by source rank) instead
of the compacted DeepEP prefix offset. The recv-pool combine path is unchanged:
it reconstructs each output token via sendHead/combineRecvIdx, so it is layout-
agnostic and needs no new kernel.
Changes:
- api.cuh: add RANK_MAJOR enum value; thread DispatchLayout + maxTokensPerRank
into the HT dispatch declaration (defaulted to TOKEN_MAJOR for compatibility).
- high-throughput/dispatch.cu: template dispatchKernel on DispatchLayout; for
RANK_MAJOR use the fixed stride rank*maxTokensPerRank; add a layout switch in
the launcher.
- ht_runtime.{hpp,cc}, bindings.cpp: pass layout + max_tokens_per_rank through
the runtime and Python bindings; expose RANK_MAJOR.
- python/mscclpp/ep/high_throughput.py: accept RANK_MAJOR; size the padded
[world*max_tokens] recv view; pass layout at both dispatch call sites.
- test/python/ep/ep_bench_mscclpp_ht.py: honor --ep-layout rank_major and add a
--validate round-trip that compares rank-major against a token-major reference.
Validated on GB200 (1/2/4 nodes): rank-major combine is bit-exact vs token-major
(max|diff|=0), padded recv = world*max_tokens_per_rank.
Drop test/python/ep/run_ep_bench.py (the old shell-out driver), test/python/ep/mscclpp_ep_bench.cu (the pure-C++ LL benchmark), and the in-process CUPTI kernel timer (cupti_kernel_timer.cpp + the standalone CMakeLists.txt that built them). The unified in-process Python driver (run_ep_bench_python.py) is the single entry point, and its kernel-only timing uses the torch-kineto path (EP_KERNEL_TIMER=kineto) exclusively; the --kernel-timing flag and all CUPTI plumbing are removed.
The EP benchmark runs on a single NVL72 domain (same-rack MNNVL, NVLink transport, EP_DISABLE_GIN=1, NCCL_IB_DISABLE=1). DeepEP hybrid mode adds a hierarchical RDMA + NVLink tier for multi-rail networks that this workload never exercises; enabling it only reserves extra buffer capacity and QPs (65/129 vs 17). Set allow_hybrid_mode=0 so the DeepEP config matches the transport being benchmarked. Verified bit-for-bit-equivalent kernel timing vs hybrid mode on a 2-node t=4096 h8704 run (kernel total 1349 vs 1344 us).
The previous gate restricted DeepEP CUDA-graph capture to a single node (num_ranks <= local_world), assuming its symmetric-memory kernels always crash under graph capture internode (CUDA 719). That crash is specific to the RDMA/IB scale-out (GIN) path, NOT the node count: on an all-NVLink / MNNVL fabric (EP_DISABLE_GIN=1, one NVL72 domain) capture works at any scale. Verified DeepEP CUDA-graph at 1/2/4 nodes (4/8/16 GPUs) on GB200 NVL72 -- all exit 0, no CUDA 719. Gate now keys off EP_DISABLE_GIN so graph capture is enabled on the NVLink path and only disabled for the RDMA/GIN scale-out.
Reword the misleading "legacy paired loop" comments. EP_KINETO_SEPARATE=0 is not legacy: it is REQUIRED whenever a backend captures dispatch+combine in one CUDA graph (single replay runs both phases, so the skew-free separate pass cannot isolate combine). SEPARATE=1 remains the correct mode for eager runs (it collapses combine recv-spin skew). Also correct the main-loop force comment: DeepEP manages EP_KINETO_SEPARATE itself inside setup_deepep (single-graphs on the NVLink/MNNVL path at any node count, keeps the separate pass on the RDMA/GIN eager fallback), so it is not forced in the loop; only nccl/flashinfer are forced under --cuda-graph. Comment-only, no behavior change.
Address review comment (remove sync here): the stream.synchronize() between dispatch_fn() and combine_fn() in the warmup paired loop is unnecessary -- the paired dispatch->combine already runs in order on the same stream, and the loop still syncs + barriers after combine each iteration. Also restore the kineto combine-pass prime sync that a prior commit removed (that one is kept). Verified kernel-only dispatch/combine correct for all four backends on a 1-node rank-major run.
Follow-up to the review comment (move the kernel name parse for specific libraries to the backend files). Instead of a per-backend name-substring constant (which read as boilerplate since every library happens to embed the phase word in its kernel names), each backend module now exposes a parse_kineto_kernels(key_averages) -> (dispatch_us, combine_us) that owns its librarys kernel-name knowledge and delegates the summation to a shared sum_matching_kernel_us() in ep_bench_common. run_ep_bench_python threads the active backends parser through run_backend into _kineto_kernel_us; the shared _parse helper is gone and the harness holds zero per-library kernel knowledge. Behavior-preserving: verified non-zero kernel-only dispatch/combine for all four backends on a 1-node rank-major run.
mscclpp now captures dispatch+combine into a SINGLE combined CUDA graph (combine_fn is a no-op), like nccl/flashinfer/deepep. But it was missing from the force-EP_KINETO_SEPARATE=0 tuple, so under --cuda-graph it kept the default separate two-pass whose combine pass runs the no-op and records nothing -> kineto captured 0 LL kernels. Add mscclpp to the tuple so the paired single-pass attributes per-phase kernel time by kernel name. Verified: mscclpp cuda-graph now reports Dispatch/Combine (was captured 0); all four backends report valid kernel-only times in both eager and cuda-graph.
…-only The separate two-pass timing method was labeled as if it were DeepEP-specific code. Reword to make explicit it is a generic method (adopted from DeepEP bench_kineto) that applies to every backend via the backend-supplied dispatch_fn/combine_fn closures; the loop has no per-library logic. Comment-only.
…helper Address review comment (move the cuda-graph logic out of the per-backend files). All four backends now capture dispatch+combine as ONE combined graph (one replay runs both phases, combine_fn a no-op), so the prime/sync/capture boilerplate was duplicated four times. Extract it into capture_dispatch_combine_graph() and have each backend pass its own op closures. The library-specific bits stay local in those closures: DeepEP cached do_cpu_sync=False dispatch args, NCCL-EP capture-stream refetch, mscclpp dispatch-output-to-combine handoff (shared via a holder), and FlashInfer best-effort capture plus external MPI barrier. The helper lives in ep_bench_common (not run_ep_bench_python) because the backend modules import from common; a helper they call cannot live in the importer without a circular import. Behavior-preserving: verified eager and cuda-graph kernel-only dispatch/combine for all four backends on a 1-node rank-major run.
…hon) Address review comment (move the cuda-graph logic to run_ep_bench_python; unified timing where dispatch_fn/combine_fn come from the lib and the harness does capture -> iterate -> results). The backends no longer build CUDA graphs themselves; each setup_* returns a dict with eager dispatch/combine ops plus an optional "graph" spec (capture-safe dispatch/combine, an optional pre-replay barrier, and an on-capture-failure reset). run_ep_bench_python owns a single _capture_paired_graph() that captures dispatch+combine as ONE graph for any backend, wraps replay, forces the paired kineto pass when it captures, and drops the graph before teardown. Per-library specifics stay in the backend closures: DeepEP cached do_cpu_sync=False dispatch (+ GIN transport gate), NCCL-EP capture-stream refetch, mscclpp dispatch-output-to-combine handoff, FlashInfer barrier-outside-graph (pre_replay) + best-effort rebuild (on_fail). This also generalizes the earlier mscclpp fix: the harness forces EP_KINETO_SEPARATE=0 whenever it actually captures a single graph. Verified: all four backends capture (no eager fallback) and report correct kernel-only dispatch/combine in both eager and cuda-graph on a 1-node rank-major run.
…UDA graph Address review comment (run multiple iterations inside the cuda graph: with torch.cuda.graph(g_all): for i in range(100): dispatch(); combine()). Follows the sglang bench_moe_ep.py pattern: --iters-per-graph N captures N dispatch->combine iterations INSIDE the single graph so one replay runs them all, then divides the measured host time back to per-iteration. This amortizes per-replay launch overhead and keeps the spin-waiting dispatch/ combine kernels from being inflated by per-replay launch skew. Kernel-only kineto is already per-iteration (its per-launch average divides by the kernel count, which scales with N). The harness _capture_paired_graph loops the ops inside the capture; run_backend divides host times by iters_per_graph. Default 1 preserves current behavior. Verified on a 1-node rank-major run: all four backends capture at N=1 and N=10, report consistent per-iteration numbers, and N=10 drops/tightens the host-observed times (mscclpp host Total 53.3->43.7us) as expected.
…azhou/ep_bench_update # Conflicts: # test/python/ep/ep_bench_flashinfer.py # test/python/ep/ep_bench_mscclpp.py # test/python/ep/mscclpp_ep_bench.cu # test/python/ep/run_ep_bench.py # test/python/ep/run_ep_bench_python.py
Capture 10 dispatch->combine iterations per graph by default (was 1) so the CUDA-graph path amortizes launch overhead and avoids per-replay launch skew out of the box; reported times remain per-iteration. Grouping only applies under --cuda-graph, so the non-1 default is auto-clamped to 1 for eager runs instead of erroring. Verified: default --cuda-graph groups by 10 and eager runs cleanly, both reporting consistent per-iteration mscclpp numbers.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…t_python Bring the review-comment refactors and newer feature/ep runtime from ep_bench_update onto the HT branch, keeping the high-throughput benchmark backend as the HT branch's value-add. Runtime (src/, python/mscclpp/): resolved to ep_bench_update's version, which is the newer feature/ep plus the official rank-major HT support (#857, "Support rank major") and multi-node NVL/NVLS (#855). This supersedes the HT branch's local rank-major prototype (32d569d): the C++ HT dispatch API (api.cuh, ht_runtime, bindings, dispatch.cu) and python/mscclpp/ep/ high_throughput.py now use the barrier-channel #857 implementation. Benchmark (test/python/ep/): took ep_bench_update's harness (per-backend parse_kineto_kernels, harness-owned CUDA-graph capture, --graph-group-size, removed C++ bench/CUPTI/run_ep_bench.py) and re-added the HT backend. Adapted ep_bench_mscclpp_ht.py to the new backend contract: it now returns the uniform dict {dispatch, combine, teardown, barrier, graph} and exposes parse_kineto_kernels; graph is None because the HT recv-pool path is not validated under CUDA-graph capture, so --cuda-graph runs eager via the harness fallback. mscclpp-ht is wired into the harness registry, choices and _PARSE_KINETO. Verified: the four standard backends (nccl/mscclpp/deepep/flashinfer) run eager and cuda-graph correctly through the merged harness; mscclpp-ht runs end-to-end at one node (RANK_MAJOR, host and kernel-only dispatch/combine both reported).
…backend The HT backend was left graph=None on a conservative assumption that its recv-pool / cached-dispatch path might not be graph-capturable. Empirically it IS: the cached dispatch (previous_handle=) skips notify_dispatch host wait, so capturing dispatch+combine as one graph is capture-safe. Add the graph spec (cached dispatch + combine via a _cap holder, like the mscclpp LL backend). Verified capturing RANK_MAJOR at 1 and 2 nodes on GB200, graph-group-size 1 and 10, with kernel-only times matching eager.
Both the low-latency and high-throughput backends drive the same MoECommunicator API, so keep them in one module. Move setup_mscclpp_ht into ep_bench_mscclpp.py and reuse the shared parse_kineto_kernels (the HT and LL kernel-name buckets are identical), then delete ep_bench_mscclpp_ht.py. The harness now imports setup_mscclpp_ht from ep_bench_mscclpp and maps mscclpp-ht to the same parser. Verified LL eager and HT rank-major cuda-graph still run from the merged module.
…ess import Complete the previous consolidation commit (434121a), which only recorded the deletion of ep_bench_mscclpp_ht.py because a stale pathspec made git add abort before staging the real changes. This adds setup_mscclpp_ht into ep_bench_mscclpp.py and repoints the harness import/parser map at it, so the mscclpp-ht backend resolves again.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Qinghua Zhou (seagater)
requested review from
Binyang Li (Binyang2014) and
Copilot
August 3, 2026 23:42
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the unified in-process Python EP benchmark harness to include an MSCCL++ high-throughput (HT) backend, while also simplifying the benchmark stack by removing the older standalone C++/CUPTI benchmark path. The result is a single Python harness intended to time dispatch→combine consistently across multiple EP implementations.
Changes:
- Add a new
--backend mscclpp-htimplementation (HT mode) inep_bench_mscclpp.py, and wire it into the unified runner. - Refactor CUDA-graph capture to be harness-owned via a single “paired graph” helper, with backends providing capture-safe ops via a uniform
{dispatch, combine, teardown, barrier, graph}contract. - Add per-backend Kineto kernel-name parsing helpers and centralize substring-based kernel-time summation in
ep_bench_common.py.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/python/ep/run_ep_bench.py | Deleted legacy multi-process driver that shelled out to external binaries. |
| test/python/ep/run_ep_bench_python.py | Adds mscclpp-ht, harness-owned single-graph capture, backend registry refactor, and pluggable Kineto parsing. |
| test/python/ep/mscclpp_ep_bench.cu | Deleted standalone C++ LL benchmark binary source. |
| test/python/ep/ep_bench_nccl.py | Moves graph capture responsibility to harness via graph_spec; adds Kineto parse helper. |
| test/python/ep/ep_bench_mscclpp.py | Adds HT backend (setup_mscclpp_ht), adds Kineto parsing, and adapts LL backend to the new harness contract. |
| test/python/ep/ep_bench_flashinfer.py | Adapts to harness-owned capture; adds Kineto parse helper; provides capture spec with pre_replay barrier. |
| test/python/ep/ep_bench_deepep.py | Adapts to harness-owned capture; updates graph-capture gating and adds Kineto parse helper. |
| test/python/ep/ep_bench_common.py | Adds shared sum_matching_kernel_us() helper for per-backend Kineto parsing. |
| test/python/ep/cupti_kernel_timer.cpp | Deleted in-process CUPTI timer implementation. |
| test/python/ep/CMakeLists.txt | Deleted standalone build for the removed C++ benchmark + CUPTI helper. |
Address review comment: _kineto_kernel_us reads as a confusing name. Rename to torch_profiler_kernel_us, which describes what it does (times the dispatch/combine kernels with torch.profiler). Pure rename, no behavior change.
…p8 note) - Default --graph-group-size 10 -> 50 (reviewer: "Maybe increase to 50 by default?"): capture 50 dispatch->combine iterations per graph by default, further amortizing launch overhead / launch skew. - Unify the internal name to graph_group_size everywhere (reviewer: "Different with iteration_per_group?" and "Why hard code to 1 here?"): the harness used iters_per_graph internally while the CLI arg is --graph-group-size, which read as two different concepts. Rename _capture_paired_graph / run_backend params and the effective-value local to graph_group_size; the "=1" default now reads as "no grouping" (1 iteration captured), which is why it is 1 when a backend is not graph-captured. - Correct the FP8 wording (reviewer: "check if nccl support fp8 right?"): NCCL-EP DOES support FP8 (nccl_ep device code has token_data_type 0=FP8/uint8, calculate_fp8_scales, use_fp8). The previous "NCCL-EP path is bf16 only" text implied the library cannot; reword the --dispatch-dtype help and the guard to say the NCCL-EP path IN THIS BENCHMARK is BF16-only (the harness does not plumb NCCL-EP dispatch scales yet), not the library. Verified: default --cuda-graph captures with graph_group_size=50, per-iteration numbers unchanged; the fp8 guard prints the corrected message.
…p_bench_ht_python
…nel sync)
Port the high-throughput RANK_MAJOR dispatch layout onto the current
barrier-channel HT runtime. RANK_MAJOR places each dispatched token at a
fixed [num_ranks, max_tokens_per_rank, hidden] slot grouped by source rank
instead of the compacted DeepEP prefix offset; the recv-pool combine path is
layout-agnostic and unchanged. Layout is threaded through dispatch.cu
(templated dispatchKernel), api.cuh, ht_runtime.{hpp,cc}, bindings.cpp, and
high_throughput.py. Defaults to TOKEN_MAJOR for compatibility.
…rror In single-graph CUDA-graph mode both dispatch and combine replay inside dispatch_fn() and combine_fn is a no-op, so the host-observed combine span is ~0us. Clamp comb_us to 1e-3 us so the downstream throughput division (comb_bytes / c_avg) cannot raise ZeroDivisionError. Kernel-only kineto still reports the true per-phase combine time.
…t a bug Reword the DeepEP CUDA-graph gate comment: the RDMA/IB scale-out (GIN/IBGDA) path is not graph-capturable because DeepEP internode transport drives NVSHMEM/IBGDA put-signal operations that are illegal inside a CUDA graph (CUDA 719 in symmetric.hpp). This is a documented DeepEP internode limitation, not a harness bug; we disable capture when GIN is active and run that path eagerly.
…omment DeepEP V2 (ElasticBuffer) scale-out uses NCCL GIN (GPU-Initiated Networking, backed by GDAKI/DOCA GPUNetIO on this stack), not the legacy NVSHMEM/IBGDA Buffer path. Fix the earlier comment that misattributed the graph-capture gate to NVSHMEM/IBGDA put-signal ops and symmetric.hpp. The real on-stream blocker is the dispatch CPU sync for exact recv-token counts (do_cpu_sync); cached dispatch forces it False, which is why the NVLink/MNNVL path is capture-safe.
Per review, keep a single CLI flag for the number of dispatch->combine iterations captured inside one CUDA graph. Rename the arg dest to iters_per_graph, drop the --graph-group-size alias, and update the validation message, help text, comments, and the captured-graph log line accordingly. The internal _capture_paired_graph/run_backend graph_group_size parameter (which receives the value) is unchanged.
…p_bench_ht_python
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the mscclpp high-throughput (HT) EP backend to the unified in-process
Python benchmark, on top of the review-comment follow-ups from #858. Benchmarks
all five backends through one shared harness with identical dispatch→combine
timing: mscclpp LL, mscclpp HT (TOKEN_MAJOR / RANK_MAJOR), NCCL-EP,
DeepEP V2, FlashInfer.
Changes
mscclpp-htbackend (--backend mscclpp-ht): drivesMoECommunicatorinHIGH_THROUGHPUTmode. Supports--ep-layout token_major(default) andrank_major;--validatechecks rank-majorreduces bit-exactly to the token-major reference.
ep_bench_mscclpp.pyalongside the LLsetup_mscclpp(both drivethe same
MoECommunicatorAPI), sharingparse_kineto_kernels.dispatch (
previous_handle=→ no host-side notify wait) + combine as onegraph, reusing the harness-owned
_capture_paired_graphand--graph-group-size.NVL/NVLS (Add multi-node NVL/NVLS algorithm support #855) from feature/ep; supersedes the branch's local rank-major prototype.
Validation (GB200 NVL72)
d8704 e512 k8 t4096 — all single-graph, no eager fallback. Kernel Total (D+C):
1n ~934 µs, 2n ~1406 µs; rank-major ≈ token-major (no measurable overhead).
the merged harness (numbers unchanged from ep(bench): unified in-process Python EP benchmark — review-comment follow-ups #858).