CollectiveX fixes - #2489
Open
Oseltamivir wants to merge 29 commits into
Open
Conversation
中文：让 roundtrip 在所有行中统一表示 dispatch->combine The chained roundtrip excluded staging only for fp8, so `stage` sat inside it for exactly two configurations — MoRI BF16 scale-up and FlashInfer BF16 — and outside it for the other 800+ rows. The headline therefore compared two different quantities and penalised those two. Gate the hoist on stage_device_work instead, keeping CX_FP8_CONSUME=dequant as the opt-out. Also stop two staging paths from moving padding. MoRI cast its whole cap-sized receive buffer under fp8 rather than the rows dispatch filled, which made that stage flat in T (61-114us across the decode ladder against 6-30us for BF16). FlashInfer copied its whole padded workspace plane where the kernel's own staging path skips slots past recv_counters; measured occupancy is ~0.66 at EP8 and ~0.41 at EP16.
中文：标记 roundtrip 契约变更并修正相关文档 methodology.md described the headline as containing expert-output staging, which is now the opposite of what it measures; three docstrings and one comment still described the old gate, including the test module's own premise. Bump the sweep version so the durable store can tell the two measurement generations apart, and emit stage_excluded_from_roundtrip per row: without it a MoRI BF16 row measured before this change is byte-indistinguishable from one measured after, while meaning something different. Also scope the dequant hatch's historical-reproduction claim, which no longer holds for MoRI fp8 now that its stage casts only the rows dispatch filled.
中文：在头条指标旁发布 entry-skew 区间 Entry stagger is charged to the cross-rank MAX, and how much depends on the code path AND the precision rather than only the fleet: on identical h200 low-latency decode cells the spread is ~9.3us for deepep-v2 and uccl-ep at BF16 (shared legacy Buffer path) against ~2.6us for nccl-ep, and collapses to ~2.8us for those two under FP8 where the in-kernel quantise makes dispatch self-align the ranks. So MAX taxes some rows more than others and the term cannot be subtracted in a principled way. Keep MAX as the headline — a layer is not done until its slowest rank is, and MIN would launder stagger the backend itself causes — but emit the bracket the harness already measures, and state the rule: rank on roundtrip p50 only where MAX and MIN agree, never on p99 of MAX for multi-node decode, and read isolated components as residual-wait diagnostics.
…'s LL cap 中文：将 nccl-ep HT combine 按实际接收数取切片，并对齐 MoRI 的 LL 容量 nccl-ep HT combine's staging copy is sized by the tensor it is handed -- upstream reads num_tokens = x->sizes[0] -- not by the group's buffer, so passing the whole ladder-max receive plane made it copy max(ladder) * world rows on every call regardless of T. That put a rung-independent floor under the measurement: ~55-80us on a decode leg against ~470-1295us on a prefill leg whose ladder maximum is 16x larger, while the per-token slope stayed within 12%. Slice to the count the metadata exchange already reports, which is what upstream's own ep_test does. LL keeps the full padded plane; its kernel asserts that shape. MoRI had no buffer_cap override, so its low-latency ladder ran to T=512 while every other backend stopped at 256 -- a top rung no cross-vendor comparison could use. 256 is also vLLM's DEFAULT_MAX_NUM_BATCHED_TOKENS_FOR_BATCHED_DP.
…g represents 中文：说明 nccl-ep 为何不计入路由绑定，以及 MoRI 配置代表什么 nccl-ep binds routing with a collective whose cost scales with group capacity rather than token count, so charging it per iteration would import a ladder-max-proportional term into dispatch — the artifact just removed from combine. It is bound during warm-up, matching NVIDIA's own ep_bench; low-latency has nothing to exclude because its update returns immediately and the kernel reads cached routing inside the timed dispatch. MoRI's pinned MANUAL launch config is what vLLM and SGLang actually run, so its numbers describe the engine-integrated configuration rather than MoRI's peak. Record the delta to its own tuned tables (~1.3-1.4x combine at T=256 on gfx950) as context, and why AUTO would not reproduce them.
中文：按 handle 缓存 nccl-ep 低延迟门控权重包装器 LL applies the routing gate in its combine kernel, and the adapter built a fresh nccl.ep.Tensor wrapper for the weights on every timed combine while its dispatch path cached every wrapper it uses. Building one costs a torch resolve, an np.asarray and a cybind allocation, and time_us charges host work inside the measured window, so this was a per-call tax no other backend paid — measured as roughly 3-4us of nccl-ep's small-T floor. Wrap it once in _ensure_handle with the rest, and use the same object on the oracle path so the two cannot diverge.
…ws incomparable 中文：对齐 MoRI 生产环境 warp 数，并在报表中暴露不可比因素 MoRI's combine ran 8 warps while dispatch ran 16; vLLM and SGLang both pin ONE warp_num_per_block of 16 that applies to both phases, so 16 matches the engine-integrated config exactly. 16 is also the kernel ceiling (kMaxWarpGroups 8 x kWarpsPerGroup 2, unguarded groupData[8] indexed by warpId/2), so it cannot go higher. Emit the copy counts behind the byte figures, keyed on the combine contract rather than on mode: deepep-v2, uccl-ep and nccl-ep low-latency receive one copy per (token, expert), but MoRI's IntraNodeLL deduplicates by destination rank, so a blanket "low-latency uses assignment bytes" rule would overstate MoRI by ~1.5x at EP8. routed_copies stays canonical. Surface topology and wire basis in the summary table. Two things cannot be fixed by changing what we time and were previously invisible: GB's "EP8" is two 4-GPU trays inside a 72-GPU MNNVL domain rather than 8 GPUs in one node, and two low-latency rows at the same token count move ~1.5x different combine traffic depending on which basis their kernels use.
…rnel 中文：以单一融合 kernel 计入生产环境实际支付的 fp8 量化开销 Production quantises bf16->fp8 once per forward pass, with a single fused kernel, immediately before the dispatch collective. This benchmark did it once per SHAPE in make_problem, outside every timed window, so it omitted a real cost. Moving the eager helper inside the window would have been worse than omitting it: eager is a 9-launch composite, measured on-metal at 19.2us on H100 and 53.6us on MI300X against 1.51us and 4.85us for the compiled single kernel, so charging it would publish this harness's kernel count rather than production's cost. So it is compiled, and both the wire and the oracle go through ONE callable — identity then holds by construction rather than by coincidence. Low-latency deliberately keeps the eager helper: deepep-v2 and uccl-ep quantise inside their dispatch kernel, so the cost is already charged, and the oracle payload gate compares against that kernel's bits. A global swap would have redded every low-latency fp8 cell without touching LL timing. MoRI needs no compile — its cast is already one elementwise kernel — but its LL is caller-prequantized, so it is cast in dispatch for both modes. Guarded rather than trusted: assert_quantize_identity checks bitwise equality AND per-row invariance across batch sizes, because the payload gate compares the sender's [T, hidden] quantize against the oracle's [receive_count, hidden] one. Verified on-metal for e4m3fn (H100) and e4m3fnuz (MI300X/MI325X, the arch with no prior precedent for a compiled fp32->fp8 convert). Also move logical_copies out of byte_provenance: every value there is a per-component byte breakdown, and a reader indexing it by component name must not meet a differently-shaped entry.
中文：为 flashinfer-ep 增加 FP8 dispatch Dispatch-side only: the scales ride as a fourth payload (the kernel's kMaxPayloads is exactly 4, and vLLM's own integration uses the same [values, scales, ids, weights] order), and combine stays BF16, so none of the 0.6.16+ combine-quant API is needed — the shipped 0.6.8.post1 wheel suffices with no image bump. The transport imposes no dtype whitelist on payloads; only the workspace, expert-id and metainfo tensors are type-checked. Same per-128-block e4m3 recipe as deepep-v2 and uccl-ep, which keeps the fp8 axis comparable: its scale overhead is identical to theirs, since 4 bytes per 128 elements is 1 byte per 32. Recorded honestly as `fp8-e4m3fn-blockwise-offpath` because vLLM accepts only nvfp4/mxfp8/bf16 on this transport today, so the row measures what the transport costs with the DeepSeek-V3 recipe rather than a configuration a deployment can currently select — the caveat travels in the artifact rather than only in prose. The cast pair is local rather than shared on purpose: deepep-v2 must use deep_ep's own helper because its low-latency kernel quantises in-kernel and the oracle compares those bits, and uccl-ep vendors one faithful to UCCL's. Nothing external pins FlashInfer's arithmetic, so these are three contracts rather than three copies of one.
…ndings / CollectiveX：修正方法论自相矛盾之处及三项评审发现 Adversarial review of this branch found two places where docs/methodology.md contradicted itself or the code. That file defines what a published number MEANS, so a wrong sentence there is worse than a wrong comment. - The headline percentile was stated twice, incompatibly: "the p50 of the per-iteration cross-rank MAX" in one paragraph and "measured roundtrip p99 is the headline latency" in another. On skew-heavy backends those differ by 2-5x. p99 is what the published cohorts rank on, so the p50 sentence was the wrong one; both percentiles are emitted and summarize.py prints both. - The low-latency copy-basis claim was blanket where the behaviour is not. MoRI's IntraNodeLL deduplicates per rank (its combine is an unweighted rank-sum), so the rank-deduplicated count is exact there, not a lower bound. The commit that surfaced this in the summary table left the prose asserting the incomparable thing was uniform. - FP8 `normal` dispatch now contains the quantize, which the doc never said. It also listed the FP8 backends without FlashInfer EP. Both corrected, with the comparability consequence stated: an FP8 `normal` dispatch covers quantize-plus-transport while its BF16 control covers transport alone. - `stage_excluded_from_roundtrip` is false in two unrelated situations, and reading it alone is unsafe. Documented the disambiguation: `stage` absent means nothing to stage; `stage` present with the flag false means the dequant hatch put it back in the chain. Three separate on-metal reports asked about this, which is enough evidence that the contract needed writing down. Code, all from the same review: - The FlashInfer wheel gate scraped digits, so `0.6.16rc1` and `0.6.16.dev0` read as 0.6.16 and selected the FP32 combine model against a kernel that still rounds per level. That direction can exceed COMBINE_REL_TOL and RED a correct run; the reverse is a few ulps. Now ordered with `packaging`, which sorts pre-releases below their release, with a conservative regex fallback and False on anything unparseable. - The duplicate-slot test could not tell keep-first from keep-last, because it used identical messages in both slots. Replaced the blind spot with cancelling contributions (256, 1, -256): keep-first is 0.0, keep-last is 1.0. Verified the function returns 0.0. - summarize.py raised IndexError on a document with zero measurement rows. It validates nothing by design, so it now degrades to "-" rather than losing the whole table. 91 tests pass with torch present (0 skips). 中文：对本分支的对抗性评审发现 docs/methodology.md 有两处与自身或代码矛盾。该文件定义了已发布 指标的**含义**，因此其中的错误句子比错误注释更严重。 - 头条百分位被以互不相容的方式表述了两次：一段称"逐次迭代跨 rank MAX 的 p50"，另一段称"实测 roundtrip p99 即头条延迟"。在 skew 较大的后端上两者相差 2-5 倍。已发布队列排名依据的是 p99， 因此 p50 那句是错的；两个百分位都会输出，summarize.py 也都会打印。 - 低延迟拷贝基准的表述过于笼统。MoRI 的 IntraNodeLL 按 rank 去重（其 combine 为非加权 rank 求 和），因此在该处按 rank 去重的计数是精确值而非下界。此前在汇总表中揭示该差异的提交，遗漏了正文 中"该不可比属性是统一的"这一断言。 - FP8 `normal` 的 dispatch 现已包含量化，而文档从未说明；FP8 后端列表也漏了 FlashInfer EP。两者 均已修正，并说明其可比性后果：FP8 `normal` 的 dispatch 涵盖量化加传输，而其 BF16 对照仅涵盖传输。 - `stage_excluded_from_roundtrip` 在两种互不相关的情形下均为 false，单看该字段并不安全。已补充 判别规则：无 `stage` 组件表示无需 staging；有 `stage` 组件而该标志为 false 表示 dequant 旁路把 转换放回了链路内。三份独立的实机报告都问到了这一点，足以说明该约定需要写清。 代码修改同样来自该评审： - FlashInfer wheel 版本判定此前直接抓取数字，导致 `0.6.16rc1` 与 `0.6.16.dev0` 被读作 0.6.16， 从而对仍逐层取整的 kernel 选用了 FP32 combine 模型。该方向可能超出 COMBINE_REL_TOL 并把正确的 运行判为失败；反方向仅差几个 ulp。现改用 `packaging` 排序（预发布版本排在正式版本之前），并保留 保守的正则回退，无法解析时一律返回 False。 - 重复 slot 的测试无法区分保留首个与保留末个，因为两个 slot 使用了相同的消息。现改用可相互抵消的 贡献值（256、1、-256）：保留首个得 0.0，保留末个得 1.0。已验证函数返回 0.0。 - summarize.py 在测量行为空的文档上会抛出 IndexError。该模块本就不做校验，现改为降级显示 "-"， 而不是丢失整张表。 在存在 torch 的环境下 91 项测试全部通过（0 项跳过）。
…s to / CollectiveX：让 MoRI 运行在与其固定 warp 数相匹配的缓冲模式下
Pinning the engines' `warp_num_per_block` of 16 was only half a configuration. MoRI's own tuning
tables key combine on `zero_copy`, selecting ~16 warps when the caller supplies an external input
buffer and 4-8 when it uses a registered one, and this adapter used a registered buffer. So the
previous commit took production's warp count into a mode production does not run, which matched
neither vLLM/SGLang nor any MoRI-authored artifact.
That mismatch is measured, not theoretical. On-metal A/B of warps 16 vs 8 in registered-buffer mode,
three chips, every arm fully correct:
T MI355X MI300X MI325X
<=64 -8..-10% (win) flat flat
128 +17.2/+18.3% +13.4% +16.9%
256 -0.1/-0.3% +14.5% +16.8%
512 +61.2/+62.3% +74.1% +77.9%
Roundtrip p50 at T=512 moved +38% on MI300X and +40% on MI325X. MI355X used an exact two-line revert
with two replicates agreeing within ~1pp; the gfx942 arms are cross-commit but decompose per
component, since dispatch warps were already 16 and dispatch/stage stayed matched.
So `use_external_inp_buf` moves to True on every path, which is MoRI's default, what vLLM leaves it
at, and what SGLang sets explicitly. The registered-buffer branches stay, so the mode remains a
one-line A/B.
Two consequences, both intended:
- BF16 rows no longer report a `stage` component. With an external input buffer the kernel does its
own staging copy, so handing over the dispatch output is a bare assignment with no device work.
FP8 rows still stage, because the received payload must be dequantized.
- Reading MoRI's source settled a claim this file had wrong. The external-input staging copy loops
`i < totalRecvTokenNum` over `inpTokenBuf` (intranode.hpp:542-560), so it never reads past the
receive count -- the comment asserting "the kernel reads the padded plane directly, so all of it
must be BF16" was false. FP8 now dequantizes only the filled rows on this path too; casting the
whole cap-sized plane would have been ~99.8% padding at T=1, the same waste the zero-copy branch
already documented.
methodology.md now states the full pinned tuple including the warp count, that the two settings move
together and why, and the one asymmetry: the low-latency arm has no engine config to match, because
SGLang's low-latency path pins AsyncLL at 8 warps while this suite uses IntraNodeLL (AsyncLL is
split-phase and fails silently under a single-call harness). Inheriting 16 there is a choice, and is
now labelled as one.
The A/B above measured the OLD mode. It has to be re-run in the new one before these numbers are
quoted as the cost of 16 warps generally.
91 tests pass.
中文：固定引擎所用的 `warp_num_per_block`=16 只完成了配置的一半。MoRI 自带的调优表按 `zero_copy`
索引 combine：调用方提供外部输入缓冲时选约 16 warps，使用注册缓冲时选 4-8。而本适配器此前使用注册
缓冲，因此上一个提交把生产环境的 warp 数用在了生产环境并不采用的模式下，既不匹配 vLLM/SGLang，也
不匹配任何 MoRI 官方产物。
该不匹配已实测，并非推测。在注册缓冲模式下对 warps 16 与 8 做实机 A/B，三款芯片，各组结果均完全
正确（表见上）。T=512 时 roundtrip p50 在 MI300X 上上升 38%，在 MI325X 上上升 40%。MI355X 使用
精确的两行回退并有两次重复，彼此相差约 1 个百分点；gfx942 各组为跨提交对比，但可按组件分解，因为
dispatch warps 本就是 16，且 dispatch/stage 在两组间保持一致。
因此 `use_external_inp_buf` 在所有路径上改为 True——这是 MoRI 的默认值、vLLM 未改动的值，也是
SGLang 显式设置的值。注册缓冲分支予以保留，使该模式仍可通过一行改动进行 A/B。
两项预期后果：
- BF16 行不再上报 `stage` 组件。使用外部输入缓冲时，kernel 自行完成 staging 拷贝，因此交出
dispatch 输出只是一次裸赋值，没有设备端工作。FP8 行仍保留 stage，因为收到的载荷需要反量化。
- 阅读 MoRI 源码纠正了本文件中的一处错误论断。外部输入的 staging 拷贝循环条件为
`i < totalRecvTokenNum`（intranode.hpp:542-560），绝不会读到接收计数之外，因此"kernel 直接读取
填充后的整个平面，故必须全部为 BF16"这一注释是错的。现在该路径下 FP8 同样只反量化已填充的行；
若转换整个按上限分配的平面，在 T=1 时约 99.8% 都是填充，正是零拷贝分支早已记录的浪费。
methodology.md 现已写明完整的固定配置元组（含 warp 数）、两项设置为何必须一同变动，以及唯一的不
对称之处：低延迟分支没有可对标的引擎配置，因为 SGLang 的低延迟路径固定使用 AsyncLL 且 warps=8，
而本套件使用 IntraNodeLL（AsyncLL 为分阶段实现，在单次调用的测量框架下会静默失败）。在该处沿用
16 是一种选择，现已如实标注。
上述 A/B 测量的是**旧**模式。在把这些数字当作 16 warps 的普遍代价引用之前，必须在新模式下重跑。
91 项测试通过。
…de 写入产物文件名
The result path was `{runner}_{backend}_{precision}_{phase}_{ts}-cNNN.json` with no mode segment, so
a low-latency case and a normal case sharing runner, backend, precision, phase and index resolved to
byte-identical paths. Driving several shards from one loop shares the second-resolution timestamp,
and the second case silently overwrote the first — no error, just a missing artifact.
CI hides this: every leg is its own job with its own timestamp. It surfaced on metal, where it cost
two runs before anyone noticed the files were gone rather than never written.
The existing comment already explained why precision is in the filename. Mode belongs there for
exactly the same reason, so this extends that fix rather than inventing a new scheme.
The four pinned filenames in the seam-contract tests move to `..._bf16_normal_decode_...`, and the
low-latency round-trip test now asserts its own `out` — it previously checked mode, phase and scope
but never the filename, which is precisely why a collision between it and its normal-mode sibling
was invisible to the suite.
91 tests pass.
中文：产物路径此前为 `{runner}_{backend}_{precision}_{phase}_{ts}-cNNN.json`，不含 mode 段，因此
runner、backend、precision、phase 与序号相同的 low-latency 用例与 normal 用例会解析出完全相同的
路径。若在一个循环中驱动多个 shard，它们共享秒级时间戳，后一个用例会静默覆盖前一个——没有报错，
只是产物缺失。
CI 掩盖了该问题：每条 leg 都是独立作业，拥有各自的时间戳。它在实机上暴露，并且在有人意识到文件是
被覆盖而非从未写入之前，已经浪费了两次运行。
原有注释已说明 precision 为何要写入文件名。mode 出于完全相同的理由同样应当写入，因此本次修改是对
该处修复的延伸，而非另立方案。
seam 契约测试中四处固定的文件名改为 `..._bf16_normal_decode_...`，并且 low-latency 往返测试现在
会断言自身的 `out`——它此前检查了 mode、phase 与 scope，却从未检查文件名，这正是它与 normal 模式
同胞用例之间的冲突对测试套件不可见的原因。
91 项测试通过。
…hy legs / CollectiveX：避免单用例挂起保护杀掉仅仅是较慢的正常运行
Two shards in sweep 30799122697 went red without any benchmark failure. The h100 one is provably not
a measurement fault: its uploaded artifact has both cases at status=success with every rung
passed=true, and the timeline says why it died anyway.
job start 09:05:45
decode artifact written 09:17:14 (case 0, fine)
prefill srun begins ~09:17:15 -> 900s deadline at 09:32:15
prefill artifact written 09:32:17 <-- 2 seconds past the deadline
job fails 09:32:37 <-- +20s, inside `timeout -k 30`'s grace
The case completed its measurement, wrote a correct artifact, and `timeout` killed the srun two
seconds later. The shard is red with good data attached.
b200's uccl-ep FP8 leg is the same fault with the kill made explicit: `ERROR: case 1 failed`, then
`STEP ... CANCELLED ... DUE to SIGNAL Killed` and `Force Terminated`. Case 1 is prefill in both, and
uccl-ep FP8 is the slowest FP8 path in the suite (it carries a per-iteration scale transpose on top
of the quantize), so it hit the wall on a single node where deepep-v2 needed two.
COLLX_RUN_TIMEOUT is a hang guard, not a work budget, and 900s no longer sits clear of the most
expensive legitimate case. FP8 got slower deliberately: moving the quantize inside the timed dispatch
costs ~233us vs ~126us per dispatch on GB300, so the heaviest FP8 prefill legs now land right on the
old line. That is the change working as intended, colliding with a cap never sized for it.
1800 is the value the AMD launcher had already picked for this exact reason, so this makes it the
shared default and drops the per-launcher override — one number, no drift, and a 2x margin over the
~900s a real FP8 prefill leg needs. AMD behaviour is unchanged.
This is orthogonal to whether the compiled quantize is the right call at all (still being measured):
the cap killed cases that had already finished, so it was too tight regardless of that outcome.
Not fixed here, recorded so nobody re-diagnoses them as this: b300's FP8 EP16 leg failed in
distributed rendezvous with `DistStoreError: Timed out after 601 seconds waiting for clients. 8/16
clients joined` — one node's ranks never arrived, an infrastructure fault unrelated to FP8 or to this
cap. gb300's FP8 EP16 leg failed in source preparation with a GitHub 502 on clone. Both need a
re-run, not a code change.
91 tests pass.
中文：扫描 30799122697 中有两个 shard 在没有任何基准测试失败的情况下变红。其中 h100 那个可以证明不是
测量问题：其上传的产物中两个 case 均为 status=success、所有 rung 均 passed=true，而时间线说明了它为何
仍然失败（见上表）。
该 case 完成了测量、写出了正确的产物，两秒后 `timeout` 杀掉了 srun。该 shard 带着正确数据变红。
b200 的 uccl-ep FP8 leg 是同一问题，且杀死过程更为明确：先 `ERROR: case 1 failed`，随后
`STEP ... CANCELLED ... DUE to SIGNAL Killed` 与 `Force Terminated`。两者的 case 1 都是 prefill，而
uccl-ep FP8 是本套件中最慢的 FP8 路径（除量化外还带有每次迭代的 scale 转置），因此它在单节点上就撞上
了上限，而 deepep-v2 需要两节点才会。
COLLX_RUN_TIMEOUT 是挂起保护，而非工作量预算，900 秒已不再明显高于最昂贵的正常用例。FP8 是有意变慢
的：把量化移入计时期的 dispatch 后，在 GB300 上每次 dispatch 约 233us（原约 126us），因此最重的 FP8
prefill leg 正好压在旧界线上。这是该改动按预期工作，却与一个从未为此设定的上限相撞。
1800 正是 AMD launcher 早已因同样原因选定的值，因此本次将其提升为共享默认值并移除各 launcher 中的重复
覆盖——一个数字、不再漂移，并相对真实 FP8 prefill leg 所需的约 900 秒留有 2 倍余量。AMD 行为不变。
这与"编译版量化本身是否正确"（仍在测量中）是两个独立问题：该上限杀掉的是已经跑完的 case，因此无论那个
结论如何，它都太紧了。
本次未修复、但记录于此以免被误诊为同一问题：b300 的 FP8 EP16 leg 在分布式 rendezvous 阶段失败，报
`DistStoreError: Timed out after 601 seconds waiting for clients. 8/16 clients joined`——某个节点的
rank 从未加入，属于与 FP8 及本上限无关的基础设施故障。gb300 的 FP8 EP16 leg 在源码准备阶段因 GitHub
clone 返回 502 而失败。两者需要重跑，而非代码改动。
91 项测试通过。
…BF16 的 stage 形态 On-metal validation of the external-input switch checked the artifact shape and found the claim in this file too strong. BF16 rows do not "report no `stage` component at all" -- the component is still declared, it just carries no percentiles, exactly as any backend whose staging is a bare pointer assignment already reports it (NCCL EP BF16 has always looked like this). What changes for MoRI BF16 is the percentiles and the `stage_excluded_from_roundtrip` flag, not the presence of the key. The distinction matters to a consumer: one keying on the presence of `stage` sees no change at all, while one reading percentiles or the flag sees the intended change. Saying "no component" would send the first kind looking for a key that is still there. 中文：对外部输入缓冲切换的实机验证检查了产物形态，发现本文件中的表述过强。BF16 行并非"完全不上报 `stage` 组件"——该组件仍会声明，只是不携带百分位数据，与任何 staging 为裸指针赋值的后端此前的上报 方式完全一致（NCCL EP 的 BF16 一直如此）。对 MoRI BF16 而言，变化的是百分位数据与 `stage_excluded_from_roundtrip` 标志，而非该键是否存在。 这一区别对消费方很重要：以 `stage` 键是否存在为判据的一方看不到任何变化，而读取百分位或该标志的一方 才会看到预期变化。若表述为"没有该组件"，会让前一类消费方去寻找一个其实仍然存在的键。
…ectiveX：修正版本判定辅助函数后的空行间距 Two blank lines after a top-level def, matching every other module-level function in this file. No behaviour change. 中文：顶层函数定义后保留两个空行，与本文件中其他所有模块级函数保持一致。行为无变化。
…per byte / CollectiveX：说明 FP8 量化开销是每次调用的固定成本而非按字节计
Moving the FP8 quantize inside the timed dispatch was the right call and stays. But the resulting
number is easy to misread at the bottom of the ladder, and nothing in the docs said so.
Measured from the sweep's own artifacts, DeepEP V2 decode, FP8 dispatch p50 minus its BF16 control:
SKU T=1 T=64 T=256 T=512
h100 65.4 65.5 21.6 -16.1
b200 58.8 57.9 45.4 +6.8
b300 27.3 26.6 1.4 -7.7
gb300 56.9 57.0 55.1 +50.4
Flat until transport dominates, then decaying, and negative by T=512 on two SKUs where halved
payload bytes more than repay it. FlashInfer EP's is larger (~107us on gb300); its codec is its own
and FP8 adds a fourth dispatch payload.
At T=1 the FP8 path moves FEWER bytes than BF16, so none of this is transport. It is per-call work,
and it exceeds the fused quantize's measured device time (1.5-3.6us depending on SKU) by more than an
order of magnitude — because `time_us` deliberately has no host sync before its start event, so a
near-idle stream at T=1 lets host-side launch cost land inside the window.
Compiling the quantize REDUCES this rather than causing it: an eager quantize measures 33-39us worse
per decode dispatch on h100 through the same window, which is why the fused compile stays. But
production issues one custom quantize op into an already-busy stream, so the small-T end of an FP8
`normal` row is the least production-representative number this suite emits. The doc now says to
compare FP8 against BF16 at the top of the ladder, where the charge is repaid, not at T=1.
中文：把 FP8 量化移入计时期的 dispatch 是正确的决定并将保留。但由此得到的数字在阶梯低端容易被误读，
而文档此前对此毫无说明。
依据扫描自身产物测得（DeepEP V2 decode，FP8 dispatch p50 减去其 BF16 对照，见上表）：在传输开销占据
主导之前基本持平，随后衰减，到 T=512 时在两款 SKU 上已转为负值——此时载荷字节减半带来的收益已超过该
开销。FlashInfer EP 的该开销更大（gb300 上约 107us）：其编解码是自有实现，且 FP8 会增加第四个
dispatch 载荷。
在 T=1 时，FP8 路径搬运的字节数比 BF16 **更少**，因此这部分完全不是传输开销，而是每次调用的固定
开销，并且比融合量化自身的实测设备时间（各 SKU 1.5-3.6us）高出一个数量级以上——原因是 `time_us`
有意在起始事件前不做主机同步，因此 T=1 时近乎空闲的流会把主机侧的启动开销计入窗口内。
编译量化是**降低**而非造成该开销：在同一窗口下，eager 量化在 h100 上每次 decode dispatch 要慢
33-39us，这正是保留融合编译的原因。但生产环境是向一个本已繁忙的流下发一个自定义量化算子，因此 FP8
`normal` 行在小 T 端是本套件给出的最不具生产代表性的数字。文档现已说明：应在阶梯高端（该开销已被
收回处）而非 T=1 处比较 FP8 与 BF16。
…ctiveX：将新增注释精简至与周边代码一致的密度 A density check on this branch's additions: 44 comment lines against 45 code lines, a ratio of 0.98, where the files being edited sit at 0.13 (ep_harness.py), 0.15 (ep_flashinfer.py), 0.16 (common.sh, summarize.py) and 0.42 (ep_mori.py). Between 2.3x and 6.5x the surrounding code. The excess was measured tables and incident history restated inline -- the warps 16-vs-8 numbers across three chips, the ~233us-vs-~126us dispatch figures, the artifact-written-2s-past-deadline timeline, the CI-gets-its-own-timestamp narrative. All of it already lives in the commit that introduced it and, where a reader of published numbers needs it, in methodology.md. A code comment is the wrong place for a third copy: it is the copy that goes stale silently. What stays is the part a reader of the code cannot derive from the code: that `use_external_inp_buf` and `combine_warps` have to move together because MoRI's tables key combine on `zero_copy`; that the kernel's staging copy is bounded by `totalRecvTokenNum` so only filled rows need converting; that the timeout is a hang guard rather than a work budget; that the version gate needs real ordering because the safe failure direction is False. Now 30 comment lines against the same 45 code lines, ratio 0.67. Still above ambient, which I think is right for changes whose justification is upstream kernel behaviour a reader cannot see from here. No behaviour change. 91 tests pass. 中文：对本分支新增内容做了注释密度检查：44 行注释对 45 行代码，比例 0.98；而被修改文件自身的比例为 0.13（ep_harness.py）、0.15（ep_flashinfer.py）、0.16（common.sh、summarize.py）与 0.42 （ep_mori.py）。即为周边代码的 2.3 至 6.5 倍。 超出部分是把实测数据表与问题经过原地复述了一遍——三款芯片上 warps 16 与 8 的对比数字、约 233us 对 约 126us 的 dispatch 数据、产物在截止时刻后 2 秒写出的时间线、CI 每条 leg 各有独立时间戳的来龙去脉。 这些内容已存在于引入它们的提交中，并且在已发布指标的读者需要时也存在于 methodology.md 中。代码注释 不该是第三份副本：它恰恰是会悄然过期的那一份。 保留下来的是读者无法从代码本身推导出的部分：`use_external_inp_buf` 与 `combine_warps` 必须同步变动， 因为 MoRI 的调优表按 `zero_copy` 索引 combine；kernel 的 staging 拷贝以 `totalRecvTokenNum` 为界， 因此只需转换已填充的行；该超时是挂起保护而非工作量预算；版本判定需要真正的版本序，因为安全的失败方向 是 False。 现为 30 行注释对同样的 45 行代码，比例 0.67。仍高于周边水平——对于其依据是读者在此处看不到的上游 kernel 行为的改动，我认为这是合适的。 行为无变化。91 项测试通过。
…es / CollectiveX：记录 warps 与缓冲模式配对的实测结果 The doc carried only half the evidence: the cost of 16 warps in the WRONG buffer mode. On-metal A/Bs across all three CDNA chips now cover the mode the engines actually run, and the sign flips. In external-input mode, 16 warps beats 8 by 14-19% combine at T=128, 26-27% at T=256, and 9-14% at every prefill rung including T=8192 — the true top of the ladder, which a decode-only A/B would have missed entirely. MI355X reads a tie at T=512 where both gfx942 chips keep a 5-7% edge, so the T=512 tie is a local quirk rather than the win petering out. Below T=32, 16 gives up 0.2-2.5us; that is the only range where 8 is ahead. Every arm was correct at every rung on every chip, so this is a throughput question and not a correctness one. Two smaller corrections from the same runs: - BF16's `stage` component is not merely "declared with no percentiles" but an explicit unavailable marker: availability unavailable, null percentile block, zero sample count. Stated that way now, because a consumer distinguishing "absent" from "present but empty" needs to know which it is. - How far the engine config sits off MoRI's achievable peak is arch-dependent and only partly measured. MI355X, with the registered mode's excluded BF16 stage added back so the cross-mode comparison is not flattered, puts a registered buffer at 8 warps 15% ahead at T=512 decode and 8% ahead at T=8192 prefill. That did not reproduce as a clear win on gfx942, so the doc gives 0-15% as a range rather than a single number, and says plainly that the faster configuration is one no engine runs. No code change. 91 tests pass. 中文：文档此前只记录了一半证据：16 warps 在**错误**缓冲模式下的代价。现在三款 CDNA 芯片的实机 A/B 已覆盖引擎真正使用的模式，结论符号发生反转。 在外部输入缓冲模式下，16 warps 的 combine 在 T=128 领先 8 约 14-19%，T=256 领先 26-27%，并在包括 T=8192（阶梯真正的顶端，仅测 decode 的 A/B 会完全遗漏该点）在内的每个 prefill 档位领先 9-14%。 MI355X 在 T=512 呈现持平，而两款 gfx942 芯片仍保持 5-7% 优势，因此 T=512 的持平属于局部特例，而非 优势逐渐消失。在 T<32 时 16 会损失 0.2-2.5us，这是 8 唯一领先的区间。所有芯片、所有档位、所有分组 的正确性均通过，因此这是吞吐问题而非正确性问题。 同批运行带来的两处小修正： - BF16 的 `stage` 组件并非仅"已声明但无百分位"，而是显式的不可用标记：availability 为 unavailable、 百分位块为 null、样本数为 0。现按此表述，因为需要区分"缺失"与"存在但为空"的消费方必须知道是哪一种。 - 引擎配置距 MoRI 可达峰值有多远，取决于架构且仅部分测得。在 MI355X 上，将注册缓冲模式被排除的 BF16 stage 计回以避免跨模式比较失真后，注册缓冲配 8 warps 在 T=512 decode 领先 15%、在 T=8192 prefill 领先 8%。该结果在 gfx942 上未复现为明确优势，因此文档给出 0-15% 的区间而非单一数值，并明确说明更快 的那个配置是没有任何引擎会采用的配置。 无代码改动。91 项测试通过。
…lectiveX：排除存在 GPU 温度降频故障的 B200 节点
`b200-dgxc-deepep-v2-ll-fp8-n1` was killed by the per-case hang guard twice, at 900s and again at
1800s, and looked like a genuine wall-clock pathology in that backend and precision. It is one bad
GPU. Sampled from inside an allocation on gpu-2-6:
index, clocks.sm, temperature.gpu, power.draw, clocks_event_reasons.active
0, 1965 MHz, 34, 269.15 W, 0x0
...
7, 120 MHz, 93, 302.26 W, 0x20 <- SW Thermal Slowdown
GPU 7 sits at 120 MHz and 93 C while its seven siblings run 1965 MHz at 34-39 C, across 23 samples
over four minutes. 93 C at ~294 W on a B200 is a cooling or sensor fault, not a workload effect.
Every EP collective is a barrier across all eight ranks, so the leg runs at rank 7's pace: 3.4-7.8s
per trial against 0.445s on a healthy node, projecting ~34 min — which is the 32 min that got killed.
Both CI kills landed on gpu-2-6; the run that passed on 2026-08-02 was on gpu-2-9.
True case time on a healthy node is fp8 2m39s and bf16 1m04s, a ratio of 2.5x. The "10x" that made
this look pathological came from dividing a kill by a pass, which is not a ratio — a killed case has
no duration, only a lower bound.
Excluding the node follows the h100-dgxc precedent in this file, which already drops nine unhealthy
pods. This is a workaround, not a repair: gpu-2-6 needs a Slurm drain and a hardware/cooling check on
its GPU 7, and the exclusion should come back out once that is done.
Note the exclusion does NOT rescue an in-flight sweep, which uses the config frozen at dispatch.
Falsified along the way, so nobody re-runs them: a recompile loop (dynamo counters total=5 ok=5), an
expensive fp8 oracle (oracle_pre 3.29s at T=1, <=0.01s at every other rung), a regression in the
`_ll_dequant_static` compile (stage p50 230-247us throughout), one dominant rung (Pass 2 flat, all 36
rung-by-component cells 2.75-3.85s), and memory (flat at 10.18 GiB/GPU, no OOM).
中文：`b200-dgxc-deepep-v2-ll-fp8-n1` 曾两次被单用例挂起保护杀掉（900 秒与 1800 秒），看起来像该后端
与精度组合下真实存在的墙钟时间病态问题。实际原因是一块故障 GPU。在 gpu-2-6 的分配内采样（见上表）：
GPU 7 停在 120 MHz、93 C，而其余七块运行在 1965 MHz、34-39 C，四分钟内 23 次采样均如此。B200 在约
294 W 下达到 93 C，属于散热或传感器故障，而非负载所致。
每次 EP 集合通信都是跨全部八个 rank 的屏障，因此整条 leg 以 rank 7 的速度运行：每个 trial 为
3.4-7.8 秒，而健康节点为 0.445 秒，据此推算约 34 分钟——正是被杀掉的那 32 分钟。两次 CI 被杀都发生在
gpu-2-6；2026-08-02 通过的那次运行在 gpu-2-9。
健康节点上的真实用例时间为 fp8 2 分 39 秒、bf16 1 分 04 秒，比值 2.5 倍。让人误以为存在病态问题的
"10 倍"来自用一次被杀的时长除以一次通过的时长，而这并不构成比值——被杀的用例没有时长，只有下界。
排除该节点沿用本文件中 h100-dgxc 的既有做法（其已排除九个不健康 pod）。这是权宜之计而非修复：
gpu-2-6 需要 Slurm drain 并对其 GPU 7 做硬件/散热检查，修好后应移除该排除项。
注意：该排除**不会**挽救正在运行中的扫描，后者使用派发时冻结的配置。
…arm-up / CollectiveX：明确 FP8 staging 的性质，并停止在预热中重复执行
Two changes, both from the same realisation: under FP8, `stage` is not a phase a serving stack has.
Its work is converting the received FP8 payload into the BF16 that combine sends, and production does
not do that as a separate step -- the FP8 goes into the expert GEMM, which reads FP8 operands natively
and emits BF16, and that output is what combine receives. This suite does not run the expert GEMM
because it measures the collective and not the layer, so `stage` stands in for it.
1. The doc now says that plainly, and adds the consequence that matters to a consumer: `stage` must
not be summed into a total or compared between backends, because each adapter converts a different
amount -- DeepEP V2 a padded low-latency plane, MoRI only the received rows, FlashInfer only the
filled slots. One component name, three different quantities. It also names the one production path
that DOES pay a separate materialised dequant (vLLM's quant-format mismatch fallback), which is
what `CX_FP8_CONSUME=dequant` models and why that is not the default.
2. `warm()` no longer stages on every iteration. Where staging is excluded from the chain, the timed
roundtrip stages nothing at all, so warming it 32 times per component per trial rehearsed a path
the measurement never takes -- and at ~247us against a 61us roundtrip that was the largest single
cost in an FP8 leg, roughly 34,816 dequant calls per rung. It now stages once and reuses the
payload, which is exactly what `benchmark_roundtrip` already does in the timed path on the same
documented reasoning ("routing is fixed for a ladder point, so the same staged tensor is valid for
every iteration"). So warm-up now resembles the region it warms instead of diverging from it.
`benchmark_stage` opts back in via `stage_every=True`, because there staging IS the timed operation
and starving its warm-up would change the number it reports. The `dequant` hatch keeps staging every
iteration too, since that configuration genuinely has the conversion in the chain.
Expected saving is about 73% of the dequant calls per rung; the remainder is `benchmark_stage`'s own
warm-up plus the untimed `pre` that has to feed each timed combine. No published number changes --
`stage` was already excluded from `roundtrip` -- so this is CI wall-clock only.
Three tests pin the hoist, and they discriminate: reverting it fails them with `5 != 1` rather than
passing quietly, which is the failure mode a staging change of this shape usually has. They stub torch
so they run without a GPU. 94 tests pass.
中文：两项改动源于同一认识：在 FP8 下，`stage` 并非服务栈中真实存在的阶段。它的工作是把收到的 FP8
载荷转换为 combine 所发送的 BF16，而生产环境并不把这作为独立步骤——FP8 直接进入 expert GEMM，后者
原生读取 FP8 操作数并输出 BF16，该输出才是 combine 收到的内容。本套件不运行 expert GEMM（它测量的是
集合通信而非整层），因此 `stage` 只是其替身。
1. 文档现已明确说明这一点，并补充了对消费方最重要的推论：`stage` 不可计入总和，也不可在后端之间比较，
因为各适配器转换的数据量并不相同——DeepEP V2 为补齐后的低延迟平面，MoRI 仅为已接收的行，
FlashInfer 仅为已填充的槽位。同一个组件名对应三种不同的量。文档同时指出确实会付出独立物化反量化
代价的那条生产路径（vLLM 的量化格式不匹配回退），这正是 `CX_FP8_CONSUME=dequant` 所建模的情形，
也是它并非默认值的原因。
2. `warm()` 不再在每次迭代都执行 staging。当 staging 被排除在链路之外时，计时的 roundtrip 完全不做
staging，因此按每组件每 trial 预热 32 次去重复该路径，等于预热了测量根本不会走的路径——而在约
247us 对 61us roundtrip 的量级下，这是 FP8leg 中最大的单项开销，每档位约 34,816 次反量化调用。
现改为 staging 一次并复用其载荷，这与 `benchmark_roundtrip` 在计时路径中早已采用的做法完全一致
（"给定阶梯点的路由是固定的，因此同一 staged 张量对每次迭代都有效"）。预热由此变得与其所预热的
区域相似，而非与之背离。
`benchmark_stage` 通过 `stage_every=True` 重新启用逐次 staging，因为在该处 staging 就是被计时的
操作，削减其预热会改变它上报的数值。`dequant` 旁路同样保持逐次 staging，因为该配置下转换确实位于
链路内。
预计可减少每档位约 73% 的反量化调用；其余部分为 `benchmark_stage` 自身的预热，以及必须为每次计时
combine 供数的未计时 `pre`。不改变任何已发布指标——`stage` 本就已被排除在 `roundtrip` 之外——因此
这仅影响 CI 的墙钟时间。
三项测试锁定该 hoist，且具备区分能力：回退该改动会以 `5 != 1` 失败而非静默通过，而后者正是此类
staging 改动通常的失效模式。测试对 torch 做了打桩，因此无需 GPU 即可运行。94 项测试通过。
…eX：拒绝包含降频 GPU 的资源分配 A B200 node had GPU 7 pinned at 120 MHz and 93 C while its seven siblings ran 1965 MHz at 34-39 C, with both SW and HW thermal slowdown Active and a cumulative slowdown counter of 55.5 hours. Its T.Limit readings were NEGATIVE, i.e. above both the slowdown and the shutdown threshold. Every collective is a barrier across all ranks, so that one device paced the entire leg: 7.1-9.0s per trial against 0.445s on a healthy node. The plateau ratio of 17.4x tracks the clock ratio of 16.4x almost exactly. The case was killed by the wall-clock guard twice — at 900s and again after the guard was raised to 1800s — having printed no rungs and produced no artifact, which is indistinguishable from a hang in that backend and precision. Its real cost is 2m39s. The launcher already validates each allocation (network profile everywhere, CUDA context on b300) and already excludes a rejected node and retries elsewhere. This adds a GPU health check to that chain, so the same fault costs seconds and one retry instead of thirty minutes and a misdiagnosis. It is retryable on every SKU: a throttled device is never this leg's to tolerate. Design notes, both learned from the incident: - **The throttle FLAG is the signal, not the clock.** This runs while the allocation is idle, and an idle B200 also reads 120 MHz — the same number the clamped one reports under load. A clock threshold cannot separate health from idleness; the flag can. - **Temperature is a second, independent signal**, because the flag can clear between samples while the fault persists. - **It fails OPEN** on a missing `nvidia-smi`, a non-zero exit, or unparseable output. A check that blocks legs when it cannot read the hardware is worse than the fault it looks for. Parsing is split from the I/O (`gpu_health_faults`) so it is testable without hardware. Six tests cover healthy, both flags, either flag alone, heat with no flag, and four unreadable shapes. One of them pins the specific trap that "Active" as a substring also matches "Not Active" and would pass every fault straight through. Wired into `launch_single-slurm.sh`, which covers h100/h200/b200/b300. `launch_gb-nv.sh` is a one-line follow-up if wanted; the AMD launchers would need the rocm-smi equivalent. 100 tests pass. 中文：某 B200 节点的 GPU 7 被固定在 120 MHz、93 C，而其余七块运行在 1965 MHz、34-39 C，SW 与 HW 温度 降频均为 Active，累计降频计数达 55.5 小时。其 T.Limit 读数为**负值**，即已超过降频与关机两个阈值。 每次集合通信都是跨全部 rank 的屏障，因此这一块设备决定了整条 leg 的节奏：每 trial 7.1-9.0 秒，而健康 节点为 0.445 秒。17.4 倍的平台期比值与 16.4 倍的时钟比值几乎完全吻合。该用例被墙钟保护杀掉两次——先是 900 秒，随后在保护放宽至 1800 秒后再次被杀——且未打印任何档位、未产出任何产物，这与该后端与精度下的 挂起完全无法区分。其真实耗时为 2 分 39 秒。 launcher 本就会校验每次分配（各处校验网络画像，b300 额外校验 CUDA 上下文），并且本就会排除被拒节点并 在别处重试。本次在该链路中加入 GPU 健康检查，使同类故障的代价从三十分钟加一次误诊，变为数秒加一次 重试。该项在所有 SKU 上均可重试：降频设备从不属于本条 leg 应当容忍的范围。 两点设计取舍均来自本次事件： - **信号是降频标志而非时钟。** 该检查在分配处于空闲时运行，而空闲的 B200 同样读到 120 MHz——与被钳制 的那块在负载下报告的数值相同。时钟阈值无法区分健康与空闲，标志可以。 - **温度作为第二个独立信号**，因为标志可能在两次采样之间清除而故障仍然存在。 - **失败时放行**：`nvidia-smi` 缺失、非零退出或输出无法解析时均视为健康。一个在读不到硬件时就阻塞 leg 的检查，比它所要寻找的故障更糟。 解析逻辑已与 I/O 分离（`gpu_health_faults`），因此无需硬件即可测试。六项测试覆盖健康、双标志、单标志、 仅高温无标志，以及四种无法解析的形态。其中一项专门锁定一个陷阱：把 "Active" 当作子串匹配会同时命中 "Not Active"，从而让所有故障直接通过。 已接入 `launch_single-slurm.sh`（覆盖 h100/h200/b200/b300）。`launch_gb-nv.sh` 如需接入只是一行改动； AMD 的 launcher 需要 rocm-smi 的对应实现。 100 项测试通过。
…from / CollectiveX：记录选定 combine 模型所依据的库版本 `implementation.combine_reduction` exists, by its own comment, so that "a wheel bump [does not] silently change the arithmetic behind `passed` with no trace". It records the outcome but not the input: a reader can see the oracle used the slot-tree model, but not which wheel selected it, and so cannot distinguish a correct selection from a mis-parse of the version string. That is not hypothetical for this backend. FlashInfer EP picks its reduction from `flashinfer.__version__`, and the parse behind that choice was rewritten in 8668338 precisely because the old one read `0.6.16rc1` as 0.6.16 and would have modelled FP32 against a kernel that rounds per level. On-metal validation of the rewrite had to read the wheel string out of the image's dist-info to check it, because the artifact does not carry it. `library_version` is additive and opt-in via `getattr`, so it is None for the four backends that do not report one; FlashInfer EP now sets it. No other backend's measurement SEMANTICS depend on its library version today — for the rest a version change alters performance, not what the oracle models — so this deliberately does not become a required field. Verified on metal at a8c0411 before this commit: the shipped wheel is flashinfer_python-0.6.8.post1 and both EP16 BF16 artifacts recorded `combine_reduction=topk-slot-tree`, i.e. the rewritten gate selects correctly on hardware. That check needed a dist-info read; after this commit the artifact answers it alone. 100 tests pass. 中文：`implementation.combine_reduction` 之所以存在，按其自身注释所述，是为了让"wheel 版本变动不会在 无任何痕迹的情况下悄然改变 `passed` 背后的算术"。但它只记录了结果而未记录输入：读者能看到 oracle 使用 了 slot-tree 模型，却看不出是哪个 wheel 选中了它，因而无法区分正确选择与版本字符串的误解析。 对该后端而言这并非假设。FlashInfer EP 依据 `flashinfer.__version__` 选择其 reduction，而该选择背后的 解析逻辑正是在 8668338 中被重写的——原因恰恰是旧逻辑会把 `0.6.16rc1` 读成 0.6.16，从而对一个逐层 取整的 kernel 套用 FP32 模型。对该重写的实机验证不得不从镜像的 dist-info 中读取 wheel 字符串来核对， 因为产物本身并不携带它。 `library_version` 通过 `getattr` 实现为附加且可选字段，因此对未上报该值的四个后端为 None；FlashInfer EP 现已设置该值。目前没有其他后端的测量**语义**依赖其库版本——对其余后端而言版本变化影响性能，而非 oracle 所建模的内容——因此本次有意不将其设为必填字段。 本提交前已在 a8c0411 于实机验证：所用 wheel 为 flashinfer_python-0.6.8.post1，且两个 EP16 BF16 产物均记录 `combine_reduction=topk-slot-tree`，即重写后的判定在硬件上选择正确。该核对此前需要读取 dist-info；本提交之后，产物本身即可回答该问题。 100 项测试通过。
…lectiveX：排除存在 GPU 温度降频故障的 h100 节点
Second node found the same way in a day, on a different cluster. `hpc-gpu-1-2` GPU 3, sampled from
inside the allocation under load:
GPU 3 570-690 MHz 87 C sw_thermal_slowdown Active
GPUs 0-2,4-7 1980 MHz ~40 C no flags
`hpc-gpu-1-3`, measured alongside as a control, is uniformly healthy: all eight at 1980 MHz, 48-62 C,
no flags.
It was found while chasing an apparent uccl-ep prefill regression — compiled quantize losing 943us at
T=8192 against eager — which had its two arms on different nodes, with the losing arm on this one. A
clamp that deepens with load reproduces that shape exactly: heaviest at the top prefill rung, invisible
at decode. So the "regression" is most likely this node, and the swapped-arm rerun will confirm it.
Joining gpu-2-6 (b200, GPU 7) in the exclusion lists. Both are workarounds pending a drain and a
cooling/sensor check; both should come back out once fixed.
Worth recording as a general rule: **suspect the node whenever an A/B's arms ran on different hosts.**
Both faults were one-directional, load-proportional, and all-green — a shape indistinguishable from a
real code regression from the measurement side alone.
Not excluded, because not confirmed: hpc-gpu-1-6 and hpc-gpu-1-14 produced a rendezvous TCPStore
timeout in one sweep leg. That is a single observation and a different signature (network, not thermal),
and h100 is already down to 10 usable pods of 20.
100 tests pass.
中文：一天之内以同样方式在另一个集群上发现第二个故障节点。`hpc-gpu-1-2` 的 GPU 3，在分配内于负载下
采样（见上表）。作为对照同时测量的 `hpc-gpu-1-3` 完全健康：八块 GPU 均为 1980 MHz、48-62 C、无标志。
该节点是在排查一个看似存在的 uccl-ep prefill 回归时发现的——编译版量化在 T=8192 时比 eager 慢 943us
——而该对比的两组分别运行在不同节点上，且落后的那一组正好在此节点。随负载加深的降频恰好能复现这一
形态：在 prefill 最高档位最严重，在 decode 处不可见。因此该"回归"很可能就是这个节点，交换分组的重跑
将予以确认。
现与 gpu-2-6（b200，GPU 7）一同加入排除列表。两者都是等待 drain 与散热/传感器检查期间的权宜之计，
修复后都应移除。
值得记录的一般规则：**当 A/B 的两组运行在不同主机上时，应首先怀疑节点。** 两次故障都是单向、随负载
成比例、且全部通过——仅从测量侧看，这与真实的代码回归无法区分。
未排除（因未确认）：hpc-gpu-1-6 与 hpc-gpu-1-14 在某条扫描 leg 中出现 rendezvous TCPStore 超时。
那是单次观察且特征不同（网络而非温度），且 h100 的可用 pod 已从 20 降至 10。
100 项测试通过。
… / CollectiveX：修正对抗性评审在我自己改动中发现的四处缺陷 All four are mine, from the GPU-health gate and the warm-up hoist. **Tests appended past the `__main__` guard, in two files.** `cat >>` put `WarmStaging` and `GpuHealthProbe` BELOW `if __name__ == "__main__": unittest.main()`, which raises SystemExit before those classes are defined. `unittest discover` imports the module and collected them, so CI was green and the omission was invisible — but running either file directly silently skipped them: test_roundtrip_staging 9 of 12, test_runtime 41 of 50. Anyone iterating on these files got false green on exactly the tests written to pin the new behaviour. Both guards now sit at EOF; direct execution collects 12 and 50, discover still 103. **The health gate could not prove it saw any GPUs.** It emitted nothing on success, so a gate gone BLIND — zero visible devices, or a driver old enough to spell the fields `clocks_throttle_reasons.*` — wrote an empty log indistinguishable from one that inspected eight healthy GPUs. That is precisely the "looks like protection" failure the gate exists to avoid. It now emits `gpu-health-checked gpus=N`, which makes per-cluster verification possible rather than assumed. Three tests pin the healthy, faulty and no-binary paths. **The probe relied on an undocumented default to see the GPUs at all.** It ran `srun` without `--gres`, trusting a step to inherit the job's GRES. That is Slurm's documented default and the allocation is `--exclusive`, so it worked — but a site config or a stray `SLURM_STEP_GRES` would have blinded it silently, and the sibling cuda-context probe already passes `--gres=gpu:N`. Now it does too, which settles the question by construction instead of by argument. `--time=5` bounds the other end: `nvidia-smi` can wedge uninterruptible on exactly the sick hardware this looks for, and Python's own timeout cannot reap a process in D-state, so the "seconds instead of thirty minutes" claim inverted on the worst case it targets. **A doc comment ended up on the wrong function.** Inserting the gate split `collx_validate_cuda_context_on_job` from its comment, leaving "A clean nvidia-smi inventory does not prove that a prior cancelled workload released every CUDA context" heading a function that IS an nvidia-smi inventory. Restored, and the gate has its own. Also corrected a docstring the hoist made stale: `_ll_dequant_static` said the dequant "runs in every timed component's warmup", which stopped being true when warm-up started staging once. 103 tests pass; direct execution of each test file now collects everything it defines. 中文：四处缺陷均出自我自己的改动（GPU 健康门与预热 hoist）。 **两个文件中的测试被追加到了 `__main__` 保护之后。** `cat >>` 把 `WarmStaging` 与 `GpuHealthProbe` 放到了 `if __name__ == "__main__": unittest.main()` 之下，而后者会在这些类定义之前抛出 SystemExit。 `unittest discover` 通过导入模块仍能收集到它们，因此 CI 是绿的、遗漏不可见——但直接运行这两个文件会 静默跳过：test_roundtrip_staging 为 9/12，test_runtime 为 41/50。在这些文件上迭代的人，恰好在为新行为 把关的测试上得到虚假的绿色。现两处保护均置于文件末尾；直接执行分别收集 12 与 50 项，discover 仍为 103。 **健康门无法证明自己看到了任何 GPU。** 它在成功时不输出任何内容，因此一个已经"失明"的门——没有可见 设备，或驱动版本老到字段名为 `clocks_throttle_reasons.*`——写出的空日志与检查了八块健康 GPU 的情形 完全无法区分。这正是该门本应避免的"看起来像保护"的失效模式。现会输出 `gpu-health-checked gpus=N`，使按集群核验成为可能而非只能假定。三项测试分别锁定健康、故障与无二进制 三条路径。 **该探针原本依赖一个未写明的默认行为才能看到 GPU。** 它调用 `srun` 时未传 `--gres`，依赖步骤继承作业 的 GRES。这确实是 Slurm 的默认行为且分配为 `--exclusive`，所以能工作——但站点配置或残留的 `SLURM_STEP_GRES` 会让它静默失明，而同类的 cuda-context 探针本就传了 `--gres=gpu:N`。现同样传入， 从构造上而非论证上解决该问题。`--time=5` 约束另一端：`nvidia-smi` 恰好可能在该门所针对的故障硬件上 陷入不可中断等待，而 Python 自身的超时无法回收处于 D 状态的进程，从而使"数秒而非三十分钟"的论断在其 针对的最坏情形下反转。 **一段文档注释落到了错误的函数上。** 插入该门时把 `collx_validate_cuda_context_on_job` 与其注释分开， 使"干净的 nvidia-smi 清单并不能证明先前被取消的负载已释放每个 CUDA 上下文"这句话，落在了一个**本身 就是** nvidia-smi 清单的函数之上。已恢复，并为该门另写注释。 同时修正了因 hoist 而过期的一处 docstring：`_ll_dequant_static` 原称该反量化"在每个计时组件的预热中 运行"，而预热改为只 stage 一次后该说法已不成立。 103 项测试通过；每个测试文件直接执行时现均能收集其定义的全部测试。
…ollectiveX：记录健康门无法据以判定的温度离散度 On-metal thermal tracing of the h100 fault showed the gate's absolute temperature arm is unreachable for that fault class. GPU 3 peaked at **87 C** under load — H100 engages software thermal slowdown at ~86-87 C, so a clamped H100 never crosses a 90 C limit. And at pre-flight the allocation is idle, so the throttle flag is clear too. Both halves of the gate are blind to a load-dependent clamp on that architecture. The one signal that WAS visible at pre-flight was relative: the sick GPU idled at 55 C against ~30 C for every sibling on both nodes. Measured healthy references, now on record: 50-66 C under load on h100 (~16 C spread), 34-39 C on b200, and node2's healthy siblings at 39-43 C — putting GPU 3's 87 C about 44 C above its own siblings. So `gpu-health-checked` now carries `hottest/median/spread`. Deliberately REPORTED, not gated: I have n=1 for idle spread predicting an under-load clamp, and a rejection threshold picked from one incident would risk failing healthy allocations — which on three consecutive attempts kills the leg, a worse outcome than the fault. Instrument first, gate once the distribution is known. `gpu_temperature_spread` is a pure function beside `gpu_health_faults`, so promoting it later is a one-line change in `validate_gpu_health`. Three tests cover the sick spread, its appearance in the marker, and the unreadable cases. One of them caught my own arithmetic: I expected a median of 34 for eight temperatures when it is the 5th sorted value, 35. 106 tests pass; direct execution of the file collects all 53 it defines. 中文：对 h100 故障的实机温度追踪表明，该门的绝对温度判据对这类故障不可达。GPU 3 在负载下峰值为 **87 C**——H100 在约 86-87 C 即启用软件温度降频，因此被钳制的 H100 永远不会越过 90 C 阈值。而在预检 时分配处于空闲，降频标志同样为清空。在该架构上，门的两个判据对随负载出现的钳制都是盲的。 预检时**确实**可见的唯一信号是相对量：故障 GPU 空闲温度为 55 C，而两个节点上所有同伴均约 30 C。现已 记录的实测健康参考值：h100 负载下 50-66 C（离散约 16 C）、b200 为 34-39 C，且 node2 的健康同伴为 39-43 C——即 GPU 3 的 87 C 高出其同伴约 44 C。 因此 `gpu-health-checked` 现携带 `hottest/median/spread`。这是有意**仅上报、不作判定**：空闲离散度可 预测负载下钳制这一结论目前样本数为 1，而依据单次事件选定的拒绝阈值有可能误判健康分配——连续三次误判 即会杀掉该 leg，其后果比故障本身更糟。先埋点，待分布已知后再设门。`gpu_temperature_spread` 是与 `gpu_health_faults` 并列的纯函数，日后提升为判定只需在 `validate_gpu_health` 中改一行。 三项测试覆盖故障离散度、其在标记中的出现，以及无法解析的情形。其中一项抓出了我自己的算术错误：八个 温度值的中位数应取排序后第 5 个（35），而我写成了 34。 106 项测试通过；该文件直接执行时可收集其定义的全部 53 项测试。
…tiveX：修正第三轮评审发现的八处错误论断 No behaviour change. The review confirmed the MoRI buffer-mode switch has no out-of-bounds read in any reachable configuration, then found that four comments and four documented facts were wrong. Comments, all stale after the switch: - `ep_mori.py` stage() cited `intranode.hpp:542` for the bound that makes the `[:rows]` cast safe. The bound is real, but line 542 sits inside `else if constexpr (UseP2PRead)`, and with an external input buffer plus combine quant_type "none" the launcher selects `EpCombineIntraNodeKernel_bf16_nop2p` — UseP2PRead FALSE. The branch that actually runs carries the same `tokenIdx < totalRecvTokenNum` bound, so the conclusion held for the wrong reason. Now cites the live branch and notes the other is compile-time dead here. - `ep_mori.py` still said "With use_external_inp_buf False the launcher takes the zero-copy branch" and that `_nop2p` sits behind a branch "we never enter". Both halves inverted by the switch. - `ep_backend.py` benchmark_roundtrip still said the staged tensor "IS the registered combine buffer" for MoRI. It is the dispatch output at BF16, or a fresh `[:rows]` cast under FP8. - `tests/test_roundtrip_staging.py` still said MoRI BF16 rows "do real device work" in stage. They do not: `self._fp8 or not self._external_input` collapses to `self._fp8`. Line 71 of the same file already stated the new behaviour, so the file contradicted itself. methodology.md, which defines what a published number means: - "issued as one fused kernel and guarded bitwise against its eager reference" was asserted for all FP8 `normal` backends. False for MoRI, whose quantize is a plain dtype cast needing neither. - Byte accounting credited per-128-block FP32 scales to "DeepEP's and UCCL-EP's blockwise codec" only. FlashInfer EP carries them too, as a fourth dispatch payload — which another section of the same file already said, so the doc contradicted itself across sections. - The `stage`-is-scaffolding parenthetical said DeepEP V2 converts "a padded low-latency plane". That is its LOW-LATENCY path; in `normal` mode it converts only received rows. UCCL-EP was omitted entirely. Since that sentence exists to explain why `stage` is not comparable between backends, getting the quantities wrong defeated it. - The AUTO-tuning aside claimed no gfx950 BF16 dispatch rule and no gfx950 IntraNodeLL combine table, hence hard-coded defaults. Only half true: gfx950 does ship an IntraNodeLL BF16 dispatch table and an IntraNode combine table with BF16 rules at both `zero_copy` values. Rewritten to say what is actually missing, and why partial tuning is its own argument against AUTO. - Added the bridge for a tension the review flagged: the doc says published cohorts rank on p99 and also says do not rank on p99 of MAX for multi-node decode. Both are true because the cohorts group into bootstrap equivalence bands rather than ordering by raw p99, so a stall-dominated cell ties instead of winning; a reader comparing two cells by hand has no such machinery, which is what the MAX/MIN bracket is for. - Scoped the "both engines pin 80/0/16" claim to the BF16 and FP8 paths this suite sweeps; SGLang has a separate FP4 intra-node override. Left alone: the vLLM-side claims (fp8+scales passed through on a block_k match, dequant as the mismatch fallback) are unverifiable from this tree and consistent with earlier findings. 106 tests pass. 中文：无行为变更。评审确认 MoRI 缓冲模式切换在所有可达配置下均无越界读取，随后发现四处注释与四项文档 事实有误。 注释（均因该切换而过期）：`ep_mori.py` 的 stage() 引用 `intranode.hpp:542` 作为 `[:rows]` 转换安全性 的依据——该界限确实存在，但第 542 行位于 `else if constexpr (UseP2PRead)` 之内，而在外部输入缓冲加 combine quant_type "none" 的组合下，launcher 选择的是 `EpCombineIntraNodeKernel_bf16_nop2p`，即 UseP2PRead 为**假**。真正执行的分支带有相同的 `tokenIdx < totalRecvTokenNum` 界限，因此结论成立但理由 错误；另三处分别为：仍称 use_external_inp_buf 为 False、仍称 staged 张量是 MoRI 的注册 combine 缓冲、 仍称 MoRI BF16 行在 stage 中有实际设备工作（同一文件第 71 行已正确表述，即文件自相矛盾）。 methodology.md（定义已发布指标含义的文件）：将"以单个融合 kernel 下发并逐位对齐 eager 参考"错误地断言 于所有 FP8 `normal` 后端（对 MoRI 不成立）；字节核算遗漏了 FlashInfer EP 的 scale 载荷（而同一文件另一 处已提及其第四个载荷，即跨节自相矛盾）；`stage` 脚手架说明中把 DeepEP V2 说成转换"补齐后的低延迟平面" （那是其低延迟路径，`normal` 模式仅转换已接收行），并完全遗漏 UCCL-EP；AUTO 调优旁注的论据半数不实， 已改为陈述实际缺失的部分；补充了 p99 与 p50 排序张力的衔接说明（队列按 bootstrap 等价带分组而非按原始 p99 排序）；并将"两个引擎均固定 80/0/16"的说法限定于本套件所扫描的 BF16 与 FP8 路径。 未改动：vLLM 侧的论断在本代码树中无法验证，且与既有结论一致。 106 项测试通过。
…p contract / CollectiveX：修正 README 的 FP8 覆盖范围并写明 roundtrip 约定 The README carried the same stale-coverage error a review had already found in methodology.md, in the entry-point document rather than the detailed one: - It listed FP8 support as "DeepEP V2, MoRI, UCCL-EP", omitting FlashInfer EP, and its FlashInfer row still said "BF16 only". FlashInfer FP8 landed in f78330a and is green on-metal at EP8 and EP16. The row now describes the shape that matters to a reader: caller-prequantized blockwise e4m3fn as a fourth dispatch payload with per-128-block FP32 scales, and the combine plane forced to BF16 — which is mandatory, not cosmetic, because the C++ `toNvDataType` accepts only fp16/bf16/fp32 for combine, so an FP8 combine buffer raises instead of corrupting. - It never mentioned `stage` or the fact that `roundtrip` excludes it, which is THE contract change of this branch. A reader starting here would have had no way to know that rows published before sweep version 2 carried the staging copy inside the chain for MoRI BF16 and FlashInfer BF16, i.e. that `roundtrip` meant different things in different rows. Now stated, with the FP8-stage-is-scaffolding point and the do-not-sum-or-compare warning, pointing at methodology.md for the full contract. Also noted that the caller-side FP8 quantize is charged inside the measured dispatch, since the README describes the precision sweep and that is a load-bearing property of those numbers. Checked and left alone: "NCCL EP is BF16-only" is still true, and the "32 synchronized full roundtrip warmups" description still holds — the warm-up hoist changed what staging those warmups repeat, not that there are 32 full roundtrips. 106 tests pass. 中文：README 中存在与评审此前在 methodology.md 中发现的同一处覆盖范围过期错误，而且出现在入口文档而 非细节文档中： - 它把 FP8 支持列为"DeepEP V2、MoRI、UCCL-EP"，遗漏了 FlashInfer EP，且其 FlashInfer 条目仍写作 "BF16 only"。FlashInfer 的 FP8 于 f78330a 落地，并已在实机 EP8 与 EP16 上通过。该条目现描述了对 读者重要的形态：调用方预量化的分块 e4m3fn 作为第四个 dispatch 载荷，附带每 128 块的 FP32 scale， 且 combine 平面强制为 BF16——这是必需而非修饰，因为 C++ 的 `toNvDataType` 对 combine 仅接受 fp16/bf16/fp32，因此 FP8 的 combine 缓冲会直接抛错而不会产生静默错误。 - 它完全没有提到 `stage`，也没有说明 `roundtrip` 将其排除，而这正是本分支的核心约定变更。从此处开始 阅读的人无从得知：在 sweep 版本 2 之前发布的行，对 MoRI BF16 与 FlashInfer BF16 是把 staging 拷贝 计入链路内的，即 `roundtrip` 在不同行中含义不同。现已写明，并附上"FP8 的 stage 属于测量脚手架"这一 要点与"不可求和、不可跨后端比较"的警示，并指向 methodology.md 获取完整约定。 同时说明调用方侧的 FP8 量化计入被测 dispatch 之内，因为 README 描述了精度扫描，而这是这些数字的关键 性质。 已核查并保留不变："NCCL EP 为 BF16 only"仍然成立；"32 次同步完整 roundtrip 预热"的描述亦仍成立—— 预热 hoist 改变的是这些预热重复的 staging，而非仍有 32 次完整 roundtrip 这一事实。 106 项测试通过。
The timing-window change altered what `roundtrip` means, which is normally a reason to bump the sweep version so two generations can never be mixed in one comparison. Keeping it at 1 instead, so this lands without requiring the frontend reader to accept a new generation first — the reader SKIPS runs whose version it does not list, so a version bump merged ahead of the frontend would make new runs silently invisible on the dashboard rather than fail loudly. The consequence is stated rather than hidden: the version tag no longer separates the generations, so `implementation.stage_excluded_from_roundtrip` and whether a `stage` component is present are the only discriminators. Both the README and methodology.md now say that explicitly, in place of the sentences that pointed at "sweep version 2". Worth being clear about what this trades away: rows measured before this branch and rows measured after it will both carry `version: 1` while `roundtrip` means something different in each — for MoRI BF16 and FlashInfer BF16 the older rows include the staging copy, and FP8 `normal` rows now charge the caller-side quantize inside dispatch. Anything reading the durable store across that boundary needs to key on the two fields above, not on the version. 106 tests pass.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
No bugs found, but given the size and complexity of this change I think it warrants a human look before merging.
What was reviewed: the roundtrip/stage measurement contract change (stage_excluded_from_roundtrip) across all five backends, the new FlashInfer FP8 dispatch path (fourth payload, filled-slot-only staging, oracle round-trip), the fused-quantize bitwise-identity gate (assert_quantize_identity), the NCCL EP HT combine-input slicing fix, and the new GPU-health pre-flight probe (fail-open design, throttle-flag vs temperature signals). Also checked the FlashInfer wheel-version parser and the two candidate slot-caching issues the bug hunter ruled out — the version parser reads real dotted-version ordering rather than scraping digits, and the cached slot indices are resolved once during the untimed warm-up pass where routing is fixed, not per timed iteration.
Extended reasoning...
Overview
This PR touches 21 files across the CollectiveX EP benchmark: the shared backend base class (ep_backend.py), all five backend adapters (deepep-v2, mori, uccl-ep, nccl-ep, flashinfer-ep), the harness, runtime probes, launchers, config codec, summarizer, and matrix/test files. It redefines what roundtrip measures (excluding staging uniformly instead of only for two backends), moves FP8 quantization into the timed dispatch window with a bitwise-identity safety gate, adds FP8 dispatch support to FlashInfer EP, fixes an NCCL EP combine-sizing bug, adds a GPU thermal-health pre-flight check, and reworks MoRI's launch configuration to match production engine settings.
Security risks
None of consequence — this is an internal benchmarking harness with no exposure to untrusted input. The GPU health probe shells out to nvidia-smi with a fixed argv (no injection surface), and the new subprocess.run calls use argument lists rather than shell strings. The one thing worth a second glance is the probe's explicit fail-open behavior on any read/parse error (deliberate, and documented as intentional so a broken probe can't block every allocation), but that's a benchmarking-availability tradeoff, not a security one.
Level of scrutiny
This warrants more than mechanical scrutiny despite living under experimental/: it redefines a measurement contract that downstream consumers key on (stage_excluded_from_roundtrip), and it adds a new bitwise-identity correctness gate (assert_quantize_identity) whose failure mode is silent measurement corruption across the whole fleet if it's ever wrong. The diff is also large enough (five backend files, the shared base class, the oracle-facing harness) that reasoning about every precision/mode combination by inspection alone is error-prone — the author's own description notes three adversarial review passes were needed to reach this state, which itself signals the subtlety involved.
Other factors
Test coverage is substantial (106 unit tests, several added directly for this change: FusedQuantizeGate, GpuHealthProbe, the reworked RoundtripStagingGate), and the description reports a full on-metal sweep across nine SKUs. No bugs were surfaced by the bug-hunting pass. Given the combination of scale, changed measurement semantics, and the number of independent backend code paths touched, I'd still like a human to confirm the cross-backend consistency of the roundtrip/stage contract change before this merges.
…teady-state period
Three changes, the first a plain bug.
**The low-latency path was disabling the fabric it should have used.** `_create_ll_buffer` passed
`allow_nvlink_for_low_latency_mode=True` but never passed `allow_mnnvl`, so the legacy Buffer took its
default of False — and a False there self-sets NVSHMEM_DISABLE_MNNVL. On GB200/GB300, whose scale-up
transport IS MNNVL, that forced the decode kernels onto IBGDA and measured the rack's slow path. Now
keyed on `scale_up_transport == "mnnvl"`, a topology fact the adapter already receives, and passed only
if the pinned wheel accepts the keyword — raising rather than silently running over IBGDA if it does
not.
**deepep-v2 low-latency was never dispatched on GB at all.** gb200/gb300 carried
`ll_backends = {"nccl-ep": [8]}`, so this is new coverage rather than a change to any published
number: 8 shards, EP8 and EP16, both precisions. EP16 is included because it stays inside the NVL72
domain with no scale-out, exactly as flashinfer-ep and nccl-ep already do EP16 there.
**A new `period` component, because `roundtrip` answers a different question than decode serving
asks.** `roundtrip` drains the GPU around every pair, so it measures the latency of an idle pipeline.
A decode loop runs dispatch->combine->dispatch->combine without stopping, and its per-layer cost is
the pipeline's period — smaller than the sum of separately-drained stages, and indifferent to how
inter-rank entry stagger gets attributed, which is the effect behind the falling small-T curves on
b200 uccl-ep low-latency. Added alongside the existing components, never replacing them, so nothing
already published changes meaning.
It is opt-in per backend (`pipeline_pairs`, default 0) for a specific reason. Issuing pairs
back-to-back lets ranks drift, and dispatch is a peer WRITE into another rank's buffer: stream order
on the receiver does not order the sender's remote writes, so overlap is only sound where the receive
buffer can absorb the drift. A collective bounds that drift to about one iteration — a dispatch cannot
complete until every rank enters it — so DeepEP's low-latency receive, double-buffered with a parity
that flips per dispatch, covers it; a single shared buffer would not. That is the same two-micro-batch
overlap SGLang and vLLM run. Only that path opts in; enabling it elsewhere would give a fast number
over corrupted data, which is why the default is off and a row without the component simply did not
measure it.
Three tests pin the opt-in: off by default, present when declared, and absent at `pipeline_pairs = 1`
since a single pair is not a pipeline and must not advertise a second name for `roundtrip`.
109 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standardizes what a CollectiveX number means, adds FP8 dispatch to FlashInfer EP, and fixes several measurement and infrastructure bugs found along the way.
Measurement contract
roundtripnow means dispatch-then-combine — the transport — in every row. Expert-output staging is reported separately asstage, where it used to sit inside the chain for MoRI BF16 and FlashInfer BF16 only.implementation.stage_excluded_from_roundtripdistinguishes the generations from the artifact alone; the sweepversionstays 1.warp_num_per_blockof 16. The two must move together: MoRI's tuning tables key combine onzero_copy, and 16 warps in the wrong mode costs +61-78% combine at T=512 on three chips.New coverage
Bugs fixed
stagep50 unchanged.Infrastructure
Three adversarial review passes; every finding fixed. 106 unit tests. One full-matrix sweep across all nine SKUs: 79 green, 4 failed — one documented b200 low-latency wall and three pre-harness rendezvous flakes, no code defects.