Add accumulate method for PortChannel

[Changho Hwang (chhwang)]

Adds accumulate, a remote 64-bit add, on Connection (host) and PortChannel (device).

The caller supplies only its own contribution, not the destination's current value. That is the difference from updateAndSync(), which assigns an absolute value and therefore requires the sender to own the address. Addition commutes, so arrival order does not matter and several peers can target one counter.

Encoding

Adds TriggerAccumulate = 6 to the opcode set provided by the chhwang/enhanced-fifo base branch. The full signed operand occupies the trigger's first word, spanning the size and srcOffset fields.

A zero operand traverses the FIFO and adds nothing. Enhanced FIFO reports readiness with lap parity in the reserved bit of snd, so trigger payload no longer doubles as a written marker.

Transport support

The addition has to be a real read-modify-write at the destination, which not every transport can do.

Transport	Concurrent writers	Mechanism
IB	any	NIC atomic fetch-and-add
IB, no-atomic mode	none — throws InvalidUsage	device has no RDMA atomics
Ethernet	any remote writer	receiving process does the RMW, serialized across its connections
CudaIpc on ROCm	any	kernel on the connection's stream
CudaIpc on CUDA	none — throws InvalidUsage	see below

On Ethernet every peer terminates its socket in the destination process, so one mutex covers all of them. Without it, a 7-writer fan-in loses ~74% of updates on MI300X (1040 of 1400). The destination GPU must not write the address concurrently, because it can be overwritten inside the RMW window.

CudaIpc differs by platform because host access to device memory differs:

• ROCm: a kernel on the connection's stream. That stream also orders the add ahead of any signal or flush issued afterwards. A kernel there runs while the caller's kernel spins — measured at 0.4 ms against a 1910 ms spinner, at 1, 64, and 304 blocks.
• CUDA: unsupported. The host reaches device memory only through the copy engines, which move values but cannot add to one, and a host-side read-modify-write is not atomic against writers in other processes. A kernel is atomic but unusable: in the caller's context it does not start until the caller's kernel finishes, which deadlocks any caller spinning on the result; in a separate context it costs 2391 us per operation against 19 us for a plain remote store.

A host-side CPU atomic on the peer pointer does not work on ROCm either. GPU memory is host-accessible to the process that allocated it, but an hipIpcOpenMemHandle mapping is device-only, and the proxy holds exactly that imported pointer. It segfaults.

Tests

PortChannelOneToOneTest, per transport:

• Accumulate — 64 blocks per rank accumulate into the peer each iteration, then signal, flush, wait. After each wait() the value is checked against a range, which asserts that every add the peer issued before its signal has landed. The lower bound is the ordering property; the upper bound allows the peer to be one iteration ahead, since it resumes on our signal.
• AccumulateSigned — negative operands, exact total.
• AccumulateZero — zero and non-zero operands interleaved. Hangs without the no-op guard.
• AccumulateIbHostNoAtomicRejected, AccumulateCudaIpcRejected — the two unsupported combinations throw.

Operands are 2^32 + 1, so a truncated operand shows up as a wrong sum.

PortChannelFanInTest is new: ranks 1..N-1 each push 200 accumulates at rank 0's single counter, and rank 0 checks the exact total. It covers IB and Ethernet everywhere, plus CudaIpc on ROCm. It polls until the total stops changing rather than sleeping, because EthernetConnection::flush() is a no-op and a sender cannot tell when the receiver applied its update.

Validation

	H200, CUDA 12.9	MI300X, ROCm 7.2
--filter=Accumulate	6/6	5/5 non-IB
fan-in, 8 ranks	IB, Ethernet	CudaIpc, Ethernet
mp_unit_tests	35 passed, 2 skipped	22 passed, 4 skipped
unit_tests	34/34	23 passed, 9 skipped
PingPongPerf, CudaIpc	19.23 us — unchanged	—

The MI300X node has no IB device, so every IB test fails there with ib.cc:685 IB transport out of range: 0 >= 0, including tests this PR does not touch. All other suites — allgather_test_cpp, allgather_test_host_offloading, nvls_test, executor_test, python/test/test_mscclpp.py — produce results identical to origin/main built the same way.

Follow-ups, not in this PR

• Move updateAndSync's running counter into the connection and drop the uint64_t* src shadow from the public API.
• Add a device-side MemoryChannel::accumulate<T>() for concurrent intra-node accumulation on CUDA.
• Batch proxy-side ROCm accumulates. One launch per batch measured 13.5x faster at batch 64 and 41.5x at batch 256.
• IbPeerToPeerTest.MemoryConsistency divides by zero when no IB device is present. Pre-existing.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.