perf(webrtc): serve bounded concurrent browser RPCs (rpc-multiplex-4)

[mickvandijke]

Summary

Browser-transport performance follow-up to #220, targeting web-support. Nodes can now serve several browser RPCs concurrently on one authenticated WebRTC DataChannel instead of one at a time.

• 614bf4e7 perf(webrtc): serve bounded concurrent RPCs and retire reset channels
  • Advertises rpc-multiplex-4 in authenticated HELLO. A client that negotiates it may have at most four uncompleted requests per ordered DataChannel; others keep one. Request IDs correlate out-of-order completions.
  • One writer serializes whole frames and AEAD sealing; one reader authenticates frames in wire order before dispatch. Frames are not interleaved and no cryptographic checks are removed.
  • The server keeps receiving while work executes and while replies are written.
  • A channel reset retires channel I/O independently of tracked storage/payment workers and their reservations. An excess replacement channel is rejected without tearing down a healthy sibling channel.
  • Bulk GET preparation queues behind per-source and global permits derived from the existing byte budgets, held through writing. Configured limits are unchanged.
• 50167d39 fix(webrtc): suspend channel idle deadlines while requests are active. Reset detection stays active, and the idle budget restarts after the last response.

ADR-0015 gains a "Bounded concurrent RPCs and cancellation" section. The matching client work landed on ant-client web-support at 04c64fa (part of WithAutonomi/ant-client#186).

Linear issue

Closes V2-803

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Proposed for human review: adds a negotiated browser wire capability and changes node-side request admission on the WebRTC listener.

Compatibility

• Wire: additive rpc-multiplex-4 HELLO capability. Request/response frame formats are unchanged. Clients that don't negotiate it keep one outstanding request per channel. Native QUIC is untouched.
• Storage: none.
• API: none. There are no new config fields, and existing connection, channel, source and byte limits are unchanged.

Semver impact

• breaking
• feature
• fix

Test evidence

• Testnet A/B: 60 minutes, two 60-node DigitalOcean testnets with 30% symmetric NAT. Each ran 4 continuous WASM uploaders and 1 continuous downloader in headless Chromium, using the matching client build. Results, method and conclusion are in the comment below. The headline versus web-support: WASM uploads +118% throughput, WASM downloads +383%, 0 failures in 340 operations, 0 node restarts, and 37% less node CPU per GiB moved.
• Automated tests on this head (50167d39): run on a clean checkout: cargo test --lib --features test-utils web_rtc: 39 passed. cargo test --test webrtc_direct_devnet --features test-utils: 6 passed, 1 ignored (the five-node devnet test, #[ignore] by design). This includes the new multiplexed_get_completes_before_blocked_put and multiplexed_full_chunks_respect_source_budget regressions.

New dependency

none

ADR

• ADR-0015 — Bounded concurrent RPCs and cancellation

Mitigation / rollback

Revert the two commits. Clients fall back to one request per channel when a node doesn't advertise rpc-multiplex-4. As a last resort, disable webrtc_direct.enabled.

🤖 Generated with Claude Code

[mickvandijke]

Testnet A/B: web-support-optimizations vs web-support (WASM clients, 60 min)

Conclusion: the optimizations are a clear improvement and should stay on web-support for the path to main. With browser (WASM) clients, the optimization stack delivered:

• 2.2× the upload throughput and 4.8× the download throughput;
• zero upload or download failures on either side (340 operations);
• no node restarts and 37% less node CPU per GiB moved.

Absolute node CPU and RSS are higher because the optimized fleet moved 2.7× more data. Follow-ups are listed at the end.

The stack is #232 (merged into web-support) plus ant-client web-support-optimizations, which was fast-forwarded into ant-client web-support and so is part of WithAutonomi/ant-client#186. Measured commits: node 50167d39, client 04c64fa. ant-node web-support has since gained 1905e107 (replication fresh-offer fix), which this run does not cover.

Setup

	baseline ws	candidate opt
ant-node	web-support @ 41acee71	web-support-optimizations @ 50167d39
WASM client	ant-client web-support @ 9b85491	ant-client web-support-optimizations @ 04c64fa
Download concurrency	3 (SDK default before adaptive)	omitted = adaptive (current SDK default)

• Fleets: two separate 60-node DigitalOcean testnets, 12 droplets each, 18/60 nodes (30%) behind symmetric NAT (MASQUERADE --random-fully). Both use the same seed, so layout, regions and NAT assignment are identical. Each has its own Anvil EVM. Before T0, every node was active and bootstrapped with a WebRTC endpoint; node logs confirm the commits.
• Clients: 5 client droplets per testnet, in the same regions for both. Each runs headless Chromium 151 (Playwright) with that branch's wasm-pack build (browser-wasm), talking to nodes over WebRTC Direct.
  • 4 uploaders (sgp1, sfo3, tor1, sfo3) loop continuously: random 50 MiB → encryptPublicFile → uploadStagedPublicFile (auto payment mode). Each pays real on-chain from its own wallet, via the same serialized NonceManager pattern as the SDK's private-key adapter. One long-lived client per page.
  • 1 downloader (nyc1) runs continuously: downloadPublicFile on addresses harvested from the uploaders, sha256-verified. It uses a fresh BrowserNetworkClient per download, because a reused client serves repeats from its in-memory chunk cache. All downloads in both fleets were fresh files.
• Window: all 10 clients started at the same T0 and ran 60 minutes (2026-09-21 17:15:39–18:15:39 CEST). Every 30 s, systemd CPU/RSS was sampled for all 120 nodes and all clients.

Uploads (4 continuous uploaders, 50 MiB files)

	ws	opt	Δ
ok / failed	74 / 0	161 / 0
data stored	3.61 GiB	7.86 GiB	+118%
aggregate throughput	1.03 MiB/s	2.24 MiB/s	+118%
wall per upload p50 / p95 / max	181 / 268 / 315 s	84 / 123 / 158 s	−54%
network phase p50	179 s	82 s	−54%
WASM self-encrypt p50	2.03 s	1.98 s	≈
payment submit + confirm p50	1.06 s	1.01 s	≈

Downloads (1 continuous downloader, cold client, sha256-verified)

	ws	opt	Δ
ok / failed	18 / 0	87 / 0
aggregate throughput	0.25 MiB/s	1.21 MiB/s	+383%
wall per download p50 / p95	176 / 203 s	40 / 56 s	−77%
per-download MiB/s p50	0.29	1.24	×4.4

Throughput held steady across all six 10-minute buckets on both sides. There was no degradation over the hour, and every uploader improved (ws 18–19 uploads each, opt 33–44).

Node resources (60 nodes each)

	ws	opt	Δ
node CPU mean / p95 / max (% of one core)	4.7 / 14.4 / 46.6	8.0 / 19.3 / 59.1	+70% mean
fleet CPU per GiB of client data moved	2,278 core-s	1,436 core-s	−37%
node RSS mean / p95 (MiB)	126 / 176	149 / 215	+18% / +22%
node restarts	0	0
node droplet host CPU mean	13.0%	21.5%
lowest free memory on any node droplet	2.36 GB	2.22 GB
fleet network rx / tx	24.6 / 21.3 MiB/s	43.9 / 38.3 MiB/s
DTLS ClientHello warnings (≈ new browser connections)	891,888	205,740	~12× fewer per GiB

• RSS: node RSS rose over the hour in both fleets at a similar pace (ws 108 → 146 MiB, opt 126 → 172 MiB) while the opt fleet stored 2.2× the data. This looks like load/data-driven growth, not an opt-specific leak, but one hour doesn't show the plateau.
• Connection churn: the far lower DTLS handshake rate per GiB matches the connection reuse and two-lane/multiplexed channels.
• Clients: Chromium on the uploaders used 24% → 39% of a core (for 2.2× the work), with RSS about 1.5 GB on both sides. The downloader used 9% → 29% of a core, RSS about 0.75–0.8 GB. Clients were not the bottleneck.

Failures and errors

• Measured window: 0 upload and 0 download failures on either testnet. With zero failures, the realistic upper bound on the failure rate is about 1.9% for opt uploads and 4.1% for ws uploads. The only failure records are one per client for the in-flight operation cut off when Chromium stopped at the end of the window; they ended after T1 and are excluded.
• Node ERROR lines: ws 21, opt 28. Almost all are webrtc_sctp "stream N not found" noise (late data on an already-reset SCTP stream).
  • opt also logged 2 replication audit timeouts out of 27,684 outbound audits (ws: 0 of 13,616). Both came from symmetric-NAT nodes on one droplet (lon1) challenging one key each on public nodes in sgp1 and fra1. There were no absent keys and no digest mismatches.
• Aborted first attempt (not counted): the benchmark harness hit nonce has already been used in its own payment code, caused by the ethers provider's request cache. I fixed it to mirror the SDK's serialized NonceManager adapter, reset all client state and re-ran from a fresh T0. This was not a client or node problem.

Native-client context (separate run, same commits, earlier the same day)

Same method, but with the native ant CLI (4 uploaders + 1 downloader, 50 MiB, 60 min, 60 nodes, 30% NAT):

native	ws	opt	Δ
uploads ok / failed	404 / 0	363 / 0
upload aggregate	5.61 MiB/s	5.04 MiB/s	−10%
upload wall p50	34.2 s	38.0 s	+11%
downloads ok / failed	89 / 0	132 / 0
download aggregate	1.24 MiB/s	1.83 MiB/s	+48%
download wall p50	39.3 s	26.4 s	−33%
node CPU mean	27.7%	25.0%	−10%

Follow-ups before web-support → main

1. Native upload −10%. It comes from one run on separate fleets and could be fleet variance. The native upload/payment code is untouched, but native reads, the adaptive controller and node request admission did change. A repeat native A/B, ideally with swapped fleets, would settle it.
2. Longer soak (≥ 3 h) to confirm node RSS plateaus under the higher throughput.
3. The two NAT-originated audit timeouts are worth watching in the next longer run. At this rate they are not a blocker.