fix(replication): bound the fresh-offer backlog and send offers without copying

[mickvandijke]

Summary

Under sustained client uploads on a 60-node testnet, nodes grew to 1–2 GiB within an hour. Heap profiles attributed ~80% of live memory to encoded FreshReplicationOffer messages queued in the fresh-write drainer: every accepted PUT was encoded immediately (~4–5 MiB) and its per-peer send tasks then waited for one of MAX_CONCURRENT_REPLICATION_SENDS (3) permits while pinning that buffer, with nothing bounding the backlog.

Five commits:

1. fix(replication): bound encoded fresh offers waiting behind send permits — FreshWriteEvent carries only key + proof; the drainer takes a MAX_PENDING_FRESH_OFFERS (8) permit before reading the chunk back and encoding, and the permit lives with the encoded offer until its last send finishes.

2. perf(replication): cut copies of fresh offers on the send path — the chunk moves into the offer instead of being copied; ReplicationMessage::encode serializes into an exactly-sized buffer (a doubling Vec had left chunk-sized messages with up to 2× capacity retained while queued).

3. perf(replication): share fresh offers with the transport as Bytes — with saorsa-core accepting impl Into<Bytes>, each send attempt hands out a reference-counted handle; together with the exactly-sized frame and the transport's owned-buffer write, an in-flight send costs one frame instead of four copies. Adds ADR-0016.

4. fix(replication): send PaidNotify before waiting for an offer permit — first step of decoupling the paid-list evidence from chunk back-pressure.

5. fix(replication): two-stage fresh replication with un-gated paid-list evidence (review follow-up) — the fresh-write drainer now never waits for a permit: for every event, at arrival rate, it records PaidForList(self) and sends PaidNotify, then forwards the event to a new offer dispatcher, the only permit-gated stage. The dispatcher reads the chunk back with get_raw (the chunk was content-checked when stored), retries a failed read up to MAX_FRESH_READ_ATTEMPTS times with the permit released in between, and skips only a chunk that is no longer stored. Chunk-carrying protocol fields are encoded as byte strings (serde_bytes; identical postcard layout, unit-tested), so sizing and encoding are single memcpy passes and an oversized body is refused before allocation. PaidNotify shares one Bytes buffer across recipients. The offer pipeline is one function used by both the dispatcher and the direct replicate_fresh entry point. A new e2e test drives the PUT pipeline through the real channel, including a missing-chunk event ahead of a real one.

Nothing is ever dropped by the permit: both queues are unbounded and FIFO, permits are released whenever a send terminates, and every offer keeps the same fan-out, retries and delayed possession check — only the bulk chunk transfer is deferred under load.

Pins: ant-protocol 79a68a80, saorsa-core 9baba9c5, saorsa-transport 0f55bfc7 (all perf/replication-send-path).

Linear issue

Closes V2-TBD

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: none — replication message encoding and the saorsa-core frame are byte-identical.
• Storage: none — the drainer reads chunks the PUT handler already stored.
• API: FreshWriteEvent drops its data field and fresh::replicate_fresh takes the chunk by value plus a pending-offer permit (crate-internal callers); ReplicationEngine::replicate_fresh is unchanged.

Semver impact

• breaking
• feature
• fix

Test evidence

• cargo test --lib -- replication: 591 passed (includes the exact-capacity encoding test and the serde_bytes wire-equivalence test).
• cargo test --features test-utils --test e2e -- fresh replication paid: 39 passed, including the new fresh_write_pipeline_replicates_queued_writes_and_skips_missing_chunks.
• Testnet validation of commit 1 (2026-09-21, two 60-node fleets, 4 uploaders + 1 downloader each, mimalloc and jemalloc builds): fleet peak live memory 1068 → 298 MiB and 674 → 262 MiB after redeploying the fix; the node that had reached 1051 MiB stayed flat at 0.0 MiB/min; the worst bootstrap's queued offers fell from 101 (463 MiB) to 6 (17.7 MiB) in heap profiles. Data and tooling: ant-testnet/state/comparisons/web-support-memory-diag-0921/.
• Commits 2–3 are validated by the test suites above; not yet re-run on a testnet.
• clippy is clean on the changed files; -D warnings on the whole crate trips a pre-existing Duration lint in src/storage/migration_signal.rs with newer clippy (unrelated).

Testnet comparison, 2026-09-22 (this stack vs current web-support). Two 60-node DigitalOcean fleets with an identical layout (8 regions, 18/60 nodes behind symmetric NAT), 4 native uploaders (50 MiB files, own wallets) + 1 continuous sha256-verified downloader each, started together and measured for 60 minutes. Baseline: ant-node 50167d39 + ant-client 04c64fa. New: ant-node f55b7df8, saorsa-core e1f1ef92, saorsa-transport 3bd451d2, ant-protocol ac25b717, ant-client f11c557 (pins only, so the client code is identical on both sides).

	base	new
Uploads OK / failed	375 / 0	412 / 0 (+10%)
Upload throughput	5.21 MiB/s	5.72 MiB/s
Upload time mean / p95 / worst	37.9 / 63.4 / 208 s	34.4 / 46.1 / 55 s
Upload throughput per 10 min	5.6 → 4.9 MiB/s (decaying)	5.5 → 5.5 MiB/s (flat)
Downloads OK / failed	130 / 0	120 / 0 (−8%)
Download time mean / p95	26.9 / 36.5 s	29.1 / 40.1 s
Node RSS mean / p95 / largest	199 / 305 / 428 MiB	182 / 262 / 364 MiB
Lowest free memory on any node droplet	1392 MiB	1859 MiB
Node CPU per GiB stored	0.82 core-h	0.82 core-h
Node WARN / ERROR log lines	52,613 / 38	3,773 / 12
Uploader client peak / p95 memory	680 / 465 MiB	453 / 327 MiB
Uploader client CPU per GiB	313 core-s	282 core-s
Node restarts	0	0

The baseline's warnings are the unbounded replication fan-out at work: 17,774 dial failures, 2,795 "Paid notify dropped at admission — paid-list evidence lost", 1,877 possession-probe timeouts, ~2,800 channel-send failures. The new stack logged no paid-notify drops, 134 dial failures and 286 probe timeouts. The 8% slower downloads on the new stack are the expected cost of deferring replication under load: every download fetched a file uploaded within the previous minute, which now has fewer replicas at that moment (MAX_PENDING_FRESH_OFFERS trades memory for burst absorption). Raw data: ant-testnet/state/comparisons/web-support-send-path-vs-base-60m-0922/.

New dependency

serde_bytes 0.11 (already present in the dependency graph transitively; now a direct dependency for the byte-string encoding of chunk payload fields)

ADR

https://github.com/WithAutonomi/ant-node/blob/perf/replication-send-path/docs/adr/ADR-0016-bounded-fresh-offers-and-copy-free-sends.md

Mitigation / rollback

Revert the branch (or re-pin the three git deps to 1764950d / 02dd65fc / 3ed66a9c). Wire format is unchanged, so a partial rollout or rollback is safe. If the pending-offer cap ever throttles replication too hard, raising MAX_PENDING_FRESH_OFFERS trades memory for burst absorption without touching the wire.

🤖 Generated with Claude Code