fix(replication): bound the fresh-offer backlog and send offers without copying

Cargo.toml

@@ -107,6 +107,9 @@ page_size = "0.6"
 
 # Protocol serialization
 postcard = { version = "1.1.3", features = ["use-std"] }
+# Byte-string encoding for chunk payloads in replication messages (same postcard
+# wire layout as a u8 sequence, but serialized and sized in one memcpy pass).
+serde_bytes = "0.11"
 bao = "0.13.1"
 
 # Shared portable browser profile. The native listener is enabled separately
@@ -231,9 +234,9 @@ webrtc-direct = [
 [patch.crates-io]
 saorsa-pqc = { git = "https://github.com/saorsa-labs/saorsa-pqc", rev = "29a2b272c4ee8b72c17102da00033dbfe1ddcd37" }
 evmlib = { git = "https://github.com/WithAutonomi/evmlib", rev = "ccd65f18bf3750af8b8c3cfe07548bc77e3c368d" }
-ant-protocol = { git = "https://github.com/WithAutonomi/ant-protocol", rev = "1764950d7af880aa13af679ac8c2d0fcbcde0776" }
-saorsa-core = { git = "https://github.com/WithAutonomi/saorsa-core", rev = "02dd65fc4df68f326b8d7ed0bd0e3ffb6c29329c" }
-saorsa-transport = { git = "https://github.com/WithAutonomi/saorsa-transport", rev = "3ed66a9c0880583ceab1e1550dce756e2716d111" }
+ant-protocol = { git = "https://github.com/WithAutonomi/ant-protocol", rev = "a66ddcfb60d50a9ec07b9aee0d063a730a96b330" }
+saorsa-core = { git = "https://github.com/WithAutonomi/saorsa-core", rev = "53f1fc6fca13109f583dd951ee3eab42d7a37121" }
+saorsa-transport = { git = "https://github.com/WithAutonomi/saorsa-transport", rev = "1766037ef159d80495515de65f96b3fd8fc94319" }
 
 [profile.release]
 lto = true

docs/adr/ADR-0016-bounded-fresh-offers-and-copy-free-sends.md

@@ -0,0 +1,144 @@
+# ADR-0016: Bounded fresh-replication offers and copy-free message sends
+
+- **Status:** Proposed
+- **Date:** 2026-09-22
+- **Decision owners:** <pending>
+- **Reviewers:** <pending>
+- **Supersedes:** none
+- **Superseded by:** none
+- **Related:** [ADR-0003](ADR-0003-full-node-detection-and-eviction.md)
+  (best-effort fresh delivery and possession checks),
+  [ADR-0005](ADR-0005-replication-repair-hardening.md),
+  ant-node `perf/replication-send-path`, saorsa-core `perf/replication-send-path`,
+  saorsa-transport `perf/replication-send-path`,
+  `ant-testnet/state/comparisons/web-support-memory-diag-0921/` (heap profiles
+  and per-minute live/RSS reports from the diagnosis)
+
+## Context
+
+Under sustained client uploads on a 60-node DigitalOcean testnet, individual
+nodes grew from ~100 MiB to 1–2 GiB of resident memory within an hour and
+kept growing for as long as writes continued. Heap profiles taken on the
+running nodes attributed roughly 80% of live memory to one site: encoded
+`FreshReplicationOffer` messages queued by the fresh-write drainer.
+
+The mechanism was structural rather than a leak:
+
+- Every accepted PUT was pushed to the drainer with the full chunk, and
+  `replicate_fresh` encoded the offer (chunk plus proof, up to ~4–5 MiB)
+  immediately, before any send permit was held.
+- One send task per close-group peer then waited for one of
+  `MAX_CONCURRENT_REPLICATION_SENDS` (3) permits while pinning that encoded
+  buffer. Nothing bounded how many chunks could be waiting in that state.
+- On a real network each send holds its permit for seconds (QUIC delivery
+  acknowledgement, retries, unreachable NAT peers), so a write rate above
+  the send rate grew the queue without limit. Loopback devnets never showed
+  it because sends complete instantly.
+
+Once the backlog was bounded, the profile showed the remaining cost per
+in-flight send: the same frame existed as the caller's serialized message
+*and* as the QUIC stream's copy for the whole transfer, plus transient
+copies made while framing (payload clone per channel attempt, owned wire
+message for signing, and a doubling `Vec` that left chunk-sized frames with
+up to twice their length in capacity).
+
+## Decision Drivers
+
+- Node memory must stay bounded under any client write rate; replication
+  may be delayed by backpressure but must not be dropped.
+- The change must not alter the wire format, storage format or payment
+  logic, so it can ship as a behavioural fix.
+- Existing callers of the send APIs in saorsa-core and saorsa-transport must
+  keep compiling and behaving the same.
+
+## Considered Options
+
+1. Bound the fresh-write channel and drop or block PUT handling when full.
+   Rejected: either silently loses replication or blocks client responses
+   on network conditions.
+2. Raise `MAX_CONCURRENT_REPLICATION_SENDS`. Rejected: only moves the
+   knee of the curve and increases bandwidth pressure on home links; the
+   queue behind the permits would still be unbounded.
+3. Keep events small and take a bounded permit before materialising an
+   offer; separately remove the avoidable copies on the send path. Chosen.
+
+## Decision
+
+We will bound the number of encoded fresh offers that can exist at once and
+make the send path hand a single owned buffer down to the QUIC stream:
+
+- `FreshWriteEvent` carries only the key and the payment proof, and fresh
+  replication runs as two stages. The fresh-write drainer never waits for
+  chunk back-pressure: for every event, at arrival rate, it records the key
+  in `PaidForList(self)` and sends `PaidNotify` to the paid close group —
+  the evidence later repair depends on — then forwards the event to the
+  offer dispatcher. The dispatcher is the only permit-gated stage: it
+  acquires a `MAX_PENDING_FRESH_OFFERS` (8) permit before it reads the
+  chunk back from storage (`get_raw`; the chunk was content-checked when
+  stored) and encodes it; the permit lives with the encoded offer until the
+  last per-peer send drops it. A backlog therefore waits as small queued
+  events, and at most ~40 MiB of encoded offers exist per node. Nothing is
+  dropped by back-pressure: both queues are unbounded and FIFO, every offer
+  is dispatched with the same fan-out, retries and delayed possession check,
+  and a failed read-back is retried `MAX_FRESH_READ_ATTEMPTS` times with the
+  permit released in between; only a chunk that is no longer stored is
+  skipped.
+- The chunk moves into the offer rather than being copied, and
+  `ReplicationMessage::encode` serializes into an exactly-sized buffer.
+- The encoded offer is shared as `Bytes`; saorsa-core's `send_message`
+  accepts `impl Into<Bytes>`, frames the payload through a borrowing
+  `WireMessageRef` (byte-identical to `WireMessage` on the wire) into an
+  exactly-sized frame, and passes that frame as `Bytes` to
+  saorsa-transport's new `send_bytes`, where the QUIC stream takes ownership
+  via `write_chunks` instead of copying it.
+
+## Consequences
+
+### Positive
+
+- Memory under write load is bounded by configuration: pending offers plus
+  the three in-flight sends, each held once, instead of growing with the
+  backlog. On the diagnostic fleets peak live memory fell from 1068 MiB to
+  298 MiB (mimalloc build) and from 674 MiB to 262 MiB (jemalloc build)
+  after the backpressure change alone.
+- Every large send node-wide (chunk GET responses included) stops paying
+  for a second copy of its frame during the transfer.
+- No wire, storage or API break: `send(&[u8])` remains and copies once as
+  before; `Vec<u8>` callers of `send_message` convert without copying.
+
+### Negative / Trade-offs
+
+- Replication of a burst of writes is spread out in time rather than
+  encoded eagerly; the delayed possession check is scheduled after each
+  offer's sends are dispatched, so it shifts by the same amount. A chunk
+  fetched seconds after its upload can therefore have fewer replicas than
+  before (the 2026-09-22 comparison measured downloads of just-uploaded
+  files 8% slower). Paid-list evidence is not affected, and the previous
+  unbounded fan-out lost that evidence outright under load (2,795
+  "paid notify dropped at admission" in one hour on the baseline fleet).
+- The drainer re-reads each chunk from disk when its permit arrives, one
+  extra read per accepted write.
+
+### Neutral / Operational
+
+- `MAX_PENDING_FRESH_OFFERS` and `MAX_CONCURRENT_REPLICATION_SENDS` are
+  the two knobs; raising the first trades memory for burst absorption.
+- The signing step still serializes the payload once to produce the signed
+  bytes; changing that would alter the signature input and is out of scope.
+
+## Validation
+
+- Unit tests: exact-capacity encoding of chunk-sized offers (ant-node) and
+  byte-for-byte equivalence of `WireMessageRef` with `WireMessage`
+  (saorsa-core); replication unit and e2e fresh-replication scenarios pass.
+- Testnet evidence (2026-09-21): with the backpressure change, the node
+  that had reached 1051 MiB live memory stayed flat at 0.0 MiB/min with a
+  150 MiB peak, and the worst bootstrap's queued offers dropped from 101
+  (463 MiB) to 6 (17.7 MiB) in heap profiles.
+- Review trigger: any change to fresh replication fan-out, send permits, or
+  the wire-message framing must re-run the memory diagnostics under
+  sustained uploads and confirm live memory stays bounded.
+
+## Notes for AI-assisted work
+
+AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR.

src/replication/config.rs

@@ -170,6 +170,28 @@ pub const SELF_LOOKUP_INTERVAL_MAX: Duration = Duration::from_secs(SELF_LOOKUP_I
 /// at most ~12 MB queued for the upload link at any instant.
 pub const MAX_CONCURRENT_REPLICATION_SENDS: usize = 3;
 
+/// Maximum number of encoded fresh-replication offers held in memory.
+///
+/// Each accepted write is encoded once (chunk plus proof, up to ~4 MB) and
+/// that buffer stays alive until the last of its per-peer sends completes.
+/// With only `MAX_CONCURRENT_REPLICATION_SENDS` transfers in flight, a write
+/// rate above the network's send rate would otherwise queue an unbounded
+/// number of encoded offers behind the send permits. The fresh-write drainer
+/// takes one of these permits before it reads and encodes a chunk, so the
+/// backlog waits as small queued events instead of chunk-sized buffers.
+pub const MAX_PENDING_FRESH_OFFERS: usize = 8;
+
+/// How many times the offer dispatcher tries to read an accepted chunk back
+/// from storage before giving up on its fresh offers.
+///
+/// The chunk was stored moments earlier, so a failed read is a transient
+/// fault (exhausted descriptors, an I/O hiccup) far more often than a lost
+/// chunk; a lost chunk reports `None` and is skipped without retry.
+pub const MAX_FRESH_READ_ATTEMPTS: u32 = 3;
+
+/// Pause before retrying a failed chunk read-back in the offer dispatcher.
+pub const FRESH_READ_RETRY_DELAY: Duration = Duration::from_secs(1);
+
 /// Maximum number of concurrent in-flight audit-responder tasks.
 ///
 /// The LIGHT audit-responder handlers — responsible-chunk audits and subtree