fix(relay): move fleet-wide NIP-43 membership sweep off the pre-bind startup path - #4554
fix(relay): move fleet-wide NIP-43 membership sweep off the pre-bind startup path#4554tlongwell-block wants to merge 2 commits into
Conversation
…startup path The relay awaited reconcile_nip43_membership_snapshots for EVERY provisioned community before binding the listener. The sweep is sequential, two queries per community plus a publication per repair (~9.5ms/community measured locally). Cost is linear in community count, so on a deployment with many communities the pre-bind sweep takes minutes — far past a typical startup probe window. The probe SIGKILLs the pod mid-sweep, the restart begins again at community number one, and the pod crashloops forever: low RSS, port never bound, zero log lines at RUST_LOG=error. Fix, minimal by design: - Pre-bind: reconcile ONLY the deployment community (one snapshot check, bounded), with explicit phase logs carrying elapsed_ms. - The fleet-wide sweep moves entirely into the periodic post-bind task, now jittered (random fraction of the interval, same rationale as the usage-metrics poller) and leader-gated via a session advisory lock so N replicas do not run N concurrent fleet-wide sweeps. Lock is held only for the duration of a sweep; a replica dying mid-sweep releases it with its session. - Sweep telemetry: start/progress(1000)/complete/fail logs with elapsed_ms, plus a buzz_nip43_membership_sweep_seconds histogram. Pagination/resume and per-query deadlines are a deliberate follow-up PR. Verified: cargo test -p buzz-relay — 837 passed, 1 pre-existing failure (api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo) which fails identically at origin/main 5e0efb0 with this diff stashed. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
4958c07 to
f0dfd21
Compare
Review found the sweep-after-bind ordering was probabilistic: the sweep task is spawned before serve(), zero jitter is a valid draw, and interval.tick() fires immediately — so the fleet sweep could still start pre-bind. Replace the timing assumption with a structural guarantee: - new after_listener_bound(oneshot::Receiver, task) helper; the gated task runs only after the signal fires and never runs if the sender is dropped (startup failed before bind) - serve() takes a listener_bound_tx: oneshot::Sender<()> and fires it immediately after the main TCP listener binds - the fleet sweep task is wrapped in after_listener_bound; jitter is now explicitly only lock-contention spreading, not ordering Adds two deterministic tests: a yield-storm test proving the task cannot run before the signal even with zero delay, and a dropped-sender test proving the task never runs when startup fails pre-bind. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Samuel7192
left a comment
There was a problem hiding this comment.
Reviewed the post-bind ordering, advisory-lock lifetime, skipped-tick behavior, and failure handling. The change structurally removes the fleet sweep from the pre-bind path and preserves per-community reconciliation. CI coverage includes the listener-bound gate and the existing relay suite.
|
Production impact confirmed on |
Problem: silent permanent crashloop after a restart ("boot wedge")
Startup awaits
reconcile_nip43_membership_snapshots— a sequential sweep over every provisioned community — before binding the listener. Per community it runs a snapshot-reconciliation check (2 queries) plus a repair publication when needed.Measured locally (isolated docker rig): ~9.5ms/community on the repair path; 1001 communities ≈ 9.5s of publishes, all logged before
buzz-relay TCP listening. The cost is linear in community count, so on a deployment with a large number of communities the pre-bind sweep takes minutes — far past a typical startup probe window (e.g. 120s). The probe then SIGKILLs the pod mid-sweep, the restart begins again at community #1, and the pod crashloops forever with the port never bound and zero log lines atRUST_LOG=error(a healthy boot at that level logs nothing until failure, so the loop is silent).Fix (deliberately minimal — emergency scope)
elapsed_msso the startup sequence is observable even when it's healthy.NIP43_SWEEP_LOCK_KEY, reusingtry_lock_usage_metrics) so N replicas never run N concurrent fleet-wide sweeps. The lock is held only for the sweep's duration; a replica dying mid-sweep releases it with its session.MissedTickBehavior::Skip) — a sweep longer than the interval doesn't queue a catch-up burst.scanned/total/reconciled/elapsed_ms) / complete / fail logs, plus abuzz_nip43_membership_sweep_secondshistogram.Pagination/resume and per-query lock/statement deadlines are a separate follow-up PR by agreed scope.
Verification (live-local)
Rig: docker compose stack, postgres seeded with 1001 communities, all 1001 kind:13534 snapshot events deleted to force the worst-case repair path. Binary built from this branch,
cargo build --release -p buzz-relay.Run 1 (1001 repairs pending):
Bind happens 12ms after the deployment-community check; the other 1000 repairs happen post-bind under the lock. DB confirms all 1001 snapshots restored.
Run 2 (restart, converged state — idempotence):
Tests:
cargo test -p buzz-relay— 837 passed, 1 failed:api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo, which fails identically at the base commit (origin/main5e0efb0) with this diff stashed — pre-existing, not introduced here.just test-unit(the pre-push gate) green. Clippy clean.