Skip to content

fix(relay): move fleet-wide NIP-43 membership sweep off the pre-bind startup path - #4554

Open
tlongwell-block wants to merge 2 commits into
mainfrom
eva/relay-postbind-nip43-sweep
Open

fix(relay): move fleet-wide NIP-43 membership sweep off the pre-bind startup path#4554
tlongwell-block wants to merge 2 commits into
mainfrom
eva/relay-postbind-nip43-sweep

Conversation

@tlongwell-block

@tlongwell-block tlongwell-block commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Problem: silent permanent crashloop after a restart ("boot wedge")

Startup awaits reconcile_nip43_membership_snapshots — a sequential sweep over every provisioned community — before binding the listener. Per community it runs a snapshot-reconciliation check (2 queries) plus a repair publication when needed.

Measured locally (isolated docker rig): ~9.5ms/community on the repair path; 1001 communities ≈ 9.5s of publishes, all logged before buzz-relay TCP listening. The cost is linear in community count, so on a deployment with a large number of communities the pre-bind sweep takes minutes — far past a typical startup probe window (e.g. 120s). The probe then SIGKILLs the pod mid-sweep, the restart begins again at community #1, and the pod crashloops forever with the port never bound and zero log lines at RUST_LOG=error (a healthy boot at that level logs nothing until failure, so the loop is silent).

Fix (deliberately minimal — emergency scope)

  • Pre-bind: reconcile only the deployment community (one bounded snapshot check), with explicit phase logs carrying elapsed_ms so the startup sequence is observable even when it's healthy.
  • Fleet sweep moves entirely post-bind into the existing periodic task, now:
    • jittered — first tick delayed by a random fraction of the interval (same rationale and mechanism as the usage-metrics poller: PID-derived seeds are unsafe when every pod is PID 1);
    • leader-gated — session advisory lock (NIP43_SWEEP_LOCK_KEY, reusing try_lock_usage_metrics) so N replicas never run N concurrent fleet-wide sweeps. The lock is held only for the sweep's duration; a replica dying mid-sweep releases it with its session.
    • skip-tick on overrun (MissedTickBehavior::Skip) — a sweep longer than the interval doesn't queue a catch-up burst.
  • Sweep telemetry: start / progress (every 1000 communities, with scanned/total/reconciled/elapsed_ms) / complete / fail logs, plus a buzz_nip43_membership_sweep_seconds histogram.

Pagination/resume and per-query lock/statement deadlines are a separate follow-up PR by agreed scope.

Verification (live-local)

Rig: docker compose stack, postgres seeded with 1001 communities, all 1001 kind:13534 snapshot events deleted to force the worst-case repair path. Binary built from this branch, cargo build --release -p buzz-relay.

Run 1 (1001 repairs pending):

15:48:44.461 NIP-43 startup phase: reconciling deployment community snapshot
15:48:44.474 NIP-43 startup phase: deployment community snapshot reconciled repaired=true elapsed_ms=12
15:48:44.475 buzz-relay TCP listening addr=127.0.0.1:33000        <- bind at +14ms, not +9.5s
15:48:46.478 NIP-43 membership sweep starting (leader)
15:48:50.024 NIP-43 membership sweep complete: snapshots repaired count=1000 elapsed_ms=3546

Bind happens 12ms after the deployment-community check; the other 1000 repairs happen post-bind under the lock. DB confirms all 1001 snapshots restored.

Run 2 (restart, converged state — idempotence):

15:49:19.054 ... deployment community snapshot reconciled repaired=false elapsed_ms=7
15:49:19.055 buzz-relay TCP listening
15:49:20.330 NIP-43 membership sweep complete: nothing to repair elapsed_ms=1273

Tests: cargo test -p buzz-relay — 837 passed, 1 failed: api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo, which fails identically at the base commit (origin/main 5e0efb0) with this diff stashed — pre-existing, not introduced here. just test-unit (the pre-push gate) green. Clippy clean.

…startup path

The relay awaited reconcile_nip43_membership_snapshots for EVERY
provisioned community before binding the listener. The sweep is
sequential, two queries per community plus a publication per repair
(~9.5ms/community measured locally). Cost is linear in community count,
so on a deployment with many communities the pre-bind sweep takes
minutes — far past a typical startup probe window. The probe SIGKILLs
the pod mid-sweep, the restart begins again at community number one,
and the pod crashloops forever: low RSS, port never bound, zero log
lines at RUST_LOG=error.

Fix, minimal by design:
- Pre-bind: reconcile ONLY the deployment community (one snapshot check,
  bounded), with explicit phase logs carrying elapsed_ms.
- The fleet-wide sweep moves entirely into the periodic post-bind task,
  now jittered (random fraction of the interval, same rationale as the
  usage-metrics poller) and leader-gated via a session advisory lock so
  N replicas do not run N concurrent fleet-wide sweeps. Lock is held
  only for the duration of a sweep; a replica dying mid-sweep releases
  it with its session.
- Sweep telemetry: start/progress(1000)/complete/fail logs with
  elapsed_ms, plus a buzz_nip43_membership_sweep_seconds histogram.

Pagination/resume and per-query deadlines are a deliberate follow-up PR.

Verified: cargo test -p buzz-relay — 837 passed, 1 pre-existing failure
(api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo) which
fails identically at origin/main 5e0efb0 with this diff stashed.

Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block force-pushed the eva/relay-postbind-nip43-sweep branch from 4958c07 to f0dfd21 Compare August 3, 2026 16:31
Review found the sweep-after-bind ordering was probabilistic: the sweep
task is spawned before serve(), zero jitter is a valid draw, and
interval.tick() fires immediately — so the fleet sweep could still start
pre-bind. Replace the timing assumption with a structural guarantee:

- new after_listener_bound(oneshot::Receiver, task) helper; the gated
  task runs only after the signal fires and never runs if the sender is
  dropped (startup failed before bind)
- serve() takes a listener_bound_tx: oneshot::Sender<()> and fires it
  immediately after the main TCP listener binds
- the fleet sweep task is wrapped in after_listener_bound; jitter is now
  explicitly only lock-contention spreading, not ordering

Adds two deterministic tests: a yield-storm test proving the task cannot
run before the signal even with zero delay, and a dropped-sender test
proving the task never runs when startup fails pre-bind.

Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>

@Samuel7192 Samuel7192 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the post-bind ordering, advisory-lock lifetime, skipped-tick behavior, and failure handling. The change structurally removes the fleet sweep from the pre-bind path and preserves per-community reconciliation. CI coverage includes the listener-bound gate and the existing relay suite.

@Samuel7192

Copy link
Copy Markdown

Production impact confirmed on postral.communities.buzz.xyz: Desktop/ACP clients saw connection closures, transient database errors, and 404 relay: no community is configured for this host during the relay restart from 17:26–17:36 UTC. The host has recovered (30/30 NIP-11 200, 20/20 WebSocket 101, 20/20 /events requests reached tenant auth rather than host-mapping 404), but the pre-bind sweep remains a restart hazard until this lands. All CI is green; requesting the required Block-member approval from @block/buzz-oss-team.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants