Skip to content

Avoid blocking auto-standby holds on persistence - #415

Open
sjmiller609 wants to merge 2 commits into
mainfrom
hypeship/unblock-autostandby-holds
Open

Avoid blocking auto-standby holds on persistence#415
sjmiller609 wants to merge 2 commits into
mainfrom
hypeship/unblock-autostandby-holds

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

summary

  • move auto-standby runtime metadata writes outside the controller state mutex
  • serialize writes per instance and discard superseded updates so persisted runtime follows controller mutation order
  • preserve operation-specific persistence error context
  • add regression coverage for blocked persistence, cross-instance isolation, superseded generations, and instance deletion

why

Runtime metadata persistence can wait on instance storage while holding the controller-wide mutex. That delays unrelated auto-standby holds and adds latency before browser traffic can be forwarded.

tests

  • go test -race -count=10 ./lib/autostandby
  • go vet ./lib/autostandby
  • go test ./... (partial: packages unrelated to this change fail because embedded binaries and mkfs.erofs are unavailable; lib/autostandby passes)

Note

Medium Risk
Concurrency and ordering changes around idle/standby runtime metadata affect recovery across restarts and hold latency; behavior is localized to autostandby with targeted race tests but storage timing can still diverge briefly from in-memory state.

Overview
Auto-standby runtime persistence no longer holds the controller-wide mutex while SetRuntime runs, so slow instance storage cannot block HoldStandby or work on other instances.

State updates still happen under the lock via prepareRuntimePersistenceLocked, which stamps a generation and a per-instance serialization lock; persistRuntime runs afterward, skips stale generations and deleted instances, and uses propagate vs best-effort error handling with operation labels in logs.

Call sites across seed/resync, connection events, standby confirmation, standby execution, and active reconcile were refactored to unlock before persisting; connection handling can batch multiple persist jobs after one locked pass.

Regression tests cover cross-instance non-blocking, latest-generation wins, deleted-instance skip, and hold completing while persistence is in flight.

Reviewed by Cursor Bugbot for commit 118f11c. Bugbot is set up for automated code reviews on this repo. Configure here.

@sjmiller609
sjmiller609 marked this pull request as ready for review August 17, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant