Repository navigation
Queue pause/resume is lost if sent while the notifier is reconnecting #1579
Description
Since #1564, a job cancel sent while the notifier isn't listening is picked up by the fallback poll. Queue pause and resume have no fallback in notifier mode (pollForSettingChanges only runs when Notifier == nil), so a QueuePause sent while the listener is down is lost until the client restarts.
Repro on master (1e8fa0d), Postgres 18, riverpgxv5. A worker client and a second client on another pool. I terminate the worker's LISTEN backend, call QueuePause from the second client right away, then insert jobs:
1.1s terminated 1 LISTEN backend
1.1s QueuePause(default) committed
3.2s job inserted 2s after the pause ran
31.2s job inserted 30s after the pause ran
91.2s job inserted 1m30s after the pause ran
91.2s river_queue.paused_at is set
With the same steps and no killed listener, none of the three jobs run.
Related: the notifier's health Ping uses the service context with no timeout. When I froze the listener's socket with a TCP proxy (silent, no error), the notifier never reconnected before my runs ended, so pause notifications stopped there too. Cancels still arrived through the #1564 poll.
Possible fix: run pollForSettingChanges in notifier mode too, at a slow interval like JobCancelPollInterval, and put listenerTimeout on the ping.
Happy to send a PR for the Go client if this direction works.
Activity
brandur commentedon Nov 4, 2026
Thanks. Just OOC, are you running into these problems in real life or is it more theoretical? I'm surprised that notifiers reconnecting happens often enough to cause issues.
yash-bitla commentedon Nov 4, 2026
Theoretical on my side: I found it with a fault-injection test (killing the LISTEN backend, and freezing its socket through a TCP proxy) while looking at a similar lost-NOTIFY issue in another Postgres queue, not from a production incident.
I agree reconnects are rare. What made me file it is the failure mode: a lost cancel now heals within a minute thanks to #1564, but a lost pause doesn't heal until the client restarts, and an operator pausing a queue during an incident is exactly when a database failover or restart is likely.
The fix I have is small: run pollForSettingChanges with a notifier too, at a 1-minute interval like the cancel poll, plus a timeout on the notifier's ping. Happy to open it if you think it's worth it, or close this if you'd rather not. @brandur
brandur commentedon Nov 4, 2026
Okay, thanks. Yeah, it's still worth fixing for sure.
I had my bot build an example fix to see what it'd look like, but happy to take one as well if you want to try your hand at it, so sure, send one over! And can you please also sign our CLA: https://github.com/riverqueue/rivercla Thx!
yash-bitla commentedon Nov 4, 2026
Thanks @brandur! I signed the CLA in riverqueue/rivercla#43.
I'll send the PR over shortly. It runs
pollForSettingChangeswith a notifier too (on a 1-minute interval, like the cancel poll), and adds a timeout to the notifier's ping so a frozen listener reconnects. Both changes have tests.Poll queue settings as a fallback when a notifier is configured (rive…