Skip to content

atepg: deliver worker watch events via a transactional change feed - #934

Open
shrutiyam-glitch wants to merge 1 commit into
agent-substrate:mainfrom
shrutiyam-glitch:atepg-change-feed
Open

atepg: deliver worker watch events via a transactional change feed#934
shrutiyam-glitch wants to merge 1 commit into
agent-substrate:mainfrom
shrutiyam-glitch:atepg-change-feed

Conversation

@shrutiyam-glitch

Copy link
Copy Markdown
Collaborator

Fixes #920 partially.

Replaces per-write pg_notify with a worker_changes outbox table written in the same transaction, and LISTEN with a 100ms polling watcher.

Motivation

  • Scalability Bottleneck: pg_notify serializes the commits of all notifying transactions through a global lock held across the commit (including fsync). This artificially caps worker writes at ~600/s on Cloud SQL regardless of instance size, whereas our target is O(10K) worker updates/s.
  • Payload Limits: Bypasses the 8KB NOTIFY payload limit that previously caused writes to fail.
  • Reliability: A cursor-based polling watcher survives reconnects and failovers without missing events, which was a known flaw with the ephemeral LISTEN approach.

(Known Postgres pathology prior art: Recall.ai, DBOS).

Performance Improvement

WorkerUpdate @ 1,000 QPS , 1M workers (preloaded) — before vs after the change feed:

p50 p90 p95 p99
Before (per-update pg_notify) 40.3s 55.6s 61.2s 63.8s
After (change-feed table) 7.08 ms 8.04 ms 8.52 ms 27.5 ms

Changes Made

  • Schema: Added transactional change feed table worker_changes.
  • Write Path: Worker writes now append to the worker_changes feed inside the same transaction instead of calling pg_notify().
  • Watch Path: Replaced LISTEN in WatchWorkers with a polling watcher that queries the feed every 100ms.
  • Cleanup: Implemented a janitor process during polling to periodically delete old feed rows.
  • Tests: Updated atomicity tests to verify feed inserts instead of pg_notify payloads.
  • Tests pass
  • Appropriate changes to documentation are included in the PR

Replaces per-write pg_notify with a worker_changes outbox table written
in the same transaction, and LISTEN with a 100ms polling watcher.

pg_notify serializes the commits of all notifying transactions through a
global lock held across the commit (fsync included), capping worker
writes at ~600/s on Cloud SQL regardless of instance size — measured
p50 610ms at 1k updates/s vs 4.9ms with the feed, which sustains >2k/s.
The requirements target O(10K) worker updates/s. Also removes NOTIFY's
8KB payload limit (which failed the write) and lets watchers survive
reconnects/failovers via their cursor instead of missing events.

Known pathology with public prior art:
https://www.recall.ai/blog/postgres-listen-notify-does-not-scale
https://www.dbos.dev/blog/postgres-listen-notify-scalability
(upstream Postgres fix lands in PG 19; unavailable on Cloud SQL).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

db optimization: worker update latency degrades sharply under load due to per-write pg_notify

1 participant