feat(ups): implement queue subs

[MasterPtato]

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[railway-app]

🚅 Deployed to the rivet-pr-4486 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
kitchen-sink	❌ Build Failed (View Logs)	Web	Mar 31, 2026 at 1:41 am
frontend-cloud	😴 Sleeping (View Logs)	Web	Mar 29, 2026 at 5:37 am
website	😴 Sleeping (View Logs)	Web	Mar 24, 2026 at 12:15 pm
frontend-inspector	😴 Sleeping (View Logs)	Web	Mar 24, 2026 at 7:28 am
mcp-hub	✅ Success (View Logs)	Web	Mar 24, 2026 at 12:31 am
ladle	❌ Build Failed (View Logs)	Web	Mar 24, 2026 at 12:31 am

[MasterPtato]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• chore: envoy client #4521 : 2 dependent PRs (#4531 , #4534 )
• feat(pb): actors v3 #4463
• feat(ups): implement queue subs #4486 👈 (View in Graphite)
• feat(cache): add in flight deduping #4459
• fix(cache): clean up lib #4457
• fix(config): allow configuring topo dcs via map, fix pg ssl mode config #4456
• fix(api): dont 500 on cross dc response #4449
• chore: update byte units on frontend #4444
• fix: runner alloc idx logic, api auth for actor get #4443
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pkg-pr-new]

More templates

• actor-actions
• actor-actions-vercel
• ai-agent
• ai-agent-vercel
• example-ai-and-user-generated-actors-freestyle
• chat-room
• chat-room-vercel
• example-cloudflare-workers
• example-cloudflare-workers-hono
• collaborative-document
• collaborative-document-vercel
• cross-actor-actions
• cross-actor-actions-vercel
• example-cursors
• example-cursors-raw-websocket
• example-cursors-raw-websocket-vercel
• example-cursors-vercel
• custom-serverless
• example-elysia
• example-experimental-durable-streams-ai-agent
• example-experimental-durable-streams-ai-agent-vercel
• geo-distributed-database
• geo-distributed-database-vercel
• hello-world
• hello-world-vercel
• example-hono
• hono-react
• hono-react-vercel
• example-kitchen-sink
• example-kitchen-sink-vercel
• multi-region-vercel
• multiplayer-game
• example-multiplayer-game-patterns
• example-multiplayer-game-patterns-vercel
• multiplayer-game-vercel
• native-websockets
• native-websockets-vercel
• next-js
• per-tenant-database
• per-tenant-database-vercel
• example-raw-fetch-handler
• example-raw-fetch-handler-vercel
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-raw-websocket-handler-proxy-vercel
• example-raw-websocket-handler-vercel
• example-react
• example-react-vercel
• sandbox
• sandbox-coding-agent
• sandbox-coding-agent-vercel
• sandbox-vercel
• scheduling
• scheduling-vercel
• example-sqlite-drizzle
• example-sqlite-raw
• state
• state-vercel
• example-stream
• example-stream-vercel
• example-trpc

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@4486

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@4486

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@4486

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@4486

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@4486

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@4486

@rivetkit/sqlite-vfs

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sqlite-vfs@4486

@rivetkit/traces

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/traces@4486

@rivetkit/workflow-engine

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/workflow-engine@4486

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@4486

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@4486

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@4486

commit: ec32bdf

[claude]

PR Review: feat(ups): implement queue subs

The core design is solid. Using FOR UPDATE SKIP LOCKED for Postgres-backed competing consumers is the right approach, and the heartbeat/TTL pattern for publisher-side discovery is appropriate given Postgres NOTIFY's lack of built-in queue semantics. A few issues need attention before merging.

Critical

DefaultHasher is not stable across processes or Rust versions
(engine/packages/universalpubsub/src/driver/postgres/mod.rs)

DefaultHasher's output is explicitly not guaranteed to be deterministic across processes or Rust versions. Using it to compute subject_hash and queue_hash stored in ups_queue_subs / ups_queue_messages means two processes, or the same process after a Rust upgrade, may compute different hashes for the same string, silently breaking all queue routing. The existing hash_subject path has the same pre-existing bug, but extending it to persisted queue keys makes this a data-integrity issue.

Fix: Replace with a stable, deterministic hash (e.g., FNV-1a or truncated SHA-256 formatted as hex).

Medium

No metrics update in queue subscription cleanup task (spawn_queue_subscription_cleanup_task)

spawn_subscription_cleanup_task updates metrics::POSTGRES_SUBSCRIPTION_COUNT when removing entries. The queue analog does not update any counter after removing from queue_subscriptions. This is an observability gap.

Unresolved TODO: retain_sync inside retain_async (engine/packages/universalpubsub/src/driver/memory/mod.rs)

The TODO comment is left in the code. retain_sync inside the retain_async closure blocks the async executor thread for the duration of the inner lock. For a 60-second GC interval this is acceptable (no deadlock risk since the nested maps are distinct objects), but the TODO should be replaced with a comment explaining why it is safe.

No queue depth limit (engine/packages/universalpubsub/src/driver/postgres/mod.rs)

QUEUE_MESSAGE_MAX_AGE_SECS is 3600s and GC runs every 5 minutes. If a consumer is down, messages accumulate unbounded for up to an hour. High-throughput queues with dead consumers could grow the table very large. Worth noting as a design-level concern even if not a blocker.

Low

Idiomatic Rust: return Ok(...) vs expression form (engine/packages/universalpubsub/src/pubsub.rs:136)

return Ok(Subscriber::new(...)) should be the bare expression Ok(Subscriber::new(...)) to match the style of the surrounding subscribe method.

SQL injection pattern is fragile (engine/packages/universalpubsub/src/driver/postgres/mod.rs)

NOTIFY "{}" with channel built from format! is safe today only because the hash output is hex. The same pre-existing pattern is now copied into two additional call sites. A helper that validates the channel name before interpolating would prevent this from becoming a vulnerability if the input ever changes.

Unused anyhow! import (engine/packages/universalpubsub/src/driver/postgres/mod.rs:1)

If all usages of anyhow! were replaced by .context() per the project convention, remove anyhow from the import line.

Informational

• Memory metric counts topics, not subscribers. MEMORY_SUBSCRIBER_COUNT uses self.subscribers.len() + self.queue_subscribers.len() which counts distinct topic entries, not individual sender handles. Pre-existing issue, but the surface is now expanded.
• Test timing sensitivity. The load-balance test uses a 500ms collection window per subscriber (vs. 5s in test_queue_subscribe_single). This may be flaky on slow CI/Postgres environments with 10 messages across database round-trips.
• NATS driver is minimal and correct - clean delegation to the NATS client.

Summary

Severity	Issue
Critical	DefaultHasher used for persistent DB keys - not stable across processes/versions
Medium	No metrics update in queue subscription cleanup task
Medium	retain_sync inside async retain - TODO unresolved
Medium	No queue depth limit; unbounded growth if consumers are down
Low	return Ok(...) vs idiomatic expression form
Low	SQL NOTIFY injection pattern fragile (safe today, not future-proof)
Low	Unused anyhow! import if all usages replaced
Info	Memory metric counts topics not subscribers (expanded surface)
Info	Load-balance test 500ms window may be flaky on slow CI

The DefaultHasher stability issue is the most important fix - it will silently corrupt queue routing in any multi-process deployment.

[claude]

Code Review summary for feat(ups): implement queue subs -- The implementation is well-structured and the test coverage is solid. Two medium issues to address before merge: (1) retain_sync on scc::HashMap inside the memory driver GC task runs synchronously inside an async closure and can block the executor; (2) The SELECT and INSERT+NOTIFY in publish_to_queues are not wrapped in a transaction, creating a race where a message can be orphaned for up to 1 hour if the subscriber dies between the two operations. Minor items: one remaining anyhow! macro call in publish() should use .context() instead; the queue parameter is missing from the tracing span in queue_subscribe (fields(%subject) should also include queue); and the Cargo.lock version downgrade from 2.1.7 to 2.1.6-rc.1 appears to be a stacked-PR artifact. On test coverage: test_queue_subscribe_load_balance only checks total message count, not per-subscriber distribution -- this is the right choice to avoid flakiness but worth noting in a comment. Also missing a test for Postgres reconnection behavior (does not drain pending ups_queue_messages on reconnect).

[claude]

PR Review: feat(ups): implement queue subs

Good implementation of queue subscriptions across all three drivers. The Postgres driver in particular is well thought out with atomic message claiming, heartbeating, and reconnection handling. A few issues worth addressing before merging.

---

BUG: Heartbeat Task Leaks After Subscriber Drop

In queue_subscribe in the Postgres driver, the heartbeat task uses a cloned CancellationToken for its stop signal, but the token stored in the struct (_heartbeat_token) is just a CancellationToken — not a DropGuard. Dropping a CancellationToken does NOT cancel it, so the heartbeat loop will run indefinitely after the subscriber is dropped.

The child token (heartbeat_token_child) inside the spawned task holds a reference, so neither token gets cancelled when the struct is dropped. Fix: store a DropGuard instead of the bare CancellationToken:

let heartbeat_drop_guard = heartbeat_token.clone().drop_guard();
// store _heartbeat_drop_guard: heartbeat_drop_guard in the struct

This means dropped queue subscriptions will keep heartbeating until the process exits, keeping dead rows alive in ups_queue_subs beyond the TTL.

---

Cleanup in Drop Has No Error Logging

The Drop impl for PostgresQueueSubscriber spawns a task to deregister from ups_queue_subs but silently ignores errors. Worth adding a tracing::warn if the delete fails.

---

Memory Driver Metric Counts Topics, Not Subscribers

metrics::MEMORY_SUBSCRIBER_COUNT
    .set((self.subscribers.len() + self.queue_subscribers.len()) as i64);

queue_subscribers.len() is the number of topics, not individual queue subscribers (which would require iterating the nested maps). This makes the metric misleading when there are multiple queue groups or multiple subscribers per group.

---

spawn_queue_subscription_cleanup_task Missing Metric Update

Unlike spawn_subscription_cleanup_task which calls metrics::POSTGRES_SUBSCRIPTION_COUNT.set(...) after cleanup, the queue variant does not update any metric on removal.

---

Minor: TODO Left in GC Loop

// TODO: Is retain_sync bad here?

Worth resolving before merge. The concern is valid — retain_sync inside retain_async blocks the async executor for the inner retain duration. For small maps this is likely fine, but a brief explanatory comment would be better than a TODO.

---

Minor: rx: Option<...> Is Unnecessary

PostgresQueueSubscriber stores rx: Option<broadcast::Receiver<Vec>>, but it is always Some at construction. Consider storing rx directly to avoid the unnecessary Option wrapping.

---

Minor: PostgresQueueSubscriber Visibility

pub struct PostgresQueueSubscriber is fully public but only used internally as Box. Should be pub(crate) or private.

---

Observation: Delivery Semantics

The FOR UPDATE SKIP LOCKED + DELETE approach means a message is gone the moment it is claimed, before the consumer processes it. If the consumer crashes after claiming but before processing, the message is lost. This is at-most-once delivery (matching NATS queue subscribe semantics). Worth a doc comment since it is a meaningful semantic choice.

---

Overall the implementation is solid. The Postgres driver correctly handles reconnection, wakeup fan-out, atomic claiming, and GC. The heartbeat leak is the one bug that should be fixed before this ships.