Epic: staged private per-step queues and queue identity

[jumski]

Outcome

Deliver private per-step queues in stages while preserving one-flow/one-queue execution for plain flows. #650 preserves existing slug validation except normalized default-flow uniqueness; #651 adds the per-step naming restrictions. The combined release requires a coordinated package/database upgrade:

one concrete flow
  -> one private queue per step
  -> one independently configured worker pool per step

This removes cross-step starvation while preserving one typed DAG, automatic dependency release, and one run. Stable aliases, custom queue names, and multi-flow sharing remain deferred follow-ups that require separate evidence.

Why staged delivery

The terminal-state, claim-visibility, and stalled-recovery prerequisites shipped in 0.15.1. #647 removed the legacy compiler through #672 and shipped separately in 0.16.0 through #674.

The queue work starts from 0.16.0. #650 stops treating flow_slug as queue identity and extends existing claiming with stored queue/message identity. #651 adds generated private per-step queues and exact step selection on that foundation. Neither stage carries a temporary legacy compiler or a general worker protocol.

#650's approved body defines Stage 1 implementation scope. This epic owns sequencing and the combined release gate; it does not add rejected inspection, classification, or audit requirements back into #650.

A downstream validation then decides whether private per-step queues are useful enough to release and whether any deferred routing work has a concrete use case.

Current ownership model

Flow
  DAG, handlers, conditions, and execution semantics

StepQueuedFlow
  opt-in private queue-per-step deployment mode

Persisted step route
  generated physical queue for future tasks and ownership evidence

Task
  concrete flow slug and queue snapshot unchanged by runtime code

Plain flow worker
  one concrete flow and its generated default queue

Step worker
  one concrete flow and one generated private step queue

The current stages add no pgflow.queues registry. Generated private ownership derives from the concrete flow's default queue or complete persisted step route.

Compatibility defaults

plain Flow
  every step queue = lower(flow.slug)
  worker queue = the same generated default
  no worker selector

withStepQueues(flow)
  every step gets a generated private queue
  worker requires typed stepSlug

No queue option is added to .step(), .array(), or .map() in these stages.

Committed stages

Correctness prerequisites

• Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #638 — skipped paths terminalize their task rows, fixed by Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #649 on 2026-08-28.
• Failed runs leave unfinished sibling tasks queued or started instead of cancelled #645 — failed runs terminalize unfinished task rows and recovery ignores terminal runs and steps, fixed by fix(core): terminalize unfinished tasks when runs fail #663 on 2026-09-02.
• requeue_stalled_tasks() uses flow timeout instead of effective step timeout #621 — stalled recovery uses the effective step timeout, fixed by fix(core): use effective step timeout for stalled tasks #664 on 2026-09-02.
• Make start_tasks() apply task visibility before handlers run #656 — start_tasks() applies task visibility before handlers run, fixed by fix(core): guarantee task visibility extension in start_tasks() #667.

The complete correctness set shipped in 0.15.1 through #666.

Stage 0 — startup-only compilation (released)

• Remove legacy flow compilation before queue identity and routing #647 — Removed ControlPlane, migration-based flow compilation, and the obsolete FlowWorkerConfig.compilation option; implemented by refactor: make worker startup the only flow compilation path #672 and released in 0.16.0 through Version Packages #674.

Queue identity and route metadata extend only this startup compiler. The published 0.16.0 worker already calls the two-argument startup function; removal of the older three-argument function does not reject that worker.

Stage 1 — database queue identity

• Persist physical queue identity for flow tasks #650 — Persist canonical queue identity on steps and task queue snapshots that runtime code leaves unchanged.

This stage preserves one-flow/one-queue routing and adds no queue registry. Dispatch uses the persisted step route; later message operations use each task's (queue_name, message_id). It owns normalized default-flow uniqueness, nullable message IDs, pair uniqueness, exact JavaScript IDs, read-once queue-aware claiming, and a focused transactional maintenance upgrade. It preserves existing slug validation otherwise and existing startup signatures where practical. Database-enforced identity immutability belongs to #678, not this release.

Stage 2 — private per-step queues

• Add private per-step queues with typed step workers #651 — Add withStepQueues(flow) and typed per-step workers.

Generated queues are lowercase, private to one concrete flow, collision-checked, and at most 47 characters. #651 owns the additional shared naming restrictions and queue mode/route-map startup checks. Plain flows keep the same worker API and routing, subject to those documented naming restrictions.

Validation gate

• Validate one concrete StepQueuedFlow in a downstream application or focused spike before the stable Stage 2 release.

The validation must measure whether generated private queues remove meaningful application-owned queue and worker code while preserving domain correctness. It must not require aliases, named shared queues, or multi-flow workers.

Choose the validation application or focused application spike before this gate, not before #650. Older linked application plans that require aliases or shared routing do not add prerequisites to the committed queue stages. An internal two-step queue test provides early feedback but does not substitute for application-level validation.

Required rollout documentation

• Document the worker-function enable fence for production deployments #654 — Document the current one-function enable fence for in-place production replacement, fixed by docs: fence worker_functions row before deprecating workers in update guide #670.
• Add private per-step queues with typed step workers #651 — Document complete per-step replacement, new-version queue coverage, and old-version drain.

#654 is complete. Finish #651's rollout documentation before the first stable release that changes worker placement.

Independent integration parity

• Allow flow handlers to select the next retry delay #655 — Allow handlers to select the next retry delay.

#655 does not change queue architecture or block #650/#651 implementation. If the chosen application needs provider-selected Retry-After, implement the minimum delay override before claiming that application's retry parity, or report the gap explicitly. Keep attempt accounting unchanged; do not treat a retry-policy gap as evidence that private queue placement failed.

Deferred follow-ups

• Add stable flow aliases through DeployedFlow startup deployment #648 — stable aliases through DeployedFlow;
• Add explicit shared queues with multi-flow dispatch #652 — explicit shared queues and multi-flow dispatch;
• Design queue-less manual completion tasks (queue: false) #661 — queue-less manual completion tasks.

These issues are not completion requirements or prerequisites for this epic. Their APIs, storage models, and release order remain separate decisions. Start #648 or #652 only after Stage 2 validation records a concrete need; #652 also requires explicit evidence that grouping or sharing queues is valuable.

Cross-stage invariants

Startup compilation

The released #647 implementation makes worker startup the only supported flow compiler. Worker startup sends the complete ordered structural definition. #650 preserves this path and existing signatures where practical through small additive changes. #651 adds queue mode and the complete ordered route map as deployment metadata, checked and persisted under the concrete flow lock before polling.

Production shape/mode/route mismatches fail without destructive recompilation. Existing automatic local same-slug recompilation remains unchanged. Neither stage intentionally rejects released workers solely to establish a future protocol version. Handler code and TypeScript contracts that shape comparison cannot observe still require developer-managed concrete versioning.

Queue identity

PGMQ message identity is:

(queue_name, message_id)

Dispatch sends through the persisted step route and copies it onto the task. Later message operations use that task snapshot, not a queue reconstructed from flow_slug. Runtime updates leave snapshots unchanged; #678 owns database-enforced immutability. Keep message_id nullable and enforce non-null queue/message pair uniqueness. JavaScript represents message IDs as exact decimal strings.

Generated private ownership

Generated default and per-step queues are private pgflow resources. Ownership derives from the concrete flow's persisted default or complete step route rather than a separate registry.

For a missing concrete definition, reject an already listed queue with the same normalized generated name. An exact existing definition may reuse its queue. Use pgflow's existing definition locks and constraints to make normalized flow uniqueness and cross-flow route checks atomic. Resolve metadata spelling through public pgmq.list_queues() when needed, and reject ambiguous listed matches before destructive work.

Use public PGMQ APIs for creation, listing, and deletion. Trust those operations instead of inspecting columns, indexes, sequences, extension membership, or physical queue layout. Do not lock pgmq.meta or add direct PGMQ row locks to claiming. Existing set_vt_batch() and direct archive pruning remain accepted integrations.

Document exclusive pgflow queue management: applications must not send directly to these queues or independently create, replace, or alter them. Protection against concurrent external PGMQ changes is outside these stages.

Whole-flow deletion uses persisted routes and the default queue for an empty plain flow. Retain existing synchronization and drop private queues and archives transactionally; required PGMQ failures roll back deletion. Do not archive individual messages immediately before dropping their entire queue and archive. #650 also updates the optional manually installed pruning snippet and provides manual replacement/adaptation instructions, without source fingerprints or automatic replacement.

If #652 is adopted, it owns the minimum metadata needed for explicit shared queues without changing task identity again.

Per-step naming rules and focused upgrade checks

#650 preserves accepted spelling and existing slug validation, while atomically rejecting distinct flows that share a normalized default queue. It does not add the following restrictions. #651 owns their implementation, focused migration checks, and documentation for plain and step-queued flows.

Use one validation contract for flow and step slugs in TypeScript and SQL:

• no leading or trailing _;
• no __ anywhere;
• single internal underscores and camelCase remain valid;
• existing character, leading-digit, length, and reserved-word restrictions remain;
• reject case-only duplicate step slugs within one flow; retain Persist physical queue identity for flow tasks #650's normalized default-flow uniqueness.

__ is reserved for generated queue names. Do not add internal-step flags, alternate validators, or bypasses for hypothetical ghost steps.

Preserve accepted spelling and exact references; only generated queue names are lowercase. #651's new restrictions are a breaking validation change. Its focused migration checks cover definitions that violate those rules, including definitions without active runs.

#650's migration checks only what its new columns and constraints need: backfill steps/tasks with lower(flow_slug), including tasks with null message IDs, then enforce non-null, queue-name validity, pair uniqueness, and normalized default-flow uniqueness. New-constraint violations fail transactionally without renaming, deleting, or repairing data. Use bounded migration lock waits. Neither stage needs a separate general-purpose audit, PGMQ body scan, or physical-object inspection.

Generated per-step names

Use one fixed compatibility contract:

maximum length = 47
readable = lower(flow_slug || '__' || step_slug)
fallback = lower(flow_slug || '__' || zero_based_step_index)

Use the readable name when it fits, otherwise the index fallback. Reject the complete flow before mutation if both exceed 47.

TypeScript mirrors the algorithm for synchronous feedback. SQL remains authoritative and also calls pgmq.validate_queue_name().

Flow immutability and versions

A concrete flow_slug identifies one immutable production shape and queue mode. Shape or queue-mode changes require a new concrete slug.

Generated queues use concrete slugs. Existing tasks never move to another concrete version.

Worker coverage

One worker polls one queue. A private step worker supports one exact (flow_slug, step_slug) pair.

Compilation does not wait for every worker. Missing workers leave tasks queued durably. Startup logs identify only the worker's selected flow, step, and queue. #651 supplies a copyable post-deployment query that compares persisted routes with live worker rows; operators check coverage before switching callers to a new concrete version. Add no startup coordinator or global readiness protocol.

Read once, claim by stored identity

#650 extends the existing claim operation with the actual polled queue, expected flow, message IDs, and worker identity. #651 adds the exact step selector. Workers read through PGMQ once; claiming uses persisted queue/message identity and existing eligibility, not message bodies or a second queue read.

Message/task state	Behavior
Exact eligible queued task for the subscription	Claim once; only successfully claimed rows reach handlers.
Matching started task	Consume no attempt; leave completion and recovery in charge.
Matching terminal or otherwise ineligible task	Do not execute or revive it; preserve existing lifecycle cleanup behavior.
No matching task	Preserve the message, warn with queue/message IDs, and continue valid work.
Matching task belongs to another flow or selected step	Never claim, mutate, or archive it through this worker; warn and skip.

Recurrence after the normal visibility timeout is accepted for unknown or wrong-route messages until an operator handles them. Ordinary database failures remain retryable. Warnings include identifiers, never bodies. Do not add body classification, automatic foreign-message archival, batch-wide fatal outcomes, forced unknown-message visibility resets, or persistent HTTP restart pauses.

Preserve current timing: successful claims use the effective step timeout plus 2 seconds; stalled recovery uses the effective timeout plus 30 seconds. A repeated read of a started task must not restart its recovery deadline, consume another attempt, or stop a healthy worker. Keep existing pgflow synchronization unless queue identity needs a specific adjustment; these stages are not a broad lifecycle or lock-order rewrite.

Production rollout

Use #654's enable fence for current in-place worker replacement. #651 extends it to the complete affected step-worker set and documents new concrete-version drain.

• In-place replacement disables, drains, deploys, and restores the affected functions.
• A new concrete version starts healthy workers on independent queues before caller switching; old workers remain until old work drains.

Do not add a cross-worker activation protocol. Durable queues hold work while cooperating workers start.

Type-quality requirements

The committed APIs must:

• preserve exact flow handler, dependency, condition, context, and environment inference;
• preserve the exact step-slug union through withStepQueues();
• require stepSlug only for private step-queued workers;
• perform complete synchronous generated-name checks before database access;
• keep SQL authoritative for the complete route.

Do not add a flow-slug generic or conditional string types solely for generated queue-name validation.

Deferred alias and shared-queue issues own their own type contracts if adopted.

Integration validation

The downstream validation must preserve application domain tables, transitions, claims, and terminal outcomes while replacing queue and worker mechanics.

It must report:

• orchestration code removed and retained;
• direct PGMQ references removed and retained;
• independent capacity and starvation behavior;
• success, retry, crash, ambiguity, and concurrency behavior;
• retry and recovery gaps;
• type and autocomplete quality;
• an adopt, revise, or stop recommendation for Stage 2.

An adopt result permits the stable private-queue release. A revise result adds only the missing scoped capability and repeats validation. A stop result publishes neither #650 nor #651; #647 remains part of the released 0.16.0 baseline. Any alias or shared-queue proposal still needs its own concrete justification.

Development and release sequence

Use sequential task deliveries: #650 implementation, review, and merge, then #651 implementation, review, and merge. Follow the repository's current delivery workflow with one issue and one editing implementer at a time; this epic is not one implementation task. Implement #650 from main using its approved requirements, not the rejected implementation branch or its old plan. Keep #651's step selector additive. Use the repository's focused, batch, and final check cadence rather than a full release rehearsal for every internal slice.

0.16.0 baseline: lifecycle hardening + #647 (released)
  -> #650 queue identity, focused upgrade fixture, and queue-aware claiming
  -> #651 private per-step queues and typed workers
  -> start application validation after the first working private-queue slice
  -> close scoped gaps and finish rollout documentation
  -> finalize the queue migration and freeze the combined release candidate
  -> validate that exact candidate downstream and run final checks
  -> adopt: release
     revise: make the scoped change and repeat affected validation
     stop: remove unreleased #650/#651 changes; retain released #647
  -> merge the Changesets release PR for the stable queue release

#650 and #651 each add the appropriate Changeset, but the generated Version Packages PR remains unmerged until this sequence completes. #650 may remain operational if #651 is delayed, but it does not publish alone. If separate branches need intermediate migrations, follow the repository's temporary-migration workflow and finalize/consolidate the queue migration before exact-candidate validation. Never rewrite a released migration.

Use a coordinated maintenance upgrade from 0.16.0, not an old/new rolling-worker promise: stop and drain workers, pause producers and definition/maintenance/recovery writers, apply the migration through Supabase's migration runner, replace or adapt any installed pruning helper, deploy matching packages, and resume. Test populated upgrade/backfill, atomic rollback on new-constraint violations, and plain-flow startup/execution after upgrade. #651 adds the actual naming and mode/route checks needed for per-step routing; this epic does not require #650 to reject released workers or introduce a protocol handshake.

Acceptance criteria

• Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #638, Failed runs leave unfinished sibling tasks queued or started instead of cancelled #645, requeue_stalled_tasks() uses flow timeout instead of effective step timeout #621, and Make start_tasks() apply task visibility before handlers run #656 establish truthful terminal state, dispatchable-only recovery, effective timeouts, and reliable claim visibility before queue work.
• Remove legacy flow compilation before queue identity and routing #647 removed the legacy compiler and optional worker compilation and shipped in 0.16.0.
• Persist physical queue identity for flow tasks #650 persists canonical queue identity without a speculative queue registry.
• Persist physical queue identity for flow tasks #650 preserves plain flow APIs, routing, and existing slug validation except normalized default-flow uniqueness; Add private per-step queues with typed step workers #651 documents its additional shared naming restrictions.
• TypeScript and SQL apply Add private per-step queues with typed step workers #651's naming rules consistently; focused transactional migration checks reject new-constraint violations without renaming, deleting, or repairing data.
• A populated 0.16.0 fixture upgrades successfully, invalid new identities roll back atomically, and plain-flow startup/execution pass afterward without a forced protocol-version boundary.
• Add private per-step queues with typed step workers #651 provides private per-step queues and independent worker capacity without aliases or shared registries.
• Generated names follow the lowercase 47-character contract and fail atomically with actionable errors.
• Every task stores its step's queue snapshot, including tasks with null message IDs; runtime updates leave snapshots unchanged. Enforcement triggers remain deferred to Enforce immutability of persisted pgflow identities #678.
• Claims use exact stored queue/message identity and flow/step eligibility without queue rereads or body classification; wrong-route and unmatched messages warn and remain untouched while valid work continues.
• Started/terminal safeguards, claim visibility, recovery deadlines, and ordinary database retry behavior remain intact without a fatal-message/HTTP-pause protocol.
• Whole-flow deletion avoids redundant archival; installed pruning functions have an explicit upgrade path.
• pgflow rejects listed external-name collisions, ambiguous normalized matches, and cross-flow references using its own locks/constraints and public PGMQ APIs; concurrent external changes remain outside scope.
• Downstream validation records an explicit Stage 2 recommendation.
• Document the worker-function enable fence for production deployments #654's current fence and Add private per-step queues with typed step workers #651's per-step rollout documentation are complete before the stable worker-placement release.
• Downstream validation records the exact candidate containing Persist physical queue identity for flow tasks #650 + Add private per-step queues with typed step workers #651 on the released 0.16.0 baseline, including the final migration artifact.
• The Changesets release PR remains unmerged until Persist physical queue identity for flow tasks #650, Add private per-step queues with typed step workers #651, downstream validation, rollout docs, and final checks finish.

Out of scope

• Stable aliases.
• Explicit named or shared queues.
• Multi-flow worker registries.
• Mutable routing for an existing concrete version.
• Polling several queues from one worker instance.
• Automatic shared-queue deletion.
• Dynamic handler loading.
• Runtime input-schema enforcement.
• Manual approval or external completion in the current stages; tracked separately in Design queue-less manual completion tasks (queue: false) #661.
• Database-enforced identity immutability (Enforce immutability of persisted pgflow identities #678), broad PGMQ inspection or locks, general upgrade audits, body classification, and fatal-message shutdown protocols.