Persist physical queue identity for flow tasks

[jumski]

Goal

Persist physical queue identity independently of flow identity, while keeping today's one-flow/one-queue behavior.

PGMQ message IDs are queue-scoped. A queued task's message identity is (queue_name, message_id), not message_id alone.

Implement from main. This is the foundation for private per-step queues in #651, not an implementation of that feature. #653 owns the combined release gate; this issue body defines the implementation scope.

Persisted identity

Add:

pgflow.steps.queue_name       text not null
pgflow.step_tasks.queue_name  text not null

• Store canonical lowercase queue names. For this stage, every step's queue is lower(flow_slug).
• When a task is created, copy its step's queue name. Runtime code never changes that snapshot.
• Keep message_id nullable and retain the task primary key (run_id, step_slug, task_index).
• Enforce uniqueness of (queue_name, message_id) where message_id is not null.
• Do not add queue identity to runs or step states, a queue registry, or an immutability trigger. Database-enforced immutability belongs to Enforce immutability of persisted pgflow identities #678.

Names and provisioning

Preserve accepted flow/step spelling, camelCase, and existing slug validation. Defer new underscore restrictions and case-insensitive step uniqueness to the per-step naming work in #651.

Two concrete flows must not address the same normalized default queue. Enforce that constraint atomically in pgflow, including direct SQL creation. Existing conflicting definitions block migration; do not rename or delete them automatically.

Use the existing 47-character queue-name limit and PGMQ's public name validation. Preserve the current provisioning lifecycle: create_flow() creates the default queue, including for an empty flow; add_step() records its resolved default route. Task dispatch performs no queue DDL. Keep production shape-mismatch rejection and existing local recompilation behavior.

Use public PGMQ APIs for creation, listing, and deletion:

• Before creating a new flow, reject an already listed queue with the same normalized name. An existing matching flow may reuse its queue.
• Coordinate pgflow's own definition operations with its existing locking approach, using normalized identity where needed. Do not lock pgmq.meta.
• Existing mixed-case queue names must remain usable. Where a public operation needs the original spelling, resolve it through pgmq.list_queues(); do not create a second metadata entry just to lowercase it. Reject an ambiguous listed match before destructive work.
• Trust PGMQ to implement its queue operations. Do not inspect table columns, indexes, sequences, extension membership, or physical queue layout.
• Document that pgflow manages its queues exclusively. Applications must not send directly to them or independently create, replace, or alter them. Protection against concurrent external PGMQ changes is outside this feature.

Use existing foreign keys and constraints when they already reject an invalid write atomically. Do not add dependency-existence queries solely to replace a database error with a custom error.

Queue-aware lifecycle

Task creation sends through steps.queue_name and persists that same value on the task. After creation, message operations use step_tasks.queue_name, not a queue reconstructed from flow_slug.

Apply this consistently to claiming, visibility, completion, failure, retries, skip/cancel cascades, late callbacks, stalled recovery, and task pruning. Batch operations across several tasks by queue; use ordinary single-task operations for one task.

Preserve existing terminal-state guards, attempt accounting, timeout behavior, and concurrency safeguards. Successful claims use the effective step timeout plus 2 seconds for visibility; stalled recovery retains the effective timeout plus 30 seconds. Only rows successfully claimed in the transaction may reach handlers.

Keep existing synchronization unless a queue-identity change needs a specific adjustment. This is not a broad rewrite of callback, recovery, or lock ordering. Do not add direct PGMQ row locks to claiming.

set_vt_batch() and the optional archive-pruning helper are accepted, focused PGMQ integrations. Their existing direct table access may remain; this feature does not replace them with slower per-message loops or a new abstraction.

Read once, claim by stored identity

The worker reads messages once through PGMQ. The claim operation receives the actual polled queue, expected flow, message IDs, and worker identity. Extend the existing operation rather than requiring a new function name or a general worker protocol.

Use persisted (queue_name, message_id) plus flow identity and existing task/run/step eligibility to select work. A message body does not authorize a claim. Represent message IDs as exact decimal strings at the JavaScript boundary and cast them to PostgreSQL bigint for SQL calls.

Required behavior:

Message/task state	Behavior
Exact eligible queued task for the worker's flow	Claim once and apply existing visibility guarantees before handler execution.
Matching started task	Do not claim again or consume another attempt; leave completion and recovery in charge.
Matching terminal or otherwise ineligible task	Do not execute or revive it. Preserve existing lifecycle cleanup behavior.
No matching task	Preserve the message, warn with queue/message IDs, and continue valid work from the batch.
Matching task belongs to another flow	Never claim, mutate, or archive it through this worker. Warn and skip it.

Unknown messages may recur after their normal visibility timeout until an operator handles them. That is accepted. Warnings contain identifiers, not bodies. Ordinary database errors retain existing retry behavior.

Do not add message-body classification, a second queue read, automatic foreign-message archival, batch-wide fatal outcomes, forced visibility resets for unknown messages, or persistent HTTP restart pauses.

Deletion and optional pruning

Delete a flow's runtime data and its private queue transactionally, using its persisted routes and default queue for an empty flow. Retain existing deletion synchronization and fail atomically if a required PGMQ operation fails. Do not archive individual messages immediately before dropping their entire queue and archive.

Update the optional pgflow.prune_data_older_than(interval) snippet and its tests to use task snapshots for message cleanup and persisted definition routes for archive cleanup. Keep it manually installed. Provide replacement instructions and tell users to adapt customized copies themselves; do not add source fingerprints or automatic replacement.

Migration and upgrade

Use the repository's schema-first migration workflow. Preserve released migrations.

1. Add the queue columns in nullable form.
2. Backfill existing steps and tasks with lower(flow_slug), including tasks whose message_id is null.
3. Enforce non-null, queue-name validity, queue/message uniqueness, and normalized default-flow uniqueness.
4. Install the corresponding functions and refresh generated artifacts.

Make the migration transactional. Duplicate identities or other violations of the new constraints leave the database unchanged. Do not rename definitions, recreate queues, or repair data to make the upgrade pass.

Document a maintenance upgrade from 0.16.0: stop and drain workers, pause producers and definition/maintenance/recovery writers, apply the migration through Supabase's migration runner, update any installed pruning helper, deploy matching packages, and resume. Use bounded migration lock waits. No online-upgrade guarantee is needed.

Limit migration checks to what the new columns and constraints need. Do not scan PGMQ message bodies, audit physical objects, or build a separate general-purpose pre-upgrade audit system. Existing unrelated queue damage is not something this migration repairs or diagnoses.

Keep plain Flow and EdgeWorker.start(flow, config) APIs and default routing. Preserve existing startup and plain SQL call signatures where practical through a small additive change. Do not intentionally reject released workers solely to establish a future protocol. This does not promise a mixed-version rolling upgrade; #651 owns any new startup contract required by per-step routing.

Checks and completion

Add focused regressions alongside existing tests:

• Identity: snapshot creation, unchanged snapshots across lifecycle updates, nullable message IDs, duplicate-pair rejection, and exact JavaScript IDs beyond the safe integer range.
• Isolation: the same message ID in different queues never crosses task identity; wrong-flow and unmatched messages remain untouched while valid work continues.
• Lifecycle: completion, failure, retries, skip/cancel, late callbacks, and recovery use the stored queue; existing concurrent-claim and terminal-state safeguards still pass.
• Provisioning and cleanup: default and empty flows, visible external-name collisions, normalized flow collisions, mixed-case existing queues through public APIs, deletion, and the updated optional pruning helper.
• Upgrade and execution: a populated 0.16.0 fixture backfills successfully, a new-constraint violation rolls back atomically, and plain-flow startup and execution pass after upgrade.

Use internal SQL test fixtures to prove queue isolation without adding a public custom-routing API. Run repository-required affected checks. Update documentation for changed behavior, upgrade steps, and the supported queue-management boundary.

Out of scope

Per-step workers or naming, shared/custom queues, mutable routes, manual completion, a general ownership/corruption framework, new immutability enforcement, a general fatal-message protocol, and unrelated lifecycle refactoring. Linked follow-up issues do not add these requirements back into #650.