Add explicit shared queues with multi-flow dispatch

[jumski]

Goal

Allow explicit step-to-queue routing, including several steps on one queue and several concrete flows or versions on one queue.

This is a deferred follow-up to #653. Keep the useful routing capability; do not bundle it with queue adoption, a corruption detector, or a new worker shutdown protocol.

Dependencies and scope

• Persist physical queue identity for flow tasks #650 supplies persisted step routes, task queue snapshots, exact queue/message identity, and read-once claiming.
• Add private per-step queues with typed step workers #651 supplies private per-step routing and typed worker selection. Plain flows and withStepQueues() remain supported unchanged.
• Begin release work after Stage 2 validation records a concrete need for grouped steps or shared queues.
• Add stable flow aliases through DeployedFlow startup deployment #648 is optional composition, not a prerequisite. Grouped steps and multi-flow workers must work with concrete flows without stable aliases.

Routing API

A complete typed step-to-queue map is sufficient. The helper name is provisional:

const routed = routeFlow(flow, {
  queues: {
    classify: 'thread_processing',
    enrich: 'thread_processing',
    deliverSlack: 'slack_delivery',
  },
})

Here classify and enrich share one queue and worker pool. Every key must refer to a real step in flow.

• Require every step exactly once; reject missing or unknown keys in TypeScript and at the SQL boundary.
• Preserve exact step-key and queue-literal unions for autocomplete, plus existing handler, dependency, condition, context, and environment inference.
• Validate names synchronously and through PGMQ's public validation in SQL. Use canonical lowercase queue identity and the established 47-character compatibility limit. Do not build conditional string types to parse queue-name syntax.
• Allow several steps, concrete flows, or versions to select the same explicit queue. Explicit names remain separate from generated private queues.
• Persist the resolved route separately from DAG shape. Production route changes require a new concrete slug; existing task snapshots never move.

Worker API and startup

One worker polls one selected queue and imports the complete definitions it supports:

EdgeWorker.start([routedV1, routedV2], {
  queueName: 'thread_processing',
  maxConcurrent: 4,
})

A one-element array supports grouped steps in one flow. queueName autocompletes from the supplied routes.

Before registration:

1. Reject duplicate concrete slugs and a queue selector absent from the supplied routes.
2. Compile or verify each complete supplied definition through the existing startup compiler.
3. Derive the exact supported (flow_slug, step_slug) pairs from persisted routes for the selected queue.
4. Register against that queue and start polling.

Validate the supplied configuration, not every active alias, historical run, or recoverable task in the database. Startup does not prove global handler coverage. If startup fails after compilation, durable work remains queued.

If #648 is available, routed flows may also carry its alias wrapper. Preserve its explicit activation rules: compiling another version never replaces an existing alias target automatically.

Claiming and dispatch

Extend the existing queue-aware claim operation with an exact support set. A flow-only allowlist is insufficient because different steps of one flow can use different queues.

Inputs are the actual polled queue, message IDs, worker identity, and supported flow/step pairs. Use exact decimal-string message IDs at the JavaScript boundary and PostgreSQL bigint for SQL calls.

The worker reads messages once through PGMQ. Claims use persisted (queue_name, message_id) and existing task/run/step eligibility. Message bodies do not authorize work.

Message/task state	Behavior
Exact eligible queued task in the support set	Claim once and apply existing visibility guarantees before executing its handler.
Matching started task	Do not consume another attempt; leave completion and recovery in charge.
Supported terminal or otherwise ineligible task	Do not execute or revive it; preserve existing lifecycle cleanup.
No matching task	Preserve the message, warn with queue/message IDs, and continue valid work.
Task outside the worker's exact support set	Do not claim, mutate, or archive it; warn and continue supported work.

Unknown or unsupported messages may recur after their normal visibility timeout. Operators correct the registry or handle the message. This does not justify stopping the whole batch or worker.

Only successfully claimed rows reach handlers. Select the handler by returned concrete flow and step identity. Reuse existing input/context construction, completion, failure, retries, and SQL progression.

Preserve existing locks, terminal guards, attempt accounting, and timing: claim visibility uses the effective timeout plus 2 seconds; stalled recovery uses it plus 30 seconds. Ordinary database errors retain existing retry behavior. Do not add body classification, queue rereads, automatic foreign-message archival, forced unknown-message visibility resets, or persistent HTTP restart pauses.

Shared queue lifecycle

Add only the pgflow-owned metadata needed to distinguish declared shared queues from generated private queues and coordinate their reuse. #650 does not prebuild this metadata. Reuse persisted routes and existing pgflow locks/constraints rather than a general ownership framework.

• Use public PGMQ creation, listing, validation, and deletion operations. Resolve original metadata spelling through pgmq.list_queues() when needed; reject ambiguous normalized matches before destructive work.
• Provision an absent declared shared queue. Reuse a queue already declared shared through pgflow. Reject a listed external queue or a generated private queue rather than silently adopting or converting it.
• Keep declaration and pgflow route changes atomic. Trust PGMQ operations; do not inspect physical objects or lock pgmq.meta. Concurrent external queue creation, replacement, or alteration is outside the supported management boundary.
• Deleting a concrete flow cleans up only that flow's tracked messages using its task snapshots. Never drop a shared queue or its archive with one flow, including local recompilation. Preserve other flows' and unknown messages.
• Keep set_vt_batch() and direct archive pruning. Shared archive cleanup must not apply one flow's retention policy to another flow's messages; if ownership cannot be selected safely, leave shared archive retention to a separate queue-wide maintenance operation.

Applications must not send unrelated messages directly to pgflow-managed queues. Importing/adopting existing queues is deferred; this issue needs no message scan or empty-or-fully-tracked adoption API.

Coverage and rollout

Document the handler set required for each queue. Provide a copyable, point-in-time check against persisted routes and relevant runtime task snapshots so operators can find missing concrete flow/step handlers. This is an operator check, not a startup-wide gate, new service, or cross-worker readiness protocol.

Retain old handlers while old runs can still reach the queue or old tasks remain executable/recoverable. Durable unsupported work stays intact, but it will not progress until a matching worker is deployed.

Use #654's existing in-place replacement procedure for the complete affected worker-function set: record and disable it, drain workers, deploy registries with required old/new versions, then restore only previously enabled functions. When aliases are present, use #648's explicit compare-and-swap activation after operator coverage checks. Do not promise an old/new rolling-worker upgrade.

Use the repository's schema-first migration workflow for any required metadata. Preserve released migrations, use transactional focused constraint checks and bounded lock waits, and document the maintenance upgrade. Do not add a general pre-upgrade audit or automatic data repair.

Acceptance criteria

Routing and typing

• A complete typed route map supports two steps on one queue and another step on a separate queue.
• Missing/unknown steps and invalid queue selectors fail with actionable errors; queue literals and existing flow types remain available.
• Several concrete flows or versions share a queue without requiring aliases; plain and private per-step APIs remain unchanged.
• Persisted routes and historical task snapshots retain exact queue identity; production route mismatches fail non-destructively.

Runtime and lifecycle

• Each worker claims only its exact supported flow/step pairs and dispatches returned identities correctly.
• Mixed supported, unmatched, and unsupported batches process valid work without mutating unknown/unsupported work or stopping the worker.
• Started/terminal guards, exact large message IDs, attempts, visibility, recovery, and concurrent-claim behavior remain correct.
• Shared provisioning rejects visible external/private collisions through public PGMQ APIs; existing declarations reuse their queues.
• Deletion, local recompilation, and maintenance preserve other flows' messages and shared queues/archives.

Completion

• Tests cover grouped steps, shared flows/versions, incomplete registries, wrong routes, collisions, deletion, and upgrade rollback without body scans or a fatal-message protocol.
• Documentation includes grouped-step and multi-flow examples, the operator coverage check, and old-version drain/replacement instructions.
• Run repository-required affected checks and record the concrete use case that justifies release.

Out of scope

• Existing-queue adoption/import, physical PGMQ inspection, and protection against external queue mutation.
• Global startup coverage enforcement, fatal-message shutdown, and cross-worker activation coordination.
• Polling multiple queues per worker, dynamic handler loading, and mutable routes for an existing concrete version.
• Automatic shared-queue deletion or old-version removal; stable aliases are separate in Add stable flow aliases through DeployedFlow startup deployment #648.
• Manual completion (Design queue-less manual completion tasks (queue: false) #661), general immutability enforcement (Enforce immutability of persisted pgflow identities #678), and unrelated lifecycle refactoring.