diff --git a/ARCHITECTURE_GUIDE.md b/ARCHITECTURE_GUIDE.md index 44c9f400b..f9487bb08 100644 --- a/ARCHITECTURE_GUIDE.md +++ b/ARCHITECTURE_GUIDE.md @@ -60,7 +60,7 @@ const CompleteExample = new Flow<{ urls: string[] }>({ async (input, context) => { // context.env - Environment variables // context.shutdownSignal - Graceful shutdown signal - // context.stepTask - Current task details (run_id, step_slug, input, msg_id, task_index) + // context.stepTask - Current task details (run_id, step_slug, input, msg_id as decimal string, task_index) // context.workerConfig - Worker configuration (read-only) // context.sql - PostgreSQL client (Supabase preset) // context.supabase - Supabase client (Supabase preset) @@ -123,7 +123,7 @@ const CompleteExample = new Flow<{ urls: string[] }>({ export default CompleteExample; ``` -**Compilation**: Worker startup extracts the complete flow shape (`extractFlowShape()`) and PostgreSQL compiles or verifies it via `pgflow.ensure_flow_compiled(flow_slug, shape)` before any registration. +**Compilation**: Worker startup extracts the complete flow shape (`extractFlowShape()`) and PostgreSQL compiles or verifies it via `pgflow.ensure_flow_compiled(flow_slug, shape, worker_protocol)` before any registration. The database answers with its protocol version and the flow's canonical queue name; a mismatched or old-looking answer stops the worker before registration (`QueueProtocolMismatchError`). New workers and the queue-aware database must be upgraded together; rolling old/new workers are unsupported. **Important**: See DSL package files for: - Type inference utilities: `ExtractFlowInput`, `ExtractFlowOutput`, `StepInput`, `StepOutput` @@ -172,7 +172,7 @@ export default CompleteExample; **Critical Cross-Cutting Concepts**: -1. **Two-Phase Polling** - Worker calls `read_with_poll()` then `start_tasks(workerId)` to prevent race conditions +1. **Two-Phase Polling** - Worker calls `read_with_poll()` then `claim_tasks(queue_name, flow_slug, message_ids, worker_id)` to prevent race conditions 2. **Empty Array Cascade** - When `initial_tasks=0`, `cascade_complete_taskless_steps()` completes entire dependent chain in one transaction 3. **Map Step `initial_tasks` Lifecycle**: - Root maps: Set at flow start from input array length @@ -198,13 +198,13 @@ export default CompleteExample; **Worker Lifecycle**: 1. `acknowledgeStart()`: - - Compile or verify the imported flow shape (`ensureFlowCompiled`) + - Compile or verify the imported flow shape (`ensureFlowCompiled`) with the worker protocol handshake - Track the worker function (`track_worker_function`) - - Insert the worker row with `workerId` in the database + - Insert the worker row with `workerId` and the flow's canonical queue in the database 2. Main loop: - `sendHeartbeat()` - Update status, check deprecation - If deprecated → exit gracefully - - Two-phase polling: `readMessages()` then `startTasks(workerId)` + - Queue-aware claiming: `readMessages()` then `claimTasks()` on the flow's canonical queue - Execute handlers (up to `maxConcurrent` parallel) - `complete_task()` or `fail_task()` 3. On shutdown: @@ -227,17 +227,16 @@ const supabase = createClient( ); // Create worker with all configuration options -const worker = createFlowWorker(supabase, MyFlow, { - // Queue configuration - queueName: 'tasks', // Default: 'tasks' - - // Polling configuration - maxPollSeconds: 2, // Default: 2 - pollIntervalMs: 100, // Default: 100 - batchSize: 10, // Default: 10 - visibilityTimeout: 2, // Default: 2 - - // Concurrency configuration +const worker = createFlowWorker( + MyFlow, + { + // Polling configuration + maxPollSeconds: 2, // Default: 2 + pollIntervalMs: 100, // Default: 100 + batchSize: 10, // Default: 10 + visibilityTimeout: 2, // Default: 2 + + // Concurrency configuration maxConcurrent: 10, // Default: 10 maxPgConnections: 4, // Default: 4 @@ -247,7 +246,10 @@ const worker = createFlowWorker(supabase, MyFlow, { delay: 1000, // Base delay in ms maxAttempts: 3, // Max retry attempts }, -}); + }, + createLogger, + platformAdapter +); // Start worker (runs until shutdown signal) await worker.start(); @@ -255,9 +257,9 @@ await worker.start(); // Worker provides context to handlers: // - context.env - Environment variables // - context.shutdownSignal - Graceful shutdown detection -// - context.stepTask - Current task (flow_slug, run_id, step_slug, input, msg_id, task_index) +// - context.stepTask - Current task (flow_slug, run_id, step_slug, input, msg_id as decimal string, task_index) // - context.workerConfig - Configuration (read-only, frozen) -// - context.rawMessage - Full pgmq message (msg_id, read_ct, enqueued_at, vt) +// - context.rawMessage - Full pgmq message (msg_id as decimal string, read_ct, enqueued_at, vt) // - context.sql - PostgreSQL client (Supabase) // - context.supabase - Supabase client (Supabase) ``` @@ -307,17 +309,27 @@ await worker.start(); ## Critical Cross-Package Concepts -### 1. Two-Phase Polling (Worker + SQL Core) +### 1. Queue-Aware Task Claiming (Worker + SQL Core) -**Why**: Prevents race condition where worker processes message before `step_tasks` record exists. +**Why**: Prevents race condition where worker processes message before `step_tasks` record exists, and binds every claim to the flow's canonical queue so message identity is unambiguous. **How**: -- Phase 1: Worker calls `read_with_poll()` - reserves messages, returns `msg_id`s -- Phase 2: Worker calls `start_tasks(flow_slug, msg_ids, workerId)` - creates `step_tasks`, returns details +- Phase 1: Worker reads from the flow's canonical queue with `read_with_poll()` - reserves messages, returns decimal-string `msg_id`s +- Phase 2: Worker calls `pgflow.claim_tasks(queue_name, flow_slug, msg_ids, workerId)` - validates the route, claims matching `step_tasks`, returns details + +Workers and the queue-aware database must be upgraded together; see [the queue identity upgrade](https://pgflow.dev/deploy/update-pgflow/#the-queue-identity-upgrade) for the fenced procedure. + +**Queue identity rules** (#650): +- A task message is identified by `(queue_name, message_id)`, not `message_id` alone. +- Flow and step slugs retain their accepted spelling; physical queue names are lowercase. +- Do not send application messages directly into pgflow-owned queues. +- Clearly foreign untracked messages are archived with body-free warnings. +- Apparently genuine or ambiguous unsupported work stops the worker and pauses HTTP restarts. +- PGMQ message IDs in JavaScript contexts are decimal strings. **See**: -- Worker implementation: `/pkgs/edge-worker/src/worker/FlowWorkerLifecycle.ts` -- SQL implementation: `/pkgs/core/src/migrations/pgflow--*.sql` (functions: `read_with_poll`, `start_tasks`) +- Worker implementation: `/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts` +- SQL implementation: `/pkgs/core/schemas/0120_function_claim_tasks.sql` (functions: `read_with_poll`, `claim_tasks`) ### 2. Empty Array Cascade (DSL + SQL Core) @@ -365,7 +377,7 @@ await worker.start(); ## Non-Negotiable Conventions -- **Slugs**: `[a-zA-Z_][a-zA-Z0-9_]*`, 1-128 chars (cannot be 'run') +- **Slugs**: start with a letter, then letters, digits, or single internal underscores; no leading, trailing, or doubled underscores (cannot be `run`). Step slugs allow 1-128 characters; current flow slugs allow 1-47 because their lowercase value is the generated queue name - **DAG Only**: No cycles or conditional edges - **Topological Order**: Steps added in dependency order (FK enforced) - **JSON Serializable**: All inputs/outputs must be JSON-compatible diff --git a/pkgs/client/__tests__/e2e/full-stack-dsl.test.ts b/pkgs/client/__tests__/e2e/full-stack-dsl.test.ts index 1f3eb6976..e68722d1b 100644 --- a/pkgs/client/__tests__/e2e/full-stack-dsl.test.ts +++ b/pkgs/client/__tests__/e2e/full-stack-dsl.test.ts @@ -40,17 +40,26 @@ describe('Full Stack DSL Integration', () => { await grantMinimalPgflowPermissions(sql); // 2. Compile the flow through startup compilation - // Remove any definition from previous test runs so compilation is deterministic. - await sql`SELECT pgflow.delete_flow_and_data(${SimpleFlow.slug})`; + // Remove any definition from previous test runs so compilation is + // deterministic (delete_flow_and_data raises for absent flows). + await sql` + SELECT pgflow.delete_flow_and_data(flow_slug) + FROM pgflow.flows WHERE flow_slug = ${SimpleFlow.slug} + `; const shape = extractFlowShape(SimpleFlow); - const [{ result }] = await sql<{ result: { status: string } }[]>` + const [{ result }] = await sql< + { result: { status: string; queue_name: string; protocol_version: number } }[] + >` SELECT pgflow.ensure_flow_compiled( ${SimpleFlow.slug}, - ${sql.json(shape as unknown as Json)}::jsonb + ${sql.json(shape as unknown as Json)}::jsonb, + '{"version": 1}'::jsonb ) AS result `; expect(result.status).toBe('compiled'); + expect(result.queue_name).toBe(SimpleFlow.slug.toLowerCase()); + expect(result.protocol_version).toBe(1); // 4. Verify flow was created correctly const flows = diff --git a/pkgs/client/__tests__/helpers/polling.ts b/pkgs/client/__tests__/helpers/polling.ts index f86a95eab..5c46167f4 100644 --- a/pkgs/client/__tests__/helpers/polling.ts +++ b/pkgs/client/__tests__/helpers/polling.ts @@ -42,20 +42,28 @@ export async function readAndStart( workerUuid: string = TEST_WORKER_UUID, functionName = 'test_worker' ) { - // 1. Ensure the worker exists / update its heartbeat - const workerId = await ensureWorker(sql, flowSlug, workerUuid, functionName); + // Canonical physical queue; the concrete flow argument stays exact (#650) + const queueName = flowSlug.toLowerCase(); + + // 1. Ensure the worker exists / update its heartbeat (canonical queue) + const workerId = await ensureWorker(sql, queueName, workerUuid, functionName); // 2. Read messages from the queue - const messages = await sqlClient.readMessages(flowSlug, vt, qty, 1, 50); + const messages = await sqlClient.readMessages(queueName, vt, qty, 1, 50); // 3. If no messages, return empty array if (messages.length === 0) { return []; } - // 4. Start the tasks and return the resulting rows + // 4. Claim the tasks for the retrieved messages; this test helper only + // supports the ok path const msgIds = messages.map(m => m.msg_id); - const tasks = await sqlClient.startTasks(flowSlug, msgIds, workerId); + const result = await sqlClient.startTasks(queueName, flowSlug, msgIds, workerId); + + if (result.status !== 'ok') { + throw new Error(`readAndStart(): unexpected claim status ${result.status}`); + } - return tasks; + return result.tasks; } \ No newline at end of file diff --git a/pkgs/core/README.md b/pkgs/core/README.md index 2b48363d1..6dbf6b99c 100644 --- a/pkgs/core/README.md +++ b/pkgs/core/README.md @@ -93,7 +93,7 @@ The SQL Core handles the workflow lifecycle through these key operations: 1. **Definition**: Workflows are defined using `create_flow` and `add_step` 2. **Instantiation**: Workflow instances are started with `start_flow`, creating a new run -3. **Task Retrieval**: The [Edge Worker](../edge-worker/README.md) uses two-phase polling - first `read_with_poll` to reserve queue messages, then `start_tasks` to convert them to executable tasks +3. **Task Retrieval**: The [Edge Worker](../edge-worker/README.md) uses two-phase polling - first `read_with_poll` to reserve queue messages, then `claim_tasks` to convert them to executable tasks 4. **State Transitions**: When the Edge Worker reports back using `complete_task` or `fail_task`, the SQL Core handles state transitions and schedules dependent steps [Flow lifecycle diagram (click to enlarge)](./assets/flow-lifecycle.svg) @@ -282,21 +282,27 @@ SELECT * FROM pgmq.read_with_poll( ); ``` -**Phase 2 - Start Tasks:** +**Phase 2 - Claim Tasks:** ```sql -SELECT * FROM pgflow.start_tasks( +SELECT pgflow.claim_tasks( + queue_name => 'analyze_website', flow_slug => 'analyze_website', - msg_ids => ARRAY[101, 102, 103], -- message IDs from phase 1 + message_ids => ARRAY[101, 102, 103]::bigint[], -- message IDs from phase 1 worker_id => '550e8400-e29b-41d4-a716-446655440000'::uuid ); ``` +The worker ID must identify a registered worker subscribed to this queue. + **How it works:** 1. **read_with_poll** reserves raw queue messages and hides them from other workers -2. **start_tasks** finds matching step_tasks, increments attempts counter, and builds task inputs -3. Task metadata and input are returned to the worker for execution +2. **claim_tasks** validates the queue subscription and classifies the complete batch before it changes task state +3. An `ok` result returns only tasks claimed by this transaction, plus body-free warnings for archived foreign messages +4. A `fatal` result claims and archives nothing, resets batch visibility, pauses HTTP restarts, and tells the worker to stop + +`start_tasks(flow_slug, msg_ids, worker_id)` remains as a SQL compatibility wrapper for plain workers. New workers call `claim_tasks` directly. This two-phase approach ensures tasks always exist before processing begins, eliminating race conditions that could occur with single-phase polling. diff --git a/pkgs/core/__tests__/types/PgflowSqlClient.test-d.ts b/pkgs/core/__tests__/types/PgflowSqlClient.test-d.ts index 6c8028966..5058c8778 100644 --- a/pkgs/core/__tests__/types/PgflowSqlClient.test-d.ts +++ b/pkgs/core/__tests__/types/PgflowSqlClient.test-d.ts @@ -8,7 +8,13 @@ vi.mock('postgres', () => { }); import { PgflowSqlClient } from '../../src/PgflowSqlClient.js'; -import type { Json, StepTaskKey, StepTaskRecord } from '../../src/types.js'; +import type { + ClaimDiagnostic, + ClaimTasksResult, + Json, + StepTaskKey, + StepTaskRecord, +} from '../../src/types.js'; import postgres from 'postgres'; import { Flow } from '@pgflow/dsl'; @@ -26,10 +32,10 @@ describe('PgflowSqlClient Type Compatibility with Flow', () => { // Check startTasks method types expectTypeOf(client.startTasks).toBeFunction(); expectTypeOf(client.startTasks).parameters.toMatchTypeOf< - [string, number[], string] + [string, string, string[], string] >(); expectTypeOf(client.startTasks).returns.toEqualTypeOf< - Promise[]> + Promise> >(); // Check completeTask method types @@ -60,25 +66,79 @@ describe('PgflowSqlClient Type Compatibility with Flow', () => { client.startFlow(flow, { url: 'string', extraneousKey: 'value' }); }); + it('types the claim contract as a discriminated union with queue-aware records', () => { + const stepFlow = new Flow<{ url: string }>({ slug: 'test_flow' }).step( + { slug: 'run' }, + () => null + ); + type FlowType = typeof stepFlow; + type Result = ClaimTasksResult; + + const ok: Result = { + status: 'ok', + tasks: [ + { + flow_slug: 'test_flow', + run_id: '11111111-1111-1111-1111-111111111111', + step_slug: 'run', + task_index: 0, + queue_name: 'test_flow', + input: { url: 'x' }, + msg_id: '9007199254740993', + flow_input: null, + }, + ], + warnings: [ + { queue_name: 'test_flow', message_id: '9007199254740994', reason: 'foreign_message' }, + ], + }; + expectTypeOf(ok.tasks).toEqualTypeOf[]>(); + expectTypeOf(ok.tasks[0]!.queue_name).toEqualTypeOf(); + expectTypeOf(ok.tasks[0]!.msg_id).toEqualTypeOf(); + + const fatal: Result = { + status: 'fatal', + tasks: [], + errors: [ + { queue_name: 'test_flow', message_id: '9007199254740995', reason: 'unsupported_work' }, + ], + }; + if (fatal.status === 'fatal') { + expectTypeOf(fatal.tasks).toEqualTypeOf<[]>(); + expectTypeOf(fatal.errors[0]!.reason).toEqualTypeOf< + 'foreign_message' | 'unsupported_work' | 'wrong_route' | 'invalid_subscription' + >(); + expectTypeOf(fatal.errors[0]!.message_id).toEqualTypeOf(); + } + + const badReason = { queue_name: 'q', message_id: '1', reason: 'something_else' }; + // @ts-expect-error - diagnostics reject arbitrary reason strings + const rejectedReason: ClaimDiagnostic = badReason; + const badId = { queue_name: 'q', message_id: 1, reason: 'foreign_message' }; + // @ts-expect-error - diagnostic message ids are strings, not numbers + const rejectedId: ClaimDiagnostic = badId; + void [rejectedReason, rejectedId]; + }); + it('should properly type startTasks method parameters', () => { const sql = postgres(); const flow = new Flow<{ url: string }>({ slug: 'test_flow' }); const client = new PgflowSqlClient(sql); - // Valid calls should compile - client.startTasks('flow_slug', [1, 2, 3], 'worker-id'); - client.startTasks('flow_slug', [], 'worker-id'); + // Valid calls should compile: queue, flow, decimal-string ids, worker. + client.startTasks('queue', 'flow_slug', ['1', '2', '3'], 'worker-id'); + client.startTasks('queue', 'flow_slug', [], 'worker-id'); - // @ts-expect-error - flowSlug must be string - client.startTasks(123, [1, 2, 3], 'worker-id'); + // @ts-expect-error - queueName must be string + client.startTasks(123, 'flow_slug', ['1'], 'worker-id'); - // @ts-expect-error - msgIds must be number array - client.startTasks('flow_slug', ['1', '2', '3'], 'worker-id'); + // @ts-expect-error - msgIds must be decimal-string array + client.startTasks('queue', 'flow_slug', [1, 2, 3], 'worker-id'); // @ts-expect-error - msgIds must be array - client.startTasks('flow_slug', 123, 'worker-id'); + client.startTasks('queue', 'flow_slug', '1', 'worker-id'); // @ts-expect-error - workerId must be string - client.startTasks('flow_slug', [1, 2, 3], 123); + client.startTasks('queue', 'flow_slug', ['1'], 123); }); }); diff --git a/pkgs/core/project.json b/pkgs/core/project.json index 32d831196..06c1b9db9 100644 --- a/pkgs/core/project.json +++ b/pkgs/core/project.json @@ -26,6 +26,23 @@ "{projectRoot}/scripts/run-upgrade-fixture", "{projectRoot}/atlas/atlas.hcl", "{projectRoot}/atlas/supabase-baseline-schema.sql" + ], + "queueUpgradeFixture": [ + "{projectRoot}/supabase/upgrade_queue_fixture/**", + "{projectRoot}/scripts/run-queue-upgrade-fixture", + "{projectRoot}/queries/PRE_MIGRATION_CHECK_650.sql", + "{projectRoot}/supabase/migrations/**", + "{projectRoot}/supabase/tests/_shared/prune_data_older_than.sql.raw", + "{workspaceRoot}/pkgs/edge-worker/src/**", + "{workspaceRoot}/pkgs/edge-worker/tests/fakes.ts", + "{workspaceRoot}/pkgs/edge-worker/tests/config.ts", + "{workspaceRoot}/pkgs/edge-worker/tests/sql.ts", + "{workspaceRoot}/pkgs/edge-worker/tests/integration/_helpers.ts", + "{workspaceRoot}/pkgs/edge-worker/tests/integration/upgrade/**", + "{workspaceRoot}/pkgs/edge-worker/deno.test.json", + "{workspaceRoot}/pkgs/edge-worker/deno.lock", + "{workspaceRoot}/pkgs/dsl/src/**", + "{workspaceRoot}/pkgs/core/src/**" ] }, "targets": { @@ -192,7 +209,8 @@ "supabaseSetup", "supabaseRuntime", "pgtapTests", - "upgradeFixture" + "upgradeFixture", + "queueUpgradeFixture" ], "cache": true, "options": { @@ -200,6 +218,16 @@ "command": "../../scripts/with-supabase-lock.sh . bash -ceu '../../scripts/ensure-migrations-body.sh .; scripts/run-test-with-colors; scripts/run-upgrade-fixture'" } }, + "test:upgrade:queue": { + "executor": "nx:run-commands", + "local": true, + "cache": false, + "inputs": ["queueUpgradeFixture"], + "options": { + "cwd": "{projectRoot}", + "command": "../../scripts/with-supabase-lock.sh . scripts/run-queue-upgrade-fixture" + } + }, "test:pgtap:file": { "executor": "@pgflow/nx-executors:focused-file", "local": true, diff --git a/pkgs/core/queries/PRE_MIGRATION_CHECK_650.sql b/pkgs/core/queries/PRE_MIGRATION_CHECK_650.sql new file mode 100644 index 000000000..a9f1ce7e3 --- /dev/null +++ b/pkgs/core/queries/PRE_MIGRATION_CHECK_650.sql @@ -0,0 +1,663 @@ +-- ================================================================================ +-- PRE-MIGRATION CHECK for the queue identity migration (#650) +-- ================================================================================ +-- Purpose: read-only audit of a pgflow 0.16.0 database before the queue +-- identity upgrade. Reports every definition, task pair, queue +-- resource, and installed pruning helper the upgrade must inspect. +-- When to run: BEFORE applying the migration, while writers are paused. +-- Requirements: PostgreSQL with pgflow at 0.16.0 and PGMQ 1.5.1. +-- +-- What to do with output: +-- - severity=info rows only? The locked migration preflight will recheck +-- everything; this stale report never replaces it. +-- - severity=error rows? Resolve the exact names/keys manually first. +-- The migration never renames, merges, deletes, or repairs anything. +-- +-- This script performs no writes: REPEATABLE READ READ ONLY + ROLLBACK. +-- It depends on no new pgflow function or column. Paste into any SQL client. +-- ================================================================================ + +BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY; + +DO $audit$ +declare + v_issue record; + v_queue text; + v_metadata_name text; + v_meta_count int; + v_meta_row pgmq.meta%ROWTYPE; + v_qtable text; + v_atable text; + v_seq text; + v_q_oid oid; + v_a_oid oid; + v_seq_oid oid; + v_ext_oid oid; + v_bad text; + v_count text; + v_samples text; + v_procedure regprocedure; + v_prosrc_md5 text; + -- md5(prosrc) of the stock 0.16.0 helper frozen in the upgrade fixture. + v_stock_prune_md5 constant text := 'd87aba89f910570d5a3c17ba1243eb76'; +begin + -- ========================================================================== + -- 1. Incompatible flow/step definitions under the new slug rules + -- ========================================================================== + for v_issue in + select 'flow'::text as kind, f.flow_slug, null::text as step_slug, + 'leading_underscore'::text as code + from pgflow.flows f where left(f.flow_slug, 1) = '_' + union all + select 'flow', f.flow_slug, null, 'trailing_underscore' + from pgflow.flows f where right(f.flow_slug, 1) = '_' + union all + select 'flow', f.flow_slug, null, 'double_underscore' + from pgflow.flows f where position('__' in f.flow_slug) > 0 + union all + select 'flow', f.flow_slug, null, 'invalid_slug' + from pgflow.flows f + where f.flow_slug !~ '^[a-zA-Z][a-zA-Z0-9_]*$' + or length(f.flow_slug) > 128 + or f.flow_slug = 'run' + union all + select 'step', s.flow_slug, s.step_slug, 'leading_underscore' + from pgflow.steps s where left(s.step_slug, 1) = '_' + union all + select 'step', s.flow_slug, s.step_slug, 'trailing_underscore' + from pgflow.steps s where right(s.step_slug, 1) = '_' + union all + select 'step', s.flow_slug, s.step_slug, 'double_underscore' + from pgflow.steps s where position('__' in s.step_slug) > 0 + union all + select 'step', s.flow_slug, s.step_slug, 'invalid_slug' + from pgflow.steps s + where s.step_slug !~ '^[a-zA-Z][a-zA-Z0-9_]*$' + or length(s.step_slug) > 128 + or s.step_slug = 'run' + loop + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', v_issue.code, + 'flow_slug', v_issue.flow_slug, + 'step_slug', v_issue.step_slug, + 'hint', 'Resolve this exact definition before upgrade; the migration does not rename or delete it' + )::text; + end loop; + + -- ========================================================================== + -- 2. Case-only conflicts (exact spelling of every member is reported) + -- ========================================================================== + for v_issue in + select f.flow_slug, null::text as step_slug, 'case_conflict_flow'::text as code + from pgflow.flows f + where exists ( + select 1 from pgflow.flows other + where other.flow_slug <> f.flow_slug + and lower(other.flow_slug) = lower(f.flow_slug) + ) + union all + select s.flow_slug, s.step_slug, 'case_conflict_step' + from pgflow.steps s + where exists ( + select 1 from pgflow.steps other + where other.flow_slug = s.flow_slug + and other.step_slug <> s.step_slug + and lower(other.step_slug) = lower(s.step_slug) + ) + loop + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', v_issue.code, + 'flow_slug', v_issue.flow_slug, + 'step_slug', v_issue.step_slug, + 'hint', 'A differently spelled flow/step shares this canonical queue; resolve exact spelling manually (no automatic rename)' + )::text; + end loop; + + -- ========================================================================== + -- 3. Denormalized runtime ownership consistency (before trusting backfill) + -- ========================================================================== + select count(*)::text into v_count + from pgflow.step_tasks t + join pgflow.runs r on r.run_id = t.run_id + where r.flow_slug is distinct from t.flow_slug; + if v_count <> '0' then + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'task_run_flow_mismatch', + 'count', v_count, + 'samples', ( + select coalesce(string_agg(t.run_id || ':' || t.step_slug || '#' || t.task_index, ', '), '') + from ( + select t.run_id, t.step_slug, t.task_index + from pgflow.step_tasks t + join pgflow.runs r on r.run_id = t.run_id + where r.flow_slug is distinct from t.flow_slug + limit 20 + ) t + ), + 'hint', 'step_tasks disagree with their run flow_slug; resolve denormalized ownership manually' + )::text; + end if; + + -- ========================================================================== + -- 4. Prospective duplicate (queue, message) pairs — all tasks, not only + -- active runs + -- ========================================================================== + for v_issue in + select lower(t.flow_slug) as queue_name, t.message_id::text as message_id, + count(*)::text as count + from pgflow.step_tasks t + where t.message_id is not null + group by lower(t.flow_slug), t.message_id + having count(*) > 1 + loop + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'duplicate_queue_message_pair', + 'queue_name', v_issue.queue_name, + 'count', v_issue.count, + 'samples', v_issue.message_id, + 'hint', 'Distinct old tasks share a prospective queue/message pair; resolve manually (no automatic merge)' + )::text; + end loop; + + -- ========================================================================== + -- 5. Backfill overview per flow (counts + bounded samples, informational) + -- ========================================================================== + for v_issue in + select f.flow_slug, + lower(f.flow_slug) as queue_name, + (select count(*)::text from pgflow.steps s where s.flow_slug = f.flow_slug) as steps, + (select count(*)::text from pgflow.step_tasks t where t.flow_slug = f.flow_slug) as tasks, + (select count(*)::text from pgflow.step_tasks t + where t.flow_slug = f.flow_slug and t.message_id is null) as null_message_tasks, + (select coalesce(string_agg( + r.run_id || ':' || r.step_slug || '#' || r.task_index || '->' || coalesce(r.message_id::text, 'NULL'), ', '), '') + from ( + select t.run_id, t.step_slug, t.task_index, t.message_id + from pgflow.step_tasks t + where t.flow_slug = f.flow_slug and t.message_id is null + limit 20 + ) r) as null_samples, + (select coalesce(string_agg( + r.run_id || ':' || r.step_slug || '#' || r.task_index || '->' || r.message_id::text, ', '), '') + from ( + select t.run_id, t.step_slug, t.task_index, t.message_id + from pgflow.step_tasks t + where t.flow_slug = f.flow_slug and t.message_id is not null + order by t.run_id, t.step_slug, t.task_index + limit 20 + ) r) as task_samples + from pgflow.flows f + order by f.flow_slug + loop + raise notice '%', jsonb_build_object( + 'severity', 'info', + 'code', 'backfill_overview', + 'flow_slug', v_issue.flow_slug, + 'queue_name', v_issue.queue_name, + 'count', v_issue.tasks, + 'samples', jsonb_build_object( + 'steps', v_issue.steps, + 'tasks', v_issue.tasks, + 'null_message_tasks', v_issue.null_message_tasks, + 'null_task_keys', v_issue.null_samples, + 'task_keys', v_issue.task_samples + ), + 'hint', 'Every step/task backfills queue_name to lower(flow_slug); NULL message IDs stay NULL' + )::text; + end loop; + + -- ========================================================================== + -- 6. Generated queue resources for every persisted flow definition + -- (global pgmq metadata lookup; only validated pgflow candidates are + -- inspected — never unrelated application queues) + -- ========================================================================== + for v_queue in select distinct lower(f.flow_slug) from pgflow.flows f order by 1 + loop + if length(v_queue) > 47 or v_queue !~ '^[a-z][a-z0-9_]*$' then + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'invalid_canonical_queue_name', + 'queue_name', v_queue, + 'hint', 'Generated queue name must be lowercase, at most 47 characters, starting with a letter' + )::text; + continue; + end if; + + select count(*), min(m.queue_name) into v_meta_count, v_metadata_name + from pgmq.meta m + where lower(m.queue_name) = v_queue; + + v_qtable := pgmq.format_table_name(v_queue, 'q'); + v_atable := pgmq.format_table_name(v_queue, 'a'); + v_seq := v_qtable || '_msg_id_seq'; + + if v_meta_count = 0 then + if to_regclass('pgmq.' || v_qtable) is not null + or to_regclass('pgmq.' || v_atable) is not null + or to_regclass('pgmq.' || v_seq) is not null then + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'objects_without_metadata', + 'queue_name', v_queue, + 'hint', 'Physical queue objects exist without pgmq metadata; resolve ownership manually (the migration repairs nothing)' + )::text; + else + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'queue_absent', + 'queue_name', v_queue, + 'hint', 'Flow definition has no generated queue resources; the migration does not reconstruct a lost live queue' + )::text; + end if; + continue; + end if; + + if v_meta_count > 1 then + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'ambiguous_metadata', + 'queue_name', v_queue, + 'count', v_meta_count::text, + 'hint', 'Multiple pgmq metadata spellings resolve to one canonical queue; remove the wrong spelling manually' + )::text; + continue; + end if; + + if to_regclass('pgmq.' || v_qtable) is null + or to_regclass('pgmq.' || v_atable) is null + or to_regclass('pgmq.' || v_seq) is null then + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'incomplete_queue_objects', + 'queue_name', v_queue, + 'samples', format('metadata=%s q=%s a=%s seq=%s', + v_metadata_name, + to_regclass('pgmq.' || v_qtable) is not null, + to_regclass('pgmq.' || v_atable) is not null, + to_regclass('pgmq.' || v_seq) is not null), + 'hint', 'Queue metadata exists but q/a/sequence objects are incomplete; resolve manually (no repair, no drop)' + )::text; + continue; + end if; + + -- Physical shape, index, sequence-dependency, and extension-membership + -- audit: the same read-only catalog contract the migration preflight + -- enforces, mirroring the complete per-column contract of + -- _inspect_generated_queue (0070_functions_generated_queues.sql) + -- including explicit missing-column rejection. Problems are reported + -- (never repaired) with bounded samples. + select * into v_meta_row from pgmq.meta m where lower(m.queue_name) = v_queue; + v_q_oid := to_regclass(format('pgmq.%I', v_qtable)); + v_a_oid := to_regclass(format('pgmq.%I', v_atable)); + v_seq_oid := to_regclass(format('pgmq.%I', v_seq)); + select e.oid into v_ext_oid from pg_extension e where e.extname = 'pgmq'; + + select string_agg(problem, '; ') into v_bad + from ( + select 'queue table is not an ordinary permanent table' as problem + from pg_class c + where c.oid = v_q_oid and (c.relkind <> 'r' or c.relpersistence <> 'p') + union all + select 'archive table is not an ordinary permanent table' + from pg_class c + where c.oid = v_a_oid and (c.relkind <> 'r' or c.relpersistence <> 'p') + union all + select 'queue msg_id must be a non-null bigint generated-always identity' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'msg_id' + and (a.atttypid <> 'int8'::regtype or not a.attnotnull or a.attidentity <> 'a') + union all + select 'queue table is missing its msg_id bigint identity column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'msg_id' and a.attnum > 0) + union all + select 'queue msg_id has no single-column primary key' + where not exists ( + select 1 from pg_index i + where i.indrelid = v_q_oid and i.indisprimary and i.indisvalid + and i.indnkeyatts = 1 + and i.indkey[0] = (select a.attnum from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'msg_id') + ) + union all + select 'queue table has no valid usable single-column index on vt' + where not exists ( + select 1 + from pg_index i + join pg_attribute a on a.attrelid = i.indrelid and a.attnum = i.indkey[0] + where i.indrelid = v_q_oid and i.indisvalid and i.indisready + and i.indpred is null and i.indexprs is null and i.indnkeyatts = 1 + and a.attname = 'vt' + ) + union all + select 'queue read_ct must be a non-null integer' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'read_ct' + and (a.atttypid <> 'int4'::regtype or not a.attnotnull) + union all + select 'queue table is missing its read_ct column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'read_ct' and a.attnum > 0) + union all + select 'queue enqueued_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'enqueued_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'queue table is missing its enqueued_at column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'enqueued_at' and a.attnum > 0) + union all + select 'queue vt must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'vt' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'queue table is missing its vt column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'vt' and a.attnum > 0) + union all + select 'queue message must be jsonb' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'message' + and a.atttypid <> 'jsonb'::regtype + union all + select 'queue table is missing its message column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'message' and a.attnum > 0) + union all + select 'queue headers must be jsonb' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'headers' + and a.atttypid <> 'jsonb'::regtype + union all + select 'queue table is missing its headers column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'headers' and a.attnum > 0) + union all + select 'archive msg_id must be a non-null bigint primary key without identity generator' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'msg_id' + and (a.atttypid <> 'int8'::regtype or not a.attnotnull or a.attidentity <> '') + union all + select 'archive table is missing its msg_id bigint column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'msg_id' and a.attnum > 0) + union all + select 'archive msg_id has no single-column primary key' + where not exists ( + select 1 from pg_index i + where i.indrelid = v_a_oid and i.indisprimary and i.indisvalid + and i.indnkeyatts = 1 + and i.indkey[0] = (select a.attnum from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'msg_id') + ) + union all + select 'archive read_ct must be a non-null integer' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'read_ct' + and (a.atttypid <> 'int4'::regtype or not a.attnotnull) + union all + select 'archive table is missing its read_ct column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'read_ct' and a.attnum > 0) + union all + select 'archive enqueued_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'enqueued_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table is missing its enqueued_at column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'enqueued_at' and a.attnum > 0) + union all + select 'archive archived_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'archived_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table is missing its archived_at column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'archived_at' and a.attnum > 0) + union all + select 'archive vt must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'vt' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table is missing its vt column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'vt' and a.attnum > 0) + union all + select 'archive message must be jsonb' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'message' + and a.atttypid <> 'jsonb'::regtype + union all + select 'archive table is missing its message column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'message' and a.attnum > 0) + union all + select 'archive headers must be jsonb' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'headers' + and a.atttypid <> 'jsonb'::regtype + union all + select 'archive table is missing its headers column' + where not exists (select 1 from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'headers' and a.attnum > 0) + union all + select 'archive table has no valid usable single-column index on archived_at' + where not exists ( + select 1 + from pg_index i + join pg_attribute a on a.attrelid = i.indrelid and a.attnum = i.indkey[0] + where i.indrelid = v_a_oid and i.indisvalid and i.indisready + and i.indpred is null and i.indexprs is null and i.indnkeyatts = 1 + and a.attname = 'archived_at' + ) + union all + select 'sequence must be a bigint sequence' + from pg_sequence s + where s.seqrelid = v_seq_oid + and s.seqtypid <> 'int8'::regtype + union all + select 'sequence is missing (the named relation is not a sequence or does not exist)' + where not exists (select 1 from pg_sequence s where s.seqrelid = v_seq_oid) + union all + select 'sequence is not associated with queue msg_id' + where not exists ( + select 1 + from pg_depend d + join pg_attribute a + on a.attrelid = d.refobjid and a.attnum = d.refobjsubid + where d.objid = v_seq_oid + and d.refobjid = v_q_oid + and a.attname = 'msg_id' + and d.deptype in ('i', 'a') + ) + union all + select 'metadata flags disagree with physical shape (partitioned/unlogged)' + where v_meta_row.is_partitioned or v_meta_row.is_unlogged + union all + select 'q/a tables or sequence are not members of the installed pgmq extension' + where v_ext_oid is not null and ( + not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_q_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) or not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_a_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) or not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_seq_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) + ) + ) problems; + + if v_bad is not null then + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'malformed_queue_objects', + 'queue_name', v_queue, + 'samples', v_bad, + 'hint', 'Queue objects fail the physical/dependency/extension contract; resolve manually (the migration repairs nothing)' + )::text; + continue; + end if; + + -- Active queue rows without a matching exact task identity, including + -- non-visible messages (vt > now() does not exempt them). A malformed + -- relation shape reports instead of failing the audit. + begin + execute format( + 'select count(*)::text from pgmq.%I q + where not exists ( + select 1 from pgflow.step_tasks t + where lower(t.flow_slug) = $1 and t.message_id = q.msg_id + )', v_qtable) + into v_count using v_queue; + exception when others then + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'queue_inspect_error', + 'queue_name', v_queue, + 'samples', sqlerrm || ' (table: ' || v_qtable || ')', + 'hint', 'The queue relation has an unexpected shape; inspect it manually before upgrade' + )::text; + continue; + end; + + if v_count <> '0' then + begin + execute format( + 'select coalesce(string_agg(x.k, '', ''), '''') + from (select q.msg_id::text as k from pgmq.%I q + where not exists ( + select 1 from pgflow.step_tasks t + where lower(t.flow_slug) = $1 and t.message_id = q.msg_id + ) + order by q.msg_id limit 20) x', v_qtable) + into v_samples using v_queue; + exception when others then + v_samples := null; + end; + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'unmatched_active_message', + 'queue_name', v_queue, + 'count', v_count, + 'samples', v_samples, + 'hint', 'Active queue messages without a matching exact task identity; resolve orphan messages manually before upgrade' + )::text; + end if; + + -- Matched messages whose envelopes identify different work than their + -- durable task identity: a valid contradicting address is corruption the + -- migration must reject, so the audit reports it before upgrade. + begin + execute format( + 'select count(*)::text from pgmq.%I q + join pgflow.step_tasks t + on lower(t.flow_slug) = $1 and t.message_id = q.msg_id + where (q.message ->> ''flow_slug'') is not null + and (q.message ->> ''flow_slug'') is distinct from t.flow_slug + or ((q.message ->> ''run_id'') ~* ''^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'' + and (q.message ->> ''run_id'')::uuid is distinct from t.run_id) + or (q.message ->> ''step_slug'') is not null + and (q.message ->> ''step_slug'') is distinct from t.step_slug + or ((q.message ->> ''task_index'') ~ ''^[0-9]{1,9}$'' + and (q.message ->> ''task_index'')::int is distinct from t.task_index)', + v_qtable) + into v_count using v_queue; + exception when others then + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'queue_inspect_error', + 'queue_name', v_queue, + 'samples', sqlerrm || ' (table: ' || v_qtable || ')', + 'hint', 'The queue relation has an unexpected shape; inspect it manually before upgrade' + )::text; + continue; + end; + + if v_count <> '0' then + begin + execute format( + 'select coalesce(string_agg(x.k, '', ''), '''') + from (select q.msg_id::text as k from pgmq.%I q + join pgflow.step_tasks t + on lower(t.flow_slug) = $1 and t.message_id = q.msg_id + where (q.message ->> ''flow_slug'') is not null + and (q.message ->> ''flow_slug'') is distinct from t.flow_slug + or ((q.message ->> ''run_id'') ~* ''^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'' + and (q.message ->> ''run_id'')::uuid is distinct from t.run_id) + or (q.message ->> ''step_slug'') is not null + and (q.message ->> ''step_slug'') is distinct from t.step_slug + or ((q.message ->> ''task_index'') ~ ''^[0-9]{1,9}$'' + and (q.message ->> ''task_index'')::int is distinct from t.task_index) + order by q.msg_id limit 20) x', v_qtable) + into v_samples using v_queue; + exception when others then + v_samples := null; + end; + raise notice '%', jsonb_build_object( + 'severity', 'error', + 'code', 'envelope_contradiction', + 'queue_name', v_queue, + 'count', v_count, + 'samples', v_samples, + 'hint', 'Matched active messages carry envelopes that identify different work than their task rows; resolve manually before upgrade (no bodies are shown)' + )::text; + end if; + end loop; + + -- ========================================================================== + -- 7. Installed pruning helper (upgrade action, never auto-replaced) + -- ========================================================================== + v_procedure := to_regprocedure('pgflow.prune_data_older_than(interval)'); + if v_procedure is null then + raise notice '%', jsonb_build_object( + 'severity', 'info', + 'code', 'pruning_helper_absent', + 'hint', 'No installed pgflow.prune_data_older_than(interval); nothing to replace or adapt' + )::text; + else + select md5(p.prosrc) into v_prosrc_md5 + from pg_proc p + join pg_namespace n on n.oid = p.pronamespace + where n.nspname = 'pgflow' + and p.proname = 'prune_data_older_than'; + raise notice '%', jsonb_build_object( + 'severity', 'warning', + 'code', case when v_prosrc_md5 = v_stock_prune_md5 then 'pruning_helper_stock' + else 'pruning_helper_customized' end, + 'samples', jsonb_build_object( + 'md5_prosrc', v_prosrc_md5, + 'stock_0_16_0_md5', v_stock_prune_md5 + ), + 'hint', case when v_prosrc_md5 = v_stock_prune_md5 + then 'Stock 0.16.0 helper detected: replace it explicitly after the migration; the migration never overwrites it' + else 'Customized helper detected: adapt it manually to queue snapshots after the migration; the migration never overwrites it' end + )::text; + end if; + + raise notice '%', jsonb_build_object( + 'severity', 'info', + 'code', 'audit_complete', + 'hint', 'This report is a point-in-time snapshot taken while writers are paused; the locked migration preflight rechecks everything and can still fail on new activity' + )::text; +end +$audit$; + +ROLLBACK; diff --git a/pkgs/core/schemas/0030_utilities.sql b/pkgs/core/schemas/0030_utilities.sql index 2a72c5dab..497485fe5 100644 --- a/pkgs/core/schemas/0030_utilities.sql +++ b/pkgs/core/schemas/0030_utilities.sql @@ -31,10 +31,32 @@ begin and slug <> '' and length(slug) <= 128 and slug ~ '^[a-zA-Z_][a-zA-Z0-9_]*$' + and left(slug, 1) <> '_' + and right(slug, 1) <> '_' + and position('__' in slug) = 0 and slug NOT IN ('run'); -- reserved words end; $$; +-- Canonical physical queue names stored by pgflow: lowercase, nonempty, at +-- most 47 characters (PGMQ compatibility limit), starting with a letter. +-- Distinct from the slug rule: generated names may contain double underscores. +create or replace function pgflow._is_valid_queue_name( + queue_name text +) +returns boolean +language sql +immutable +parallel safe +set search_path = '' +as $$ + select + queue_name is not null + and queue_name <> '' + and length(queue_name) <= 47 + and queue_name ~ '^[a-z][a-z0-9_]*$' +$$; + create or replace function pgflow.calculate_retry_delay( base_delay numeric, attempts_count int diff --git a/pkgs/core/schemas/0050_tables_definitions.sql b/pkgs/core/schemas/0050_tables_definitions.sql index 452540c99..95a26d22f 100644 --- a/pkgs/core/schemas/0050_tables_definitions.sql +++ b/pkgs/core/schemas/0050_tables_definitions.sql @@ -20,6 +20,7 @@ create table pgflow.steps ( step_type text not null default 'single', step_index int not null default 0, deps_count int not null default 0 check (deps_count >= 0), + queue_name text not null, opt_max_attempts int, opt_base_delay int, opt_timeout int, @@ -44,7 +45,8 @@ create table pgflow.steps ( constraint opt_timeout_is_positive check (opt_timeout is null or opt_timeout > 0), constraint opt_start_delay_is_nonnegative check (opt_start_delay is null or opt_start_delay >= 0), constraint when_unmet_is_valid check (when_unmet in ('fail', 'skip', 'skip-cascade')), - constraint when_exhausted_is_valid check (when_exhausted in ('fail', 'skip', 'skip-cascade')) + constraint when_exhausted_is_valid check (when_exhausted in ('fail', 'skip', 'skip-cascade')), + constraint queue_name_is_valid check (pgflow._is_valid_queue_name(queue_name)) ); -- Dependencies table - stores relationships between steps @@ -63,3 +65,9 @@ create table pgflow.deps ( create index if not exists idx_deps_by_flow_step on pgflow.deps (flow_slug, step_slug); create index if not exists idx_deps_by_flow_dep on pgflow.deps (flow_slug, dep_slug); + +-- Case-insensitive namespace uniqueness: concrete spelling is preserved, but +-- case-only aliases would collide on generated queue names, so they are +-- rejected atomically by declarative indexes (#650). +create unique index if not exists idx_flows_slug_lower on pgflow.flows (lower(flow_slug)); +create unique index if not exists idx_steps_slug_lower on pgflow.steps (flow_slug, lower(step_slug)); diff --git a/pkgs/core/schemas/0060_tables_runtime.sql b/pkgs/core/schemas/0060_tables_runtime.sql index c487d687f..9fc205bce 100644 --- a/pkgs/core/schemas/0060_tables_runtime.sql +++ b/pkgs/core/schemas/0060_tables_runtime.sql @@ -88,6 +88,7 @@ create table pgflow.step_tasks ( message_id bigint, task_index int not null default 0, status text not null default 'queued', + queue_name text not null, attempts_count int not null default 0, error_message text, output jsonb, @@ -117,10 +118,40 @@ create table pgflow.step_tasks ( constraint completed_at_is_after_started_at check ( completed_at is null or started_at is null or completed_at >= started_at ), - constraint failed_at_is_after_started_at check (failed_at is null or started_at is null or failed_at >= started_at) + constraint failed_at_is_after_started_at check (failed_at is null or started_at is null or failed_at >= started_at), + constraint queue_name_is_valid check (pgflow._is_valid_queue_name(queue_name)) ); -create index if not exists idx_step_tasks_message_id on pgflow.step_tasks (message_id); +-- Queue/message pair identity for queue-scoped PGMQ message IDs. Replaces the +-- former message-only lookups; NULL message IDs stay legal (cleanup paths). +create unique index if not exists idx_step_tasks_queue_message +on pgflow.step_tasks (queue_name, message_id) +where message_id is not null; + +-- Task queue snapshots are immutable after insertion. Statement transition +-- tables also reject a task-address move that a key-based comparison would miss. +create or replace function pgflow._keep_task_queue_name() +returns trigger +language plpgsql +set search_path = '' +as $$ +begin + if exists ( + select run_id, step_slug, task_index, queue_name from old_tasks + except + select run_id, step_slug, task_index, queue_name from new_tasks + ) then + raise exception 'step_tasks.queue_name is immutable'; + end if; + return null; +end; +$$; + +create trigger keep_task_queue_name +after update on pgflow.step_tasks +referencing old table as old_tasks new table as new_tasks +for each statement execute function pgflow._keep_task_queue_name(); + create index if not exists idx_step_tasks_queued on pgflow.step_tasks (run_id, step_slug) where status = 'queued'; create index if not exists idx_step_tasks_completed on pgflow.step_tasks (run_id, step_slug) where status = 'completed'; create index if not exists idx_step_tasks_failed on pgflow.step_tasks (run_id, step_slug) where status = 'failed'; @@ -128,5 +159,4 @@ create index if not exists idx_step_tasks_flow_run_step on pgflow.step_tasks (fl -- New indexes for refactored polling behavior create index if not exists idx_step_tasks_started on pgflow.step_tasks (started_at) where status = 'started'; -create index if not exists idx_step_tasks_queued_msg on pgflow.step_tasks (message_id) where status = 'queued'; create index if not exists idx_step_tasks_last_worker on pgflow.step_tasks (last_worker_id) where status = 'started'; diff --git a/pkgs/core/schemas/0062_function_requeue_stalled_tasks.sql b/pkgs/core/schemas/0062_function_requeue_stalled_tasks.sql index 76b2a1b23..e2f84c548 100644 --- a/pkgs/core/schemas/0062_function_requeue_stalled_tasks.sql +++ b/pkgs/core/schemas/0062_function_requeue_stalled_tasks.sql @@ -1,6 +1,6 @@ -- Requeue stalled tasks that have been in 'started' status longer than their effective -- timeout (step override with flow fallback) + 30s buffer. This matches the effective --- timeout used by start_tasks() for PGMQ visibility, without its +2s visibility margin. +-- timeout used by claim_tasks() for PGMQ visibility, without its +2s visibility margin. -- This handles tasks that got stuck when workers crashed without completing them create or replace function pgflow.requeue_stalled_tasks() returns int @@ -17,26 +17,77 @@ begin -- but status left as 'started' for easy identification via requeued_count column -- Eligibility requires the parent run AND parent step to still be 'started': -- stale rows on failed runs or terminal steps must not be revived (#645). - with stalled_tasks as ( + -- + -- Lock order (#650): eligible parent runs are locked first (ordered by + -- run_id), then eligible step states (ordered by (run_id, step_slug)), then + -- task rows (ordered by (run_id, step_slug, task_index)) - as three + -- sequential lock sets, not one joined FOR UPDATE, so parent rows are + -- always locked before their children. SKIP LOCKED is preserved at every + -- level: a blocked parent/run/state/task is skipped, not waited on, and no + -- later-order lock is held while waiting. Status and timeout predicates + -- are restated in each phase so EvalPlanQual rechecks them under the locks. + with locked_runs as ( + select r.run_id + from pgflow.runs r + where r.status = 'started' + and exists ( + select 1 + from pgflow.step_tasks st + join pgflow.step_states ss on ss.run_id = st.run_id and ss.step_slug = st.step_slug + join pgflow.flows f on f.flow_slug = r.flow_slug + join pgflow.steps s on s.flow_slug = r.flow_slug and s.step_slug = st.step_slug + where st.run_id = r.run_id + and st.status = 'started' + and ss.status = 'started' + and st.permanently_stalled_at is null + and st.started_at < now() + - (coalesce(s.opt_timeout, f.opt_timeout) * interval '1 second') + - interval '30 seconds' + ) + order by r.run_id + for update skip locked + ), + locked_states as ( + select ss.run_id, ss.step_slug + from pgflow.step_states ss + join locked_runs lr on lr.run_id = ss.run_id + where ss.status = 'started' + and exists ( + select 1 + from pgflow.step_tasks st + join pgflow.flows f on f.flow_slug = ss.flow_slug + join pgflow.steps s on s.flow_slug = ss.flow_slug and s.step_slug = st.step_slug + where st.run_id = ss.run_id + and st.step_slug = ss.step_slug + and st.status = 'started' + and st.permanently_stalled_at is null + and st.started_at < now() + - (coalesce(s.opt_timeout, f.opt_timeout) * interval '1 second') + - interval '30 seconds' + ) + order by ss.run_id, ss.step_slug + for update of ss skip locked + ), + stalled_tasks as ( select st.run_id, st.step_slug, st.task_index, st.message_id, - r.flow_slug, + st.queue_name, st.requeued_count from pgflow.step_tasks st + join locked_states ls on ls.run_id = st.run_id and ls.step_slug = st.step_slug join pgflow.runs r on r.run_id = st.run_id - join pgflow.step_states ss on ss.run_id = st.run_id and ss.step_slug = st.step_slug join pgflow.flows f on f.flow_slug = r.flow_slug join pgflow.steps s on s.flow_slug = r.flow_slug and s.step_slug = st.step_slug where st.status = 'started' and r.status = 'started' - and ss.status = 'started' and st.permanently_stalled_at is null and st.started_at < now() - (coalesce(s.opt_timeout, f.opt_timeout) * interval '1 second') - interval '30 seconds' + order by st.run_id, st.step_slug, st.task_index for update of st skip locked ), -- Separate tasks that can be requeued from those that exceeded max requeues @@ -46,7 +97,7 @@ begin to_archive as ( select * from stalled_tasks where requeued_count >= max_requeues ), - -- Update tasks that will be requeued + -- Update tasks that will be requeued; the queue comes from the task snapshot requeued as ( update pgflow.step_tasks st set @@ -59,14 +110,14 @@ begin where st.run_id = tr.run_id and st.step_slug = tr.step_slug and st.task_index = tr.task_index - returning tr.flow_slug as queue_name, tr.message_id + returning tr.queue_name as queue_name, tr.message_id ), - -- Make requeued messages visible immediately (batched per queue) + -- Make requeued messages visible immediately (batched per queue snapshot) visibility_reset as ( select pgflow.set_vt_batch( r.queue_name, - array_agg(r.message_id), - array_agg(0) -- all offsets are 0 (immediate visibility) + array_agg(r.message_id order by r.message_id), + array_agg(0 order by r.message_id) -- all offsets are 0 (immediate visibility) ) from requeued r where r.message_id is not null @@ -82,21 +133,26 @@ begin and st.task_index = ta.task_index returning st.run_id ), - -- Archive messages for tasks that exceeded max requeues (batched per queue) + -- Archive messages for tasks that exceeded max requeues (batched per queue + -- snapshot; never grouped across queues) archived as ( - select pgmq.archive(ta.flow_slug, array_agg(ta.message_id)) + select pgmq.archive(ta.queue_name, array_agg(ta.message_id)) from to_archive ta where ta.message_id is not null - group by ta.flow_slug - ), - -- Force execution of visibility_reset CTE - _vr as (select count(*) from visibility_reset), - -- Force execution of mark_permanently_stalled CTE - _mps as (select count(*) from mark_permanently_stalled), - -- Force execution of archived CTE - _ar as (select count(*) from archived) - select count(*) into result_count - from requeued, _vr, _mps, _ar; + group by ta.queue_name + ) + -- Force execution of every side-effecting CTE regardless of join order: + -- a cross join with an empty relation could skip scanning the forcing + -- wrappers, so they are evaluated as scalar subqueries that always run. + select + (select count(*) from requeued) + + 0 * coalesce( + (select count(*) from visibility_reset) + + (select count(*) from mark_permanently_stalled) + + (select count(*) from archived), + 0 + ) + into result_count; return result_count; end; diff --git a/pkgs/core/schemas/0070_functions_generated_queues.sql b/pkgs/core/schemas/0070_functions_generated_queues.sql new file mode 100644 index 000000000..b7ab6cf3e --- /dev/null +++ b/pkgs/core/schemas/0070_functions_generated_queues.sql @@ -0,0 +1,465 @@ +-- Internal generated-queue ownership helpers (#650). +-- +-- These helpers inspect and provision the deterministic private queues derived +-- from persisted flow definitions. They are internal ownership machinery, not +-- routing APIs. Prohibited usage: hot-path polling/send/archive (they never +-- call these), and runtime task operations (they use task queue snapshots). + +-- Inspect the physical PGMQ objects and pgflow ownership evidence for a +-- generated queue. Performs no writes; takes the pgmq.meta topology fence so +-- inspection cannot interleave with a concurrent external PGMQ create. +-- +-- Returns: +-- {"state":"absent"} when no metadata and no physical objects exist +-- {"state":"present","metadata_name":} when the complete +-- valid object set exists under this flow's ownership +-- +-- Raises on every uncertain state: ambiguous metadata, partial objects, +-- objects without metadata, metadata without objects, malformed shape, or a +-- resource referenced/derived by another flow. p_require_existing=true +-- additionally rejects the absent state and a missing concrete definition. +create or replace function pgflow._inspect_generated_queue( + p_flow_slug text, + p_queue_name text, + p_require_existing boolean +) +returns jsonb +language plpgsql +volatile +set search_path = '' +as $$ +declare + v_canonical_queue text := lower(p_flow_slug); + v_qtable text; + v_atable text; + v_sequence text; + v_meta_count int; + v_metadata_name text; + v_meta_row pgmq.meta%ROWTYPE; + v_flow_exists boolean; + v_other_flow text; + v_routed_step text; + v_other_route text; + v_q_oid oid; + v_a_oid oid; + v_seq_oid oid; + v_ext_oid oid; + v_bad text; +begin + if not pgflow._is_valid_queue_name(p_queue_name) then + raise exception 'Flow %: "%" is not a valid generated queue name (lowercase, at most 47 characters, starting with a letter)', + p_flow_slug, p_queue_name; + end if; + + if p_queue_name <> v_canonical_queue then + raise exception 'Flow %: queue "%" is not its canonical generated route "%" (custom routes do not exist in #650)', + p_flow_slug, p_queue_name, v_canonical_queue; + end if; + + select exists(select 1 from pgflow.flows f where f.flow_slug = p_flow_slug) + into v_flow_exists; + + -- Topology fence: serialize against external PGMQ create/drop on this + -- namespace (PGMQ 1.5.1 inserts metadata only after creating objects). All + -- ownership checks and physical inspection happen after the fence so a + -- wait cannot invalidate them. + lock table pgmq.meta in share row exclusive mode; + + -- Ownership evidence: no other flow may derive, persist, or reference this route + select f.flow_slug into v_other_flow + from pgflow.flows f + where lower(f.flow_slug) = p_queue_name + and f.flow_slug <> p_flow_slug + limit 1; + if v_other_flow is not null then + raise exception 'Generated queue "%" for flow % collides with the derived route of flow %', + p_queue_name, p_flow_slug, v_other_flow; + end if; + + select s.flow_slug into v_other_flow + from pgflow.steps s + where s.queue_name = p_queue_name + and s.flow_slug <> p_flow_slug + limit 1; + if v_other_flow is not null then + raise exception 'Generated queue "%" for flow % is persisted as a route of flow %', + p_queue_name, p_flow_slug, v_other_flow; + end if; + + select t.flow_slug into v_other_flow + from pgflow.step_tasks t + where t.queue_name = p_queue_name + and t.flow_slug <> p_flow_slug + limit 1; + if v_other_flow is not null then + raise exception 'Generated queue "%" for flow % is referenced by tasks of flow %', + p_queue_name, p_flow_slug, v_other_flow; + end if; + + -- The current flow's persisted route must actually be this queue: a step + -- persisting a different route is an invalid definition, and verifying or + -- deleting this queue while such a step exists would bypass it. + select s.step_slug, s.queue_name into v_routed_step, v_other_route + from pgflow.steps s + where s.flow_slug = p_flow_slug + and s.queue_name is distinct from p_queue_name + limit 1; + if v_routed_step is not null then + raise exception 'Flow %: step "%" persists route "%" instead of the generated queue "%"; the definition is invalid and no queue operation may proceed', + p_flow_slug, v_routed_step, v_other_route, p_queue_name; + end if; + + select count(*), min(m.queue_name) into v_meta_count, v_metadata_name + from pgmq.meta as m + where lower(m.queue_name) = p_queue_name; + + if v_meta_count > 1 then + raise exception 'Generated queue "%" for flow % has ambiguous PGMQ metadata (% rows share the name case-insensitively)', + p_queue_name, p_flow_slug, v_meta_count; + end if; + + v_qtable := pgmq.format_table_name(p_queue_name, 'q'); + v_atable := pgmq.format_table_name(p_queue_name, 'a'); + v_sequence := v_qtable || '_msg_id_seq'; + + v_q_oid := to_regclass(format('pgmq.%I', v_qtable)); + v_a_oid := to_regclass(format('pgmq.%I', v_atable)); + v_seq_oid := to_regclass(format('pgmq.%I', v_sequence)); + + if v_meta_count = 0 and v_q_oid is null and v_a_oid is null and v_seq_oid is null then + -- Only genuinely absent when zero metadata AND zero objects + if p_require_existing then + raise exception 'Flow %: generated queue "%" is missing (no PGMQ metadata, no objects)', + p_flow_slug, p_queue_name; + end if; + return jsonb_build_object('state', 'absent'); + end if; + + -- Some metadata or physical evidence exists: the concrete definition must + -- own it. A missing definition plus any resource is a collision. + if not v_flow_exists then + raise exception 'Generated queue "%" exists (metadata: %, queue table: %, archive table: %, sequence: %) but flow % has no definition; refusing to adopt external resources', + p_queue_name, v_meta_count, v_qtable, v_atable, v_sequence, p_flow_slug; + end if; + + if v_meta_count = 0 then + raise exception 'Flow %: generated queue "%" has physical objects without PGMQ metadata (queue table: %, archive table: %, sequence: %)', + p_flow_slug, p_queue_name, v_qtable, v_atable, v_sequence; + end if; + + if v_q_oid is null or v_a_oid is null or v_seq_oid is null then + raise exception 'Flow %: generated queue "%" has incomplete PGMQ objects (queue table: %, archive table: %, sequence: %)', + p_flow_slug, p_queue_name, v_qtable, v_atable, v_sequence; + end if; + + select * into v_meta_row from pgmq.meta m where lower(m.queue_name) = p_queue_name; + + -- ========================================== + -- QUEUE TABLE CONTRACT + -- ========================================== + select reason into v_bad from ( + select 'queue table %s is not an ordinary permanent table' as reason + from pg_class c + where c.oid = v_q_oid + and (c.relkind <> 'r' or c.relpersistence <> 'p') + union all + select 'queue table %s: msg_id must be a non-null bigint generated-always identity primary key' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'msg_id' + and (a.atttypid <> 'int8'::regtype or not a.attnotnull or a.attidentity <> 'a') + union all + select 'queue table %s: missing msg_id bigint identity column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'msg_id' and a.attnum > 0) + union all + select 'queue table %s: msg_id has no single-column primary key' + where not exists ( + select 1 from pg_index i + where i.indrelid = v_q_oid and i.indisprimary and i.indisvalid + and i.indnkeyatts = 1 + and i.indkey[0] = (select a.attnum from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'msg_id') + ) + union all + select 'queue table %s: read_ct must be a non-null integer' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'read_ct' + and (a.atttypid <> 'int4'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing read_ct column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'read_ct' and a.attnum > 0) + union all + select 'queue table %s: enqueued_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'enqueued_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing enqueued_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'enqueued_at' and a.attnum > 0) + union all + select 'queue table %s: vt must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'vt' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing vt column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'vt' and a.attnum > 0) + union all + select 'queue table %s: message must be jsonb' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'message' + and a.atttypid <> 'jsonb'::regtype + union all + select 'queue table %s: missing message column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'message' and a.attnum > 0) + union all + select 'queue table %s: headers must be jsonb' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'headers' + and a.atttypid <> 'jsonb'::regtype + union all + select 'queue table %s: missing headers column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'headers' and a.attnum > 0) + union all + select 'queue table %s has no valid usable single-column index on vt' + where not exists ( + select 1 + from pg_index i + join pg_attribute a on a.attrelid = i.indrelid and a.attnum = i.indkey[0] + where i.indrelid = v_q_oid + and i.indisvalid + and i.indisready + and i.indpred is null + and i.indexprs is null + and i.indnkeyatts = 1 + and a.attname = 'vt' + ) + union all + select 'queue table %s metadata flags disagree with physical shape (partitioned/unlogged)' + where v_meta_row.is_partitioned or v_meta_row.is_unlogged + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Flow %: generated queue "%" failed physical inspection: %', + p_flow_slug, p_queue_name, format(v_bad, v_qtable); + end if; + + -- ========================================== + -- ARCHIVE TABLE CONTRACT + -- ========================================== + select reason into v_bad from ( + select 'archive table %s is not an ordinary permanent table' as reason + from pg_class c + where c.oid = v_a_oid + and (c.relkind <> 'r' or c.relpersistence <> 'p') + union all + select 'archive table %s: msg_id must be a non-null bigint primary key without identity generator' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'msg_id' + and (a.atttypid <> 'int8'::regtype or not a.attnotnull or a.attidentity <> '') + union all + select 'archive table %s: missing msg_id bigint column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'msg_id' and a.attnum > 0) + union all + select 'archive table %s: msg_id has no single-column primary key' + where not exists ( + select 1 from pg_index i + where i.indrelid = v_a_oid and i.indisprimary and i.indisvalid + and i.indnkeyatts = 1 + and i.indkey[0] = (select a.attnum from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'msg_id') + ) + union all + select 'archive table %s: read_ct must be a non-null integer' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'read_ct' + and (a.atttypid <> 'int4'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing read_ct column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'read_ct' and a.attnum > 0) + union all + select 'archive table %s: enqueued_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'enqueued_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing enqueued_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'enqueued_at' and a.attnum > 0) + union all + select 'archive table %s: archived_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'archived_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing archived_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'archived_at' and a.attnum > 0) + union all + select 'archive table %s: vt must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'vt' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing vt column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'vt' and a.attnum > 0) + union all + select 'archive table %s: message must be jsonb' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'message' + and a.atttypid <> 'jsonb'::regtype + union all + select 'archive table %s: missing message column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'message' and a.attnum > 0) + union all + select 'archive table %s: headers must be jsonb' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'headers' + and a.atttypid <> 'jsonb'::regtype + union all + select 'archive table %s: missing headers column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'headers' and a.attnum > 0) + union all + select 'archive table %s has no valid usable single-column index on archived_at' + where not exists ( + select 1 + from pg_index i + join pg_attribute a on a.attrelid = i.indrelid and a.attnum = i.indkey[0] + where i.indrelid = v_a_oid + and i.indisvalid + and i.indisready + and i.indpred is null + and i.indexprs is null + and i.indnkeyatts = 1 + and a.attname = 'archived_at' + ) + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Flow %: generated queue "%" failed physical inspection: %', + p_flow_slug, p_queue_name, format(v_bad, v_atable); + end if; + + -- ========================================== + -- SEQUENCE CONTRACT + -- ========================================== + select reason into v_bad from ( + select 'sequence %s must be a bigint sequence' as reason + from pg_sequence s + where s.seqrelid = v_seq_oid + and s.seqtypid <> 'int8'::regtype + union all + select 'sequence %s is missing' + where not exists (select 1 from pg_sequence s where s.seqrelid = v_seq_oid) + union all + select 'sequence %s is not associated with queue msg_id' + where not exists ( + select 1 + from pg_depend d + join pg_attribute a + on a.attrelid = d.refobjid and a.attnum = d.refobjsubid + where d.objid = v_seq_oid + and d.refobjid = v_q_oid + and a.attname = 'msg_id' + and d.deptype in ('i', 'a') + ) + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Flow %: generated queue "%" failed physical inspection: %', + p_flow_slug, p_queue_name, format(v_bad, v_sequence); + end if; + + -- ========================================== + -- EXTENSION MEMBERSHIP CONTRACT + -- ========================================== + -- When pgmq is installed as an extension, PGMQ's own create/drop path + -- marks the q/a tables and the identity sequence as extension members + -- (pg_depend deptype 'e'). Objects without that membership were created + -- outside PGMQ's implementation and must not be treated as owned + -- generated queues. + select e.oid into v_ext_oid from pg_extension e where e.extname = 'pgmq'; + if v_ext_oid is not null then + select reason into v_bad from ( + select 'queue table %s is not a member of the installed pgmq extension' as reason + where not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_q_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) + union all + select 'archive table %s is not a member of the installed pgmq extension' + where not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_a_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) + union all + select 'sequence %s is not a member of the installed pgmq extension' + where not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_seq_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Flow %: generated queue "%" failed extension-membership inspection: %', + p_flow_slug, p_queue_name, format(v_bad, v_qtable); + end if; + end if; + + return jsonb_build_object('state', 'present', 'metadata_name', v_metadata_name); +end; +$$; + +-- Provision (or verify) the canonical generated queue for a flow under the +-- canonical flow advisory lock, the concrete flow row lock, and the pgmq.meta +-- topology fence. Never adopts foreign resources and never reconstructs a +-- lost queue that live definitions/tasks still reference. +create or replace function pgflow._ensure_generated_queue( + p_flow_slug text, + p_queue_name text +) +returns void +language plpgsql +volatile +set search_path = '' +as $$ +declare + v_state jsonb; +begin + perform pg_advisory_xact_lock(1, hashtext(lower(p_flow_slug))); + + if not exists (select 1 from pgflow.flows f where f.flow_slug = p_flow_slug for update) then + raise exception 'Flow % does not exist; cannot provision generated queue "%"', + p_flow_slug, p_queue_name; + end if; + + v_state := pgflow._inspect_generated_queue(p_flow_slug, p_queue_name, false); + + if v_state ->> 'state' = 'absent' then + -- A live definition or task snapshot that still references this route + -- means the physical queue was lost; recreating it would hide that loss. + if exists ( + select 1 + from pgflow.steps s + where s.flow_slug = p_flow_slug and s.queue_name = p_queue_name + ) or exists ( + select 1 + from pgflow.step_tasks t + where t.flow_slug = p_flow_slug and t.queue_name = p_queue_name + ) then + raise exception 'Flow %: generated queue "%" is absent but existing steps/tasks reference it; refusing to reconstruct a lost live queue', + p_flow_slug, p_queue_name; + end if; + + perform pgmq.create(p_queue_name); + end if; + + -- Post-create verification under the same fence and locks. + perform pgflow._inspect_generated_queue(p_flow_slug, p_queue_name, true); +end; +$$; diff --git a/pkgs/core/schemas/0100_function__cascade_force_skip_steps.sql b/pkgs/core/schemas/0100_function__cascade_force_skip_steps.sql index 5cc14c16c..c62f26ffc 100644 --- a/pkgs/core/schemas/0100_function__cascade_force_skip_steps.sql +++ b/pkgs/core/schemas/0100_function__cascade_force_skip_steps.sql @@ -12,10 +12,12 @@ DECLARE v_flow_slug text; v_total_skipped int := 0; BEGIN - -- Get flow_slug for this run + -- Lock the parent run at direct entry before any run-mutating work; + -- callers that already hold the run lock re-acquire it harmlessly. SELECT r.flow_slug INTO v_flow_slug FROM pgflow.runs r - WHERE r.run_id = _cascade_force_skip_steps.run_id; + WHERE r.run_id = _cascade_force_skip_steps.run_id + FOR UPDATE; IF v_flow_slug IS NULL THEN RAISE EXCEPTION 'Run not found: %', _cascade_force_skip_steps.run_id; @@ -100,14 +102,17 @@ BEGIN FROM skipped AS skipped_step ) AND task.status IN ('queued', 'started') - RETURNING task.message_id + RETURNING task.queue_name, task.message_id ), -- ---------- Archive queued/started task messages for skipped steps ---------- + -- Grouped by the task's queue snapshot; only newly skipped steps' tasks are + -- archived (preexisting skipped steps were already archived) (#650) archived_messages AS ( - SELECT pgmq.archive(v_flow_slug, ARRAY_AGG(task.message_id)) as result - FROM skipped_tasks AS task - WHERE task.message_id IS NOT NULL - HAVING COUNT(task.message_id) > 0 + SELECT pgmq.archive(st.queue_name, ARRAY_AGG(st.message_id)) as result + FROM skipped_tasks AS st + WHERE st.message_id IS NOT NULL + GROUP BY st.queue_name + HAVING COUNT(st.message_id) > 0 ), -- ---------- Update run counters ---------- run_updates AS ( diff --git a/pkgs/core/schemas/0100_function_add_step.sql b/pkgs/core/schemas/0100_function_add_step.sql index bb3475979..bf377e40b 100644 --- a/pkgs/core/schemas/0100_function_add_step.sql +++ b/pkgs/core/schemas/0100_function_add_step.sql @@ -10,21 +10,45 @@ create or replace function pgflow.add_step( required_input_pattern jsonb default null, forbidden_input_pattern jsonb default null, when_unmet text default 'skip', - when_exhausted text default 'fail' + when_exhausted text default 'fail', + queue_name text default null ) returns pgflow.steps language plpgsql -set search_path to '' +set search_path = '' volatile as $$ DECLARE result_step pgflow.steps; next_idx int; + v_queue_name text; + v_alias_slug text; BEGIN - -- Validate map step constraints - -- Map steps can have either: - -- 0 dependencies (root map - maps over flow input array) - -- 1 dependency (dependent map - maps over dependency output array) + -- Canonical flow lock shared with compilation and deletion + PERFORM pg_advisory_xact_lock(1, hashtext(lower(add_step.flow_slug))); + + -- Lock the exact concrete flow row before calculating the next step index + IF NOT EXISTS ( + SELECT 1 FROM pgflow.flows f + WHERE f.flow_slug = add_step.flow_slug + FOR UPDATE + ) THEN + RAISE EXCEPTION 'Flow % does not exist', add_step.flow_slug; + END IF; + + -- Resolve the route: omitted defaults to the canonical generated queue; + -- an explicit equal canonical route is accepted, a different route is not. + v_queue_name := COALESCE(add_step.queue_name, lower(add_step.flow_slug)); + IF add_step.queue_name IS NOT NULL AND add_step.queue_name <> lower(add_step.flow_slug) THEN + RAISE EXCEPTION 'Flow %: step "%" cannot use queue "%" (custom routes do not exist in #650; the canonical route is "%")', + add_step.flow_slug, add_step.step_slug, add_step.queue_name, lower(add_step.flow_slug); + END IF; + + -- Validate the step slug and map constraints before any provisioning + IF NOT pgflow.is_valid_slug(add_step.step_slug) THEN + RAISE EXCEPTION 'Flow %: "%" is not a valid step slug', add_step.flow_slug, add_step.step_slug; + END IF; + IF COALESCE(add_step.step_type, 'single') = 'map' AND COALESCE(array_length(add_step.deps_slugs, 1), 0) > 1 THEN RAISE EXCEPTION 'Map step "%" can have at most one dependency, but % were provided: %', add_step.step_slug, @@ -32,14 +56,43 @@ BEGIN array_to_string(add_step.deps_slugs, ', '); END IF; - -- Get next step index + -- Dependencies must reference existing steps of this exact flow + PERFORM 1 + FROM unnest(COALESCE(add_step.deps_slugs, '{}')) AS d(dep_slug) + WHERE NOT EXISTS ( + SELECT 1 FROM pgflow.steps s + WHERE s.flow_slug = add_step.flow_slug + AND s.step_slug = d.dep_slug + ); + IF FOUND THEN + RAISE EXCEPTION 'Flow %: step "%" has a dependency that does not exist', add_step.flow_slug, add_step.step_slug; + END IF; + + -- Case-alias precheck before the unique index; names both spellings + SELECT s.step_slug INTO v_alias_slug + FROM pgflow.steps s + WHERE s.flow_slug = add_step.flow_slug + AND lower(s.step_slug) = lower(add_step.step_slug) + AND s.step_slug <> add_step.step_slug + LIMIT 1; + IF v_alias_slug IS NOT NULL THEN + RAISE SQLSTATE '23505' USING MESSAGE = format( + 'Flow %s: step "%s" conflicts with existing step "%s" (case-insensitive step namespace)', + add_step.flow_slug, add_step.step_slug, v_alias_slug + ); + END IF; + + -- Provision/verify the generated queue through the shared ownership path + PERFORM pgflow._ensure_generated_queue(add_step.flow_slug, v_queue_name); + + -- Get next step index (under the flow row lock) SELECT COALESCE(MAX(s.step_index) + 1, 0) INTO next_idx FROM pgflow.steps s WHERE s.flow_slug = add_step.flow_slug; - -- Create the step + -- Create the step with its resolved route snapshot INSERT INTO pgflow.steps ( - flow_slug, step_slug, step_type, step_index, deps_count, + flow_slug, step_slug, step_type, step_index, deps_count, queue_name, opt_max_attempts, opt_base_delay, opt_timeout, opt_start_delay, required_input_pattern, forbidden_input_pattern, when_unmet, when_exhausted ) @@ -49,6 +102,7 @@ BEGIN COALESCE(add_step.step_type, 'single'), next_idx, COALESCE(array_length(add_step.deps_slugs, 1), 0), + v_queue_name, add_step.max_attempts, add_step.base_delay, add_step.timeout, diff --git a/pkgs/core/schemas/0100_function_archive_task_message.sql b/pkgs/core/schemas/0100_function_archive_task_message.sql index 3ed804114..004a3d4c3 100644 --- a/pkgs/core/schemas/0100_function_archive_task_message.sql +++ b/pkgs/core/schemas/0100_function_archive_task_message.sql @@ -1,23 +1,46 @@ +-- Archive a single task's queue message using the task's queue snapshot. +-- Lock order: parent/run, step state, then task row, then queue row (#650). +-- Re-entrant with callers that already hold these row locks. create or replace function pgflow._archive_task_message( p_run_id uuid, p_step_slug text, p_task_index int ) returns void -language sql +language plpgsql volatile -set search_path to '' +set search_path = '' as $$ - SELECT pgmq.archive( - r.flow_slug, - ARRAY_AGG(st.message_id) - ) - FROM pgflow.step_tasks st - JOIN pgflow.runs r ON st.run_id = r.run_id - WHERE st.run_id = p_run_id - AND st.step_slug = p_step_slug - AND st.task_index = p_task_index - AND st.message_id IS NOT NULL - GROUP BY r.flow_slug - HAVING COUNT(st.message_id) > 0; +declare + v_batch record; +begin + PERFORM 1 FROM pgflow.runs r + WHERE r.run_id = p_run_id + FOR UPDATE; + + PERFORM 1 FROM pgflow.step_states ss + WHERE ss.run_id = p_run_id + AND ss.step_slug = p_step_slug + FOR UPDATE; + + FOR v_batch IN + WITH locked_tasks AS ( + SELECT task.queue_name, task.message_id + FROM pgflow.step_tasks task + WHERE task.run_id = p_run_id + AND task.step_slug = p_step_slug + AND task.task_index = p_task_index + AND task.message_id IS NOT NULL + ORDER BY task.task_index + FOR UPDATE + ) + SELECT + lt.queue_name, + ARRAY_AGG(lt.message_id ORDER BY lt.message_id) AS ids + FROM locked_tasks lt + GROUP BY lt.queue_name + LOOP + PERFORM pgmq.archive(v_batch.queue_name, v_batch.ids); + END LOOP; +END; $$; diff --git a/pkgs/core/schemas/0100_function_cascade_resolve_conditions.sql b/pkgs/core/schemas/0100_function_cascade_resolve_conditions.sql index 8fe445c8e..4c577eb09 100644 --- a/pkgs/core/schemas/0100_function_cascade_resolve_conditions.sql +++ b/pkgs/core/schemas/0100_function_cascade_resolve_conditions.sql @@ -19,14 +19,17 @@ DECLARE v_processed_count int; v_run_transitioned boolean; v_flow_slug text; - v_cancelled_message_ids bigint[]; + v_archive_batch record; BEGIN -- ========================================== - -- GUARD: Early return if run is already terminal + -- GUARD: lock the parent run at direct entry, then early-return if the + -- run is already terminal. Callers that already hold the run lock (for + -- example complete_task) re-acquire it harmlessly in the same transaction. -- ========================================== SELECT r.status, r.input INTO v_run_status, v_run_input FROM pgflow.runs r - WHERE r.run_id = cascade_resolve_conditions.run_id; + WHERE r.run_id = cascade_resolve_conditions.run_id + FOR UPDATE; IF v_run_status IN ('failed', 'completed') THEN RETURN v_run_status != 'failed'; @@ -158,24 +161,25 @@ BEGIN ); -- Terminalize every unfinished task across all branches as cancelled, - -- capturing their message ids for archival below. Lock-order invariant: - -- always lock/update step_tasks before PGMQ queue rows. - WITH cancelled_tasks AS ( - UPDATE pgflow.step_tasks AS task - SET status = 'cancelled' - WHERE task.run_id = cascade_resolve_conditions.run_id - AND task.status IN ('queued', 'started') - RETURNING task.message_id - ) - SELECT ARRAY_AGG(ct.message_id) INTO v_cancelled_message_ids - FROM cancelled_tasks ct - WHERE ct.message_id IS NOT NULL; - - -- Archive the cancelled task messages captured above (only after their - -- task rows are terminalized) - IF v_cancelled_message_ids IS NOT NULL THEN - PERFORM pgmq.archive(v_first_fail.flow_slug, v_cancelled_message_ids); - END IF; + -- capturing their queue/message pairs for archival below. Lock-order + -- invariant: always lock/update step_tasks before PGMQ queue rows. + FOR v_archive_batch IN + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = cascade_resolve_conditions.run_id + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ) + SELECT + ct.queue_name, + ARRAY_AGG(ct.message_id ORDER BY ct.message_id) AS ids + FROM cancelled_tasks ct + WHERE ct.message_id IS NOT NULL + GROUP BY ct.queue_name + LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); + END LOOP; END IF; RETURN false; diff --git a/pkgs/core/schemas/0100_function_complete_task.sql b/pkgs/core/schemas/0100_function_complete_task.sql index 6b593bebd..d7daa7d63 100644 --- a/pkgs/core/schemas/0100_function_complete_task.sql +++ b/pkgs/core/schemas/0100_function_complete_task.sql @@ -14,13 +14,21 @@ declare v_dependent_map_slug text; v_run_record pgflow.runs%ROWTYPE; v_step_record pgflow.step_states%ROWTYPE; - v_violation_archived_ids bigint[]; + v_archive_batch record; begin -- ========================================== -- GUARD: No mutations on failed runs -- ========================================== IF EXISTS (SELECT 1 FROM pgflow.runs WHERE pgflow.runs.run_id = complete_task.run_id AND pgflow.runs.status = 'failed') THEN + -- Archive the late callback message through the locked single-task + -- helper (run/step/task locks are its own acquisition); the message must + -- not stay visible for re-reading after a failed run (#650). + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); RETURN QUERY SELECT * FROM pgflow.step_tasks WHERE pgflow.step_tasks.run_id = complete_task.run_id AND pgflow.step_tasks.step_slug = complete_task.step_slug @@ -68,16 +76,13 @@ END IF; -- If the step is not in 'started' state, this is a late callback. -- Do not mutate step_states or runs, archive message, return task row. IF v_step_record.status != 'started' THEN - -- Archive the task message if present (prevents stuck work) - PERFORM pgmq.archive( - v_run_record.flow_slug, - st.message_id - ) - FROM pgflow.step_tasks st - WHERE st.run_id = complete_task.run_id - AND st.step_slug = complete_task.step_slug - AND st.task_index = complete_task.task_index - AND st.message_id IS NOT NULL; + -- Archive the task message if present (prevents stuck work) through the + -- locked single-task helper; run/step locks are already held here + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); -- Return the current task row without any mutations RETURN QUERY SELECT * FROM pgflow.step_tasks WHERE pgflow.step_tasks.run_id = complete_task.run_id @@ -172,36 +177,44 @@ IF v_dependent_map_slug IS NOT NULL THEN ); -- Terminalize every other unfinished task as cancelled, capturing their - -- message ids for archival below. Lock-order invariant: always lock/update - -- step_tasks before PGMQ queue rows. The culprit task is already terminal - -- (failed above), so it is excluded from the cancellation set. - WITH cancelled_tasks AS ( - UPDATE pgflow.step_tasks AS task - SET status = 'cancelled' - WHERE task.run_id = complete_task.run_id - AND task.status IN ('queued', 'started') - RETURNING task.message_id - ), - culprit_task AS ( - -- Terminal culprit row: safe to read for its message id after terminalization - SELECT st.message_id - FROM pgflow.step_tasks st - WHERE st.run_id = complete_task.run_id - AND st.step_slug = complete_task.step_slug - AND st.task_index = complete_task.task_index - AND st.message_id IS NOT NULL - ) - SELECT ARRAY_AGG(ids.message_id) INTO v_violation_archived_ids - FROM ( - SELECT message_id FROM culprit_task - UNION ALL - SELECT message_id FROM cancelled_tasks WHERE message_id IS NOT NULL - ) ids; - - -- Archive the culprit and cancelled task messages (only after their task rows are terminalized) - IF v_violation_archived_ids IS NOT NULL THEN - PERFORM pgmq.archive(v_run_record.flow_slug, v_violation_archived_ids); - END IF; + -- queue/message pairs for archival below. Lock-order invariant: always + -- lock/update step_tasks before PGMQ queue rows. The culprit task is + -- already terminal (failed above), so it is excluded from the cancellation + -- set. The grouped FOR forces the cancellation UPDATE to run before any + -- archive call. + FOR v_archive_batch IN + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = complete_task.run_id + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ), + culprit_task AS ( + -- Terminal culprit row: safe to read for its queue/message pair after + -- terminalization + SELECT st.queue_name, st.message_id + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.message_id IS NOT NULL + ), + archived_pairs AS ( + SELECT + ids.queue_name, + ARRAY_AGG(ids.message_id ORDER BY ids.message_id) AS ids + FROM ( + SELECT queue_name, message_id FROM culprit_task + UNION ALL + SELECT queue_name, message_id FROM cancelled_tasks WHERE message_id IS NOT NULL + ) ids + GROUP BY ids.queue_name + ) + SELECT ap.queue_name, ap.ids FROM archived_pairs ap + LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); + END LOOP; -- Return the failed task row (API contract: always return task row) RETURN QUERY @@ -378,13 +391,12 @@ IF v_step_state.status = 'completed' THEN -- skipped steps can set initial_tasks=0 for their map dependents IF NOT pgflow.cascade_resolve_conditions(complete_task.run_id) THEN -- Run was failed due to a condition with when_unmet='fail' - -- Archive the current task's message before returning - PERFORM pgmq.archive( - (SELECT r.flow_slug FROM pgflow.runs r WHERE r.run_id = complete_task.run_id), - (SELECT st.message_id FROM pgflow.step_tasks st - WHERE st.run_id = complete_task.run_id - AND st.step_slug = complete_task.step_slug - AND st.task_index = complete_task.task_index) + -- Archive the current task's message before returning through the + -- locked single-task helper + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index ); RETURN QUERY SELECT * FROM pgflow.step_tasks WHERE pgflow.step_tasks.run_id = complete_task.run_id @@ -399,18 +411,18 @@ IF v_step_state.status = 'completed' THEN END IF; -- ---------- Archive completed task message ---------- --- Move message from active queue to archive table +-- Move message from active queue to archive table using the task's queue +-- snapshot (#650) PERFORM ( WITH completed_tasks AS ( - SELECT r.flow_slug, st.message_id + SELECT st.queue_name, st.message_id FROM pgflow.step_tasks st - JOIN pgflow.runs r ON st.run_id = r.run_id WHERE st.run_id = complete_task.run_id AND st.step_slug = complete_task.step_slug AND st.task_index = complete_task.task_index AND st.status = 'completed' ) - SELECT pgmq.archive(ct.flow_slug, ct.message_id) + SELECT pgmq.archive(ct.queue_name, ct.message_id) FROM completed_tasks ct WHERE EXISTS (SELECT 1 FROM completed_tasks) ); diff --git a/pkgs/core/schemas/0100_function_create_flow.sql b/pkgs/core/schemas/0100_function_create_flow.sql index 5c56312a3..52ff79ee7 100644 --- a/pkgs/core/schemas/0100_function_create_flow.sql +++ b/pkgs/core/schemas/0100_function_create_flow.sql @@ -1,6 +1,10 @@ --- Create a new flow with optional configuration. --- NULL parameters use defaults defined in the 'defaults' CTE below. --- This allows callers to pass NULL to explicitly use the default value. +-- Create a new flow definition with optional configuration. +-- NULL parameters use defaults defined below; callers may pass NULL to +-- explicitly use the default value. +-- +-- Definition-only (#650): this function performs no queue DDL. The generated +-- default queue is provisioned by add_step()/_create_flow_from_shape() through +-- the shared generated-queue path. create or replace function pgflow.create_flow( flow_slug text, max_attempts int default null, @@ -8,33 +12,56 @@ create or replace function pgflow.create_flow( timeout int default null ) returns pgflow.flows -language sql -set search_path to '' +language plpgsql volatile +set search_path = '' as $$ -WITH - defaults AS ( - SELECT 3 AS def_max_attempts, 5 AS def_base_delay, 60 AS def_timeout - ), - flow_upsert AS ( - INSERT INTO pgflow.flows (flow_slug, opt_max_attempts, opt_base_delay, opt_timeout) - SELECT - flow_slug, - COALESCE(max_attempts, defaults.def_max_attempts), - COALESCE(base_delay, defaults.def_base_delay), - COALESCE(timeout, defaults.def_timeout) - FROM defaults - ON CONFLICT (flow_slug) DO UPDATE - SET flow_slug = pgflow.flows.flow_slug -- Dummy update - RETURNING * - ), - ensure_queue AS ( - SELECT pgmq.create(flow_slug) - WHERE NOT EXISTS ( - SELECT 1 FROM pgmq.list_queues() WHERE queue_name = flow_slug - ) +#variable_conflict use_column +declare + result_flow pgflow.flows; + v_alias_slug text; +begin + -- Canonical advisory lock: case aliases share the lock so concurrent + -- compilation cannot create both spellings. + perform pg_advisory_xact_lock(1, hashtext(lower(create_flow.flow_slug))); + + -- Precheck case aliases before the unique index does; the message names + -- both exact spellings. + select f.flow_slug into v_alias_slug + from pgflow.flows f + where lower(f.flow_slug) = lower(create_flow.flow_slug) + and f.flow_slug <> create_flow.flow_slug + limit 1; + + if v_alias_slug is not null then + raise sqlstate '23505' using message = format( + 'Flow "%s" conflicts with existing flow "%s" (case-insensitive flow namespace)', + create_flow.flow_slug, v_alias_slug + ); + end if; + + if not exists (select 1 from pgflow.flows f where f.flow_slug = create_flow.flow_slug) then + -- New identity: the canonical generated route must be genuinely absent so + -- create_flow cannot launder an external queue into apparent ownership + -- before add_step() runs. The inspection takes the pgmq.meta fence. + perform pgflow._inspect_generated_queue( + create_flow.flow_slug, + lower(create_flow.flow_slug), + false + ); + end if; + + insert into pgflow.flows as flow (flow_slug, opt_max_attempts, opt_base_delay, opt_timeout) + values ( + create_flow.flow_slug, + coalesce(max_attempts, 3), + coalesce(base_delay, 5), + coalesce(timeout, 60) ) -SELECT f.* -FROM flow_upsert f -LEFT JOIN (SELECT 1 FROM ensure_queue) _dummy ON true; -- Left join ensures flow is returned + on conflict (flow_slug) do update + set flow_slug = flow.flow_slug -- Dummy update: idempotent + returning * into result_flow; + + return result_flow; +end; $$; diff --git a/pkgs/core/schemas/0100_function_create_flow_from_shape.sql b/pkgs/core/schemas/0100_function_create_flow_from_shape.sql index a6006432d..ec2767f73 100644 --- a/pkgs/core/schemas/0100_function_create_flow_from_shape.sql +++ b/pkgs/core/schemas/0100_function_create_flow_from_shape.sql @@ -1,6 +1,11 @@ -- Compile a flow from a JSONB shape -- Creates the flow and all its steps using existing create_flow/add_step functions -- Includes options from shape (NULL values = use default) +-- +-- #650: preflights the complete shape and all required canonical queues under +-- locks before creating anything, persists the flow, provisions the generated +-- default (even for a zero-step plain flow), then creates every step with its +-- explicitly resolved canonical route. create or replace function pgflow._create_flow_from_shape( p_flow_slug text, p_shape jsonb @@ -8,18 +13,24 @@ create or replace function pgflow._create_flow_from_shape( returns void language plpgsql volatile -set search_path to '' +set search_path = '' as $$ DECLARE v_step jsonb; v_deps text[]; v_flow_options jsonb; v_step_options jsonb; + v_canonical_queue text := lower(p_flow_slug); BEGIN + -- Preflight the complete shape under the canonical flow lock before any + -- mutation: a late invalid step must not leave earlier queues/definitions. + PERFORM pg_advisory_xact_lock(1, hashtext(lower(p_flow_slug))); + PERFORM pgflow._validate_flow_shape(p_flow_slug, p_shape); + -- Extract flow-level options (may be null) v_flow_options := p_shape->'options'; - -- Create the flow with options (NULL = use default) + -- Create the flow definition (definition-only; no queue DDL) PERFORM pgflow.create_flow( p_flow_slug, (v_flow_options->>'maxAttempts')::int, @@ -27,7 +38,10 @@ BEGIN (v_flow_options->>'timeout')::int ); - -- Iterate over steps in order and add each one + -- Provision the generated default queue for the persisted identity + PERFORM pgflow._ensure_generated_queue(p_flow_slug, v_canonical_queue); + + -- Iterate over steps in order and add each one with its resolved route FOR v_step IN SELECT * FROM jsonb_array_elements(p_shape->'steps') LOOP -- Convert dependencies jsonb array to text array @@ -59,7 +73,8 @@ BEGIN WHEN (v_step->'forbiddenInputPattern'->>'defined')::boolean THEN v_step->'forbiddenInputPattern'->'value' ELSE NULL - END + END, + queue_name => v_canonical_queue ); END LOOP; END; diff --git a/pkgs/core/schemas/0100_function_delete_flow_and_data.sql b/pkgs/core/schemas/0100_function_delete_flow_and_data.sql index 7cd14e110..92bc01b9a 100644 --- a/pkgs/core/schemas/0100_function_delete_flow_and_data.sql +++ b/pkgs/core/schemas/0100_function_delete_flow_and_data.sql @@ -1,22 +1,154 @@ -- Deletes a flow and all its associated data -- WARNING: This is destructive - deletes flow definition AND all runtime data -- Used by ensure_flow_compiled for development mode recompilation +-- +-- #650 lock order: canonical advisory lock, concrete flow row, runtime rows +-- (runs, step states, tasks in (run_id, step_slug, task_index) order), then +-- the pgmq.meta topology fence, then physical queue table locks. Ownership is +-- retained until every validated queue is dropped; the flow identity row is +-- deleted last. create or replace function pgflow.delete_flow_and_data(p_flow_slug text) returns void language plpgsql volatile -set search_path to '' +set search_path = '' as $$ +DECLARE + v_route text[]; + v_route_names text[]; + v_metadata_names text[]; + v_snapshot_violation record; + v_queue text; + v_qtable text; + v_atable text; + v_sequence text; + v_inspect_result jsonb; + v_idx int; BEGIN - -- Drop queue and archive table (pgmq) - PERFORM pgmq.drop_queue(p_flow_slug); + PERFORM pg_advisory_xact_lock(1, hashtext(lower(p_flow_slug))); - -- Delete all associated data in the correct order (respecting FK constraints) + -- Retain and lock the concrete identity; reject a missing flow + IF NOT EXISTS ( + SELECT 1 FROM pgflow.flows f + WHERE f.flow_slug = p_flow_slug + FOR UPDATE + ) THEN + RAISE EXCEPTION 'Flow % does not exist', p_flow_slug; + END IF; + + -- Runtime locks in the established order, before any queue/metadata access + PERFORM 1 FROM pgflow.runs r + WHERE r.flow_slug = p_flow_slug + ORDER BY r.run_id + FOR UPDATE; + + PERFORM 1 FROM pgflow.step_states s + WHERE s.flow_slug = p_flow_slug + ORDER BY s.run_id, s.step_slug + FOR UPDATE; + + PERFORM 1 FROM pgflow.step_tasks t + WHERE t.flow_slug = p_flow_slug + ORDER BY t.run_id, t.step_slug, t.task_index + FOR UPDATE; + + -- Topology fence after runtime locks (never the reverse order) + LOCK TABLE pgmq.meta IN SHARE ROW EXCLUSIVE MODE; + + -- Capture the complete private route: persisted steps plus the default for + -- an empty plain flow. An unprovisioned definition-only flow is not + -- permission to drop a same-named resource; _inspect_generated_queue + -- rejects that below. + SELECT COALESCE( + ARRAY_AGG(DISTINCT s.queue_name ORDER BY s.queue_name), + ARRAY[lower(p_flow_slug)] + ) INTO v_route + FROM pgflow.steps s + WHERE s.flow_slug = p_flow_slug; + + -- Validate every task snapshot against the captured route + SELECT t.run_id, t.step_slug, t.task_index, t.queue_name + INTO v_snapshot_violation + FROM pgflow.step_tasks t + WHERE t.flow_slug = p_flow_slug + AND NOT (t.queue_name = ANY(v_route)) + ORDER BY t.run_id, t.step_slug, t.task_index + LIMIT 1; + + IF v_snapshot_violation IS NOT NULL THEN + RAISE EXCEPTION 'Flow %: task %/%/% snapshot queue "%" is outside the validated private route; refusing deletion', + p_flow_slug, + v_snapshot_violation.run_id, + v_snapshot_violation.step_slug, + v_snapshot_violation.task_index, + v_snapshot_violation.queue_name; + END IF; + + v_route_names := v_route; + v_metadata_names := ARRAY[]::text[]; + + -- Resolve exact metadata spelling and physical validity per route queue; + -- this also rejects missing, ambiguous, malformed, or differently owned + -- resources instead of dropping an uncertain physical queue. + FOR v_idx IN 1..COALESCE(array_length(v_route, 1), 0) + LOOP + v_queue := v_route[v_idx]; + v_inspect_result := pgflow._inspect_generated_queue(p_flow_slug, v_queue, true); + + -- Lock the validated physical queue/archive tables before mutation + EXECUTE format( + 'LOCK TABLE pgmq.%I, pgmq.%I IN ACCESS EXCLUSIVE MODE', + pgmq.format_table_name(v_queue, 'q'), + pgmq.format_table_name(v_queue, 'a') + ); + + -- Recheck ownership/shape after the physical locks are held + v_inspect_result := pgflow._inspect_generated_queue(p_flow_slug, v_queue, true); + + -- Remember the exact metadata spelling for the drop below: the flow row + -- and step definitions may be gone by then. + v_metadata_names[v_idx] := v_inspect_result ->> 'metadata_name'; + END LOOP; + + -- Delete runtime rows and step definitions in FK order while retaining the + -- flow identity and captured validated queue names DELETE FROM pgflow.step_tasks AS task WHERE task.flow_slug = p_flow_slug; DELETE FROM pgflow.step_states AS state WHERE state.flow_slug = p_flow_slug; DELETE FROM pgflow.runs AS run WHERE run.flow_slug = p_flow_slug; DELETE FROM pgflow.deps AS dep WHERE dep.flow_slug = p_flow_slug; DELETE FROM pgflow.steps AS step WHERE step.flow_slug = p_flow_slug; + + -- Drop each validated private queue using its exact metadata spelling. + -- No per-message archival/deletion happens before the whole-queue drop. + FOR v_idx IN 1..COALESCE(array_length(v_route_names, 1), 0) + LOOP + v_queue := v_route_names[v_idx]; + v_qtable := pgmq.format_table_name(v_queue, 'q'); + v_atable := pgmq.format_table_name(v_queue, 'a'); + v_sequence := v_qtable || '_msg_id_seq'; + + PERFORM pgmq.drop_queue(v_metadata_names[v_idx]); + + -- Post-drop completeness: pgmq.drop_queue must have removed the + -- metadata row, both physical tables, and the identity sequence. A + -- partial drop leaves the namespace ambiguous and must abort before + -- the flow identity row is deleted. + IF EXISTS ( + SELECT 1 FROM pgmq.meta m WHERE lower(m.queue_name) = v_queue + ) THEN + RAISE EXCEPTION 'Flow %: dropping generated queue "%" left its PGMQ metadata behind; deletion aborted with everything rolled back', + p_flow_slug, v_queue; + END IF; + + IF to_regclass(format('pgmq.%I', v_qtable)) IS NOT NULL + OR to_regclass(format('pgmq.%I', v_atable)) IS NOT NULL + OR to_regclass(format('pgmq.%I', v_sequence)) IS NOT NULL THEN + RAISE EXCEPTION 'Flow %: dropping generated queue "%" left physical objects behind (queue table: %, archive table: %, sequence: %); deletion aborted with everything rolled back', + p_flow_slug, v_queue, v_qtable, v_atable, v_sequence; + END IF; + END LOOP; + + -- Delete the concrete flow identity row last DELETE FROM pgflow.flows AS flow WHERE flow.flow_slug = p_flow_slug; END; $$; diff --git a/pkgs/core/schemas/0100_function_ensure_flow_compiled.sql b/pkgs/core/schemas/0100_function_ensure_flow_compiled.sql index a2bfa7d07..347c1987b 100644 --- a/pkgs/core/schemas/0100_function_ensure_flow_compiled.sql +++ b/pkgs/core/schemas/0100_function_ensure_flow_compiled.sql @@ -1,14 +1,17 @@ -- Ensure a flow is compiled in the database -- Auto-detects environment via is_local(): local -> auto-recompile, production -> fail on mismatch --- Returns: { status: 'compiled' | 'verified' | 'recompiled' | 'mismatch', differences: text[] } +-- Returns: { status: 'compiled' | 'verified' | 'recompiled' | 'mismatch', differences: text[], queue_name } +-- #650: all non-mismatch results carry the checked canonical queue; the +-- verified branch also proves the persisted route's physical resources. create or replace function pgflow.ensure_flow_compiled( flow_slug text, - shape jsonb + shape jsonb, + worker_protocol jsonb ) returns jsonb language plpgsql volatile -set search_path to '' +set search_path = '' as $$ DECLARE v_lock_key int; @@ -16,9 +19,20 @@ DECLARE v_db_shape jsonb; v_differences text[]; v_is_local boolean; + v_canonical_queue text := lower(ensure_flow_compiled.flow_slug); BEGIN - -- Generate lock key from flow_slug (deterministic hash) - v_lock_key := hashtext(ensure_flow_compiled.flow_slug); + -- Queue-capable startup handshake (#650): the required third argument has + -- no default and no fallback wrapper. Version 1 identifies the queue-aware + -- startup/claim semantics. Reject missing/non-object/wrong-version values + -- before any definition mutation. + IF jsonb_typeof(worker_protocol) IS DISTINCT FROM 'object' + OR worker_protocol -> 'version' IS DISTINCT FROM '1'::jsonb THEN + RAISE EXCEPTION 'Queue-capable worker protocol version 1 is required'; + END IF; + -- Generate lock key from the canonical slug (deterministic hash). + -- Case aliases share the lock so concurrent compilation of 'Orders' and + -- 'orders' serializes against each other. + v_lock_key := hashtext(lower(ensure_flow_compiled.flow_slug)); -- Acquire transaction-level advisory lock -- Serializes concurrent compilation attempts for same flow @@ -31,7 +45,12 @@ BEGIN -- 2. If flow missing: compile (both environments) IF NOT v_flow_exists THEN PERFORM pgflow._create_flow_from_shape(ensure_flow_compiled.flow_slug, ensure_flow_compiled.shape); - RETURN jsonb_build_object('status', 'compiled', 'differences', '[]'::jsonb); + RETURN jsonb_build_object( + 'status', 'compiled', + 'differences', '[]'::jsonb, + 'protocol_version', 1, + 'queue_name', v_canonical_queue + ); END IF; -- 3. Get current shape from DB @@ -40,9 +59,20 @@ BEGIN -- 4. Compare shapes v_differences := pgflow._compare_flow_shapes(ensure_flow_compiled.shape, v_db_shape); - -- 5. If shapes match: return verified + -- 5. If shapes match: inspect the persisted route/resources before + -- returning verified. A shape match alone does not prove a valid queue. IF array_length(v_differences, 1) IS NULL THEN - RETURN jsonb_build_object('status', 'verified', 'differences', '[]'::jsonb); + PERFORM pgflow._inspect_generated_queue( + ensure_flow_compiled.flow_slug, + v_canonical_queue, + true + ); + RETURN jsonb_build_object( + 'status', 'verified', + 'differences', '[]'::jsonb, + 'protocol_version', 1, + 'queue_name', v_canonical_queue + ); END IF; -- 6. Shapes differ - auto-detect environment via is_local() @@ -51,13 +81,25 @@ BEGIN -- Local mode is the only destructive branch; production mismatches never -- delete data and return mismatch so worker startup fails. IF v_is_local THEN - -- Recompile in local/dev: full deletion + fresh compile + -- Preflight the entire replacement shape before any deletion so a bad + -- late step cannot destroy old data. Deletion follows the established + -- runtime-before-metadata lock order. + PERFORM pgflow._validate_flow_shape(ensure_flow_compiled.flow_slug, ensure_flow_compiled.shape); + PERFORM pgflow.delete_flow_and_data(ensure_flow_compiled.flow_slug); PERFORM pgflow._create_flow_from_shape(ensure_flow_compiled.flow_slug, ensure_flow_compiled.shape); - RETURN jsonb_build_object('status', 'recompiled', 'differences', to_jsonb(v_differences)); + RETURN jsonb_build_object( + 'status', 'recompiled', + 'differences', to_jsonb(v_differences), + 'protocol_version', 1, + 'queue_name', v_canonical_queue + ); ELSE -- Fail in production - RETURN jsonb_build_object('status', 'mismatch', 'differences', to_jsonb(v_differences)); + RETURN jsonb_build_object( + 'status', 'mismatch', + 'differences', to_jsonb(v_differences) + ); END IF; END; $$; diff --git a/pkgs/core/schemas/0100_function_fail_task.sql b/pkgs/core/schemas/0100_function_fail_task.sql index 6da065e4f..854af350e 100644 --- a/pkgs/core/schemas/0100_function_fail_task.sql +++ b/pkgs/core/schemas/0100_function_fail_task.sql @@ -19,8 +19,7 @@ DECLARE v_prev_step_status text; v_run_status text; v_flow_slug text; - v_skipped_message_ids bigint[]; - v_cancelled_message_ids bigint[]; + v_archive_batch record; begin -- If run is already failed, no retries allowed. @@ -59,14 +58,9 @@ IF v_run_status = 'failed' THEN END IF; IF v_prev_step_status IS NOT NULL AND v_prev_step_status != 'started' THEN - -- Archive the task message if present - PERFORM pgmq.archive(v_flow_slug, ARRAY_AGG(st.message_id)) - FROM pgflow.step_tasks st - WHERE st.run_id = fail_task.run_id - AND st.step_slug = fail_task.step_slug - AND st.task_index = fail_task.task_index - AND st.message_id IS NOT NULL - HAVING COUNT(st.message_id) > 0; + -- Archive the task message if present, through the locked single-task + -- helper (locks already held above) + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); RETURN QUERY SELECT * FROM pgflow.step_tasks WHERE pgflow.step_tasks.run_id = fail_task.run_id @@ -218,23 +212,25 @@ END IF; -- requeue_stalled_tasks() uses the same order; archiving queue rows first -- deadlocks the two transactions against each other. -- Terminalize all still-active sibling task rows for the skipped step, - -- capturing their message ids for archival below. - WITH skipped_tasks AS ( - UPDATE pgflow.step_tasks AS task - SET status = 'skipped' - WHERE task.run_id = fail_task.run_id - AND task.step_slug = fail_task.step_slug - AND task.status IN ('queued', 'started') - RETURNING task.message_id - ) - SELECT ARRAY_AGG(st.message_id) INTO v_skipped_message_ids - FROM skipped_tasks st - WHERE st.message_id IS NOT NULL; - - -- Archive the sibling task messages captured above (only after their task rows are terminalized) - IF v_skipped_message_ids IS NOT NULL THEN - PERFORM pgmq.archive(v_flow_slug, v_skipped_message_ids); - END IF; + -- capturing their queue/message pairs for archival below. + FOR v_archive_batch IN + WITH skipped_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'skipped' + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ) + SELECT + st.queue_name, + ARRAY_AGG(st.message_id ORDER BY st.message_id) AS ids + FROM skipped_tasks st + WHERE st.message_id IS NOT NULL + GROUP BY st.queue_name + LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); + END LOOP; -- Send broadcast event for step skipped PERFORM realtime.send( @@ -331,21 +327,23 @@ END IF; -- PGMQ queue rows. The culprit task is already terminal (failed or requeued by -- fail_or_retry_task), so only unfinished queued/started siblings are cancelled. IF v_run_failed THEN - WITH cancelled_tasks AS ( - UPDATE pgflow.step_tasks AS task - SET status = 'cancelled' - WHERE task.run_id = fail_task.run_id - AND task.status IN ('queued', 'started') - RETURNING task.message_id - ) - SELECT ARRAY_AGG(ct.message_id) INTO v_cancelled_message_ids - FROM cancelled_tasks ct - WHERE ct.message_id IS NOT NULL; - - -- Archive the cancelled task messages captured above (only after their task rows are terminalized) - IF v_cancelled_message_ids IS NOT NULL THEN - PERFORM pgmq.archive(v_flow_slug, v_cancelled_message_ids); - END IF; + FOR v_archive_batch IN + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = fail_task.run_id + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ) + SELECT + ct.queue_name, + ARRAY_AGG(ct.message_id ORDER BY ct.message_id) AS ids + FROM cancelled_tasks ct + WHERE ct.message_id IS NOT NULL + GROUP BY ct.queue_name + LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); + END LOOP; END IF; -- For queued tasks: delay the message for retry with exponential backoff @@ -361,32 +359,36 @@ PERFORM ( ), queued_tasks AS ( SELECT - r.flow_slug, + st.queue_name, st.message_id, pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay FROM pgflow.step_tasks st - JOIN pgflow.runs r ON st.run_id = r.run_id WHERE st.run_id = fail_task.run_id AND st.step_slug = fail_task.step_slug AND st.task_index = fail_task.task_index AND st.status = 'queued' ) - SELECT pgmq.set_vt(qt.flow_slug, qt.message_id, qt.calculated_delay) + SELECT pgmq.set_vt(qt.queue_name, qt.message_id, qt.calculated_delay) FROM queued_tasks qt WHERE EXISTS (SELECT 1 FROM queued_tasks) ); --- For failed tasks: archive the message -PERFORM pgmq.archive(r.flow_slug, ARRAY_AGG(st.message_id)) -FROM pgflow.step_tasks st -JOIN pgflow.runs r ON st.run_id = r.run_id -WHERE st.run_id = fail_task.run_id - AND st.step_slug = fail_task.step_slug - AND st.task_index = fail_task.task_index - AND st.status = 'failed' - AND st.message_id IS NOT NULL -GROUP BY r.flow_slug -HAVING COUNT(st.message_id) > 0; +-- For failed tasks: archive the message grouped by the task's queue snapshot +FOR v_archive_batch IN + SELECT + st.queue_name, + ARRAY_AGG(st.message_id ORDER BY st.message_id) AS ids + FROM pgflow.step_tasks st + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'failed' + AND st.message_id IS NOT NULL + GROUP BY st.queue_name + HAVING COUNT(st.message_id) > 0 +LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); +END LOOP; return query select * from pgflow.step_tasks st diff --git a/pkgs/core/schemas/0100_function_start_flow.sql b/pkgs/core/schemas/0100_function_start_flow.sql index abae94696..c4a35b403 100644 --- a/pkgs/core/schemas/0100_function_start_flow.sql +++ b/pkgs/core/schemas/0100_function_start_flow.sql @@ -13,6 +13,14 @@ declare v_root_map_count int; begin +-- ========================================== +-- LOCK: Hold the concrete flow definition against deletion/recompilation +-- while this producer reads step definitions (#650). +-- ========================================== +perform 1 from pgflow.flows f +where f.flow_slug = start_flow.flow_slug +for key share; + -- ========================================== -- VALIDATION: Root map array input -- ========================================== diff --git a/pkgs/core/schemas/0100_function_start_ready_steps.sql b/pkgs/core/schemas/0100_function_start_ready_steps.sql index a70ca9f26..fd9632852 100644 --- a/pkgs/core/schemas/0100_function_start_ready_steps.sql +++ b/pkgs/core/schemas/0100_function_start_ready_steps.sql @@ -75,6 +75,7 @@ message_batches AS ( started_step.flow_slug, started_step.run_id, started_step.step_slug, + step.queue_name, COALESCE(step.opt_start_delay, 0) as delay, array_agg( jsonb_build_object( @@ -91,32 +92,37 @@ message_batches AS ( AND step.step_slug = started_step.step_slug -- Generate task indices from 0 to initial_tasks-1 CROSS JOIN LATERAL generate_series(0, started_step.initial_tasks - 1) AS task_idx(task_index) - GROUP BY started_step.flow_slug, started_step.run_id, started_step.step_slug, step.opt_start_delay + GROUP BY started_step.flow_slug, started_step.run_id, started_step.step_slug, step.queue_name, step.opt_start_delay ), -- ---------- Send messages to queue ---------- -- Uses batch sending for performance with large arrays +-- Sends to each step's resolved route; performs no queue DDL (#650) sent_messages AS ( SELECT mb.flow_slug, mb.run_id, mb.step_slug, + mb.queue_name, task_indices.task_index, msg_ids.msg_id FROM message_batches mb CROSS JOIN LATERAL unnest(mb.task_indices) WITH ORDINALITY AS task_indices(task_index, idx_ord) - CROSS JOIN LATERAL pgmq.send_batch(mb.flow_slug, mb.messages, mb.delay) WITH ORDINALITY AS msg_ids(msg_id, msg_ord) + CROSS JOIN LATERAL pgmq.send_batch(mb.queue_name, mb.messages, mb.delay) WITH ORDINALITY AS msg_ids(msg_id, msg_ord) WHERE task_indices.idx_ord = msg_ids.msg_ord ) -- ========================================== -- PHASE 3: RECORD TASKS IN DATABASE -- ========================================== -INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, message_id) +-- The task snapshots the resolved queue; the snapshot never changes after +-- insertion (#650). +INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, queue_name, message_id) SELECT sent_messages.flow_slug, sent_messages.run_id, sent_messages.step_slug, sent_messages.task_index, + sent_messages.queue_name, sent_messages.msg_id FROM sent_messages; diff --git a/pkgs/core/schemas/0100_function_validate_flow_shape.sql b/pkgs/core/schemas/0100_function_validate_flow_shape.sql new file mode 100644 index 000000000..99ce6458f --- /dev/null +++ b/pkgs/core/schemas/0100_function_validate_flow_shape.sql @@ -0,0 +1,70 @@ +-- Complete pre-mutation validation of a compiled flow shape (#650). +-- The compiler calls this before any definition change so a late invalid +-- step name cannot leave earlier queues/definitions behind. +create or replace function pgflow._validate_flow_shape( + p_flow_slug text, + p_shape jsonb +) +returns void +language plpgsql +volatile +set search_path = '' +as $$ +declare + v_step jsonb; + v_step_slug text; + v_dep text; + v_canonical_queue text := lower(p_flow_slug); + v_conflict text; +begin + if not pgflow.is_valid_slug(p_flow_slug) then + raise exception 'Flow "%" is not a valid flow slug', p_flow_slug; + end if; + + if jsonb_typeof(p_shape) is distinct from 'object' + or jsonb_typeof(p_shape->'steps') is distinct from 'array' then + raise exception 'Flow % requires a complete steps array', p_flow_slug; + end if; + + -- Resolve all required canonical queue names before mutation: in #650 the + -- complete required route is exactly the canonical default. + if not pgflow._is_valid_queue_name(v_canonical_queue) then + raise exception 'Flow % resolves to generated queue name "%" longer than the 47-character compatibility limit or otherwise invalid', + p_flow_slug, v_canonical_queue; + end if; + + -- Every step name satisfies the shared slug rules + for v_step in select * from jsonb_array_elements(p_shape->'steps') loop + v_step_slug := v_step->>'slug'; + + if not pgflow.is_valid_slug(v_step_slug) then + raise exception 'Flow % contains invalid step slug "%"', p_flow_slug, v_step_slug; + end if; + + if jsonb_typeof(v_step->'dependencies') is distinct from 'array' then + raise exception 'Flow % step "%" requires a dependencies array', p_flow_slug, v_step_slug; + end if; + + -- Dependency names preserve exact spelling and satisfy the slug rules + for v_dep in select * from jsonb_array_elements_text(v_step->'dependencies') loop + if not pgflow.is_valid_slug(v_dep) then + raise exception 'Flow % step "%" has invalid dependency slug "%"', p_flow_slug, v_step_slug, v_dep; + end if; + end loop; + end loop; + + -- Case-only duplicate step identities inside the shape are rejected while + -- exact spelling is preserved. + select s1->>'slug' into v_conflict + from jsonb_array_elements(p_shape->'steps') s1, + jsonb_array_elements(p_shape->'steps') s2 + where lower(s1->>'slug') = lower(s2->>'slug') + and s1->>'slug' <> s2->>'slug' + limit 1; + + if v_conflict is not null then + raise exception 'Flow % contains case-only duplicate step identities (first conflict: "%")', + p_flow_slug, v_conflict; + end if; +end; +$$; diff --git a/pkgs/core/schemas/0120_function_claim_tasks.sql b/pkgs/core/schemas/0120_function_claim_tasks.sql new file mode 100644 index 000000000..c9af34502 --- /dev/null +++ b/pkgs/core/schemas/0120_function_claim_tasks.sql @@ -0,0 +1,507 @@ +-- Queue-aware task claim (#650). +-- +-- Complete-batch classification before any mutation: every read message is +-- classified by its durable (queue_name, message_id) pair and envelope +-- identity, under the ordered parent/run, step-state, task, and queue row +-- locks, so a concurrent claim cannot change a classification after it was +-- made. Claimed tasks are built exclusively from the guarded UPDATE's +-- RETURNING rows: a task this transaction did not claim is never returned. +-- With any fatal classification, no task is claimed or archived: the complete +-- read batch is reset to immediate visibility, the registered worker's HTTP +-- function is paused, and a normal fatal JSON result is returned (never a +-- SQL error after those writes). PGMQ message IDs are cast to text before +-- JSON conversion. +create or replace function pgflow.claim_tasks( + queue_name text, + flow_slug text, + message_ids bigint [], + worker_id uuid +) +returns jsonb +language plpgsql +volatile +set search_path = '' +as $$ +declare + v_qtable text := pgmq.format_table_name(queue_name, 'q'); + v_worker record; + v_flow_exists boolean; + v_route_violation text; + v_ids bigint[]; + v_bodies jsonb; + v_classification record; + v_claim_ids bigint[]; + v_defer_ids bigint[]; + v_terminal_ids bigint[]; + v_foreign_ids bigint[]; + v_fatal boolean := false; + v_errors jsonb := '[]'::jsonb; + v_warnings jsonb := '[]'::jsonb; + v_claimed_tasks jsonb; + v_body jsonb; + v_body_flow text; + v_body_run text; + v_body_step text; + v_body_index text; + v_run_valid boolean; + v_index_valid boolean; + v_reason text; + v_addr record; + v_vt_offsets int[]; + v_updated_count int; + v_claimed_count int; +begin + -- Deduplicate the read batch + select array_agg(distinct id order by id) into v_ids + from unnest(message_ids) as u(id) + where id is not null; + + if v_ids is null then + return jsonb_build_object('status', 'ok', 'tasks', '[]'::jsonb, 'warnings', '[]'::jsonb); + end if; + + -- ========================================== + -- SUBSCRIPTION VALIDATION (before body use) + -- ========================================== + select w.queue_name, w.function_name + into v_worker + from pgflow.workers w + where w.worker_id = claim_tasks.worker_id; + + if v_worker is null then + -- Missing registration supplies no invented function to pause; one + -- diagnostic per batch member keeps every message ID non-null + select coalesce(jsonb_agg( + jsonb_build_object('queue_name', queue_name, 'message_id', id::text, 'reason', 'invalid_subscription') + order by id), '[]'::jsonb) + into v_errors + from unnest(v_ids) as u(id); + perform pgflow.set_vt_batch(queue_name, v_ids, array_fill(0, array[cardinality(v_ids)])); + return jsonb_build_object('status', 'fatal', 'tasks', '[]'::jsonb, 'errors', v_errors); + end if; + + if v_worker.queue_name is distinct from queue_name then + select coalesce(jsonb_agg( + jsonb_build_object('queue_name', queue_name, 'message_id', id::text, 'reason', 'invalid_subscription') + order by id), '[]'::jsonb) + into v_errors + from unnest(v_ids) as u(id); + perform pgflow.set_vt_batch(queue_name, v_ids, array_fill(0, array[cardinality(v_ids)])); + update pgflow.worker_functions wf + set enabled = false, updated_at = clock_timestamp() + where wf.function_name = v_worker.function_name + and wf.start_mode = 'http'; + return jsonb_build_object('status', 'fatal', 'tasks', '[]'::jsonb, 'errors', v_errors); + end if; + + -- ========================================== + -- ROUTE VALIDATION + -- ========================================== + select exists(select 1 from pgflow.flows f where f.flow_slug = claim_tasks.flow_slug) + into v_flow_exists; + + select s.step_slug into v_route_violation + from pgflow.steps s + where s.flow_slug = claim_tasks.flow_slug + and s.queue_name is distinct from claim_tasks.queue_name + limit 1; + + if not v_flow_exists + or claim_tasks.queue_name is distinct from lower(claim_tasks.flow_slug) + or v_route_violation is not null then + select coalesce(jsonb_agg( + jsonb_build_object('queue_name', queue_name, 'message_id', id::text, 'reason', 'wrong_route') + order by id), '[]'::jsonb) + into v_errors + from unnest(v_ids) as u(id); + perform pgflow.set_vt_batch(queue_name, v_ids, array_fill(0, array[cardinality(v_ids)])); + update pgflow.worker_functions wf + set enabled = false, updated_at = clock_timestamp() + where wf.function_name = v_worker.function_name + and wf.start_mode = 'http'; + return jsonb_build_object('status', 'fatal', 'tasks', '[]'::jsonb, 'errors', v_errors); + end if; + + -- ========================================== + -- READ-ONLY DISCOVERY + -- ========================================== + -- Read the bodies once (ordinary SQL error if the physical table is gone). + -- Bodies are immutable in PGMQ, so reading them before the locks is safe. + -- Envelope inspection is identity classification only; application input + -- JSON is never validated here. + execute format( + 'select coalesce(jsonb_agg(jsonb_build_object(''msg_id'', q.msg_id, ''message'', q.message)), ''[]''::jsonb) + from pgmq.%I q where q.msg_id = any($1)', + v_qtable + ) into v_bodies using v_ids; + + -- ========================================== + -- ORDERED LOCKS + -- ========================================== + -- Parent runs, step states, task rows, then queue rows in message-ID + -- order: the established parent-first order shared with every other + -- runtime operation. + perform 1 + from pgflow.runs r + where r.run_id in ( + select t.run_id from pgflow.step_tasks t + where t.queue_name = claim_tasks.queue_name and t.message_id = any(v_ids) + ) + order by r.run_id + for update; + + perform 1 + from pgflow.step_states ss + where ss.run_id in ( + select t.run_id from pgflow.step_tasks t + where t.queue_name = claim_tasks.queue_name and t.message_id = any(v_ids) + ) + order by ss.run_id, ss.step_slug + for update; + + perform 1 + from pgflow.step_tasks t + where t.queue_name = claim_tasks.queue_name and t.message_id = any(v_ids) + order by t.run_id, t.step_slug, t.task_index + for update; + + execute format( + 'select q.msg_id from pgmq.%I q where q.msg_id = any($1) order by q.msg_id for update', + v_qtable + ) using v_ids; + + -- ========================================== + -- CLASSIFICATION (under the locks above) + -- ========================================== + for v_classification in + with pairs as ( + select + t.run_id, + t.step_slug, + t.task_index, + t.message_id, + t.status as task_status, + t.permanently_stalled_at, + t.started_at, + r.status as run_status, + ss.status as step_status + from pgflow.step_tasks t + left join pgflow.runs r on r.run_id = t.run_id + left join pgflow.step_states ss on ss.run_id = t.run_id and ss.step_slug = t.step_slug + where t.queue_name = claim_tasks.queue_name + and t.message_id = any(v_ids) + ) + select + u.id as msg_id, + p.run_id as task_run_id, + p.step_slug as task_step, + p.task_index as task_index, + p.task_status, + p.permanently_stalled_at, + p.started_at, + p.run_status, + p.step_status, + b.msg -> 'message' as body + from unnest(v_ids) as u(id) + left join pairs p on p.message_id = u.id + left join lateral jsonb_array_elements(v_bodies) b(msg) on (b.msg->>'msg_id')::bigint = u.id + order by u.id + loop + v_body := v_classification.body; + v_body_flow := v_body ->> 'flow_slug'; + v_body_run := v_body ->> 'run_id'; + v_body_step := v_body ->> 'step_slug'; + v_body_index := v_body ->> 'task_index'; + -- Safe-cast gates: only well-formed components may identify work + v_run_valid := v_body_run is not null + and v_body_run ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'; + v_index_valid := v_body_index is not null + and v_body_index ~ '^[0-9]{1,9}$'; + + if v_classification.task_run_id is not null then + -- ========================================== + -- EXACT DURABLE PAIR: the pair wins over the envelope. Malformed or + -- absent components never contradict it; only a VALID address that + -- positively identifies different work is fatal. + -- ========================================== + if v_body_flow is not null and v_body_flow is distinct from claim_tasks.flow_slug then + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'wrong_route'); + v_fatal := true; + elsif v_run_valid and v_body_run::uuid is distinct from v_classification.task_run_id then + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'unsupported_work'); + v_fatal := true; + elsif v_body_step is not null and v_body_step is distinct from v_classification.task_step then + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'unsupported_work'); + v_fatal := true; + elsif v_index_valid and (v_body_index)::int is distinct from v_classification.task_index then + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'unsupported_work'); + v_fatal := true; + elsif v_classification.task_status in ('completed', 'failed', 'skipped', 'cancelled') then + -- Terminal task: idempotent archive (archive ignores already-archived) + v_terminal_ids := array_append(v_terminal_ids, v_classification.msg_id); + elsif v_classification.permanently_stalled_at is not null then + -- Permanent stall: preserve status/history, archive idempotently + v_terminal_ids := array_append(v_terminal_ids, v_classification.msg_id); + elsif v_classification.task_status = 'started' + and v_classification.run_status = 'started' + and v_classification.step_status = 'started' then + v_defer_ids := array_append(v_defer_ids, v_classification.msg_id); + elsif v_classification.task_status = 'queued' + and v_classification.run_status = 'started' + and v_classification.step_status = 'started' then + v_claim_ids := array_append(v_claim_ids, v_classification.msg_id); + else + -- Active task with incompatible parent state + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'unsupported_work'); + v_fatal := true; + end if; + else + -- ========================================== + -- NO EXACT PAIR: key presence decides. Present-null identity keys are + -- pgflow-shaped evidence, not foreign silence. + -- ========================================== + if v_body is null + or not (v_body ? 'flow_slug' or v_body ? 'run_id' + or v_body ? 'step_slug' or v_body ? 'task_index') then + -- Clearly foreign: archive and warn (no bodies in diagnostics) + v_foreign_ids := array_append(v_foreign_ids, v_classification.msg_id); + v_warnings := v_warnings || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'foreign_message'); + else + -- Apparently genuine or ambiguous pgflow work without an exact pair: + -- fatal. Inspect valid address components to name the reason; a + -- malformed or null-valued component stays unsupported work. + v_reason := 'unsupported_work'; + if v_body_flow is not null and v_body_flow is distinct from claim_tasks.flow_slug then + v_reason := 'wrong_route'; + elsif v_run_valid then + select r.flow_slug into v_addr + from pgflow.runs r + where r.run_id = v_body_run::uuid; + if not found then + v_reason := 'unsupported_work'; + elsif v_addr.flow_slug is distinct from claim_tasks.flow_slug then + v_reason := 'wrong_route'; + elsif v_body_step is not null and v_index_valid then + -- A complete valid address that belongs to this flow but to a + -- task in another queue is wrong-route work + if exists ( + select 1 + from pgflow.step_tasks t + where t.run_id = v_body_run::uuid + and t.step_slug = v_body_step + and t.task_index = (v_body_index)::int + and t.queue_name is distinct from claim_tasks.queue_name + ) then + v_reason := 'wrong_route'; + end if; + end if; + end if; + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', v_reason); + v_fatal := true; + end if; + end if; + end loop; + + -- A live deferred task whose queue message disappeared is an ordinary + -- integrity/visibility failure with total rollback (#656 protection) + if not v_fatal and v_defer_ids is not null then + perform 1 + from unnest(v_defer_ids) as d(id) + where not exists ( + select 1 from jsonb_array_elements(v_bodies) b(msg) where (b.msg->>'msg_id')::bigint = d.id + ); + if found then + raise exception 'claim_tasks(): deferred live task message is missing from queue %', queue_name; + end if; + end if; + + -- ========================================== + -- FATAL BRANCH: reset the whole read batch, pause, return normally + -- ========================================== + if v_fatal then + perform pgflow.set_vt_batch( + queue_name, v_ids, + array_fill(0, array[cardinality(v_ids)]) + ); + update pgflow.worker_functions wf + set enabled = false, updated_at = clock_timestamp() + where wf.function_name = v_worker.function_name + and wf.start_mode = 'http'; + return jsonb_build_object('status', 'fatal', 'tasks', '[]'::jsonb, 'errors', v_errors); + end if; + + -- ========================================== + -- NONFATAL BRANCH (under the locks above) + -- ========================================== + -- Defer started tasks to their existing recovery deadline (effective + -- timeout + 30s from started_at); repeated reads never move that deadline + if v_defer_ids is not null then + with deadlines as ( + select + t.message_id, + greatest(0, ceil(extract(epoch from ( + t.started_at + + make_interval(secs => coalesce(s.opt_timeout, f.opt_timeout) + 30) + - clock_timestamp() + )))::integer) as vt_delay + from pgflow.step_tasks t + join pgflow.runs r on r.run_id = t.run_id + join pgflow.flows f on f.flow_slug = r.flow_slug + join pgflow.steps s on s.flow_slug = r.flow_slug and s.step_slug = t.step_slug + where t.queue_name = claim_tasks.queue_name + and t.message_id = any(v_defer_ids) + ) + select array_agg(d.vt_delay order by d.message_id) into v_vt_offsets + from (select message_id from unnest(v_defer_ids) as x(message_id)) ids + join deadlines d on d.message_id = ids.message_id; + + perform pgflow.set_vt_batch(queue_name, v_defer_ids, v_vt_offsets); + end if; + + -- Idempotent archival of terminal and clearly foreign groups after task locks + if v_terminal_ids is not null then + perform pgmq.archive(queue_name, v_terminal_ids); + end if; + if v_foreign_ids is not null then + perform pgmq.archive(queue_name, v_foreign_ids); + end if; + + -- ========================================== + -- CLAIM: guarded update; the returned tasks are built ONLY from the + -- UPDATE ... RETURNING rows, never from a re-query by message ID. + -- ========================================== + if v_claim_ids is not null then + with + updated as ( + update pgflow.step_tasks task + set + attempts_count = attempts_count + 1, + status = 'started', + started_at = now(), + last_worker_id = claim_tasks.worker_id + where task.queue_name = claim_tasks.queue_name + and task.message_id = any(v_claim_ids) + and task.status = 'queued' + returning + task.flow_slug, + task.run_id, + task.step_slug, + task.task_index, + task.queue_name, + task.message_id + ), + runs as ( + select r.run_id, r.input + from pgflow.runs r + where r.run_id in (select run_id from updated) + ), + deps as ( + select + st.run_id, + st.step_slug, + dep.dep_slug, + dep_state.output as dep_output + from updated st + join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug + join pgflow.step_states dep_state on + dep_state.run_id = st.run_id and + dep_state.step_slug = dep.dep_slug and + dep_state.status = 'completed' + ), + deps_outputs as ( + select + d.run_id, + d.step_slug, + jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output, + count(*) as dep_count + from deps d + group by d.run_id, d.step_slug + ), + timeouts as ( + select + u.message_id, + coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay + from updated u + join pgflow.flows flow on flow.flow_slug = u.flow_slug + join pgflow.steps step on step.flow_slug = u.flow_slug and step.step_slug = u.step_slug + ), + visibility_reset as ( + select pgflow.set_vt_batch( + claim_tasks.queue_name, + (select array_agg(t.message_id order by t.message_id) from timeouts t), + (select array_agg(t.vt_delay order by t.message_id) from timeouts t) + ) + ), + counts as ( + select + (select count(*) from visibility_reset) as updated_count, + (select count(*) from updated) as claimed_count + ) + select + c.updated_count, + c.claimed_count, + coalesce(( + select jsonb_agg( + jsonb_build_object( + 'flow_slug', st.flow_slug, + 'run_id', st.run_id, + 'step_slug', st.step_slug, + 'task_index', st.task_index, + 'queue_name', st.queue_name, + 'msg_id', st.message_id::text, + 'input', + case + when step.step_type = 'map' then + case + when step.deps_count = 0 then jsonb_array_element(r.input, st.task_index) + else (select jsonb_array_element(value, st.task_index) from jsonb_each(dep_out.deps_output) limit 1) + end + else coalesce(dep_out.deps_output, '{}'::jsonb) + end, + 'flow_input', + case + when step.step_type != 'map' and step.deps_count = 0 then r.input + else null + end + ) + order by st.message_id + ) + from updated st + join runs r on st.run_id = r.run_id + join pgflow.steps step on + step.flow_slug = st.flow_slug and + step.step_slug = st.step_slug + left join deps_outputs dep_out on + dep_out.run_id = st.run_id and + dep_out.step_slug = st.step_slug + ), '[]'::jsonb) + into v_updated_count, v_claimed_count, v_claimed_tasks + from counts c; + + -- Guard completeness: every task the guarded update actually claimed + -- must have its visibility extension; otherwise the whole statement + -- fails atomically (#656). A guarded update that claims fewer rows + -- than classified (e.g. a concurrent skip winning the row lock, #638) + -- simply returns only the claimed rows. + if v_updated_count is distinct from v_claimed_count then + raise exception 'claim_tasks(): visibility updated % of % claimed messages', + v_updated_count, v_claimed_count; + end if; + else + v_claimed_tasks := '[]'::jsonb; + end if; + + return jsonb_build_object( + 'status', 'ok', + 'tasks', v_claimed_tasks, + 'warnings', v_warnings + ); +end; +$$; diff --git a/pkgs/core/schemas/0120_function_start_tasks.sql b/pkgs/core/schemas/0120_function_start_tasks.sql index 5905b06c0..b7aa9f739 100644 --- a/pkgs/core/schemas/0120_function_start_tasks.sql +++ b/pkgs/core/schemas/0120_function_start_tasks.sql @@ -1,217 +1,52 @@ +-- Compatibility wrapper for the existing plain-worker SQL claim signature +-- (#650). Resolves the exact flow's canonical default queue and delegates to +-- the queue-aware claim boundary pgflow.claim_tasks(). +-- +-- On a fatal classification this wrapper returns no tasks and emits a +-- body-free SQL warning; it cannot express the JSON fatal result and is not a +-- supported worker startup compatibility layer. The new worker calls +-- claim_tasks directly. create or replace function pgflow.start_tasks( flow_slug text, msg_ids bigint [], worker_id uuid ) returns setof pgflow.step_task_record +language plpgsql volatile -set search_path to '' -language sql +set search_path = '' as $$ - with task_candidates as ( - select - task.flow_slug, - task.run_id, - task.step_slug, - task.task_index, - task.message_id - from pgflow.step_tasks as task - join pgflow.runs r on r.run_id = task.run_id - where task.flow_slug = start_tasks.flow_slug - and task.message_id = any(msg_ids) - and task.status = 'queued' - and r.status = 'started' - and exists ( - select 1 - from pgflow.step_states ss - where ss.run_id = task.run_id - and ss.step_slug = task.step_slug - and ss.status = 'started' - ) - ), - -- Claim rows with a guarded update and return only what was actually - -- claimed. A concurrent skip can win the row lock between the candidate - -- select and this update; the status = 'queued' recheck then claims nothing, - -- so no stale candidate row must escape to the worker (#638). - tasks as ( - update pgflow.step_tasks - set - attempts_count = attempts_count + 1, - status = 'started', - started_at = now(), - last_worker_id = worker_id - from task_candidates as candidate - where step_tasks.message_id = candidate.message_id - and step_tasks.flow_slug = candidate.flow_slug - and step_tasks.status = 'queued' - returning - step_tasks.flow_slug, - step_tasks.run_id, - step_tasks.step_slug, - step_tasks.task_index, - step_tasks.message_id - ), - runs as ( - select - r.run_id, - r.input - from pgflow.runs r - where r.run_id in (select run_id from tasks) - ), - deps as ( - select - st.run_id, - st.step_slug, - dep.dep_slug, - -- Read output directly from step_states (already aggregated by writers) - dep_state.output as dep_output - from tasks st - join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug - join pgflow.step_states dep_state on - dep_state.run_id = st.run_id and - dep_state.step_slug = dep.dep_slug and - dep_state.status = 'completed' -- Only include completed deps (not skipped) - ), - deps_outputs as ( - select - d.run_id, - d.step_slug, - jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output, - count(*) as dep_count - from deps d - group by d.run_id, d.step_slug - ), - timeouts as ( - select - task.message_id, - task.flow_slug, - coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay - from tasks task - join pgflow.flows flow on flow.flow_slug = task.flow_slug - join pgflow.steps step on step.flow_slug = task.flow_slug and step.step_slug = task.step_slug - ), - -- Batch update visibility timeouts for all messages. - -- The final statement must force this CTE to run: an unreferenced SELECT - -- CTE is not guaranteed to execute, which would leave a claimed task with - -- only the shorter initial PGMQ read visibility (#656). - visibility_reset as ( - select pgflow.set_vt_batch( - start_tasks.flow_slug, - array_agg(t.message_id order by t.message_id), - array_agg(t.vt_delay order by t.message_id) - ) - from timeouts t - ), - -- Force execution of the visibility_reset CTE (same pattern as - -- requeue_stalled_tasks) and guard completeness: set_vt_batch updates - -- only queue rows it finds, so fewer returned rows than claimed tasks - -- means a visibility extension did not run (#656). SQL functions cannot - -- RAISE, so the mismatch branch casts a descriptive message to int4: - -- the cast error fails the whole statement, rolling back the task - -- transition and attempt increment, and returns nothing. - _vr as ( - select case - when updated.updated_count = claimed.claimed_count then updated.updated_count - else format( - 'start_tasks(): visibility updated %s of %s claimed messages', - updated.updated_count, - claimed.claimed_count - )::int4 - end as visibility_updates - from (select count(*) as updated_count from visibility_reset) as updated - cross join (select count(*) as claimed_count from tasks) as claimed - ) - select - st.flow_slug, - st.run_id, - st.step_slug, - -- ========================================== - -- INPUT CONSTRUCTION LOGIC - -- ========================================== - -- This nested CASE statement determines how to construct the input - -- for each task based on the step type (map vs non-map). - -- - -- The fundamental difference: - -- - Map steps: Receive RAW array elements (e.g., just 42 or "hello") - -- - Non-map steps: Receive structured objects with named keys - -- (e.g., {"run": {...}, "dependency1": {...}}) - -- ========================================== - CASE - -- -------------------- MAP STEPS -------------------- - -- Map steps process arrays element-by-element. - -- Each task receives ONE element from the array at its task_index position. - WHEN step.step_type = 'map' THEN - -- Map steps get raw array elements without any wrapper object - CASE - -- ROOT MAP: Gets array from run input - -- Example: run input = [1, 2, 3] - -- task 0 gets: 1 - -- task 1 gets: 2 - -- task 2 gets: 3 - WHEN step.deps_count = 0 THEN - -- Root map (deps_count = 0): no dependencies, reads from run input. - -- Extract the element at task_index from the run's input array. - -- Note: If run input is not an array, this will return NULL - -- and the flow will fail (validated in start_flow). - jsonb_array_element(r.input, st.task_index) +declare + v_result jsonb; + v_task jsonb; +begin + select pgflow.claim_tasks( + lower(start_tasks.flow_slug), + start_tasks.flow_slug, + start_tasks.msg_ids, + start_tasks.worker_id + ) into v_result; - -- DEPENDENT MAP: Gets array from its single dependency - -- Example: dependency output = ["a", "b", "c"] - -- task 0 gets: "a" - -- task 1 gets: "b" - -- task 2 gets: "c" - ELSE - -- Has dependencies (should be exactly 1 for map steps). - -- Extract the element at task_index from the dependency's output array. - -- - -- Why the subquery with jsonb_each? - -- - The dependency outputs a raw array: [1, 2, 3] - -- - deps_outputs aggregates it into: {"dep_name": [1, 2, 3]} - -- - We need to unwrap and get just the array value - -- - Map steps have exactly 1 dependency (enforced by add_step) - -- - So jsonb_each will return exactly 1 row - -- - We extract the 'value' which is the raw array [1, 2, 3] - -- - Then get the element at task_index from that array - (SELECT jsonb_array_element(value, st.task_index) - FROM jsonb_each(dep_out.deps_output) - LIMIT 1) - END + if v_result ->> 'status' = 'fatal' then + raise warning 'start_tasks(): fatal claim classification for flow % (no tasks started)', start_tasks.flow_slug; + return; + end if; - -- -------------------- NON-MAP STEPS -------------------- - -- Regular (non-map) steps receive dependency outputs as a structured object. - -- Root steps (no dependencies) get empty object - they access flowInput via context. - -- Dependent steps get only their dependency outputs. - ELSE - -- Non-map steps get structured input with dependency keys only - -- Example for dependent step: { - -- "step1": {"output": "from_step1"}, - -- "step2": {"output": "from_step2"} - -- } - -- Example for root step: {} - -- - -- Note: flow_input is available separately in the returned record - -- for workers to access via context.flowInput - coalesce(dep_out.deps_output, '{}'::jsonb) - END as input, - st.message_id as msg_id, - st.task_index as task_index, - -- flow_input: Original run input for worker context - -- Only included for root non-map steps to avoid data duplication. - -- Root map steps: flowInput IS the array, useless to include - -- Dependent steps: lazy load via ctx.flowInput when needed - CASE - WHEN step.step_type != 'map' AND step.deps_count = 0 - THEN r.input - ELSE NULL - END as flow_input - from tasks st - join runs r on st.run_id = r.run_id - join pgflow.steps step on - step.flow_slug = st.flow_slug and - step.step_slug = st.step_slug - left join deps_outputs dep_out on - dep_out.run_id = st.run_id and - dep_out.step_slug = st.step_slug - cross join _vr - where _vr.visibility_updates >= 0 + for v_task in select * from jsonb_array_elements(v_result -> 'tasks') + loop + return query + select + (v_task ->> 'flow_slug')::text, + (v_task ->> 'run_id')::uuid, + (v_task ->> 'step_slug')::text, + v_task -> 'input', + (v_task ->> 'msg_id')::bigint, + (v_task ->> 'task_index')::int, + case + when jsonb_typeof(v_task -> 'flow_input') is distinct from 'null' + then v_task -> 'flow_input' + else null + end; + end loop; +end; $$; diff --git a/pkgs/core/scripts/run-queue-upgrade-fixture b/pkgs/core/scripts/run-queue-upgrade-fixture new file mode 100755 index 000000000..906011138 --- /dev/null +++ b/pkgs/core/scripts/run-queue-upgrade-fixture @@ -0,0 +1,501 @@ +#!/bin/bash +set -euo pipefail + +# 0.16.0 queue upgrade fixture for the persist_queue migration (#650). +# +# Proves on freshly seeded old databases that: +# 1. the copyable audit is read-only and reports every category, +# 2. the real new worker fails startup on the old database with an +# actionable protocol mismatch and changes nothing, +# 3. the migration backfills canonical queue identity and preserves +# history/metadata spelling atomically, +# 4. every rejection scenario rolls back completely, +# 5. the locked preflight respects a 5-second lock bound under open +# producers/definition/topology work. +# +# Owns its own container and databases; never touches the Nx Supabase stack. +# Invoked by the test:upgrade:queue target (under the stack lock) and by +# run-upgrade-fixture at the end of the full pgTAP target (lock already held). + +cd "$(dirname "$0")/.." + +IMAGE="jumski/atlas-postgres-pgflow:17.6.1.054" +suffix=$(printf '%s' "$PWD" | sha256sum | cut -c1-12) +CONTAINER="pgflow-queue-upgrade-${suffix}" +BASELINE_MIGRATION="20260907082520_pgflow_remove_legacy_flow_compilation.sql" +MIGRATION=$(ls supabase/migrations/*_pgflow_persist_queue.sql) +AUDIT=queries/PRE_MIGRATION_CHECK_650.sql +STOCK_PRUNE_MD5_FILE=.nx-inputs/queue_fixture_stock_prune_md5 + +cleanup() { docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; } +trap cleanup EXIT +cleanup + +echo "queue upgrade fixture: starting postgres container" +docker run -d --name "$CONTAINER" -p 127.0.0.1::5432 "$IMAGE" >/dev/null + +for _ in $(seq 1 30); do + if docker exec "$CONTAINER" pg_isready -U postgres >/dev/null 2>&1; then + break + fi + sleep 1 +done + +port=$(docker port "$CONTAINER" 5432/tcp | awk -F: '{print $NF}' | head -1) +export PGFLOW_UPGRADE_DB_URL="postgresql://postgres:postgres@127.0.0.1:${port}/postgres" + +psql_in() { # db (stdin: heredoc or file redirect) + docker exec -i "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -q -U postgres -d "$1" +} +psql_at() { # db (unaligned output for signatures; stdin: heredoc) + docker exec -i "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -A -t -U postgres -d "$1" +} +# -c/-f invocations never read stdin; docker exec -i would wait forever on the +# never-closing stdin pipe Nx provides, so these run without -i. +psql_exec() { # db args... + docker exec "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -q -U postgres -d "$1" "${@:2}" +} +psql_exec_at() { # db args... + docker exec "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -A -t -U postgres -d "$1" "${@:2}" +} + +# Builds a fresh 0.16.0 database in ~6s (image-local baseline, fast +# migrations). pg_dump/restore cannot clone the seeded state because pgmq 1.5.1 +# marks queue tables and metadata as extension members, and CREATE DATABASE +# TEMPLATE is blocked by the pg_cron/pg_net workers attached to postgres. +kill_client_backends() { # db + docker exec "$CONTAINER" psql -X -q -U postgres -d postgres -c \ + "select pg_terminate_backend(pid) from pg_stat_activity + where datname = '$1' and pid <> pg_backend_pid() + and backend_type = 'client backend'" >/dev/null 2>&1 || true +} + +seed_old_db() { # db (postgres itself is built in place, never dropped) + if [[ "$1" != "postgres" ]]; then + kill_client_backends "$1" || true + docker exec "$CONTAINER" psql -X -q -U postgres -d postgres \ + -c "drop database if exists $1" >/dev/null 2>&1 || true + docker exec "$CONTAINER" psql -X -q -U postgres -d postgres \ + -c "create database $1" >/dev/null + fi + psql_in "$1" < atlas/supabase-baseline-schema.sql + local f reached_baseline=false + for f in supabase/migrations/*.sql; do + if [[ "$(basename "$f")" == "$(basename "$MIGRATION")" ]]; then + break + fi + psql_in "$1" < "$f" + if [[ "$(basename "$f")" == "$BASELINE_MIGRATION" ]]; then + reached_baseline=true + break + fi + done + if [[ "$reached_baseline" != true ]]; then + echo "queue upgrade fixture: baseline migration not found" >&2 + exit 1 + fi + psql_in "$1" < supabase/upgrade_queue_fixture/seed.sql + psql_in "$1" < supabase/upgrade_queue_fixture/prune_0_16_0.sql +} + +echo "queue upgrade fixture: building seeded 0.16.0 database" +seed_old_db postgres + +# Stock 0.16.0 pruning helper digest: the audit's known-stock comparison and +# the fixture assertions both use this exact literal. +STOCK_PRUNE_MD5=$(psql_exec_at postgres -c \ + "select md5(prosrc) from pg_proc p join pg_namespace n on n.oid = p.pronamespace + where n.nspname = 'pgflow' and p.proname = 'prune_data_older_than'") +if grep -q REPLACE_WITH_MEASURED_DIGEST "$AUDIT"; then + echo "queue upgrade fixture: $AUDIT still carries REPLACE_WITH_MEASURED_DIGEST." >&2 + echo " measured stock 0.16.0 md5(prosrc): $STOCK_PRUNE_MD5" >&2 + echo " paste it into v_stock_prune_md5 in $AUDIT and rerun." >&2 + exit 1 +fi +mkdir -p .nx-inputs +printf '%s' "$STOCK_PRUNE_MD5" > "$STOCK_PRUNE_MD5_FILE" + +# Clones restore the pgflow schema from this dump and recreate PGMQ queues +# through dblink: pg_cron cannot be created outside postgres +# (cron.database_name), so migrations never run inside a clone, and pgmq 1.5.1 +# marks queue tables as extension members that pg_dump skips. +docker exec "$CONTAINER" pg_dump -Fc --schema=pgflow \ + -U postgres -f /tmp/old_seed_pgflow.dump postgres + +clone_db() { # new_db: restored pgflow schema + dblink-recreated queues + echo "queue upgrade fixture: seeding clone $1" + kill_client_backends "$1" || true + docker exec "$CONTAINER" psql -X -q -U postgres -d postgres \ + -c "drop database if exists $1" >/dev/null 2>&1 || true + docker exec "$CONTAINER" psql -X -q -U postgres -d postgres \ + -c "create database $1" >/dev/null + docker exec "$CONTAINER" psql -X -q -U postgres -d "$1" \ + -c "create extension pgmq" >/dev/null + if ! docker exec "$CONTAINER" pg_restore -U postgres -d "$1" \ + --no-owner /tmp/old_seed_pgflow.dump > "$TMP/restore_$1.log" 2>&1; then + echo "queue upgrade fixture: restoring clone $1 failed" >&2 + cat "$TMP/restore_$1.log" >&2 + exit 1 + fi + docker exec -i "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -q -U postgres -d "$1" \ + <<'SQL' > "$TMP/queues_$1.log" 2>&1 || { cat "$TMP/queues_$1.log" >&2; exit 1; } +create extension if not exists dblink; +create schema if not exists realtime; +create or replace function realtime.send( + payload jsonb, event text, key text, dedupe boolean default false +) returns void language sql as $$ select $$; +do $clone_queues$ +declare + r record; + v_q text; + v_a text; +begin + for r in + select * from dblink('dbname=postgres', 'select queue_name from pgmq.meta order by 1') + as m(queue_name text) + loop + perform pgmq.create(r.queue_name); + v_q := pgmq.format_table_name(r.queue_name, 'q'); + v_a := pgmq.format_table_name(r.queue_name, 'a'); + execute format( + 'insert into pgmq.%I (msg_id, read_ct, enqueued_at, vt, message, headers) + overriding system value + select msg_id, read_ct, enqueued_at, vt, message, headers + from dblink(''dbname=postgres'', %L) + as q(msg_id bigint, read_ct int, enqueued_at timestamptz, vt timestamptz, message jsonb, headers jsonb)', + v_q, 'select msg_id, read_ct, enqueued_at, vt, message, headers from pgmq.' || v_q); + execute format( + 'insert into pgmq.%I (msg_id, read_ct, enqueued_at, vt, message, headers, archived_at) + overriding system value + select msg_id, read_ct, enqueued_at, vt, message, headers, archived_at + from dblink(''dbname=postgres'', %L) + as a(msg_id bigint, read_ct int, enqueued_at timestamptz, vt timestamptz, message jsonb, headers jsonb, archived_at timestamptz)', + v_a, 'select msg_id, read_ct, enqueued_at, vt, message, headers, archived_at from pgmq.' || v_a); + execute format( + 'select setval(pg_get_serial_sequence(''pgmq.%I'', ''msg_id''), + greatest((select coalesce(max(msg_id), 0) + 1 from pgmq.%I), 1), false)', + v_q, v_q); + end loop; +end +$clone_queues$; +SQL +} + +drop_db() { # db + kill_client_backends "$1" + docker exec "$CONTAINER" psql -X -q -U postgres -d postgres \ + -c "drop database if exists $1" >/dev/null +} + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"; cleanup' EXIT + +audit_sql="$TMP/audit_run.sql" +sed "s/REPLACE_WITH_MEASURED_DIGEST/${STOCK_PRUNE_MD5}/" "$AUDIT" > "$audit_sql" + +# Canonical state signature: pgflow rows, function surface, PGMQ metadata, +# per-queue q/a row counts, and sequence positions. +state_signature() { # db + psql_at "$1" <<'SQL' +select 'flows=' || coalesce(md5(string_agg(f::text, '|' order by f::text)), 'empty') from pgflow.flows f; +select 'steps=' || coalesce(md5(string_agg(s::text, '|' order by s::text)), 'empty') from pgflow.steps s; +select 'deps=' || coalesce(md5(string_agg(d::text, '|' order by d::text)), 'empty') from pgflow.deps d; +select 'runs=' || coalesce(md5(string_agg(r::text, '|' order by r::text)), 'empty') from pgflow.runs r; +select 'states=' || coalesce(md5(string_agg(s::text, '|' order by s::text)), 'empty') from pgflow.step_states s; +select 'tasks=' || coalesce(md5(string_agg(t::text, '|' order by t::text)), 'empty') from pgflow.step_tasks t; +select 'workers=' || coalesce(md5(string_agg(w::text, '|' order by w::text)), 'empty') from pgflow.workers w; +select 'functions=' || coalesce(md5(string_agg(w::text, '|' order by w::text)), 'empty') from pgflow.worker_functions w; +select 'meta=' || coalesce(string_agg(queue_name || ':' || created_at::text, ',' order by queue_name), '') from pgmq.meta; +select 'pgflow_funcs=' || coalesce(string_agg(p.oid::regprocedure::text, ',' order by 1), '') from pg_proc p + join pg_namespace n on n.oid = p.pronamespace where n.nspname = 'pgflow'; +select 'new_columns=' || count(*) from information_schema.columns + where table_schema = 'pgflow' and column_name = 'queue_name'; +do $sig$ +declare r record; +begin + for r in select queue_name from pgmq.meta order by 1 loop + if to_regclass(format('pgmq.%I', pgmq.format_table_name(r.queue_name, 'q'))) is not null then + execute format('select ''q_%s='' || count(*)::text from pgmq.%I', + lower(r.queue_name), pgmq.format_table_name(r.queue_name, 'q')); + else + raise notice 'q_%s=MISSING', lower(r.queue_name); + end if; + if to_regclass(format('pgmq.%I', pgmq.format_table_name(r.queue_name, 'a'))) is not null then + execute format('select ''a_%s='' || count(*)::text from pgmq.%I', + lower(r.queue_name), pgmq.format_table_name(r.queue_name, 'a')); + else + raise notice 'a_%s=MISSING', lower(r.queue_name); + end if; + if to_regclass(format('pgmq.%I', pgmq.format_table_name(r.queue_name, 'q') || '_msg_id_seq')) is not null then + execute format('select ''seq_%s='' || last_value::text from pgmq.%I', + lower(r.queue_name), pgmq.format_table_name(r.queue_name, 'q') || '_msg_id_seq'); + else + raise notice 'seq_%s=MISSING', lower(r.queue_name); + end if; + end loop; +end +$sig$; +SQL +} + +# History signature: task core columns that must survive the migration +# byte-for-byte (queue_name excluded because it is the migration's addition). +task_core_signature() { # db + psql_exec_at "$1" -c \ + "select coalesce(md5(string_agg( + t.run_id || ':' || t.step_slug || ':' || t.task_index || ':' || + coalesce(t.message_id::text, 'N') || ':' || t.status || ':' || + t.attempts_count || ':' || coalesce(t.error_message, '') || ':' || + coalesce(t.output::text, ''), + '|' order by t.run_id, t.step_slug, t.task_index)), 'empty') + from pgflow.step_tasks t" +} + +# ========================================== +# Phase 1: audit is read-only and complete +# ========================================== +echo "queue upgrade fixture: audit on clean old database (expect no errors)" +psql_at postgres < "$audit_sql" | tee "$TMP/audit_clean.log" >/dev/null +if grep -q '"severity": "error"' "$TMP/audit_clean.log"; then + echo "queue upgrade fixture: clean database reported errors" >&2 + exit 1 +fi + +echo "queue upgrade fixture: audit on corrupted old database" +clone_db queue_audit +awk '/^-- scenario: inject/{p=1;next} /^-- =+$/{p=0} p' \ + supabase/upgrade_queue_fixture/audit_assertions.sql \ + | psql_in queue_audit >/dev/null +state_signature queue_audit > "$TMP/audit_pre.sig" +psql_at queue_audit < "$audit_sql" > "$TMP/audit.log" 2>&1 +state_signature queue_audit > "$TMP/audit_post.sig" +if ! diff -u "$TMP/audit_pre.sig" "$TMP/audit_post.sig" > "$TMP/audit_sig.diff"; then + echo "queue upgrade fixture: audit mutated the database" >&2 + cat "$TMP/audit_sig.diff" >&2 + exit 1 +fi +for expect in \ + '"code": "leading_underscore"' \ + '"flow_slug": "_bad_flow"' \ + '"flow_slug": "bad_"' \ + '"flow_slug": "a__b"' \ + '"code": "case_conflict_flow"' \ + '"code": "ambiguous_metadata"' \ + '"code": "unmatched_active_message"' \ + '"code": "malformed_queue_objects"' \ + '"code": "pruning_helper_stock"' \ + '"code": "backfill_overview"'; do + if ! grep -qF "$expect" "$TMP/audit.log"; then + echo "queue upgrade fixture: audit log missing $expect" >&2 + exit 1 + fi +done +if ! grep -qF 'archive table is missing its headers column' "$TMP/audit.log"; then + echo "queue upgrade fixture: audit log missing the missing-archive-column report" >&2 + exit 1 +fi +null_keys=$(grep '"flow_slug": "many_null"' "$TMP/audit.log" | grep -o -e '->NULL' | wc -l) +if [[ "$null_keys" -ne 20 ]]; then + echo "queue upgrade fixture: expected exactly 20 sampled NULL keys, got $null_keys" >&2 + exit 1 +fi +if grep -q 'app-token-XYZ-3f9' "$TMP/audit.log"; then + echo "queue upgrade fixture: audit exposed unrelated queue contents" >&2 + exit 1 +fi +psql_in queue_audit <<'SQL' >/dev/null +do $$ +begin + if exists (select 1 from pgflow.step_tasks t + join pgflow.runs r on r.run_id = t.run_id + where r.flow_slug is distinct from t.flow_slug) then + raise exception 'fixture broken: ownership mismatch expected in audit db'; + end if; +end $$; +SQL +# Post-audit SQL assertions from the fixture file (everything after the +# scenario section). +awk '/^-- =+$/{n++; next} n >= 1' \ + supabase/upgrade_queue_fixture/audit_assertions.sql \ + | psql_in queue_audit >/dev/null +drop_db queue_audit + +# ========================================== +# Phase 2: real new worker against the old database +# ========================================== +echo "queue upgrade fixture: startup probe (protocol mismatch required)" +if ! command -v deno >/dev/null 2>&1; then + echo "queue upgrade fixture: deno not found on PATH" >&2 + exit 1 +fi +deno run --config ../edge-worker/deno.test.json --allow-all \ + ../edge-worker/tests/integration/upgrade/startup_probe.ts + +# ========================================== +# Phase 3: migration succeeds on the populated old database +# ========================================== +echo "queue upgrade fixture: applying $(basename "$MIGRATION") to the seeded database" +clone_db main_upgraded +task_core_signature main_upgraded > "$TMP/task_core_old.txt" +psql_in main_upgraded < "$MIGRATION" +task_core_signature main_upgraded > "$TMP/task_core_new.txt" +if ! diff -u "$TMP/task_core_old.txt" "$TMP/task_core_new.txt" > "$TMP/task_core.diff"; then + echo "queue upgrade fixture: task history changed across the migration" >&2 + cat "$TMP/task_core.diff" >&2 + exit 1 +fi + +echo "queue upgrade fixture: post-migration assertions" +psql_in main_upgraded < supabase/upgrade_queue_fixture/assertions.sql + +installed_md5=$(psql_exec_at main_upgraded -c \ + "select md5(prosrc) from pg_proc p join pg_namespace n on n.oid = p.pronamespace + where n.nspname = 'pgflow' and p.proname = 'prune_data_older_than'") +if [[ "$installed_md5" != "$STOCK_PRUNE_MD5" ]]; then + echo "queue upgrade fixture: migration overwrote the installed pruning helper" >&2 + exit 1 +fi + +echo "queue upgrade fixture: explicit pruning helper replacement and execution" +psql_in main_upgraded < supabase/tests/_shared/prune_data_older_than.sql.raw +psql_exec main_upgraded -c "select pgflow.prune_data_older_than(interval '1 second')" >/dev/null +psql_in main_upgraded <<'SQL' >/dev/null +do $$ +begin + if exists (select 1 from pgflow.runs r where r.flow_slug = 'billing') then + raise exception 'prune did not remove expired old runs; runs now: %', + (select string_agg( + flow_slug || ':' || status || ':c=' || coalesce(completed_at::text, '-') || + ':f=' || coalesce(failed_at::text, '-'), '; ') + from pgflow.runs); + end if; + if exists (select 1 from pgflow.step_tasks t where t.flow_slug = 'billing') then + raise exception 'prune left task rows of expired runs'; + end if; + if not exists (select 1 from pgmq.q_app_events) then + raise exception 'prune touched an unrelated application queue'; + end if; + + -- A run started after pruning still executes on the snapshot-based schema. + perform pgflow.start_flow('fresh', '"fresh-2"'::jsonb); + if not exists ( + select 1 from pgflow.step_tasks + where flow_slug = 'fresh' and queue_name = 'fresh' and status = 'queued' + ) then + raise exception 'post-prune run did not create a canonical task'; + end if; +end $$; +SQL + +# ========================================== +# Phase 4: customized helper is reported and never overwritten +# ========================================== +echo "queue upgrade fixture: customized pruning helper scenario" +clone_db queue_custom +# The marker comment lands inside the function body so md5(prosrc) differs +# from the stock digest while behavior stays identical. +sed 's|cutoff_timestamp TIMESTAMPTZ := now() - retention_interval;|cutoff_timestamp TIMESTAMPTZ := now() - retention_interval; -- customized marker|' \ + supabase/upgrade_queue_fixture/prune_0_16_0.sql > "$TMP/prune_custom.sql" +psql_in queue_custom < "$TMP/prune_custom.sql" +psql_at queue_custom < "$audit_sql" > "$TMP/audit_custom.log" 2>&1 +if ! grep -qF '"code": "pruning_helper_customized"' "$TMP/audit_custom.log"; then + echo "queue upgrade fixture: audit missed the customized helper" >&2 + exit 1 +fi +custom_before=$(psql_exec_at queue_custom -c \ + "select md5(prosrc) from pg_proc p join pg_namespace n on n.oid = p.pronamespace + where n.nspname = 'pgflow' and p.proname = 'prune_data_older_than'") +psql_in queue_custom < "$MIGRATION" +custom_after=$(psql_exec_at queue_custom -c \ + "select md5(prosrc) from pg_proc p join pg_namespace n on n.oid = p.pronamespace + where n.nspname = 'pgflow' and p.proname = 'prune_data_older_than'") +if [[ "$custom_after" != "$custom_before" ]]; then + echo "queue upgrade fixture: migration overwrote the customized helper" >&2 + exit 1 +fi +drop_db queue_custom + +# ========================================== +# Phase 5: every rejection scenario rolls back completely +# ========================================== +scenarios=$(awk '/^-- scenario: /{print $3}' supabase/upgrade_queue_fixture/rejections.sql) +for scenario in $scenarios; do + echo "queue upgrade fixture: rejection scenario $scenario" + clone_db "reject_${scenario}" + awk -v s="$scenario" '/^-- scenario: /{p = ($3 == s); next} p' \ + supabase/upgrade_queue_fixture/rejections.sql \ + | psql_in "reject_${scenario}" >/dev/null + state_signature "reject_${scenario}" > "$TMP/${scenario}_pre.sig" + if psql_in "reject_${scenario}" < "$MIGRATION" > "$TMP/${scenario}_mig.log" 2>&1; then + echo "queue upgrade fixture: scenario $scenario unexpectedly migrated" >&2 + exit 1 + fi + state_signature "reject_${scenario}" > "$TMP/${scenario}_post.sig" + if ! diff -u "$TMP/${scenario}_pre.sig" "$TMP/${scenario}_post.sig" > "$TMP/${scenario}.diff"; then + echo "queue upgrade fixture: scenario $scenario left changes behind" >&2 + cat "$TMP/${scenario}.diff" >&2 + exit 1 + fi + drop_db "reject_${scenario}" +done + +# ========================================== +# Phase 6: locked preflight under open writers (5-second bound) +# ========================================== +blockers=$(awk '/^-- blocker: /{print $3}' supabase/upgrade_queue_fixture/concurrency.sql) +for blocker in $blockers; do + echo "queue upgrade fixture: concurrency blocker $blocker" + clone_db conc_target + awk -v s="$blocker" '/^-- blocker: /{p = ($3 == s); next} p' \ + supabase/upgrade_queue_fixture/concurrency.sql > "$TMP/blocker_${blocker}.sql" + docker cp "$TMP/blocker_${blocker}.sql" "$CONTAINER:/tmp/blocker.sql" >/dev/null + docker exec -d -e PGAPPNAME=queue_fixture_blocker \ + "$CONTAINER" psql -X -q -U postgres -d conc_target -f /tmp/blocker.sql + + held=false + for _ in $(seq 1 30); do + n=$(psql_exec_at postgres -c \ + "select count(*) from pg_stat_activity + where datname = 'conc_target' and application_name = 'queue_fixture_blocker' + and xact_start is not null") + if [[ "$n" -ge 1 ]]; then + held=true + break + fi + sleep 0.2 + done + if [[ "$held" != true ]]; then + echo "queue upgrade fixture: blocker $blocker never opened its transaction" >&2 + exit 1 + fi + + state_signature conc_target > "$TMP/conc_${blocker}_pre.sig" + if timeout 60 docker exec -i "$CONTAINER" psql -v ON_ERROR_STOP=1 -X -q -U postgres -d conc_target < "$MIGRATION" > "$TMP/conc_${blocker}_mig.log" 2>&1; then + echo "queue upgrade fixture: migration succeeded while $blocker held locks" >&2 + exit 1 + fi + grep -q 'lock_timeout\|deadlock detected\|canceling statement' "$TMP/conc_${blocker}_mig.log" || { + echo "queue upgrade fixture: $blocker failure was not a bounded lock wait" >&2 + cat "$TMP/conc_${blocker}_mig.log" >&2 + exit 1 + } + state_signature conc_target > "$TMP/conc_${blocker}_post.sig" + if ! diff -u "$TMP/conc_${blocker}_pre.sig" "$TMP/conc_${blocker}_post.sig" > "$TMP/conc_${blocker}.diff"; then + echo "queue upgrade fixture: $blocker rejection left changes behind" >&2 + cat "$TMP/conc_${blocker}.diff" >&2 + exit 1 + fi + docker exec "$CONTAINER" psql -X -q -U postgres -d postgres -c \ + "select pg_terminate_backend(pid) from pg_stat_activity + where application_name = 'queue_fixture_blocker'" >/dev/null + sleep 0.5 +done + +echo "queue upgrade fixture: migration succeeds with no blockers" +if ! psql_in conc_target < "$MIGRATION"; then + echo "queue upgrade fixture: fenced migration failed without blockers" >&2 + exit 1 +fi +drop_db conc_target + +echo "queue upgrade fixture: PASS" diff --git a/pkgs/core/scripts/run-upgrade-fixture b/pkgs/core/scripts/run-upgrade-fixture index 78efa55f9..dcaebe38a 100755 --- a/pkgs/core/scripts/run-upgrade-fixture +++ b/pkgs/core/scripts/run-upgrade-fixture @@ -65,4 +65,10 @@ echo "upgrade fixture: applying consolidated migration $(basename "$consolidated psql_in < "$consolidated" psql_in < supabase/upgrade_fixture/assertions.sql + echo "upgrade fixture: PASS" + +# The 0.16.0 queue upgrade fixture runs only after this fixture passes. It +# owns its own container and databases; it must not reacquire this +# nonreentrant stack lock (the lock is already held here). +scripts/run-queue-upgrade-fixture diff --git a/pkgs/core/src/PgflowSqlClient.ts b/pkgs/core/src/PgflowSqlClient.ts index e32e701b1..95a53db7f 100644 --- a/pkgs/core/src/PgflowSqlClient.ts +++ b/pkgs/core/src/PgflowSqlClient.ts @@ -1,10 +1,11 @@ import type postgres from 'postgres'; import type { - StepTaskRecord, IPgflowClient, StepTaskKey, RunRow, MessageRecord, + ClaimTasksResult, + MessageId, } from './types.js'; import type { Json } from './types.js'; import type { AnyFlow, ExtractFlowInput } from '@pgflow/dsl'; @@ -24,8 +25,10 @@ export class PgflowSqlClient maxPollSeconds = 5, pollIntervalMs = 200 ): Promise { + // msg_id is projected to text so PGMQ bigint IDs cross the JavaScript + // boundary without precision loss (#650) return await this.sql` - SELECT * + SELECT msg_id::text as msg_id, read_ct, enqueued_at, vt, message, headers FROM pgmq.read_with_poll( queue_name => ${queueName}, vt => ${visibilityTimeout}, @@ -37,18 +40,20 @@ export class PgflowSqlClient } async startTasks( + queueName: string, flowSlug: string, - msgIds: number[], + messageIds: MessageId[], workerId: string - ): Promise[]> { - return await this.sql[]>` - SELECT * - FROM pgflow.start_tasks( + ): Promise> { + const [row] = await this.sql<{ result: ClaimTasksResult }[]>` + select pgflow.claim_tasks( + queue_name => ${queueName}, flow_slug => ${flowSlug}, - msg_ids => ${msgIds}::bigint[], + message_ids => ${messageIds}::bigint[], worker_id => ${workerId}::uuid - ); + ) as result `; + return row.result; } async completeTask(stepTask: StepTaskKey, output?: Json): Promise { diff --git a/pkgs/core/src/database-types.ts b/pkgs/core/src/database-types.ts index 393ef0b2c..96d8316eb 100644 --- a/pkgs/core/src/database-types.ts +++ b/pkgs/core/src/database-types.ts @@ -208,6 +208,7 @@ export type Database = { message_id: number | null output: Json | null permanently_stalled_at: string | null + queue_name: string queued_at: string requeued_count: number run_id: string @@ -227,6 +228,7 @@ export type Database = { message_id?: number | null output?: Json | null permanently_stalled_at?: string | null + queue_name: string queued_at?: string requeued_count?: number run_id: string @@ -246,6 +248,7 @@ export type Database = { message_id?: number | null output?: Json | null permanently_stalled_at?: string | null + queue_name?: string queued_at?: string requeued_count?: number run_id?: string @@ -295,6 +298,7 @@ export type Database = { opt_max_attempts: number | null opt_start_delay: number | null opt_timeout: number | null + queue_name: string required_input_pattern: Json | null step_index: number step_slug: string @@ -311,6 +315,7 @@ export type Database = { opt_max_attempts?: number | null opt_start_delay?: number | null opt_timeout?: number | null + queue_name: string required_input_pattern?: Json | null step_index?: number step_slug: string @@ -327,6 +332,7 @@ export type Database = { opt_max_attempts?: number | null opt_start_delay?: number | null opt_timeout?: number | null + queue_name?: string required_input_pattern?: Json | null step_index?: number step_slug?: string @@ -425,7 +431,24 @@ export type Database = { Args: { p_flow_slug: string; p_shape: Json } Returns: undefined } + _ensure_generated_queue: { + Args: { p_flow_slug: string; p_queue_name: string } + Returns: undefined + } _get_flow_shape: { Args: { p_flow_slug: string }; Returns: Json } + _inspect_generated_queue: { + Args: { + p_flow_slug: string + p_queue_name: string + p_require_existing: boolean + } + Returns: Json + } + _is_valid_queue_name: { Args: { queue_name: string }; Returns: boolean } + _validate_flow_shape: { + Args: { p_flow_slug: string; p_shape: Json } + Returns: undefined + } add_step: { Args: { base_delay?: number @@ -433,6 +456,7 @@ export type Database = { flow_slug: string forbidden_input_pattern?: Json max_attempts?: number + queue_name?: string required_input_pattern?: Json start_delay?: number step_slug: string @@ -450,6 +474,7 @@ export type Database = { opt_max_attempts: number | null opt_start_delay: number | null opt_timeout: number | null + queue_name: string required_input_pattern: Json | null step_index: number step_slug: string @@ -473,6 +498,15 @@ export type Database = { Returns: number } cascade_resolve_conditions: { Args: { run_id: string }; Returns: boolean } + claim_tasks: { + Args: { + flow_slug: string + message_ids: number[] + queue_name: string + worker_id: string + } + Returns: Json + } cleanup_ensure_workers_logs: { Args: { retention_hours?: number } Returns: { @@ -497,6 +531,7 @@ export type Database = { message_id: number | null output: Json | null permanently_stalled_at: string | null + queue_name: string queued_at: string requeued_count: number run_id: string @@ -538,7 +573,7 @@ export type Database = { Returns: undefined } ensure_flow_compiled: { - Args: { flow_slug: string; shape: Json } + Args: { flow_slug: string; shape: Json; worker_protocol: Json } Returns: Json } ensure_workers: { @@ -567,6 +602,7 @@ export type Database = { message_id: number | null output: Json | null permanently_stalled_at: string | null + queue_name: string queued_at: string requeued_count: number run_id: string diff --git a/pkgs/core/src/types.ts b/pkgs/core/src/types.ts index 5789dbd8c..085aea177 100644 --- a/pkgs/core/src/types.ts +++ b/pkgs/core/src/types.ts @@ -10,6 +10,29 @@ import type { Database } from './database-types.js'; export type { Json }; +/** + * PGMQ message id: a SQL bigint, always delivered to JavaScript as an exact + * decimal string through explicit SQL text projection (#650). + */ +export type MessageId = string; + +/** + * Diagnostic row returned by claim_tasks for messages that were skipped, + * archived, or rejected during a claim batch (#650). One row per message; + * no message bodies ever appear here. + */ +export type ClaimDiagnosticReason = + | 'foreign_message' + | 'unsupported_work' + | 'wrong_route' + | 'invalid_subscription'; + +export type ClaimDiagnostic = { + queue_name: string; + message_id: MessageId; + reason: ClaimDiagnosticReason; +}; + /** * Record representing a task from pgflow.start_tasks * @@ -27,8 +50,9 @@ export type StepTaskRecord = { run_id: string; step_slug: StepSlug; task_index: number; + queue_name: string; input: Simplify>; - msg_id: number; + msg_id: MessageId; flow_input: ExtractFlowInput | null; }; }[Extract, string>]; @@ -41,11 +65,21 @@ export type StepTaskKey = Pick, 'run_id' | 'step_slug' | +/** + * Result of pgflow.claim_tasks (#650): one committed outcome for the whole + * read batch. `ok` carries the claimed tasks and body-free warnings for + * archived members; `fatal` carries no tasks and the reasons the batch was + * rejected - the SQL side already reset visibility and paused the worker. + */ +export type ClaimTasksResult = + | { status: 'ok'; tasks: StepTaskRecord[]; warnings: ClaimDiagnostic[] } + | { status: 'fatal'; tasks: []; errors: ClaimDiagnostic[] }; + /** * Record representing a message from queue polling */ export type MessageRecord = { - msg_id: number; + msg_id: MessageId; read_ct: number; enqueued_at: string; vt: string; @@ -82,16 +116,21 @@ export interface IPgflowClient { ): Promise; /** - * Starts tasks for given message IDs (phase 2 of two-phase approach) - * @param flowSlug - The flow slug to start tasks from + * Claims tasks for given message IDs via pgflow.claim_tasks (phase 2 of + * two-phase approach). Returns started tasks plus claim diagnostics; a + * `fatal` status means the SQL side reset visibility and paused the + * worker, so the caller must stop (#650). + * @param queueName - Name of the queue the messages were read from + * @param flowSlug - The flow slug to claim tasks from * @param msgIds - Array of message IDs from readMessages - * @param workerId - ID of the worker starting the tasks + * @param workerId - ID of the worker claiming the tasks */ startTasks( + queueName: string, flowSlug: string, - msgIds: number[], + msgIds: MessageId[], workerId: string - ): Promise[]>; + ): Promise>; /** * Mark a task as completed with output @@ -130,6 +169,11 @@ export type RunRow = Database['pgflow']['Tables']['runs']['Row']; export type StepStateRow = Database['pgflow']['Tables']['step_states']['Row']; /** - * Record representing a step from pgflow.step_tasks + * Record representing a step from pgflow.step_tasks. The generated type + * labels the PGMQ bigint message ID as `number`; the runtime value arrives + * as a decimal string through explicit SQL text projection (#650). */ -export type StepTaskRow = Database['pgflow']['Tables']['step_tasks']['Row']; +export type StepTaskRow = Omit< + Database['pgflow']['Tables']['step_tasks']['Row'], + 'message_id' +> & { message_id: MessageId | null }; diff --git a/pkgs/core/supabase/migrations/20260910104929_pgflow_persist_queue.sql b/pkgs/core/supabase/migrations/20260910104929_pgflow_persist_queue.sql new file mode 100644 index 000000000..05fa823be --- /dev/null +++ b/pkgs/core/supabase/migrations/20260910104929_pgflow_persist_queue.sql @@ -0,0 +1,3668 @@ +-- The Supabase migration runner does not wrap files in a transaction, so this +-- migration wraps its own body (plan Task 9: prove both runners). The offline +-- upgrade fixture applies this file WITHOUT --single-transaction for the same +-- reason. SET LOCAL/LOCK TABLE below require this wrapper. +BEGIN; +-- Migration-only locked preflight (#650). Hand-authored per the approved plan; +-- Atlas cannot infer this section. Rejects every audit category atomically +-- before any structural change. Offline writer fence is assumed outside. +SET LOCAL lock_timeout = '5s'; +LOCK TABLE pgflow.flows IN ACCESS EXCLUSIVE MODE; +LOCK TABLE pgflow.steps IN ACCESS EXCLUSIVE MODE; +LOCK TABLE pgflow.deps IN ACCESS EXCLUSIVE MODE; +LOCK TABLE pgflow.runs IN ACCESS EXCLUSIVE MODE; +LOCK TABLE pgflow.step_states IN ACCESS EXCLUSIVE MODE; +LOCK TABLE pgflow.step_tasks IN ACCESS EXCLUSIVE MODE; +LOCK TABLE pgflow.workers IN ACCESS EXCLUSIVE MODE; +LOCK TABLE pgflow.worker_functions IN ACCESS EXCLUSIVE MODE; +LOCK TABLE pgmq.meta IN SHARE ROW EXCLUSIVE MODE; +DO $preflight$ +declare + v_queue text; + v_meta_count int; + v_metadata_name text; + v_meta_row pgmq.meta%ROWTYPE; + v_qtable text; + v_atable text; + v_seq text; + v_q_oid oid; + v_a_oid oid; + v_seq_oid oid; + v_ext_oid oid; + v_bad text; + v_detail text; + v_count bigint; +begin + -- Lock each validated candidate's physical objects in canonical queue + -- order (q then a) and recheck metadata resolution under those locks. + for v_queue in select distinct lower(f.flow_slug) as q from pgflow.flows f order by 1 + loop + if length(v_queue) > 47 or v_queue !~ '^[a-z][a-z0-9_]*$' then + raise exception 'Migration preflight: canonical queue name "%" is not a valid generated queue name (lowercase, at most 47 characters, starting with a letter); rename this flow manually before upgrade', v_queue; + end if; + + select count(*), min(m.queue_name) into v_meta_count, v_metadata_name + from pgmq.meta m + where lower(m.queue_name) = v_queue; + if v_meta_count <> 1 then + raise exception 'Migration preflight: canonical queue "%" has % pgmq metadata rows (expected exactly one); resolve exact metadata spelling manually before upgrade', v_queue, v_meta_count; + end if; + + v_qtable := pgmq.format_table_name(v_queue, 'q'); + v_atable := pgmq.format_table_name(v_queue, 'a'); + v_seq := v_qtable || '_msg_id_seq'; + if to_regclass('pgmq.' || v_qtable) is null + or to_regclass('pgmq.' || v_atable) is null + or to_regclass('pgmq.' || v_seq) is null then + raise exception 'Migration preflight: queue "%" (metadata "%") is missing physical q/a/sequence objects; the migration neither reconstructs nor drops resources', v_queue, v_metadata_name; + end if; + + execute format('lock table pgmq.%I, pgmq.%I in access exclusive mode', v_qtable, v_atable); + + -- Complete physical shape, index, sequence-dependency, and + -- extension-membership contract, rechecked under the table locks: the + -- exact per-column contract of _inspect_generated_queue + -- (0070_functions_generated_queues.sql), including explicit + -- missing-column rejection for every queue and archive column. + select * into v_meta_row from pgmq.meta m where lower(m.queue_name) = v_queue; + v_q_oid := to_regclass(format('pgmq.%I', v_qtable)); + v_a_oid := to_regclass(format('pgmq.%I', v_atable)); + v_seq_oid := to_regclass(format('pgmq.%I', v_seq)); + select e.oid into v_ext_oid from pg_extension e where e.extname = 'pgmq'; + + select problem into v_bad from ( + select 'queue table %s is not an ordinary permanent table' as problem + from pg_class c + where c.oid = v_q_oid and (c.relkind <> 'r' or c.relpersistence <> 'p') + union all + select 'archive table %s is not an ordinary permanent table' + from pg_class c + where c.oid = v_a_oid and (c.relkind <> 'r' or c.relpersistence <> 'p') + union all + select 'queue table %s: msg_id must be a non-null bigint generated-always identity' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'msg_id' + and (a.atttypid <> 'int8'::regtype or not a.attnotnull or a.attidentity <> 'a') + union all + select 'queue table %s: missing msg_id bigint identity column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'msg_id' and a.attnum > 0) + union all + select 'queue table %s: msg_id has no single-column primary key' + where not exists ( + select 1 from pg_index i + where i.indrelid = v_q_oid and i.indisprimary and i.indisvalid + and i.indnkeyatts = 1 + and i.indkey[0] = (select a.attnum from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'msg_id') + ) + union all + select 'queue table %s: read_ct must be a non-null integer' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'read_ct' + and (a.atttypid <> 'int4'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing read_ct column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'read_ct' and a.attnum > 0) + union all + select 'queue table %s: enqueued_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'enqueued_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing enqueued_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'enqueued_at' and a.attnum > 0) + union all + select 'queue table %s: vt must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'vt' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing vt column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'vt' and a.attnum > 0) + union all + select 'queue table %s: message must be jsonb' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'message' + and a.atttypid <> 'jsonb'::regtype + union all + select 'queue table %s: missing message column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'message' and a.attnum > 0) + union all + select 'queue table %s: headers must be jsonb' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'headers' + and a.atttypid <> 'jsonb'::regtype + union all + select 'queue table %s: missing headers column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'headers' and a.attnum > 0) + union all + select 'queue table %s has no valid usable single-column index on vt' + where not exists ( + select 1 + from pg_index i + join pg_attribute a on a.attrelid = i.indrelid and a.attnum = i.indkey[0] + where i.indrelid = v_q_oid and i.indisvalid and i.indisready + and i.indpred is null and i.indexprs is null and i.indnkeyatts = 1 + and a.attname = 'vt' + ) + union all + select 'archive table %s: msg_id must be a non-null bigint primary key without identity generator' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'msg_id' + and (a.atttypid <> 'int8'::regtype or not a.attnotnull or a.attidentity <> '') + union all + select 'archive table %s: missing msg_id bigint column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'msg_id' and a.attnum > 0) + union all + select 'archive table %s: msg_id has no single-column primary key' + where not exists ( + select 1 from pg_index i + where i.indrelid = v_a_oid and i.indisprimary and i.indisvalid + and i.indnkeyatts = 1 + and i.indkey[0] = (select a.attnum from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'msg_id') + ) + union all + select 'archive table %s: read_ct must be a non-null integer' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'read_ct' + and (a.atttypid <> 'int4'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing read_ct column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'read_ct' and a.attnum > 0) + union all + select 'archive table %s: enqueued_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'enqueued_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing enqueued_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'enqueued_at' and a.attnum > 0) + union all + select 'archive table %s: archived_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'archived_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing archived_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'archived_at' and a.attnum > 0) + union all + select 'archive table %s: vt must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'vt' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing vt column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'vt' and a.attnum > 0) + union all + select 'archive table %s: message must be jsonb' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'message' + and a.atttypid <> 'jsonb'::regtype + union all + select 'archive table %s: missing message column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'message' and a.attnum > 0) + union all + select 'archive table %s: headers must be jsonb' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'headers' + and a.atttypid <> 'jsonb'::regtype + union all + select 'archive table %s: missing headers column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'headers' and a.attnum > 0) + union all + select 'archive table %s has no valid usable single-column index on archived_at' + where not exists ( + select 1 + from pg_index i + join pg_attribute a on a.attrelid = i.indrelid and a.attnum = i.indkey[0] + where i.indrelid = v_a_oid and i.indisvalid and i.indisready + and i.indpred is null and i.indexprs is null and i.indnkeyatts = 1 + and a.attname = 'archived_at' + ) + union all + select 'sequence %s must be a bigint sequence' + from pg_sequence s + where s.seqrelid = v_seq_oid + and s.seqtypid <> 'int8'::regtype + union all + select 'sequence %s is missing' + where not exists (select 1 from pg_sequence s where s.seqrelid = v_seq_oid) + union all + select 'sequence %s is not associated with queue msg_id' + where not exists ( + select 1 + from pg_depend d + join pg_attribute a + on a.attrelid = d.refobjid and a.attnum = d.refobjsubid + where d.objid = v_seq_oid + and d.refobjid = v_q_oid + and a.attname = 'msg_id' + and d.deptype in ('i', 'a') + ) + union all + select 'queue %s metadata flags disagree with physical shape (partitioned/unlogged)' + where v_meta_row.is_partitioned or v_meta_row.is_unlogged + union all + select 'queue %s objects are not members of the installed pgmq extension' + where v_ext_oid is not null and ( + not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_q_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) or not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_a_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) or not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_seq_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) + ) + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Migration preflight: queue "%" (metadata "%") failed physical/dependency/extension inspection: %', + v_queue, v_metadata_name, format(v_bad, v_queue); + end if; + end loop; + + -- New slug rules (leading/trailing underscore, double underscore) on every + -- existing definition; the stricter is_valid_slug does not revalidate rows. + for v_detail in + select 'flow "' || f.flow_slug || '": leading underscore not allowed' + from pgflow.flows f where left(f.flow_slug, 1) = '_' + union all + select 'flow "' || f.flow_slug || '": trailing underscore not allowed' + from pgflow.flows f where right(f.flow_slug, 1) = '_' + union all + select 'flow "' || f.flow_slug || '": double underscore not allowed' + from pgflow.flows f where position('__' in f.flow_slug) > 0 + union all + select 'flow "' || s.flow_slug || '" step "' || s.step_slug || '": leading underscore not allowed' + from pgflow.steps s where left(s.step_slug, 1) = '_' + union all + select 'flow "' || s.flow_slug || '" step "' || s.step_slug || '": trailing underscore not allowed' + from pgflow.steps s where right(s.step_slug, 1) = '_' + union all + select 'flow "' || s.flow_slug || '" step "' || s.step_slug || '": double underscore not allowed' + from pgflow.steps s where position('__' in s.step_slug) > 0 + loop + raise exception 'Migration preflight: incompatible definition %; resolve the exact name manually before upgrade (no automatic rename)', v_detail; + end loop; + + -- Case-only aliases share a canonical queue and are rejected atomically. + select string_agg(f.flow_slug, ', ' order by f.flow_slug) into v_detail + from pgflow.flows f + group by lower(f.flow_slug) + having count(*) > 1; + if v_detail is not null then + raise exception 'Migration preflight: case-only flow alias group [%] shares one canonical queue; resolve exact spelling manually (no automatic rename)', v_detail; + end if; + + select string_agg(s.flow_slug || '/' || s.step_slug, ', ' order by s.flow_slug, s.step_slug) into v_detail + from pgflow.steps s + group by s.flow_slug, lower(s.step_slug) + having count(*) > 1; + if v_detail is not null then + raise exception 'Migration preflight: case-only step alias group [%]; resolve exact spelling manually (no automatic rename)', v_detail; + end if; + + -- Prospective duplicate (queue, message) pairs; backfill must not collide. + select string_agg(lower(t.flow_slug) || '#' || t.message_id::text, ', ') into v_detail + from pgflow.step_tasks t + where t.message_id is not null + group by lower(t.flow_slug), t.message_id + having count(*) > 1; + if v_detail is not null then + raise exception 'Migration preflight: distinct old tasks share a prospective queue/message pair [%]; resolve manually (no automatic merge)', v_detail; + end if; + + -- Denormalized runtime ownership must be consistent before backfill. + select count(*) into v_count + from pgflow.step_tasks t + join pgflow.runs r on r.run_id = t.run_id + where r.flow_slug is distinct from t.flow_slug; + if v_count > 0 then + raise exception 'Migration preflight: % step_tasks rows disagree with their run''s flow_slug; resolve denormalized ownership manually', v_count; + end if; + + -- Full runtime ownership: every run references an existing flow + -- definition and every task references an existing step of its flow. + select count(*) into v_count + from pgflow.runs r + where not exists ( + select 1 from pgflow.flows f where f.flow_slug = r.flow_slug + ); + if v_count > 0 then + raise exception 'Migration preflight: % runs reference a flow definition that does not exist; resolve orphaned runs manually', v_count; + end if; + + select string_agg(t.flow_slug || '/' || t.step_slug, ', ' order by t.flow_slug, t.step_slug) into v_detail + from ( + select distinct t.flow_slug, t.step_slug + from pgflow.step_tasks t + where not exists ( + select 1 from pgflow.steps s + where s.flow_slug = t.flow_slug and s.step_slug = t.step_slug + ) + limit 20 + ) t; + if v_detail is not null then + raise exception 'Migration preflight: step_tasks reference steps that do not exist in their flow [%]; resolve orphaned task rows manually', v_detail; + end if; + + -- Active queue rows without an exact durable task identity, including + -- non-visible messages (vt > now() does not exempt them). + for v_queue in select distinct lower(f.flow_slug) as q from pgflow.flows f order by 1 + loop + v_qtable := pgmq.format_table_name(v_queue, 'q'); + execute format( + 'select count(*) from pgmq.%I q + where not exists ( + select 1 from pgflow.step_tasks t + where lower(t.flow_slug) = $1 and t.message_id = q.msg_id + )', v_qtable) + into v_count using v_queue; + if v_count > 0 then + raise exception 'Migration preflight: queue "%" holds % active message(s) without a matching exact task identity; resolve orphan messages manually before upgrade', v_queue, v_count; + end if; + + -- Matched messages whose envelopes identify different work than their + -- durable task identity. Only valid (safe-castable) contradicting + -- components count; malformed components are left to runtime claim + -- classification, where the durable pair wins. + execute format( + 'select count(*) from pgmq.%I q + join pgflow.step_tasks t + on lower(t.flow_slug) = $1 and t.message_id = q.msg_id + where (q.message ->> ''flow_slug'') is not null + and (q.message ->> ''flow_slug'') is distinct from t.flow_slug + or ((q.message ->> ''run_id'') ~* ''^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'' + and (q.message ->> ''run_id'')::uuid is distinct from t.run_id) + or (q.message ->> ''step_slug'') is not null + and (q.message ->> ''step_slug'') is distinct from t.step_slug + or ((q.message ->> ''task_index'') ~ ''^[0-9]{1,9}$'' + and (q.message ->> ''task_index'')::int is distinct from t.task_index)', + v_qtable) + into v_count using v_queue; + if v_count > 0 then + raise exception 'Migration preflight: queue "%" holds % matched active message(s) whose envelope identifies different work than the task row; resolve envelope corruption manually before upgrade', v_queue, v_count; + end if; + end loop; +end +$preflight$; +-- Create index "idx_flows_slug_lower" to table: "flows" +CREATE UNIQUE INDEX "idx_flows_slug_lower" ON "pgflow"."flows" ((lower(flow_slug))); +-- Create "_is_valid_queue_name" function +CREATE FUNCTION "pgflow"."_is_valid_queue_name" ("queue_name" text) RETURNS boolean LANGUAGE sql IMMUTABLE PARALLEL SAFE SET "search_path" = '' AS $$ +select + queue_name is not null + and queue_name <> '' + and length(queue_name) <= 47 + and queue_name ~ '^[a-z][a-z0-9_]*$' +$$; +-- Drop index "idx_step_tasks_message_id" from table: "step_tasks" +DROP INDEX "pgflow"."idx_step_tasks_message_id"; +-- Drop index "idx_step_tasks_queued_msg" from table: "step_tasks" +DROP INDEX "pgflow"."idx_step_tasks_queued_msg"; +-- Migration-only backfill (#650): between nullable column addition and +-- enforcement, exactly as approved in the plan. The immutability trigger +-- below intentionally does not exist yet during this UPDATE. +ALTER TABLE "pgflow"."step_tasks" ADD COLUMN "queue_name" text; +ALTER TABLE "pgflow"."steps" ADD COLUMN "queue_name" text; +UPDATE pgflow.steps SET queue_name = lower(flow_slug); +UPDATE pgflow.step_tasks SET queue_name = lower(flow_slug); +ALTER TABLE "pgflow"."step_tasks" ALTER COLUMN "queue_name" SET NOT NULL; +ALTER TABLE "pgflow"."steps" ALTER COLUMN "queue_name" SET NOT NULL; +ALTER TABLE "pgflow"."step_tasks" ADD CONSTRAINT "queue_name_is_valid" CHECK (pgflow._is_valid_queue_name(queue_name)); +ALTER TABLE "pgflow"."steps" ADD CONSTRAINT "queue_name_is_valid" CHECK (pgflow._is_valid_queue_name(queue_name)); +-- Create index "idx_step_tasks_queue_message" to table: "step_tasks" +CREATE UNIQUE INDEX "idx_step_tasks_queue_message" ON "pgflow"."step_tasks" ("queue_name", "message_id") WHERE (message_id IS NOT NULL); +-- Create "_keep_task_queue_name" function +CREATE FUNCTION "pgflow"."_keep_task_queue_name" () RETURNS trigger LANGUAGE plpgsql SET "search_path" = '' AS $$ +begin + if exists ( + select run_id, step_slug, task_index, queue_name from old_tasks + except + select run_id, step_slug, task_index, queue_name from new_tasks + ) then + raise exception 'step_tasks.queue_name is immutable'; + end if; + return null; +end; +$$; +-- Create trigger "keep_task_queue_name" +CREATE TRIGGER "keep_task_queue_name" AFTER UPDATE ON "pgflow"."step_tasks" REFERENCING OLD TABLE AS "old_tasks" NEW TABLE AS "new_tasks" FOR EACH STATEMENT EXECUTE FUNCTION "pgflow"."_keep_task_queue_name"(); +-- Create index "idx_steps_slug_lower" to table: "steps" +CREATE UNIQUE INDEX "idx_steps_slug_lower" ON "pgflow"."steps" ("flow_slug", (lower(step_slug))); +-- Modify "_archive_task_message" function +CREATE OR REPLACE FUNCTION "pgflow"."_archive_task_message" ("p_run_id" uuid, "p_step_slug" text, "p_task_index" integer) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_batch record; +begin + PERFORM 1 FROM pgflow.runs r + WHERE r.run_id = p_run_id + FOR UPDATE; + + PERFORM 1 FROM pgflow.step_states ss + WHERE ss.run_id = p_run_id + AND ss.step_slug = p_step_slug + FOR UPDATE; + + FOR v_batch IN + WITH locked_tasks AS ( + SELECT task.queue_name, task.message_id + FROM pgflow.step_tasks task + WHERE task.run_id = p_run_id + AND task.step_slug = p_step_slug + AND task.task_index = p_task_index + AND task.message_id IS NOT NULL + ORDER BY task.task_index + FOR UPDATE + ) + SELECT + lt.queue_name, + ARRAY_AGG(lt.message_id ORDER BY lt.message_id) AS ids + FROM locked_tasks lt + GROUP BY lt.queue_name + LOOP + PERFORM pgmq.archive(v_batch.queue_name, v_batch.ids); + END LOOP; +END; +$$; +-- Modify "_cascade_force_skip_steps" function +CREATE OR REPLACE FUNCTION "pgflow"."_cascade_force_skip_steps" ("run_id" uuid, "step_slug" text, "skip_reason" text) RETURNS integer LANGUAGE plpgsql AS $$ +DECLARE + v_flow_slug text; + v_total_skipped int := 0; +BEGIN + -- Lock the parent run at direct entry before any run-mutating work; + -- callers that already hold the run lock re-acquire it harmlessly. + SELECT r.flow_slug INTO v_flow_slug + FROM pgflow.runs r + WHERE r.run_id = _cascade_force_skip_steps.run_id + FOR UPDATE; + + IF v_flow_slug IS NULL THEN + RAISE EXCEPTION 'Run not found: %', _cascade_force_skip_steps.run_id; + END IF; + + -- ========================================== + -- SKIP STEPS IN TOPOLOGICAL ORDER + -- ========================================== + -- Use recursive CTE to find all downstream dependents, + -- then skip them in topological order (by step_index) + WITH RECURSIVE + -- ---------- Find all downstream steps ---------- + downstream_steps AS ( + -- Base case: the trigger step + SELECT + s.flow_slug, + s.step_slug, + s.step_index, + _cascade_force_skip_steps.skip_reason AS reason -- Original reason for trigger step + FROM pgflow.steps s + WHERE s.flow_slug = v_flow_slug + AND s.step_slug = _cascade_force_skip_steps.step_slug + + UNION ALL + + -- Recursive case: steps that depend on already-found steps + SELECT + s.flow_slug, + s.step_slug, + s.step_index, + 'dependency_skipped'::text AS reason -- Downstream steps get this reason + FROM pgflow.steps s + JOIN pgflow.deps d ON d.flow_slug = s.flow_slug AND d.step_slug = s.step_slug + JOIN downstream_steps ds ON ds.flow_slug = d.flow_slug AND ds.step_slug = d.dep_slug + ), + -- ---------- Deduplicate and order by step_index ---------- + steps_to_skip AS ( + SELECT DISTINCT ON (ds.step_slug) + ds.flow_slug, + ds.step_slug, + ds.step_index, + ds.reason + FROM downstream_steps ds + ORDER BY ds.step_slug, ds.step_index -- Keep first occurrence (trigger step has original reason) + ), + -- ---------- Skip the steps ---------- + skipped AS ( + UPDATE pgflow.step_states ss + SET status = 'skipped', + skip_reason = sts.reason, + skipped_at = now(), + remaining_tasks = NULL -- Clear remaining_tasks for skipped steps + FROM steps_to_skip sts + WHERE ss.run_id = _cascade_force_skip_steps.run_id + AND ss.step_slug = sts.step_slug + AND ss.status IN ('created', 'started') -- Only skip non-terminal steps + RETURNING + ss.*, + -- Broadcast step:skipped event + realtime.send( + jsonb_build_object( + 'event_type', 'step:skipped', + 'run_id', ss.run_id, + 'flow_slug', ss.flow_slug, + 'step_slug', ss.step_slug, + 'status', 'skipped', + 'skip_reason', ss.skip_reason, + 'skipped_at', ss.skipped_at + ), + concat('step:', ss.step_slug, ':skipped'), + concat('pgflow:run:', ss.run_id), + false + ) as _broadcast_result + ), + -- ---------- Terminalize active tasks of newly skipped steps ---------- + skipped_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'skipped' + WHERE task.run_id = _cascade_force_skip_steps.run_id + AND task.step_slug IN ( + SELECT skipped_step.step_slug + FROM skipped AS skipped_step + ) + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ), + -- ---------- Archive queued/started task messages for skipped steps ---------- + -- Grouped by the task's queue snapshot; only newly skipped steps' tasks are + -- archived (preexisting skipped steps were already archived) (#650) + archived_messages AS ( + SELECT pgmq.archive(st.queue_name, ARRAY_AGG(st.message_id)) as result + FROM skipped_tasks AS st + WHERE st.message_id IS NOT NULL + GROUP BY st.queue_name + HAVING COUNT(st.message_id) > 0 + ), + -- ---------- Update run counters ---------- + run_updates AS ( + UPDATE pgflow.runs r + SET remaining_steps = r.remaining_steps - skipped_count.count + FROM (SELECT COUNT(*) AS count FROM skipped) skipped_count + WHERE r.run_id = _cascade_force_skip_steps.run_id + AND skipped_count.count > 0 + ) + SELECT skipped_count.count + INTO v_total_skipped + FROM (SELECT COUNT(*) AS count FROM skipped) skipped_count + LEFT JOIN archived_messages ON true; + + RETURN v_total_skipped; +END; +$$; +-- Create "_inspect_generated_queue" function +CREATE FUNCTION "pgflow"."_inspect_generated_queue" ("p_flow_slug" text, "p_queue_name" text, "p_require_existing" boolean) RETURNS jsonb LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_canonical_queue text := lower(p_flow_slug); + v_qtable text; + v_atable text; + v_sequence text; + v_meta_count int; + v_metadata_name text; + v_meta_row pgmq.meta%ROWTYPE; + v_flow_exists boolean; + v_other_flow text; + v_routed_step text; + v_other_route text; + v_q_oid oid; + v_a_oid oid; + v_seq_oid oid; + v_ext_oid oid; + v_bad text; +begin + if not pgflow._is_valid_queue_name(p_queue_name) then + raise exception 'Flow %: "%" is not a valid generated queue name (lowercase, at most 47 characters, starting with a letter)', + p_flow_slug, p_queue_name; + end if; + + if p_queue_name <> v_canonical_queue then + raise exception 'Flow %: queue "%" is not its canonical generated route "%" (custom routes do not exist in #650)', + p_flow_slug, p_queue_name, v_canonical_queue; + end if; + + select exists(select 1 from pgflow.flows f where f.flow_slug = p_flow_slug) + into v_flow_exists; + + -- Topology fence: serialize against external PGMQ create/drop on this + -- namespace (PGMQ 1.5.1 inserts metadata only after creating objects). All + -- ownership checks and physical inspection happen after the fence so a + -- wait cannot invalidate them. + lock table pgmq.meta in share row exclusive mode; + + -- Ownership evidence: no other flow may derive, persist, or reference this route + select f.flow_slug into v_other_flow + from pgflow.flows f + where lower(f.flow_slug) = p_queue_name + and f.flow_slug <> p_flow_slug + limit 1; + if v_other_flow is not null then + raise exception 'Generated queue "%" for flow % collides with the derived route of flow %', + p_queue_name, p_flow_slug, v_other_flow; + end if; + + select s.flow_slug into v_other_flow + from pgflow.steps s + where s.queue_name = p_queue_name + and s.flow_slug <> p_flow_slug + limit 1; + if v_other_flow is not null then + raise exception 'Generated queue "%" for flow % is persisted as a route of flow %', + p_queue_name, p_flow_slug, v_other_flow; + end if; + + select t.flow_slug into v_other_flow + from pgflow.step_tasks t + where t.queue_name = p_queue_name + and t.flow_slug <> p_flow_slug + limit 1; + if v_other_flow is not null then + raise exception 'Generated queue "%" for flow % is referenced by tasks of flow %', + p_queue_name, p_flow_slug, v_other_flow; + end if; + + -- The current flow's persisted route must actually be this queue: a step + -- persisting a different route is an invalid definition, and verifying or + -- deleting this queue while such a step exists would bypass it. + select s.step_slug, s.queue_name into v_routed_step, v_other_route + from pgflow.steps s + where s.flow_slug = p_flow_slug + and s.queue_name is distinct from p_queue_name + limit 1; + if v_routed_step is not null then + raise exception 'Flow %: step "%" persists route "%" instead of the generated queue "%"; the definition is invalid and no queue operation may proceed', + p_flow_slug, v_routed_step, v_other_route, p_queue_name; + end if; + + select count(*), min(m.queue_name) into v_meta_count, v_metadata_name + from pgmq.meta as m + where lower(m.queue_name) = p_queue_name; + + if v_meta_count > 1 then + raise exception 'Generated queue "%" for flow % has ambiguous PGMQ metadata (% rows share the name case-insensitively)', + p_queue_name, p_flow_slug, v_meta_count; + end if; + + v_qtable := pgmq.format_table_name(p_queue_name, 'q'); + v_atable := pgmq.format_table_name(p_queue_name, 'a'); + v_sequence := v_qtable || '_msg_id_seq'; + + v_q_oid := to_regclass(format('pgmq.%I', v_qtable)); + v_a_oid := to_regclass(format('pgmq.%I', v_atable)); + v_seq_oid := to_regclass(format('pgmq.%I', v_sequence)); + + if v_meta_count = 0 and v_q_oid is null and v_a_oid is null and v_seq_oid is null then + -- Only genuinely absent when zero metadata AND zero objects + if p_require_existing then + raise exception 'Flow %: generated queue "%" is missing (no PGMQ metadata, no objects)', + p_flow_slug, p_queue_name; + end if; + return jsonb_build_object('state', 'absent'); + end if; + + -- Some metadata or physical evidence exists: the concrete definition must + -- own it. A missing definition plus any resource is a collision. + if not v_flow_exists then + raise exception 'Generated queue "%" exists (metadata: %, queue table: %, archive table: %, sequence: %) but flow % has no definition; refusing to adopt external resources', + p_queue_name, v_meta_count, v_qtable, v_atable, v_sequence, p_flow_slug; + end if; + + if v_meta_count = 0 then + raise exception 'Flow %: generated queue "%" has physical objects without PGMQ metadata (queue table: %, archive table: %, sequence: %)', + p_flow_slug, p_queue_name, v_qtable, v_atable, v_sequence; + end if; + + if v_q_oid is null or v_a_oid is null or v_seq_oid is null then + raise exception 'Flow %: generated queue "%" has incomplete PGMQ objects (queue table: %, archive table: %, sequence: %)', + p_flow_slug, p_queue_name, v_qtable, v_atable, v_sequence; + end if; + + select * into v_meta_row from pgmq.meta m where lower(m.queue_name) = p_queue_name; + + -- ========================================== + -- QUEUE TABLE CONTRACT + -- ========================================== + select reason into v_bad from ( + select 'queue table %s is not an ordinary permanent table' as reason + from pg_class c + where c.oid = v_q_oid + and (c.relkind <> 'r' or c.relpersistence <> 'p') + union all + select 'queue table %s: msg_id must be a non-null bigint generated-always identity primary key' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'msg_id' + and (a.atttypid <> 'int8'::regtype or not a.attnotnull or a.attidentity <> 'a') + union all + select 'queue table %s: missing msg_id bigint identity column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'msg_id' and a.attnum > 0) + union all + select 'queue table %s: msg_id has no single-column primary key' + where not exists ( + select 1 from pg_index i + where i.indrelid = v_q_oid and i.indisprimary and i.indisvalid + and i.indnkeyatts = 1 + and i.indkey[0] = (select a.attnum from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'msg_id') + ) + union all + select 'queue table %s: read_ct must be a non-null integer' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'read_ct' + and (a.atttypid <> 'int4'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing read_ct column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'read_ct' and a.attnum > 0) + union all + select 'queue table %s: enqueued_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'enqueued_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing enqueued_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'enqueued_at' and a.attnum > 0) + union all + select 'queue table %s: vt must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'vt' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'queue table %s: missing vt column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'vt' and a.attnum > 0) + union all + select 'queue table %s: message must be jsonb' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'message' + and a.atttypid <> 'jsonb'::regtype + union all + select 'queue table %s: missing message column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'message' and a.attnum > 0) + union all + select 'queue table %s: headers must be jsonb' + from pg_attribute a + where a.attrelid = v_q_oid and a.attname = 'headers' + and a.atttypid <> 'jsonb'::regtype + union all + select 'queue table %s: missing headers column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_q_oid and a.attname = 'headers' and a.attnum > 0) + union all + select 'queue table %s has no valid usable single-column index on vt' + where not exists ( + select 1 + from pg_index i + join pg_attribute a on a.attrelid = i.indrelid and a.attnum = i.indkey[0] + where i.indrelid = v_q_oid + and i.indisvalid + and i.indisready + and i.indpred is null + and i.indexprs is null + and i.indnkeyatts = 1 + and a.attname = 'vt' + ) + union all + select 'queue table %s metadata flags disagree with physical shape (partitioned/unlogged)' + where v_meta_row.is_partitioned or v_meta_row.is_unlogged + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Flow %: generated queue "%" failed physical inspection: %', + p_flow_slug, p_queue_name, format(v_bad, v_qtable); + end if; + + -- ========================================== + -- ARCHIVE TABLE CONTRACT + -- ========================================== + select reason into v_bad from ( + select 'archive table %s is not an ordinary permanent table' as reason + from pg_class c + where c.oid = v_a_oid + and (c.relkind <> 'r' or c.relpersistence <> 'p') + union all + select 'archive table %s: msg_id must be a non-null bigint primary key without identity generator' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'msg_id' + and (a.atttypid <> 'int8'::regtype or not a.attnotnull or a.attidentity <> '') + union all + select 'archive table %s: missing msg_id bigint column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'msg_id' and a.attnum > 0) + union all + select 'archive table %s: msg_id has no single-column primary key' + where not exists ( + select 1 from pg_index i + where i.indrelid = v_a_oid and i.indisprimary and i.indisvalid + and i.indnkeyatts = 1 + and i.indkey[0] = (select a.attnum from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'msg_id') + ) + union all + select 'archive table %s: read_ct must be a non-null integer' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'read_ct' + and (a.atttypid <> 'int4'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing read_ct column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'read_ct' and a.attnum > 0) + union all + select 'archive table %s: enqueued_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'enqueued_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing enqueued_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'enqueued_at' and a.attnum > 0) + union all + select 'archive table %s: archived_at must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'archived_at' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing archived_at column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'archived_at' and a.attnum > 0) + union all + select 'archive table %s: vt must be a non-null timestamptz' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'vt' + and (a.atttypid <> 'timestamptz'::regtype or not a.attnotnull) + union all + select 'archive table %s: missing vt column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'vt' and a.attnum > 0) + union all + select 'archive table %s: message must be jsonb' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'message' + and a.atttypid <> 'jsonb'::regtype + union all + select 'archive table %s: missing message column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'message' and a.attnum > 0) + union all + select 'archive table %s: headers must be jsonb' + from pg_attribute a + where a.attrelid = v_a_oid and a.attname = 'headers' + and a.atttypid <> 'jsonb'::regtype + union all + select 'archive table %s: missing headers column' + where not exists (select 1 from pg_attribute a where a.attrelid = v_a_oid and a.attname = 'headers' and a.attnum > 0) + union all + select 'archive table %s has no valid usable single-column index on archived_at' + where not exists ( + select 1 + from pg_index i + join pg_attribute a on a.attrelid = i.indrelid and a.attnum = i.indkey[0] + where i.indrelid = v_a_oid + and i.indisvalid + and i.indisready + and i.indpred is null + and i.indexprs is null + and i.indnkeyatts = 1 + and a.attname = 'archived_at' + ) + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Flow %: generated queue "%" failed physical inspection: %', + p_flow_slug, p_queue_name, format(v_bad, v_atable); + end if; + + -- ========================================== + -- SEQUENCE CONTRACT + -- ========================================== + select reason into v_bad from ( + select 'sequence %s must be a bigint sequence' as reason + from pg_sequence s + where s.seqrelid = v_seq_oid + and s.seqtypid <> 'int8'::regtype + union all + select 'sequence %s is missing' + where not exists (select 1 from pg_sequence s where s.seqrelid = v_seq_oid) + union all + select 'sequence %s is not associated with queue msg_id' + where not exists ( + select 1 + from pg_depend d + join pg_attribute a + on a.attrelid = d.refobjid and a.attnum = d.refobjsubid + where d.objid = v_seq_oid + and d.refobjid = v_q_oid + and a.attname = 'msg_id' + and d.deptype in ('i', 'a') + ) + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Flow %: generated queue "%" failed physical inspection: %', + p_flow_slug, p_queue_name, format(v_bad, v_sequence); + end if; + + -- ========================================== + -- EXTENSION MEMBERSHIP CONTRACT + -- ========================================== + -- When pgmq is installed as an extension, PGMQ's own create/drop path + -- marks the q/a tables and the identity sequence as extension members + -- (pg_depend deptype 'e'). Objects without that membership were created + -- outside PGMQ's implementation and must not be treated as owned + -- generated queues. + select e.oid into v_ext_oid from pg_extension e where e.extname = 'pgmq'; + if v_ext_oid is not null then + select reason into v_bad from ( + select 'queue table %s is not a member of the installed pgmq extension' as reason + where not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_q_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) + union all + select 'archive table %s is not a member of the installed pgmq extension' + where not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_a_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) + union all + select 'sequence %s is not a member of the installed pgmq extension' + where not exists ( + select 1 from pg_depend d + where d.classid = 'pg_class'::regclass and d.objid = v_seq_oid and d.objsubid = 0 + and d.refclassid = 'pg_extension'::regclass + and d.refobjid = v_ext_oid and d.deptype = 'e' + ) + ) problems + limit 1; + + if v_bad is not null then + raise exception 'Flow %: generated queue "%" failed extension-membership inspection: %', + p_flow_slug, p_queue_name, format(v_bad, v_qtable); + end if; + end if; + + return jsonb_build_object('state', 'present', 'metadata_name', v_metadata_name); +end; +$$; +-- Modify "is_valid_slug" function +CREATE OR REPLACE FUNCTION "pgflow"."is_valid_slug" ("slug" text) RETURNS boolean LANGUAGE plpgsql IMMUTABLE SET "search_path" = '' AS $$ +begin + return + slug is not null + and slug <> '' + and length(slug) <= 128 + and slug ~ '^[a-zA-Z_][a-zA-Z0-9_]*$' + and left(slug, 1) <> '_' + and right(slug, 1) <> '_' + and position('__' in slug) = 0 + and slug NOT IN ('run'); -- reserved words +end; +$$; +-- Create "_ensure_generated_queue" function +CREATE FUNCTION "pgflow"."_ensure_generated_queue" ("p_flow_slug" text, "p_queue_name" text) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_state jsonb; +begin + perform pg_advisory_xact_lock(1, hashtext(lower(p_flow_slug))); + + if not exists (select 1 from pgflow.flows f where f.flow_slug = p_flow_slug for update) then + raise exception 'Flow % does not exist; cannot provision generated queue "%"', + p_flow_slug, p_queue_name; + end if; + + v_state := pgflow._inspect_generated_queue(p_flow_slug, p_queue_name, false); + + if v_state ->> 'state' = 'absent' then + -- A live definition or task snapshot that still references this route + -- means the physical queue was lost; recreating it would hide that loss. + if exists ( + select 1 + from pgflow.steps s + where s.flow_slug = p_flow_slug and s.queue_name = p_queue_name + ) or exists ( + select 1 + from pgflow.step_tasks t + where t.flow_slug = p_flow_slug and t.queue_name = p_queue_name + ) then + raise exception 'Flow %: generated queue "%" is absent but existing steps/tasks reference it; refusing to reconstruct a lost live queue', + p_flow_slug, p_queue_name; + end if; + + perform pgmq.create(p_queue_name); + end if; + + -- Post-create verification under the same fence and locks. + perform pgflow._inspect_generated_queue(p_flow_slug, p_queue_name, true); +end; +$$; +-- Create "add_step" function +CREATE FUNCTION "pgflow"."add_step" ("flow_slug" text, "step_slug" text, "deps_slugs" text[] DEFAULT '{}', "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer, "start_delay" integer DEFAULT NULL::integer, "step_type" text DEFAULT 'single', "required_input_pattern" jsonb DEFAULT NULL::jsonb, "forbidden_input_pattern" jsonb DEFAULT NULL::jsonb, "when_unmet" text DEFAULT 'skip', "when_exhausted" text DEFAULT 'fail', "queue_name" text DEFAULT NULL::text) RETURNS "pgflow"."steps" LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + result_step pgflow.steps; + next_idx int; + v_queue_name text; + v_alias_slug text; +BEGIN + -- Canonical flow lock shared with compilation and deletion + PERFORM pg_advisory_xact_lock(1, hashtext(lower(add_step.flow_slug))); + + -- Lock the exact concrete flow row before calculating the next step index + IF NOT EXISTS ( + SELECT 1 FROM pgflow.flows f + WHERE f.flow_slug = add_step.flow_slug + FOR UPDATE + ) THEN + RAISE EXCEPTION 'Flow % does not exist', add_step.flow_slug; + END IF; + + -- Resolve the route: omitted defaults to the canonical generated queue; + -- an explicit equal canonical route is accepted, a different route is not. + v_queue_name := COALESCE(add_step.queue_name, lower(add_step.flow_slug)); + IF add_step.queue_name IS NOT NULL AND add_step.queue_name <> lower(add_step.flow_slug) THEN + RAISE EXCEPTION 'Flow %: step "%" cannot use queue "%" (custom routes do not exist in #650; the canonical route is "%")', + add_step.flow_slug, add_step.step_slug, add_step.queue_name, lower(add_step.flow_slug); + END IF; + + -- Validate the step slug and map constraints before any provisioning + IF NOT pgflow.is_valid_slug(add_step.step_slug) THEN + RAISE EXCEPTION 'Flow %: "%" is not a valid step slug', add_step.flow_slug, add_step.step_slug; + END IF; + + IF COALESCE(add_step.step_type, 'single') = 'map' AND COALESCE(array_length(add_step.deps_slugs, 1), 0) > 1 THEN + RAISE EXCEPTION 'Map step "%" can have at most one dependency, but % were provided: %', + add_step.step_slug, + COALESCE(array_length(add_step.deps_slugs, 1), 0), + array_to_string(add_step.deps_slugs, ', '); + END IF; + + -- Dependencies must reference existing steps of this exact flow + PERFORM 1 + FROM unnest(COALESCE(add_step.deps_slugs, '{}')) AS d(dep_slug) + WHERE NOT EXISTS ( + SELECT 1 FROM pgflow.steps s + WHERE s.flow_slug = add_step.flow_slug + AND s.step_slug = d.dep_slug + ); + IF FOUND THEN + RAISE EXCEPTION 'Flow %: step "%" has a dependency that does not exist', add_step.flow_slug, add_step.step_slug; + END IF; + + -- Case-alias precheck before the unique index; names both spellings + SELECT s.step_slug INTO v_alias_slug + FROM pgflow.steps s + WHERE s.flow_slug = add_step.flow_slug + AND lower(s.step_slug) = lower(add_step.step_slug) + AND s.step_slug <> add_step.step_slug + LIMIT 1; + IF v_alias_slug IS NOT NULL THEN + RAISE SQLSTATE '23505' USING MESSAGE = format( + 'Flow %s: step "%s" conflicts with existing step "%s" (case-insensitive step namespace)', + add_step.flow_slug, add_step.step_slug, v_alias_slug + ); + END IF; + + -- Provision/verify the generated queue through the shared ownership path + PERFORM pgflow._ensure_generated_queue(add_step.flow_slug, v_queue_name); + + -- Get next step index (under the flow row lock) + SELECT COALESCE(MAX(s.step_index) + 1, 0) INTO next_idx + FROM pgflow.steps s + WHERE s.flow_slug = add_step.flow_slug; + + -- Create the step with its resolved route snapshot + INSERT INTO pgflow.steps ( + flow_slug, step_slug, step_type, step_index, deps_count, queue_name, + opt_max_attempts, opt_base_delay, opt_timeout, opt_start_delay, + required_input_pattern, forbidden_input_pattern, when_unmet, when_exhausted + ) + VALUES ( + add_step.flow_slug, + add_step.step_slug, + COALESCE(add_step.step_type, 'single'), + next_idx, + COALESCE(array_length(add_step.deps_slugs, 1), 0), + v_queue_name, + add_step.max_attempts, + add_step.base_delay, + add_step.timeout, + add_step.start_delay, + add_step.required_input_pattern, + add_step.forbidden_input_pattern, + add_step.when_unmet, + add_step.when_exhausted + ) + ON CONFLICT ON CONSTRAINT steps_pkey + DO UPDATE SET step_slug = EXCLUDED.step_slug + RETURNING * INTO result_step; + + -- Insert dependencies + INSERT INTO pgflow.deps (flow_slug, dep_slug, step_slug) + SELECT add_step.flow_slug, d.dep_slug, add_step.step_slug + FROM unnest(COALESCE(add_step.deps_slugs, '{}')) AS d(dep_slug) + WHERE add_step.deps_slugs IS NOT NULL AND array_length(add_step.deps_slugs, 1) > 0 + ON CONFLICT ON CONSTRAINT deps_pkey DO NOTHING; + + RETURN result_step; +END; +$$; +-- Modify "create_flow" function +CREATE OR REPLACE FUNCTION "pgflow"."create_flow" ("flow_slug" text, "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer) RETURNS "pgflow"."flows" LANGUAGE plpgsql SET "search_path" = '' AS $$ +#variable_conflict use_column +declare + result_flow pgflow.flows; + v_alias_slug text; +begin + -- Canonical advisory lock: case aliases share the lock so concurrent + -- compilation cannot create both spellings. + perform pg_advisory_xact_lock(1, hashtext(lower(create_flow.flow_slug))); + + -- Precheck case aliases before the unique index does; the message names + -- both exact spellings. + select f.flow_slug into v_alias_slug + from pgflow.flows f + where lower(f.flow_slug) = lower(create_flow.flow_slug) + and f.flow_slug <> create_flow.flow_slug + limit 1; + + if v_alias_slug is not null then + raise sqlstate '23505' using message = format( + 'Flow "%s" conflicts with existing flow "%s" (case-insensitive flow namespace)', + create_flow.flow_slug, v_alias_slug + ); + end if; + + if not exists (select 1 from pgflow.flows f where f.flow_slug = create_flow.flow_slug) then + -- New identity: the canonical generated route must be genuinely absent so + -- create_flow cannot launder an external queue into apparent ownership + -- before add_step() runs. The inspection takes the pgmq.meta fence. + perform pgflow._inspect_generated_queue( + create_flow.flow_slug, + lower(create_flow.flow_slug), + false + ); + end if; + + insert into pgflow.flows as flow (flow_slug, opt_max_attempts, opt_base_delay, opt_timeout) + values ( + create_flow.flow_slug, + coalesce(max_attempts, 3), + coalesce(base_delay, 5), + coalesce(timeout, 60) + ) + on conflict (flow_slug) do update + set flow_slug = flow.flow_slug -- Dummy update: idempotent + returning * into result_flow; + + return result_flow; +end; +$$; +-- Create "_validate_flow_shape" function +CREATE FUNCTION "pgflow"."_validate_flow_shape" ("p_flow_slug" text, "p_shape" jsonb) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_step jsonb; + v_step_slug text; + v_dep text; + v_canonical_queue text := lower(p_flow_slug); + v_conflict text; +begin + if not pgflow.is_valid_slug(p_flow_slug) then + raise exception 'Flow "%" is not a valid flow slug', p_flow_slug; + end if; + + if jsonb_typeof(p_shape) is distinct from 'object' + or jsonb_typeof(p_shape->'steps') is distinct from 'array' then + raise exception 'Flow % requires a complete steps array', p_flow_slug; + end if; + + -- Resolve all required canonical queue names before mutation: in #650 the + -- complete required route is exactly the canonical default. + if not pgflow._is_valid_queue_name(v_canonical_queue) then + raise exception 'Flow % resolves to generated queue name "%" longer than the 47-character compatibility limit or otherwise invalid', + p_flow_slug, v_canonical_queue; + end if; + + -- Every step name satisfies the shared slug rules + for v_step in select * from jsonb_array_elements(p_shape->'steps') loop + v_step_slug := v_step->>'slug'; + + if not pgflow.is_valid_slug(v_step_slug) then + raise exception 'Flow % contains invalid step slug "%"', p_flow_slug, v_step_slug; + end if; + + if jsonb_typeof(v_step->'dependencies') is distinct from 'array' then + raise exception 'Flow % step "%" requires a dependencies array', p_flow_slug, v_step_slug; + end if; + + -- Dependency names preserve exact spelling and satisfy the slug rules + for v_dep in select * from jsonb_array_elements_text(v_step->'dependencies') loop + if not pgflow.is_valid_slug(v_dep) then + raise exception 'Flow % step "%" has invalid dependency slug "%"', p_flow_slug, v_step_slug, v_dep; + end if; + end loop; + end loop; + + -- Case-only duplicate step identities inside the shape are rejected while + -- exact spelling is preserved. + select s1->>'slug' into v_conflict + from jsonb_array_elements(p_shape->'steps') s1, + jsonb_array_elements(p_shape->'steps') s2 + where lower(s1->>'slug') = lower(s2->>'slug') + and s1->>'slug' <> s2->>'slug' + limit 1; + + if v_conflict is not null then + raise exception 'Flow % contains case-only duplicate step identities (first conflict: "%")', + p_flow_slug, v_conflict; + end if; +end; +$$; +-- Modify "_create_flow_from_shape" function +CREATE OR REPLACE FUNCTION "pgflow"."_create_flow_from_shape" ("p_flow_slug" text, "p_shape" jsonb) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_step jsonb; + v_deps text[]; + v_flow_options jsonb; + v_step_options jsonb; + v_canonical_queue text := lower(p_flow_slug); +BEGIN + -- Preflight the complete shape under the canonical flow lock before any + -- mutation: a late invalid step must not leave earlier queues/definitions. + PERFORM pg_advisory_xact_lock(1, hashtext(lower(p_flow_slug))); + PERFORM pgflow._validate_flow_shape(p_flow_slug, p_shape); + + -- Extract flow-level options (may be null) + v_flow_options := p_shape->'options'; + + -- Create the flow definition (definition-only; no queue DDL) + PERFORM pgflow.create_flow( + p_flow_slug, + (v_flow_options->>'maxAttempts')::int, + (v_flow_options->>'baseDelay')::int, + (v_flow_options->>'timeout')::int + ); + + -- Provision the generated default queue for the persisted identity + PERFORM pgflow._ensure_generated_queue(p_flow_slug, v_canonical_queue); + + -- Iterate over steps in order and add each one with its resolved route + FOR v_step IN SELECT * FROM jsonb_array_elements(p_shape->'steps') + LOOP + -- Convert dependencies jsonb array to text array + SELECT COALESCE(array_agg(dep), '{}') + INTO v_deps + FROM jsonb_array_elements_text(COALESCE(v_step->'dependencies', '[]'::jsonb)) AS dep; + + -- Extract step options (may be null) + v_step_options := v_step->'options'; + + -- Add the step with options (NULL = use default/inherit) + PERFORM pgflow.add_step( + flow_slug => p_flow_slug, + step_slug => v_step->>'slug', + deps_slugs => v_deps, + max_attempts => (v_step_options->>'maxAttempts')::int, + base_delay => (v_step_options->>'baseDelay')::int, + timeout => (v_step_options->>'timeout')::int, + start_delay => (v_step_options->>'startDelay')::int, + step_type => v_step->>'stepType', + when_unmet => COALESCE(v_step->>'whenUnmet', 'skip'), + when_exhausted => COALESCE(v_step->>'whenExhausted', 'fail'), + required_input_pattern => CASE + WHEN (v_step->'requiredInputPattern'->>'defined')::boolean + THEN v_step->'requiredInputPattern'->'value' + ELSE NULL + END, + forbidden_input_pattern => CASE + WHEN (v_step->'forbiddenInputPattern'->>'defined')::boolean + THEN v_step->'forbiddenInputPattern'->'value' + ELSE NULL + END, + queue_name => v_canonical_queue + ); + END LOOP; +END; +$$; +-- Modify "cascade_resolve_conditions" function +CREATE OR REPLACE FUNCTION "pgflow"."cascade_resolve_conditions" ("run_id" uuid) RETURNS boolean LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_run_input jsonb; + v_run_status text; + v_first_fail record; + v_iteration_count int := 0; + v_max_iterations int := 50; + v_processed_count int; + v_run_transitioned boolean; + v_flow_slug text; + v_archive_batch record; +BEGIN + -- ========================================== + -- GUARD: lock the parent run at direct entry, then early-return if the + -- run is already terminal. Callers that already hold the run lock (for + -- example complete_task) re-acquire it harmlessly in the same transaction. + -- ========================================== + SELECT r.status, r.input INTO v_run_status, v_run_input + FROM pgflow.runs r + WHERE r.run_id = cascade_resolve_conditions.run_id + FOR UPDATE; + + IF v_run_status IN ('failed', 'completed') THEN + RETURN v_run_status != 'failed'; + END IF; + + -- ========================================== + -- ITERATE UNTIL CONVERGENCE + -- ========================================== + -- After skipping steps, dependents may become ready and need evaluation. + -- Loop until no more steps are processed. + LOOP + v_iteration_count := v_iteration_count + 1; + IF v_iteration_count > v_max_iterations THEN + RAISE EXCEPTION 'cascade_resolve_conditions exceeded safety limit of % iterations', v_max_iterations; + END IF; + + v_processed_count := 0; + + -- ========================================== + -- PHASE 1a: CHECK FOR FAIL CONDITIONS + -- ========================================== + -- Find first step (by topological order) with unmet condition and 'fail' mode. + -- Condition is unmet when: + -- (required_input_pattern is set AND input does NOT contain it) OR + -- (forbidden_input_pattern is set AND input DOES contain it) + WITH steps_with_conditions AS ( + SELECT + step_state.flow_slug, + step_state.step_slug, + step.required_input_pattern, + step.forbidden_input_pattern, + step.when_unmet, + step.deps_count, + step.step_index + FROM pgflow.step_states AS step_state + JOIN pgflow.steps AS step + ON step.flow_slug = step_state.flow_slug + AND step.step_slug = step_state.step_slug + WHERE step_state.run_id = cascade_resolve_conditions.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + AND (step.required_input_pattern IS NOT NULL OR step.forbidden_input_pattern IS NOT NULL) + ), + step_deps_output AS ( + SELECT + swc.step_slug, + jsonb_object_agg(dep_state.step_slug, dep_state.output) AS deps_output + FROM steps_with_conditions swc + JOIN pgflow.deps dep ON dep.flow_slug = swc.flow_slug AND dep.step_slug = swc.step_slug + JOIN pgflow.step_states dep_state + ON dep_state.run_id = cascade_resolve_conditions.run_id + AND dep_state.step_slug = dep.dep_slug + AND dep_state.status = 'completed' -- Only completed deps (not skipped) + WHERE swc.deps_count > 0 + GROUP BY swc.step_slug + ), + condition_evaluations AS ( + SELECT + swc.*, + -- condition_met = (if IS NULL OR input @> if) AND (ifNot IS NULL OR NOT(input @> ifNot)) + (swc.required_input_pattern IS NULL OR + CASE WHEN swc.deps_count = 0 THEN v_run_input ELSE COALESCE(sdo.deps_output, '{}'::jsonb) END @> swc.required_input_pattern) + AND + (swc.forbidden_input_pattern IS NULL OR + NOT (CASE WHEN swc.deps_count = 0 THEN v_run_input ELSE COALESCE(sdo.deps_output, '{}'::jsonb) END @> swc.forbidden_input_pattern)) + AS condition_met + FROM steps_with_conditions swc + LEFT JOIN step_deps_output sdo ON sdo.step_slug = swc.step_slug + ) + SELECT + flow_slug, + step_slug, + required_input_pattern, + forbidden_input_pattern + INTO v_first_fail + FROM condition_evaluations + WHERE NOT condition_met AND when_unmet = 'fail' + ORDER BY step_index + LIMIT 1; + + -- Handle fail mode: fail step and run, return false + -- Note: Cannot use "v_first_fail IS NOT NULL" because records with NULL fields + -- evaluate to NULL in IS NOT NULL checks. Use FOUND instead. + IF FOUND THEN + -- Fail the run only if it is still started. The conditional UPDATE takes + -- the run row lock and rechecks status atomically, so replayed or + -- concurrent calls cannot duplicate the terminal transition or its events. + UPDATE pgflow.runs + SET status = 'failed', + failed_at = now() + WHERE pgflow.runs.run_id = cascade_resolve_conditions.run_id + AND pgflow.runs.status = 'started' + RETURNING true INTO v_run_transitioned; + + IF v_run_transitioned THEN + UPDATE pgflow.step_states + SET status = 'failed', + failed_at = now(), + error_message = 'Condition not met' + WHERE pgflow.step_states.run_id = cascade_resolve_conditions.run_id + AND pgflow.step_states.step_slug = v_first_fail.step_slug; + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', cascade_resolve_conditions.run_id, + 'step_slug', v_first_fail.step_slug, + 'status', 'failed', + 'error_message', 'Condition not met', + 'failed_at', now() + ), + concat('step:', v_first_fail.step_slug, ':failed'), + concat('pgflow:run:', cascade_resolve_conditions.run_id), + false + ); + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', cascade_resolve_conditions.run_id, + 'flow_slug', v_first_fail.flow_slug, + 'status', 'failed', + 'error_message', 'Condition not met', + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', cascade_resolve_conditions.run_id), + false + ); + + -- Terminalize every unfinished task across all branches as cancelled, + -- capturing their queue/message pairs for archival below. Lock-order + -- invariant: always lock/update step_tasks before PGMQ queue rows. + FOR v_archive_batch IN + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = cascade_resolve_conditions.run_id + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ) + SELECT + ct.queue_name, + ARRAY_AGG(ct.message_id ORDER BY ct.message_id) AS ids + FROM cancelled_tasks ct + WHERE ct.message_id IS NOT NULL + GROUP BY ct.queue_name + LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); + END LOOP; + END IF; + + RETURN false; + END IF; + + -- ========================================== + -- PHASE 1b: HANDLE SKIP CONDITIONS (with propagation) + -- ========================================== + -- Skip steps with unmet conditions and whenUnmet='skip'. + -- Also decrement remaining_deps on dependents and set initial_tasks=0 for map dependents. + WITH steps_with_conditions AS ( + SELECT + step_state.flow_slug, + step_state.step_slug, + step.required_input_pattern, + step.forbidden_input_pattern, + step.when_unmet, + step.deps_count, + step.step_index + FROM pgflow.step_states AS step_state + JOIN pgflow.steps AS step + ON step.flow_slug = step_state.flow_slug + AND step.step_slug = step_state.step_slug + WHERE step_state.run_id = cascade_resolve_conditions.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + AND (step.required_input_pattern IS NOT NULL OR step.forbidden_input_pattern IS NOT NULL) + ), + step_deps_output AS ( + SELECT + swc.step_slug, + jsonb_object_agg(dep_state.step_slug, dep_state.output) AS deps_output + FROM steps_with_conditions swc + JOIN pgflow.deps dep ON dep.flow_slug = swc.flow_slug AND dep.step_slug = swc.step_slug + JOIN pgflow.step_states dep_state + ON dep_state.run_id = cascade_resolve_conditions.run_id + AND dep_state.step_slug = dep.dep_slug + AND dep_state.status = 'completed' -- Only completed deps (not skipped) + WHERE swc.deps_count > 0 + GROUP BY swc.step_slug + ), + condition_evaluations AS ( + SELECT + swc.*, + -- condition_met = (if IS NULL OR input @> if) AND (ifNot IS NULL OR NOT(input @> ifNot)) + (swc.required_input_pattern IS NULL OR + CASE WHEN swc.deps_count = 0 THEN v_run_input ELSE COALESCE(sdo.deps_output, '{}'::jsonb) END @> swc.required_input_pattern) + AND + (swc.forbidden_input_pattern IS NULL OR + NOT (CASE WHEN swc.deps_count = 0 THEN v_run_input ELSE COALESCE(sdo.deps_output, '{}'::jsonb) END @> swc.forbidden_input_pattern)) + AS condition_met + FROM steps_with_conditions swc + LEFT JOIN step_deps_output sdo ON sdo.step_slug = swc.step_slug + ), + unmet_skip_steps AS ( + SELECT * FROM condition_evaluations + WHERE NOT condition_met AND when_unmet = 'skip' + ), + skipped_steps AS ( + UPDATE pgflow.step_states ss + SET status = 'skipped', + skip_reason = 'condition_unmet', + skipped_at = now() + FROM unmet_skip_steps uss + WHERE ss.run_id = cascade_resolve_conditions.run_id + AND ss.step_slug = uss.step_slug + AND ss.status = 'created' + RETURNING + ss.*, + realtime.send( + jsonb_build_object( + 'event_type', 'step:skipped', + 'run_id', ss.run_id, + 'flow_slug', ss.flow_slug, + 'step_slug', ss.step_slug, + 'status', 'skipped', + 'skip_reason', 'condition_unmet', + 'skipped_at', ss.skipped_at + ), + concat('step:', ss.step_slug, ':skipped'), + concat('pgflow:run:', ss.run_id), + false + ) AS _broadcast_result + ), + -- NEW: Update dependent steps (decrement remaining_deps by count of skipped parents, set initial_tasks=0 for maps) + skipped_parent_counts AS ( + -- Count how many skipped parents each child has + SELECT + dep.step_slug AS child_step_slug, + dep.flow_slug AS child_flow_slug, + COUNT(*) AS skipped_parent_count + FROM skipped_steps parent + JOIN pgflow.deps dep ON dep.flow_slug = parent.flow_slug AND dep.dep_slug = parent.step_slug + GROUP BY dep.step_slug, dep.flow_slug + ), + dependent_updates AS ( + UPDATE pgflow.step_states child_state + SET remaining_deps = child_state.remaining_deps - spc.skipped_parent_count, + -- If child is a map step and this skipped step is its only dependency, + -- set initial_tasks = 0 (skipped dep = empty array) + initial_tasks = CASE + WHEN child_step.step_type = 'map' AND child_step.deps_count = 1 THEN 0 + ELSE child_state.initial_tasks + END + FROM skipped_parent_counts spc + JOIN pgflow.steps child_step ON child_step.flow_slug = spc.child_flow_slug AND child_step.step_slug = spc.child_step_slug + WHERE child_state.run_id = cascade_resolve_conditions.run_id + AND child_state.step_slug = spc.child_step_slug + ), + run_update AS ( + UPDATE pgflow.runs r + SET remaining_steps = r.remaining_steps - (SELECT COUNT(*) FROM skipped_steps) + WHERE r.run_id = cascade_resolve_conditions.run_id + AND (SELECT COUNT(*) FROM skipped_steps) > 0 + ) + SELECT COUNT(*)::int INTO v_processed_count FROM skipped_steps; + + -- ========================================== + -- PHASE 1c: HANDLE SKIP-CASCADE CONDITIONS + -- ========================================== + -- Call _cascade_force_skip_steps for each step with unmet condition and whenUnmet='skip-cascade'. + -- Process in topological order; _cascade_force_skip_steps is idempotent. + PERFORM pgflow._cascade_force_skip_steps(cascade_resolve_conditions.run_id, ready_step.step_slug, 'condition_unmet') + FROM pgflow.step_states AS ready_step + JOIN pgflow.steps AS step + ON step.flow_slug = ready_step.flow_slug + AND step.step_slug = ready_step.step_slug + LEFT JOIN LATERAL ( + SELECT jsonb_object_agg(dep_state.step_slug, dep_state.output) AS deps_output + FROM pgflow.deps dep + JOIN pgflow.step_states dep_state + ON dep_state.run_id = cascade_resolve_conditions.run_id + AND dep_state.step_slug = dep.dep_slug + AND dep_state.status = 'completed' -- Only completed deps (not skipped) + WHERE dep.flow_slug = ready_step.flow_slug + AND dep.step_slug = ready_step.step_slug + ) AS agg_deps ON step.deps_count > 0 + WHERE ready_step.run_id = cascade_resolve_conditions.run_id + AND ready_step.status = 'created' + AND ready_step.remaining_deps = 0 + AND (step.required_input_pattern IS NOT NULL OR step.forbidden_input_pattern IS NOT NULL) + AND step.when_unmet = 'skip-cascade' + -- Condition is NOT met when: (if fails) OR (ifNot fails) + AND NOT ( + (step.required_input_pattern IS NULL OR + CASE WHEN step.deps_count = 0 THEN v_run_input ELSE COALESCE(agg_deps.deps_output, '{}'::jsonb) END @> step.required_input_pattern) + AND + (step.forbidden_input_pattern IS NULL OR + NOT (CASE WHEN step.deps_count = 0 THEN v_run_input ELSE COALESCE(agg_deps.deps_output, '{}'::jsonb) END @> step.forbidden_input_pattern)) + ) + ORDER BY step.step_index; + + -- Check if run was failed during cascade (e.g., if _cascade_force_skip_steps triggers fail) + SELECT r.status INTO v_run_status + FROM pgflow.runs r + WHERE r.run_id = cascade_resolve_conditions.run_id; + + IF v_run_status IN ('failed', 'completed') THEN + RETURN v_run_status != 'failed'; + END IF; + + -- Exit loop if no steps were processed in this iteration + EXIT WHEN v_processed_count = 0; + END LOOP; + + RETURN true; +END; +$$; +-- Modify "start_ready_steps" function +CREATE OR REPLACE FUNCTION "pgflow"."start_ready_steps" ("run_id" uuid) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +BEGIN +-- ========================================== +-- GUARD: No mutations on terminal runs +-- ========================================== +IF EXISTS ( + SELECT 1 FROM pgflow.runs + WHERE pgflow.runs.run_id = start_ready_steps.run_id + AND pgflow.runs.status IN ('failed', 'completed') +) THEN + RETURN; +END IF; + +-- ========================================== +-- PHASE 1: START READY STEPS +-- ========================================== +-- NOTE: Condition evaluation and empty map handling are done by +-- cascade_resolve_conditions() and cascade_complete_taskless_steps() +-- which are called before this function. +WITH +-- ---------- Find ready steps ---------- +-- Steps with no remaining deps and known task count +ready_steps AS ( + SELECT * + FROM pgflow.step_states AS step_state + WHERE step_state.run_id = start_ready_steps.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + AND step_state.initial_tasks IS NOT NULL -- Cannot start with unknown count + AND step_state.initial_tasks > 0 -- Don't start taskless steps (handled by cascade_complete_taskless_steps) + ORDER BY step_state.step_slug + FOR UPDATE +), +-- ---------- Mark steps as started ---------- +started_step_states AS ( + UPDATE pgflow.step_states + SET status = 'started', + started_at = now(), + remaining_tasks = ready_steps.initial_tasks -- Copy initial_tasks to remaining_tasks when starting + FROM ready_steps + WHERE pgflow.step_states.run_id = start_ready_steps.run_id + AND pgflow.step_states.step_slug = ready_steps.step_slug + RETURNING pgflow.step_states.*, + -- Broadcast step:started event atomically with the UPDATE + -- Using RETURNING ensures this executes during row processing + -- and cannot be optimized away by the query planner + realtime.send( + jsonb_build_object( + 'event_type', 'step:started', + 'run_id', pgflow.step_states.run_id, + 'step_slug', pgflow.step_states.step_slug, + 'status', 'started', + 'started_at', pgflow.step_states.started_at, + 'remaining_tasks', pgflow.step_states.remaining_tasks, + 'remaining_deps', pgflow.step_states.remaining_deps + ), + concat('step:', pgflow.step_states.step_slug, ':started'), + concat('pgflow:run:', pgflow.step_states.run_id), + false + ) as _broadcast_result -- Prefix with _ to indicate internal use only +), + +-- ========================================== +-- PHASE 2: TASK GENERATION AND QUEUE MESSAGES +-- ========================================== +-- ---------- Generate tasks and batch messages ---------- +-- Single steps: 1 task (index 0) +-- Map steps: N tasks (indices 0..N-1) +message_batches AS ( + SELECT + started_step.flow_slug, + started_step.run_id, + started_step.step_slug, + step.queue_name, + COALESCE(step.opt_start_delay, 0) as delay, + array_agg( + jsonb_build_object( + 'flow_slug', started_step.flow_slug, + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'task_index', task_idx.task_index + ) ORDER BY task_idx.task_index + ) AS messages, + array_agg(task_idx.task_index ORDER BY task_idx.task_index) AS task_indices + FROM started_step_states AS started_step + JOIN pgflow.steps AS step + ON step.flow_slug = started_step.flow_slug + AND step.step_slug = started_step.step_slug + -- Generate task indices from 0 to initial_tasks-1 + CROSS JOIN LATERAL generate_series(0, started_step.initial_tasks - 1) AS task_idx(task_index) + GROUP BY started_step.flow_slug, started_step.run_id, started_step.step_slug, step.queue_name, step.opt_start_delay +), +-- ---------- Send messages to queue ---------- +-- Uses batch sending for performance with large arrays +-- Sends to each step's resolved route; performs no queue DDL (#650) +sent_messages AS ( + SELECT + mb.flow_slug, + mb.run_id, + mb.step_slug, + mb.queue_name, + task_indices.task_index, + msg_ids.msg_id + FROM message_batches mb + CROSS JOIN LATERAL unnest(mb.task_indices) WITH ORDINALITY AS task_indices(task_index, idx_ord) + CROSS JOIN LATERAL pgmq.send_batch(mb.queue_name, mb.messages, mb.delay) WITH ORDINALITY AS msg_ids(msg_id, msg_ord) + WHERE task_indices.idx_ord = msg_ids.msg_ord +) + +-- ========================================== +-- PHASE 3: RECORD TASKS IN DATABASE +-- ========================================== +-- The task snapshots the resolved queue; the snapshot never changes after +-- insertion (#650). +INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, queue_name, message_id) +SELECT + sent_messages.flow_slug, + sent_messages.run_id, + sent_messages.step_slug, + sent_messages.task_index, + sent_messages.queue_name, + sent_messages.msg_id +FROM sent_messages; + +END; +$$; +-- Modify "complete_task" function +CREATE OR REPLACE FUNCTION "pgflow"."complete_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "output" jsonb) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_step_state pgflow.step_states%ROWTYPE; + v_dependent_map_slug text; + v_run_record pgflow.runs%ROWTYPE; + v_step_record pgflow.step_states%ROWTYPE; + v_archive_batch record; +begin + +-- ========================================== +-- GUARD: No mutations on failed runs +-- ========================================== +IF EXISTS (SELECT 1 FROM pgflow.runs WHERE pgflow.runs.run_id = complete_task.run_id AND pgflow.runs.status = 'failed') THEN + -- Archive the late callback message through the locked single-task + -- helper (run/step/task locks are its own acquisition); the message must + -- not stay visible for re-reading after a failed run (#650). + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; +END IF; + +-- ========================================== +-- LOCK ACQUISITION AND TYPE VALIDATION +-- ========================================== +-- Acquire locks first to prevent race conditions +SELECT * INTO v_run_record FROM pgflow.runs +WHERE pgflow.runs.run_id = complete_task.run_id +FOR UPDATE; + +SELECT * INTO v_step_record FROM pgflow.step_states +WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug +FOR UPDATE; + +-- ========================================== +-- GUARD: Run failed while this callback waited for the lock +-- ========================================== +-- The failed-run guard above ran before the failure committed. Recheck under +-- lock so cancellation wins: archived message stays archived, task row keeps +-- its terminal status, and no events or counters are emitted. +IF v_run_record.status = 'failed' THEN + -- Archive the task message if present (no-op when already archived) + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); + -- Return the current task row without any mutations + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; +END IF; + +-- ========================================== +-- GUARD: Late callback - step not started +-- ========================================== +-- If the step is not in 'started' state, this is a late callback. +-- Do not mutate step_states or runs, archive message, return task row. +IF v_step_record.status != 'started' THEN + -- Archive the task message if present (prevents stuck work) through the + -- locked single-task helper; run/step locks are already held here + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); + -- Return the current task row without any mutations + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; +END IF; + +-- Check for type violations AFTER acquiring locks +SELECT child_step.step_slug INTO v_dependent_map_slug +FROM pgflow.deps dependency +JOIN pgflow.steps child_step ON child_step.flow_slug = dependency.flow_slug + AND child_step.step_slug = dependency.step_slug +JOIN pgflow.steps parent_step ON parent_step.flow_slug = dependency.flow_slug + AND parent_step.step_slug = dependency.dep_slug +JOIN pgflow.step_states child_state ON child_state.flow_slug = child_step.flow_slug + AND child_state.step_slug = child_step.step_slug +WHERE dependency.dep_slug = complete_task.step_slug -- parent is the completing step + AND dependency.flow_slug = v_run_record.flow_slug + AND parent_step.step_type = 'single' -- Only validate single steps + AND child_step.step_type = 'map' + AND child_state.run_id = complete_task.run_id + AND child_state.initial_tasks IS NULL + AND (complete_task.output IS NULL OR jsonb_typeof(complete_task.output) != 'array') +LIMIT 1; + +-- Handle type violation if detected +IF v_dependent_map_slug IS NOT NULL THEN + -- Mark current task as failed FIRST and store the output that caused the + -- violation, so the task row is terminal before any queue row is touched. + UPDATE pgflow.step_tasks + SET status = 'failed', + failed_at = now(), + output = complete_task.output, -- Store the output that caused the violation + error_message = '[TYPE_VIOLATION] Produced ' || + CASE WHEN complete_task.output IS NULL THEN 'null' + ELSE jsonb_typeof(complete_task.output) END || + ' instead of array' + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + + -- Mark run as failed immediately + UPDATE pgflow.runs + SET status = 'failed', + failed_at = now() + WHERE pgflow.runs.run_id = complete_task.run_id; + + -- Broadcast run:failed event + -- Uses PERFORM pattern to ensure execution (proven reliable pattern in this function) + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', complete_task.run_id, + 'flow_slug', v_run_record.flow_slug, + 'status', 'failed', + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', complete_task.run_id), + false + ); + + -- Mark step state as failed + UPDATE pgflow.step_states + SET status = 'failed', + failed_at = now(), + error_message = '[TYPE_VIOLATION] Map step ' || v_dependent_map_slug || + ' expects array input but dependency ' || complete_task.step_slug || + ' produced ' || CASE WHEN complete_task.output IS NULL THEN 'null' + ELSE jsonb_typeof(complete_task.output) END + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug; + + -- Broadcast step:failed event + -- Uses PERFORM pattern to ensure execution (proven reliable pattern in this function) + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', complete_task.run_id, + 'step_slug', complete_task.step_slug, + 'status', 'failed', + 'error_message', '[TYPE_VIOLATION] Map step ' || v_dependent_map_slug || + ' expects array input but dependency ' || complete_task.step_slug || + ' produced ' || CASE WHEN complete_task.output IS NULL THEN 'null' + ELSE jsonb_typeof(complete_task.output) END, + 'failed_at', now() + ), + concat('step:', complete_task.step_slug, ':failed'), + concat('pgflow:run:', complete_task.run_id), + false + ); + + -- Terminalize every other unfinished task as cancelled, capturing their + -- queue/message pairs for archival below. Lock-order invariant: always + -- lock/update step_tasks before PGMQ queue rows. The culprit task is + -- already terminal (failed above), so it is excluded from the cancellation + -- set. The grouped FOR forces the cancellation UPDATE to run before any + -- archive call. + FOR v_archive_batch IN + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = complete_task.run_id + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ), + culprit_task AS ( + -- Terminal culprit row: safe to read for its queue/message pair after + -- terminalization + SELECT st.queue_name, st.message_id + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.message_id IS NOT NULL + ), + archived_pairs AS ( + SELECT + ids.queue_name, + ARRAY_AGG(ids.message_id ORDER BY ids.message_id) AS ids + FROM ( + SELECT queue_name, message_id FROM culprit_task + UNION ALL + SELECT queue_name, message_id FROM cancelled_tasks WHERE message_id IS NOT NULL + ) ids + GROUP BY ids.queue_name + ) + SELECT ap.queue_name, ap.ids FROM archived_pairs ap + LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); + END LOOP; + + -- Return the failed task row (API contract: always return task row) + RETURN QUERY + SELECT * FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index; + RETURN; +END IF; + +-- ========================================== +-- MAIN CTE CHAIN: Update task and propagate changes +-- ========================================== +WITH +-- ---------- Task completion ---------- +-- Update the task record with completion status and output +task AS ( + UPDATE pgflow.step_tasks + SET + status = 'completed', + completed_at = now(), + output = complete_task.output + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index + AND pgflow.step_tasks.status = 'started' + RETURNING * +), +-- ---------- Get step type for output handling ---------- +step_def AS ( + SELECT step.step_type + FROM pgflow.steps step + JOIN pgflow.runs run ON run.flow_slug = step.flow_slug + WHERE run.run_id = complete_task.run_id + AND step.step_slug = complete_task.step_slug +), +-- ---------- Step state update ---------- +-- Decrement remaining_tasks and potentially mark step as completed +-- Also store output atomically with status transition to completed +step_state AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN 'completed' -- Will be 0 after decrement + ELSE 'started' + END, + completed_at = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN now() -- Will be 0 after decrement + ELSE NULL + END, + remaining_tasks = pgflow.step_states.remaining_tasks - 1, + -- Store output atomically with completion (only when remaining_tasks = 1, meaning step completes) + output = CASE + -- Single step: store task output directly when completing + WHEN (SELECT step_type FROM step_def) = 'single' AND pgflow.step_states.remaining_tasks = 1 THEN + complete_task.output + -- Map step: aggregate on completion (ordered by task_index) + WHEN (SELECT step_type FROM step_def) = 'map' AND pgflow.step_states.remaining_tasks = 1 THEN + (SELECT COALESCE(jsonb_agg(all_outputs.output ORDER BY all_outputs.task_index), '[]'::jsonb) + FROM ( + -- All previously completed tasks + SELECT st.output, st.task_index + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.status = 'completed' + UNION ALL + -- Current task being completed (not yet visible as completed in snapshot) + SELECT complete_task.output, complete_task.task_index + ) all_outputs) + ELSE pgflow.step_states.output + END + FROM task + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + RETURNING pgflow.step_states.* +), +-- ---------- Dependency resolution ---------- +-- Find all child steps that depend on the completed parent step (only if parent completed) +child_steps AS ( + SELECT deps.step_slug AS child_step_slug + FROM pgflow.deps deps + JOIN step_state parent_state ON parent_state.status = 'completed' AND deps.flow_slug = parent_state.flow_slug + WHERE deps.dep_slug = complete_task.step_slug -- dep_slug is the parent, step_slug is the child + ORDER BY deps.step_slug -- Ensure consistent ordering +), +-- ---------- Lock child steps ---------- +-- Acquire locks on all child steps before updating them +child_steps_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug IN (SELECT child_step_slug FROM child_steps) + FOR UPDATE +), +-- ---------- Update child steps ---------- +-- Decrement remaining_deps and resolve NULL initial_tasks for map steps +child_steps_update AS ( + UPDATE pgflow.step_states child_state + SET remaining_deps = child_state.remaining_deps - 1, + -- Resolve NULL initial_tasks for child map steps + -- This is where child maps learn their array size from the parent + -- This CTE only runs when the parent step is complete (see child_steps JOIN) + initial_tasks = CASE + WHEN child_step.step_type = 'map' AND child_state.initial_tasks IS NULL THEN + CASE + WHEN parent_step.step_type = 'map' THEN + -- Map->map: Count all completed tasks from parent map + -- We add 1 because the current task is being completed in this transaction + -- but isn't yet visible as 'completed' in the step_tasks table + -- TODO: Refactor to use future column step_states.total_tasks + -- Would eliminate the COUNT query and just use parent_state.total_tasks + (SELECT COUNT(*)::int + 1 + FROM pgflow.step_tasks parent_tasks + WHERE parent_tasks.run_id = complete_task.run_id + AND parent_tasks.step_slug = complete_task.step_slug + AND parent_tasks.status = 'completed' + AND parent_tasks.task_index != complete_task.task_index) + ELSE + -- Single->map: Use output array length (single steps complete immediately) + CASE + WHEN complete_task.output IS NOT NULL + AND jsonb_typeof(complete_task.output) = 'array' THEN + jsonb_array_length(complete_task.output) + ELSE NULL -- Keep NULL if not an array + END + END + ELSE child_state.initial_tasks -- Keep existing value (including NULL) + END + FROM child_steps children + JOIN pgflow.steps child_step ON child_step.flow_slug = (SELECT r.flow_slug FROM pgflow.runs r WHERE r.run_id = complete_task.run_id) + AND child_step.step_slug = children.child_step_slug + JOIN pgflow.steps parent_step ON parent_step.flow_slug = (SELECT r.flow_slug FROM pgflow.runs r WHERE r.run_id = complete_task.run_id) + AND parent_step.step_slug = complete_task.step_slug + WHERE child_state.run_id = complete_task.run_id + AND child_state.step_slug = children.child_step_slug +) +-- ---------- Update run remaining_steps ---------- +-- Decrement the run's remaining_steps counter if step completed +UPDATE pgflow.runs +SET remaining_steps = pgflow.runs.remaining_steps - 1 +FROM step_state +WHERE pgflow.runs.run_id = complete_task.run_id + AND step_state.status = 'completed'; + +-- ========================================== +-- POST-COMPLETION ACTIONS +-- ========================================== + +-- ---------- Get updated state for broadcasting ---------- +SELECT * INTO v_step_state FROM pgflow.step_states +WHERE pgflow.step_states.run_id = complete_task.run_id AND pgflow.step_states.step_slug = complete_task.step_slug; + +-- ---------- Handle step completion ---------- +IF v_step_state.status = 'completed' THEN + -- Broadcast step:completed event FIRST (before cascade) + -- This ensures parent broadcasts before its dependent children + -- Use stored output from step_states (set atomically during status transition) + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:completed', + 'run_id', complete_task.run_id, + 'step_slug', complete_task.step_slug, + 'status', 'completed', + 'output', v_step_state.output, -- Use stored output instead of re-aggregating + 'completed_at', v_step_state.completed_at + ), + concat('step:', complete_task.step_slug, ':completed'), + concat('pgflow:run:', complete_task.run_id), + false + ); + + -- THEN evaluate conditions on newly-ready dependent steps + -- This must happen before cascade_complete_taskless_steps so that + -- skipped steps can set initial_tasks=0 for their map dependents + IF NOT pgflow.cascade_resolve_conditions(complete_task.run_id) THEN + -- Run was failed due to a condition with when_unmet='fail' + -- Archive the current task's message before returning through the + -- locked single-task helper + PERFORM pgflow._archive_task_message( + complete_task.run_id, + complete_task.step_slug, + complete_task.task_index + ); + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; + END IF; + + -- THEN cascade complete any taskless steps that are now ready + -- This ensures dependent children broadcast AFTER their parent + PERFORM pgflow.cascade_complete_taskless_steps(complete_task.run_id); +END IF; + +-- ---------- Archive completed task message ---------- +-- Move message from active queue to archive table using the task's queue +-- snapshot (#650) +PERFORM ( + WITH completed_tasks AS ( + SELECT st.queue_name, st.message_id + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.status = 'completed' + ) + SELECT pgmq.archive(ct.queue_name, ct.message_id) + FROM completed_tasks ct + WHERE EXISTS (SELECT 1 FROM completed_tasks) +); + +-- ---------- Trigger next steps ---------- +-- Start any steps that are now ready (deps satisfied) +PERFORM pgflow.start_ready_steps(complete_task.run_id); + +-- Check if the entire run is complete +PERFORM pgflow.maybe_complete_run(complete_task.run_id); + +-- ---------- Return completed task ---------- +RETURN QUERY SELECT * +FROM pgflow.step_tasks AS step_task +WHERE step_task.run_id = complete_task.run_id + AND step_task.step_slug = complete_task.step_slug + AND step_task.task_index = complete_task.task_index; + +end; +$$; +-- Modify "delete_flow_and_data" function +CREATE OR REPLACE FUNCTION "pgflow"."delete_flow_and_data" ("p_flow_slug" text) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_route text[]; + v_route_names text[]; + v_metadata_names text[]; + v_snapshot_violation record; + v_queue text; + v_qtable text; + v_atable text; + v_sequence text; + v_inspect_result jsonb; + v_idx int; +BEGIN + PERFORM pg_advisory_xact_lock(1, hashtext(lower(p_flow_slug))); + + -- Retain and lock the concrete identity; reject a missing flow + IF NOT EXISTS ( + SELECT 1 FROM pgflow.flows f + WHERE f.flow_slug = p_flow_slug + FOR UPDATE + ) THEN + RAISE EXCEPTION 'Flow % does not exist', p_flow_slug; + END IF; + + -- Runtime locks in the established order, before any queue/metadata access + PERFORM 1 FROM pgflow.runs r + WHERE r.flow_slug = p_flow_slug + ORDER BY r.run_id + FOR UPDATE; + + PERFORM 1 FROM pgflow.step_states s + WHERE s.flow_slug = p_flow_slug + ORDER BY s.run_id, s.step_slug + FOR UPDATE; + + PERFORM 1 FROM pgflow.step_tasks t + WHERE t.flow_slug = p_flow_slug + ORDER BY t.run_id, t.step_slug, t.task_index + FOR UPDATE; + + -- Topology fence after runtime locks (never the reverse order) + LOCK TABLE pgmq.meta IN SHARE ROW EXCLUSIVE MODE; + + -- Capture the complete private route: persisted steps plus the default for + -- an empty plain flow. An unprovisioned definition-only flow is not + -- permission to drop a same-named resource; _inspect_generated_queue + -- rejects that below. + SELECT COALESCE( + ARRAY_AGG(DISTINCT s.queue_name ORDER BY s.queue_name), + ARRAY[lower(p_flow_slug)] + ) INTO v_route + FROM pgflow.steps s + WHERE s.flow_slug = p_flow_slug; + + -- Validate every task snapshot against the captured route + SELECT t.run_id, t.step_slug, t.task_index, t.queue_name + INTO v_snapshot_violation + FROM pgflow.step_tasks t + WHERE t.flow_slug = p_flow_slug + AND NOT (t.queue_name = ANY(v_route)) + ORDER BY t.run_id, t.step_slug, t.task_index + LIMIT 1; + + IF v_snapshot_violation IS NOT NULL THEN + RAISE EXCEPTION 'Flow %: task %/%/% snapshot queue "%" is outside the validated private route; refusing deletion', + p_flow_slug, + v_snapshot_violation.run_id, + v_snapshot_violation.step_slug, + v_snapshot_violation.task_index, + v_snapshot_violation.queue_name; + END IF; + + v_route_names := v_route; + v_metadata_names := ARRAY[]::text[]; + + -- Resolve exact metadata spelling and physical validity per route queue; + -- this also rejects missing, ambiguous, malformed, or differently owned + -- resources instead of dropping an uncertain physical queue. + FOR v_idx IN 1..COALESCE(array_length(v_route, 1), 0) + LOOP + v_queue := v_route[v_idx]; + v_inspect_result := pgflow._inspect_generated_queue(p_flow_slug, v_queue, true); + + -- Lock the validated physical queue/archive tables before mutation + EXECUTE format( + 'LOCK TABLE pgmq.%I, pgmq.%I IN ACCESS EXCLUSIVE MODE', + pgmq.format_table_name(v_queue, 'q'), + pgmq.format_table_name(v_queue, 'a') + ); + + -- Recheck ownership/shape after the physical locks are held + v_inspect_result := pgflow._inspect_generated_queue(p_flow_slug, v_queue, true); + + -- Remember the exact metadata spelling for the drop below: the flow row + -- and step definitions may be gone by then. + v_metadata_names[v_idx] := v_inspect_result ->> 'metadata_name'; + END LOOP; + + -- Delete runtime rows and step definitions in FK order while retaining the + -- flow identity and captured validated queue names + DELETE FROM pgflow.step_tasks AS task WHERE task.flow_slug = p_flow_slug; + DELETE FROM pgflow.step_states AS state WHERE state.flow_slug = p_flow_slug; + DELETE FROM pgflow.runs AS run WHERE run.flow_slug = p_flow_slug; + DELETE FROM pgflow.deps AS dep WHERE dep.flow_slug = p_flow_slug; + DELETE FROM pgflow.steps AS step WHERE step.flow_slug = p_flow_slug; + + -- Drop each validated private queue using its exact metadata spelling. + -- No per-message archival/deletion happens before the whole-queue drop. + FOR v_idx IN 1..COALESCE(array_length(v_route_names, 1), 0) + LOOP + v_queue := v_route_names[v_idx]; + v_qtable := pgmq.format_table_name(v_queue, 'q'); + v_atable := pgmq.format_table_name(v_queue, 'a'); + v_sequence := v_qtable || '_msg_id_seq'; + + PERFORM pgmq.drop_queue(v_metadata_names[v_idx]); + + -- Post-drop completeness: pgmq.drop_queue must have removed the + -- metadata row, both physical tables, and the identity sequence. A + -- partial drop leaves the namespace ambiguous and must abort before + -- the flow identity row is deleted. + IF EXISTS ( + SELECT 1 FROM pgmq.meta m WHERE lower(m.queue_name) = v_queue + ) THEN + RAISE EXCEPTION 'Flow %: dropping generated queue "%" left its PGMQ metadata behind; deletion aborted with everything rolled back', + p_flow_slug, v_queue; + END IF; + + IF to_regclass(format('pgmq.%I', v_qtable)) IS NOT NULL + OR to_regclass(format('pgmq.%I', v_atable)) IS NOT NULL + OR to_regclass(format('pgmq.%I', v_sequence)) IS NOT NULL THEN + RAISE EXCEPTION 'Flow %: dropping generated queue "%" left physical objects behind (queue table: %, archive table: %, sequence: %); deletion aborted with everything rolled back', + p_flow_slug, v_queue, v_qtable, v_atable, v_sequence; + END IF; + END LOOP; + + -- Delete the concrete flow identity row last + DELETE FROM pgflow.flows AS flow WHERE flow.flow_slug = p_flow_slug; +END; +$$; +-- Modify "requeue_stalled_tasks" function +CREATE OR REPLACE FUNCTION "pgflow"."requeue_stalled_tasks" () RETURNS integer LANGUAGE plpgsql SECURITY DEFINER SET "search_path" = '' AS $$ +declare + result_count int := 0; + max_requeues constant int := 3; +begin + -- Find and requeue stalled tasks (where started_at > effective timeout + 30s buffer) + -- Tasks with requeued_count >= max_requeues will have their message archived + -- but status left as 'started' for easy identification via requeued_count column + -- Eligibility requires the parent run AND parent step to still be 'started': + -- stale rows on failed runs or terminal steps must not be revived (#645). + -- + -- Lock order (#650): eligible parent runs are locked first (ordered by + -- run_id), then eligible step states (ordered by (run_id, step_slug)), then + -- task rows (ordered by (run_id, step_slug, task_index)) - as three + -- sequential lock sets, not one joined FOR UPDATE, so parent rows are + -- always locked before their children. SKIP LOCKED is preserved at every + -- level: a blocked parent/run/state/task is skipped, not waited on, and no + -- later-order lock is held while waiting. Status and timeout predicates + -- are restated in each phase so EvalPlanQual rechecks them under the locks. + with locked_runs as ( + select r.run_id + from pgflow.runs r + where r.status = 'started' + and exists ( + select 1 + from pgflow.step_tasks st + join pgflow.step_states ss on ss.run_id = st.run_id and ss.step_slug = st.step_slug + join pgflow.flows f on f.flow_slug = r.flow_slug + join pgflow.steps s on s.flow_slug = r.flow_slug and s.step_slug = st.step_slug + where st.run_id = r.run_id + and st.status = 'started' + and ss.status = 'started' + and st.permanently_stalled_at is null + and st.started_at < now() + - (coalesce(s.opt_timeout, f.opt_timeout) * interval '1 second') + - interval '30 seconds' + ) + order by r.run_id + for update skip locked + ), + locked_states as ( + select ss.run_id, ss.step_slug + from pgflow.step_states ss + join locked_runs lr on lr.run_id = ss.run_id + where ss.status = 'started' + and exists ( + select 1 + from pgflow.step_tasks st + join pgflow.flows f on f.flow_slug = ss.flow_slug + join pgflow.steps s on s.flow_slug = ss.flow_slug and s.step_slug = st.step_slug + where st.run_id = ss.run_id + and st.step_slug = ss.step_slug + and st.status = 'started' + and st.permanently_stalled_at is null + and st.started_at < now() + - (coalesce(s.opt_timeout, f.opt_timeout) * interval '1 second') + - interval '30 seconds' + ) + order by ss.run_id, ss.step_slug + for update of ss skip locked + ), + stalled_tasks as ( + select + st.run_id, + st.step_slug, + st.task_index, + st.message_id, + st.queue_name, + st.requeued_count + from pgflow.step_tasks st + join locked_states ls on ls.run_id = st.run_id and ls.step_slug = st.step_slug + join pgflow.runs r on r.run_id = st.run_id + join pgflow.flows f on f.flow_slug = r.flow_slug + join pgflow.steps s on s.flow_slug = r.flow_slug and s.step_slug = st.step_slug + where st.status = 'started' + and r.status = 'started' + and st.permanently_stalled_at is null + and st.started_at < now() + - (coalesce(s.opt_timeout, f.opt_timeout) * interval '1 second') + - interval '30 seconds' + order by st.run_id, st.step_slug, st.task_index + for update of st skip locked + ), + -- Separate tasks that can be requeued from those that exceeded max requeues + to_requeue as ( + select * from stalled_tasks where requeued_count < max_requeues + ), + to_archive as ( + select * from stalled_tasks where requeued_count >= max_requeues + ), + -- Update tasks that will be requeued; the queue comes from the task snapshot + requeued as ( + update pgflow.step_tasks st + set + status = 'queued', + started_at = null, + last_worker_id = null, + requeued_count = st.requeued_count + 1, + last_requeued_at = now() + from to_requeue tr + where st.run_id = tr.run_id + and st.step_slug = tr.step_slug + and st.task_index = tr.task_index + returning tr.queue_name as queue_name, tr.message_id + ), + -- Make requeued messages visible immediately (batched per queue snapshot) + visibility_reset as ( + select pgflow.set_vt_batch( + r.queue_name, + array_agg(r.message_id order by r.message_id), + array_agg(0 order by r.message_id) -- all offsets are 0 (immediate visibility) + ) + from requeued r + where r.message_id is not null + group by r.queue_name + ), + -- Mark tasks as permanently stalled before archiving + mark_permanently_stalled as ( + update pgflow.step_tasks st + set permanently_stalled_at = now() + from to_archive ta + where st.run_id = ta.run_id + and st.step_slug = ta.step_slug + and st.task_index = ta.task_index + returning st.run_id + ), + -- Archive messages for tasks that exceeded max requeues (batched per queue + -- snapshot; never grouped across queues) + archived as ( + select pgmq.archive(ta.queue_name, array_agg(ta.message_id)) + from to_archive ta + where ta.message_id is not null + group by ta.queue_name + ) + -- Force execution of every side-effecting CTE regardless of join order: + -- a cross join with an empty relation could skip scanning the forcing + -- wrappers, so they are evaluated as scalar subqueries that always run. + select + (select count(*) from requeued) + + 0 * coalesce( + (select count(*) from visibility_reset) + + (select count(*) from mark_permanently_stalled) + + (select count(*) from archived), + 0 + ) + into result_count; + + return result_count; +end; +$$; +-- Modify "fail_task" function +CREATE OR REPLACE FUNCTION "pgflow"."fail_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "error_message" text) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_run_failed boolean; + v_step_failed boolean; + v_step_skipped boolean; + v_when_exhausted text; + v_task_exhausted boolean; + v_flow_slug_for_deps text; + v_prev_step_status text; + v_run_status text; + v_flow_slug text; + v_archive_batch record; +begin + +-- If run is already failed, no retries allowed. +-- Cancellation wins: tasks terminalized by the run failure (failed culprit or +-- cancelled siblings) keep their terminal status. This late callback only +-- archives any still-active message and returns the current row unchanged. +IF EXISTS (SELECT 1 FROM pgflow.runs WHERE pgflow.runs.run_id = fail_task.run_id AND pgflow.runs.status = 'failed') THEN + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); + + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; +END IF; + +-- Late callback guard: lock run + step rows and use current statuses +-- under lock so concurrent fail_task calls cannot read stale status. +SELECT ss.status, r.status, r.flow_slug INTO v_prev_step_status, v_run_status, v_flow_slug +FROM pgflow.runs r +JOIN pgflow.step_states ss ON ss.run_id = r.run_id +WHERE ss.run_id = fail_task.run_id + AND ss.step_slug = fail_task.step_slug +FOR UPDATE OF r, ss; + +-- Recheck under lock: the run may have failed while this callback waited +-- for the lock (the EXISTS guard above ran before the failure committed). +IF v_run_status = 'failed' THEN + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); + + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; +END IF; + +IF v_prev_step_status IS NOT NULL AND v_prev_step_status != 'started' THEN + -- Archive the task message if present, through the locked single-task + -- helper (locks already held above) + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); + + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; +END IF; + +WITH flow_info AS ( + SELECT r.flow_slug + FROM pgflow.runs r + WHERE r.run_id = fail_task.run_id +), + config AS ( + SELECT + COALESCE(s.opt_max_attempts, f.opt_max_attempts) AS opt_max_attempts, + COALESCE(s.opt_base_delay, f.opt_base_delay) AS opt_base_delay, + s.when_exhausted + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN flow_info fi ON fi.flow_slug = s.flow_slug + WHERE s.flow_slug = fi.flow_slug AND s.step_slug = fail_task.step_slug +), +fail_or_retry_task as ( + UPDATE pgflow.step_tasks as task + SET + status = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN 'queued' + ELSE 'failed' + END, + failed_at = CASE + WHEN task.attempts_count >= (SELECT opt_max_attempts FROM config) THEN now() + ELSE NULL + END, + started_at = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN NULL + ELSE task.started_at + END, + error_message = fail_task.error_message + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.task_index = fail_task.task_index + AND task.status = 'started' + RETURNING * +), + -- Determine if task exhausted retries and get when_exhausted mode + task_status AS ( + SELECT + (select status from fail_or_retry_task) AS new_task_status, + (select when_exhausted from config) AS when_exhausted_mode, + -- Task is exhausted when it's failed (no more retries) + ((select status from fail_or_retry_task) = 'failed') AS is_exhausted +), +maybe_fail_step AS ( + UPDATE pgflow.step_states + SET + -- Status logic: + -- - If task not exhausted (retrying): keep current status + -- - If exhausted AND when_exhausted='fail': set to 'failed' + -- - If exhausted AND when_exhausted IN ('skip', 'skip-cascade'): set to 'skipped' + status = CASE + WHEN NOT (select is_exhausted from task_status) THEN pgflow.step_states.status + WHEN (select when_exhausted_mode from task_status) = 'fail' THEN 'failed' + ELSE 'skipped' -- skip or skip-cascade + END, + failed_at = CASE + WHEN (select is_exhausted from task_status) AND (select when_exhausted_mode from task_status) = 'fail' THEN now() + ELSE NULL + END, + error_message = CASE + WHEN (select is_exhausted from task_status) THEN fail_task.error_message + ELSE NULL + END, + skip_reason = CASE + WHEN (select is_exhausted from task_status) AND (select when_exhausted_mode from task_status) IN ('skip', 'skip-cascade') THEN 'handler_failed' + ELSE pgflow.step_states.skip_reason + END, + skipped_at = CASE + WHEN (select is_exhausted from task_status) AND (select when_exhausted_mode from task_status) IN ('skip', 'skip-cascade') THEN now() + ELSE pgflow.step_states.skipped_at + END, + -- Clear remaining_tasks when skipping (required by remaining_tasks_state_consistency constraint) + remaining_tasks = CASE + WHEN (select is_exhausted from task_status) AND (select when_exhausted_mode from task_status) IN ('skip', 'skip-cascade') THEN NULL + ELSE pgflow.step_states.remaining_tasks + END + FROM fail_or_retry_task + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + RETURNING pgflow.step_states.* +), +run_update AS ( + -- Update run status: only fail when when_exhausted='fail' and step was failed + UPDATE pgflow.runs + SET status = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN 'failed' + ELSE status + END, + failed_at = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN now() + ELSE NULL + END, + -- Decrement remaining_steps only on FIRST transition to skipped + -- (not when step was already skipped and a second task fails) + -- Uses PL/pgSQL variable captured before CTE chain + remaining_steps = CASE + WHEN (select status from maybe_fail_step) = 'skipped' + AND v_prev_step_status != 'skipped' + THEN pgflow.runs.remaining_steps - 1 + ELSE pgflow.runs.remaining_steps + END + WHERE pgflow.runs.run_id = fail_task.run_id + RETURNING pgflow.runs.status +) +SELECT + COALESCE((SELECT status = 'failed' FROM run_update), false), + COALESCE((SELECT status = 'failed' FROM maybe_fail_step), false), + COALESCE((SELECT status = 'skipped' FROM maybe_fail_step), false), + COALESCE((SELECT is_exhausted FROM task_status), false) +INTO v_run_failed, v_step_failed, v_step_skipped, v_task_exhausted; + + -- Capture when_exhausted mode for later skip handling + SELECT s.when_exhausted INTO v_when_exhausted + FROM pgflow.steps s +JOIN pgflow.runs r ON r.flow_slug = s.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug; + +-- Send broadcast event for step failure if the step was failed +IF v_task_exhausted AND v_step_failed THEN + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', fail_task.run_id, + 'step_slug', fail_task.step_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + concat('step:', fail_task.step_slug, ':failed'), + concat('pgflow:run:', fail_task.run_id), + false + ); +END IF; + +-- Handle step skipping (when_exhausted = 'skip' or 'skip-cascade') + IF v_task_exhausted AND v_step_skipped THEN + -- Lock-order invariant: always lock/update step_tasks before PGMQ queue rows. + -- requeue_stalled_tasks() uses the same order; archiving queue rows first + -- deadlocks the two transactions against each other. + -- Terminalize all still-active sibling task rows for the skipped step, + -- capturing their queue/message pairs for archival below. + FOR v_archive_batch IN + WITH skipped_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'skipped' + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ) + SELECT + st.queue_name, + ARRAY_AGG(st.message_id ORDER BY st.message_id) AS ids + FROM skipped_tasks st + WHERE st.message_id IS NOT NULL + GROUP BY st.queue_name + LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); + END LOOP; + + -- Send broadcast event for step skipped + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:skipped', + 'run_id', fail_task.run_id, + 'step_slug', fail_task.step_slug, + 'status', 'skipped', + 'skip_reason', 'handler_failed', + 'error_message', fail_task.error_message, + 'skipped_at', now() + ), + concat('step:', fail_task.step_slug, ':skipped'), + concat('pgflow:run:', fail_task.run_id), + false + ); + + -- For skip-cascade: cascade skip to all downstream dependents + IF v_when_exhausted = 'skip-cascade' THEN + PERFORM pgflow._cascade_force_skip_steps(fail_task.run_id, fail_task.step_slug, 'handler_failed'); + ELSE + -- For plain 'skip': decrement remaining_deps on dependent steps + -- (This mirrors the pattern in cascade_resolve_conditions.sql for when_unmet='skip') + SELECT flow_slug INTO v_flow_slug_for_deps + FROM pgflow.runs + WHERE pgflow.runs.run_id = fail_task.run_id; + + UPDATE pgflow.step_states AS child_state + SET remaining_deps = child_state.remaining_deps - 1, + -- If child is a map step and this skipped step is its only dependency, + -- set initial_tasks = 0 (skipped dep = empty array) + initial_tasks = CASE + WHEN child_step.step_type = 'map' AND child_step.deps_count = 1 THEN 0 + ELSE child_state.initial_tasks + END + FROM pgflow.deps AS dep + JOIN pgflow.steps AS child_step ON child_step.flow_slug = dep.flow_slug AND child_step.step_slug = dep.step_slug + WHERE child_state.run_id = fail_task.run_id + AND dep.flow_slug = v_flow_slug_for_deps + AND dep.dep_slug = fail_task.step_slug + AND child_state.step_slug = dep.step_slug; + + -- Evaluate conditions on newly-ready dependent steps + -- This must happen before cascade_complete_taskless_steps so that + -- skipped steps can set initial_tasks=0 for their map dependents + IF NOT pgflow.cascade_resolve_conditions(fail_task.run_id) THEN + -- Run was failed due to a condition with when_unmet='fail' + -- Archive the failed task's message before returning + PERFORM pgflow._archive_task_message(fail_task.run_id, fail_task.step_slug, fail_task.task_index); + -- Return the task row (API contract) + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; + END IF; + + -- Auto-complete taskless steps (e.g., map steps with initial_tasks=0 from skipped dep) + PERFORM pgflow.cascade_complete_taskless_steps(fail_task.run_id); + + -- Start steps that became ready after condition resolution and taskless completion + PERFORM pgflow.start_ready_steps(fail_task.run_id); + END IF; + + -- Try to complete the run (remaining_steps may now be 0) + PERFORM pgflow.maybe_complete_run(fail_task.run_id); +END IF; + +-- Send broadcast event for run failure if the run was failed +IF v_run_failed THEN + DECLARE + v_flow_slug text; + BEGIN + SELECT flow_slug INTO v_flow_slug FROM pgflow.runs WHERE pgflow.runs.run_id = fail_task.run_id; + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', fail_task.run_id, + 'flow_slug', v_flow_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', fail_task.run_id), + false + ); + END; +END IF; + +-- Terminalize unfinished tasks as cancelled when the run fails, then archive +-- their messages. Lock-order invariant: always lock/update step_tasks before +-- PGMQ queue rows. The culprit task is already terminal (failed or requeued by +-- fail_or_retry_task), so only unfinished queued/started siblings are cancelled. +IF v_run_failed THEN + FOR v_archive_batch IN + WITH cancelled_tasks AS ( + UPDATE pgflow.step_tasks AS task + SET status = 'cancelled' + WHERE task.run_id = fail_task.run_id + AND task.status IN ('queued', 'started') + RETURNING task.queue_name, task.message_id + ) + SELECT + ct.queue_name, + ARRAY_AGG(ct.message_id ORDER BY ct.message_id) AS ids + FROM cancelled_tasks ct + WHERE ct.message_id IS NOT NULL + GROUP BY ct.queue_name + LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); + END LOOP; +END IF; + +-- For queued tasks: delay the message for retry with exponential backoff +PERFORM ( + WITH retry_config AS ( + SELECT + COALESCE(s.opt_base_delay, f.opt_base_delay) AS base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN pgflow.runs r ON r.flow_slug = f.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug + ), + queued_tasks AS ( + SELECT + st.queue_name, + st.message_id, + pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay + FROM pgflow.step_tasks st + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'queued' + ) + SELECT pgmq.set_vt(qt.queue_name, qt.message_id, qt.calculated_delay) + FROM queued_tasks qt + WHERE EXISTS (SELECT 1 FROM queued_tasks) +); + +-- For failed tasks: archive the message grouped by the task's queue snapshot +FOR v_archive_batch IN + SELECT + st.queue_name, + ARRAY_AGG(st.message_id ORDER BY st.message_id) AS ids + FROM pgflow.step_tasks st + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'failed' + AND st.message_id IS NOT NULL + GROUP BY st.queue_name + HAVING COUNT(st.message_id) > 0 +LOOP + PERFORM pgmq.archive(v_archive_batch.queue_name, v_archive_batch.ids); +END LOOP; + +return query select * +from pgflow.step_tasks st +where st.run_id = fail_task.run_id + and st.step_slug = fail_task.step_slug + and st.task_index = fail_task.task_index; + +end; +$$; +-- Modify "start_flow" function +CREATE OR REPLACE FUNCTION "pgflow"."start_flow" ("flow_slug" text, "input" jsonb, "run_id" uuid DEFAULT NULL::uuid) RETURNS SETOF "pgflow"."runs" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_created_run pgflow.runs%ROWTYPE; + v_root_map_count int; +begin + +-- ========================================== +-- LOCK: Hold the concrete flow definition against deletion/recompilation +-- while this producer reads step definitions (#650). +-- ========================================== +perform 1 from pgflow.flows f +where f.flow_slug = start_flow.flow_slug +for key share; + +-- ========================================== +-- VALIDATION: Root map array input +-- ========================================== +WITH root_maps AS ( + SELECT step_slug + FROM pgflow.steps + WHERE steps.flow_slug = start_flow.flow_slug + AND steps.step_type = 'map' + AND steps.deps_count = 0 +) +SELECT COUNT(*) INTO v_root_map_count FROM root_maps; + +-- If we have root map steps, validate that input is an array +IF v_root_map_count > 0 THEN + -- First check for NULL (should be caught by NOT NULL constraint, but be defensive) + IF start_flow.input IS NULL THEN + RAISE EXCEPTION 'Flow % has root map steps but input is NULL', start_flow.flow_slug; + END IF; + + -- Then check if it's not an array + IF jsonb_typeof(start_flow.input) != 'array' THEN + RAISE EXCEPTION 'Flow % has root map steps but input is not an array (got %)', + start_flow.flow_slug, jsonb_typeof(start_flow.input); + END IF; +END IF; + +-- ========================================== +-- MAIN CTE CHAIN: Create run and step states +-- ========================================== +WITH + -- ---------- Gather flow metadata ---------- + flow_steps AS ( + SELECT steps.flow_slug, steps.step_slug, steps.step_type, steps.deps_count + FROM pgflow.steps + WHERE steps.flow_slug = start_flow.flow_slug + ), + -- ---------- Create run record ---------- + created_run AS ( + INSERT INTO pgflow.runs (run_id, flow_slug, input, remaining_steps) + VALUES ( + COALESCE(start_flow.run_id, gen_random_uuid()), + start_flow.flow_slug, + start_flow.input, + (SELECT count(*) FROM flow_steps) + ) + RETURNING * + ), + -- ---------- Create step states ---------- + -- Sets initial_tasks: known for root maps, NULL for dependent maps + created_step_states AS ( + INSERT INTO pgflow.step_states (flow_slug, run_id, step_slug, remaining_deps, initial_tasks) + SELECT + fs.flow_slug, + (SELECT created_run.run_id FROM created_run), + fs.step_slug, + fs.deps_count, + -- Updated logic for initial_tasks: + CASE + WHEN fs.step_type = 'map' AND fs.deps_count = 0 THEN + -- Root map: get array length from input + CASE + WHEN jsonb_typeof(start_flow.input) = 'array' THEN + jsonb_array_length(start_flow.input) + ELSE + 1 + END + WHEN fs.step_type = 'map' AND fs.deps_count > 0 THEN + -- Dependent map: unknown until dependencies complete + NULL + ELSE + -- Single steps: always 1 task + 1 + END + FROM flow_steps fs + ) +SELECT * FROM created_run INTO v_created_run; + +-- ========================================== +-- POST-CREATION ACTIONS +-- ========================================== + +-- ---------- Broadcast run:started event ---------- +PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:started', + 'run_id', v_created_run.run_id, + 'flow_slug', v_created_run.flow_slug, + 'input', v_created_run.input, + 'status', 'started', + 'remaining_steps', v_created_run.remaining_steps, + 'started_at', v_created_run.started_at + ), + 'run:started', + concat('pgflow:run:', v_created_run.run_id), + false +); + +-- ---------- Evaluate conditions on ready steps ---------- +-- Skip steps with unmet conditions, propagate to dependents +IF NOT pgflow.cascade_resolve_conditions(v_created_run.run_id) THEN + -- Run was failed due to a condition with when_unmet='fail' + RETURN QUERY SELECT * FROM pgflow.runs where pgflow.runs.run_id = v_created_run.run_id; + RETURN; +END IF; + +-- ---------- Complete taskless steps ---------- +-- Handle empty array maps that should auto-complete +PERFORM pgflow.cascade_complete_taskless_steps(v_created_run.run_id); + +-- ---------- Start initial steps ---------- +-- Start root steps (those with no dependencies) +PERFORM pgflow.start_ready_steps(v_created_run.run_id); + +-- ---------- Check for run completion ---------- +-- If cascade completed all steps (zero-task flows), finalize the run +PERFORM pgflow.maybe_complete_run(v_created_run.run_id); + +RETURN QUERY SELECT * FROM pgflow.runs where pgflow.runs.run_id = v_created_run.run_id; + +end; +$$; +-- Create "claim_tasks" function +CREATE FUNCTION "pgflow"."claim_tasks" ("queue_name" text, "flow_slug" text, "message_ids" bigint[], "worker_id" uuid) RETURNS jsonb LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_qtable text := pgmq.format_table_name(queue_name, 'q'); + v_worker record; + v_flow_exists boolean; + v_route_violation text; + v_ids bigint[]; + v_bodies jsonb; + v_classification record; + v_claim_ids bigint[]; + v_defer_ids bigint[]; + v_terminal_ids bigint[]; + v_foreign_ids bigint[]; + v_fatal boolean := false; + v_errors jsonb := '[]'::jsonb; + v_warnings jsonb := '[]'::jsonb; + v_claimed_tasks jsonb; + v_body jsonb; + v_body_flow text; + v_body_run text; + v_body_step text; + v_body_index text; + v_run_valid boolean; + v_index_valid boolean; + v_reason text; + v_addr record; + v_vt_offsets int[]; + v_updated_count int; + v_claimed_count int; +begin + -- Deduplicate the read batch + select array_agg(distinct id order by id) into v_ids + from unnest(message_ids) as u(id) + where id is not null; + + if v_ids is null then + return jsonb_build_object('status', 'ok', 'tasks', '[]'::jsonb, 'warnings', '[]'::jsonb); + end if; + + -- ========================================== + -- SUBSCRIPTION VALIDATION (before body use) + -- ========================================== + select w.queue_name, w.function_name + into v_worker + from pgflow.workers w + where w.worker_id = claim_tasks.worker_id; + + if v_worker is null then + -- Missing registration supplies no invented function to pause; one + -- diagnostic per batch member keeps every message ID non-null + select coalesce(jsonb_agg( + jsonb_build_object('queue_name', queue_name, 'message_id', id::text, 'reason', 'invalid_subscription') + order by id), '[]'::jsonb) + into v_errors + from unnest(v_ids) as u(id); + perform pgflow.set_vt_batch(queue_name, v_ids, array_fill(0, array[cardinality(v_ids)])); + return jsonb_build_object('status', 'fatal', 'tasks', '[]'::jsonb, 'errors', v_errors); + end if; + + if v_worker.queue_name is distinct from queue_name then + select coalesce(jsonb_agg( + jsonb_build_object('queue_name', queue_name, 'message_id', id::text, 'reason', 'invalid_subscription') + order by id), '[]'::jsonb) + into v_errors + from unnest(v_ids) as u(id); + perform pgflow.set_vt_batch(queue_name, v_ids, array_fill(0, array[cardinality(v_ids)])); + update pgflow.worker_functions wf + set enabled = false, updated_at = clock_timestamp() + where wf.function_name = v_worker.function_name + and wf.start_mode = 'http'; + return jsonb_build_object('status', 'fatal', 'tasks', '[]'::jsonb, 'errors', v_errors); + end if; + + -- ========================================== + -- ROUTE VALIDATION + -- ========================================== + select exists(select 1 from pgflow.flows f where f.flow_slug = claim_tasks.flow_slug) + into v_flow_exists; + + select s.step_slug into v_route_violation + from pgflow.steps s + where s.flow_slug = claim_tasks.flow_slug + and s.queue_name is distinct from claim_tasks.queue_name + limit 1; + + if not v_flow_exists + or claim_tasks.queue_name is distinct from lower(claim_tasks.flow_slug) + or v_route_violation is not null then + select coalesce(jsonb_agg( + jsonb_build_object('queue_name', queue_name, 'message_id', id::text, 'reason', 'wrong_route') + order by id), '[]'::jsonb) + into v_errors + from unnest(v_ids) as u(id); + perform pgflow.set_vt_batch(queue_name, v_ids, array_fill(0, array[cardinality(v_ids)])); + update pgflow.worker_functions wf + set enabled = false, updated_at = clock_timestamp() + where wf.function_name = v_worker.function_name + and wf.start_mode = 'http'; + return jsonb_build_object('status', 'fatal', 'tasks', '[]'::jsonb, 'errors', v_errors); + end if; + + -- ========================================== + -- READ-ONLY DISCOVERY + -- ========================================== + -- Read the bodies once (ordinary SQL error if the physical table is gone). + -- Bodies are immutable in PGMQ, so reading them before the locks is safe. + -- Envelope inspection is identity classification only; application input + -- JSON is never validated here. + execute format( + 'select coalesce(jsonb_agg(jsonb_build_object(''msg_id'', q.msg_id, ''message'', q.message)), ''[]''::jsonb) + from pgmq.%I q where q.msg_id = any($1)', + v_qtable + ) into v_bodies using v_ids; + + -- ========================================== + -- ORDERED LOCKS + -- ========================================== + -- Parent runs, step states, task rows, then queue rows in message-ID + -- order: the established parent-first order shared with every other + -- runtime operation. + perform 1 + from pgflow.runs r + where r.run_id in ( + select t.run_id from pgflow.step_tasks t + where t.queue_name = claim_tasks.queue_name and t.message_id = any(v_ids) + ) + order by r.run_id + for update; + + perform 1 + from pgflow.step_states ss + where ss.run_id in ( + select t.run_id from pgflow.step_tasks t + where t.queue_name = claim_tasks.queue_name and t.message_id = any(v_ids) + ) + order by ss.run_id, ss.step_slug + for update; + + perform 1 + from pgflow.step_tasks t + where t.queue_name = claim_tasks.queue_name and t.message_id = any(v_ids) + order by t.run_id, t.step_slug, t.task_index + for update; + + execute format( + 'select q.msg_id from pgmq.%I q where q.msg_id = any($1) order by q.msg_id for update', + v_qtable + ) using v_ids; + + -- ========================================== + -- CLASSIFICATION (under the locks above) + -- ========================================== + for v_classification in + with pairs as ( + select + t.run_id, + t.step_slug, + t.task_index, + t.message_id, + t.status as task_status, + t.permanently_stalled_at, + t.started_at, + r.status as run_status, + ss.status as step_status + from pgflow.step_tasks t + left join pgflow.runs r on r.run_id = t.run_id + left join pgflow.step_states ss on ss.run_id = t.run_id and ss.step_slug = t.step_slug + where t.queue_name = claim_tasks.queue_name + and t.message_id = any(v_ids) + ) + select + u.id as msg_id, + p.run_id as task_run_id, + p.step_slug as task_step, + p.task_index as task_index, + p.task_status, + p.permanently_stalled_at, + p.started_at, + p.run_status, + p.step_status, + b.msg -> 'message' as body + from unnest(v_ids) as u(id) + left join pairs p on p.message_id = u.id + left join lateral jsonb_array_elements(v_bodies) b(msg) on (b.msg->>'msg_id')::bigint = u.id + order by u.id + loop + v_body := v_classification.body; + v_body_flow := v_body ->> 'flow_slug'; + v_body_run := v_body ->> 'run_id'; + v_body_step := v_body ->> 'step_slug'; + v_body_index := v_body ->> 'task_index'; + -- Safe-cast gates: only well-formed components may identify work + v_run_valid := v_body_run is not null + and v_body_run ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'; + v_index_valid := v_body_index is not null + and v_body_index ~ '^[0-9]{1,9}$'; + + if v_classification.task_run_id is not null then + -- ========================================== + -- EXACT DURABLE PAIR: the pair wins over the envelope. Malformed or + -- absent components never contradict it; only a VALID address that + -- positively identifies different work is fatal. + -- ========================================== + if v_body_flow is not null and v_body_flow is distinct from claim_tasks.flow_slug then + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'wrong_route'); + v_fatal := true; + elsif v_run_valid and v_body_run::uuid is distinct from v_classification.task_run_id then + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'unsupported_work'); + v_fatal := true; + elsif v_body_step is not null and v_body_step is distinct from v_classification.task_step then + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'unsupported_work'); + v_fatal := true; + elsif v_index_valid and (v_body_index)::int is distinct from v_classification.task_index then + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'unsupported_work'); + v_fatal := true; + elsif v_classification.task_status in ('completed', 'failed', 'skipped', 'cancelled') then + -- Terminal task: idempotent archive (archive ignores already-archived) + v_terminal_ids := array_append(v_terminal_ids, v_classification.msg_id); + elsif v_classification.permanently_stalled_at is not null then + -- Permanent stall: preserve status/history, archive idempotently + v_terminal_ids := array_append(v_terminal_ids, v_classification.msg_id); + elsif v_classification.task_status = 'started' + and v_classification.run_status = 'started' + and v_classification.step_status = 'started' then + v_defer_ids := array_append(v_defer_ids, v_classification.msg_id); + elsif v_classification.task_status = 'queued' + and v_classification.run_status = 'started' + and v_classification.step_status = 'started' then + v_claim_ids := array_append(v_claim_ids, v_classification.msg_id); + else + -- Active task with incompatible parent state + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'unsupported_work'); + v_fatal := true; + end if; + else + -- ========================================== + -- NO EXACT PAIR: key presence decides. Present-null identity keys are + -- pgflow-shaped evidence, not foreign silence. + -- ========================================== + if v_body is null + or not (v_body ? 'flow_slug' or v_body ? 'run_id' + or v_body ? 'step_slug' or v_body ? 'task_index') then + -- Clearly foreign: archive and warn (no bodies in diagnostics) + v_foreign_ids := array_append(v_foreign_ids, v_classification.msg_id); + v_warnings := v_warnings || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', 'foreign_message'); + else + -- Apparently genuine or ambiguous pgflow work without an exact pair: + -- fatal. Inspect valid address components to name the reason; a + -- malformed or null-valued component stays unsupported work. + v_reason := 'unsupported_work'; + if v_body_flow is not null and v_body_flow is distinct from claim_tasks.flow_slug then + v_reason := 'wrong_route'; + elsif v_run_valid then + select r.flow_slug into v_addr + from pgflow.runs r + where r.run_id = v_body_run::uuid; + if not found then + v_reason := 'unsupported_work'; + elsif v_addr.flow_slug is distinct from claim_tasks.flow_slug then + v_reason := 'wrong_route'; + elsif v_body_step is not null and v_index_valid then + -- A complete valid address that belongs to this flow but to a + -- task in another queue is wrong-route work + if exists ( + select 1 + from pgflow.step_tasks t + where t.run_id = v_body_run::uuid + and t.step_slug = v_body_step + and t.task_index = (v_body_index)::int + and t.queue_name is distinct from claim_tasks.queue_name + ) then + v_reason := 'wrong_route'; + end if; + end if; + end if; + v_errors := v_errors || jsonb_build_object( + 'queue_name', queue_name, 'message_id', v_classification.msg_id::text, 'reason', v_reason); + v_fatal := true; + end if; + end if; + end loop; + + -- A live deferred task whose queue message disappeared is an ordinary + -- integrity/visibility failure with total rollback (#656 protection) + if not v_fatal and v_defer_ids is not null then + perform 1 + from unnest(v_defer_ids) as d(id) + where not exists ( + select 1 from jsonb_array_elements(v_bodies) b(msg) where (b.msg->>'msg_id')::bigint = d.id + ); + if found then + raise exception 'claim_tasks(): deferred live task message is missing from queue %', queue_name; + end if; + end if; + + -- ========================================== + -- FATAL BRANCH: reset the whole read batch, pause, return normally + -- ========================================== + if v_fatal then + perform pgflow.set_vt_batch( + queue_name, v_ids, + array_fill(0, array[cardinality(v_ids)]) + ); + update pgflow.worker_functions wf + set enabled = false, updated_at = clock_timestamp() + where wf.function_name = v_worker.function_name + and wf.start_mode = 'http'; + return jsonb_build_object('status', 'fatal', 'tasks', '[]'::jsonb, 'errors', v_errors); + end if; + + -- ========================================== + -- NONFATAL BRANCH (under the locks above) + -- ========================================== + -- Defer started tasks to their existing recovery deadline (effective + -- timeout + 30s from started_at); repeated reads never move that deadline + if v_defer_ids is not null then + with deadlines as ( + select + t.message_id, + greatest(0, ceil(extract(epoch from ( + t.started_at + + make_interval(secs => coalesce(s.opt_timeout, f.opt_timeout) + 30) + - clock_timestamp() + )))::integer) as vt_delay + from pgflow.step_tasks t + join pgflow.runs r on r.run_id = t.run_id + join pgflow.flows f on f.flow_slug = r.flow_slug + join pgflow.steps s on s.flow_slug = r.flow_slug and s.step_slug = t.step_slug + where t.queue_name = claim_tasks.queue_name + and t.message_id = any(v_defer_ids) + ) + select array_agg(d.vt_delay order by d.message_id) into v_vt_offsets + from (select message_id from unnest(v_defer_ids) as x(message_id)) ids + join deadlines d on d.message_id = ids.message_id; + + perform pgflow.set_vt_batch(queue_name, v_defer_ids, v_vt_offsets); + end if; + + -- Idempotent archival of terminal and clearly foreign groups after task locks + if v_terminal_ids is not null then + perform pgmq.archive(queue_name, v_terminal_ids); + end if; + if v_foreign_ids is not null then + perform pgmq.archive(queue_name, v_foreign_ids); + end if; + + -- ========================================== + -- CLAIM: guarded update; the returned tasks are built ONLY from the + -- UPDATE ... RETURNING rows, never from a re-query by message ID. + -- ========================================== + if v_claim_ids is not null then + with + updated as ( + update pgflow.step_tasks task + set + attempts_count = attempts_count + 1, + status = 'started', + started_at = now(), + last_worker_id = claim_tasks.worker_id + where task.queue_name = claim_tasks.queue_name + and task.message_id = any(v_claim_ids) + and task.status = 'queued' + returning + task.flow_slug, + task.run_id, + task.step_slug, + task.task_index, + task.queue_name, + task.message_id + ), + runs as ( + select r.run_id, r.input + from pgflow.runs r + where r.run_id in (select run_id from updated) + ), + deps as ( + select + st.run_id, + st.step_slug, + dep.dep_slug, + dep_state.output as dep_output + from updated st + join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug + join pgflow.step_states dep_state on + dep_state.run_id = st.run_id and + dep_state.step_slug = dep.dep_slug and + dep_state.status = 'completed' + ), + deps_outputs as ( + select + d.run_id, + d.step_slug, + jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output, + count(*) as dep_count + from deps d + group by d.run_id, d.step_slug + ), + timeouts as ( + select + u.message_id, + coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay + from updated u + join pgflow.flows flow on flow.flow_slug = u.flow_slug + join pgflow.steps step on step.flow_slug = u.flow_slug and step.step_slug = u.step_slug + ), + visibility_reset as ( + select pgflow.set_vt_batch( + claim_tasks.queue_name, + (select array_agg(t.message_id order by t.message_id) from timeouts t), + (select array_agg(t.vt_delay order by t.message_id) from timeouts t) + ) + ), + counts as ( + select + (select count(*) from visibility_reset) as updated_count, + (select count(*) from updated) as claimed_count + ) + select + c.updated_count, + c.claimed_count, + coalesce(( + select jsonb_agg( + jsonb_build_object( + 'flow_slug', st.flow_slug, + 'run_id', st.run_id, + 'step_slug', st.step_slug, + 'task_index', st.task_index, + 'queue_name', st.queue_name, + 'msg_id', st.message_id::text, + 'input', + case + when step.step_type = 'map' then + case + when step.deps_count = 0 then jsonb_array_element(r.input, st.task_index) + else (select jsonb_array_element(value, st.task_index) from jsonb_each(dep_out.deps_output) limit 1) + end + else coalesce(dep_out.deps_output, '{}'::jsonb) + end, + 'flow_input', + case + when step.step_type != 'map' and step.deps_count = 0 then r.input + else null + end + ) + order by st.message_id + ) + from updated st + join runs r on st.run_id = r.run_id + join pgflow.steps step on + step.flow_slug = st.flow_slug and + step.step_slug = st.step_slug + left join deps_outputs dep_out on + dep_out.run_id = st.run_id and + dep_out.step_slug = st.step_slug + ), '[]'::jsonb) + into v_updated_count, v_claimed_count, v_claimed_tasks + from counts c; + + -- Guard completeness: every task the guarded update actually claimed + -- must have its visibility extension; otherwise the whole statement + -- fails atomically (#656). A guarded update that claims fewer rows + -- than classified (e.g. a concurrent skip winning the row lock, #638) + -- simply returns only the claimed rows. + if v_updated_count is distinct from v_claimed_count then + raise exception 'claim_tasks(): visibility updated % of % claimed messages', + v_updated_count, v_claimed_count; + end if; + else + v_claimed_tasks := '[]'::jsonb; + end if; + + return jsonb_build_object( + 'status', 'ok', + 'tasks', v_claimed_tasks, + 'warnings', v_warnings + ); +end; +$$; +-- Modify "start_tasks" function +CREATE OR REPLACE FUNCTION "pgflow"."start_tasks" ("flow_slug" text, "msg_ids" bigint[], "worker_id" uuid) RETURNS SETOF "pgflow"."step_task_record" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_result jsonb; + v_task jsonb; +begin + select pgflow.claim_tasks( + lower(start_tasks.flow_slug), + start_tasks.flow_slug, + start_tasks.msg_ids, + start_tasks.worker_id + ) into v_result; + + if v_result ->> 'status' = 'fatal' then + raise warning 'start_tasks(): fatal claim classification for flow % (no tasks started)', start_tasks.flow_slug; + return; + end if; + + for v_task in select * from jsonb_array_elements(v_result -> 'tasks') + loop + return query + select + (v_task ->> 'flow_slug')::text, + (v_task ->> 'run_id')::uuid, + (v_task ->> 'step_slug')::text, + v_task -> 'input', + (v_task ->> 'msg_id')::bigint, + (v_task ->> 'task_index')::int, + case + when jsonb_typeof(v_task -> 'flow_input') is distinct from 'null' + then v_task -> 'flow_input' + else null + end; + end loop; +end; +$$; +-- Create "ensure_flow_compiled" function +CREATE FUNCTION "pgflow"."ensure_flow_compiled" ("flow_slug" text, "shape" jsonb, "worker_protocol" jsonb) RETURNS jsonb LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_lock_key int; + v_flow_exists boolean; + v_db_shape jsonb; + v_differences text[]; + v_is_local boolean; + v_canonical_queue text := lower(ensure_flow_compiled.flow_slug); +BEGIN + -- Queue-capable startup handshake (#650): the required third argument has + -- no default and no fallback wrapper. Version 1 identifies the queue-aware + -- startup/claim semantics. Reject missing/non-object/wrong-version values + -- before any definition mutation. + IF jsonb_typeof(worker_protocol) IS DISTINCT FROM 'object' + OR worker_protocol -> 'version' IS DISTINCT FROM '1'::jsonb THEN + RAISE EXCEPTION 'Queue-capable worker protocol version 1 is required'; + END IF; + -- Generate lock key from the canonical slug (deterministic hash). + -- Case aliases share the lock so concurrent compilation of 'Orders' and + -- 'orders' serializes against each other. + v_lock_key := hashtext(lower(ensure_flow_compiled.flow_slug)); + + -- Acquire transaction-level advisory lock + -- Serializes concurrent compilation attempts for same flow + PERFORM pg_advisory_xact_lock(1, v_lock_key); + + -- 1. Check if flow exists + SELECT EXISTS(SELECT 1 FROM pgflow.flows AS flow WHERE flow.flow_slug = ensure_flow_compiled.flow_slug) + INTO v_flow_exists; + + -- 2. If flow missing: compile (both environments) + IF NOT v_flow_exists THEN + PERFORM pgflow._create_flow_from_shape(ensure_flow_compiled.flow_slug, ensure_flow_compiled.shape); + RETURN jsonb_build_object( + 'status', 'compiled', + 'differences', '[]'::jsonb, + 'protocol_version', 1, + 'queue_name', v_canonical_queue + ); + END IF; + + -- 3. Get current shape from DB + v_db_shape := pgflow._get_flow_shape(ensure_flow_compiled.flow_slug); + + -- 4. Compare shapes + v_differences := pgflow._compare_flow_shapes(ensure_flow_compiled.shape, v_db_shape); + + -- 5. If shapes match: inspect the persisted route/resources before + -- returning verified. A shape match alone does not prove a valid queue. + IF array_length(v_differences, 1) IS NULL THEN + PERFORM pgflow._inspect_generated_queue( + ensure_flow_compiled.flow_slug, + v_canonical_queue, + true + ); + RETURN jsonb_build_object( + 'status', 'verified', + 'differences', '[]'::jsonb, + 'protocol_version', 1, + 'queue_name', v_canonical_queue + ); + END IF; + + -- 6. Shapes differ - auto-detect environment via is_local() + v_is_local := pgflow.is_local(); + + -- Local mode is the only destructive branch; production mismatches never + -- delete data and return mismatch so worker startup fails. + IF v_is_local THEN + -- Preflight the entire replacement shape before any deletion so a bad + -- late step cannot destroy old data. Deletion follows the established + -- runtime-before-metadata lock order. + PERFORM pgflow._validate_flow_shape(ensure_flow_compiled.flow_slug, ensure_flow_compiled.shape); + + PERFORM pgflow.delete_flow_and_data(ensure_flow_compiled.flow_slug); + PERFORM pgflow._create_flow_from_shape(ensure_flow_compiled.flow_slug, ensure_flow_compiled.shape); + RETURN jsonb_build_object( + 'status', 'recompiled', + 'differences', to_jsonb(v_differences), + 'protocol_version', 1, + 'queue_name', v_canonical_queue + ); + ELSE + -- Fail in production + RETURN jsonb_build_object( + 'status', 'mismatch', + 'differences', to_jsonb(v_differences) + ); + END IF; +END; +$$; +-- Drop "add_step" function +DROP FUNCTION "pgflow"."add_step" (text, text, text[], integer, integer, integer, integer, text, jsonb, jsonb, text, text); +-- Drop "ensure_flow_compiled" function +DROP FUNCTION "pgflow"."ensure_flow_compiled" (text, jsonb); +COMMIT; diff --git a/pkgs/core/supabase/migrations/atlas.sum b/pkgs/core/supabase/migrations/atlas.sum index 34ed859b5..30745aac4 100644 --- a/pkgs/core/supabase/migrations/atlas.sum +++ b/pkgs/core/supabase/migrations/atlas.sum @@ -1,4 +1,4 @@ -h1:9ItIcsLHe6xhjbaHTXXGIUkY4L65yjgul6QLoVfawo8= +h1:lrZuxZ6Ez2BA/Bd7JO7J1uZNXoWMVvDhNFHiVZAog6s= 20250429164909_pgflow_initial.sql h1:I3n/tQIg5Q5nLg7RDoU3BzqHvFVjmumQxVNbXTPG15s= 20250517072017_pgflow_fix_poll_for_tasks_to_use_separate_statement_for_polling.sql h1:wTuXuwMxVniCr3ONCpodpVWJcHktoQZIbqMZ3sUHKMY= 20250609105135_pgflow_add_start_tasks_and_started_status.sql h1:ggGanW4Wyt8Kv6TWjnZ00/qVb3sm+/eFVDjGfT8qyPg= @@ -22,3 +22,4 @@ h1:9ItIcsLHe6xhjbaHTXXGIUkY4L65yjgul6QLoVfawo8= 20260607175525_pgflow_worker_start_mode.sql h1:PFAfoGaHe5stKF7YAFg6AqBxmRisqDvV60vVpnnVdBE= 20260904095427_pgflow_task_lifecycle_hardening.sql h1:27b0BfBcQxeu5XSqVtQYvDCRzTsvLqzS/5hx14/2VyM= 20260907082520_pgflow_remove_legacy_flow_compilation.sql h1:LNFDz+ZZlWb19FmWNPK57eiD+FXySMbStVij8MTSvDw= +20260910104929_pgflow_persist_queue.sql h1:YpED3BFxlYLy5bcldqTzsbSuiTWXHmgYNzx8F+oXq/8= diff --git a/pkgs/core/supabase/seed.sql b/pkgs/core/supabase/seed.sql index 6b3f1ecad..8bdb179e7 100644 --- a/pkgs/core/supabase/seed.sql +++ b/pkgs/core/supabase/seed.sql @@ -14,6 +14,7 @@ BEGIN DELETE FROM pgflow.steps; DELETE FROM pgflow.flows; DELETE FROM pgflow.worker_functions; + DELETE FROM pgflow.workers; -- Also clear the realtime.messages table if it exists BEGIN @@ -73,7 +74,7 @@ create or replace function pgflow_tests.ensure_worker( function_name text default 'test_worker' ) returns uuid as $$ INSERT INTO pgflow.workers (worker_id, queue_name, function_name, last_heartbeat_at) - VALUES (worker_uuid, queue_name, function_name, now()) + VALUES (worker_uuid, lower(queue_name), function_name, now()) ON CONFLICT (worker_id) DO UPDATE SET last_heartbeat_at = now(), queue_name = EXCLUDED.queue_name, @@ -93,18 +94,19 @@ create or replace function pgflow_tests.read_and_start( ) returns setof pgflow.step_task_record language sql as $$ - -- 1. make sure the worker exists / update its heartbeat + -- 1. make sure the worker exists / update its heartbeat (canonical queue; + -- the concrete flow argument stays exact) (#650) WITH w AS ( SELECT pgflow_tests.ensure_worker( - queue_name => flow_slug, + queue_name => lower(flow_slug), worker_uuid => worker_uuid, function_name => function_name ) AS wid ), - -- 2. read messages from the queue + -- 2. read messages from the queue (canonical physical route) msgs AS ( SELECT * - FROM pgmq.read_with_poll(flow_slug, vt, qty, 1, 50) + FROM pgmq.read_with_poll(lower(flow_slug), vt, qty, 1, 50) LIMIT qty ), -- 3. collect their msg_ids @@ -191,10 +193,10 @@ BEGIN q.message, extract(epoch from (q.vt - q.enqueued_at))::int as vt_seconds FROM pgmq.%s q - JOIN pgflow.step_tasks st ON st.message_id = q.msg_id + JOIN pgflow.step_tasks st ON st.queue_name = lower($2) AND st.message_id = q.msg_id WHERE st.step_slug = $1', qtable); - RETURN QUERY EXECUTE query USING step_slug; + RETURN QUERY EXECUTE query USING step_slug, queue_name; END; $$; diff --git a/pkgs/core/supabase/tests/_shared/delete_flow_and_data.sql.raw b/pkgs/core/supabase/tests/_shared/delete_flow_and_data.sql.raw deleted file mode 100644 index c2b8391ee..000000000 --- a/pkgs/core/supabase/tests/_shared/delete_flow_and_data.sql.raw +++ /dev/null @@ -1,24 +0,0 @@ -/** - * Deletes a flow and all its associated data. - * WARNING: This is destructive and should only be used during development. - * - * @param flow_slug - The slug of the flow to delete - */ -create or replace function pgflow.delete_flow_and_data( - flow_slug TEXT -) returns void language plpgsql as $$ -BEGIN - -- Drop queue and archive table - PERFORM pgmq.drop_queue(delete_flow_and_data.flow_slug); - - -- Delete all associated data in the correct order - DELETE FROM pgflow.step_tasks WHERE step_tasks.flow_slug = delete_flow_and_data.flow_slug; - DELETE FROM pgflow.step_states WHERE step_states.flow_slug = delete_flow_and_data.flow_slug; - DELETE FROM pgflow.runs WHERE runs.flow_slug = delete_flow_and_data.flow_slug; - DELETE FROM pgflow.deps WHERE deps.flow_slug = delete_flow_and_data.flow_slug; - DELETE FROM pgflow.steps WHERE steps.flow_slug = delete_flow_and_data.flow_slug; - DELETE FROM pgflow.flows WHERE flows.flow_slug = delete_flow_and_data.flow_slug; - - RAISE NOTICE 'Flow % and all associated data has been deleted', delete_flow_and_data.flow_slug; -END -$$; \ No newline at end of file diff --git a/pkgs/core/supabase/tests/_shared/prune_data_older_than.sql.raw b/pkgs/core/supabase/tests/_shared/prune_data_older_than.sql.raw index b68b823ab..2930cdc62 100644 --- a/pkgs/core/supabase/tests/_shared/prune_data_older_than.sql.raw +++ b/pkgs/core/supabase/tests/_shared/prune_data_older_than.sql.raw @@ -11,52 +11,159 @@ * * WARNING: Ensure retention_interval is longer than your longest start_delay to avoid * deleting tasks before they have a chance to execute. + * + * #650: queue-aware version. Active-message cleanup groups by the task's + * queue_name snapshot (never by the denormalized flow slug), archive pruning + * enumerates persisted definition routes (including the empty compiled plain + * default), and task rows are locked/deleted before their queue rows. + * Lock order matches delete_flow_and_data: retained flow definitions are + * locked (FOR KEY SHARE, ordered) before any runtime row, then expired + * parent runs, their step states, and the complete affected task set - + * including tasks of still-active runs whose last_worker_id will be + * FK-set NULL by the worker deletion below - before any queue row is + * touched. After all runtime locks and before any queue access, every + * route this pass touches (task snapshots plus archive routes) is + * validated with _inspect_generated_queue(flow_slug, queue_name, true), + * which also checks metadata ambiguity, physical shape, sequence + * ownership, and extension membership under the pgmq.meta fence; a + * corrupted or replaced queue stops pruning instead of letting pgflow + * delete from a queue it does not own. */ create or replace function pgflow.prune_data_older_than( retention_interval INTERVAL ) returns void language plpgsql as $$ DECLARE cutoff_timestamp TIMESTAMPTZ := now() - retention_interval; - flow_record RECORD; + batch_record RECORD; + route_record RECORD; + bad_route RECORD; + validate_route RECORD; + archive_queue TEXT; archive_table TEXT; dynamic_sql TEXT; BEGIN - -- Delete old worker records + -- Lock retained flow definitions first (before any runtime row), the same + -- definition-first order delete_flow_and_data uses. Taking these locks + -- after runtime locks could deadlock with flow deletion. + PERFORM 1 FROM pgflow.flows f + ORDER BY f.flow_slug + FOR KEY SHARE; + + -- Validate every persisted route as the flow's canonical generated queue + -- before any queue access: a corrupted steps.queue_name must never direct + -- pruning into a queue pgflow does not own. + SELECT f.flow_slug, s.step_slug, s.queue_name INTO bad_route + FROM pgflow.flows f + JOIN pgflow.steps s ON s.flow_slug = f.flow_slug + WHERE s.queue_name IS DISTINCT FROM lower(f.flow_slug) + ORDER BY f.flow_slug, s.step_slug + LIMIT 1; + IF bad_route IS NOT NULL THEN + RAISE EXCEPTION 'prune_data_older_than(): flow "%" step "%" persists non-canonical route "%" (expected "%"); fix the corrupted definition before pruning', + bad_route.flow_slug, bad_route.step_slug, bad_route.queue_name, lower(bad_route.flow_slug); + END IF; + + -- Lock parent runs of every task this pass will touch, then their step + -- states, then the complete affected task set. The task set includes + -- tasks of still-active runs whose last_worker_id references a worker + -- that will be deleted below (the FK ON DELETE SET NULL update touches + -- those task rows after this point, so they must already be locked). + PERFORM 1 FROM pgflow.runs r + WHERE ( + (r.completed_at IS NOT NULL AND r.completed_at < cutoff_timestamp) OR + (r.failed_at IS NOT NULL AND r.failed_at < cutoff_timestamp) + ) + ORDER BY r.run_id + FOR UPDATE; + + PERFORM 1 FROM pgflow.step_states ss + WHERE ss.run_id IN ( + SELECT r.run_id FROM pgflow.runs r + WHERE ( + (r.completed_at IS NOT NULL AND r.completed_at < cutoff_timestamp) OR + (r.failed_at IS NOT NULL AND r.failed_at < cutoff_timestamp) + ) + ) + ORDER BY ss.run_id, ss.step_slug + FOR UPDATE; + + PERFORM 1 FROM pgflow.step_tasks t + WHERE t.run_id IN ( + SELECT r.run_id FROM pgflow.runs r + WHERE ( + (r.completed_at IS NOT NULL AND r.completed_at < cutoff_timestamp) OR + (r.failed_at IS NOT NULL AND r.failed_at < cutoff_timestamp) + ) + ) OR ( + t.last_worker_id IS NOT NULL + AND t.last_worker_id IN ( + SELECT w.worker_id FROM pgflow.workers w + WHERE w.last_heartbeat_at < cutoff_timestamp + ) + ) + ORDER BY t.run_id, t.step_slug, t.task_index + FOR UPDATE; + + -- Delete old worker records (FK sets task.last_worker_id NULL on rows that + -- are already in the locked set above; no new task locks are taken after + -- queue access begins) DELETE FROM pgflow.workers WHERE last_heartbeat_at < cutoff_timestamp; - -- Delete PGMQ messages from active queues BEFORE deleting step_tasks - -- This prevents orphaned messages that would appear after tasks are deleted - FOR flow_record IN - SELECT - r.flow_slug, - ARRAY_AGG(st.message_id) FILTER (WHERE st.message_id IS NOT NULL) as message_ids - FROM pgflow.runs r - JOIN pgflow.step_tasks st ON st.run_id = r.run_id + -- After all runtime locks and before any queue access, validate every + -- route this pass will touch with the shared ownership inspector (#650 + -- review): the queue snapshots of tasks about to be deleted plus every + -- persisted archive route (including empty compiled plain defaults). The + -- inspector rejects metadata ambiguity, incomplete or malformed physical + -- shape (missing columns among them), sequence ownership, and extension + -- membership, so a dropped-and-recreated external queue with the same + -- canonical name stops pruning instead of being deleted from. It takes + -- the pgmq.meta topology fence here, after runtime locks, matching + -- delete_flow_and_data's lock order. + FOR validate_route IN + SELECT DISTINCT t.flow_slug, t.queue_name + FROM pgflow.step_tasks t + JOIN pgflow.runs r ON r.run_id = t.run_id WHERE ( (r.completed_at IS NOT NULL AND r.completed_at < cutoff_timestamp) OR (r.failed_at IS NOT NULL AND r.failed_at < cutoff_timestamp) ) - GROUP BY r.flow_slug + UNION + SELECT f.flow_slug, COALESCE(s.queue_name, lower(f.flow_slug)) + FROM pgflow.flows f + LEFT JOIN pgflow.steps s ON s.flow_slug = f.flow_slug LOOP - -- Delete messages in batch (pgmq.delete ignores non-existent messages) - IF flow_record.message_ids IS NOT NULL AND array_length(flow_record.message_ids, 1) > 0 THEN - PERFORM pgmq.delete(flow_record.flow_slug, flow_record.message_ids); - END IF; + PERFORM pgflow._inspect_generated_queue( + validate_route.flow_slug, validate_route.queue_name, true + ); END LOOP; - -- Delete ALL step_tasks for old runs (regardless of individual task status) - -- This fixes FK constraint violation when deleting runs with unexecuted steps - DELETE FROM pgflow.step_tasks - WHERE run_id IN ( - SELECT run_id FROM pgflow.runs - WHERE ( - (completed_at IS NOT NULL AND completed_at < cutoff_timestamp) OR - (failed_at IS NOT NULL AND failed_at < cutoff_timestamp) + -- Delete PGMQ messages from active queues BEFORE deleting step_tasks, using + -- each task's queue snapshot. This prevents orphaned messages that would + -- appear after tasks are deleted. + FOR batch_record IN + WITH removed AS ( + DELETE FROM pgflow.step_tasks t + USING pgflow.runs r + WHERE r.run_id = t.run_id + AND ( + (r.completed_at IS NOT NULL AND r.completed_at < cutoff_timestamp) OR + (r.failed_at IS NOT NULL AND r.failed_at < cutoff_timestamp) + ) + RETURNING t.queue_name, t.message_id ) - ); + SELECT + removed.queue_name, + ARRAY_AGG(removed.message_id ORDER BY removed.message_id) AS message_ids + FROM removed + WHERE removed.message_id IS NOT NULL + GROUP BY removed.queue_name + ORDER BY removed.queue_name + LOOP + PERFORM pgmq.delete(batch_record.queue_name, batch_record.message_ids); + END LOOP; - -- Delete ALL step_states for old runs (regardless of individual step status) + -- Delete ALL step_states for old runs (regardless of individual task status) DELETE FROM pgflow.step_states WHERE run_id IN ( SELECT run_id FROM pgflow.runs @@ -73,26 +180,40 @@ BEGIN (failed_at IS NOT NULL AND failed_at < cutoff_timestamp) ); - -- Prune archived messages from PGMQ archive tables (pgmq.a_{flow_slug}) - -- For each flow, delete old archived messages - FOR flow_record IN SELECT DISTINCT flow_slug FROM pgflow.flows + -- Prune archived messages from PGMQ archive tables, enumerating validated + -- persisted definition routes (including the empty compiled plain default) + -- rather than deriving archive names from task IDs or only active runs. + -- The definitions above are still locked against deletion. + FOR route_record IN + SELECT COALESCE( + ARRAY_AGG(DISTINCT s.queue_name ORDER BY s.queue_name) + FILTER (WHERE s.queue_name IS NOT NULL), + ARRAY[lower(f.flow_slug)] + ) AS route_queues + FROM pgflow.flows f + LEFT JOIN pgflow.steps s ON s.flow_slug = f.flow_slug + GROUP BY f.flow_slug + ORDER BY f.flow_slug LOOP - -- Build the archive table name - archive_table := pgmq.format_table_name(flow_record.flow_slug, 'a'); + FOREACH archive_queue IN ARRAY route_record.route_queues + LOOP + -- Build the archive table name + archive_table := pgmq.format_table_name(archive_queue, 'a'); - -- Check if the archive table exists - IF EXISTS ( - SELECT 1 FROM information_schema.tables - WHERE table_schema = 'pgmq' AND table_name = archive_table - ) THEN - -- Build and execute a dynamic SQL statement to delete old archive records - dynamic_sql := format(' - DELETE FROM pgmq.%I - WHERE archived_at < $1 - ', archive_table); + -- Check if the archive table exists + IF EXISTS ( + SELECT 1 FROM information_schema.tables + WHERE table_schema = 'pgmq' AND table_name = archive_table + ) THEN + -- Build and execute a dynamic SQL statement to delete old archive records + dynamic_sql := format(' + DELETE FROM pgmq.%I + WHERE archived_at < $1 + ', archive_table); - EXECUTE dynamic_sql USING cutoff_timestamp; - END IF; + EXECUTE dynamic_sql USING cutoff_timestamp; + END IF; + END LOOP; END LOOP; END -$$; \ No newline at end of file +$$; diff --git a/pkgs/core/supabase/tests/add_step/circular_dependency.test.sql b/pkgs/core/supabase/tests/add_step/circular_dependency.test.sql index 354dd004e..5045014a6 100644 --- a/pkgs/core/supabase/tests/add_step/circular_dependency.test.sql +++ b/pkgs/core/supabase/tests/add_step/circular_dependency.test.sql @@ -13,7 +13,7 @@ select -- Test select throws_ok( $$ SELECT pgflow.add_step('test_flow', 'circular_step', ARRAY['fourth_step', 'circular_step']) $$, - 'new row for relation "deps" violates check constraint "deps_check"', + 'Flow test_flow: step "circular_step" has a dependency that does not exist', 'Should not allow self-depending steps' ); diff --git a/pkgs/core/supabase/tests/add_step/invalid_step_slug.test.sql b/pkgs/core/supabase/tests/add_step/invalid_step_slug.test.sql index 8284f3efc..746085332 100644 --- a/pkgs/core/supabase/tests/add_step/invalid_step_slug.test.sql +++ b/pkgs/core/supabase/tests/add_step/invalid_step_slug.test.sql @@ -8,7 +8,7 @@ select pgflow.create_flow('test_flow'); -- Test select throws_ok( $$ SELECT pgflow.add_step('test_flow', '1invalid-slug') $$, - 'new row for relation "steps" violates check constraint "steps_step_slug_check"', + 'Flow test_flow: "1invalid-slug" is not a valid step slug', 'Should detect and prevent invalid step slug' ); diff --git a/pkgs/core/supabase/tests/add_step/nonexistent_dependency.test.sql b/pkgs/core/supabase/tests/add_step/nonexistent_dependency.test.sql index e4b6e10c8..c29a0727b 100644 --- a/pkgs/core/supabase/tests/add_step/nonexistent_dependency.test.sql +++ b/pkgs/core/supabase/tests/add_step/nonexistent_dependency.test.sql @@ -8,7 +8,7 @@ select pgflow.create_flow('test_flow'); -- Test select throws_ok( $$ SELECT pgflow.add_step('test_flow', 'invalid_dep_step', ARRAY['nonexistent_step']) $$, - 'insert or update on table "deps" violates foreign key constraint "deps_flow_slug_dep_slug_fkey"', + 'Flow test_flow: step "invalid_dep_step" has a dependency that does not exist', 'Should detect and prevent dependency on non-existent step' ); diff --git a/pkgs/core/supabase/tests/add_step/nonexistent_flow.test.sql b/pkgs/core/supabase/tests/add_step/nonexistent_flow.test.sql index cb0cd0356..9b4aed86c 100644 --- a/pkgs/core/supabase/tests/add_step/nonexistent_flow.test.sql +++ b/pkgs/core/supabase/tests/add_step/nonexistent_flow.test.sql @@ -5,7 +5,7 @@ select pgflow_tests.reset_db(); -- Test select throws_ok( $$ SELECT pgflow.add_step('nonexistent_flow', 'some_step') $$, - 'insert or update on table "steps" violates foreign key constraint "steps_flow_slug_fkey"', + 'Flow nonexistent_flow does not exist', 'Should not allow adding step to non-existent flow' ); diff --git a/pkgs/core/supabase/tests/add_step/step_index_uniqueness.test.sql b/pkgs/core/supabase/tests/add_step/step_index_uniqueness.test.sql index 236f6cf7b..6ed06d358 100644 --- a/pkgs/core/supabase/tests/add_step/step_index_uniqueness.test.sql +++ b/pkgs/core/supabase/tests/add_step/step_index_uniqueness.test.sql @@ -19,8 +19,8 @@ select is( -- Test: Cannot have two steps with the same index in the same flow select throws_ok( $$ - INSERT INTO pgflow.steps (flow_slug, step_slug, step_index) - VALUES ('test_flow', 'duplicate_index_step', 0) + INSERT INTO pgflow.steps (flow_slug, step_slug, step_index, queue_name) + VALUES ('test_flow', 'duplicate_index_step', 0, 'test_flow') $$, '23505', -- Unique violation error code 'duplicate key value violates unique constraint "steps_flow_slug_step_index_key"', diff --git a/pkgs/core/supabase/tests/create_flow/flow_creation.test.sql b/pkgs/core/supabase/tests/create_flow/flow_creation.test.sql index 9ea81159f..6dfd39e3c 100644 --- a/pkgs/core/supabase/tests/create_flow/flow_creation.test.sql +++ b/pkgs/core/supabase/tests/create_flow/flow_creation.test.sql @@ -1,13 +1,7 @@ begin; -select plan(2); +select plan(5); select pgflow_tests.reset_db(); --- Clean up any existing test queue -select pgmq.drop_queue('test_flow') -from pgmq.list_queues() -where queue_name = 'test_flow' -limit 1; - -- TEST: Flow should be added to the flows table select pgflow.create_flow('test_flow'); select results_eq( @@ -16,11 +10,34 @@ select results_eq( 'Flow should be added to the flows table' ); --- TEST: Creating a flow should create a PGMQ queue with the same name +-- TEST: Creating a flow is definition-only: no queue DDL (#650) select results_eq( $$ SELECT EXISTS(SELECT 1 FROM pgmq.list_queues() WHERE queue_name = 'test_flow') $$, - array[true], - 'Creating a flow should create a PGMQ queue with the same name' + array[false], + 'Creating a flow does not create a PGMQ queue' +); + +-- TEST: add_step provisions the canonical default through the shared path +select pgflow.add_step('test_flow', 'first_step'); +select is( + (select queue_name::text FROM pgmq.list_queues() WHERE queue_name = 'test_flow'), + 'test_flow', + 'add_step creates the canonical default queue' +); + +-- TEST: The step persists its canonical route +select is( + (select queue_name from pgflow.steps where step_slug = 'first_step'), + 'test_flow', + 'step route is canonical' +); + +-- TEST: An empty startup-compiled plain flow provisions its default +select pgflow.ensure_flow_compiled('empty_flow', '{"steps": []}'::jsonb, '{"version": 1}'::jsonb); +select is( + (select queue_name::text FROM pgmq.list_queues() WHERE queue_name = 'empty_flow'), + 'empty_flow', + 'empty plain flow compiles with its generated default queue' ); select * from finish(); diff --git a/pkgs/core/supabase/tests/create_flow/invalid_slug.test.sql b/pkgs/core/supabase/tests/create_flow/invalid_slug.test.sql index d1ae0b748..58f81194a 100644 --- a/pkgs/core/supabase/tests/create_flow/invalid_slug.test.sql +++ b/pkgs/core/supabase/tests/create_flow/invalid_slug.test.sql @@ -5,7 +5,7 @@ select pgflow_tests.reset_db(); -- TEST: Should detect and prevent invalid flow slug select throws_ok( $$ SELECT pgflow.create_flow('invalid-flow') $$, - 'new row for relation "flows" violates check constraint "slug_is_valid"', + 'Flow invalid-flow: "invalid-flow" is not a valid generated queue name (lowercase, at most 47 characters, starting with a letter)', 'Should detect and prevent invalid flow slug' ); diff --git a/pkgs/core/supabase/tests/delete_flow_and_data/queue_ownership.test.sql b/pkgs/core/supabase/tests/delete_flow_and_data/queue_ownership.test.sql new file mode 100644 index 000000000..31fb3bb3f --- /dev/null +++ b/pkgs/core/supabase/tests/delete_flow_and_data/queue_ownership.test.sql @@ -0,0 +1,113 @@ +-- Deletion validates private queue ownership before dropping anything (#650). +-- Missing/ambiguous/malformed/differently-owned resources and out-of-route +-- task snapshots leave all rows and resources unchanged. +begin; +select plan(13); +select pgflow_tests.reset_db(); + +-- ---------- Happy path: camelCase compiled flow with live data ---------- +select pgflow.create_flow('Orders'); +select pgflow.add_step('Orders', 'first'); +select pgflow.start_flow('Orders', '{}'); +-- Archived history for the same flow +select pgflow_tests.ensure_worker('orders'); +select pgflow_tests.read_and_start('Orders'); +select pgflow.complete_task( + (select run_id from pgflow.step_tasks where flow_slug = 'Orders'), + 'first', 0, '"done"'::jsonb +); +-- Another queue that must remain untouched +select pgmq.create('unrelated_app'); + +select lives_ok( + $$select pgflow.delete_flow_and_data('Orders')$$, + 'deleting a valid camelCase flow with live and archived data succeeds' +); +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'Orders'), + 0, + 'flow definition removed' +); +select is( + (select count(*)::int from pgflow.runs where flow_slug = 'Orders'), + 0, + 'runtime rows removed' +); +select is( + (select count(*)::int from pgmq.list_queues() where lower(queue_name) = 'orders'), + 0, + 'private queue and archive dropped' +); +select is( + (select to_regclass('pgmq.q_orders')) is null, + true, + 'queue table gone' +); +select is( + (select to_regclass('pgmq.q_orders_msg_id_seq')) is null, + true, + 'identity sequence gone' +); +select is( + (select count(*)::int from pgmq.list_queues() where queue_name = 'unrelated_app'), + 1, + 'an unrelated application queue is untouched' +); + +-- ---------- Missing metadata fails safely ---------- +select pgflow_tests.reset_db(); +select pgflow.create_flow('Ghost'); +select pgflow.add_step('Ghost', 'first'); +delete from pgmq.meta where queue_name = 'ghost'; +select throws_ok( + $$select pgflow.delete_flow_and_data('Ghost')$$, + 'P0001', null, + 'missing metadata fails safely' +); +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'Ghost'), + 1, + 'failed deletion leaves the flow row (rollback)' +); +select is( + (select count(*)::int from pgflow.steps where flow_slug = 'Ghost'), + 1, + 'failed deletion leaves step definitions (rollback)' +); + +-- ---------- Ambiguous metadata spelling fails safely ---------- +select pgflow_tests.reset_db(); +select pgflow.create_flow('Ambig'); +select pgflow.add_step('Ambig', 'first'); +-- Simulate legacy mixed-case metadata plus a case alias of it +update pgmq.meta set queue_name = 'Ambig' where queue_name = 'ambig'; +insert into pgmq.meta (queue_name, is_partitioned, is_unlogged) +values ('ambig', false, false); +select throws_ok( + $$select pgflow.delete_flow_and_data('Ambig')$$, + 'P0001', null, + 'two metadata spellings fail safely' +); + +-- ---------- Task snapshot outside the private route fails safely ---------- +select pgflow_tests.reset_db(); +select pgflow.create_flow('Routed'); +select pgflow.add_step('Routed', 'first'); +select pgflow.start_flow('Routed', '{}'); +-- Direct insert (the immutability trigger guards updates, not inserts) +insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, queue_name, message_id) +select 'Routed', run_id, 'first', 99, 'elsewhere', null +from pgflow.runs where flow_slug = 'Routed'; +select is( + (select count(*)::int from pgflow.step_tasks where flow_slug = 'Routed' and queue_name = 'elsewhere'), + 1, + 'fixture: task snapshot outside the route' +); +select throws_ok( + $$select pgflow.delete_flow_and_data('Routed')$$, + 'P0001', null, + 'a task snapshot outside the private route rejects deletion' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/ensure_flow_compiled/auto_recompiles_when_local.test.sql b/pkgs/core/supabase/tests/ensure_flow_compiled/auto_recompiles_when_local.test.sql index 3e394615b..913c39355 100644 --- a/pkgs/core/supabase/tests/ensure_flow_compiled/auto_recompiles_when_local.test.sql +++ b/pkgs/core/supabase/tests/ensure_flow_compiled/auto_recompiles_when_local.test.sql @@ -19,7 +19,7 @@ select is( "steps": [ {"slug": "new_step", "stepType": "single", "dependencies": [], "whenUnmet": "skip", "whenExhausted": "fail"} ] - }'::jsonb + }'::jsonb, '{"version": 1}'::jsonb ) as result ), 'recompiled', diff --git a/pkgs/core/supabase/tests/ensure_flow_compiled/compiles_missing_flow.test.sql b/pkgs/core/supabase/tests/ensure_flow_compiled/compiles_missing_flow.test.sql index 203e2e891..775a3e020 100644 --- a/pkgs/core/supabase/tests/ensure_flow_compiled/compiles_missing_flow.test.sql +++ b/pkgs/core/supabase/tests/ensure_flow_compiled/compiles_missing_flow.test.sql @@ -12,7 +12,7 @@ select is( "steps": [ {"slug": "first", "stepType": "single", "dependencies": [], "whenUnmet": "skip", "whenExhausted": "fail"} ] - }'::jsonb + }'::jsonb, '{"version": 1}'::jsonb ) as result ), 'compiled', diff --git a/pkgs/core/supabase/tests/ensure_flow_compiled/fails_mismatch_when_not_local.test.sql b/pkgs/core/supabase/tests/ensure_flow_compiled/fails_mismatch_when_not_local.test.sql index 90953a596..fe4353d55 100644 --- a/pkgs/core/supabase/tests/ensure_flow_compiled/fails_mismatch_when_not_local.test.sql +++ b/pkgs/core/supabase/tests/ensure_flow_compiled/fails_mismatch_when_not_local.test.sql @@ -19,7 +19,7 @@ select is( "steps": [ {"slug": "new_step", "stepType": "single", "dependencies": []} ] - }'::jsonb + }'::jsonb, '{"version": 1}'::jsonb ) as result ), 'mismatch', @@ -36,7 +36,7 @@ select ok( "steps": [ {"slug": "new_step", "stepType": "single", "dependencies": []} ] - }'::jsonb + }'::jsonb, '{"version": 1}'::jsonb ) as result ), 'Should return differences for production mismatch' diff --git a/pkgs/core/supabase/tests/ensure_flow_compiled/signature.test.sql b/pkgs/core/supabase/tests/ensure_flow_compiled/signature.test.sql index 85f2714b9..c28aa44b4 100644 --- a/pkgs/core/supabase/tests/ensure_flow_compiled/signature.test.sql +++ b/pkgs/core/supabase/tests/ensure_flow_compiled/signature.test.sql @@ -1,11 +1,16 @@ begin; -select plan(2); +select plan(5); select has_function( 'pgflow', 'ensure_flow_compiled', - array['text', 'jsonb'], - 'ensure_flow_compiled(text, jsonb) should exist' + array['text', 'jsonb', 'jsonb'], + 'ensure_flow_compiled(text, jsonb, jsonb) should exist' +); + +select ok( + to_regprocedure('pgflow.ensure_flow_compiled(text,jsonb)') is null, + 'ensure_flow_compiled(text, jsonb) should not exist' ); select ok( @@ -13,5 +18,19 @@ select ok( 'ensure_flow_compiled(text, jsonb, boolean) should not exist' ); +-- Missing/non-object/wrong-version protocol is rejected before any +-- definition mutation +select throws_ok( + $$select pgflow.ensure_flow_compiled('proto_flow', '{"steps": []}'::jsonb, null)$$, + 'P0001', 'Queue-capable worker protocol version 1 is required', + 'null protocol is rejected' +); + +select throws_ok( + $$select pgflow.ensure_flow_compiled('proto_flow', '{"steps": []}'::jsonb, '{"version": 2}'::jsonb)$$, + 'P0001', 'Queue-capable worker protocol version 1 is required', + 'wrong protocol version is rejected' +); + select * from finish(); rollback; diff --git a/pkgs/core/supabase/tests/ensure_flow_compiled/verifies_matching_shape.test.sql b/pkgs/core/supabase/tests/ensure_flow_compiled/verifies_matching_shape.test.sql index 206bbc314..f78e9af4e 100644 --- a/pkgs/core/supabase/tests/ensure_flow_compiled/verifies_matching_shape.test.sql +++ b/pkgs/core/supabase/tests/ensure_flow_compiled/verifies_matching_shape.test.sql @@ -18,7 +18,7 @@ select is( {"slug": "first", "stepType": "single", "dependencies": [], "whenUnmet": "skip", "whenExhausted": "fail", "requiredInputPattern": {"defined": false}, "forbiddenInputPattern": {"defined": false}}, {"slug": "second", "stepType": "single", "dependencies": ["first"], "whenUnmet": "skip", "whenExhausted": "fail", "requiredInputPattern": {"defined": false}, "forbiddenInputPattern": {"defined": false}} ] - }'::jsonb + }'::jsonb, '{"version": 1}'::jsonb ) as result ), 'verified', @@ -36,7 +36,7 @@ select is( {"slug": "first", "stepType": "single", "dependencies": [], "whenUnmet": "skip", "whenExhausted": "fail", "requiredInputPattern": {"defined": false}, "forbiddenInputPattern": {"defined": false}}, {"slug": "second", "stepType": "single", "dependencies": ["first"], "whenUnmet": "skip", "whenExhausted": "fail", "requiredInputPattern": {"defined": false}, "forbiddenInputPattern": {"defined": false}} ] - }'::jsonb + }'::jsonb, '{"version": 1}'::jsonb ) as result ), 0, diff --git a/pkgs/core/supabase/tests/is_valid_slug.test.sql b/pkgs/core/supabase/tests/is_valid_slug.test.sql index 86e30b5da..a2f5dc731 100644 --- a/pkgs/core/supabase/tests/is_valid_slug.test.sql +++ b/pkgs/core/supabase/tests/is_valid_slug.test.sql @@ -1,5 +1,5 @@ begin; -select plan(11); +select plan(17); select pgflow_tests.reset_db(); -- TEST: Null input @@ -46,22 +46,58 @@ select ok( -- TEST: Valid with underscore select ok( - pgflow.is_valid_slug('valid_slug'), - 'is_valid_slug returns true for string with underscore' + pgflow.is_valid_slug('a_b'), + 'is_valid_slug returns true for single internal underscore' ); -- TEST: Valid with numbers (not at start) select ok( - pgflow.is_valid_slug('valid123'), + pgflow.is_valid_slug('a1'), 'is_valid_slug returns true for string with numbers not at start' ); -- TEST: Valid mixed case select ok( - pgflow.is_valid_slug('validSlug'), + pgflow.is_valid_slug('camelCase'), 'is_valid_slug returns true for mixed case string' ); +-- TEST: Exact 128-character slug stays valid +select ok( + pgflow.is_valid_slug(repeat('a', 128)), + 'is_valid_slug returns true for exact 128-character slug' +); + +-- TEST: Leading underscore +select ok( + not pgflow.is_valid_slug('_a'), + 'is_valid_slug returns false for leading underscore' +); + +-- TEST: Trailing underscore +select ok( + not pgflow.is_valid_slug('a_'), + 'is_valid_slug returns false for trailing underscore' +); + +-- TEST: Double underscore +select ok( + not pgflow.is_valid_slug('a__b'), + 'is_valid_slug returns false for embedded double underscore' +); + +-- TEST: Bare underscore +select ok( + not pgflow.is_valid_slug('_'), + 'is_valid_slug returns false for bare underscore' +); + +-- TEST: Triple underscore inside +select ok( + not pgflow.is_valid_slug('a___b'), + 'is_valid_slug returns false for triple underscore' +); + -- TEST: select ok( not pgflow.is_valid_slug('run'), diff --git a/pkgs/core/supabase/tests/maintenance/prune_deletes_all_child_statuses.test.sql b/pkgs/core/supabase/tests/maintenance/prune_deletes_all_child_statuses.test.sql index 979c5bfaa..b5ecabdc7 100644 --- a/pkgs/core/supabase/tests/maintenance/prune_deletes_all_child_statuses.test.sql +++ b/pkgs/core/supabase/tests/maintenance/prune_deletes_all_child_statuses.test.sql @@ -58,7 +58,7 @@ set failed_at = NULL where flow_slug = 'status_test_flow' and step_slug = 'step2'; -insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, status, queued_at, started_at) +insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, status, queued_at, started_at, queue_name) select 'status_test_flow', run_id, @@ -66,7 +66,8 @@ select 0, 'started', now() - interval '36 days', - now() - interval '35 days' + now() - interval '35 days', + 'status_test_flow' from pgflow.runs where flow_slug = 'status_test_flow'; -- step3: created but never started diff --git a/pkgs/core/supabase/tests/maintenance/prune_rejects_malformed_queue.test.sql b/pkgs/core/supabase/tests/maintenance/prune_rejects_malformed_queue.test.sql new file mode 100644 index 000000000..b0072dedd --- /dev/null +++ b/pkgs/core/supabase/tests/maintenance/prune_rejects_malformed_queue.test.sql @@ -0,0 +1,41 @@ +-- #650 review: pruning validates every route it will touch with +-- _inspect_generated_queue before any queue access. A malformed queue with +-- the same canonical name (here: the valid vt index replaced by a partial +-- one, as a dropped-and-recreated external replacement could produce) must +-- stop pruning atomically instead of deleting from that queue. +\i _shared/prune_data_older_than.sql.raw +begin; +select plan(3); +select pgflow_tests.reset_db(); + +select pgflow.create_flow('prMal'); +select pgflow.add_step('prMal', 'first'); +select pgflow.start_flow('prMal', '{}'); +update pgflow.runs +set started_at = now() - interval '45 days', + status = 'completed', completed_at = now() - interval '40 days' +where flow_slug = 'prMal'; + +-- Corrupt the generated queue's physical shape in place. +drop index pgmq.q_prmal_vt_idx; + +select throws_ok( + $$select pgflow.prune_data_older_than(make_interval(days => 30))$$, + 'P0001', + null, + 'pruning refuses a malformed queue with the same canonical name' +); + +select is( + (select count(*)::int from pgflow.runs where flow_slug = 'prMal'), + 1, + 'the expired run survives the rejected pruning pass' +); +select is( + (select count(*)::int from pgflow.step_tasks where flow_slug = 'prMal'), + 1, + 'task rows survive the rejected pruning pass' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/maintenance/queue_snapshot_pruning.test.sql b/pkgs/core/supabase/tests/maintenance/queue_snapshot_pruning.test.sql new file mode 100644 index 000000000..733f06178 --- /dev/null +++ b/pkgs/core/supabase/tests/maintenance/queue_snapshot_pruning.test.sql @@ -0,0 +1,93 @@ +-- Queue-aware optional pruning (#650): active cleanup groups by task queue +-- snapshot, archive retention enumerates persisted routes (including empty +-- compiled plain defaults), and NULL message IDs stay valid. +\i _shared/prune_data_older_than.sql.raw +begin; +select plan(8); +select pgflow_tests.reset_db(); + +-- Flow A: completed run with live message (worker crashed mid-processing) +select pgflow.create_flow('prA', timeout => 1); +select pgflow.add_step('prA', 'first'); +select pgflow.start_flow('prA', '{}'); +update pgflow.runs +set started_at = now() - interval '45 days', + status = 'completed', completed_at = now() - interval '40 days' +where flow_slug = 'prA'; +update pgflow.step_tasks set queued_at = now() - interval '45 days' where flow_slug = 'prA'; + +-- Flow B: recent run that must survive entirely +select pgflow.create_flow('prB', timeout => 1); +select pgflow.add_step('prB', 'first'); +select pgflow.start_flow('prB', '{}'); +update pgflow.runs +set started_at = now() - interval '6 days', + status = 'completed', completed_at = now() - interval '5 days' +where flow_slug = 'prB'; + +-- Flow C: empty compiled plain flow (no steps, no runs) with an old archived +-- message in its default archive table +select pgflow.create_flow('prC'); +select pgmq.create('prc'); +select pgmq.send('prc', '{"hello":"world"}'); +select pgmq.archive('prc', (select msg_id from pgmq.q_prc limit 1)); +update pgmq.a_prc set archived_at = now() - interval '40 days'; + +-- A second queue whose archive holds both old and recent messages; it is +-- routed by a persisted definition route, so it must be pruned by route name +select pgflow.create_flow('prD'); +select pgflow.add_step('prD', 'only');select pgmq.send('prd', '{"old":1}'); +select pgmq.archive('prd', (select msg_id from pgmq.q_prd limit 1)); +update pgmq.a_prd set archived_at = now() - interval '40 days'; +select pgmq.send('prd', '{"new":1}'); +select pgmq.archive('prd', (select msg_id from pgmq.q_prd limit 1)); + +-- An unrelated application queue's archive must not be scanned +select pgmq.create('prOther'); +select pgmq.send('prOther', '{"old":1}'); +select pgmq.archive('prOther', (select msg_id from pgmq.q_prOther limit 1)); +update pgmq.a_prother set archived_at = now() - interval '40 days'; + +select lives_ok( + $$select pgflow.prune_data_older_than(make_interval(days => 30))$$, + 'pruning runs cleanly' +); + +select is( + (select count(*)::int from pgmq.q_prA), + 0, + 'old completed run active message is deleted by task snapshot' +); +select is( + (select count(*)::int from pgflow.step_tasks where flow_slug = 'prA'), + 0, + 'old completed run task rows are deleted' +); +select is( + (select count(*)::int from pgflow.step_tasks where flow_slug = 'prB'), + 1, + 'recent run tasks survive' +); +select is( + (select count(*)::int from pgmq.q_prB), + 1, + 'recent run message survives' +); +select is( + (select count(*)::int from pgmq.a_prc), + 0, + 'empty compiled plain default archive is pruned by route' +); +select is( + (select message->>'new' from pgmq.a_prd), + '1', + 'recent archive entries survive while old ones are pruned' +); +select is( + (select count(*)::int from pgmq.a_prOther), + 1, + 'an unrelated application archive is untouched' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/case_uniqueness.test.sql b/pkgs/core/supabase/tests/queue_identity/case_uniqueness.test.sql new file mode 100644 index 000000000..a0c90ebcd --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/case_uniqueness.test.sql @@ -0,0 +1,63 @@ +-- Case-only duplicate flow and step slugs must be rejected atomically by +-- declarative unique indexes, even with no runs and no SQL function precheck. +begin; +select plan(10); +select pgflow_tests.reset_db(); + +-- Exact spelling is preserved +select lives_ok( + $$select pgflow.create_flow('Orders')$$, + 'create_flow accepts a camelCase flow with exact spelling' +); + +select is( + (select flow_slug from pgflow.flows where flow_slug = 'Orders'), + 'Orders', + 'flow spelling is preserved' +); + +select throws_ok( + $$select pgflow.create_flow('orders')$$, + '23505', null, 'case-only flow alias is rejected atomically' +); + +select lives_ok( + $$select pgflow.create_flow('Orders')$$, + 'exact repeated create_flow stays idempotent' +); + +select lives_ok( + $$select pgflow.add_step('Orders', 'saveItem')$$, + 'add_step accepts a camelCase step with exact spelling' +); + +select throws_ok( + $$select pgflow.add_step('Orders', 'SaveItem')$$, + '23505', null, 'case-only step alias is rejected atomically' +); + +select lives_ok( + $$select pgflow.add_step('Orders', 'saveItem')$$, + 'exact repeated add_step stays idempotent' +); + +-- Direct inserts hit the same atomic boundary +select throws_ok( + $$insert into pgflow.flows (flow_slug) values ('ORDERS')$$, + '23505', null, 'direct case-alias flow insert is rejected' +); + +select throws_ok( + $$insert into pgflow.steps (flow_slug, step_slug, queue_name) values ('Orders', 'SAVEITEM', 'orders')$$, + '23505', null, 'direct case-alias step insert is rejected' +); + +-- Definitions exist without any runs +select is( + (select count(*)::int from pgflow.flows), + 1, + 'no-run definitions remain (single flow after aliases rejected)' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/claim_classification.test.sql b/pkgs/core/supabase/tests/queue_identity/claim_classification.test.sql new file mode 100644 index 000000000..bd1ae68f9 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/claim_classification.test.sql @@ -0,0 +1,94 @@ +-- Complete-batch claim classification (#650): one mixed read batch produces +-- distinct outcomes - claim, defer, terminal archive, foreign archive with a +-- body-free warning - and no second attempt on the started task. +begin; +select plan(11); +select pgflow_tests.reset_db(); + +select pgflow.create_flow('Mixed', timeout => 5); +select pgflow.add_step('Mixed', 'eachItem', step_type => 'map'); +select pgflow.start_flow('Mixed', '[10, 20, 30]'); +select pgflow_tests.ensure_worker('mixed'); + +-- t1 (index 1): claim once so it is started, then make its message visible again +select message_id as msg_id_1 from pgflow.step_tasks where task_index = 1 \gset +select pgflow.claim_tasks('mixed', 'Mixed', ARRAY[:msg_id_1]::bigint[], + '11111111-1111-1111-1111-111111111111'::uuid); + +-- t2 (index 2): terminal status with its message still in the queue +update pgflow.step_tasks +set status = 'completed', completed_at = now() +where flow_slug = 'Mixed' and task_index = 2; + +-- Foreign untracked message +select pgmq.send('mixed', '{"hello":"world"}'); + +-- Make every active message visible for one mixed read batch +select pgflow_tests.reset_message_visibility('mixed'); + +create temporary table before_claim as +select task_index, started_at, attempts_count from pgflow.step_tasks; + +create temporary table mixed_result as +select pgflow.claim_tasks( + 'mixed', 'Mixed', + (select array_agg(msg_id order by msg_id) from pgmq.q_mixed), + '11111111-1111-1111-1111-111111111111'::uuid +) as result; + +select is( + (select result->>'status' from mixed_result), + 'ok', + 'mixed batch with no fatal member claims successfully' +); +select is( + (select jsonb_array_length(result->'tasks') from mixed_result), + 1, + 'exactly one task is claimed' +); +select is( + (select result->'tasks'->0->>'task_index' from mixed_result), + '0', + 'the queued task is the claimed one' +); +select is( + (select attempts_count from pgflow.step_tasks where task_index = 1), + (select attempts_count from before_claim where task_index = 1), + 'the started task gets no second attempt' +); +select is( + (select started_at from pgflow.step_tasks where task_index = 1), + (select started_at from before_claim where task_index = 1), + 'started_at is untouched by deferral' +); +select is( + (select count(*)::int from pgmq.q_mixed), + 2, + 'claimed and deferred messages remain in the queue' +); +select is( + (select count(*)::int from pgmq.a_mixed), + 2, + 'terminal and foreign messages are archived' +); +select is( + (select jsonb_array_length(result->'warnings') from mixed_result), + 1, + 'exactly one body-free warning is returned' +); +select is( + (select result->'warnings'->0->>'reason' from mixed_result), + 'foreign_message', + 'the warning names the foreign reason' +); +select ok( + (select result::text from mixed_result) not like '%hello%', + 'no message body leaks into the result' +); +select ok( + (select vt from pgmq.q_mixed where msg_id = :msg_id_1) > clock_timestamp() + interval '25 seconds', + 'the deferred started task keeps its +30 recovery deadline (not the +2 claim margin)' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/claim_envelope.test.sql b/pkgs/core/supabase/tests/queue_identity/claim_envelope.test.sql new file mode 100644 index 000000000..f48f25de9 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/claim_envelope.test.sql @@ -0,0 +1,101 @@ +-- Claim envelope identity rules (#650): exact durable pairs win over +-- malformed-looking components, a valid address that positively identifies +-- different work is fatal, and present-null identity keys are pgflow +-- evidence rather than foreign silence. +begin; +select plan(7); +select pgflow_tests.reset_db(); + +select pgflow.create_flow('Env', timeout => 5); +select pgflow.add_step('Env', 'first'); +select pgflow.start_flow('Env', '{}'); +select pgflow_tests.ensure_worker('env'); + +-- Case 1: exact durable pair with a malformed-looking body wins; the task +-- is claimed, not rejected. +update pgmq.q_env +set message = jsonb_set(message, '{run_id}', '"not-a-uuid"') +where msg_id = (select message_id from pgflow.step_tasks where task_index = 0); + +create temporary table env_case1 as +select pgflow.claim_tasks( + 'env', 'Env', + (select array_agg(message_id) from pgflow.step_tasks where task_index = 0), + '11111111-1111-1111-1111-111111111111'::uuid +) as result; + +select is( + (select result->>'status' from env_case1), + 'ok', + 'exact pair with malformed run_id component still claims' +); +select is( + (select jsonb_array_length(result->'tasks') from env_case1), + 1, + 'the durable pair is claimed once' +); + +-- Case 2: a valid step address identifying different work is fatal. The +-- run_id is restored so only the step contradicts the durable pair. +update pgmq.q_env q +set message = jsonb_set( + jsonb_set(q.message, '{step_slug}', '"otherStep"'), + '{run_id}', to_jsonb(t.run_id::text) +) +from pgflow.step_tasks t +where t.queue_name = 'env' and t.task_index = 0 and q.msg_id = t.message_id; + +select is( + (select r->>'status' from pgflow.claim_tasks( + 'env', 'Env', + (select array_agg(message_id) from pgflow.step_tasks where task_index = 0), + '11111111-1111-1111-1111-111111111111'::uuid) as r), + 'fatal', + 'exact pair with a contradicting valid step address is fatal' +); + +-- Case 3: present-null identity keys are pgflow evidence, not foreign +-- silence: fatal unsupported work. +select pgmq.send('env', '{"run_id": null}'); + +create temporary table env_case3 as +select pgflow.claim_tasks( + 'env', 'Env', + ARRAY[(select max(msg_id) from pgmq.q_env)]::bigint[], + '11111111-1111-1111-1111-111111111111'::uuid +) as result; + +select is( + (select result->>'status' from env_case3), + 'fatal', + 'present-null run_id without an exact pair is fatal' +); +select is( + (select result->'errors'->0->>'reason' from env_case3), + 'unsupported_work', + 'the present-null envelope is unsupported work, not foreign' +); + +-- Case 4: a valid flow address naming a different flow is wrong-route work. +select pgmq.send('env', '{"flow_slug":"Elsewhere"}'); + +create temporary table env_case4 as +select pgflow.claim_tasks( + 'env', 'Env', + ARRAY[(select max(msg_id) from pgmq.q_env)]::bigint[], + '11111111-1111-1111-1111-111111111111'::uuid +) as result; + +select is( + (select result->>'status' from env_case4), + 'fatal', + 'a different-flow envelope without an exact pair is fatal' +); +select is( + (select result->'errors'->0->>'reason' from env_case4), + 'wrong_route', + 'the diagnostic names the wrong route' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/claim_fatal_atomicity.test.sql b/pkgs/core/supabase/tests/queue_identity/claim_fatal_atomicity.test.sql new file mode 100644 index 000000000..57fe2186c --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/claim_fatal_atomicity.test.sql @@ -0,0 +1,79 @@ +-- Fatal batch atomicity (#650): with any fatal member, claim none, mutate no +-- task history, archive nothing; commit the complete-batch visibility reset +-- and the persistent HTTP restart pause, and return a normal fatal result. +begin; +select plan(9); +select pgflow_tests.reset_db(); + +select pgflow.create_flow('Orders', max_attempts => 3); +select pgflow.add_step('Orders', 'first'); +select pgflow.start_flow('Orders', '{}'); + +select pgflow.track_worker_function('orders_worker'); +select pgflow_tests.ensure_worker('orders', function_name => 'orders_worker'); + +-- Capture pre-claim state +create temporary table before_claim as +select status, attempts_count, started_at from pgflow.step_tasks; + +-- The batch: the valid queued task's message plus unsupported identity work +select message_id as msg_id from pgflow.step_tasks \gset +select pgmq.send('orders', '{"flow_slug":"Orders","run_id":"not-a-uuid"}'); +select pgflow_tests.reset_message_visibility('orders'); + +create temporary table claim_ids as +select array_agg(msg_id order by msg_id) as ids from pgmq.q_orders; + +create temporary table claim_result as +select pgflow.claim_tasks( + 'orders', 'Orders', + (select ids from claim_ids), + '11111111-1111-1111-1111-111111111111'::uuid +) as result; + +select is( + (select result->>'status' from claim_result), + 'fatal', + 'unsupported identity is fatal' +); +select is( + (select jsonb_array_length(result->'tasks') from claim_result), + 0, + 'fatal claims no tasks' +); +select is( + (select result->'errors'->0->>'reason' from claim_result), + 'unsupported_work', + 'the fatal diagnostic names the reason' +); +select ok( + (select result::text from claim_result) not like '%not-a-uuid%', + 'the fatal diagnostic carries no envelope body' +); +select results_eq( + $$ select status, attempts_count from pgflow.step_tasks $$, + $$ select status, attempts_count from before_claim $$, + 'no task history changes on a fatal batch' +); +select is( + (select count(*)::int from pgmq.a_orders), + 0, + 'nothing is archived on a fatal batch' +); +select is( + (select enabled from pgflow.worker_functions where function_name = 'orders_worker'), + false, + 'fatal claim persists HTTP restart pause' +); +select ok( + (select bool_and(vt <= clock_timestamp()) from pgmq.q_orders), + 'every member of the read batch is reset to immediate visibility' +); +select is( + (select count(*)::int from pgflow.runs where status = 'started'), + 1, + 'the run is untouched by the fatal claim' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/claim_races.test.sql b/pkgs/core/supabase/tests/queue_identity/claim_races.test.sql new file mode 100644 index 000000000..e834872b9 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/claim_races.test.sql @@ -0,0 +1,180 @@ +-- Concurrent claim races (#650): only committed guarded claims return, a +-- visible started message consumes no attempt from a competing claim, and a +-- committed fatal outcome is observable from a third connection with its +-- pause/reset committed while no task mutation escapes. +begin; +select plan(11); + +create extension if not exists dblink; + +select count(pg_terminate_backend(pid)) as terminated_stale_sessions +from pg_stat_activity +where application_name in ('crace_ctrl', 'crace_a', 'crace_b', 'crace_probe') + and pid <> pg_backend_pid(); + +select format( + 'hostaddr=%s port=%s dbname=%s user=postgres password=postgres application_name=', + coalesce(host(inet_server_addr()), '127.0.0.1'), + inet_server_port(), + current_database() +) as conn_base \gset + +select dblink_connect('ctrl', :'conn_base' || 'crace_ctrl'); +select dblink_exec('ctrl', 'set lock_timeout = 20000'); +select dblink_connect('probe', :'conn_base' || 'crace_probe'); + +create function pg_temp.wait_locked(app_name text, deadline_s int default 10) +returns boolean language plpgsql as $fn$ +declare + blocked boolean; + deadline timestamptz := clock_timestamp() + make_interval(secs => deadline_s); +begin + perform pg_sleep(0.2); + loop + select exists( + select 1 from dblink('probe', format( + 'select 1 from pg_stat_activity where application_name = %L and wait_event_type = ''Lock''', + app_name + )) as t(x int) + ) into blocked; + if blocked or clock_timestamp() > deadline then + return blocked; + end if; + perform pg_sleep(0.1); + end loop; +end; +$fn$; + +create function pg_temp.capture_result(conn text) +returns text language plpgsql as $fn$ +declare + v_first text; + v_count int; + v_guard int := 0; +begin + begin + select r into v_first from dblink_get_result(conn) as t(r text) limit 1; + exception when others then + v_first := 'error:' || sqlstate; + end; + loop + v_guard := v_guard + 1; + exit when v_guard > 5; + begin + select count(*) into v_count from dblink_get_result(conn) as t(r text); + exception when others then + v_count := 1; + end; + exit when v_count = 0; + end loop; + return coalesce(v_first, 'ok:empty'); +exception when others then + return 'error:' || sqlstate; +end; +$fn$; + +select dblink_connect('a', :'conn_base' || 'crace_a'); +select dblink_exec('a', 'set lock_timeout = 20000'); +select dblink_connect('b', :'conn_base' || 'crace_b'); +select dblink_exec('b', 'set lock_timeout = 20000'); + +-- ============================================================ +-- RACE 1: two competing claims of the same read batch +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('crace1', timeout => 5); perform pgflow.add_step('crace1', 'first'); perform pgflow.start_flow('crace1', '{}'); end $do$;$$); +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.ensure_worker('crace1'); end $do$;$$); + +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.step_tasks where flow_slug = 'crace1' for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.claim_tasks('crace1','crace1',ARRAY[1]::bigint[],'11111111-1111-1111-1111-111111111111'::uuid)$$); +select ok(pg_temp.wait_locked('crace_a'), 'first claim queues on the task row lock'); + +select dblink_send_query('b', $$select pgflow.claim_tasks('crace1','crace1',ARRAY[1]::bigint[],'11111111-1111-1111-1111-111111111111'::uuid)$$); +select ok(pg_temp.wait_locked('crace_b'), 'competing claim queues behind the first'); + +select dblink_exec('ctrl', 'commit'); + +create temporary table race1 as +select pg_temp.capture_result('a') as a_outcome, + pg_temp.capture_result('b') as b_outcome; + +select is( + (select attempts_count from dblink('probe', 'select attempts_count from pgflow.step_tasks where flow_slug = ''crace1''') as t(attempts_count int)), + 1, + 'exactly one committed claim consumes the single attempt' +); +select is( + (select status from dblink('probe', 'select status from pgflow.step_tasks where flow_slug = ''crace1''') as t(status text)), + 'started', + 'the task is started once' +); + +-- ============================================================ +-- RACE 2: claim vs fail_task cannot partially mutate +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('crace2', timeout => 5); perform pgflow.add_step('crace2', 'first'); perform pgflow.start_flow('crace2', '{}'); end $do$;$$); +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.ensure_worker('crace2'); end $do$;$$); + +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.runs where flow_slug = 'crace2' for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.claim_tasks('crace2','crace2',ARRAY[1]::bigint[],'11111111-1111-1111-1111-111111111111'::uuid)$$); +select ok(pg_temp.wait_locked('crace_a'), 'claim queues on the run row lock'); + +select dblink_send_query('b', $$select pgflow.fail_task((select run_id from pgflow.runs where flow_slug='crace2'), 'first', 0, 'boom')$$); +select ok(pg_temp.wait_locked('crace_b'), 'fail_task queues on the run row lock'); + +select dblink_exec('ctrl', 'commit'); +select pg_temp.capture_result('a'); +select pg_temp.capture_result('b'); + +select ok( + (select attempts from dblink('probe', 'select attempts_count as attempts from pgflow.step_tasks where flow_slug = ''crace2''') as t(attempts int)) <= 1, + 'claim and failure contention consumes at most one attempt (no partial mutation)' +); + +-- ============================================================ +-- RACE 3: committed fatal outcome inspected from a third connection +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('crace3', timeout => 5); perform pgflow.add_step('crace3', 'first'); perform pgflow.start_flow('crace3', '{}'); perform pgflow_tests.ensure_worker('crace3', function_name => 'crace3_worker'); perform pgflow.track_worker_function('crace3_worker'); perform pgmq.send('crace3', '{"flow_slug":"crace3","run_id":"not-a-uuid"}'); perform pgflow_tests.reset_message_visibility('crace3'); end $do$;$$); + +-- Read (autocommit connection A), then claim (autocommit connection B): the +-- real two-transaction worker path. B claims the same visible batch A read. +select count(*) as read_rows +from dblink('a', $$select msg_id from pgmq.q_crace3$$) as t(msg_id bigint); + +select is( + (select r->>'status' from dblink('b', $$ + select pgflow.claim_tasks( + 'crace3', 'crace3', + (select array_agg(msg_id order by msg_id) from pgmq.q_crace3), + '11111111-1111-1111-1111-111111111111'::uuid) as r + $$) as t(r jsonb)), + 'fatal', + 'committed fatal result returns normally from an autocommit claim' +); + +-- Third connection (probe) observes the committed pause and reset +select is( + (select enabled from dblink('probe', 'select enabled from pgflow.worker_functions where function_name = ''crace3_worker''') as t(enabled boolean)), + false, + 'HTTP restart pause is committed and visible to a third connection' +); +select ok( + (select bool_and(vt <= clock_timestamp()) from dblink('probe', 'select vt from pgmq.q_crace3') as t(vt timestamptz)), + 'the complete read batch is reset and visible to a third connection' +); +select is( + (select attempts_count from dblink('probe', 'select attempts_count from pgflow.step_tasks where flow_slug = ''crace3''') as t(attempts_count int)), + 0, + 'no task attempt escapes the committed fatal outcome' +); + +select dblink_disconnect('a'); +select dblink_disconnect('b'); +select dblink_disconnect('ctrl'); +select dblink_disconnect('probe'); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/deletion_races.test.sql b/pkgs/core/supabase/tests/queue_identity/deletion_races.test.sql new file mode 100644 index 000000000..b7761584a --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/deletion_races.test.sql @@ -0,0 +1,265 @@ +-- Deletion lock-order races (#650): deletion versus complete_task, fail_task, +-- recovery, pruning, start_flow, and external queue loss. pg_blocking_pids +-- barriers pin the interleaving; after release every cooperating pair must +-- complete or reject boundedly, with no queue-first/task-later deadlock. +begin; +select plan(18); + +create extension if not exists dblink; + +select count(pg_terminate_backend(pid)) as terminated_stale_sessions +from pg_stat_activity +where application_name in ('drace_ctrl', 'drace_a', 'drace_b', 'drace_probe') + and pid <> pg_backend_pid(); + +select format( + 'hostaddr=%s port=%s dbname=%s user=postgres password=postgres application_name=', + coalesce(host(inet_server_addr()), '127.0.0.1'), + inet_server_port(), + current_database() +) as conn_base \gset + +select dblink_connect('ctrl', :'conn_base' || 'drace_ctrl'); +select dblink_exec('ctrl', 'set lock_timeout = 20000'); +select dblink_connect('probe', :'conn_base' || 'drace_probe'); + +create function pg_temp.wait_locked(app_name text, deadline_s int default 10) +returns boolean language plpgsql as $fn$ +declare + blocked boolean; + deadline timestamptz := clock_timestamp() + make_interval(secs => deadline_s); +begin + perform pg_sleep(0.2); + loop + select exists( + select 1 from dblink('probe', format( + 'select 1 from pg_stat_activity where application_name = %L and wait_event_type = ''Lock''', + app_name + )) as t(x int) + ) into blocked; + if blocked or clock_timestamp() > deadline then + return blocked; + end if; + perform pg_sleep(0.1); + end loop; +end; +$fn$; + +create function pg_temp.wait_settled(app_a text, app_b text, deadline_s int default 20) +returns void language plpgsql as $fn$ +declare + still_running int; + deadline timestamptz := clock_timestamp() + make_interval(secs => deadline_s); +begin + loop + select count(*) into still_running + from dblink('probe', format( + 'select pid from pg_stat_activity where application_name in (%L, %L) and state = ''active'' and pid <> pg_backend_pid()', + app_a, app_b + )) as t(pid bigint); + exit when still_running = 0 or clock_timestamp() > deadline; + perform pg_sleep(0.2); + end loop; +end; +$fn$; + +create function pg_temp.capture_result(conn text) +returns text language plpgsql as $fn$ +declare + v_first text; + v_count int; + v_guard int := 0; +begin + begin + select r into v_first from dblink_get_result(conn) as t(r text) limit 1; + exception when others then + v_first := 'error:' || sqlstate; + end; + loop + v_guard := v_guard + 1; + exit when v_guard > 5; + begin + select count(*) into v_count from dblink_get_result(conn) as t(r text); + exception when others then + v_count := 1; + end; + exit when v_count = 0; + end loop; + return coalesce(v_first, 'ok:empty'); +exception when others then + return 'error:' || sqlstate; +end; +$fn$; + +select dblink_connect('a', :'conn_base' || 'drace_a'); +select dblink_exec('a', 'set lock_timeout = 20000'); +select dblink_connect('b', :'conn_base' || 'drace_b'); +select dblink_exec('b', 'set lock_timeout = 20000'); + +-- ============================================================ +-- RACE 1: deletion vs late complete_task on the same run +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('delrace1', max_attempts => 1); perform pgflow.add_step('delrace1', 'first'); perform pgflow.start_flow('delrace1', '{}'); end $do$;$$); + +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.runs where flow_slug = 'delrace1' for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.delete_flow_and_data('delrace1')$$); +select ok(pg_temp.wait_locked('drace_a'), 'deletion queues on the run row lock'); + +select run_id from dblink('ctrl', $$select run_id from pgflow.runs where flow_slug='delrace1'$$) as t(run_id uuid) \gset +select dblink_send_query('b', format($$select pgflow.complete_task(%L, 'first', 0, '"late"'::jsonb)$$, :'run_id')); +select ok(pg_temp.wait_locked('drace_b'), 'late completion queues on the run row lock'); + +select dblink_exec('ctrl', 'commit'); +select pg_temp.wait_settled('drace_a', 'drace_b'); + +create temporary table race1 as +select pg_temp.capture_result('a') as del_outcome, + pg_temp.capture_result('b') as cb_outcome; +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'delrace1') + + (select count(*)::int from pgmq.list_queues() where queue_name = 'delrace1'), + (select case when (select del_outcome from race1) like 'error:%' then 1 else 0 end), + 'deletion/completion contention: full deletion or full rollback, never a partial drop' +); + +-- ============================================================ +-- RACE 2: deletion vs fail_task (task-row lock) +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('delrace2', max_attempts => 1); perform pgflow.add_step('delrace2', 'first'); perform pgflow.start_flow('delrace2', '{}'); end $do$;$$); + +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.step_tasks where flow_slug = 'delrace2' for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.delete_flow_and_data('delrace2')$$); +select ok(pg_temp.wait_locked('drace_a'), 'deletion queues on the task row lock'); + +select dblink_send_query('b', $$select pgflow.fail_task((select run_id from pgflow.runs where flow_slug='delrace2'), 'first', 0, 'boom')$$); +select ok(pg_temp.wait_locked('drace_b'), 'fail_task queues behind the same lock'); + +select dblink_exec('ctrl', 'commit'); +select pg_temp.wait_settled('drace_a', 'drace_b'); + +create temporary table race2 as +select pg_temp.capture_result('a') as del_outcome, + pg_temp.capture_result('b') as fail_outcome; +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'delrace2') + + (select count(*)::int from pgmq.list_queues() where queue_name = 'delrace2'), + (select case when (select del_outcome from race2) like 'error:%' then 1 else 0 end), + 'deletion/failure contention: full deletion or full rollback, no deadlock' +); + +-- ============================================================ +-- RACE 3: deletion vs recovery (SKIP LOCKED skips the contested task) +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('delrace3', timeout => 1); perform pgflow.add_step('delrace3', 'first'); perform pgflow.start_flow('delrace3', '{}'); end $do$;$$); +select dblink_exec('ctrl', $$do $do$ begin update pgflow.step_tasks set queued_at = now() - interval '120 seconds', started_at = now() - interval '120 seconds' where flow_slug = 'delrace3'; end $do$;$$); + +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.step_tasks where flow_slug = 'delrace3' for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.delete_flow_and_data('delrace3')$$); +select ok(pg_temp.wait_locked('drace_a'), 'deletion queues on the task row lock'); + +select is( + (select r::int from dblink('b', 'select pgflow.requeue_stalled_tasks()') as t(r int)), + 0, + 'recovery skips the locked task instead of waiting' +); + +select dblink_exec('ctrl', 'commit'); +select pg_temp.capture_result('a'); +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'delrace3'), + 0, + 'deletion completes after the recovery skip' +); + +-- ============================================================ +-- RACE 4: deletion vs pruning (run_id-ordered locks, no deadlock) +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('delrace4', timeout => 1); perform pgflow.add_step('delrace4', 'first'); perform pgflow.start_flow('delrace4', '{}'); perform pgflow.create_flow('delrace4b', timeout => 1); perform pgflow.add_step('delrace4b', 'first'); perform pgflow.start_flow('delrace4b', '{}'); end $do$;$$); + +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.runs where flow_slug in ('delrace4','delrace4b') order by run_id for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.delete_flow_and_data('delrace4')$$); +select ok(pg_temp.wait_locked('drace_a'), 'deletion queues on the run lock'); + +select dblink_send_query('b', $$do $do$ begin perform pgflow.prune_data_older_than(interval '1 day'); end $do$;$$); +select ok(pg_temp.wait_locked('drace_b'), 'pruning queues on the run lock'); + +select dblink_exec('ctrl', 'commit'); +select pg_temp.wait_settled('drace_a', 'drace_b'); +create temporary table race4 as +select pg_temp.capture_result('a') as del_outcome, + pg_temp.capture_result('b') as prune_outcome; +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'delrace4') + + (select count(*)::int from pgmq.list_queues() where queue_name = 'delrace4'), + (select case when (select del_outcome from race4) like 'error:%' then 1 else 0 end), + 'deletion/pruning contention: full deletion or full rollback, no deadlock' +); + +-- ============================================================ +-- RACE 5: deletion vs producer start_flow (definition lock) +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('delrace5', timeout => 1); perform pgflow.add_step('delrace5', 'first'); end $do$;$$); + +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.flows where flow_slug = 'delrace5' for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.delete_flow_and_data('delrace5')$$); +select ok(pg_temp.wait_locked('drace_a'), 'deletion queues on the flow row lock'); + +select dblink_send_query('b', $$select pgflow.start_flow('delrace5', '{}')$$); +select ok(pg_temp.wait_locked('drace_b'), 'producer queues on the flow row lock'); + +select dblink_exec('ctrl', 'commit'); +select pg_temp.wait_settled('drace_a', 'drace_b'); +create temporary table race5 as +select pg_temp.capture_result('a') as del_outcome, + pg_temp.capture_result('b') as produce_outcome; +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'delrace5'), + (select case when (select del_outcome from race5) like 'error:%' then 1 else 0 end), + 'deletion/producer contention: full deletion or full rollback, no deadlock' +); + +-- ============================================================ +-- RACE 6: external queue loss while deletion is fenced at the flow row +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('delrace6'); perform pgflow.add_step('delrace6', 'first'); end $do$;$$); + +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.flows where flow_slug = 'delrace6' for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.delete_flow_and_data('delrace6')$$); +select ok(pg_temp.wait_locked('drace_a'), 'deletion queues on the flow row lock'); + +-- External PGMQ drop commits while deletion is fenced: a rolled-back +-- conflict is acceptable; silent adoption/drop of uncertain resources is not. +select dblink_exec('b', $$do $do$ begin perform pgmq.drop_queue('delrace6'); end $do$;$$); + +select dblink_exec('ctrl', 'commit'); +select pg_temp.wait_settled('drace_a', 'drace_a'); +select is( + (select pg_temp.capture_result('a') like 'error:%'), + true, + 'deletion rejects after external queue loss' +); +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'delrace6'), + 1, + 'rolled-back deletion leaves the flow intact' +); + +select dblink_disconnect('a'); +select dblink_disconnect('b'); +select dblink_disconnect('ctrl'); +select dblink_disconnect('probe'); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/lifecycle.test.sql b/pkgs/core/supabase/tests/queue_identity/lifecycle.test.sql new file mode 100644 index 000000000..9f4184e51 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/lifecycle.test.sql @@ -0,0 +1,170 @@ +-- Queue-aware lifecycle coverage (#650): every archive/retry/cancel path uses +-- the task's queue snapshot. PGMQ message IDs are queue-scoped, so two flows +-- each holding ID 1 must never interfere. +begin; +select plan(17); +select pgflow_tests.reset_db(); + +-- ============================================================ +-- BLOCK 1: cross-queue isolation with shared message ID 1 +-- ============================================================ +select pgflow.create_flow('Orders', max_attempts => 1); +select pgflow.add_step('Orders', 'first'); +select pgflow.create_flow('Invoices', max_attempts => 1); +select pgflow.add_step('Invoices', 'first'); +select pgflow.start_flow('Orders', '{}'); +select pgflow.start_flow('Invoices', '{}'); + +select is( + (select message_id from pgflow.step_tasks where flow_slug = 'Orders'), + (select message_id from pgflow.step_tasks where flow_slug = 'Invoices'), + 'fixture deliberately reuses an ID across queues' +); + +-- Complete Orders' task: only the orders queue/archive change +select pgflow_tests.poll_and_complete('Orders'); + +select is( + (select count(*)::int from pgmq.a_orders), + 1, + 'completion archives into the orders archive by snapshot' +); +select is( + (select count(*)::int from pgmq.q_orders), + 0, + 'orders queue is drained' +); +select is( + (select count(*)::int from pgmq.q_invoices), + 1, + 'the invoices queue keeps its own ID 1' +); +select is( + (select count(*)::int from pgmq.a_invoices), + 0, + 'the invoices archive is untouched' +); + +-- Fail Invoices' task (max_attempts=1: exhaustion, run fails) +select pgflow_tests.poll_and_fail('Invoices'); + +select is( + (select count(*)::int from pgmq.a_invoices), + 1, + 'exhaustion archives into the invoices archive by snapshot' +); +select is( + (select count(*)::int from pgflow.step_tasks where flow_slug = 'Invoices' and status = 'cancelled'), + 0, + 'single-task flow: the culprit itself terminalizes as failed' +); +select is( + (select queue_name from pgflow.step_tasks where flow_slug = 'Orders'), + 'orders', + 'completion leaves the task snapshot unchanged' +); +select is( + (select queue_name from pgflow.step_tasks where flow_slug = 'Invoices'), + 'invoices', + 'failure leaves the task snapshot unchanged' +); + +-- ============================================================ +-- BLOCK 2: SQL-only multi-queue cancellation fixture +-- ============================================================ +-- Deliberately malformed internal state (a step routed off the canonical +-- default) to prove cancellation groups keep queue identity. Not an allowed +-- user route; the surrounding transaction and reset_db own cleanup. +select pgflow_tests.reset_db(); +select pgflow.create_flow('GroupFixture', max_attempts => 1); +select pgflow.add_step('GroupFixture', 'leftTask'); +select pgflow.add_step('GroupFixture', 'rightTask'); +select pgmq.create('fixture_other'); +update pgflow.steps set queue_name = 'fixture_other' +where flow_slug = 'GroupFixture' and step_slug = 'rightTask'; +select pgflow.start_flow('GroupFixture', '{}'); + +-- An unrelated third queue must stay untouched +select pgflow.create_flow('Bystander', max_attempts => 1); +select pgflow.add_step('Bystander', 'only'); +select pgflow.start_flow('Bystander', '{}'); + +select is( + (select queue_name from pgflow.step_tasks where step_slug = 'rightTask'), + 'fixture_other', + 'rightTask task snapshots the non-default route' +); + +-- Start the culprit through direct test-only task state (the public claim +-- correctly rejects this invalid route) +select pgflow_tests.ensure_worker('groupfixture'); +update pgflow.step_tasks +set status = 'started', + started_at = now(), + attempts_count = attempts_count + 1, + last_worker_id = '11111111-1111-1111-1111-111111111111'::uuid +where flow_slug = 'GroupFixture' and step_slug = 'rightTask'; + +select pgflow.fail_task( + (select run_id from pgflow.step_tasks where step_slug = 'rightTask'), + 'rightTask', + 0, + 'boom' +); + +select is( + (select count(*)::int from pgmq.a_fixture_other), + 1, + 'failure archives the culprit from its snapshot queue' +); +select is( + (select count(*)::int from pgmq.a_groupfixture), + 1, + 'cancellation archives the sibling from the canonical snapshot queue' +); +select is( + (select count(*)::int from pgmq.q_bystander), + 1, + 'a third queue is never touched' +); +select is( + (select status from pgflow.step_tasks where step_slug = 'leftTask'), + 'cancelled', + 'sibling task is cancelled' +); + +-- ============================================================ +-- BLOCK 3: NULL message ID follows state cleanup without PGMQ calls +-- ============================================================ +select pgflow_tests.reset_db(); +select pgflow.create_flow('NullMsg', max_attempts => 1); +select pgflow.add_step('NullMsg', 'first'); +select pgflow.start_flow('NullMsg', '{}'); +select pgflow_tests.ensure_worker('nullmsg'); +update pgflow.step_tasks +set message_id = null, + status = 'started', + started_at = now(), + attempts_count = attempts_count + 1, + last_worker_id = '11111111-1111-1111-1111-111111111111'::uuid +where flow_slug = 'NullMsg'; + +select lives_ok( + $$select pgflow.complete_task( + (select run_id from pgflow.step_tasks where flow_slug = 'NullMsg'), + 'first', 0, '"done"'::jsonb)$$, + 'a NULL-ID task completes without any PGMQ call' +); +select is( + (select status from pgflow.step_tasks where flow_slug = 'NullMsg'), + 'completed', + 'NULL-ID task reaches completed' +); +select is( + (select status from pgflow.runs where flow_slug = 'NullMsg'), + 'completed', + 'the run completes for a NULL-ID task' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/provisioning.test.sql b/pkgs/core/supabase/tests/queue_identity/provisioning.test.sql new file mode 100644 index 000000000..363596779 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/provisioning.test.sql @@ -0,0 +1,112 @@ +-- Generated-queue provisioning rejection coverage (#650): silent adoption, +-- malformed objects, ambiguous metadata, cross-flow references, oversized +-- generated names, and late invalid steps all fail without mutation. +begin; +select plan(13); +select pgflow_tests.reset_db(); + +-- ---------- External preexisting queue is a collision, not adopted ---------- +select pgmq.create('occupied'); +select throws_ok( + $$select pgflow.create_flow('Occupied')$$, + 'P0001', null, + 'external preexisting queue is rejected, not adopted' +); +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'Occupied'), + 0, + 'no flow definition is inserted on collision' +); +select is( + (select count(*)::int from pgmq.q_occupied), + 0, + 'the external queue stays empty and intact' +); + +-- ---------- Partial objects: metadata without the archive table ---------- +select pgflow_tests.reset_db(); +select pgmq.create('partial'); +alter extension pgmq drop table pgmq.a_partial; +drop table pgmq.a_partial; +select throws_ok( + $$select pgflow.create_flow('Partial')$$, + 'P0001', null, + 'partial objects are rejected, not repaired' +); + +-- ---------- Ambiguous case-aliased metadata ---------- +select pgflow_tests.reset_db(); +select pgmq.create('Orders'); +insert into pgmq.meta (queue_name, is_partitioned, is_unlogged) +values ('orders', false, false); +select throws_ok( + $$select pgflow.create_flow('Orders')$$, + 'P0001', null, + 'ambiguous Orders/orders metadata is rejected' +); + +-- ---------- Cross-flow route reference ---------- +select pgflow_tests.reset_db(); +select pgflow.create_flow('Owner'); +select pgflow.add_step('Owner', 'first'); +select pgflow.create_flow('Thief'); +select pgflow.add_step('Thief', 'first'); +update pgflow.steps set queue_name = 'owner' where flow_slug = 'Thief'; +select throws_ok( + $$select pgflow.add_step('Owner', 'another')$$, + 'P0001', null, + 'a route claimed by another flow is rejected' +); + +-- ---------- 48-character generated name exceeds the queue limit ---------- +select pgflow_tests.reset_db(); +select throws_ok( + $$select pgflow.create_flow(repeat('a', 48))$$, + 'P0001', null, + 'a 48-character generated queue name is rejected' +); + +-- The generic 128-character slug limit still applies to step slugs +select pgflow.create_flow('longstep'); +select lives_ok( + $$select pgflow.add_step('longstep', repeat('s', 127))$$, + 'a 127-character step slug is accepted' +); + +-- ---------- Late invalid step rolls the whole compile back ---------- +select pgflow_tests.reset_db(); +select throws_ok( + $$select pgflow.ensure_flow_compiled('late_invalid', $json$ + {"steps": [ + {"slug": "good_step", "dependencies": []}, + {"slug": "bad__step", "dependencies": []} + ]}$json$::jsonb, '{"version": 1}'::jsonb)$$, + 'P0001', null, + 'a late invalid step rejects the whole compile' +); +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'late_invalid'), + 0, + 'no flow definition survives a late invalid step' +); +select is( + (select count(*)::int from pgmq.list_queues() where queue_name = 'late_invalid'), + 0, + 'no queue is provisioned for a rejected compile' +); + +-- ---------- Explicit non-canonical route in add_step is rejected ---------- +select pgflow_tests.reset_db(); +select pgflow.create_flow('Routed'); +select throws_ok( + $$select pgflow.add_step('Routed', 'first', queue_name => 'elsewhere')$$, + 'P0001', null, + 'an explicit different route is rejected in #650' +); +select lives_ok( + $$select pgflow.add_step('Routed', 'first', queue_name => 'routed')$$, + 'an explicit equal canonical route is accepted' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/provisioning_races.test.sql b/pkgs/core/supabase/tests/queue_identity/provisioning_races.test.sql new file mode 100644 index 000000000..77b67e4d0 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/provisioning_races.test.sql @@ -0,0 +1,335 @@ +-- Concurrent provisioning races for generated queue identity (#650). +-- +-- Uses dblink sessions with pg_blocking_pids()/wait-state barriers (no elapsed +-- sleeps as proof): each race pins the interleaving by holding a lock the +-- victim must queue behind, observes the wait chain, then releases and requires +-- either one valid owner or total rollback of the losing transaction. +begin; +select plan(22); + +create extension if not exists dblink; + +-- Self-heal stale sessions leaked by previously crashed runs +select count(pg_terminate_backend(pid)) as terminated_stale_sessions +from pg_stat_activity +where application_name in ('qrace_ctrl', 'qrace_a', 'qrace_b', 'qrace_e', 'qrace_probe') + and pid <> pg_backend_pid(); + +select format( + 'hostaddr=%s port=%s dbname=%s user=postgres password=postgres application_name=', + coalesce(host(inet_server_addr()), '127.0.0.1'), + inet_server_port(), + current_database() +) as conn_base \gset + +select dblink_connect('ctrl', :'conn_base' || 'qrace_ctrl'); +select dblink_exec('ctrl', 'set lock_timeout = 20000'); +select dblink_connect('probe', :'conn_base' || 'qrace_probe'); + +-- Deterministic wait until a named session waits on a Lock event, observed +-- through the probe connection (fresh activity snapshot per query). +create function pg_temp.wait_locked(app_name text, deadline_s int default 10) +returns boolean language plpgsql as $fn$ +declare + blocked boolean; + deadline timestamptz := clock_timestamp() + make_interval(secs => deadline_s); +begin + perform pg_sleep(0.2); + loop + select exists( + select 1 from dblink('probe', format( + 'select 1 from pg_stat_activity where application_name = %L and wait_event_type = ''Lock''', + app_name + )) as t(x int) + ) into blocked; + if blocked or clock_timestamp() > deadline then + return blocked; + end if; + perform pg_sleep(0.1); + end loop; +end; +$fn$; + +-- Bounded wait until both named sessions stop running their async query. +create function pg_temp.wait_settled(app_a text, app_b text, deadline_s int default 20) +returns void language plpgsql as $fn$ +declare + still_running int; + deadline timestamptz := clock_timestamp() + make_interval(secs => deadline_s); +begin + loop + select count(*) into still_running + from dblink('probe', format( + 'select pid from pg_stat_activity where application_name in (%L, %L) and state = ''active'' and pid <> pg_backend_pid()', + app_a, app_b + )) as t(pid bigint); + exit when still_running = 0 or clock_timestamp() > deadline; + perform pg_sleep(0.2); + end loop; +end; +$fn$; + +-- Capture an async dblink result without letting an expected contention +-- error abort the test file: records the first row as text or the SQLSTATE +-- of the failure. Drains the connection so it accepts new async queries. +create function pg_temp.capture_result(conn text) +returns text language plpgsql as $fn$ +declare + v_first text; + v_count int; + v_guard int := 0; +begin + begin + select r into v_first from dblink_get_result(conn) as t(r text) limit 1; + exception when others then + v_first := 'error:' || sqlstate; + end; + -- Drain: dblink connections stay busy until get_result returns no rows. + loop + v_guard := v_guard + 1; + exit when v_guard > 5; + begin + select count(*) into v_count from dblink_get_result(conn) as t(r text); + exception when others then + v_count := 1; + end; + exit when v_count = 0; + end loop; + return coalesce(v_first, 'ok:empty'); +end; +$fn$; + +-- Drain a connection after a direct dblink_get_result assertion (including +-- errored queries) so later dblink_send_query calls do not fail. +create function pg_temp.drain_result(conn text) +returns void language plpgsql as $fn$ +declare + v_count int; + v_guard int := 0; +begin + loop + v_guard := v_guard + 1; + exit when v_guard > 5; + begin + select count(*) into v_count from dblink_get_result(conn) as t(r text); + exception when others then + v_count := 1; + end; + exit when v_count = 0; + end loop; +end; +$fn$; + +-- ============================================================ +-- RACE 1: concurrent 'Orders'/'orders' compilation +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); end $do$;$$); + +-- ctrl holds the pgmq.meta topology fence; session A (compile 'Orders') must +-- queue behind it inside create_flow's collision inspection. +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', 'lock table pgmq.meta in share row exclusive mode'); + +select dblink_connect('a', :'conn_base' || 'qrace_a'); +select dblink_exec('a', 'set lock_timeout = 20000'); +select dblink_send_query('a', + $$select pgflow.ensure_flow_compiled('Orders', '{"steps":[{"slug":"first","stepType":"single","dependencies":[],"whenUnmet":"skip","whenExhausted":"fail","requiredInputPattern":{"defined":false},"forbiddenInputPattern":{"defined":false}}]}'::jsonb, '{"version": 1}'::jsonb)$$); + +select ok( + pg_temp.wait_locked('qrace_a'), + 'compile A queues behind the metadata fence' +); + +-- B compiles the case alias; it must queue behind A (shared canonical advisory lock) +select dblink_connect('b', :'conn_base' || 'qrace_b'); +select dblink_exec('b', 'set lock_timeout = 20000'); +select dblink_send_query('b', + $$select pgflow.ensure_flow_compiled('orders', '{"steps":[{"slug":"first","stepType":"single","dependencies":[],"whenUnmet":"skip","whenExhausted":"fail","requiredInputPattern":{"defined":false},"forbiddenInputPattern":{"defined":false}}]}'::jsonb, '{"version": 1}'::jsonb)$$); + +select ok( + pg_temp.wait_locked('qrace_b'), + 'compile B queues behind the case-shared flow lock' +); + +-- Release the fence: A wins, B must reject the alias atomically +select dblink_exec('ctrl', 'commit'); + +select is( + (select r->>'status' from dblink_get_result('a') as t(r jsonb)), + 'compiled', + 'first case spelling compiles' +); +select pg_temp.drain_result('a'); +select throws_ok( + $$select r->>'status' from dblink_get_result('b') as t(r jsonb)$$, + '23505', null, + 'case-alias compilation is rejected after losing the race' +); +select pg_temp.drain_result('b'); +select is( + (select flow_slug from pgflow.flows where lower(flow_slug) = 'orders'), + 'Orders', + 'exactly one spelling owns the definition' +); +select is( + (select count(*)::int from pgmq.list_queues() where queue_name = 'orders'), + 1, + 'exactly one generated queue exists' +); + +-- ============================================================ +-- RACE 2: same exact compilation from two sessions +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); end $do$;$$); +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', 'lock table pgmq.meta in share row exclusive mode'); + +select dblink_send_query('a', + $$select pgflow.ensure_flow_compiled('Same', '{"steps":[{"slug":"first","stepType":"single","dependencies":[],"whenUnmet":"skip","whenExhausted":"fail","requiredInputPattern":{"defined":false},"forbiddenInputPattern":{"defined":false}}]}'::jsonb, '{"version": 1}'::jsonb)$$); +select ok( + pg_temp.wait_locked('qrace_a'), + 'first compile queues behind the fence' +); +select dblink_send_query('b', + $$select pgflow.ensure_flow_compiled('Same', '{"steps":[{"slug":"first","stepType":"single","dependencies":[],"whenUnmet":"skip","whenExhausted":"fail","requiredInputPattern":{"defined":false},"forbiddenInputPattern":{"defined":false}}]}'::jsonb, '{"version": 1}'::jsonb)$$); +select ok( + pg_temp.wait_locked('qrace_b'), + 'second compile queues behind the first' +); +select dblink_exec('ctrl', 'commit'); + +select is( + (select r->>'status' from dblink_get_result('a') as t(r jsonb)), + 'compiled', + 'first exact compile succeeds' +); +select pg_temp.drain_result('a'); +select is( + (select r->>'status' from dblink_get_result('b') as t(r jsonb)), + 'verified', + 'second exact compile verifies idempotently' +); +select pg_temp.drain_result('b'); + +-- ============================================================ +-- RACE 3: concurrent add_step calls take distinct indexes +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); perform pgflow.create_flow('added'); perform pgflow.add_step('added', 'first'); end $do$;$$); +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', $$do $do$ begin perform 1 from pgflow.flows where flow_slug = 'added' for update; end $do$;$$); + +select dblink_send_query('a', $$select pgflow.add_step('added', 'alpha')$$); +select ok( + pg_temp.wait_locked('qrace_a'), + 'first add_step queues behind the flow row lock' +); +select dblink_send_query('b', $$select pgflow.add_step('added', 'beta')$$); +select ok( + pg_temp.wait_locked('qrace_b'), + 'second add_step queues behind the first' +); +select dblink_exec('ctrl', 'commit'); + +select ok( + pg_temp.capture_result('a') not like 'error:%', + 'first add_step succeeds' +); +select ok( + pg_temp.capture_result('b') not like 'error:%', + 'second add_step succeeds' +); +select results_eq( + $$ select step_slug || '=' || step_index::text from pgflow.steps where flow_slug = 'added' order by step_slug $$, + $$ values ('alpha=1'), ('beta=2'), ('first=0') $$, + 'concurrent add_step calls take distinct sequential indexes' +); + +-- ============================================================ +-- RACE 4: direct building-block case alias while a compile is fenced +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); end $do$;$$); +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', 'lock table pgmq.meta in share row exclusive mode'); + +select dblink_send_query('a', + $$select pgflow.ensure_flow_compiled('Direct', '{"steps":[{"slug":"first","stepType":"single","dependencies":[],"whenUnmet":"skip","whenExhausted":"fail","requiredInputPattern":{"defined":false},"forbiddenInputPattern":{"defined":false}}]}'::jsonb, '{"version": 1}'::jsonb)$$); +select ok( + pg_temp.wait_locked('qrace_a'), + 'compile queues behind the fence before inserting the identity' +); + +-- Direct building-block INSERT (not a pgflow function) commits the alias +select dblink_exec('b', $$insert into pgflow.flows (flow_slug) values ('direct')$$); +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'direct'), + 1, + 'direct case-alias insert commits while the compile is fenced' +); + +select dblink_exec('ctrl', 'commit'); +create temporary table race4_outcome as +select pg_temp.capture_result('a') as a_outcome; + +select is( + (select a_outcome from race4_outcome) in ('error:23505', 'error:P0001'), + true, + 'fenced compile rejects the committed case alias atomically' +); +select is( + (select count(*)::int from pgflow.flows where flow_slug = 'Direct'), + 0, + 'losing compile leaves no definition' +); + +-- ============================================================ +-- RACE 5: external pgmq.create lands at the topology fence +-- ============================================================ +select dblink_exec('ctrl', $$do $do$ begin perform pgflow_tests.reset_db(); end $do$;$$); +select dblink_exec('ctrl', 'begin'); +select dblink_exec('ctrl', 'lock table pgmq.meta in share row exclusive mode'); + +select dblink_send_query('a', + $$select pgflow.ensure_flow_compiled('Fresh', '{"steps":[{"slug":"first","stepType":"single","dependencies":[],"whenUnmet":"skip","whenExhausted":"fail","requiredInputPattern":{"defined":false},"forbiddenInputPattern":{"defined":false}}]}'::jsonb, '{"version": 1}'::jsonb)$$); +select ok( + pg_temp.wait_locked('qrace_a'), + 'compile queues behind the fence' +); + +-- External create: its metadata insert queues behind the fence +select dblink_connect('e', :'conn_base' || 'qrace_e'); +select dblink_send_query('e', $$select pgmq.create('fresh')$$); +select ok( + pg_temp.wait_locked('qrace_e'), + 'external create queues on the metadata fence' +); + +-- Release: A and E contend; one must win, the loser rolls back entirely +select dblink_exec('ctrl', 'commit'); +select pg_temp.wait_settled('qrace_a', 'qrace_e'); + +create temporary table race5_outcome as +select pg_temp.capture_result('a') as a_outcome; + +-- Either one valid owner (flow + single metadata row) or total compile +-- rollback; never silent adoption of unowned objects into a live definition. +select ok( + ( + (select a_outcome from race5_outcome) not like 'error:%' + and exists(select 1 from pgflow.flows where flow_slug = 'Fresh') + and (select count(*)::int from pgmq.meta where lower(queue_name) = 'fresh') = 1 + ) or ( + (select a_outcome from race5_outcome) like 'error:%' + and not exists(select 1 from pgflow.flows where flow_slug = 'Fresh') + ), + 'fence contention yields one valid owner or a full rollback' +); + +-- Cleanup sessions +select dblink_disconnect('a'); +select dblink_disconnect('b'); +select dblink_disconnect('e'); +select dblink_disconnect('ctrl'); +select dblink_disconnect('probe'); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/recovery.test.sql b/pkgs/core/supabase/tests/queue_identity/recovery.test.sql new file mode 100644 index 000000000..8c8cc2329 --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/recovery.test.sql @@ -0,0 +1,89 @@ +-- Recovery keeps queue snapshots and margins (#650): requeue visibility uses +-- the task snapshot, permanent stall archives without revival, and terminal +-- parents are excluded. The +30s recovery margin is distinct from the +2s +-- claim margin. +begin; +select plan(9); +select pgflow_tests.reset_db(); + +-- ============================================================ +-- Requeue: snapshot-based visibility reset, unchanged queue_name +-- ============================================================ +select pgflow.create_flow('RecoverMe', timeout => 1); +select pgflow.add_step('RecoverMe', 'first'); +select pgflow.start_flow('RecoverMe', '{}'); +select pgflow_tests.ensure_worker('recoverme'); + +-- Direct test-only started-task setup (recovery, not claiming, is under test) +update pgflow.step_tasks +set queued_at = now() - interval '120 seconds', + started_at = now() - interval '120 seconds', + status = 'started', + attempts_count = attempts_count + 1, + last_worker_id = '11111111-1111-1111-1111-111111111111'::uuid +where flow_slug = 'RecoverMe'; + +select is( + (select pgflow.requeue_stalled_tasks()), + 1, + 'one stalled task is requeued' +); +select is( + (select status from pgflow.step_tasks where flow_slug = 'RecoverMe'), + 'queued', + 'requeued task returns to queued' +); +select is( + (select queue_name from pgflow.step_tasks where flow_slug = 'RecoverMe'), + 'recoverme', + 'requeue leaves the queue snapshot unchanged' +); +select ok( + (select vt <= clock_timestamp() from pgmq.q_recoverme), + 'requeued message is visible immediately in its snapshot queue' +); +select is( + (select requeued_count from pgflow.step_tasks where flow_slug = 'RecoverMe'), + 1, + 'requeue increments the counter once' +); + +-- ============================================================ +-- Permanent stall: stays started, never reanimated +-- ============================================================ +select pgflow_tests.reset_db(); +select pgflow.create_flow('Permanent', timeout => 1); +select pgflow.add_step('Permanent', 'first'); +select pgflow.start_flow('Permanent', '{}'); +select pgflow_tests.ensure_worker('permanent'); +update pgflow.step_tasks +set queued_at = now() - interval '120 seconds', + started_at = now() - interval '120 seconds', + status = 'started', + attempts_count = attempts_count + 1, + last_worker_id = '11111111-1111-1111-1111-111111111111'::uuid, + requeued_count = 3 +where flow_slug = 'Permanent'; + +select is( + (select pgflow.requeue_stalled_tasks()), + 0, + 'permanently stalled task is not requeued' +); +select is( + (select status from pgflow.step_tasks where flow_slug = 'Permanent'), + 'started', + 'permanent stall stays started' +); +select ok( + (select permanently_stalled_at is not null from pgflow.step_tasks where flow_slug = 'Permanent'), + 'permanent stall is timestamped' +); +select is( + (select count(*)::int from pgmq.a_permanent), + 1, + 'permanent stall archives its message from the snapshot queue' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/queue_identity/storage.test.sql b/pkgs/core/supabase/tests/queue_identity/storage.test.sql new file mode 100644 index 000000000..1e5e0a73f --- /dev/null +++ b/pkgs/core/supabase/tests/queue_identity/storage.test.sql @@ -0,0 +1,125 @@ +-- Storage and producer snapshot coverage for persisted queue identity (#650). +-- Exact concrete spelling is preserved; definition and task snapshots are +-- canonical. Includes map ordering and start-delay coverage. +begin; +select plan(14); +select pgflow_tests.reset_db(); + +-- ---------- Single-step flow: spelling and snapshots ---------- +select pgflow.create_flow('Orders'); +select pgflow.add_step('Orders', 'first'); +select pgflow.start_flow('Orders', '{}'); + +select is( + (select flow_slug from pgflow.flows), + 'Orders', + 'flow spelling is preserved' +); + +select is( + (select queue_name from pgflow.steps), + 'orders', + 'step route is canonical' +); + +select is( + (select queue_name from pgflow.step_tasks), + 'orders', + 'task snapshots the route' +); + +select ok( + (select message_id is not null from pgflow.step_tasks), + 'producer stores a message ID' +); + +select is( + (select message->>'flow_slug' from pgmq.q_orders), + 'Orders', + 'queue message carries exact concrete flow spelling' +); + +-- ---------- Map ordering: task_index ordinality matches message order ---------- +select pgflow_tests.reset_db(); +select pgflow.create_flow('Invoices'); +select pgflow.add_step('Invoices', 'first'); +select pgflow.add_step('Invoices', 'eachItem', step_type => 'map'); +select pgflow.start_flow('Invoices', '[10, 20, 30]'); + +select results_eq( + $$ + select message->>'task_index' + from pgmq.q_invoices + where message->>'step_slug' = 'eachItem' + order by msg_id + $$, + $$ values ('0'), ('1'), ('2') $$, + 'map task indices preserve per-step ordinality' +); + +select results_eq( + $$ select task_index::text from pgflow.step_tasks where step_slug = 'eachItem' order by task_index $$, + $$ values ('0'), ('1'), ('2') $$, + 'map tasks are recorded in ordinal order' +); + +-- ---------- Start delay: messages sent to the resolved route carry the delay ---------- +select pgflow_tests.reset_db(); +select pgflow.create_flow('Delays', timeout => 5); +select pgflow.add_step('Delays', 'later', start_delay => 17); +select pgflow.start_flow('Delays', '{}'); + +select ok( + (select vt > enqueued_at + interval '10 seconds' from pgmq.q_delays), + 'start delay is applied on the resolved route' +); + +-- ---------- Snapshot immutability and pair uniqueness ---------- +select pgflow_tests.reset_db(); +select pgflow.create_flow('SnapA'); +select pgflow.add_step('SnapA', 'first'); +select pgflow.create_flow('SnapB'); +select pgflow.add_step('SnapB', 'first'); +select pgflow.start_flow('SnapA', '{}'); +select pgflow.start_flow('SnapB', '{}'); + +select ok( + (select message_id from pgflow.step_tasks where flow_slug = 'SnapA') + = (select message_id from pgflow.step_tasks where flow_slug = 'SnapB'), + 'two private queues share message ID 1 by design' +); + +select lives_ok( + $$update pgflow.step_tasks set status = 'started' where flow_slug = 'SnapA'$$, + 'an identical queue update is harmless' +); + +select lives_ok( + $$insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, queue_name, message_id) + select 'SnapA', run_id, 'first', 1, 'snapa', null from pgflow.runs where flow_slug = 'SnapA'$$, + 'null-ID task insert succeeds for the immutability probe' +); + +select throws_ok( + $$update pgflow.step_tasks set queue_name = 'snapb' where flow_slug = 'SnapA' and task_index = 1$$, + 'step_tasks.queue_name is immutable', + 'a changed queue rejects' +); + +select throws_ok( + $$insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, queue_name, message_id) + select 'SnapA', run_id, 'first', 1, 'snapa', + (select message_id from pgflow.step_tasks where flow_slug = 'SnapB') + from pgflow.runs where flow_slug = 'SnapA'$$, + '23505', null, + 'the same non-null queue/message pair rejects' +); + +select lives_ok( + $$insert into pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, queue_name, message_id) + select 'SnapA', run_id, 'first', 2, 'snapa', null from pgflow.runs where flow_slug = 'SnapA'$$, + 'multiple null-ID tasks succeed' +); + +select finish(); +rollback; diff --git a/pkgs/core/supabase/tests/start_tasks/map_task_creation_scaling_performance.test.sql b/pkgs/core/supabase/tests/start_tasks/map_task_creation_scaling_performance.test.sql index edc547fb7..025266a1a 100644 --- a/pkgs/core/supabase/tests/start_tasks/map_task_creation_scaling_performance.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/map_task_creation_scaling_performance.test.sql @@ -47,6 +47,10 @@ BEGIN v_end_time := clock_timestamp(); v_task_creation_ms := EXTRACT(EPOCH FROM (v_end_time - v_flow_start)) * 1000; + -- Refresh planner stats so start_tasks timing measures steady-state + -- indexed plans, not autovacuum lag on the freshly inserted rows + ANALYZE pgflow.step_tasks; + -- Get sample of message IDs for start_tasks test SELECT array_agg(message_id) INTO v_msg_ids FROM ( SELECT message_id FROM pgflow.step_tasks @@ -103,6 +107,10 @@ BEGIN v_end_time := clock_timestamp(); v_task_creation_ms := EXTRACT(EPOCH FROM (v_end_time - v_flow_start)) * 1000; + -- Refresh planner stats so start_tasks timing measures steady-state + -- indexed plans, not autovacuum lag on the freshly inserted rows + ANALYZE pgflow.step_tasks; + -- Get sample of message IDs SELECT array_agg(message_id) INTO v_msg_ids FROM ( SELECT message_id FROM pgflow.step_tasks @@ -159,6 +167,10 @@ BEGIN v_end_time := clock_timestamp(); v_task_creation_ms := EXTRACT(EPOCH FROM (v_end_time - v_flow_start)) * 1000; + -- Refresh planner stats so start_tasks timing measures steady-state + -- indexed plans, not autovacuum lag on the freshly inserted rows + ANALYZE pgflow.step_tasks; + -- Get sample of message IDs SELECT array_agg(message_id) INTO v_msg_ids FROM ( SELECT message_id FROM pgflow.step_tasks @@ -215,6 +227,10 @@ BEGIN v_end_time := clock_timestamp(); v_task_creation_ms := EXTRACT(EPOCH FROM (v_end_time - v_flow_start)) * 1000; + -- Refresh planner stats so start_tasks timing measures steady-state + -- indexed plans, not autovacuum lag on the freshly inserted rows + ANALYZE pgflow.step_tasks; + -- Get sample of message IDs SELECT array_agg(message_id) INTO v_msg_ids FROM ( SELECT message_id FROM pgflow.step_tasks diff --git a/pkgs/core/supabase/tests/start_tasks/visibility_failure_rollback.test.sql b/pkgs/core/supabase/tests/start_tasks/visibility_failure_rollback.test.sql index 03a99e0e6..d876d0985 100644 --- a/pkgs/core/supabase/tests/start_tasks/visibility_failure_rollback.test.sql +++ b/pkgs/core/supabase/tests/start_tasks/visibility_failure_rollback.test.sql @@ -96,7 +96,8 @@ select throws_ok( (select ids from vispartial_msgs), '11111111-1111-1111-1111-111111111111'::uuid ) $$, - 'invalid input syntax for type integer: "start_tasks(): visibility updated 1 of 2 claimed messages"', + 'P0001', + 'claim_tasks(): visibility updated 1 of 2 claimed messages', 'partial visibility mismatch fails the whole statement and returns nothing' ); diff --git a/pkgs/core/supabase/upgrade_queue_fixture/assertions.sql b/pkgs/core/supabase/upgrade_queue_fixture/assertions.sql new file mode 100644 index 000000000..2be6ff73a --- /dev/null +++ b/pkgs/core/supabase/upgrade_queue_fixture/assertions.sql @@ -0,0 +1,190 @@ +-- 0.16.0 queue upgrade fixture assertions (#650). +-- Runs AFTER the queue identity migration on the main seeded old database. +-- Plain DO-block asserts (the fixture container image has no pgTAP); +-- any failure raises, psql runs with ON_ERROR_STOP=1, the script exits +-- non-zero. The runner additionally diffs captured pre/post state signatures. + +-- ========================================== +-- 1. Backfill: canonical snapshots everywhere, NULL IDs preserved +-- ========================================== +do $$ +declare + v_count int; +begin + select count(*) into v_count from pgflow.steps + where queue_name is distinct from lower(flow_slug); + if v_count <> 0 then + raise exception 'steps backfill: % rows not canonical', v_count; + end if; + + select count(*) into v_count from pgflow.step_tasks + where queue_name is distinct from lower(flow_slug); + if v_count <> 0 then + raise exception 'step_tasks backfill: % rows not canonical', v_count; + end if; + + if not exists ( + select 1 from pgflow.step_tasks t + join pgflow.runs r on r.run_id = t.run_id + where r.flow_slug = 'billing' and r.input = '"inv-2"'::jsonb + and t.message_id is null and t.status = 'completed' + ) then + raise exception 'NULL-message completed task lost by migration'; + end if; +end $$; + +-- ========================================== +-- 2. Exact PGMQ metadata spelling preserved until pgflow deletes it +-- ========================================== +do $$ +begin + if not exists (select 1 from pgmq.meta where queue_name = 'Orders') then + raise exception 'metadata spelling ''Orders'' was not preserved'; + end if; +end $$; + +-- ========================================== +-- 3. Partial unique (queue, message) pair constraint works +-- ========================================== +do $$ +declare + v_dup bigint; +begin + select t.message_id into v_dup + from pgflow.step_tasks t + where t.queue_name = 'orders' and t.message_id is not null + limit 1; + + begin + insert into pgflow.step_tasks ( + flow_slug, run_id, step_slug, task_index, queue_name, message_id, status + ) + select t.flow_slug, t.run_id, 'packBoxes', 1, 'orders', v_dup, 'queued' + from pgflow.step_tasks t + where t.queue_name = 'orders' and t.message_id = v_dup + limit 1; + raise exception 'duplicate (queue, message) pair insert was accepted'; + exception when unique_violation then + -- expected + end; +end $$; + +-- ========================================== +-- 4. Startup handshake: new signature only, canonical queue answer +-- ========================================== +do $$ +declare + v_sig_count int; + v_result jsonb; +begin + select count(*) into v_sig_count + from pg_proc p + join pg_namespace n on n.oid = p.pronamespace + where n.nspname = 'pgflow' and p.proname = 'ensure_flow_compiled'; + if v_sig_count <> 1 then + raise exception 'ensure_flow_compiled overloads present: % (expected exactly the 3-arg function)', v_sig_count; + end if; + + select pgflow.ensure_flow_compiled( + 'Orders', pgflow._get_flow_shape('Orders'), '{"version":1}'::jsonb + ) into v_result; + if v_result ->> 'queue_name' is distinct from 'orders' + or v_result ->> 'protocol_version' is distinct from '1' then + raise exception 'ensure_flow_compiled answered %', v_result; + end if; + + begin + perform pgflow.ensure_flow_compiled('Orders', pgflow._get_flow_shape('Orders'), null); + raise exception 'null protocol argument accepted'; + exception when others then + if sqlerrm not like '%Queue-capable worker protocol%' then + raise; + end if; + end; +end $$; + +-- ========================================== +-- 5. Claims and callbacks work on mixed-case identity (started deferral) +-- ========================================== +do $$ +declare + v_ids bigint[]; + v_result jsonb; +begin + insert into pgflow.workers (worker_id, queue_name, function_name, started_at, last_heartbeat_at) + values ('22222222-2222-2222-2222-222222222222', 'orders', 'orders_worker', now(), now()) + on conflict (worker_id) do nothing; + + select array_agg(msg_id) into v_ids from pgmq.read('orders', 5, 10); + select pgflow.claim_tasks('orders', 'Orders', v_ids, '22222222-2222-2222-2222-222222222222'::uuid) + into v_result; + + if v_result ->> 'status' is distinct from 'ok' + or jsonb_array_length(v_result -> 'tasks') <> 0 then + raise exception 'mixed-case started-task deferral failed: %', v_result; + end if; +end $$; + +-- ========================================== +-- 6. Fresh plain flow end-to-end on the migrated database +-- ========================================== +do $$ +declare + v_ids bigint[]; + v_result jsonb; + v_run uuid; +begin + perform pgflow.create_flow('fresh'); + perform pgflow.add_step('fresh', 'work'); + select run_id into v_run from pgflow.start_flow('fresh', '"fresh-1"'::jsonb); + + insert into pgflow.workers (worker_id, queue_name, function_name, started_at, last_heartbeat_at) + values ('33333333-3333-3333-3333-333333333333', 'fresh', 'fresh_worker', now(), now()) + on conflict (worker_id) do nothing; + + select array_agg(msg_id) into v_ids from pgmq.read('fresh', 30, 5); + select pgflow.claim_tasks('fresh', 'fresh', v_ids, '33333333-3333-3333-3333-333333333333'::uuid) + into v_result; + if v_result ->> 'status' <> 'ok' or jsonb_array_length(v_result -> 'tasks') <> 1 then + raise exception 'fresh claim failed: %', v_result; + end if; + + perform pgflow.complete_task(v_run, 'work', 0, '"done"'::jsonb); + if not exists (select 1 from pgmq.a_fresh) then + raise exception 'completed fresh task message was not archived via its snapshot'; + end if; +end $$; + +-- ========================================== +-- 7. PGMQ 1.5.1 mixed-case lifecycle: delete through pgflow, recreate +-- ========================================== +do $$ +begin + perform pgflow.delete_flow_and_data('Orders'); + + if exists (select 1 from pgmq.meta where lower(queue_name) = 'orders') then + raise exception 'Orders metadata survived deletion'; + end if; + if to_regclass('pgmq.q_orders') is not null + or to_regclass('pgmq.a_orders') is not null + or to_regclass('pgmq.q_orders_msg_id_seq') is not null then + raise exception 'Orders physical objects survived deletion'; + end if; + if exists (select 1 from pgflow.flows where flow_slug = 'Orders') then + raise exception 'Orders flow row survived deletion'; + end if; + if not exists (select 1 from pgmq.meta where queue_name = 'app_events') then + raise exception 'unrelated application queue was disturbed by deletion'; + end if; + + -- Recreate with canonical metadata and execute again. + perform pgflow.create_flow('Orders'); + perform pgflow.add_step('Orders', 'saveItem'); + if not exists (select 1 from pgmq.meta where queue_name = 'orders') then + raise exception 'recompiled Orders did not provision canonical metadata ''orders'''; + end if; + perform pgflow.start_flow('Orders', '"ord-2"'::jsonb); + if not exists (select 1 from pgflow.step_tasks where flow_slug = 'Orders' and queue_name = 'orders') then + raise exception 'recreated Orders task lacks canonical snapshot'; + end if; +end $$; diff --git a/pkgs/core/supabase/upgrade_queue_fixture/audit_assertions.sql b/pkgs/core/supabase/upgrade_queue_fixture/audit_assertions.sql new file mode 100644 index 000000000..93ac15cc3 --- /dev/null +++ b/pkgs/core/supabase/upgrade_queue_fixture/audit_assertions.sql @@ -0,0 +1,62 @@ +-- 0.16.0 queue upgrade fixture audit assertions (#650). +-- Part 1 (sections "-- scenario:") injects every audit report category into a +-- freshly restored old database, INCLUDING more than 20 NULL-message tasks to +-- exercise the sample cap and an unrelated application queue with a +-- distinctive body token. +-- Part 2 runs AFTER the audit script and asserts the database is unchanged +-- (read-only report). The runner additionally greps the NOTICE log for exact +-- incompatible names, the 20-key sample cap, NULL counts, and that the +-- unrelated queue's body token never appears. + +-- scenario: inject +select pgflow.create_flow('_bad_flow'); +select pgflow.create_flow('bad_'); +select pgflow.create_flow('a__b'); +select pgflow.create_flow('orders'); + +select pgflow.create_flow('many_null'); +select pgflow.add_step('many_null', 'fan', step_type => 'map'); +select pgflow.start_flow('many_null', ( + select jsonb_agg(g) from generate_series(1, 25) g +)); +select pgmq.archive('many_null', array_agg(message_id)) +from pgflow.step_tasks where flow_slug = 'many_null'; +update pgflow.step_tasks set message_id = null where flow_slug = 'many_null'; + +select pgmq.send('billing', '{"orphan":"visible"}'); +select pgmq.send('billing', '{"orphan":"invisible"}'); +update pgmq.q_billing +set vt = now() + interval '5 minutes' +where message = '{"orphan":"invisible"}'::jsonb; + +-- Missing archive column on a separate queue (empty_flow): the audit must +-- report the malformed physical shape; billing keeps reporting its orphan +-- messages because the malformed report skips only empty_flow. +alter table pgmq.a_empty_flow drop column headers; + +-- ========================================== +-- Post-audit: read-only proof (run after PRE_MIGRATION_CHECK_650.sql) +-- ========================================== +do $$ +declare + v_count int; +begin + select count(*) into v_count + from information_schema.columns + where table_schema = 'pgflow' + and table_name in ('steps', 'step_tasks') + and column_name = 'queue_name'; + if v_count <> 0 then + raise exception 'audit added pgflow columns (was not read-only)'; + end if; + + if not exists (select 1 from pgflow.flows where flow_slug = '_bad_flow') then + raise exception 'audit mutated definitions'; + end if; + + if not exists ( + select 1 from pgmq.q_app_events where message ->> 'secret' = 'app-token-XYZ-3f9' + ) then + raise exception 'audit disturbed the unrelated application queue'; + end if; +end $$; diff --git a/pkgs/core/supabase/upgrade_queue_fixture/concurrency.sql b/pkgs/core/supabase/upgrade_queue_fixture/concurrency.sql new file mode 100644 index 000000000..c595298c1 --- /dev/null +++ b/pkgs/core/supabase/upgrade_queue_fixture/concurrency.sql @@ -0,0 +1,36 @@ +-- 0.16.0 queue upgrade fixture concurrency blockers (#650). +-- Each section starts with "-- blocker: " and holds ONE transaction that +-- holds a lock the migration must respect. The runner starts a section on its +-- own connection (application_name=queue_fixture_blocker), starts the real +-- migration, and requires either a bounded lock-timeout failure with unchanged +-- state, or completion. Cleanup terminates the blocker backend. + +-- blocker: producer +-- Open producer transaction holding a run row lock. +begin; +select 1 from pgflow.runs where flow_slug = 'Orders' for update; +select pg_sleep(60); +rollback; + +-- blocker: definition +-- Open definition transaction holding pgflow.flows locks. +begin; +update pgflow.flows set opt_timeout = opt_timeout where flow_slug = 'Orders'; +select pg_sleep(60); +rollback; + +-- blocker: queue_topology +-- External PGMQ topology change holding the metadata fence. +begin; +lock table pgmq.meta in share row exclusive mode; +select pg_sleep(60); +rollback; + +-- blocker: queue_create +-- External PGMQ queue creation at the topology fence: it creates physical +-- objects and holds the pgmq.meta row lock. The migration must fail within +-- the 5-second lock bound and change nothing. +begin; +select pgmq.create('concurrent_app_queue'); +select pg_sleep(60); +rollback; diff --git a/pkgs/core/supabase/upgrade_queue_fixture/prune_0_16_0.sql b/pkgs/core/supabase/upgrade_queue_fixture/prune_0_16_0.sql new file mode 100644 index 000000000..b68b823ab --- /dev/null +++ b/pkgs/core/supabase/upgrade_queue_fixture/prune_0_16_0.sql @@ -0,0 +1,98 @@ +/** + * Prunes old records from pgflow tables and PGMQ archive tables. + * + * @param retention_interval - Interval of recent records to keep (e.g., interval '28 days', interval '3 months') + * + * IMPORTANT: This function deletes ALL associated records for runs that completed or failed + * more than retention_interval ago, regardless of individual record status. This includes: + * - step_states with status='created' or status='started' (never executed) + * - step_tasks with status='queued' or status='started' (never completed) + * - PGMQ messages in active queues + * + * WARNING: Ensure retention_interval is longer than your longest start_delay to avoid + * deleting tasks before they have a chance to execute. + */ +create or replace function pgflow.prune_data_older_than( + retention_interval INTERVAL +) returns void language plpgsql as $$ +DECLARE + cutoff_timestamp TIMESTAMPTZ := now() - retention_interval; + flow_record RECORD; + archive_table TEXT; + dynamic_sql TEXT; +BEGIN + -- Delete old worker records + DELETE FROM pgflow.workers + WHERE last_heartbeat_at < cutoff_timestamp; + + -- Delete PGMQ messages from active queues BEFORE deleting step_tasks + -- This prevents orphaned messages that would appear after tasks are deleted + FOR flow_record IN + SELECT + r.flow_slug, + ARRAY_AGG(st.message_id) FILTER (WHERE st.message_id IS NOT NULL) as message_ids + FROM pgflow.runs r + JOIN pgflow.step_tasks st ON st.run_id = r.run_id + WHERE ( + (r.completed_at IS NOT NULL AND r.completed_at < cutoff_timestamp) OR + (r.failed_at IS NOT NULL AND r.failed_at < cutoff_timestamp) + ) + GROUP BY r.flow_slug + LOOP + -- Delete messages in batch (pgmq.delete ignores non-existent messages) + IF flow_record.message_ids IS NOT NULL AND array_length(flow_record.message_ids, 1) > 0 THEN + PERFORM pgmq.delete(flow_record.flow_slug, flow_record.message_ids); + END IF; + END LOOP; + + -- Delete ALL step_tasks for old runs (regardless of individual task status) + -- This fixes FK constraint violation when deleting runs with unexecuted steps + DELETE FROM pgflow.step_tasks + WHERE run_id IN ( + SELECT run_id FROM pgflow.runs + WHERE ( + (completed_at IS NOT NULL AND completed_at < cutoff_timestamp) OR + (failed_at IS NOT NULL AND failed_at < cutoff_timestamp) + ) + ); + + -- Delete ALL step_states for old runs (regardless of individual step status) + DELETE FROM pgflow.step_states + WHERE run_id IN ( + SELECT run_id FROM pgflow.runs + WHERE ( + (completed_at IS NOT NULL AND completed_at < cutoff_timestamp) OR + (failed_at IS NOT NULL AND failed_at < cutoff_timestamp) + ) + ); + + -- Delete old runs records + DELETE FROM pgflow.runs + WHERE ( + (completed_at IS NOT NULL AND completed_at < cutoff_timestamp) OR + (failed_at IS NOT NULL AND failed_at < cutoff_timestamp) + ); + + -- Prune archived messages from PGMQ archive tables (pgmq.a_{flow_slug}) + -- For each flow, delete old archived messages + FOR flow_record IN SELECT DISTINCT flow_slug FROM pgflow.flows + LOOP + -- Build the archive table name + archive_table := pgmq.format_table_name(flow_record.flow_slug, 'a'); + + -- Check if the archive table exists + IF EXISTS ( + SELECT 1 FROM information_schema.tables + WHERE table_schema = 'pgmq' AND table_name = archive_table + ) THEN + -- Build and execute a dynamic SQL statement to delete old archive records + dynamic_sql := format(' + DELETE FROM pgmq.%I + WHERE archived_at < $1 + ', archive_table); + + EXECUTE dynamic_sql USING cutoff_timestamp; + END IF; + END LOOP; +END +$$; \ No newline at end of file diff --git a/pkgs/core/supabase/upgrade_queue_fixture/rejections.sql b/pkgs/core/supabase/upgrade_queue_fixture/rejections.sql new file mode 100644 index 000000000..a4228845f --- /dev/null +++ b/pkgs/core/supabase/upgrade_queue_fixture/rejections.sql @@ -0,0 +1,73 @@ +-- 0.16.0 queue upgrade fixture rejection injections (#650). +-- Each section starts with "-- scenario: " and holds ONLY the old-database +-- corruption for that rejection row. The runner extracts one section with awk, +-- injects it into a freshly restored old database, snapshots state, applies the +-- real migration (which must fail atomically), and diffs the snapshots. + +-- scenario: invalid_names +-- Leading/trailing/double underscores and an invalid step name, with no runs. +select pgflow.create_flow('_flow'); +select pgflow.create_flow('flow_'); +select pgflow.create_flow('flow__name'); +select pgflow.create_flow('invnames'); +select pgflow.add_step('invnames', 'bad__step'); + +-- scenario: case_alias +-- Case-only flow definitions share the canonical queue 'orders'. +select pgflow.create_flow('orders'); + +-- scenario: duplicate_pair +-- Two distinct old tasks share a prospective (queue, message) pair. +select pgflow.add_step('Orders', 'audit2'); +insert into pgflow.step_states (flow_slug, run_id, step_slug, status) +select 'Orders', r.run_id, 'audit2', 'created' +from pgflow.runs r where r.flow_slug = 'Orders'; + +insert into pgflow.step_tasks ( + flow_slug, run_id, step_slug, task_index, status, message_id, attempts_count +) +select 'Orders', r.run_id, 'audit2', 0, 'queued', t.message_id, 0 +from pgflow.runs r +join pgflow.step_tasks t on t.run_id = r.run_id and t.step_slug = 'audit' +where r.flow_slug = 'Orders'; + +-- scenario: orphan_message +-- Unmatched active messages: one immediately visible, one non-visible +-- (vt pushed into the future; vt > now() does not exempt it). +select pgmq.send('billing', '{"orphan":"visible"}'); +select pgmq.send('billing', '{"orphan":"invisible"}'); +update pgmq.q_billing +set vt = now() + interval '5 minutes' +where message = '{"orphan":"invisible"}'::jsonb; + +-- scenario: missing_objects +-- Queue metadata exists but physical objects are incomplete (the archive +-- table is renamed away; queue tables are extension members and cannot be +-- dropped directly); the migration must neither reconstruct nor drop anything. +alter table pgmq.a_empty_flow rename to a_empty_flow_broken; + +-- scenario: ambiguous_metadata +-- Two distinct metadata spellings ('Orders' from the seed, 'ORDERS' injected) +-- addressing the same canonical queue; exact metadata must stay unchanged. +insert into pgmq.meta (queue_name, is_partitioned, is_unlogged, created_at) +values ('ORDERS', false, false, now()); + +-- scenario: ownership_mismatch +-- Denormalized task/run flow_slug disagreement; no snapshot is guessed. +update pgflow.step_tasks set flow_slug = 'billing' +where run_id in (select run_id from pgflow.runs where flow_slug = 'Orders') + and step_slug = 'audit'; + +-- scenario: missing_column +-- The archive table lost its headers column (a dropped-and-recreated or +-- manually altered external queue with the same canonical name); a +-- physical shape missing columns must reject the migration atomically +-- instead of passing incomplete objects through the preflight. +alter table pgmq.a_billing drop column headers; + +-- scenario: malformed_objects +-- The queue's valid single-column vt index is replaced by a partial one; the +-- preflight's physical shape contract must reject the migration atomically +-- (a malformed or unusable index is not ownership evidence). +drop index pgmq.q_billing_vt_idx; +create index q_billing_vt_partial on pgmq.q_billing (vt) where read_ct > 0; diff --git a/pkgs/core/supabase/upgrade_queue_fixture/seed.sql b/pkgs/core/supabase/upgrade_queue_fixture/seed.sql new file mode 100644 index 000000000..d9cb94543 --- /dev/null +++ b/pkgs/core/supabase/upgrade_queue_fixture/seed.sql @@ -0,0 +1,101 @@ +-- 0.16.0 queue upgrade fixture seed (#650). +-- Runs on a database at 0.16.0 (migrations up to +-- 20260907082520_pgflow_remove_legacy_flow_compilation.sql only), BEFORE the +-- queue identity migration is applied. Produces the populated old database +-- the audit, startup probe, migration assertions, and rejection templates use. + +-- camelCase flow: exact spelling 'Orders', PGMQ metadata 'Orders', +-- physical tables q_orders/a_orders (PGMQ lowercases table names only). +select pgflow.create_flow('Orders'); +select pgflow.add_step('Orders', 'saveItem'); +select pgflow.add_step('Orders', 'packBoxes', array['saveItem']); +select pgflow.add_step('Orders', 'audit'); + +-- Definition-only flow: no steps, no runs; old create_flow still +-- provisioned its queue. +select pgflow.create_flow('empty_flow'); + +-- Lowercase flow that will reach task failure exhaustion. +-- 1s retry base delay keeps the exhaustion loop inside a short poll window. +select pgflow.create_flow('billing', null, 1); +select pgflow.add_step('billing', 'charge'); + +-- Register an old worker the way 0.16.0 did (exact flow spelling). +select pgflow.track_worker_function('orders_worker'); +insert into pgflow.workers (worker_id, queue_name, function_name, started_at, last_heartbeat_at) +values ('11111111-1111-1111-1111-111111111111', 'Orders', 'orders_worker', now(), now()); + +-- ========================================== +-- Orders run 1: complete the whole chain +-- ========================================== +select pgflow.start_flow('Orders', '"ord-1"'::jsonb); + +-- Claim and complete saveItem (audit task stays queued with its message). +do $$ +declare + v_run uuid; + v_ids bigint[]; +begin + select run_id into v_run from pgflow.runs where flow_slug = 'Orders'; + select array_agg(msg_id) into v_ids from pgmq.read('Orders', 30, 5); + perform pgflow.start_tasks('Orders', v_ids, '11111111-1111-1111-1111-111111111111'::uuid); + perform pgflow.complete_task(v_run, 'saveItem', 0, '"saved"'::jsonb); +end $$; + +-- packBoxes became ready with its own message; complete it so the run ends. +do $$ +declare + v_run uuid; + v_ids bigint[]; +begin + select run_id into v_run from pgflow.runs where flow_slug = 'Orders'; + select array_agg(msg_id) into v_ids from pgmq.read('Orders', 30, 5); + perform pgflow.start_tasks('Orders', v_ids, '11111111-1111-1111-1111-111111111111'::uuid); + perform pgflow.complete_task(v_run, 'packBoxes', 0, '"packed"'::jsonb); +end $$; + +-- ========================================== +-- billing run 1: drive charge to exhaustion (message archived) +-- ========================================== +select pgflow.start_flow('billing', '"inv-1"'::jsonb); + +do $$ +declare + v_run uuid; + v_ids bigint[]; + v_attempt int; +begin + select run_id into v_run from pgflow.runs where flow_slug = 'billing'; + for v_attempt in 1..3 loop + select array_agg(msg_id) into v_ids from pgmq.read_with_poll('billing', 30, 5, 30, 500); + perform pgflow.start_tasks('billing', v_ids, '11111111-1111-1111-1111-111111111111'::uuid); + perform pgflow.fail_task(v_run, 'charge', 0, 'fixture: attempt ' || v_attempt); + end loop; +end $$; + +-- ========================================== +-- billing run 2: completed task that later lost its message ID +-- (historical NULL-message success row; the former message is removed as +-- fixture setup so the old queue holds no unmatched active message) +-- ========================================== +select pgflow.start_flow('billing', '"inv-2"'::jsonb); + +do $$ +declare + v_run uuid; + v_ids bigint[]; +begin + select run_id into v_run from pgflow.runs where flow_slug = 'billing' and input = '"inv-2"'::jsonb; + select array_agg(msg_id) into v_ids from pgmq.read_with_poll('billing', 30, 5, 30, 500); + perform pgflow.start_tasks('billing', v_ids, '11111111-1111-1111-1111-111111111111'::uuid); + perform pgflow.complete_task(v_run, 'charge', 0, '"paid"'::jsonb); + perform pgmq.archive('billing', v_ids); + update pgflow.step_tasks set message_id = null + where run_id = v_run and step_slug = 'charge'; +end $$; + +-- ========================================== +-- Unrelated application queue: never inspected by the audit +-- ========================================== +select pgmq.create('app_events'); +select pgmq.send('app_events', '{"secret":"app-token-XYZ-3f9"}'); diff --git a/pkgs/dsl/README.md b/pkgs/dsl/README.md index d0a9eee6e..f6ae752c7 100644 --- a/pkgs/dsl/README.md +++ b/pkgs/dsl/README.md @@ -240,10 +240,10 @@ All platforms provide these core resources: - **`ctx.env`** - Environment variables (`Record`) - **`ctx.flowInput`** - Original flow input (typed as the flow's input type) - **`ctx.shutdownSignal`** - AbortSignal for graceful shutdown handling -- **`ctx.rawMessage`** - Original pgmq message with metadata +- **`ctx.rawMessage`** - Original pgmq message with metadata (msg_id is an exact decimal string; #650) ```typescript interface PgmqMessageRecord { - msg_id: number; + msg_id: string; // decimal-string PGMQ bigint read_ct: number; enqueued_at: Date; vt: Date; @@ -256,7 +256,7 @@ All platforms provide these core resources: flow_slug: string; run_id: string; step_slug: string; - msg_id: number; + msg_id: string; // decimal-string PGMQ bigint } ``` - **`ctx.workerConfig`** - Resolved worker configuration with all defaults applied @@ -314,9 +314,17 @@ new Flow({ }); ``` +### Slug Rules and Queue Identity + +Flow and step slugs are up to 128 characters and may use letters, digits, and underscores, but they must not start with a digit or underscore, end with an underscore, or contain two consecutive underscores. The word `run` is reserved. + +Accepted slugs keep their exact spelling, and every reference must match it. Uniqueness checks ignore case: flow slugs across the database, and step slugs within each flow. Different flows may reuse a step slug. + +Each flow owns one generated queue whose physical name is the lowercase flow slug. Because PGMQ plain queue names are limited to 47 characters, current flow slugs cannot exceed 47 characters. Step slugs retain the generic 128-character limit. Handlers still see the exact flow and step spelling they defined; only the physical queue name is lowercase. + ## Deploying Flows -Flow workers deploy definitions during startup. The worker extracts the complete flow shape, then PostgreSQL compiles a missing definition or verifies an existing one before polling begins. +Flow workers deploy definitions during startup. The worker extracts the complete flow shape, then PostgreSQL compiles a missing definition or verifies an existing one before polling begins. The database answers with its protocol version and the flow's canonical queue name. A new worker against an older database stops with `QueueProtocolMismatchError`; an old worker against the queue-aware database fails on the removed SQL signature before registration or polling. Upgrade worker packages and the database together. See [Startup Compilation](https://pgflow.dev/concepts/startup-compilation/) for local recompilation and production versioning behavior. diff --git a/pkgs/dsl/__tests__/runtime/flow.test.ts b/pkgs/dsl/__tests__/runtime/flow.test.ts index 5b05fdeac..ce231c396 100644 --- a/pkgs/dsl/__tests__/runtime/flow.test.ts +++ b/pkgs/dsl/__tests__/runtime/flow.test.ts @@ -51,6 +51,25 @@ describe('Flow', () => { it('rejects invalid slugs during flow creation', () => { expect(() => new Flow({ slug: '1invalid' })).toThrowError(); }); + + it.each(['_a', 'a_', 'a__b', '_', 'a___b'])('rejects %s', (slug) => { + expect(() => new Flow({ slug })).toThrow(); + expect(() => new Flow({ slug: 'valid' }).step({ slug }, () => null)).toThrow(); + }); + + it.each(['a_b', 'camelCase', 'a1'])('preserves %s', (slug) => { + expect(new Flow({ slug }).slug).toBe(slug); + }); + + it('preserves exact 128-character slugs and rejects longer', () => { + const exact = 'a'.repeat(128); + expect(new Flow({ slug: exact }).slug).toBe(exact); + expect(() => new Flow({ slug: 'a'.repeat(129) })).toThrow(); + }); + + it('rejects the reserved slug run', () => { + expect(() => new Flow({ slug: 'run' })).toThrow(); + }); }); describe('runtime options validation', () => { diff --git a/pkgs/dsl/__tests__/runtime/steps.test.ts b/pkgs/dsl/__tests__/runtime/steps.test.ts index a675eea57..4be6ed885 100644 --- a/pkgs/dsl/__tests__/runtime/steps.test.ts +++ b/pkgs/dsl/__tests__/runtime/steps.test.ts @@ -25,6 +25,22 @@ describe('Steps', () => { 'Step "test_step" already exists in flow "test_flow"' ); }); + + it('rejects a case-only duplicate step', () => { + const stepFlow = new Flow({ slug: 'Orders' }).step({ slug: 'saveItem' }, () => null); + expect(() => stepFlow.step({ slug: 'SaveItem' }, () => null)).toThrow(); + }); + + it('rejects case-only duplicate steps in map steps', () => { + const mapFlow = new Flow({ slug: 'orders' }).step({ slug: 'first' }, () => null); + expect(() => mapFlow.map({ slug: 'FIRST' }, () => null)).toThrow(); + }); + + it.each(['_a', 'a_', 'a__b', '_', 'a___b'])('rejects step slug %s', (slug) => { + expect(() => flow.step({ slug }, noop)).toThrow(); + expect(() => flow.map({ slug }, noop)).toThrow(); + expect(() => flow.array({ slug }, noop)).toThrow(); + }); }); describe('slug validation', () => { diff --git a/pkgs/dsl/__tests__/runtime/utils.test.ts b/pkgs/dsl/__tests__/runtime/utils.test.ts index 9bcd9aa03..56987c03b 100644 --- a/pkgs/dsl/__tests__/runtime/utils.test.ts +++ b/pkgs/dsl/__tests__/runtime/utils.test.ts @@ -6,7 +6,18 @@ describe('validateSlug', () => { expect(() => validateSlug('valid_slug')).not.toThrowError(); expect(() => validateSlug('valid_slug_123')).not.toThrowError(); expect(() => validateSlug('validSlug123')).not.toThrowError(); - expect(() => validateSlug('_valid_slug')).not.toThrowError(); + }); + + it('rejects slugs with leading, trailing, or double underscores', () => { + expect(() => validateSlug('_valid_slug')).toThrowError( + `Slug '_valid_slug' cannot start or end with an underscore or contain double underscores` + ); + expect(() => validateSlug('valid_slug_')).toThrowError( + `Slug 'valid_slug_' cannot start or end with an underscore or contain double underscores` + ); + expect(() => validateSlug('valid__slug')).toThrowError( + `Slug 'valid__slug' cannot start or end with an underscore or contain double underscores` + ); }); it('rejects slugs that start with numbers', () => { diff --git a/pkgs/dsl/__tests__/types/context-inference.test-d.ts b/pkgs/dsl/__tests__/types/context-inference.test-d.ts index 0d2ddf6e6..fc1c7134e 100644 --- a/pkgs/dsl/__tests__/types/context-inference.test-d.ts +++ b/pkgs/dsl/__tests__/types/context-inference.test-d.ts @@ -21,7 +21,7 @@ describe('Context Type Inference Tests', () => { expectTypeOf(context.env).toEqualTypeOf>(); expectTypeOf(context.shutdownSignal).toEqualTypeOf(); expectTypeOf(context.stepTask.run_id).toEqualTypeOf(); - expectTypeOf(context.rawMessage.msg_id).toEqualTypeOf(); + expectTypeOf(context.rawMessage.msg_id).toEqualTypeOf(); return { processed: true }; }); diff --git a/pkgs/dsl/__tests__/types/supabase-context-inference.test-d.ts b/pkgs/dsl/__tests__/types/supabase-context-inference.test-d.ts index 76df19397..7b717ee26 100644 --- a/pkgs/dsl/__tests__/types/supabase-context-inference.test-d.ts +++ b/pkgs/dsl/__tests__/types/supabase-context-inference.test-d.ts @@ -13,7 +13,7 @@ describe('Supabase Flow Context Inference', () => { // FlowContext properties expectTypeOf(context.stepTask.run_id).toEqualTypeOf(); - expectTypeOf(context.rawMessage.msg_id).toEqualTypeOf(); + expectTypeOf(context.rawMessage.msg_id).toEqualTypeOf(); expectTypeOf(context.workerConfig.maxConcurrent).toEqualTypeOf(); expectTypeOf(context.env).toMatchTypeOf>(); expectTypeOf(context.shutdownSignal).toEqualTypeOf(); diff --git a/pkgs/dsl/src/dsl.ts b/pkgs/dsl/src/dsl.ts index 4cc611000..642cd17e8 100644 --- a/pkgs/dsl/src/dsl.ts +++ b/pkgs/dsl/src/dsl.ts @@ -394,7 +394,7 @@ export interface WorkerConfig { // Message record interface (minimal contract - actual type defined in @pgflow/core) export interface MessageRecord { - msg_id: number; + msg_id: string; // decimal-string PGMQ bigint (#650) read_ct: number; enqueued_at: string; vt: string; @@ -406,8 +406,9 @@ export interface StepTaskRecord { flow_slug: string; run_id: string; step_slug: string; + queue_name: string; // canonical physical queue snapshot (#650) input: Json; // JSON-serializable input from database (JSONB column) - msg_id: number; + msg_id: string; // decimal-string PGMQ bigint (#650) } // Base context for queue workers (no stepTask) @@ -740,6 +741,17 @@ export class Flow< return this.stepDefinitions[slug as string]; } + /** + * Case-insensitive duplicate step detection. Case-only aliases would map to + * conflicting generated names downstream, so they are rejected here while + * exact spelling is preserved. + */ + private hasStepWithSlug(slug: string): boolean { + return Object.keys(this.stepDefinitions).some( + (existing) => existing.toLowerCase() === slug.toLowerCase() + ); + } + // Overload 1: Root step without conditions step< Slug extends string, @@ -962,7 +974,7 @@ export class Flow< // Validate the step slug validateSlug(slug); - if (this.stepDefinitions[slug]) { + if (this.hasStepWithSlug(slug)) { throw new Error(`Step "${slug}" already exists in flow "${this.slug}"`); } @@ -1229,7 +1241,7 @@ export class Flow< // Validate the step slug validateSlug(slug); - if (this.stepDefinitions[slug]) { + if (this.hasStepWithSlug(slug)) { throw new Error(`Step "${slug}" already exists in flow "${this.slug}"`); } diff --git a/pkgs/dsl/src/utils.ts b/pkgs/dsl/src/utils.ts index 70844f533..d64d82ddf 100644 --- a/pkgs/dsl/src/utils.ts +++ b/pkgs/dsl/src/utils.ts @@ -31,6 +31,10 @@ export function validateSlug(slug: string): void { `Slug '${slug}' can only contain letters, numbers, and underscores` ); } + + if (slug.startsWith('_') || slug.endsWith('_') || slug.includes('__')) { + throw new Error(`Slug '${slug}' cannot start or end with an underscore or contain double underscores`); + } } /** diff --git a/pkgs/edge-worker/README.md b/pkgs/edge-worker/README.md index 28e761dec..0c9dd3f4f 100644 --- a/pkgs/edge-worker/README.md +++ b/pkgs/edge-worker/README.md @@ -102,10 +102,10 @@ These resources are provided regardless of platform: - **`env`** - Environment variables (`Record`) - **`shutdownSignal`** - AbortSignal for graceful shutdown handling -- **`rawMessage`** - Original pgmq message with metadata +- **`rawMessage`** - Original pgmq message with metadata (msg_id is an exact decimal string; #650) ```typescript interface PgmqMessageRecord { - msg_id: number; + msg_id: string; // decimal-string PGMQ bigint read_ct: number; enqueued_at: Date; vt: Date; @@ -118,11 +118,14 @@ These resources are provided regardless of platform: flow_slug: string; run_id: string; step_slug: string; + queue_name: string; // canonical physical queue snapshot (#650) input: StepInput; - msg_id: number; + msg_id: string; // decimal-string PGMQ bigint } ``` +Queue identity rules (#650): a task message is identified by `(queue_name, message_id)`, not `message_id` alone. Flow workers claim tasks from their flow's canonical lowercase queue; do not send application messages directly into pgflow-owned queues. New workers and the queue-aware database must be upgraded together - a startup protocol mismatch stops the worker with `QueueProtocolMismatchError` instead of polling. + ### Supabase Platform Resources When running on Supabase (the default), these additional resources are available: diff --git a/pkgs/edge-worker/deno.lock b/pkgs/edge-worker/deno.lock index 9cac9b9ba..e204e7024 100644 --- a/pkgs/edge-worker/deno.lock +++ b/pkgs/edge-worker/deno.lock @@ -14,6 +14,8 @@ "jsr:@std/internal@0.224": "0.224.0", "jsr:@std/io@0.225.0": "0.225.0", "jsr:@std/testing@0.224": "0.224.0", + "npm:@pgflow/dsl@0.16.0": "0.16.0", + "npm:@pgflow/edge-worker@0.16.0": "0.16.0", "npm:@supabase/supabase-js@^2.39.0": "2.86.0", "npm:@types/node@*": "22.5.4" }, @@ -75,6 +77,29 @@ } }, "npm": { + "@henrygd/queue@1.2.0": { + "integrity": "sha512-jW/BLSTpcvExDhqJGxtIPgGr2O0IFF8XUNDwEbfCfhrXT8a4xztQ9Lv6U/vbYzYC0xVWn+3zv6YnLUh3bEFUKA==" + }, + "@pgflow/core@0.16.0": { + "integrity": "sha512-kCcPb+qEo1Opr3PVR/dAlf9N9Xh+kBtakvw5yMTy6F3TRV8yyDsX47pjDb3v2DFMgjqLyZWp4y6/e6sszx6kpg==", + "dependencies": [ + "@pgflow/dsl", + "postgres" + ] + }, + "@pgflow/dsl@0.16.0": { + "integrity": "sha512-430U9P1AlbL+479q7xczGyjmOfaU+HKsJl45NRgfZRlYjBPrd9Y/js5L9Xeb1JuD277jF6Qn7UosxUO6Pey5lQ==" + }, + "@pgflow/edge-worker@0.16.0": { + "integrity": "sha512-XbLSmSPjABW+g2SEY9FCWHYb1WmeUy+GfoEO6SpZ3DpBRmjCufxuBsuqRQP5C2h9HUnMSXta9BZYmf693flDSg==", + "dependencies": [ + "@henrygd/queue", + "@pgflow/core", + "@pgflow/dsl", + "@supabase/supabase-js", + "postgres" + ] + }, "@supabase/auth-js@2.86.0": { "integrity": "sha512-3xPqMvBWC6Haqpr6hEWmSUqDq+6SA1BAEdbiaHdAZM9QjZ5uiQJ+6iD9pZOzOa6MVXZh4GmwjhC9ObIG0K1NcA==", "dependencies": [ @@ -137,6 +162,9 @@ "iceberg-js@0.8.0": { "integrity": "sha512-kmgmea2nguZEvRqW79gDqNXyxA3OS5WIgMVffrHpqXV4F/J4UmNIw2vstixioLTNSkd5rFB8G0s3Lwzogm6OFw==" }, + "postgres@3.4.5": { + "integrity": "sha512-cDWgoah1Gez9rN3H4165peY9qfpEo+SA61oQv65O3cRUE1pOEoJWwddwcqKE8XZYjbblOJlYDlLV4h67HrEVDg==" + }, "tslib@2.8.1": { "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==" }, diff --git a/pkgs/edge-worker/src/core/Queries.ts b/pkgs/edge-worker/src/core/Queries.ts index 393e87573..8476b7dc4 100644 --- a/pkgs/edge-worker/src/core/Queries.ts +++ b/pkgs/edge-worker/src/core/Queries.ts @@ -1,13 +1,23 @@ import type postgres from 'postgres'; import type { WorkerRow, WorkerStartMode } from './types.js'; import type { FlowShape, Json } from '@pgflow/dsl'; +import { QueueProtocolMismatchError } from '../flow/errors.js'; export type EnsureFlowCompiledStatus = 'compiled' | 'verified' | 'recompiled' | 'mismatch'; -export interface EnsureFlowCompiledResult { - status: EnsureFlowCompiledStatus; - differences: string[]; -} +/** + * Startup handshake result (#650). Every non-mismatch status carries the + * checked canonical queue and protocol version; the worker validates both + * before registration/polling and does not trust an old-looking result. + */ +export type EnsureFlowCompiledResult = + | { status: 'mismatch'; differences: string[] } + | { + status: Exclude; + differences: string[]; + protocol_version: 1; + queue_name: string; + }; export class Queries { constructor(private readonly sql: postgres.Sql) {} @@ -65,13 +75,32 @@ export class Queries { // TODO: If FlowShape ever becomes part of a public API or accepts external input, // add a runtime assertion function (assertJsonCompatible) to validate at the boundary. const shapeJson = this.sql.json(shape as unknown as Json); - const [result] = await this.sql<{ result: EnsureFlowCompiledResult }[]>` - SELECT pgflow.ensure_flow_compiled( - ${flowSlug}, - ${shapeJson}::jsonb - ) as result - `; - return result.result; + const protocolJson = this.sql.json({ version: 1 } as Json); + let result: { result: EnsureFlowCompiledResult } | undefined; + try { + [result] = await this.sql<{ result: EnsureFlowCompiledResult }[]>` + SELECT pgflow.ensure_flow_compiled( + ${flowSlug}, + ${shapeJson}::jsonb, + ${protocolJson}::jsonb + ) as result + `; + } catch (error) { + // Translate only the missing queue-capable startup function into an + // actionable coordinated-upgrade error; ordinary connection errors stay + // database errors and there is no fallback to the old signature. + if ( + error instanceof Error && + /function pgflow\.ensure_flow_compiled.*does not exist/.test(error.message) + ) { + throw new QueueProtocolMismatchError( + flowSlug, + `The database has no queue-capable pgflow.ensure_flow_compiled(text, jsonb, jsonb) function.` + ); + } + throw error; + } + return result!.result; } /** diff --git a/pkgs/edge-worker/src/core/Worker.ts b/pkgs/edge-worker/src/core/Worker.ts index 7ff49b3e1..14e5a5689 100644 --- a/pkgs/edge-worker/src/core/Worker.ts +++ b/pkgs/edge-worker/src/core/Worker.ts @@ -1,5 +1,6 @@ import type { IBatchProcessor, ILifecycle, WorkerBootstrap } from './types.js'; import type { Logger } from '../platform/types.js'; +import { FatalWorkerError } from './errors.js'; /** Initial delay before retrying a failed main-loop iteration. */ const RETRY_DELAY_MS = 100; @@ -86,6 +87,18 @@ export class Worker { try { await this.batchProcessor.processBatch(); } catch (error: unknown) { + // A committed fatal claim batch is terminal: log once and stop + // without a retry cycle. The scheduled stop below drains/aborts + // execution and cleans up after the loop exits. It must NOT be + // awaited here: performStop() waits for mainLoopPromise, creating + // a self-wait. + if (error instanceof FatalWorkerError) { + this.logger.error(error.message); + void this.stop().catch((stopError) => { + this.logger.error('Worker cleanup failed after fatal batch', stopError); + }); + break; + } this.logger.error(`Error processing batch: ${error}`); iterationFailed = true; } diff --git a/pkgs/edge-worker/src/core/context.ts b/pkgs/edge-worker/src/core/context.ts index 38490d762..e16e3e142 100644 --- a/pkgs/edge-worker/src/core/context.ts +++ b/pkgs/edge-worker/src/core/context.ts @@ -1,6 +1,6 @@ /* DSL‐level ------------------------------------------------------------ */ import type { BaseContext, AnyFlow, AllStepInputs, ExtractFlowInput } from '@pgflow/dsl'; -import type { Json } from './types.js'; +import type { Json, MessageId } from './types.js'; import type { PgmqMessageRecord } from '../queue/types.js'; import type { StepTaskRecord } from '@pgflow/core'; import type { QueueWorkerConfig, FlowWorkerConfig } from './workerConfigTypes.js'; @@ -63,7 +63,7 @@ export type StepTaskContext< * immediately (if provided) or lazy-loads from the runs table. */ export interface StepTaskWithMessage { - msg_id : number; + msg_id : MessageId; message: PgmqMessageRecord>; task : StepTaskRecord; flowInput: ExtractFlowInput | null; diff --git a/pkgs/edge-worker/src/core/errors.ts b/pkgs/edge-worker/src/core/errors.ts new file mode 100644 index 000000000..8f8900fc3 --- /dev/null +++ b/pkgs/edge-worker/src/core/errors.ts @@ -0,0 +1,12 @@ +/** + * Terminal worker error: a committed fatal claim batch. The SQL side already + * committed the visibility reset and the HTTP restart pause; the worker must + * stop without a retry cycle. The message carries only reason/queue/message + * IDs - never message bodies. + */ +export class FatalWorkerError extends Error { + constructor(message: string) { + super(message); + this.name = 'FatalWorkerError'; + } +} diff --git a/pkgs/edge-worker/src/core/types.ts b/pkgs/edge-worker/src/core/types.ts index e74b1e1dc..63e79969f 100644 --- a/pkgs/edge-worker/src/core/types.ts +++ b/pkgs/edge-worker/src/core/types.ts @@ -1,4 +1,7 @@ export type { Json } from '@pgflow/core'; +import type { MessageId } from '@pgflow/core'; + +export type { MessageId }; // TODO: This Supplier pattern is a temporary measure to defer workerId access // until after worker startup. Consider refactoring initialization to pass @@ -11,12 +14,12 @@ export interface IPoller { } export interface IExecutor { - get msgId(): number; + get msgId(): MessageId; execute(): Promise; } export interface IMessage { - msg_id: number; + msg_id: MessageId; } export interface ILifecycle { diff --git a/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts b/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts index 1ca8b6dad..646ba98f0 100644 --- a/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts +++ b/pkgs/edge-worker/src/flow/FlowWorkerLifecycle.ts @@ -4,7 +4,8 @@ import type { Logger, StartupContext } from '../platform/types.js'; import { States, WorkerState } from '../core/WorkerState.js'; import type { AnyFlow } from '@pgflow/dsl'; import { extractFlowShape } from '@pgflow/dsl'; -import { FlowShapeMismatchError } from './errors.js'; +import { FlowShapeMismatchError, QueueProtocolMismatchError } from './errors.js'; +import type { EnsureFlowCompiledResult } from '../core/Queries.js'; export interface FlowLifecycleConfig { heartbeatInterval?: number; @@ -24,6 +25,8 @@ export class FlowWorkerLifecycle implements InternalLifec private queries: Queries; private workerRow?: WorkerRow; private flow: TFlow; + /** Verified canonical physical queue returned by the startup handshake */ + private verifiedQueueName?: string; // TODO: Temporary field for supplier pattern until we refactor initialization private _workerId?: string; private _edgeFunctionName?: string; @@ -53,7 +56,7 @@ export class FlowWorkerLifecycle implements InternalLifec await this.queries.trackWorkerFunction(workerBootstrap.edgeFunctionName, startMode); // Log startup banner with compilation status - this.logStartupBanner(compilationStatus); + this.logStartupBanner(compilationStatus.status); this.workerRow = await this.queries.onWorkerStarted({ queueName: this.queueName, @@ -63,16 +66,37 @@ export class FlowWorkerLifecycle implements InternalLifec this.workerState.transitionTo(States.Running); } - private async ensureFlowCompiled(): Promise { + private async ensureFlowCompiled(): Promise> { const shape = extractFlowShape(this.flow); const result = await this.queries.ensureFlowCompiled(this.flow.slug, shape); + // The runtime mismatch check narrows this union before the queue/version + // checks below. if (result.status === 'mismatch') { throw new FlowShapeMismatchError(this.flow.slug, result.differences); } - return result.status; + // The result is untrusted until these explicit checks pass: a + // queue-aware database must answer with protocol version 1 and the + // flow's canonical queue. + if (result.protocol_version !== 1) { + throw new QueueProtocolMismatchError( + this.flow.slug, + `ensure_flow_compiled() answered without queue-capable protocol version 1.` + ); + } + const canonicalQueue = this.flow.slug.toLowerCase(); + if (result.queue_name !== canonicalQueue) { + throw new QueueProtocolMismatchError( + this.flow.slug, + `ensure_flow_compiled() returned queue '${result.queue_name}' instead of the canonical '${canonicalQueue}'.` + ); + } + + this.verifiedQueueName = result.queue_name; + + return result; } /** @@ -111,8 +135,12 @@ export class FlowWorkerLifecycle implements InternalLifec return this._edgeFunctionName ?? this.workerRow?.function_name; } + /** + * The canonical physical queue: only valid after the startup handshake + * verified it against the database. + */ get queueName() { - return this.flow.slug; + return this.verifiedQueueName ?? this.flow.slug.toLowerCase(); } // TODO: Temporary getter for supplier pattern until we refactor initialization diff --git a/pkgs/edge-worker/src/flow/StepTaskExecutor.ts b/pkgs/edge-worker/src/flow/StepTaskExecutor.ts index 6f932cffd..675f49538 100644 --- a/pkgs/edge-worker/src/flow/StepTaskExecutor.ts +++ b/pkgs/edge-worker/src/flow/StepTaskExecutor.ts @@ -68,8 +68,7 @@ export class StepTaskExecutor `Two-phase polling for flow tasks with batch size ${batchSize}, maxPollSeconds: ${this.config.maxPollSeconds}, pollIntervalMs: ${this.config.pollIntervalMs}` ); - try { - // Phase 1: Read messages from queue - const messages = await this.adapter.readMessages( - this.config.queueName, - this.config.visibilityTimeout ?? 2, - batchSize, - this.config.maxPollSeconds, - this.config.pollIntervalMs - ); - - if (messages.length === 0) { - this.logger.debug('No messages found in queue'); - return []; - } - - this.logger.debug(`Found ${messages.length} messages, starting tasks`); - - // Phase 2: Start tasks for the retrieved messages - const msgIds = messages.map((msg) => msg.msg_id); - const tasks = await this.adapter.startTasks( - this.config.queueName, - msgIds, - workerId - ); - - this.logger.debug( - `Started ${tasks.length} tasks from ${messages.length} messages` - ); - - // Log if we got fewer tasks than messages (indicates some messages had no matching queued tasks) - if (tasks.length < messages.length) { - this.logger.debug( - `Note: Started ${tasks.length} tasks from ${messages.length} messages. ` + - `${ - messages.length - tasks.length - } messages had no queued tasks (may retry later).` - ); - } - - // Create a map of message ID to message for quick lookup - const messageMap = new Map>>(); - for (const msg of messages) { - messageMap.set(msg.msg_id, msg as PgmqMessageRecord>); - } - - // Pair each task with its corresponding message - const taskWithMessages: StepTaskWithMessage[] = tasks - .map(task => { - const message = messageMap.get(task.msg_id); - if (!message) { - this.logger.error(`No message found for task ${task.run_id}:${task.step_slug} with msg_id ${task.msg_id}`); - return null; - } - return { - message, - task, - msg_id: task.msg_id, - flowInput: task.flow_input - }; - }) - .filter((item): item is StepTaskWithMessage => item !== null); - - return taskWithMessages; - } catch (err: unknown) { - this.logger.error(`Error in two-phase polling for flow tasks: ${err}`); - // Rethrow so Worker can distinguish a failed poll (which drives its - // retry backoff) from an empty successful poll. - throw err; + // Phase 1: Read messages from the queue (own transaction; commits before claim) + const messages = await this.adapter.readMessages( + this.config.queueName, + this.config.visibilityTimeout ?? 2, + batchSize, + this.config.maxPollSeconds, + this.config.pollIntervalMs + ); + + if (messages.length === 0) { + this.logger.debug('No messages found in queue'); + return []; + } + + this.logger.debug(`Found ${messages.length} messages, starting tasks`); + + // Phase 2: Claim tasks for the retrieved messages + const msgIds: MessageId[] = messages.map((msg) => msg.msg_id); + const result = await this.adapter.startTasks( + this.config.queueName, + this.config.flowSlug, + msgIds, + workerId + ); + + if (result.status === 'fatal') { + // The SQL side committed the visibility reset and restart pause; stop + // the worker without a retry cycle. Ordinary SQL/network exceptions + // stay ordinary and retryable. + throw new FatalWorkerError(formatDiagnostics(result.errors)); + } + + // Log only supplied body-free warning diagnostics + for (const warning of result.warnings) { + this.logger.warn(formatDiagnostic(warning)); + } + + const tasks = result.tasks; + + this.logger.debug( + `Started ${tasks.length} tasks from ${messages.length} messages` + ); + + // Create a map of message ID to message for quick lookup + const messageMap = new Map>>(); + for (const msg of messages) { + messageMap.set(msg.msg_id, msg as PgmqMessageRecord>); } + + // Pair each task with its corresponding message + const taskWithMessages: StepTaskWithMessage[] = tasks + .map(task => { + const message = messageMap.get(task.msg_id); + if (!message) { + this.logger.error(`No message found for task ${task.run_id}:${task.step_slug} with msg_id ${task.msg_id}`); + return null; + } + return { + message, + task, + msg_id: task.msg_id, + flowInput: task.flow_input + }; + }) + .filter((item): item is StepTaskWithMessage => item !== null); + + return taskWithMessages; } private isAborted(): boolean { return this.signal.aborted; } } + +function formatDiagnostic(diagnostic: ClaimDiagnostic): string { + return `Claim warning: reason=${diagnostic.reason} queue=${diagnostic.queue_name} message_id=${diagnostic.message_id}`; +} + +function formatDiagnostics(diagnostics: ClaimDiagnostic[]): string { + return [ + 'Fatal claim batch: the worker must stop (visibility reset and HTTP restart pause are committed).', + ...diagnostics.map(formatDiagnostic), + ].join(' | '); +} diff --git a/pkgs/edge-worker/src/flow/createFlowWorker.ts b/pkgs/edge-worker/src/flow/createFlowWorker.ts index ceb68a135..82f0df7e0 100644 --- a/pkgs/edge-worker/src/flow/createFlowWorker.ts +++ b/pkgs/edge-worker/src/flow/createFlowWorker.ts @@ -95,8 +95,10 @@ export function createFlowWorker< // Create the pgflow adapter const pgflowAdapter = new PgflowSqlClient(sql); - // Use flow slug as queue name, or fallback to 'tasks' - const queueName = flow.slug || 'tasks'; + // Use the canonical physical queue derived from the flow slug; the + // lifecycle's handshake equality check must pass before the main loop can + // use it. A flow slug is mandatory: no 'tasks' fallback (#650). + const queueName = flow.slug.toLowerCase(); logger.debug(`Using queue name: ${queueName}`); // Create specialized FlowWorkerLifecycle with the proxied queue and flow @@ -113,10 +115,12 @@ export function createFlowWorker< // Create FlowInputProvider for lazy loading and caching flow input const flowInputProvider = new FlowInputProvider(sql); - // Create StepTaskPoller with two-phase approach + // Create StepTaskPoller with two-phase approach: queue (physical + // subscription) and flow (handler identity) travel separately (#650) const pollerConfig: StepTaskPollerConfig = { batchSize: resolvedConfig.batchSize, - queueName: flow.slug, + queueName, + flowSlug: flow.slug, visibilityTimeout: resolvedConfig.visibilityTimeout, maxPollSeconds: resolvedConfig.maxPollSeconds, pollIntervalMs: resolvedConfig.pollIntervalMs, diff --git a/pkgs/edge-worker/src/flow/errors.ts b/pkgs/edge-worker/src/flow/errors.ts index b9301f541..24aa335f2 100644 --- a/pkgs/edge-worker/src/flow/errors.ts +++ b/pkgs/edge-worker/src/flow/errors.ts @@ -16,3 +16,23 @@ export class FlowShapeMismatchError extends Error { this.name = 'FlowShapeMismatchError'; } } + +/** + * Error thrown when the database's worker startup protocol does not match + * this worker's queue-capable protocol (#650). Old and new sides must be + * upgraded together; rolling old/new workers are unsupported. + */ +export class QueueProtocolMismatchError extends Error { + constructor( + public readonly flowSlug: string, + detail: string + ) { + super( + `Flow '${flowSlug}' requires a coordinated pgflow upgrade.\n` + + `${detail}\n` + + `Deploy the queue-aware pgflow packages and database migration together; ` + + `running old and new workers side by side is unsupported.` + ); + this.name = 'QueueProtocolMismatchError'; + } +} diff --git a/pkgs/edge-worker/src/queue/Queue.ts b/pkgs/edge-worker/src/queue/Queue.ts index 081e857c4..dd4d338a1 100644 --- a/pkgs/edge-worker/src/queue/Queue.ts +++ b/pkgs/edge-worker/src/queue/Queue.ts @@ -42,7 +42,7 @@ export class Queue { `; } - async archive(msgId: number): Promise { + async archive(msgId: string): Promise { this.logger.debug( `Archiving message ${msgId} from queue '${this.queueName}'` ); @@ -51,7 +51,7 @@ export class Queue { `; } - async archiveBatch(msgIds: number[]): Promise { + async archiveBatch(msgIds: string[]): Promise { this.logger.debug( `Archiving ${msgIds.length} messages from queue '${this.queueName}'` ); @@ -77,8 +77,10 @@ export class Queue { this.logger.debug( `Reading messages from queue '${this.queueName}' with poll` ); + // msg_id is projected to text so PGMQ bigint IDs cross the JavaScript + // boundary without precision loss (#650) return await this.sql[]>` - SELECT * + SELECT msg_id::text as msg_id, read_ct, enqueued_at, vt, message, headers FROM pgmq.read_with_poll( queue_name => ${this.queueName}, vt => ${visibilityTimeout}, @@ -99,17 +101,17 @@ export class Queue { * The only change made is now() replaced with clock_timestamp(). */ async setVt( - msgId: number, + msgId: string, vtOffsetSeconds: number ): Promise> { this.logger.debug( `Setting visibility timeout for message ${msgId} to ${vtOffsetSeconds} seconds` ); - const records = await this.sql[]>` + const records = await this.sql<(PgmqMessageRecord & { msg_id: string })[]>` UPDATE ${this.sql('pgmq.q_' + this.queueName)} SET vt = (clock_timestamp() + make_interval(secs => ${vtOffsetSeconds})) WHERE msg_id = ${msgId}::bigint - RETURNING *; + RETURNING msg_id::text as msg_id, read_ct, enqueued_at, vt, message, headers; `; return records[0]; } diff --git a/pkgs/edge-worker/tests/integration/flow/compilationAtStartup.test.ts b/pkgs/edge-worker/tests/integration/flow/compilationAtStartup.test.ts index 85f605301..f8e1e7d3d 100644 --- a/pkgs/edge-worker/tests/integration/flow/compilationAtStartup.test.ts +++ b/pkgs/edge-worker/tests/integration/flow/compilationAtStartup.test.ts @@ -379,13 +379,15 @@ Deno.test( try { // Fire all compilations simultaneously on separate connections - // Note: Must use conn.json() for proper jsonb parameter passing + // Note: Must use conn.json() for proper jsonb parameter passing; the + // required queue-capable protocol argument is part of the call (#650). const results = await Promise.all( connections.map( (conn) => conn`SELECT pgflow.ensure_flow_compiled( ${flowSlug}, - ${conn.json(shape)} + ${conn.json(shape)}, + ${conn.json({ version: 1 })}::jsonb ) as result` ) ); diff --git a/pkgs/edge-worker/tests/integration/flow/queueClaim.test.ts b/pkgs/edge-worker/tests/integration/flow/queueClaim.test.ts new file mode 100644 index 000000000..4710fda93 --- /dev/null +++ b/pkgs/edge-worker/tests/integration/flow/queueClaim.test.ts @@ -0,0 +1,151 @@ +import { assertEquals } from '@std/assert'; +import { withPgNoTransaction } from '../../db.ts'; +import { Flow } from '@pgflow/dsl'; +import { createFlowWorker } from '../../../src/flow/createFlowWorker.ts'; +import { createTestPlatformAdapter } from '../_helpers.ts'; +import { fakeLogger } from '../../fakes.ts'; +import { Queue } from '../../../src/queue/Queue.ts'; +import type { postgres } from '../../sql.ts'; +import { delay } from '@std/async'; + +// #650 queue identity end-to-end claim: PGMQ identity sequence starts above +// Number.MAX_SAFE_INTEGER, so every ID that crosses the JavaScript boundary +// must survive as an exact decimal string — in contexts, in the client, and +// in queue/archive operations. A numeric round-trip anywhere in the path +// silently corrupts these IDs (9007199254740993 === 9007199254740992 in +// IEEE-754 doubles). +const MAX_SAFE = 9007199254740992; +const ID_FIRST = '9007199254740993'; +const ID_SECOND = '9007199254740994'; + +const ClaimFlow = new Flow({ + slug: 'queue_claim_flow', +}) + .map({ slug: 'work' }, (input: number, ctx) => { + // Both exposure paths must carry the exact decimal-string ID. + const rawId = ctx.rawMessage.msg_id; + const taskId = ctx.stepTask.msg_id; + if (typeof rawId !== 'string' || typeof taskId !== 'string') { + throw new Error( + `msg_id lost string identity: raw=${typeof rawId} task=${typeof taskId}` + ); + } + if (rawId !== taskId) { + throw new Error(`cross-ID operation: raw=${rawId} task=${taskId}`); + } + if (BigInt(rawId) <= BigInt(MAX_SAFE)) { + throw new Error(`unexpected small msg_id ${rawId}`); + } + if (input === 2) { + // Fails and requeues; the deadline loop below waits for exhaustion + // and archival by queue snapshot. + throw new Error('intentional failure for the second item'); + } + return input * 10; + }); + +Deno.test( + 'queue claim keeps bigint message IDs lossless end-to-end', + withPgNoTransaction(async (sql: postgres.Sql) => { + await sql`select pgflow_tests.reset_db()`; + + // Seed the flow definition through the new canonical path. + const worker = createFlowWorker( + ClaimFlow, + { sql, maxConcurrent: 1, batchSize: 10, maxPollSeconds: 1, pollIntervalMs: 100 }, + () => fakeLogger, + createTestPlatformAdapter(sql) + ); + await worker.startOnlyOnce({ + edgeFunctionName: 'queue_claim_test', + workerId: crypto.randomUUID(), + }); + + // Push the identity sequence above MAX_SAFE_INTEGER before producing. + await sql` + select setval( + pg_get_serial_sequence('pgmq.q_queue_claim_flow', 'msg_id'), + ${MAX_SAFE}::bigint, true + ) + `; + + const [run] = await sql<{ run_id: string }[]>` + select run_id from pgflow.start_flow( + 'queue_claim_flow', '[1, 2]'::jsonb + ) + `; + assertEquals(run.run_id.length, 36, 'run started'); + + // Raw queue read shares the same string ID contract (#650). + const rawQueue = new Queue(sql, 'queue_claim_flow', fakeLogger); + const raw = await rawQueue.readWithPoll(10, 5, 5, 100); + assertEquals(raw.length, 2, 'raw queue read sees both messages'); + for (const msg of raw) { + assertEquals(typeof msg.msg_id, 'string', 'raw queue msg_id is a string'); + assertEquals( + BigInt(msg.msg_id) > BigInt(MAX_SAFE), + true, + `raw queue msg_id above MAX_SAFE_INTEGER: ${msg.msg_id}` + ); + } + + // Requeue both for the worker (the raw read moved visibility). + await sql` + select pgflow.set_vt_batch( + 'queue_claim_flow', + array[${raw[0].msg_id}::bigint, ${raw[1].msg_id}::bigint], + array[0, 0] + ) + `; + + try { + // Let the worker claim and execute both tasks. + const deadline = Date.now() + 60_000; + for (;;) { + const [{ done }] = await sql<{ done: boolean }[]>` + select not exists ( + select 1 from pgflow.step_tasks + where flow_slug = 'queue_claim_flow' and status in ('queued', 'started') + ) as done + `; + if (done || Date.now() > deadline) break; + await delay(200); + } + } finally { + await worker.stop(); + } + + const tasks = await sql< + { task_index: number; status: string; message_id: string | null; output: unknown }[] + >` + select task_index, status, message_id::text as message_id, output + from pgflow.step_tasks + where flow_slug = 'queue_claim_flow' + order by task_index + `; + assertEquals(tasks.length, 2, 'both tasks present'); + + const [first, second] = [...tasks].sort((a, b) => (a.message_id! < b.message_id! ? -1 : 1)); + assertEquals(first.status, 'completed', 'lower-ID task completed'); + assertEquals(first.output, 10, 'lower-ID task output is its own'); + assertEquals(first.message_id, ID_FIRST, 'first task exact string ID'); + assertEquals(second.status, 'failed', 'higher-ID task exhausted'); + assertEquals(second.message_id, ID_SECOND, 'second task exact string ID'); + + // The exhausted task's message was archived via its queue snapshot; the + // completed task's message was archived by complete_task. No active + // message remains, and both archive rows keep their exact IDs. + const [{ active }] = await sql<{ active: number }[]>` + select count(*)::int as active from pgmq.q_queue_claim_flow + `; + assertEquals(active, 0, 'no active messages remain'); + const archived = await sql<{ msg_id: string }[]>` + select msg_id::text as msg_id from pgmq.a_queue_claim_flow order by msg_id + `; + assertEquals( + archived.map((a) => a.msg_id), + [ID_FIRST, ID_SECOND], + 'archive holds exactly both string IDs' + ); + }) +); diff --git a/pkgs/edge-worker/tests/integration/flow/queueStartup.test.ts b/pkgs/edge-worker/tests/integration/flow/queueStartup.test.ts new file mode 100644 index 000000000..e1f9d8ed9 --- /dev/null +++ b/pkgs/edge-worker/tests/integration/flow/queueStartup.test.ts @@ -0,0 +1,219 @@ +import { assertRejects, assertEquals } from '@std/assert'; +import { withPgNoTransaction } from '../../db.ts'; +import { Flow } from '@pgflow/dsl'; +import { createFlowWorker } from '../../../src/flow/createFlowWorker.ts'; +import { QueueProtocolMismatchError } from '../../../src/flow/errors.ts'; +import { createTestPlatformAdapter } from '../_helpers.ts'; +import { fakeLogger } from '../../fakes.ts'; +import type { postgres } from '../../sql.ts'; + +// #650 startup compatibility matrix (migrated database side): +// 1. migrated DB + new plain worker -> compiles/verifies, registers the +// canonical queue, handlers run with exact camelCase step identity; +// 2. migrated DB + old-looking handshake answer -> rejected before +// registration (the result is untrusted until protocol/queue checks +// pass); +// 3. migrated DB + released 0.16.0 worker startup call -> the two-argument +// ensure_flow_compiled lookup fails before any worker/function +// registration or polling. +// The opposite direction (0.16.0 database + new worker) runs in the +// upgrade_queue_fixture startup probe, not against this stack. + +const OrdersFlow = new Flow<{ order: string }>({ slug: 'StartupOrders' }).step( + { slug: 'saveItem' }, + () => 'saved' +); + +Deno.test( + 'new worker verifies and registers the canonical queue', + withPgNoTransaction(async (sql: postgres.Sql) => { + await sql`select pgflow_tests.reset_db()`; + + const worker = createFlowWorker( + OrdersFlow, + { sql, maxConcurrent: 1, batchSize: 5, maxPollSeconds: 1, pollIntervalMs: 100 }, + () => fakeLogger, + createTestPlatformAdapter(sql) + ); + await worker.startOnlyOnce({ + edgeFunctionName: 'startup_orders_worker', + workerId: crypto.randomUUID(), + }); + + try { + // Canonical queue was provisioned through the new handshake and the + // worker registered against it with the lowercase physical name. + const [registered] = await sql<{ queue_name: string }[]>` + select queue_name from pgflow.workers + where function_name = 'startup_orders_worker' + `; + assertEquals(registered.queue_name, 'startuporders'); + + // Exact camelCase step identity survives compilation. + const steps = await sql<{ step_slug: string; queue_name: string }[]>` + select step_slug, queue_name from pgflow.steps + where flow_slug = 'StartupOrders' + `; + assertEquals(steps.length, 1); + assertEquals(steps[0].step_slug, 'saveItem'); + assertEquals(steps[0].queue_name, 'startuporders'); + + // A run executes through the canonical route. + await sql` + select run_id from pgflow.start_flow('StartupOrders', '{"order":"o1"}'::jsonb) + `; + const [{ queued }] = await sql<{ queued: number }[]>` + select count(*)::int as queued from pgflow.step_tasks + where flow_slug = 'StartupOrders' and queue_name = 'startuporders' + `; + assertEquals(queued, 1, 'task snapshotted the canonical queue'); + } finally { + await worker.stop(); + } + }) +); + +Deno.test( + 'new worker rejects an old-looking handshake answer before registration', + withPgNoTransaction(async (sql: postgres.Sql) => { + await sql`select pgflow_tests.reset_db()`; + + // Tamper with the database answer so it looks like a pre-#650 result: + // no protocol_version, no queue_name. The worker must not trust it. + // The real handshake is restored afterwards: reset_db clears rows only. + const restoreUrl = new URL( + '../../../../core/schemas/0100_function_ensure_flow_compiled.sql', + import.meta.url + ); + try { + await sql` + create or replace function pgflow.ensure_flow_compiled( + flow_slug text, shape jsonb, worker_protocol jsonb + ) returns jsonb language sql as $fn$ + select jsonb_build_object('status', 'verified', 'differences', '[]'::jsonb) + $fn$; + `; + + const worker = createFlowWorker( + OrdersFlow, + { sql, maxConcurrent: 1, batchSize: 5, maxPollSeconds: 1, pollIntervalMs: 100 }, + () => fakeLogger, + createTestPlatformAdapter(sql) + ); + + await assertRejects( + () => + worker.startOnlyOnce({ + edgeFunctionName: 'startup_orders_tampered', + workerId: crypto.randomUUID(), + }), + QueueProtocolMismatchError + ); + + const [{ n }] = await sql<{ n: number }[]>` + select count(*)::int as n from pgflow.workers + where function_name = 'startup_orders_tampered' + `; + assertEquals(n, 0, 'no worker registered from an untrusted answer'); + } finally { + await sql.file(restoreUrl); + } + }) +); + +Deno.test( + 'released 0.16.0 startup call fails before registration on a migrated database', + withPgNoTransaction(async (sql: postgres.Sql) => { + await sql`select pgflow_tests.reset_db()`; + + // Ordinary SQL signature probe: the fast explanation of the failure. + const err = await sql`select pgflow.ensure_flow_compiled('x', '{}'::jsonb)` + .catch((e: Error) => e); + assertEquals( + err instanceof Error && /function pgflow.ensure_flow_compiled.*does not exist/i.test(err.message), + true, + `two-argument lookup must fail: ${String(err)}` + ); + + // The real released worker through pinned test-only fixture imports. + // These are fixture dependencies, not restored production APIs (#650). + // A missing import or export fails the test: the released-worker + // compatibility direction must never silently pass as untested. + const internal = await import('npm:@pgflow/edge-worker@0.16.0/_internal'); + // The published _internal entry nests createFlowWorker (default/core/flow). + const carrier = (internal as { + default?: { createFlowWorker?: unknown }; + core?: { createFlowWorker?: unknown }; + flow?: { createFlowWorker?: unknown }; + }); + const releasedCreate = carrier.default?.createFlowWorker + ?? carrier.core?.createFlowWorker + ?? carrier.flow?.createFlowWorker; + const dsl = await import('npm:@pgflow/dsl@0.16.0'); + const releasedFlow = (dsl as { Flow?: unknown }).Flow; + assertEquals( + typeof releasedCreate === 'function' && typeof releasedFlow === 'function', + true, + 'pinned 0.16.0 fixture exports must expose createFlowWorker and Flow' + ); + + { + const OldFlow = releasedFlow as new (opts: { slug: string }) => { + step: (o: { slug: string }, h: () => unknown) => unknown; + }; + const oldFlow = new OldFlow({ slug: 'StartupOrders' }).step( + { slug: 'saveItem' }, + () => 'saved' + ); + const oldCreate = releasedCreate as ( + flow: unknown, + opts: Record, + logger: () => unknown, + adapter: unknown + ) => { startOnlyOnce: (b: { edgeFunctionName: string; workerId: string }) => Promise }; + + const oldWorker = oldCreate( + oldFlow, + { sql, maxConcurrent: 1, batchSize: 5 }, + () => fakeLogger, + createTestPlatformAdapter(sql) + ); + + const [beforeWorkers] = await sql<{ n: number }[]>` + select count(*)::int as n from pgflow.workers + `; + const [beforeFunctions] = await sql<{ n: number }[]>` + select count(*)::int as n from pgflow.worker_functions + `; + + let releasedError: unknown; + try { + await oldWorker.startOnlyOnce({ + edgeFunctionName: 'startup_orders_old_worker', + workerId: crypto.randomUUID(), + }); + } catch (e) { + releasedError = e; + } + // The released worker must fail at the removed two-argument startup + // signature itself - not at an adapter, fixture, or unrelated error. + assertEquals( + releasedError instanceof Error && + /function pgflow\.ensure_flow_compiled\(unknown, jsonb\) does not exist/ + .test(releasedError.message), + true, + `released 0.16.0 startup must fail at the removed signature: ${String(releasedError)}` + ); + + const [afterWorkers] = await sql<{ n: number }[]>` + select count(*)::int as n from pgflow.workers + `; + const [afterFunctions] = await sql<{ n: number }[]>` + select count(*)::int as n from pgflow.worker_functions + `; + assertEquals(afterWorkers.n, beforeWorkers.n, 'no worker registered'); + assertEquals(afterFunctions.n, beforeFunctions.n, 'no function registered'); + } + }) +); + diff --git a/pkgs/edge-worker/tests/integration/messageExecutorContext.test.ts b/pkgs/edge-worker/tests/integration/messageExecutorContext.test.ts index e42821887..7849bf312 100644 --- a/pkgs/edge-worker/tests/integration/messageExecutorContext.test.ts +++ b/pkgs/edge-worker/tests/integration/messageExecutorContext.test.ts @@ -22,7 +22,7 @@ Deno.test( await queue.safeCreate(); const mockMessage: PgmqMessageRecord<{ data: string }> = { - msg_id: 123, + msg_id: '123', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -79,7 +79,7 @@ Deno.test( await queue.safeCreate(); const mockMessage: PgmqMessageRecord<{ data: string }> = { - msg_id: 456, + msg_id: '456', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -113,7 +113,7 @@ Deno.test( await queue.safeCreate(); const mockMessage: PgmqMessageRecord<{ id: number; name: string }> = { - msg_id: 789, + msg_id: '789', read_ct: 2, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -146,7 +146,7 @@ Deno.test( // Verify rawMessage in context matches the original message assertExists(receivedRawMessage); - assertEquals(receivedRawMessage.msg_id, 789); + assertEquals(receivedRawMessage.msg_id, '789'); assertEquals(receivedRawMessage.read_ct, 2); assertEquals(receivedRawMessage.message, { id: 42, name: 'test item' }); }) @@ -156,7 +156,7 @@ Deno.test( 'MessageExecutor - Supabase clients are available when env vars exist', withTransaction(async (sql) => { const mockMessage: PgmqMessageRecord<{ test: string }> = { - msg_id: 999, + msg_id: '999', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', diff --git a/pkgs/edge-worker/tests/integration/messageExecutorContextWorkerConfig.test.ts b/pkgs/edge-worker/tests/integration/messageExecutorContextWorkerConfig.test.ts index 8f42da49a..64f8166c3 100644 --- a/pkgs/edge-worker/tests/integration/messageExecutorContextWorkerConfig.test.ts +++ b/pkgs/edge-worker/tests/integration/messageExecutorContextWorkerConfig.test.ts @@ -29,7 +29,7 @@ Deno.test( }; const mockMessage: PgmqMessageRecord<{test: string}> = { - msg_id: 123, + msg_id: '123', read_ct: 2, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', diff --git a/pkgs/edge-worker/tests/integration/stepTaskExecutorContext.test.ts b/pkgs/edge-worker/tests/integration/stepTaskExecutorContext.test.ts index 534c7b322..3b8c95076 100644 --- a/pkgs/edge-worker/tests/integration/stepTaskExecutorContext.test.ts +++ b/pkgs/edge-worker/tests/integration/stepTaskExecutorContext.test.ts @@ -53,7 +53,8 @@ Deno.test( // Mock step task record - root steps get flow input directly const mockTask: StepTaskRecord = { flow_slug: 'context_test_flow', - msg_id: 123, + queue_name: 'context_test_flow', + msg_id: '123', run_id: 'test-run-id', step_slug: 'test_step', task_index: 0, @@ -63,7 +64,7 @@ Deno.test( // Create context with mock task and message using proper flow worker context creation const mockMessage = { - msg_id: 123, + msg_id: '123', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -75,7 +76,7 @@ Deno.test( sql: _sql, abortSignal: abortController.signal, taskWithMessage: { - msg_id: 123, + msg_id: '123', message: mockMessage, task: mockTask, flowInput: { data: 'test data' }, @@ -117,7 +118,8 @@ Deno.test( // Mock step task record - input is the unwrapped flowInput for root steps const mockTask: StepTaskRecord = { flow_slug: 'legacy_flow', - msg_id: 456, + queue_name: 'legacy_flow', + msg_id: '456', run_id: 'legacy_run_id', step_slug: 'legacy_step', task_index: 0, @@ -130,7 +132,7 @@ Deno.test( // Create proper context for legacy handler test const mockMessage = { - msg_id: 456, + msg_id: '456', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -142,7 +144,7 @@ Deno.test( sql: _sql, abortSignal: new AbortController().signal, taskWithMessage: { - msg_id: 456, + msg_id: '456', message: mockMessage, task: mockTask, flowInput: { value: 42 }, @@ -176,7 +178,7 @@ Deno.test( // Mock message - root steps get flow input directly (empty object for this flow) const mockMessage = { - msg_id: 789, + msg_id: '789', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -186,7 +188,8 @@ Deno.test( // Mock step task record - root steps get flow input directly const mockTask: StepTaskRecord = { flow_slug: 'rawmessage_flow', - msg_id: 789, + queue_name: 'rawmessage_flow', + msg_id: '789', run_id: 'raw_run_id', step_slug: 'check_raw', task_index: 0, @@ -196,7 +199,7 @@ Deno.test( // Create context - for this test we need a mock taskWithMessage const mockTaskWithMessage = { - msg_id: 789, + msg_id: '789', message: mockMessage, task: mockTask, flowInput: {}, @@ -238,7 +241,7 @@ Deno.test( // Mock message - root steps get flow input directly (empty object for this flow) const mockMessage = { - msg_id: 999, + msg_id: '999', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -248,7 +251,8 @@ Deno.test( // Mock step task record const mockTask: StepTaskRecord = { flow_slug: 'supabase_flow', - msg_id: 999, + queue_name: 'supabase_flow', + msg_id: '999', run_id: 'supabase_run_id', step_slug: 'check_clients', task_index: 0, @@ -258,7 +262,7 @@ Deno.test( // Create context with Supabase env vars const mockTaskWithMessage = { - msg_id: 999, + msg_id: '999', message: mockMessage, task: mockTask, flowInput: {}, @@ -322,7 +326,7 @@ Deno.test( // Create context - root steps get flow input directly const mockMessageForComplex = { - msg_id: 456, + msg_id: '456', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -331,7 +335,8 @@ Deno.test( const mockTaskForComplex: StepTaskRecord = { flow_slug: 'complex_context_flow', - msg_id: 456, + queue_name: 'complex_context_flow', + msg_id: '456', run_id: 'complex_run', step_slug: 'fetch_data', task_index: 0, @@ -344,7 +349,7 @@ Deno.test( sql: _sql, abortSignal: abortController.signal, taskWithMessage: { - msg_id: 456, + msg_id: '456', message: mockMessageForComplex, task: mockTaskForComplex, flowInput: { id: 123 }, diff --git a/pkgs/edge-worker/tests/integration/upgrade/startup_probe.ts b/pkgs/edge-worker/tests/integration/upgrade/startup_probe.ts new file mode 100644 index 000000000..5b0b7ca66 --- /dev/null +++ b/pkgs/edge-worker/tests/integration/upgrade/startup_probe.ts @@ -0,0 +1,109 @@ +// #650 upgrade fixture startup probe: the real new worker against a populated +// 0.16.0 database. Requires PGFLOW_UPGRADE_DB_URL (published loopback port of +// the fixture container); never falls back to an ordinary integration +// database. Invoked by scripts/run-queue-upgrade-fixture while the old schema +// and data are still installed. Deno script, not a test file. +import postgres from 'postgres'; +import { Flow } from '@pgflow/dsl'; +import { createFlowWorker } from '../../../src/flow/createFlowWorker.ts'; +import { QueueProtocolMismatchError } from '../../../src/flow/errors.ts'; +import { createTestPlatformAdapter } from '../_helpers.ts'; +import { fakeLogger } from '../../fakes.ts'; + +const dbUrl = Deno.env.get('PGFLOW_UPGRADE_DB_URL'); + +// The failed worker keeps a heartbeat/stop path that may write after the +// probe closes its SQL connection; those dead-socket rejections are expected +// and must not fail the probe. The handler is named and removed before the +// verification connection is created: a CONNECTION_ENDED rejection from +// captureState(verify) is a real verification failure and must fail the +// probe (Deno exits non-zero on unhandled rejections by default). +const suppressConnectionEnded = (event: PromiseRejectionEvent) => { + if (String(event.reason).includes('CONNECTION_ENDED')) { + event.preventDefault(); + } +}; +globalThis.addEventListener('unhandledrejection', suppressConnectionEnded); +if (!dbUrl) { + console.error('startup_probe: PGFLOW_UPGRADE_DB_URL is required'); + Deno.exit(1); +} +if (dbUrl.includes('@127.0.0.1:5432') || dbUrl.includes('localhost:5432')) { + console.error('startup_probe: refusing the ordinary integration database port'); + Deno.exit(1); +} + +const sql = postgres(dbUrl, { prepare: false, onnotice: () => {} }); + +interface WorkerCountRow { + workers: string; + worker_functions: string; + queue_rows: string; +} + +async function captureState(client: postgres.Sql): Promise { + const [row] = await client` + select + (select coalesce(string_agg(worker_id::text || ':' || queue_name, ',' order by worker_id::text), '') + from pgflow.workers) as workers, + (select coalesce(string_agg(function_name || ':' || enabled::text, ',' order by function_name), '') + from pgflow.worker_functions) as worker_functions, + (select count(*)::text from pgmq.q_orders) as queue_rows + `; + return row; +} + +const before = await captureState(sql); + +const OrdersFlow = new Flow<{ order: string }>({ slug: 'Orders' }).step( + { slug: 'saveItem' }, + () => null, +); + +const worker = createFlowWorker( + OrdersFlow, + { sql, maxConcurrent: 1, batchSize: 10 }, + () => fakeLogger, + createTestPlatformAdapter(sql), +); + +try { + await worker.startOnlyOnce({ + edgeFunctionName: 'orders_worker', + workerId: crypto.randomUUID(), + }); + console.error('startup_probe: startup unexpectedly succeeded on the old database'); + Deno.exit(1); +} catch (error) { + const mismatch = error instanceof QueueProtocolMismatchError; + if (!mismatch) { + console.error('startup_probe: expected QueueProtocolMismatchError, got:', error); + Deno.exit(1); + } + console.log('startup_probe: protocol mismatch as required'); + console.log(String(error).split('\n')[0]); +} finally { + await new Promise((resolve) => setTimeout(resolve, 1500)); + await sql.end(); +} + +// Reconnect: the old database must be untouched after the failed startup. +// captureState runs on the fresh connection, so a CONNECTION_ENDED rejection +// here is a real verification failure and must fail the probe. The cleanup +// handler is removed first so it can no longer suppress any rejection. +globalThis.removeEventListener('unhandledrejection', suppressConnectionEnded); +const verify = postgres(dbUrl, { prepare: false, onnotice: () => {} }); +try { + const after = await captureState(verify); + if ( + after.workers !== before.workers || + after.worker_functions !== before.worker_functions || + after.queue_rows !== before.queue_rows + ) { + console.error('startup_probe: state changed after failed startup', { before, after }); + Deno.exit(1); + } + console.log('startup_probe: PASS (mismatch before registration, state unchanged)'); +} finally { + await verify.end(); +} diff --git a/pkgs/edge-worker/tests/unit/FlowWorkerLifecycle.compilation.test.ts b/pkgs/edge-worker/tests/unit/FlowWorkerLifecycle.compilation.test.ts index 74b4466b7..6a0e2350b 100644 --- a/pkgs/edge-worker/tests/unit/FlowWorkerLifecycle.compilation.test.ts +++ b/pkgs/edge-worker/tests/unit/FlowWorkerLifecycle.compilation.test.ts @@ -15,6 +15,8 @@ class MockQueries extends Queries { nextCompilationResult: EnsureFlowCompiledResult = { status: 'verified', differences: [], + protocol_version: 1, + queue_name: 'test_flow', }; constructor() { diff --git a/pkgs/edge-worker/tests/unit/FlowWorkerLifecycle.deprecation.test.ts b/pkgs/edge-worker/tests/unit/FlowWorkerLifecycle.deprecation.test.ts index e524b8d70..8d6a39fe6 100644 --- a/pkgs/edge-worker/tests/unit/FlowWorkerLifecycle.deprecation.test.ts +++ b/pkgs/edge-worker/tests/unit/FlowWorkerLifecycle.deprecation.test.ts @@ -49,7 +49,7 @@ class MockQueries extends Queries { _flowSlug: string, _shape: FlowShape ): Promise { - return Promise.resolve({ status: 'verified', differences: [] }); + return Promise.resolve({ status: 'verified', differences: [], protocol_version: 1, queue_name: 'test_flow' }); } override trackWorkerFunction(_functionName: string): Promise { diff --git a/pkgs/edge-worker/tests/unit/Poller.batchSize.test.ts b/pkgs/edge-worker/tests/unit/Poller.batchSize.test.ts index 44cd94f91..dfbf126a8 100644 --- a/pkgs/edge-worker/tests/unit/Poller.batchSize.test.ts +++ b/pkgs/edge-worker/tests/unit/Poller.batchSize.test.ts @@ -104,6 +104,7 @@ Deno.test('StepTaskPoller caps limit at configured batchSize', async () => { { batchSize: 5, queueName: 'test_flow', + flowSlug: 'TestFlow', visibilityTimeout: 10, maxPollSeconds: 1, pollIntervalMs: 100, @@ -137,6 +138,7 @@ Deno.test('StepTaskPoller uses smaller available slot limit', async () => { { batchSize: 5, queueName: 'test_flow', + flowSlug: 'TestFlow', visibilityTimeout: 10, maxPollSeconds: 1, pollIntervalMs: 100, @@ -170,6 +172,7 @@ Deno.test('StepTaskPoller uses configured batchSize without limit', async () => { batchSize: 5, queueName: 'test_flow', + flowSlug: 'TestFlow', visibilityTimeout: 10, maxPollSeconds: 1, pollIntervalMs: 100, @@ -195,6 +198,7 @@ Deno.test('StepTaskPoller rethrows readMessages failures instead of returning an { batchSize: 5, queueName: 'test_flow', + flowSlug: 'TestFlow', visibilityTimeout: 10, maxPollSeconds: 1, pollIntervalMs: 100, diff --git a/pkgs/edge-worker/tests/unit/StepTaskPoller.classification.test.ts b/pkgs/edge-worker/tests/unit/StepTaskPoller.classification.test.ts new file mode 100644 index 000000000..865ec30fc --- /dev/null +++ b/pkgs/edge-worker/tests/unit/StepTaskPoller.classification.test.ts @@ -0,0 +1,162 @@ +import { assertEquals, assertRejects } from '@std/assert'; +import { StepTaskPoller } from '../../src/flow/StepTaskPoller.ts'; +import { FatalWorkerError } from '../../src/core/errors.ts'; +import { fakeLogger } from '../fakes.ts'; + +interface RecordedLogger { + warnings: string[]; + errors: string[]; +} + +function recordingLogger(): { logger: typeof fakeLogger; recorded: RecordedLogger } { + const recorded: RecordedLogger = { warnings: [], errors: [] }; + const logger = { + ...fakeLogger, + warn: (message: string) => recorded.warnings.push(message), + error: (message: string) => recorded.errors.push(message), + }; + return { logger, recorded }; +} + +const BIG_ID = '9007199254740993'; + +function makeAdapter(overrides: Record = {}) { + return { + readMessages: () => + Promise.resolve([ + { + msg_id: BIG_ID, + read_ct: 1, + enqueued_at: '2026-01-01T00:00:00Z', + vt: '2026-01-01T00:00:02Z', + message: { flow_slug: 'Orders', step_slug: 'saveItem' }, + }, + { + msg_id: '9007199254740994', + read_ct: 1, + enqueued_at: '2026-01-01T00:00:00Z', + vt: '2026-01-01T00:00:02Z', + message: { hello: 'SECRET_BODY_TOKEN' }, + }, + ]), + startTasks: () => Promise.resolve({ status: 'ok', tasks: [], warnings: [] }), + ...overrides, + }; +} + +function makePoller(adapter: unknown, logger = fakeLogger) { + return new StepTaskPoller( + adapter as never, + new AbortController().signal, + { + batchSize: 10, + queueName: 'orders', + flowSlug: 'Orders', + visibilityTimeout: 2, + maxPollSeconds: 1, + pollIntervalMs: 10, + }, + () => 'worker-id', + logger + ); +} + +Deno.test('StepTaskPoller passes queue and flow separately and returns paired tasks', async () => { + let captured: { queueName: string; flowSlug: string; messageIds: string[] } | undefined; + const adapter = makeAdapter({ + startTasks: (queueName: string, flowSlug: string, messageIds: string[]) => { + captured = { queueName, flowSlug, messageIds }; + return Promise.resolve({ + status: 'ok', + tasks: [ + { + flow_slug: 'Orders', + run_id: '11111111-1111-1111-1111-111111111111', + step_slug: 'saveItem', + task_index: 0, + queue_name: 'orders', + input: {}, + msg_id: BIG_ID, + flow_input: null, + }, + ], + warnings: [], + }); + }, + }); + + const result = await makePoller(adapter).poll(); + + assertEquals(captured!.queueName, 'orders'); + assertEquals(captured!.flowSlug, 'Orders'); + assertEquals(captured!.messageIds, [BIG_ID, '9007199254740994']); + assertEquals(result.length, 1); + assertEquals(result[0].msg_id, BIG_ID); + assertEquals(result[0].task.queue_name, 'orders'); +}); + +Deno.test('StepTaskPoller logs one body-free warning per foreign message', async () => { + const { logger, recorded } = recordingLogger(); + const adapter = makeAdapter({ + startTasks: () => + Promise.resolve({ + status: 'ok', + tasks: [], + warnings: [ + { + queue_name: 'orders', + message_id: '9007199254740994', + reason: 'foreign_message', + }, + ], + }), + }); + + const result = await makePoller(adapter, logger).poll(); + + assertEquals(result.length, 0); + assertEquals(recorded.warnings.length, 1); + assertEquals( + recorded.warnings[0], + 'Claim warning: reason=foreign_message queue=orders message_id=9007199254740994' + ); + assertEquals(recorded.warnings[0].includes('SECRET_BODY_TOKEN'), false); +}); + +Deno.test('StepTaskPoller translates a committed fatal result into FatalWorkerError', async () => { + const adapter = makeAdapter({ + startTasks: () => + Promise.resolve({ + status: 'fatal', + tasks: [], + errors: [ + { + queue_name: 'orders', + message_id: '9007199254740994', + reason: 'unsupported_work', + }, + ], + }), + }); + + const error = await assertRejects( + () => makePoller(adapter).poll(), + FatalWorkerError + ); + + assertEquals(error.name, 'FatalWorkerError'); + assertEquals(error.message.includes('reason=unsupported_work'), true); + assertEquals(error.message.includes('queue=orders'), true); + assertEquals(error.message.includes('message_id=9007199254740994'), true); + assertEquals(error.message.includes('SECRET_BODY_TOKEN'), false); +}); + +Deno.test('StepTaskPoller keeps ordinary SQL exceptions retryable', async () => { + const adapter = makeAdapter({ + readMessages: () => Promise.reject(new Error('connection refused')), + }); + + const error = await assertRejects(() => makePoller(adapter).poll(), Error); + assertEquals(error instanceof FatalWorkerError, false); + assertEquals(error.message, 'connection refused'); +}); diff --git a/pkgs/edge-worker/tests/unit/Worker.mainLoop.test.ts b/pkgs/edge-worker/tests/unit/Worker.mainLoop.test.ts index 7438922c1..e9d839f31 100644 --- a/pkgs/edge-worker/tests/unit/Worker.mainLoop.test.ts +++ b/pkgs/edge-worker/tests/unit/Worker.mainLoop.test.ts @@ -3,6 +3,7 @@ import { FakeTime } from '@std/testing/time'; import { Worker } from '../../src/core/Worker.ts'; import type { IBatchProcessor, ILifecycle, WorkerBootstrap } from '../../src/core/types.ts'; import { States, WorkerState } from '../../src/core/WorkerState.ts'; +import { FatalWorkerError } from '../../src/core/errors.ts'; import { fakeLogger } from '../fakes.ts'; /** @@ -194,3 +195,64 @@ Deno.test('Worker stop completes immediately while a retry delay is pending', as time.restore(); } }); + +Deno.test('Worker main loop stops terminally after a fatal batch', async () => { + const time = new FakeTime(); + try { + const batchTimes: number[] = []; + const fatalErrors: string[] = []; + const logger = { + ...fakeLogger, + error: (message: string) => fatalErrors.push(message), + }; + let stopAcknowledged = 0; + let cleanups = 0; + const lifecycle = createRunningLifecycle(); + const origAcknowledgeStop = lifecycle.acknowledgeStop; + lifecycle.acknowledgeStop = () => { + stopAcknowledged++; + origAcknowledgeStop.call(lifecycle); + }; + + const fatal = () => + Promise.reject( + new FatalWorkerError( + 'Committed fatal claim batch: reason=unsupported_work queue=orders message_id=9007199254740994' + ) + ); + const worker = new Worker( + createBatchProcessor(fatal, batchTimes), + lifecycle, + logger, + { cleanup: () => { cleanups++; return Promise.resolve(); } } + ); + + await worker.startOnlyOnce(workerBootstrap); + // Drain the fatal branch, the scheduled stop, and performStop's wait on + // the main-loop promise. + for (let i = 0; i < 5; i++) { + await time.runMicrotasks(); + } + + assertEquals(batchTimes.length, 1, 'exactly one poll happens before the fatal stop'); + + // No retry timer is scheduled after a fatal batch. + await advance(time, 10_000); + assertEquals(batchTimes.length, 1, 'no retry iteration after a fatal batch'); + + await worker.stop(); + + assertEquals(worker.isStopped, true, 'the worker reaches the stopped state'); + assertEquals(stopAcknowledged, 1, 'stop is acknowledged exactly once'); + assertEquals(cleanups, 1, 'cleanup runs exactly once'); + assertEquals(fatalErrors.length, 1, 'the fatal outcome is logged exactly once'); + assertEquals( + fatalErrors[0]!.includes('reason=unsupported_work') && + fatalErrors[0]!.includes('message_id=9007199254740994'), + true, + 'the fatal log is the body-free diagnostic line' + ); + } finally { + time.restore(); + } +}); diff --git a/pkgs/edge-worker/tests/unit/contextUtils.test.ts b/pkgs/edge-worker/tests/unit/contextUtils.test.ts index 5ab5001fa..c54e6c169 100644 --- a/pkgs/edge-worker/tests/unit/contextUtils.test.ts +++ b/pkgs/edge-worker/tests/unit/contextUtils.test.ts @@ -29,7 +29,7 @@ const minimalEnv = { // Mock pgmq message record const mockMessage: PgmqMessageRecord<{ test: string }> = { - msg_id: 123, + msg_id: '123', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -39,7 +39,7 @@ const mockMessage: PgmqMessageRecord<{ test: string }> = { // Mock pgmq message record with step input structure const mockStepMessage: PgmqMessageRecord<{ run: { test: string } }> = { - msg_id: 123, + msg_id: '123', read_ct: 1, enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -53,10 +53,11 @@ const mockFlowInput = { test: 'flow-input' }; // Mock step task (using generic typing) const mockStepTask = { flow_slug: 'test-flow', + queue_name: 'test-flow', run_id: 'run-456', step_slug: 'test-step', input: { run: { test: 'input' } }, - msg_id: 123, + msg_id: '123', flow_input: mockFlowInput, // Can be actual value or null - test helper wraps in Promise task_index: 0 } as unknown as StepTaskRecord; @@ -134,7 +135,7 @@ Deno.test('context - rawMessage is accessible', () => { sql: mockSql }); - assertEquals(context.rawMessage.msg_id, 123); + assertEquals(context.rawMessage.msg_id, '123'); assertEquals(context.rawMessage.message, { test: 'data' }); }); diff --git a/pkgs/edge-worker/tests/unit/workerConfigContext.test.ts b/pkgs/edge-worker/tests/unit/workerConfigContext.test.ts index 6dcc7f4bb..41d892e04 100644 --- a/pkgs/edge-worker/tests/unit/workerConfigContext.test.ts +++ b/pkgs/edge-worker/tests/unit/workerConfigContext.test.ts @@ -31,7 +31,7 @@ Deno.test('createContextSafeConfig excludes sql field and freezes result', () => Deno.test('Queue worker context includes workerConfig for GitHub issue use case', async () => { const mockMessage: PgmqMessageRecord<{test: string}> = { - msg_id: 123, + msg_id: '123', read_ct: 2, // Current retry attempt enqueued_at: '2024-01-01T00:00:00Z', vt: '2024-01-01T00:01:00Z', @@ -91,7 +91,7 @@ Deno.test('Queue worker config immutability prevents handler modifications', asy const context = { env: {}, shutdownSignal: new AbortController().signal, - rawMessage: { msg_id: 1, read_ct: 1, message: {} }, + rawMessage: { msg_id: '1', read_ct: 1, message: {} }, workerConfig: createContextSafeConfig(mockConfig), }; @@ -120,8 +120,8 @@ Deno.test('Flow worker context includes workerConfig', async () => { const context = { env: {}, shutdownSignal: new AbortController().signal, - rawMessage: { msg_id: 456, read_ct: 1, message: {} }, - stepTask: { flow_slug: 'test', step_slug: 'step', msg_id: 456, run_id: 'run', input: {} }, + rawMessage: { msg_id: '456', read_ct: 1, message: {} }, + stepTask: { flow_slug: 'test', step_slug: 'step', msg_id: '456', run_id: 'run', queue_name: 'test', input: {} }, workerConfig: createContextSafeConfig(mockConfig), }; diff --git a/pkgs/example-flows/src/example-flow.ts b/pkgs/example-flows/src/example-flow.ts index a524d34d1..7115b5e50 100644 --- a/pkgs/example-flows/src/example-flow.ts +++ b/pkgs/example-flows/src/example-flow.ts @@ -34,12 +34,13 @@ export const stepTaskRecord: StepTaskRecord = { run_id: '123', step_slug: 'normalStep', task_index: 0, + queue_name: 'example_flow', input: { rootStep: { doubledValue: 23 }, // thirdStep: { finalValue: 23 }, --- this should be an error // normalStep: { doubledValueArray: [1, 2, 3] }, --- this should be an error }, - msg_id: 1, + msg_id: '1', flow_input: { value: 23 }, }; diff --git a/pkgs/website/src/content/docs/build/delete-flows.mdx b/pkgs/website/src/content/docs/build/delete-flows.mdx index a68456646..d700820b8 100644 --- a/pkgs/website/src/content/docs/build/delete-flows.mdx +++ b/pkgs/website/src/content/docs/build/delete-flows.mdx @@ -40,6 +40,16 @@ SELECT pgflow.delete_flow_and_data('analyzeWebsite'); This deletes the flow definition, all runs, queued messages, and task outputs for the specified flow. +### Failure modes and deletion order + +Deletion is all-or-nothing: it either removes the flow and everything it owns, or it rolls back and removes nothing. It fails with an explicit error, before any destructive step, when the queue state is not exactly what the flow owns (#650): + +- the queue metadata is missing or ambiguous (for example two case-distinct metadata rows); +- physical queue objects are malformed or incomplete; +- task rows claim queue snapshots outside the flow's canonical route. + +On success, deletion locks and deletes the runtime data and step definitions first, then drops the validated PGMQ queue objects, and removes the flow identity row last - so an interrupted attempt cannot leave a live flow without its queue or history. + ## After Deleting Once you've deleted the flow: diff --git a/pkgs/website/src/content/docs/concepts/data-model.mdx b/pkgs/website/src/content/docs/concepts/data-model.mdx index 44393eb17..aad38d28e 100644 --- a/pkgs/website/src/content/docs/concepts/data-model.mdx +++ b/pkgs/website/src/content/docs/concepts/data-model.mdx @@ -13,7 +13,7 @@ pgflow's data model separates flow definitions from runtime execution state. Flo ### 🏷️ Slugs as Identifiers -Flows and steps are identified by slugs - simple text identifiers like `'analyzeWebsite'` or `'fetchData'`. Slugs use camelCase and must be valid identifiers (alphanumeric, max 128 characters), serving as natural, readable keys throughout the system. +Flows and steps are identified by exact, case-preserving slugs such as `'analyzeWebsite'` or `'fetchData'`. Uniqueness checks ignore case, but references must match the accepted spelling. See [Naming conventions](/concepts/naming-conventions/#exact-rules) for the character and length limits. ### 🔑 Composite Keys with Denormalization @@ -81,6 +81,9 @@ These tables track the execution state of flow instances: - Single steps create 1 task, map steps create N tasks - Each task has retry counter and attempts tracking - Contains `task_index` for map task array elements +- Each task stores an immutable `queue_name` snapshot: the canonical queue the claim, archival, and cleanup operations use. Steps store the same queue at definition time (#650) +- A task message is identified by the partial unique pair `(queue_name, message_id)`, not by `message_id` alone +- `message_id` stays nullable for task records without a PGMQ message; taskless steps create no task rows, and the task primary key remains unchanged - Tracks task status (`queued`, `started`, `completed`, `failed`, `skipped`, `cancelled`) - `skipped` marks the logical orchestration state: the parent step was skipped, so the task will never run; an already-running handler is not forcibly terminated - `cancelled` marks unfinished tasks whose run failed: the task that caused the failure stays `failed` (with its error), completed work stays `completed`, and every remaining queued or started task becomes `cancelled`. `runs.failed_at` is the cancellation time; there is no separate `cancelled_at` column diff --git a/pkgs/website/src/content/docs/concepts/naming-conventions.mdx b/pkgs/website/src/content/docs/concepts/naming-conventions.mdx index fc068ed20..d8c535563 100644 --- a/pkgs/website/src/content/docs/concepts/naming-conventions.mdx +++ b/pkgs/website/src/content/docs/concepts/naming-conventions.mdx @@ -31,6 +31,22 @@ export const AnalyzeWebsite = new Flow({ slug: 'analyzeWebsite' }) Flow slugs are stored in the database and used to identify flows when starting runs. Using camelCase keeps them consistent with step slugs and JavaScript conventions. +### Exact rules + +Slug acceptance is checked at definition time: + +- up to 128 characters (letters, digits, underscores); +- must not start with a digit or an underscore; +- must not end with an underscore; +- no two consecutive underscores; +- the word `run` is reserved. + +Accepted slugs keep their exact spelling, including case. References such as `start_flow()` and `dependsOn` must use that exact spelling. + +Uniqueness checks ignore case. Flow slugs must be unique across the database, while step slugs must be unique only within their flow. For example, `fetchUser` and `FetchUser` cannot coexist in one flow, but different flows may each define `fetchUser`. + +One extra limit applies to generated queue names. A flow's physical queue name is its lowercase flow slug, and PGMQ limits that name to 47 characters. As a result, current flow slugs cannot exceed 47 characters. Step slugs retain the generic 128-character limit. + ## File naming Flow files use **kebab-case** (industry standard for TypeScript): diff --git a/pkgs/website/src/content/docs/concepts/startup-compilation.mdx b/pkgs/website/src/content/docs/concepts/startup-compilation.mdx index a0ecdf30d..01df10881 100644 --- a/pkgs/website/src/content/docs/concepts/startup-compilation.mdx +++ b/pkgs/website/src/content/docs/concepts/startup-compilation.mdx @@ -15,13 +15,17 @@ Every startup follows this ordered contract: ```text extract complete shape - -> compile or verify under the flow lock + -> compile or verify under the flow lock (queue-aware protocol handshake) -> track the worker function -> insert the worker row -> start polling ``` -Compilation happens under the flow's advisory lock, so concurrent workers for the same flow serialize safely. If compilation fails, the worker performs no registration write: the `worker_functions` row and the `pgflow.workers` row stay unchanged, and startup fails. +Compilation happens under the flow's advisory lock, so concurrent workers for the same flow serialize safely. The database answers the compilation handshake with its protocol version and the flow's canonical queue name; the worker registers against that queue and claims tasks from it. If compilation fails, the worker performs no registration write: the `worker_functions` row and the `pgflow.workers` row stay unchanged, and startup fails. + + ```d2 ...@../../../assets/pgflow-theme.d2 @@ -77,9 +81,17 @@ register -> polling When a worker starts, it: 1. **Extracts the complete flow shape** from the imported TypeScript definition -2. **Compiles the flow if missing** - PostgreSQL creates the flow, steps, and queue from the shape -3. **Verifies the shape if the flow exists** - PostgreSQL compares the worker's shape with the database definition -4. **Registers the worker** - tracks the edge function, inserts the worker row, then starts polling +2. **Compiles the flow if missing** - PostgreSQL creates the flow, steps, and the canonical queue from the shape +3. **Verifies the shape if the flow exists** - PostgreSQL compares the worker's shape with the database definition and returns the canonical queue name +4. **Registers the worker** - tracks the edge function, inserts the worker row bound to the canonical queue, then starts polling + +### Queue provisioning and ownership + +A flow owns one generated private queue whose name is its lowercase flow slug. Compilation checks the complete definition and queue namespace before it changes either one. It provisions the queue even when the flow has no steps. + +An existing physical queue is reusable only when the same persisted flow definition already owns that exact route and its PGMQ objects are valid. A queue with the generated name but no matching flow definition is a collision, so startup fails instead of adopting it. Ambiguous metadata or malformed queue objects also stop startup. + +The SQL building blocks use the same boundary. `create_flow()` creates only the flow definition; `add_step()` provisions or verifies its generated queue. Starting a run performs no queue DDL. ## Local Development diff --git a/pkgs/website/src/content/docs/deploy/prune-records.mdx b/pkgs/website/src/content/docs/deploy/prune-records.mdx index 37c16a860..217cc2a9d 100644 --- a/pkgs/website/src/content/docs/deploy/prune-records.mdx +++ b/pkgs/website/src/content/docs/deploy/prune-records.mdx @@ -31,7 +31,7 @@ When a run (completed or failed) exceeds the retention period, **ALL** associate **Deleted:** - Run records (`pgflow.runs`) - Step states and tasks - all statuses (`pgflow.step_states`, `pgflow.step_tasks`) -- PGMQ messages - active queue (`pgmq.q_{flow_slug}`) and archived (`pgmq.a_{flow_slug}`) +- PGMQ messages - active queue (`pgmq.q_{lowercase flow slug}`) and archive (`pgmq.a_{lowercase flow slug}`) - Inactive workers (based on `last_heartbeat_at`) **Preserved:** @@ -47,6 +47,10 @@ pgflow includes a pruning function that accepts an INTERVAL parameter specifying pgflow.prune_data_older_than(retention_interval INTERVAL) ``` + + Examples: ```sql -- Keep 90 days of data (recommended default) diff --git a/pkgs/website/src/content/docs/deploy/update-pgflow.mdx b/pkgs/website/src/content/docs/deploy/update-pgflow.mdx index 9ab211264..d3b17d8fe 100644 --- a/pkgs/website/src/content/docs/deploy/update-pgflow.mdx +++ b/pkgs/website/src/content/docs/deploy/update-pgflow.mdx @@ -8,7 +8,8 @@ sidebar: variant: tip --- -import { Aside, Steps } from "@astrojs/starlight/components"; +import { Aside, Steps, Code } from "@astrojs/starlight/components"; +import preMigrationCheck650 from '../../../../../core/queries/PRE_MIGRATION_CHECK_650.sql?raw'; This guide explains how to update pgflow to the latest version, including package updates and database migrations. pgflow updates involve two main components that need to be updated together. @@ -99,6 +100,10 @@ pgflow is tested on PostgreSQL 17 for compatibility. ### 5. Apply new migrations + + Apply the new migrations to your database: ```bash frame="none" @@ -130,6 +135,90 @@ Where: The original timestamp allows pgflow to search for and detect which migrations have already been installed, while the new timestamp prefix ensures Supabase can apply the migration without timestamp ordering errors. +## The queue identity upgrade + +The pgflow database migration `20260910104929_pgflow_persist_queue.sql` gives every flow a persisted, canonical queue: step and task rows store the queue name they will use, task messages are identified by `(queue_name, message_id)` instead of `message_id` alone, and physical queue names are the lowercase flow slug while your flow and step slugs keep their exact spelling. + +The migration also tightens flow and step slug validation. Leading or trailing underscores, doubled underscores, and case-only duplicates now block the upgrade, including definitions without active runs. Review the complete [naming rules](/concepts/naming-conventions/#exact-rules) before the maintenance window. The migration never renames incompatible definitions. + +This change is coordinated: new workers speak a new startup protocol, and old workers call startup functions the migrated database no longer provides. The two sides must move together. + + + +### 1. Audit the old database and resolve findings + +While writers are still paused or quiet, run the read-only audit. It reports invalid definitions, duplicate task pairs, queue ownership or topology problems, and the installed pruning helper - all as bounded JSON notices with samples. It changes nothing. + + + +Only `severity=info` rows means the audit found nothing to fix. `severity=error` rows name exact flow or step slugs and keys: resolve them manually before proceeding. The migration never renames, merges, deletes, or repairs anything on your behalf. + +### 2. Save worker and schedule state + +```sql frame="none" +-- Save HTTP worker enabled flags and cron schedules so you can restore +-- exact values afterwards. Nothing is restored to a default on=true. +select function_name, enabled from pgflow.worker_functions; +select jobid, jobname, schedule, active from cron.job; +``` + +### 3. Pause producers and maintenance + +Pause the producers that call `pgflow.start_flow`, and any jobs that change flow definitions, delete flows, run the installed pruning helper, or recover stalled tasks. Keep the old workers running while their current handlers finish. + +### 4. Stop and drain workers + +Disable HTTP worker restarts, and stop process-mode supervisors from starting new workers. Wait for active handlers to finish and commit, then stop the remaining worker processes. Queues do not need to be empty: the migration validates and preserves matched queued messages and task state. Do not delete queued messages to prepare the upgrade. + +### 5. Apply the migration transaction with a bounded lock wait + +Apply pending migrations through the Supabase CLI so it uses the installer-generated filename and records the migration in Supabase's history: + +```bash frame="none" +npx supabase migration up --db-url "$DATABASE_URL" +``` + +The queue identity migration is self-transactional and refuses to start if another session holds a conflicting lock for longer than five seconds. On refusal or error the database is unchanged: fix the blocker and run the migration command again. + +### 6. Replace or adapt the installed pruning helper + +The migration never overwrites `pgflow.prune_data_older_than(interval)`. If you have the stock 0.16.0 helper installed, replace it explicitly with the current version from [the pruning function](/deploy/prune-records/#the-pruning-function) and keep maintenance paused until that is done. If you customized the helper, compare your installed source first: + +```sql frame="none" +select pg_get_functiondef('pgflow.prune_data_older_than(interval)'::regprocedure); +``` + +Adapt your customization manually to the current source's queue-snapshot grouping and lock order instead of pasting over your custom behavior. Keep cron schedules disabled until the replacement or adaptation is complete. + +### 7. Deploy matching packages and workers + +Update all pgflow packages to the same version (see the update process above), redeploy each worker function, and start process-mode workers. Workers now register against their flow's canonical lowercase queue. + +### 8. Restore the saved state and verify + +Re-enable exactly the worker functions and cron jobs you saved in step 2 - nothing turns itself back on. Then verify the upgrade: + +```sql frame="none" +-- Canonical snapshots are populated for every step and task. +select step_slug, queue_name from pgflow.steps order by 1; +select queue_name, count(*) from pgflow.step_tasks group by 1 order by 1; + +-- A fresh run completes through the canonical route. +select run_id, status from pgflow.start_flow('YourFlow', '{}'::jsonb); +``` + +Handlers receive message IDs as exact decimal strings, including values above `Number.MAX_SAFE_INTEGER`. Compare IDs for equality as strings, or use `BigInt` when numeric ordering is necessary. Never convert them with `Number()`. + +Do not send application messages directly into pgflow-owned queues: clearly foreign untracked messages are archived with body-free warnings, and apparently genuine or ambiguous unsupported work stops the worker and pauses HTTP restarts. + ## Remove manual flow compilation pgflow 0.16.0 removed `compileFlow()`, ControlPlane, `pgflow compile`, and the `FlowWorkerConfig.compilation` option, and replaced `pgflow.ensure_flow_compiled(text, jsonb, boolean)` with a two-argument startup-only signature. Old workers cannot start after the database migration. diff --git a/pkgs/website/src/content/docs/reference/configuration/worker.mdx b/pkgs/website/src/content/docs/reference/configuration/worker.mdx index f5de34087..4d0bdf995 100644 --- a/pkgs/website/src/content/docs/reference/configuration/worker.mdx +++ b/pkgs/website/src/content/docs/reference/configuration/worker.mdx @@ -181,7 +181,7 @@ my-worker: ↻ retry 1/3 in 5s When using Edge Worker in [Background Jobs Mode](/get-started/faq/#what-are-the-two-edge-worker-modes) (without pgflow orchestration), there are a few key differences: - **No flow/step configuration**: Queue workers don't have `maxAttempts`, `baseDelay`, `timeout`, or `startDelay` options in the flow definition. Instead, these are configured directly in the worker options. -- **`queueName` option**: Queue workers can specify a custom queue name (default is `tasks`), while flow workers automatically use the flow slug as the queue name. +- **`queueName` option**: Queue workers can specify a custom queue name (default is `tasks`), while flow workers automatically use the lowercase flow slug as their canonical queue name. - **Handler signature**: Queue workers receive a simple payload, while flow workers receive a context object with `input`, `run`, and previous step outputs. For complete queue worker configuration, see [Queue Worker Configuration](/reference/queue-worker/configuration/). diff --git a/pkgs/website/src/content/docs/reference/context.mdx b/pkgs/website/src/content/docs/reference/context.mdx index cb36d0249..07e94e730 100644 --- a/pkgs/website/src/content/docs/reference/context.mdx +++ b/pkgs/website/src/content/docs/reference/context.mdx @@ -55,9 +55,11 @@ An AbortSignal that triggers when the worker is shutting down. Use this to grace The original message from the pgmq queue, containing metadata like message ID, read count, and enqueued timestamp. Useful for debugging and advanced queue operations. +Message IDs are decimal strings. PGMQ stores them as SQL `bigint`, which exceeds JavaScript's safe integer range, so pgflow delivers the exact decimal string instead of a rounded number (#650). Compare IDs for equality as strings. Use `BigInt` for numeric ordering, and never convert them with `Number()`. + ```typescript interface PgmqMessageRecord { - msg_id: number; // Unique message ID from pgmq + msg_id: string; // Unique message ID from pgmq (exact decimal string) read_ct: number; // How many times this message has been read enqueued_at: string; // ISO timestamp when message was enqueued vt: string; // ISO timestamp for visibility timeout @@ -85,8 +87,9 @@ interface StepTaskRecord { run_id: string; // UUID of the current flow run step_slug: string; // Slug identifier of the current step task_index: number; // Task index (0 for single steps, 0..N-1 for map steps) + queue_name: string; // Canonical physical queue snapshot the task was claimed from input: StepInput; // Typed input for this specific step (inferred from flow) - msg_id: number; // pgmq message ID + msg_id: string; // pgmq message ID (exact decimal string) } ```