[jumski]

Summary

Fixes #689. Independent of the worker startup handshake in #711.

recovery transaction
  retry budget exhausted → mark permanently stalled + archive message
  COMMIT

separate escalation cron transaction
  lock and recheck run → step → task
  marked started/queued task → fail_task()
    apply whenExhausted: fail | skip | skip-cascade
    preserve attempts_count as committed claims

The existing recovery setup configures both cron jobs with one interval. Upgrade wiring preserves installed schedules and inactive/absent recovery choices. Historical marked queued leftovers are included; unmarked queued and terminal tasks stay unchanged.

Upgrade and deployment guidance restores recorded activation values instead of enabling everything. Saved cron values must be JSON booleans; unknown or malformed state never enables a job. Includes a core changeset, generated migration/types, pgTAP coverage, and upgrade fixtures.

Evidence

• Before: recovery stamped and archived tasks after its requeue limit but left them active. Late failure callbacks could retry an archived task below max_attempts.
• After: marker-aware exhaustion reuses the existing policy transitions, while the separate sweep repairs historical started/queued rows without synthetic claims. Regression coverage includes policies, callback races, map/private queues, and cron state preservation.
• Fresh runtime validation on this base (commit c0fa0d14e): 300 pgTAP files, all tests successful, plus every upgrade fixture: 0.15.0, 0.16.0 backfill/runtime, all conflict rollbacks, and the new escalation cron states (active/paused/absent). Core prepush gate (sqruff, eslint, build, strict and vitest type checks), unit tests, and generated-types/migration verification. Edge-worker: 279 unit tests, 70 integration tests, 13 Edge Function E2E tests, 14 portable-runtime E2E tests, and the full prepush gate. Website build, lint, and typecheck. dsl and cli builds. The final force-push went through the normal pre-push hooks.
• Relocation correction: the escalation upgrade fixture originally named 20261102104928_pgflow_worker_startup_handshake.sql as its baseline — a migration that exists only on fix(worker): coordinate pending starts with invocation handshakes #711's branch. It now stops at 20261001194538_pgflow_telemetry_endpoint_fix.sql, the last migration before the escalation one on this base. The fixture container readiness wait also moved from pg_isready to waiting for the second database system is ready to accept connections log line: the pinned image initializes PGDATA on every start, and pg_isready can pass against the short-lived initdb temp server whose socket then disappears (observed on the leading_step fixture).
• Review follow-up: the escalation error message now interpolates each task's own requeued_count instead of a hardcoded 3 that could drift from requeue_stalled_tasks()'s max_requeues constant. The migration was regenerated with the documented atlas flow as 20261103185005_pgflow_escalate_permanently_stalled.sql, keeping the hand-authored cron-state preservation block Atlas cannot express. That regeneration's fix-sql step also reformats four pre-existing schema files (0100_function_add_step, 0100_function_compare_flow_shapes, 0110_function_set_vt_batch, 0120_function_start_tasks) to sqruff 0.39.0's LT01 spacing (text[] → text []) — whitespace only, unblocks core:lint:sqruff and the pre-push nx-affected-checks hook, which fail identically on main. One environmental stop occurred during validation: the stopped_at CPU-clock-limit E2E test timed out once under load and passed unchanged after a fresh test environment.

Merge Danger

Door: one-way for data effects after escalation runs. Code can be reverted; completed transitions and downstream side effects cannot be undone automatically.

Blast Radius: recovery

Historical marked tasks follow the existing exhaustion policy after upgrade. In particular, skip can enqueue dependent business work. Inspect the documented backlog before upgrade. Pause both cron jobs and the relevant workers/callbacks when downstream execution must stop, then restore each recorded state exactly. Inactive cron jobs alone do not prevent marker-aware failure callbacks. No production migration or deployment is part of this PR submission.

[changeset-bot]

🦋 Changeset detected

Latest commit: 35ac37a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@pgflow/core	Minor
pgflow	Minor
@pgflow/client	Minor
@pgflow/edge-worker	Minor
@pgflow/dsl	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jumski]

• fix(core): terminalize permanently stalled tasks through whenExhausted #713 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[nx-cloud]

View your CI Pipeline Execution ↗ for commit 35ac37a

Command	Status	Duration	Result
nx run-many -t build --projects=dsl,core,cli	✅ Succeeded	5s	View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗

---

☁️ Nx Cloud last updated this comment at 2026-11-04 00:17:05 UTC

[github-actions]

🔍 Preview Deployment: Website

✅ Deployment successful!

🔗 Preview URL: https://pr-713.pgflow.pages.dev

📝 Details:

• Branch: 689-terminalize-stalled
• Commit: 6b2ecf31dc4c31e27b1dfae4801609a5bfadca9d
• View Logs

_Last updated: _