# Fix: Add automatic data migration for map step type migration

 (#234)

## Summary

Updates migration `20251006073122_pgflow_add_map_step_type.sql` to automatically handle existing production data when transitioning from single-task-only schema to map step schema.

## Changes

### Migration Updates
- Split schema changes into 3 steps to avoid constraint violations:
  1. Drop old constraints and NOT NULL requirement
  2. Run automatic data backfill (UPDATE statements)
  3. Add new constraints

### Automatic Data Migration
The migration now automatically:
- Backfills `initial_tasks = 1` for all existing steps (correct for pre-map-step schema where all steps had exactly 1 task)
- Sets `remaining_tasks = NULL` for 'created' status steps (new semantics: NULL = not started)

### Pre-Migration Verification
Added `pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql`:
- Read-only query to verify migration will succeed
- Shows exactly which runs/steps will be auto-fixed
- Provides copy/paste-friendly output for Discord support
- Only needed for production databases with existing data

### Documentation
Updated `.changeset/add-map-step-type-infrastructure.md`:
- Added WARNING callout about automatic data migration
- Added TIP callout with verification instructions
- Includes expected output example
- Clarifies this is for production databases only (not local dev)

## Testing

Migration tested against production-like data with existing `created` and `started` steps. All constraint violations are automatically resolved.

## User Impact

**No manual intervention required.** Migration handles all data cleanup automatically. Users can optionally verify beforehand using the pre-migration check query.

Author: jumski

.changeset/add-map-step-type-infrastructure.md

@@ -4,24 +4,41 @@
 
 Add map step type infrastructure in SQL core
 
-## 🚨🚨🚨 CRITICAL MIGRATION WARNING 🚨🚨🚨
+> [!WARNING]
+> **This migration includes automatic data migration**
+>
+> The migration will automatically update existing `step_states` rows to satisfy new constraints. This should complete without issues due to strict check constraints enforced in previous versions.
 
-**THIS MIGRATION REQUIRES MANUAL DATA UPDATE BEFORE DEPLOYMENT!**
+> [!TIP]
+> **Optional: Verify before deploying to production**
+>
+> If you have existing production data and want to verify the migration will succeed cleanly, run this **read-only check query** (does not modify data) in **Supabase Studio** against your **production database**:
+>
+> 1. Open Supabase Studio → SQL Editor
+> 2. Copy contents of `pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql`
+> 3. Execute against your production database (not local dev!)
+> 4. Review results
+>
+> **Expected output for successful migration:**
+> ```
+> type                  | identifier                | details
+> ----------------------|---------------------------|---------------------------
+> ISSUE_1_AUTO_FIXED    | run=abc12345 step=load    | status=created remaining_tasks=1 → will set to NULL
+> ISSUE_2_AUTO_FIXED    | run=def67890 step=process | status=started → will set initial_tasks=1
+> INFO_SUMMARY          | total_step_states=42      | created=5 started=2 completed=30 failed=5
+> ```
+>
+> **Interpretation:**
+> - ✅ Only `ISSUE_1_AUTO_FIXED`, `ISSUE_2_AUTO_FIXED`, and `INFO_SUMMARY` rows? **Safe to migrate**
+> - 🆘 Unexpected issues or errors? Copy output and share on Discord for help
+>
+> **Note:** This check only returns results indicating correctness or problems - it does not modify any data. Only useful for production databases with existing runs.
 
-The migration adds a new constraint `remaining_tasks_state_consistency` that will **FAIL ON EXISTING DATA** if not handled properly.
+**Automatic data updates:**
+- Sets `initial_tasks = 1` for all existing steps (correct for pre-map-step schema)
+- Sets `remaining_tasks = NULL` for 'created' status steps (new semantics)
 
-### Required Data Migration:
-
-Before applying this migration to any environment with existing data, you MUST include:
-
-```sql
--- CRITICAL: Update existing step_states to satisfy new constraint
-UPDATE pgflow.step_states 
-SET remaining_tasks = NULL 
-WHERE status = 'created';
-```
-
-**Without this update, the migration WILL FAIL in production!** The new constraint requires that `remaining_tasks` can only be set when `status != 'created'`.
+No manual intervention required.
 
 ---
 

pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql

@@ -0,0 +1,77 @@
+-- ================================================================================
+-- PRE-MIGRATION CHECK for 20251006073122_pgflow_add_map_step_type.sql
+-- ================================================================================
+-- Purpose: Verify the migration will succeed cleanly on your database
+-- When to run: BEFORE applying the migration
+-- What to do with output:
+--   - ✅ Only INFO rows? You're good to migrate
+--   - ⚠️  ISSUE_1/ISSUE_2? Safe - migration auto-fixes these
+--   - 🆘 Unexpected issues? Copy/paste output to Discord for help
+-- ================================================================================
+
+WITH issues AS (
+  -- Issue 1: Created steps with remaining_tasks (will be set to NULL)
+  SELECT
+    1 AS priority,
+    'ISSUE_1_AUTO_FIXED' AS type,
+    format('run=%s step=%s',
+      LEFT(run_id::text, 8),
+      step_slug
+    ) AS identifier,
+    format('status=%s remaining_tasks=%s → will set to NULL',
+      status,
+      remaining_tasks
+    ) AS details
+  FROM pgflow.step_states
+  WHERE status = 'created' AND remaining_tasks IS NOT NULL
+
+  UNION ALL
+
+  -- Issue 2: Started steps (will backfill initial_tasks)
+  SELECT
+    2 AS priority,
+    'ISSUE_2_AUTO_FIXED' AS type,
+    format('run=%s step=%s',
+      LEFT(run_id::text, 8),
+      step_slug
+    ) AS identifier,
+    format('status=%s → will set initial_tasks=%s',
+      status,
+      COALESCE(remaining_tasks::text, '1')
+    ) AS details
+  FROM pgflow.step_states
+  WHERE status = 'started'
+
+  UNION ALL
+
+  -- Info: Summary stats
+  SELECT
+    999 AS priority,
+    'INFO_SUMMARY' AS type,
+    format('total_step_states=%s', COUNT(*)) AS identifier,
+    format('created=%s started=%s completed=%s failed=%s',
+      COUNT(*) FILTER (WHERE status = 'created'),
+      COUNT(*) FILTER (WHERE status = 'started'),
+      COUNT(*) FILTER (WHERE status = 'completed'),
+      COUNT(*) FILTER (WHERE status = 'failed')
+    ) AS details
+  FROM pgflow.step_states
+)
+SELECT
+  type,
+  identifier,
+  details
+FROM issues
+ORDER BY priority, identifier;
+
+-- ================================================================================
+-- HOW TO READ THE OUTPUT:
+-- ================================================================================
+-- type                  | identifier              | details
+-- ----------------------|-------------------------|---------------------------
+-- ISSUE_1_AUTO_FIXED    | run=abc12345 step=foo   | Will be cleaned up
+-- ISSUE_2_AUTO_FIXED    | run=def67890 step=bar   | Will be backfilled
+-- INFO_SUMMARY          | total_step_states=42    | Overall stats
+--
+-- ✅ Safe to migrate if you only see ISSUE_1/ISSUE_2 + INFO_SUMMARY
+-- ================================================================================

pkgs/core/queries/fail_permanently.sql

@@ -1,14 +1,31 @@
 -- Modify "step_task_record" composite type
 ALTER TYPE "pgflow"."step_task_record" ADD ATTRIBUTE "task_index" integer;
--- MANUAL DATA MIGRATION: Prepare existing data for new constraint
--- This UPDATE must run BEFORE the new constraint is added to avoid failures
--- The new constraint "remaining_tasks_state_consistency" requires that
--- remaining_tasks is NULL when status = 'created'
+-- Modify "step_states" table - Step 1: Drop old constraint and NOT NULL
+ALTER TABLE "pgflow"."step_states"
+  DROP CONSTRAINT "step_states_remaining_tasks_check",
+  ALTER COLUMN "remaining_tasks" DROP NOT NULL,
+  ALTER COLUMN "remaining_tasks" DROP DEFAULT,
+  ADD COLUMN "initial_tasks" integer NULL;
+-- AUTOMATIC DATA MIGRATION: Prepare existing data for new constraints
+-- This runs AFTER dropping NOT NULL but BEFORE adding new constraints
+-- All old steps had exactly 1 task (enforced by old only_single_task_per_step constraint)
+
+-- Backfill initial_tasks = 1 for all existing steps
+-- (Old schema enforced exactly 1 task per step, so all steps had initial_tasks=1)
+UPDATE "pgflow"."step_states"
+SET "initial_tasks" = 1
+WHERE "initial_tasks" IS NULL;
+
+-- Set remaining_tasks to NULL for 'created' status
+-- (New semantics: NULL = not started, old semantics: 1 = not started)
 UPDATE "pgflow"."step_states"
 SET "remaining_tasks" = NULL
-WHERE "status" = 'created';
--- Modify "step_states" table
-ALTER TABLE "pgflow"."step_states" DROP CONSTRAINT "step_states_remaining_tasks_check", ADD CONSTRAINT "initial_tasks_known_when_started" CHECK ((status <> 'started'::text) OR (initial_tasks IS NOT NULL)), ADD CONSTRAINT "remaining_tasks_state_consistency" CHECK ((remaining_tasks IS NULL) OR (status <> 'created'::text)), ADD CONSTRAINT "step_states_initial_tasks_check" CHECK ((initial_tasks IS NULL) OR (initial_tasks >= 0)), ALTER COLUMN "remaining_tasks" DROP NOT NULL, ALTER COLUMN "remaining_tasks" DROP DEFAULT, ADD COLUMN "initial_tasks" integer NULL;
+WHERE "status" = 'created' AND "remaining_tasks" IS NOT NULL;
+-- Modify "step_states" table - Step 2: Add new constraints
+ALTER TABLE "pgflow"."step_states"
+  ADD CONSTRAINT "initial_tasks_known_when_started" CHECK ((status <> 'started'::text) OR (initial_tasks IS NOT NULL)),
+  ADD CONSTRAINT "remaining_tasks_state_consistency" CHECK ((remaining_tasks IS NULL) OR (status <> 'created'::text)),
+  ADD CONSTRAINT "step_states_initial_tasks_check" CHECK ((initial_tasks IS NULL) OR (initial_tasks >= 0));
 -- Modify "step_tasks" table
 ALTER TABLE "pgflow"."step_tasks" DROP CONSTRAINT "only_single_task_per_step", DROP CONSTRAINT "output_valid_only_for_completed", ADD CONSTRAINT "output_valid_only_for_completed" CHECK ((output IS NULL) OR (status = ANY (ARRAY['completed'::text, 'failed'::text])));
 -- Modify "steps" table

pkgs/core/queries/fail_task.sql

@@ -1,4 +1,4 @@
-h1:A6hL4r68jggxXAl+YKBn+dGczQ3JWLsXvIh1QqnF4hU=
+h1:eDZ+F7VXqXxJeuJJTxxV6UTGnFGgvM/++AJVQc+Ao4Y=
 20250429164909_pgflow_initial.sql h1:5K7OqB/vj73TWJTQquUzn+i6H2wWduaW+Ir1an3QYmQ=
 20250517072017_pgflow_fix_poll_for_tasks_to_use_separate_statement_for_polling.sql h1:gnT6hYn43p5oIfr0HqoGlqX/4Si+uxMsCBtBa0/Z2Cg=
 20250609105135_pgflow_add_start_tasks_and_started_status.sql h1:9Yv/elMz9Nht9rCJOybx62eNrUyghsEMbMKeOJPUMVc=
@@ -8,4 +8,4 @@ h1:A6hL4r68jggxXAl+YKBn+dGczQ3JWLsXvIh1QqnF4hU=
 20250627090700_pgflow_fix_function_search_paths.sql h1:NRMbmDKkOww7pOx1TVERMP5UdjmgfH0wE9QhzfBU3co=
 20250707210212_pgflow_add_opt_start_delay.sql h1:11J7SDgS6EVFUwxSi0bRZnNQgVGTV0EJGj9yuC0vczY=
 20250719205006_pgflow_worker_deprecation.sql h1:L3LDsVrUeABlRBXhHsu60bilfgDKEJHci5xWknH9XIg=
-20251006073122_pgflow_add_map_step_type.sql h1:TOAhJ9WcG4mb67nBX+7DRl5UchezNM4CgTilS0wjM94=
+20251006073122_pgflow_add_map_step_type.sql h1:CPYy9lf7qUlZrkLzDI/0nH3CSUZZqFwXC6pO84E7Ezg=