supabase db reset/start: migration-apply SQL pipeline stops mid-stream with no server-side counterpart

[bisidore]

supabase db reset/start: migration-apply SQL pipeline stops mid-stream with no server-side counterpart

Repository: LinkFault/linkfault-foundation
Workflow: LinkFault Verification (.github/workflows/verify.yml)
Runner: ubuntu-24.04 (GitHub-hosted)
Failing step: supabase start --debug or supabase db reset --local --debug (same underlying stall, surfaces in whichever step happens to be applying migrations at the time)

Summary

Six workflow_dispatch/push runs, across four unrelated branches (including main itself, post-fix) with no shared content beyond a common ancestor, all stalled at the identical point: mid-way through a pipelined SQL statement stream the CLI sends while applying local migrations, specifically on

INSERT INTO supabase_migrations.schema_migrations(version, name, statements) VALUES($1, $2, $3)

immediately following the same migration file (20260817160209_assessments.sql). The step produces no error text, no exception, no timeout message — the log stream simply stops until the step's own external timeout (or GitHub's default) kills it. Reproduced on two pinned CLI versions (2.109.1 and 2.114.0), and both before and after an unrelated, fully-diagnosed, already-fixed content bug in this repo (a duplicate CREATE TYPE across two migrations) — ruling out both the CLI version and repo migration content as the cause.

A live database snapshot taken during one occurrence (run 6, below) shows the database itself idle and unblocked at the moment of the stall, with no connection matching the CLI's migration-apply session present at all. We can't see past that boundary — whether the stall is inside the CLI process itself, or in something between the CLI and the database (a Docker networking hiccup between client and container, for instance) isn't something we can distinguish from where we're looking. Filing here since the pipelined statement-batching behavior visible in --debug output is the last thing we can observe before the stall, not because we've localized the fault to this repo specifically.

The six runs

#	Run ID	Branch	Started (UTC)	Ended (UTC)	Duration	CLI version	Notes
1	32091737259	feat/step-9-billing-enforcement	2026-08-18 02:24:47	02:27:45	~3m	2.109.1
2	32091982094	feat/step-9-billing-enforcement	2026-08-18 02:28:48	02:31:56	~3m	2.109.1	retry of #1
3	32092494820	feat/step-9-billing-enforcement	2026-08-18 02:37:15	02:39:49	~2.5m	2.109.1	retry of #1/#2
4	32093153813	chore/ci-resource-sampling	2026-08-18 02:48:11	02:50:56	~2.75m	2.109.1	host-resource telemetry captured (below)
5	32095227294	feat/lean-admin	2026-08-18 03:22:17	03:25:04	~2.75m	2.109.1	unrelated migration content
6	32099052341	main	2026-08-18 04:30:37	04:32:18	~1.75m	2.109.1	post-fix, with DB-level watchdog — see below

Runs 1–3 are the same branch, retried per an internal "retry once on suspected flakiness" policy — three identical failures is what first raised this to a real investigation. Runs 4–5 are unrelated branches added specifically to test content-dependence; the stall is identical regardless of migration content. Run 6 is main itself, after fixing an unrelated duplicate-CREATE TYPE bug that had briefly been suspected as the cause, and with a container/DB-level watchdog added independently by another contributor — this is the run with the evidence below.

All six: the relevant step (Start local Supabase in runs 1–5, Clean local database reset in run 6 — see "Why the step differs" below) ends in failure with RESULT_TESTS: skipped; the job never reaches the pgTAP suite.

Why the failing step differs between run 6 and runs 1–5

Runs 1–5 used a workflow without a --debug flag on supabase db reset. Run 6's workflow (updated independently, unrelated to this investigation) added --debug to the reset step and increased timeouts, which appears to have shifted which step the CLI's internal migration-apply sequence was mid-way through when the stall occurred — but it is the same stall, at the same statement, following the same migration.

Evidence (run 6): the database shows no counterpart activity during the stall

Run 6's workflow includes a 30-second-interval background watchdog (added independently) that snapshots docker ps, pg_stat_activity, and blocking-lock state throughout the step, if: always() so it's captured even on failure. The snapshot taken during this stall:

--- pg_stat_activity (supabase_db_cjuamnnwhddycyicsqxu) ---
pid|state|wait_event_type|wait_event|query
384|active|Client|WalSenderWaitForWal|START_REPLICATION SLOT cainophile_5dciytpl LOGICAL ...
439|active|||SELECT pid, state, wait_event_type, wait_event, left(query, 200) AS query ... (the watchdog's own query)
205|idle|Extension|Extension|
230|idle|Client|ClientRead|LISTEN "pgrst"
... [23 more rows, all idle Supabase-internal services: Realtime, Logflare,
    PostgREST connection-pool housekeeping — none related to migration apply]
(27 rows)

--- blocking locks (supabase_db_cjuamnnwhddycyicsqxu) ---
pid|blocked_by|query
(0 rows)

No row corresponds to the CLI's migration-applying connection at all, and zero blocking locks exist. The database has no active or blocked query anywhere near the stalled INSERT INTO supabase_migrations.schema_migrations statement — it's sitting idle. Whatever's stopped making progress, it isn't visible from inside Postgres: no lock, no long-running query, no session matching the CLI's last Parse/Bind/Describe/Execute burst. That's as far as we can see from this side — we can't tell from here whether the CLI process itself stopped, or whether something between it and the database (e.g. a dropped or hung connection at the Docker networking layer) is why the database never saw the rest of the burst.

Host-level resource telemetry (run 4)

Before the watchdog above existed, we independently added host-level sampling (free -m / df -h / docker stats / docker ps, every 15s) around the same step, to rule out resource exhaustion as a simpler explanation. From run 4:

• Memory: stable throughout — available stayed at 6.6–6.8GB of 7938MB total; buff/cache grew (normal page caching), no pressure.
• Disk: climbed from 82% used (14G avail) to 91% used (7.2G avail) over ~90s before the Postgres container started — consistent with Docker image pull/extraction. Never exhausted; bottomed at 7.2G free.
• The DB container appeared ~94s into the step, health: starting, then healthy 22s later, using a trivial 65–76MiB memory / 0.04–4.4% CPU throughout.
• The stall occurs ~8–9s after the container reports healthy, mid-pipeline in the same fast Parse/Bind/Describe/Execute burst as every other run. No OOM/kill/disk-full signature anywhere in any of the six full logs (grepped explicitly — zero matches).

Combined with run 6's finding, this rules out host-level resource exhaustion (memory/disk/CPU on the runner) and database-level lock contention as explanations. What's left is that the stall has no visible counterpart on either the host or the database side — consistent with something happening in the path between the CLI process and the database, but we can't localize it further than that from here.

What's been ruled out

• Memory exhaustion — ruled out directly (run 4 telemetry).
• Disk exhaustion — real churn during image pull, never hit 0; not a clean explanation on its own.
• Database-side lock contention — ruled out directly (run 6: zero blocking locks, DB idle).
• Migration content — ruled out; identical stall across branches with materially different migration sets, always at the same pre-existing migration boundary, both before and after fixing an unrelated content bug that had briefly looked like a candidate cause.
• CLI version — tested 2.109.1 (five of six stalls) and 2.114.0 (one separate run, 32093728092, not counted above since it hit a different, self-inflicted, fully-diagnosed content bug before reaching the stall point — not conclusive either way on CLI version, but the stall independently reproduced on both versions across other runs).

What we have not been able to do

No Docker or Supabase CLI available in any environment outside this GitHub Actions pipeline (checked directly — nothing on PATH, no local Docker install), so no local reproduction, bisection, or testing against a different Docker daemon version was possible. Everything above comes directly from the GitHub Actions run logs and the watchdog/sampler artifacts.

For maintainers

• CLI versions observed: 2.109.1, and possibly 2.114.0 (inconclusive — see above).
• Environment: GitHub Actions ubuntu-24.04 hosted runner, Docker-based local stack (supabase start / supabase db reset --local).
• Symptom: mid-migration-apply, the CLI's SQL pipeline (visible via --debug's raw PG Send frame logging) stops emitting further Parse/Bind/Describe/Execute frames after a specific statement, with no error, no exception, no exit code reported until the step's own external timeout kills it.
• Database-side snapshot at the time of the stall (attached below) shows no active or blocked query matching the stalled statement — whatever stopped making progress isn't visible from inside Postgres.
• Reproduced 6 times across 4 branches and 2 CLI versions; not migration-content-dependent (always at the same relative point in a fixed sequence of already-applied migrations, regardless of what else is in the branch).
• Happy to share full --debug logs / run links for any of the six runs above if useful.