perf(cli): warm-restore the local database baseline on db reset and fresh start - #6223
perf(cli): warm-restore the local database baseline on db reset and fresh start#6223avallete wants to merge 3 commits into
Conversation
The shadow baseline cache is ON by default and roots its tar directory at the ambient `SUPABASE_HOME`, which these four suites never pinned. Any cold shadow provision they drove therefore read from — and published ~90MB-shaped tars into — the developer's real `~/.supabase/cache/shadow-baseline/`, on every test run. None of them has the cache as its subject, so they now opt out file-wide with `useLegacyShadowCacheDisabled()`, the same escape hatch `db diff`/`db pull`/ declarative sync/`migration squash` already use. Behavior under test is unchanged; the suites just stop reaching outside their own temp directories. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…resh start `supabase db reset` and the fresh-volume path of `supabase start`/`supabase db start` rebuilt the platform baseline from scratch every single run: init schema, the three PG15+ one-shot realtime/storage/auth migrate jobs, webhooks convergence, API privileges, vault, `roles.sql`. That is the exact cluster state `shadow-cache.ts` already snapshots as a PGDATA tar for the throwaway shadow, so the local `db` container now restores from — and publishes into — the SAME tar pool, under the same key, with the same LRU/TTL sweep. A `db diff` warms the snapshot a later `db reset` restores, and vice versa. Warm: the tar is unpacked into the created-but-unstarted Postgres container via `preStartArchives` (`docker cp -`), so the entrypoint finds an initialized PGDATA and skips `initdb` plus the whole baseline. `Restoring cached baseline...` prints where `Initialising schema...`/`Seeding globals from roles.sql...` would have, and the one-shot migrate jobs do not run. Migrations, seeding, satellite restarts and `_current_branch` are unchanged. A restore that comes up broken warns, force- removes the container AND its volume, and re-runs the bring-up cold — the same escape hatch the shadow already has. Cold: after `legacySetupDatabase` and strictly before `MigrateAndSeed`, the container is stopped, its PGDATA exported, restarted, and waited on with a direct connect probe. Publication is best-effort (`Warning: database baseline not cached: ...`); only a cluster that fails to come back fails the run. The pause is invisible to the rest of the stack: `supabase start` finishes the DB bootstrap before any other service starts, and `db reset` restarts its satellites straight after anyway. Structure: `main-db-baseline.ts` owns peek -> spec decoration -> skip-setup decision -> export seam. `baseline-state.ts` hoists the baseline-state seam so `db-setup.ts` can take it without importing `shadow-database.ts` back. `shadow-cache.ts`'s key resolution, peek, artifact helpers and export step are now keyed structurally, so one implementation serves both clusters. `legacyStartSetupLocalDatabase` folds into `legacyRunFreshDbSetup`, which owns the config load, the session lifecycle (two sessions when a snapshot splits them) and the prelude — the seam a warm hit has to skip. Gated exactly like the shadow: PG<=14, OrioleDB, an unreadable `roles.sql` and `SUPABASE_SHADOW_CACHE=false` all fall through to today's flow byte for byte, and `db start --from-backup` bypasses the cache outright since it owns `preStartArchives` itself. No new CLI flags. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e34f35651
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| yield* legacyRemoveContainer(spawner, args.dbContainerId); | ||
| yield* legacyRemoveVolume(spawner, args.dbContainerId); |
There was a problem hiding this comment.
When a cached tar is corrupt or truncated, legacyCreateContainer handles the failed docker cp by removing the newly created container before returning the error. This fallback then calls the strict legacyRemoveContainer, which treats “no such container” as an error, so execution stops before removing the named PGDATA volume or retrying the cold bring-up. The advertised corrupt-cache recovery therefore fails precisely for archive extraction errors; make this cleanup tolerate an already-removed container while still removing the volume.
Useful? React with 👍 / 👎.
| const exported = yield* Effect.result( | ||
| Effect.gen(function* () { | ||
| yield* legacyShadowContainerVerb(spawner, "stop", containerId); | ||
| yield* legacyShadowContainerVerb(spawner, "stop", args.containerId); |
There was a problem hiding this comment.
During a cold local db reset, a SIGINT after this successful docker stop but before the later restart interrupts the effect and skips revive, leaving the newly recreated database container stopped; unlike the two start flows, reset has no outer rollback hook. The stop/export/restart lifecycle needs an interruption-safe mask or finalizer that always attempts the restart before propagating interruption.
AGENTS.md reference: AGENTS.md:L64-L66
Useful? React with 👍 / 👎.
| const toml = yield* legacyCheckDbToml(fs, path, workdir, undefined, { | ||
| warnOnUnresolvedEnv: false, | ||
| }); |
There was a problem hiding this comment.
If config.toml changes while the container is coming up, the cache key was already computed from the handler's earlier legacyCheckDbToml result, but this second read supplies the webhooks policy, API grants, and vault secrets actually written before publication. The resulting tar is then stored under a key describing different contents, so a later run matching the old configuration can restore the wrong privileges, extensions, or secrets. Thread the same resolved TOML values used for keying through the baseline setup, as is already done for roles.sql.
Useful? React with 👍 / 👎.
| : { ...args.spec, preStartArchives: [restoreArchive] }; | ||
| yield* legacyCreateContainer(spawner, spec, args.containerOpts); | ||
| yield* args.waitReady; |
There was a problem hiding this comment.
This adds real Docker behavior to start—restoring a tar into a named PGDATA volume, restarting after publication, and falling back through container/volume removal—but the added command tests use mocked spawners, while the existing live cache test only restores a nameless shadow container without a named volume. Add a gated live scenario covering a cold then warm main-database start/reset so ownership, named-volume extraction, and actual restart behavior are exercised as required for changes to start's Docker interaction.
AGENTS.md reference: apps/cli/AGENTS.md:L520-L523
Useful? React with 👍 / 👎.
…n-db-baseline-restore # Conflicts: # apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts # apps/cli/src/legacy/commands/db/start/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/start/start.integration.test.ts # apps/cli/src/legacy/commands/start/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/start/start.integration.test.ts # apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts # apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts # apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts # apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts # apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9829de466a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| yield* output.raw("Restoring cached baseline...\n", "stderr"); | ||
| return { | ||
| kind: "warm", | ||
| restoreArchive: legacyPgDataRestoreArchive(input.fs, tarPath), |
There was a problem hiding this comment.
When a valid, bootable PGDATA tar is copied or renamed over another cache key, this path restores it without calling legacyValidatePgDataArchive; legacyPeekShadowBaseline only checks that the file exists. If the differing key represents roles, vault secrets, API grants, or service versions while retaining the same database password, the health check succeeds, the state remains warm, and the correct baseline is skipped, silently running with the wrong database contents. Validate the archive's embedded marker against peek.key before constructing the restore archive, as the shadow restore path already does, and fall back cold when it does not match.
Useful? React with 👍 / 👎.
| toml: input.toml, | ||
| spec: postgresSpec, | ||
| containerOpts: input.containerOpts, | ||
| cacheEligible: true, |
There was a problem hiding this comment.
During a local PG15 reset with the rest of the stack running, this enables publication of the recreated database even though the satellite containers and Kong are not restarted until after legacyRunFreshDbSetup completes. Because the replacement database uses the same stable network identity, auth, storage, or realtime can reconnect and handle traffic after it becomes healthy but before snapshotBaseline stops it; any rows or background state written in that interval are captured in the globally shared tar even though they are absent from the cache key, so later resets or another matching local project can warm-restore that runtime data. Stop or isolate database clients through the snapshot seam, or avoid publishing reset-produced baselines.
Useful? React with 👍 / 👎.
1 participant
Summary
Stacked on #6220 (← #6215 ← #6203 ← #6184 ← #6102).
supabase db resetand a fresh-volumesupabase start/db startran the full platform baseline (init schema + the PG15+ realtime/storage/auth one-shot migrate jobs, ~15s) against the realdbcontainer on every invocation — the exact cost the shadow-baseline cache already eliminates for shadows, produced by the same sharedlegacySetupDatabase. The main database now shares that cache pool:preStartArchivesentry into thedbcontainer spec (unpacked betweendocker createanddocker start, the mechanicspgdata-snapshot.ts/container-lifecycle.tsalready provide), the platform baseline is skipped entirely, and only the project's own migrations + seed replay. The transcript prints an explicitRestoring cached baseline...in place ofInitialising schema.../Seeding globals from roles.sql....legacySetupDatabase, before migrate/seed) via a stop →docker cp→ restart-with-connect-probe dance. Publication is best-effort: any failure warns and the command completes normally. Safe onsupabase startbecause the DB bootstrap fully completes before any other service starts.SUPABASE_SHADOW_CACHEdefault-on env as escape hatch);db start --from-backupbypasses entirely; the PG≤14 reset path is untouched.Structure:
main-db-baseline.tsowns the whole bring-up (legacyBringUpMainDbWithBaseline— peek → spec decoration → create with warm-fallback), bothstart-database.tsandrecreate-local-database.tsreduce to one call.baseline-state.tshoists the cluster baseline state as awarm | cold | uncacheddiscriminated union plus the session-opening branch bothdb-setup.tsandshadow-database.tspreviously duplicated;legacyRunFreshDbSetupabsorbs the deletedlegacyStartSetupLocalDatabase.shadow-cache.ts's key resolution, export step, and eviction are generalized structurally (LegacyBaselineCacheInput) so one implementation serves both cluster kinds; shadow warn/transcript strings are preserved verbatim.Supporting fixes that fell out:
config.tomlvalues are threaded through instead of a redundant re-parse (fresh runs stay at the pre-existing two parses, and nothing runs when the cache is ineligible).roles.sqlis read once: the peeked bytes that were hashed into the cache key are the exact bytes the seeding step executes, closing a narrow TOCTOU where the key could describe content that was never applied (also fixed for shadows).legacyMemoizeSuccesson the input), so key resolution and the setup prelude share one resolution.LegacyBaselineSnapshotRevivalFailureis aData.TaggedError.-c max_worker_processes=0while the main db does not, and this is deliberately unkeyed — safe today because the flag is command-line-only and no PG15+ baseline step persists worker-dependent state; any future baseline step that depends on live background workers must add the discriminator to the key.The first commit is standalone test hygiene: four integration suites drove cold shadow provisions with the cache on and
SUPABASE_HOMEunpinned, writing stray tars into the developer's real~/.supabaseon every test run.New integration coverage: reset publish/reuse/broken-restore-recreate/export-failure-warns/PG14-never-consults,
db startwarm fresh start and--from-backupbypass, driven by real cold-run-then-warm-run cycles rather than hardcoded keys.Linked issue
Closes #
open-for-contributionlabel (or I'm a Supabase maintainer).Checklist
fix(cli): …).pnpm check:allandpnpm testpass for the workspace(s) I touched.🤖 Generated with Claude Code