diff --git a/apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md index 3596528eeb..e3524dc9b5 100644 --- a/apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md @@ -41,16 +41,19 @@ removed `LegacyDeclarativeSeam.execInherit` seam — see those commands' own | seed files from `--sql-paths` or `[db.seed].sql_paths` | SQL | when seeding is enabled (not `--no-seed`); `--sql-paths` overrides config | | schema files from `[db.migrations].schema_paths` | SQL | when the `--experimental` schema-files branch is taken, either target (see Notes) | | `/supabase/buckets/` | files | local path, when storage is up and `[storage.buckets]` configure objects | -| `/supabase/roles.sql` | SQL | local PG15 path only, via the reused `legacyStartSetupLocalDatabase` pipeline — missing file tolerated | +| `/supabase/roles.sql` | SQL | local PG15 path only, via the reused `legacyRunFreshDbSetup` pipeline — missing file tolerated | +| `$SUPABASE_HOME/cache/shadow-baseline/shadow-baseline-.tar` | tar | local PG15 path only, on a warm baseline-cache hit — streamed into the created-but-unstarted container | | `~/.docker/config.json` | JSON | via the `docker`/`podman` CLI itself, for registry auth — never read directly by this process | ## Files Written -| Path | Format | When | -| ------------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | always (post-run telemetry flush) | -| `/supabase/.temp/pgdelta/catalog--migrations--.json` | JSON | best-effort, after migrations/seeding succeed, when no `--version`/`--last` resolved a version AND pg-delta is enabled (`[experimental.pgdelta].enabled` or `SUPABASE_EXPERIMENTAL_PG_DELTA`) AND the legacy engine is selected (`SUPABASE_USE_PG_DELTA_NEXT=false`); the default next engine skips this warmup entirely; a failure only warns on stderr and never fails the reset — see Notes. Native TS on both targets: **remote path** (`` = the project ref/URL hash) after either apply branch (schema-files or migrations); **local path** (`` = `"local"`) PG15 only, via the reused `legacyStartSetupLocalDatabase` pipeline (`db-setup.ts`) after `MigrateAndSeed` — the PG≤14 branch never calls this at all, so a PG≤14 local project never writes this file regardless of pg-delta config | +| Path | Format | When | +| ------------------------------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | always (post-run telemetry flush) | +| `/supabase/.temp/pgdelta/catalog--migrations--.json` | JSON | best-effort, after migrations/seeding succeed, when no `--version`/`--last` resolved a version AND pg-delta is enabled (`[experimental.pgdelta].enabled` or `SUPABASE_EXPERIMENTAL_PG_DELTA`); a failure only warns on stderr and never fails the reset — see Notes. Native TS on both targets: **remote path** (`` = the project ref/URL hash) after either apply branch (schema-files or migrations); **local path** (`` = `"local"`) PG15 only, via the reused `legacyRunFreshDbSetup` pipeline (`db-setup.ts`) after `MigrateAndSeed` — the PG≤14 branch never calls this at all, so a PG≤14 local project never writes this file regardless of pg-delta config | + +| `$SUPABASE_HOME/cache/shadow-baseline/shadow-baseline-.tar` (default `~/.supabase/…`) | tar | local PG15 path only, on a COLD baseline-cache miss: the freshly provisioned cluster's `PGDATA` is snapshotted at the baseline/migrations seam and published atomically (temp file + `rename`, mode `0o600`), then an LRU/TTL sweep may DELETE other `shadow-baseline-*.tar` entries in the same directory. Best-effort — a failure only prints `Warning: database baseline not cached: ` and never fails the reset. Skipped entirely when the cache is off (`SUPABASE_SHADOW_CACHE`), on PG≤14, and on OrioleDB. A warm restore that never becomes ready DELETES the offending tar before recreating cold | On the local path, the native recreate additionally recreates the `supabase_db_` container/volume (PG15) or the `postgres`/`_supabase` @@ -62,9 +65,11 @@ equivalent, PG15) or `InitSchema14`/`ApplyApiPrivileges` (PG14). | Command | When | Purpose | | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `docker container inspect supabase_db_` | local path | `AssertSupabaseDbIsRunning` probe (Podman fallback) | -| `docker container rm -f supabase_db_` / `docker volume rm -f ` | local path, PG15 | remove the existing container/volume before recreating (Podman fallback) | +| `docker container rm -f supabase_db_` / `docker volume rm -f ` | local path, PG15 | remove the existing container/volume before recreating (Podman fallback); issued a SECOND time when a warm baseline restore never becomes ready — the tar was unpacked INTO the named volume, so both must go before recreating cold | | `docker network create` / `docker volume create` / `docker create` / `docker start` | local path, PG15 | recreate the Postgres container (same primitives `db start` uses) | -| `docker run --rm ` | local path, PG15, per enabled service | the one-shot `initSchema15` migrate jobs (`legacyStartSetupLocalDatabase`) | +| `docker cp - :/var/lib/postgresql` | local path, PG15, warm cache hit | streams the cached baseline tar into the created-but-unstarted container so the entrypoint finds an initialized `PGDATA` and skips the whole platform baseline | +| `docker stop supabase_db_` → `docker cp supabase_db_:/var/lib/postgresql/data -` → `docker start ` | local path, PG15, cold cache miss | the baseline snapshot, taken at the baseline/migrations seam (a running Postgres's `PGDATA` is not coherent to copy). The stop+export degrade to a warning; a cluster that does not come back after the restart FAILS the reset | +| `docker run --rm ` | local path, PG15, per enabled service | the one-shot `initSchema15` migrate jobs (`legacyRunFreshDbSetup`) | | `docker restart ` | local path, PG14 | `RestartDatabase` — pg_cron must restart after `pg_terminate_backend` | | `docker restart ` | local path, both PG14 and PG15 | concurrent satellite-container restart, not-found tolerated per service | | `docker container inspect ` + `docker exec kong reload --nginx-conf /home/kong/custom_nginx.template` | local path, both PG14 and PG15 | reload Kong so it re-resolves the restarted containers' addresses (issue #6016) — the `--nginx-conf` flag is load-bearing: a bare `kong reload` regenerates nginx.conf from Kong's default template and drops the custom `email_templates` server (#6059) | @@ -90,7 +95,7 @@ child) is fully native as of CLI-1958. ### Local path (native, in TS) **PG15+:** the container/volume are removed and recreated (see "Subprocesses"), then -the reused `legacyStartSetupLocalDatabase` pipeline runs the initial schema (as +the reused `legacyRunFreshDbSetup` pipeline runs the initial schema (as one-shot Docker jobs, not SQL over a session), `ApplyApiPrivileges`, a vault upsert, a `roles.sql` seed, and `MigrateAndSeed` (migrations `≤ --version`, seed unless `--no-seed`) — over a fresh host-facing Postgres connection. @@ -128,21 +133,22 @@ the whole reset** (not just "skip buckets"). ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | -| `SUPABASE_ACCESS_TOKEN` | auth token for the `--linked` resolver path | no (falls back to keyring → `~/.supabase/access-token`) | -| `SUPABASE_DB_PASSWORD` | password for the linked/remote connection | no | -| `SUPABASE_YES` | auto-confirm the reset prompt | no (also `--yes`) | -| `SUPABASE_EXPERIMENTAL` | selects the schema-files apply branch on either target | no (also `--experimental`) | -| `SUPABASE_EXPERIMENTAL_PGDELTA_ENABLED` | overrides `[experimental.pgdelta].enabled`; a truthy value flips the reset gate (`experimental && resolvedVersion === "" && !toml.pgDelta.enabled`) back to timestamped migrations even with `--experimental` set — switches between two different destructive code paths | no | -| `SUPABASE_DB_MIGRATIONS_SCHEMA_PATHS` | overrides `[db.migrations].schema_paths` (viper `AutomaticEnv`, beats the config-file value) for the schema-files apply branch — genuinely effective on both targets now | no (no dedicated flag — config-file-only otherwise) | -| `SUPABASE_PROJECT_ID` | overrides the local container id; ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | -| `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the post-reset migrations-catalog cache (see Files Written) when `[experimental.pgdelta].enabled` is unset — distinct from `SUPABASE_EXPERIMENTAL_PGDELTA_ENABLED` above, which switches the reset's own apply branch instead | no (project `.env` or shell) | -| `SUPABASE_USE_PG_DELTA_NEXT` | selects the pg-delta implementation; `false` selects the legacy edge-runtime engine and thereby restores the migrations-catalog cache (unset/unrecognized defaults to the next engine, which skips it); shell presence wins over project `.env`, even an empty shell value | no (project `.env` or shell) | -| `SUPABASE_INTERNAL_IMAGE_REGISTRY` | overrides the pg-delta edge-runtime image registry for the migrations-catalog cache export (scoped for the whole run via `legacyApplyProjectEnv`, matching `db push`) | no (project `.env` or shell) | -| `PGDELTA_NPM_REGISTRY` | overrides the pg-delta edge-runtime npm registry (`.npmrc` + `NPM_CONFIG_REGISTRY` forward) for the migrations-catalog cache export (scoped for the whole run via `legacyApplyProjectEnv`, matching `db push`) | no (project `.env` or shell) | -| `SUPABASE_DB_PORT` / `SUPABASE_DB_MAJOR_VERSION` / `SUPABASE_DB_HEALTH_TIMEOUT` / `SUPABASE_DB_SETTINGS_*` | local-path container-recreate config overrides, same as `db start` | no | -| `SUPABASE_NETWORK_ID` (`--network-id`) | forces the recreated container/network onto an existing Docker network | no | +| Variable | Purpose | Required? | +| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | +| `SUPABASE_ACCESS_TOKEN` | auth token for the `--linked` resolver path | no (falls back to keyring → `~/.supabase/access-token`) | +| `SUPABASE_DB_PASSWORD` | password for the linked/remote connection | no | +| `SUPABASE_YES` | auto-confirm the reset prompt | no (also `--yes`) | +| `SUPABASE_EXPERIMENTAL` | selects the schema-files apply branch on either target | no (also `--experimental`) | +| `SUPABASE_EXPERIMENTAL_PGDELTA_ENABLED` | overrides `[experimental.pgdelta].enabled`; a truthy value flips the reset gate (`experimental && resolvedVersion === "" && !toml.pgDelta.enabled`) back to timestamped migrations even with `--experimental` set — switches between two different destructive code paths | no | +| `SUPABASE_DB_MIGRATIONS_SCHEMA_PATHS` | overrides `[db.migrations].schema_paths` (viper `AutomaticEnv`, beats the config-file value) for the schema-files apply branch — genuinely effective on both targets now | no (no dedicated flag — config-file-only otherwise) | +| `SUPABASE_PROJECT_ID` | overrides the local container id; ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | +| `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the post-reset migrations-catalog cache (see Files Written) when `[experimental.pgdelta].enabled` is unset — distinct from `SUPABASE_EXPERIMENTAL_PGDELTA_ENABLED` above, which switches the reset's own apply branch instead | no (project `.env` or shell) | +| `SUPABASE_INTERNAL_IMAGE_REGISTRY` | overrides the pg-delta edge-runtime image registry for the migrations-catalog cache export (scoped for the whole run via `legacyApplyProjectEnv`, matching `db push`) | no (project `.env` or shell) | +| `PGDELTA_NPM_REGISTRY` | overrides the pg-delta edge-runtime npm registry (`.npmrc` + `NPM_CONFIG_REGISTRY` forward) for the migrations-catalog cache export (scoped for the whole run via `legacyApplyProjectEnv`, matching `db push`) | no (project `.env` or shell) | +| `SUPABASE_DB_PORT` / `SUPABASE_DB_MAJOR_VERSION` / `SUPABASE_DB_HEALTH_TIMEOUT` / `SUPABASE_DB_SETTINGS_*` | local-path container-recreate config overrides, same as `db start` | no | +| `SUPABASE_NETWORK_ID` (`--network-id`) | forces the recreated container/network onto an existing Docker network | no | +| `SUPABASE_SHADOW_CACHE` | gates the local PG15 database-baseline cache (on by default); a falsy value provisions the baseline from scratch on every reset and never reads or writes a tar | no (project `.env` or shell) | +| `SUPABASE_HOME` | roots the baseline tar pool at `$SUPABASE_HOME/cache/shadow-baseline/` instead of `~/.supabase/…` | no | ## Exit Codes @@ -264,7 +270,7 @@ path has no confirmation prompt. post-apply cache (see that command's SIDE_EFFECTS Notes) — rather than a second copy. - **Local path** (native since CLI-1955/2062, no Go child involved): the reused - `legacyStartSetupLocalDatabase` pipeline (`db-setup.ts`) calls the same + `legacyRunFreshDbSetup` pipeline (`db-setup.ts`) calls the same `legacyTryCacheMigrationsCatalog` (with prefix `"local"`) right after `MigrateAndSeed` succeeds, warning the same way on failure. `reset.layers.ts` composes `legacyEdgeRuntimeScriptLayer`/`legacyPgDeltaSslProbeLayer` for this — diff --git a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts index 4addec1d0e..b4110204c6 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts @@ -1,9 +1,16 @@ -import { chmodSync, mkdirSync, writeFileSync } from "node:fs"; +import { + chmodSync, + existsSync, + mkdirSync, + readdirSync, + readFileSync, + writeFileSync, +} from "node:fs"; import { dirname, join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; -import { Cause, Effect, Exit, Layer, Option, PlatformError, Sink, Stream } from "effect"; +import { Cause, Effect, Exit, Layer, Option, Path, PlatformError, Sink, Stream } from "effect"; import { ChildProcessSpawner } from "effect/unstable/process"; import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; @@ -21,7 +28,9 @@ import { mockLegacyLinkedProjectCacheTracked, mockLegacyPlatformApiService, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, + withLegacyShadowCacheEnabled, legacySequentialExecBatch, } from "../../../../../tests/helpers/legacy-mocks.ts"; import { LegacyPlatformApi } from "../../../auth/legacy-platform-api.service.ts"; @@ -47,6 +56,7 @@ import { type LegacyEdgeRuntimeRunOpts, } from "../../../shared/legacy-edge-runtime-script.service.ts"; import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { legacyShadowBaselineCacheDir } from "../../../shared/legacy-pgdelta.paths.ts"; import { LegacyDbConfigResolver } from "../../../shared/legacy-db-config.service.ts"; import type { LegacyDbConfigFlags, @@ -159,6 +169,8 @@ function mockConnection( /** When set, an `exec` whose SQL contains this substring fails instead of succeeding. */ execFailsOn?: string; execFailsMessage?: string; + /** Shared SQL/Docker ordering log — see {@link setup}'s own `timeline` option. */ + timeline?: Array; } = {}, ) { const execs: Array = []; @@ -178,6 +190,7 @@ function mockConnection( ); } execs.push(sql); + opts.timeline?.push(`sql ${sql}`); if (opts.failStatement !== undefined && sql === opts.failStatement.sql) { return Effect.fail( new LegacyDbExecError({ @@ -257,7 +270,10 @@ type RouteResult = { readonly stderr?: ReadonlyArray; }; -function mockContainerCliSpawner(route: (args: ReadonlyArray) => RouteResult) { +function mockContainerCliSpawner( + route: (args: ReadonlyArray) => RouteResult, + timeline?: Array, +) { const spawned: Array = []; const encoder = new TextEncoder(); @@ -267,6 +283,7 @@ function mockContainerCliSpawner(route: (args: ReadonlyArray) => RouteRe Effect.gen(function* () { const args = command._tag === "StandardCommand" ? command.args : []; spawned.push({ args }); + timeline?.push(`docker ${args.join(" ")}`); if (command._tag !== "StandardCommand") { return yield* Effect.fail( @@ -339,7 +356,7 @@ const restartedContainers = (spawned: ReadonlyArray): ReadonlyArray const kongReloadCalls = (spawned: ReadonlyArray): ReadonlyArray => spawned.filter((s) => s.args[0] === "exec" && s.args[1] === KONG_ID); -/** The three PG15+ one-shot migrate jobs (`legacyStartSetupLocalDatabase`'s `LegacyDockerRun` calls). */ +/** The three PG15+ one-shot migrate jobs (`legacyRunFreshDbSetup`'s `LegacyDockerRun` calls). */ const dbSetupJobCalls = (spawned: ReadonlyArray): ReadonlyArray => spawned.filter((s) => s.args[0] === "run" && s.args[1] === "--rm"); @@ -406,6 +423,79 @@ function defaultLocalResetRoute(opts: DefaultRouteOpts = {}) { }; } +// --------------------------------------------------------------------------- +// Baseline cache (`db-bootstrap/main-db-baseline.ts`) — per-test `SUPABASE_HOME` +// --------------------------------------------------------------------------- + +/** + * The per-test `SUPABASE_HOME` every cache-enabled scenario is rooted at, as a child of the + * suite's own temp workdir. NEVER the developer's real `~/.supabase`: a tar leaked there is a + * ~90MB artefact in production and would be restored by their next real `supabase start`. + */ +const SHADOW_HOME_DIR = "_supabase_home"; + +/** A `Path` for the pure helper below, which has no effect context of its own to read one from. */ +const POSIX_PATH = Effect.runSync(Path.Path.pipe(Effect.provide(Path.layer))); + +/** Derived exactly like production's, so a relocated cache directory cannot pass unnoticed. */ +const shadowBaselineDir = (workdir: string): string => + legacyShadowBaselineCacheDir(POSIX_PATH, { SUPABASE_HOME: join(workdir, SHADOW_HOME_DIR) }); + +/** Every published `shadow-baseline-.tar` under the per-test home (`[]` when none exist). */ +const publishedBaselineTars = (workdir: string): ReadonlyArray => + existsSync(shadowBaselineDir(workdir)) + ? readdirSync(shadowBaselineDir(workdir)).filter((name) => name.endsWith(".tar")) + : []; + +/** `PGDATA` inside every `supabase/postgres` image — the path the export/restore argv names. */ +const PGDATA_PATH = "/var/lib/postgresql/data"; + +/** + * The baseline restore's own `docker cp - :/var/lib/postgresql` (PGDATA's PARENT — the + * export tar carries `data/` as its own top-level member). Deliberately narrower than + * `cp -`: every container create also copies its secret files in that way, targeting `:/`. + */ +const isBaselineRestoreCp = (args: ReadonlyArray): boolean => + args[0] === "cp" && args[1] === "-" && (args[2] ?? "").endsWith(":/var/lib/postgresql"); + +const isBaselineRestoreEntry = (entry: string): boolean => + entry.startsWith("docker cp - ") && entry.endsWith(":/var/lib/postgresql"); + +/** Last index in an ordering {@link setup} `timeline` matching `pred`, or `-1`. */ +const lastIndexWhere = ( + timeline: ReadonlyArray, + pred: (entry: string) => boolean, +): number => timeline.reduce((found, entry, index) => (pred(entry) ? index : found), -1); + +/** + * The whole recreate, but the `db` container only reports healthy once a SECOND one has been + * created — a warm restore that extracts cleanly and then never answers, which is the one failure + * shape that implicates the tar's own contents. + */ +function unhealthyUntilRecreatedRoute(): (args: ReadonlyArray) => RouteResult { + const base = defaultLocalResetRoute(); + let creates = 0; + return (args: ReadonlyArray): RouteResult => { + if (args[0] === "create") { + creates += 1; + return base(args); + } + if (args[0] === "container" && args[1] === "inspect" && args[2] === DB_ID && creates < 2) { + return { stdout: [STARTING_STATE] }; + } + return base(args); + }; +} + +/** Fails the snapshot's own `docker cp : -`, leaving the restore direction working. */ +function exportFailsRoute(): (args: ReadonlyArray) => RouteResult { + const base = defaultLocalResetRoute(); + return (args: ReadonlyArray): RouteResult => + args[0] === "cp" && args[1] !== "-" + ? { exitCode: 1, stderr: ["Error: No such container:path"] } + : base(args); +} + const alwaysReadyHttpClientLayer = Layer.succeed( HttpClient.HttpClient, HttpClient.make((request) => @@ -446,6 +536,15 @@ function setup( // `LegacyProjectNotLinkedError` absent an explicit `--project-ref` flag, // instead of silently falling back to `opts.ref ?? LEGACY_VALID_REF`. linkedFails?: boolean; + /** + * Opt-in interleaved log of every `docker ` spawn AND every SQL statement executed + * over the mocked session, in the order they actually happened. `child.spawned` and + * `conn.execs` are each ordered on their own but say nothing about how the two relate — and + * the baseline snapshot's ONE hard ordering contract (`db-bootstrap/main-db-baseline.ts`) is + * exactly a cross-boundary one: the `docker stop`/`cp`/`start` seam sits after the setup + * jobs and strictly before the first migration statement. + */ + timeline?: Array; }, ) { if (opts.toml !== undefined) { @@ -472,7 +571,7 @@ function setup( resolveFails: opts.resolveFails, }); const route = opts.route ?? defaultLocalResetRoute(opts.routeOpts); - const child = mockContainerCliSpawner(route); + const child = mockContainerCliSpawner(route, opts.timeline); // Backs both the local recreate's post-setup pg-delta migrations-catalog warmup // (`db-setup.ts`'s `legacyTryCacheMigrationsCatalog`) and the remote path's own // post-reset catalog-cache call — tracked so tests can assert on it directly @@ -570,6 +669,12 @@ const FAST_HEALTH_TOML = '[db]\nhealth_timeout = "1s"\n'; describe("legacy db reset", () => { const tmp = useLegacyTempWorkdir("supabase-db-reset-"); + // The baseline cache (`db-bootstrap/main-db-baseline.ts`) is ON by default and reads + // `process.env` directly, so every PG15 recreate below would otherwise restore or publish a real + // tar under the developer's own `~/.supabase`. This suite's subject is the reset pipeline; the + // cache-focused scenarios opt back in per-test with `withLegacyShadowCacheEnabled`. + useLegacyShadowCacheDisabled(); + describe("local reset — PG15+", () => { it.live("recreates the container, waits healthy, and runs the setup pipeline", () => { const { layer, out, child, telemetry } = setup(tmp.current, { @@ -889,6 +994,200 @@ describe("legacy db reset", () => { }); }); + // The baseline cache is ON in production; the suite-wide `useLegacyShadowCacheDisabled` above + // turns it off everywhere else so the other scenarios assert the plain recreate. These turn it + // back on — under a per-test `SUPABASE_HOME`, so the ~90MB-in-production tar never lands in the + // developer's real `~/.supabase` — and drive `db reset` for real against it. The cache's own + // mechanics (key derivation, atomic publish, retention) are covered at their own level in + // `shared/db-bootstrap/shadow-cache.integration.test.ts`. + describe("local reset — cached database baseline", () => { + const CACHE_TOML = 'project_id = "test"\n[db]\nhealth_timeout = "1s"\n'; + + /** Re-enables the cache the suite-wide gate turned off, rooted at this test's own home. */ + const withCacheEnabled = (body: Effect.Effect): Effect.Effect => + withLegacyShadowCacheEnabled(join(tmp.current, SHADOW_HOME_DIR), body); + + /** One full `db reset --local` over a project with a single, identifiable migration. */ + const runReset = Effect.fnUntraced(function* ( + marker: string, + opts: { route?: (args: ReadonlyArray) => RouteResult } = {}, + ) { + const timeline: Array = []; + const s = setup(tmp.current, { + toml: CACHE_TOML, + files: migrationFile("20240101000000", `create table ${marker} ();`), + args: ["db", "reset", "--local"], + isLocal: true, + timeline, + ...(opts.route === undefined ? {} : { route: opts.route }), + }); + yield* legacyDbReset(DEFAULT_FLAGS).pipe(Effect.provide(s.layer)); + return { ...s, timeline }; + }); + + it.live( + "publishes a database baseline on a first reset, snapshotting between the setup jobs and the migrations", + () => + withCacheEnabled( + Effect.gen(function* () { + const cold = yield* runReset("cold_marker"); + + // Cold: the platform baseline really ran, and nothing was restored. + expect(cold.out.stderrText).toContain("Initialising schema..."); + expect(cold.out.stderrText).not.toContain("Restoring cached baseline..."); + expect(dbSetupJobCalls(cold.child.spawned)).toHaveLength(3); + + // Exactly one tar, under this test's own home — never the developer's. + expect(publishedBaselineTars(tmp.current)).toHaveLength(1); + + // The snapshot seam is the contract: stop -> export -> start, strictly after the + // baseline's one-shot jobs and strictly before the first user migration replays. + const lastSetupJob = lastIndexWhere(cold.timeline, (entry) => + entry.startsWith("docker run --rm"), + ); + const stop = cold.timeline.indexOf(`docker stop ${DB_ID}`); + const exportCp = cold.timeline.indexOf(`docker cp ${DB_ID}:${PGDATA_PATH} -`); + const restart = cold.timeline.indexOf(`docker start ${DB_ID}`); + const migration = cold.timeline.findIndex((entry) => + entry.includes("create table cold_marker ()"), + ); + expect(lastSetupJob).toBeGreaterThanOrEqual(0); + expect(stop).toBeGreaterThan(lastSetupJob); + expect(exportCp).toBeGreaterThan(stop); + expect(restart).toBeGreaterThan(exportCp); + expect(migration).toBeGreaterThan(restart); + }), + ), + 20_000, + ); + + it.live( + "reuses the published baseline on the next reset instead of re-running it", + () => + withCacheEnabled( + Effect.gen(function* () { + yield* runReset("warm_marker"); + expect(publishedBaselineTars(tmp.current)).toHaveLength(1); + + const warm = yield* runReset("warm_marker"); + + // The restored cluster already carries the baseline, so neither its progress text + // nor its one-shot migrate jobs happen — and nothing is re-snapshotted. + expect(warm.out.stderrText).toContain("Restoring cached baseline...\n"); + expect(warm.out.stderrText).not.toContain("Initialising schema..."); + expect(warm.out.stderrText).not.toContain("Seeding globals from roles.sql..."); + expect(dbSetupJobCalls(warm.child.spawned)).toHaveLength(0); + expect(warm.timeline).not.toContain(`docker stop ${DB_ID}`); + + // The tar is unpacked into the created-but-unstarted container: the entrypoint has + // to find an initialized PGDATA, so the copy MUST precede `docker start`. + const restoreCp = warm.timeline.findIndex(isBaselineRestoreEntry); + const start = warm.timeline.findIndex((entry) => entry.startsWith("docker start ")); + expect(restoreCp).toBeGreaterThanOrEqual(0); + expect(start).toBeGreaterThan(restoreCp); + + // Everything downstream of the baseline is unchanged: migrations still replay and + // the satellites still restart behind a reloaded Kong. + expect(warm.conn.execs.some((sql) => sql.includes("create table warm_marker ()"))).toBe( + true, + ); + expect(restartedContainers(warm.child.spawned)).toEqual( + expect.arrayContaining([ + "supabase_storage_test", + "supabase_auth_test", + "supabase_realtime_test", + "supabase_pooler_test", + ]), + ); + expect(kongReloadCalls(warm.child.spawned)).toHaveLength(1); + expect(warm.out.stderrText).toContain("Finished "); + expect(publishedBaselineTars(tmp.current)).toHaveLength(1); + }), + ), + 20_000, + ); + + it.live( + "recreates the database from scratch when the cached baseline never comes up", + () => + withCacheEnabled( + Effect.gen(function* () { + yield* runReset("fallback_marker"); + const [tarName] = publishedBaselineTars(tmp.current); + expect(tarName).not.toBeUndefined(); + const tarPath = join(shadowBaselineDir(tmp.current), tarName ?? ""); + // Mark the published tar so the fallback's own republish is distinguishable from it. + writeFileSync(tarPath, "SUSPECT-BASELINE"); + + const recovered = yield* runReset("fallback_marker", { + route: unhealthyUntilRecreatedRoute(), + }); + + // The user is told, rather than left to debug a cluster the cache broke. + expect(recovered.out.stderrText).toContain("Restoring cached baseline...\n"); + expect(recovered.out.stderrText).toContain( + "Warning: cached database baseline unusable (", + ); + expect(recovered.out.stderrText).toContain("); recreating.\n"); + + // Container AND volume are force-removed a second time (the recreate already did it + // once) — the tar was unpacked INTO the named volume, so reusing it would just boot + // the same broken cluster. + expect(removedContainers(recovered.child.spawned).filter((id) => id === DB_ID)).toEqual( + [DB_ID, DB_ID], + ); + expect(removedVolumes(recovered.child.spawned).filter((id) => id === DB_ID)).toEqual([ + DB_ID, + DB_ID, + ]); + + // Exactly one replacement container, created WITHOUT a restore archive. + const creates = recovered.child.spawned.filter((s) => s.args[0] === "create"); + const restores = recovered.child.spawned.filter((s) => isBaselineRestoreCp(s.args)); + expect(creates).toHaveLength(2); + expect(restores).toHaveLength(1); + + // The replacement runs the full baseline, and the reset still succeeds. + expect(recovered.out.stderrText).toContain("Initialising schema..."); + expect(dbSetupJobCalls(recovered.child.spawned)).toHaveLength(3); + expect( + recovered.conn.execs.some((sql) => sql.includes("create table fallback_marker ()")), + ).toBe(true); + expect(recovered.out.stderrText).toContain("Finished "); + + // The unusable tar does not survive the run — a later reset can never restore it + // again. It was deleted as suspect, then replaced by the fallback's own snapshot. + expect(publishedBaselineTars(tmp.current)).toEqual([tarName]); + expect(readFileSync(tarPath, "utf8")).not.toContain("SUSPECT-BASELINE"); + }), + ), + 30_000, + ); + + it.live( + "warns but still completes the reset when the baseline cannot be exported", + () => + withCacheEnabled( + Effect.gen(function* () { + const s = yield* runReset("uncached_marker", { route: exportFailsRoute() }); + + expect(s.out.stderrText).toContain("Warning: database baseline not cached: "); + // Nothing published — a half-written tar must never be observable under the final + // name, so a failed export leaves the pool exactly as it found it. + expect(publishedBaselineTars(tmp.current)).toEqual([]); + + // The run itself is untouched by the cache's failure. + expect(s.out.stderrText).toContain("Initialising schema..."); + expect( + s.conn.execs.some((sql) => sql.includes("create table uncached_marker ()")), + ).toBe(true); + expect(s.out.stderrText).toContain("Finished "); + }), + ), + 20_000, + ); + }); + describe("local reset — PG14", () => { it.live( "recreates via the four-statement DROP/CREATE sequence, then initDatabase + RestartDatabase", @@ -1197,6 +1496,33 @@ describe("legacy db reset", () => { }); }); + it.live("never consults the database baseline cache on a PG14 reset", () => + // PG <= 14 is cache-ineligible: its setup path execs the bundled globals SQL, whose + // `ALTER ROLE … SET` defaults only take effect on NEW sessions, so a snapshot boundary's + // forced reconnect would change what the following migrations see. With the cache + // explicitly ON, this path must still behave byte for byte like an uncached run. + withLegacyShadowCacheEnabled( + join(tmp.current, SHADOW_HOME_DIR), + Effect.gen(function* () { + const { layer, out, child } = setup(tmp.current, { + toml: PG14_TOML, + args: ["db", "reset", "--local"], + isLocal: true, + }); + yield* legacyDbReset(DEFAULT_FLAGS).pipe(Effect.provide(layer)); + expect(out.stderrText).not.toContain("Restoring cached baseline..."); + expect(out.stderrText).not.toContain("Warning: database baseline not cached"); + // No snapshot seam at all — the PG14 path's own `docker restart` is its only + // container verb (`restart`, never `stop`/`cp`/`start`). + expect(child.spawned.some((s) => s.args[0] === "stop")).toBe(false); + expect(child.spawned.some((s) => s.args[0] === "start")).toBe(false); + expect(child.spawned.some((s) => s.args[0] === "cp")).toBe(false); + // The cache directory is never even created, let alone written to. + expect(existsSync(shadowBaselineDir(tmp.current))).toBe(false); + }), + ), + ); + it.live( "passes the resolved --version cutoff through to the final MigrateAndSeed step (PG14)", () => { diff --git a/apps/cli/src/legacy/commands/db/reset/reset.layers.ts b/apps/cli/src/legacy/commands/db/reset/reset.layers.ts index 14c07e3a07..8a4298a975 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.layers.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.layers.ts @@ -27,7 +27,7 @@ import { legacyTelemetryStateLayer } from "../../../telemetry/legacy-telemetry-s * command, so `LegacyGoProxy` is not composed here. * * `legacyDockerRunLayer` backs the native local recreate's PG15+ one-shot migrate - * jobs (`legacyStartSetupLocalDatabase`, reused via `legacyRecreateLocalDatabase`) + * jobs (`legacyRunFreshDbSetup`, reused via `legacyRecreateLocalDatabase`) * — same reasoning as `db start`'s own `start.layers.ts`. * `legacyEdgeRuntimeScriptLayer`/`legacyPgDeltaSslProbeLayer` back that same shared * setup pipeline's best-effort pg-delta migrations-catalog warmup (`db-setup.ts`'s diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index 74c0597144..3a7ff71044 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -26,6 +26,7 @@ import { mockLegacyPlatformApiService, mockLegacyTelemetryStateTracked, useLegacyTempWorkdir, + useLegacyShadowCacheDisabled, legacySequentialExecBatch, } from "../../../../../../../tests/helpers/legacy-mocks.ts"; import { CliArgs } from "../../../../../../shared/cli/cli-args.service.ts"; @@ -317,6 +318,11 @@ const flags = ( const failError = (exit: Exit.Exit) => Exit.isFailure(exit) ? exit.cause.reasons.find(Cause.isFailReason)?.error : undefined; +// The baseline PGDATA cache is ON by default and roots its tar directory at the ambient +// `SUPABASE_HOME` — which this suite does not pin — so a cold provision here would read from, and +// publish into, the developer's real `~/.supabase`. This suite's subject is not the cache. +useLegacyShadowCacheDisabled(); + describe("legacy db schema declarative generate integration", () => { const tmp = useLegacyTempWorkdir(); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts index d73968cf6c..e1b0190fa8 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts @@ -10,6 +10,7 @@ import { LegacyPgDeltaEngine } from "./legacy-pgdelta-engine.service.ts"; import { LegacyPgDeltaNextAdapter } from "./legacy-pgdelta-next-adapter.service.ts"; import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; import type { LegacyDbTomlValues } from "../../../shared/legacy-db-config.toml-read.ts"; +import { useLegacyShadowCacheDisabled } from "../../../../../tests/helpers/legacy-mocks.ts"; const common = { context: { @@ -104,6 +105,11 @@ function setup() { }; } +// The baseline PGDATA cache is ON by default and roots its tar directory at the ambient +// `SUPABASE_HOME` — which this suite does not pin — so a cold provision here would read from, and +// publish into, the developer's real `~/.supabase`. This suite's subject is not the cache. +useLegacyShadowCacheDisabled(); + describe("pg-delta next shadow selection", () => { it.effect("does not provision a second shadow for prepared database diffs", () => { const { state, layer } = setup(); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index e05d2ddf1a..0cb80a0369 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -92,7 +92,6 @@ interface NativeShadowBase { interface ProvisionedDeclarativeShadow { readonly declarativeUrl: string; readonly restoredFromPgDataSnapshot: boolean; - readonly snapshotKey: string | undefined; } /** @@ -282,14 +281,14 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( // reaches that seam when its export publishes the tar; any other handle (warm because // another process published between peek and acquire, or uncached) never runs a // snapshot, so signal immediately — the waiter then just re-peeks current disk state. - const seamWillRun = handle.snapshotRequired && !handle.baselinePresent; - const seamHandle: LegacyShadowAcquiredHandle = seamWillRun - ? { - ...handle, - snapshotBaseline: handle.snapshotBaseline.pipe(Effect.ensuring(onBaselineSeam)), - } - : handle; - if (!seamWillRun) yield* onBaselineSeam; + const seamHandle: LegacyShadowAcquiredHandle = + handle._tag === "cold" + ? { + ...handle, + snapshotBaseline: handle.snapshotBaseline.pipe(Effect.ensuring(onBaselineSeam)), + } + : handle; + if (handle._tag !== "cold") yield* onBaselineSeam; yield* awaitShadowReady(input, seamHandle); const setup = setupRunInput(input, seamHandle); yield* legacyMigrateNextShadowDatabase(input.spawner, setup, seamHandle); @@ -319,8 +318,7 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( ); return { declarativeUrl: legacyToPostgresURL(setup.connConfig), - restoredFromPgDataSnapshot: handle.baselinePresent, - snapshotKey: handle.snapshotKey, + restoredFromPgDataSnapshot: handle._tag === "warm", } satisfies ProvisionedDeclarativeShadow; }).pipe(Effect.provide(runtimeWith(outputService)), Effect.mapError(nextShadowError)); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.plan.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.plan.ts index 10d92a3997..0871022d6b 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.plan.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.plan.ts @@ -10,7 +10,7 @@ * * - `parallel` — both snapshots are published: both provisions warm-restore concurrently. A warm * provision skips the platform baseline entirely (`legacySetupShadowDatabase`'s - * `baselinePresent` branch), so the declarative fiber prints nothing and the migrations fiber's + * `warm` branch), so the declarative fiber prints nothing and the migrations fiber's * `Applying migration ...` lines stream live and in order. * - `baseline-handoff` — both are cold with the SAME cache key (webhooks agree): the baseline is * paid exactly once. The migrations shadow cold-provisions; its snapshot export runs at the diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.apply.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.apply.integration.test.ts index 6522c8f877..8f6be2a93b 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.apply.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.apply.integration.test.ts @@ -16,6 +16,7 @@ import { import { LegacyEdgeRuntimeScriptError } from "../../../shared/legacy-edge-runtime-script.errors.ts"; import { legacyApplyDeclarativePgDelta } from "./legacy-pgdelta.apply.ts"; import type { LegacyPgDeltaContext } from "../../../shared/legacy-pgdelta.ts"; +import { useLegacyShadowCacheDisabled } from "../../../../../tests/helpers/legacy-mocks.ts"; const CTX: LegacyPgDeltaContext = { projectId: "ref", @@ -52,6 +53,11 @@ function makeDeclarativeDir(): string { const failError = (exit: Exit.Exit) => Exit.isFailure(exit) ? exit.cause.reasons.find(Cause.isFailReason)?.error : undefined; +// The baseline PGDATA cache is ON by default and roots its tar directory at the ambient +// `SUPABASE_HOME` — which this suite does not pin — so a cold provision here would read from, and +// publish into, the developer's real `~/.supabase`. This suite's subject is not the cache. +useLegacyShadowCacheDisabled(); + describe("legacyApplyDeclarativePgDelta", () => { it.effect( "fails with LegacyPgDeltaDeclarativeApplyError interpolating the RELATIVE dir, not the absolute one, when the declarative dir doesn't exist", diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.integration.test.ts index ec0927af3c..ac7c408046 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.integration.test.ts @@ -10,6 +10,7 @@ import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; import { mockLegacyCliConfig, mockLegacyShadowContainerCliSpawner, + useLegacyShadowCacheDisabled, } from "../../../../../tests/helpers/legacy-mocks.ts"; import { mockOutput, mockRuntimeInfo } from "../../../../../tests/helpers/mocks.ts"; import { CliArgs } from "../../../../shared/cli/cli-args.service.ts"; @@ -178,6 +179,11 @@ function setup( const failError = (exit: Exit.Exit) => Exit.isFailure(exit) ? exit.cause.reasons.find(Cause.isFailReason)?.error : undefined; +// The baseline PGDATA cache is ON by default and roots its tar directory at the ambient +// `SUPABASE_HOME` — which this suite does not pin — so a cold provision here would read from, and +// publish into, the developer's real `~/.supabase`. This suite's subject is not the cache. +useLegacyShadowCacheDisabled(); + describe("legacyDeclarativeSeamLayer.exportCatalog", () => { it.effect( "provisions a shadow on a baseline cache miss, then reuses the cached catalog with no further container work", diff --git a/apps/cli/src/legacy/commands/db/start/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/start/SIDE_EFFECTS.md index 88457f78aa..93a0d40a21 100644 --- a/apps/cli/src/legacy/commands/db/start/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/start/SIDE_EFFECTS.md @@ -67,36 +67,41 @@ volume was confirmed fresh this run). | `/supabase/roles.sql` | SQL | on a fresh volume with no `--from-backup` — the "Seeding globals..." message always prints first; a missing file is tolerated | | `/supabase/migrations/*.sql`, `supabase/seed.sql` | SQL | on a fresh volume with no `--from-backup`, via the standard migration-apply + seed pipeline | | `/supabase/` (files/directories/globs) | SQL | on a fresh volume with no `--from-backup`, INSTEAD of `migrations/*.sql`, when `--experimental`/`SUPABASE_EXPERIMENTAL` is set and `[experimental.pgdelta] enabled` is false | +| `$SUPABASE_HOME/cache/shadow-baseline/shadow-baseline-.tar` | tar | on a fresh volume with no `--from-backup`, on a warm baseline-cache hit — streamed into the created-but-unstarted container | | `/supabase/.branches/_current_branch` | text | always, existence check before writing (see "Files Written") | | `~/.docker/config.json` | JSON | via the `docker`/`podman` CLI itself, for registry auth — never read directly by this process | ## Files Written -| Path | Format | When | -| ---------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/.branches/_current_branch` | text | only if absent — writes `"main"` (see the step-by-step sequence above for exactly when) | -| `/supabase/.temp/pgdelta/catalog-local-migrations--.json` | JSON | best-effort, on a fresh volume with no `--from-backup`, after `MigrateAndSeed`, when pg-delta is enabled (`[experimental.pgdelta] enabled` or `SUPABASE_EXPERIMENTAL_PG_DELTA`) AND the legacy engine is selected (`SUPABASE_USE_PG_DELTA_NEXT=false`); the default next engine skips this warmup entirely; a failure only warns on stderr and never fails `db start` | -| local Docker volume `supabase_db_` | — | the Postgres data volume, created on first start (or first `--from-backup` restore) | -| local Docker network `supabase_network_` (or `--network-id`) | — | created if it doesn't already exist | -| `~/.supabase/telemetry.json` | JSON | always — telemetry flush (`Effect.ensuring(telemetryState.flush)`), success and failure | +| Path | Format | When | +| ------------------------------------------------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/.branches/_current_branch` | text | only if absent — writes `"main"` (see the step-by-step sequence above for exactly when) | +| `/supabase/.temp/pgdelta/catalog-local-migrations--.json` | JSON | best-effort, on a fresh volume with no `--from-backup`, after `MigrateAndSeed`, when pg-delta is enabled (`[experimental.pgdelta] enabled` or `SUPABASE_EXPERIMENTAL_PG_DELTA`); a failure only warns on stderr and never fails `db start` | +| `$SUPABASE_HOME/cache/shadow-baseline/shadow-baseline-.tar` (default `~/.supabase/…`) | tar | on a fresh volume with no `--from-backup`, on a COLD baseline-cache miss: the freshly provisioned cluster's `PGDATA` is snapshotted at the baseline/migrations seam and published atomically (temp file + `rename`, mode `0o600`), then an LRU/TTL sweep may DELETE other `shadow-baseline-*.tar` entries in the same directory. Best-effort — a failure only prints `Warning: database baseline not cached: ` and never fails `db start`. Skipped entirely when the cache is off (`SUPABASE_SHADOW_CACHE`), on PG≤14, on OrioleDB, and on `--from-backup` (which owns the container's `preStartArchives` itself and provisions no platform baseline). A warm restore that never becomes ready DELETES the offending tar before recreating cold | +| local Docker volume `supabase_db_` | — | the Postgres data volume, created on first start (or first `--from-backup` restore) | +| local Docker network `supabase_network_` (or `--network-id`) | — | created if it doesn't already exist | +| `~/.supabase/telemetry.json` | JSON | always — telemetry flush (`Effect.ensuring(telemetryState.flush)`), success and failure | ## Subprocesses Every step below shells out to `docker` (falling back to `podman`), matching every other native container command in this codebase — never `supabase-go`. -| Command | When | -| -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `docker container inspect supabase_db_` | always — the already-running probe | -| `docker network create --label ... ` | when not already running, unless `--network-id` names a built-in network | -| `docker volume inspect supabase_db_` | when not already running — the pre-create fresh-volume probe | -| `docker image inspect` / `docker pull` (registry-fallback resolve) | when not already running — resolves the Postgres image | -| `docker volume create --label ...` | when not already running, unless a `--from-backup` restore onto an existing volume (fails first) | -| `docker create` + optional `docker cp - :/` + `docker start` | when not already running — `cp` streams one in-memory tar archive containing the pgsodium root key at its exact path when the selected entrypoint needs it | -| `docker container inspect` (repeated) | health-wait polling, 1s constant backoff up to `db.health_timeout` | -| `docker logs ` | on a health-check timeout (either path — swallowed or not) | -| `docker run --rm ...` | fresh volume, no `--from-backup`, `db.major_version >= 15`: up to 3 one-shot migrate jobs (realtime/storage/auth) | -| `docker ps` / `docker stop` / `docker container prune` / `docker volume prune` (fresh-volume runs only) / `docker network prune` | on ANY failure from network-ensure through `_current_branch` — the rollback | +| Command | When | +| -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `docker container inspect supabase_db_` | always — the already-running probe | +| `docker network create --label ... ` | when not already running, unless `--network-id` names a built-in network | +| `docker volume inspect supabase_db_` | when not already running — the pre-create fresh-volume probe | +| `docker image inspect` / `docker pull` (registry-fallback resolve) | when not already running — resolves the Postgres image | +| `docker volume create --label ...` | when not already running, unless a `--from-backup` restore onto an existing volume (fails first) | +| `docker create` + optional `docker cp - :/` + `docker start` | when not already running — `cp` streams one in-memory tar archive containing the pgsodium root key at its exact path when the selected entrypoint needs it | +| `docker cp - :/var/lib/postgresql` | fresh volume, no `--from-backup`, warm baseline-cache hit — streams the cached tar into the created-but-unstarted container, so the entrypoint finds an initialized `PGDATA` and skips the whole platform baseline | +| `docker container rm -f supabase_db_` + `docker volume rm -f ` | fresh volume, when a warm baseline restore never becomes ready — the tar was unpacked INTO the named volume, so both go before the cold recreate | +| `docker container inspect` (repeated) | health-wait polling, 1s constant backoff up to `db.health_timeout` | +| `docker logs ` | on a health-check timeout (either path — swallowed or not) | +| `docker run --rm ...` | fresh volume, no `--from-backup`, `db.major_version >= 15`: up to 3 one-shot migrate jobs (realtime/storage/auth) | +| `docker stop supabase_db_` → `docker cp supabase_db_:/var/lib/postgresql/data -` → `docker start ` | fresh volume, no `--from-backup`, cold baseline-cache miss: the baseline snapshot, taken at the baseline/migrations seam (a running Postgres's `PGDATA` is not coherent to copy). The stop+export degrade to a warning; a cluster that does not come back after the restart FAILS the command | +| `docker ps` / `docker stop` / `docker container prune` / `docker volume prune` (fresh-volume runs only) / `docker network prune` | on ANY failure from network-ensure through `_current_branch` — the rollback | ## API Routes @@ -125,6 +130,8 @@ native container command in this codebase — never `supabase-go`. | `SUPABASE_EXPERIMENTAL` (or `--experimental`) | fresh volume + no pg-delta: applies `db.migrations.schema_paths` files instead of `migrations/*.sql` | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the post-`MigrateAndSeed` migrations-catalog cache warmup when `[experimental.pgdelta].enabled` is unset | no | | `SUPABASE_USE_PG_DELTA_NEXT` | selects the pg-delta implementation; `false` selects the legacy edge-runtime engine and thereby restores the migrations-catalog cache warmup (unset/unrecognized defaults to the next engine, which skips it) | no | +| `SUPABASE_SHADOW_CACHE` | gates the fresh-volume database-baseline cache (on by default); a falsy value provisions the baseline from scratch every time and never reads or writes a tar | no | +| `SUPABASE_HOME` | roots the baseline tar pool at `$SUPABASE_HOME/cache/shadow-baseline/` instead of `~/.supabase/…` | no | | `DOCKER_HOST` / `DOCKER_CONTEXT` / `DOCKER_TLS_VERIFY` / `DOCKER_CERT_PATH` / `DOCKER_API_VERSION` / `DOCKER_CONFIG` | Read (ambient shell OR a project `.env`/`.env.`/`.env.local` file, installed into the process environment before any Docker work) to pick the Docker daemon this whole command talks to | no | `--network-id` (a global CLI flag, not an environment variable — `shared/legacy/global-flags.ts`) diff --git a/apps/cli/src/legacy/commands/db/start/start.integration.test.ts b/apps/cli/src/legacy/commands/db/start/start.integration.test.ts index 5f5244be1f..f03b708204 100644 --- a/apps/cli/src/legacy/commands/db/start/start.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/start/start.integration.test.ts @@ -3,7 +3,7 @@ import { join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { afterEach, describe, expect, it } from "@effect/vitest"; -import { Cause, Effect, Exit, Layer, Option, PlatformError, Sink, Stream } from "effect"; +import { Cause, Effect, Exit, Layer, Option, Path, PlatformError, Sink, Stream } from "effect"; import { ChildProcessSpawner } from "effect/unstable/process"; import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; @@ -17,7 +17,9 @@ import { import { mockLegacyCliConfig, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, + withLegacyShadowCacheEnabled, legacySequentialExecBatch, } from "../../../../../tests/helpers/legacy-mocks.ts"; import { CliArgs } from "../../../../shared/cli/cli-args.service.ts"; @@ -39,6 +41,7 @@ import { type LegacyEdgeRuntimeRunOpts, } from "../../../shared/legacy-edge-runtime-script.service.ts"; import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { legacyShadowBaselineCacheDir } from "../../../shared/legacy-pgdelta.paths.ts"; import { legacyDbStart } from "./start.handler.ts"; import type { LegacyDbStartFlags } from "./start.command.ts"; @@ -142,7 +145,7 @@ function bindsFromCreateArgs(args: ReadonlyArray): ReadonlyArray return binds; } -/** The three PG15+ one-shot migrate jobs (`legacyStartSetupLocalDatabase`'s `LegacyDockerRun` calls). */ +/** The three PG15+ one-shot migrate jobs (`legacyRunFreshDbSetup`'s `LegacyDockerRun` calls). */ function dbSetupJobCalls(spawned: ReadonlyArray): ReadonlyArray { return spawned.filter((s) => s.args[0] === "run" && s.args[1] === "--rm"); } @@ -259,6 +262,12 @@ function fakeDbSession() { const tempRoot = useLegacyTempWorkdir("supabase-db-start-int-"); +// The baseline cache (`db-bootstrap/main-db-baseline.ts`) is ON by default and reads +// `process.env` directly, so every fresh-volume bring-up below would otherwise restore or publish +// a real tar under the developer's own `~/.supabase`. This suite's subject is the bring-up itself; +// the cache-focused scenarios opt back in per-test with `withLegacyShadowCacheEnabled`. +useLegacyShadowCacheDisabled(); + function writeConfig(workdir: string, contents: string) { mkdirSync(join(workdir, "supabase"), { recursive: true }); writeFileSync(join(workdir, "supabase", "config.toml"), contents); @@ -1803,4 +1812,111 @@ describe("legacy db start", () => { expect(out.stderrText).toContain("Starting database from backup...\n"); }); }); + + // The baseline cache (`shared/db-bootstrap/main-db-baseline.ts`) is ON in production; the + // suite-wide `useLegacyShadowCacheDisabled` above turns it off everywhere else so the other + // scenarios assert the plain bring-up. These turn it back on — under a per-test + // `SUPABASE_HOME`, so the ~90MB-in-production tar never lands in the developer's real + // `~/.supabase` — and drive `db start` for real against it. The cache's own mechanics (key + // derivation, atomic publish, retention, degradation) are covered at their own level in + // `shared/db-bootstrap/shadow-cache.integration.test.ts`. + describe("cached database baseline", () => { + const SHADOW_HOME_DIR = "_supabase_home"; + + /** Re-enables the cache the suite-wide gate turned off, rooted at this test's own home. */ + const withCacheEnabled = (body: Effect.Effect): Effect.Effect => + withLegacyShadowCacheEnabled(join(tempRoot.current, SHADOW_HOME_DIR), body); + + /** + * Every published `shadow-baseline-.tar` under the per-test home, at the directory + * production itself derives — a relocated cache directory cannot pass unnoticed here. + */ + const publishedBaselineTars = (): ReadonlyArray => { + const dir = legacyShadowBaselineCacheDir( + Effect.runSync(Path.Path.pipe(Effect.provide(Path.layer))), + { SUPABASE_HOME: join(tempRoot.current, SHADOW_HOME_DIR) }, + ); + return existsSync(dir) ? readdirSync(dir).filter((name) => name.endsWith(".tar")) : []; + }; + + /** + * The baseline restore's own `docker cp - :/var/lib/postgresql` (PGDATA's PARENT). + * Deliberately narrower than `cp -`: every container create also copies its secret files + * that way, targeting `:/`. + */ + const isBaselineRestoreCp = (args: ReadonlyArray): boolean => + args[0] === "cp" && args[1] === "-" && (args[2] ?? "").endsWith(":/var/lib/postgresql"); + + /** One full fresh-volume `db start`. */ + const runFreshStart = Effect.fnUntraced(function* (fromBackup?: string) { + const s = setup({ route: freshVolumeRoute(defaultRoute()) }); + yield* legacyDbStart(flags(fromBackup)).pipe(Effect.provide(s.layer)); + return s; + }); + + it.live( + "reuses the published baseline on the next fresh-volume start instead of re-running it", + () => + withCacheEnabled( + Effect.gen(function* () { + const cold = yield* runFreshStart(); + // Cold: the platform baseline really ran, and the snapshot was taken at its seam. + expect(cold.out.stderrText).toContain("Initialising schema..."); + expect(dbSetupJobCalls(cold.child.spawned)).toHaveLength(3); + expect(cold.child.spawned.filter((s) => s.args[0] === "stop")).toHaveLength(1); + expect(publishedBaselineTars()).toHaveLength(1); + + const warm = yield* runFreshStart(); + // Warm: the restored cluster already carries the baseline, so neither its progress + // text nor its one-shot migrate jobs happen — and nothing is re-snapshotted. + expect(warm.out.stderrText).toContain("Restoring cached baseline...\n"); + expect(warm.out.stderrText).not.toContain("Initialising schema..."); + expect(warm.out.stderrText).not.toContain("Seeding globals from roles.sql..."); + expect(dbSetupJobCalls(warm.child.spawned)).toHaveLength(0); + expect(warm.child.spawned.filter((s) => s.args[0] === "stop")).toHaveLength(0); + + // The tar is unpacked into the created-but-unstarted container: the entrypoint has + // to find an initialized PGDATA, so the copy MUST precede `docker start`. + const restoreCp = warm.child.spawned.findIndex((s) => isBaselineRestoreCp(s.args)); + const start = warm.child.spawned.findIndex((s) => s.args[0] === "start"); + expect(restoreCp).toBeGreaterThanOrEqual(0); + expect(start).toBeGreaterThan(restoreCp); + + // `initCurrentBranch` is the last line of `StartDatabase` on every path. + expect(readFileSync(currentBranchPath(tempRoot.current), "utf8")).toBe("main"); + expect(publishedBaselineTars()).toHaveLength(1); + }), + ), + 20_000, + ); + + it.live( + "--from-backup never restores a cached baseline, even when one is published for this project", + () => + withCacheEnabled( + Effect.gen(function* () { + // `--from-backup` and a plain fresh start hash to the SAME key (the restore + // entrypoint is not a key input), so this leaves a tar the backup run below would + // warm-hit if it consulted the cache at all. + yield* runFreshStart(); + expect(publishedBaselineTars()).toHaveLength(1); + + const backup = yield* runFreshStart("/abs/host/backup.sql"); + + expect(backup.out.stderrText).not.toContain("Restoring cached baseline..."); + // The only archive this path delivers is its own backup bind. + expect(bindsFromCreateArgs(createArgs(backup.child.spawned) ?? [])).toContain( + "/abs/host/backup.sql:/etc/backup.sql:ro", + ); + expect(backup.child.spawned.some((s) => isBaselineRestoreCp(s.args))).toBe(false); + // The whole `SetupLocalDatabase`-equivalent pipeline is skipped, so there is no + // platform baseline to publish either — the pool is left exactly as it was. + expect(dbSetupJobCalls(backup.child.spawned)).toHaveLength(0); + expect(backup.child.spawned.filter((s) => s.args[0] === "stop")).toHaveLength(0); + expect(publishedBaselineTars()).toHaveLength(1); + }), + ), + 20_000, + ); + }); }); diff --git a/apps/cli/src/legacy/commands/start/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/start/SIDE_EFFECTS.md index 4af5e52658..8d6fe71083 100644 --- a/apps/cli/src/legacy/commands/start/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/start/SIDE_EFFECTS.md @@ -21,7 +21,7 @@ One piece of the old Go CLI's `start` remains explicitly **out of scope**: an "update available" hint. Omitted entirely — this port has zero Management API dependency for `start`, by design. -### Fresh-volume DB setup (`legacyStartSetupLocalDatabase`) +### Fresh-volume DB setup (`legacyRunFreshDbSetup`) Runs the initial schema/migrations/seed pipeline. Gated on `isFreshVolume` (`legacyVolumeExists` on the Postgres volume, checked BEFORE the @@ -41,6 +41,32 @@ enabled` is false, in which case `db.migrations.schema_paths` files are applied `migrations/*.sql`; seed still runs either way. A failure at any step rolls back the whole `start` run (same as any other bring-up failure). +On PG15+, this pipeline is fronted by the **baseline PGDATA cache** +(`db-bootstrap/main-db-baseline.ts`), which shares its tar pool with the throwaway shadow +database `db diff`/`db pull`/`migration squash` provision. Before Postgres's container is +created, the run computes a settings key and looks for +`$SUPABASE_HOME/cache/shadow-baseline/shadow-baseline-.tar` (default +`~/.supabase/cache/...`, shared across worktrees): + +- **Warm** — the tar is unpacked into the created-but-not-yet-started container via + `docker cp - :/var/lib/postgresql`, so the entrypoint finds an initialized PGDATA and + skips `initdb`. `Restoring cached baseline...` prints to stderr INSTEAD of + `Initialising schema...`/`Seeding globals from roles.sql...`, and the three PG15+ one-shot + migrate jobs do NOT run. Migrations and seeding are unchanged. If the restored cluster does + not become healthy, `Warning: cached database baseline unusable (...); recreating.` prints, + the container and its volume are force-removed, and the whole bring-up re-runs cold. +- **Cold** — everything runs exactly as before, and the baseline is then published: after the + schema/vault/`roles.sql` steps and BEFORE `MigrateAndSeed`, the `db` container is + `docker stop`ped, its `/var/lib/postgresql/data` exported with `docker cp` to the tar + (atomic temp-then-rename, mode `0600`), `docker start`ed again, and waited on with a direct + connect probe. Publication is best-effort: any failure prints + `Warning: database baseline not cached: ` and the run continues uncached. A cluster + that does not come back up after the snapshot DOES fail the run. +- **Skipped entirely** on PG<=14, on OrioleDB clusters, when `supabase/roles.sql` is + unreadable, and when `SUPABASE_SHADOW_CACHE` is `false`/`0`. + +Retention is LRU (newest 8) plus a 14-day mtime TTL, swept on every cold export and warm hit. + `legacyStartInitCurrentBranch` (writes `supabase/.branches/_current_branch` = `"main"` if absent) is NOT part of this fresh-volume-gated pipeline — it runs unconditionally on every `start`, immediately after this pipeline's gate closes (whether or not the pipeline itself @@ -93,11 +119,12 @@ command. ## Files Written -| Path | Format | When | -| ---------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `/supabase/.branches/_current_branch` | text | on every start, only if absent — writes `"main"` | -| `/supabase/.temp/start-secrets//{env,multiline-env}/` | varies | Edge Runtime's own JWT/service-role-key/secret env artifacts — see below | -| `/supabase/.temp/pgdelta/catalog-local-migrations--.json` | JSON | best-effort, on a fresh volume, after `MigrateAndSeed`, when pg-delta is enabled (`[experimental.pgdelta] enabled` or `SUPABASE_EXPERIMENTAL_PG_DELTA`) AND the legacy engine is selected (`SUPABASE_USE_PG_DELTA_NEXT=false`); the default next engine skips this warmup entirely; a failure only warns on stderr and never fails `start` | +| Path | Format | When | +| --------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/.branches/_current_branch` | text | on every start, only if absent — writes `"main"` | +| `/supabase/.temp/start-secrets//{env,multiline-env,main}/` | varies | Edge Runtime's own JWT/service-role-key/secret env artifacts and bootstrap template — see below | +| `/supabase/.temp/pgdelta/catalog-local-migrations--.json` | JSON | best-effort, on a fresh volume, after `MigrateAndSeed`, when pg-delta is enabled (`[experimental.pgdelta] enabled` or `SUPABASE_EXPERIMENTAL_PG_DELTA`); a failure only warns on stderr and never fails `start` | +| `$SUPABASE_HOME/cache/shadow-baseline/shadow-baseline-.tar` (default `~/.supabase/…`) | tar | best-effort, on a fresh PG15+ volume with the baseline cache on, between the platform baseline and `MigrateAndSeed` — mode `0600`, published by an atomic rename, shared with the shadow-database cache; a failure only warns on stderr and never fails `start` | Kong's `custom_nginx.template`, Vector's `vector.yaml`, and Postgres's own bootstrap script (`postgresql.conf`-equivalent setup) are all rendered in memory and injected @@ -164,6 +191,8 @@ not implemented. | `SUPABASE_*` (any dotted config field) | Generic Viper-style `AutomaticEnv` override of any `config.toml` field (e.g. `SUPABASE_AUTH_ENABLED`, `SUPABASE_API_PORT`) | no | | `SUPABASE_EXPERIMENTAL` (or `--experimental`) | Fresh volume + no pg-delta: applies `db.migrations.schema_paths` files instead of `migrations/*.sql` (see "Fresh-volume DB setup" above) | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | Enables the post-`MigrateAndSeed` migrations-catalog cache warmup when `[experimental.pgdelta].enabled` is unset | no | +| `SUPABASE_SHADOW_CACHE` | `false`/`0` opts out of the fresh-volume baseline PGDATA cache (default ON) — see "Fresh-volume DB setup" above | no | +| `SUPABASE_HOME` | Roots the baseline PGDATA cache directory (`$SUPABASE_HOME/cache/shadow-baseline`, default `~/.supabase`) | no | | `SUPABASE_USE_PG_DELTA_NEXT` | Selects the pg-delta implementation; `false` selects the legacy edge-runtime engine and thereby restores the migrations-catalog cache warmup (unset/unrecognized defaults to the next engine, which skips it) | no | | `SUPABASE_INTERNAL_IMAGE_REGISTRY` | Overrides the image registry used to resolve every service's image | no | | `SUPABASE_PROJECT_ID` | Overrides the resolved local project id (env → config.toml → workdir basename) | no | diff --git a/apps/cli/src/legacy/commands/start/start.command.ts b/apps/cli/src/legacy/commands/start/start.command.ts index 5e54778ede..1a9b31448d 100644 --- a/apps/cli/src/legacy/commands/start/start.command.ts +++ b/apps/cli/src/legacy/commands/start/start.command.ts @@ -53,7 +53,7 @@ export type LegacyStartFlags = CliCommand.Command.Config.Infer; // the same `FetchHttpClient`-backed layer `db reset`/`seed buckets` use, needed for the // health-check probes (`legacyWaitForHealthyServices`) and `legacySeedBucketsRun`. // `legacyDockerRunLayer`/`legacyDbConnectionLayer` ARE listed here — the fresh-volume -// `SetupLocalDatabase` equivalent (`start.handler.ts`'s `legacyStartSetupLocalDatabase` +// `SetupLocalDatabase` equivalent (`start.handler.ts`'s `legacyRunFreshDbSetup` // call) needs both: the PG15+ one-shot migrate jobs run through `LegacyDockerRun`, and // the schema/globals/API-privileges SQL runs over a direct `LegacyDbConnection` session. // `legacyEdgeRuntimeScriptLayer`/`legacyPgDeltaSslProbeLayer` back that same fresh-volume diff --git a/apps/cli/src/legacy/commands/start/start.handler.ts b/apps/cli/src/legacy/commands/start/start.handler.ts index 7d92e89285..f74f1d30c3 100644 --- a/apps/cli/src/legacy/commands/start/start.handler.ts +++ b/apps/cli/src/legacy/commands/start/start.handler.ts @@ -599,11 +599,11 @@ export const legacyStart = Effect.fn("legacy.start")(function* (flags: LegacySta // `legacyCheckDbToml` resolves `[db.vault]`/`[db.seed]`/`db.migrations.enabled`/the effective // `api.auto_expose_new_tables` tri-state — this must run unconditionally, // before any Docker work. The port only ran this inside - // `legacyStartSetupLocalDatabase`, which is itself gated on the DB container's + // `legacyRunFreshDbSetup`, which is itself gated on the DB container's // healthcheck passing AND a fresh volume (the `NoBackupVolume` gate) — so a malformed // `SUPABASE_DB_SEED_ENABLED`/an undecryptable `[db.vault]` secret went completely unvalidated // whenever `start` reused an existing volume. The resolved Webhooks flag is also retained so - // existing volumes can converge `pg_net`; `legacyStartSetupLocalDatabase`'s own internal call + // existing volumes can converge `pg_net`; `legacyRunFreshDbSetup`'s own internal call // (an already-accepted duplicate config-load pass, matching `db start`'s own independent // resolution — see `../../shared/db-bootstrap/db-setup.ts`'s header) still resolves fresh-setup // values for its own use when it runs. @@ -1584,7 +1584,14 @@ export const legacyStart = Effect.fn("legacy.start")(function* (flags: LegacySta // `db start` (see `legacyStartDatabase`'s header for why this is caller-supplied). resolvePostgresImage: Effect.succeed(resolveImage(postgresImage)), dbHealthTimeoutSeconds, - webhooksEnabled: dbTomlValues.webhooksEnabled, + // This run's own already-validated `[db]` values: `webhooksEnabled` converges an + // existing volume's pg_net, and all three are what the baseline cache keys on — see + // `LegacyMainDbBaselineTomlInputs` (`db-bootstrap/main-db-baseline.ts`). + toml: { + webhooksEnabled: dbTomlValues.webhooksEnabled, + apiAutoExposeNewTables: dbTomlValues.baseline.apiAutoExposeNewTables, + vault: dbTomlValues.vault, + }, setup: { majorVersion, experimental, diff --git a/apps/cli/src/legacy/commands/start/start.integration.test.ts b/apps/cli/src/legacy/commands/start/start.integration.test.ts index 4ea6ca0b73..20c0eadb6d 100644 --- a/apps/cli/src/legacy/commands/start/start.integration.test.ts +++ b/apps/cli/src/legacy/commands/start/start.integration.test.ts @@ -21,6 +21,7 @@ import { import { mockLegacyCliConfig, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, legacySequentialExecBatch, } from "../../../../tests/helpers/legacy-mocks.ts"; @@ -88,6 +89,12 @@ vi.mock("../../shared/legacy-local-config-values.ts", async () => { const tempRoot = useLegacyTempWorkdir("supabase-start-int-"); +// The baseline cache (`db-bootstrap/main-db-baseline.ts`) is ON by default and reads +// `process.env` directly, so every fresh-volume bring-up below would otherwise restore or publish +// a real tar under the developer's own `~/.supabase`. This suite's subject is the bring-up itself; +// the cache-focused scenarios opt back in per-test with `withLegacyShadowCacheEnabled`. +useLegacyShadowCacheDisabled(); + function flags(overrides: Partial = {}): LegacyStartFlags { return { exclude: overrides.exclude ?? [], @@ -367,7 +374,7 @@ function mockStorageBucketHttpClient() { /** * A fake `LegacyDbSession` recording every `exec`/`query` call — the fresh-volume - * `SetupLocalDatabase`-equivalent path (`legacyStartSetupLocalDatabase`) needs an + * `SetupLocalDatabase`-equivalent path (`legacyRunFreshDbSetup`) needs an * open session for PG<=14's schema SQL / `ApplyApiPrivileges`; PG15+ (this suite's * default) never calls `exec`/`query` at all (its schema init is three one-shot * `LegacyDockerRun` jobs instead — see `db-setup.ts`'s header), so this mostly just @@ -463,7 +470,7 @@ function setup(opts: SetupOpts = {}) { opts.httpClientLayer ?? alwaysReadyHttpClientLayer, // Only ever exercised by a fresh-volume scenario (`volume inspect` exiting // non-zero) — every other scenario's default "volume already exists" route - // never reaches `legacyStartSetupLocalDatabase`/`legacySeedBucketsRun`, but + // never reaches `legacyRunFreshDbSetup`/`legacySeedBucketsRun`, but // both are still part of `legacyStart`'s aggregate Effect type, so every // scenario needs these satisfied regardless of whether it exercises them. Layer.succeed(LegacyDbConnection, { connect: () => Effect.succeed(dbSession.session) }), @@ -2388,7 +2395,7 @@ content_path = "./supabase/templates/custom_notice.html" }); describe("fresh volume: DB setup + bucket seeding", () => { - /** The three PG15+ one-shot migrate jobs (`legacyStartSetupLocalDatabase`'s `LegacyDockerRun` calls) — a plain `docker run --rm ...`, distinct from Edge Runtime's own create/cp/start bring-up. */ + /** The three PG15+ one-shot migrate jobs (`legacyRunFreshDbSetup`'s `LegacyDockerRun` calls) — a plain `docker run --rm ...`, never `-d`, distinct from Edge Runtime's own detached `docker run -d`. */ function dbSetupJobCalls(spawned: ReadonlyArray): ReadonlyArray { return spawned.filter((s) => s.args[0] === "run" && s.args[1] === "--rm"); } @@ -2535,7 +2542,7 @@ content_path = "./supabase/templates/custom_notice.html" it.live( "fails on an undecryptable [db.vault] secret even on a non-fresh volume, matching Go's Config.Load", () => { - // `legacyCheckDbToml`'s own internal call inside `legacyStartSetupLocalDatabase` only + // `legacyCheckDbToml`'s own internal call inside `legacyRunFreshDbSetup` only // runs on a fresh volume — an undecryptable `[db.vault]` // secret (a DB-specific field `@supabase/config`'s own schema never decrypts, only // `legacyCheckDbToml`'s pipeline does) must still fail eagerly, before any Docker work, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/baseline-state.ts b/apps/cli/src/legacy/shared/db-bootstrap/baseline-state.ts new file mode 100644 index 0000000000..fb9cb3b0cf --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/baseline-state.ts @@ -0,0 +1,149 @@ +/** + * What a provisioning step hands the pipeline that follows it about the CLUSTER'S CONTENTS — the + * one seam the baseline cache (`shadow-cache.ts`) needs, and nothing else uses — plus the ONE + * session-shaped consequence every consumer draws from it ({@link legacyOpenBaselineSession}). + * + * Lives in its own module because both consumers sit on opposite sides of an import edge: the + * throwaway shadow (`shadow-database.ts`, which imports `db-setup.ts`) and the long-running local + * `db` container (`db-setup.ts`'s own `legacyRunFreshDbSetup`, via `main-db-baseline.ts`). Keeping + * the shape here lets `db-setup.ts` accept it without importing `shadow-database.ts` back. + * + * Deliberately a value the provisioning step OWNS and returns (alongside the container id), not an + * `afterBaseline` callback threaded down through the setup pipeline: the cache is the only party + * that knows whether a cluster already carries a baseline and what to do once a fresh one exists, + * so both answers travel together with the container the cache handed over. + */ + +import { Effect } from "effect"; + +import type { Output } from "../../../shared/output/output.service.ts"; +import type { LegacyDbConnection, LegacyDbSession } from "../legacy-db-connection.service.ts"; + +/** + * A cluster restored from this key's PGDATA snapshot: it already carries the platform baseline + * (`legacySetupDatabase`'s init schema + API privileges + vault + `roles.sql`), so re-running it + * would be wasted work at best and a double-applied baseline at worst — and there is nothing left + * to snapshot. + */ +export interface LegacyWarmBaselineState { + readonly _tag: "warm"; +} + +/** + * A cache-enabled COLD provision: the baseline still has to run, and the cluster it produces is + * the one the cache publishes. The ONLY state whose {@link snapshotBaseline} really stops the + * container, which is what the setup compositions key their SESSION structure on — see + * {@link legacyOpenBaselineSession}. + */ +export interface LegacyColdBaselineState { + readonly _tag: "cold"; + /** + * Runs immediately after the FRESHLY provisioned baseline and strictly before anything else + * touches the cluster (the shadow's template database/user migrations, the local `db` + * container's `MigrateAndSeed`) — the only point at which `postgres` holds the pristine baseline + * and nothing else. + * + * Takes NO session, and {@link legacyOpenBaselineSession} guarantees none is open against the + * cluster while it runs: the snapshot is a disk-level PGDATA export that has to stop the + * container. + * + * A cache that cannot SNAPSHOT degrades silently (warn + uncached run) — but the error channel + * is not `never`, for the one failure that is the run's problem rather than the cache's: a + * cluster that does not come back up after the export. Reporting success there would send the + * caller's next connect to a dead (or worse, someone else's) Postgres on the published port — + * see `legacyExportBaselineSnapshot`'s doc comment (`shadow-cache.ts`). + * + * `SnapshotError` is each cluster's own error vocabulary for that one failure + * (`LegacyShadowDbError` for the shadow, `LegacyDbSetupError` for the local `db` container). + */ + readonly snapshotBaseline: Effect.Effect; + /** + * The `supabase/roles.sql` bytes the cache ALREADY read when it hashed this run's key + * (`legacyResolveShadowCacheKeyInputs`, `shadow-cache.ts`; `""` when the file is absent). + * Threaded into `legacySetupDatabase` so the seed executes exactly the bytes the key describes: + * re-reading the file at seed time would let an edit landing in between publish a snapshot whose + * contents disagree with the key it is published under. + */ + readonly rolesSql: string; +} + +/** + * A run the cache never applies to (PG<=14, OrioleDB, `SUPABASE_SHADOW_CACHE=false`, an unreadable + * `roles.sql`, `db start --from-backup`, `--no-cache`): provision the baseline, snapshot nothing. + * Deliberately distinct from {@link LegacyColdBaselineState} — splitting sessions when no snapshot + * will run would be a gratuitous behavior change, since a reconnect picks up role-level defaults + * `roles.sql` may have just installed (e.g. `ALTER ROLE postgres SET statement_timeout`), which + * Go's single-connection flow never exposed to migrations (review: Codex on #6184). + */ +export interface LegacyUncachedBaselineState { + readonly _tag: "uncached"; +} + +/** + * The three states a provisioning step can hand over, as a discriminated union: a cluster either + * already carries the baseline (warm), or still needs one that will be snapshotted (cold), or + * still needs one nobody will snapshot (uncached). Modelling it this way is what makes + * "already has a baseline AND owes a snapshot" unrepresentable. + */ +export type LegacyClusterBaselineState = + | LegacyWarmBaselineState + | LegacyColdBaselineState + | LegacyUncachedBaselineState; + +/** + * The baseline state every uncached caller passes. `never` in the error position makes this + * assignable to any cluster's own {@link LegacyClusterBaselineState}. + */ +export const LEGACY_BASELINE_UNCACHED: LegacyClusterBaselineState = { _tag: "uncached" }; + +/** + * Opens the session everything AFTER the baseline runs on, running the baseline itself when this + * state says it still has to — the identical three-way branch both setup compositions + * (`db-setup.ts`'s `legacyRunFreshDbSetup` for the long-running local `db` container, + * `shadow-database.ts`'s `legacyOpenShadowBaselineSession` for the throwaway shadow) used to spell + * out separately: + * + * - **warm**: no baseline, no snapshot — just connect. The restored cluster already carries the + * platform baseline, so neither `Initialising schema...` nor `Seeding globals from roles.sql...` + * prints and the PG15+ one-shot migrate jobs never run. + * - **cold**: run the baseline in its OWN scope so that session is CLOSED before + * {@link LegacyColdBaselineState.snapshotBaseline} stops the container (a disk-level export + * severs any live backend, and an open session holds SIGTERM's smart shutdown open until the + * grace period expires), then connect again for whatever follows. + * - **uncached**: Go's single-connection flow, verbatim — baseline and everything after it on one + * session. + * + * The returned session's lifetime is the CALLER's enclosing `Scope.Scope` (Go's + * `defer conn.Close(...)`), which is why `connect`'s own requirements leak through instead of + * being wrapped in `Effect.scoped` here. + */ +export const legacyOpenBaselineSession = ( + connect: Effect.Effect, + runBaseline: (session: LegacyDbSession) => Effect.Effect, + baseline: LegacyClusterBaselineState, +): Effect.Effect => + Effect.gen(function* () { + if (baseline._tag === "cold") { + yield* Effect.scoped( + Effect.gen(function* () { + yield* runBaseline(yield* connect); + }), + ); + yield* baseline.snapshotBaseline; + } + const session = yield* connect; + if (baseline._tag === "uncached") { + yield* runBaseline(session); + } + return session; + }); + +/** + * The `roles.sql` bytes the cache already read for this run's key, when it read any — see + * {@link LegacyColdBaselineState.rolesSql}. `undefined` on every other state: a warm cluster + * never runs the seed at all, and an uncached run has no peek to inherit from, so it reads the + * file itself exactly as before. + */ +export const legacyBaselineRolesSql = ( + baseline: LegacyClusterBaselineState, +): string | undefined => (baseline._tag === "cold" ? baseline.rolesSql : undefined); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/bootstrap-config.ts b/apps/cli/src/legacy/shared/db-bootstrap/bootstrap-config.ts index c34ea6b783..934bd06d4f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/bootstrap-config.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/bootstrap-config.ts @@ -222,7 +222,7 @@ export const legacyResolveDbBootstrapConfig = ( ); // Both the long-running Realtime container (`supabase start` only) AND the PG15+ one-shot - // Realtime setup job (both callers, via `legacyStartSetupLocalDatabase`) must see the SAME + // Realtime setup job (both callers, via `legacyRunFreshDbSetup`) must see the SAME // already-overridden values (Go's single `utils.Config.Realtime` source of truth, // `internal/start/start.go:922,928`, `internal/db/start/start.go:283,290`). const realtimeIpVersion = yield* wrapConfigOverride( diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index af334db801..85ba9fa136 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -4,7 +4,7 @@ * the `db` container's healthcheck passes on a FRESH volume (Go's `NoBackupVolume` * gate, `start.go:184` — the caller decides whether to invoke this at all; see * `legacyVolumeExists` in `./container-lifecycle.ts`). The single exported - * entry point, {@link legacyStartSetupLocalDatabase}, runs the exact Go call chain + * entry point, {@link legacyRunFreshDbSetup}, runs the exact Go call chain * in order: * * 1. **`initSchema`** (`start.go:243-266`) — prints `Initialising schema...`, then @@ -49,14 +49,16 @@ * it's actually present. A missing file is tolerated (Go's `errors.Is(err, * os.ErrNotExist)` check, reproduced here as an existence check ahead of the read * rather than a caught not-found error — see the call site's own comment for why); - * any other read/exec error propagates. + * any other read/exec error propagates. A run whose baseline cache already read + * the file for its key execs THOSE bytes instead — see + * {@link LegacySetupDatabaseInput.rolesSql}. * 6. **`apply.MigrateAndSeed`** (`start.go:368`, via the already-ported * `legacyMigrateAndSeed`) with the caller-supplied {@link - * LegacyStartSetupLocalDatabaseInput.version} — `""` (every pending migration) for + * LegacyRunFreshDbSetupInput.version} — `""` (every pending migration) for * `db start`'s own call, matching `SetupLocalDatabase`'s call in the `start` * context; `db reset`'s PG15 recreate (the function's OTHER real Go caller, * `resetDatabase15`, `reset.go:169`) passes its own resolved reset version instead. - * {@link LegacyStartSetupLocalDatabaseInput.seedFlags} applies `db reset`'s + * {@link LegacyRunFreshDbSetupInput.seedFlags} applies `db reset`'s * `--no-seed`/`--sql-paths` overrides on top of the loaded `[db.seed]` config first * (a no-op for `db start`, which has neither flag) — see * {@link legacyResolveResetSeedConfig}. @@ -73,7 +75,7 @@ * Go's exact warning (`Warning: failed to cache migrations catalog: `, * `start.go:378`) to stderr and is otherwise swallowed, reusing the identical * best-effort catch/warn shape `legacy-db-push-core.ts` already established for - * its own call — this step never fails {@link legacyStartSetupLocalDatabase} or + * its own call — this step never fails {@link legacyRunFreshDbSetup} or * the caller's `start`/`db start`/`db reset` run. Requires * `LegacyEdgeRuntimeScript`/`LegacyPgDeltaSslProbe` in this function's own effect * environment (widened accordingly below), so `start.command.ts`, @@ -101,7 +103,7 @@ * called by `StartDatabase` (the caller of `SetupLocalDatabase`) UNCONDITIONALLY, * regardless of `NoBackupVolume` (`start.go:184-189`) — unlike everything above, * which only runs on a fresh volume. `start.handler.ts` calls it directly, outside - * the `isFreshVolume` gate that wraps {@link legacyStartSetupLocalDatabase}; `db + * the `isFreshVolume` gate that wraps {@link legacyRunFreshDbSetup}; `db * reset` never calls it at all (Go's own `resetDatabase`/`resetDatabase15` never * call `initCurrentBranch` either). * @@ -115,7 +117,7 @@ */ import type { ProjectConfig } from "@supabase/config"; -import { Data, Effect, type FileSystem, Option, type Path, Schedule } from "effect"; +import { Data, Effect, type FileSystem, Option, type Path, Schedule, type Scope } from "effect"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import type { LocalServiceVersionOverrides } from "../../../shared/services/services.shared.ts"; @@ -158,6 +160,12 @@ import { type LegacyVaultSecret, legacyUpsertVaultSecrets, } from "../legacy-vault.ts"; +import { + LEGACY_BASELINE_UNCACHED, + legacyBaselineRolesSql, + legacyOpenBaselineSession, + type LegacyClusterBaselineState, +} from "./baseline-state.ts"; import { legacyEnsureImagesCached, type LegacyImagePrepullError } from "./image-prepull.ts"; import { legacyResolvePinnedImage } from "./pinned-image.ts"; import { LEGACY_COMPOSE_PROJECT_LABEL } from "./container-lifecycle.ts"; @@ -247,7 +255,7 @@ function legacyDbSetupDockerReason( return "image_inspect"; } -/** Every failure {@link legacyStartSetupLocalDatabase} can produce. */ +/** Every failure the `SetupLocalDatabase`-equivalent half of {@link legacyRunFreshDbSetup} can produce. */ export type LegacyStartSetupLocalDatabaseError = | LegacyDbConfigLoadError | LegacyDbSetupError @@ -352,7 +360,7 @@ export const legacyResolveDbSetupPrelude = (setup: { * Input to {@link legacySetupDatabase} — Go's EXPORTED `SetupDatabase(ctx, conn, host, w, * fsys)` (`start.go:383-399`): `initSchema -> ApplyApiPrivileges -> vault upsert -> * SeedGlobals(roles.sql)`, deliberately WITHOUT `apply.MigrateAndSeed` (that extra step is - * what makes {@link LegacyStartSetupLocalDatabaseInput}/{@link legacyStartSetupLocalDatabase} + * what makes {@link LegacyRunFreshDbSetupInput}/{@link legacyRunFreshDbSetup} * bigger — see that interface's own doc comment). Extracted as its own exported shape * (CLI-1956) so shadow-database provisioning (`shadow-database.ts`) can reach the exact same * platform-baseline pipeline the real local `db` container's fresh-volume setup does, without @@ -467,6 +475,15 @@ export interface LegacySetupDatabaseInput { readonly apiAutoExposeNewTables: Option.Option; /** `toml.vault` — Go's `utils.Config.Db.Vault`, threaded straight into {@link legacyUpsertVaultSecrets}. */ readonly vault: ReadonlyArray; + /** + * The `supabase/roles.sql` bytes the baseline cache ALREADY read while hashing this run's key + * (`legacyBaselineRolesSql`, `baseline-state.ts`) — `""` when the file was absent then. When + * set, the custom-roles seed execs exactly these bytes rather than re-reading the file, so an + * edit landing between the key computation and this step can never make the published snapshot + * disagree with the key it is published under. Absent on every run that never peeked (cache + * off, PG<=14, OrioleDB), which reads the file exactly as before. + */ + readonly rolesSql?: string | undefined; } /** Controls the extension side effects of {@link legacySetupDatabase}. */ @@ -474,19 +491,16 @@ export interface LegacySetupDatabaseOptions { readonly webhooks?: "config" | "enabled" | "disabled"; } -/** Input to {@link legacyStartSetupLocalDatabase}. */ -export interface LegacyStartSetupLocalDatabaseInput extends Omit< - LegacySetupDatabaseInput, - "apiAutoExposeNewTables" | "vault" | "webhooksEnabled" -> { - /** - * `--experimental`/`SUPABASE_EXPERIMENTAL`, resolved by the caller (Go's - * `viper.GetBool("EXPERIMENTAL")`) — threaded straight into - * {@link legacyMigrateAndSeed}'s own `experimental` gate (`internal/migration/apply/ - * apply.go:19`); `legacySetupDatabase`/Go's own `SetupDatabase` have no use for it — - * only this function's own trailing `MigrateAndSeed` call does. - */ - readonly experimental: boolean; +/** Input to {@link legacyRunFreshDbSetup} — the whole `SetupLocalDatabase`-equivalent pipeline. */ +export interface LegacyRunFreshDbSetupInput { + readonly fs: FileSystem.FileSystem; + readonly path: Path.Path; + /** The Supabase project root (parent of `supabase/`). */ + readonly workdir: string; + readonly projectId: string; + readonly networkId: string; + readonly hostname: string; + readonly dbPort: number; /** * The migration version to reapply (Go's `apply.MigrateAndSeed(ctx, version, ...)`). * `db start`'s own caller always passes `""` (Go's `SetupLocalDatabase(ctx, "", ...)`, @@ -504,6 +518,7 @@ export interface LegacyStartSetupLocalDatabaseInput extends Omit< * unchanged. */ readonly seedFlags: { readonly noSeed: boolean; readonly sqlPaths: ReadonlyArray }; + readonly setup: LegacyFreshDbSetupInput; } /** @@ -534,12 +549,67 @@ const errMessage = (e: unknown): string => ? e.message : String(e); +/** + * Memoizes `effect`'s first SUCCESS; failures are never cached, so a retry re-runs the real + * effect. Deliberately not `Effect.cached` (which returns `Effect>` and needs an + * effectful construction site — every construction site here is a plain function) and not + * concurrency-guarded: the consumers of the one field this wraps (`jwks`) evaluate sequentially + * on the same fiber. + * + * Lives here rather than next to either consumer because this module is their common ancestor: + * `shadow-database.ts`'s `legacyShadowRunInputFromLocalContainerInputs` and + * `local-container-inputs.ts`'s own `setup.jwks` both wrap with it, and both already import from + * here. With the baseline cache enabled that effect is evaluated TWICE on a cold run — once for + * the cache key, once by {@link legacyResolveDbSetupPrelude} for the baseline itself — and + * third-party JWKS discovery can be a real network request. Memoizing the first success keeps the + * run to one request AND guarantees the published snapshot carries the exact value its key was + * computed from, even if the issuer rotates mid-run (review: Codex on #6184). + */ +export function legacyMemoizeSuccess(effect: Effect.Effect): Effect.Effect { + let succeeded: Effect.Effect | undefined; + return Effect.suspend( + () => + succeeded ?? + effect.pipe( + Effect.tap((value) => + Effect.sync(() => { + succeeded = Effect.succeed(value); + }), + ), + ), + ); +} + +/** + * The scoped scratch directory every in-memory SQL exec in this module writes through — see + * {@link legacyExecSqlConstant} for why an on-disk file is needed at all. Removed with the + * enclosing scope. + */ +const legacySetupTempDir = ( + fs: FileSystem.FileSystem, + prefix: string, +): Effect.Effect => + fs.makeTempDirectoryScoped({ prefix }).pipe( + Effect.mapError( + (error) => + new LegacyDbSetupError({ + message: `failed to create temp directory: ${errMessage(error)}`, + reason: "filesystem", + }), + ), + ); + /** * Writes `sql` to `/` and execs it via `legacyExecSqlFile` * (Go's `migration.NewMigrationFromReader(strings.NewReader(sql))` + * `file.ExecBatch(ctx, conn)` on an in-memory string — there is no on-disk file in * Go at all; this port needs one only because `legacyExecSqlFile` reads from the * filesystem like every other `execMigrationBatch` caller). + * + * `displayPath`, when given, is the path a failure reports instead of the scratch file — used by + * the one caller whose SQL is not a constant at all but a real project file already read into + * memory (`supabase/roles.sql`, see {@link LegacySetupDatabaseInput.rolesSql}), so its errors + * still name the file the user actually wrote. */ const legacyExecSqlConstant = Effect.fnUntraced(function* ( session: LegacyDbSession, @@ -548,6 +618,7 @@ const legacyExecSqlConstant = Effect.fnUntraced(function* ( tmpDir: string, filename: string, sql: string, + displayPath?: string, ) { const filePath = path.join(tmpDir, filename); yield* fs.writeFileString(filePath, sql).pipe( @@ -565,6 +636,7 @@ const legacyExecSqlConstant = Effect.fnUntraced(function* ( path, filePath, (message) => new LegacyDbSetupError({ message, reason: "database" }), + displayPath, ); }); @@ -909,7 +981,7 @@ const legacyStartInitSchema = Effect.fnUntraced(function* ( * regardless of PG major version (unlike `initSchema`, this always execs SQL over * `session` directly — it is never part of the PG15+ one-shot Docker jobs). Exported * (and taking `session`/`fs`/`path` directly, not the whole - * {@link LegacyStartSetupLocalDatabaseInput}) because Go's `ApplyApiPrivileges` is + * {@link LegacySetupDatabaseInput}) because Go's `ApplyApiPrivileges` is * the SAME exported function `db reset`'s PG14 `initDatabase` calls * (`reset.go:176-186`), after its own `InitSchema14` call and with none of * `SetupDatabase`'s other steps (vault/roles.sql/MigrateAndSeed) — see @@ -982,7 +1054,7 @@ export const legacyRemoveDatabaseWebhooks = Effect.fnUntraced(function* ( * TS constant for this path — `legacy/commands/db/branch/*` are Management-API * cloud-branch commands, unrelated to this local file — so it's inlined here, * the only current consumer (per "Hoist Before You Duplicate"). Exported (rather - * than folded into {@link legacyStartSetupLocalDatabase}) because Go calls it + * than folded into {@link legacyRunFreshDbSetup}) because Go calls it * unconditionally, not just on a fresh volume — see this module's header. */ export const legacyStartInitCurrentBranch = Effect.fnUntraced(function* ( @@ -1030,7 +1102,7 @@ export const legacyStartInitCurrentBranch = Effect.fnUntraced(function* ( /** * Runs Go's EXPORTED `SetupDatabase(ctx, conn, host, w, fsys)` (`start.go:383-399`) — * see {@link LegacySetupDatabaseInput}'s own doc comment for exactly what's in and out of - * scope. Extracted out of {@link legacyStartSetupLocalDatabase} (CLI-1956) so shadow-database + * scope. Extracted out of {@link legacyRunFreshDbSetup} (CLI-1956) so shadow-database * provisioning can reuse this exact sequence without also reaching `apply.MigrateAndSeed`. */ export const legacySetupDatabase = ( @@ -1054,17 +1126,7 @@ export const legacySetupDatabase = ( // initSchema -> user/baseline extension activation -> ApplyApiPrivileges. yield* Effect.scoped( Effect.gen(function* () { - const tmpDir = yield* fs - .makeTempDirectoryScoped({ prefix: "supabase-start-db-setup-" }) - .pipe( - Effect.mapError( - (error) => - new LegacyDbSetupError({ - message: `failed to create temp directory: ${errMessage(error)}`, - reason: "filesystem", - }), - ), - ); + const tmpDir = yield* legacySetupTempDir(fs, "supabase-start-db-setup-"); const requiresPg14WebhooksCleanup = input.majorVersion === 14; yield* legacyStartInitSchema(spawner, input, tmpDir); if (requiresPg14WebhooksCleanup) { @@ -1099,6 +1161,33 @@ export const legacySetupDatabase = ( const customRolesPath = path.join(workdir, "supabase", "roles.sql"); const output = yield* Output; yield* output.raw(`Seeding globals from ${path.basename(customRolesPath)}...\n`, "stderr"); + + // A run whose baseline cache already read this file for its key seeds THOSE bytes instead of + // re-reading it here — see {@link LegacySetupDatabaseInput.rolesSql}. `""` means the cache + // found no file, which is the same tolerated-missing-file case the read path below handles by + // skipping the exec; anything else is written back out to a scratch file, since + // `legacyExecSqlFile` (like every `execMigrationBatch` caller) reads from disk. Failures still + // name `supabase/roles.sql`, not the scratch copy. + const peekedRolesSql = input.rolesSql; + if (peekedRolesSql !== undefined) { + if (peekedRolesSql.length === 0) return; + yield* Effect.scoped( + Effect.gen(function* () { + const tmpDir = yield* legacySetupTempDir(fs, "supabase-start-db-roles-"); + yield* legacyExecSqlConstant( + session, + fs, + path, + tmpDir, + "roles.sql", + peekedRolesSql, + customRolesPath, + ); + }), + ); + return; + } + const rolesExist = yield* fs.exists(customRolesPath).pipe( Effect.mapError( (error) => @@ -1120,178 +1209,25 @@ export const legacySetupDatabase = ( }); /** - * Runs the full `SetupLocalDatabase`-equivalent sequence — see this module's - * header for the exact Go call chain and line-range citations. Call once, right - * after the `db` container's healthcheck passes on a fresh volume (Go's - * `NoBackupVolume` gate); the caller decides that gating, this function performs - * no health/readiness checks of its own. - */ -export const legacyStartSetupLocalDatabase = ( - spawner: Spawner, - input: LegacyStartSetupLocalDatabaseInput, -): Effect.Effect< - void, - // `LegacyDbConnectError` rides alongside the alias (as in `legacyRunFreshDbSetup`) - // because a batch that cannot check a connection out of the pool fails with the - // driver's connect error verbatim, suggestion included. - LegacyStartSetupLocalDatabaseError | LegacyDbConnectError, - | Output - | LegacyDockerRun - | RuntimeInfo - | LegacyEdgeRuntimeScript - | LegacyPgDeltaSslProbe - // `legacyTryCacheMigrationsCatalog`'s own pg-delta export call resolves - // `FileSystem.FileSystem`/`Path.Path` from the effect context itself (not from - // the `fs`/`path` values this function already threads through as plain data — - // see `legacy-pgdelta.ts`'s `legacyExportCatalogPgDelta`), so both must be - // ambient here too; every real caller already gets them from `BunServices.layer` - // at the CLI root runtime, same as `db push`'s own composition. - | FileSystem.FileSystem - | Path.Path -> => - Effect.gen(function* () { - const { session, fs, path, workdir } = input; - - // `warnOnUnresolvedEnv: false` — both `start.handler.ts` and `db/start/ - // start.handler.ts` already ran an earlier, same-invocation `legacyCheckDbToml` - // purely for its Go-parity validation side effect (their own callers discard the - // result) before ever reaching this fresh-volume setup, so that earlier call - // already printed Go's single `assertEnvLoaded` OrioleDB S3 WARN, if any. Without - // this, this module's own accepted duplicate config-load pass (see this module's - // header) would print the SAME warning a second time — a real, observable stderr - // divergence from Go's exactly-once `flags.LoadConfig`, unlike the harmless - // resolved-value duplication the header describes. - const toml = yield* legacyCheckDbToml(fs, path, workdir, undefined, { - warnOnUnresolvedEnv: false, - }); - - // SetupDatabase: initSchema -> ApplyApiPrivileges -> vault secrets -> custom-roles seed - // (start.go:383-399) — extracted to {@link legacySetupDatabase} so shadow-database - // provisioning (CLI-1956) can reuse this exact sequence without also reaching - // `apply.MigrateAndSeed` below. - yield* legacySetupDatabase(spawner, { - ...input, - webhooksEnabled: toml.webhooksEnabled, - apiAutoExposeNewTables: toml.baseline.apiAutoExposeNewTables, - vault: toml.vault, - }); - - // apply.MigrateAndSeed(ctx, version, conn, fsys) — `db start`'s own caller always - // passes `version: ""` (every pending migration, matching `SetupLocalDatabase`'s - // own call in the `start` context, `start.go:185,368`); `db reset`'s PG15 recreate - // passes its own resolved reset version instead (`resetDatabase15`, `reset.go:169`) - // — see `input.version`'s own doc comment. `experimental`/`pgDeltaEnabled`/ - // `schemaPaths` gate `legacyMigrateAndSeed`'s own declarative-schema-files branch - // (apply.go:19) — see its doc comment; `toml.pgDelta.enabled` and `toml.schemaPaths` - // are this module's own already-loaded config (the latter already resolved + - // `SUPABASE_DB_MIGRATIONS_SCHEMA_PATHS` env-overridden by `legacyCheckDbToml`, - // `legacy-db-config.toml-read.ts`), not re-read from the caller's raw, unresolved - // `ProjectConfig`. `input.seedFlags` applies `db reset`'s own `--no-seed`/ - // `--sql-paths` overrides on top of the loaded `[db.seed]` config — a no-op for - // `db start`, which has neither flag. - yield* legacyMigrateAndSeed(session, fs, path, workdir, input.version, { - migrationsEnabled: toml.migrationsEnabled, - seed: legacyResolveResetSeedConfig(toml.seed, input.seedFlags, path), - experimental: input.experimental, - pgDeltaEnabled: toml.pgDelta.enabled, - schemaPaths: toml.schemaPaths, - localDatabaseWebhooksEnabled: toml.webhooksEnabled, - }); - - const output = yield* Output; - - // pgcache.TryCacheMigrationsCatalog(ctx, pgconn.Config{Host: Config.Hostname, - // Port: Config.Db.Port, User: "postgres", Password: Config.Db.Password, Database: - // "postgres"}, "local", version, fsys, ...) (start.go:371-379): best-effort, run - // immediately after MigrateAndSeed above, for BOTH real Go callers of this shared - // function — `db start` (always `version: ""`) and `db reset`'s PG15 recreate - // (its own resolved reset `input.version`, usually also `""`). `cacheEnabled` - // reproduces Go's `ShouldCacheMigrationsCatalog()` gate exactly - // (`pgcache/cache.go:93-95`): `len(version) == 0` AND (`toml.pgDelta.enabled` OR - // `SUPABASE_EXPERIMENTAL_PG_DELTA`) — the same formula `legacy-db-push-core.ts` - // already uses for its own call. `input.dbUrl` is already the HOST-facing - // `postgresql://postgres:@:/postgres` address (see its - // own doc comment) — the exact same shape Go's `utils.ToPostgresURL(config)` builds - // from that literal `pgconn.Config` here, so it's reused directly as `targetUrl` - // rather than re-derived. `conn`'s fields are only ever read by - // `legacyCatalogPrefixFromConfig` on a non-local prefix fallback, unreachable here - // since `isLocal` is always `true`. - const cacheEnabled = - input.version.length === 0 && - (toml.pgDelta.enabled || - legacyParseBoolEnv(toml.envLookup("SUPABASE_EXPERIMENTAL_PG_DELTA"))); - const pgDeltaImplementation = legacyResolvePgDeltaImplementation( - legacyPgDeltaImplementationFlag( - process.env[LEGACY_PG_DELTA_NEXT_FLAG_NAME], - toml.projectEnv[LEGACY_PG_DELTA_NEXT_FLAG_NAME], - ), - ); - const pgDeltaCtx: LegacyPgDeltaContext = { - projectId: input.projectId, - cwd: workdir, - npmVersion: Option.getOrUndefined(toml.pgDelta.npmVersion), - denoVersion: toml.denoVersion, - projectEnv: toml.projectEnv, - }; - const hostDbUrl = new URL(input.dbUrl); - // Scope the `PGDELTA_NPM_REGISTRY`-from-project-`.env` apply to just this call: - // `legacyExportCatalogPgDelta` reads it off bare `process.env` - // (`legacyPgDeltaNpmRegistryOption`), same as `db push`/`db pull`/`db dump`/ - // `bootstrap`'s own calls into pg-delta — Go's `loadNestedEnv` already made it - // process-wide by this point (`config.go:788`), but this module otherwise threads - // every override through `projectEnvValues` explicitly rather than mutating - // `process.env`, so this one shared-code call needs the same opt-in helper those - // other commands use. `legacyApplyProjectEnv` registers a finalizer that reverts it. - yield* Effect.scoped( - Effect.gen(function* () { - yield* legacyApplyProjectEnv(input.projectEnvValues ?? {}); - yield* legacyTryCacheMigrationsCatalog(fs, path, pgDeltaCtx, { - // The catalog is a legacy-engine artifact with no in-process consumer. - enabled: cacheEnabled && pgDeltaImplementation === "legacy", - targetUrl: input.dbUrl, - conn: { - host: hostDbUrl.hostname, - port: Number(hostDbUrl.port), - user: "postgres", - database: "postgres", - }, - isLocal: true, - migrationsDir: path.join(workdir, "supabase", "migrations"), - }).pipe( - // Best-effort: Go's own `TryCacheMigrationsCatalog` failure only ever warns - // (`fmt.Fprintln(os.Stderr, "Warning: failed to cache migrations catalog:", err)`, - // start.go:378) and never fails `legacyStartSetupLocalDatabase` — same shape - // `legacy-db-push-core.ts` already established for this exact call. - Effect.catch((error) => - output.raw( - `Warning: failed to cache migrations catalog: ${redactLegacyConnectionString(error.message)}\n`, - "stderr", - ), - ), - ); - }), - ); - - // `initCurrentBranch` (start.go:233-241) is NOT called here — see this - // module's header for why it moved to the caller instead. - }); - -/** - * The `setup` shape shared by BOTH real Go callers of {@link - * legacyStartSetupLocalDatabase} — `db start`'s own fresh-volume branch - * (`start-database.ts`'s `legacyStartDatabase`) and `db reset`'s PG15 recreate - * composition (`recreate-local-database.ts`'s `legacyRecreateLocalDatabase15`) — - * everything {@link legacyStartSetupLocalDatabase} needs, minus what {@link - * legacyRunFreshDbSetup} itself already resolves/threads through (`session`, - * `images`). The two callers used to each declare an identical copy of this - * interface; hoisted here alongside {@link legacyRunFreshDbSetup} itself - * (CLI-1955 review follow-up). + * The `setup` shape shared by BOTH real Go callers of {@link legacyRunFreshDbSetup} — `db + * start`/`supabase start`'s own fresh-volume branch (`start-database.ts`'s + * `legacyStartDatabase`) and `db reset`'s PG15 recreate composition + * (`recreate-local-database.ts`'s `legacyRecreateLocalDatabase15`) — everything that pipeline + * needs, minus what it already resolves itself (the session, the JWKS/one-shot-job-image + * prelude, `[db]` config). The two callers used to each declare an identical copy of this + * interface; hoisted here alongside {@link legacyRunFreshDbSetup} itself (CLI-1955 review + * follow-up). */ export interface LegacyFreshDbSetupInput { readonly majorVersion: number; /** Already spliced with the caller's own realtime/storage/auth enabled-for-setup + ip_version/max_header_length/file_size_limit overrides — see `bootstrap-config.ts`'s `LegacyDbBootstrapConfig`. */ - readonly config: LegacyStartSetupLocalDatabaseInput["config"]; - /** Threaded straight through to {@link LegacyStartSetupLocalDatabaseInput.experimental} — see its own doc comment. */ + readonly config: LegacySetupDatabaseInput["config"]; + /** + * `--experimental`/`SUPABASE_EXPERIMENTAL`, resolved by the caller (Go's + * `viper.GetBool("EXPERIMENTAL")`) — threaded straight into {@link legacyMigrateAndSeed}'s own + * `experimental` gate (`internal/migration/apply/apply.go:19`); `legacySetupDatabase`/Go's own + * `SetupDatabase` have no use for it — only the trailing `MigrateAndSeed` call does. + */ readonly experimental: boolean; readonly dbUrl: string; readonly jwtSecret: string; @@ -1310,7 +1246,7 @@ export interface LegacyFreshDbSetupInput { readonly projectEnvValues: Readonly> | undefined; /** * `--debug` — threaded straight through to {@link - * LegacyStartSetupLocalDatabaseInput.debug}; see its own doc comment. + * LegacySetupDatabaseInput.debug}; see its own doc comment. */ readonly debug: boolean; } @@ -1372,17 +1308,7 @@ export const legacyRunDatabaseWebhooksSetup = (input: { ); if (pgNetOwnedByMigrations) return; } - const tmpDir = yield* input.fs - .makeTempDirectoryScoped({ prefix: "supabase-start-db-webhooks-" }) - .pipe( - Effect.mapError( - (error) => - new LegacyDbSetupError({ - message: `failed to create temp directory: ${errMessage(error)}`, - reason: "filesystem", - }), - ), - ); + const tmpDir = yield* legacySetupTempDir(input.fs, "supabase-start-db-webhooks-"); if (!input.enabled) { yield* legacyRemoveDatabaseWebhooks(session, input.fs, input.path, tmpDir); return; @@ -1392,34 +1318,33 @@ export const legacyRunDatabaseWebhooksSetup = (input: { ); /** - * Runs {@link legacyStartSetupLocalDatabase} against a freshly-provisioned local - * Postgres — the exact sequence BOTH real Go callers run once Postgres's own - * healthcheck passes on a fresh database (`db start`'s fresh-volume branch and - * `db reset`'s PG15 recreate, see {@link LegacyFreshDbSetupInput}'s own doc - * comment): dial the host-facing session (Go's `ConnectLocalPostgres`), resolve - * JWKS + the three PG15+ one-shot job images' PINNED names via {@link - * legacyResolveDbSetupPrelude} (the same hoisted prelude the shadow-database variant - * uses), then run {@link legacyStartSetupLocalDatabase} - * itself. `version`/`seedFlags` are the one genuine difference between the two - * callers (`db start` always passes `""`/`{noSeed:false, sqlPaths:[]}`; `db - * reset` passes its own resolved reset version/flags) — threaded straight - * through by the caller, matching each one's own `LegacyStartSetupLocalDatabaseInput` - * field of the same name. + * Runs the full `SetupLocalDatabase`-equivalent sequence against a freshly-provisioned local + * Postgres — the exact pipeline BOTH real Go callers run once Postgres's own healthcheck passes + * on a fresh database (`db start`/`supabase start`'s fresh-volume branch and `db reset`'s PG15 + * recreate, see {@link LegacyFreshDbSetupInput}'s own doc comment), and the single entry point + * this module's header describes step by step: load `[db]` config, dial the host-facing session + * (Go's `ConnectLocalPostgres`), resolve JWKS + the three PG15+ one-shot job images' PINNED names + * via {@link legacyResolveDbSetupPrelude} (the same hoisted prelude the shadow-database variant + * uses), run {@link legacySetupDatabase}, then `apply.MigrateAndSeed` and the best-effort + * `pgcache.TryCacheMigrationsCatalog` warmup. + * + * `version`/`seedFlags` are the one genuine difference between the two callers (`db start` always + * passes `""`/`{noSeed:false, sqlPaths:[]}`; `db reset` passes its own resolved reset + * version/flags) — threaded straight through by the caller. + * + * `baseline` is the baseline cache's seam (`main-db-baseline.ts`), applied through the SAME + * {@link legacyOpenBaselineSession} the throwaway shadow's own composition uses + * (`legacyOpenShadowBaselineSession`, `shadow-database.ts`): a `warm` hit skips the prelude AND + * {@link legacySetupDatabase} outright (so neither `Initialising schema...` nor `Seeding globals + * from roles.sql...` prints and the PG15+ one-shot migrate jobs never run), a `cold` provision + * runs the baseline in its own scope and snapshots before reconnecting for `MigrateAndSeed`, and + * the default {@link LEGACY_BASELINE_UNCACHED} is today's single-session, no-snapshot flow, byte + * for byte. */ export const legacyRunFreshDbSetup = ( spawner: Spawner, - input: { - readonly fs: FileSystem.FileSystem; - readonly path: Path.Path; - readonly workdir: string; - readonly projectId: string; - readonly networkId: string; - readonly hostname: string; - readonly dbPort: number; - readonly version: string; - readonly seedFlags: { readonly noSeed: boolean; readonly sqlPaths: ReadonlyArray }; - readonly setup: LegacyFreshDbSetupInput; - }, + input: LegacyRunFreshDbSetupInput, + baseline: LegacyClusterBaselineState = LEGACY_BASELINE_UNCACHED, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyDbConnectError | LegacyImagePrepullError | E, @@ -1429,54 +1354,188 @@ export const legacyRunFreshDbSetup = ( | RuntimeInfo | LegacyEdgeRuntimeScript | LegacyPgDeltaSslProbe + // `legacyTryCacheMigrationsCatalog`'s own pg-delta export call resolves + // `FileSystem.FileSystem`/`Path.Path` from the effect context itself (not from + // the `fs`/`path` values this function already threads through as plain data — + // see `legacy-pgdelta.ts`'s `legacyExportCatalogPgDelta`), so both must be + // ambient here too; every real caller already gets them from `BunServices.layer` + // at the CLI root runtime, same as `db push`'s own composition. | FileSystem.FileSystem | Path.Path > => Effect.scoped( Effect.gen(function* () { - const { setup } = input; + const { setup, fs, path, workdir } = input; const dbPassword = legacyStartInternalDbPassword(setup.dbUrl); // Go's `SetupLocalDatabase` dials this first host-facing connect exactly // once (`start.go:360-363`); we deliberately diverge and retry dial-level // failures: the container's internal health check says nothing about the // HOST side, where Docker Desktop (Windows/WSL2) can publish the port a - // few seconds late (#6136). - const session = yield* legacyConnectLocalPostgres({ + // few seconds late (#6136). Held as an unevaluated Effect because the + // snapshotting cold path dials TWICE — see `baseline` above. + const connect = legacyConnectLocalPostgres({ hostname: input.hostname, dbPort: input.dbPort, password: dbPassword, }); - const { jwks, images: dbSetupImages } = yield* legacyResolveDbSetupPrelude(setup); + // `warnOnUnresolvedEnv: false` — both `start.handler.ts` and `db/start/ + // start.handler.ts` already ran an earlier, same-invocation `legacyCheckDbToml` + // purely for its Go-parity validation side effect (their own callers discard the + // result) before ever reaching this fresh-volume setup, so that earlier call + // already printed Go's single `assertEnvLoaded` OrioleDB S3 WARN, if any. Without + // this, this module's own accepted duplicate config-load pass (see this module's + // header) would print the SAME warning a second time — a real, observable stderr + // divergence from Go's exactly-once `flags.LoadConfig`, unlike the harmless + // resolved-value duplication the header describes. + // + // Loaded once, up front, rather than mid-pipeline: EVERY path below needs it (a warm + // cache hit skips `legacySetupDatabase` entirely but still migrates and seeds). That + // is the one ordering divergence from before — the load used to sit between the + // prelude's banner and `SetupDatabase` — and it is unobservable in practice: with + // `warnOnUnresolvedEnv: false` the read prints nothing, and each caller's handler + // already failed the whole command on a malformed `config.toml` long before here. + const toml = yield* legacyCheckDbToml(fs, path, workdir, undefined, { + warnOnUnresolvedEnv: false, + }); - yield* legacyStartSetupLocalDatabase(spawner, { - session, - fs: input.fs, - path: input.path, - workdir: input.workdir, - config: setup.config, + // SetupDatabase: initSchema -> ApplyApiPrivileges -> vault secrets -> custom-roles seed + // (start.go:383-399), preceded by the banner + JWKS + one-shot job image prelude. + const runBaseline = (session: LegacyDbSession) => + Effect.gen(function* () { + const { jwks, images: dbSetupImages } = yield* legacyResolveDbSetupPrelude(setup); + yield* legacySetupDatabase(spawner, { + session, + fs, + path, + workdir, + config: setup.config, + webhooksEnabled: toml.webhooksEnabled, + majorVersion: setup.majorVersion, + // Go's `utils.DbId` — the internal Docker-network address the PG15+ one-shot + // jobs connect through. + dbHost: localDbContainerId(input.projectId), + projectId: input.projectId, + networkId: input.networkId, + dbUrl: setup.dbUrl, + jwtSecret: setup.jwtSecret, + jwks, + apiUrl: setup.apiUrl, + authExternalUrl: setup.authExternalUrl, + siteUrl: setup.siteUrl, + anonKey: setup.anonKey, + serviceRoleKey: setup.serviceRoleKey, + storageTargetMigration: setup.storageTargetMigration, + images: dbSetupImages, + projectEnvValues: setup.projectEnvValues, + debug: setup.debug, + apiAutoExposeNewTables: toml.baseline.apiAutoExposeNewTables, + vault: toml.vault, + rolesSql: legacyBaselineRolesSql(baseline), + }); + }); + + // Warm: connect only. Cold: baseline in its own scope, snapshot, reconnect. Uncached: Go's + // single-connection flow — see {@link legacyOpenBaselineSession}. + const session = yield* legacyOpenBaselineSession(connect, runBaseline, baseline); + + // apply.MigrateAndSeed(ctx, version, conn, fsys) — `db start`'s own caller always + // passes `version: ""` (every pending migration, matching `SetupLocalDatabase`'s + // own call in the `start` context, `start.go:185,368`); `db reset`'s PG15 recreate + // passes its own resolved reset version instead (`resetDatabase15`, `reset.go:169`) + // — see `input.version`'s own doc comment. `experimental`/`pgDeltaEnabled`/ + // `schemaPaths` gate `legacyMigrateAndSeed`'s own declarative-schema-files branch + // (apply.go:19) — see its doc comment; `toml.pgDelta.enabled` and `toml.schemaPaths` + // are this module's own already-loaded config (the latter already resolved + + // `SUPABASE_DB_MIGRATIONS_SCHEMA_PATHS` env-overridden by `legacyCheckDbToml`, + // `legacy-db-config.toml-read.ts`), not re-read from the caller's raw, unresolved + // `ProjectConfig`. `input.seedFlags` applies `db reset`'s own `--no-seed`/ + // `--sql-paths` overrides on top of the loaded `[db.seed]` config — a no-op for + // `db start`, which has neither flag. + yield* legacyMigrateAndSeed(session, fs, path, workdir, input.version, { + migrationsEnabled: toml.migrationsEnabled, + seed: legacyResolveResetSeedConfig(toml.seed, input.seedFlags, path), experimental: setup.experimental, - majorVersion: setup.majorVersion, - // Go's `utils.DbId` — the internal Docker-network address the PG15+ one-shot - // jobs connect through. Unchanged from before CLI-1956, just now an explicit - // parameter on `LegacySetupDatabaseInput` instead of computed inside it. - dbHost: localDbContainerId(input.projectId), - projectId: input.projectId, - networkId: input.networkId, - dbUrl: setup.dbUrl, - jwtSecret: setup.jwtSecret, - jwks, - apiUrl: setup.apiUrl, - authExternalUrl: setup.authExternalUrl, - siteUrl: setup.siteUrl, - anonKey: setup.anonKey, - serviceRoleKey: setup.serviceRoleKey, - storageTargetMigration: setup.storageTargetMigration, - images: dbSetupImages, - projectEnvValues: setup.projectEnvValues, - debug: setup.debug, - version: input.version, - seedFlags: input.seedFlags, + pgDeltaEnabled: toml.pgDelta.enabled, + schemaPaths: toml.schemaPaths, + localDatabaseWebhooksEnabled: toml.webhooksEnabled, }); + + const output = yield* Output; + + // pgcache.TryCacheMigrationsCatalog(ctx, pgconn.Config{Host: Config.Hostname, + // Port: Config.Db.Port, User: "postgres", Password: Config.Db.Password, Database: + // "postgres"}, "local", version, fsys, ...) (start.go:371-379): best-effort, run + // immediately after MigrateAndSeed above, for BOTH real Go callers of this shared + // function — `db start` (always `version: ""`) and `db reset`'s PG15 recreate + // (its own resolved reset `input.version`, usually also `""`). `cacheEnabled` + // reproduces Go's `ShouldCacheMigrationsCatalog()` gate exactly + // (`pgcache/cache.go:93-95`): `len(version) == 0` AND (`toml.pgDelta.enabled` OR + // `SUPABASE_EXPERIMENTAL_PG_DELTA`) — the same formula `legacy-db-push-core.ts` + // already uses for its own call. `setup.dbUrl` is already the HOST-facing + // `postgresql://postgres:@:/postgres` address (see its + // own doc comment) — the exact same shape Go's `utils.ToPostgresURL(config)` builds + // from that literal `pgconn.Config` here, so it's reused directly as `targetUrl` + // rather than re-derived. `conn`'s fields are only ever read by + // `legacyCatalogPrefixFromConfig` on a non-local prefix fallback, unreachable here + // since `isLocal` is always `true`. + const cacheEnabled = + input.version.length === 0 && + (toml.pgDelta.enabled || + legacyParseBoolEnv(toml.envLookup("SUPABASE_EXPERIMENTAL_PG_DELTA"))); + const pgDeltaImplementation = legacyResolvePgDeltaImplementation( + legacyPgDeltaImplementationFlag( + process.env[LEGACY_PG_DELTA_NEXT_FLAG_NAME], + toml.projectEnv[LEGACY_PG_DELTA_NEXT_FLAG_NAME], + ), + ); + const pgDeltaCtx: LegacyPgDeltaContext = { + projectId: input.projectId, + cwd: workdir, + npmVersion: Option.getOrUndefined(toml.pgDelta.npmVersion), + denoVersion: toml.denoVersion, + projectEnv: toml.projectEnv, + }; + const hostDbUrl = new URL(setup.dbUrl); + // Scope the `PGDELTA_NPM_REGISTRY`-from-project-`.env` apply to just this call: + // `legacyExportCatalogPgDelta` reads it off bare `process.env` + // (`legacyPgDeltaNpmRegistryOption`), same as `db push`/`db pull`/`db dump`/ + // `bootstrap`'s own calls into pg-delta — Go's `loadNestedEnv` already made it + // process-wide by this point (`config.go:788`), but this module otherwise threads + // every override through `projectEnvValues` explicitly rather than mutating + // `process.env`, so this one shared-code call needs the same opt-in helper those + // other commands use. `legacyApplyProjectEnv` registers a finalizer that reverts it. + yield* Effect.scoped( + Effect.gen(function* () { + yield* legacyApplyProjectEnv(setup.projectEnvValues ?? {}); + yield* legacyTryCacheMigrationsCatalog(fs, path, pgDeltaCtx, { + // The catalog is a legacy-engine artifact with no in-process consumer. + enabled: cacheEnabled && pgDeltaImplementation === "legacy", + targetUrl: setup.dbUrl, + conn: { + host: hostDbUrl.hostname, + port: Number(hostDbUrl.port), + user: "postgres", + database: "postgres", + }, + isLocal: true, + migrationsDir: path.join(workdir, "supabase", "migrations"), + }).pipe( + // Best-effort: Go's own `TryCacheMigrationsCatalog` failure only ever warns + // (`fmt.Fprintln(os.Stderr, "Warning: failed to cache migrations catalog:", err)`, + // start.go:378) and never fails this pipeline — same shape + // `legacy-db-push-core.ts` already established for this exact call. + Effect.catch((error) => + output.raw( + `Warning: failed to cache migrations catalog: ${redactLegacyConnectionString(error.message)}\n`, + "stderr", + ), + ), + ); + }), + ); + + // `initCurrentBranch` (start.go:233-241) is NOT called here — see this + // module's header for why it moved to the caller instead. }), ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts index 0c3d2bad12..ff22342f07 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts @@ -18,15 +18,19 @@ import { LegacyEdgeRuntimeScript, type LegacyEdgeRuntimeRunOpts, } from "../legacy-edge-runtime-script.service.ts"; +import { legacyGetRegistryImageUrl } from "../legacy-docker-registry.ts"; import { LegacyPgDeltaSslProbe } from "../legacy-pgdelta-ssl-probe.service.ts"; +import type { LocalServiceVersionOverrides } from "../../../shared/services/services.shared.ts"; import { LegacyDbSetupError, legacyResolveDbSetupPrelude, legacyRunDatabaseWebhooksSetup, + legacyRunFreshDbSetup, legacyStartInitCurrentBranch, - legacyStartSetupLocalDatabase, - type LegacyStartSetupLocalDatabaseInput, + type LegacyFreshDbSetupInput, + type LegacyRunFreshDbSetupInput, } from "./db-setup.ts"; +import { legacyResolvePinnedImage } from "./pinned-image.ts"; const decodeConfig = Schema.decodeUnknownSync(ProjectConfigSchema); @@ -167,7 +171,7 @@ function mockDockerRunFails() { /** * `LegacyEdgeRuntimeScript`/`LegacyPgDeltaSslProbe` back * `legacyTryCacheMigrationsCatalog`'s own pg-delta catalog-export call (`db-setup.ts`'s - * pgcache-warmup step) — required by {@link legacyStartSetupLocalDatabase}'s own widened + * pgcache-warmup step) — required by {@link legacyRunFreshDbSetup}'s own widened * effect environment regardless of whether a given test's config actually enables * pg-delta (the early `!params.enabled` return means these mocks are never invoked at * runtime unless a test opts in via `writeConfigToml`'s `[experimental.pgdelta]`). @@ -205,52 +209,102 @@ function writeConfigToml(workdir: string, content: string): void { const defaultConfig: ProjectConfig = decodeConfig({}); +/** + * Flat override keys accepted by {@link baseInput} — the SAME shape the pre-merge test suite + * used against the (now-deleted) flat `LegacyStartSetupLocalDatabaseInput`, distributed here + * into `legacyRunFreshDbSetup`'s split top-level/`setup` shape. Kept as a standalone interface + * (not `Partial>`) since several keys (`dbHost`/`images`) no + * longer exist on the real input at all, and several others (`majorVersion`, `config`, `jwks`, …) + * now live nested under `setup` instead of flat. + */ +interface RunFreshDbSetupOverrides { + readonly config?: ProjectConfig; + readonly experimental?: boolean; + readonly majorVersion?: number; + readonly projectId?: string; + readonly networkId?: string; + readonly hostname?: string; + readonly dbPort?: number; + readonly dbUrl?: string; + readonly jwtSecret?: string; + /** A plain string, matching the pre-merge flat shape — wrapped as `Effect.succeed(...)` for `setup.jwks`. */ + readonly jwks?: string; + readonly apiUrl?: string; + readonly authExternalUrl?: string; + readonly siteUrl?: string; + readonly anonKey?: string; + readonly serviceRoleKey?: string; + readonly storageTargetMigration?: string; + readonly realtimeEnabledForSetup?: boolean; + readonly storageEnabledForSetup?: boolean; + readonly authEnabledForSetup?: boolean; + readonly serviceVersionOverrides?: LocalServiceVersionOverrides; + readonly projectEnvValues?: Readonly>; + readonly debug?: boolean; + readonly version?: string; + readonly seedFlags?: { readonly noSeed: boolean; readonly sqlPaths: ReadonlyArray }; +} + +/** + * `session` travels alongside the real `legacyRunFreshDbSetup` input as an extra property — + * `run` below pulls it off and turns it into the `LegacyDbConnection` mock layer instead of + * passing it through (the real function opens its own session via that service; it no longer + * accepts one as input). + */ +type RunFreshDbSetupTestInput = Omit, "fs" | "path"> & { + readonly session: LegacyDbSession; +}; + function baseInput( workdir: string, session: LegacyDbSession, - overrides: Partial = {}, -): Omit { + overrides: RunFreshDbSetupOverrides = {}, +): RunFreshDbSetupTestInput { + const setup: LegacyFreshDbSetupInput = { + majorVersion: overrides.majorVersion ?? 17, + config: overrides.config ?? defaultConfig, + experimental: overrides.experimental ?? false, + dbUrl: overrides.dbUrl ?? "postgresql://postgres:postgrespassword@127.0.0.1:54322/postgres", + jwtSecret: overrides.jwtSecret ?? "super-secret-jwt-token-with-at-least-32-characters-long", + jwks: Effect.succeed(overrides.jwks ?? '{"keys":[]}'), + apiUrl: overrides.apiUrl ?? "http://127.0.0.1:54321", + authExternalUrl: overrides.authExternalUrl, + siteUrl: overrides.siteUrl ?? defaultConfig.auth.site_url, + anonKey: overrides.anonKey ?? "anon-key", + serviceRoleKey: overrides.serviceRoleKey ?? "service-role-key", + storageTargetMigration: overrides.storageTargetMigration ?? "", + realtimeEnabledForSetup: overrides.realtimeEnabledForSetup ?? true, + storageEnabledForSetup: overrides.storageEnabledForSetup ?? true, + authEnabledForSetup: overrides.authEnabledForSetup ?? true, + serviceVersionOverrides: overrides.serviceVersionOverrides ?? {}, + projectEnvValues: overrides.projectEnvValues, + debug: overrides.debug ?? false, + }; return { session, workdir, - config: defaultConfig, - experimental: false, - majorVersion: 17, - dbHost: "supabase_db_proj", - projectId: "proj", - networkId: "supabase_network_proj", - dbUrl: "postgresql://postgres:postgrespassword@127.0.0.1:54322/postgres", - jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", - jwks: '{"keys":[]}', - apiUrl: "http://127.0.0.1:54321", - siteUrl: defaultConfig.auth.site_url, - anonKey: "anon-key", - serviceRoleKey: "service-role-key", - storageTargetMigration: "", - images: { - realtime: "public.ecr.aws/supabase/realtime:v2.34.7", - storage: "public.ecr.aws/supabase/storage-api:v1.0.0", - auth: "public.ecr.aws/supabase/gotrue:v2.170.0", - }, - projectEnvValues: undefined, - debug: false, - version: "", - seedFlags: { noSeed: false, sqlPaths: [] }, - ...overrides, + projectId: overrides.projectId ?? "proj", + networkId: overrides.networkId ?? "supabase_network_proj", + hostname: overrides.hostname ?? "127.0.0.1", + dbPort: overrides.dbPort ?? 54322, + version: overrides.version ?? "", + seedFlags: overrides.seedFlags ?? { noSeed: false, sqlPaths: [] }, + setup, }; } const run = ( - input: Omit, + input: RunFreshDbSetupTestInput, out: ReturnType, docker: ReturnType | ReturnType, edgeRuntime: ReturnType = mockEdgeRuntime(), -) => - Effect.gen(function* () { +) => { + const { session, ...rest } = input; + return Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; - return yield* legacyStartSetupLocalDatabase(mockAlwaysCachedSpawner(), { - ...input, + return yield* legacyRunFreshDbSetup(mockAlwaysCachedSpawner(), { + ...rest, fs, path, }); @@ -263,11 +317,15 @@ const run = ( mockRuntimeInfo({ platform: "darwin" }), edgeRuntime.layer, mockPgDeltaSslProbeLayer(), + Layer.succeed(LegacyDbConnection, { + connect: () => Effect.succeed(session), + }), ), ), ); +}; -describe("legacyStartSetupLocalDatabase", () => { +describe("legacyRunFreshDbSetup", () => { describe("PG <= 14 vs PG >= 15 schema branch", () => { it.effect("PG14: execs globals + the PG14 initial schema, runs no one-shot docker jobs", () => { const workdir = makeWorkdir(); @@ -337,9 +395,20 @@ describe("legacyStartSetupLocalDatabase", () => { Effect.map(() => { expect(docker.runs.length).toBe(2); const images = docker.runs.map((r) => r.image); - expect(images).toContain("public.ecr.aws/supabase/realtime:v2.34.7"); - expect(images).toContain("public.ecr.aws/supabase/gotrue:v2.170.0"); - expect(images).not.toContain("public.ecr.aws/supabase/storage-api:v1.0.0"); + // `mockAlwaysCachedSpawner` reports the FIRST registry candidate + // (`legacyGetRegistryImageUrlCandidates`'s default-registry entry) as already + // cached, so the job's resolved `image` is the registry-qualified form of the + // pinned image, not the bare pinned string — same derivation + // `squash.integration.test.ts`/`serve.integration.test.ts` already use. + expect(images).toContain( + legacyGetRegistryImageUrl(legacyResolvePinnedImage("realtime", "realtime", {})), + ); + expect(images).toContain( + legacyGetRegistryImageUrl(legacyResolvePinnedImage("gotrue", "auth", {})), + ); + expect(images).not.toContain( + legacyGetRegistryImageUrl(legacyResolvePinnedImage("storage", "storage", {})), + ); rmSync(workdir, { recursive: true, force: true }); }), ); @@ -402,7 +471,10 @@ describe("legacyStartSetupLocalDatabase", () => { baseInput(workdir, session, { majorVersion: 15, config, - dbHost: "supabase_db_myproj", + // `dbHost` is no longer caller-supplied — it's derived internally via + // `localDbContainerId(projectId)` (`supabase_db_`), so overriding + // `projectId` here reproduces the same `supabase_db_myproj` DB_HOST value. + projectId: "myproj", jwks: '{"keys":["stub"]}', }), out, @@ -825,33 +897,25 @@ describe("legacyStartSetupLocalDatabase", () => { }, ); - it.effect( - "warns without failing legacyStartSetupLocalDatabase when the catalog export fails", - () => { - const workdir = makeWorkdir(); - writeConfigToml(workdir, "[experimental.pgdelta]\nenabled = true\n"); - writeFileSync(join(workdir, "supabase", ".env"), "SUPABASE_USE_PG_DELTA_NEXT=false\n"); - const { session } = fakeSession(); - const out = mockOutput(); - const docker = mockDockerRun(); - const edgeRuntime = mockEdgeRuntime({ - failWith: "edge-runtime script produced no output", - }); - return run( - baseInput(workdir, session, { majorVersion: 14 }), - out, - docker, - edgeRuntime, - ).pipe( - Effect.map(() => { - expect(out.stderrText).toContain( - "Warning: failed to cache migrations catalog: edge-runtime script produced no output", - ); - rmSync(workdir, { recursive: true, force: true }); - }), - ); - }, - ); + it.effect("warns without failing legacyRunFreshDbSetup when the catalog export fails", () => { + const workdir = makeWorkdir(); + writeConfigToml(workdir, "[experimental.pgdelta]\nenabled = true\n"); + writeFileSync(join(workdir, "supabase", ".env"), "SUPABASE_USE_PG_DELTA_NEXT=false\n"); + const { session } = fakeSession(); + const out = mockOutput(); + const docker = mockDockerRun(); + const edgeRuntime = mockEdgeRuntime({ + failWith: "edge-runtime script produced no output", + }); + return run(baseInput(workdir, session, { majorVersion: 14 }), out, docker, edgeRuntime).pipe( + Effect.map(() => { + expect(out.stderrText).toContain( + "Warning: failed to cache migrations catalog: edge-runtime script produced no output", + ); + rmSync(workdir, { recursive: true, force: true }); + }), + ); + }); }); }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/local-container-inputs.ts b/apps/cli/src/legacy/shared/db-bootstrap/local-container-inputs.ts index 03d332ed77..dc24d876bc 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/local-container-inputs.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/local-container-inputs.ts @@ -55,7 +55,7 @@ import { legacyResolveDbBootstrapConfig, type LegacyDbBootstrapConfig, } from "./bootstrap-config.ts"; -import type { LegacyFreshDbSetupInput } from "./db-setup.ts"; +import { legacyMemoizeSuccess, type LegacyFreshDbSetupInput } from "./db-setup.ts"; import type { LegacyContainerOpts } from "./container-lifecycle.ts"; import { legacyEnsureImagesCached, type LegacyImagePrepullError } from "./image-prepull.ts"; import type { LegacyPostgresStartServiceInput } from "./postgres.service.ts"; @@ -280,17 +280,25 @@ export const legacyBuildLocalDbContainerInputs = ( // Go's `initSchema15`'s realtime job resolves JWKS itself, LOCALLY, gated on // `Realtime.Enabled` (`internal/db/start/start.go:337-341`) — `legacyRunFreshDbSetup` only // evaluates this Effect when reached AND `realtimeEnabledForSetup`. - jwks: Effect.tryPromise({ - try: () => - legacyResolveLocalJwks( - config, - workdir, - values.jwtSecret, - projectEnvValues, - remoteOverrideKeys, - ), - catch: (cause) => mapError(cause instanceof Error ? cause.message : String(cause)), - }), + // + // Memoized for the same reason the shadow's own run input memoizes it + // (`legacyShadowRunInputFromLocalContainerInputs`): with the baseline cache enabled a cold + // run evaluates this TWICE — once for the cache key (`main-db-baseline.ts`'s peek, before + // the container is even created), once by `legacyResolveDbSetupPrelude` for the baseline + // itself — and third-party JWKS discovery can be a real network request. + jwks: legacyMemoizeSuccess( + Effect.tryPromise({ + try: () => + legacyResolveLocalJwks( + config, + workdir, + values.jwtSecret, + projectEnvValues, + remoteOverrideKeys, + ), + catch: (cause) => mapError(cause instanceof Error ? cause.message : String(cause)), + }), + ), apiUrl: values.apiUrl, authExternalUrl: legacyResolveAuthExternalUrl( loaded?.document, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/main-db-baseline.ts b/apps/cli/src/legacy/shared/db-bootstrap/main-db-baseline.ts new file mode 100644 index 0000000000..4b4c566415 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/main-db-baseline.ts @@ -0,0 +1,491 @@ +/** + * The baseline cache, applied to the LONG-RUNNING local `db` container — `supabase db reset`'s + * PG15 recreate and the fresh-volume branch of `supabase start`/`supabase db start`. The cluster + * those flows build from scratch is, up to the seam `legacySetupDatabase` ends at, the same thing + * `shadow-cache.ts` already snapshots for the throwaway shadow: the same init schema, the same + * PG15+ one-shot realtime/storage/auth migrate jobs, the same webhooks convergence, API-privilege + * pass, vault upsert, and `roles.sql` seed. So it restores from — and publishes into — the SAME + * tar pool, under the same `legacyShadowCacheKey`. + * + * - **Warm** (a snapshot exists for this key): the tar is unpacked into the created-but-unstarted + * Postgres container ({@link legacyPgDataRestoreArchive} via `preStartArchives`, exactly as the + * warm shadow does), the entrypoint finds an initialized PGDATA and skips `initdb` plus the + * whole baseline, and `legacyRunFreshDbSetup` goes straight to `MigrateAndSeed`. `Restoring + * cached baseline...` prints where `Initialising schema...` would have. + * - **Cold** (no snapshot): the container is created exactly as before, and the baseline is + * published at the baseline/migrations seam — after `legacySetupDatabase`, before + * `MigrateAndSeed` — via {@link legacyExportBaselineSnapshot}. + * - **Uncachable** (PG<=14, OrioleDB, `SUPABASE_SHADOW_CACHE=false`, unreadable `roles.sql`, or + * `db start --from-backup`, which owns `preStartArchives` itself): today's flow, byte for byte. + * + * A warm restore that does not come up is never the user's problem to debug: + * {@link legacyBringUpMainDbWithBaseline} warns, destroys the restored container AND its volume, + * and re-runs the bring-up cold — the same escape hatch `legacyAcquireShadowDatabase` gives the + * shadow. An INTERRUPT during that restore destroys them too, for a sharper reason: a + * half-restored PGDATA volume left behind would read as "not a fresh volume" to the next + * `supabase start`, which would then skip the fresh-volume setup entirely. + * + * **What sharing the pool with the shadow rests on.** The shadow bootstraps under + * `-c max_worker_processes=0` (`LEGACY_SHADOW_ENTRYPOINT_ARGS`, `postgres.service.ts`) and this + * container does not, and that difference is deliberately NOT in the key — see + * `LegacyShadowCacheKeyInputs`'s own doc comment (`shadow-cache.ts`) for why that is safe today + * and what would break it. + * + * Cold publication is best-effort: `Warning: database baseline not cached: ` and the run + * continues. The ONE failure that still propagates is the cluster not coming back after the + * snapshot's `docker stop`/`docker start` — see {@link legacyExportBaselineSnapshot} for why + * reporting success over a dead container is worse than failing. + * + * **Why the mid-setup stop is safe on every flow.** `supabase start` finishes the whole DB + * bootstrap — container, health wait, fresh-volume setup — BEFORE it starts any other service + * (`start.handler.ts`'s `bringUp`), so nothing is up to observe the pause. `db start` brings up + * the database alone. `db reset` runs with the satellites up, but they already tolerate a `db` + * container that was force-removed and recreated from scratch two steps earlier, and the reset + * restarts them itself immediately after (`Restarting containers...`). + */ + +import { Effect, Result, type FileSystem, type Option, type Path } from "effect"; +import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; + +import { Output } from "../../../shared/output/output.service.ts"; +import type { LegacyDbConnection } from "../legacy-db-connection.service.ts"; +import type { LegacyPgConnInput } from "../legacy-db-connection.service.ts"; +import type { LegacyVaultSecret } from "../legacy-vault.ts"; +import { LEGACY_BASELINE_UNCACHED, type LegacyClusterBaselineState } from "./baseline-state.ts"; +import { + legacyCreateContainer, + legacyRemoveContainer, + legacyRemoveVolume, + type LegacyContainerError, + type LegacyContainerOpts, + type LegacyContainerRemoveError, + type LegacyVolumeRemoveError, +} from "./container-lifecycle.ts"; +import { LegacyDbSetupError, type LegacyFreshDbSetupInput } from "./db-setup.ts"; +import type { LegacyStartContainerSpec } from "./docker-create-args.ts"; +import type { LegacyHealthCheckTimeoutError } from "./health-check.ts"; +import { legacyStartInternalDbPassword } from "./internal-db-connection.ts"; +import { legacyPgDataRestoreArchive } from "./pgdata-snapshot.ts"; +import type { LegacyPostgresStartServiceInput } from "./postgres.service.ts"; +import { + legacyExportBaselineSnapshot, + legacyForgetShadowBaselineTar, + legacyPeekShadowBaseline, + legacyRefreshShadowBaselineOnWarmHit, + legacyShadowBaselineTarPath, + type LegacyBaselineCacheInput, +} from "./shadow-cache.ts"; +import { legacyTimeShadowPhase } from "./shadow-debug.ts"; + +type Spawner = ChildProcessSpawner["Service"]; + +/** The local `db` container's own binding of the shared baseline-state seam. */ +export type LegacyMainDbBaselineState = LegacyClusterBaselineState; + +type LegacyPreStartArchive = NonNullable[number]; + +/** + * The three `[db]`-config values the cache key needs that the container bring-up inputs do not + * already carry. Every caller already ran its own validating `legacyCheckDbToml` pass — Go's + * `flags.LoadConfig`-equivalent gate, long before any Docker work — so these are threaded down + * from THAT result rather than re-read here: the key MUST describe the baseline the provisioning + * actually configures (they are the same values `legacyRunFreshDbSetup` hands + * `legacySetupDatabase`), and a third silent pass over `config.toml` could only ever drift from + * it. + */ +export interface LegacyMainDbBaselineTomlInputs { + readonly webhooksEnabled: boolean; + readonly apiAutoExposeNewTables: Option.Option; + readonly vault: ReadonlyArray; +} + +/** + * Everything the main-db baseline cache needs — assembled by {@link legacyMainDbBaselineInput} + * from exactly the values both bring-up compositions already hold, so neither one has to know + * which of them the cache key reads. + */ +interface LegacyMainDbBaselineInput { + readonly fs: FileSystem.FileSystem; + readonly path: Path.Path; + readonly workdir: string; + /** + * The local `db` container's own name, which is ALSO its PGDATA volume's name + * (`localDbContainerId(projectId)` — Go's `utils.DbId` names both). The warm-failure fallback + * removes both under this one name. + */ + readonly dbContainerId: string; + /** The already registry-resolved `supabase/postgres` image the container runs. */ + readonly image: string; + /** + * The `db` container's host-facing connect target, used for the post-snapshot readiness probe. + * Its `password` is also the key's `dbPassword`: for this container that is always the + * `"postgres"` literal `legacyBuildPostgresStartContainerSpec` bakes in as `POSTGRES_PASSWORD`, + * deliberately NOT `[db] password`, which only the shadow honors. + */ + readonly connConfig: LegacyPgConnInput; + readonly healthTimeoutSeconds: number; + /** The container-spec fields the cluster's own identity depends on. */ + readonly postgresSpec: Omit; + readonly setup: LegacyFreshDbSetupInput; +} + +/** + * Assembles a {@link LegacyMainDbBaselineInput} from what `legacyStartDatabase`/ + * `legacyRecreateLocalDatabase15` already have in scope. The connect target is built the exact + * same way `legacyRunFreshDbSetup` builds its own (`legacyStartInternalDbPassword(setup.dbUrl)`), + * so the key's `dbPassword` can never drift from the password the run authenticates with. + */ +const legacyMainDbBaselineInput = (args: { + readonly fs: FileSystem.FileSystem; + readonly path: Path.Path; + readonly workdir: string; + readonly dbContainerId: string; + readonly hostname: string; + readonly dbPort: number; + readonly healthTimeoutSeconds: number; + readonly image: string; + readonly postgresSpec: Omit; + readonly setup: LegacyFreshDbSetupInput; +}): LegacyMainDbBaselineInput => ({ + fs: args.fs, + path: args.path, + workdir: args.workdir, + dbContainerId: args.dbContainerId, + image: args.image, + connConfig: { + host: args.hostname, + port: args.dbPort, + user: "postgres", + password: legacyStartInternalDbPassword(args.setup.dbUrl), + database: "postgres", + }, + healthTimeoutSeconds: args.healthTimeoutSeconds, + postgresSpec: args.postgresSpec, + setup: args.setup, +}); + +/** What {@link legacyPlanMainDbBaseline} decided, before any container exists. */ +type LegacyMainDbBaselinePlan = + /** The cache does not apply to this run — the spec and the pipeline are untouched. */ + | { readonly kind: "uncached"; readonly state: LegacyMainDbBaselineState } + /** Provision normally, then publish the baseline at the baseline/migrations seam. */ + | { readonly kind: "cold"; readonly state: LegacyMainDbBaselineState } + | { + readonly kind: "warm"; + /** Spliced into the Postgres container spec's `preStartArchives` before `docker create`. */ + readonly restoreArchive: LegacyPreStartArchive; + readonly state: LegacyMainDbBaselineState; + readonly tarPath: string; + /** + * What a failed restore degrades to — same key, same tar path, but publishing WITHOUT the + * `skipIfPublished` dedupe: unless it was deleted as suspect, the unusable tar is still + * sitting at that path, deliberately retained so this export atomically REPLACES it + * (review: Codex on #6215). + */ + readonly coldFallbackState: LegacyMainDbBaselineState; + }; + +/** + * The plan for a run the cache never even looks at — `db start --from-backup`, and every + * non-fresh-volume restart, neither of which provisions a platform baseline at all. + */ +const LEGACY_MAIN_DB_BASELINE_UNCACHED: LegacyMainDbBaselinePlan = { + kind: "uncached", + state: LEGACY_BASELINE_UNCACHED, +}; + +/** + * Peeks the shared tar pool and decides warm / cold / uncachable for the local `db` container, + * WITHOUT creating anything. Runs before `docker create`, because a warm restore has to be + * unpacked into the container between `docker create` and `docker start`. + * + * The `webhooks` policy is left at the config-following default: `legacySetupDatabase` on this + * path is called with no override at all, so it resolves `toml.webhooksEnabled` from + * `config.toml`, and the key must say the same thing (see `diff.handler.ts`'s own `webhooks` + * comment for what goes wrong when the key and the provisioning disagree). + * + * The error channel is the key resolution's own `E` — a JWKS resolution failure, which a real + * cold provision at this same input would have hit anyway inside `legacyResolveDbSetupPrelude`, + * just a few seconds later. That is the one ordering change on a failing run: JWKS discovery now + * fails before the container is created rather than after its health wait. + */ +const legacyPlanMainDbBaseline = ( + spawner: Spawner, + input: LegacyMainDbBaselineInput, + toml: LegacyMainDbBaselineTomlInputs, +): Effect.Effect => + Effect.gen(function* () { + const peek = yield* legacyPeekShadowBaseline(legacyMainDbCacheInput(input, toml)); + if (peek.state === "uncachable") return LEGACY_MAIN_DB_BASELINE_UNCACHED; + + const tarPath = legacyShadowBaselineTarPath(input.path, peek.key); + const publishing = (skipIfPublished: boolean): LegacyMainDbBaselineState => ({ + _tag: "cold", + // The exact `roles.sql` bytes this key was hashed from — the seed runs THESE rather than + // re-reading the file, so the published snapshot cannot disagree with its own key. + rolesSql: peek.keyInputs.rolesSql, + snapshotBaseline: legacyExportMainDbBaseline( + spawner, + input, + peek.key, + tarPath, + skipIfPublished, + ), + }); + + if (peek.state === "cold") { + // The tar was absent at peek time, so one found at export time can only be a same-key + // sibling's fresh publish — dedupe against it, exactly like the shadow's `!cached` path. + return { kind: "cold", state: publishing(true) } as const; + } + + yield* legacyRefreshShadowBaselineOnWarmHit(input, tarPath); + const output = yield* Output; + // Prints where `Initialising schema...` would have (`legacyResolveDbSetupPrelude`), on the + // same channel: a warm run really does skip that step, and silence would leave the seconds + // of `docker cp -` that replace it unexplained. + yield* output.raw("Restoring cached baseline...\n", "stderr"); + return { + kind: "warm", + restoreArchive: legacyPgDataRestoreArchive(input.fs, tarPath), + state: { _tag: "warm" }, + tarPath, + coldFallbackState: publishing(false), + } as const; + }); + +/** + * Creates + starts the Postgres container for `plan`, waits for it with the caller's own gate, + * and — on a warm restore that does not come up — warns, destroys the container AND its restored + * volume, and retries the bring-up cold. Resolves to the baseline state + * `legacyRunFreshDbSetup` should actually run with, which is NOT `plan.state` after a fallback. + * + * Removing the volume is what makes the fallback a real recovery rather than a loop: the tar was + * unpacked INTO the named PGDATA volume, so recreating the container over it would just boot the + * same broken cluster. `legacyRemoveContainer` is intolerant of "not found" while + * `legacyRemoveVolume`'s `-f` makes a missing volume a no-op — correct either way here, since a + * failed archive extraction already removes the container itself (see `legacyCreateContainer`) + * but never the named volume. + * + * Both warm failure shapes are covered: a corrupt tar fails the `docker cp -` inside + * `legacyCreateContainer`; a tar that extracts but yields an unusable cluster fails `waitReady`. + * The tar is deleted only in the SECOND case, mirroring + * `LegacyShadowCacheUnavailable.tarSuspect` — an extraction or daemon failure says nothing about + * the tar's contents, and the fallback's own export atomically replaces a genuinely bad one + * anyway. + */ +const legacyCreateMainDbWithBaseline = ( + spawner: Spawner, + args: { + readonly plan: LegacyMainDbBaselinePlan; + /** The undecorated Postgres container spec — the warm path splices `preStartArchives` on. */ + readonly spec: LegacyStartContainerSpec; + readonly containerOpts: LegacyContainerOpts; + /** `localDbContainerId(projectId)` — the container's name and its volume's name. */ + readonly dbContainerId: string; + readonly fs: FileSystem.FileSystem; + /** + * The caller's own readiness gate for the created container, already carrying that caller's + * semantics (`db start --from-backup` swallows a timeout; `db reset` never does). Not + * parameterized by container id: both callers watch the container by its stable NAME. + */ + readonly waitReady: Effect.Effect; + }, +): Effect.Effect< + LegacyMainDbBaselineState, + | LegacyContainerError + | LegacyHealthCheckTimeoutError + | LegacyContainerRemoveError + | LegacyVolumeRemoveError, + Output | R +> => + Effect.gen(function* () { + const bringUp = ( + restoreArchive: LegacyPreStartArchive | undefined, + ): Effect.Effect => + Effect.gen(function* () { + const spec: LegacyStartContainerSpec = + restoreArchive === undefined + ? args.spec + : { ...args.spec, preStartArchives: [restoreArchive] }; + yield* legacyCreateContainer(spawner, spec, args.containerOpts); + yield* args.waitReady; + }); + + if (args.plan.kind !== "warm") { + yield* bringUp(undefined); + return args.plan.state; + } + const plan = args.plan; + + // Container AND volume, in that order: the tar was unpacked INTO the named PGDATA volume, and + // `docker rm -v` does not remove a NAMED one — see this function's own doc comment. + const discardRestored = Effect.gen(function* () { + yield* legacyRemoveContainer(spawner, args.dbContainerId); + yield* legacyRemoveVolume(spawner, args.dbContainerId); + }); + + const restored = yield* legacyTimeShadowPhase( + "baseline-restore", + bringUp(plan.restoreArchive), + ).pipe( + // The same guard `legacyWarmShadow` puts on its own restored cluster (`shadow-cache.ts`). + // `onInterrupt` is the substantive half here: a Ctrl-C landing anywhere in the restore + // must not stand the run down leaving a half-restored VOLUME behind, because the next + // `supabase start` would find that volume already present, take it for a provisioned one, + // skip the fresh-volume setup entirely, and run against a cluster whose baseline never + // finished landing. Best-effort — an interrupt is already the caller's answer, so a failed + // teardown has nowhere to go (the shadow's own remove is non-failing for the same reason). + // `interruptible` matches the shadow's own restore and keeps the guarantee independent of + // whether a future caller wraps this bring-up in an uninterruptible acquire, where the + // readiness wait would otherwise pin that Ctrl-C for the whole health-timeout budget. + Effect.onInterrupt(() => + Effect.gen(function* () { + yield* Effect.ignore(legacyRemoveContainer(spawner, args.dbContainerId)); + yield* Effect.ignore(legacyRemoveVolume(spawner, args.dbContainerId)); + }), + ), + Effect.interruptible, + Effect.result, + ); + if (!Result.isFailure(restored)) return plan.state; + + const cause = restored.failure; + const output = yield* Output; + yield* output.raw( + `Warning: cached database baseline unusable (${cause.message}); recreating.\n`, + "stderr", + ); + // The ONE failure that implicates the tar's CONTENTS: it extracted cleanly, the container + // started, and the cluster never answered. + if (cause._tag === "LegacyHealthCheckTimeoutError") { + yield* legacyForgetShadowBaselineTar(args.fs, plan.tarPath); + } + yield* discardRestored; + yield* bringUp(undefined); + return plan.coldFallbackState; + }); + +/** + * The whole cache-aware bring-up of the local `db` container, as ONE call: decide (peek the tar + * pool), create + start, wait with the caller's own gate, fall back cold if a warm restore does + * not come up, and resolve to the baseline state `legacyRunFreshDbSetup` should run with. Both + * compositions that create this container (`start-database.ts`'s `legacyStartDatabase` and + * `recreate-local-database.ts`'s `legacyRecreateLocalDatabase15`) ran the identical five-call + * sequence before this existed; everything that genuinely differs between them is a parameter + * here — {@link cacheEligible}, the {@link waitReady} policy, and the container spec. + * + * Slots into both callers exactly where `docker create` used to be, because the decision has to + * land BEFORE it: a warm restore is a `docker cp -` into the created-but-unstarted container. + */ +export const legacyBringUpMainDbWithBaseline = ( + spawner: Spawner, + args: { + readonly fs: FileSystem.FileSystem; + readonly path: Path.Path; + readonly workdir: string; + /** `localDbContainerId(projectId)` — the container's name and its PGDATA volume's name. */ + readonly dbContainerId: string; + readonly hostname: string; + readonly dbPort: number; + readonly healthTimeoutSeconds: number; + /** The already registry-resolved `supabase/postgres` image the container runs. */ + readonly image: string; + /** The container-spec fields the cluster's own identity depends on. */ + readonly postgresSpec: Omit; + readonly setup: LegacyFreshDbSetupInput; + /** The caller's own already-validated `[db]` values — see {@link LegacyMainDbBaselineTomlInputs}. */ + readonly toml: LegacyMainDbBaselineTomlInputs; + /** The built, undecorated Postgres container spec — the warm path splices the archive on. */ + readonly spec: LegacyStartContainerSpec; + readonly containerOpts: LegacyContainerOpts; + /** + * Whether this run provisions a platform baseline at all: only such a run can restore or + * publish one. `false` for `db start --from-backup` (which owns `preStartArchives` itself and + * skips the whole baseline) and for every non-fresh-volume start; always `true` for `db + * reset`'s PG15 recreate, which just removed the volume. + */ + readonly cacheEligible: boolean; + /** + * The caller's own readiness gate, already carrying that caller's semantics (`db start + * --from-backup` swallows a timeout; `db reset` never does). + */ + readonly waitReady: Effect.Effect; + }, +): Effect.Effect< + LegacyMainDbBaselineState, + | E + | LegacyContainerError + | LegacyHealthCheckTimeoutError + | LegacyContainerRemoveError + | LegacyVolumeRemoveError, + Output | R +> => + Effect.gen(function* () { + const plan = args.cacheEligible + ? yield* legacyPlanMainDbBaseline(spawner, legacyMainDbBaselineInput(args), args.toml) + : LEGACY_MAIN_DB_BASELINE_UNCACHED; + return yield* legacyCreateMainDbWithBaseline(spawner, { + plan, + spec: args.spec, + containerOpts: args.containerOpts, + dbContainerId: args.dbContainerId, + fs: args.fs, + waitReady: args.waitReady, + }); + }); + +/** Adapts the main-db bring-up inputs into the shared, structurally-keyed cache input. */ +const legacyMainDbCacheInput = ( + input: LegacyMainDbBaselineInput, + toml: LegacyMainDbBaselineTomlInputs, +): LegacyBaselineCacheInput => ({ + fs: input.fs, + path: input.path, + workdir: input.workdir, + image: input.image, + db: input.postgresSpec.db, + experimental: input.postgresSpec.experimental, + jwtSecret: input.postgresSpec.jwtSecret, + jwtExpiry: input.postgresSpec.jwtExpiry, + ...(input.postgresSpec.rootKey === undefined ? {} : { rootKey: input.postgresSpec.rootKey }), + // Read off the connect config rather than re-derived, so the key stays pinned to the password + // the run actually authenticates with — see {@link LegacyMainDbBaselineInput.connConfig}. + password: input.connConfig.password, + setup: { + majorVersion: input.setup.majorVersion, + config: input.setup.config, + jwks: input.setup.jwks, + storageTargetMigration: input.setup.storageTargetMigration, + serviceVersionOverrides: input.setup.serviceVersionOverrides, + projectEnvValues: input.setup.projectEnvValues, + webhooksEnabled: toml.webhooksEnabled, + apiAutoExposeNewTables: toml.apiAutoExposeNewTables, + vault: toml.vault, + }, +}); + +/** The local `db` container's binding of {@link legacyExportBaselineSnapshot}. */ +const legacyExportMainDbBaseline = ( + spawner: Spawner, + input: LegacyMainDbBaselineInput, + key: string, + tarPath: string, + skipIfPublished: boolean, +): Effect.Effect => + legacyExportBaselineSnapshot(spawner, input, { + containerId: input.dbContainerId, + tarPath, + key, + skipIfPublished, + connConfig: input.connConfig, + healthTimeoutSeconds: input.healthTimeoutSeconds, + image: input.image, + warnLabel: "database", + clusterLabel: "local database", + readyLabel: "re-started database", + }).pipe( + Effect.mapError( + (cause) => new LegacyDbSetupError({ message: cause.reason, reason: "docker_daemon" }), + ), + ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts index a6f79c3c30..5bc0192f17 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts @@ -8,7 +8,7 @@ * different composition over the SAME underlying primitives * (`legacyEnsureNetwork`, `legacyBuildPostgresStartContainerSpec`, * `legacyCreateContainer`, `legacyWaitForHealthyServices`, - * `legacyStartSetupLocalDatabase`), matching Go's own structure: + * `legacyRunFreshDbSetup`), matching Go's own structure: * * **PG >= 15** (`resetDatabase15`, `reset.go:114-142`): * 1. `docker container rm -f ` — NOT tolerant of "not found" (a genuine @@ -27,7 +27,7 @@ * `legacyBuildPostgresStartContainerSpec`/`legacyCreateContainer`. * 6. Health wait — NEVER swallowed (no `--from-backup`-equivalent gate here at * all). - * 7. `legacyStartSetupLocalDatabase` — UNCONDITIONALLY (no fresh-volume gate: a + * 7. `legacyRunFreshDbSetup` — UNCONDITIONALLY (no fresh-volume gate: a * reset just removed the volume, so it's always fresh) and with the * RESOLVED reset `version`/`seedFlags` (not `""` like `db start`'s own call) * — see `db-setup.ts`'s own header for this one genuine parameter @@ -61,7 +61,7 @@ * the PG14 path too, after the db container restart. * 4. Final connect as `postgres`/`postgres` → `apply.MigrateAndSeed` with the * resolved reset `version`/`seedFlags` — the same seed-override logic PG15's - * `legacyStartSetupLocalDatabase` call applies. + * `legacyRunFreshDbSetup` call applies. * * Deliberately absent, matching Go exactly: no volume-existence probe (a reset * just removed the volume, so there is nothing to probe), no `NoBackupVolume`/ @@ -70,9 +70,19 @@ * call it), and no rollback on failure (Go's `cmd/db.go` only wraps `--mode * start` in a `DockerRemoveAll` cleanup — the recreate dispatch has none). * + * Steps 5-7 on the PG15 path additionally run through the baseline PGDATA cache + * (`main-db-baseline.ts`) — a TS-only addition with no Go counterpart, and the one + * place this composition diverges from `resetDatabase15`'s literal shape. It decides + * warm/cold BEFORE the container is created (a warm restore is a `docker cp -` into + * the created-but-unstarted container), owns the create + health wait so a failed + * restore can fall back to a cold one, and hands `legacyRunFreshDbSetup` the baseline + * state that decides whether the platform baseline is re-run or snapshotted. With + * `SUPABASE_SHADOW_CACHE=false` — and on the PG<=14 path, which never consults it at + * all — every step above is byte-for-byte what it was. + * * `pgcache.TryCacheMigrationsCatalog`'s best-effort catalog warmup (part of Go's * `SetupLocalDatabase`, reachable from the PG15 path above via - * `legacyStartSetupLocalDatabase`) IS reached here too — see `db-setup.ts`'s own + * `legacyRunFreshDbSetup`) IS reached here too — see `db-setup.ts`'s own * header for the exact gate/citations. `reset.layers.ts` composes * `legacyEdgeRuntimeScriptLayer`/`legacyPgDeltaSslProbeLayer` for it, matching * `db start`'s own layer composition (`db/start/start.layers.ts`). @@ -108,7 +118,6 @@ import { legacyEnsureNetwork, legacyRemoveContainer, legacyRemoveVolume, - legacyCreateContainer, LEGACY_COMPOSE_PROJECT_LABEL, type LegacyContainerRemoveError, type LegacyContainerError, @@ -133,6 +142,10 @@ import { } from "./health-check.ts"; import type { LegacyImagePrepullError } from "./image-prepull.ts"; import { legacyStartInternalDbPassword } from "./internal-db-connection.ts"; +import { + legacyBringUpMainDbWithBaseline, + type LegacyMainDbBaselineTomlInputs, +} from "./main-db-baseline.ts"; import { legacyBuildPostgresStartContainerSpec, type LegacyPostgresStartServiceInput, @@ -212,6 +225,13 @@ export interface LegacyRecreateLocalDatabaseInput { /** Lazy — evaluated right where Go's `DockerStart` would resolve it (PG15 path only). */ readonly resolvePostgresImage: Effect.Effect; readonly dbHealthTimeoutSeconds: number; + /** + * The `[db]` values from the caller's OWN already-validated `legacyCheckDbToml` pass — they + * describe the baseline the cache keys the PG15 recreate on (see + * {@link LegacyMainDbBaselineTomlInputs}). The PG14 branch below loads config itself, for the + * many other fields it needs. + */ + readonly toml: LegacyMainDbBaselineTomlInputs; /** The resolved reset migration version (`""` for every pending migration). */ readonly version: string; /** `db reset`'s `--no-seed`/`--sql-paths` — see {@link legacyResolveResetSeedConfig}. */ @@ -387,30 +407,53 @@ const legacyRecreateLocalDatabase15 = ( ...input.postgresSpec, image: resolvedPostgresImage, }); - yield* legacyCreateContainer(spawner, postgresSpec, input.containerOpts); - - // Never swallowed — reset has no `--from-backup`-equivalent gate at all. - yield* legacyWaitForHealthyServices(spawner, [postgresSpec.containerName], { - timeoutSeconds: input.dbHealthTimeoutSeconds, - images: new Map([[postgresSpec.containerName, resolvedPostgresImage]]), - }); - // UNCONDITIONAL — no fresh-volume gate: a reset just removed the volume above, so - // it's always fresh. Passes the RESOLVED reset `version`/`seedFlags`, unlike `db - // start`'s own call — see `db-setup.ts`'s header for this one real difference. - yield* legacyRunFreshDbSetup(spawner, { + // The baseline cache's decision has to land BEFORE `docker create`: a warm restore is a + // `docker cp -` into the created-but-unstarted container, so the whole create + health wait + // belongs to it. Always eligible: a reset just removed the volume, so this run always + // provisions the platform baseline. See `main-db-baseline.ts`. + const baseline = yield* legacyBringUpMainDbWithBaseline(spawner, { fs: input.fs, path: input.path, workdir: input.workdir, - projectId: input.projectId, - networkId: input.networkId, + dbContainerId: input.dbContainerId, hostname: input.hostname, dbPort: input.dbPort, - version: input.version, - seedFlags: input.seedFlags, + healthTimeoutSeconds: input.dbHealthTimeoutSeconds, + image: resolvedPostgresImage, + postgresSpec: input.postgresSpec, setup: input.setup, + toml: input.toml, + spec: postgresSpec, + containerOpts: input.containerOpts, + cacheEligible: true, + // Never swallowed — reset has no `--from-backup`-equivalent gate at all. + waitReady: legacyWaitForHealthyServices(spawner, [postgresSpec.containerName], { + timeoutSeconds: input.dbHealthTimeoutSeconds, + images: new Map([[postgresSpec.containerName, resolvedPostgresImage]]), + }), }); + // UNCONDITIONAL — no fresh-volume gate: a reset just removed the volume above, so + // it's always fresh. Passes the RESOLVED reset `version`/`seedFlags`, unlike `db + // start`'s own call — see `db-setup.ts`'s header for this one real difference. + yield* legacyRunFreshDbSetup( + spawner, + { + fs: input.fs, + path: input.path, + workdir: input.workdir, + projectId: input.projectId, + networkId: input.networkId, + hostname: input.hostname, + dbPort: input.dbPort, + version: input.version, + seedFlags: input.seedFlags, + setup: input.setup, + }, + baseline, + ); + yield* output.raw("Restarting containers...\n", "stderr"); yield* legacyRestartServicesAndReloadKong(spawner, input.projectId); }); @@ -421,7 +464,7 @@ const legacyRecreateLocalDatabase15 = ( * `initDatabase` (needs `api.auto_expose_new_tables`) and the final * `MigrateAndSeed` (needs `db.migrations.enabled`/`[db.seed]`/pg-delta gate) — * the same "each caller re-loads its own config" duplication `db start`'s own - * handler and `legacyStartSetupLocalDatabase` both already take independently. + * handler and `legacyRunFreshDbSetup` both already take independently. */ const legacyRecreateLocalDatabase14 = ( spawner: Spawner, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/reset-local-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/reset-local-database.ts index 43ee93b2eb..c486fe8009 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/reset-local-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/reset-local-database.ts @@ -132,7 +132,7 @@ export const legacyResetLocalDatabase = Effect.fnUntraced(function* ( // (`internal/db/reset/reset.go:57-61`). Re-validate here as an explicit, independent gate // (the same pattern `db start`/`db push` use), so "a malformed config aborts before the // local database is recreated" is enforced by this function directly. - yield* legacyCheckDbToml(fs, path, workdir); + const dbTomlValues = yield* legacyCheckDbToml(fs, path, workdir); // AssertSupabaseDbIsRunning — error if the local db container is down. const running = yield* legacyIsLocalDbRunning( @@ -190,6 +190,13 @@ export const legacyResetLocalDatabase = Effect.fnUntraced(function* ( postgresSpec: postgresSpecBase, resolvePostgresImage, dbHealthTimeoutSeconds: bootstrapConfig.dbHealthTimeoutSeconds, + // The `[db]` values this function's own validating load above already resolved — what the + // PG15 recreate's baseline cache keys on, rather than a third silent `config.toml` pass. + toml: { + webhooksEnabled: dbTomlValues.webhooksEnabled, + apiAutoExposeNewTables: dbTomlValues.baseline.apiAutoExposeNewTables, + vault: dbTomlValues.vault, + }, version: input.version, seedFlags: input.seedFlags, // `db reset` resolves `--experimental` EARLIER than this prelude (it gates the diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 4e90ae87c3..337c315c20 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -42,6 +42,7 @@ import { LEGACY_SHADOW_CACHE_ENV, legacyAcquireShadowDatabase, legacyPeekShadowBaseline, + type LegacyShadowAcquiredHandle, type LegacyShadowCacheOpts, } from "./shadow-cache.ts"; import { LEGACY_SHADOW_DEBUG_ENV } from "./shadow-debug.ts"; @@ -186,6 +187,14 @@ const soleTarName = Effect.fnUntraced(function* (fs: FileSystem.FileSystem, path return entries.filter((entry) => entry.endsWith(".tar")); }); +/** + * The snapshot step of whichever handle this is — the export on a cache-enabled cold acquire, and + * nothing at all on a warm or uncached one, which is exactly what the real consumer + * (`legacyOpenBaselineSession`) does with the same three states. + */ +const snapshotOf = (handle: LegacyShadowAcquiredHandle) => + handle._tag === "cold" ? handle.snapshotBaseline : Effect.void; + /** The cache key a published snapshot's filename (`shadow-baseline-.tar`) is stored under. */ const keyOf = (tarName: string) => tarName.slice("shadow-baseline-".length, -".tar".length); @@ -205,7 +214,7 @@ const coldRun = ( ) => Effect.gen(function* () { const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input, opts); - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); yield* legacyRemoveShadowDatabase(docker.spawner, handle.containerId); return handle; }); @@ -222,12 +231,12 @@ describe("legacyAcquireShadowDatabase", () => { const path = yield* Path.Path; const input = shadowInput(fs, path); const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); - expect(handle.baselinePresent).toBe(false); + expect(handle._tag).toBe("uncached"); // `--rm` intact, no PGDATA copies either way, and the snapshot step is a no-op. The one // `cp-secret` is the pgsodium root key every shadow has always been given. expect(docker.calls("create")[0] ?? []).toContain("--rm"); - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); expect(docker.steps()).toEqual(["create", "cp-secret", "start"]); // Nothing is written to disk at all. expect(yield* soleTarName(fs, path)).toEqual([]); @@ -255,12 +264,12 @@ describe("legacyAcquireShadowDatabase", () => { const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input, { bypassCache: true, }); - expect(handle.baselinePresent).toBe(false); + expect(handle._tag).toBe("uncached"); // No restore in, no export out: the bypassed run neither reads nor rewrites the tar. const bypassSteps = docker.steps().slice(docker.steps().lastIndexOf("create")); expect(bypassSteps).toEqual(["create", "cp-secret", "start"]); expect(docker.calls("create").at(-1) ?? []).toContain("--rm"); - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); expect(docker.stepCalls("cp-out")).toHaveLength(1); // the initial cold run's only }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); @@ -285,10 +294,9 @@ describe("legacyAcquireShadowDatabase", () => { setup: { ...base.setup, majorVersion: 14 }, }; const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); - expect(handle.baselinePresent).toBe(false); - expect(handle.snapshotRequired).toBe(false); + expect(handle._tag).toBe("uncached"); expect(docker.calls("create")[0] ?? []).toContain("--rm"); - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); expect(yield* soleTarName(fs, path)).toEqual([]); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); @@ -317,7 +325,7 @@ describe("legacyAcquireShadowDatabase", () => { yield* fs.utimes(abandoned, twoHoursAgo, twoHoursAgo); const warm = yield* legacyAcquireShadowDatabase(docker.spawner, input); - expect(warm.baselinePresent).toBe(true); + expect(warm._tag).toBe("warm"); expect(yield* fs.exists(abandoned)).toBe(false); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); @@ -339,9 +347,9 @@ describe("legacyAcquireShadowDatabase", () => { experimental: { ...defaultConfig.experimental, orioledb_version: "15" }, }; const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); - expect(handle.baselinePresent).toBe(false); + expect(handle._tag).toBe("uncached"); expect(docker.calls("create")[0] ?? []).toContain("--rm"); - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); expect(docker.calls("stop")).toEqual([]); expect(yield* soleTarName(fs, path)).toEqual([]); }), @@ -358,7 +366,7 @@ describe("legacyAcquireShadowDatabase", () => { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); expect(yield* soleTarName(fs, path)).toHaveLength(1); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); @@ -375,11 +383,11 @@ describe("legacyAcquireShadowDatabase", () => { const path = yield* Path.Path; const input = shadowInput(fs, path); const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); - expect(handle.baselinePresent).toBe(false); + expect(handle._tag).toBe("cold"); // The cold container must survive its own `docker stop`, so it carries no `--rm`. expect(docker.calls("create")[0] ?? []).not.toContain("--rm"); - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); // Ordering IS the contract here: stop before the copy (a live PGDATA is not coherent to // copy), the baseline marker stamped in between (it must be the LAST thing written to @@ -405,7 +413,6 @@ describe("legacyAcquireShadowDatabase", () => { // this run's own cache key, which is what binds the artifact to the name it is filed under. const stamp = docker.containers.get(handle.containerId)?.stamp ?? ""; expect(stamp).toContain(LEGACY_PGDATA_BASELINE_MARKER_NAME); - expect(stamp).toContain(legacyPgDataBaselineMarkerContent(handle.snapshotKey ?? "")); expect(docker.stepCalls("cp-out")[0]).toEqual([ "cp", `${handle.containerId}:${LEGACY_PGDATA_PATH}`, @@ -420,7 +427,9 @@ describe("legacyAcquireShadowDatabase", () => { expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); const published = yield* fs.readFileString(path.join(shadowCacheDir(path), tars[0] ?? "")); expect(published).toBe(expectedTarFor(tars[0] ?? "")); - expect(keyOf(tars[0] ?? "")).toBe(handle.snapshotKey); + // The stamp's content is the published filename's own key — the binding the next run's + // pre-restore scan verifies. + expect(stamp).toContain(legacyPgDataBaselineMarkerContent(keyOf(tars[0] ?? ""))); // The stamp made it all the way into the artifact — this is the entry the next run's // pre-restore scan requires, so a cold export that skipped it would never warm anything. expect(published).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); @@ -449,7 +458,7 @@ describe("legacyAcquireShadowDatabase", () => { const warm = yield* legacyAcquireShadowDatabase(docker.spawner, input); // A brand new container every time — the cache keeps a file, never a container. expect(warm.containerId).not.toBe(cold.containerId); - expect(warm.baselinePresent).toBe(true); + expect(warm._tag).toBe("warm"); // Throwaway again: the warm container never gets stopped, so `--rm` is back. expect(docker.calls("create").at(-1) ?? []).toContain("--rm"); @@ -467,7 +476,7 @@ describe("legacyAcquireShadowDatabase", () => { ); // Nothing more is exported: the baseline is already on disk. - yield* warm.snapshotBaseline; + yield* snapshotOf(warm); expect(docker.calls("stop")).toHaveLength(1); expect(yield* soleTarName(fs, path)).toHaveLength(1); }), @@ -560,10 +569,10 @@ describe("legacyAcquireShadowDatabase", () => { ], { concurrency: 2 }, ); - expect(first.baselinePresent).toBe(false); - expect(second.baselinePresent).toBe(false); + expect(first._tag).toBe("cold"); + expect(second._tag).toBe("cold"); - yield* Effect.all([first.snapshotBaseline, second.snapshotBaseline], { concurrency: 2 }); + yield* Effect.all([snapshotOf(first), snapshotOf(second)], { concurrency: 2 }); // One export, one tar with the exported bytes intact, no leftover partials. expect(docker.stepCalls("cp-out")).toHaveLength(1); @@ -607,7 +616,7 @@ describe("legacyAcquireShadowDatabase", () => { const both = yield* soleTarName(fs, path); expect(both).toHaveLength(2); expect(both).toContain(first[0]); - expect(rekeyed.baselinePresent).toBe(false); + expect(rekeyed._tag).toBe("cold"); // An unrelated file in the cache directory is untouched by retention. const stray = path.join(shadowCacheDir(path), "catalog-abc.json"); @@ -647,7 +656,7 @@ describe("legacyAcquireShadowDatabase", () => { yield* fs.makeDirectory(path.join(worktreeB, "supabase"), { recursive: true }); const cold = yield* coldRun(docker, { ...shadowInput(fs, path), workdir: worktreeA }); - expect(cold.baselinePresent).toBe(false); + expect(cold._tag).toBe("cold"); expect(yield* soleTarName(fs, path)).toHaveLength(1); // Same settings, different project path — the second worktree must restore, not re-export. @@ -655,7 +664,7 @@ describe("legacyAcquireShadowDatabase", () => { ...shadowInput(fs, path), workdir: worktreeB, }); - expect(warm.baselinePresent).toBe(true); + expect(warm._tag).toBe("warm"); expect(warm.containerId).not.toBe(cold.containerId); expect(yield* soleTarName(fs, path)).toHaveLength(1); yield* legacyRemoveShadowDatabase(docker.spawner, warm.containerId); @@ -673,7 +682,7 @@ describe("legacyAcquireShadowDatabase", () => { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; const cold = yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54320 })); - expect(cold.baselinePresent).toBe(false); + expect(cold._tag).toBe("cold"); expect(yield* soleTarName(fs, path)).toHaveLength(1); // pg-delta next allocates an ephemeral host port per shadow; the published port is @@ -682,7 +691,7 @@ describe("legacyAcquireShadowDatabase", () => { docker.spawner, shadowInput(fs, path, { shadowPort: 54399 }), ); - expect(warm.baselinePresent).toBe(true); + expect(warm._tag).toBe("warm"); expect(yield* soleTarName(fs, path)).toHaveLength(1); yield* legacyRemoveShadowDatabase(docker.spawner, warm.containerId); }), @@ -706,17 +715,17 @@ describe("legacyAcquireShadowDatabase", () => { expect(yield* soleTarName(fs, path)).toHaveLength(1); const forcedOn = yield* coldRun(docker, input, { webhooks: "enabled" }); - expect(forcedOn.baselinePresent).toBe(false); + expect(forcedOn._tag).toBe("cold"); expect(yield* soleTarName(fs, path)).toHaveLength(2); const warmConfig = yield* legacyAcquireShadowDatabase(docker.spawner, input, { webhooks: "config", }); - expect(warmConfig.baselinePresent).toBe(true); + expect(warmConfig._tag).toBe("warm"); const warmDisabled = yield* legacyAcquireShadowDatabase(docker.spawner, input, { webhooks: "disabled", }); - expect(warmDisabled.baselinePresent).toBe(true); + expect(warmDisabled._tag).toBe("warm"); expect(yield* soleTarName(fs, path)).toHaveLength(2); yield* legacyRemoveShadowDatabase(docker.spawner, warmConfig.containerId); yield* legacyRemoveShadowDatabase(docker.spawner, warmDisabled.containerId); @@ -746,7 +755,7 @@ describe("legacyAcquireShadowDatabase", () => { "mirror.internal.example", coldRun(docker, input), ); - expect(mirrored.baselinePresent).toBe(false); + expect(mirrored._tag).toBe("cold"); const mirroredTar = yield* soleTarName(fs, path); // Distinct keys coexist in the global cache — the default-registry tar is not swept. expect(mirroredTar).toHaveLength(2); @@ -772,7 +781,7 @@ describe("legacyAcquireShadowDatabase", () => { // Reporting success here would send the caller's next connect to a dead container's port // — possibly answered by a DIFFERENT Postgres by then — so this must be a failure, not a // "not cached" warning. - const exit = yield* handle.snapshotBaseline.pipe(Effect.exit); + const exit = yield* snapshotOf(handle).pipe(Effect.exit); expect(Exit.isFailure(exit)).toBe(true); expect(out.stderrText).not.toContain("Warning: shadow baseline not cached"); }), @@ -791,7 +800,7 @@ describe("legacyAcquireShadowDatabase", () => { const path = yield* Path.Path; const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); // The run itself must never fail for a cache problem. - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); expect(out.stderrText).toContain("Warning: shadow baseline not cached"); // The caller is about to reconnect, so the container is running again regardless. @@ -825,7 +834,7 @@ describe("legacyAcquireShadowDatabase", () => { const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); expect(out.stderrText).toContain("cached shadow baseline unusable"); // Falls all the way back to a cold provision — a fresh container with no baseline. - expect(fallback.baselinePresent).toBe(false); + expect(fallback._tag).toBe("cold"); expect(fallback.containerId).not.toBe(cold.containerId); // The container whose restore failed is removed, not orphaned: with the cold run's own // container already released by `coldRun`, only the fallback's remains. @@ -836,7 +845,7 @@ describe("legacyAcquireShadowDatabase", () => { // ...and the cold fallback's own export atomically REPLACES it rather than treating the // retained-but-unusable tar as a sibling's fresh publish and dedupe-skipping — the // warm-fallback cold path must pass `skipIfPublished: false` (review: Codex on #6215). - yield* fallback.snapshotBaseline; + yield* snapshotOf(fallback); expect(docker.stepCalls("cp-out")).toHaveLength(2); expect(yield* soleTarName(fs, path)).toHaveLength(1); expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); @@ -870,13 +879,13 @@ describe("legacyAcquireShadowDatabase", () => { expect(out.stderrText).toContain("cached shadow baseline unusable"); expect(out.stderrText).toContain("data/PG_VERSION"); - expect(fallback.baselinePresent).toBe(false); + expect(fallback._tag).toBe("cold"); // Caught before any container was created, so nothing was ever restored. expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); // The contents ARE the problem, so the tar goes — and the cold fallback republishes a // good one within the same run, which is what keeps this fail-open. expect(yield* soleTarName(fs, path)).toEqual([]); - yield* fallback.snapshotBaseline; + yield* snapshotOf(fallback); expect(yield* soleTarName(fs, path)).toHaveLength(1); expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); @@ -910,13 +919,13 @@ describe("legacyAcquireShadowDatabase", () => { expect(out.stderrText).toContain("cached shadow baseline unusable"); expect(out.stderrText).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); - expect(fallback.baselinePresent).toBe(false); + expect(fallback._tag).toBe("cold"); // Caught before any container was created, so nothing was ever restored. expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); // The contents ARE the problem, so the tar goes — and the cold fallback republishes a // marked one within the same run. expect(yield* soleTarName(fs, path)).toEqual([]); - yield* fallback.snapshotBaseline; + yield* snapshotOf(fallback); expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); }), @@ -936,11 +945,13 @@ describe("legacyAcquireShadowDatabase", () => { const path = yield* Path.Path; const inputA = shadowInput(fs, path, { jwtExpiry: 3600 }); const inputB = shadowInput(fs, path, { jwtExpiry: 7200 }); - // Two genuinely different configurations, each with its own published snapshot. - const coldA = yield* coldRun(docker, inputA); - const coldB = yield* coldRun(docker, inputB); - const keyA = coldA.snapshotKey ?? ""; - const keyB = coldB.snapshotKey ?? ""; + // Two genuinely different configurations, each with its own published snapshot. The + // handle no longer carries its key, so each run's key is read off the tar it filed. + yield* coldRun(docker, inputA); + const [tarA = ""] = yield* soleTarName(fs, path); + const keyA = keyOf(tarA); + yield* coldRun(docker, inputB); + const keyB = (yield* soleTarName(fs, path)).map(keyOf).find((k) => k !== keyA) ?? ""; expect(keyA).not.toBe(keyB); const tarPathB = path.join(shadowCacheDir(path), `shadow-baseline-${keyB}.tar`); const tarPathA = path.join(shadowCacheDir(path), `shadow-baseline-${keyA}.tar`); @@ -957,7 +968,7 @@ describe("legacyAcquireShadowDatabase", () => { expect(out.stderrText).toContain("cached shadow baseline unusable"); expect(out.stderrText).toContain(`snapshot is stamped with key ${keyA}, not ${keyB}`); - expect(fallback.baselinePresent).toBe(false); + expect(fallback._tag).toBe("cold"); // Caught before any container was created, so nothing was ever restored. expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); // Only the MISNAMED COPY goes: nothing else can ever be filed under B's name, while A's @@ -967,7 +978,7 @@ describe("legacyAcquireShadowDatabase", () => { expectedTarFor(`shadow-baseline-${keyA}.tar`), ); // ...and the cold fallback republishes B's real baseline within the same run. - yield* fallback.snapshotBaseline; + yield* snapshotOf(fallback); expect(yield* fs.readFileString(tarPathB)).toBe( expectedTarFor(`shadow-baseline-${keyB}.tar`), ); @@ -988,7 +999,7 @@ describe("legacyAcquireShadowDatabase", () => { const path = yield* Path.Path; const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); // Same fail-open contract as every other export failure: the run itself never fails... - yield* handle.snapshotBaseline; + yield* snapshotOf(handle); expect(out.stderrText).toContain("Warning: shadow baseline not cached"); expect(out.stderrText).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); @@ -1023,7 +1034,7 @@ describe("legacyAcquireShadowDatabase", () => { Effect.provide(broken.layer), ); - expect(fallback.baselinePresent).toBe(false); + expect(fallback._tag).toBe("cold"); expect(fallback.containerId).not.toBe(cold.containerId); // The suspect container is gone before the replacement is created — it holds the shadow's // published port. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 4024bafd4a..c62a2c04ba 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -1,9 +1,17 @@ /** - * Shadow baseline cache — the acquire/release pair `db diff`/`db pull`/the migrations-catalog + * Baseline cache — the acquire/release pair `db diff`/`db pull`/the migrations-catalog * resolution path use in place of bare `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` - * (`shadow-database.ts`). Caches the shadow's platform baseline (init schema + the PG15+ one-shot + * (`shadow-database.ts`). Caches a cluster's platform baseline (init schema + the PG15+ one-shot * realtime/storage/auth jobs) as a disk-level PGDATA snapshot, never a kept container. * + * The tar pool is SHARED with the long-running local `db` container: `main-db-baseline.ts` keys, + * restores, and publishes through this same module — same directory, same `shadow-baseline-` + * names (kept as-is so both producers stay interchangeable), same LRU/TTL sweep. A `db diff` run + * therefore warms the snapshot a later `supabase db reset` restores, and vice versa. Everything + * that genuinely differs between the two clusters is already a key input (the `postgres` password + * in particular: the main container is always initialized with the `"postgres"` literal, the + * shadow with `[db] password`), so a tar is only ever reachable from a cluster it fits. + * * - **Cold** (no snapshot for this key): provision the shadow as an uncached run does, then — * right after the baseline and before `contrib_regression`/any user migration — stop the * container, export its PGDATA via {@link legacyExportPgDataTar} (`pgdata-snapshot.ts`), and @@ -32,10 +40,16 @@ import { createHash } from "node:crypto"; import type { ProjectConfig } from "@supabase/config"; -import { Clock, Effect, Option, Result, Semaphore, type FileSystem } from "effect"; +import { Clock, Data, Effect, Option, Result, Semaphore, type FileSystem, type Path } from "effect"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import { Output } from "../../../shared/output/output.service.ts"; +import type { LocalServiceVersionOverrides } from "../../../shared/services/services.shared.ts"; +import { + actionability, + type CliErrorActionabilityDeclaration, + ErrorActionabilityId, +} from "../../../shared/telemetry/error-actionability.ts"; import { containerCliExitCode, legacyDescribeContainerCliFailure, @@ -179,6 +193,16 @@ export interface LegacyShadowCacheServiceInput { * secrets' values and the versioned `auth`/`storage`/`_realtime` schema those jobs write. This * key mirrors that function's hashing STYLE (sha256 over newline-joined formatted fields) without * reusing it. + * + * **One container-shape difference is deliberately NOT hashed**, and it is the one to re-check + * before adding a baseline step: the shadow bootstraps under `-c max_worker_processes=0` + * (`legacyBuildShadowPostgresContainerSpec`, `postgres.service.ts`) while the long-running local + * `db` container this pool is shared with does not. That is safe today because the flag is + * command-line-only — the entrypoint never persists it into PGDATA, so a tar produced under it + * behaves identically once restored into either cluster — and because no baseline step depends on + * a live background worker. A future step that DOES (waiting on pg_net's queue worker to drain, + * say) would silently produce two different baselines under one key, and must add the worker + * setting to this shape before it lands. */ export interface LegacyShadowCacheKeyInputs { /** The resolved, full `supabase/postgres` image (tag included — a major version is not enough). */ @@ -402,7 +426,50 @@ export function legacyEffectiveShadowWebhooksEnabled( } /** - * Resolves {@link LegacyShadowCacheKeyInputs} from the same run input the shadow container + * The filesystem handles every artifact-side helper in this module needs — nothing cluster- or + * key-specific. Both {@link LegacyBaselineCacheInput} and `main-db-baseline.ts`'s own input + * satisfy it structurally. + */ +export interface LegacyBaselineCacheFiles { + readonly fs: FileSystem.FileSystem; + readonly path: Path.Path; +} + +/** + * Everything the cache key is computed from, as a STRUCTURAL shape rather than the shadow's own + * run input: `LegacyShadowSetupInput` (`shadow-database.ts`) satisfies it as-is, and so does the + * long-running local `db` container's equivalent (`main-db-baseline.ts`), which shares this + * module's tar pool. Deliberately narrow — a field only belongs here if + * {@link legacyResolveShadowCacheKeyInputs} actually reads it, so neither producer is forced to + * invent a value for a field the other one owns (the shadow's own host port, the main + * container's volume name). + */ +export interface LegacyBaselineCacheInput extends LegacyBaselineCacheFiles { + readonly workdir: string; + /** The resolved, registry-rewritten `supabase/postgres` image the cluster runs. */ + readonly image: string; + readonly db: Pick; + readonly experimental: ProjectConfig["experimental"]; + readonly jwtSecret: string; + readonly jwtExpiry: number; + readonly rootKey?: string; + /** The cluster's own `POSTGRES_PASSWORD` — see {@link LegacyShadowCacheKeyInputs.dbPassword}. */ + readonly password: string; + readonly setup: { + readonly majorVersion: number; + readonly config: ProjectConfig; + readonly webhooksEnabled: boolean; + readonly jwks: Effect.Effect; + readonly storageTargetMigration: string; + readonly serviceVersionOverrides: LocalServiceVersionOverrides; + readonly projectEnvValues: Readonly> | undefined; + readonly apiAutoExposeNewTables: Option.Option; + readonly vault: ReadonlyArray; + }; +} + +/** + * Resolves {@link LegacyShadowCacheKeyInputs} from the same run input the cluster * itself is built from, plus `supabase/roles.sql` off disk. The service enabled flags come from * `setup.config` (NOT the `*EnabledForSetup` fields, which only gate JWKS resolution) because * `legacySetupDatabase`'s own one-shot job gates read exactly those config fields. @@ -421,7 +488,7 @@ export function legacyEffectiveShadowWebhooksEnabled( */ const legacyResolveShadowCacheKeyInputs = ( - input: LegacyShadowSetupInput, + input: LegacyBaselineCacheInput, opts: LegacyShadowCacheOpts = {}, ): Effect.Effect, E> => Effect.gen(function* () { @@ -529,6 +596,11 @@ export function legacyShadowBaselineTarFileName(key: string): string { return `${LEGACY_SHADOW_BASELINE_TAR_PREFIX}${key}${LEGACY_SHADOW_BASELINE_TAR_SUFFIX}`; } +/** This key's absolute snapshot path in the global cache directory. */ +export function legacyShadowBaselineTarPath(path: Path.Path, key: string): string { + return path.join(legacyShadowBaselineCacheDir(path), legacyShadowBaselineTarFileName(key)); +} + /** * Whether `fileName` is a published baseline snapshot (`shadow-baseline-.tar`). Pure and * deliberately conservative: only this module's own prefix AND suffix, so partials @@ -579,7 +651,7 @@ export function legacyShadowBaselineTarsToEvict( } /** Best-effort removal — a leftover tar only ever costs disk, never correctness. */ -const legacyForgetShadowBaselineTar = ( +export const legacyForgetShadowBaselineTar = ( fs: FileSystem.FileSystem, filePath: string, ): Effect.Effect => fs.remove(filePath).pipe(Effect.orElseSucceed(() => undefined)); @@ -609,8 +681,8 @@ const LEGACY_SHADOW_PARTIAL_ABANDON_MS = 60 * 60 * 1000; * `.partial` names (review: Codex on #6184). Runs before every cold export and on warm hits (so * orphans cannot accumulate once every later run goes warm) — best-effort throughout. */ -const legacySweepAbandonedShadowBaselinePartials = ( - input: LegacyShadowSetupInput, +const legacySweepAbandonedShadowBaselinePartials = ( + input: LegacyBaselineCacheFiles, ): Effect.Effect => Effect.gen(function* () { const cacheDir = legacyShadowBaselineCacheDir(input.path); @@ -638,9 +710,7 @@ const legacySweepAbandonedShadowBaselinePartials = ( * Best-effort throughout — a snapshot that cannot be swept costs ~90MB of disk, so it must never * fail the export or warm hit that just succeeded. */ -const legacySweepShadowBaselineRetention = ( - input: LegacyShadowSetupInput, -): Effect.Effect => +const legacySweepShadowBaselineRetention = (input: LegacyBaselineCacheFiles): Effect.Effect => Effect.gen(function* () { const cacheDir = legacyShadowBaselineCacheDir(input.path); const names = yield* input.fs @@ -675,6 +745,23 @@ const legacyTouchShadowBaselineTar = ( yield* fs.utimes(tarPath, now, now); }).pipe(Effect.orElseSucceed(() => undefined)); +/** + * The bookkeeping every warm hit runs before restoring: refresh the tar's mtime (so frequently + * used keys survive LRU/TTL), then sweep abandoned partials and over-cap/aged tars — a killed + * concurrent writer's leftover would otherwise persist indefinitely once every later run goes + * warm, since the cold export's own sweep never runs again (review: Codex on #6184). Best-effort + * and cheap throughout. Shared by the shadow acquire below and `main-db-baseline.ts`. + */ +export const legacyRefreshShadowBaselineOnWarmHit = ( + input: LegacyBaselineCacheFiles, + tarPath: string, +): Effect.Effect => + Effect.gen(function* () { + yield* legacyTouchShadowBaselineTar(input.fs, tarPath); + yield* legacySweepAbandonedShadowBaselinePartials(input); + yield* legacySweepShadowBaselineRetention(input); + }); + // --------------------------------------------------------------------------- // Container primitives the cache adds on top of `shadow-database.ts` // --------------------------------------------------------------------------- @@ -719,21 +806,42 @@ const legacyShadowConnConfig = (input: LegacyShadowSetupInput): LegacyPgCo * `legacyWaitForShadowReady` self-instruments (`ready-attempt`/`ready-wait`, `health-check.ts`) * whenever `SUPABASE_SHADOW_DEBUG` is on, so neither call site here needs an extra timing wrapper. */ -const legacyAwaitShadowReady = ( +const legacyAwaitClusterReady = ( spawner: Spawner, - input: LegacyShadowSetupInput, containerId: string, + args: { + readonly connConfig: LegacyPgConnInput; + readonly healthTimeoutSeconds: number; + readonly image: string; + }, what: string, ): Effect.Effect => - legacyWaitForShadowReady(spawner, containerId, legacyShadowConnConfig(input), { - timeoutSeconds: input.healthTimeoutSeconds, - image: input.image, + legacyWaitForShadowReady(spawner, containerId, args.connConfig, { + timeoutSeconds: args.healthTimeoutSeconds, + image: args.image, }).pipe( Effect.mapError((cause) => legacyShadowCacheUnavailable(`${what} never became ready: ${cause.message}`), ), ); +const legacyAwaitShadowReady = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + containerId: string, + what: string, +): Effect.Effect => + legacyAwaitClusterReady( + spawner, + containerId, + { + connConfig: legacyShadowConnConfig(input), + healthTimeoutSeconds: input.healthTimeoutSeconds, + image: input.image, + }, + what, + ); + // --------------------------------------------------------------------------- // Cold export // --------------------------------------------------------------------------- @@ -767,9 +875,9 @@ const legacyShadowExportMutex = Semaphore.makeUnsafe(1); * tar in place forever, failing every later warm restore into another cold provision (review: * Codex on #6215). */ -const legacyWriteShadowBaselineTar = ( +const legacyWriteShadowBaselineTar = ( spawner: Spawner, - input: LegacyShadowSetupInput, + input: LegacyBaselineCacheFiles, tarPath: string, containerId: string, skipIfPublished: boolean, @@ -798,34 +906,73 @@ const legacyWriteShadowBaselineTar = ( }), ); +/** + * The one failure {@link legacyExportBaselineSnapshot} does NOT degrade: the cluster did not come + * back up after its snapshot. Every caller re-maps it into its OWN error vocabulary + * (`LegacyShadowDbError` for the shadow, `LegacyDbSetupError` for the local `db` container) rather + * than letting this module pick one for both, so this class never reaches a command's own error + * rendering — its declaration exists because every error type in this codebase carries one, and + * mirrors what both of those mappings resolve to (`reason: "docker_daemon"`). + */ +export class LegacyBaselineSnapshotRevivalFailure extends Data.TaggedError( + "LegacyBaselineSnapshotRevivalFailure", +)<{ + readonly reason: string; +}> { + get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { + return { ...actionability.dockerNotRunning, fingerprint_suffix: "docker_not_running" }; + } +} + /** * The cold path's snapshot step, run at the baseline/migrations seam — after - * `legacySetupDatabase` and strictly before `contrib_regression` or any user migration, with no - * session open against the shadow ({@link LegacyShadowBaselineState.snapshotBaseline}). + * `legacySetupDatabase` and strictly before `contrib_regression` or any user migration (the + * shadow) / `MigrateAndSeed` (the local `db` container), with no session open against the cluster + * ({@link LegacyShadowBaselineState.snapshotBaseline}). * * `docker stop` -> export -> `docker start` -> readiness wait. The container is stopped because a * live Postgres's PGDATA is not a coherent thing to copy; the stop is fast (~1s) because the * entrypoint `exec`s Postgres, so PID 1 receives the SIGTERM instead of `sh` swallowing it and - * burning the full 10s grace period. + * burning the full 10s grace period — and because a session left open would hold SIGTERM's smart + * shutdown until the grace period expires, forcing a SIGKILL and an unclean snapshot. * * Two failure classes, deliberately NOT one broad catch: a stop/export failure only means this * run stays uncached — it warns and the run continues. A restart/readiness failure is the RUN'S - * problem: the caller is about to reconnect to the shadow's published port, and reporting success + * problem: the caller is about to reconnect to the cluster's published port, and reporting success * over a dead container would make that connect a blind dial — which can even reach a DIFFERENT * Postgres that claimed the port while the container was down (matching default credentials are * common locally), applying the template + migrations to the wrong database. So the restart runs * whether the export succeeded or not (`docker start` on an already-running container — e.g. when * the stop itself failed — is a no-op success), and its failure PROPAGATES as a - * {@link LegacyShadowDbError} instead of degrading (review: Codex on #6184). + * {@link LegacyBaselineSnapshotRevivalFailure} instead of degrading (review: Codex on #6184). + * + * The readiness gate is {@link legacyWaitForShadowReady} (a direct connect probe) for BOTH + * clusters, never the Docker health gate the long-running `db` container's own bring-up uses: that + * healthcheck has a 10-second interval and no start period, so waiting on it here would add ~6.5s + * to every cold run for a verdict Postgres can already give. */ -const legacyExportShadowBaseline = ( +export const legacyExportBaselineSnapshot = ( spawner: Spawner, - input: LegacyShadowSetupInput, - key: string, - tarPath: string, - containerId: string, - skipIfPublished: boolean, -): Effect.Effect => + input: LegacyBaselineCacheFiles, + args: { + readonly containerId: string; + readonly tarPath: string; + /** The resolved cache key — stamped into PGDATA right before the copy-out. */ + readonly key: string; + readonly skipIfPublished: boolean; + /** The cluster's own host-facing connect target, for the post-restart readiness probe. */ + readonly connConfig: LegacyPgConnInput; + readonly healthTimeoutSeconds: number; + /** The cluster's resolved image, named in the readiness gate's exec-format recovery hint. */ + readonly image: string; + /** How the cache names itself in the degraded-publish warning (`shadow`/`database`). */ + readonly warnLabel: string; + /** How the cluster names itself in a revival failure (`shadow database`/`local database`). */ + readonly clusterLabel: string; + /** How the readiness gate names the restarted cluster (`re-started shadow`/`re-started database`). */ + readonly readyLabel: string; + }, +): Effect.Effect => legacyTimeShadowPhase( "baseline-export", Effect.gen(function* () { @@ -833,18 +980,18 @@ const legacyExportShadowBaseline = ( // which the container is simply still up) only costs this run its snapshot. const exported = yield* Effect.result( Effect.gen(function* () { - yield* legacyShadowContainerVerb(spawner, "stop", containerId); - // The stamp is what makes the published tar mean "the baseline THIS key promises" rather - // than "some PostgreSQL cluster". Two things give it that meaning. Its POSITION in the - // sequence: this whole step runs from `snapshotBaseline`, which `legacySetupShadowDatabase` - // invokes strictly after `legacySetupDatabase` returns (`shadow-database.ts`), and the - // stamp is the last mutation before the copy-out — so a future regression that snapshots - // EARLIER cannot produce a marked tar, it just stays uncached instead of silently - // publishing a bare cluster under a baseline key. And its CONTENT: `key` itself, which - // `legacyWarmShadow` compares against the key it resolved this run, so a valid snapshot - // of a DIFFERENT key that was copied over this filename is rejected too (review: Codex - // on #6184). - yield* legacyStampPgDataBaselineMarker(spawner, containerId, key).pipe( + yield* legacyShadowContainerVerb(spawner, "stop", args.containerId); + // The stamp is what makes the published tar mean "the baseline THIS key promises" + // rather than "some PostgreSQL cluster". Two things give it that meaning. Its POSITION + // in the sequence: this whole step runs from the cold state's `snapshotBaseline`, + // which every provisioning pipeline invokes strictly after the platform baseline is in + // place, and the stamp is the last mutation before the copy-out — so a future + // regression that snapshots EARLIER cannot produce a marked tar, it just stays + // uncached instead of silently publishing a bare cluster under a baseline key. And its + // CONTENT: `key` itself, which the warm restore compares against the key it resolved + // this run, so a valid snapshot of a DIFFERENT key that was copied over this filename + // is rejected too (review: Codex on #6184). + yield* legacyStampPgDataBaselineMarker(spawner, args.containerId, args.key).pipe( Effect.mapError((cause: LegacyPgDataSnapshotUnavailable) => legacyShadowCacheUnavailable(cause.reason), ), @@ -852,37 +999,62 @@ const legacyExportShadowBaseline = ( yield* legacyWriteShadowBaselineTar( spawner, input, - tarPath, - containerId, - skipIfPublished, + args.tarPath, + args.containerId, + args.skipIfPublished, ); }), ); - // Run-critical phase: the shadow must be back up and answering before this step reports + // Run-critical phase: the cluster must be back up and answering before this step reports // success — see this function's own doc comment for why these failures must propagate. const revive = Effect.gen(function* () { - yield* legacyShadowContainerVerb(spawner, "start", containerId); - yield* legacyAwaitShadowReady(spawner, input, containerId, "re-started shadow"); + yield* legacyShadowContainerVerb(spawner, "start", args.containerId); + yield* legacyAwaitClusterReady(spawner, args.containerId, args, args.readyLabel); }); yield* revive.pipe( Effect.mapError( (cause) => - new LegacyShadowDbError({ - message: `shadow database did not come back after the baseline snapshot: ${cause.reason}`, - reason: "docker_daemon", + new LegacyBaselineSnapshotRevivalFailure({ + reason: `${args.clusterLabel} did not come back after the baseline snapshot: ${cause.reason}`, }), ), ); if (Result.isFailure(exported)) { const output = yield* Output; yield* output.raw( - `Warning: shadow baseline not cached: ${exported.failure.reason}\n`, + `Warning: ${args.warnLabel} baseline not cached: ${exported.failure.reason}\n`, "stderr", ); } }), ); +/** The shadow's own binding of {@link legacyExportBaselineSnapshot}, in its error vocabulary. */ +const legacyExportShadowBaseline = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + key: string, + tarPath: string, + containerId: string, + skipIfPublished: boolean, +): Effect.Effect => + legacyExportBaselineSnapshot(spawner, input, { + containerId, + tarPath, + key, + skipIfPublished, + connConfig: legacyShadowConnConfig(input), + healthTimeoutSeconds: input.healthTimeoutSeconds, + image: input.image, + warnLabel: "shadow", + clusterLabel: "shadow database", + readyLabel: "re-started shadow", + }).pipe( + Effect.mapError( + (cause) => new LegacyShadowDbError({ message: cause.reason, reason: "docker_daemon" }), + ), + ); + // --------------------------------------------------------------------------- // Acquire / release // --------------------------------------------------------------------------- @@ -939,7 +1111,7 @@ export type LegacyShadowBaselinePeek = * the same way, so it must not be folded into `uncachable`. */ export const legacyPeekShadowBaseline = ( - input: LegacyShadowSetupInput, + input: LegacyBaselineCacheInput, opts: LegacyShadowCacheOpts = {}, ): Effect.Effect => Effect.gen(function* () { @@ -952,10 +1124,7 @@ export const legacyPeekShadowBaseline = ( const keyInputs = yield* legacyResolveShadowCacheKeyInputs(input, opts); if (Option.isNone(keyInputs)) return { state: "uncachable" } as const; const key = legacyShadowCacheKey(keyInputs.value); - const tarPath = input.path.join( - legacyShadowBaselineCacheDir(input.path), - legacyShadowBaselineTarFileName(key), - ); + const tarPath = legacyShadowBaselineTarPath(input.path, key); const cached = yield* input.fs.exists(tarPath).pipe(Effect.orElseSucceed(() => false)); return { state: cached ? ("warm" as const) : ("cold" as const), @@ -970,18 +1139,9 @@ export const legacyPeekShadowBaseline = ( * baseline is in place. Release needs nothing extra — every shadow this module hands out is * removed the same way an uncached one is. */ -export interface LegacyShadowAcquiredHandle extends LegacyShadowBaselineState { +export type LegacyShadowAcquiredHandle = LegacyShadowBaselineState & { readonly containerId: string; - /** - * The resolved shadow-baseline cache key this handle's cluster is keyed under — present - * exactly when the acquisition was cache-eligible (a cold export or a warm restore), absent - * for an uncached, `bypassCache`d, or uncachable one. Two handles carrying the SAME key share - * the same tar's lineage: one either restored it or exported it this run, so their clusters - * are physical clones of each other. `legacy-pgdelta-next-shadow.layer.ts` reads it to decide - * whether pg-delta's same-database-identity guard must be bypassed for a plan's two shadows. - */ - readonly snapshotKey?: string; -} +}; /** A throwaway shadow with no snapshot step — the cache-off path. */ const legacyUncachedShadow = ( @@ -989,12 +1149,7 @@ const legacyUncachedShadow = ( input: LegacyShadowSetupInput, ): Effect.Effect => legacyCreateShadowDatabase(spawner, input).pipe( - Effect.map(({ containerId }) => ({ - containerId, - baselinePresent: false, - snapshotRequired: false, - snapshotBaseline: Effect.void, - })), + Effect.map(({ containerId }) => ({ containerId, _tag: "uncached" as const })), ); /** @@ -1016,13 +1171,13 @@ const legacyColdCachedShadow = ( key: string, tarPath: string, skipIfPublished: boolean, + rolesSql: string, ): Effect.Effect => legacyCreateShadowDatabase(spawner, { ...input, autoRemove: false }).pipe( Effect.map(({ containerId }) => ({ containerId, - snapshotKey: key, - baselinePresent: false, - snapshotRequired: true, + _tag: "cold" as const, + rolesSql, snapshotBaseline: legacyExportShadowBaseline( spawner, input, @@ -1130,13 +1285,7 @@ const legacyWarmShadow = ( Effect.onInterrupt(() => legacyRemoveShadowDatabase(spawner, containerId)), Effect.interruptible, ); - return { - containerId, - snapshotKey: key, - baselinePresent: true, - snapshotRequired: false, - snapshotBaseline: Effect.void, - } satisfies LegacyShadowAcquiredHandle; + return { containerId, _tag: "warm" } satisfies LegacyShadowAcquiredHandle; }); /** @@ -1194,23 +1343,21 @@ export const legacyAcquireShadowDatabase = ( : yield* Effect.interruptible(legacyResolveShadowCacheKeyInputs(input, opts)); if (Option.isNone(keyInputs)) return yield* legacyUncachedShadow(spawner, input); const key = legacyShadowCacheKey(keyInputs.value); - const tarPath = input.path.join( - legacyShadowBaselineCacheDir(input.path), - legacyShadowBaselineTarFileName(key), - ); + const tarPath = legacyShadowBaselineTarPath(input.path, key); + + // The exact `roles.sql` bytes this key was computed from — carried into the cold provision so + // the seed executes them rather than re-reading the file (see + // {@link LegacyColdBaselineState.rolesSql}). + const rolesSql = keyInputs.value.rolesSql; const cached = yield* input.fs.exists(tarPath).pipe(Effect.orElseSucceed(() => false)); // The tar is absent at acquire time, so a tar found at export time can only be a // same-key sibling's fresh publish — dedupe against it. - if (!cached) return yield* legacyColdCachedShadow(spawner, input, key, tarPath, true); + if (!cached) { + return yield* legacyColdCachedShadow(spawner, input, key, tarPath, true, rolesSql); + } - // Warm hits refresh mtime (so frequently used keys survive LRU/TTL) and sweep abandoned - // partials — a killed concurrent writer's leftover would otherwise persist indefinitely once - // every later run goes warm, since the cold export's own sweep never runs again (review: - // Codex on #6184). Best-effort and cheap. - yield* legacyTouchShadowBaselineTar(input.fs, tarPath); - yield* legacySweepAbandonedShadowBaselinePartials(input); - yield* legacySweepShadowBaselineRetention(input); + yield* legacyRefreshShadowBaselineOnWarmHit(input, tarPath); return yield* legacyWarmShadow(spawner, input, key, tarPath).pipe( Effect.catch((cause) => @@ -1231,7 +1378,7 @@ export const legacyAcquireShadowDatabase = ( // still sitting at this exact path, deliberately retained so this fallback's own // export atomically REPLACES it — skipping because "a tar exists" would leave a // genuinely corrupt one in place forever (review: Codex on #6215). - return yield* legacyColdCachedShadow(spawner, input, key, tarPath, false); + return yield* legacyColdCachedShadow(spawner, input, key, tarPath, false, rolesSql); }), ), ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index f4870358a1..3688bc9ccf 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -67,6 +67,12 @@ import { LEGACY_CLI_PROJECT_LABEL } from "../legacy-docker-ids.ts"; import type { LegacyDockerRun } from "../legacy-docker-run.service.ts"; import { legacyApplyMigrations } from "../legacy-migration-apply.ts"; import type { LegacyVaultSecret } from "../legacy-vault.ts"; +import { + LEGACY_BASELINE_UNCACHED, + legacyBaselineRolesSql, + legacyOpenBaselineSession, + type LegacyClusterBaselineState, +} from "./baseline-state.ts"; import { legacyEnsureNetwork, legacyCreateContainer, @@ -88,6 +94,7 @@ import { type LegacySetupDatabaseOptions, type LegacyStartDbSetupImages, type LegacyStartSetupLocalDatabaseError, + legacyMemoizeSuccess, legacyResolveDbSetupPrelude, legacySetupDatabase, } from "./db-setup.ts"; @@ -414,28 +421,6 @@ export interface LegacyShadowSetupInput extends LegacyShadowConnectionInput { readonly setup: LegacyShadowDbSetupInput; } -/** - * Memoizes `effect`'s first SUCCESS; failures are never cached, so a retry re-runs the real - * effect. Deliberately not `Effect.cached` (which returns `Effect>` and needs an - * effectful construction site — {@link legacyShadowRunInputFromLocalContainerInputs} is a plain - * function) and not concurrency-guarded: the two consumers of the one field this wraps (`jwks` — - * see its construction inside that function) evaluate sequentially on the same fiber. - */ -function legacyMemoizeSuccess(effect: Effect.Effect): Effect.Effect { - let succeeded: Effect.Effect | undefined; - return Effect.suspend( - () => - succeeded ?? - effect.pipe( - Effect.tap((value) => - Effect.sync(() => { - succeeded = Effect.succeed(value); - }), - ), - ), - ); -} - /** * Adapts {@link LegacyLocalDbContainerInputs} (`local-container-inputs.ts`, the SAME * config/image/JWKS resolution prelude `db start`/`db reset` share) plus the caller's own @@ -662,7 +647,7 @@ const legacyCreateShadowTemplateDatabase = ( * need to resolve JWKS/images and run {@link legacySetupDatabase} — derived from `db-setup.ts`'s * `LegacyFreshDbSetupInput` (the exact same shape `legacyRunFreshDbSetup` resolves for the real * local `db` container) rather than hand-copied, so the two never silently drift: swap - * `experimental` (which only `legacyStartSetupLocalDatabase`'s trailing `MigrateAndSeed` call + * `experimental` (which only `legacyRunFreshDbSetup`'s trailing `MigrateAndSeed` call * needs — irrelevant to the shadow's `SetupDatabase`-only pipeline, see {@link * LegacySetupDatabaseInput}'s own doc comment) for the two fields the shadow's own caller * (`legacy-shadow-source.ts`) resolves from an already-loaded `config.toml` instead @@ -700,7 +685,9 @@ export const legacyBuildShadowSetupDatabaseInput = ( input: LegacyShadowSetupRunInput, session: LegacyDbSession, resolved: { readonly jwks: string; readonly images: LegacyStartDbSetupImages }, + rolesSql?: string, ): LegacySetupDatabaseInput => ({ + rolesSql, session, fs: input.fs, path: input.path, @@ -744,10 +731,12 @@ export const legacyBuildShadowSetupDatabaseInput = ( * `defer conn.Close(...)`), which is why this function leaks that requirement instead of * wrapping itself in `Effect.scoped`. * - * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_COLD}, i.e. exactly the sequence above. - * A warm shadow-cache hit skips the prelude + `SetupDatabase` entirely (the restored cluster + * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_UNCACHED}, i.e. exactly the sequence + * above. The three-way branch itself is {@link legacyOpenBaselineSession} (`baseline-state.ts`), + * shared verbatim with the long-running local `db` container's own fresh-setup composition: a + * warm shadow-cache hit skips the prelude + `SetupDatabase` entirely (the restored cluster * already carries them); a cache-enabled COLD provision runs the baseline in its OWN scope, so - * its session is closed before {@link LegacyShadowBaselineState.snapshotBaseline} stops the + * its session is closed before the cold state's own `snapshotBaseline` stops the * container, and returns a second session opened against the restarted one. * * Shared by all three baseline-running shadow compositions — {@link legacySetupShadowDatabase} @@ -760,54 +749,37 @@ export const legacyOpenShadowBaselineSession = ( spawner: Spawner, input: LegacyShadowSetupRunInput, options: LegacySetupDatabaseOptions = {}, - baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_UNCACHED, ): Effect.Effect< LegacyDbSession, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, Output | LegacyDockerRun | RuntimeInfo | LegacyDbConnection | Scope.Scope > => - Effect.gen(function* () { - if (!baseline.baselinePresent && baseline.snapshotRequired) { - // Own scope: the baseline session must be closed before `snapshotBaseline` — see - // {@link LegacyShadowBaselineState.snapshotRequired}. - yield* Effect.scoped( - Effect.gen(function* () { - const setupSession = yield* legacyConnectShadowDatabase(input.connConfig); - const resolved = yield* legacyResolveDbSetupPrelude(input.setup); - yield* legacySetupDatabase( - spawner, - legacyBuildShadowSetupDatabaseInput(input, setupSession, resolved), - options, - ).pipe( - // The baseline's batched SQL files check their own connection out of the pool; - // failing to acquire one is a shadow CONNECT failure, like - // `legacyConnectShadowDatabase`'s, never a setup/statement failure. - Effect.catchTag("LegacyDbConnectError", (cause) => - Effect.fail(new LegacyShadowDbError({ message: cause.message, reason: "connect" })), - ), - ); - }), - ); - yield* baseline.snapshotBaseline; - } - const session = yield* legacyConnectShadowDatabase(input.connConfig); - if (!baseline.baselinePresent && !baseline.snapshotRequired) { - // Go's single-connection flow, verbatim: baseline and everything after it on this one - // session — see {@link LegacyShadowBaselineState.snapshotRequired}. - const resolved = yield* legacyResolveDbSetupPrelude(input.setup); - yield* legacySetupDatabase( - spawner, - legacyBuildShadowSetupDatabaseInput(input, session, resolved), - options, - ).pipe( - // Same pooled-connection failure mapping as the snapshotting branch above. - Effect.catchTag("LegacyDbConnectError", (cause) => - Effect.fail(new LegacyShadowDbError({ message: cause.message, reason: "connect" })), - ), - ); - } - return session; - }); + legacyOpenBaselineSession( + legacyConnectShadowDatabase(input.connConfig), + (session) => + Effect.gen(function* () { + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput( + input, + session, + resolved, + legacyBaselineRolesSql(baseline), + ), + options, + ).pipe( + // The baseline's batched SQL files check their own connection out of the pool; + // failing to acquire one is a shadow CONNECT failure, like + // `legacyConnectShadowDatabase`'s, never a setup/statement failure. + Effect.catchTag("LegacyDbConnectError", (cause) => + Effect.fail(new LegacyShadowDbError({ message: cause.message, reason: "connect" })), + ), + ); + }), + baseline, + ); /** * Port of Go's `SetupShadowDatabase` (`apps/cli-go/internal/db/diff/diff.go:181-193`): @@ -816,7 +788,7 @@ export const legacyOpenShadowBaselineSession = ( * this resolves (Go's `defer conn.Close(...)`), matching `Effect.scoped`'s finalizer running at * the end of this function rather than leaking a `Scope.Scope` requirement to the caller. * - * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_COLD}. A warm shadow-cache hit skips the + * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_UNCACHED}. A warm shadow-cache hit skips the * prelude + `SetupDatabase` (the restored cluster already has them) and only recreates * `contrib_regression`; a cache-enabled COLD provision snapshots between the baseline and the * template, matching {@link migrateShadowDatabase}. @@ -825,7 +797,7 @@ export const legacySetupShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, options: LegacySetupDatabaseOptions = {}, - baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_UNCACHED, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, @@ -839,59 +811,15 @@ export const legacySetupShadowDatabase = ( ); /** - * What an `acquire` hands the `use` phase about the shadow cluster's CONTENTS — the seam the warm - * shadow-container cache (`shadow-cache.ts`) needs and nothing else uses. - * - * Deliberately a value the acquire OWNS and returns (alongside the container id), not an - * `afterBaseline` callback threaded down through `legacyPrepareShadowSource`: the cache is the - * only party that knows whether a cluster already carries a baseline and what to do once a fresh - * one exists, so both answers travel together with the container the cache handed over. - * {@link LEGACY_SHADOW_BASELINE_COLD} is what every uncached caller passes. + * What an `acquire` hands the `use` phase about the shadow cluster's CONTENTS — the shadow's own + * binding of the shared {@link LegacyClusterBaselineState} (`baseline-state.ts`, which the local + * `db` container's fresh-setup pipeline binds to its own error type instead). See that module for + * the full contract of each field. */ -export interface LegacyShadowBaselineState { - /** - * `true` only on a warm cache hit: the cluster already carries the platform baseline - * (`legacySetupDatabase`'s init schema + API privileges + vault + `roles.sql`), restored from - * the cache's own PGDATA snapshot, so re-running it would be wasted work at best and a - * double-applied baseline at worst. - */ - readonly baselinePresent: boolean; - /** - * `true` ONLY for a cache-enabled COLD provision — the one state whose - * {@link snapshotBaseline} really stops the container. This is what - * {@link legacyMigrateShadowDatabase} keys its session structure on: the baseline session must - * be closed before a real snapshot (a disk-level export severs any live backend), but when no - * snapshot will run, splitting sessions would be a gratuitous behavior change — a reconnect - * picks up role-level defaults `roles.sql` may have just installed (e.g. `ALTER ROLE postgres - * SET statement_timeout`), which Go's single-connection flow never exposed to migrations - * (review: Codex on #6184). So uncached and warm runs keep exactly one session. - */ - readonly snapshotRequired: boolean; - /** - * Runs immediately after a FRESHLY provisioned baseline and strictly before the template - * database/user migrations — the only point at which `postgres` holds the pristine baseline and - * nothing else. - * - * Takes NO session, and {@link legacyMigrateShadowDatabase} guarantees no session is open - * against the shadow while it runs when {@link snapshotRequired} is set: the snapshot is a - * disk-level PGDATA export that has to stop the container, which would sever any live backend. - * - * A cache that cannot SNAPSHOT degrades silently (warn + uncached run) — but the error channel - * is {@link LegacyShadowDbError}, not `never`, for the one failure that is the run's problem - * rather than the cache's: a shadow that does not come back up after the export. Reporting - * success there would send the caller's next connect to a dead (or worse, someone else's) - * Postgres on the shadow port — see `legacyExportShadowBaseline`'s doc comment - * (`shadow-cache.ts`). - */ - readonly snapshotBaseline: Effect.Effect; -} +export type LegacyShadowBaselineState = LegacyClusterBaselineState; /** The baseline state every uncached caller passes: provision it, snapshot nothing. */ -export const LEGACY_SHADOW_BASELINE_COLD: LegacyShadowBaselineState = { - baselinePresent: false, - snapshotRequired: false, - snapshotBaseline: Effect.void, -}; +export const LEGACY_SHADOW_BASELINE_UNCACHED: LegacyShadowBaselineState = LEGACY_BASELINE_UNCACHED; /** * Port of Go's `MigrateShadowDatabase` (`apps/cli-go/internal/db/diff/diff.go:195-209`): @@ -906,8 +834,8 @@ export const LEGACY_SHADOW_BASELINE_COLD: LegacyShadowBaselineState = { * own doc comment for why the ordering matters. Connection closed once this resolves, matching * Go's `defer conn.Close(...)`. * - * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_COLD}, i.e. exactly the sequence above. - * A warm shadow-cache hit passes a state whose `baselinePresent` is `true`, which skips the + * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_UNCACHED}, i.e. exactly the sequence above. + * A warm shadow-cache hit passes a `warm` state, which skips the * prelude + `SetupDatabase` steps (the restored cluster already has them) and goes straight to * the template database and the user migrations; a COLD cache-enabled provision passes the same * cold sequence plus a `snapshotBaseline` step between the baseline and the template database. @@ -920,7 +848,7 @@ const migrateShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, setupOptions: LegacySetupDatabaseOptions, - baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_UNCACHED, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, @@ -971,7 +899,7 @@ const migrateShadowDatabase = ( export const legacyMigrateShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, - baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_UNCACHED, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, @@ -986,7 +914,7 @@ export const legacyMigrateShadowDatabase = ( export const legacyMigrateNextShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, - baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_UNCACHED, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index 3a785d7383..6d5de0a253 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -777,11 +777,7 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { }), }, {}, - { - baselinePresent: true, - snapshotRequired: false, - snapshotBaseline: Effect.void, - }, + { _tag: "warm" }, ); expect(jwksEvaluated).toBe(false); expect(calls.some((c) => c.sql === LEGACY_SHADOW_CREATE_TEMPLATE_SQL)).toBe(true); @@ -846,9 +842,9 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { setup: baseShadowSetup(), }); // ONE connect, matching Go's single-connection flow: the default baseline state - // (`LEGACY_SHADOW_BASELINE_COLD`) requires no snapshot, so baseline + template + + // (`LEGACY_SHADOW_BASELINE_UNCACHED`) requires no snapshot, so baseline + template + // migrations all share one session — the split-session shape is reserved for the - // cache's own snapshotting cold provision (`snapshotRequired: true`), whose disk-level + // cache's own snapshotting cold provision (a `cold` state), whose disk-level // export must close the session before stopping the container. The ordering under test // is unaffected — the migration listing still precedes the connect. expect(events).toEqual(["list", "connect"]); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/start-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/start-database.ts index 19bfb04569..e98093d851 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/start-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/start-database.ts @@ -16,6 +16,15 @@ * pipeline (skipped IN FULL when `fromBackup` is set) -> `initCurrentBranch`, unconditionally (the * LAST line of `StartDatabase`, reached on every path that doesn't already return/fail above). * + * One TS-only addition sits inside that order, with no Go counterpart: the baseline PGDATA cache + * (`main-db-baseline.ts`), consulted only on the run that would provision the platform baseline + * (fresh volume, no `--from-backup`). Its decision lands between the container spec being built + * and `docker create`, because a warm restore is a `docker cp -` into the created-but-unstarted + * container; it then owns create + health wait, so a restore that comes up broken can fall back to + * a cold provision, and hands `legacyRunFreshDbSetup` the baseline state that decides whether the + * platform baseline is re-run or snapshotted. With `SUPABASE_SHADOW_CACHE=false`, on PG<=14, and + * on `--from-backup`, every step above is byte-for-byte what it was. + * * Deliberately has ZERO knowledge of `--ignore-health-check` — matching Go exactly: that flag is * `internal/start/start.go`'s `Run()`'s own concern, entirely OUTSIDE `StartDatabase` (Go's * `StartDatabase` has no `ignoreHealthCheck` parameter at all). `supabase start`'s own caller @@ -47,7 +56,7 @@ * baked into those containers' envs). */ -import { Data, Effect, type FileSystem, type Path, Result } from "effect"; +import { Data, Effect, type FileSystem, type Path } from "effect"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import type * as HttpClient from "effect/unstable/http/HttpClient"; @@ -65,15 +74,16 @@ import { LEGACY_CLI_PROJECT_LABEL } from "../legacy-docker-ids.ts"; import type { LegacyDockerRun } from "../legacy-docker-run.service.ts"; import { legacyEnsureNetwork, - legacyCreateContainer, legacyVolumeExists, LEGACY_COMPOSE_PROJECT_LABEL, type LegacyContainerCreateError, type LegacyContainerOpts, + type LegacyContainerRemoveError, type LegacyContainerStartError, type LegacyNetworkCreateError, type LegacyVolumeCreateError, type LegacyVolumeInspectError, + type LegacyVolumeRemoveError, } from "./container-lifecycle.ts"; import { legacyRunDatabaseWebhooksSetup, @@ -87,6 +97,10 @@ import { legacyWaitForHealthyServices, type LegacyHealthCheckTimeoutError, } from "./health-check.ts"; +import { + legacyBringUpMainDbWithBaseline, + type LegacyMainDbBaselineTomlInputs, +} from "./main-db-baseline.ts"; import type { LegacyEdgeRuntimeScript } from "../legacy-edge-runtime-script.service.ts"; import type { LegacyPgDeltaSslProbe } from "../legacy-pgdelta-ssl-probe.service.ts"; import { @@ -132,7 +146,11 @@ export type LegacyStartDatabaseError = | LegacyImagePrepullError | LegacyHealthCheckTimeoutError | LegacyDbConnectError - | LegacyStartSetupLocalDatabaseError; + | LegacyStartSetupLocalDatabaseError + // Only reachable through the baseline cache's warm-restore fallback, which force-removes the + // restored container and its volume before recreating them cold — see `main-db-baseline.ts`. + | LegacyContainerRemoveError + | LegacyVolumeRemoveError; export interface LegacyStartDatabaseInput { readonly fs: FileSystem.FileSystem; @@ -156,8 +174,12 @@ export interface LegacyStartDatabaseInput { */ readonly resolvePostgresImage: Effect.Effect; readonly dbHealthTimeoutSeconds: number; - /** Effective `[experimental.webhooks].enabled`, used to converge existing volumes. */ - readonly webhooksEnabled: boolean; + /** + * The `[db]` values from the caller's OWN already-validated `legacyCheckDbToml` pass — + * `webhooksEnabled` converges existing volumes here, and all three describe the baseline the + * cache keys on (see {@link LegacyMainDbBaselineTomlInputs}). + */ + readonly toml: LegacyMainDbBaselineTomlInputs; readonly setup: LegacyFreshDbSetupInput; /** * Fired synchronously, exactly once, right after the pre-create volume probe resolves — @@ -243,46 +265,77 @@ export const legacyStartDatabase = ( ...input.postgresSpec, image: resolvedPostgresImage, }); - yield* legacyCreateContainer(spawner, postgresSpec, input.containerOpts); - const postgresHealthResult = yield* legacyWaitForHealthyServices( + // Go's `if utils.NoBackupVolume && len(fromBackup) == 0 { SetupLocalDatabase(...) }` + // (`start.go:184-188`) — SKIPPED IN FULL when `fromBackup` is set, not merely reduced: no + // initSchema/ApplyApiPrivileges/vault/roles.sql/MigrateAndSeed on that path at all. Resolved + // ahead of the container create because the baseline cache below needs the same answer: only + // a run that would provision the platform baseline can restore or publish one. + const runsFreshSetup = isFreshVolume && fromBackup === undefined; + + const rawPostgresHealthWait = legacyWaitForHealthyServices( spawner, [postgresSpec.containerName], { timeoutSeconds: input.dbHealthTimeoutSeconds, images: new Map([[postgresSpec.containerName, resolvedPostgresImage]]), }, - ).pipe(Effect.result); - if (Result.isFailure(postgresHealthResult)) { - // Go's `StartDatabase` (`start.go:179-181`): `WaitForHealthyService`'s error is discarded - // ONLY when `len(fromBackup) > 0` — the log dump to stderr already happened inside - // `legacyWaitForHealthyServices` regardless of this branch. Any OTHER failure propagates - // BARE — this function has no `--ignore-health-check` knowledge at all, see this module's - // header for why that's entirely the caller's concern. - if (fromBackup === undefined) { - return yield* Effect.fail(postgresHealthResult.failure); - } - } + ); + // Go's `StartDatabase` (`start.go:179-181`): `WaitForHealthyService`'s error is discarded + // ONLY when `len(fromBackup) > 0` — the log dump to stderr already happened inside + // `legacyWaitForHealthyServices` regardless of this branch. Any OTHER failure propagates + // BARE — this function has no `--ignore-health-check` knowledge at all, see this module's + // header for why that's entirely the caller's concern. Folded into the effect itself (rather + // than a `Result` branch after the fact) so the baseline cache's own bring-up can reuse it + // verbatim as its readiness gate. + const postgresHealthWait = + fromBackup === undefined + ? rawPostgresHealthWait + : rawPostgresHealthWait.pipe(Effect.catch(() => Effect.void)); - // Go's `if utils.NoBackupVolume && len(fromBackup) == 0 { SetupLocalDatabase(...) }` - // (`start.go:184-188`) — SKIPPED IN FULL when `fromBackup` is set, not merely reduced: no - // initSchema/ApplyApiPrivileges/vault/roles.sql/MigrateAndSeed on that path at all. - if (isFreshVolume && fromBackup === undefined) { - yield* legacyRunFreshDbSetup(spawner, { - fs: input.fs, - path: input.path, - workdir: input.workdir, - projectId: input.projectId, - networkId: input.networkId, - hostname: input.hostname, - dbPort: input.dbPort, - // Go's own `StartDatabase` -> `SetupLocalDatabase(ctx, "", ...)` call - // (`start.go:185`) — every pending migration, no `db reset`-only seed - // override (`db start` has neither `--no-seed` nor `--sql-paths`). - version: "", - seedFlags: { noSeed: false, sqlPaths: [] }, - setup: input.setup, - }); + // The baseline cache's decision has to land BEFORE `docker create`: a warm restore is a + // `docker cp -` into the created-but-unstarted container, so the whole create + health wait + // belongs to it. `--from-backup` and a non-fresh volume bypass it outright (`cacheEligible`) + // — the former owns `preStartArchives` (its own logical dump) and skips the whole baseline + // anyway, and neither provisions one. See `main-db-baseline.ts`. + const baseline = yield* legacyBringUpMainDbWithBaseline(spawner, { + fs: input.fs, + path: input.path, + workdir: input.workdir, + dbContainerId: input.dbContainerId, + hostname: input.hostname, + dbPort: input.dbPort, + healthTimeoutSeconds: input.dbHealthTimeoutSeconds, + image: resolvedPostgresImage, + postgresSpec: input.postgresSpec, + setup: input.setup, + toml: input.toml, + spec: postgresSpec, + containerOpts: input.containerOpts, + cacheEligible: runsFreshSetup, + waitReady: postgresHealthWait, + }); + + if (runsFreshSetup) { + yield* legacyRunFreshDbSetup( + spawner, + { + fs: input.fs, + path: input.path, + workdir: input.workdir, + projectId: input.projectId, + networkId: input.networkId, + hostname: input.hostname, + dbPort: input.dbPort, + // Go's own `StartDatabase` -> `SetupLocalDatabase(ctx, "", ...)` call + // (`start.go:185`) — every pending migration, no `db reset`-only seed + // override (`db start` has neither `--no-seed` nor `--sql-paths`). + version: "", + seedFlags: { noSeed: false, sqlPaths: [] }, + setup: input.setup, + }, + baseline, + ); } else if (fromBackup === undefined) { yield* legacyRunDatabaseWebhooksSetup({ fs: input.fs, @@ -290,7 +343,7 @@ export const legacyStartDatabase = ( hostname: input.hostname, dbPort: input.dbPort, dbUrl: input.setup.dbUrl, - enabled: input.webhooksEnabled, + enabled: input.toml.webhooksEnabled, }); } diff --git a/apps/cli/src/legacy/shared/db-bootstrap/start-local-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/start-local-database.ts index a7c4b2301f..59605f2eb5 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/start-local-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/start-local-database.ts @@ -720,7 +720,14 @@ export const legacyStartLocalDatabase = Effect.fnUntraced(function* (fromBackupF resolvePostgresImage, dbHealthTimeoutSeconds: bootstrapConfig.dbHealthTimeoutSeconds, setup, - webhooksEnabled: dbTomlValues.webhooksEnabled, + // This run's own already-validated `[db]` values: `webhooksEnabled` converges an existing + // volume's pg_net, and all three are what the baseline cache keys on — see + // `LegacyMainDbBaselineTomlInputs` (`main-db-baseline.ts`). + toml: { + webhooksEnabled: dbTomlValues.webhooksEnabled, + apiAutoExposeNewTables: dbTomlValues.baseline.apiAutoExposeNewTables, + vault: dbTomlValues.vault, + }, onFreshVolumeResolved: (resolved) => { isFreshVolume = resolved; }, diff --git a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts index cee3825b7d..a374a4170c 100644 --- a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts +++ b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts @@ -1371,7 +1371,7 @@ const readDbTomlCore = Effect.fnUntraced(function* ( // side effect (`start.handler.ts:614`, `db/start/start.handler.ts:125`, both discard // the result), then again internally wherever a resolved value is actually needed // (`legacyIsLocalDbRunning`'s best-effort `projectId` probe, - // `legacyStartSetupLocalDatabase`'s own accepted duplicate config-load pass — + // `legacyRunFreshDbSetup`'s own accepted duplicate config-load pass — // see `db-bootstrap/db-setup.ts`'s header). Those internal re-reads pass // `false` so the warning still fires exactly once per invocation instead of // two or three times. diff --git a/apps/cli/src/legacy/shared/legacy-docker-run.service.ts b/apps/cli/src/legacy/shared/legacy-docker-run.service.ts index 10950df25a..dc722923ba 100644 --- a/apps/cli/src/legacy/shared/legacy-docker-run.service.ts +++ b/apps/cli/src/legacy/shared/legacy-docker-run.service.ts @@ -46,7 +46,7 @@ export interface LegacyDockerRunOpts { * Skips this layer's own image resolution (`legacyMakeDockerImageResolver`) when the * caller already resolved `image` itself through a `projectEnvValues`-aware path (e.g. * `start`'s one-shot fresh-DB setup jobs, resolved via `legacyEnsureImagesCached` before - * `legacyStartSetupLocalDatabase` ever calls this service). This layer's own resolver is + * `legacyRunFreshDbSetup` ever calls this service). This layer's own resolver is * built once, statically, with no `projectEnvValues` in scope (see * `legacy-docker-run.layer.ts`'s header) — re-resolving an ALREADY-resolved image (e.g. * `registry.example.com/supabase/gotrue:v2.192.0`) would treat it as a fresh, unresolved diff --git a/apps/cli/tests/helpers/cli.ts b/apps/cli/tests/helpers/cli.ts index 139c5b4528..c4e773a189 100644 --- a/apps/cli/tests/helpers/cli.ts +++ b/apps/cli/tests/helpers/cli.ts @@ -254,10 +254,10 @@ export function spawnSupabase( SUPABASE_HOME: homeDir, SUPABASE_NO_KEYRING: "1", SUPABASE_TELEMETRY_DISABLED: "1", - // The shadow baseline cache (`db-bootstrap/shadow-cache.ts`) is ON by default. Off here so an - // e2e/live run observes the plain shadow lifecycle and never leaves a ~90MB tar in a temp - // project — a test whose subject IS the cache opts back in through `options.env`, which is - // spread after this. + // The baseline PGDATA cache (`db-bootstrap/shadow-cache.ts`, and `main-db-baseline.ts` for + // the long-running `db` container) is ON by default. Off here so an e2e/live run observes the + // plain shadow/bring-up lifecycle and never leaves a ~90MB tar behind — a test whose subject + // IS the cache opts back in through `options.env`, which is spread after this. SUPABASE_SHADOW_CACHE: "0", ...options?.env, }; diff --git a/apps/cli/tests/helpers/legacy-mocks.ts b/apps/cli/tests/helpers/legacy-mocks.ts index 1489eb6d1c..79acebdab2 100644 --- a/apps/cli/tests/helpers/legacy-mocks.ts +++ b/apps/cli/tests/helpers/legacy-mocks.ts @@ -770,13 +770,19 @@ export const legacyWithEnv = ( * had afterwards. Like {@link useLegacyTempWorkdir} it calls vitest's `beforeEach`/`afterEach` * internally, so it must be invoked at module scope (or inside the surrounding `describe`). * - * The shadow baseline cache (`db-bootstrap/shadow-cache.ts`) is ON by default and reads - * `process.env` directly, so ANY suite that provisions a shadow through - * `legacyWithShadowDatabase` with a mocked spawner — `db diff`, `db pull`, declarative sync — now - * exercises the cache path unless it opts out: the cold path adds a `docker stop`/`docker cp`/ - * `docker start` round trip and writes a ~90MB-shaped tar into the test's workdir. Suites whose - * subject is anything OTHER than the cache should call this so they keep asserting the plain - * container lifecycle; the cache's own suites deliberately do not. + * The baseline PGDATA cache (`db-bootstrap/shadow-cache.ts`) is ON by default and reads + * `process.env` directly, so ANY suite that provisions a cluster whose platform baseline it + * covers, with a mocked spawner, now exercises the cache path unless it opts out: the cold path + * adds a `docker stop`/`docker cp`/`docker start` round trip and publishes a tar. That is both + * shadow-provisioning suites (`db diff`, `db pull`, declarative sync, `migration squash`, via + * `legacyWithShadowDatabase`) and the long-running `db` container's own bring-up (`supabase + * start`, `db start`, `db reset`, via `main-db-baseline.ts`). + * + * Critically, the tar directory is rooted at `SUPABASE_HOME` — which these suites do NOT pin — so + * without this the cache reads from, and writes into, the DEVELOPER'S REAL `~/.supabase`. Suites + * whose subject is anything OTHER than the cache should call this so they keep asserting the plain + * container lifecycle; the cache's own scenarios opt back in per-test with + * {@link withLegacyShadowCacheEnabled}, which pins both variables together. */ export function useLegacyShadowCacheDisabled(): void { const name = "SUPABASE_SHADOW_CACHE";