diff --git a/apps/cli/docs/go-cli-divergences.md b/apps/cli/docs/go-cli-divergences.md index ce946eb4d5..b7784563cf 100644 --- a/apps/cli/docs/go-cli-divergences.md +++ b/apps/cli/docs/go-cli-divergences.md @@ -86,6 +86,34 @@ These commands exist in the TS CLI today but have no direct top-level equivalent ## Behavioral divergences from the Go reference +- `db diff`/`db pull`/`db schema declarative sync`/`db schema declarative generate` shadow + baseline cache (#6184): the shadow + database's platform baseline is cached as a PGDATA snapshot under + `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides + the root; LRU keep-8 + 14-day mtime TTL, shared across worktrees with the same settings) and + restored into a + fresh container on later runs, cutting shadow provisioning from ~15s to a few seconds. The key + includes the effective Webhooks/`pg_net` policy (legacy migrate forces enabled; next migrate + follows config; next declarative forces disabled). Covers + migra/`db pull` via `legacyWithShadowDatabase`, the bundled pg-delta next sync/diff + shadows via `legacyAcquireShadowDatabase`, and `generate`/`sync`/`diff`'s legacy pg-delta + opt-out (`SUPABASE_USE_PG_DELTA_NEXT=false`), whose catalog exports provision a shadow through + the same acquire on a catalog cache miss (ephemeral host ports are not part of the cache + key — they are not baked into PGDATA). TS-only, + default ON; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (ambient env or project dotenv), `sync +--no-cache` bypasses it per-invocation, and `SUPABASE_SHADOW_DEBUG=1` prints stderr-only phase + timings. OrioleDB clusters and PG <= 14 are cache-ineligible (external S3 state and mid-session + role-default mutation respectively — see `shadow-cache.ts`). Known session-semantics caveat on + the cached paths: migrations run on a session opened after the baseline, so role-level defaults + a user's `roles.sql` installs (`ALTER ROLE … SET …`) apply to migrations, whereas Go's + single-connection flow ran migrations before those defaults took effect; opting out restores + Go's exact single-session behavior. +- Postgres container entrypoint (`postgres.service.ts`): the init script `exec`s + `docker-entrypoint.sh` so Postgres is PID 1. Go leaves `sh` as PID 1, so SIGTERM is never + forwarded and every `docker stop` burns the full 10s grace period. Applies to + `supabase start`, `db start`, `--from-backup`, and shadow containers (the last is why the + shadow baseline cache's cold export can stop/start in ~1s). Timing is not part of the + Go-parity surface (ADR 0016). - `db schema declarative generate`/`sync` default declarative directory is `supabase/schemas`; the old Go CLI reference (pre-`7b469f5b3`) used `supabase/database`. The move aligns the default with the product-wide declarative-schemas convention. To keep the upgrade visible, diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 2f9466e69b..f075c41d17 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -19,30 +19,34 @@ it, and JSON `null` disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| --------------------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | -| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | -| `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | -| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | -| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit `--from/--to migrations` catalog cache | +| Path | Format | When | +| --------------------------------------------------------------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | +| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); also hashed into the shadow-baseline cache key on every cache-eligible acquire, warm hits included (where no baseline is applied at all); missing file tolerated | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | abandoned-partial sweep on every cache-eligible acquire (warm hit and cold export) — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | +| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit `--from/--to migrations` catalog cache | ## Files Written -| Path | Format | When | -| ----------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------ | -| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | -| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output`; flattened review representation, not a portable apply script | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | +| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output`; flattened review representation, not a portable apply script | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision creates the current key's snapshot (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`); a warm hit `touch`es its mtime (LRU); every cache-eligible acquire may delete other keys under LRU keep-8 + 14-day mtime TTL — ~90MB (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker @@ -92,6 +96,9 @@ of this command's own target resolve, ahead of the differ container. | `SUPABASE_DB_MAJOR_VERSION` / `SUPABASE_DB_HEALTH_TIMEOUT` / `SUPABASE_DB_SETTINGS_*` | shadow container-config overrides, same as `db start`/`db reset` | no | | `SUPABASE_PROJECT_ID` | overrides the shadow container's project id/labels, same as `db start`/`db reset` (`utils.DbId`); ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the shadow baseline cache (and other CLI state) | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | +| `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta engine | no | | `PGDELTA_DEBUG` | pg-delta debug capture | no | | `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | @@ -223,6 +230,24 @@ transaction metadata. no longer the `db __shadow` seam) plus a native pg-delta catalog export. No hidden Go `db schema declarative __catalog` subprocess runs for this path any more. +### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) + +ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the ambient env AND the +project's dotenv, e.g. `supabase/.env`), restoring the documented uncached lifecycle. A warm hit +skips the platform baseline, so the `Initialising schema...` progress line does not print — +progress text reflects the work actually performed. +Artifact: `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` +overrides the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster +(including the effective Webhooks/`pg_net` policy); +shared across worktrees with the same settings; retention is LRU (keep 8) + 14-day mtime TTL +(warm hits refresh mtime; sibling tars may be deleted). +Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed +on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, +a cold export failure only warns and leaves the run uncached (one exception: a shadow that +fails to come back up after the snapshot fails the run rather than reporting a false success). See `shared/db-bootstrap/ +shadow-cache.ts`'s doc comment for the mechanics. `--use-pgadmin` is NOT cached — its shadow keeps +the plain create/remove lifecycle. + ### `--use-pgadmin` parity quirks and deliberate divergence (CLI-1968) - `source`/`target` are INVERTED relative to the migra/pg-delta path: `source` is the diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index ed9bebe00a..c063c10c1e 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -30,6 +30,7 @@ import { legacyFindDropStatements } from "../../../shared/legacy-sql-split.ts"; import { legacyBuildLocalDbContainerInputs } from "../../../shared/db-bootstrap/local-container-inputs.ts"; import { legacyIsLocalDbRunning } from "../../../shared/db-bootstrap/local-db-running.ts"; import { legacyWaitForHealthyServices } from "../../../shared/db-bootstrap/health-check.ts"; +import { legacyWithShadowDatabase } from "../../../shared/db-bootstrap/shadow-cache.ts"; import { legacyCreateShadowDatabase, legacyMigrateShadowDatabase, @@ -690,9 +691,21 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy pgDelta: cfg.pgDelta, ctx, }; - // Register cleanup atomically with creation; prepare and diff remain interruptible. - diffResult = yield* Effect.acquireUseRelease( - legacyCreateShadowDatabase(spawner, shadowInput), + // `legacyWithShadowDatabase` (`shadow-cache.ts`) rather than a bare + // `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` pair — see its doc comment for + // both halves of the rationale: why the lifecycle is an `Effect.acquireUseRelease` (an + // interrupt must not be able to land between creation and the finalizer being attached) and + // why the cache seam sits here (with `SUPABASE_SHADOW_CACHE` unset it IS today's + // create/remove pair; otherwise a key-matching PGDATA snapshot is restored into the fresh + // container in a few seconds instead of cold-provisioning the baseline in ~15s). + // The key's webhooks policy mirrors the migrate `legacyPrepareShadowSource` will actually + // select for this mode: legacy's `legacyMigrateShadowDatabase` forces `pg_net` on, next's + // `legacyMigrateNextShadowDatabase` follows project config — a key that said "enabled" for + // a config-following baseline would let the two engines restore each other's tars + // (review: Codex on #6184). + diffResult = yield* legacyWithShadowDatabase( + spawner, + shadowInput, (handle) => Effect.gen(function* () { const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); @@ -740,7 +753,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // single migration file. return { sql, files: undefined }; }), - (handle) => legacyRemoveShadowDatabase(spawner, handle.containerId), + { webhooks: migrationMode === "pgdelta-next" ? "config" : "enabled" }, ); } const out = diffResult.sql; diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 68bf7909ee..20d749ddbe 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -10,11 +10,15 @@ import { stripAnsi } from "../../../../../tests/helpers/ansi.ts"; import { LEGACY_FAKE_SHADOW_CONTAINER_ID, LEGACY_VALID_REF, + legacyFailWriteStringMatchingFsLayer, legacyFailWriteStringOnNthCallFsLayer, + legacyWithEnv, mockLegacyCliConfig, + mockLegacyDockerDaemonCliSpawner, mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, legacySequentialExecBatch, } from "../../../../../tests/helpers/legacy-mocks.ts"; @@ -41,6 +45,7 @@ import { type LegacyDbSession, type LegacyPgConnInput, } from "../../../shared/legacy-db-connection.service.ts"; +import { LegacyDbConnectError } from "../../../shared/legacy-db-connection.errors.ts"; import { LegacyDockerRunError } from "../../../shared/legacy-docker-run.errors.ts"; import { LegacyDockerRun, @@ -88,11 +93,21 @@ interface SetupOpts { readonly networkId?: string; // --network-id value forwarded to docker runs // When set, the Nth `writeFileString` fails, exercising cleanup-on-failure. readonly failWriteOnCall?: number; + // When set, the first `writeFileString` whose path matches fails. Prefer this + // over `failWriteOnCall` when shadow setup writes extra SQL before the + // command's `--file` migration. + readonly failWriteMatching?: (path: string) => boolean; // When set, the shadow container never reports healthy — for the interrupt-during- // health-wait regression coverage (review: PRRT_kwDOErm0O86XMrID). See // `mockLegacyShadowContainerCliSpawner`'s own doc comment for why this is required - // (not `Effect.never`) to observe a genuinely suspended retry loop. + // (not `Effect.never`) to observe a genuinely suspended retry loop. Only the + // `--use-pgadmin` branch still gates on the Docker healthcheck; the shadow-source + // branch gates on `neverConnectableShadow` below instead. readonly neverHealthyShadow?: boolean; + // When set, every connect to the shadow's own port is refused, so the readiness gate + // (`legacyWaitForShadowReady`) keeps polling — the shadow-source branch's equivalent of + // `neverHealthyShadow`, since that wait no longer consults the Docker healthcheck. + readonly neverConnectableShadow?: boolean; // `LegacyCliConfig.projectId` (the `SUPABASE_PROJECT_ID` env-only reader). Defaults // to `Option.some("test")`; pass `Option.none()` to exercise the // config.toml/workdir-basename fallback `legacyResolveLocalProjectId` provides for @@ -128,6 +143,10 @@ interface SetupOpts { // host-gateway` (Linux-only). Defaults to `"linux"` (every other test's implicit // baseline); pass `"darwin"`/`"win32"` to exercise the no-add-host branch. readonly platform?: NodeJS.Platform; + // Swaps the stateless shadow spawner for the stateful Docker model, whose + // `stop`/`cp`/`start` really move bytes. Required by (and only by) the tests that + // enable the shadow BASELINE CACHE — see `mockLegacyDockerDaemonCliSpawner`. + readonly statefulDocker?: boolean; } const alwaysReadyHttpClientLayer = Layer.succeed( @@ -137,14 +156,28 @@ const alwaysReadyHttpClientLayer = Layer.succeed( ), ); -/** Records every `LegacyDbConnection.connect` target's database name, and every `exec`/`query` SQL run against it. */ -function fakeShadowDbConnection() { +/** `[db] shadow_port`'s schema default — the port every connect to the shadow itself dials. */ +const LEGACY_SHADOW_PORT = 54320; + +/** + * Records every `LegacyDbConnection.connect` target's database name, and every `exec`/`query` + * SQL run against it. + * + * `neverConnectableShadow` makes every connect to the SHADOW port fail (leaving the local + * target's own connects untouched) — the shadow's readiness gate is now a direct connect probe + * (`legacyWaitForShadowReady`), so a shadow that never accepts a connection is what keeps a + * provisioning fiber genuinely suspended inside that retry loop. + */ +function fakeShadowDbConnection(opts: { readonly neverConnectableShadow?: boolean } = {}) { const connectedDatabases: Array = []; const execCalls: Array = []; const layer = Layer.succeed(LegacyDbConnection, { connect: (cfg: LegacyPgConnInput) => - Effect.sync(() => { + Effect.gen(function* () { connectedDatabases.push(cfg.database); + if (opts.neverConnectableShadow === true && cfg.port === LEGACY_SHADOW_PORT) { + return yield* Effect.fail(new LegacyDbConnectError({ message: "connection refused" })); + } const session: LegacyDbSession = { exec: (sql) => Effect.sync(() => { @@ -175,7 +208,14 @@ function setup(workdir: string, opts: SetupOpts = {}) { dbNotRunning: opts.dbNotRunning ?? false, dbInspectFailsWith: opts.dbInspectFailsWith, }); - const shadowDbConnection = fakeShadowDbConnection(); + // The shadow baseline cache's cold export and warm restore only mean anything against a + // daemon that actually holds container state and carries `docker cp` bytes, so the cache + // tests below opt into the stateful model instead. + const dockerDaemon = + opts.statefulDocker === true ? mockLegacyDockerDaemonCliSpawner() : undefined; + const shadowDbConnection = fakeShadowDbConnection({ + neverConnectableShadow: opts.neverConnectableShadow ?? false, + }); const explicitDiffCalls: LegacyPgDeltaExplicitDiffInput[] = []; const databaseDiffCalls: LegacyPgDeltaDatabaseDiffInput[] = []; @@ -411,7 +451,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { edge, docker, shadowDbConnection.layer, - shadowSpawner.layer, + dockerDaemon?.layer ?? shadowSpawner.layer, alwaysReadyHttpClientLayer, resolver, projectRefResolver, @@ -432,10 +472,13 @@ function setup(workdir: string, opts: SetupOpts = {}) { mockRuntimeInfo({ platform: opts.platform ?? "linux" }), ); // Merged last so its `FileSystem` overrides everything above (last-wins). - const layer = - opts.failWriteOnCall === undefined - ? baseLayer - : Layer.merge(baseLayer, legacyFailWriteStringOnNthCallFsLayer(opts.failWriteOnCall)); + const failWriteLayer = + opts.failWriteMatching !== undefined + ? legacyFailWriteStringMatchingFsLayer(opts.failWriteMatching) + : opts.failWriteOnCall !== undefined + ? legacyFailWriteStringOnNthCallFsLayer(opts.failWriteOnCall) + : undefined; + const layer = failWriteLayer === undefined ? baseLayer : Layer.merge(baseLayer, failWriteLayer); return { layer, @@ -454,6 +497,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { differRegistryEnvAtCall, shadowSetupJobCalls, shadowSpawned: shadowSpawner.spawned, + dockerDaemon, shadowConnectedDatabases: shadowDbConnection.connectedDatabases, shadowExecCalls: shadowDbConnection.execCalls, }; @@ -492,6 +536,10 @@ const stderr = (out: ReturnType) => ); const tmp = useLegacyTempWorkdir(); +// The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker cp`/ +// `docker start` round trip plus a snapshot tar to every shadow this suite provisions. This suite +// is about the command, not the cache, so it asserts the plain shadow lifecycle. +useLegacyShadowCacheDisabled(); // --- native --use-pgadmin fixtures --- @@ -1954,37 +2002,36 @@ describe("legacy db diff", () => { }); it.live( - "removes the shadow container on a SIGINT-style interruption during the health wait, without waiting for the health-check timeout", + "removes the shadow container on a SIGINT-style interruption during the readiness wait, without waiting for the readiness timeout", () => { // Regression test for the acquireUseRelease restructuring (review: // PRRT_kwDOErm0O86XMrID): an earlier shape passed the ENTIRE // `legacyPrepareShadowSource` (create -> health-wait -> migrate -> // declarative-apply) as `acquireUseRelease`'s `acquire`, which Effect's // `uninterruptibleMask` (no `restore` around `acquire`) made completely - // uninterruptible — a SIGINT landing during the health wait (which can run for - // up to 30 real seconds, `LEGACY_HEALTH_CHECK_TIMEOUT_SECONDS`) was silently - // swallowed until the health check gave up on its own. `acquire` is now ONLY - // `legacyCreateShadowDatabase` - // (container creation); the health wait runs inside the interruptible `use` - // phase instead, so a `Fiber.interrupt` here must land promptly. - const s = setup(tmp.current, { neverHealthyShadow: true }); + // uninterruptible — a SIGINT landing during the readiness wait (which can run + // for up to 30 real seconds, `LEGACY_HEALTH_CHECK_TIMEOUT_SECONDS`) was silently + // swallowed until the wait gave up on its own. `acquire` is now ONLY + // `legacyCreateShadowDatabase` (container creation); the readiness wait runs + // inside the interruptible `use` phase instead, so a `Fiber.interrupt` here must + // land promptly. + const s = setup(tmp.current, { neverConnectableShadow: true }); return Effect.gen(function* () { const fiber = yield* legacyDbDiff(flags()).pipe( Effect.provide(s.layer), Effect.forkChild({ startImmediately: true }), ); - // Wait until the shadow's own health check has actually probed the - // never-healthy container at least once — proving the fiber is genuinely - // suspended inside `legacyWaitForHealthyServices`'s retry loop, not merely - // past the `create` call. - while (!s.shadowSpawned.some((c) => c.args[0] === "container" && c.args[1] === "inspect")) { + // Wait until the shadow's readiness gate has actually refused a connect at + // least once — proving the fiber is genuinely suspended inside + // `legacyWaitForShadowReady`'s retry loop, not merely past the `create` call. + while (s.shadowConnectedDatabases.length === 0) { yield* Effect.sleep("5 millis"); } // `Fiber.interrupt` only resolves once the target fiber (and its finalizers, // including `legacyRemoveShadowDatabase`) has fully completed — if `acquire` - // still covered the health wait, this call would hang for up to 30 real + // still covered the readiness wait, this call would hang for up to 30 real // seconds (or until this test's own timeout), instead of resolving as soon - // as the in-flight probe's own subprocess call returns. + // as the in-flight probe returns. yield* Fiber.interrupt(fiber); expect(s.shadowSpawned.filter((c) => c.args[0] === "create")).toHaveLength(1); expect(s.shadowSpawned.filter((c) => c.args[0] === "rm")).toHaveLength(1); @@ -2469,12 +2516,9 @@ describe("legacy db diff", () => { it.effect( "fails with LegacyDbDiffWriteError when writing the pgAdmin --file migration fails", () => { - // Shadow setup writes the branch marker and `revoke-api-privileges.sql` - // before the command writes the pgAdmin migration, so call #3 is the - // diff-file write exercised here. const s = setup(tmp.current, { pgadminStdout: [JSON.stringify([pgadminEntry()])], - failWriteOnCall: 3, + failWriteMatching: (path) => path.includes("pgadmin_diff"), }); return Effect.gen(function* () { const error = yield* legacyDbDiff( @@ -2566,4 +2610,62 @@ describe("legacy db diff", () => { }, ); }); + + describe("shadow baseline cache", () => { + /** The `.tar` files published under the per-test `SUPABASE_HOME` this block pins. */ + const publishedTars = () => { + const dir = join(tmp.current, "_supabase_home", "cache", "shadow-baseline"); + return existsSync(dir) ? readdirSync(dir).filter((entry) => entry.endsWith(".tar")) : []; + }; + + /** + * Runs `db diff` with the shadow baseline cache ENABLED (this file pins it off for every + * other test) and its artifacts isolated under the workdir, against the stateful Docker + * model the export/restore round trip needs. + */ + const runCached = (implementation: "legacy" | "next") => { + const s = setup(tmp.current, { + statefulDocker: true, + pgDeltaImplementation: implementation, + diffSql: "create table t ();\n", + }); + return legacyWithEnv( + "SUPABASE_HOME", + join(tmp.current, "_supabase_home"), + legacyWithEnv( + "SUPABASE_SHADOW_CACHE", + "1", + legacyDbDiff(flags({ usePgDelta: Option.some(true) })).pipe(Effect.provide(s.layer)), + ), + ).pipe(Effect.as(s)); + }; + + // Regression: both migrate paths used to pass a hardcoded `{ webhooks: "enabled" }`, so the + // legacy run's forced-`pg_net` baseline and the next run's config-following baseline keyed + // to the SAME tar and silently restored each other's cluster. The handler now forks the + // policy on `migrationMode`; `shadow-cache.integration.test.ts` covers the cache's half of + // the contract, this covers `db diff`'s call site. + it.live("a legacy-engine baseline is never restored into a pg-delta-next run", () => { + mkdirSync(join(tmp.current, "supabase"), { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + "[experimental.pgdelta]\nenabled = true\n", + ); + return Effect.gen(function* () { + // Legacy migrate forces `pg_net` on regardless of config, and publishes that baseline. + const legacyRun = yield* runCached("legacy"); + expect(legacyRun.dockerDaemon?.stepCalls("cp-out")).toHaveLength(1); + const legacyTars = publishedTars(); + expect(legacyTars).toHaveLength(1); + + // pg-delta next follows the config (webhooks are off here), so it must cold-provision + // and publish its OWN baseline rather than restore the forced-on one above. + const nextRun = yield* runCached("next"); + expect(nextRun.dockerDaemon?.stepCalls("cp-in")).toHaveLength(0); + expect(nextRun.dockerDaemon?.stepCalls("cp-out")).toHaveLength(1); + expect(publishedTars()).toHaveLength(2); + expect(publishedTars()).toEqual(expect.arrayContaining(legacyTars)); + }); + }); + }); }); diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 1550e6efbb..1bddbe510c 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -35,31 +35,35 @@ disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ----------------------------------------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | -| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | -| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | -| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | +| Path | Format | When | +| ----------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | +| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); also hashed into the shadow-baseline cache key on every cache-eligible acquire, warm hits included (where no baseline is applied at all); missing file tolerated | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | abandoned-partial sweep on every cache-eligible acquire (warm hit and cold export) — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | ## Files Written -| Path | Format | When | -| ---------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | -| `/supabase/schemas/**` | SQL | `--declarative` | -| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | -| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | +| `/supabase/schemas/**` | SQL | `--declarative` | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision creates the current key's snapshot, migration-style pull only (never `--declarative`'s bare shadow or the delegated `--experimental` path); a warm hit `touch`es its mtime (LRU); every cache-eligible acquire may delete other keys under LRU keep-8 + 14-day mtime TTL — ~90MB (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker @@ -67,11 +71,31 @@ disables formatting without disabling safe compaction. - Shadow Postgres container — provisioned and torn down natively (`legacyPrepareShadowSource` in `legacy/commands/db/shared/legacy-shadow-source.ts` / `legacyPrepareRawShadow` in `legacy/shared/db-bootstrap/shadow-database.ts`, which also owns the lower-level primitives - both build on), no longer via a Go seam. + both build on), no longer via a Go seam. Torn down with `docker rm -f -v` on every run, + cache or no cache — see the shadow baseline cache section below. - `supabase/migra` container — the migra OOM bash fallback only. - `pg_dump` container — the initial-migra pull's native remote-schema dump (`legacyStreamPgDump`, shared with `db dump`). +### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) + +ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the ambient env AND the +project's dotenv, e.g. `supabase/.env`), restoring the documented uncached lifecycle. A warm hit +skips the platform baseline, so the `Initialising schema...` progress line does not print — +progress text reflects the work actually performed. +Artifact: `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` +overrides the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster +(including the effective Webhooks/`pg_net` policy); +shared across worktrees with the same settings; retention is LRU (keep 8) + 14-day mtime TTL +(warm hits refresh mtime; sibling tars may be deleted). +Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed +on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, +a cold export failure only warns and leaves the run uncached (one exception: a shadow that +fails to come back up after the snapshot fails the run rather than reporting a false success). See `shared/db-bootstrap/ +shadow-cache.ts`'s doc comment for the mechanics. Each pooler-retry attempt acquires/releases its +own shadow (a warm hit restores the same tar each time); `--declarative`'s bare shadow runs no +baseline, so it is never cached. + ## API Routes / DB | Method | Path / SQL | Auth | Purpose | @@ -92,6 +116,9 @@ disables formatting without disabling safe compaction. | `SUPABASE_DB_MAJOR_VERSION` / `SUPABASE_DB_HEALTH_TIMEOUT` / `SUPABASE_DB_SETTINGS_*` | shadow container-config overrides, same as `db start`/`db reset` | no | | `SUPABASE_PROJECT_ID` | overrides the shadow container's project id/labels, same as `db start`/`db reset` (`utils.DbId`); ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the shadow baseline cache (and other CLI state) | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | +| `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta diff engine | no | | `SUPABASE_EXPERIMENTAL` | selects the deprecated structured-dump branch (still delegates to Go, see below) | no | | `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | diff --git a/apps/cli/src/legacy/commands/db/pull/pull.debug.ts b/apps/cli/src/legacy/commands/db/pull/pull.debug.ts index 0282f2b3b1..558ae84e6e 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.debug.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.debug.ts @@ -7,7 +7,7 @@ import { legacyDebugBundleMessage, legacySaveDebugBundle, } from "../shared/legacy-debug-bundle.ts"; -import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.cache.ts"; +import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.paths.ts"; import { type LegacyPgDeltaContext, legacyExportCatalogPgDelta, diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 813bc383ea..d3266d8c0d 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -40,6 +40,7 @@ import { legacyBuildLocalDbContainerInputs, type LegacyLocalDbContainerInputs, } from "../../../shared/db-bootstrap/local-container-inputs.ts"; +import { legacyWithShadowDatabase } from "../../../shared/db-bootstrap/shadow-cache.ts"; import { legacyCreateShadowDatabase, legacyPrepareRawShadow, @@ -786,9 +787,22 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy pgDelta: toml.pgDelta, ctx, }; - // Register cleanup atomically with shadow acquisition. - return yield* Effect.acquireUseRelease( - legacyCreateShadowDatabase(spawner, shadowInput), + // `legacyWithShadowDatabase` (`shadow-cache.ts`) rather than a bare + // `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` pair — see its doc + // comment for both halves of the rationale: why the lifecycle is an + // `Effect.acquireUseRelease` (an interrupt must not be able to land between creation + // and the finalizer being attached) and why the cache seam sits here. Note each + // pooler-retry attempt still acquires and releases its own shadow — on the warm path + // each attempt restores its own fresh container from the same cached snapshot, + // sequentially. + // The key's webhooks policy mirrors the migrate `legacyPrepareShadowSource` will + // actually select for this mode: legacy's `legacyMigrateShadowDatabase` forces + // `pg_net` on, next's `legacyMigrateNextShadowDatabase` follows project config — + // a key that said "enabled" for a config-following baseline would let the two + // engines restore each other's tars (review: Codex on #6184). + return yield* legacyWithShadowDatabase( + spawner, + shadowInput, (handle) => Effect.gen(function* () { const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); @@ -838,7 +852,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy }); return { sql, files: undefined, debug: undefined }; }), - (handle) => legacyRemoveShadowDatabase(spawner, handle.containerId), + { webhooks: migrationMode === "pgdelta-next" ? "config" : "enabled" }, ); }); const diffOutcome = yield* withPoolerFallback(targetEndpoint, runShadowDiff); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index bb940110f7..42aa50f962 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -9,10 +9,13 @@ import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; import { stripAnsi } from "../../../../../tests/helpers/ansi.ts"; import { LEGACY_VALID_REF, + legacyWithEnv, mockLegacyCliConfig, + mockLegacyDockerDaemonCliSpawner, mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, } from "../../../../../tests/helpers/legacy-mocks.ts"; import { @@ -131,6 +134,10 @@ interface SetupOpts { // `LegacyProjectNotLinkedError` absent an explicit `--project-ref` flag, // instead of silently falling back to `opts.resolvedRef ?? LEGACY_VALID_REF`. readonly linkedFails?: boolean; + // Swaps the stateless shadow spawner for the stateful Docker model, whose + // `stop`/`cp`/`start` really move bytes. Required by (and only by) the tests that + // enable the shadow BASELINE CACHE — see `mockLegacyDockerDaemonCliSpawner`. + readonly statefulDocker?: boolean; } function setup(workdir: string, opts: SetupOpts = {}) { @@ -143,6 +150,11 @@ function setup(workdir: string, opts: SetupOpts = {}) { // A real docker-spawner fake backs container create/start/health-inspect/cleanup. const shadowSpawner = mockLegacyShadowContainerCliSpawner(); + // The shadow baseline cache's cold export and warm restore only mean anything against a + // daemon that actually holds container state and carries `docker cp` bytes, so the cache + // tests below opt into the stateful model instead. + const dockerDaemon = + opts.statefulDocker === true ? mockLegacyDockerDaemonCliSpawner() : undefined; const engineCalls: Array<{ operation: "diff" | "export"; @@ -321,6 +333,8 @@ function setup(workdir: string, opts: SetupOpts = {}) { const execLog: string[] = []; const historyUpserts: ReadonlyArray[] = []; const connectedDatabases: Array = []; + /** Same connects as {@link connectedDatabases}, keeping the port that tells target from shadow apart. */ + const connectTargets: Array<{ readonly database: string; readonly port: number }> = []; // The resolver mock's own target connection always dials port 5432; the native // shadow (platform baseline, `CREATE_TEMPLATE`, migrations, and — on the // declarative branch — the `contrib_regression` override) always dials the @@ -360,6 +374,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { connect: (cfg: { readonly database: string; readonly port: number }) => Effect.sync(() => { connectedDatabases.push(cfg.database); + connectTargets.push({ database: cfg.database, port: cfg.port }); return cfg.port === TARGET_PORT ? targetSession : shadowSession; }), }); @@ -450,7 +465,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { edge, docker, dbConnection, - shadowSpawner.layer, + dockerDaemon?.layer ?? shadowSpawner.layer, alwaysReadyHttpClientLayer, resolver, projectRefResolver, @@ -481,11 +496,13 @@ function setup(workdir: string, opts: SetupOpts = {}) { historyUpserts, execLog, connectedDatabases, + connectTargets, poolerFallbackCalls, resolveCalls, dumpCalls, engineCalls, shadowSpawned: shadowSpawner.spawned, + dockerDaemon, get edgeRunCount() { return edgeRunCount; }, @@ -523,6 +540,10 @@ const seedMigration = (workdir: string, version: string) => { }; const tmp = useLegacyTempWorkdir(); +// The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker cp`/ +// `docker start` round trip plus a snapshot tar to every shadow this suite provisions. This suite +// is about the command, not the cache, so it asserts the plain shadow lifecycle. +useLegacyShadowCacheDisabled(); describe("legacy db pull", () => { it.effect("pulls a migration (pgdelta engine) and updates remote history under --yes", () => { @@ -884,9 +905,15 @@ describe("legacy db pull", () => { files: ["public/t.sql"], }); // Declarative mode's bare shadow (`legacyPrepareRawShadow`) never connects to set - // up a platform baseline or `contrib_regression` template — the only connect is - // the top-level target connect (`resolved.conn`, database "postgres"). - expect(s.connectedDatabases).toEqual(["postgres"]); + // up a platform baseline or `contrib_regression` template. The only connects are + // the top-level target connect (`resolved.conn`, port 5432, database "postgres") + // and the shadow's own readiness probe on the shadow port — a single short-lived + // connect that is now the provisioning gate (`legacyWaitForShadowReady`) in place + // of waiting on the shadow container's 10s-interval Docker healthcheck. + expect(s.connectTargets).toEqual([ + { database: "postgres", port: 5432 }, + { database: "postgres", port: 54320 }, + ]); expect(s.shadowSpawned.filter((call) => call.args[0] === "create")).toHaveLength(1); expect(s.shadowSpawned.filter((call) => call.args[0] === "rm")).toHaveLength(1); }).pipe(Effect.provide(s.layer)); @@ -2289,4 +2316,69 @@ describe("legacy db pull", () => { expect(Exit.isFailure(exit)).toBe(true); }).pipe(Effect.provide(s.layer)); }); + + describe("shadow baseline cache", () => { + /** The `.tar` files published under the per-test `SUPABASE_HOME` this block pins. */ + const publishedTars = () => { + const dir = join(tmp.current, "_supabase_home", "cache", "shadow-baseline"); + return existsSync(dir) ? readdirSync(dir).filter((entry) => entry.endsWith(".tar")) : []; + }; + + /** + * Runs `db pull` with the shadow baseline cache ENABLED (this file pins it off for every + * other test) and its artifacts isolated under the temp root, against the stateful Docker + * model the export/restore round trip needs. + * + * Each run gets its OWN workdir so the migration file the previous pull wrote cannot shift + * the second run's behaviour — the cache key is global and deliberately workdir-independent, + * so two worktrees with identical settings still collide on the same tar. + */ + const runCached = (implementation: "legacy" | "next") => { + const workdir = join(tmp.current, `${implementation}-worktree`); + seedMigration(workdir, "20240101000000"); + writeFileSync( + join(workdir, "supabase", "config.toml"), + "[experimental.pgdelta]\nenabled = true\n", + ); + const s = setup(workdir, { + statefulDocker: true, + engineImplementation: implementation, + remoteVersions: ["20240101000000"], + edgeStdout: pgDeltaDiffEnvelope([{ name: "schema_changes", sql: "create table t ();" }]), + yes: true, + }); + return legacyWithEnv( + "SUPABASE_HOME", + join(tmp.current, "_supabase_home"), + legacyWithEnv( + "SUPABASE_SHADOW_CACHE", + "1", + legacyDbPull(flags()).pipe(Effect.provide(s.layer)), + ), + ).pipe(Effect.as(s)); + }; + + // Regression: both migrate paths used to pass a hardcoded `{ webhooks: "enabled" }`, so the + // legacy run's forced-`pg_net` baseline and the next run's config-following baseline keyed + // to the SAME tar and silently restored each other's cluster. The handler now forks the + // policy on `migrationMode`; `shadow-cache.integration.test.ts` covers the cache's half of + // the contract, this covers `db pull`'s call site. + it.live("a legacy-engine baseline is never restored into a pg-delta-next run", () => { + return Effect.gen(function* () { + // Legacy migrate forces `pg_net` on regardless of config, and publishes that baseline. + const legacyRun = yield* runCached("legacy"); + expect(legacyRun.dockerDaemon?.stepCalls("cp-out")).toHaveLength(1); + const legacyTars = publishedTars(); + expect(legacyTars).toHaveLength(1); + + // pg-delta next follows the config (webhooks are off here), so it must cold-provision + // and publish its OWN baseline rather than restore the forced-on one above. + const nextRun = yield* runCached("next"); + expect(nextRun.dockerDaemon?.stepCalls("cp-in")).toHaveLength(0); + expect(nextRun.dockerDaemon?.stepCalls("cp-out")).toHaveLength(1); + expect(publishedTars()).toHaveLength(2); + expect(publishedTars()).toEqual(expect.arrayContaining(legacyTars)); + }); + }); + }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 62c85f1f7c..1e0c6c51b2 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -5,7 +5,10 @@ import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; import { Cause, Effect, Exit, FileSystem, Layer, Option, Path } from "effect"; -import { mockLegacyShadowContainerCliSpawner } from "../../../../../../tests/helpers/legacy-mocks.ts"; +import { + mockLegacyShadowContainerCliSpawner, + useLegacyShadowCacheDisabled, +} from "../../../../../../tests/helpers/legacy-mocks.ts"; import { alwaysReadyHttpClientLayer } from "../../../../../../tests/helpers/legacy-local-reset.ts"; import { mockOutput, mockRuntimeInfo } from "../../../../../../tests/helpers/mocks.ts"; import { CliArgs } from "../../../../../shared/cli/cli-args.service.ts"; @@ -390,6 +393,10 @@ const toml: LegacyDbTomlValues = { }; describe("legacyDiffDeclarativeToMigrations", () => { + // The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker + // cp`/`docker start` round trip plus a snapshot tar to every shadow this suite provisions. This + // suite is about the orchestration, not the cache, so it asserts the plain shadow lifecycle. + useLegacyShadowCacheDisabled(); it.effect( "resolves the migrations catalog natively and diffs it against the seam-provisioned declarative catalog", () => { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index 0d2782eb87..bf322f47ee 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -7,30 +7,36 @@ Pg-delta runs in-process by default. Set `SUPABASE_USE_PG_DELTA_NEXT=false` for the legacy catalog/edge-runtime implementation; there is no automatic fallback. Coverage gaps warn; `--strict-coverage` makes them fatal, and `PGDELTA_DEBUG` writes diagnostic JSON under `supabase/.temp/pgdelta/v2/debug//`. -`--no-cache` affects only the legacy opt-out. The bundled formatter defaults to +`--no-cache` affects only the legacy opt-out (its catalog cache and the shadow +baseline snapshot those catalog exports use). The bundled formatter defaults to lowercase SQL at width 180; config overrides it, and JSON `null` disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ----------------------------------------------- | ---------- | --------------------------------------------- | -| `/supabase/config.toml` | TOML | always — pg-delta gate, ports, format options | -| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | -| `/supabase/.temp/postgres-version` | plain text | legacy opt-out's shadow-DB image resolution | -| `/supabase/migrations/*.sql` | SQL | smart mode — detect whether migrations exist | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog cache | -| `~/.supabase/access-token` | plain text | `--linked` (token resolution) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always — pg-delta gate, ports, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/.temp/postgres-version` | plain text | legacy opt-out's shadow-DB image resolution | +| `/supabase/migrations/*.sql` | SQL | smart mode — detect whether migrations exist | +| `/supabase/roles.sql` | SQL | legacy opt-out — hashed into the catalog cache key, and on a catalog miss also into the shadow-baseline cache key (on warm hits too, not just cold ones) and applied to a cold shadow's baseline; missing file tolerated (hashed as empty) | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog cache | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | legacy opt-out catalog miss, warm shadow-cache hit — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU/TTL (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | legacy opt-out catalog miss — abandoned-partial sweep on every cache-eligible acquire; removed when older than an hour | +| `~/.supabase/access-token` | plain text | `--linked` (token resolution) | ## Files Written -| Path | Format | When | -| ----------------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------ | -| `/supabase/schemas/**/*.sql` (default declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | -| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| ----------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/schemas/**/*.sql` (default declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | +| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | legacy opt-out catalog miss, cache-enabled COLD shadow provision creates the current key's snapshot; a warm hit `touch`es its mtime; LRU/TTL may delete other keys (`SUPABASE_HOME` overrides the root; `--no-cache` neither reads nor writes) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | legacy opt-out catalog miss, during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers @@ -42,15 +48,18 @@ formatting without disabling safe compaction. ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------- | -------------------------------------------------- | --------- | -| `SUPABASE_ACCESS_TOKEN` | auth token for `--linked` | no | -| `DB_PASSWORD` | password for `--linked` / `--db-url` | no | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | -| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | -| `SUPABASE_SERVICES_HOSTNAME` | local DB host for `--local` | no | -| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | +| Variable | Purpose | Required? | +| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | +| `SUPABASE_ACCESS_TOKEN` | auth token for `--linked` | no | +| `DB_PASSWORD` | password for `--linked` / `--db-url` | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the legacy opt-out's shadow baseline cache | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache for the legacy opt-out's catalog-miss shadows; ON by default, set to `false`/`0` to opt out | no | +| `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines) for the legacy opt-out's catalog-miss shadows; never touches stdout/exit codes | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | +| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | +| `SUPABASE_SERVICES_HOSTNAME` | local DB host for `--local` | no | +| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | ## Exit Codes diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index f10ab8b050..edb37e6a17 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -9,37 +9,42 @@ there is no automatic fallback. Coverage gaps warn; `--strict-coverage` makes them fatal, while `PGDELTA_DEBUG` writes diagnostic JSON under `supabase/.temp/pgdelta/v2/debug//`. Bundled output may use different SQL and ordered transaction-aware files but must apply and converge. `--no-cache` -affects only the legacy opt-out. The bundled formatter defaults to lowercase SQL +bypasses the bundled engine's shadow baseline cache and the legacy opt-out's +catalog + snapshot caches. The bundled formatter defaults to lowercase SQL at width 180; config overrides it, and JSON `null` disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| --------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------ | -| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | -| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | -| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | -| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | -| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | -| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | +| Path | Format | When | +| --------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | +| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | +| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent); separately hashed into the shadow-baseline cache key on every cache-eligible acquire — bundled-engine shadows and the legacy opt-out's catalog miss alike, warm hits included — and applied to a cold shadow's baseline | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — bundled-engine migrations/declarative shadows, and the legacy opt-out's catalog miss; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | abandoned-partial sweep on every cache-eligible acquire (warm hit and cold export) — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | ## Files Written -| Path | Format | When | -| ------------------------------------------------------------------ | ------ | ------------------------------------------------- | -| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | -| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | +| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision creates the current key's snapshot — bundled-engine migrations/declarative shadows, and the legacy opt-out's catalog miss (a catalog hit provisions no shadow; `--no-cache` bypasses the snapshot cache entirely — neither read nor written); a warm hit `touch`es its mtime (LRU); every cache-eligible acquire may delete other keys under LRU keep-8 + 14-day mtime TTL — ~90MB (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | ## Subprocesses / Containers | What | When | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| Two natively-provisioned shadows: migrated source and declarative target | bundled engine | +| Two natively-provisioned shadows (migrated source + declarative target) via `legacyAcquireShadowDatabase` — ephemeral host ports, settings-keyed global baseline cache | bundled engine | | Natively-provisioned shadow Postgres container (`legacyCreateShadowDatabase`/`legacyPrepareShadowSource`) + native migrate; the catalog itself is exported via edge-runtime | legacy opt-out, migrations-catalog cache miss | | Natively-provisioned shadow Postgres container (platform-baseline setup via one-shot auth/storage/realtime migrate jobs, then the declarative directory applied via the pg-delta edge-runtime apply script) → catalog export | legacy opt-out, declarative-catalog cache miss | | Edge-runtime container running the pg-delta diff and, on a catalog cache miss, catalog-export/declarative-apply scripts | legacy opt-out | @@ -47,13 +52,16 @@ disabling safe compaction. ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------- | -------------------------------------------------- | --------- | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | -| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | -| `SUPABASE_SERVICES_HOSTNAME` | local DB host for the bootstrap generate | no | -| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | +| Variable | Purpose | Required? | +| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the shadow baseline cache (and other CLI state) | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | +| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | +| `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | +| `SUPABASE_SERVICES_HOSTNAME` | local DB host for the bootstrap generate | no | +| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | ## Exit Codes @@ -130,3 +138,32 @@ existing SQL or creates an export manifest. shadows. Under the legacy opt-out, both catalog shadows are provisioned in-process using the same primitives as `db diff`; catalog export, declarative apply, and diff run through the edge-runtime pg-delta scripts. + +### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) + +The bundled (pg-delta next) engine provisions both plan shadows through +`legacyAcquireShadowDatabase` (`legacy-pgdelta-next-shadow.layer.ts`): ON by default, +`SUPABASE_SHADOW_CACHE=false`/`=0` opts out (ambient env or project dotenv), and `--no-cache` +bypasses restore and publish for that invocation. Next allocates an ephemeral host port per +shadow; the cache key hashes the cluster recipe (including the effective Webhooks/`pg_net` +policy), not the published port, so worktrees and repeated syncs with the same settings share +a warm hit. The migrations shadow follows project config; the declarative shadow forces +`pg_net` off — those are distinct keys when Webhooks are enabled. A warm hit skips the +platform baseline on both shadows (`legacyMigrateNextShadowDatabase` / +`legacySetupShadowDatabase` are baseline-state-aware). Artifact: +`~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides +the root), keyed by a hash of every input baked into the cluster (including the effective +Webhooks/`pg_net` policy); shared across worktrees with the same settings; retention is LRU +(keep 8) + 14-day mtime TTL (warm hits refresh mtime; sibling tars may be deleted). Container +lifecycle is identical to the uncached path +except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the +command — a warm-path anomaly cold-provisions instead, a cold export failure only warns and +leaves the run uncached (one exception: a shadow that fails to come back up after the snapshot +fails the run rather than reporting a false success). See `shared/db-bootstrap/shadow-cache.ts`. + +Under the legacy opt-out, every catalog-miss shadow (migrations, baseline, declarative) goes +through `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), the same +`legacyWithShadowDatabase` seam `db diff`/`db pull` use. `--no-cache` bypasses that snapshot +cache along with the catalog cache. Catalog provisioners wait with `legacyWaitForShadowReady` +and thread baseline state through `legacySetupShadowDatabase` so a warm hit does not +double-apply the baseline. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 6147f565bc..c32b74101c 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -26,9 +26,9 @@ import { LegacyLinkedProjectCache } from "../../../../../telemetry/legacy-linked import { LegacyTelemetryState } from "../../../../../telemetry/legacy-telemetry-state.service.ts"; import { legacyListLocalMigrations, - legacyPgDeltaTempPath, legacyResolveSetupInputs, } from "../../../../../shared/legacy-pgdelta.cache.ts"; +import { legacyPgDeltaTempPath } from "../../../../../shared/legacy-pgdelta.paths.ts"; import { LegacyPgDeltaEngine } from "../../../shared/legacy-pgdelta-engine.service.ts"; import { legacyIsPgDeltaDebugEnabled, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 26d19fa12f..cf166ed900 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -24,6 +24,7 @@ import { mockLegacyLinkedProjectCacheTracked, mockLegacyPlatformApiService, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, } from "../../../../../../../tests/helpers/legacy-mocks.ts"; import { CliArgs } from "../../../../../../shared/cli/cli-args.service.ts"; @@ -452,6 +453,10 @@ const legacyUuidLoadError = () => describe("legacy db schema declarative sync integration", () => { const tmp = useLegacyTempWorkdir(); + // The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker cp`/ + // `docker start` round trip plus a snapshot tar to every shadow this suite provisions. This suite + // is about the command, not the cache, so it asserts the plain shadow lifecycle. + useLegacyShadowCacheDisabled(); it.effect("gate: fails when pg-delta is not enabled", () => { seedDeclarative(tmp.current); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts index bddfaf7edd..d73968cf6c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts @@ -61,7 +61,7 @@ const toml: LegacyDbTomlValues = { }; function setup() { - const state = { migrations: 0, plan: 0 }; + const state = { migrations: 0, plan: 0, planBypassCache: undefined as boolean | undefined }; const shadow = Layer.succeed(LegacyPgDeltaNextShadow, { provisionMigrations: () => Effect.sync(() => { @@ -71,9 +71,10 @@ function setup() { Effect.fail(new LegacyDeclarativeShadowDbError({ message: "stop after routing" })), ), ), - provisionPlan: () => + provisionPlan: (opts) => Effect.sync(() => { state.plan += 1; + state.planBypassCache = opts.bypassCache; }).pipe( Effect.andThen( Effect.fail(new LegacyDeclarativeShadowDbError({ message: "stop after routing" })), @@ -124,7 +125,8 @@ describe("pg-delta next shadow selection", () => { }) .pipe(Effect.exit); - expect(state).toEqual({ migrations: 0, plan: 0 }); + expect(state.migrations).toBe(0); + expect(state.plan).toBe(0); }).pipe(Effect.provide(layer)); }); @@ -145,7 +147,8 @@ describe("pg-delta next shadow selection", () => { }) .pipe(Effect.exit); - expect(state).toEqual({ migrations: 1, plan: 0 }); + expect(state.migrations).toBe(1); + expect(state.plan).toBe(0); }).pipe(Effect.provide(layer)); }); @@ -172,7 +175,37 @@ describe("pg-delta next shadow selection", () => { }) .pipe(Effect.exit); - expect(state).toEqual({ migrations: 0, plan: 1 }); + expect(state.migrations).toBe(0); + expect(state.plan).toBe(1); + expect(state.planBypassCache).toBeUndefined(); + }).pipe(Effect.provide(layer)); + }); + + it.effect("forwards --no-cache as bypassCache on the declarative plan shadows", () => { + const { state, layer } = setup(); + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + yield* engine + .planDeclarativeSchema({ + ...common, + toml, + files: [{ name: "schema.sql", sql: "create table example(id int);" }], + noCache: true, + setupInputs: { + image: "postgres:17", + majorVersion: 17, + authEnabled: true, + storageEnabled: true, + realtimeEnabled: true, + autoExpose: false, + vaultNames: [], + rolesSql: "", + }, + }) + .pipe(Effect.exit); + + expect(state.plan).toBe(1); + expect(state.planBypassCache).toBe(true); }).pipe(Effect.provide(layer)); }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index b925e23d40..aaeafcf1ec 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -331,6 +331,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( context: input.context, toml: input.toml, ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), + ...(input.noCache ? { bypassCache: true } : {}), }); const migrations = parseLegacyConnectionString(shadow.migrationsUrl); const declarative = parseLegacyConnectionString(shadow.declarativeUrl); @@ -354,6 +355,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( shadowPool: declarativePool, files: input.files, allowDrops: true, + ...(shadow.allowSameDatabaseIdentity ? { allowSameDatabaseIdentity: true } : {}), debug: input.debug, schema: input.schema, formatOptions: input.formatOptions, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts index ebff9b0a4a..222beda809 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts @@ -1,6 +1,6 @@ import { Effect, type FileSystem, type Path } from "effect"; -import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.cache.ts"; +import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.paths.ts"; import type { LegacyPgDeltaNextDiagnostic, LegacyPgDeltaNextOperation, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts index 0c58965625..40a77a4d32 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts @@ -10,7 +10,7 @@ import { legacyPgDeltaNextTempPath, legacySavePgDeltaNextDebugArtifacts, } from "./legacy-pgdelta-next-artifacts.ts"; -import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.cache.ts"; +import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.paths.ts"; describe("pg-delta next artifact generation", () => { it.effect("writes structured non-cache artifacts and metadata under v2", () => { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index 8319962425..24d6423513 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -19,15 +19,18 @@ import { legacyBuildLocalDbContainerInputs, type LegacyLocalDbContainerInputs, } from "../../../shared/db-bootstrap/local-container-inputs.ts"; -import { legacyWaitForHealthyServices } from "../../../shared/db-bootstrap/health-check.ts"; +import { legacyWaitForShadowReady } from "../../../shared/db-bootstrap/health-check.ts"; +import { + legacyAcquireShadowDatabase, + type LegacyShadowAcquiredHandle, + type LegacyShadowCacheOpts, +} from "../../../shared/db-bootstrap/shadow-cache.ts"; import { legacyConnectShadowDatabase, - legacyCreateShadowDatabase, legacyMigrateNextShadowDatabase, legacyRemoveShadowDatabase, legacyShadowRunInputFromLocalContainerInputs, legacySetupShadowDatabase, - type LegacyShadowDatabaseHandle, } from "../../../shared/db-bootstrap/shadow-database.ts"; import { ChildProcessSpawner } from "effect/unstable/process"; import type { ChildProcessSpawner as ChildProcessSpawnerType } from "effect/unstable/process/ChildProcessSpawner"; @@ -79,6 +82,37 @@ interface NativeShadowBase { readonly image: string; } +interface ProvisionedMigrationsShadow extends LegacyPgDeltaNextMigrationsShadow { + readonly snapshotKey: string | undefined; +} + +interface ProvisionedDeclarativeShadow { + readonly declarativeUrl: string; + readonly restoredFromPgDataSnapshot: boolean; + readonly snapshotKey: string | undefined; +} + +/** + * Whether pg-delta's same-database guard must be bypassed for this plan's two shadows — i.e. + * whether they can legitimately report the same PostgreSQL identity (system identifier + + * database OID). That happens exactly when the declarative shadow was physically RESTORED from + * the same snapshot key that also produced the migrations shadow's cluster: same key means same + * tar, and the migrations side is that tar's lineage whether it warm-restored FROM the tar or + * cold-exported it this very run — the baseline handoff, where requiring the migrations handle + * itself to be a warm restore would leave the guard armed against its own clone and fail the + * first cold plan (review: Codex on #6184, P1). A freshly initdb'd declarative shadow always + * carries its own new identity, and different keys mean tars exported from different clusters, + * so both of those stay `false` and keep the guard armed. A `true` alongside identities that + * happen to differ is harmless by design: pg-delta's bypass only takes effect on an exact + * identity match, never on a same-lineage sibling. + */ +export function legacyAllowSameDatabaseIdentityForPlanShadows(opts: { + readonly declarativeRestoredFromPgDataSnapshot: boolean; + readonly sameSnapshotKey: boolean; +}): boolean { + return opts.declarativeRestoredFromPgDataSnapshot && opts.sameSnapshotKey; +} + /** * Removes extensions that the legacy PG14 platform baseline installs implicitly * so the declarative shadow reflects only extension declarations in schema files. @@ -98,7 +132,7 @@ export const legacyPreparePgDeltaNextDeclarativeBaseline = Effect.fnUntraced(fun yield* session.exec('DROP EXTENSION IF EXISTS "uuid-ossp"'); }); -const setupRunInput = (input: NativeShadowInput, handle: LegacyShadowDatabaseHandle) => ({ +const setupRunInput = (input: NativeShadowInput, handle: LegacyShadowAcquiredHandle) => ({ fs: input.base.fs, path: input.base.path, workdir: input.base.workdir, @@ -198,34 +232,55 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( ), }); - const acquireShadow = (input: NativeShadowInput) => - Effect.acquireRelease(legacyCreateShadowDatabase(input.spawner, input.base), (handle) => - legacyRemoveShadowDatabase(input.spawner, handle.containerId).pipe( - Effect.provideService(Output, output), - ), + /** + * Cache-aware acquire, released when the current scope closes — next returns a URL the + * engine keeps using after provision, so this cannot be `legacyWithShadowDatabase` + * (that wrapper removes the container when `use` returns). + */ + const acquireShadow = (input: NativeShadowInput, opts: LegacyShadowCacheOpts) => + Effect.acquireRelease( + legacyAcquireShadowDatabase(input.spawner, input.base, opts), + (handle) => + legacyRemoveShadowDatabase(input.spawner, handle.containerId).pipe( + Effect.provideService(Output, output), + ), ); - const provisionMigrations = (input: NativeShadowInput) => - Effect.gen(function* () { - const handle = yield* acquireShadow(input); - yield* legacyWaitForHealthyServices(input.spawner, [handle.containerId], { + const awaitShadowReady = (input: NativeShadowInput, handle: LegacyShadowAcquiredHandle) => + legacyWaitForShadowReady( + input.spawner, + handle.containerId, + { + host: input.base.hostname, + port: input.base.shadowPort, + user: "postgres", + password: input.base.password, + database: "postgres", + }, + { timeoutSeconds: input.base.healthTimeoutSeconds, - }); + image: input.base.image, + }, + ); + + const provisionMigrations = (input: NativeShadowInput, opts: LegacyShadowCacheOpts) => + Effect.gen(function* () { + const handle = yield* acquireShadow(input, opts); + yield* awaitShadowReady(input, handle); const setup = setupRunInput(input, handle); - yield* legacyMigrateNextShadowDatabase(input.spawner, setup); + yield* legacyMigrateNextShadowDatabase(input.spawner, setup, handle); return { migrationsUrl: legacyToPostgresURL(setup.connConfig), - } satisfies LegacyPgDeltaNextMigrationsShadow; + snapshotKey: handle.snapshotKey, + } satisfies ProvisionedMigrationsShadow; }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); - const provisionDeclarative = (input: NativeShadowInput) => + const provisionDeclarative = (input: NativeShadowInput, opts: LegacyShadowCacheOpts) => Effect.gen(function* () { - const handle = yield* acquireShadow(input); - yield* legacyWaitForHealthyServices(input.spawner, [handle.containerId], { - timeoutSeconds: input.base.healthTimeoutSeconds, - }); + const handle = yield* acquireShadow(input, opts); + yield* awaitShadowReady(input, handle); const setup = setupRunInput(input, handle); - yield* legacySetupShadowDatabase(input.spawner, setup, { webhooks: "disabled" }); + yield* legacySetupShadowDatabase(input.spawner, setup, { webhooks: "disabled" }, handle); yield* Effect.scoped( Effect.gen(function* () { const session = yield* legacyConnectShadowDatabase(setup.connConfig); @@ -235,16 +290,28 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( ); }), ); - return legacyToPostgresURL(setup.connConfig); + return { + declarativeUrl: legacyToPostgresURL(setup.connConfig), + restoredFromPgDataSnapshot: handle.baselinePresent, + snapshotKey: handle.snapshotKey, + } satisfies ProvisionedDeclarativeShadow; }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); + const cacheOpts = ( + opts: LegacyPgDeltaNextShadowInput, + webhooks: NonNullable, + ): LegacyShadowCacheOpts => ({ + webhooks, + ...(opts.bypassCache === true ? { bypassCache: true } : {}), + }); + return LegacyPgDeltaNextShadow.of({ provisionMigrations: (opts) => Effect.gen(function* () { const port = yield* nextPort(); const built = yield* buildNativeBase(opts); const input = buildNativeInput(opts, built, port); - return yield* provisionMigrations(input); + return yield* provisionMigrations(input, cacheOpts(opts, "config")); }).pipe(Effect.mapError(nextShadowError)), provisionPlan: (opts) => Effect.gen(function* () { @@ -253,11 +320,23 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( const built = yield* buildNativeBase(opts); const migrationsInput = buildNativeInput(opts, built, migrationsPort); const declarativeInput = buildNativeInput(opts, built, declarativePort); - const migrations = yield* provisionMigrations(migrationsInput); - const declarativeUrl = yield* provisionDeclarative(declarativeInput); + const migrations = yield* provisionMigrations(migrationsInput, cacheOpts(opts, "config")); + const declarative = yield* provisionDeclarative( + declarativeInput, + cacheOpts(opts, "disabled"), + ); return { - ...migrations, - declarativeUrl, + migrationsUrl: migrations.migrationsUrl, + declarativeUrl: declarative.declarativeUrl, + // Key equality is what encodes lineage: the declarative shadow restored the very tar + // the migrations side either restored or exported this run, so the two clusters are + // physical clones. An absent key (uncached/bypassed/uncachable) is never lineage. + allowSameDatabaseIdentity: legacyAllowSameDatabaseIdentityForPlanShadows({ + declarativeRestoredFromPgDataSnapshot: declarative.restoredFromPgDataSnapshot, + sameSnapshotKey: + migrations.snapshotKey !== undefined && + migrations.snapshotKey === declarative.snapshotKey, + }), } satisfies LegacyPgDeltaNextPlanShadows; }).pipe(Effect.mapError(nextShadowError)), }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts index aa40d07057..9ce2a54731 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts @@ -1,8 +1,11 @@ import { it } from "@effect/vitest"; import { Effect } from "effect"; -import { describe, expect } from "vitest"; +import { describe, expect, it as vitestIt } from "vitest"; -import { legacyPreparePgDeltaNextDeclarativeBaseline } from "./legacy-pgdelta-next-shadow.layer.ts"; +import { + legacyAllowSameDatabaseIdentityForPlanShadows, + legacyPreparePgDeltaNextDeclarativeBaseline, +} from "./legacy-pgdelta-next-shadow.layer.ts"; function recordingSession() { const statements: string[] = []; @@ -42,3 +45,54 @@ describe("legacyPreparePgDeltaNextDeclarativeBaseline", () => { }); }); }); + +describe("legacyAllowSameDatabaseIdentityForPlanShadows", () => { + vitestIt.each([ + { + // The baseline handoff on a cold cache: the migrations shadow exported the tar this very + // run and the declarative shadow warm-restored that same key — an exact physical clone, + // so the guard has to be bypassed even though the migrations handle is not a restore. + scenario: "the declarative shadow restored the tar the migrations shadow just exported", + restored: true, + sameKey: true, + expected: true, + }, + { + // Both sides warm off the same tar on a later run — same lineage, same conclusion. + scenario: "both shadows warm-restored the same key", + restored: true, + sameKey: true, + expected: true, + }, + { + // A freshly initdb'd declarative shadow always carries a brand-new identity, so the guard + // stays armed no matter what the migrations side did. + scenario: "the declarative shadow was cold-provisioned", + restored: false, + sameKey: true, + expected: false, + }, + { + // Restored from a DIFFERENT key's tar: a different originating cluster, own identity. This + // also covers an absent key on either side (uncached/bypassed/uncachable acquisitions), + // which the caller folds into `sameSnapshotKey: false`. + scenario: "the shadows carry different or absent snapshot keys", + restored: true, + sameKey: false, + expected: false, + }, + { + scenario: "neither shadow came from a snapshot", + restored: false, + sameKey: false, + expected: false, + }, + ])("returns $expected when $scenario", ({ restored, sameKey, expected }) => { + expect( + legacyAllowSameDatabaseIdentityForPlanShadows({ + declarativeRestoredFromPgDataSnapshot: restored, + sameSnapshotKey: sameKey, + }), + ).toBe(expected); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts index c71e08311d..d2e7e4f75e 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts @@ -14,12 +14,19 @@ export interface LegacyPgDeltaNextMigrationsShadow { export interface LegacyPgDeltaNextPlanShadows extends LegacyPgDeltaNextMigrationsShadow { /** Independent platform baseline owned by `planSchemaFiles` while loading desired SQL. */ readonly declarativeUrl: string; + /** Both databases are separate servers restored from CLI-owned PGDATA snapshots. */ + readonly allowSameDatabaseIdentity: boolean; } export interface LegacyPgDeltaNextShadowInput { readonly context: LegacyPgDeltaContext; readonly toml: LegacyDbTomlValues; readonly projectRef?: string; + /** + * `db schema declarative sync --no-cache` (and generate's same flag): force a fresh + * shadow baseline instead of restoring/publishing the global snapshot cache. + */ + readonly bypassCache?: boolean; } interface LegacyPgDeltaNextShadowShape { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts index 19fbf800b1..a01b105325 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts @@ -42,15 +42,15 @@ import { import type { LegacyDockerRun } from "../../../shared/legacy-docker-run.service.ts"; import type { LegacyImagePrepullError } from "../../../shared/db-bootstrap/image-prepull.ts"; import type { LegacyHealthCheckTimeoutError } from "../../../shared/db-bootstrap/health-check.ts"; -import { legacyWaitForHealthyServices } from "../../../shared/db-bootstrap/health-check.ts"; +import { legacyWaitForShadowReady } from "../../../shared/db-bootstrap/health-check.ts"; import { legacySeedGlobals } from "../../../shared/legacy-migration-apply.ts"; import { LEGACY_BAD_PATTERN_MESSAGE, legacyPathMatch } from "../../../shared/legacy-path-match.ts"; import { legacyToPostgresURL } from "../../../shared/legacy-postgres-url.ts"; +import type { LegacyShadowAcquiredHandle } from "../../../shared/db-bootstrap/shadow-cache.ts"; import { legacyMigrateShadowDatabase, legacyMigrateNextShadowDatabase, LegacyShadowDbError, - type LegacyShadowDatabaseHandle, type LegacyShadowSetupInput, type LegacyShadowSourceResult, } from "../../../shared/db-bootstrap/shadow-database.ts"; @@ -95,7 +95,10 @@ export type LegacyPrepareShadowSourceError = /** * Port of Go's `PrepareShadowSource` (`apps/cli-go/internal/db/diff/shadow.go:37-91`): - * health-wait against an already-`legacyCreateShadowDatabase`-created shadow -> + * readiness-wait against an already-`legacyCreateShadowDatabase`-created shadow (a direct + * connect probe, `legacyWaitForShadowReady` — NOT the Docker-health gate the long-running `db` + * container uses, which the shadow's own 10s-interval healthcheck cannot satisfy until well + * after Postgres is connectable; see that function's own doc comment) -> * `MigrateShadowDatabase` (platform baseline + local migrations + the `contrib_regression` * template database) -> build the diff-source config -> for legacy local targets, the * declarative-schema override branch. Pg-delta next always compares that migrations shadow @@ -123,7 +126,7 @@ export type LegacyPrepareShadowSourceError = */ export const legacyPrepareShadowSource = ( spawner: Spawner, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, input: LegacyPrepareShadowSourceInput, ): Effect.Effect< LegacyShadowSourceResult, @@ -145,10 +148,6 @@ export const legacyPrepareShadowSource = ( Effect.gen(function* () { const { containerId } = handle; - yield* legacyWaitForHealthyServices(spawner, [containerId], { - timeoutSeconds: input.healthTimeoutSeconds, - }); - const connConfig: LegacyPgConnInput = { host: input.hostname, port: input.shadowPort, @@ -156,20 +155,35 @@ export const legacyPrepareShadowSource = ( password: input.password, database: "postgres", }; + + yield* legacyWaitForShadowReady(spawner, containerId, connConfig, { + timeoutSeconds: input.healthTimeoutSeconds, + image: input.image, + }); + + // `handle` doubles as the baseline state: on a warm shadow-cache hit the cluster it carries + // already holds the platform baseline (so only the template database + user migrations run), + // and on a cache-enabled cold provision it carries the snapshot step that runs between the + // two — see `shadow-cache.ts`/`LegacyShadowBaselineState`. An uncached acquire hands over the + // always-cold state, which reproduces today's sequence exactly. const migrateShadow = input.migrationMode === "pgdelta-next" ? legacyMigrateNextShadowDatabase : legacyMigrateShadowDatabase; - yield* migrateShadow(spawner, { - fs: input.fs, - path: input.path, - workdir: input.workdir, - projectId: input.projectId, - container: containerId, - networkId: input.networkId, - connConfig, - setup: input.setup, - }); + yield* migrateShadow( + spawner, + { + fs: input.fs, + path: input.path, + workdir: input.workdir, + projectId: input.projectId, + container: containerId, + networkId: input.networkId, + connConfig, + setup: input.setup, + }, + handle, + ); const sourceUrl = legacyToPostgresURL(connConfig); diff --git a/apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts b/apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts index 5d1f5dcb75..dafd73c57d 100644 --- a/apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts +++ b/apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts @@ -15,6 +15,7 @@ import { mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, legacySequentialExecBatch, } from "../../../../../tests/helpers/legacy-mocks.ts"; @@ -432,6 +433,10 @@ const failureTag = (exit: Exit.Exit): string | undefined => { }; const tmp = useLegacyTempWorkdir(); +// The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker cp`/ +// `docker start` round trip plus a snapshot tar to every shadow this suite provisions. This suite +// is about the command, not the cache, so it asserts the plain shadow lifecycle. +useLegacyShadowCacheDisabled(); describe("legacy migration squash", () => { describe("flag surface & ordering", () => { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts index 2714e8bad2..2292b254dc 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts @@ -45,6 +45,11 @@ import { /** Structural element type of {@link LegacyStartContainerSpec.secretFiles} — not exported from `docker-create-args.ts`, so referenced positionally here. */ type LegacyStartSecretFileSpec = NonNullable[number]; +/** Structural element type of {@link LegacyStartContainerSpec.preStartArchives} — same reasoning as {@link LegacyStartSecretFileSpec}. */ +type LegacyStartPreStartArchiveSpec = NonNullable< + LegacyStartContainerSpec["preStartArchives"] +>[number]; + type Spawner = ChildProcessSpawner["Service"]; /** @@ -765,6 +770,50 @@ function legacyCopyStartSecretFilesIntoContainer( ); } +/** + * `docker cp - :` with one + * {@link LegacyStartContainerSpec.preStartArchives} entry's tar bytes on stdin — the ONE `docker + * cp` form that preserves each archive member's uid/gid inside the container (the host-path form + * rewrites ownership to root, which a restored Postgres data directory cannot survive; see that + * field's own doc comment). + * + * Sequenced by {@link legacyCreateContainer} between `docker create` and `docker start`, for the + * same two reasons the secret-file copies are: the container must exist for `docker cp` to have a + * target, and must not be running yet so its entrypoint never races the copy — which for an + * archive is not merely a race but the whole point, since the entrypoint's behavior depends on + * what it finds already unpacked. + */ +function legacyExtractPreStartArchiveIntoContainer( + spawner: Spawner, + containerId: string, + archive: LegacyStartPreStartArchiveSpec, +): Effect.Effect { + const failure = (detail: string) => + new LegacyContainerCreateError({ + message: `failed to create docker container: failed to restore archive into container${detail}`, + reason: "runtime", + }); + return Effect.scoped( + Effect.gen(function* () { + const child = yield* spawnContainerCli( + spawner, + ["cp", "-", `${containerId}:${archive.containerPath}`], + { stdin: archive.tar, stdout: "ignore", stderr: "pipe" }, + ).pipe(Effect.mapError((cause) => failure(`: ${legacyDescribeContainerCliFailure(cause)}`))); + const [exitCode, stderr] = yield* Effect.all( + [child.exitCode.pipe(Effect.map(Number)), legacyCollectText(child.stderr)], + { concurrency: "unbounded" }, + ).pipe(Effect.mapError(() => failure(""))); + if (exitCode !== 0) { + const message = stderr.trim(); + return yield* Effect.fail( + failure(message.length > 0 ? `: ${message}` : `: exit ${exitCode}`), + ); + } + }), + ); +} + /** * Port of Go's `DockerStart` (`apps/cli-go/internal/utils/docker.go:363-440`), * minus image resolution (already done by `image-prepull.ts`) and network @@ -783,7 +832,11 @@ function legacyCopyStartSecretFilesIntoContainer( * Runs strictly between `docker create` and `docker start`: the container * must already exist for `docker cp` to have a target, and must not be * running yet so its entrypoint never races the copy. - * 6. `docker start`. + * 6. Unpack any `preStartArchives` into the same created-but-unstarted + * container via `docker cp -` (`legacyExtractPreStartArchiveIntoContainer`) + * — also TS-port-only, and for the shadow baseline cache's restored PGDATA + * the "not started yet" half of step 5's ordering is the entire point. + * 7. `docker start`. * * Resolves to the created container's id/name on success. */ @@ -837,6 +890,29 @@ export function legacyCreateContainer( containerId, finalSpec.secretFiles ?? [], ); + // Sequentially, not concurrently like the secret files: two archives could legitimately + // overlap in the container's filesystem, so the spec's own order has to be the applied order. + // + // A failed extraction removes the just-created container (best-effort, `-v` so an anonymous + // volume goes with it) before failing. The secret-file/`docker start` steps deliberately do + // NOT do this (Go-parity leak window — see `legacyCreateShadowDatabase`'s doc comment, + // `shadow-database.ts`), but `preStartArchives` is TS-only with no Go counterpart, and its one + // producer (the shadow baseline cache's warm restore) recovers from this exact failure by + // provisioning a replacement — which must not accumulate an orphaned created container per + // recovery (review: Codex on #6184). + yield* Effect.forEach( + finalSpec.preStartArchives ?? [], + (archive) => legacyExtractPreStartArchiveIntoContainer(spawner, containerId, archive), + { discard: true }, + ).pipe( + Effect.tapError(() => + containerCliExitCode(spawner, ["rm", "-f", "-v", containerId], { + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + }).pipe(Effect.orElseSucceed(() => 0)), + ), + ); yield* legacyDockerStartContainer(spawner, containerId, finalSpec); return containerId; }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index 9378a2d360..af334db801 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -175,9 +175,11 @@ type Spawner = ChildProcessSpawner["Service"]; /** * Go's inline `RevokeDefaultDataApiPrivilegesSql` constant (`start.go:405-412`) — * NOT a `//go:embed` file (unlike the three large SQL templates), so transcribed - * directly here rather than as a sibling `templates/*.sql.ts` module. + * directly here rather than as a sibling `templates/*.sql.ts` module. Exported for the + * shadow baseline cache's embedded-SQL digest (`shadow-cache.ts`), which must re-key + * whenever this text changes across CLI releases. */ -const LEGACY_START_REVOKE_API_PRIVILEGES_SQL = ` +export const LEGACY_START_REVOKE_API_PRIVILEGES_SQL = ` alter default privileges for role postgres in schema public revoke select, insert, update, delete on tables from anon, authenticated, service_role; alter default privileges for role postgres in schema public @@ -186,7 +188,12 @@ alter default privileges for role postgres in schema public revoke execute on functions from anon, authenticated, service_role; `; -const LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL = +/** + * Exported for the shadow baseline cache's embedded-SQL digest (`shadow-cache.ts`), same as + * {@link LEGACY_START_REVOKE_API_PRIVILEGES_SQL}: a webhooks-enabled baseline bakes this + * statement into PGDATA, so the digest must re-key whenever this text changes across releases. + */ +export const LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL = "create extension if not exists pg_net schema extensions;"; // The historical PG14 dump installs pg_net because later statements grant on diff --git a/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts b/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts index 7219fe2b9b..99bae326ef 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts @@ -45,14 +45,20 @@ * call site actually needs one; there is no value in modelling Docker surface * this builder never has to reproduce. * - * One field has no Go struct equivalent at all: {@link LegacyStartContainerSpec.secretFiles}. - * It exists purely because this module's own "shell out to `docker create`" - * architecture (unlike Go's direct Engine API calls) has an argv-exposure - * problem `container.Config`/`container.HostConfig` never had — see that - * field's doc comment, and `container-lifecycle.ts`'s `legacyCreateContainer`, - * for the mitigation. + * Two fields have no Go struct equivalent at all: + * {@link LegacyStartContainerSpec.secretFiles} and + * {@link LegacyStartContainerSpec.preStartArchives}. The first exists purely + * because this module's own "shell out to `docker create`" architecture (unlike + * Go's direct Engine API calls) has an argv-exposure problem + * `container.Config`/`container.HostConfig` never had; the second because Go + * destroys its shadow container on every run and so never needs to seed a + * container's filesystem before it starts. See each field's doc comment, and + * `container-lifecycle.ts`'s `legacyCreateContainer`, for how both are + * delivered. */ +import type { PlatformError, Stream } from "effect"; + import { legacyBindMountSpecSource, legacyIsBindMountSource, @@ -123,6 +129,26 @@ interface LegacyStartSecretFileSpec { readonly content: string; } +/** + * One tar archive to extract into the created-but-not-yet-started container — see + * {@link LegacyStartContainerSpec.preStartArchives}'s doc comment for the full contract. Not + * exported on its own: callers reference it structurally through that field. + */ +interface LegacyStartPreStartArchiveSpec { + /** + * The directory INSIDE the container the archive's members are unpacked relative to, i.e. + * `docker cp - :`. Always a POSIX container path, never a host path. + */ + readonly containerPath: string; + /** + * The tar bytes, as a lazily-consumed stream (typically `FileSystem.stream(hostTarPath)`). + * A stream rather than a host path on purpose: it keeps the archive's own storage decisions + * with the producer, and `legacyCreateContainer` needs no `FileSystem` in its own context to + * deliver it. + */ + readonly tar: Stream.Stream; +} + export interface LegacyStartContainerSpec { /** `container.Config.Image` (already resolved/pulled — resolution is out of scope here). */ readonly image: string; @@ -192,6 +218,25 @@ export interface LegacyStartContainerSpec { * over the Engine API) for that same reason. */ readonly secretFiles?: ReadonlyArray; + /** + * Tar archives to unpack into the container's own filesystem AFTER `docker create` and + * strictly BEFORE `docker start` — the shape `docker cp - :` (tar on + * stdin) implements. No Go equivalent: Go's shadow container is destroyed on every run, so + * nothing in the Go CLI ever seeds a container's filesystem ahead of its entrypoint. + * + * The one consumer today is the shadow baseline cache (`shadow-cache.ts`), which restores a + * previously exported PGDATA directory so the `supabase/postgres` entrypoint finds a + * `PG_VERSION` file and skips `initdb` entirely. The delivery form matters and is not + * interchangeable with {@link secretFiles}': `docker cp :` resets the + * copied tree's ownership to the archive-extracting user (root), and Postgres refuses to + * start on a data directory it does not own, whereas the tar-STREAM form preserves each + * member's uid/gid verbatim. + * + * NOT consumed here — {@link legacyBuildStartContainerCreateArgs} stays pure/no-I/O and never + * reads this field, exactly like {@link secretFiles}. `container-lifecycle.ts`'s + * `legacyCreateContainer` is the sole consumer. + */ + readonly preStartArchives?: ReadonlyArray; /** * `container.Config.Entrypoint`'s first element. Docker CLI's `--entrypoint` * only accepts a single executable/script name (unlike the Engine API field, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts index ab421387e2..f82c85e373 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts @@ -10,7 +10,7 @@ * and only the final timeout's failures surface to the caller. */ -import { Data, Effect, Schedule, Stream } from "effect"; +import { Clock, Data, Duration, Effect, Result, Schedule, Stream } from "effect"; import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientRequest from "effect/unstable/http/HttpClientRequest"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; @@ -24,8 +24,10 @@ import { legacySpawnContainerCliWithRuntime, type LegacyContainerRuntime, } from "../legacy-container-cli.ts"; +import { LegacyDbConnection, type LegacyPgConnInput } from "../legacy-db-connection.service.ts"; import { legacyInspectContainerState } from "../legacy-docker-lifecycle.ts"; import { legacyKongAuthHeaders } from "../legacy-kong-auth.ts"; +import { legacyShadowDebugEnabled, legacyShadowDebugTruncate } from "./shadow-debug.ts"; type Spawner = ChildProcessSpawner["Service"]; @@ -437,3 +439,236 @@ export function legacyWaitForHealthyServices( ); }); } + +/** + * Bounds a single readiness connect so a dial that hangs (rather than being + * refused outright) cannot swallow the whole poll budget. Explicit rather than + * inherited from the driver's own local default (`legacy-db-connection.sql-pg. + * layer.ts` — `cfg.connectTimeoutSeconds ?? (isLocal ? 2 : 10)`), which happens + * to be the same 2 seconds: this probe's budget is a property of the polling + * loop, not of whichever driver layer answers it. + */ +const LEGACY_SHADOW_READY_CONNECT_TIMEOUT_SECONDS = 2; + +/** + * One round's verdict. `fatal` is what makes an exited container fail fast + * instead of burning the remaining budget: nothing about a dead container can + * change on a later round, whereas a refused connect (or a transient `docker + * container inspect` failure) is just "not ready yet". + */ +interface LegacyShadowReadyFailure { + readonly reason: string; + readonly fatal: boolean; +} + +const legacyShadowNotReady = (reason: string): LegacyShadowReadyFailure => ({ + reason, + fatal: false, +}); + +/** + * A single short-lived connect attempt against the shadow, dialled exactly the + * way `legacyConnectShadowDatabase` (`shadow-database.ts`) dials it — same + * `isLocal`/`dnsResolver` pair, so a config that authenticates for the probe + * authenticates for the real connection too. `Effect.scoped` closes the session + * the moment the probe resolves: the caller opens (and owns) its own connection + * afterwards through `legacyConnectShadowDatabase`. + */ +const legacyProbeShadowConnect = ( + connConfig: LegacyPgConnInput, +): Effect.Effect => + Effect.scoped( + Effect.gen(function* () { + const dbConnection = yield* LegacyDbConnection; + yield* dbConnection.connect( + { ...connConfig, connectTimeoutSeconds: LEGACY_SHADOW_READY_CONNECT_TIMEOUT_SECONDS }, + { isLocal: true, dnsResolver: "native" }, + ); + }), + ).pipe(Effect.mapError((cause) => legacyShadowNotReady(cause.message))); + +export interface LegacyWaitForShadowReadyOptions { + readonly timeoutSeconds?: number; + /** The shadow container's already-resolved postgres image, named in the exec-format recovery hint. */ + readonly image?: string; +} + +/** + * The shadow database's readiness gate — {@link legacyWaitForHealthyServices}'s + * counterpart for the ONE container whose Docker healthcheck cannot answer in + * time. The shadow's healthcheck is `interval=10s` with no + * `start_period`/`start_interval` (`postgres.service.ts`, deliberately left + * unchanged — other tooling reads that config), so Docker's very first probe + * runs at t+10s while Postgres has been accepting connections since ~3.5s: + * gating on `Health.Status` spends ~6.5s per provision waiting for a verdict + * that is already knowable. `--health-start-interval` would fix the container + * side, but it needs Docker Engine 25+/API 1.44 and is not reliably supported + * by Podman (which `spawnContainerCli` falls back to), so the CLI-side wait + * asks Postgres directly instead. + * + * Each round, on the same 1-second constant backoff and the same + * `timeoutSeconds` budget as the health gate: + * + * 1. {@link legacyInspectContainerState} — still `running`? This preserves the + * crash detection the health gate provided; an exited container fails + * immediately rather than at the end of the budget. + * 2. a short {@link legacyProbeShadowConnect} — success ⇒ ready, return now. + * + * The round count is not the only bound: the whole wait also carries a wall-clock cap, since a + * round that hangs costs its own connect timeout on top of the 1-second delay — see + * `boundSeconds` below. + * + * Failure shape is deliberately identical to the health gate's: the same + * {@link LegacyHealthCheckTimeoutError}, the same `unhealthy` payload, and the + * same `docker logs` dump teed to stderr on the way out, so a broken shadow + * still surfaces exactly what it surfaced before. It additionally attaches the + * exec-format recovery {@link suggestion} when the caller names the shadow's + * image via {@link LegacyWaitForShadowReadyOptions.image} — something the old + * shadow call sites never got, since none of them ever passed `images` to + * {@link legacyWaitForHealthyServices} either. This closes that pre-existing + * gap rather than restoring parity with a behavior that previously existed. + */ +export function legacyWaitForShadowReady( + spawner: Spawner, + containerId: string, + connConfig: LegacyPgConnInput, + opts: LegacyWaitForShadowReadyOptions = {}, +): Effect.Effect { + const timeoutSeconds = opts.timeoutSeconds ?? LEGACY_HEALTH_CHECK_TIMEOUT_SECONDS; + // Checked once per call (never cached at module load — a test, or a long-lived process, + // mutates `process.env` and expects the next call to see it) — see `shadow-debug.ts`'s own + // doc comment. `Output` is not in this function's own context (only `LegacyDbConnection` is), + // so debug lines here go straight to stderr via `process.stderr.write` rather than widening + // this function's `R` just for a debug-only concern — the same tradeoff + // `legacyStreamContainerLogsOnce` above already makes for its own log-teeing. + const debug = legacyShadowDebugEnabled(); + + // The most recent attempt's failure, kept even once a later attempt succeeds — the debug + // summary line reports it either way (see the doc comment below on the completion line), and + // the elapsed-time bound below replays it so an expiry fails exactly like an exhausted retry. + let lastFailure: LegacyShadowReadyFailure | undefined; + + const rawProbe: Effect.Effect = Effect.gen( + function* () { + const state = yield* legacyInspectContainerState(spawner, containerId).pipe( + Effect.mapError((cause) => legacyShadowNotReady(cause.message)), + ); + if (!state.running) { + return yield* Effect.fail({ + reason: `container is not running: ${state.status}`, + fatal: true, + }); + } + yield* legacyProbeShadowConnect(connConfig); + }, + ).pipe( + Effect.tapError((failure) => + Effect.sync(() => { + lastFailure = failure; + }), + ), + ); + + let attempts = 0; + + // Debug-only per-attempt line: `shadow-debug: ready-attempt ms `. + // Exploration 1a's key data point — whether a cold attempt burns the full connect timeout. + const probe: Effect.Effect = !debug + ? rawProbe + : Effect.gen(function* () { + attempts += 1; + const attemptNumber = attempts; + const start = yield* Clock.currentTimeMillis; + const outcome = yield* Effect.result(rawProbe); + const elapsed = (yield* Clock.currentTimeMillis) - start; + yield* Effect.sync(() => { + globalThis.process.stderr.write( + `shadow-debug: ready-attempt ${attemptNumber} ${elapsed}ms ${ + Result.isSuccess(outcome) + ? "ok" + : `error: ${legacyShadowDebugTruncate(outcome.failure.reason)}` + }\n`, + ); + }); + return yield* Effect.fromResult(outcome); + }); + + // Same policy as `legacyWaitForHealthyServices` (Go's + // `NewBackoffPolicy(ctx, timeout)`): a 1-second constant delay, capped at + // `timeoutSeconds` retries after the initial attempt. + const schedule = Schedule.max([Schedule.spaced("1 seconds"), Schedule.recurs(timeoutSeconds)]); + + /** + * The wall-clock cap over the WHOLE wait. `Schedule.recurs` counts attempts, not seconds, and + * an attempt is not free: a dial that hangs burns its full + * {@link LEGACY_SHADOW_READY_CONNECT_TIMEOUT_SECONDS} before the next 1-second delay even + * starts, so a nominal 30-second budget could run ~90 seconds of real time (review: Codex on + * #6184). The cap is the schedule's own total spacing PLUS one attempt's connect allowance, so + * it lands strictly after a clean retries-exhausted finish: the retry count still decides every + * ordinary run (the last round's probe is never cut short), and this only fires when attempts + * themselves overrun. + */ + const boundSeconds = timeoutSeconds + LEGACY_SHADOW_READY_CONNECT_TIMEOUT_SECONDS; + + const waited: Effect.Effect = probe.pipe( + Effect.retry({ schedule, while: (failure) => !failure.fatal }), + // Expiry re-raises the last attempt's own failure (or, if nothing has failed yet, the first + // attempt is still hanging) so it flows into the same handler below — a timed-out wait + // reports the same error, the same log dump, and the same stderr text as an exhausted one. + Effect.timeoutOrElse({ + duration: Duration.seconds(boundSeconds), + orElse: () => + Effect.fail( + lastFailure ?? + legacyShadowNotReady(`not ready after ${boundSeconds}s: connection attempt hung`), + ), + }), + Effect.catch((failure) => + Effect.gen(function* () { + // Go skips this dump on context cancellation (`start.go:215`) — an + // interrupted fiber never reaches this handler, so no separate check. + const scan = yield* legacyDumpContainerLogs(spawner, containerId); + // Unlike the health gate, the old shadow call sites never named an + // image here at all — so this `opts.image` check is what makes a + // shadow's exec-format failure carry the hint for the first time. + const suggestion = + scan.found && opts.image !== undefined + ? legacyExecFormatRecoveryHint( + [containerId], + new Map([[containerId, opts.image]]), + scan.runtime ?? "docker", + ) + : undefined; + return yield* Effect.fail( + new LegacyHealthCheckTimeoutError({ + // The health gate's own ` ` joining, so both waits read + // identically on stderr. + message: `${containerId} ${failure.reason}`, + unhealthy: [{ containerId, reason: failure.reason }], + ...(suggestion === undefined ? {} : { suggestion }), + }), + ); + }), + ), + ); + + if (!debug) return waited; + + // Debug-only completion line, emitted on success OR failure: + // `shadow-debug: ready-wait ms attempts=[ last-error=""]`. + return Effect.gen(function* () { + const start = yield* Clock.currentTimeMillis; + const outcome = yield* Effect.result(waited); + const elapsed = (yield* Clock.currentTimeMillis) - start; + const lastErrorSegment = + lastFailure === undefined + ? "" + : ` last-error="${legacyShadowDebugTruncate(lastFailure.reason)}"`; + yield* Effect.sync(() => { + globalThis.process.stderr.write( + `shadow-debug: ready-wait ${elapsed}ms attempts=${attempts}${lastErrorSegment}\n`, + ); + }); + return yield* Effect.fromResult(outcome); + }); +} diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts index 6e64b732e5..cbee13eb4e 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts @@ -6,9 +6,16 @@ import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; import * as TestClock from "effect/testing/TestClock"; +import { LegacyDbConnectError } from "../legacy-db-connection.errors.ts"; +import { + LegacyDbConnection, + type LegacyDbSession, + type LegacyPgConnInput, +} from "../legacy-db-connection.service.ts"; import { LegacyHealthCheckTimeoutError, legacyWaitForHealthyServices, + legacyWaitForShadowReady, type LegacyHealthCheckPostgrestGateway, } from "./health-check.ts"; @@ -774,3 +781,329 @@ describe("legacyWaitForHealthyServices", () => { ); }); }); + +const SHADOW_CONTAINER_ID = "abc123456789shadow"; + +const shadowConnConfig: LegacyPgConnInput = { + host: "127.0.0.1", + port: 54320, + user: "postgres", + password: "postgres", + database: "postgres", +}; + +/** + * A `LegacyDbConnection` whose `connect` fails the first `failTimes` calls + * (`Number.POSITIVE_INFINITY` never succeeds), recording every dialled config + * and how many probe sessions were released. `closedSessions` is what proves + * the readiness probe hands nothing back to the caller: the downstream code + * opens its own connection through `legacyConnectShadowDatabase` afterwards. + */ +function mockShadowDbConnection( + opts: { readonly failTimes?: number; readonly connectMillis?: number } = {}, +) { + const failTimes = opts.failTimes ?? 0; + const session: LegacyDbSession = { + exec: () => Effect.void, + query: () => Effect.succeed([]), + execBatch: () => Effect.void, + extensionExists: () => Effect.succeed(false), + copyToCsv: () => Effect.succeed(new Uint8Array()), + queryRaw: () => Effect.succeed({ fields: [], rows: [], commandTag: "" }), + }; + const attempts: Array = []; + let closedSessions = 0; + const layer = Layer.succeed(LegacyDbConnection, { + connect: (cfg) => + Effect.gen(function* () { + attempts.push(cfg); + // A dial that hangs before answering — how a real attempt burns its own connect timeout. + if (opts.connectMillis !== undefined) yield* Effect.sleep(opts.connectMillis); + if (attempts.length <= failTimes) { + return yield* Effect.fail(new LegacyDbConnectError({ message: "connection refused" })); + } + yield* Effect.addFinalizer(() => + Effect.sync(() => { + closedSessions += 1; + }), + ); + return session; + }), + }); + return { + layer, + attempts, + get closedSessions() { + return closedSessions; + }, + }; +} + +/** The timeout path tees the container's logs to the real stderr — swallow them. */ +function withSilencedStderr(effect: Effect.Effect): Effect.Effect { + return Effect.suspend(() => { + const originalWrite = globalThis.process.stderr.write.bind(globalThis.process.stderr); + globalThis.process.stderr.write = (() => true) as typeof globalThis.process.stderr.write; + return effect.pipe( + Effect.ensuring( + Effect.sync(() => { + globalThis.process.stderr.write = originalWrite; + }), + ), + ); + }); +} + +const inspectCalls = (mock: ReturnType) => + mock.spawned.filter((args) => args[0] === "container" && args[1] === "inspect"); + +describe("legacyWaitForShadowReady", () => { + it.effect( + "resolves on the first successful connect, while Docker still reports the healthcheck as starting", + () => + Effect.gen(function* () { + // The shadow container's healthcheck runs on a 10s interval with no + // start period, so Docker cannot report `healthy` before t+10s even + // though Postgres accepts connections at ~3.5s — the whole point of + // this wait is that the health status is never consulted at all. + const mock = mockHealthSpawner(() => runningStarting); + const db = mockShadowDbConnection(); + + const exit = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 30 }, + ).pipe(Effect.provide(db.layer), Effect.exit); + + expect(Exit.isSuccess(exit)).toBe(true); + expect(inspectCalls(mock)).toHaveLength(1); + expect(db.attempts).toHaveLength(1); + // Bounded, so a hung dial cannot eat a whole poll round's budget. + expect(db.attempts[0]?.connectTimeoutSeconds).toBe(2); + // Released immediately: the caller opens its own connection afterwards. + expect(db.closedSessions).toBe(1); + }), + ); + + it.effect("keeps polling a refused connect on a 1-second backoff until it succeeds", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting); + const db = mockShadowDbConnection({ failTimes: 2 }); + + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 30 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + yield* TestClock.adjust("1 seconds"); + const exit = yield* Fiber.await(fiber); + + expect(Exit.isSuccess(exit)).toBe(true); + // 2 refused attempts, then the 3rd that connects — a refused connect is + // "not ready yet", never an error in its own right. + expect(db.attempts).toHaveLength(3); + expect(inspectCalls(mock)).toHaveLength(3); + expect(db.closedSessions).toBe(1); + }), + ); + + it.effect( + "fails with LegacyHealthCheckTimeoutError and dumps the container's logs when it never becomes connectable", + () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 2 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.unhealthy).toEqual([ + { containerId: SHADOW_CONTAINER_ID, reason: "connection refused" }, + ]); + expect(error.message).toBe(`${SHADOW_CONTAINER_ID} connection refused`); + // 1 initial attempt + `timeoutSeconds` retries, same budget as the + // Docker-health gate this replaces. + expect(db.attempts).toHaveLength(3); + expect( + mock.spawned.some((args) => args[0] === "logs" && args[1] === SHADOW_CONTAINER_ID), + ).toBe(true); + }), + ); + + it.effect("stops on elapsed time, not on the retry count, when every attempt hangs", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting); + // 1.2s per dial: the round trip is now 2.2s (dial + the 1-second backoff), so the + // `timeoutSeconds` retry count alone would keep polling until t=7.8s. + const db = mockShadowDbConnection({ + failTimes: Number.POSITIVE_INFINITY, + connectMillis: 1200, + }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 3 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + // The whole wait's wall-clock cap: `timeoutSeconds` of backoff plus one dial's + // 2-second connect allowance. Joining here would hang if the wait were still counting + // rounds instead of seconds. + yield* TestClock.adjust("5 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + // Cut off mid-way through the third dial — the 4-attempt retry budget never ran out. + expect(db.attempts).toHaveLength(3); + // Same failure the exhausted path produces: the last completed attempt's own reason, + // the same ` ` message, and the same log dump. + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.message).toBe(`${SHADOW_CONTAINER_ID} connection refused`); + expect(error.unhealthy).toEqual([ + { containerId: SHADOW_CONTAINER_ID, reason: "connection refused" }, + ]); + expect( + mock.spawned.some((args) => args[0] === "logs" && args[1] === SHADOW_CONTAINER_ID), + ).toBe(true); + }), + ); + + it.effect("fails fast, well inside the budget, once the container has exited", () => + Effect.gen(function* () { + // Running on the first round, gone on every round after it. + const mock = mockHealthSpawner((_id, callIndex) => + callIndex === 0 ? runningStarting : notRunning, + ); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 30 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.unhealthy).toEqual([ + { containerId: SHADOW_CONTAINER_ID, reason: "container is not running: exited" }, + ]); + // A dead container can never become connectable — one second in, not 30. + expect(inspectCalls(mock)).toHaveLength(2); + expect(db.attempts).toHaveLength(1); + expect( + mock.spawned.some((args) => args[0] === "logs" && args[1] === SHADOW_CONTAINER_ID), + ).toBe(true); + }), + ); + + describe("exec format error recovery advice", () => { + it.effect("names the shadow's image when its dumped logs show exec format error", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting, { + [SHADOW_CONTAINER_ID]: ["exec /docker-entrypoint.sh: exec format error\n"], + }); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 1, image: "public.ecr.aws/supabase/postgres:15.1.0.147" }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.suggestion).toContain( + `${SHADOW_CONTAINER_ID}'s image public.ecr.aws/supabase/postgres:15.1.0.147`, + ); + expect(error.suggestion).toContain( + "supabase stop\n docker image rm -f public.ecr.aws/supabase/postgres:15.1.0.147\n supabase start", + ); + }), + ); + + it.effect("stays silent when no image is named, even if the logs show exec format error", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting, { + [SHADOW_CONTAINER_ID]: ["exec /docker-entrypoint.sh: exec format error\n"], + }); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 1 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.suggestion).toBeUndefined(); + }), + ); + + it.effect("stays silent for a plain not-ready timeout even when an image is named", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 1, image: "public.ecr.aws/supabase/postgres:15.1.0.147" }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.suggestion).toBeUndefined(); + }), + ); + }); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts new file mode 100644 index 0000000000..6974cc5996 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts @@ -0,0 +1,669 @@ +/** + * Generic PGDATA snapshot/restore primitives — container-agnostic on purpose. `shadow-cache.ts` + * is the only caller today, but nothing here assumes "shadow": these are the building blocks for + * savepointing ANY local Postgres container and restoring it into a fresh one — the shadow's + * disk-level baseline cache today, a future save/restore for the long-running + * `supabase_db_` stack container tomorrow. + * + * **Coherence contract:** the container must be STOPPED before {@link legacyExportPgDataTar} + * runs — a snapshot of a running Postgres's data directory is not a consistent thing to copy. + * Callers own the stop/start around the export; this module only moves bytes. + * + * TODO(hot-save): the STOPPED contract could generalize to a consistency MODE. `frozen` — + * `docker pause` → copy → `docker unpause` — yields a crash-consistent copy (connections stall + * ~1s instead of dropping; restore boots through normal WAL recovery). `online` — + * `pg_backup_start()` → fuzzy copy → `pg_backup_stop()`, writing the returned `backup_label` + * into the artifact — is the zero-stall, Postgres-native form, and the ONLY one that also works + * for a future NATIVE (non-container) Postgres process, where no freezer exists and recovery + * replays the backup-labeled WAL range instead. Neither is worth the surface for the shadow + * cache (its export runs once per key on an already-cold path); implement when a live-stack + * savepoint feature needs to export without downtime. + * + * **Ownership caveat:** the restore side ({@link legacyPgDataRestoreArchive}) MUST be delivered as + * a tar stream unpacked via `docker cp - :`, never a directory copy — a directory copy + * resets ownership to the extracting user (root) and Postgres refuses to start on a data directory + * it does not own, whereas the tar-stream form preserves each member's uid/gid verbatim. + * + * The artifact is a plain file, so it fits a future NATIVE (non-Docker) Postgres service just as + * well — nothing about the format is container-specific. + */ + +import { Effect, Option, Stream, type FileSystem } from "effect"; +import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; + +import { + legacyCollectText, + legacyDescribeContainerCliFailure, + spawnContainerCli, +} from "../legacy-container-cli.ts"; +import type { LegacyStartContainerSpec } from "./docker-create-args.ts"; + +type Spawner = ChildProcessSpawner["Service"]; + +/** + * `PGDATA` in every `supabase/postgres` image — the directory {@link legacyExportPgDataTar} + * exports and {@link legacyPgDataRestoreArchive} restores. Hardcoded rather than read from the + * container's own `PGDATA` env: the entrypoint scripts this codebase generates + * (`postgres.service.ts`) never override it, and the value is part of the tar's own layout, so a + * mismatch has to be a deliberate change here. + */ +export const LEGACY_PGDATA_PATH = "/var/lib/postgresql/data"; + +/** + * `docker cp - :` unpacks the archive's members RELATIVE to `dest`, and + * {@link LEGACY_PGDATA_PATH}'s export tar has `data/` as its own top-level member, so the restore + * target is PGDATA's parent. A POSIX constant, not `path.dirname` — this is a container path and + * must not follow the host's separator. + */ +export const LEGACY_PGDATA_PARENT_PATH = "/var/lib/postgresql"; + +// --------------------------------------------------------------------------- +// What a valid snapshot must contain +// --------------------------------------------------------------------------- + +/** + * PGDATA's own directory name — `docker cp : -` names its members after the source + * BASENAME, so `data/` is the export tar's top-level entry (see {@link LEGACY_PGDATA_PARENT_PATH}). + */ +const LEGACY_PGDATA_DIR_NAME = LEGACY_PGDATA_PATH.slice(LEGACY_PGDATA_PARENT_PATH.length + 1); + +/** + * The entry whose presence proves an archive really carries an exported cluster: every Postgres + * data directory has a `PG_VERSION` file at its root, and `initdb` writes it first. An archive that + * unpacks cleanly but lacks it is the corruption a restore cannot otherwise notice. + * + * On its own it proves only "*a* PostgreSQL cluster", which is strictly weaker than what a cache + * key promises — hence {@link LEGACY_PGDATA_BASELINE_MARKER_ENTRY}. + */ +export const LEGACY_PGDATA_CLUSTER_ENTRY = `${LEGACY_PGDATA_DIR_NAME}/PG_VERSION`; + +/** + * A file this module writes into PGDATA's ROOT ({@link legacyStampPgDataBaselineMarker}) as the + * last step before the export copies the directory out. Postgres ignores unknown regular files at + * the data directory's root (`pg_upgrade` and friends routinely leave some there), and a restored + * container simply carries it along, so the cost of the stamp is one 512-byte tar member. + * + * SCREAMING_SNAKE on purpose, and not by taste: `docker cp` tars a directory through Go's + * `filepath.Walk`, which visits each level in sorted order, so an uppercase root file lands + * immediately next to `PG_VERSION` — near the front of a ~90MB archive rather than behind every + * `base/` page. That is a PERFORMANCE hint for {@link legacyValidatePgDataArchive} only: + * the scan is correct at any position, and settles late (not wrongly) if a Docker release ever + * reorders its walk. + */ +export const LEGACY_PGDATA_BASELINE_MARKER_NAME = "SUPABASE_BASELINE"; + +/** + * The marker's tar entry — what {@link legacyValidatePgDataArchive} looks for, and the reason a + * cached snapshot means "the Supabase platform baseline this key promises" rather than merely "a + * PostgreSQL cluster". + * + * Its whole value is WHEN it is written: {@link legacyStampPgDataBaselineMarker} is called from the + * export step alone, after the caller's own baseline has completed and immediately before the + * copy-out. So an archive produced before the baseline ran — a wiring regression that moves the + * snapshot earlier, or a hand-placed bare PGDATA tar dropped into the cache directory — cannot + * carry it, and is rejected before anything is restored. + */ +export const LEGACY_PGDATA_BASELINE_MARKER_ENTRY = `${LEGACY_PGDATA_DIR_NAME}/${LEGACY_PGDATA_BASELINE_MARKER_NAME}`; + +/** + * The marker's CONTENT: the caller's own identity token for what the snapshot carries — for the + * shadow baseline cache, the cache key the archive is published under, which is also its + * filename's stem (`legacyShadowBaselineTarFileName`, `shadow-cache.ts`). + * + * Presence alone is strictly weaker than a snapshot's own filename claims: a perfectly valid + * archive COPIED or RENAMED over another key's cache file passes a name-only check and warm-restores + * a baseline built from different roles/vault values/service versions. Binding the marker to the + * key — stamped at export, compared at validation ({@link legacyValidatePgDataArchive}) — is what + * makes an archive vouch for the filename it is stored under, not merely for "some baseline". + * + * The trailing newline is the canonical form on BOTH sides: it makes the stamped file a normal + * one-line text file (`cat`-able while debugging a cache directory) and both halves of the contract + * go through this one function, so the two can never drift. + */ +export const legacyPgDataBaselineMarkerContent = (key: string): string => `${key}\n`; + +/** Every entry {@link legacyValidatePgDataArchive} requires of a restorable snapshot. */ +export const LEGACY_PGDATA_REQUIRED_ENTRIES: ReadonlyArray = [ + LEGACY_PGDATA_CLUSTER_ENTRY, + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, +]; + +/** + * Internal-only "the snapshot could not be produced" signal — deliberately NOT a + * `Data.TaggedError`: every caller of {@link legacyExportPgDataTar} decides for itself how to + * degrade (the shadow baseline cache warns and continues uncached), so this must not be mistaken + * for a CLI-facing error. + */ +export interface LegacyPgDataSnapshotUnavailable { + readonly reason: string; +} + +const legacyPgDataSnapshotUnavailable = (reason: string): LegacyPgDataSnapshotUnavailable => ({ + reason, +}); + +/** + * The one-member tar {@link legacyStampPgDataBaselineMarker} pushes into the container: + * `SUPABASE_BASELINE` relative to the `docker cp` destination, which is PGDATA itself, carrying + * {@link legacyPgDataBaselineMarkerContent}'s token for `key`. + * + * Exported for the unit test that round-trips it through this module's own scanner — the stamp and + * the check have to agree on the entry name AND on the content encoding, and nothing else proves + * that they do. + */ +export const legacyPgDataBaselineMarkerTar = ( + key: string, +): Effect.Effect => + Effect.tryPromise({ + try: () => + new Bun.Archive({ + [LEGACY_PGDATA_BASELINE_MARKER_NAME]: legacyPgDataBaselineMarkerContent(key), + }).bytes(), + catch: (cause) => + legacyPgDataSnapshotUnavailable( + `failed to build the baseline marker archive: ${cause instanceof Error ? cause.message : String(cause)}`, + ), + }); + +/** + * Writes {@link LEGACY_PGDATA_BASELINE_MARKER_ENTRY} into the container's PGDATA, so the export + * that follows carries it and {@link legacyValidatePgDataArchive} can tell a snapshot of a + * COMPLETED baseline for `key` apart from any other cluster — including a valid snapshot of a + * DIFFERENT key that was copied over this key's cache file. + * + * Delivered as a stdin tar through `docker cp -`, the same form the restore side uses + * (`legacyExtractPreStartArchiveIntoContainer`, `container-lifecycle.ts`) and for the same reason: + * it needs no daemon-visible host path, so it works against local, remote-context, and confined + * Docker clients alike. `docker cp` into a STOPPED container is fully supported — which is exactly + * the state this module's coherence contract already requires of the export. + * + * The caller owns the ORDERING that gives the marker its meaning: this must be the last mutation + * before {@link legacyExportPgDataTar}, and must run only once whatever the snapshot is supposed to + * capture is genuinely in place. + */ +export const legacyStampPgDataBaselineMarker = ( + spawner: Spawner, + containerId: string, + key: string, +): Effect.Effect => + Effect.gen(function* () { + const tar = yield* legacyPgDataBaselineMarkerTar(key); + const failure = (detail: string) => + legacyPgDataSnapshotUnavailable( + `failed to stamp ${LEGACY_PGDATA_BASELINE_MARKER_ENTRY}: ${detail}`, + ); + yield* Effect.scoped( + Effect.gen(function* () { + const child = yield* spawnContainerCli( + spawner, + ["cp", "-", `${containerId}:${LEGACY_PGDATA_PATH}`], + { stdin: Stream.make(tar), stdout: "ignore", stderr: "pipe" }, + ).pipe(Effect.mapError((cause) => failure(legacyDescribeContainerCliFailure(cause)))); + const [exitCode, stderr] = yield* Effect.all( + [child.exitCode.pipe(Effect.map(Number)), legacyCollectText(child.stderr)], + { concurrency: "unbounded" }, + ).pipe(Effect.mapError((cause) => failure(legacyDescribeContainerCliFailure(cause)))); + if (exitCode !== 0) { + const message = stderr.trim(); + return yield* Effect.fail( + failure(`docker cp exited ${exitCode}${message.length > 0 ? `: ${message}` : ""}`), + ); + } + }), + ); + }); + +/** + * Streams `docker cp :${LEGACY_PGDATA_PATH} -`'s tar straight to a temp file next to + * `tarPath` and `rename`s it into place. The stream never lands in memory: the child's stdout is + * piped into `FileSystem.sink`, so a large snapshot costs one buffer's worth of heap. + * + * The container must already be STOPPED (see this module's own header) — the caller owns the + * stop/start around this call. The `rename` is the LAST step and is what publishes the entry: a + * partially written tar must never be observable under the final name. Any failure removes the + * temp file; nothing is left behind for a later run to find. + */ +export const legacyExportPgDataTar = ( + spawner: Spawner, + containerId: string, + fs: FileSystem.FileSystem, + tarPath: string, +): Effect.Effect => { + const tempPath = `${tarPath}.${process.pid}.partial`; + return Effect.gen(function* () { + // Clear any pre-existing file at the temp path (a crashed same-pid predecessor, or an + // adversarially pre-created one on a shared host) so the exclusive-create below starts from + // a genuinely fresh inode — see the sink's own comment. + yield* fs.remove(tempPath).pipe(Effect.orElseSucceed(() => undefined)); + yield* Effect.scoped( + Effect.gen(function* () { + const child = yield* spawnContainerCli( + spawner, + ["cp", `${containerId}:${LEGACY_PGDATA_PATH}`, "-"], + { stdin: "ignore", stdout: "pipe", stderr: "pipe" }, + ).pipe( + Effect.mapError((cause) => + legacyPgDataSnapshotUnavailable( + `failed to export ${LEGACY_PGDATA_PATH}: ${legacyDescribeContainerCliFailure(cause)}`, + ), + ), + ); + // stdout is consumed concurrently with awaiting the exit code, not after it: an + // unread pipe would block `docker cp` long before it finished writing the archive. + const [exitCode, , stderr] = yield* Effect.all( + [ + child.exitCode.pipe(Effect.map(Number)), + // `0o600`: the archive is a full PGDATA — vault secret values, the JWT secret, and + // role password hashes are all in its pages — so it must not be group/world-readable + // on a shared host. `rename` preserves the mode, so the published tar inherits it. + // `wx` (O_EXCL), not `w`: a plain truncating open would inherit an attacker- + // PRE-CREATED file's permissive mode instead of applying `mode` (which only governs + // creation). With the best-effort remove above, `wx` only ever fails if someone + // recreated the path in the race window — and that failure degrades to an uncached + // run, never to a world-readable tar (review: depthfirst on #6184). + Stream.run(child.stdout, fs.sink(tempPath, { flag: "wx", mode: 0o600 })), + legacyCollectText(child.stderr), + ], + { concurrency: "unbounded" }, + ).pipe( + Effect.mapError((cause) => + legacyPgDataSnapshotUnavailable( + `failed to export ${LEGACY_PGDATA_PATH}: ${legacyDescribeContainerCliFailure(cause)}`, + ), + ), + ); + if (exitCode !== 0) { + const message = stderr.trim(); + return yield* Effect.fail( + legacyPgDataSnapshotUnavailable( + `docker cp exited ${exitCode}${message.length > 0 ? `: ${message}` : ""}`, + ), + ); + } + }), + ); + yield* fs + .rename(tempPath, tarPath) + .pipe( + Effect.mapError((cause) => + legacyPgDataSnapshotUnavailable(`failed to publish ${tarPath}: ${cause.message}`), + ), + ); + }).pipe(Effect.onError(() => fs.remove(tempPath).pipe(Effect.orElseSucceed(() => undefined)))); +}; + +// --------------------------------------------------------------------------- +// Archive validation +// --------------------------------------------------------------------------- + +/** POSIX tar's fixed block size: headers, file content, and the end marker are all multiples of it. */ +const LEGACY_TAR_BLOCK_SIZE = 512; + +const LEGACY_TAR_NO_BYTES = new Uint8Array(0); + +const legacyTarDecoder = new TextDecoder(); + +/** + * The most content {@link legacyScanTarChunkForEntries} will ever buffer for `captureEntry`. The + * only entry any caller captures is the baseline marker, whose content is one short identity token, + * so a larger member cannot be a marker this module wrote: it is left UNCAPTURED (`captured` stays + * `undefined`, which every reader treats as "does not match"), rather than being read into memory + * on the word of an untrusted archive's own size field. Keeps the scan's O(1) memory property + * intact whatever a hand-placed tar in the cache directory claims. + */ +const LEGACY_TAR_CAPTURE_MAX_BYTES = 1024; + +/** + * {@link legacyScanTarChunkForEntries}'s carry-over state — everything needed to resume a header + * walk at an arbitrary chunk boundary, and nothing else. `carry` holds the bytes of a header block + * a chunk ended in the middle of (always `< 512`); `skip` counts the file-content bytes still to be + * STEPPED OVER without buffering, which is what keeps a ~90MB archive off the heap. + */ +export interface LegacyTarScanState { + readonly carry: Uint8Array; + readonly skip: number; + /** Consecutive all-zero blocks seen; two in a row is tar's end-of-archive marker. */ + readonly zeroBlocks: number; + /** + * The required entries not seen yet. Empty means every one of them was found; whatever is left + * once the scan settles is what the archive is missing, which is what the caller reports. + */ + readonly missing: ReadonlySet; + readonly ended: boolean; + /** A block that is neither zero nor a checksum-valid header: not a tar (or a truncated one). */ + readonly malformed: boolean; + /** + * The one entry whose CONTENT the walk reads rather than steps over — the baseline marker, whose + * bytes say which key the archive belongs to. `undefined` for a presence-only walk. + */ + readonly captureEntry: string | undefined; + /** + * {@link captureEntry}'s content bytes. `undefined` until its header is seen, and STILL undefined + * afterwards when the entry was too large to capture ({@link LEGACY_TAR_CAPTURE_MAX_BYTES}) — + * both mean "no content to compare against", which is the safe verdict either way. + */ + readonly captured: Uint8Array | undefined; + /** Content bytes of {@link captureEntry} still to be captured; `0` once complete or not capturing. */ + readonly capturePending: number; +} + +/** + * A fresh walk looking for `required` — every entry of which must appear for the scan to pass — + * capturing `captureEntry`'s content along the way when given (it must be one of `required`, so + * that "found everything" also means "the captured entry's header was seen"). + */ +export const legacyInitialTarScanState = ( + required: Iterable, + captureEntry?: string, +): LegacyTarScanState => ({ + carry: LEGACY_TAR_NO_BYTES, + skip: 0, + zeroBlocks: 0, + missing: new Set(required), + ended: false, + malformed: false, + captureEntry, + captured: undefined, + capturePending: 0, +}); + +/** Whether every required entry has been seen AND its captured content (if any) is complete. */ +export const legacyTarScanFound = (state: LegacyTarScanState): boolean => + state.missing.size === 0 && state.capturePending === 0; + +/** Whether the scan has reached a verdict — nothing later in the archive can change it. */ +export const legacyTarScanSettled = (state: LegacyTarScanState): boolean => + legacyTarScanFound(state) || state.ended || state.malformed; + +/** {@link LegacyTarScanState.captured} decoded, or `undefined` when there is nothing to compare. */ +export const legacyTarScanCapturedText = (state: LegacyTarScanState): string | undefined => + state.captured === undefined ? undefined : legacyTarDecoder.decode(state.captured); + +/** A NUL-terminated text field of a tar header block. */ +const legacyTarTextField = (block: Uint8Array, offset: number, length: number): string => { + const raw = block.subarray(offset, offset + length); + const end = raw.indexOf(0); + return legacyTarDecoder.decode(end === -1 ? raw : raw.subarray(0, end)); +}; + +/** + * A numeric header field: NUL/space-padded octal, or GNU's base-256 form (high bit of the first + * byte) for sizes past what 11 octal digits can hold. `undefined` when neither parses. + */ +const legacyTarNumericField = ( + block: Uint8Array, + offset: number, + length: number, +): number | undefined => { + const first = block[offset] ?? 0; + if ((first & 0x80) !== 0) { + let value = first & 0x7f; + for (let index = offset + 1; index < offset + length; index += 1) { + value = value * 256 + (block[index] ?? 0); + } + return value; + } + const text = legacyTarTextField(block, offset, length).trim(); + if (text.length === 0) return 0; + if (!/^[0-7]+$/u.test(text)) return undefined; + return Number.parseInt(text, 8); +}; + +/** + * Tar's own integrity check on a header block: the stored checksum is the sum of all 512 bytes + * with the checksum field itself read as spaces. Both the unsigned and the (historical) signed + * summation are accepted, as every tar reader does. This is what tells a genuine header apart from + * arbitrary bytes, so a non-tar file cannot be walked as if it were one. + */ +const legacyTarChecksumValid = (block: Uint8Array): boolean => { + const stored = legacyTarNumericField(block, 148, 8); + if (stored === undefined) return false; + let unsigned = 0; + let signed = 0; + for (let index = 0; index < LEGACY_TAR_BLOCK_SIZE; index += 1) { + const byte = index >= 148 && index < 156 ? 0x20 : (block[index] ?? 0); + unsigned += byte; + signed += byte > 127 ? byte - 256 : byte; + } + return stored === unsigned || stored === signed; +}; + +/** The header's full member path: ustar's `prefix` field rejoined, with a leading `./` dropped. */ +const legacyTarEntryName = (block: Uint8Array): string => { + const name = legacyTarTextField(block, 0, 100); + const prefix = legacyTarTextField(block, 345, 155); + const joined = prefix.length > 0 ? `${prefix}/${name}` : name; + return joined.startsWith("./") ? joined.slice(2) : joined; +}; + +const legacyTarBlockIsZero = (block: Uint8Array): boolean => block.every((byte) => byte === 0); + +/** + * Folds one stream chunk into a tar HEADER walk looking for every entry still in `state.missing`, + * capturing `state.captureEntry`'s content if it has one. Pure and chunk-boundary-agnostic: every + * other member's content is stepped over by byte count rather than buffered, so the whole scan + * costs one partial header block plus (at most) + * {@link LEGACY_TAR_CAPTURE_MAX_BYTES} of memory no matter how large the archive is. Stops (and + * stays stopped) at the first of: the last required entry found (with its capture complete), the + * end-of-archive marker, or a block that is not a valid header. + */ +export const legacyScanTarChunkForEntries = ( + state: LegacyTarScanState, + chunk: Uint8Array, +): LegacyTarScanState => { + if (legacyTarScanSettled(state)) return state; + let carry = state.carry; + let skip = state.skip; + let zeroBlocks = state.zeroBlocks; + let missing = state.missing; + let captured = state.captured; + let capturePending = state.capturePending; + const { captureEntry } = state; + /** + * Appends the leading `capturePending` bytes of a content run being consumed. Content bytes are + * always consumed front-to-back, and `capturePending` never exceeds what is left of the capture + * entry's own content, so this is correct whether the run is the tail carried over from the + * previous chunk or a fresh member's content in this one. + */ + const takeCapture = (bytes: Uint8Array): void => { + if (capturePending <= 0 || captured === undefined) return; + const take = Math.min(capturePending, bytes.length); + const merged = new Uint8Array(captured.length + take); + merged.set(captured); + merged.set(bytes.subarray(0, take), captured.length); + captured = merged; + capturePending -= take; + }; + // A verdict keeps `missing`/`captured` (they are the report) and drops the walk's resumption state. + const settle = ( + verdict: Pick, + ): LegacyTarScanState => ({ + carry: LEGACY_TAR_NO_BYTES, + skip: 0, + zeroBlocks: 0, + missing, + captureEntry, + captured, + capturePending, + ...verdict, + }); + // Content bytes carried over from the previous chunk come first — they are not headers. + let offset = Math.min(skip, chunk.length); + takeCapture(chunk.subarray(0, offset)); + skip -= offset; + // The capture may have been the only thing outstanding when the previous chunk ran out. + if (missing.size === 0 && capturePending === 0) return settle({ ended: false, malformed: false }); + while (offset < chunk.length) { + const available = chunk.length - offset; + let block: Uint8Array; + if (carry.length > 0) { + const take = Math.min(LEGACY_TAR_BLOCK_SIZE - carry.length, available); + const merged = new Uint8Array(carry.length + take); + merged.set(carry); + merged.set(chunk.subarray(offset, offset + take), carry.length); + offset += take; + if (merged.length < LEGACY_TAR_BLOCK_SIZE) { + carry = merged; + break; + } + carry = LEGACY_TAR_NO_BYTES; + block = merged; + } else if (available < LEGACY_TAR_BLOCK_SIZE) { + carry = chunk.slice(offset); + break; + } else { + block = chunk.subarray(offset, offset + LEGACY_TAR_BLOCK_SIZE); + offset += LEGACY_TAR_BLOCK_SIZE; + } + + if (legacyTarBlockIsZero(block)) { + zeroBlocks += 1; + if (zeroBlocks >= 2) return settle({ ended: true, malformed: false }); + continue; + } + zeroBlocks = 0; + if (!legacyTarChecksumValid(block)) { + return settle({ ended: false, malformed: true }); + } + const name = legacyTarEntryName(block); + const wasMissing = missing.has(name); + if (wasMissing) { + const remaining = new Set(missing); + remaining.delete(name); + missing = remaining; + } + const size = legacyTarNumericField(block, 124, 12); + if (size === undefined || size < 0) { + return settle({ ended: false, malformed: true }); + } + // Arm the capture on the capture entry's FIRST occurrence only (`wasMissing`), so a duplicate + // member later in the archive cannot overwrite what the real one said. An oversized entry is + // left uncaptured — see {@link LEGACY_TAR_CAPTURE_MAX_BYTES}. + if (wasMissing && name === captureEntry && size <= LEGACY_TAR_CAPTURE_MAX_BYTES) { + captured = LEGACY_TAR_NO_BYTES; + capturePending = size; + } + // Content is padded up to the next block boundary; directories and links carry size 0. + const content = Math.ceil(size / LEGACY_TAR_BLOCK_SIZE) * LEGACY_TAR_BLOCK_SIZE; + const stepped = Math.min(content, chunk.length - offset); + takeCapture(chunk.subarray(offset, offset + stepped)); + offset += stepped; + skip = content - stepped; + // Only now, with the capture (if any) armed and fed: settling at the header would have + // discarded the very bytes the marker check needs. + if (missing.size === 0 && capturePending === 0) { + return settle({ ended: false, malformed: false }); + } + } + return { + carry, + skip, + zeroBlocks, + missing, + ended: false, + malformed: false, + captureEntry, + captured, + capturePending, + }; +}; + +/** + * Why an archive at `tarPath` must not be restored under `key` — the two distinguishable ways + * {@link legacyValidatePgDataArchive} can reject one. Both implicate the FILE, never the + * infrastructure, so both are safe for a caller to act on by discarding it. + */ +export type LegacyPgDataArchiveProblem = + /** The header stream never carried one of {@link LEGACY_PGDATA_REQUIRED_ENTRIES}. */ + | { readonly _tag: "missing-entries"; readonly entries: ReadonlyArray } + /** + * Every entry is there, but the marker vouches for a DIFFERENT key than the one this archive is + * stored under. `found` is the marker's own token (trimmed), or `undefined` when it carried none + * that could be read (an oversized or truncated marker member). + */ + | { readonly _tag: "wrong-key"; readonly expected: string; readonly found: string | undefined }; + +/** + * Whether `tarPath` is a snapshot that may be restored as `key`'s baseline — `Option.none()` when + * it is, the reason it is not otherwise. + * + * Three failures, each invisible to the restore itself. Without `PG_VERSION`, an archive that is + * syntactically fine but carries no cluster (an EMPTY tar qualifies) restores SILENTLY: + * `docker cp -` extracts nothing, the Postgres entrypoint finds an empty PGDATA and runs a fresh + * `initdb`, readiness passes, and the caller is handed a bare cluster it believes carries the + * platform baseline. Without the baseline marker, that same silent-success shape survives one level + * up: a REAL but bare PGDATA tar (dropped into the cache directory by hand, or produced by a future + * regression that exports before the baseline runs) restores, starts, and answers — and only the + * resulting diff would ever show it. And without the marker's CONTENT, it survives one level up + * again: a genuine, fully baselined snapshot of ANOTHER key, copied or renamed over this key's + * cache file, passes every name-only check while carrying different roles, vault values, and + * service-version schema — see {@link legacyPgDataBaselineMarkerContent}. Validating the header + * stream up front is the only place any of the three is observable, so callers must check BEFORE + * restoring. + * + * Reads the file locally — no Docker, no extraction — and stops as soon as both entries have been + * seen and the marker's few bytes read; see {@link LEGACY_PGDATA_BASELINE_MARKER_NAME} for why that + * is normally within the archive's first blocks. Even a full walk only parses HEADERS (every other + * member's content is stepped over by byte count), so the cost is one sequential read with O(1) + * memory. Only a genuine read failure fails; a valid tar that does not qualify simply reports why. + */ +export const legacyValidatePgDataArchive = ( + fs: FileSystem.FileSystem, + tarPath: string, + key: string, +): Effect.Effect, LegacyPgDataSnapshotUnavailable> => + fs.stream(tarPath).pipe( + Stream.mapAccum( + () => + legacyInitialTarScanState( + LEGACY_PGDATA_REQUIRED_ENTRIES, + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + ), + (state: LegacyTarScanState, chunk: Uint8Array) => { + const next = legacyScanTarChunkForEntries(state, chunk); + return [next, [next]] as const; + }, + ), + Stream.takeUntil(legacyTarScanSettled), + Stream.runLast, + // An empty file yields no chunks at all, so `None` means nothing was found: everything missing. + Effect.map((last) => + Option.isSome(last) + ? legacyPgDataArchiveProblem(last.value, key) + : Option.some({ + _tag: "missing-entries", + entries: LEGACY_PGDATA_REQUIRED_ENTRIES, + }), + ), + Effect.mapError((cause) => + legacyPgDataSnapshotUnavailable(`failed to read ${tarPath}: ${cause.message}`), + ), + ); + +/** {@link legacyValidatePgDataArchive}'s verdict, split out so it is pure and directly testable. */ +const legacyPgDataArchiveProblem = ( + state: LegacyTarScanState, + key: string, +): Option.Option => { + const entries = LEGACY_PGDATA_REQUIRED_ENTRIES.filter((entry) => state.missing.has(entry)); + if (entries.length > 0) return Option.some({ _tag: "missing-entries", entries }); + const stamped = legacyTarScanCapturedText(state); + if (stamped === legacyPgDataBaselineMarkerContent(key)) return Option.none(); + // Trimmed for the report only — the comparison above is on the exact canonical form, so a marker + // padded with whitespace is a mismatch rather than something to normalize into a match. + return Option.some({ _tag: "wrong-key", expected: key, found: stamped?.trim() }); +}; + +/** + * Builds the {@link LegacyStartContainerSpec.preStartArchives} entry that restores a + * {@link legacyExportPgDataTar} tar into a container between `docker create` and `docker start`. + * `containerPath` is PGDATA's PARENT, not PGDATA itself — see {@link LEGACY_PGDATA_PARENT_PATH}'s + * own doc comment for why. + */ +export const legacyPgDataRestoreArchive = ( + fs: FileSystem.FileSystem, + tarPath: string, +): NonNullable[number] => ({ + containerPath: LEGACY_PGDATA_PARENT_PATH, + tar: fs.stream(tarPath), +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts new file mode 100644 index 0000000000..bb13014a08 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts @@ -0,0 +1,245 @@ +/** + * The pure tar-header walk behind {@link legacyValidatePgDataArchive}'s pre-restore check. + * Unit tests rather than integration ones because the interesting cases are all in the format + * handling: chunk boundaries landing mid-header (or mid-MARKER-CONTENT), content that must be + * stepped over rather than parsed, and bytes that are not a tar at all. + */ + +import { describe, expect, it } from "@effect/vitest"; +import { Effect } from "effect"; + +import { + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + LEGACY_PGDATA_BASELINE_MARKER_NAME, + LEGACY_PGDATA_CLUSTER_ENTRY, + LEGACY_PGDATA_REQUIRED_ENTRIES, + legacyInitialTarScanState, + legacyPgDataBaselineMarkerContent, + legacyPgDataBaselineMarkerTar, + legacyScanTarChunkForEntries, + legacyTarScanCapturedText, + legacyTarScanFound, + legacyTarScanSettled, + type LegacyTarScanState, +} from "./pgdata-snapshot.ts"; + +const BLOCK = 512; + +const encoder = new TextEncoder(); + +const tarBlock = (fields: ReadonlyArray): Uint8Array => { + const block = new Uint8Array(BLOCK); + for (const [offset, text] of fields) block.set(encoder.encode(text), offset); + return block; +}; + +const octal = (value: number, width: number) => `${value.toString(8).padStart(width, "0")}\0`; + +/** One ustar member: a checksummed header block plus its NUL-padded content blocks. */ +const tarEntry = (name: string, content: string, typeFlag = "0"): Uint8Array => { + const header = tarBlock([ + [0, name], + [100, octal(0o600, 7)], + [108, octal(0, 7)], + [116, octal(0, 7)], + [124, octal(content.length, 11)], + [136, octal(0, 11)], + [148, " "], + [156, typeFlag], + [257, "ustar\0"], + [263, "00"], + ]); + let checksum = 0; + for (const byte of header) checksum += byte; + header.set(encoder.encode(`${octal(checksum, 6)} `), 148); + const padded = new Uint8Array(Math.ceil(content.length / BLOCK) * BLOCK); + padded.set(encoder.encode(content)); + const out = new Uint8Array(header.length + padded.length); + out.set(header); + out.set(padded, header.length); + return out; +}; + +const TAR_END = new Uint8Array(2 * BLOCK); + +const concat = (...parts: ReadonlyArray): Uint8Array => { + const total = parts.reduce((sum, part) => sum + part.length, 0); + const out = new Uint8Array(total); + let offset = 0; + for (const part of parts) { + out.set(part, offset); + offset += part.length; + } + return out; +}; + +/** Feeds `tar` through the scanner in fixed-size chunks, stopping as the real stream would. */ +const scan = ( + tar: Uint8Array, + chunkSize: number, + required: ReadonlyArray = LEGACY_PGDATA_REQUIRED_ENTRIES, + captureEntry: string | undefined = LEGACY_PGDATA_BASELINE_MARKER_ENTRY, +) => { + let state: LegacyTarScanState = legacyInitialTarScanState( + required, + required.includes(captureEntry ?? "") ? captureEntry : undefined, + ); + for (let offset = 0; offset < tar.length; offset += chunkSize) { + state = legacyScanTarChunkForEntries(state, tar.subarray(offset, offset + chunkSize)); + if (legacyTarScanSettled(state)) break; + } + return { + ...state, + found: legacyTarScanFound(state), + capturedText: legacyTarScanCapturedText(state), + }; +}; + +const KEY = "0011223344556677"; + +const MARKER_ENTRY = tarEntry( + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + legacyPgDataBaselineMarkerContent(KEY), +); + +describe("legacyScanTarChunkForEntries", () => { + const pgdataTar = concat( + tarEntry("data/", "", "5"), + tarEntry("data/postgresql.conf", "listen_addresses = '*'\n"), + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + MARKER_ENTRY, + TAR_END, + ); + + it("finds every required entry regardless of where the chunk boundaries fall", () => { + // 7 and 513 both split header blocks; the required entries are the last two members, behind a + // directory and a file whose content must be stepped over rather than mistaken for a header. + // The marker's own CONTENT is read rather than stepped over, so those same boundaries also + // land mid-token — the capture has to survive being fed one byte at a time. + for (const chunkSize of [1, 7, 100, 512, 513, 4096, pgdataTar.length]) { + const state = scan(pgdataTar, chunkSize); + expect(state.found, `chunk size ${chunkSize}`).toBe(true); + expect(state.malformed).toBe(false); + expect(state.capturedText, `chunk size ${chunkSize}`).toBe( + legacyPgDataBaselineMarkerContent(KEY), + ); + } + }); + + it("captures the marker's own key, not a same-named entry's, and not an oversized one", () => { + // The whole point of the content: two archives with identical member LISTS, telling apart the + // key each one actually vouches for. + const other = concat( + tarEntry("data/", "", "5"), + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + tarEntry( + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + legacyPgDataBaselineMarkerContent("ffffffff00000000"), + ), + TAR_END, + ); + expect(scan(other, 512).found).toBe(true); + expect(scan(other, 512).capturedText).toBe("ffffffff00000000\n"); + + // A marker member larger than the cap is left UNCAPTURED rather than buffered on the say-so of + // an untrusted archive's own size field — "no content" is the safe verdict, never a match. + const oversized = concat( + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + tarEntry(LEGACY_PGDATA_BASELINE_MARKER_ENTRY, "x".repeat(2048)), + TAR_END, + ); + const state = scan(oversized, 512); + expect(state.found).toBe(true); + expect(state.capturedText).toBeUndefined(); + }); + + it("stops at the last required entry without walking the rest of the archive", () => { + // Everything after them is garbage: reaching it would settle `malformed` instead. + const trailing = concat( + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + MARKER_ENTRY, + encoder.encode("x".repeat(2048)), + ); + expect(scan(trailing, 512).found).toBe(true); + }); + + it("keeps looking when only some of the required entries have been seen", () => { + // A REAL cluster with no baseline marker — the shape a hand-placed PGDATA tar, or a snapshot + // exported before the platform baseline ran, would have. It must not pass. + const bare = concat( + tarEntry("data/", "", "5"), + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + TAR_END, + ); + const state = scan(bare, 512); + expect(state.found).toBe(false); + expect(state.ended).toBe(true); + expect([...state.missing]).toEqual([LEGACY_PGDATA_BASELINE_MARKER_ENTRY]); + // ...and the same bytes DO pass once the marker is there, so nothing else about them is wrong. + const stamped = concat(bare.subarray(0, bare.length - TAR_END.length), MARKER_ENTRY, TAR_END); + expect(scan(stamped, 512).found).toBe(true); + }); + + it("reports a valid but cluster-less archive as not found", () => { + expect(scan(TAR_END, 512)).toMatchObject({ found: false, ended: true, malformed: false }); + expect([...scan(TAR_END, 512).missing]).toEqual([...LEGACY_PGDATA_REQUIRED_ENTRIES]); + const otherEntries = concat(tarEntry("data/base/1/2345", "rows"), TAR_END); + expect(scan(otherEntries, 512)).toMatchObject({ found: false, ended: true }); + }); + + it("never mistakes file content for a header", () => { + // A member whose CONTENT is itself a valid `data/PG_VERSION` header — only the real member + // list counts, so this archive must come back cluster-less. + const decoy = new TextDecoder().decode(tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n")); + expect(scan(concat(tarEntry("data/decoy", decoy), TAR_END), 512).found).toBe(false); + }); + + it("settles as malformed on bytes that are not a tar", () => { + expect(scan(encoder.encode("x".repeat(4096)), 512).malformed).toBe(true); + // A header whose checksum does not add up (a flipped byte in an otherwise real archive). + const corrupted = concat(tarEntry("data/PG_VERSION", "17\n"), MARKER_ENTRY, TAR_END); + corrupted[5] = 0x41; + expect(scan(corrupted, 512).malformed).toBe(true); + }); + + it("treats a truncated trailing block as the end of what it can read", () => { + // No end marker and a half header: nothing found, but nothing claimed either. + const truncated = concat(tarEntry("data/base/1/2345", "rows")).subarray(0, BLOCK + 100); + expect(scan(truncated, 512)).toMatchObject({ found: false, ended: false, malformed: false }); + }); +}); + +describe("legacyPgDataBaselineMarkerTar", () => { + it("stamps the very entry the pre-restore scan requires, carrying its key", () => + Effect.runPromise( + Effect.gen(function* () { + // The stamp and the check are two halves of one contract, and only a round trip proves + // they agree — on the entry name AND on the content encoding: `docker cp - :` + // unpacks this archive's members RELATIVE to PGDATA, so its bare `SUPABASE_BASELINE` + // member is what a later export tars up as `data/SUPABASE_BASELINE`. + const bytes = yield* legacyPgDataBaselineMarkerTar(KEY); + const stamped = scan( + bytes, + 512, + [LEGACY_PGDATA_BASELINE_MARKER_NAME], + LEGACY_PGDATA_BASELINE_MARKER_NAME, + ); + expect(stamped.found).toBe(true); + // What the warm path compares against: the key this snapshot is published under, in the + // one canonical form both halves go through. + expect(stamped.capturedText).toBe(legacyPgDataBaselineMarkerContent(KEY)); + // ...so a snapshot stamped with ANOTHER key does not read as this one's. + const otherKey = yield* legacyPgDataBaselineMarkerTar("ffffffff00000000"); + expect( + scan( + otherKey, + 512, + [LEGACY_PGDATA_BASELINE_MARKER_NAME], + LEGACY_PGDATA_BASELINE_MARKER_NAME, + ).capturedText, + ).not.toBe(legacyPgDataBaselineMarkerContent(KEY)); + // Nothing else rides along — one member, so the stamp costs a couple of blocks. + expect(scan(bytes, 512, [LEGACY_PGDATA_BASELINE_MARKER_ENTRY]).found).toBe(false); + }), + )); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts index f3ec4879a4..461744bf1f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts @@ -258,6 +258,14 @@ function legacyPostgresExtraEnv( * {@link legacyBuildPostgresStartContainerSpec}), so it never appears in this * process's own `docker create` argv (CWE-214/522). * + * The final command is `exec`'d — a deliberate divergence from Go's script + * (which leaves `sh` as PID 1, so SIGTERM is never forwarded and every + * `docker stop` burns the full 10s grace period before SIGKILL; with `exec`, + * Postgres is PID 1 and stops in ~1s). Applies to all three entrypoint + * variants below. Timing is not part of the Go-parity surface (ADR 0016); + * see `shadow-cache.ts`'s own doc comment for why fast shutdown matters to the shadow baseline + * cache's cold path. + * * Otherwise byte-for-byte derived from Go's raw-string concatenation — * `NewContainerConfig(args ...string)` splices `strings.Join(args, " ")` * straight after the literal trailing space following `/etc/postgresql` @@ -276,7 +284,7 @@ function legacyPostgresEntrypointScriptPg15(postgresConfig: string, args = ""): "\n" + "cat <<'EOF' > /etc/postgresql.schema.sql && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - `docker-entrypoint.sh postgres -D /etc/postgresql ${args}\n` + + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${args}\n` + `${LEGACY_START_DB_SCHEMA_SQL}\n` + `${LEGACY_START_DB_WEBHOOK_SQL}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + @@ -301,7 +309,7 @@ function legacyPostgresEntrypointScriptPg14(postgresConfig: string, args = ""): "\n" + "cat <<'EOF' > /docker-entrypoint-initdb.d/supabase_schema.sql && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - `docker-entrypoint.sh postgres -D /etc/postgresql ${args}\n` + + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${args}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + "EOF\n" + `${postgresConfig}\n` + @@ -328,7 +336,7 @@ function legacyPostgresEntrypointScriptRestore(postgresConfig: string): string { "cat <<'EOF' > /etc/postgresql.schema.sql && \\\n" + "cat <<'EOF' > /docker-entrypoint-initdb.d/migrate.sh && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - "docker-entrypoint.sh postgres -D /etc/postgresql\n" + + "exec docker-entrypoint.sh postgres -D /etc/postgresql\n" + `${LEGACY_START_DB_SCHEMA_SQL}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + "EOF\n" + diff --git a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts index 122418f02f..16969daeb7 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts @@ -79,7 +79,7 @@ describe("legacyBuildPostgresStartContainerSpec", () => { "\n" + "cat <<'EOF' > /etc/postgresql.schema.sql && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - "docker-entrypoint.sh postgres -D /etc/postgresql \n" + + "exec docker-entrypoint.sh postgres -D /etc/postgresql \n" + `${LEGACY_START_DB_SCHEMA_SQL}\n` + `${LEGACY_START_DB_WEBHOOK_SQL}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + @@ -108,7 +108,7 @@ describe("legacyBuildPostgresStartContainerSpec", () => { "\n" + "cat <<'EOF' > /docker-entrypoint-initdb.d/supabase_schema.sql && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - "docker-entrypoint.sh postgres -D /etc/postgresql \n" + + "exec docker-entrypoint.sh postgres -D /etc/postgresql \n" + `${LEGACY_START_DB_SUPABASE_SQL}\n` + "EOF\n" + `${POSTGRES_CONFIG_HEADER}\n` + @@ -252,7 +252,7 @@ describe("legacyBuildPostgresStartContainerSpec", () => { "cat <<'EOF' > /etc/postgresql.schema.sql && \\\n" + "cat <<'EOF' > /docker-entrypoint-initdb.d/migrate.sh && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - "docker-entrypoint.sh postgres -D /etc/postgresql\n" + + "exec docker-entrypoint.sh postgres -D /etc/postgresql\n" + `${LEGACY_START_DB_SCHEMA_SQL}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + "EOF\n" + @@ -405,7 +405,7 @@ describe("legacyBuildShadowPostgresContainerSpec", () => { ); const script = spec.cmd?.[1]; expect(script).toContain( - `docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}\n`, + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}\n`, ); expect(spec.secretFiles).toEqual([ { @@ -422,7 +422,7 @@ describe("legacyBuildShadowPostgresContainerSpec", () => { ); const script = spec.cmd?.[1]; expect(script).toContain( - `docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}\n`, + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}\n`, ); expect(spec.secretFiles).toBeUndefined(); expect(spec.tmpfs).toEqual({ "/docker-entrypoint-initdb.d": "" }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/realtime-env.ts b/apps/cli/src/legacy/shared/db-bootstrap/realtime-env.ts index 8210ec20b4..9d31d45676 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/realtime-env.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/realtime-env.ts @@ -24,7 +24,7 @@ import { * `authenticator`, Storage's `supabase_storage_admin`), so it is not hoisted * alongside `legacyStartInternalDbUrl`. */ -const LEGACY_REALTIME_DB_USER = "supabase_admin"; +export const LEGACY_REALTIME_DB_USER = "supabase_admin"; /** * Go's `realtime.TenantId` default (`pkg/config/config.go:481`) — `toml:"-"` @@ -37,7 +37,7 @@ const LEGACY_REALTIME_DB_USER = "supabase_admin"; export const LEGACY_REALTIME_TENANT_ID = "realtime-dev"; /** Go's `realtime.EncryptionKey` default (`pkg/config/config.go:482`) — `toml:"-"`, never configurable. */ -const LEGACY_REALTIME_ENCRYPTION_KEY = "supabaserealtime"; +export const LEGACY_REALTIME_ENCRYPTION_KEY = "supabaserealtime"; /** Go's `realtime.SecretKeyBase` default (`pkg/config/config.go:483`) — `toml:"-"`, never configurable. */ const LEGACY_REALTIME_SECRET_KEY_BASE = diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts new file mode 100644 index 0000000000..a0f8d52e8c --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -0,0 +1,1032 @@ +/** + * The shadow baseline cache's acquire/export/restore flow, driven end to end against a tiny + * in-test Docker model (create/start/stop/rm all mutate the same container table, and `docker cp` + * really moves bytes in and out of it) plus the REAL filesystem under a per-test temp workdir, so + * the tar artifact, its atomic publish, and its retention rule are exercised for real. + * + * Scenario-oriented on purpose: every test is a sequence of real acquires and releases, and the + * assertions are on the resulting Docker state, the tar on disk, and what the caller was told + * about the baseline — not on internal call ordering, except where the ordering IS the contract + * (the export must stop the container before copying and start it again afterwards). + */ + +import { join } from "node:path"; + +import type { ProjectConfig } from "@supabase/config"; +import { ProjectConfigSchema } from "@supabase/config"; +import { BunServices } from "@effect/platform-bun"; +import { describe, expect, it } from "@effect/vitest"; +import { Effect, Exit, FileSystem, Layer, Option, Path, Schema } from "effect"; + +import { + LEGACY_FAKE_EMPTY_TAR, + LEGACY_FAKE_UNSTAMPED_PGDATA_TAR, + legacyFakePgDataTar, + legacyWithEnv, + mockLegacyDockerDaemonCliSpawner, + useLegacyTempWorkdir, +} from "../../../../tests/helpers/legacy-mocks.ts"; +import { mockOutput } from "../../../../tests/helpers/mocks.ts"; +import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; +import { LegacyDbConnectError } from "../legacy-db-connection.errors.ts"; +import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; +import { + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + LEGACY_PGDATA_BASELINE_MARKER_NAME, + LEGACY_PGDATA_PARENT_PATH, + LEGACY_PGDATA_PATH, + legacyPgDataBaselineMarkerContent, +} from "./pgdata-snapshot.ts"; +import { + LEGACY_SHADOW_BASELINE_KEEP, + LEGACY_SHADOW_CACHE_ENV, + legacyAcquireShadowDatabase, + type LegacyShadowCacheOpts, +} from "./shadow-cache.ts"; +import { LEGACY_SHADOW_DEBUG_ENV } from "./shadow-debug.ts"; +import { legacyRemoveShadowDatabase } from "./shadow-database.ts"; +import type { LegacyShadowDbSetupInput, LegacyShadowSetupInput } from "./shadow-database.ts"; + +const decodeConfig = Schema.decodeUnknownSync(ProjectConfigSchema); +const defaultConfig: ProjectConfig = decodeConfig({}); + +const tempRoot = useLegacyTempWorkdir("legacy-shadow-cache-"); + +const withShadowCacheEnv = (value: string | undefined, body: Effect.Effect) => + legacyWithEnv(LEGACY_SHADOW_CACHE_ENV, value, body); + +/** + * Isolates the global shadow-baseline cache under a per-test `SUPABASE_HOME` so tests never + * write into the developer's real `~/.supabase`. Nested with the opt-in/opt-out gate. + */ +const withShadowCacheHome = ( + value: string | undefined, + body: Effect.Effect, +): Effect.Effect => + legacyWithEnv( + "SUPABASE_HOME", + join(tempRoot.current, "_supabase_home"), + withShadowCacheEnv(value, body), + ); + +const withShadowDebugEnv = (value: string | undefined, body: Effect.Effect) => + legacyWithEnv(LEGACY_SHADOW_DEBUG_ENV, value, body); + +/** + * Captures every write `body` makes directly to the real `process.stderr` — the channel + * `legacyWaitForShadowReady`'s own `ready-attempt`/`ready-wait` debug lines use, since that + * function has no `Output` in its context (see `health-check.ts`'s own doc comment). Mirrors + * `health-check.unit.test.ts`'s own capture/restore pattern. + */ +const captureStderr = ( + body: Effect.Effect, +): Effect.Effect<{ readonly result: A; readonly writes: ReadonlyArray }, E, R> => + Effect.gen(function* () { + const writes: Array = []; + const originalWrite = globalThis.process.stderr.write.bind(globalThis.process.stderr); + globalThis.process.stderr.write = ((chunk: string | Uint8Array) => { + writes.push(typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk)); + return true; + }) as typeof globalThis.process.stderr.write; + const result = yield* body.pipe( + Effect.ensuring( + Effect.sync(() => { + globalThis.process.stderr.write = originalWrite; + }), + ), + ); + return { result, writes }; + }); + +// --------------------------------------------------------------------------- +// A fake Postgres the readiness probe can connect to +// --------------------------------------------------------------------------- + +function fakeCluster(opts: { readonly failConnect?: boolean } = {}) { + const connected: Array = []; + const layer = Layer.succeed(LegacyDbConnection, { + connect: (cfg) => + Effect.suspend(() => { + connected.push(cfg.database); + return opts.failConnect === true + ? Effect.fail(new LegacyDbConnectError({ message: "connection refused" })) + : Effect.succeed({ + exec: () => Effect.void, + query: () => Effect.succeed([]), + execBatch: () => Effect.void, + extensionExists: () => Effect.succeed(false), + copyToCsv: () => Effect.succeed(new Uint8Array()), + queryRaw: () => Effect.succeed({ fields: [], rows: [], commandTag: "" }), + }); + }), + }); + return { layer, connected }; +} + +// --------------------------------------------------------------------------- +// Inputs +// --------------------------------------------------------------------------- + +const shadowSetup = (): LegacyShadowDbSetupInput => ({ + majorVersion: 17, + config: defaultConfig, + dbUrl: "postgresql://postgres:postgres@127.0.0.1:54320/postgres", + jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", + jwks: Effect.succeed('{"keys":[]}'), + apiUrl: "http://127.0.0.1:54321", + authExternalUrl: undefined, + siteUrl: defaultConfig.auth.site_url, + anonKey: "anon-key", + serviceRoleKey: "service-role-key", + storageTargetMigration: "", + realtimeEnabledForSetup: false, + storageEnabledForSetup: false, + authEnabledForSetup: false, + serviceVersionOverrides: {}, + projectEnvValues: undefined, + debug: false, + webhooksEnabled: false, + apiAutoExposeNewTables: Option.some(true), + vault: [], +}); + +const shadowInput = ( + fs: FileSystem.FileSystem, + path: Path.Path, + overrides: { readonly shadowPort?: number; readonly jwtExpiry?: number } = {}, +): LegacyShadowSetupInput => ({ + db: { major_version: 17, settings: {} }, + experimental: defaultConfig.experimental, + jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", + jwtExpiry: overrides.jwtExpiry ?? 3600, + networkId: "supabase_network_proj", + image: "public.ecr.aws/supabase/postgres:17.4.1.030", + configImage: "supabase/postgres:17.4.1.030", + shadowPort: overrides.shadowPort ?? 54320, + password: "postgres", + projectId: "proj", + isBitbucketPipeline: false, + workdir: tempRoot.current, + extraHosts: [], + fs, + path, + hostname: "127.0.0.1", + healthTimeoutSeconds: 2, + setup: shadowSetup(), +}); + +const shadowCacheDir = (path: Path.Path) => legacyShadowBaselineCacheDir(path); + +/** The snapshot tars in the global cache dir, whatever keys they belong to. */ +const soleTarName = Effect.fnUntraced(function* (fs: FileSystem.FileSystem, path: Path.Path) { + const entries = yield* fs + .readDirectory(shadowCacheDir(path)) + .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); + return entries.filter((entry) => entry.endsWith(".tar")); +}); + +/** The cache key a published snapshot's filename (`shadow-baseline-.tar`) is stored under. */ +const keyOf = (tarName: string) => tarName.slice("shadow-baseline-".length, -".tar".length); + +/** + * What a correct cold export publishes under `tarName`: the fake PGDATA archive carrying the + * baseline marker stamped with THAT filename's own key. Derived from the name rather than hardcoded + * so the assertion fails if the export ever stamps a different key than it publishes under. + */ +const expectedTarFor = (tarName: string) => + legacyFakePgDataTar(legacyPgDataBaselineMarkerContent(keyOf(tarName))); + +/** A full cold run: acquire, export the baseline, release. */ +const coldRun = ( + docker: ReturnType, + input: LegacyShadowSetupInput, + opts: LegacyShadowCacheOpts = {}, +) => + Effect.gen(function* () { + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input, opts); + yield* handle.snapshotBaseline; + yield* legacyRemoveShadowDatabase(docker.spawner, handle.containerId); + return handle; + }); + +describe("legacyAcquireShadowDatabase", () => { + it.live("is today's bare create when the cache is explicitly disabled", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "0", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(handle.baselinePresent).toBe(false); + + // `--rm` intact, no PGDATA copies either way, and the snapshot step is a no-op. The one + // `cp-secret` is the pgsodium root key every shadow has always been given. + expect(docker.calls("create")[0] ?? []).toContain("--rm"); + yield* handle.snapshotBaseline; + expect(docker.steps()).toEqual(["create", "cp-secret", "start"]); + // Nothing is written to disk at all. + expect(yield* soleTarName(fs, path)).toEqual([]); + + yield* legacyRemoveShadowDatabase(docker.spawner, handle.containerId); + expect(docker.calls("rm")[0]).toEqual(["rm", "-f", "-v", handle.containerId]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("bypassCache acquires an uncached shadow even when a warm tar exists", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + // A published tar for this exact key — `sync --no-cache`'s bypass must ignore it. + yield* coldRun(docker, input); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input, { + bypassCache: true, + }); + expect(handle.baselinePresent).toBe(false); + // No restore in, no export out: the bypassed run neither reads nor rewrites the tar. + const bypassSteps = docker.steps().slice(docker.steps().lastIndexOf("create")); + expect(bypassSteps).toEqual(["create", "cp-secret", "start"]); + expect(docker.calls("create").at(-1) ?? []).toContain("--rm"); + yield* handle.snapshotBaseline; + expect(docker.stepCalls("cp-out")).toHaveLength(1); // the initial cold run's only + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("stays uncached on PG14, whose setup mutates role defaults mid-session", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + // PG<=14's globals SQL runs `ALTER ROLE … SET …` on the setup session; a snapshot + // boundary would force migrations onto a fresh session that observes those defaults, + // unlike Go's single-connection flow — so the cache must stand down entirely. + const base = shadowInput(fs, path); + const input = { + ...base, + db: { ...base.db, major_version: 14 }, + setup: { ...base.setup, majorVersion: 14 }, + }; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(handle.baselinePresent).toBe(false); + expect(handle.snapshotRequired).toBe(false); + expect(docker.calls("create")[0] ?? []).toContain("--rm"); + yield* handle.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a warm hit also sweeps abandoned partials left by a killed concurrent writer", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + yield* coldRun(docker, input); + // A concurrent writer that lost the publish race and was SIGKILLed mid-export: its + // partial predates the hour threshold. Every later run is warm, so the warm branch + // must be the one to sweep it. + const abandoned = path.join( + shadowCacheDir(path), + "shadow-baseline-0011223344556677.tar.4242.partial", + ); + yield* fs.writeFileString(abandoned, "stale"); + const twoHoursAgo = new Date(Date.now() - 2 * 60 * 60 * 1000); + yield* fs.utimes(abandoned, twoHoursAgo, twoHoursAgo); + + const warm = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(warm.baselinePresent).toBe(true); + expect(yield* fs.exists(abandoned)).toBe(false); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("stays uncached for an OrioleDB cluster even with the cache enabled", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + // OrioleDB runs the shadow with an external S3 storage backend, so a disk-level PGDATA + // tar is not a coherent snapshot — the acquire must degrade to the bare uncached shadow. + const input = { + ...shadowInput(fs, path), + experimental: { ...defaultConfig.experimental, orioledb_version: "15" }, + }; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(handle.baselinePresent).toBe(false); + expect(docker.calls("create")[0] ?? []).toContain("--rm"); + yield* handle.snapshotBaseline; + expect(docker.calls("stop")).toEqual([]); + expect(yield* soleTarName(fs, path)).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("takes the cache path when the env var is unset (default ON)", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + undefined, + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); + yield* handle.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("cold run stops, exports the tar, and starts the container again", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(handle.baselinePresent).toBe(false); + // The cold container must survive its own `docker stop`, so it carries no `--rm`. + expect(docker.calls("create")[0] ?? []).not.toContain("--rm"); + + yield* handle.snapshotBaseline; + + // Ordering IS the contract here: stop before the copy (a live PGDATA is not coherent to + // copy), the baseline marker stamped in between (it must be the LAST thing written to + // PGDATA, so nothing after the platform baseline can be missing from what it vouches + // for), start plus a readiness probe after it (the caller is about to reconnect). + expect(docker.steps()).toEqual([ + "create", + "cp-secret", + "start", + "stop", + "cp-stamp", + "cp-out", + "start", + "inspect", + ]); + expect(docker.stepCalls("cp-stamp")[0]).toEqual([ + "cp", + "-", + `${handle.containerId}:${LEGACY_PGDATA_PATH}`, + ]); + // The stamp really carries the marker file, delivered as a tar so `docker cp` unpacks it + // relative to PGDATA rather than rewriting the directory's ownership — and its content is + // this run's own cache key, which is what binds the artifact to the name it is filed under. + const stamp = docker.containers.get(handle.containerId)?.stamp ?? ""; + expect(stamp).toContain(LEGACY_PGDATA_BASELINE_MARKER_NAME); + expect(stamp).toContain(legacyPgDataBaselineMarkerContent(handle.snapshotKey ?? "")); + expect(docker.stepCalls("cp-out")[0]).toEqual([ + "cp", + `${handle.containerId}:${LEGACY_PGDATA_PATH}`, + "-", + ]); + expect(docker.containers.get(handle.containerId)?.running).toBe(true); + + // Exactly one tar, published under its final name with the exported bytes intact — no + // `.partial` left behind. + const tars = yield* soleTarName(fs, path); + expect(tars).toHaveLength(1); + expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); + const published = yield* fs.readFileString(path.join(shadowCacheDir(path), tars[0] ?? "")); + expect(published).toBe(expectedTarFor(tars[0] ?? "")); + expect(keyOf(tars[0] ?? "")).toBe(handle.snapshotKey); + // The stamp made it all the way into the artifact — this is the entry the next run's + // pre-restore scan requires, so a cold export that skipped it would never warm anything. + expect(published).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); + const leftovers = yield* fs.readDirectory(shadowCacheDir(path)); + expect(leftovers.filter((entry) => entry.includes("partial"))).toEqual([]); + + // Release is the uncached removal, same as ever — nothing is kept. + yield* legacyRemoveShadowDatabase(docker.spawner, handle.containerId); + expect(docker.ids()).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("warm run restores the tar into a FRESH container before starting it", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const cold = yield* coldRun(docker, input); + + const warm = yield* legacyAcquireShadowDatabase(docker.spawner, input); + // A brand new container every time — the cache keeps a file, never a container. + expect(warm.containerId).not.toBe(cold.containerId); + expect(warm.baselinePresent).toBe(true); + // Throwaway again: the warm container never gets stopped, so `--rm` is back. + expect(docker.calls("create").at(-1) ?? []).toContain("--rm"); + + // The restore lands BEFORE the start, and carries the exported bytes into PGDATA's parent. + const warmSteps = docker.steps().slice(docker.steps().lastIndexOf("create")); + expect(warmSteps).toEqual(["create", "cp-secret", "cp-in", "start", "inspect"]); + expect(docker.stepCalls("cp-in").at(-1)).toEqual([ + "cp", + "-", + `${warm.containerId}:${LEGACY_PGDATA_PARENT_PATH}`, + ]); + const [warmTarName = ""] = yield* soleTarName(fs, path); + expect(docker.containers.get(warm.containerId)?.restored).toBe( + `${LEGACY_PGDATA_PARENT_PATH}::${expectedTarFor(warmTarName)}`, + ); + + // Nothing more is exported: the baseline is already on disk. + yield* warm.snapshotBaseline; + expect(docker.calls("stop")).toHaveLength(1); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a pre-created permissive temp file cannot leak into the published tar's mode", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const tempDir = shadowCacheDir(path); + yield* fs.makeDirectory(tempDir, { recursive: true }); + // An adversarially (or crash-) pre-created temp file at THIS process's own temp path, + // world-readable. The export must not inherit its mode: the pre-remove + `wx` + // exclusive-create guarantees a fresh 0600 inode. + yield* coldRun(docker, input); // publish once to learn the tar name + const [tarName = ""] = yield* soleTarName(fs, path); + const tarPath = path.join(tempDir, tarName); + const tempPath = `${tarPath}.${process.pid}.partial`; + yield* fs.remove(tarPath); // force the next run cold + yield* fs.writeFileString(tempPath, "poisoned"); + yield* fs.chmod(tempPath, 0o666); + + yield* coldRun(docker, input); + + const info = yield* fs.stat(tarPath); + // 0o600 exactly — not the pre-created file's 0o666. + expect((Number(info.mode) & 0o777).toString(8)).toBe("600"); + expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a cold export sweeps abandoned partial temp files but never fresh ones", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const tempDir = shadowCacheDir(path); + yield* fs.makeDirectory(tempDir, { recursive: true }); + // A SIGKILLed export's leftover (writer long gone — hour-plus-old mtime) and a + // concurrent writer's live temp file (fresh mtime). + const abandoned = path.join(tempDir, "shadow-baseline-0123456789abcdef.tar.99999.partial"); + const live = path.join(tempDir, "shadow-baseline-fedcba9876543210.tar.88888.partial"); + yield* fs.writeFileString(abandoned, "stale"); + yield* fs.writeFileString(live, "in-flight"); + const twoHoursAgo = new Date(Date.now() - 2 * 60 * 60 * 1000); + yield* fs.utimes(abandoned, twoHoursAgo, twoHoursAgo); + + yield* coldRun(docker, shadowInput(fs, path)); + + expect(yield* fs.exists(abandoned)).toBe(false); + expect(yield* fs.exists(live)).toBe(true); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("publishing distinct keys keeps both tars until LRU/TTL eviction", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + yield* coldRun(docker, shadowInput(fs, path)); + const first = yield* soleTarName(fs, path); + expect(first).toHaveLength(1); + + // A changed baked-in input (jwt expiry) is a different cluster / different key — both + // must coexist in the global cache (unlike the old project-local current-key-only sweep, + // which would have deleted the first). Host publish port is NOT a key input. + const rekeyed = yield* coldRun(docker, shadowInput(fs, path, { jwtExpiry: 7200 })); + const both = yield* soleTarName(fs, path); + expect(both).toHaveLength(2); + expect(both).toContain(first[0]); + expect(rekeyed.baselinePresent).toBe(false); + + // An unrelated file in the cache directory is untouched by retention. + const stray = path.join(shadowCacheDir(path), "catalog-abc.json"); + yield* fs.writeFileString(stray, "{}"); + + // mtime is the LRU ordinal, and the rapid-fire publishes below can land within the + // filesystem's timestamp granularity — an mtime tie makes "oldest" ambiguous and the + // eviction pick arbitrary (observed as a CI-only failure). Age the first tar explicitly: + // this test asserts the keep-cap behavior, not tie-breaking. + const anHourAgo = new Date(Date.now() - 60 * 60 * 1000); + yield* fs.utimes(path.join(shadowCacheDir(path), first[0] ?? ""), anHourAgo, anHourAgo); + + // Fill to the keep-cap + 1 with more distinct keys; the oldest (first) is evicted. + for (let i = 0; i < LEGACY_SHADOW_BASELINE_KEEP - 1; i++) { + yield* coldRun(docker, shadowInput(fs, path, { jwtExpiry: 8000 + i })); + } + const afterCap = yield* soleTarName(fs, path); + expect(afterCap).toHaveLength(LEGACY_SHADOW_BASELINE_KEEP); + expect(afterCap).not.toContain(first[0]); + expect(yield* fs.exists(stray)).toBe(true); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("worktrees with identical settings share a warm hit from the global cache", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const worktreeA = path.join(tempRoot.current, "worktree-a"); + const worktreeB = path.join(tempRoot.current, "worktree-b"); + yield* fs.makeDirectory(path.join(worktreeA, "supabase"), { recursive: true }); + yield* fs.makeDirectory(path.join(worktreeB, "supabase"), { recursive: true }); + + const cold = yield* coldRun(docker, { ...shadowInput(fs, path), workdir: worktreeA }); + expect(cold.baselinePresent).toBe(false); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + // Same settings, different project path — the second worktree must restore, not re-export. + const warm = yield* legacyAcquireShadowDatabase(docker.spawner, { + ...shadowInput(fs, path), + workdir: worktreeB, + }); + expect(warm.baselinePresent).toBe(true); + expect(warm.containerId).not.toBe(cold.containerId); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + yield* legacyRemoveShadowDatabase(docker.spawner, warm.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a changed published host port is still a warm hit", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const cold = yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54320 })); + expect(cold.baselinePresent).toBe(false); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + // pg-delta next allocates an ephemeral host port per shadow; the published port is + // not in PGDATA, so a later run on a different port must restore the same tar. + const warm = yield* legacyAcquireShadowDatabase( + docker.spawner, + shadowInput(fs, path, { shadowPort: 54399 }), + ); + expect(warm.baselinePresent).toBe(true); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + yield* legacyRemoveShadowDatabase(docker.spawner, warm.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("legacy forced-on webhooks and next config-following webhooks do not share a tar", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + // `shadowSetup.webhooksEnabled` is false, so `"config"` (next migrate) and + // `"disabled"` (next declarative) bake the same cluster; `"enabled"` (legacy + // migrate) does not and must not restore that tar. + yield* coldRun(docker, input, { webhooks: "config" }); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + const forcedOn = yield* coldRun(docker, input, { webhooks: "enabled" }); + expect(forcedOn.baselinePresent).toBe(false); + expect(yield* soleTarName(fs, path)).toHaveLength(2); + + const warmConfig = yield* legacyAcquireShadowDatabase(docker.spawner, input, { + webhooks: "config", + }); + expect(warmConfig.baselinePresent).toBe(true); + const warmDisabled = yield* legacyAcquireShadowDatabase(docker.spawner, input, { + webhooks: "disabled", + }); + expect(warmDisabled.baselinePresent).toBe(true); + expect(yield* soleTarName(fs, path)).toHaveLength(2); + yield* legacyRemoveShadowDatabase(docker.spawner, warmConfig.containerId); + yield* legacyRemoveShadowDatabase(docker.spawner, warmDisabled.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a changed internal image registry is a different key, not a warm hit", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + yield* coldRun(docker, input); + const defaultRegistryTar = yield* soleTarName(fs, path); + expect(defaultRegistryTar).toHaveLength(1); + + // The one-shot migrate jobs resolve their images through + // `SUPABASE_INTERNAL_IMAGE_REGISTRY`, so a different registry can bake different + // realtime/storage/auth schema under identical tags — the snapshot must not be shared. + const mirrored = yield* legacyWithEnv( + "SUPABASE_INTERNAL_IMAGE_REGISTRY", + "mirror.internal.example", + coldRun(docker, input), + ); + expect(mirrored.baselinePresent).toBe(false); + const mirroredTar = yield* soleTarName(fs, path); + // Distinct keys coexist in the global cache — the default-registry tar is not swept. + expect(mirroredTar).toHaveLength(2); + expect(mirroredTar).toContain(defaultRegistryTar[0]); + expect(mirroredTar.some((name) => name !== defaultRegistryTar[0])).toBe(true); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live( + "a shadow that cannot come back after the snapshot fails the run, not just the cache", + () => { + const docker = mockLegacyDockerDaemonCliSpawner({ failRestart: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); + // The export itself succeeds (stop + copy-out are fine); the revive `docker start` fails. + // Reporting success here would send the caller's next connect to a dead container's port + // — possibly answered by a DIFFERENT Postgres by then — so this must be a failure, not a + // "not cached" warning. + const exit = yield* handle.snapshotBaseline.pipe(Effect.exit); + expect(Exit.isFailure(exit)).toBe(true); + expect(out.stderrText).not.toContain("Warning: shadow baseline not cached"); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }, + ); + + it.live("a failed export warns, leaves no tar, and still brings the container back up", () => { + const docker = mockLegacyDockerDaemonCliSpawner({ failCopyOut: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); + // The run itself must never fail for a cache problem. + yield* handle.snapshotBaseline; + + expect(out.stderrText).toContain("Warning: shadow baseline not cached"); + // The caller is about to reconnect, so the container is running again regardless. + expect(docker.containers.get(handle.containerId)?.running).toBe(true); + // Neither a published tar nor a half-written temp file survives. + const entries = yield* fs.readDirectory(shadowCacheDir(path)); + expect(entries).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live( + "a failed warm restore falls back cold, keeping the tar until its export replaces it", + () => { + const docker = mockLegacyDockerDaemonCliSpawner({ failCopyIn: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + // A cold run first, so a snapshot for this exact key exists to be restored. + const cold = yield* coldRun(docker, input); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(out.stderrText).toContain("cached shadow baseline unusable"); + // Falls all the way back to a cold provision — a fresh container with no baseline. + expect(fallback.baselinePresent).toBe(false); + expect(fallback.containerId).not.toBe(cold.containerId); + // The container whose restore failed is removed, not orphaned: with the cold run's own + // container already released by `coldRun`, only the fallback's remains. + expect(docker.ids()).toEqual([fallback.containerId]); + // An extraction failure does NOT implicate the tar's contents (it could just as well be + // a daemon hiccup), so the tar survives the fallback decision... + expect(yield* soleTarName(fs, path)).toHaveLength(1); + // ...and the cold fallback's own export atomically republishes over it, so a genuinely + // corrupt tar still self-heals within this one run. + yield* fallback.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }, + ); + + it.live("a published tar carrying no cluster is discarded instead of restored", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + yield* coldRun(docker, input); + const [tarName = ""] = yield* soleTarName(fs, path); + const tarPath = path.join(shadowCacheDir(path), tarName); + + // The published artifact is replaced by a tar that is perfectly well-formed but carries no + // PGDATA: `docker cp -` would extract nothing, the entrypoint would `initdb` a fresh + // cluster, readiness would pass, and the caller would diff against a BARE database while + // being told the platform baseline was present. + yield* fs.writeFileString(tarPath, LEGACY_FAKE_EMPTY_TAR); + const stepsBefore = docker.steps().length; + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); + + expect(out.stderrText).toContain("cached shadow baseline unusable"); + expect(out.stderrText).toContain("data/PG_VERSION"); + expect(fallback.baselinePresent).toBe(false); + // Caught before any container was created, so nothing was ever restored. + expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); + // The contents ARE the problem, so the tar goes — and the cold fallback republishes a + // good one within the same run, which is what keeps this fail-open. + expect(yield* soleTarName(fs, path)).toEqual([]); + yield* fallback.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toHaveLength(1); + expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); + yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a published tar carrying a bare cluster is discarded instead of restored", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + yield* coldRun(docker, input); + const [tarName = ""] = yield* soleTarName(fs, path); + const tarPath = path.join(shadowCacheDir(path), tarName); + + // A REAL, perfectly restorable PGDATA — but one that never ran the platform baseline. This + // is the failure `data/PG_VERSION` alone cannot see: `docker cp -` extracts a genuine + // cluster, the entrypoint SKIPS `initdb`, readiness passes, and the caller would be told + // the baseline is present while diffing against a bare database. Only the missing marker + // separates it from a usable snapshot. + yield* fs.writeFileString(tarPath, LEGACY_FAKE_UNSTAMPED_PGDATA_TAR); + const stepsBefore = docker.steps().length; + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); + + expect(out.stderrText).toContain("cached shadow baseline unusable"); + expect(out.stderrText).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); + expect(fallback.baselinePresent).toBe(false); + // Caught before any container was created, so nothing was ever restored. + expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); + // The contents ARE the problem, so the tar goes — and the cold fallback republishes a + // marked one within the same run. + expect(yield* soleTarName(fs, path)).toEqual([]); + yield* fallback.snapshotBaseline; + expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); + yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live( + "another key's snapshot copied over this key's filename is discarded, not restored", + () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const inputA = shadowInput(fs, path, { jwtExpiry: 3600 }); + const inputB = shadowInput(fs, path, { jwtExpiry: 7200 }); + // Two genuinely different configurations, each with its own published snapshot. + const coldA = yield* coldRun(docker, inputA); + const coldB = yield* coldRun(docker, inputB); + const keyA = coldA.snapshotKey ?? ""; + const keyB = coldB.snapshotKey ?? ""; + expect(keyA).not.toBe(keyB); + const tarPathB = path.join(shadowCacheDir(path), `shadow-baseline-${keyB}.tar`); + const tarPathA = path.join(shadowCacheDir(path), `shadow-baseline-${keyA}.tar`); + + // A's snapshot copied over B's cache file — the shape a copied `~/.supabase/cache` + // directory, a restored backup, or a hand-renamed tar produces. Every entry the + // presence check requires is there (it IS a real, fully baselined cluster), so only the + // marker's key separates it from B's own baseline: restoring it would silently diff + // against A's roles, vault values and service schema. + yield* fs.writeFileString(tarPathB, yield* fs.readFileString(tarPathA)); + const stepsBefore = docker.steps().length; + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, inputB); + + expect(out.stderrText).toContain("cached shadow baseline unusable"); + expect(out.stderrText).toContain(`snapshot is stamped with key ${keyA}, not ${keyB}`); + expect(fallback.baselinePresent).toBe(false); + // Caught before any container was created, so nothing was ever restored. + expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); + // Only the MISNAMED COPY goes: nothing else can ever be filed under B's name, while A's + // own tar — still correctly named — is left completely alone. + expect(yield* fs.exists(tarPathB)).toBe(false); + expect(yield* fs.readFileString(tarPathA)).toBe( + expectedTarFor(`shadow-baseline-${keyA}.tar`), + ); + // ...and the cold fallback republishes B's real baseline within the same run. + yield* fallback.snapshotBaseline; + expect(yield* fs.readFileString(tarPathB)).toBe( + expectedTarFor(`shadow-baseline-${keyB}.tar`), + ); + yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }, + ); + + it.live("a failed baseline stamp leaves the run uncached rather than publishing a tar", () => { + const docker = mockLegacyDockerDaemonCliSpawner({ failStamp: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); + // Same fail-open contract as every other export failure: the run itself never fails... + yield* handle.snapshotBaseline; + + expect(out.stderrText).toContain("Warning: shadow baseline not cached"); + expect(out.stderrText).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); + // ...the shadow is back up for the caller to reconnect to... + expect(docker.containers.get(handle.containerId)?.running).toBe(true); + // ...and nothing is published, because an UNMARKED tar would only be thrown away on the + // next run anyway. The export never even runs. + expect(docker.stepCalls("cp-out")).toHaveLength(0); + expect(yield* soleTarName(fs, path)).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a restored shadow that never becomes ready is removed before the cold retry", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const healthy = fakeCluster(); + const cold = yield* coldRun(docker, input).pipe(Effect.provide(healthy.layer)); + const tarBefore = yield* soleTarName(fs, path); + expect(tarBefore).toHaveLength(1); + + // The restored cluster refuses every connection — the restore produced something + // unstartable, so the tar itself is suspect. + const broken = fakeCluster({ failConnect: true }); + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input).pipe( + Effect.provide(broken.layer), + ); + + expect(fallback.baselinePresent).toBe(false); + expect(fallback.containerId).not.toBe(cold.containerId); + // The suspect container is gone before the replacement is created — it holds the shadow's + // published port. + const removeIndex = docker.spawned.findIndex((args) => args[0] === "rm"); + const recreateIndex = docker.spawned.findLastIndex((args) => args[0] === "create"); + expect(removeIndex).toBeGreaterThanOrEqual(0); + expect(removeIndex).toBeLessThan(recreateIndex); + expect(yield* soleTarName(fs, path)).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, fakeCluster().layer))); + }); +}); + +describe("SUPABASE_SHADOW_DEBUG phase-timing instrumentation", () => { + it.live("emits export and restore phase lines when the debug env var is set", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const { writes } = yield* withShadowDebugEnv( + "1", + captureStderr( + Effect.gen(function* () { + yield* coldRun(docker, input); + yield* legacyAcquireShadowDatabase(docker.spawner, input); + }), + ), + ); + + // Routed through the mocked `Output` (both phases run inside shadow-cache.ts, which + // always has `Output` in context). + expect(out.stderrText).toContain("shadow-debug: baseline-export"); + expect(out.stderrText).toContain("shadow-debug: baseline-restore"); + // Written straight to the real `process.stderr` by `legacyWaitForShadowReady` + // (`health-check.ts`), which has no `Output` in its own context. + expect(writes.some((chunk) => chunk.includes("shadow-debug: ready-wait"))).toBe(true); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("emits no shadow-debug lines when the debug env var is unset", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + withShadowDebugEnv( + undefined, + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const { writes } = yield* captureStderr(coldRun(docker, shadowInput(fs, path))); + expect(out.stderrText).not.toContain("shadow-debug:"); + expect(writes.some((chunk) => chunk.includes("shadow-debug:"))).toBe(false); + }), + ), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts new file mode 100644 index 0000000000..c76445b069 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts @@ -0,0 +1,235 @@ +/** + * The shadow baseline cache's ONE live scenario (golden path only, per the repo's live-test + * policy): the SAME `db diff` invocation run TWICE against a real local stack must cold-publish a + * `shadow-baseline-.tar` on the first run, warm-restore that exact tar on the second, and + * produce byte-identical diff output either way. + * + * A black-box `runSupabaseLive` subprocess test, like every other `*.live.test.ts` in this + * workspace: the facts it is here to prove are the ones only the real wiring can — that `db diff` + * actually routes through `legacyAcquireShadowDatabase`, that the cache's env gates + * (`SUPABASE_SHADOW_CACHE`/`SUPABASE_SHADOW_DEBUG`) and its `${SUPABASE_HOME}/cache/shadow-baseline` + * artifact location survive a real process boundary, that the cache key is STABLE across two + * separate CLI processes (an in-process test computes it once), and that a warm-restored cluster + * yields the same migration SQL as a cold-provisioned one. It replaces an earlier in-process + * version of this file that called `legacyAcquireShadowDatabase` directly with a synthetic layer + * graph — that shape could stay green while the `db diff` wiring, the env propagation, or the cache + * enablement was broken. + * + * The acquire/export/restore MECHANICS (cold export, warm restore, tar validation and rejection, + * retention/LRU, cache-off and bypass paths) are covered exhaustively by + * `shadow-cache.integration.test.ts` against its in-test Docker model plus a real filesystem, and + * the pure key/retention logic by `shadow-cache.unit.test.ts`. Nothing branch-shaped belongs here. + * + * Gated with `describeDockerLive` (the cli-e2e-ci signal composed with a `docker info` probe, since + * this is a Docker-only local-stack suite). + */ + +import { mkdtemp, readdir, rm, stat } from "node:fs/promises"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, expect, test } from "vitest"; + +import { makeTempHome } from "../../../../tests/helpers/cli.ts"; +import { describeDockerLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; + +const START_TIMEOUT_MS = 280_000; +const DIFF_TIMEOUT_MS = 180_000; +// One full `start` plus the cold/warm `db diff` pair, with lifecycle overhead for `init`, the +// filesystem inspection between runs, and the fast-failing port-conflict retries below (a +// conflicting publish fails in `docker create`/`start`, i.e. seconds, never a whole +// `DIFF_TIMEOUT_MS`) — same "budget each subprocess separately" shape as `diff.live.test.ts`. +const LIFECYCLE_OVERHEAD_MS = 90_000; + +/** + * `db diff`'s shadow port, published on the host by the shadow container. Docker itself has to + * bind it, so a test CANNOT truly reserve it up front: binding a listener and releasing it proves + * nothing about the window between the release and the container's own bind. The honest mitigation + * is therefore two-part — pick ports far from the `[db] shadow_port` default (54320) that a stray + * local stack or a neighbouring suite would be holding, and retry the scenario on the next + * candidate when the CLI reports a real bind conflict. + * + * Fed through `SUPABASE_DB_SHADOW_PORT` (`legacy-db-config.toml-read.ts`'s `envOverride`) rather + * than by rewriting the generated `config.toml`, so the `init` template stays exactly as a user's + * would be. The port is deliberately NOT part of the cache key (see `legacyShadowCacheKey`), so + * retrying on a different one cannot change which tar the run looks for. + * + * The candidate sequence is derived from this process's own pid, so two independently concurrent + * runs of this suite start from different bases instead of racing for one shared pair, and a + * locally-occupied port only costs one retry step. The base stays inside the IANA dynamic range + * (49152-65535) with room for every candidate below its ceiling. + */ +const SHADOW_PORT_CANDIDATE_COUNT = 8; +const SHADOW_PORT_BASE = 49152 + ((process.pid * 37) % (16384 - SHADOW_PORT_CANDIDATE_COUNT)); +const SHADOW_PORT_CANDIDATES: ReadonlyArray = Array.from( + { length: SHADOW_PORT_CANDIDATE_COUNT }, + (_, index) => SHADOW_PORT_BASE + index, +); + +const DIFF_ARGS = ["db", "diff", "--local", "--use-pg-delta"] as const; + +/** `shadow-cache.ts`'s published artifact name — `shadow-baseline-<16 hex key>.tar`. */ +const BASELINE_TAR_PATTERN = /^shadow-baseline-[0-9a-f]{16}\.tar$/u; + +/** + * Docker's own bind-conflict wording, as it reaches stderr through the shadow's + * `docker create`/`docker start` failure. Only used to decide whether to retry on another + * candidate port — never asserted on. + */ +function isShadowPortConflict(stderr: string): boolean { + return /port is already allocated|address already in use|Bind for \S+ failed/iu.test(stderr); +} + +async function baselineTars(cacheDir: string): Promise> { + const entries = await readdir(cacheDir).catch(() => [] as Array); + return entries.filter((entry) => entry.endsWith(".tar")).sort(); +} + +describeDockerLive("shadow baseline cache (live Docker)", () => { + let projectDir: string | undefined; + let home: ReturnType | undefined; + + afterEach(async () => { + if (projectDir !== undefined) { + // Best-effort cleanup even if an assertion above failed mid-lifecycle — a leaked local + // stack would otherwise pollute the CI runner for later jobs. + await runSupabaseLive(["stop", "--no-backup"], { + cwd: projectDir, + ...(home === undefined ? {} : { home: home.dir }), + }).catch(() => undefined); + await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); + } + // Disposes the temp `SUPABASE_HOME`, and with it the ~90MB baseline tar this suite published. + home?.[Symbol.dispose](); + projectDir = undefined; + home = undefined; + }); + + test( + "publishes a baseline snapshot on the first db diff, then restores it on the second with identical output", + { timeout: START_TIMEOUT_MS + 2 * DIFF_TIMEOUT_MS + LIFECYCLE_OVERHEAD_MS }, + async () => { + projectDir = await mkdtemp(path.join(tmpdir(), "sb-shadow-cache-live-")); + // One temp `SUPABASE_HOME` for every run in this test, so the two `db diff` processes share + // the global `${SUPABASE_HOME}/cache/shadow-baseline` directory the cache publishes into — + // `runSupabase` otherwise mints (and disposes) a fresh home per invocation, which would make + // every run a cold one. + home = makeTempHome(); + const cacheDir = path.join(home.dir, "cache", "shadow-baseline"); + + const init = await runSupabaseLive(["init"], { cwd: projectDir, home: home.dir }); + expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + + // Same declarative setup as `db/diff/diff.live.test.ts`: point `[db.migrations] + // schema_paths` at a schema directory so `db diff --local` has real, deterministic SQL to + // produce — the payload whose byte-identity across the cold and warm runs is the actual + // user-visible contract here. Paths are relative to `supabase/`. + const configPath = path.join(projectDir, "supabase", "config.toml"); + const config = readFileSync(configPath, "utf8"); + expect(config).toContain("schema_paths = []"); + writeFileSync( + configPath, + config.replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]'), + ); + const schemasDir = path.join(projectDir, "supabase", "schemas"); + mkdirSync(schemasDir, { recursive: true }); + writeFileSync( + path.join(schemasDir, "01_probe_fn.sql"), + `create function public.probe_fn() +returns void +language sql +as $$ select 1; $$; +`, + ); + + // Exclude the heaviest, least relevant services — `db diff` only needs the local Postgres + // container reachable, same rationale as stop/status/diff. + const start = await runSupabaseLive( + ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], + { cwd: projectDir, home: home.dir, exitTimeoutMs: START_TIMEOUT_MS }, + ); + expect(start.exitCode, `stdout:\n${start.stdout}\nstderr:\n${start.stderr}`).toBe(0); + + let cold: Awaited> | undefined; + let warm: Awaited> | undefined; + let coldTars: ReadonlyArray = []; + let coldMtimeMs = 0; + + for (const [index, shadowPort] of SHADOW_PORT_CANDIDATES.entries()) { + const canRetry = index < SHADOW_PORT_CANDIDATES.length - 1; + // Each attempt must start from an empty cache, or the previous attempt's tar would make + // this attempt's first run a warm one. + await rm(cacheDir, { recursive: true, force: true }); + const diffOptions = { + cwd: projectDir, + home: home.dir, + exitTimeoutMs: DIFF_TIMEOUT_MS, + env: { + // The e2e/live harness pins this to "0" by default so ordinary suites never leave a + // ~90MB tar behind; this suite's subject IS the cache, so it opts back in. + SUPABASE_SHADOW_CACHE: "1", + // Turns on `shadow-debug.ts`'s stderr phase lines, which name the path actually taken + // (`baseline-export` cold, `baseline-restore` warm). + SUPABASE_SHADOW_DEBUG: "1", + SUPABASE_DB_SHADOW_PORT: String(shadowPort), + }, + }; + + const first = await runSupabaseLive([...DIFF_ARGS], diffOptions); + if (first.exitCode !== 0 && canRetry && isShadowPortConflict(first.stderr)) continue; + cold = first; + coldTars = await baselineTars(cacheDir); + if (coldTars.length === 1) { + coldMtimeMs = (await stat(path.join(cacheDir, coldTars[0]!))).mtimeMs; + } + // A genuine cold-run failure is reported below rather than spending another full + // `DIFF_TIMEOUT_MS` on a warm run that has no snapshot to restore. + if (first.exitCode !== 0) break; + + const second = await runSupabaseLive([...DIFF_ARGS], diffOptions); + if (second.exitCode !== 0 && canRetry && isShadowPortConflict(second.stderr)) { + cold = undefined; + continue; + } + warm = second; + break; + } + + // --- Run 1: cold. The baseline was provisioned and exported as one keyed tar. --- + expect(cold, "every candidate shadow port reported a bind conflict").toBeDefined(); + if (cold === undefined) return; + expect(cold.exitCode, `stdout:\n${cold.stdout}\nstderr:\n${cold.stderr}`).toBe(0); + expect(cold.stderr).toContain("shadow-debug: baseline-export"); + expect(cold.stderr).not.toContain("shadow-debug: baseline-restore"); + // The artifact is a plain file under the global per-settings cache — the property that lets + // worktrees with the same settings share a warm hit, and a future native (non-Docker) + // Postgres service consume the same snapshot. + expect(coldTars, `cache dir: ${cacheDir}\nstderr:\n${cold.stderr}`).toHaveLength(1); + expect(coldTars[0]).toMatch(BASELINE_TAR_PATTERN); + + // --- Run 2: warm. The same key restored that snapshot instead of rebuilding it. --- + expect(warm).toBeDefined(); + if (warm === undefined) return; + expect(warm.exitCode, `stdout:\n${warm.stdout}\nstderr:\n${warm.stderr}`).toBe(0); + expect(warm.stderr).toContain("shadow-debug: baseline-restore"); + expect(warm.stderr).not.toContain("shadow-debug: baseline-export"); + // Neither degradation path may have engaged — both warn on stderr before falling back to a + // cold provision, and either would otherwise hide a broken warm path behind a passing run. + expect(warm.stderr).not.toContain("cached shadow baseline unusable"); + expect(warm.stderr).not.toContain("shadow baseline not cached"); + // Same single tar, same filename: the key is reproducible across processes, and the warm run + // published nothing of its own. + const warmTars = await baselineTars(cacheDir); + expect(warmTars).toEqual(coldTars); + // Warm hits refresh mtime so a frequently used key survives LRU/TTL retention. + const warmMtimeMs = (await stat(path.join(cacheDir, warmTars[0]!))).mtimeMs; + expect(warmMtimeMs).toBeGreaterThan(coldMtimeMs); + + // The user-visible contract is unchanged by which path ran: stdout carries the migration SQL + // (no `-f`, so `db diff` prints it), and a restored cluster must diff to exactly the same + // statements as a freshly baselined one. + expect(cold.stdout).toContain("CREATE FUNCTION public.probe_fn()"); + expect(warm.stdout).toBe(cold.stdout); + }, + ); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts new file mode 100644 index 0000000000..c4a852c4c1 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -0,0 +1,1161 @@ +/** + * Shadow baseline cache — the acquire/release pair `db diff`/`db pull`/the migrations-catalog + * resolution path use in place of bare `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` + * (`shadow-database.ts`). Caches the shadow's platform baseline (init schema + the PG15+ one-shot + * realtime/storage/auth jobs) as a disk-level PGDATA snapshot, never a kept container. + * + * - **Cold** (no snapshot for this key): provision the shadow as an uncached run does, then — + * right after the baseline and before `contrib_regression`/any user migration — stop the + * container, export its PGDATA via {@link legacyExportPgDataTar} (`pgdata-snapshot.ts`), and + * start it again. + * - **Warm** (that tar exists): create the shadow with the tar unpacked into it before it starts + * ({@link legacyPgDataRestoreArchive}), so the entrypoint skips `initdb` and the whole baseline; + * the caller applies user migrations straight onto the restored `postgres`. + * + * Invariants: the artifact is a plain file, not a Docker object (native-services friendly — see + * `pgdata-snapshot.ts`'s own header); no container outlives a run — cold, warm, and cache-off + * shadows are all removed with `docker rm -f -v` on release; the cold path alone drops `--rm` (see + * {@link LegacyCreateShadowDatabaseInput.autoRemove} for the consequence); the tar is published by + * an atomic rename, so concurrent writers need no lock file; a cache anomaly never fails the run — + * a warm-path anomaly cold-provisions (deleting the tar only when its contents are implicated — + * see `LegacyShadowCacheUnavailable.tarSuspect`), a cold export failure only warns and leaves the + * run uncached (with ONE deliberate exception: a shadow that fails to come back up after the + * snapshot fails the run — see `legacyExportShadowBaseline`); tars live under the global + * `${SUPABASE_HOME}/cache/shadow-baseline/` (shared across worktrees with the same settings), + * with LRU (keep 8) + 14-day mtime TTL retention. `SUPABASE_SHADOW_CACHE` is ON by default; + * `false`/`0` opts out. + */ + +import { createHash } from "node:crypto"; + +import type { ProjectConfig } from "@supabase/config"; +import { Clock, Effect, Option, Result, type FileSystem } from "effect"; +import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; + +import { Output } from "../../../shared/output/output.service.ts"; +import { + containerCliExitCode, + legacyDescribeContainerCliFailure, +} from "../legacy-container-cli.ts"; +import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; +import type { LegacyPgConnInput } from "../legacy-db-connection.service.ts"; +import { legacyGetRegistryImageUrl } from "../legacy-docker-registry.ts"; +import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; +import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; +import { LEGACY_POSTGRES_DEFAULT_ROOT_KEY } from "../legacy-local-config-values.ts"; +import { + LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL, + LEGACY_START_REVOKE_API_PRIVILEGES_SQL, + type LegacySetupDatabaseOptions, +} from "./db-setup.ts"; +import { + LEGACY_START_INTERNAL_DB_NAME, + LEGACY_START_INTERNAL_DB_PORT, +} from "./internal-db-connection.ts"; +import { + LEGACY_REALTIME_DB_USER, + LEGACY_REALTIME_ENCRYPTION_KEY, + LEGACY_REALTIME_TENANT_ID, +} from "./realtime-env.ts"; +import { LEGACY_START_DB_SCHEMA_SQL } from "./templates/db-schema.sql.ts"; +import { LEGACY_START_DB_SUPABASE_SQL } from "./templates/db-supabase.sql.ts"; +import { LEGACY_START_DB_WEBHOOK_SQL } from "./templates/db-webhook.sql.ts"; +import { + LEGACY_CREATE_VAULT_KV, + LEGACY_READ_VAULT_KV, + LEGACY_UPDATE_VAULT_KV, + type LegacyVaultSecret, +} from "../legacy-vault.ts"; +import { legacyWaitForShadowReady } from "./health-check.ts"; +import { + legacyExportPgDataTar, + legacyPgDataRestoreArchive, + legacyStampPgDataBaselineMarker, + legacyValidatePgDataArchive, +} from "./pgdata-snapshot.ts"; +import type { + LegacyPgDataArchiveProblem, + LegacyPgDataSnapshotUnavailable, +} from "./pgdata-snapshot.ts"; +import { legacyResolvePinnedImage } from "./pinned-image.ts"; +import { legacyTimeShadowPhase } from "./shadow-debug.ts"; +import { + legacyCreateShadowDatabase, + legacyRemoveShadowDatabase, + type LegacyShadowBaselineState, + LegacyShadowDbError, + type LegacyShadowSetupInput, +} from "./shadow-database.ts"; + +type Spawner = ChildProcessSpawner["Service"]; + +/** `SUPABASE_SHADOW_CACHE` — the opt-OUT gate for this whole module (default ON). */ +export const LEGACY_SHADOW_CACHE_ENV = "SUPABASE_SHADOW_CACHE"; + +/** + * Internal-only "the cache cannot be used" signal. Deliberately NOT a `Data.TaggedError`: it + * never reaches a user or telemetry — every producer is caught by + * {@link legacyAcquireShadowDatabase} (which falls back to a cold provision) or by + * {@link legacyExportShadowBaseline} (which warns and continues uncached), so classifying it as a + * CLI error would be misleading, and would pollute the error-actionability vocabulary with a + * failure the user can neither see nor act on. + */ +interface LegacyShadowCacheUnavailable { + readonly reason: string; + /** + * `true` only when the failure implicates the TAR'S CONTENTS — today, exactly three producers, + * all in {@link legacyWarmShadow}: an archive whose header stream is missing a required entry — + * the PGDATA cluster file or the baseline marker; one whose marker vouches for a DIFFERENT cache + * key than the filename it is stored under (both checked before any container is created); and a + * restored cluster that started but never + * accepted connections (the readiness wait). The wrong-key case is the one where the bytes may + * be a perfectly good snapshot — of another key — so what is discarded is only the MISNAMED + * COPY, which is exactly right: nothing else can be keyed by this filename. + * Everything else (a `docker create`/`cp`/`start` + * failure — daemon outage, + * port collision, or even a corrupt archive's failed extraction) leaves the tar in place: an + * infra failure says nothing about the tar, and a genuinely corrupt one is atomically + * REPLACED by the cold fallback's own export in the same run, so deleting up front would only + * throw away a valid ~15s-to-rebuild baseline on transient Docker failures (review: Codex on + * #6184). + */ + readonly tarSuspect?: boolean; +} + +const legacyShadowCacheUnavailable = ( + reason: string, + opts: { readonly tarSuspect?: boolean } = {}, +): LegacyShadowCacheUnavailable => ({ reason, ...opts }); + +/** + * Whether the shadow baseline cache is enabled for this invocation. + * + * Unset (or empty) means ON — this is a default-on optimization, not a feature flag. A value that + * IS set goes through the repo's `viper.GetBool` parser, so `SUPABASE_SHADOW_CACHE=false` and + * `=0` opt out while `=1`/`=true` are explicit opt-ins. + * + * `projectEnvValues` (the project's dotenv-merged env, ambient-wins — see + * `legacyGetRegistryOverride`'s identical parameter, `legacy-docker-registry.ts`) is consulted + * first so an opt-out set only in `supabase/.env` is honored, matching how Go's `loadNestedEnv` + * makes project dotenv visible to every viper read (review: Codex on #6184). Falls back to `env` + * (ambient) when the record is absent or lacks the key. + */ +export function legacyShadowCacheEnabled( + env: Readonly> = process.env, + projectEnvValues?: Readonly>, +): boolean { + const raw = projectEnvValues?.[LEGACY_SHADOW_CACHE_ENV] ?? env[LEGACY_SHADOW_CACHE_ENV]; + if (raw === undefined || raw.length === 0) return true; + return legacyParseBoolEnv(raw); +} + +// --------------------------------------------------------------------------- +// Cache key +// --------------------------------------------------------------------------- + +/** One of the three PG15+ one-shot migrate jobs, as the cache key sees it. */ +export interface LegacyShadowCacheServiceInput { + readonly enabled: boolean; + /** + * `legacyResolvePinnedImage`'s pinned image passed through `legacyGetRegistryImageUrl` — the + * REGISTRY-RESOLVED ref, not the bare `supabase/:` pin, because that is the identity + * `legacyRunStartMigrateJob`'s own `legacyEnsureImagesCached` resolve actually runs: a changed + * `SUPABASE_INTERNAL_IMAGE_REGISTRY` (ambient or project-`.env`) can serve a different image + * under the same tag, and the postgres image field above is already the registry-resolved form + * (review: Codex on #6184). Hashed ONLY when {@link enabled}, since a disabled service's job + * never ran into the baseline. + */ + readonly image: string; +} + +/** + * Every input baked into the shadow cluster during a cold provision. Deliberately NOT + * `legacySetupInputsToken`'s shape (`legacy-pgdelta.cache.ts`): that key is a Go-parity + * byte-for-byte contract shared with the Go binary AND hashes vault NAMES only while omitting + * the one-shot job image tags — both fatal for a CLUSTER snapshot, which carries the vault + * secrets' values and the versioned `auth`/`storage`/`_realtime` schema those jobs write. This + * key mirrors that function's hashing STYLE (sha256 over newline-joined formatted fields) without + * reusing it. + */ +export interface LegacyShadowCacheKeyInputs { + /** The resolved, full `supabase/postgres` image (tag included — a major version is not enough). */ + readonly postgresImage: string; + readonly majorVersion: number; + readonly jwtSecret: string; + readonly jwtExpiry: number; + readonly rootKey: string; + /** `[db] password` — baked into the cluster as the `postgres` role's password. */ + readonly dbPassword: string; + /** + * The Storage migration pin read from `supabase/.temp/storage-migration` (written by + * `supabase link`), fed as `DB_MIGRATIONS_FREEZE_AT` to the Storage one-shot migrate job + * (`legacyStartStorageMigrateEnv`, `db-setup.ts`) — it decides WHICH Storage migrations the + * baseline carries, independently of the job image's own tag. `""` when absent (unlinked + * project), matching the setup input's own zero value. Hashed ONLY when `services.storage` + * is enabled AND `majorVersion >= 15` — the exact compound gate `legacyStartInitSchema15` + * (`db-setup.ts`) puts the consuming job behind, mirroring {@link jwks}'s treatment + * (review: depthfirst/Codex on #6184). + */ + readonly storageTargetMigration: string; + readonly dbSettings: ProjectConfig["db"]["settings"]; + /** + * `api.auto_expose_new_tables` as config carries it. Hashed as the EFFECTIVE two-state behavior, + * not the raw tri-state — see {@link legacyEffectiveShadowApiGrantsKept}. + */ + readonly autoExposeNewTables: Option.Option; + /** + * Effective Webhooks/`pg_net` policy baked into the cluster — the same boolean + * `legacySetupDatabase` applies (`options.webhooks` resolved against + * `setup.webhooksEnabled`). Distinct from the raw config flag: legacy migrate + * forces enabled, next declarative forces disabled, and next migrate follows + * config. Hashed so those callers cannot share a snapshot (review: Codex/ + * depthfirst on #6184). + */ + readonly webhooksEnabled: boolean; + /** `supabase/roles.sql`'s contents, `""` when absent. */ + readonly rolesSql: string; + /** + * `[db.vault]` secrets — names AND values, both of which land in `vault.secrets`. Only + * RESOLVED entries are hashed ({@link legacyShadowCacheKey}), because the upsert skips + * unresolved ones entirely — see the loop's own comment there. + */ + readonly vault: ReadonlyArray; + readonly services: { + readonly realtime: LegacyShadowCacheServiceInput; + readonly storage: LegacyShadowCacheServiceInput; + readonly auth: LegacyShadowCacheServiceInput; + }; + /** + * The resolved JWKS string (`LegacyShadowDbSetupInput.jwks`, an `Effect` the caller resolves + * lazily) that realtime's one-shot tenant-seed job bakes into the cluster + * (`legacyBuildRealtimeEnv`'s `jwks` field, `db-setup.ts`) — a config change under + * `auth.third_party` changes this value without touching anything else in this struct, so it + * needs its own field rather than folding into `services.realtime`. Hashed ONLY when + * `services.realtime` is enabled AND `majorVersion >= 15` — the exact compound gate + * {@link legacyResolveDbSetupPrelude} itself uses to decide whether to resolve (and therefore + * whether the one-shot job ever consumes) this same effect during a real setup: a disabled or + * pre-PG15 realtime never reaches the job that reads it. `""` when excluded, mirroring + * {@link LegacyShadowCacheServiceInput.image}. + */ + readonly jwks: string; +} + +/** + * Digest of every CLI-EMBEDDED literal baked into the baseline cluster — the inputs that change + * with a CLI release rather than with the project's config: the PG15+ entrypoint's initdb heredocs + * (schema/webhook/_supabase — `postgres.service.ts`), the API privilege revocation, and the + * Realtime one-shot job's seed constants. Without this line, a CLI upgrade that edits a grant, + * schema statement, revocation, or seeded literal WITHOUT bumping the corresponding image would + * warm-restore the previous release's baseline (review: depthfirst/Codex on #6184). Computed once + * at module load — these are compile-time constants. When adding a new embedded step to the + * baseline (`legacySetupDatabase`/the entrypoint scripts/a one-shot job's env), add its text here + * too. + * + * Deliberately EXCLUDES PG<=14's own setup SQL (`LEGACY_START_DB_GLOBALS_SQL`, + * `LEGACY_START_DB_INITIAL_SCHEMA_13_SQL`/`_14_SQL`): PG<=14 is cache-ineligible — + * {@link legacyResolveShadowCacheKeyInputs} returns `Option.none()` for `majorVersion <= 14` before + * any key is computed, so no cluster keyed by this digest can ever have run through that SQL. If + * PG<=14 ever becomes cache-eligible, those templates must be re-added here. + */ +const LEGACY_SHADOW_BASELINE_EMBEDDED_DIGEST = createHash("sha256") + .update( + [ + LEGACY_START_DB_SCHEMA_SQL, + LEGACY_START_DB_WEBHOOK_SQL, + LEGACY_START_DB_SUPABASE_SQL, + LEGACY_START_REVOKE_API_PRIVILEGES_SQL, + // The webhooks-enable statement `legacySetupDatabase` runs for a webhooks-enabled baseline + // (`db-setup.ts`). `webhooksEnabled` above only says WHETHER it ran; this line covers the + // text it ran, so editing the statement re-keys those tars too (review: Codex on #6184). + LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL, + // The vault upsert's own SQL (`legacyUpsertVaultSecrets`, `legacy-vault.ts`) runs into the + // baseline right after the privilege pass — same digest rationale as every line above. + LEGACY_READ_VAULT_KV, + LEGACY_UPDATE_VAULT_KV, + LEGACY_CREATE_VAULT_KV, + // The Realtime one-shot job's CLI-embedded seed literals. `SEED_SELF_HOST=true` + // (`legacyBuildRealtimeEnv`, `realtime-env.ts`) makes that job PERSIST a tenant plus its + // `postgres_cdc_rls` extension settings into `_realtime`, encrypted with `DB_ENC_KEY` — so + // these values are baked into the snapshot exactly like the SQL above, and every one of them + // is a `toml:"-"`/hardcoded literal a CLI release can edit. `services.realtime.image` only + // re-keys when the IMAGE moves, so without these lines such an edit would warm-restore the + // old tenant identity and encryption key (review: Codex on #6184). Only the CONSTANTS + // belong here: the job's per-run env (the shadow's own short container id as `DB_HOST`, its + // password, the resolved JWKS) is either already a key field or deliberately excluded. + LEGACY_REALTIME_TENANT_ID, + LEGACY_REALTIME_ENCRYPTION_KEY, + LEGACY_REALTIME_DB_USER, + LEGACY_START_INTERNAL_DB_NAME, + String(LEGACY_START_INTERNAL_DB_PORT), + ].join("\n--8<--\n"), + "utf8", + ) + .digest("hex"); + +/** JSON with recursively key-sorted objects, so `db.settings`' own property order cannot change the key. */ +function legacyCanonicalJson(value: unknown): string { + if (value === null || typeof value !== "object") return JSON.stringify(value) ?? "null"; + if (Array.isArray(value)) return `[${value.map(legacyCanonicalJson).join(",")}]`; + const entries = Object.entries(value) + .filter(([, entryValue]) => entryValue !== undefined) + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)); + return `{${entries.map(([key, entryValue]) => `${JSON.stringify(key)}:${legacyCanonicalJson(entryValue)}`).join(",")}}`; +} + +const legacyBoolToken = (value: boolean) => (value ? "true" : "false"); + +/** + * The two-state behavior `legacyApplyApiPrivileges` (`db-setup.ts`) actually derives from + * `api.auto_expose_new_tables`' tri-state: it returns early ONLY for an explicit `true`, so unset + * and explicit `false` both exec {@link LEGACY_START_REVOKE_API_PRIVILEGES_SQL} and bake the exact + * same cluster. Hashing the raw tri-state would split those two into different keys and force a + * spurious ~90MB re-snapshot for a config edit that changes nothing on disk (review: Codex on + * #6184). + */ +const legacyEffectiveShadowApiGrantsKept = (value: Option.Option): boolean => + Option.getOrElse(value, () => false); + +/** + * The cache key: a 16-hex-char (64-bit) sha256 prefix over a fixed field order. 64 bits is + * ample for a per-settings global cache whose only cost of a collision would be a wrong baseline + * — and every genuinely divergent input is in the payload, so a collision needs an actual hash + * collision, not a missed field. Short enough to read in a filename. + */ +export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string { + // Every UNRESTRICTED string is JSON-encoded before interpolation (`quoted`): a raw newline in + // one field could otherwise forge a whole extra payload line, letting two distinct + // configurations collide (review: Codex on #6184 — same class as the vault tuples below). + // Numbers and closed tokens need no quoting; `rolesSql` stays raw because it is the + // documented LAST field, with nothing after it to forge. + const quoted = (value: string) => JSON.stringify(value); + const lines: Array = [ + `postgres_image=${quoted(inputs.postgresImage)}`, + `major_version=${inputs.majorVersion}`, + // Host publish port is deliberately excluded: it is not baked into PGDATA, and + // pg-delta next allocates an ephemeral port per shadow. Hashing it would miss + // every warm hit on that path. Restore always uses the current run's port. + `jwt_secret=${quoted(inputs.jwtSecret)}`, + `jwt_expiry=${inputs.jwtExpiry}`, + `root_key=${quoted(inputs.rootKey)}`, + `db_password=${quoted(inputs.dbPassword)}`, + `db_settings=${legacyCanonicalJson(inputs.dbSettings)}`, + `api_grants_kept=${legacyBoolToken(legacyEffectiveShadowApiGrantsKept(inputs.autoExposeNewTables))}`, + `webhooks_enabled=${legacyBoolToken(inputs.webhooksEnabled)}`, + // Not a per-run input — see the digest's own doc comment for what it covers and why. + `baseline_embedded_digest=${LEGACY_SHADOW_BASELINE_EMBEDDED_DIGEST}`, + ]; + for (const name of ["realtime", "storage", "auth"] as const) { + const service = inputs.services[name]; + lines.push( + service.enabled + ? `service=${name} enabled=true image=${quoted(service.image)}` + : `service=${name} enabled=false`, + ); + } + // Realtime's resolved JWKS — see the field's own doc comment for the compound + // enabled+majorVersion gate (mirrors `service=realtime`'s own `enabled` exclusion above, plus + // the PG15+ gate the one-shot job itself is behind). + lines.push( + inputs.services.realtime.enabled && inputs.majorVersion >= 15 + ? `realtime_jwks=${quoted(inputs.jwks)}` + : "realtime_jwks=excluded", + ); + // Storage's migration pin — same compound enabled+majorVersion gate as the JWKS line above, + // because the consuming one-shot job (`legacyStartInitSchema15`) is behind the same gate. + lines.push( + inputs.services.storage.enabled && inputs.majorVersion >= 15 + ? `storage_target_migration=${quoted(inputs.storageTargetMigration)}` + : "storage_target_migration=excluded", + ); + // ONLY resolved entries: `legacyUpsertVaultSecrets` (`legacy-vault.ts`) filters on + // `secret.resolved` before touching `vault.secrets`, so an unresolved entry never lands in + // the cluster and must not affect the key — hashing exactly what the upsert processes + // (review: Codex on #6184). + for (const secret of inputs.vault + .filter((secret) => secret.resolved) + .sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0))) { + // JSON-encoded tuple, not `name=value`: both halves are unrestricted strings, so a bare + // `=` join would let (`a=b`, `c`) and (`a`, `b=c`) collide — and a value containing a + // newline could forge a whole extra payload line (review: Codex on #6184). + lines.push(`vault=${JSON.stringify([secret.name, secret.value])}`); + } + // Last, raw (it can contain anything, including newlines) — mirroring `setupInputsToken`, + // which also appends `roles.sql` verbatim at the end of its own payload. + const payload = `${lines.join("\n")}\nroles_sql=\n${inputs.rolesSql}`; + return createHash("sha256").update(payload, "utf8").digest("hex").slice(0, 16); +} + +/** + * Same resolution `legacySetupDatabase` applies (`db-setup.ts`): `"enabled"` always + * installs `pg_net`, `"disabled"` always removes it, `"config"` (the default) follows + * `setup.webhooksEnabled`. + */ +export function legacyEffectiveShadowWebhooksEnabled( + policy: LegacySetupDatabaseOptions["webhooks"], + webhooksEnabled: boolean, +): boolean { + const webhooks = policy ?? "config"; + return webhooks === "enabled" || (webhooks === "config" && webhooksEnabled); +} + +/** + * Resolves {@link LegacyShadowCacheKeyInputs} from the same run input the shadow container + * itself is built from, plus `supabase/roles.sql` off disk. The service enabled flags come from + * `setup.config` (NOT the `*EnabledForSetup` fields, which only gate JWKS resolution) because + * `legacySetupDatabase`'s own one-shot job gates read exactly those config fields. + * + * Returns `Option.none` (never a failure) for the two conditions that make caching + * unavailable — an OrioleDB cluster (whose state is partly external, see the body's own + * comment) and an unreadable `roles.sql` — so this function's OWN error channel carries + * nothing but `E`: the `input.setup.jwks` effect below is `yield*`ed unguarded (only when the + * consuming realtime job is actually reachable — see the field's own doc comment), and a real + * JWKS failure must fail this whole acquire exactly as it would have failed a real setup, never + * be folded into a cache-miss the way {@link LegacyShadowCacheUnavailable} failures are + * elsewhere in this module. Keeping the two failure modes on separate channels (`Option.none` + * vs. a genuine `Effect` failure) is what lets the caller (`legacyAcquireShadowDatabase`) + * degrade the former to an uncached shadow while letting the latter propagate, without an `as` + * cast to tell them apart. + */ + +const legacyResolveShadowCacheKeyInputs = ( + input: LegacyShadowSetupInput, + opts: LegacyShadowCacheOpts = {}, +): Effect.Effect, E> => + Effect.gen(function* () { + // OrioleDB (`experimental.orioledb_version`) makes the WHOLE cache ineligible, not just a + // key input: that branch runs the shadow with an external S3 storage backend + // (`S3_ENABLED=true` — `legacyPostgresExtraEnv`, `postgres.service.ts`), so part of the + // cluster's state lives outside PGDATA and a disk-level tar is not a coherent snapshot to + // begin with (review: Codex on #6184). Same `Option.none` degradation as an unreadable + // `roles.sql` below: the run proceeds uncached. + const orioledbVersion = input.experimental.orioledb_version; + if (orioledbVersion !== undefined && orioledbVersion.length > 0) return Option.none(); + + // PG <= 14 is cache-ineligible too: its setup path executes the bundled globals SQL + // (`LEGACY_START_DB_GLOBALS_SQL`, `db-setup.ts`'s pre-15 branch), whose `ALTER ROLE … SET` + // statements (statement_timeout, `postgres`'s search_path) only take effect on NEW sessions. + // Go/uncached runs apply migrations on the SAME session that ran the setup — before those + // defaults exist — while any snapshot boundary forces a reconnect that picks them up, so + // unqualified names in user migrations could resolve into different schemas and change the + // diff (review: Codex on #6184). PG15+ moves that SQL into the entrypoint's initdb heredoc, + // which runs before any CLI session, so no session ever observes a mid-run change there. + if (input.setup.majorVersion <= 14) return Option.none(); + + const rolesPath = input.path.join(input.workdir, "supabase", "roles.sql"); + const rolesSql = yield* input.fs + .readFileString(rolesPath) + .pipe( + Effect.catchTag("PlatformError", (error) => + error.reason._tag === "NotFound" ? Effect.succeed("") : Effect.succeed(undefined), + ), + ); + if (rolesSql === undefined) return Option.none(); + + const overrides = input.setup.serviceVersionOverrides; + // The same registry rewrite `legacyRunStartMigrateJob`'s `legacyEnsureImagesCached` applies + // when the job actually runs (same `projectEnvValues`-then-ambient precedence) — see + // {@link LegacyShadowCacheServiceInput.image}. + const resolveJobImage = (image: string): string => + legacyGetRegistryImageUrl(image, input.setup.projectEnvValues); + // The exact compound gate {@link legacyResolveDbSetupPrelude} uses to decide whether the + // realtime one-shot job's JWKS effect is reached (and therefore run) at all during a real + // setup — see `db-setup.ts`. Gating on anything looser here would resolve (and risk failing + // on) an effect a real cold provision at this same `majorVersion`/`enabled` combination would + // never have touched. + const realtimeConsumesJwks = + input.setup.majorVersion >= 15 && input.setup.config.realtime.enabled; + const jwks = realtimeConsumesJwks ? yield* input.setup.jwks : ""; + return Option.some({ + postgresImage: input.image, + majorVersion: input.db.major_version, + jwtSecret: input.jwtSecret, + jwtExpiry: input.jwtExpiry, + // The EFFECTIVE value, not the raw input: `legacyBuildShadowPostgresContainerSpec` + // (`postgres.service.ts`) falls back to the embedded default when unset, so hashing `""` + // would fail to re-key if that security-sensitive default ever rotates between CLI + // releases (review: depthfirst on #6184). + rootKey: input.rootKey ?? LEGACY_POSTGRES_DEFAULT_ROOT_KEY, + dbPassword: input.password, + dbSettings: input.db.settings, + storageTargetMigration: input.setup.storageTargetMigration, + autoExposeNewTables: input.setup.apiAutoExposeNewTables, + webhooksEnabled: legacyEffectiveShadowWebhooksEnabled( + opts.webhooks, + input.setup.webhooksEnabled, + ), + rolesSql, + vault: input.setup.vault, + jwks, + services: { + realtime: { + enabled: input.setup.config.realtime.enabled, + image: resolveJobImage(legacyResolvePinnedImage("realtime", "realtime", overrides)), + }, + storage: { + enabled: input.setup.config.storage.enabled, + image: resolveJobImage(legacyResolvePinnedImage("storage", "storage", overrides)), + }, + auth: { + enabled: input.setup.config.auth.enabled, + image: resolveJobImage(legacyResolvePinnedImage("gotrue", "auth", overrides)), + }, + }, + } satisfies LegacyShadowCacheKeyInputs); + }); + +// --------------------------------------------------------------------------- +// The tar artifact +// --------------------------------------------------------------------------- + +/** Filename prefix shared by every key's snapshot — the handle the retention sweep enumerates by. */ +const LEGACY_SHADOW_BASELINE_TAR_PREFIX = "shadow-baseline-"; + +const LEGACY_SHADOW_BASELINE_TAR_SUFFIX = ".tar"; + +/** Cap on published tars in the global cache (~90MB each → ~720MB). */ +export const LEGACY_SHADOW_BASELINE_KEEP = 8; + +/** Drop published tars whose mtime is older than this (warm hits refresh mtime). */ +export const LEGACY_SHADOW_BASELINE_MAX_AGE_MS = 14 * 24 * 60 * 60 * 1000; + +/** + * `shadow-baseline-.tar` under `${SUPABASE_HOME}/cache/shadow-baseline/` — one ~90MB file + * per settings key, shared across worktrees. + */ +export function legacyShadowBaselineTarFileName(key: string): string { + return `${LEGACY_SHADOW_BASELINE_TAR_PREFIX}${key}${LEGACY_SHADOW_BASELINE_TAR_SUFFIX}`; +} + +/** + * Whether `fileName` is a published baseline snapshot (`shadow-baseline-.tar`). Pure and + * deliberately conservative: only this module's own prefix AND suffix, so partials + * (`…tar..partial`) and any unrelated file in the cache dir are never eviction candidates. + */ +export function legacyIsShadowBaselineTar(fileName: string): boolean { + return ( + fileName.startsWith(LEGACY_SHADOW_BASELINE_TAR_PREFIX) && + fileName.endsWith(LEGACY_SHADOW_BASELINE_TAR_SUFFIX) && + fileName.length === + LEGACY_SHADOW_BASELINE_TAR_PREFIX.length + 16 + LEGACY_SHADOW_BASELINE_TAR_SUFFIX.length && + /^shadow-baseline-[0-9a-f]{16}\.tar$/u.test(fileName) + ); +} + +/** One published tar as the LRU/TTL rule sees it — name + mtime, no filesystem. */ +export interface LegacyShadowBaselineTarEntry { + readonly fileName: string; + readonly mtimeMs: number; +} + +export interface LegacyShadowBaselineRetentionOpts { + readonly keep?: number; + readonly maxAgeMs?: number; +} + +/** + * Pure LRU + age eviction: drop every published tar older than `maxAgeMs`, then among the + * survivors keep the newest `keep` by mtime. Never returns non-tar names (catalogs, partials). + * Unit-testable without a filesystem. + */ +export function legacyShadowBaselineTarsToEvict( + entries: ReadonlyArray, + now: number, + opts: LegacyShadowBaselineRetentionOpts = {}, +): ReadonlyArray { + const keep = opts.keep ?? LEGACY_SHADOW_BASELINE_KEEP; + const maxAgeMs = opts.maxAgeMs ?? LEGACY_SHADOW_BASELINE_MAX_AGE_MS; + const candidates = entries.filter((entry) => legacyIsShadowBaselineTar(entry.fileName)); + const aged = new Set( + candidates.filter((entry) => now - entry.mtimeMs > maxAgeMs).map((entry) => entry.fileName), + ); + const newestFirst = candidates + .filter((entry) => !aged.has(entry.fileName)) + .sort((left, right) => right.mtimeMs - left.mtimeMs); + const overCap = newestFirst.slice(keep).map((entry) => entry.fileName); + return [...aged, ...overCap]; +} + +/** Best-effort removal — a leftover tar only ever costs disk, never correctness. */ +const legacyForgetShadowBaselineTar = ( + fs: FileSystem.FileSystem, + filePath: string, +): Effect.Effect => fs.remove(filePath).pipe(Effect.orElseSucceed(() => undefined)); + +/** + * Whether `fileName` is one of {@link legacyExportPgDataTar}'s in-flight temp files + * (`shadow-baseline-.tar..partial`). Pure name check only — whether it is ABANDONED + * (crashed/SIGKILLed writer, whose `Effect.onError` cleanup never ran) is an mtime question the + * sweep answers separately, so a concurrent writer's live temp file is never a candidate by name + * alone. + */ +export function legacyIsShadowBaselinePartial(fileName: string): boolean { + return /^shadow-baseline-[0-9a-f]{16}\.tar\.\d+\.partial$/u.test(fileName); +} + +/** + * A partial older than this is abandoned: the export itself streams ~90MB in seconds, so an + * hour-old temp file's writer is long gone. Deliberately enormous relative to a real export so a + * slow disk can never get a LIVE temp file swept out from under its writer. + */ +const LEGACY_SHADOW_PARTIAL_ABANDON_MS = 60 * 60 * 1000; + +/** + * Removes abandoned `.partial` temp files (see {@link legacyIsShadowBaselinePartial}) — the one + * artifact a SIGKILLed/crashed cold export leaves behind that nothing else ever cleans: later + * runs use their own pid in the temp name, and the tar retention sweep deliberately ignores + * `.partial` names (review: Codex on #6184). Runs before every cold export and on warm hits (so + * orphans cannot accumulate once every later run goes warm) — best-effort throughout. + */ +const legacySweepAbandonedShadowBaselinePartials = ( + input: LegacyShadowSetupInput, +): Effect.Effect => + Effect.gen(function* () { + const cacheDir = legacyShadowBaselineCacheDir(input.path); + const entries = yield* input.fs + .readDirectory(cacheDir) + .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); + const now = yield* Clock.currentTimeMillis; + yield* Effect.forEach( + entries.filter(legacyIsShadowBaselinePartial), + (entry) => + Effect.gen(function* () { + const filePath = input.path.join(cacheDir, entry); + const info = yield* input.fs.stat(filePath); + const mtime = Option.getOrUndefined(info.mtime); + if (mtime !== undefined && now - mtime.getTime() > LEGACY_SHADOW_PARTIAL_ABANDON_MS) { + yield* legacyForgetShadowBaselineTar(input.fs, filePath); + } + }).pipe(Effect.orElseSucceed(() => undefined)), + { discard: true }, + ); + }); + +/** + * Applies the global-cache LRU + TTL retention rule (see {@link legacyShadowBaselineTarsToEvict}). + * Best-effort throughout — a snapshot that cannot be swept costs ~90MB of disk, so it must never + * fail the export or warm hit that just succeeded. + */ +const legacySweepShadowBaselineRetention = ( + input: LegacyShadowSetupInput, +): Effect.Effect => + Effect.gen(function* () { + const cacheDir = legacyShadowBaselineCacheDir(input.path); + const names = yield* input.fs + .readDirectory(cacheDir) + .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); + const now = yield* Clock.currentTimeMillis; + const entries: Array = []; + for (const fileName of names) { + if (!legacyIsShadowBaselineTar(fileName)) continue; + const info = yield* input.fs + .stat(input.path.join(cacheDir, fileName)) + .pipe(Effect.orElseSucceed(() => undefined)); + if (info === undefined) continue; + const mtime = Option.getOrUndefined(info.mtime); + if (mtime === undefined) continue; + entries.push({ fileName, mtimeMs: mtime.getTime() }); + } + yield* Effect.forEach( + legacyShadowBaselineTarsToEvict(entries, now), + (fileName) => legacyForgetShadowBaselineTar(input.fs, input.path.join(cacheDir, fileName)), + { discard: true }, + ); + }); + +/** Refresh mtime on a warm hit so frequently used keys survive LRU/TTL. Best-effort. */ +const legacyTouchShadowBaselineTar = ( + fs: FileSystem.FileSystem, + tarPath: string, +): Effect.Effect => + Effect.gen(function* () { + const now = new Date(yield* Clock.currentTimeMillis); + yield* fs.utimes(tarPath, now, now); + }).pipe(Effect.orElseSucceed(() => undefined)); + +// --------------------------------------------------------------------------- +// Container primitives the cache adds on top of `shadow-database.ts` +// --------------------------------------------------------------------------- + +/** + * `docker `, resolving to {@link LegacyShadowCacheUnavailable} on anything but a clean + * exit. Both verbs this is used for (`stop`, `start`) are steps the export cannot proceed without, + * so a non-zero exit is an anomaly rather than something to tolerate. + */ +const legacyShadowContainerVerb = ( + spawner: Spawner, + verb: "start" | "stop", + containerId: string, +): Effect.Effect => + containerCliExitCode(spawner, [verb, containerId], { + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + }).pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable( + `failed to ${verb} shadow container: ${legacyDescribeContainerCliFailure(cause)}`, + ), + ), + Effect.flatMap((exitCode) => + exitCode === 0 + ? Effect.void + : Effect.fail(legacyShadowCacheUnavailable(`docker ${verb} exited ${exitCode}`)), + ), + ); + +/** The shadow's own connect target, shared by the readiness waits on both paths. */ +const legacyShadowConnConfig = (input: LegacyShadowSetupInput): LegacyPgConnInput => ({ + host: input.hostname, + port: input.shadowPort, + user: "postgres", + password: input.password, + database: "postgres", +}); + +/** + * `legacyWaitForShadowReady` self-instruments (`ready-attempt`/`ready-wait`, `health-check.ts`) + * whenever `SUPABASE_SHADOW_DEBUG` is on, so neither call site here needs an extra timing wrapper. + */ +const legacyAwaitShadowReady = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + containerId: string, + what: string, +): Effect.Effect => + legacyWaitForShadowReady(spawner, containerId, legacyShadowConnConfig(input), { + timeoutSeconds: input.healthTimeoutSeconds, + image: input.image, + }).pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable(`${what} never became ready: ${cause.message}`), + ), + ); + +// --------------------------------------------------------------------------- +// Cold export +// --------------------------------------------------------------------------- + +/** + * Ensures the tar's global cache directory exists, delegates the actual export to + * {@link legacyExportPgDataTar} (`pgdata-snapshot.ts` — see that function's own doc comment for + * the atomic-publish mechanics), then applies the LRU + TTL retention rule. + */ +const legacyWriteShadowBaselineTar = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + tarPath: string, + containerId: string, +): Effect.Effect => + Effect.gen(function* () { + const cacheDir = legacyShadowBaselineCacheDir(input.path); + yield* input.fs + .makeDirectory(cacheDir, { recursive: true, mode: 0o700 }) + .pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable(`failed to create ${cacheDir}: ${cause.message}`), + ), + ); + yield* legacySweepAbandonedShadowBaselinePartials(input); + yield* legacyExportPgDataTar(spawner, containerId, input.fs, tarPath).pipe( + Effect.mapError((cause: LegacyPgDataSnapshotUnavailable) => + legacyShadowCacheUnavailable(cause.reason), + ), + ); + yield* legacySweepShadowBaselineRetention(input); + }); + +/** + * The cold path's snapshot step, run at the baseline/migrations seam — after + * `legacySetupDatabase` and strictly before `contrib_regression` or any user migration, with no + * session open against the shadow ({@link LegacyShadowBaselineState.snapshotBaseline}). + * + * `docker stop` -> export -> `docker start` -> readiness wait. The container is stopped because a + * live Postgres's PGDATA is not a coherent thing to copy; the stop is fast (~1s) because the + * entrypoint `exec`s Postgres, so PID 1 receives the SIGTERM instead of `sh` swallowing it and + * burning the full 10s grace period. + * + * Two failure classes, deliberately NOT one broad catch: a stop/export failure only means this + * run stays uncached — it warns and the run continues. A restart/readiness failure is the RUN'S + * problem: the caller is about to reconnect to the shadow's published port, and reporting success + * over a dead container would make that connect a blind dial — which can even reach a DIFFERENT + * Postgres that claimed the port while the container was down (matching default credentials are + * common locally), applying the template + migrations to the wrong database. So the restart runs + * whether the export succeeded or not (`docker start` on an already-running container — e.g. when + * the stop itself failed — is a no-op success), and its failure PROPAGATES as a + * {@link LegacyShadowDbError} instead of degrading (review: Codex on #6184). + */ +const legacyExportShadowBaseline = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + key: string, + tarPath: string, + containerId: string, +): Effect.Effect => + legacyTimeShadowPhase( + "baseline-export", + Effect.gen(function* () { + // Cache-degradable phase: stop + export. A failure here (including a failed stop, after + // which the container is simply still up) only costs this run its snapshot. + const exported = yield* Effect.result( + Effect.gen(function* () { + yield* legacyShadowContainerVerb(spawner, "stop", containerId); + // The stamp is what makes the published tar mean "the baseline THIS key promises" rather + // than "some PostgreSQL cluster". Two things give it that meaning. Its POSITION in the + // sequence: this whole step runs from `snapshotBaseline`, which `legacySetupShadowDatabase` + // invokes strictly after `legacySetupDatabase` returns (`shadow-database.ts`), and the + // stamp is the last mutation before the copy-out — so a future regression that snapshots + // EARLIER cannot produce a marked tar, it just stays uncached instead of silently + // publishing a bare cluster under a baseline key. And its CONTENT: `key` itself, which + // `legacyWarmShadow` compares against the key it resolved this run, so a valid snapshot + // of a DIFFERENT key that was copied over this filename is rejected too (review: Codex + // on #6184). + yield* legacyStampPgDataBaselineMarker(spawner, containerId, key).pipe( + Effect.mapError((cause: LegacyPgDataSnapshotUnavailable) => + legacyShadowCacheUnavailable(cause.reason), + ), + ); + yield* legacyWriteShadowBaselineTar(spawner, input, tarPath, containerId); + }), + ); + // Run-critical phase: the shadow must be back up and answering before this step reports + // success — see this function's own doc comment for why these failures must propagate. + const revive = Effect.gen(function* () { + yield* legacyShadowContainerVerb(spawner, "start", containerId); + yield* legacyAwaitShadowReady(spawner, input, containerId, "re-started shadow"); + }); + yield* revive.pipe( + Effect.mapError( + (cause) => + new LegacyShadowDbError({ + message: `shadow database did not come back after the baseline snapshot: ${cause.reason}`, + reason: "docker_daemon", + }), + ), + ); + if (Result.isFailure(exported)) { + const output = yield* Output; + yield* output.raw( + `Warning: shadow baseline not cached: ${exported.failure.reason}\n`, + "stderr", + ); + } + }), + ); + +// --------------------------------------------------------------------------- +// Acquire / release +// --------------------------------------------------------------------------- + +/** + * Per-invocation cache controls a CALLER passes (as opposed to the user-level + * {@link LEGACY_SHADOW_CACHE_ENV} env gate). `bypassCache` is `db schema declarative sync + * --no-cache`'s hook: that flag documents "force fresh shadow database setup", so a run carrying + * it must neither restore an existing snapshot nor publish a new one — exactly the uncached + * lifecycle, regardless of the env gate (review: Codex on #6184). + * + * `webhooks` is the same policy the caller will pass to `legacySetupDatabase` / + * `legacySetupShadowDatabase` / `legacyMigrate*ShadowDatabase`. Defaults to `"config"` + * (follow `setup.webhooksEnabled`), matching {@link LegacySetupDatabaseOptions}. Hashed as + * the effective boolean so a forced-on legacy migrate snapshot cannot warm-restore into a + * next path that follows config, or a declarative shadow that forces webhooks off. + */ +export interface LegacyShadowCacheOpts { + readonly bypassCache?: boolean; + readonly webhooks?: LegacySetupDatabaseOptions["webhooks"]; +} + +/** + * What `Effect.acquireUseRelease`'s `acquire` hands the `use` phase: the container, whether its + * cluster already carries the platform baseline, and the snapshot step to run once a fresh + * baseline is in place. Release needs nothing extra — every shadow this module hands out is + * removed the same way an uncached one is. + */ +export interface LegacyShadowAcquiredHandle extends LegacyShadowBaselineState { + readonly containerId: string; + /** + * The resolved shadow-baseline cache key this handle's cluster is keyed under — present + * exactly when the acquisition was cache-eligible (a cold export or a warm restore), absent + * for an uncached, `bypassCache`d, or uncachable one. Two handles carrying the SAME key share + * the same tar's lineage: one either restored it or exported it this run, so their clusters + * are physical clones of each other. `legacy-pgdelta-next-shadow.layer.ts` reads it to decide + * whether pg-delta's same-database-identity guard must be bypassed for a plan's two shadows. + */ + readonly snapshotKey?: string; +} + +/** A throwaway shadow with no snapshot step — the cache-off path. */ +const legacyUncachedShadow = ( + spawner: Spawner, + input: LegacyShadowSetupInput, +): Effect.Effect => + legacyCreateShadowDatabase(spawner, input).pipe( + Effect.map(({ containerId }) => ({ + containerId, + baselinePresent: false, + snapshotRequired: false, + snapshotBaseline: Effect.void, + })), + ); + +/** + * A cold, cache-enabled shadow: today's container plus the export step at the baseline seam. + * + * `autoRemove: false` is the one and only container-shape difference the cache introduces, and it + * is forced: the export has to `docker stop` the container and `docker start` it again, and Docker + * destroys an `--rm` container the moment it exits. Release still removes it with `docker rm -f + * -v`, so the container's lifetime is unchanged — see + * {@link LegacyCreateShadowDatabaseInput.autoRemove}. + */ +const legacyColdCachedShadow = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + key: string, + tarPath: string, +): Effect.Effect => + legacyCreateShadowDatabase(spawner, { ...input, autoRemove: false }).pipe( + Effect.map(({ containerId }) => ({ + containerId, + snapshotKey: key, + baselinePresent: false, + snapshotRequired: true, + snapshotBaseline: legacyExportShadowBaseline(spawner, input, key, tarPath, containerId), + })), + ); + +/** A cache key as {@link legacyShadowCacheKey} produces it — 16 hex chars, nothing else. */ +const LEGACY_SHADOW_CACHE_KEY_PATTERN = /^[0-9a-f]{16}$/u; + +/** + * The warm-path warning's wording for a rejected snapshot. Names WHICH of the two content failures + * happened, because they mean different things to whoever reads the line: a missing entry is a + * broken or hand-placed artifact, while a wrong key is a real snapshot of another configuration + * sitting under this one's filename (a copied cache directory, a renamed file) — and only the + * misnamed copy is being discarded, not that other key's own tar. + * + * The marker's own bytes are NOT echoed verbatim: they come from a file this run did not write, + * capped at a KiB but otherwise arbitrary, and stderr is not the place to render them. Only a token + * that is shaped like a cache key is shown. + */ +const legacyDescribeShadowArchiveProblem = (problem: LegacyPgDataArchiveProblem): string => { + if (problem._tag === "missing-entries") { + return `snapshot has no ${problem.entries.join(" or ")} entry`; + } + const found = + problem.found !== undefined && LEGACY_SHADOW_CACHE_KEY_PATTERN.test(problem.found) + ? `key ${problem.found}` + : "an unreadable key"; + return `snapshot is stamped with ${found}, not ${problem.expected}`; +}; + +/** + * The warm path proper: verify the snapshot tar really carries a baselined cluster, create the + * shadow with it unpacked into it before it starts ({@link LegacyCreateShadowDatabaseInput.restoreArchive}), + * then wait for the restored Postgres. Every failure resolves to + * {@link LegacyShadowCacheUnavailable}, which the caller turns into the escape hatch. + * + * The readiness failure removes the container here rather than leaving it to the caller, because + * the caller's fallback creates a REPLACEMENT container and the suspect one must be gone by then + * (it holds the shadow's published port). + * + * The readiness wait is explicitly `Effect.interruptible`: this whole function runs inside + * `Effect.acquireUseRelease`'s uninterruptible `acquire` ({@link legacyWithShadowDatabase}), and + * while the restore itself is short (~2s of `docker create` + `docker cp -`), a restored container + * that starts but never accepts connections would otherwise pin a Ctrl-C for the full + * `healthTimeoutSeconds` budget — exactly the swallowed-SIGINT shape `legacyPrepareShadowSource`'s + * own doc comment (`legacy-shadow-source.ts`) was restructured to avoid. `Effect.onInterrupt` + * removes the container on that path, so re-enabling interruption cannot leak it (review: Codex + * on #6184). + */ +const legacyWarmShadow = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + key: string, + tarPath: string, +): Effect.Effect< + LegacyShadowAcquiredHandle, + LegacyShadowCacheUnavailable, + Output | LegacyDbConnection +> => + Effect.gen(function* () { + // An archive that unpacks cleanly but carries the wrong thing is the ONE corruption the restore + // itself cannot report: `docker cp -` extracts whatever it is given, the entrypoint skips + // `initdb` (or runs one over an empty PGDATA), readiness passes, and this function would hand + // back `baselinePresent: true` for a cluster that never saw `legacySetupDatabase` — the caller + // then skips it too and diffs against a BARE database, silently producing wrong SQL. The same + // shape hides a second lie: a fully baselined snapshot of ANOTHER key, copied over this key's + // filename, restores just as cleanly while carrying different roles/vault values/service + // schema. So the tar's own headers are scanned BEFORE anything is created (locally, no Docker) + // for the cluster file, for the baseline marker this module stamps immediately before every + // export, AND for that marker's key — see {@link legacyValidatePgDataArchive}. A read failure + // is infra and leaves the tar in place; either verdict implicates its CONTENTS (review: Codex + // on #6184). + const problem = yield* legacyValidatePgDataArchive(input.fs, tarPath, key).pipe( + Effect.mapError((cause) => legacyShadowCacheUnavailable(cause.reason)), + ); + if (Option.isSome(problem)) { + return yield* Effect.fail( + legacyShadowCacheUnavailable(legacyDescribeShadowArchiveProblem(problem.value), { + tarSuspect: true, + }), + ); + } + const { containerId } = yield* legacyTimeShadowPhase( + "baseline-restore", + legacyCreateShadowDatabase(spawner, { + ...input, + restoreArchive: legacyPgDataRestoreArchive(input.fs, tarPath), + }), + ).pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable(`failed to restore shadow baseline: ${cause.message}`), + ), + ); + yield* legacyAwaitShadowReady(spawner, input, containerId, "restored shadow").pipe( + // The ONE failure that implicates the tar's contents: the restored cluster started but + // never accepted connections — see {@link LegacyShadowCacheUnavailable.tarSuspect}. + Effect.mapError((cause) => legacyShadowCacheUnavailable(cause.reason, { tarSuspect: true })), + Effect.tapError(() => legacyRemoveShadowDatabase(spawner, containerId)), + Effect.onInterrupt(() => legacyRemoveShadowDatabase(spawner, containerId)), + Effect.interruptible, + ); + return { + containerId, + snapshotKey: key, + baselinePresent: true, + snapshotRequired: false, + snapshotBaseline: Effect.void, + } satisfies LegacyShadowAcquiredHandle; + }); + +/** + * `Effect.acquireUseRelease`'s `acquire` for every shadow-provisioning call site that runs the + * platform baseline (`db diff`'s migra/pg-delta branch, `db pull`'s migration diff, + * `legacy-pgdelta.cache.ts`'s catalog export, and pg-delta next's scoped shadows) — see + * {@link legacyWithShadowDatabase} for the acquire/use/release wrapper, and + * `legacy-pgdelta-next-shadow.layer.ts` for the scoped `acquireRelease` form next uses so the + * container outlives provision (the engine keeps using the URL after this returns). + * + * With {@link LEGACY_SHADOW_CACHE_ENV} set to a falsey value this IS `legacyCreateShadowDatabase`, + * byte for byte. Otherwise it resolves the cache key and either restores this key's snapshot into + * a fresh container (warm) or creates one and arranges for its baseline to be exported (cold). + * Either way the container itself is created identically to an uncached one. + * + * Runs inside `acquireUseRelease`'s uninterruptible `acquire`, same as + * `legacyCreateShadowDatabase` — a warm restore adds ~2s of uninterruptible work (a `docker cp -`); + * its readiness wait re-enables interruption explicitly (see {@link legacyWarmShadow}), and the + * multi-second baseline/migration sequence stays in the interruptible `use` phase exactly as + * before. + * + * The `E` in the error channel is {@link legacyResolveShadowCacheKeyInputs}'s own JWKS + * resolution alone (see that function's doc comment): every OTHER failure this function's own + * body can produce while computing the key or restoring the snapshot is caught and degraded to a + * cold provision — a genuine JWKS failure is the one case that must reach the caller instead, + * since a real cold provision at this input would have failed the same way. + */ +export const legacyAcquireShadowDatabase = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + opts: LegacyShadowCacheOpts = {}, +): Effect.Effect< + LegacyShadowAcquiredHandle, + LegacyShadowDbError | E, + Output | LegacyDbConnection +> => + Effect.gen(function* () { + if ( + opts.bypassCache === true || + !legacyShadowCacheEnabled(process.env, input.setup.projectEnvValues) + ) { + return yield* legacyUncachedShadow(spawner, input); + } + + // Interruptible: this runs inside `acquireUseRelease`'s uninterruptible `acquire`, but + // nothing has been acquired yet — and the JWKS effect inside can be a real third-party + // discovery request, which must not pin a Ctrl-C for its whole duration (review: Codex on + // #6184). Interruption here simply means no container was ever created, so there is nothing + // for a finalizer to release. + const keyInputs = yield* Effect.interruptible(legacyResolveShadowCacheKeyInputs(input, opts)); + if (Option.isNone(keyInputs)) return yield* legacyUncachedShadow(spawner, input); + const key = legacyShadowCacheKey(keyInputs.value); + const tarPath = input.path.join( + legacyShadowBaselineCacheDir(input.path), + legacyShadowBaselineTarFileName(key), + ); + + const cached = yield* input.fs.exists(tarPath).pipe(Effect.orElseSucceed(() => false)); + if (!cached) return yield* legacyColdCachedShadow(spawner, input, key, tarPath); + + // Warm hits refresh mtime (so frequently used keys survive LRU/TTL) and sweep abandoned + // partials — a killed concurrent writer's leftover would otherwise persist indefinitely once + // every later run goes warm, since the cold export's own sweep never runs again (review: + // Codex on #6184). Best-effort and cheap. + yield* legacyTouchShadowBaselineTar(input.fs, tarPath); + yield* legacySweepAbandonedShadowBaselinePartials(input); + yield* legacySweepShadowBaselineRetention(input); + + return yield* legacyWarmShadow(spawner, input, key, tarPath).pipe( + Effect.catch((cause) => + Effect.gen(function* () { + const output = yield* Output; + yield* output.raw( + `Warning: cached shadow baseline unusable (${cause.reason}); recreating.\n`, + "stderr", + ); + // Delete ONLY when the failure implicates the tar's contents (a restored cluster that + // came up broken) — see {@link LegacyShadowCacheUnavailable.tarSuspect} for why an + // infra or extraction failure leaves it in place (the cold fallback's own export + // republishes over a genuinely bad tar anyway). + if (cause.tarSuspect === true) { + yield* legacyForgetShadowBaselineTar(input.fs, tarPath); + } + return yield* legacyColdCachedShadow(spawner, input, key, tarPath); + }), + ), + ); + }); + +/** + * The acquire/use/release triple every shadow-provisioning call site that runs the platform + * baseline uses (`db diff`'s migra/pg-delta branch, `db pull`'s migration diff, + * `legacy-pgdelta.cache.ts`'s catalog export). + * + * `Effect.acquireUseRelease`, NOT a `yield* acquire` followed by a later + * `.pipe(Effect.ensuring(release))`: the latter leaves a real gap between the shadow's successful + * creation and the finalizer actually being attached — a fiber interrupt landing between those two + * statements would skip the release entirely, leaking the live container and leaving the shadow + * port occupied. `acquireUseRelease` closes that: `acquire` runs inside an `uninterruptibleMask` + * and the release finalizer is registered in the SAME uninterruptible continuation `acquire` + * resolves into, matching Go's `defer DockerRemove` immediately after successful creation + * (review: PRRT_kwDOErm0O86XDr4Y). It does NOT make removal unconditional — see + * `legacyCreateShadowDatabase`'s own doc comment (`shadow-database.ts`) for the still-present, + * deliberate-Go-parity leak window when `acquire` itself fails partway through (a `docker create` + * success followed by a `docker cp`/`docker start` failure). + * + * `acquire` is ONLY the container acquisition — NOT the health-wait/migrate/declarative-apply + * `legacyPrepareShadowSource` performs. Those belong in `use`, where a SIGINT can still interrupt + * them (matching Go's single cancellable `ctx` threaded through the equivalent calls); passing all + * of `legacyPrepareShadowSource` as `acquire` made that whole sequence uninterruptible too, since + * `acquireUseRelease`'s `uninterruptibleMask` has no `restore` around `acquire` — see + * `legacy-shadow-source.ts`'s own doc comment on `legacyPrepareShadowSource` for the full + * rationale (review: PRRT_kwDOErm0O86XMrID). + * + * `release` is `legacyRemoveShadowDatabase` unconditionally — the shadow baseline cache keeps a + * file, never a container, so there is nothing here for it to special-case. + * + * The handle `use` receives also carries the baseline state `legacyPrepareShadowSource` needs. + * + * `E` (also present in `use`'s own `E2`, since every real caller's `use` phase already resolves + * the same `input.setup.jwks` effect via `legacyMigrateShadowDatabase`/`legacyResolveDbSetupPrelude`) + * can now ALSO surface straight out of `acquire`: see {@link legacyAcquireShadowDatabase}'s own + * doc comment for why a JWKS failure discovered while computing the cache key must propagate + * rather than degrade to an uncached shadow. + */ +export const legacyWithShadowDatabase = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + use: (handle: LegacyShadowAcquiredHandle) => Effect.Effect, + opts: LegacyShadowCacheOpts = {}, +): Effect.Effect => + Effect.acquireUseRelease(legacyAcquireShadowDatabase(spawner, input, opts), use, (handle) => + legacyRemoveShadowDatabase(spawner, handle.containerId), + ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts new file mode 100644 index 0000000000..7ec071700a --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -0,0 +1,424 @@ +import { describe, expect, it } from "@effect/vitest"; +import { Option } from "effect"; + +import { + LEGACY_SHADOW_BASELINE_KEEP, + LEGACY_SHADOW_BASELINE_MAX_AGE_MS, + legacyEffectiveShadowWebhooksEnabled, + legacyIsShadowBaselinePartial, + legacyIsShadowBaselineTar, + legacyShadowBaselineTarFileName, + legacyShadowBaselineTarsToEvict, + legacyShadowCacheEnabled, + legacyShadowCacheKey, + type LegacyShadowCacheKeyInputs, +} from "./shadow-cache.ts"; + +const baseKeyInputs = (): LegacyShadowCacheKeyInputs => ({ + postgresImage: "public.ecr.aws/supabase/postgres:17.6.1.158", + majorVersion: 17, + jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", + jwtExpiry: 3600, + rootKey: "d4dc5b6d4a1d6a10b2c1e5b6a7c8d9e0", + dbPassword: "postgres", + dbSettings: { effective_cache_size: "128MB", max_connections: 100 }, + autoExposeNewTables: Option.none(), + storageTargetMigration: "20240101000000", + webhooksEnabled: true, + rolesSql: "create role custom_role;\n", + vault: [{ name: "secret", value: "value", resolved: true }], + jwks: '{"keys":[]}', + services: { + realtime: { enabled: true, image: "supabase/realtime:v2.34.47" }, + storage: { enabled: true, image: "supabase/storage-api:v1.25.7" }, + auth: { enabled: true, image: "supabase/gotrue:v2.177.0" }, + }, +}); + +describe("legacyShadowCacheEnabled", () => { + it("is ON unless the env var explicitly opts out", () => { + // Unset and empty both mean "the user never expressed a preference" — the cache is a + // default-on optimization, not a feature flag. + expect(legacyShadowCacheEnabled({})).toBe(true); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "" })).toBe(true); + // A value that IS set goes through the repo's `viper.GetBool` parser. + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "0" })).toBe(false); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "false" })).toBe(false); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "no" })).toBe(false); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "1" })).toBe(true); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "true" })).toBe(true); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "TRUE" })).toBe(true); + }); + + it("honors an opt-out set only in the project dotenv values", () => { + // `supabase/.env` is loaded into `projectEnvValues` (ambient-wins merge, upstream), not into + // `process.env` — the gate must consult it, same as the registry override does. + expect(legacyShadowCacheEnabled({}, { SUPABASE_SHADOW_CACHE: "false" })).toBe(false); + expect(legacyShadowCacheEnabled({}, { SUPABASE_SHADOW_CACHE: "1" })).toBe(true); + expect(legacyShadowCacheEnabled({}, {})).toBe(true); + // The record's own value is what counts once present — upstream merging already applied + // ambient-wins precedence when building it. + expect( + legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "1" }, { SUPABASE_SHADOW_CACHE: "0" }), + ).toBe(false); + }); +}); + +describe("legacyEffectiveShadowWebhooksEnabled", () => { + it("matches legacySetupDatabase: enabled/disabled override config, config follows the flag", () => { + expect(legacyEffectiveShadowWebhooksEnabled("enabled", false)).toBe(true); + expect(legacyEffectiveShadowWebhooksEnabled("enabled", true)).toBe(true); + expect(legacyEffectiveShadowWebhooksEnabled("disabled", true)).toBe(false); + expect(legacyEffectiveShadowWebhooksEnabled("disabled", false)).toBe(false); + expect(legacyEffectiveShadowWebhooksEnabled("config", true)).toBe(true); + expect(legacyEffectiveShadowWebhooksEnabled("config", false)).toBe(false); + expect(legacyEffectiveShadowWebhooksEnabled(undefined, true)).toBe(true); + expect(legacyEffectiveShadowWebhooksEnabled(undefined, false)).toBe(false); + }); +}); + +describe("legacyShadowCacheKey", () => { + it("is stable for identical inputs and independent of object key order", () => { + const first = legacyShadowCacheKey(baseKeyInputs()); + expect(legacyShadowCacheKey(baseKeyInputs())).toBe(first); + expect( + legacyShadowCacheKey({ + ...baseKeyInputs(), + dbSettings: { max_connections: 100, effective_cache_size: "128MB" }, + }), + ).toBe(first); + expect(first).toMatch(/^[0-9a-f]{16}$/u); + expect(legacyShadowBaselineTarFileName(first)).toBe(`shadow-baseline-${first}.tar`); + }); + + it("changes when ANY baked-in input changes", () => { + const base = baseKeyInputs(); + const mutations: ReadonlyArray<{ + readonly label: string; + readonly inputs: LegacyShadowCacheKeyInputs; + }> = [ + { label: "postgres image tag", inputs: { ...base, postgresImage: "postgres:17.6.1.159" } }, + { label: "major version", inputs: { ...base, majorVersion: 15 } }, + { label: "jwt secret", inputs: { ...base, jwtSecret: "other-secret" } }, + { label: "jwt expiry", inputs: { ...base, jwtExpiry: 7200 } }, + { label: "root key", inputs: { ...base, rootKey: "0000" } }, + { label: "db password", inputs: { ...base, dbPassword: "hunter2" } }, + { label: "db settings", inputs: { ...base, dbSettings: { max_connections: 200 } } }, + { + label: "auto expose new tables", + inputs: { ...base, autoExposeNewTables: Option.some(true) }, + }, + { label: "effective webhooks / pg_net", inputs: { ...base, webhooksEnabled: false } }, + { label: "roles.sql", inputs: { ...base, rolesSql: "" } }, + { + label: "storage migration pin (storage enabled, majorVersion >= 15)", + inputs: { ...base, storageTargetMigration: "20250607080910" }, + }, + { + label: "storage migration pin (pinned vs unpinned)", + inputs: { ...base, storageTargetMigration: "" }, + }, + { + label: "jwks (realtime enabled, majorVersion >= 15)", + inputs: { ...base, jwks: '{"keys":["rotated"]}' }, + }, + { + label: "vault secret name", + inputs: { ...base, vault: [{ name: "other", value: "value", resolved: true }] }, + }, + { + label: "vault secret value", + inputs: { ...base, vault: [{ name: "secret", value: "rotated", resolved: true }] }, + }, + { + label: "realtime image", + inputs: { + ...base, + services: { ...base.services, realtime: { enabled: true, image: "realtime:next" } }, + }, + }, + { + label: "storage image", + inputs: { + ...base, + services: { ...base.services, storage: { enabled: true, image: "storage:next" } }, + }, + }, + { + label: "auth image", + inputs: { + ...base, + services: { ...base.services, auth: { enabled: true, image: "gotrue:next" } }, + }, + }, + { + label: "realtime enabled flag", + inputs: { + ...base, + services: { + ...base.services, + realtime: { enabled: false, image: base.services.realtime.image }, + }, + }, + }, + { + label: "storage enabled flag", + inputs: { + ...base, + services: { + ...base.services, + storage: { enabled: false, image: base.services.storage.image }, + }, + }, + }, + { + label: "auth enabled flag", + inputs: { + ...base, + services: { ...base.services, auth: { enabled: false, image: base.services.auth.image } }, + }, + }, + ]; + const baseKey = legacyShadowCacheKey(base); + const seen = new Map([[baseKey, "base"]]); + for (const mutation of mutations) { + const key = legacyShadowCacheKey(mutation.inputs); + const collision = seen.get(key); + expect(collision, `${mutation.label} must change the cache key`).toBeUndefined(); + seen.set(key, mutation.label); + } + }); + + it("collapses auto_expose_new_tables to the behavior legacyApplyApiPrivileges actually takes", () => { + // `legacyApplyApiPrivileges` (`db-setup.ts`) returns early ONLY for an explicit `true`; unset + // and explicit `false` both exec the revoke SQL, so they bake the identical cluster and must + // share a snapshot instead of forcing a ~90MB re-export. + const base = baseKeyInputs(); + const unset = legacyShadowCacheKey({ ...base, autoExposeNewTables: Option.none() }); + const explicitFalse = legacyShadowCacheKey({ + ...base, + autoExposeNewTables: Option.some(false), + }); + const explicitTrue = legacyShadowCacheKey({ ...base, autoExposeNewTables: Option.some(true) }); + expect(explicitFalse).toBe(unset); + expect(explicitTrue).not.toBe(unset); + }); + + it("excludes a disabled service's image tag entirely", () => { + const base = baseKeyInputs(); + const withRealtimeA: LegacyShadowCacheKeyInputs = { + ...base, + services: { ...base.services, realtime: { enabled: false, image: "supabase/realtime:v1" } }, + }; + const withRealtimeB: LegacyShadowCacheKeyInputs = { + ...base, + services: { ...base.services, realtime: { enabled: false, image: "supabase/realtime:v2" } }, + }; + expect(legacyShadowCacheKey(withRealtimeA)).toBe(legacyShadowCacheKey(withRealtimeB)); + }); + + it("excludes the resolved jwks when realtime is disabled", () => { + const base = baseKeyInputs(); + const disabledRealtime: LegacyShadowCacheKeyInputs = { + ...base, + services: { + ...base.services, + realtime: { ...base.services.realtime, enabled: false }, + }, + }; + const withJwksA: LegacyShadowCacheKeyInputs = { ...disabledRealtime, jwks: '{"keys":["a"]}' }; + const withJwksB: LegacyShadowCacheKeyInputs = { ...disabledRealtime, jwks: '{"keys":["b"]}' }; + expect(legacyShadowCacheKey(withJwksA)).toBe(legacyShadowCacheKey(withJwksB)); + }); + + it("excludes the resolved jwks when majorVersion is below 15, even with realtime enabled", () => { + const base = baseKeyInputs(); + const pre15: LegacyShadowCacheKeyInputs = { ...base, majorVersion: 14 }; + const withJwksA: LegacyShadowCacheKeyInputs = { ...pre15, jwks: '{"keys":["a"]}' }; + const withJwksB: LegacyShadowCacheKeyInputs = { ...pre15, jwks: '{"keys":["b"]}' }; + expect(legacyShadowCacheKey(withJwksA)).toBe(legacyShadowCacheKey(withJwksB)); + }); + + it("excludes the storage migration pin when storage is disabled", () => { + const base = baseKeyInputs(); + const disabledStorage: LegacyShadowCacheKeyInputs = { + ...base, + services: { + ...base.services, + storage: { ...base.services.storage, enabled: false }, + }, + }; + const withPinA: LegacyShadowCacheKeyInputs = { + ...disabledStorage, + storageTargetMigration: "20240101000000", + }; + const withPinB: LegacyShadowCacheKeyInputs = { + ...disabledStorage, + storageTargetMigration: "20250607080910", + }; + expect(legacyShadowCacheKey(withPinA)).toBe(legacyShadowCacheKey(withPinB)); + }); + + it("excludes the storage migration pin when majorVersion is below 15, even with storage enabled", () => { + const base = baseKeyInputs(); + const pre15: LegacyShadowCacheKeyInputs = { ...base, majorVersion: 14 }; + const withPinA: LegacyShadowCacheKeyInputs = { ...pre15, storageTargetMigration: "a" }; + const withPinB: LegacyShadowCacheKeyInputs = { ...pre15, storageTargetMigration: "b" }; + expect(legacyShadowCacheKey(withPinA)).toBe(legacyShadowCacheKey(withPinB)); + }); + + it("cannot collide scalar fields across line boundaries", () => { + const base = baseKeyInputs(); + // A newline embedded in one unrestricted scalar must not be able to forge the next + // payload line: rootKey `p\ndb_password="q"` + password `r` vs rootKey `p` + a password + // whose tail mimics the same text. + const left = legacyShadowCacheKey({ + ...base, + rootKey: 'p"\ndb_password="q', + dbPassword: "r", + }); + const right = legacyShadowCacheKey({ + ...base, + rootKey: "p", + dbPassword: 'q"\ndb_password="r', + }); + expect(left).not.toBe(right); + }); + + it("excludes unresolved vault secrets, which the upsert never processes", () => { + const base = baseKeyInputs(); + const withUnresolved = legacyShadowCacheKey({ + ...base, + vault: [...base.vault, { name: "pending", value: "", resolved: false }], + }); + expect(withUnresolved).toBe(legacyShadowCacheKey(base)); + // A RESOLVED empty value does land in the cluster, so it must re-key. + const withResolvedEmpty = legacyShadowCacheKey({ + ...base, + vault: [...base.vault, { name: "pending", value: "", resolved: true }], + }); + expect(withResolvedEmpty).not.toBe(legacyShadowCacheKey(base)); + }); + + it("cannot collide vault name/value pairs across the tuple boundary", () => { + const base = baseKeyInputs(); + // `name=a=b, value=c` vs `name=a, value=b=c` — a bare `=`-joined encoding serializes both + // as `vault=a=b=c`. + const left = legacyShadowCacheKey({ + ...base, + vault: [{ name: "a=b", value: "c", resolved: true }], + }); + const right = legacyShadowCacheKey({ + ...base, + vault: [{ name: "a", value: "b=c", resolved: true }], + }); + expect(left).not.toBe(right); + }); + + it("hashes vault secrets in a name-stable order", () => { + const base = baseKeyInputs(); + const ascending = legacyShadowCacheKey({ + ...base, + vault: [ + { name: "a", value: "1", resolved: true }, + { name: "b", value: "2", resolved: true }, + ], + }); + const descending = legacyShadowCacheKey({ + ...base, + vault: [ + { name: "b", value: "2", resolved: true }, + { name: "a", value: "1", resolved: true }, + ], + }); + expect(ascending).toBe(descending); + }); +}); +describe("shadow baseline tar retention", () => { + const key = "0123456789abcdef"; + const now = 1_700_000_000_000; + + it("recognizes only this module's own published snapshots", () => { + expect(legacyIsShadowBaselineTar(legacyShadowBaselineTarFileName(key))).toBe(true); + for (const other of [ + "catalog-local-migrations-abc-123.json", + "shadow-baseline.tar", + `shadow-baseline-${key}.tar.4242.partial`, + `shadow-cache-${key}.json`, + "pgdelta-debug.zip", + // Wrong key length / non-hex. + "shadow-baseline-0123456789abcde.tar", + "shadow-baseline-0123456789abcdefg.tar", + "shadow-baseline-0123456789ABCDEF.tar", + ]) { + expect(legacyIsShadowBaselineTar(other), other).toBe(false); + } + }); + + it("recognizes only this module's own partial temp files as abandoned-sweep candidates", () => { + expect(legacyIsShadowBaselinePartial(`shadow-baseline-${key}.tar.4242.partial`)).toBe(true); + for (const other of [ + legacyShadowBaselineTarFileName(key), + "shadow-baseline-fedcba9876543210.tar", + "shadow-baseline.tar.4242.partial", + `shadow-baseline-${key}.tar.partial`, + `shadow-baseline-${key}.tar.4242.partial.bak`, + "catalog-local-migrations-abc-123.json", + ]) { + expect(legacyIsShadowBaselinePartial(other), other).toBe(false); + } + }); + + it("evicts aged tars and keeps the newest N among survivors", () => { + const aged = now - LEGACY_SHADOW_BASELINE_MAX_AGE_MS - 1; + const fresh = now - 1_000; + const entries = [ + { fileName: legacyShadowBaselineTarFileName("aaaaaaaaaaaaaaaa"), mtimeMs: aged }, + { fileName: legacyShadowBaselineTarFileName("bbbbbbbbbbbbbbbb"), mtimeMs: fresh }, + { fileName: legacyShadowBaselineTarFileName("cccccccccccccccc"), mtimeMs: fresh - 10 }, + { fileName: "catalog-abc.json", mtimeMs: aged }, + { fileName: `shadow-baseline-${key}.tar.1.partial`, mtimeMs: aged }, + ]; + expect( + legacyShadowBaselineTarsToEvict(entries, now, { + keep: 1, + maxAgeMs: LEGACY_SHADOW_BASELINE_MAX_AGE_MS, + }), + ).toEqual([ + legacyShadowBaselineTarFileName("aaaaaaaaaaaaaaaa"), + legacyShadowBaselineTarFileName("cccccccccccccccc"), + ]); + }); + + it("evicts the oldest beyond the keep cap when all are fresh", () => { + const entries = Array.from({ length: LEGACY_SHADOW_BASELINE_KEEP + 2 }, (_, index) => ({ + fileName: legacyShadowBaselineTarFileName(index.toString(16).padStart(16, "0")), + mtimeMs: now - index * 1_000, + })); + const evicted = legacyShadowBaselineTarsToEvict(entries, now); + expect(evicted).toHaveLength(2); + expect(evicted).toContain( + legacyShadowBaselineTarFileName(LEGACY_SHADOW_BASELINE_KEEP.toString(16).padStart(16, "0")), + ); + expect(evicted).toContain( + legacyShadowBaselineTarFileName( + (LEGACY_SHADOW_BASELINE_KEEP + 1).toString(16).padStart(16, "0"), + ), + ); + }); + + it("never returns a file that is not one of this module's own snapshots", () => { + expect( + legacyShadowBaselineTarsToEvict( + [ + { fileName: "catalog-local-migrations-abc-123.json", mtimeMs: 0 }, + { fileName: "shadow-baseline.tar", mtimeMs: 0 }, + { fileName: `shadow-baseline-${key}.tar.4242.partial`, mtimeMs: 0 }, + { fileName: `shadow-cache-${key}.json`, mtimeMs: 0 }, + { fileName: "pgdelta-debug.zip", mtimeMs: 0 }, + ], + now, + ), + ).toEqual([]); + }); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index dd31042f69..ecba06ebed 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -46,7 +46,6 @@ import { type Path, type Scope, } from "effect"; -import type * as HttpClient from "effect/unstable/http/HttpClient"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import { Output } from "../../../shared/output/output.service.ts"; @@ -75,12 +74,14 @@ import { type LegacyContainerError, type LegacyContainerOpts, } from "./container-lifecycle.ts"; +import type { LegacyStartContainerSpec } from "./docker-create-args.ts"; import type { LegacyImagePrepullError } from "./image-prepull.ts"; import type { LegacyHealthCheckTimeoutError } from "./health-check.ts"; -import { legacyWaitForHealthyServices } from "./health-check.ts"; +import { legacyWaitForShadowReady } from "./health-check.ts"; import type { LegacyLocalDbContainerInputs } from "./local-container-inputs.ts"; import { legacyListLocalMigrationPaths } from "../legacy-migration-history.ts"; import { legacyToPostgresURL } from "../legacy-postgres-url.ts"; +import { legacyTimeShadowPhase } from "./shadow-debug.ts"; import { type LegacyFreshDbSetupInput, type LegacySetupDatabaseInput, @@ -215,6 +216,32 @@ export interface LegacyCreateShadowDatabaseInput extends LegacyShadowPostgresCon readonly isBitbucketPipeline: boolean; readonly workdir: string; readonly extraHosts: ReadonlyArray; + /** + * Set ONLY by the shadow baseline cache's warm path (`shadow-cache.ts`): a previously exported + * PGDATA tar to unpack into the container between `docker create` and `docker start`, so the + * `supabase/postgres` entrypoint finds an initialized data directory and skips `initdb` plus + * the whole platform baseline. Everything else about the container — including `--rm` and the + * project-labels-only label set — is identical to an uncached shadow, so a restored shadow is + * still a throwaway container removed on release. + * + * Delivered as {@link LegacyStartContainerSpec.preStartArchives}; see that field's doc comment + * for why the tar-stream form of `docker cp` is the only one that works here. + */ + readonly restoreArchive?: NonNullable[number]; + /** + * Set ONLY by the shadow baseline cache's COLD path (`shadow-cache.ts`), to `false`. That path + * has to `docker stop` the container mid-run to take a coherent disk-level PGDATA snapshot and + * then `docker start` it again — and Docker removes an `AutoRemove` container the moment it + * exits, `docker stop` included (verified against Docker 29: the container is gone ~1-2s after + * the stop returns), which would leave nothing to restart. + * + * The container is still removed by `docker rm -f -v` on release exactly like every other + * shadow, so the only externally visible difference is what a SIGKILLed CLI leaves behind: a + * stopped shadow container carrying the usual project labels — which `supabase stop` sweeps — + * rather than nothing. Omitted (i.e. Go's `--rm`) on the warm path and whenever the cache is + * off, since neither ever stops the container. + */ + readonly autoRemove?: boolean; } /** Resolved by {@link legacyCreateShadowDatabase} — everything a caller needs to both use and later tear down the shadow. */ @@ -266,7 +293,17 @@ export const legacyCreateShadowDatabase = ( }), ), ); - const spec = legacyBuildShadowPostgresContainerSpec(input); + // Both overrides are the shadow baseline cache's and nobody else's: a warm restore adds one + // `docker cp -` between `docker create` and `docker start` (no argv change at all), and a cold + // cache-enabled provision drops `--rm` so the container survives its own snapshot's `docker + // stop`. The spec builder itself stays on Go's `autoRemove: true` default. See each field's + // doc comment on {@link LegacyCreateShadowDatabaseInput}. + const baseSpec = legacyBuildShadowPostgresContainerSpec(input); + const spec: LegacyStartContainerSpec = { + ...baseSpec, + ...(input.autoRemove === undefined ? {} : { autoRemove: input.autoRemove }), + ...(input.restoreArchive === undefined ? {} : { preStartArchives: [input.restoreArchive] }), + }; // The shadow container has no name (Docker auto-generates one) and no network alias — // see this module's own header for why that's still enough for the shadow's own one-shot // setup jobs to reach it. The pgsodium root key itself (PG15+ only) never touches host @@ -377,6 +414,28 @@ export interface LegacyShadowSetupInput extends LegacyShadowConnectionInput { readonly setup: LegacyShadowDbSetupInput; } +/** + * Memoizes `effect`'s first SUCCESS; failures are never cached, so a retry re-runs the real + * effect. Deliberately not `Effect.cached` (which returns `Effect>` and needs an + * effectful construction site — {@link legacyShadowRunInputFromLocalContainerInputs} is a plain + * function) and not concurrency-guarded: the two consumers of the one field this wraps (`jwks` — + * see its construction inside that function) evaluate sequentially on the same fiber. + */ +function legacyMemoizeSuccess(effect: Effect.Effect): Effect.Effect { + let succeeded: Effect.Effect | undefined; + return Effect.suspend( + () => + succeeded ?? + effect.pipe( + Effect.tap((value) => + Effect.sync(() => { + succeeded = Effect.succeed(value); + }), + ), + ), + ); +} + /** * Adapts {@link LegacyLocalDbContainerInputs} (`local-container-inputs.ts`, the SAME * config/image/JWKS resolution prelude `db start`/`db reset` share) plus the caller's own @@ -460,7 +519,14 @@ export function legacyShadowRunInputFromLocalContainerInputs( database: "postgres", }), jwtSecret: localInputs.setup.jwtSecret, - jwks: localInputs.setup.jwks, + // Memoized: with the shadow baseline cache enabled this effect is evaluated TWICE on a + // cold run — once by `legacyResolveShadowCacheKeyInputs` (`shadow-cache.ts`) for the cache + // key, once by `legacyResolveDbSetupPrelude` for the baseline itself — and third-party + // JWKS discovery can be a real network request. Memoizing the first success keeps the run + // to one request AND guarantees the published snapshot carries the exact value its key was + // computed from, even if the issuer rotates mid-run (review: Codex on #6184). Failures are + // not cached — a transient discovery failure fails the run either way. + jwks: legacyMemoizeSuccess(localInputs.setup.jwks), apiUrl: localInputs.setup.apiUrl, authExternalUrl: localInputs.setup.authExternalUrl, siteUrl: localInputs.setup.siteUrl, @@ -480,26 +546,31 @@ export function legacyShadowRunInputFromLocalContainerInputs( } /** - * Port of Go's `PrepareRawShadow` (`apps/cli-go/internal/db/diff/shadow.go:93-116`): health-wait - * against an already-{@link legacyCreateShadowDatabase}-created shadow (created + healthy, no - * platform baseline or migrations applied) — used inline (`db pull --declarative`'s empty - * declarative-export source), not the `ok`-sentinel error-path pattern + * Port of Go's `PrepareRawShadow` (`apps/cli-go/internal/db/diff/shadow.go:93-116`): readiness + * wait against an already-{@link legacyCreateShadowDatabase}-created shadow (created + accepting + * connections, no platform baseline or migrations applied) — used inline (`db pull + * --declarative`'s empty declarative-export source), not the `ok`-sentinel error-path pattern * `legacy-shadow-source.ts`'s `legacyPrepareShadowSource` uses, since there is only ONE step - * here that can fail (the health wait) rather than several. Lives here (not + * here that can fail (the readiness wait) rather than several. Lives here (not * `legacy-shadow-source.ts`) because it has zero pg-delta/declarative dependency — see this * module's own header. * + * Gates on {@link legacyWaitForShadowReady}, NOT on the Docker-health + * `legacyWaitForHealthyServices` the long-running `db` container still uses: the shadow's + * own healthcheck cannot report `healthy` before its first 10-second-interval probe, ~6.5s after + * Postgres is already connectable — see that function's own doc comment. + * * Deliberately does NOT call {@link legacyCreateShadowDatabase} itself — the caller does, as the * `acquire` of an `Effect.acquireUseRelease` whose `use` phase is this function (see * `diff.handler.ts`/`pull.handler.ts`'s call sites). Go's `PrepareRawShadow` threads a single - * cancellable `ctx` through both creation and the health wait, so a SIGINT can interrupt either; - * an earlier shape here instead passed the WHOLE create-then-health-wait effect as `acquire`, + * cancellable `ctx` through both creation and the readiness wait, so a SIGINT can interrupt + * either; an earlier shape here instead passed the WHOLE create-then-wait effect as `acquire`, * which Effect's `uninterruptibleMask` (`acquireUseRelease(acquire, use, release) => * uninterruptibleMask(restore => flatMap(acquire, a => onExitPrimitive(restore(use(a)), ...)))`) - * makes entirely uninterruptible — a SIGINT during the health wait (which can run for up to + * makes entirely uninterruptible — a SIGINT during the readiness wait (which can run for up to * `healthTimeoutSeconds`) was silently swallowed until the wait finished or timed out on its * own, unlike Go. Splitting `legacyCreateShadowDatabase` out as the (brief, Docker-API-bound) - * `acquire` and keeping this health-wait as part of the interruptible `use` restores that parity + * `acquire` and keeping this wait as part of the interruptible `use` restores that parity * — a SIGINT here now lands immediately, same as Go's ctx cancellation, while * `legacyRemoveShadowDatabase` still runs as the `release` finalizer regardless of how `use` * exits (review: PRRT_kwDOErm0O86XMrID). @@ -508,16 +579,9 @@ export const legacyPrepareRawShadow = ( spawner: Spawner, handle: LegacyShadowDatabaseHandle, input: LegacyPrepareRawShadowInput, -): Effect.Effect< - LegacyShadowSourceResult, - LegacyHealthCheckTimeoutError, - Output | LegacyDockerRun | RuntimeInfo | HttpClient.HttpClient -> => +): Effect.Effect => Effect.gen(function* () { const { containerId } = handle; - yield* legacyWaitForHealthyServices(spawner, [containerId], { - timeoutSeconds: input.healthTimeoutSeconds, - }); const connConfig: LegacyPgConnInput = { host: input.hostname, port: input.shadowPort, @@ -525,6 +589,10 @@ export const legacyPrepareRawShadow = ( password: input.password, database: "postgres", }; + yield* legacyWaitForShadowReady(spawner, containerId, connConfig, { + timeoutSeconds: input.healthTimeoutSeconds, + image: input.image, + }); return { container: containerId, sourceUrl: legacyToPostgresURL(connConfig), @@ -562,7 +630,23 @@ export const legacySetupShadowConn = ( Effect.fail(new LegacyShadowDbError({ message: cause.message, reason: "connect" })), ), ); - yield* input.session.exec(LEGACY_SHADOW_CREATE_TEMPLATE_SQL).pipe( + yield* legacyCreateShadowTemplateDatabase(input.session); + }); + +/** + * {@link legacySetupShadowConn}'s trailing {@link LEGACY_SHADOW_CREATE_TEMPLATE_SQL} step on its + * own (Go's `setupShadowConn`'s second half, `diff.go:178`). Split out because it is the ONE part + * of Go's `setupShadowConn` a warm shadow-cache hit still has to run: the cache's PGDATA snapshot + * is taken strictly BEFORE this statement (`shadow-cache.ts`), so a restored cluster carries the + * platform baseline but no `contrib_regression`, and the template database must be recreated even + * though the baseline itself is skipped. + */ +const legacyCreateShadowTemplateDatabase = ( + session: LegacyDbSession, +): Effect.Effect => + legacyTimeShadowPhase( + "contrib-regression-create", + session.exec(LEGACY_SHADOW_CREATE_TEMPLATE_SQL).pipe( Effect.mapError( (cause) => new LegacyShadowDbError({ @@ -570,8 +654,8 @@ export const legacySetupShadowConn = ( reason: "database", }), ), - ); - }); + ), + ); /** * Shared fields both {@link legacySetupShadowDatabase} and {@link legacyMigrateShadowDatabase} @@ -657,11 +741,17 @@ export const legacyBuildShadowSetupDatabaseInput = ( * connect error immediately, not pay for JWKS work first. The connection is closed once this * resolves (Go's `defer conn.Close(...)`), matching `Effect.scoped`'s finalizer running at the * end of this function rather than leaking a `Scope.Scope` requirement to the caller. + * + * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_COLD}, i.e. exactly the sequence above. + * A warm shadow-cache hit skips the prelude + `SetupDatabase` (the restored cluster already + * has them) and only recreates `contrib_regression`; a cache-enabled COLD provision snapshots + * between the baseline and the template, matching {@link migrateShadowDatabase}. */ export const legacySetupShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, options: LegacySetupDatabaseOptions = {}, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, @@ -669,16 +759,100 @@ export const legacySetupShadowDatabase = ( > => Effect.scoped( Effect.gen(function* () { + if (!baseline.baselinePresent && baseline.snapshotRequired) { + yield* Effect.scoped( + Effect.gen(function* () { + const setupSession = yield* legacyConnectShadowDatabase(input.connConfig); + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput(input, setupSession, resolved), + options, + ).pipe( + // The baseline's batched SQL files check their own connection out of the pool; + // failing to acquire one is a shadow CONNECT failure, like + // `legacyConnectShadowDatabase`'s, never a setup/statement failure. + Effect.catchTag("LegacyDbConnectError", (cause) => + Effect.fail(new LegacyShadowDbError({ message: cause.message, reason: "connect" })), + ), + ); + }), + ); + yield* baseline.snapshotBaseline; + } const session = yield* legacyConnectShadowDatabase(input.connConfig); - const resolved = yield* legacyResolveDbSetupPrelude(input.setup); - yield* legacySetupShadowConn( - spawner, - legacyBuildShadowSetupDatabaseInput(input, session, resolved), - options, - ); + if (!baseline.baselinePresent && !baseline.snapshotRequired) { + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput(input, session, resolved), + options, + ).pipe( + // Same connect-vs-setup classification as the snapshot branch above. + Effect.catchTag("LegacyDbConnectError", (cause) => + Effect.fail(new LegacyShadowDbError({ message: cause.message, reason: "connect" })), + ), + ); + } + yield* legacyCreateShadowTemplateDatabase(session); }), ); +/** + * What an `acquire` hands the `use` phase about the shadow cluster's CONTENTS — the seam the warm + * shadow-container cache (`shadow-cache.ts`) needs and nothing else uses. + * + * Deliberately a value the acquire OWNS and returns (alongside the container id), not an + * `afterBaseline` callback threaded down through `legacyPrepareShadowSource`: the cache is the + * only party that knows whether a cluster already carries a baseline and what to do once a fresh + * one exists, so both answers travel together with the container the cache handed over. + * {@link LEGACY_SHADOW_BASELINE_COLD} is what every uncached caller passes. + */ +export interface LegacyShadowBaselineState { + /** + * `true` only on a warm cache hit: the cluster already carries the platform baseline + * (`legacySetupDatabase`'s init schema + API privileges + vault + `roles.sql`), restored from + * the cache's own PGDATA snapshot, so re-running it would be wasted work at best and a + * double-applied baseline at worst. + */ + readonly baselinePresent: boolean; + /** + * `true` ONLY for a cache-enabled COLD provision — the one state whose + * {@link snapshotBaseline} really stops the container. This is what + * {@link legacyMigrateShadowDatabase} keys its session structure on: the baseline session must + * be closed before a real snapshot (a disk-level export severs any live backend), but when no + * snapshot will run, splitting sessions would be a gratuitous behavior change — a reconnect + * picks up role-level defaults `roles.sql` may have just installed (e.g. `ALTER ROLE postgres + * SET statement_timeout`), which Go's single-connection flow never exposed to migrations + * (review: Codex on #6184). So uncached and warm runs keep exactly one session. + */ + readonly snapshotRequired: boolean; + /** + * Runs immediately after a FRESHLY provisioned baseline and strictly before the template + * database/user migrations — the only point at which `postgres` holds the pristine baseline and + * nothing else. + * + * Takes NO session, and {@link legacyMigrateShadowDatabase} guarantees no session is open + * against the shadow while it runs when {@link snapshotRequired} is set: the snapshot is a + * disk-level PGDATA export that has to stop the container, which would sever any live backend. + * + * A cache that cannot SNAPSHOT degrades silently (warn + uncached run) — but the error channel + * is {@link LegacyShadowDbError}, not `never`, for the one failure that is the run's problem + * rather than the cache's: a shadow that does not come back up after the export. Reporting + * success there would send the caller's next connect to a dead (or worse, someone else's) + * Postgres on the shadow port — see `legacyExportShadowBaseline`'s doc comment + * (`shadow-cache.ts`). + */ + readonly snapshotBaseline: Effect.Effect; +} + +/** The baseline state every uncached caller passes: provision it, snapshot nothing. */ +export const LEGACY_SHADOW_BASELINE_COLD: LegacyShadowBaselineState = { + baselinePresent: false, + snapshotRequired: false, + snapshotBaseline: Effect.void, +}; + /** * Port of Go's `MigrateShadowDatabase` (`apps/cli-go/internal/db/diff/diff.go:195-209`): * lists local migrations FIRST (Go's `migration.ListLocalMigrations`, fails fast on a bad @@ -691,11 +865,27 @@ export const legacySetupShadowDatabase = ( * (`db-setup.ts`) for the real local `db` container — see {@link legacySetupShadowDatabase}'s * own doc comment for why the ordering matters. Connection closed once this resolves, matching * Go's `defer conn.Close(...)`. + * + * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_COLD}, i.e. exactly the sequence above. + * A warm shadow-cache hit passes a state whose `baselinePresent` is `true`, which skips the + * prelude + `SetupDatabase` steps (the restored cluster already has them) and goes straight to + * the template database and the user migrations; a COLD cache-enabled provision passes the same + * cold sequence plus a `snapshotBaseline` step between the baseline and the template database. + * + * The one structural divergence from Go is confined to the SNAPSHOTTING cold branch + * (`baseline.snapshotRequired`): there the baseline runs in its own scope, its session is CLOSED + * before {@link LegacyShadowBaselineState.snapshotBaseline} (the disk-level PGDATA snapshot stops + * the container, which severs any live backend), and the template database + migrations run on a + * second session. Every OTHER state — uncached (cache off / `--no-cache` / OrioleDB) and warm — + * uses exactly one session, matching Go's single connection: see + * {@link LegacyShadowBaselineState.snapshotRequired} for why the split must not leak into the + * uncached path. The SQL every path issues is unchanged. */ const migrateShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, setupOptions: LegacySetupDatabaseOptions, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, @@ -714,13 +904,46 @@ const migrateShadowDatabase = ( ), ); + if (!baseline.baselinePresent && baseline.snapshotRequired) { + // Own scope: the baseline session must be closed before `snapshotBaseline` — see this + // function's own doc comment. + yield* Effect.scoped( + Effect.gen(function* () { + const setupSession = yield* legacyConnectShadowDatabase(input.connConfig); + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput(input, setupSession, resolved), + setupOptions, + ).pipe( + // The baseline's batched SQL files check their own connection out of the pool; + // failing to acquire one is a shadow CONNECT failure, like + // `legacyConnectShadowDatabase`'s, never a setup/statement failure. + Effect.catchTag("LegacyDbConnectError", (cause) => + Effect.fail(new LegacyShadowDbError({ message: cause.message, reason: "connect" })), + ), + ); + }), + ); + yield* baseline.snapshotBaseline; + } const session = yield* legacyConnectShadowDatabase(input.connConfig); - const resolved = yield* legacyResolveDbSetupPrelude(input.setup); - yield* legacySetupShadowConn( - spawner, - legacyBuildShadowSetupDatabaseInput(input, session, resolved), - setupOptions, - ); + if (!baseline.baselinePresent && !baseline.snapshotRequired) { + // Go's single-connection flow, verbatim: baseline + template + migrations all on this + // one session — see this function's own doc comment. + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput(input, session, resolved), + setupOptions, + ).pipe( + // Same connect-vs-setup classification as the snapshot branch above. + Effect.catchTag("LegacyDbConnectError", (cause) => + Effect.fail(new LegacyShadowDbError({ message: cause.message, reason: "connect" })), + ), + ); + } + yield* legacyCreateShadowTemplateDatabase(session); yield* legacyApplyMigrations( session, input.fs, @@ -746,11 +969,12 @@ const migrateShadowDatabase = ( export const legacyMigrateShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, Output | LegacyDockerRun | RuntimeInfo | LegacyDbConnection -> => migrateShadowDatabase(spawner, input, { webhooks: "enabled" }); +> => migrateShadowDatabase(spawner, input, { webhooks: "enabled" }, baseline); /** * Migrates a shadow for the in-process pg-delta engine. Unlike the legacy engine, @@ -760,8 +984,9 @@ export const legacyMigrateShadowDatabase = ( export const legacyMigrateNextShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, Output | LegacyDockerRun | RuntimeInfo | LegacyDbConnection -> => migrateShadowDatabase(spawner, input, {}); +> => migrateShadowDatabase(spawner, input, {}, baseline); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index 7249ea4ab0..3a785d7383 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -212,7 +212,7 @@ describe("legacyCreateShadowDatabase / legacyRemoveShadowDatabase", () => { // in argv. const script = mock.spawned[createIdx]?.at(-1) ?? ""; expect(script).toContain( - `docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}`, + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}`, ); }), ); @@ -741,6 +741,64 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { ); }); + it.effect( + "skips the platform baseline on a warm cache hit but still creates the template", + () => { + const { session, calls } = fakeSession(); + const workdir = tempRoot.current; + const mock = mockSpawner(); + let jwksEvaluated = false; + return Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + yield* legacySetupShadowDatabase( + mock.spawner, + { + fs, + path, + workdir, + projectId: "proj", + container: "shadow-container-id-0123456789abcdef", + networkId: "supabase_network_proj", + connConfig: { + host: "127.0.0.1", + port: 54320, + user: "postgres", + password: "postgres", + database: "postgres", + }, + setup: baseShadowSetup({ + majorVersion: 17, + realtimeEnabledForSetup: true, + jwks: Effect.sync(() => { + jwksEvaluated = true; + return '{"keys":[]}'; + }), + }), + }, + {}, + { + baselinePresent: true, + snapshotRequired: false, + snapshotBaseline: Effect.void, + }, + ); + expect(jwksEvaluated).toBe(false); + expect(calls.some((c) => c.sql === LEGACY_SHADOW_CREATE_TEMPLATE_SQL)).toBe(true); + }).pipe( + Effect.provide( + Layer.mergeAll( + BunServices.layer, + mockOutput().layer, + mockDockerRun(), + mockRuntimeInfo(), + mockDbConnection(session), + ), + ), + ); + }, + ); + it.effect( "legacyMigrateShadowDatabase lists local migrations BEFORE connecting, tolerating a missing migrations directory as an empty list rather than a failure", () => { @@ -787,6 +845,12 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { }, setup: baseShadowSetup(), }); + // ONE connect, matching Go's single-connection flow: the default baseline state + // (`LEGACY_SHADOW_BASELINE_COLD`) requires no snapshot, so baseline + template + + // migrations all share one session — the split-session shape is reserved for the + // cache's own snapshotting cold provision (`snapshotRequired: true`), whose disk-level + // export must close the session before stopping the container. The ordering under test + // is unaffected — the migration listing still precedes the connect. expect(events).toEqual(["list", "connect"]); }).pipe( Effect.provide( diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-debug.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-debug.ts new file mode 100644 index 0000000000..c7c9248d68 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-debug.ts @@ -0,0 +1,80 @@ +/** + * Debug-only phase-timing instrumentation for the shadow baseline cache + * (`shadow-cache.ts`'s `baseline-export`/`baseline-restore`) and the shadow's readiness gate + * (`health-check.ts`'s `legacyWaitForShadowReady`, which emits `ready-attempt`/`ready-wait`). + * `SUPABASE_SHADOW_DEBUG` is the opt-in gate — with it unset every + * helper here is a pure pass-through (not even a `Clock` read), so leaving instrumentation in + * place costs one boolean check per call site and nothing else. This module changes nothing about + * behavior, stdout, exit codes, or flags: it only ever writes to STDERR, and only when the env + * var is set (Go-parity constraint — this is a TS-only debugging aid with no Go counterpart). + * + * Lives in its own leaf module, not inside `shadow-cache.ts` itself, so `shadow-database.ts` (the + * `contrib_regression` template-database creation) and `health-check.ts` (the readiness gate) can + * use the same timing primitive without an import cycle back through `shadow-cache.ts`, which + * already imports from both of those modules. + * + * Every line has the fixed shape `shadow-debug: ms` — see + * {@link legacyTimeShadowPhase}'s own doc comment for the `detail` contract. + */ + +import { Clock, Effect, Result } from "effect"; + +import { Output } from "../../../shared/output/output.service.ts"; +import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; + +/** `SUPABASE_SHADOW_DEBUG` — the opt-in gate for shadow phase-timing instrumentation (default OFF). */ +export const LEGACY_SHADOW_DEBUG_ENV = "SUPABASE_SHADOW_DEBUG"; + +/** + * Whether shadow phase-timing debug lines are enabled for this invocation. Mirrors + * `legacyShadowCacheEnabled`'s own env-parsing style (`shadow-cache.ts`, itself + * `legacyParseBoolEnv` over `process.env`) — checked fresh on every call, never cached at module + * load, so a test (or a long-lived process) that mutates `process.env` sees the change + * immediately. + */ +export function legacyShadowDebugEnabled( + env: Readonly> = process.env, +): boolean { + return legacyParseBoolEnv(env[LEGACY_SHADOW_DEBUG_ENV]); +} + +/** Truncates a debug-line detail string to `maxLength` (default ~120, matching the brief's per-line budget), appending `…` when cut. */ +export function legacyShadowDebugTruncate(message: string, maxLength = 120): string { + return message.length > maxLength ? `${message.slice(0, maxLength)}…` : message; +} + +/** `shadow-debug: ms\n` — the one line shape every emitter in this module produces. */ +function legacyShadowDebugLine(phase: string, elapsedMs: number, detail: string): string { + return `shadow-debug: ${phase} ${elapsedMs}ms${detail}\n`; +} + +/** + * Times `effect` and, only when {@link legacyShadowDebugEnabled}, emits one + * `shadow-debug: ms` line to stderr via `Output.raw` once `effect` completes — + * on success OR failure, so a failing phase's elapsed time is never silently lost. + * + * `phase` may be a thunk instead of a plain string for a label that is itself non-trivial to + * build (e.g. embedding a SQL statement's first ~60 chars) — the thunk is only ever invoked when + * debug is on, so a disabled run pays neither the label construction nor the `Clock` read. + * + * `detail` receives the effect's own {@link Result.Result} and returns the trailing text (e.g. + * ` attempts=5` or ` error="..."`) appended after the bare `ms`; the default emits nothing + * extra. A complete no-op when debug is off — the input `effect` is returned unchanged, so + * leaving this wrapper in place costs exactly one boolean check per call site. + */ +export const legacyTimeShadowPhase = ( + phase: string | (() => string), + effect: Effect.Effect, + detail: (result: Result.Result) => string = () => "", +): Effect.Effect => { + if (!legacyShadowDebugEnabled()) return effect; + const label = typeof phase === "function" ? phase() : phase; + return Effect.gen(function* () { + const start = yield* Clock.currentTimeMillis; + const outcome = yield* Effect.result(effect); + const elapsed = (yield* Clock.currentTimeMillis) - start; + const output = yield* Output; + yield* output.raw(legacyShadowDebugLine(label, elapsed, detail(outcome)), "stderr"); + return yield* Effect.fromResult(outcome); + }); +}; diff --git a/apps/cli/src/legacy/shared/legacy-docker-lifecycle.ts b/apps/cli/src/legacy/shared/legacy-docker-lifecycle.ts index 77cb13cf4d..1dfe3cb60b 100644 --- a/apps/cli/src/legacy/shared/legacy-docker-lifecycle.ts +++ b/apps/cli/src/legacy/shared/legacy-docker-lifecycle.ts @@ -81,7 +81,7 @@ function splitNonEmptyLines(text: string): ReadonlyArray { } /** - * Shared `docker ps --filter label= [--all] --format + * Shared `docker ps --filter label= [--filter label=…] [--all] --format * ` spawn: one Docker CLI invocation is one underlying `GET * /containers/json` Docker Engine API request regardless of `--format` * (`--format` only controls how the CLI renders the already-returned JSON @@ -89,17 +89,23 @@ function splitNonEmptyLines(text: string): ReadonlyArray { * through here so two differently-formatted needs never accidentally cost * two real requests. See {@link legacyListContainerIdsAndNames}'s doc * comment for why that distinction matters for Go-parity request-log tests. + * + * Multiple `labelFilters` are ANDed by Docker itself (repeating `--filter label=` narrows the + * match), so scoping a listing to a second label costs no extra request either. */ function spawnDockerPsLines( spawner: Spawner, - opts: { readonly projectIdFilter: string; readonly all: boolean; readonly formatArg: string }, + opts: { + readonly labelFilters: ReadonlyArray; + readonly all: boolean; + readonly formatArg: string; + }, ): Effect.Effect, LegacyDockerLifecycleListError> { return Effect.scoped( Effect.gen(function* () { const args = [ "ps", - "--filter", - `label=${opts.projectIdFilter}`, + ...opts.labelFilters.flatMap((filterValue) => ["--filter", `label=${filterValue}`]), ...(opts.all ? ["--all"] : []), "--format", opts.formatArg, @@ -164,7 +170,7 @@ export const legacyListContainersByLabel = ( }, ) => spawnDockerPsLines(spawner, { - projectIdFilter: opts.projectIdFilter, + labelFilters: [opts.projectIdFilter], all: opts.all, formatArg: opts.format === "names" ? "{{.Names}}" : "{{.ID}}", }); @@ -207,7 +213,7 @@ export const legacyListContainerIdsAndNames = ( }, ): Effect.Effect, LegacyDockerLifecycleListError> => spawnDockerPsLines(spawner, { - projectIdFilter: opts.projectIdFilter, + labelFilters: [opts.projectIdFilter], all: opts.all, formatArg: `{{.ID}}\t{{.Names}}\t{{.Label "${LEGACY_CLI_WORKDIR_LABEL}"}}`, }).pipe( diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts index f53595e08d..11b4349e42 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts @@ -22,15 +22,18 @@ import { legacyBuildLocalDbContainerInputs, type LegacyLocalDbContainerInputs, } from "./db-bootstrap/local-container-inputs.ts"; -import { legacyWaitForHealthyServices } from "./db-bootstrap/health-check.ts"; +import { legacyWaitForShadowReady } from "./db-bootstrap/health-check.ts"; +import { + legacyWithShadowDatabase, + type LegacyShadowAcquiredHandle, + type LegacyShadowCacheOpts, +} from "./db-bootstrap/shadow-cache.ts"; import { - legacyCreateShadowDatabase, - legacyRemoveShadowDatabase, legacySetupShadowDatabase, legacyShadowRunInputFromLocalContainerInputs, - type LegacyShadowDatabaseHandle, type LegacyShadowSetupInput, } from "./db-bootstrap/shadow-database.ts"; +import { legacyPgDeltaTempPath } from "./legacy-pgdelta.paths.ts"; import { legacyCompareUtf8Bytes } from "./legacy-glob.ts"; import { LegacyMigrationsReadError } from "./legacy-migration.errors.ts"; import { legacyToPostgresURL } from "./legacy-postgres-url.ts"; @@ -227,11 +230,6 @@ export function legacyDeclarativeCatalogFileName( return `catalog-${legacySanitizedCatalogPrefix(prefix)}-declarative-${hash}-${timestampMillis}.json`; } -/** `supabase/.temp/pgdelta` — where catalog snapshots + debug bundles live. */ -export function legacyPgDeltaTempPath(path: Path.Path, workdir: string): string { - return path.join(workdir, "supabase", ".temp", "pgdelta"); -} - /** * Lists local migration file paths under `migrationsDir`. Mirrors Go's * `migration.ListLocalMigrations` (`pkg/migration/list.go:33`): entries are sorted by name — Go's @@ -768,10 +766,13 @@ const exportViaShadowCatalog = ( built: LegacyShadowCatalogInputs, provision: ( spawner: Spawner, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, shadowInput: LegacyShadowSetupInput, ) => Effect.Effect, persist: (snapshot: string) => Effect.Effect, + // No default: every caller must declare its provisioner's effective webhooks policy, or the + // key records config-following while the baseline forced `pg_net` on (review: Codex on #6184). + shadowCacheOpts: LegacyShadowCacheOpts, ) => Effect.gen(function* () { const { spawner, localInputs } = built; @@ -783,8 +784,16 @@ const exportViaShadowCatalog = ( fs, path, ); - const written = yield* Effect.acquireUseRelease( - legacyCreateShadowDatabase(spawner, shadowInput), + // `legacyWithShadowDatabase` (`db-bootstrap/shadow-cache.ts`) rather than a bare + // `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` pair — see its doc comment: with + // `SUPABASE_SHADOW_CACHE` unset it IS that pair (identical Docker argv, identical labels), and + // with it set a catalog cache miss restores a key-matching PGDATA snapshot into the fresh + // shadow instead of paying the full cold provision — the same swap `db diff`/`db pull`'s own + // call sites make. `shadowCacheOpts` carries `sync --no-cache`'s bypass and the + // caller's effective Webhooks policy — see `LegacyShadowCacheOpts`. + const written = yield* legacyWithShadowDatabase( + spawner, + shadowInput, (handle) => Effect.gen(function* () { const shadow = yield* provision(spawner, handle, shadowInput); @@ -794,7 +803,7 @@ const exportViaShadowCatalog = ( }); return yield* persist(snapshot); }), - (handle) => legacyRemoveShadowDatabase(spawner, handle.containerId), + shadowCacheOpts, ); return path.relative(ctx.cwd, written); }); @@ -811,7 +820,7 @@ const legacyProvisionMigrationsShadow = ( ctx: LegacyPgDeltaContext, toml: LegacyDbTomlValues, spawner: Spawner, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, shadowInput: LegacyShadowSetupInput, ) => legacyPrepareShadowSource(spawner, handle, { @@ -881,6 +890,10 @@ export const legacyResolveMigrationsCatalogRef = Effect.fnUntraced(function* ( timestamp, ); }), + // `legacyProvisionMigrationsShadow` migrates via `legacyMigrateShadowDatabase`, which forces + // `pg_net` on — the key must record that, not the config-following default (no `bypassCache`: + // `db diff` has no `--no-cache` on this path). + { webhooks: "enabled" }, ); }); @@ -988,6 +1001,10 @@ export const legacyGetMigrationsCatalogRef = Effect.fnUntraced(function* ( timestamp, ); }), + // `--no-cache` promises "force fresh shadow database setup" (`declarative.shared.ts`), so it + // must ALSO bypass the shadow baseline snapshot, not just the catalog cache — otherwise a + // warm tar would skip the very setup the flag exists to force (review: Codex on #6184). + { bypassCache: params.noCache, webhooks: "enabled" }, ); }); @@ -1050,13 +1067,10 @@ const legacyProvisionBaselineShadow = ( fs: FileSystem.FileSystem, path: Path.Path, ctx: LegacyPgDeltaContext, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, shadowInput: LegacyShadowSetupInput, ) => Effect.gen(function* () { - yield* legacyWaitForHealthyServices(spawner, [handle.containerId], { - timeoutSeconds: shadowInput.healthTimeoutSeconds, - }); const connConfig: LegacyPgConnInput = { host: shadowInput.hostname, port: shadowInput.shadowPort, @@ -1064,16 +1078,25 @@ const legacyProvisionBaselineShadow = ( password: shadowInput.password, database: "postgres", }; - yield* legacySetupShadowDatabase(spawner, { - fs, - path, - workdir: ctx.cwd, - projectId: shadowInput.projectId, - container: handle.containerId, - networkId: shadowInput.networkId, - connConfig, - setup: shadowInput.setup, + yield* legacyWaitForShadowReady(spawner, handle.containerId, connConfig, { + timeoutSeconds: shadowInput.healthTimeoutSeconds, + image: shadowInput.image, }); + yield* legacySetupShadowDatabase( + spawner, + { + fs, + path, + workdir: ctx.cwd, + projectId: shadowInput.projectId, + container: handle.containerId, + networkId: shadowInput.networkId, + connConfig, + setup: shadowInput.setup, + }, + {}, + handle, + ); return { sourceUrl: legacyToPostgresURL(connConfig) } satisfies LegacyProvisionedShadow; }); @@ -1092,13 +1115,10 @@ const legacyProvisionDeclarativeShadow = ( ctx: LegacyPgDeltaContext, declarativeDirAbs: string, declarativeDirRel: string, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, shadowInput: LegacyShadowSetupInput, ) => Effect.gen(function* () { - yield* legacyWaitForHealthyServices(spawner, [handle.containerId], { - timeoutSeconds: shadowInput.healthTimeoutSeconds, - }); const connConfig: LegacyPgConnInput = { host: shadowInput.hostname, port: shadowInput.shadowPort, @@ -1106,16 +1126,25 @@ const legacyProvisionDeclarativeShadow = ( password: shadowInput.password, database: "postgres", }; - yield* legacySetupShadowDatabase(spawner, { - fs, - path, - workdir: ctx.cwd, - projectId: shadowInput.projectId, - container: handle.containerId, - networkId: shadowInput.networkId, - connConfig, - setup: shadowInput.setup, + yield* legacyWaitForShadowReady(spawner, handle.containerId, connConfig, { + timeoutSeconds: shadowInput.healthTimeoutSeconds, + image: shadowInput.image, }); + yield* legacySetupShadowDatabase( + spawner, + { + fs, + path, + workdir: ctx.cwd, + projectId: shadowInput.projectId, + container: handle.containerId, + networkId: shadowInput.networkId, + connConfig, + setup: shadowInput.setup, + }, + {}, + handle, + ); const targetUrl = legacyToPostgresURL(connConfig); yield* legacyApplyDeclarativePgDelta(ctx, { fs, @@ -1202,6 +1231,7 @@ export const legacyExportBaselineCatalogRef = ( snapshot, ) : legacyWriteCatalogFile(fs, tempDir, cachePath, snapshot), + { bypassCache: params.noCache, webhooks: "config" }, ); }); @@ -1292,5 +1322,6 @@ export const legacyExportDeclarativeCatalogRef = ( timestamp, ); }), + { bypassCache: params.noCache, webhooks: "config" }, ); }); diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts new file mode 100644 index 0000000000..fdb2912c92 --- /dev/null +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts @@ -0,0 +1,40 @@ +/** + * On-disk locations for pg-delta-adjacent cache/snapshot artefacts. + * + * Split out of `legacy-pgdelta.cache.ts` (which owns the catalog cache's keys AND its + * shadow-provisioning resolution path) so `db-bootstrap/shadow-cache.ts` — the warm + * shadow-container cache, which `legacy-pgdelta.cache.ts` itself consumes for its own + * shadow provisioning — can reach path helpers without an import cycle between the two. + * + * Two roots: + * - {@link legacyPgDeltaTempPath}: project-local (`supabase/.temp/pgdelta`) — catalog + * snapshots and debug bundles (Go-shared, workspace-mounted). + * - {@link legacyShadowBaselineCacheDir}: global under `SUPABASE_HOME` — the shadow + * baseline PGDATA tars, shared across worktrees with the same settings. + */ + +import { homedir } from "node:os"; + +import type { Path } from "effect"; + +import { resolveSupabaseHome } from "../../shared/config/supabase-home.ts"; + +/** `supabase/.temp/pgdelta` — catalog snapshots and debug bundles (`declarative.go:44`). */ +export function legacyPgDeltaTempPath(path: Path.Path, workdir: string): string { + return path.join(workdir, "supabase", ".temp", "pgdelta"); +} + +/** + * Global shadow-baseline cache directory: + * `${SUPABASE_HOME}/cache/shadow-baseline` (default `~/.supabase/cache/shadow-baseline`). + * + * Pure: callers may pass `env`/`homeDir` for tests; production uses `process.env` and + * `os.homedir()`. + */ +export function legacyShadowBaselineCacheDir( + path: Path.Path, + env: Readonly> = process.env, + homeDir: string = homedir(), +): string { + return path.join(resolveSupabaseHome(env, homeDir), "cache", "shadow-baseline"); +} diff --git a/apps/cli/src/legacy/shared/legacy-vault.ts b/apps/cli/src/legacy/shared/legacy-vault.ts index 6b415baf80..6b596ab62d 100644 --- a/apps/cli/src/legacy/shared/legacy-vault.ts +++ b/apps/cli/src/legacy/shared/legacy-vault.ts @@ -24,9 +24,11 @@ export interface LegacyVaultSecret { readonly resolved: boolean; } -const READ_VAULT_KV = "SELECT id, name FROM vault.secrets WHERE name = ANY($1)"; -const UPDATE_VAULT_KV = "SELECT vault.update_secret($1, $2)"; -const CREATE_VAULT_KV = "SELECT vault.create_secret($1, $2)"; +// Exported for the shadow baseline cache's embedded-SQL digest (`shadow-cache.ts`), which must +// re-key whenever the SQL this module bakes into a baseline changes across CLI releases. +export const LEGACY_READ_VAULT_KV = "SELECT id, name FROM vault.secrets WHERE name = ANY($1)"; +export const LEGACY_UPDATE_VAULT_KV = "SELECT vault.update_secret($1, $2)"; +export const LEGACY_CREATE_VAULT_KV = "SELECT vault.create_secret($1, $2)"; /** * Upserts `[db.vault]` secrets into `vault.secrets`. Port of Go's @@ -46,7 +48,7 @@ export const legacyUpsertVaultSecrets = ( yield* output.raw("Updating vault secrets...\n", "stderr"); const existing = yield* session - .query(READ_VAULT_KV, [resolved.map((secret) => secret.name)]) + .query(LEGACY_READ_VAULT_KV, [resolved.map((secret) => secret.name)]) .pipe( Effect.mapError( (cause) => @@ -63,9 +65,9 @@ export const legacyUpsertVaultSecrets = ( for (const secret of resolved) { const id = existingByName.get(secret.name); if (id !== undefined) { - yield* session.query(UPDATE_VAULT_KV, [id, secret.value]); + yield* session.query(LEGACY_UPDATE_VAULT_KV, [id, secret.value]); } else { - yield* session.query(CREATE_VAULT_KV, [secret.value, secret.name]); + yield* session.query(LEGACY_CREATE_VAULT_KV, [secret.value, secret.name]); } } yield* session.exec("COMMIT"); diff --git a/apps/cli/tests/helpers/cli.ts b/apps/cli/tests/helpers/cli.ts index 5e043a3964..139c5b4528 100644 --- a/apps/cli/tests/helpers/cli.ts +++ b/apps/cli/tests/helpers/cli.ts @@ -254,6 +254,11 @@ export function spawnSupabase( SUPABASE_HOME: homeDir, SUPABASE_NO_KEYRING: "1", SUPABASE_TELEMETRY_DISABLED: "1", + // The shadow baseline cache (`db-bootstrap/shadow-cache.ts`) is ON by default. Off here so an + // e2e/live run observes the plain shadow lifecycle and never leaves a ~90MB tar in a temp + // project — a test whose subject IS the cache opts back in through `options.env`, which is + // spread after this. + SUPABASE_SHADOW_CACHE: "0", ...options?.env, }; if (entrypoint === "legacy") { diff --git a/apps/cli/tests/helpers/legacy-mocks.ts b/apps/cli/tests/helpers/legacy-mocks.ts index fd9b230930..e9b097a6df 100644 --- a/apps/cli/tests/helpers/legacy-mocks.ts +++ b/apps/cli/tests/helpers/legacy-mocks.ts @@ -4,8 +4,9 @@ import { join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { type ApiClient, makeApiClient, type SupabaseApiConfigError } from "@supabase/api/effect"; -import { Effect, FileSystem, Layer, Option, Redacted, Sink, Stream } from "effect"; +import { Effect, FileSystem, Layer, Option, Predicate, Redacted, Sink, Stream } from "effect"; import { PlatformError, SystemError } from "effect/PlatformError"; +import type { ChildProcess } from "effect/unstable/process"; import { ChildProcessSpawner } from "effect/unstable/process"; import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientError from "effect/unstable/http/HttpClientError"; @@ -40,6 +41,10 @@ import { LegacyLoginVerificationError, } from "../../src/legacy/commands/login/login.errors.ts"; import { LegacyCliConfig } from "../../src/legacy/config/legacy-cli-config.service.ts"; +import { + LEGACY_PGDATA_BASELINE_MARKER_NAME, + LEGACY_PGDATA_PATH, +} from "../../src/legacy/shared/db-bootstrap/pgdata-snapshot.ts"; import { legacyProjectRefLayer } from "../../src/legacy/config/legacy-project-ref.layer.ts"; import { LegacyLinkedProjectCache } from "../../src/legacy/telemetry/legacy-linked-project-cache.service.ts"; import { LegacyTelemetryState } from "../../src/legacy/telemetry/legacy-telemetry-state.service.ts"; @@ -733,6 +738,60 @@ export function useLegacyTempWorkdir(prefix = "supabase-legacy-test-"): { }; } +/** + * Sets `name` to `value` (or unsets it when `value` is `undefined`) for the duration of `body`, + * restoring whatever was there before — including whatever a surrounding `beforeEach` such as + * {@link useLegacyShadowCacheDisabled} put there. Scoping the override to the effect rather than + * to a nested `describe` keeps it independent of vitest's hook ordering, so a single test can + * opt back INTO a variable its file pins off. + */ +export const legacyWithEnv = ( + name: string, + value: string | undefined, + body: Effect.Effect, +): Effect.Effect => + Effect.acquireUseRelease( + Effect.sync(() => { + const previous = process.env[name]; + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + return previous; + }), + () => body, + (previous) => + Effect.sync(() => { + if (previous === undefined) delete process.env[name]; + else process.env[name] = previous; + }), + ); + +/** + * Pins `SUPABASE_SHADOW_CACHE=0` for every test in the calling file, restoring whatever the host + * had afterwards. Like {@link useLegacyTempWorkdir} it calls vitest's `beforeEach`/`afterEach` + * internally, so it must be invoked at module scope (or inside the surrounding `describe`). + * + * The shadow baseline cache (`db-bootstrap/shadow-cache.ts`) is ON by default and reads + * `process.env` directly, so ANY suite that provisions a shadow through + * `legacyWithShadowDatabase` with a mocked spawner — `db diff`, `db pull`, declarative sync — now + * exercises the cache path unless it opts out: the cold path adds a `docker stop`/`docker cp`/ + * `docker start` round trip and writes a ~90MB-shaped tar into the test's workdir. Suites whose + * subject is anything OTHER than the cache should call this so they keep asserting the plain + * container lifecycle; the cache's own suites deliberately do not. + */ +export function useLegacyShadowCacheDisabled(): void { + const name = "SUPABASE_SHADOW_CACHE"; + let previous: string | undefined; + beforeEach(() => { + previous = process.env[name]; + process.env[name] = "0"; + }); + afterEach(() => { + if (previous === undefined) delete process.env[name]; + else process.env[name] = previous; + previous = undefined; + }); +} + /** * Ambient isolation for tests that construct the REAL `legacyCliConfigLayer` / * `legacyCredentialsLayer` (directly or inside a command runtime layer) against @@ -754,8 +813,8 @@ export function legacyIsolatedHomeLayer( } // --------------------------------------------------------------------------- -// Failing filesystem — wraps the real Bun `FileSystem` and fails the Nth -// `writeFileString` call with a `PlatformError`, so cleanup-on-failure paths +// Failing filesystem — wraps the real Bun `FileSystem` and fails a chosen +// `writeFileString` with a `PlatformError`, so cleanup-on-failure paths // (e.g. the pg-delta multi-file migration writer) can be exercised // deterministically. Every other call delegates to the real filesystem, so // config reads / earlier writes behave normally. Merge this AFTER @@ -765,6 +824,23 @@ export function legacyIsolatedHomeLayer( export function legacyFailWriteStringOnNthCallFsLayer( failOnCall: number, +): Layer.Layer { + return legacyFailWriteStringFsLayer((_, calls) => calls === failOnCall); +} + +/** + * Same as {@link legacyFailWriteStringOnNthCallFsLayer}, but fails the first + * `writeFileString` whose path matches `match`. Prefer this when earlier + * setup writes (shadow SQL, branch markers) make a fixed call index brittle. + */ +export function legacyFailWriteStringMatchingFsLayer( + match: (path: string) => boolean, +): Layer.Layer { + return legacyFailWriteStringFsLayer((path) => match(path)); +} + +function legacyFailWriteStringFsLayer( + shouldFail: (path: string, calls: number) => boolean, ): Layer.Layer { return Layer.effect( FileSystem.FileSystem, @@ -774,7 +850,7 @@ export function legacyFailWriteStringOnNthCallFsLayer( ...real, writeFileString: (path, data, options) => { calls += 1; - if (calls === failOnCall) { + if (shouldFail(path, calls)) { return Effect.fail( new PlatformError( new SystemError({ @@ -957,6 +1033,334 @@ export function mockLegacyShadowContainerCliSpawner( return { layer, spawned }; } +// --------------------------------------------------------------------------- +// A minimal, stateful Docker model — the shadow BASELINE CACHE's round trip +// (`docker stop` -> `docker cp - :PGDATA` (the baseline stamp) -> +// `docker cp :PGDATA -` -> `docker start`, and the warm +// `docker cp - :` restore) really moves bytes, which +// `mockLegacyShadowContainerCliSpawner` above deliberately does not model. +// --------------------------------------------------------------------------- + +/** + * A real (if tiny) POSIX tar, byte for byte — `legacyValidatePgDataArchive` + * (`db-bootstrap/pgdata-snapshot.ts`) walks the header stream and checksum-validates every block + * before a warm restore, so the fake export has to be a genuine archive rather than a stand-in + * string. Every byte stays ASCII (padding is NUL), which is why the spawner's `TextEncoder` and + * the tests' `readFileString` round-trip it unchanged. + */ +const legacyFakeTarBlock = (fields: ReadonlyArray): string => { + const block = Array.from({ length: 512 }, () => "\0"); + for (const [offset, text] of fields) { + for (let index = 0; index < text.length; index += 1) { + block[offset + index] = text[index] ?? "\0"; + } + } + return block.join(""); +}; + +const legacyFakeTarOctal = (value: number, width: number) => + `${value.toString(8).padStart(width, "0")}\0`; + +/** One ustar member: a checksummed 512-byte header plus its NUL-padded content blocks. */ +const legacyFakeTarEntry = (name: string, content: string, typeFlag: "0" | "5"): string => { + const header = legacyFakeTarBlock([ + [0, name], + [100, legacyFakeTarOctal(typeFlag === "5" ? 0o755 : 0o600, 7)], + [108, legacyFakeTarOctal(0, 7)], + [116, legacyFakeTarOctal(0, 7)], + [124, legacyFakeTarOctal(content.length, 11)], + [136, legacyFakeTarOctal(0, 11)], + // The checksum is computed over the header with this field read as 8 spaces. + [148, " "], + [156, typeFlag], + [257, "ustar\0"], + [263, "00"], + ]); + let checksum = 0; + for (let index = 0; index < header.length; index += 1) checksum += header.charCodeAt(index); + const padding = "\0".repeat((512 - (content.length % 512)) % 512); + return `${header.slice(0, 148)}${legacyFakeTarOctal(checksum, 6)} ${header.slice(156)}${content}${padding}`; +}; + +/** Tar's end-of-archive marker: two all-zero blocks. */ +const LEGACY_FAKE_TAR_END = "\0".repeat(1024); + +/** + * What the fake `docker cp :PGDATA -` emits for a container the export path has NOT stamped — + * a real cluster (`data/` + `data/PG_VERSION`) and nothing more. Stands in both for a hand-placed + * bare PGDATA archive and for a snapshot taken before the platform baseline ran: it restores, + * starts, and answers, so only the missing marker tells it apart from a usable baseline. + */ +export const LEGACY_FAKE_UNSTAMPED_PGDATA_TAR = `${legacyFakeTarEntry("data/", "", "5")}${legacyFakeTarEntry("data/PG_VERSION", "17\n", "0")}${LEGACY_FAKE_TAR_END}`; + +/** + * The bytes the fake `docker cp :PGDATA -` emits once the export path has stamped the + * container — stands in for a real ~90MB PGDATA tar, with the same top-level `data/` member, + * `data/PG_VERSION` cluster file, and `data/SUPABASE_BASELINE` marker a real export carries. + * + * `markerContent` is whatever the export ACTUALLY stamped, carried through rather than fixed: the + * marker binds a snapshot to its cache key, so a fake that always emitted the same marker would + * make every key-binding assertion pass vacuously — a production regression stamping the wrong key + * (or a fixed one) has to show up as a warm-path mismatch here too. + */ +export const legacyFakePgDataTar = (markerContent: string): string => + `${legacyFakeTarEntry("data/", "", "5")}${legacyFakeTarEntry("data/PG_VERSION", "17\n", "0")}${legacyFakeTarEntry("data/SUPABASE_BASELINE", markerContent, "0")}${LEGACY_FAKE_TAR_END}`; + +/** + * The `SUPABASE_BASELINE` member's content inside a `docker cp - :` stamp payload — a + * plain 512-block walk over the archive `legacyPgDataBaselineMarkerTar` built, so the fake export + * below can carry the real stamp forward. `undefined` when the payload has no such member, which + * is the fake's "this container was never really stamped" signal. + */ +const legacyFakeStampedMarkerContent = (stamp: string): string | undefined => { + let offset = 0; + while (offset + 512 <= stamp.length) { + const header = stamp.slice(offset, offset + 512); + offset += 512; + const name = (header.slice(0, 100).split("\0")[0] ?? "").replace(/^\.\//u, ""); + if (name.length === 0) return undefined; // the end-of-archive marker + const size = Number.parseInt((header.slice(124, 136).split("\0")[0] ?? "").trim(), 8); + if (!Number.isInteger(size) || size < 0) return undefined; + if (name === LEGACY_PGDATA_BASELINE_MARKER_NAME) return stamp.slice(offset, offset + size); + offset += Math.ceil(size / 512) * 512; + } + return undefined; +}; + +/** + * A syntactically VALID tar carrying no members at all — what a replaced or truncated cache + * artifact looks like to `docker cp -`, which extracts it happily and lets the entrypoint `initdb` + * a fresh cluster over the top. The pre-restore marker check is what catches it. + */ +export const LEGACY_FAKE_EMPTY_TAR = LEGACY_FAKE_TAR_END; + +interface LegacyFakeContainer { + readonly labels: Readonly>; + readonly autoRemove: boolean; + running: boolean; + /** Whether this container has ever been `docker start`ed — distinguishes a RE-start for `failRestart`. */ + everStarted?: boolean; + /** What a previous `docker cp - :` unpacked into this container, if anything. */ + restored: string | undefined; + /** + * What a `docker cp - :` delivered — the export path's baseline stamp. Its presence + * is what makes the fake's copy-OUT emit a marked archive, so a production regression that stops + * stamping publishes {@link LEGACY_FAKE_UNSTAMPED_PGDATA_TAR} and every warm-path test notices. + */ + stamp: string | undefined; +} + +/** + * Every `docker` call a shadow provision issues, backed by ONE stateful container table: + * `create`/`start`/`stop`/`rm` mutate it, and `docker cp` really carries bytes in and out. Use + * this instead of {@link mockLegacyShadowContainerCliSpawner} whenever the shadow BASELINE CACHE + * is enabled for the test — the cold export's stop/copy-out/start round trip and the warm + * restore's copy-in have no meaning against a stateless spawner, and a failed export is + * fail-open (a warning, no tar), so a stateless fake makes cache assertions pass vacuously. + * + * `container inspect supabase_db_` reports "no such container" here (the table only + * holds shadows), i.e. the local stack is DOWN — fine for every non-`--use-pgadmin` path, which + * never issues that probe. + */ +export function mockLegacyDockerDaemonCliSpawner( + opts: { + readonly failStart?: boolean; + /** Fails only RE-starts (a `docker start` after the container has already run once) — the cold export's revive step. */ + readonly failRestart?: boolean; + readonly failCopyOut?: boolean; + readonly failCopyIn?: boolean; + /** Fails the export path's baseline stamp (`docker cp - :`) only. */ + readonly failStamp?: boolean; + } = {}, +) { + const containers = new Map(); + const spawned: Array> = []; + const encoder = new TextEncoder(); + const decoder = new TextDecoder(); + let nextId = 0; + + /** + * Drains a `Stream` passed as a command's `stdin` into a string, the way the real + * `NodeChildProcessSpawner` runs a user-supplied stdin stream into the child's stdin sink — a + * fake that ignored it would never notice the restore tar was not actually delivered. + */ + const readStdin = Effect.fnUntraced(function* (command: ChildProcess.Command) { + const configured = command._tag === "StandardCommand" ? command.options.stdin : undefined; + // A caller may pass either a bare `CommandInput` or a `StdinConfig` wrapping one. + const input = Stream.isStream(configured) + ? configured + : Predicate.hasProperty(configured, "stream") + ? configured.stream + : undefined; + if (!Stream.isStream(input)) return ""; + return yield* Stream.runFold( + input, + () => "", + (text: string, chunk) => text + decoder.decode(chunk as Uint8Array, { stream: true }), + ).pipe(Effect.orElseSucceed(() => "")); + }); + + const spawner = ChildProcessSpawner.make((command) => + Effect.gen(function* () { + const args = command._tag === "StandardCommand" ? command.args : []; + spawned.push(args); + + let exitCode = 0; + let stdout = ""; + let stderr = ""; + + if (args[0] === "network" && args[1] === "inspect") { + exitCode = 1; + } else if (args[0] === "create") { + nextId += 1; + const id = `shadowcontainer${String(nextId).padStart(2, "0")}`.padEnd(64, "0"); + const labels: Record = {}; + for (let index = 0; index < args.length; index += 1) { + if (args[index] !== "--label") continue; + const [key = "", ...rest] = (args[index + 1] ?? "").split("="); + labels[key] = rest.join("="); + } + containers.set(id, { + labels, + autoRemove: args.includes("--rm"), + running: false, + restored: undefined, + stamp: undefined, + }); + stdout = id; + } else if (args[0] === "start") { + const container = containers.get(args[1] ?? ""); + if ( + opts.failStart === true || + container === undefined || + (opts.failRestart === true && container.everStarted) + ) { + exitCode = 1; + } else { + container.running = true; + container.everStarted = true; + } + } else if (args[0] === "stop") { + const id = args[1] ?? ""; + const container = containers.get(id); + if (container === undefined) exitCode = 1; + else { + container.running = false; + // Docker destroys an `--rm` container the moment it exits, `docker stop` included — + // verified against Docker 29. This is exactly why the cold export path drops `--rm`. + if (container.autoRemove) containers.delete(id); + } + } else if (args[0] === "rm") { + containers.delete(args[args.length - 1] ?? ""); + } else if (args[0] === "cp" && args[1] === "-") { + // Three different stdin copies share this form, so each failure switch has to pick out + // exactly one: the pgsodium root key goes to `:/`, the export path's baseline stamp to + // `:`, and the warm restore to `:`. Otherwise a warm + // fallback test would kill the root-key copy and never reach the archive. + const [id = "", containerPath = ""] = (args[2] ?? "").split(":"); + const container = containers.get(id); + const received = yield* readStdin(command); + const isSecret = containerPath === "" || containerPath === "/"; + const isStamp = containerPath === LEGACY_PGDATA_PATH; + const failed = + (isStamp && opts.failStamp === true) || + (!isSecret && !isStamp && opts.failCopyIn === true); + if (container === undefined || failed) { + exitCode = 1; + stderr = "no such container"; + } else if (isStamp) { + container.stamp = received; + } else if (!isSecret) { + container.restored = `${containerPath}::${received}`; + } + } else if (args[0] === "cp" && args[2] === "-") { + // Export: `docker cp : -`, tar on stdout. What comes out depends on + // whether the container was stamped first — that is the whole point of the marker. + const [id = ""] = (args[1] ?? "").split(":"); + const container = containers.get(id); + if (opts.failCopyOut === true || container === undefined) { + exitCode = 1; + stderr = "no such container"; + } else { + // The marker the stamp really delivered rides along into the exported archive; a + // container that was never stamped (or whose stamp carried no marker member) exports the + // bare cluster the warm path must reject. + const marker = + container.stamp === undefined + ? undefined + : legacyFakeStampedMarkerContent(container.stamp); + stdout = + marker === undefined ? LEGACY_FAKE_UNSTAMPED_PGDATA_TAR : legacyFakePgDataTar(marker); + } + } else if (args[0] === "container" && args[1] === "inspect") { + const container = containers.get(args[2] ?? ""); + exitCode = container === undefined ? 1 : 0; + stdout = + container === undefined + ? "" + : JSON.stringify({ + Running: container.running, + Status: container.running ? "running" : "exited", + Health: { Status: "healthy" }, + }); + } + + return ChildProcessSpawner.makeHandle({ + pid: ChildProcessSpawner.ProcessId(1), + // `docker cp … -` writes the raw archive, every other call a trailing newline. + stdout: Stream.fromIterable(stdout.length > 0 ? [encoder.encode(stdout)] : []), + stderr: Stream.fromIterable(stderr.length > 0 ? [encoder.encode(stderr)] : []), + all: Stream.empty, + exitCode: Effect.succeed(ChildProcessSpawner.ExitCode(exitCode)), + isRunning: Effect.succeed(false), + stdin: Sink.drain, + kill: () => Effect.void, + unref: Effect.succeed(Effect.void), + getInputFd: () => Sink.drain, + getOutputFd: () => Stream.empty, + }); + }), + ); + + /** + * One readable label per Docker call, so a test can assert the SEQUENCE of meaningful steps + * rather than raw argv. `cp` is split four ways because the shadow issues four different copies: + * the pgsodium root key every shadow gets (`cp-secret`, `container-lifecycle.ts`), the baseline + * marker stamped into PGDATA just before an export (`cp-stamp`), the baseline export (`cp-out`), + * and the baseline restore (`cp-in`). + */ + const stepOf = (args: ReadonlyArray): string => { + if (args[0] === "network") return "network"; + if (args[0] === "container" && args[1] === "inspect") return "inspect"; + if (args[0] === "cp") { + if (args[1] === "-") { + const dest = args[2] ?? ""; + const containerPath = dest.slice(dest.indexOf(":") + 1); + if (containerPath === "" || containerPath === "/") return "cp-secret"; + return containerPath === LEGACY_PGDATA_PATH ? "cp-stamp" : "cp-in"; + } + if (args[2] === "-") return "cp-out"; + return "cp-secret"; + } + return args[0] ?? ""; + }; + + return { + /** Ready to merge into a test runtime; `spawner` is the bare handler for direct callers. */ + layer: Layer.succeed(ChildProcessSpawner.ChildProcessSpawner, spawner), + spawner, + spawned, + containers, + /** Every argv whose first token is `verb` (`create`/`rm`/`stop`/`start`/`cp`). */ + calls: (verb: string) => spawned.filter((args) => args[0] === verb), + /** Every argv classified as `step` — see {@link stepOf}. */ + stepCalls: (step: string) => spawned.filter((args) => stepOf(args) === step), + /** The full step sequence, with the `network` bookkeeping calls dropped as noise. */ + steps: () => spawned.map(stepOf).filter((step) => step !== "network"), + ids: () => [...containers.keys()], + }; +} + // --------------------------------------------------------------------------- // Runtime composition — bundles the entire Layer.mergeAll(...) graph that // every native-port integration test re-builds, including the easy-to-mis-wire diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md new file mode 100644 index 0000000000..760e1df470 --- /dev/null +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -0,0 +1,100 @@ +# pg-delta / shadow-database follow-ups + +Deferred review findings that are valid but out of scope for the PR that surfaced them. +Each entry names the PR it came out of so the context is recoverable. + +## ~~Shadow baseline cache key does not cover CLI-embedded init SQL~~ (resolved in PR #6184) + +Resolved on the PR itself after depthfirst independently flagged it: the key now folds in a +digest of the embedded init/privilege SQL constants (`LEGACY_SHADOW_BASELINE_SQL_DIGEST`, +`shadow-cache.ts`). Original write-up kept below for the rationale and the alternative considered. + +## Original: shadow baseline cache key does not cover CLI-embedded init SQL (PR #6184) + +The cache key (`legacyShadowCacheKey`, `apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts`) +hashes every *config-derived* input baked into the shadow cluster, but not the CLI-embedded init +SQL the entrypoint heredocs into the cluster at `initdb` time (`LEGACY_START_DB_SCHEMA_SQL`, +`LEGACY_START_DB_WEBHOOK_SQL`, `LEGACY_START_DB_SUPABASE_SQL` — `postgres.service.ts`), nor the +baseline steps `legacySetupDatabase` itself performs (API privileges SQL, vault upsert SQL). If a +CLI release changes any of those without a `supabase/postgres` image bump, a warm tar produced by +the older CLI silently restores the older baseline. + +Realistic frequency is low (these constants track Go's `start.go` templates and change rarely, +usually alongside image bumps), which is why it did not block #6184. Two candidate fixes: + +- Fold the CLI version into the key — over-invalidates once per release (~one 15s cold run per + upgrade), trivially safe, one line. +- Hash the embedded SQL constants themselves — precise, no per-release invalidation, slightly more + surface. + +Either way, add a unit-test mutation case alongside the existing ones in +`shadow-cache.unit.test.ts`. + +## ~~Make the baseline/declarative catalog shadows use `legacyWaitForShadowReady`~~ (resolved in PR #6184) + +Resolved on the PR itself: both catalog provisioners now wait with `legacyWaitForShadowReady` +(`legacy-pgdelta.cache.ts`). Original write-up kept below. + +## Original: make the baseline/declarative catalog shadows use `legacyWaitForShadowReady` (PR #6184 × CLI-1970 merge) + +CLI-1970 (#6162) made `legacyExportBaselineCatalogRef`/`legacyExportDeclarativeCatalogRef` +(`legacy-pgdelta.cache.ts`) native. `legacySetupShadowDatabase` is now baseline-state-aware +(skip when `baselinePresent`; snapshot on a cache-enabled cold provision), and those catalog +provisions thread the acquire handle through, so a warm hit no longer double-applies the +baseline. They still wait with `legacyWaitForHealthyServices` (docker healthcheck, 10s interval) +instead of `legacyWaitForShadowReady` (container-state + short connect), so a warm restore can +sit until the first healthcheck tick. + +The bundled pg-delta next sync/diff shadows (`legacy-pgdelta-next-shadow.layer.ts`) already use +the cache-aware acquire + `legacyWaitForShadowReady`. + +## Shadow-cache robustness follow-ups from the #6184 review (Codex, deferred) + +Five valid-but-deferred findings from the #6184 review rounds. None block the feature: each is an +edge on an already-degraded path (infra failures, races) where the cache's fail-open design keeps +the command correct, at worst at cold-provision speed. + +- **Warm readiness failures always mark the tar suspect** (`shadow-cache.ts`, + `legacyWarmShadow`): a Docker inspect/daemon hiccup during the restored container's readiness + wait reaches the unconditional `tarSuspect: true` mapping, so a valid tar can be deleted on a + pure infra blip (the cold fallback then re-exports it, so the cost is one cold run). Fix: only + set `tarSuspect` when the container stayed inspectable/running and Postgres itself failed + readiness. (Codex comment 3786040107.) +- **Snapshot-revive failures all report `reason: "docker_daemon"`** (`shadow-cache.ts`, + `legacyExportShadowBaseline`): the post-snapshot `docker start` and the follow-up readiness + wait share one error mapping, so a Postgres-side failure after a successful start is + fingerprinted as a daemon outage in telemetry. Fix: map the two stages separately. + (Codex comment 3786040112.) +- **A failed warm restore can leak the created container** (`container-lifecycle.ts` + + `shadow-cache.ts`): `legacyCreateContainer`'s post-create cleanup covers a failed archive + extraction, but a failure in the secret-file copy or the final `docker start` leaves the + created container behind while the warm→cold fallback retries with a replacement. At most one + stopped project-labeled container; `supabase stop` sweeps it. Fix: extend the tapError cleanup + to every post-create step. (Codex comment 3789148842.) +- **Cache-key JWKS resolution runs before the `Initialising schema...` banner** + (`shadow-cache.ts`, `legacyResolveShadowCacheKeyInputs`): a third-party JWKS discovery failure + on a cache-eligible acquire surfaces before the banner that the uncached flow prints first + (CLI-1956 shape). Fix: defer the key's JWKS resolution until after the prelude has printed, + reusing the resolved value for setup. (Codex comment 3789363067.) +- **`roles.sql` TOCTOU between key hash and execution** (`shadow-cache.ts` + + `db-setup.ts`): the key hashes `roles.sql` once, and `legacySetupDatabase` re-reads the path + later; an edit in that window publishes a tar under a key describing the old contents. Fix: + thread the captured contents into setup, or re-hash before publishing. (Codex comment + 3789363070.) +- **Warm restores keep the cold container's id as Realtime's seeded `DB_HOST`** + (`shadow-database.ts` + `realtime-env.ts`): the cold one-shot Realtime job seeds + `_realtime.extensions` with the exporting container's 12-char id, and a warm restore (different + container, job skipped) leaves that dead id in the encrypted settings. Only migrations that + inspect or act on Realtime tenant configuration can observe the difference. Fix: normalize the + seeded host before snapshotting, or refresh it after restore. (Codex comment 3789481478.) +- **Cache key hashes image tags, not immutable digests** (`shadow-cache.ts`, + `legacyShadowCacheKey`): a registry tag republished with different bytes (or a locally retagged + image) keeps the same key, so a warm hit would restore the previous image's baseline and skip + the updated service migrations. Accepted risk: every keyed image is an exact pinned version tag + the release pipeline treats as immutable, a changed `SUPABASE_INTERNAL_IMAGE_REGISTRY` is + already in the key, and the 14-day TTL bounds staleness. Hashing digests is structurally + costly — the key is computed before images are pulled, so `docker image inspect` would have to + either move pulls ahead of the cache decision or fall back to tags for unpulled images (making + the first published tar never warm-hit), and it adds per-acquire inspect round-trips to the + warm path. Revisit only if the image-resolution pipeline ever produces mutable tags (`latest`, + branch tags) — those should become cache-ineligible at that point. (Codex comment 3802804637.)