From 1baf8ea1066352a6b3d4497a1087fef1e25fcc95 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 6 Aug 2026 09:04:30 +0200 Subject: [PATCH 01/82] feat(cli): bundle pg-delta next engine --- apps/cli-go/CONTRIBUTING.md | 56 +- apps/cli-go/cmd/db.go | 12 +- apps/cli-go/docs/supabase/db/diff.md | 4 + apps/cli-go/docs/supabase/db/pull.md | 15 +- .../db/schema-declarative-generate.md | 4 + .../supabase/db/schema-declarative-sync.md | 4 + apps/cli-go/internal/db/diff/shadow.go | 85 +++ apps/cli-go/internal/db/diff/shadow_test.go | 116 ++++ apps/cli/docs/go-cli-porting-status.md | 14 +- apps/cli/package.json | 2 + .../legacy/commands/bootstrap/SIDE_EFFECTS.md | 36 +- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 64 +- .../legacy/commands/db/diff/diff.handler.ts | 212 ++++-- .../commands/db/diff/diff.integration.test.ts | 278 ++++++-- .../legacy/commands/db/diff/diff.layers.ts | 19 +- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 45 +- .../legacy/commands/db/pull/pull.handler.ts | 228 ++++--- .../commands/db/pull/pull.integration.test.ts | 190 +++++- .../legacy/commands/db/pull/pull.layers.ts | 18 +- .../legacy/commands/db/push/SIDE_EFFECTS.md | 65 +- .../commands/db/push/push.integration.test.ts | 48 +- ...eclarative.orchestrate.integration.test.ts | 134 +++- .../declarative/declarative.orchestrate.ts | 102 ++- .../declarative/declarative.smart-target.ts | 64 +- .../declarative/generate/SIDE_EFFECTS.md | 66 +- .../declarative/generate/generate.handler.ts | 28 +- .../generate/generate.integration.test.ts | 52 +- .../declarative/generate/generate.layers.ts | 20 +- .../schema/declarative/sync/SIDE_EFFECTS.md | 68 +- .../schema/declarative/sync/sync.handler.ts | 63 +- .../declarative/sync/sync.integration.test.ts | 78 ++- .../db/schema/declarative/sync/sync.layers.ts | 13 + .../db/shared/legacy-pgdelta-engine.layer.ts | 67 ++ .../legacy-pgdelta-engine.layer.unit.test.ts | 196 ++++++ .../legacy-pgdelta-engine.legacy.layer.ts | 183 ++++++ .../legacy-pgdelta-engine.next.layer.ts | 368 +++++++++++ .../legacy-pgdelta-engine.next.unit.test.ts | 44 ++ .../shared/legacy-pgdelta-engine.service.ts | 135 ++++ .../db/shared/legacy-pgdelta-files.ts | 168 +++++ .../shared/legacy-pgdelta-migrations.write.ts | 12 +- .../legacy-pgdelta-next-adapter.layer.ts | 615 ++++++++++++++++++ .../legacy-pgdelta-next-adapter.service.ts | 189 ++++++ .../legacy-pgdelta-next-adapter.unit.test.ts | 520 +++++++++++++++ .../shared/legacy-pgdelta-next-artifacts.ts | 81 +++ ...legacy-pgdelta-next-artifacts.unit.test.ts | 74 +++ .../shared/legacy-pgdelta-next-diagnostics.ts | 30 + ...gacy-pgdelta-next-diagnostics.unit.test.ts | 58 ++ .../db/shared/legacy-pgdelta-next-flag.ts | 22 + .../legacy-pgdelta-next-flag.unit.test.ts | 23 + .../legacy-pgdelta-next-shadow.layer.ts | 54 ++ .../legacy-pgdelta-next-shadow.service.ts | 32 + .../legacy-pgdelta-next-shadow.unit.test.ts | 154 +++++ .../shared/legacy-pgdelta-next.live.test.ts | 448 +++++++++++++ .../db/shared/legacy-pgdelta.seam.layer.ts | 6 +- .../db/shared/legacy-pgdelta.seam.service.ts | 13 +- .../commands/db/shared/legacy-pgdelta.ts | 2 +- .../db/shared/legacy-pgdelta.write.ts | 33 +- .../shared/legacy-pgdelta.write.unit.test.ts | 32 +- .../shared/legacy-db-config.toml-read.ts | 34 +- ...y-db-connection.sql-pg.integration.test.ts | 39 +- .../legacy-db-connection.sql-pg.layer.ts | 117 ++-- .../src/legacy/shared/legacy-db-push-core.ts | 9 +- apps/cli/src/legacy/shared/legacy-seed-ops.ts | 3 + apps/cli/tests/helpers/live.ts | 18 + pnpm-lock.yaml | 251 +++++++ pnpm-workspace.yaml | 2 + 66 files changed, 5696 insertions(+), 539 deletions(-) create mode 100644 apps/cli-go/internal/db/diff/shadow_test.go create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-flag.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-flag.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts diff --git a/apps/cli-go/CONTRIBUTING.md b/apps/cli-go/CONTRIBUTING.md index 39d0d33d85..25fa9709eb 100644 --- a/apps/cli-go/CONTRIBUTING.md +++ b/apps/cli-go/CONTRIBUTING.md @@ -44,9 +44,48 @@ The Supabase API client is generated from OpenAPI spec. See [our guide](api/READ ## Testing local pg-delta builds -To exercise unpublished `@supabase/pg-delta` changes inside CLI edge-runtime scripts (`db pull`, `db diff`, `db push`, etc.), publish a local build via Verdaccio in [pg-toolbelt](https://github.com/supabase/pg-toolbelt) and point the CLI at that registry. - -### 1. Start Verdaccio (pg-toolbelt) +Pg-delta and pg-topo run in-process by default and are bundled into the CLI binary. +Their versions are fixed by `apps/cli/package.json` and the lockfile at CLI build +time. `PGDELTA_NPM_REGISTRY` and a test project's +`supabase/.temp/pgdelta-version` do not affect this implementation. + +### Default bundled implementation + +To exercise unpublished [pg-toolbelt](https://github.com/supabase/pg-toolbelt) +changes locally: + +1. Build the `@supabase/pg-delta` and `@supabase/pg-topo` packages in pg-toolbelt. +2. Temporarily point both dependencies in `apps/cli/package.json` at those local + package directories (for a sibling checkout, use + `file:../../../pg-toolbelt/packages/pg-delta` and + `file:../../../pg-toolbelt/packages/pg-topo`). +3. Run `pnpm install` from the CLI repository root, then run the CLI from source + with `pnpm --dir apps/cli dev:legacy -- db diff ...`, or rebuild the legacy + binary before testing it. + +Both packages must be updated together so local source runs and built binaries +use the same planner/reorder implementation. Restore the package references and +lockfile after local testing. New-engine SQL need not byte-match the legacy +renderer; verify that the SQL executes and a subsequent operation converges to +an empty diff. + +When `PGDELTA_DEBUG` is enabled, the bundled engine writes non-reusable debug +artifacts under `supabase/.temp/pgdelta/v2/debug//`: `metadata.json` plus +`source-snapshot.json`, `desired-snapshot.json`, `plan.json`, and +`diagnostics.json` when those values are available. Legacy catalogs remain at +the `supabase/.temp/pgdelta/` root and are never consumed by the bundled engine. +For declarative `generate` and `sync`, `--no-cache` bypasses legacy catalog +reuse/warming; the bundled engine already extracts live state and has no +reusable catalog cache. + +### Legacy edge-runtime implementation + +To exercise unpublished legacy `@supabase/pg-delta` changes inside edge-runtime +scripts, select `SUPABASE_USE_PG_DELTA_NEXT=false`, publish a local build via +Verdaccio, and point the CLI at that registry. These instructions describe the +temporary compatibility implementation only. + +#### 1. Start Verdaccio (pg-toolbelt) ```sh cd pg-toolbelt @@ -55,7 +94,7 @@ bun run verdaccio:start Verdaccio listens on `http://localhost:4873`. `@supabase/*` packages you publish locally are served from local storage; other `@supabase/*` dependencies (for example `@supabase/pg-topo`) are proxied to npmjs. -### 2. Publish a local pg-delta build +#### 2. Publish a local pg-delta build After changing `packages/pg-delta`: @@ -68,7 +107,7 @@ This publishes a fresh `0.0.0-local.` version and restores `package.j Re-run whenever you change pg-delta source. -### 3. Run the CLI against the local registry +#### 3. Run the CLI against the local registry Set `PGDELTA_NPM_REGISTRY` to a URL reachable **from inside the edge-runtime Docker container**: @@ -79,6 +118,8 @@ export PGDELTA_NPM_REGISTRY=http://host.docker.internal:4873 # Linux (Docker 20.10+) export PGDELTA_NPM_REGISTRY=http://host.docker.internal:4873 # or: export PGDELTA_NPM_REGISTRY=http://172.17.0.1:4873 + +export SUPABASE_USE_PG_DELTA_NEXT=false ``` Then run any pg-delta-backed command, for example: @@ -89,4 +130,7 @@ supabase db pull --db-url "$DATABASE_URL" --diff-engine pg-delta When set, the CLI injects a scoped `.npmrc` and forwards `NPM_CONFIG_REGISTRY` into the edge-runtime container (`PgDeltaNpmRegistryOption` in `internal/utils/pgdelta_local.go`). -Unset `PGDELTA_NPM_REGISTRY` to return to the npmjs version pinned in config / `supabase/.temp/pgdelta-version`. +Unset `PGDELTA_NPM_REGISTRY` to return to the legacy npmjs version pinned in +config / `supabase/.temp/pgdelta-version`. Unset +`SUPABASE_USE_PG_DELTA_NEXT` (or set it to `true`) to return to the bundled +default implementation. diff --git a/apps/cli-go/cmd/db.go b/apps/cli-go/cmd/db.go index 66df4311cc..fedb35b30c 100644 --- a/apps/cli-go/cmd/db.go +++ b/apps/cli-go/cmd/db.go @@ -247,6 +247,16 @@ var ( if err := flags.LoadConfig(fsys); err != nil { return err } + if shadowMode == "pgdelta-next" { + nextShadow, err := diff.PreparePgDeltaNextShadow(cmd.Context(), fsys) + if err != nil { + return err + } + fmt.Println(nextShadow.Container) + fmt.Println(utils.ToPostgresURLWithoutPassword(nextShadow.Migrated)) + fmt.Println(utils.ToPostgresURLWithoutPassword(nextShadow.Scratch)) + return nil + } var src diff.ShadowSource var err error switch shadowMode { @@ -680,7 +690,7 @@ func init() { dbCmd.AddCommand(dbPullCmd) // Build hidden shadow-provisioning seam command shadowFlags := dbShadowCmd.Flags() - shadowFlags.StringVar(&shadowMode, "mode", "diff", "Shadow mode: diff (baseline + migrations) or declarative (bare shadow).") + shadowFlags.StringVar(&shadowMode, "mode", "diff", "Shadow mode: diff (baseline + migrations), declarative (bare shadow), or pgdelta-next (migrated + empty scratch).") shadowFlags.BoolVar(&shadowTargetLocal, "target-local", false, "Whether the diff target is the local database (enables the declarative-schema branch).") shadowFlags.BoolVar(&shadowUsePgDelta, "use-pg-delta", false, "Whether pg-delta is the active diff engine (selects the declarative-apply path).") shadowFlags.StringSliceVarP(&shadowSchema, "schema", "s", []string{}, "Comma separated list of schema to include.") diff --git a/apps/cli-go/docs/supabase/db/diff.md b/apps/cli-go/docs/supabase/db/diff.md index 0c0cf05a4d..497d371a95 100644 --- a/apps/cli-go/docs/supabase/db/diff.md +++ b/apps/cli-go/docs/supabase/db/diff.md @@ -10,8 +10,12 @@ By default, all schemas in the target database are diffed. Use the `--schema pub Projects created by a recent `supabase init` default to the pg-delta diff engine (`[experimental.pgdelta] enabled = true` in `config.toml`). Existing projects are unaffected and keep using migra unless they opt in. To fall back to the legacy migra engine, set `enabled = false` under `[experimental.pgdelta]`, or pass `--use-migra` for a single run. +The pg-delta engine runs in-process by default and is bundled into the CLI together with pg-topo at build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy edge-runtime implementation. `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs under `supabase/.temp/pgdelta/` affect only that opt-out; there is no automatic fallback. + With the pg-delta engine the diff SQL is formatted by default with the same settings the declarative export uses (uppercase keywords, wrapped at a max width of 180, indented and column-aligned); execution-aware transaction boundaries are preserved as per-unit header comments in the output. Configure overrides with `[experimental.pgdelta] format_options`, or set `format_options = "null"` to emit raw, unformatted statements. +The bundled and legacy renderers can produce different SQL bytes or file segmentation. The compatibility contract is executable SQL and convergence: after applying the result, a subsequent diff should be empty. With `PGDELTA_DEBUG=1`, bundled-engine snapshots, plans, and diagnostics are stored under `supabase/.temp/pgdelta/v2/debug//`; those files are diagnostic artifacts, not reusable catalogs. + While the diff command is able to capture most schema changes, there are cases where it is known to fail. Currently, this could happen if you schema contains: - Changes to publication diff --git a/apps/cli-go/docs/supabase/db/pull.md b/apps/cli-go/docs/supabase/db/pull.md index e10c0679f7..50c01be1ac 100644 --- a/apps/cli-go/docs/supabase/db/pull.md +++ b/apps/cli-go/docs/supabase/db/pull.md @@ -12,6 +12,8 @@ If no entries exist in the migration history table, the default diff engine uses Pass `--diff-engine pg-delta` to keep the migration-file `db pull` workflow while using pg-delta for the shadow diff step. On initial pull, pg-delta replaces `pg_dump` and produces the full migration from the shadow diff alone. Pass `--declarative` to switch to the declarative pg-delta export workflow instead. +Pg-delta runs in-process by default and is bundled with pg-topo at CLI build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily use the legacy edge-runtime implementation. `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs directly under `supabase/.temp/pgdelta/` affect only that opt-out; the CLI never falls back automatically. + pg-delta plans are execution-aware: when a plan crosses a transaction boundary — for example `ALTER TYPE ... ADD VALUE` followed by a statement that uses the new enum value, which cannot run in the same transaction — `db pull` writes one ordered migration file per plan unit instead of a single file (for example `_remote_schema_schema_changes.sql` and `_remote_schema_after_enum_values.sql`), each recorded in the migration history. The common case (a single unit) still produces exactly one `_remote_schema.sql` file. By default the emitted SQL is formatted with the same settings the declarative export uses (uppercase keywords, wrapped at a max width of 180, indented and column-aligned). Configure overrides with `[experimental.pgdelta] format_options` in `config.toml`, or set `format_options = "null"` to opt out and emit raw, unformatted statements. @@ -28,7 +30,16 @@ If `db pull --diff-engine pg-delta` reports `No schema changes found` but you ex PGDELTA_DEBUG=1 supabase db pull --db-url "$DATABASE_URL" --diff-engine pg-delta ``` -When pg-delta returns zero statements, the CLI writes a debug bundle under `supabase/.temp/pgdelta/debug//`: +The bundled engine writes a debug bundle under `supabase/.temp/pgdelta/v2/debug//` and includes its path in the empty-pull error. It contains `metadata.json` and, when available: + +- `source-snapshot.json` — serialized shadow database state +- `desired-snapshot.json` — serialized remote database state +- `plan.json` — serialized pg-delta plan +- `diagnostics.json` — extraction/planning diagnostics + +These files are diagnostic artifacts and are never reused as catalogs. New-engine SQL bytes and transaction-split filenames may differ from the legacy renderer; successful execution and an empty subsequent pull/diff are the contract. + +Under `SUPABASE_USE_PG_DELTA_NEXT=false`, the CLI instead writes the legacy debug bundle under `supabase/.temp/pgdelta/debug//`: - `source-catalog.json` — shadow database baseline pg-delta extracted - `target-catalog.json` — remote database pg-delta extracted @@ -36,6 +47,6 @@ When pg-delta returns zero statements, the CLI writes a debug bundle under `supa - `connection.txt` — redacted connection metadata - `error.txt` — error summary -Catalog files are not written during normal `db pull` runs. The `.temp/pgdelta` directory is also used by migration catalog caching (`db push`, local `db start`) when `[experimental.pgdelta] enabled = true`. +Legacy catalog files are not written during normal default-engine `db pull` runs. The `.temp/pgdelta` root is used only by legacy compatibility paths; default-engine artifacts are generation-separated under `.temp/pgdelta/v2/`. For TLS tracing without disabling SSL, use `SUPABASE_SSL_DEBUG=true` alongside `PGDELTA_DEBUG=1`. diff --git a/apps/cli-go/docs/supabase/db/schema-declarative-generate.md b/apps/cli-go/docs/supabase/db/schema-declarative-generate.md index 6c39004e5e..d82cae431a 100644 --- a/apps/cli-go/docs/supabase/db/schema-declarative-generate.md +++ b/apps/cli-go/docs/supabase/db/schema-declarative-generate.md @@ -4,4 +4,8 @@ Generate declarative schema files from a database. Exports the schema of a live database (local, linked, or custom URL) into SQL files under the declarative schema directory. This is the entrypoint for bootstrapping declarative mode. +Pg-delta and pg-topo run in-process and are bundled into the CLI at build time. The export includes `.pgdelta-export.json` policy metadata. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy catalog/edge-runtime implementation; `PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs at the `.temp/pgdelta/` root are legacy-only. + +`--no-cache` bypasses legacy catalog reuse/warming. The bundled engine always extracts live state and has no reusable catalog cache. With `PGDELTA_DEBUG=1`, structured diagnostics are written under `.temp/pgdelta/v2/debug//`. SQL bytes and grouping may differ between engines; reloading the export to the same managed state is the contract. + Requires `--experimental` flag or `[experimental.pgdelta] enabled = true` in config. diff --git a/apps/cli-go/docs/supabase/db/schema-declarative-sync.md b/apps/cli-go/docs/supabase/db/schema-declarative-sync.md index 1932b16f11..a6cf5e5729 100644 --- a/apps/cli-go/docs/supabase/db/schema-declarative-sync.md +++ b/apps/cli-go/docs/supabase/db/schema-declarative-sync.md @@ -4,4 +4,8 @@ Generate a new migration by diffing your declarative schema files against the cu When no declarative schema exists yet, the command offers to run `generate` first. After computing the diff, you can optionally name the migration and apply it to the local database. +Pg-delta and pg-topo run in-process and are bundled into the CLI at build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy catalog/edge-runtime implementation; `PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs at the `.temp/pgdelta/` root are legacy-only. + +`--no-cache` bypasses legacy catalog reuse/warming; the bundled engine extracts current state and has no reusable catalog cache. It may emit multiple ordered migration files to preserve transaction boundaries. SQL bytes may differ from the legacy renderer; successful application followed by an empty sync is the contract. With `PGDELTA_DEBUG=1`, snapshots, the plan, and diagnostics are written under `.temp/pgdelta/v2/debug//`. + Requires `--experimental` flag or `[experimental.pgdelta] enabled = true` in config. diff --git a/apps/cli-go/internal/db/diff/shadow.go b/apps/cli-go/internal/db/diff/shadow.go index 2ebd13591f..eba3fb2358 100644 --- a/apps/cli-go/internal/db/diff/shadow.go +++ b/apps/cli-go/internal/db/diff/shadow.go @@ -2,9 +2,11 @@ package diff import ( "context" + "time" "github.com/jackc/pgconn" "github.com/jackc/pgx/v4" + "github.com/pkg/errors" "github.com/spf13/afero" "github.com/supabase/cli/internal/db/start" "github.com/supabase/cli/internal/pgdelta" @@ -28,6 +30,89 @@ type ShadowSource struct { TargetOverride *pgconn.Config } +// PgDeltaNextShadow is a provisioned shadow container exposing both database +// states needed by the native pg-delta engine. Migrated contains the platform +// baseline plus local migrations. Scratch is an empty sibling database owned +// by pg-delta's declarative planner while it loads the desired schema files. +type PgDeltaNextShadow struct { + // Container is left running for the caller, which MUST remove it after use. + Container string + Migrated pgconn.Config + Scratch pgconn.Config +} + +type pgDeltaNextShadowDependencies struct { + create func(context.Context, uint16) (string, error) + wait func(context.Context, time.Duration, ...string) error + migrate func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error + createScratch func(context.Context, ...func(*pgx.ConnConfig)) error + remove func(string) +} + +const createPgDeltaNextScratch = "CREATE DATABASE pgdelta_declarative TEMPLATE template0" + +// createPgDeltaNextScratchDatabase creates the empty same-cluster database that +// planSchemaFiles owns. Using template0 guarantees it does not inherit the +// platform baseline or local migrations from postgres. +func createPgDeltaNextScratchDatabase(ctx context.Context, options ...func(*pgx.ConnConfig)) error { + conn, err := ConnectShadowDatabase(ctx, 10*time.Second, options...) + if err != nil { + return err + } + defer conn.Close(context.Background()) + if _, err := conn.Exec(ctx, createPgDeltaNextScratch); err != nil { + return errors.Wrap(err, "failed to create pg-delta declarative scratch database") + } + return nil +} + +// PreparePgDeltaNextShadow provisions the migrated target and an empty live +// sibling database used by the native pg-delta declarative planner. It never +// loads or applies the legacy declarative schemas. On failure, the container is +// removed best-effort without replacing the provisioning error. +func PreparePgDeltaNextShadow(ctx context.Context, fsys afero.Fs, options ...func(*pgx.ConnConfig)) (PgDeltaNextShadow, error) { + return preparePgDeltaNextShadow(ctx, fsys, pgDeltaNextShadowDependencies{ + create: CreateShadowDatabase, + wait: start.WaitForHealthyService, + migrate: MigrateShadowDatabase, + createScratch: createPgDeltaNextScratchDatabase, + remove: utils.DockerRemove, + }, options...) +} + +func preparePgDeltaNextShadow(ctx context.Context, fsys afero.Fs, dependencies pgDeltaNextShadowDependencies, options ...func(*pgx.ConnConfig)) (PgDeltaNextShadow, error) { + shadow, err := dependencies.create(ctx, utils.Config.Db.ShadowPort) + if err != nil { + return PgDeltaNextShadow{}, err + } + ok := false + defer func() { + if !ok { + dependencies.remove(shadow) + } + }() + if err := dependencies.wait(ctx, utils.Config.Db.HealthTimeout, shadow); err != nil { + return PgDeltaNextShadow{}, err + } + if err := dependencies.migrate(ctx, shadow, fsys, options...); err != nil { + return PgDeltaNextShadow{}, err + } + if err := dependencies.createScratch(ctx, options...); err != nil { + return PgDeltaNextShadow{}, err + } + migrated := pgconn.Config{ + Host: utils.Config.Hostname, + Port: utils.Config.Db.ShadowPort, + User: "postgres", + Password: utils.Config.Db.Password, + Database: "postgres", + } + scratch := migrated + scratch.Database = "pgdelta_declarative" + ok = true + return PgDeltaNextShadow{Container: shadow, Migrated: migrated, Scratch: scratch}, nil +} + // PrepareShadowSource provisions the shadow database that DiffDatabase diffs // against, but returns it running instead of diffing + removing, so a native // caller can run the differ itself. targetLocal mirrors diff --git a/apps/cli-go/internal/db/diff/shadow_test.go b/apps/cli-go/internal/db/diff/shadow_test.go new file mode 100644 index 0000000000..6cb1d400bd --- /dev/null +++ b/apps/cli-go/internal/db/diff/shadow_test.go @@ -0,0 +1,116 @@ +package diff + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/jackc/pgx/v4" + "github.com/spf13/afero" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/supabase/cli/internal/utils" +) + +func TestPreparePgDeltaNextShadow(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + utils.Config.Hostname = "shadow-host" + utils.Config.Db.ShadowPort = 6543 + utils.Config.Db.Password = "secret" + utils.Config.Db.HealthTimeout = 7 * time.Second + + var waitedContainer string + var migratedContainer string + var scratchCreated bool + var removedContainer string + dependencies := pgDeltaNextShadowDependencies{ + create: func(_ context.Context, port uint16) (string, error) { + assert.Equal(t, uint16(6543), port) + return "shadow-container", nil + }, + wait: func(_ context.Context, timeout time.Duration, containers ...string) error { + assert.Equal(t, 7*time.Second, timeout) + require.Len(t, containers, 1) + waitedContainer = containers[0] + return nil + }, + migrate: func(_ context.Context, container string, _ afero.Fs, _ ...func(*pgx.ConnConfig)) error { + migratedContainer = container + return nil + }, + createScratch: func(_ context.Context, _ ...func(*pgx.ConnConfig)) error { + scratchCreated = true + return nil + }, + remove: func(container string) { removedContainer = container }, + } + + result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) + + require.NoError(t, err) + assert.Equal(t, "shadow-container", result.Container) + assert.Equal(t, "shadow-container", waitedContainer) + assert.Equal(t, "shadow-container", migratedContainer) + assert.True(t, scratchCreated) + assert.Empty(t, removedContainer) + assert.Equal(t, "shadow-host", result.Migrated.Host) + assert.Equal(t, uint16(6543), result.Migrated.Port) + assert.Equal(t, "postgres", result.Migrated.User) + assert.Equal(t, "secret", result.Migrated.Password) + assert.Equal(t, "postgres", result.Migrated.Database) + assert.Equal(t, result.Migrated.Host, result.Scratch.Host) + assert.Equal(t, result.Migrated.Port, result.Scratch.Port) + assert.Equal(t, result.Migrated.User, result.Scratch.User) + assert.Equal(t, result.Migrated.Password, result.Scratch.Password) + assert.Equal(t, "pgdelta_declarative", result.Scratch.Database) +} + +func TestPreparePgDeltaNextShadowRemovesContainerAfterFailure(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + wantErr := errors.New("migration failed") + var removedContainer string + dependencies := pgDeltaNextShadowDependencies{ + create: func(context.Context, uint16) (string, error) { + return "failed-shadow", nil + }, + wait: func(context.Context, time.Duration, ...string) error { return nil }, + migrate: func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error { + return wantErr + }, + createScratch: func(context.Context, ...func(*pgx.ConnConfig)) error { return nil }, + remove: func(container string) { removedContainer = container }, + } + + result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) + + assert.ErrorIs(t, err, wantErr) + assert.Empty(t, result.Container) + assert.Equal(t, "failed-shadow", removedContainer) +} + +func TestPreparePgDeltaNextShadowRemovesContainerAfterScratchFailure(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + wantErr := errors.New("scratch creation failed") + var removedContainer string + dependencies := pgDeltaNextShadowDependencies{ + create: func(context.Context, uint16) (string, error) { + return "failed-scratch-shadow", nil + }, + wait: func(context.Context, time.Duration, ...string) error { return nil }, + migrate: func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error { + return nil + }, + createScratch: func(context.Context, ...func(*pgx.ConnConfig)) error { return wantErr }, + remove: func(container string) { removedContainer = container }, + } + + result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) + + assert.ErrorIs(t, err, wantErr) + assert.Empty(t, result.Container) + assert.Equal(t, "failed-scratch-shadow", removedContainer) +} diff --git a/apps/cli/docs/go-cli-porting-status.md b/apps/cli/docs/go-cli-porting-status.md index 5f461d0990..9757fa7956 100644 --- a/apps/cli/docs/go-cli-porting-status.md +++ b/apps/cli/docs/go-cli-porting-status.md @@ -82,11 +82,11 @@ These commands exist in the TS CLI today but have no direct top-level equivalent | Old command | TS status | TS command path or `missing` | Missing flags/params | Extra TS flags/params | Notes | | --------------------------------- | --------- | -------------------------------------------------- | -------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `db diff` | `ported` | `legacy/commands/db/diff/` | `n/a` | `n/a` | Native TS port. Native pg-delta / migra diff via edge-runtime against a Go-seam-provisioned live shadow (`db __shadow`); `--use-pgadmin` / `--use-pg-schema` delegate to the Go binary. | +| `db diff` | `ported` | `legacy/commands/db/diff/` | `n/a` | `n/a` | Native TS port. Pg-delta runs in-process by default with bundled pg-topo against a Go-seam-provisioned live shadow (`db __shadow`); `SUPABASE_USE_PG_DELTA_NEXT=false` retains the legacy edge-runtime implementation. Migra remains edge-runtime-backed; `--use-pgadmin` / `--use-pg-schema` delegate to Go. | | `db dump` | `ported` | `legacy/commands/db/dump/` | `n/a` | `n/a` | Native TS port. Streams `pg_dump`/`pg_dumpall` via a Docker container (`LegacyDockerRun`); schema/data/role modes, `--dry-run` script print, IPv4 transaction-pooler fallback. | | `db lint` | `ported` | `legacy/commands/db/lint/` | `n/a` | `n/a` | Native TS port. Runs `plpgsql_check` in a rolled-back transaction via LegacyDbConnection; emits Go-parity pretty JSON. | -| `db pull` | `ported` | `legacy/commands/db/pull/` | `n/a` | `n/a` | Native TS port. Native pg-delta / migra migration + `--declarative` pg-delta export; reconciles `schema_migrations`. The initial-migra pull dumps the remote schema natively (`pg_dump`) then appends the migra diff. Only `--experimental` (structured dump) still delegates to Go, deprecated in favor of `--declarative` (CLI-1957) — it needs a TS PostgreSQL DDL parser for Go's `format.WriteStructuredSchemas` that has no equivalent in this repo, and `--declarative` already delivers the same per-object schema split via pg-delta catalog introspection. | -| `db push` | `ported` | `legacy/commands/db/push/` | `n/a` | `n/a` | Native TS port. Connects local/linked/`--db-url`; pushes pending migrations, `--include-seed` seeds (`seed_files` hash tracking), `--include-roles`, `[db.vault]` secrets including decrypted `encrypted:` values; `--dry-run`; best-effort pg-delta migrations-catalog cache (warning-only on failure). Pipeline-incompatible statements (`CREATE INDEX CONCURRENTLY`, `VACUUM`, …) run standalone outside the batch transaction — from the closed Go PR supabase/cli#5156, also ported into `apps/cli-go` (CLI-1989 ruling). | +| `db pull` | `ported` | `legacy/commands/db/pull/` | `n/a` | `n/a` | Native TS port. Native bundled pg-delta / migra migration + `--declarative` pg-delta export; `SUPABASE_USE_PG_DELTA_NEXT=false` retains legacy edge-runtime pg-delta. Reconciles `schema_migrations`. The initial-migra pull dumps the remote schema natively (`pg_dump`) then appends the migra diff. Only `--experimental` (structured dump) still delegates to Go, deprecated in favor of `--declarative` (CLI-1957) — it needs a TS PostgreSQL DDL parser for Go's `format.WriteStructuredSchemas` that has no equivalent in this repo. | +| `db push` | `ported` | `legacy/commands/db/push/` | `n/a` | `n/a` | Native TS port. Connects local/linked/`--db-url`; pushes pending migrations, `--include-seed` seeds (`seed_files` hash tracking), `--include-roles`, `[db.vault]` secrets including decrypted `encrypted:` values; `--dry-run`. The best-effort pg-delta migrations-catalog warmup is retained only under `SUPABASE_USE_PG_DELTA_NEXT=false`. Pipeline-incompatible statements (`CREATE INDEX CONCURRENTLY`, `VACUUM`, …) run standalone outside the batch transaction — from the closed Go PR supabase/cli#5156, also ported into `apps/cli-go` (CLI-1989 ruling). | | `db reset` | `ported` | `legacy/commands/db/reset/` | `n/a` | `n/a` | Remote path native (drop user schemas, vault upsert, MigrateAndSeed, `--version`/`--last`, `--sql-paths` seed override). Local path native: running check, recreate + migrate + seed via the hidden Go `db __db-bootstrap` seam, storage-gated bucket seeding (reuses `seed buckets`), git-branch `Finished…` line. Only the niche `--experimental` remote schema-files path still delegates to the Go binary (telemetry-disabled). Pipeline-incompatible statements run standalone outside the batch transaction, same as `db push` (closed Go PR supabase/cli#5156, CLI-1989 ruling). | | `db start` | `ported` | `legacy/commands/db/start/` | `n/a` | `n/a` | Native TS port. Validates config, checks "already running" (prints Go's line), else delegates the container bootstrap (create + health + initial schema/roles/migrations/seed + `_current_branch`) to the hidden Go `db __db-bootstrap --mode start` seam. No status table / `cli_stack_started` (those are `supabase start`). `--from-backup` supported. | | `inspect report` | `ported` | `legacy/commands/inspect/report/` | `n/a` | `n/a` | Native TS port. Runs every inspect query via server-side `COPY ... CSV`, writes 14 CSVs under `//`, then renders a Go-parity Glamour rules summary (bounded csvq-subset evaluator; custom `[experimental.inspect.rules]` supported). | @@ -298,10 +298,10 @@ Legend: | `test db` | `ported` | [`../src/legacy/commands/test/db/db.command.ts`](../src/legacy/commands/test/db/db.command.ts) | | `test new` | `ported` | [`../src/legacy/commands/test/new/new.command.ts`](../src/legacy/commands/test/new/new.command.ts) | | `seed buckets` | `ported` | [`../src/legacy/commands/seed/buckets/buckets.command.ts`](../src/legacy/commands/seed/buckets/buckets.command.ts) | -| `db diff` | `ported` | [`../src/legacy/commands/db/diff/diff.command.ts`](../src/legacy/commands/db/diff/diff.command.ts) — native pg-delta / migra; `--use-pgadmin` / `--use-pg-schema` delegate to Go | +| `db diff` | `ported` | [`../src/legacy/commands/db/diff/diff.command.ts`](../src/legacy/commands/db/diff/diff.command.ts) — bundled in-process pg-delta by default (`SUPABASE_USE_PG_DELTA_NEXT=false` retains legacy edge-runtime); native migra; `--use-pgadmin` / `--use-pg-schema` delegate to Go | | `db dump` | `ported` | [`../src/legacy/commands/db/dump/dump.command.ts`](../src/legacy/commands/db/dump/dump.command.ts) | | `db push` | `ported` | [`../src/legacy/commands/db/push/push.command.ts`](../src/legacy/commands/db/push/push.command.ts) | -| `db pull` | `ported` | [`../src/legacy/commands/db/pull/pull.command.ts`](../src/legacy/commands/db/pull/pull.command.ts) — native pg-delta / migra; `--declarative` (deprecated alias `--use-pg-delta`) + `--diff-engine` (migra\|pg-delta); initial-migra pull dumps the schema natively (`pg_dump`) + appends the diff; `--experimental` structured dump still delegates to Go, deprecated in favor of `--declarative` (CLI-1957) | +| `db pull` | `ported` | [`../src/legacy/commands/db/pull/pull.command.ts`](../src/legacy/commands/db/pull/pull.command.ts) — bundled in-process pg-delta by default with legacy opt-out; native migra; `--declarative` (deprecated alias `--use-pg-delta`) + `--diff-engine` (migra\|pg-delta); initial-migra pull dumps the schema natively (`pg_dump`) + appends the diff; `--experimental` structured dump still delegates to Go, deprecated in favor of `--declarative` (CLI-1957) | | `db reset` | `ported` | [`../src/legacy/commands/db/reset/reset.command.ts`](../src/legacy/commands/db/reset/reset.command.ts) — includes Go-parity `--sql-paths` override for `[db.seed].sql_paths` | | `db lint` | `ported` | [`../src/legacy/commands/db/lint/lint.command.ts`](../src/legacy/commands/db/lint/lint.command.ts) | | `db start` | `ported` | [`../src/legacy/commands/db/start/start.command.ts`](../src/legacy/commands/db/start/start.command.ts) | @@ -314,8 +314,8 @@ Legend: | `db branch switch` | `wrapped` | [`../src/legacy/commands/db/branch/switch/switch.command.ts`](../src/legacy/commands/db/branch/switch/switch.command.ts) | | `db remote changes` | `wrapped` | [`../src/legacy/commands/db/remote/changes/changes.command.ts`](../src/legacy/commands/db/remote/changes/changes.command.ts) | | `db remote commit` | `wrapped` | [`../src/legacy/commands/db/remote/commit/commit.command.ts`](../src/legacy/commands/db/remote/commit/commit.command.ts) | -| `db schema declarative sync` | `ported` | [`../src/legacy/commands/db/schema/declarative/sync/sync.command.ts`](../src/legacy/commands/db/schema/declarative/sync/sync.command.ts) | -| `db schema declarative generate` | `ported` | [`../src/legacy/commands/db/schema/declarative/generate/generate.command.ts`](../src/legacy/commands/db/schema/declarative/generate/generate.command.ts) | +| `db schema declarative sync` | `ported` | [`../src/legacy/commands/db/schema/declarative/sync/sync.command.ts`](../src/legacy/commands/db/schema/declarative/sync/sync.command.ts) — bundled in-process pg-delta/pg-topo by default; legacy catalog/edge-runtime opt-out | +| `db schema declarative generate` | `ported` | [`../src/legacy/commands/db/schema/declarative/generate/generate.command.ts`](../src/legacy/commands/db/schema/declarative/generate/generate.command.ts) — bundled in-process export by default; writes `.pgdelta-export.json`; legacy catalog/edge-runtime opt-out | Flag divergences from the Go reference: diff --git a/apps/cli/package.json b/apps/cli/package.json index 7f80212f3d..10dd167e55 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -55,6 +55,8 @@ "@parcel/watcher": "^2.6.0", "@supabase/api": "workspace:*", "@supabase/config": "workspace:*", + "@supabase/pg-delta": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb", + "@supabase/pg-topo": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb", "@supabase/process-compose": "workspace:*", "@supabase/stack": "workspace:*", "@tsconfig/bun": "catalog:", diff --git a/apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md index e5a7694ba9..a510d61e41 100644 --- a/apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md @@ -6,6 +6,12 @@ health poll → write `.env` → `db push` → start suggestion. Every step is n including the migration push (`legacyDbPushCore`, shared with the standalone `supabase db push` command — see Notes). +The embedded push step does not warm pg-delta state under the default bundled +engine: no next-engine consumer uses the legacy catalog. Setting +`SUPABASE_USE_PG_DELTA_NEXT=false` retains Go's edge-runtime catalog warmup. +`PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs directly under +`.temp/pgdelta/` are meaningful only for that legacy opt-out. + ## Files Read | Path | Format | When | @@ -20,7 +26,8 @@ command — see Notes). | `/supabase/migrations/*.sql` | SQL | native push step, for each pending migration applied | | seed files from `[db.seed].sql_paths` | SQL | native push step (`--include-seed` is always set; gated on `[db.seed].enabled`) | | `/supabase/roles.sql` | SQL | native push step (`--include-roles` is always set; existence check + apply) | -| `/supabase/.temp/edge-runtime-version` | plain text | native push step's migrations-catalog cache (pg-delta), when a pinned edge-runtime image tag exists — resolved against the bootstrap workdir explicitly, not `cliConfig.workdir` (which is stale after this handler's own `process.chdir`) | +| `/supabase/.temp/pgdelta-version` | plain text | always read by push config loading for compatibility; affects the legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only: push-step catalog warmup image tag, resolved against the bootstrap workdir | ## Files Written @@ -31,8 +38,8 @@ command — see Notes). | `/supabase/.temp/project-ref` | plain text | always (mandatory; fails the command on write error) | | `/supabase/.temp/{pooler-url,rest-version,gotrue-version,storage-version,storage-migration}` | plain text | best-effort, from `link.LinkServices` | | `/.env` | dotenv | best-effort (write failure prints a warning and continues) | -| `/supabase/.temp/pgdelta/catalog--migrations--.json` | JSON | native push step, best-effort, after a successful migration apply, when pg-delta is enabled (a failure only warns on stderr and never fails the push) | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | native push step, same pg-delta gate, when the target requires SSL | +| `/supabase/.temp/pgdelta/catalog--migrations--.json` | JSON | legacy opt-out push step, best-effort after a successful migration apply when pg-delta is enabled; failure only warns | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out catalog export when the target requires SSL | | `/supabase/.temp/linked-project.json` | JSON | PersistentPostRun linked-project cache (`Effect.ensuring`); resolves against the bootstrap workdir (the prompted/`--workdir`/env target), not `cliConfig.workdir` | | `~/.supabase/telemetry.json` | JSON | PersistentPostRun telemetry flush (`Effect.ensuring`) | @@ -64,17 +71,18 @@ neither branch ever reaches the temp-login-role/Management-API path a passwordle ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | -| `SUPABASE_WORKDIR` | target dir (`--workdir` flag → env → prompt → cwd) | no | -| `SUPABASE_DB_PASSWORD` | DB password (`-p` flag → env → prompt/generate) | no | -| `GITHUB_TOKEN` | raise the GitHub API rate limit for template fetch | no | -| `SUPABASE_ACCESS_TOKEN` | auth bypass for ensure-login | no | -| `SUPABASE_PROFILE` | profile name/path (env → `~/.supabase/profile` → `supabase`) | no | -| `SUPABASE_YES` | auto-confirm the native push step's prompts (Go's viper `YES`), read project-`.env`-aware like the standalone `db push` | no | -| `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the push step's migrations-catalog cache when `[experimental.pgdelta].enabled` is unset, read project-`.env`-aware (see Files Read) | no | -| `SUPABASE_INTERNAL_IMAGE_REGISTRY` | overrides the push step's pg-delta edge-runtime image registry, read project-`.env`-aware (see Files Read) | no | -| `PGDELTA_NPM_REGISTRY` | overrides the push step's pg-delta edge-runtime npm registry (`.npmrc` + `NPM_CONFIG_REGISTRY` forward), read project-`.env`-aware (see Files Read) | no | +| Variable | Purpose | Required? | +| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | --------- | +| `SUPABASE_WORKDIR` | target dir (`--workdir` flag → env → prompt → cwd) | no | +| `SUPABASE_DB_PASSWORD` | DB password (`-p` flag → env → prompt/generate) | no | +| `GITHUB_TOKEN` | raise the GitHub API rate limit for template fetch | no | +| `SUPABASE_ACCESS_TOKEN` | auth bypass for ensure-login | no | +| `SUPABASE_PROFILE` | profile name/path (env → `~/.supabase/profile` → `supabase`) | no | +| `SUPABASE_YES` | auto-confirm the native push step's prompts (Go's viper `YES`), read project-`.env`-aware like the standalone `db push` | no | +| `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the legacy opt-out push cache when `[experimental.pgdelta].enabled` is unset, read project-`.env`-aware | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` to retain the legacy push-step catalog warmup, read project-`.env`-aware | no | +| `SUPABASE_INTERNAL_IMAGE_REGISTRY` | legacy opt-out only: overrides the push step's edge-runtime image registry, read project-`.env`-aware | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: overrides the push-step edge-runtime npm registry, read project-`.env`-aware | no | ## Exit Codes diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index a019c95594..620e7e2649 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -2,8 +2,29 @@ Native Effect port. Diffs the local project's expected schema (a throwaway shadow database) against a target database (local / linked / `--db-url`), using either -the native pg-delta or migra engine (both run inside Docker via edge-runtime). The -`--use-pgadmin` / `--use-pg-schema` engines delegate to the bundled Go binary. +pg-delta or migra. Pg-delta runs in-process by default; migra still runs in Docker +via edge-runtime. The `--use-pgadmin` / `--use-pg-schema` engines delegate to the +bundled Go binary. + +## Pg-delta implementation and compatibility + +- The default implementation is the in-process pg-delta engine bundled into the + CLI binary together with pg-topo. Its version is fixed when the CLI is built; + there is no runtime package download or automatic fallback to the legacy engine. +- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy edge-runtime implementation. + Only that opt-out reads legacy catalogs under `supabase/.temp/pgdelta/`, + `supabase/.temp/pgdelta-version`, or `PGDELTA_NPM_REGISTRY`. +- With `PGDELTA_DEBUG`, default-engine snapshots, plans, and diagnostics are written + under `supabase/.temp/pgdelta/v2/debug//`. The directory contains + `metadata.json` and, when available, `source-snapshot.json`, + `desired-snapshot.json`, `plan.json`, and `diagnostics.json`. These are diagnostic + artifacts, not reusable catalogs. +- The default engine refuses to emit a diff when extraction reports an error or a + strict coverage gap (`unmodeled_kind` or `unresolved_security_label`). The error + identifies the diagnostic origin, code, subject, and message; when debug capture + is enabled, the bundle is saved before the refusal. +- SQL text and file segmentation may differ from the legacy renderer. Applicable + output and convergence (a subsequent diff is empty) are the compatibility contract. ## Files Read @@ -14,21 +35,25 @@ the native pg-delta or migra engine (both run inside Docker via edge-runtime). T | `/supabase/database/**` (declarative dir) | SQL | local target when declarative schemas exist | | `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | | `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution | -| `/supabase/.temp/pgdelta/*.json` | JSON | explicit `--from/--to migrations` catalog (cache) | +| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only: edge-runtime image tag | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: explicit `--from/--to migrations` catalog | ## Files Written -| Path | Format | When | -| ----------------------------------------------------------- | ------ | ----------------------------------------------- | -| `/supabase/migrations/_.sql` | SQL | `--file ` and the diff is non-empty | -| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | -| `/supabase/.temp/pgdelta/*.json` | JSON | explicit `--from/--to migrations` catalog cache | -| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| ----------------------------------------------------------- | ------ | ------------------------------------------- | +| `/supabase/migrations/_.sql` | SQL | `--file ` and the diff is non-empty | +| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: migrations catalog | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out only: Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | +| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker -- Edge-runtime container (pg-delta / migra diff scripts). +- Edge-runtime container (migra, or pg-delta only under the legacy opt-out). - Shadow Postgres container (provisioned + torn down via the Go `db __shadow` seam). - `supabase/migra` container — the migra OOM bash fallback only. @@ -43,14 +68,15 @@ the native pg-delta or migra engine (both run inside Docker via edge-runtime). T ## Environment Variables -| Variable | Purpose | Required? | -| -------------------------------- | ------------------------------------------------ | --------- | -| `SUPABASE_ACCESS_TOKEN` | auth for `--linked` | no | -| `SUPABASE_DB_PASSWORD` | remote DB password (linked) | no | -| `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta engine | no | -| `PGDELTA_DEBUG` | pg-delta debug capture | no | -| `PGDELTA_NPM_REGISTRY` | scoped `@supabase` npm registry for edge-runtime | no | -| `SUPABASE_SSL_DEBUG` | migra SSL debug logging | no | +| Variable | Purpose | Required? | +| -------------------------------- | ------------------------------------------------- | --------- | +| `SUPABASE_ACCESS_TOKEN` | auth for `--linked` | no | +| `SUPABASE_DB_PASSWORD` | remote DB password (linked) | no | +| `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta engine | no | +| `PGDELTA_DEBUG` | pg-delta debug capture | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for the legacy edge-runtime engine | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: scoped npm registry | no | +| `SUPABASE_SSL_DEBUG` | migra SSL debug logging | no | ## Exit Codes diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index 61fb818dbc..98fd04c383 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -6,7 +6,10 @@ import { detectGitBranch } from "../../../../shared/git/git-branch.ts"; import { Output } from "../../../../shared/output/output.service.ts"; import { LegacyCliConfig } from "../../../config/legacy-cli-config.service.ts"; import { legacyAqua, legacyYellow } from "../../../shared/legacy-colors.ts"; -import { legacyReadDbToml } from "../../../shared/legacy-db-config.toml-read.ts"; +import { + legacyReadDbToml, + legacyResolveDeclarativeDir, +} from "../../../shared/legacy-db-config.toml-read.ts"; import { LegacyDbConfigResolver } from "../../../shared/legacy-db-config.service.ts"; import type { LegacyDbConnType } from "../../../shared/legacy-db-target-flags.ts"; import { legacyGetHostname } from "../../../shared/legacy-hostname.ts"; @@ -26,8 +29,23 @@ import { legacyGetMigrationPath, } from "../../../shared/legacy-migration-file.ts"; import { legacyDiffMigra } from "../shared/legacy-migra.ts"; +import { + LegacyPgDeltaEngine, + type LegacyPgDeltaDatabaseEndpoint, + type LegacyPgDeltaEndpoint, + type LegacyPgDeltaExportManifest, + type LegacyPgDeltaSqlFile, +} from "../shared/legacy-pgdelta-engine.service.ts"; +import { + LegacyLoadPgDeltaSqlFiles, + LegacyLoadPgDeltaSqlPaths, + LegacyReadPgDeltaExportManifest, +} from "../shared/legacy-pgdelta-files.ts"; import { legacyWritePgDeltaMigrations } from "../shared/legacy-pgdelta-migrations.write.ts"; -import { type LegacyPgDeltaContext, legacyDiffPgDelta } from "../shared/legacy-pgdelta.ts"; +import { + legacyIsPgDeltaDebugEnabled, + type LegacyPgDeltaContext, +} from "../shared/legacy-pgdelta.ts"; import { LegacyDeclarativeSeam } from "../shared/legacy-pgdelta.seam.service.ts"; import type { LegacyDbDiffFlags } from "./diff.command.ts"; import { legacyClassifyExplicitRef, legacyUnknownTargetMessage } from "./diff.explicit.ts"; @@ -85,6 +103,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy const output = yield* Output; const resolver = yield* LegacyDbConfigResolver; const seam = yield* LegacyDeclarativeSeam; + const pgDelta = yield* LegacyPgDeltaEngine; const proxy = yield* LegacyGoProxy; const cliConfig = yield* LegacyCliConfig; const telemetryState = yield* LegacyTelemetryState; @@ -193,17 +212,24 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // runs `LoadConfig(ref)` (`explicit.go:78-86`), re-merging the matching // `[remotes.]` block so a later `local` ref read and the trailing // `pgDeltaFormatOptions()` see the override. Thread the merged config through. - const resolveRef = (ref: string) => + const resolveRef = (ref: string): Effect.Effect => Effect.gen(function* () { switch (legacyClassifyExplicitRef(ref)) { - case "local": - return legacyToPostgresURL({ + case "local": { + const connection = { host: legacyGetHostname(), port: cfg.port, user: "postgres", password: cfg.password, database: "postgres", - }); + }; + return { + kind: "database", + ref: legacyToPostgresURL(connection), + connection, + connectOptions: { isLocal: true, dnsResolver }, + } satisfies LegacyPgDeltaDatabaseEndpoint; + } case "linked": { const resolved = yield* resolver.resolve({ dbUrl: Option.none(), @@ -217,39 +243,49 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy mergedLinkedRef = ref2; cfg = yield* legacyReadDbToml(fs, path, cliConfig.workdir, ref2); } - return legacyToPostgresURL(resolved.conn); + return { + kind: "database", + ref: legacyToPostgresURL(resolved.conn), + connection: resolved.conn, + connectOptions: { isLocal: resolved.isLocal, dnsResolver }, + } satisfies LegacyPgDeltaDatabaseEndpoint; } case "migrations": - return yield* seam.exportCatalog({ - mode: "migrations", - noCache: false, - // Pass the linked ref only if one resolved earlier in the cascade, - // so the `__catalog` child merges the same remote override Go's - // in-process migrations catalog sees (`explicit.go:88-126`). Absent - // otherwise → base config, matching Go's resolution order. + return { + kind: "migrations", + // Preserve resolution order: only refs resolved before this endpoint + // influence the migrations shadow/catalog. ...(mergedLinkedRef !== undefined ? { projectRef: mergedLinkedRef } : {}), - }); + } satisfies LegacyPgDeltaEndpoint; case "url": - return ref; + return { + kind: "database", + ref, + // The next engine parses arbitrary explicit URLs itself. They are + // remote by default, matching Go's TLS-safe connection path. + connectOptions: { isLocal: false, dnsResolver }, + } satisfies LegacyPgDeltaDatabaseEndpoint; default: return yield* Effect.fail( new LegacyDbDiffUnknownTargetError({ message: legacyUnknownTargetMessage(ref) }), ); } }); - const sourceRef = yield* resolveRef(from); - const targetRef = yield* resolveRef(to); + const source = yield* resolveRef(from); + const desired = yield* resolveRef(to); const explicitCtx: LegacyPgDeltaContext = { projectId: Option.getOrElse(cliConfig.projectId, () => ""), cwd: cliConfig.workdir, npmVersion: Option.getOrUndefined(cfg.pgDelta.npmVersion), denoVersion: cfg.denoVersion, }; - const result = yield* legacyDiffPgDelta(explicitCtx, { - sourceRef, - targetRef, + const result = yield* pgDelta.diffExplicit({ + context: explicitCtx, + source, + desired, schema: flags.schema, formatOptions: Option.getOrElse(cfg.pgDelta.formatOptions, () => ""), + debug: legacyIsPgDeltaDebugEnabled(), }); // Explicit-mode output: `--output` file (Go's `writeOutput`) or stdout // (Go's `fmt.Print`, no trailing newline — pg-delta ends each statement `;\n`). @@ -378,47 +414,93 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy }); yield* output.raw("Creating shadow database...\n", "stderr"); - const shadow = yield* seam.provisionShadow({ - mode: "diff", - targetLocal: resolved.isLocal, - usePgDelta: useDelta, - schema: flags.schema, - // Linked path only: the shadow merges the same `[remotes.]` override - // the engine/format read above (Go builds the shadow from the remote-merged - // config). Default `db diff` is local, which never merges a remote block. - projectRef: connType === "linked" ? linkedRef : undefined, - }); - - const diffResult = yield* Effect.gen(function* () { - const target = shadow.targetUrlOverride ?? targetUrl; - yield* output.raw( - flags.schema.length > 0 - ? `Diffing schemas: ${flags.schema.join(",")}\n` - : "Diffing schemas...\n", - "stderr", - ); - if (useDelta) { - const result = yield* legacyDiffPgDelta(ctx, { - sourceRef: shadow.sourceUrl, - targetRef: target, - schema: flags.schema, - formatOptions, + const diffingMessage = + flags.schema.length > 0 + ? `Diffing schemas: ${flags.schema.join(",")}\n` + : "Diffing schemas...\n"; + const diffResult = useDelta + ? yield* Effect.gen(function* () { + // The selected strategy owns pg-delta shadow/pool lifecycles. This message + // precedes the call because the high-level boundary intentionally exposes + // no partially-provisioned resource to the handler. + yield* output.raw(diffingMessage, "stderr"); + let declarativeFiles: ReadonlyArray | undefined; + let declarativeManifest: LegacyPgDeltaExportManifest | undefined; + if (pgDelta.implementation === "next" && resolved.isLocal) { + if (cfg.migrationSchemaPaths !== undefined && cfg.migrationSchemaPaths.length > 0) { + declarativeFiles = yield* LegacyLoadPgDeltaSqlPaths( + fs, + path, + cliConfig.workdir, + cfg.migrationSchemaPaths, + ); + } else { + const declarativeDirSetting = legacyResolveDeclarativeDir(path, cfg.pgDelta); + const declarativeDir = path.isAbsolute(declarativeDirSetting) + ? declarativeDirSetting + : path.join(cliConfig.workdir, declarativeDirSetting); + const hasDeclarativeDir = cfg.pgDelta.enabled + ? yield* fs.exists(declarativeDir).pipe(Effect.orElseSucceed(() => false)) + : false; + if (hasDeclarativeDir) { + const loaded = yield* LegacyLoadPgDeltaSqlFiles(fs, path, declarativeDir); + if (loaded.length > 0) { + declarativeFiles = loaded; + declarativeManifest = yield* LegacyReadPgDeltaExportManifest( + fs, + path, + declarativeDir, + ); + } + } else { + const schemasDir = path.join(cliConfig.workdir, "supabase", "schemas"); + const hasSchemasDir = yield* fs + .exists(schemasDir) + .pipe(Effect.orElseSucceed(() => false)); + if (hasSchemasDir) { + const loaded = yield* LegacyLoadPgDeltaSqlFiles(fs, path, schemasDir); + if (loaded.length > 0) declarativeFiles = loaded; + } + } + } + } + const result = yield* pgDelta.diffDatabase({ + context: ctx, + target: { + kind: "database", + ref: targetUrl, + connection: resolved.conn, + connectOptions: { isLocal: resolved.isLocal, dnsResolver }, + }, + targetLocal: resolved.isLocal, + schema: flags.schema, + formatOptions, + ...(connType === "linked" && linkedRef !== undefined ? { projectRef: linkedRef } : {}), + ...(declarativeFiles !== undefined ? { declarativeFiles } : {}), + ...(declarativeManifest !== undefined ? { declarativeManifest } : {}), + debug: legacyIsPgDeltaDebugEnabled(), + }); + return { sql: result.sql, files: result.files }; + }) + : yield* Effect.gen(function* () { + const shadow = yield* seam.provisionShadow({ + mode: "diff", + targetLocal: resolved.isLocal, + usePgDelta: false, + schema: flags.schema, + ...(connType === "linked" && linkedRef !== undefined ? { projectRef: linkedRef } : {}), + }); + return yield* Effect.gen(function* () { + yield* output.raw(diffingMessage, "stderr"); + const sql = yield* legacyDiffMigra(ctx, { + source: shadow.sourceUrl, + target: shadow.targetUrlOverride ?? targetUrl, + schema: flags.schema, + connectOptions: { isLocal: resolved.isLocal, dnsResolver }, + }); + return { sql, files: undefined }; + }).pipe(Effect.ensuring(seam.removeShadowContainer(shadow.container))); }); - // Keep the per-unit plan files so a multi-unit plan can be written as one - // migration file each (Go's `DatabaseDiff.Files`); `sql` stays the flattened - // join for stdout review + machine payloads. - return { sql: result.sql, files: result.files }; - } - const sql = yield* legacyDiffMigra(ctx, { - source: shadow.sourceUrl, - target, - schema: flags.schema, - connectOptions: { isLocal: resolved.isLocal, dnsResolver }, - }); - // The migra engine has no execution-aware plan units, so it always writes a - // single migration file (Go's `SaveDiff` single-file path). - return { sql, files: undefined }; - }).pipe(Effect.ensuring(seam.removeShadowContainer(shadow.container))); const out = diffResult.sql; // Detect the branch from the resolved workdir, not the caller's CWD: Go @@ -458,7 +540,13 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy workdir: cliConfig.workdir, baseMillis: yield* Clock.currentTimeMillis, name: fileName, - files: planFiles.map((file) => ({ name: file.name, sql: file.sql })), + files: planFiles.map((file) => ({ + name: + file.suffix !== undefined && file.suffix !== null + ? file.suffix.replace(/^_/u, "") + : file.name, + sql: file.sql, + })), }).pipe(Effect.mapError((cause) => new LegacyDbDiffWriteError({ message: cause.message }))); for (const unit of writtenUnits) writtenFiles.push(unit.path); } else { diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index f1a531c7a9..a00db0b094 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -28,6 +28,11 @@ import { LegacyEdgeRuntimeScript, } from "../../../shared/legacy-edge-runtime-script.service.ts"; import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { + LegacyPgDeltaEngine, + type LegacyPgDeltaDatabaseDiffInput, + type LegacyPgDeltaExplicitDiffInput, +} from "../shared/legacy-pgdelta-engine.service.ts"; import { LegacyDeclarativeSeam } from "../shared/legacy-pgdelta.seam.service.ts"; import type { LegacyDbDiffFlags } from "./diff.command.ts"; import { legacyDbDiff } from "./diff.handler.ts"; @@ -37,9 +42,11 @@ interface SetupOpts { readonly isLocal?: boolean; readonly linkedRef?: string; readonly diffSql?: string; - // When set, the pg-delta edge mock emits a multi-unit plan envelope (one file - // per entry) instead of the single-unit wrap of `diffSql`. + // When set, the pg-delta strategy mock returns one rendered file per entry. readonly diffFiles?: ReadonlyArray<{ readonly name: string; readonly sql: string }>; + // Exact suffixes returned by the next renderer, parallel to `diffFiles`. + readonly diffSuffixes?: ReadonlyArray; + readonly pgDeltaImplementation?: "legacy" | "next"; readonly targetOverride?: string; readonly oom?: boolean; // edge-runtime OOMs; the bash fallback returns `diffSql` readonly delegateStdout?: string; // stdout returned by a captured Go-delegate run @@ -60,14 +67,8 @@ function setup(workdir: string, opts: SetupOpts = {}) { projectRef?: string; }> = []; const removedContainers: string[] = []; - const exportCalls: string[] = []; - const exportCatalogCalls: Array<{ mode: string; projectRef?: string }> = []; const seam = Layer.succeed(LegacyDeclarativeSeam, { - exportCatalog: ({ mode, projectRef }) => { - exportCalls.push(mode); - exportCatalogCalls.push({ mode, projectRef }); - return Effect.succeed("supabase/.temp/pgdelta/migrations.json"); - }, + exportCatalog: () => Effect.succeed("supabase/.temp/pgdelta/migrations.json"), execInherit: () => Effect.succeed(0), ensureLocalDatabaseStarted: () => Effect.void, ensureLocalPostgresImageCurrent: () => Effect.void, @@ -85,6 +86,51 @@ function setup(workdir: string, opts: SetupOpts = {}) { }), }); + const explicitDiffCalls: LegacyPgDeltaExplicitDiffInput[] = []; + const databaseDiffCalls: LegacyPgDeltaDatabaseDiffInput[] = []; + const pgDeltaResult = () => { + const sql = opts.diffSql ?? ""; + const files = + opts.diffFiles !== undefined + ? opts.diffFiles.map((file, index) => ({ + sequence: index + 1, + name: file.name, + ...(opts.diffSuffixes?.[index] !== undefined + ? { suffix: opts.diffSuffixes[index] } + : {}), + sql: file.sql, + transactional: true, + })) + : sql.length > 0 + ? [{ sequence: 1, name: "schema_changes", sql, transactional: true }] + : []; + return { + changes: files.length > 0, + sql: opts.diffFiles !== undefined ? files.map((file) => file.sql).join("\n\n") : sql, + files, + }; + }; + const pgDeltaEngine = Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + // The handler must route through this strategy even when the selected + // implementation is legacy; the strategy owns edge runtime and shadows. + implementation: opts.pgDeltaImplementation ?? "legacy", + diffExplicit: (input) => + Effect.sync(() => { + explicitDiffCalls.push(input); + return pgDeltaResult(); + }), + diffDatabase: (input) => + Effect.sync(() => { + databaseDiffCalls.push(input); + return pgDeltaResult(); + }), + exportDeclarativeSchema: () => Effect.die("exportDeclarativeSchema unused"), + planDeclarativeSchema: () => Effect.die("planDeclarativeSchema unused"), + }), + ); + const edgeCalls: LegacyEdgeRuntimeRunOpts[] = []; const edge = Layer.succeed(LegacyEdgeRuntimeScript, { run: (runOpts: LegacyEdgeRuntimeRunOpts) => { @@ -94,28 +140,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { new LegacyEdgeRuntimeScriptError({ message: "Fatal JavaScript out of memory" }), ); } - const diffSql = opts.diffSql ?? ""; - // The pg-delta diff script (uniquely identified by `renderPlanFiles`) prints a - // JSON envelope with one file per plan unit; wrap the test's raw SQL into a - // single-unit envelope so `legacyDiffPgDelta` parses it. The migra script - // returns raw SQL unchanged. - const isPgDelta = runOpts.script.includes("renderPlanFiles"); - const planFiles = - opts.diffFiles !== undefined - ? opts.diffFiles.map((file, i) => ({ - order: i + 1, - name: file.name, - transactionMode: "transactional", - sql: file.sql, - })) - : diffSql.length > 0 - ? [{ order: 1, name: "schema_changes", transactionMode: "transactional", sql: diffSql }] - : []; - const stdout = - isPgDelta && planFiles.length > 0 - ? JSON.stringify({ version: 1, files: planFiles }) - : diffSql; - return Effect.succeed({ stdout, stderr: "" }); + return Effect.succeed({ stdout: opts.diffSql ?? "", stderr: "" }); }, }); @@ -174,6 +199,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { telemetry.layer, cache.layer, seam, + pgDeltaEngine, edge, docker, dbConnection, @@ -206,8 +232,8 @@ function setup(workdir: string, opts: SetupOpts = {}) { telemetry, provisionCalls, removedContainers, - exportCalls, - exportCatalogCalls, + explicitDiffCalls, + databaseDiffCalls, edgeCalls, resolverCalls, proxyCalls, @@ -267,12 +293,107 @@ describe("legacy db diff", () => { const s = setup(tmp.current, { diffSql: "create table p ();\n" }); return Effect.gen(function* () { yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), schema: ["public"] })); - expect(s.provisionCalls).toEqual([{ mode: "diff", targetLocal: true, usePgDelta: true }]); + expect(s.provisionCalls).toEqual([]); + expect(s.databaseDiffCalls).toHaveLength(1); + expect(s.databaseDiffCalls[0]).toMatchObject({ + targetLocal: true, + schema: ["public"], + target: { + kind: "database", + connection: { + host: "127.0.0.1", + port: 54322, + user: "postgres", + password: "postgres", + database: "postgres", + }, + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + }); + // Even the legacy implementation is hidden behind LegacyPgDeltaEngine; + // the handler no longer invokes edge runtime itself. + expect(s.edgeCalls).toEqual([]); expect(stderr(s.out)).toContain("Diffing schemas: public"); expect(stdout(s.out)).toBe("create table p ();\n\n"); }).pipe(Effect.provide(s.layer)); }); + it.effect("next local diff gives configured schema_paths precedence", () => { + mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + [ + "[db.migrations]", + 'schema_paths = ["configured.sql"]', + "", + "[experimental.pgdelta]", + "enabled = true", + "", + ].join("\n"), + ); + writeFileSync(join(tmp.current, "supabase", "configured.sql"), "create table configured ();\n"); + writeFileSync( + join(tmp.current, "supabase", "database", "ignored.sql"), + "create table ignored ();\n", + ); + const s = setup(tmp.current, { + pgDeltaImplementation: "next", + diffSql: "create table result ();\n", + }); + return Effect.gen(function* () { + yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); + expect(s.databaseDiffCalls[0]?.declarativeFiles).toEqual([ + { name: "supabase/configured.sql", sql: "create table configured ();\n" }, + ]); + expect(s.databaseDiffCalls[0]?.declarativeManifest).toBeUndefined(); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("next local diff loads the enabled declarative directory and manifest", () => { + const declarativeDir = join(tmp.current, "supabase", "database"); + mkdirSync(declarativeDir, { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + ["[experimental.pgdelta]", "enabled = true", ""].join("\n"), + ); + writeFileSync(join(declarativeDir, "public.sql"), "create table public.t ();\n"); + writeFileSync( + join(declarativeDir, ".pgdelta-export.json"), + JSON.stringify({ formatVersion: 1, redactSecrets: true, scope: "database" }), + ); + const s = setup(tmp.current, { + pgDeltaImplementation: "next", + diffSql: "create table result ();\n", + }); + return Effect.gen(function* () { + yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); + expect(s.databaseDiffCalls[0]?.declarativeFiles).toEqual([ + { name: "public.sql", sql: "create table public.t ();\n" }, + ]); + expect(s.databaseDiffCalls[0]?.declarativeManifest).toEqual({ + redactSecrets: true, + scope: "database", + }); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("next local diff falls back to supabase/schemas", () => { + const schemasDir = join(tmp.current, "supabase", "schemas"); + mkdirSync(schemasDir, { recursive: true }); + writeFileSync(join(schemasDir, "fallback.sql"), "create table fallback ();\n"); + const s = setup(tmp.current, { + pgDeltaImplementation: "next", + diffSql: "create table result ();\n", + }); + return Effect.gen(function* () { + yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); + expect(s.databaseDiffCalls[0]?.declarativeFiles).toEqual([ + { name: "fallback.sql", sql: "create table fallback ();\n" }, + ]); + expect(s.databaseDiffCalls[0]?.declarativeManifest).toBeUndefined(); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("a linked [remotes.] block enabling pg-delta selects the pg-delta engine", () => { // Go loads the project ref before LoadConfig on the linked path, merging the // matching [remotes.] block before experimental.pgdelta.enabled is read @@ -301,10 +422,9 @@ describe("legacy db diff", () => { }); return Effect.gen(function* () { yield* legacyDbDiff(flags({ linked: Option.some(true) })); - expect(s.provisionCalls[0]?.usePgDelta).toBe(true); - // The shadow is provisioned with the resolved ref so the `db __shadow` child - // merges the same `[remotes.]` override into the shadow baseline. - expect(s.provisionCalls[0]?.projectRef).toBe("abcdefghijklmnopqrst"); + expect(s.provisionCalls).toEqual([]); + expect(s.databaseDiffCalls[0]?.projectRef).toBe("abcdefghijklmnopqrst"); + expect(s.databaseDiffCalls[0]?.target.connectOptions.isLocal).toBe(false); }).pipe(Effect.provide(s.layer)); }); @@ -498,6 +618,24 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); + it.effect("uses exact next-renderer suffixes for multi-file migration names", () => { + const s = setup(tmp.current, { + diffFiles: [ + { name: "ignored_legacy_name", sql: "a" }, + { name: "ignored_legacy_name", sql: "b" }, + ], + diffSuffixes: ["_1", "_2"], + }); + return Effect.gen(function* () { + yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), file: Option.some("my_diff") })); + const dir = join(tmp.current, "supabase", "migrations"); + expect(readdirSync(dir).sort()).toEqual([ + "19700101000000_my_diff_1.sql", + "19700101000001_my_diff_2.sql", + ]); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("creates nested parent directories for a nested single-unit --file name", () => { // `db diff -f snapshots/remote` must create the `_snapshots/` parent dir // before writing, mirroring Go's `utils.WriteFile`. @@ -590,10 +728,54 @@ describe("legacy db diff", () => { yield* legacyDbDiff(flags({ from: Option.some("local"), to: Option.some("linked") })); // Explicit mode is pg-delta and never provisions a shadow. expect(s.provisionCalls).toEqual([]); + expect(s.explicitDiffCalls[0]).toMatchObject({ + source: { + kind: "database", + connection: { + host: "127.0.0.1", + user: "postgres", + database: "postgres", + }, + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + desired: { + kind: "database", + connection: { + host: "127.0.0.1", + port: 54322, + user: "postgres", + password: "postgres", + database: "postgres", + }, + connectOptions: { isLocal: false, dnsResolver: "native" }, + }, + }); expect(stdout(s.out)).toBe("create table e ();\n"); }).pipe(Effect.provide(s.layer)); }); + it.effect("explicit URL endpoints retain the raw ref and remote connection options", () => { + const s = setup(tmp.current, { diffSql: "create table u ();\n" }); + return Effect.gen(function* () { + yield* legacyDbDiff( + flags({ + from: Option.some("postgresql://source.example/postgres"), + to: Option.some("postgresql://desired.example/postgres"), + }), + ); + expect(s.explicitDiffCalls[0]?.source).toEqual({ + kind: "database", + ref: "postgresql://source.example/postgres", + connectOptions: { isLocal: false, dnsResolver: "native" }, + }); + expect(s.explicitDiffCalls[0]?.desired).toEqual({ + kind: "database", + ref: "postgresql://desired.example/postgres", + connectOptions: { isLocal: false, dnsResolver: "native" }, + }); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("explicit --output writes raw SQL to the given path", () => { const s = setup(tmp.current, { diffSql: "create table w ();\n" }); return Effect.gen(function* () { @@ -652,11 +834,12 @@ describe("legacy db diff", () => { }, ); - it.effect("explicit --from migrations resolves a shadow catalog via the seam", () => { + it.effect("explicit --from migrations routes the migrations endpoint to the strategy", () => { const s = setup(tmp.current, { diffSql: "create table m ();\n" }); return Effect.gen(function* () { yield* legacyDbDiff(flags({ from: Option.some("migrations"), to: Option.some("local") })); - expect(s.exportCalls).toEqual(["migrations"]); + expect(s.explicitDiffCalls[0]?.source).toEqual({ kind: "migrations" }); + expect(s.edgeCalls).toEqual([]); }).pipe(Effect.provide(s.layer)); }); @@ -672,8 +855,10 @@ describe("legacy db diff", () => { }); return Effect.gen(function* () { yield* legacyDbDiff(flags({ from: Option.some("linked"), to: Option.some("migrations") })); - const migrations = s.exportCatalogCalls.find((c) => c.mode === "migrations"); - expect(migrations?.projectRef).toBe("abcdefghijklmnopqrst"); + expect(s.explicitDiffCalls[0]?.desired).toEqual({ + kind: "migrations", + projectRef: "abcdefghijklmnopqrst", + }); }).pipe(Effect.provide(s.layer)); }, ); @@ -688,8 +873,7 @@ describe("legacy db diff", () => { }); return Effect.gen(function* () { yield* legacyDbDiff(flags({ from: Option.some("migrations"), to: Option.some("linked") })); - const migrations = s.exportCatalogCalls.find((c) => c.mode === "migrations"); - expect(migrations?.projectRef).toBeUndefined(); + expect(s.explicitDiffCalls[0]?.source).toEqual({ kind: "migrations" }); }).pipe(Effect.provide(s.layer)); }); @@ -711,8 +895,10 @@ describe("legacy db diff", () => { linked: Option.some(true), }), ); - const migrations = s.exportCatalogCalls.find((c) => c.mode === "migrations"); - expect(migrations?.projectRef).toBe("abcdefghijklmnopqrst"); + expect(s.explicitDiffCalls[0]?.desired).toEqual({ + kind: "migrations", + projectRef: "abcdefghijklmnopqrst", + }); }).pipe(Effect.provide(s.layer)); }); diff --git a/apps/cli/src/legacy/commands/db/diff/diff.layers.ts b/apps/cli/src/legacy/commands/db/diff/diff.layers.ts index 8c2ab09380..e53312db93 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.layers.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.layers.ts @@ -12,13 +12,18 @@ import { legacyLinkedDbResolverRuntimeLayer } from "../../../shared/legacy-manag import { legacyPgDeltaSslProbeLayer } from "../../../shared/legacy-pgdelta-ssl-probe.layer.ts"; import { legacyTelemetryStateLayer } from "../../../telemetry/legacy-telemetry-state.layer.ts"; import { legacyDeclarativeSeamLayer } from "../shared/legacy-pgdelta.seam.layer.ts"; +import { legacyPgDeltaEngineLayer } from "../shared/legacy-pgdelta-engine.layer.ts"; +import { legacyPgDeltaNextAdapterLayer } from "../shared/legacy-pgdelta-next-adapter.layer.ts"; +import { legacyPgDeltaNextShadowLayer } from "../shared/legacy-pgdelta-next-shadow.layer.ts"; /** * Runtime layer for `supabase db diff`. * * Mirrors `db schema declarative generate` (`generate.layers.ts`): the db-config - * resolver plus the native pg-delta / migra stack — the edge-runtime runner, the - * SSL probe, and the Go shadow-database seam (`provisionShadow`). `LegacyDockerRun` + * resolver plus both pg-delta implementations, migra, the SSL probe, and the Go + * shadow-database seam (`provisionShadow`). The default pg-delta runs in-process; + * the edge-runtime runner is retained only for migra and the explicit legacy opt-out. + * `LegacyDockerRun` * is exposed in the merge (not just provided to the edge-runtime layer) because the * migra OOM bash fallback runs the `supabase/migra` container directly. * Per the "provide doesn't share to siblings" rule, `LegacyCliConfig` is provided @@ -42,6 +47,15 @@ const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( ); const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); +const nextShadow = legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seam)); +const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(legacyPgDeltaNextAdapterLayer), + Layer.provide(nextShadow), + Layer.provide(edgeRuntime), + Layer.provide(legacyPgDeltaSslProbeLayer), + Layer.provide(seam), + Layer.provide(legacyDebugLoggerLayer), +); export const legacyDbDiffRuntimeLayer = Layer.mergeAll( dbConfig, @@ -50,6 +64,7 @@ export const legacyDbDiffRuntimeLayer = Layer.mergeAll( edgeRuntime, legacyPgDeltaSslProbeLayer, seam, + pgDeltaEngine, cliConfig, legacyIdentityStitchLayer, legacyTelemetryStateLayer, diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 8e6fcf5d65..a5c49e8a65 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -10,7 +10,7 @@ structured-dump sub-branch (Go's `format.WriteStructuredSchemas`) stays delegated to the bundled Go binary rather than retired or ported (CLI-1957): it needs a TS PostgreSQL DDL AST parser with no equivalent in this repo. `--declarative` covers the same per-object-files outcome for schema objects via -pg-delta catalog introspection, though its output tree and cluster-object +pg-delta managed-state extraction, though its output tree and cluster-object coverage differ (see Files Written below), so this mode is on a deprecation path — the same DECISION CLI-1960 makes for `db diff --use-pg-schema` (keep delegating, flag for removal), not the same output: Go's own `--use-pg-schema` @@ -24,14 +24,36 @@ Go checks `usePgDelta` before `EXPERIMENTAL`, so that combination never delegates and just runs the declarative export normally (see the Notes/Delegation section below). +## Pg-delta implementation and compatibility + +- Pg-delta diff and declarative export use the in-process engine bundled into the + CLI binary by default. Pg-topo is bundled with it and the version is fixed at + CLI build time; the command never downloads it or falls back automatically. +- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy edge-runtime path. + `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs + directly below `supabase/.temp/pgdelta/` apply only to that opt-out. +- With `PGDELTA_DEBUG`, default-engine diagnostic data is stored under + `supabase/.temp/pgdelta/v2/debug//` as `metadata.json` plus available + snapshot, plan, and diagnostics JSON files. These artifacts are never catalog + cache inputs. +- The default engine refuses migration or declarative output when extraction + reports an error or a strict coverage gap (`unmodeled_kind` or + `unresolved_security_label`). The refusal names the diagnostic, and debug + artifacts are saved first when capture is enabled. +- New-engine SQL bytes and transaction-split filenames may differ. Successful + execution and convergence on a subsequent pull/diff are the contract. + ## Files Read -| Path | Format | When | -| -------------------------------------- | ---------- | --------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | -| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | -| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | linked ref resolution | +| Path | Format | When | +| ----------------------------------------------- | ---------- | --------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | +| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | +| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | linked ref resolution | +| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only: edge-runtime image tag | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: migrations/baseline catalogs | ## Files Written @@ -39,13 +61,17 @@ Notes/Delegation section below). | ---------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | | `/supabase/database/**` | SQL | `--declarative` | +| `/supabase/database/.pgdelta-export.json` | JSON | default-engine `--declarative` export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: catalog snapshots | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out only: Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default pg-delta engine with `PGDELTA_DEBUG` | | `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | | `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | | `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker -- Edge-runtime container (pg-delta export / pg-delta or migra diff). +- Edge-runtime container (migra, or pg-delta only under the legacy opt-out). - Shadow Postgres container (provisioned + torn down via the Go `db __shadow` seam). - `supabase/migra` container — the migra OOM bash fallback only. - `pg_dump` container — the initial-migra pull's native remote-schema dump @@ -69,7 +95,8 @@ Notes/Delegation section below). | `SUPABASE_DB_PASSWORD` | remote DB password (overridden by `-p`) | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta diff engine | no | | `SUPABASE_EXPERIMENTAL` | selects the deprecated structured-dump branch (still delegates to Go, see below) | no | -| `PGDELTA_NPM_REGISTRY` | scoped npm registry for edge-runtime | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for the legacy edge-runtime engine | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: scoped npm registry for edge-runtime | no | ## Exit Codes diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index c510d15bca..baa9830f33 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -58,14 +58,20 @@ import { legacyFormatMigrationTimestamp, legacyGetMigrationPath, } from "../../../shared/legacy-migration-file.ts"; -import { legacyFormatDebugId } from "../shared/legacy-debug-bundle.ts"; +import { legacyDebugBundleMessage, legacyFormatDebugId } from "../shared/legacy-debug-bundle.ts"; +import type { LegacyPgDeltaContext } from "../shared/legacy-pgdelta.ts"; import { - type LegacyPgDeltaContext, - legacyDeclarativeExportPgDelta, - legacyDiffPgDelta, - legacyExportCatalogPgDelta, - legacyIsPgDeltaDebugEnabled, -} from "../shared/legacy-pgdelta.ts"; + LegacyPgDeltaEngine, + type LegacyPgDeltaDatabaseEndpoint, + type LegacyPgDeltaExportManifest, + type LegacyPgDeltaSqlFile, +} from "../shared/legacy-pgdelta-engine.service.ts"; +import { + LegacyLoadPgDeltaSqlFiles, + LegacyLoadPgDeltaSqlPaths, + LegacyReadPgDeltaExportManifest, +} from "../shared/legacy-pgdelta-files.ts"; +import { legacyIsPgDeltaDebugEnabled } from "../shared/legacy-pgdelta.ts"; import { legacySaveEmptyPgDeltaPullDebug } from "./pull.debug.ts"; import { LegacyDeclarativeSeam } from "../shared/legacy-pgdelta.seam.service.ts"; import type { LegacyDbPullFlags } from "./pull.command.ts"; @@ -160,6 +166,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy const resolver = yield* LegacyDbConfigResolver; const connection = yield* LegacyDbConnection; const seam = yield* LegacyDeclarativeSeam; + const pgDeltaEngine = yield* LegacyPgDeltaEngine; const proxy = yield* LegacyGoProxy; const cliConfig = yield* LegacyCliConfig; const telemetryState = yield* LegacyTelemetryState; @@ -304,9 +311,15 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // `db..` connection (Go's `PoolerFallbackEligible` + // `ProjectRefFromDirectDbHost`). The error message embeds the container stderr // (edge-runtime/migra errors wrap it), which is what Go classifies. + const targetEndpoint: LegacyPgDeltaDatabaseEndpoint = { + kind: "database", + ref: targetUrl, + connection: resolved.conn, + connectOptions: { isLocal: resolved.isLocal, dnsResolver }, + }; const withPoolerFallback = ( - directTarget: string, - attempt: (targetRef: string) => Effect.Effect, + directTarget: LegacyPgDeltaDatabaseEndpoint, + attempt: (target: LegacyPgDeltaDatabaseEndpoint) => Effect.Effect, ) => attempt(directTarget).pipe( Effect.catch((error) => @@ -333,7 +346,12 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy .pipe(Effect.orElseSucceed(() => Option.none())); if (Option.isSome(pooler)) { yield* legacyEmitPoolerFallbackWarning(resolved.conn.host); - return yield* attempt(legacyToPostgresURL(pooler.value)); + return yield* attempt({ + kind: "database", + ref: legacyToPostgresURL(pooler.value), + connection: pooler.value, + connectOptions: { isLocal: false, dnsResolver }, + }); } } return yield* Effect.fail(error); @@ -407,23 +425,17 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy yield* output.raw("Preparing declarative schema export using pg-delta...\n", "stderr"); const declarativeDirRel = legacyResolveDeclarativeDir(path, toml.pgDelta); const declarativeDir = path.resolve(cliConfig.workdir, declarativeDirRel); - const shadow = yield* seam.provisionShadow({ - mode: "declarative", - targetLocal: false, - usePgDelta: true, - schema: flags.schema, - // Linked path only: merge the same `[remotes.]` override into the - // shadow baseline (Go builds the shadow from the remote-merged config). - projectRef: connType === "linked" ? linkedRef : undefined, - }); - const exported = yield* withPoolerFallback(targetUrl, (targetRef) => - legacyDeclarativeExportPgDelta(ctx, { - sourceRef: shadow.sourceUrl, - targetRef, + const exported = yield* withPoolerFallback(targetEndpoint, (target) => + pgDeltaEngine.exportDeclarativeSchema({ + context: ctx, + target, schema: flags.schema, formatOptions, + projectRef: connType === "linked" ? linkedRef : undefined, + debug: legacyIsPgDeltaDebugEnabled(), + noCache: false, }), - ).pipe(Effect.ensuring(seam.removeShadowContainer(shadow.container))); + ); yield* legacyWriteDeclarativeSchemas(fs, path, declarativeDir, exported).pipe( Effect.mapError((cause) => new LegacyDbPullWriteError({ message: cause.message })), ); @@ -472,7 +484,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // dumped statement with a PostgreSQL DDL AST parser (`multigres`, ~50 node // types) to route objects into structured files. No Postgres DDL parser // exists in TS yet, and `--declarative` already covers the same per-object - // outcome via pg-delta catalog introspection, so this path is deprecated + // outcome via pg-delta managed-state extraction, so this path is deprecated // rather than ported (CLI-1957) — see the deprecation line printed above. if (delegatesExperimentalPull) { // Go's structured-dump path returns before writing a migration or @@ -621,76 +633,87 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // (`internal/db/diff/diff.go:189,234-237`); the shadow seam doesn't, so the // pull handler emits them itself to match the migration-style `db pull` output. yield* output.raw("Creating shadow database...\n", "stderr"); - const shadow = yield* seam.provisionShadow({ - mode: "diff", - // Mirror Go's `DiffDatabase` → `PrepareShadowSource(ctx, schema, - // utils.IsLocalDatabase(config), …)` (`internal/db/diff/diff.go:190`): - // a local target with declarative schema files gets a second - // `contrib_regression` shadow returned as the target override. - targetLocal: resolved.isLocal, - usePgDelta: usePgDeltaDiff, - schema: diffSchema, - // Linked path only: merge the same `[remotes.]` override into the - // shadow baseline (Go builds the shadow from the remote-merged config). - projectRef: connType === "linked" ? linkedRef : undefined, - }); - const diffOutcome = yield* Effect.gen(function* () { - // Use the declarative target override when present (Go substitutes it - // for the diff target, `diff.go:196-197`); for remote pulls it's - // undefined, so this is the direct target URL as before. - const target = shadow.targetUrlOverride ?? targetUrl; - yield* output.raw( - diffSchema.length > 0 - ? `Diffing schemas: ${diffSchema.join(",")}\n` - : "Diffing schemas...\n", - "stderr", - ); - return yield* withPoolerFallback(target, (targetRef) => - // Wrap the engine choice in a gen so both branches' error/requirement - // channels unify into one `Effect` the helper can retry generically. - Effect.gen(function* () { - if (usePgDeltaDiff) { - // With PGDELTA_DEBUG set, capture the shadow baseline catalog so an - // empty diff can be inspected later (Go's DiffDatabase, - // `internal/db/diff/diff.go:205-214`); a failed export only warns. - const debug = legacyIsPgDeltaDebugEnabled(); - const sourceCatalog = debug - ? yield* legacyExportCatalogPgDelta(ctx, { - targetRef: shadow.sourceUrl, - role: "postgres", - }).pipe( - Effect.catch((error) => - output - .raw( - `Warning: failed to export shadow pg-delta catalog: ${error.message}\n`, - "stderr", - ) - .pipe(Effect.as(undefined)), - ), - ) - : undefined; - const result = yield* legacyDiffPgDelta(ctx, { - sourceRef: shadow.sourceUrl, - targetRef, - schema: diffSchema, - formatOptions, - }); - return { - sql: result.sql, - files: result.files, - capture: debug ? { sourceCatalog, stderr: result.stderr } : undefined, - }; + yield* output.raw( + diffSchema.length > 0 + ? `Diffing schemas: ${diffSchema.join(",")}\n` + : "Diffing schemas...\n", + "stderr", + ); + let declarativeFiles: ReadonlyArray | undefined; + let declarativeManifest: LegacyPgDeltaExportManifest | undefined; + if (usePgDeltaDiff && pgDeltaEngine.implementation === "next" && resolved.isLocal) { + if (toml.migrationSchemaPaths !== undefined && toml.migrationSchemaPaths.length > 0) { + declarativeFiles = yield* LegacyLoadPgDeltaSqlPaths( + fs, + path, + cliConfig.workdir, + toml.migrationSchemaPaths, + ); + } else { + const declarativeDirSetting = legacyResolveDeclarativeDir(path, toml.pgDelta); + const declarativeDir = path.isAbsolute(declarativeDirSetting) + ? declarativeDirSetting + : path.join(cliConfig.workdir, declarativeDirSetting); + const hasDeclarativeDir = toml.pgDelta.enabled + ? yield* fs.exists(declarativeDir).pipe(Effect.orElseSucceed(() => false)) + : false; + if (hasDeclarativeDir) { + const loaded = yield* LegacyLoadPgDeltaSqlFiles(fs, path, declarativeDir); + if (loaded.length > 0) { + declarativeFiles = loaded; + declarativeManifest = yield* LegacyReadPgDeltaExportManifest( + fs, + path, + declarativeDir, + ); + } + } else { + const schemasDir = path.join(cliConfig.workdir, "supabase", "schemas"); + if (yield* fs.exists(schemasDir).pipe(Effect.orElseSucceed(() => false))) { + const loaded = yield* LegacyLoadPgDeltaSqlFiles(fs, path, schemasDir); + if (loaded.length > 0) declarativeFiles = loaded; } - const sql = yield* legacyDiffMigra(ctx, { + } + } + } + + const diffOutcome = usePgDeltaDiff + ? yield* withPoolerFallback(targetEndpoint, (target) => + pgDeltaEngine.diffDatabase({ + context: ctx, + target, + targetLocal: resolved.isLocal, + schema: diffSchema, + formatOptions, + projectRef: connType === "linked" ? linkedRef : undefined, + debug: legacyIsPgDeltaDebugEnabled(), + ...(declarativeFiles !== undefined ? { declarativeFiles } : {}), + ...(declarativeManifest !== undefined ? { declarativeManifest } : {}), + }), + ) + : yield* Effect.gen(function* () { + const shadow = yield* seam.provisionShadow({ + mode: "diff", + targetLocal: resolved.isLocal, + usePgDelta: false, + schema: diffSchema, + projectRef: connType === "linked" ? linkedRef : undefined, + }); + return yield* legacyDiffMigra(ctx, { source: shadow.sourceUrl, - target: targetRef, + target: shadow.targetUrlOverride ?? targetUrl, schema: diffSchema, connectOptions: { isLocal: resolved.isLocal, dnsResolver }, - }); - return { sql, files: undefined, capture: undefined }; - }), - ); - }).pipe(Effect.ensuring(seam.removeShadowContainer(shadow.container))); + }).pipe( + Effect.map((sql) => ({ + changes: sql.trim().length > 0, + sql, + files: [], + debug: undefined, + })), + Effect.ensuring(seam.removeShadowContainer(shadow.container)), + ); + }); const out = diffOutcome.sql; const diffEmpty = out.trim().length === 0; @@ -702,13 +725,13 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // Go saves a pg-delta debug bundle and embeds its path in the in-sync // error when PGDELTA_DEBUG is set (`internal/db/pull/pull.go:176-185`); a // bundle-save failure falls through to the plain in-sync error. - if (diffOutcome.capture !== undefined) { + if (pgDeltaEngine.implementation === "legacy" && diffOutcome.debug !== undefined) { const debugDir = yield* legacySaveEmptyPgDeltaPullDebug({ ctx, conn: resolved.conn, targetUrl, - sourceCatalog: diffOutcome.capture.sourceCatalog, - pgDeltaStderr: diffOutcome.capture.stderr, + sourceCatalog: diffOutcome.debug.sourceSnapshot, + pgDeltaStderr: diffOutcome.debug.stderr, id: legacyFormatDebugId(yield* Clock.currentTimeMillis), fs, path, @@ -731,6 +754,17 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy ); } } + if ( + pgDeltaEngine.implementation === "next" && + diffOutcome.debug?.directory !== undefined + ) { + yield* output.raw(legacyDebugBundleMessage(diffOutcome.debug.directory), "stderr"); + return yield* Effect.fail( + new LegacyDbPullInSyncError({ + message: `No schema changes found (debug bundle: ${diffOutcome.debug.directory})`, + }), + ); + } return yield* Effect.fail( new LegacyDbPullInSyncError({ message: "No schema changes found" }), ); @@ -756,7 +790,11 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy workdir: cliConfig.workdir, baseMillis: nowMillis, name, - files: planFiles.map((file) => ({ name: file.name, sql: file.sql })), + files: planFiles.map((file) => ({ + name: file.name, + suffix: file.suffix, + sql: file.sql, + })), }).pipe( Effect.mapError((cause) => new LegacyDbPullWriteError({ message: cause.message })), ); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index f22dd4e230..292e640e8f 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -37,6 +37,10 @@ import { LegacyEdgeRuntimeScript, } from "../../../shared/legacy-edge-runtime-script.service.ts"; import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { + LegacyPgDeltaEngine, + LegacyPgDeltaEngineError, +} from "../shared/legacy-pgdelta-engine.service.ts"; import { LegacyDeclarativeSeam } from "../shared/legacy-pgdelta.seam.service.ts"; import type { LegacyDbPullFlags } from "./pull.command.ts"; import { legacyDbPull } from "./pull.handler.ts"; @@ -63,6 +67,8 @@ const pgDeltaDiffEnvelope = ( }); interface SetupOpts { + readonly engineImplementation?: "next" | "legacy"; + readonly nextDebugDirectory?: string; readonly format?: OutputFormat; readonly remoteVersions?: ReadonlyArray; readonly edgeStdout?: string; // diff SQL or declarative export JSON @@ -134,6 +140,116 @@ function setup(workdir: string, opts: SetupOpts = {}) { }), }); + const engineCalls: Array<{ + operation: "diff" | "export"; + targetRef: string; + projectRef?: string; + targetLocal?: boolean; + }> = []; + let engineDiffCount = 0; + const pgDeltaEngine = Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation: opts.engineImplementation ?? "legacy", + diffExplicit: () => Effect.die("diffExplicit unused"), + diffDatabase: (input) => { + engineCalls.push({ + operation: "diff", + targetRef: input.target.ref, + projectRef: input.projectRef, + targetLocal: input.targetLocal, + }); + engineDiffCount += 1; + if (opts.edgeFailFirstWith !== undefined && engineDiffCount === 1) { + return Effect.fail( + new LegacyPgDeltaEngineError({ + message: opts.edgeFailFirstWith, + cause: opts.edgeFailFirstWith, + }), + ); + } + const stdout = opts.edgeStdout ?? ""; + if (stdout.trim().length === 0) { + return Effect.succeed({ + changes: false, + sql: "", + files: [], + ...(process.env["PGDELTA_DEBUG"] !== undefined + ? { + debug: + opts.engineImplementation === "next" + ? { + sourceSnapshot: opts.catalogStdout ?? "", + ...(opts.nextDebugDirectory !== undefined + ? { directory: opts.nextDebugDirectory } + : {}), + } + : { sourceSnapshot: opts.catalogStdout ?? "", stderr: "" }, + } + : {}), + }); + } + try { + const parsed: unknown = JSON.parse(stdout); + if (typeof parsed !== "object" || parsed === null) throw new Error("invalid envelope"); + const rawFiles = Reflect.get(parsed, "files"); + if (!Array.isArray(rawFiles)) throw new Error("invalid envelope"); + const files = rawFiles.map((raw, index) => { + if (typeof raw !== "object" || raw === null) throw new Error("invalid file"); + const sql = Reflect.get(raw, "sql"); + const name = Reflect.get(raw, "name"); + const transactionMode = Reflect.get(raw, "transactionMode"); + if (typeof sql !== "string" || typeof name !== "string") { + throw new Error("invalid file"); + } + return { + sequence: index + 1, + name, + sql, + transactional: transactionMode !== "none", + }; + }); + return Effect.succeed({ + changes: files.length > 0, + sql: files.map((file) => file.sql).join("\n"), + files, + }); + } catch (cause) { + return Effect.fail( + new LegacyPgDeltaEngineError({ + message: "failed to parse pg-delta diff output", + cause, + }), + ); + } + }, + exportDeclarativeSchema: (input) => { + engineCalls.push({ + operation: "export", + targetRef: input.target.ref, + projectRef: input.projectRef, + }); + if (opts.edgeFailFirstWith !== undefined && engineCalls.length === 1) { + return Effect.fail( + new LegacyPgDeltaEngineError({ + message: opts.edgeFailFirstWith, + cause: opts.edgeFailFirstWith, + }), + ); + } + return Effect.succeed({ + files: [{ name: "schemas/public/t.sql", sql: "create table t ();" }], + manifest: { + redactSecrets: true, + scope: "database", + profile: "supabase", + }, + }); + }, + planDeclarativeSchema: () => Effect.die("planDeclarativeSchema unused"), + }), + ); + let edgeRunCount = 0; const edge = Layer.succeed(LegacyEdgeRuntimeScript, { run: (runOpts: LegacyEdgeRuntimeRunOpts) => { @@ -260,6 +376,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { telemetry.layer, cache.layer, seam, + pgDeltaEngine, edge, docker, dbConnection, @@ -303,6 +420,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { execLog, poolerFallbackCalls, dumpCalls, + engineCalls, get edgeRunCount() { return edgeRunCount; }, @@ -366,6 +484,9 @@ describe("legacy db pull", () => { ); expect(streamText(s.out, "stderr")).not.toContain(tmp.current); expect(s.historyUpserts.length).toBe(1); + expect(s.engineCalls).toHaveLength(1); + expect(s.engineCalls[0]?.operation).toBe("diff"); + expect(s.edgeRunCount).toBe(0); expect(streamText(s.out, "stdout")).toContain("Finished supabase db pull."); // The linked ref is pre-loaded (cheap, local-only) before `resolve()` runs, so // the post-run linked-project cache still gets the ref Go would cache via @@ -536,6 +657,8 @@ describe("legacy db pull", () => { const s = setup(tmp.current, { edgeStdout: EXPORT_JSON }); return Effect.gen(function* () { yield* legacyDbPull(flags({ declarative: Option.some(true) })); + expect(s.engineCalls[0]?.operation).toBe("export"); + expect(s.edgeRunCount).toBe(0); const err = streamText(s.out, "stderr"); // Go's order: `ConnectByConfig` prints Connecting (`pull.go:40`), then // `pullDeclarativePgDelta` prints Preparing (`pull.go:93`). @@ -550,7 +673,17 @@ describe("legacy db pull", () => { expect( existsSync(join(tmp.current, "supabase", "database", "schemas", "public", "t.sql")), ).toBe(true); - expect(s.provisionCalls[0]?.mode).toBe("declarative"); + expect( + JSON.parse( + readFileSync(join(tmp.current, "supabase", "database", ".pgdelta-export.json"), "utf8"), + ), + ).toMatchObject({ + formatVersion: 1, + redactSecrets: true, + scope: "database", + files: ["schemas/public/t.sql"], + }); + expect(s.provisionCalls).toHaveLength(0); }).pipe(Effect.provide(s.layer)); }); @@ -667,7 +800,8 @@ describe("legacy db pull", () => { }); return Effect.gen(function* () { yield* legacyDbPull(flags({ declarative: Option.some(true), usePgDelta: Option.some(true) })); - expect(s.provisionCalls[0]?.mode).toBe("declarative"); + expect(s.provisionCalls).toHaveLength(0); + expect(s.engineCalls[0]?.operation).toBe("export"); }).pipe(Effect.provide(s.layer)); }); @@ -934,6 +1068,40 @@ describe("legacy db pull", () => { }).pipe(Effect.provide(s.layer)); }); + it.effect("reports the next-generation debug directory for an empty pg-delta diff", () => { + seedMigration(tmp.current, "20240101000000"); + const debugDir = join( + tmp.current, + "supabase", + ".temp", + "pgdelta", + "v2", + "debug", + "20240102-030405-678-diff", + ); + const s = setup(tmp.current, { + remoteVersions: ["20240101000000"], + edgeStdout: "", + engineImplementation: "next", + nextDebugDirectory: debugDir, + }); + return Effect.gen(function* () { + const previous = process.env["PGDELTA_DEBUG"]; + process.env["PGDELTA_DEBUG"] = "1"; + try { + const error = yield* legacyDbPull(flags({ diffEngine: Option.some("pg-delta") })).pipe( + Effect.flip, + ); + expect(error.message).toBe(`No schema changes found (debug bundle: ${debugDir})`); + expect(streamText(s.out, "stderr")).toContain(`Debug information saved to`); + expect(streamText(s.out, "stderr")).toContain(debugDir); + } finally { + if (previous === undefined) delete process.env["PGDELTA_DEBUG"]; + else process.env["PGDELTA_DEBUG"] = previous; + } + }).pipe(Effect.provide(s.layer)); + }); + it.effect("prompts to update history and inserts on yes (tty)", () => { seedMigration(tmp.current, "20240101000000"); const s = setup(tmp.current, { @@ -1471,7 +1639,7 @@ describe("legacy db pull", () => { }); return Effect.gen(function* () { yield* legacyDbPull(flags()); - expect(s.provisionCalls[0]?.usePgDelta).toBe(true); + expect(s.engineCalls[0]?.operation).toBe("diff"); }).pipe(Effect.provide(s.layer)); }); @@ -1598,10 +1766,10 @@ describe("legacy db pull", () => { }); return Effect.gen(function* () { yield* legacyDbPull(flags({ linked: Option.some(true) })); - expect(s.provisionCalls[0]?.usePgDelta).toBe(true); - // The resolved ref is forwarded to the shadow so the `db __shadow` child - // merges the same `[remotes.]` override into the shadow baseline. - expect(s.provisionCalls[0]?.projectRef).toBe("abcdefghijklmnopqrst"); + expect(s.engineCalls[0]?.operation).toBe("diff"); + // The resolved ref is forwarded through the strategy so the selected + // implementation can build the remote-merged shadow baseline. + expect(s.engineCalls[0]?.projectRef).toBe("abcdefghijklmnopqrst"); }).pipe(Effect.provide(s.layer)); }); @@ -1624,7 +1792,7 @@ describe("legacy db pull", () => { ); expect(streamText(s.out, "stderr")).toContain("does not support IPv6"); expect(streamText(s.out, "stderr")).toContain("Retrying via the IPv4 connection pooler"); - expect(s.edgeRunCount).toBe(2); + expect(s.engineCalls.filter((call) => call.operation === "diff")).toHaveLength(2); expect(streamText(s.out, "stderr")).toMatch( /Schema written to supabase[/\\]migrations[/\\]\d{14}_remote_schema\.sql\n/u, ); @@ -1642,7 +1810,7 @@ describe("legacy db pull", () => { return Effect.gen(function* () { yield* legacyDbPull(flags({ linked: Option.some(true), declarative: Option.some(true) })); expect(streamText(s.out, "stderr")).toContain("Retrying via the IPv4 connection pooler"); - expect(s.edgeRunCount).toBe(2); + expect(s.engineCalls.filter((call) => call.operation === "export")).toHaveLength(2); expect(streamText(s.out, "stderr")).toContain( `Declarative schema written to ${join("supabase", "database")}\n`, ); @@ -1665,7 +1833,7 @@ describe("legacy db pull", () => { ).pipe(Effect.exit); expect(Exit.isFailure(exit)).toBe(true); expect(streamText(s.out, "stderr")).not.toContain("Retrying via the IPv4 connection pooler"); - expect(s.edgeRunCount).toBe(1); + expect(s.engineCalls.filter((call) => call.operation === "diff")).toHaveLength(1); }).pipe(Effect.provide(s.layer)); }); @@ -1685,7 +1853,7 @@ describe("legacy db pull", () => { ).pipe(Effect.exit); expect(Exit.isFailure(exit)).toBe(true); expect(s.poolerFallbackCalls).toHaveLength(0); - expect(s.edgeRunCount).toBe(1); + expect(s.engineCalls.filter((call) => call.operation === "diff")).toHaveLength(1); }).pipe(Effect.provide(s.layer)); }); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.layers.ts b/apps/cli/src/legacy/commands/db/pull/pull.layers.ts index 821fd07acd..6a9401ac98 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.layers.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.layers.ts @@ -13,11 +13,15 @@ import { legacyPgDeltaSslProbeLayer } from "../../../shared/legacy-pgdelta-ssl-p import { legacyTelemetryStateLayer } from "../../../telemetry/legacy-telemetry-state.layer.ts"; import { stdinLayer } from "../../../../shared/runtime/stdin.layer.ts"; import { legacyDeclarativeSeamLayer } from "../shared/legacy-pgdelta.seam.layer.ts"; +import { legacyPgDeltaEngineLayer } from "../shared/legacy-pgdelta-engine.layer.ts"; +import { legacyPgDeltaNextAdapterLayer } from "../shared/legacy-pgdelta-next-adapter.layer.ts"; +import { legacyPgDeltaNextShadowLayer } from "../shared/legacy-pgdelta-next-shadow.layer.ts"; /** * Runtime layer for `supabase db pull`. Same composition as `db diff`: the - * db-config resolver, the native pg-delta / migra stack (edge-runtime, SSL probe, - * the Go shadow seam), `LegacyDbConnection` (remote connect + `schema_migrations` + * db-config resolver, both pg-delta implementations, migra, the SSL probe, and + * the Go shadow seam. The default pg-delta runs in-process; edge-runtime remains + * for migra and the explicit legacy opt-out. `LegacyDbConnection` (remote connect + `schema_migrations` * reconciliation / history update), and `LegacyDockerRun` for the migra fallback. */ const cliConfig = legacyCliConfigLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); @@ -35,6 +39,15 @@ const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( ); const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); +const nextShadow = legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seam)); +const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(legacyPgDeltaNextAdapterLayer), + Layer.provide(nextShadow), + Layer.provide(edgeRuntime), + Layer.provide(legacyPgDeltaSslProbeLayer), + Layer.provide(seam), + Layer.provide(legacyDebugLoggerLayer), +); export const legacyDbPullRuntimeLayer = Layer.mergeAll( dbConfig, @@ -43,6 +56,7 @@ export const legacyDbPullRuntimeLayer = Layer.mergeAll( edgeRuntime, legacyPgDeltaSslProbeLayer, seam, + pgDeltaEngine, cliConfig, legacyIdentityStitchLayer, legacyTelemetryStateLayer, diff --git a/apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md index d193dd88b6..58f37c142d 100644 --- a/apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md @@ -4,26 +4,33 @@ Native TypeScript port of `apps/cli-go/internal/db/push/push.go`. Applies pendin local migrations (and optionally seed data and custom roles) to the local or linked/remote Postgres database. +Pg-delta's default bundled engine has no reusable migrations-catalog consumer, +so a normal push does not warm a pg-delta cache or start edge-runtime. Setting +`SUPABASE_USE_PG_DELTA_NEXT=false` preserves Go's legacy best-effort catalog +warmup. Legacy catalogs remain directly under `supabase/.temp/pgdelta/` and are +never read by the default engine. + ## Files Read -| Path | Format | When | -| ------------------------------------- | ---------- | ----------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (embedded defaults used when absent) | -| `~/.supabase//project-ref` | plain text | on the `--linked` path (and the default target), to resolve the ref | -| `~/.supabase/access-token` | plain text | when `SUPABASE_ACCESS_TOKEN` unset and a linked temp-role is minted | -| `/supabase/migrations/` | directory | when `[db.migrations].enabled` (default true), to list local files | -| `/supabase/migrations/*.sql` | SQL | for each pending migration, when applied (and not `--dry-run`) | -| seed files from `[db.seed].sql_paths` | SQL | when `--include-seed` and `[db.seed].enabled` (paths under `supabase/`) | -| `/supabase/roles.sql` | SQL | when `--include-roles` (existence check + apply) | +| Path | Format | When | +| ------------------------------------------ | ---------- | ----------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (embedded defaults used when absent) | +| `~/.supabase//project-ref` | plain text | on the `--linked` path (and the default target), to resolve the ref | +| `~/.supabase/access-token` | plain text | when `SUPABASE_ACCESS_TOKEN` unset and a linked temp-role is minted | +| `/supabase/migrations/` | directory | when `[db.migrations].enabled` (default true), to list local files | +| `/supabase/migrations/*.sql` | SQL | for each pending migration, when applied (and not `--dry-run`) | +| seed files from `[db.seed].sql_paths` | SQL | when `--include-seed` and `[db.seed].enabled` (paths under `supabase/`) | +| `/supabase/roles.sql` | SQL | when `--include-roles` (existence check + apply) | +| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy opt-out only | ## Files Written -| Path | Format | When | -| ------------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `~/.supabase//linked-project.json` | JSON | on the `--linked` path (post-run cache, Go's `ensureProjectGroupsCached`) | -| `~/.supabase/telemetry.json` | JSON | always (post-run telemetry flush) | -| `/supabase/.temp/pgdelta/catalog--migrations--.json` | JSON | best-effort, after a successful migration apply, when pg-delta is enabled (`[experimental.pgdelta] enabled` or `SUPABASE_EXPERIMENTAL_PG_DELTA`); a failure only warns on stderr and never fails the push (Go's `pgcache.TryCacheMigrationsCatalog`) | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | same gate as above, when the target requires SSL (`legacyPreparePgDeltaRef`) | +| Path | Format | When | +| ------------------------------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `~/.supabase//linked-project.json` | JSON | on the `--linked` path (post-run cache, Go's `ensureProjectGroupsCached`) | +| `~/.supabase/telemetry.json` | JSON | always (post-run telemetry flush) | +| `/supabase/.temp/pgdelta/catalog--migrations--.json` | JSON | legacy opt-out only, best-effort after a successful migration apply when pg-delta is enabled; failure only warns (Go's `pgcache.TryCacheMigrationsCatalog`) | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out cache export when the target requires SSL | ## Database Mutations @@ -43,14 +50,15 @@ linked/remote Postgres database. ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | -| `SUPABASE_ACCESS_TOKEN` | auth token for the `--linked` resolver path | no (falls back to keyring → `~/.supabase/access-token`) | -| `SUPABASE_DB_PASSWORD` | password for the linked/remote connection | no (`--password`/`-p` takes precedence) | -| `SUPABASE_YES` | auto-confirm prompts (Go's `viper YES`) | no (also `--yes`) | -| `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the migrations-catalog cache when `[experimental.pgdelta].enabled` is unset | no (project `.env` or shell) | -| `SUPABASE_INTERNAL_IMAGE_REGISTRY` | overrides the pg-delta edge-runtime image registry for the cache export | no (project `.env` or shell) | -| `PGDELTA_NPM_REGISTRY` | overrides the pg-delta edge-runtime npm registry (`.npmrc` + `NPM_CONFIG_REGISTRY` forward) for the cache export | no (project `.env` or shell) | +| Variable | Purpose | Required? | +| ---------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------- | +| `SUPABASE_ACCESS_TOKEN` | auth token for the `--linked` resolver path | no (falls back to keyring → `~/.supabase/access-token`) | +| `SUPABASE_DB_PASSWORD` | password for the linked/remote connection | no (`--password`/`-p` takes precedence) | +| `SUPABASE_YES` | auto-confirm prompts (Go's `viper YES`) | no (also `--yes`) | +| `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the legacy opt-out cache when `[experimental.pgdelta].enabled` is unset | no (project `.env` or shell) | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` to retain the legacy migrations-catalog warmup | no (project `.env` or shell) | +| `SUPABASE_INTERNAL_IMAGE_REGISTRY` | legacy opt-out only: overrides the edge-runtime image registry for the cache export | no (project `.env` or shell) | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: overrides the edge-runtime npm registry for the cache export | no (project `.env` or shell) | ## Exit Codes @@ -117,12 +125,15 @@ stdout is payload-only. A single `result` object is emitted: directly into TS in PR supabase/cli#5671 (landed on develop as `b48fad60`) and back-ported to the pinned `apps/cli-go` oracle under the CLI-1989 parity ruling (2026-07-30). -- **Migrations catalog cache**: ported (Go's best-effort `pgcache.TryCacheMigrationsCatalog`). - After a successful migration apply, when pg-delta is enabled, exports the target's - pg-delta catalog via the edge-runtime stack and writes it under +- **Migrations catalog cache**: retained only for + `SUPABASE_USE_PG_DELTA_NEXT=false` (Go's best-effort + `pgcache.TryCacheMigrationsCatalog`). After a successful migration apply, when + pg-delta is enabled, the legacy implementation exports the target's catalog via + the edge-runtime stack and writes it under `supabase/.temp/pgdelta/`, pruning older snapshots for the same prefix (retains 2). A failure only warns on stderr (`Warning: failed to cache migrations catalog: …`) and never fails the push, matching Go exactly. Reuses `legacyExportCatalogPgDelta` (the same pg-delta export path `db pull`/`db diff` use, which always mounts the project root at `/workspace`) rather than a second copy, so the ENOENT bug fixed in - Go's `pgcache/cache.go` (supabase/cli#5921) has no TS equivalent. + Go's `pgcache/cache.go` (supabase/cli#5921) has no TS equivalent. The bundled + default engine extracts live state for commands that need it and never reads this cache. diff --git a/apps/cli/src/legacy/commands/db/push/push.integration.test.ts b/apps/cli/src/legacy/commands/db/push/push.integration.test.ts index 97731ea39a..3abcc35f87 100644 --- a/apps/cli/src/legacy/commands/db/push/push.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/push/push.integration.test.ts @@ -357,12 +357,25 @@ describe("legacy db push", () => { }); }); + it.live("does not start edge-runtime for the obsolete catalog warmup under default next", () => { + const { layer, edgeRunCalls } = setup(tmp.current, { + toml: 'project_id = "test"\n[experimental.pgdelta]\nenabled = true\n', + files: migrationFile("20240101000000"), + confirm: [true], + }); + return Effect.gen(function* () { + yield* legacyDbPush(DEFAULT_FLAGS).pipe(Effect.provide(layer)); + expect(edgeRunCalls).toHaveLength(0); + expect(existsSync(join(tmp.current, "supabase", ".temp", "pgdelta"))).toBe(false); + }); + }); + it.live("caches the migrations catalog when project .env enables pg-delta", () => { const { layer, out, edgeRunCalls } = setup(tmp.current, { toml: 'project_id = "test"\n', files: { ...migrationFile("20240101000000"), - "supabase/.env": "SUPABASE_EXPERIMENTAL_PG_DELTA=true\n", + "supabase/.env": "SUPABASE_EXPERIMENTAL_PG_DELTA=true\nSUPABASE_USE_PG_DELTA_NEXT=false\n", }, confirm: [true], catalogStdout: '{"snapshot":"ok"}', @@ -382,7 +395,10 @@ describe("legacy db push", () => { it.live("caches the migrations catalog after a successful push when pg-delta is enabled", () => { const { layer, out, edgeRunCalls } = setup(tmp.current, { toml: 'project_id = "test"\n[experimental.pgdelta]\nenabled = true\n', - files: migrationFile("20240101000000"), + files: { + ...migrationFile("20240101000000"), + "supabase/.env": "SUPABASE_USE_PG_DELTA_NEXT=false\n", + }, confirm: [true], catalogStdout: '{"snapshot":"ok"}', }); @@ -404,7 +420,10 @@ describe("legacy db push", () => { () => { const { layer, out, edgeRunCalls } = setup(tmp.current, { toml: 'project_id = "test"\n[experimental.pgdelta]\nenabled = true\n', - files: migrationFile("20240101000000"), + files: { + ...migrationFile("20240101000000"), + "supabase/.env": "SUPABASE_USE_PG_DELTA_NEXT=false\n", + }, confirm: [true], catalogStdout: '{"snapshot":"ok"}', noProjectId: true, @@ -426,7 +445,10 @@ describe("legacy db push", () => { () => { const { layer, out, edgeRunCalls } = setup(tmp.current, { toml: "[experimental.pgdelta]\nenabled = true\n", - files: migrationFile("20240101000000"), + files: { + ...migrationFile("20240101000000"), + "supabase/.env": "SUPABASE_USE_PG_DELTA_NEXT=false\n", + }, confirm: [true], catalogStdout: '{"snapshot":"ok"}', noProjectId: true, @@ -457,7 +479,10 @@ describe("legacy db push", () => { args: ["db", "push", "--linked"], isLocal: false, projectRef: LEGACY_VALID_REF, - files: migrationFile("20240101000000"), + files: { + ...migrationFile("20240101000000"), + "supabase/.env": "SUPABASE_USE_PG_DELTA_NEXT=false\n", + }, confirm: [true], catalogStdout: '{"snapshot":"ok"}', noProjectId: true, @@ -478,7 +503,10 @@ describe("legacy db push", () => { it.live("sanitizes an invalid config.toml project_id before naming the pg-delta volume", () => { const { layer, out, edgeRunCalls } = setup(tmp.current, { toml: 'project_id = "my app"\n[experimental.pgdelta]\nenabled = true\n', - files: migrationFile("20240101000000"), + files: { + ...migrationFile("20240101000000"), + "supabase/.env": "SUPABASE_USE_PG_DELTA_NEXT=false\n", + }, confirm: [true], catalogStdout: '{"snapshot":"ok"}', noProjectId: true, @@ -498,7 +526,10 @@ describe("legacy db push", () => { it.live("warns without failing the push when the catalog export fails", () => { const { layer, out } = setup(tmp.current, { toml: 'project_id = "test"\n[experimental.pgdelta]\nenabled = true\n', - files: migrationFile("20240101000000"), + files: { + ...migrationFile("20240101000000"), + "supabase/.env": "SUPABASE_USE_PG_DELTA_NEXT=false\n", + }, confirm: [true], catalogExportFailWith: "edge-runtime script produced no output", }); @@ -521,7 +552,8 @@ describe("legacy db push", () => { toml: 'project_id = "test"\n[experimental.pgdelta]\nenabled = true\n', files: { ...migrationFile("20240101000000"), - "supabase/.env": "SUPABASE_INTERNAL_IMAGE_REGISTRY=my-mirror.example.com\n", + "supabase/.env": + "SUPABASE_INTERNAL_IMAGE_REGISTRY=my-mirror.example.com\nSUPABASE_USE_PG_DELTA_NEXT=false\n", }, confirm: [true], catalogStdout: '{"snapshot":"ok"}', diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 2b581df3b0..0828abc62a 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -1,4 +1,4 @@ -import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { BunServices } from "@effect/platform-bun"; @@ -10,6 +10,11 @@ import { LegacyEdgeRuntimeScript, } from "../../../../shared/legacy-edge-runtime-script.service.ts"; import { LegacyPgDeltaSslProbe } from "../../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { legacyPgDeltaLegacyEngineLayer } from "../../shared/legacy-pgdelta-engine.legacy.layer.ts"; +import { + LegacyPgDeltaEngine, + type LegacyPgDeltaDeclarativePlanInput, +} from "../../shared/legacy-pgdelta-engine.service.ts"; import { type LegacyCatalogMode, LegacyDeclarativeSeam, @@ -73,9 +78,67 @@ const ctx = (declarativeDir: string): LegacyDeclarativeRunContext => ({ declarativeDir, schema: [], noCache: false, + debug: false, + dnsResolver: "native", }); +const engineLayer = ( + seam: Layer.Layer, + edge: Layer.Layer, +) => + legacyPgDeltaLegacyEngineLayer.pipe( + Layer.provide(Layer.mergeAll(seam, edge, probe, BunServices.layer)), + ); + describe("legacyDiffDeclarativeToMigrations", () => { + it.effect("loads nested SQL and its manifest in stable order for the engine", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-orch-")); + const declDir = join(dir, "supabase", "database"); + mkdirSync(join(declDir, "nested"), { recursive: true }); + writeFileSync(join(declDir, "z.sql"), "select 'z';"); + writeFileSync(join(declDir, "nested", "a.sql"), "select 'a';"); + writeFileSync(join(declDir, "ignored.txt"), "ignored"); + writeFileSync( + join(declDir, ".pgdelta-export.json"), + JSON.stringify({ formatVersion: 1, redactSecrets: true, scope: "database" }), + ); + const calls: LegacyPgDeltaDeclarativePlanInput[] = []; + const engine = Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation: "next", + diffExplicit: () => Effect.die("diffExplicit not used"), + diffDatabase: () => Effect.die("diffDatabase not used"), + exportDeclarativeSchema: () => Effect.die("exportDeclarativeSchema not used"), + planDeclarativeSchema: (input) => { + calls.push(input); + return Effect.succeed({ + changes: false, + sql: "", + files: [], + sourceRef: "migrations", + targetRef: "declarative", + }); + }, + }), + ); + return legacyDiffDeclarativeToMigrations({ ...ctx(declDir), debug: true, noCache: true }).pipe( + Effect.tap(() => + Effect.sync(() => { + expect(calls[0]?.files).toEqual([ + { name: "nested/a.sql", sql: "select 'a';" }, + { name: "z.sql", sql: "select 'z';" }, + ]); + expect(calls[0]?.manifest).toEqual({ redactSecrets: true, scope: "database" }); + expect(calls[0]?.debug).toBe(true); + expect(calls[0]?.noCache).toBe(true); + rmSync(dir, { recursive: true, force: true }); + }), + ), + Effect.provide(Layer.mergeAll(engine, BunServices.layer)), + ); + }); + it.effect("provisions migrations + declarative catalogs via the seam and diffs them", () => { const dir = mkdtempSync(join(tmpdir(), "legacy-decl-orch-")); const declDir = join(dir, "supabase", "database"); @@ -100,7 +163,15 @@ describe("legacyDiffDeclarativeToMigrations", () => { rmSync(dir, { recursive: true, force: true }); }), ), - Effect.provide(Layer.mergeAll(seam.layer, edge.layer, probe, BunServices.layer)), + Effect.provide( + Layer.mergeAll( + seam.layer, + edge.layer, + probe, + engineLayer(seam.layer, edge.layer), + BunServices.layer, + ), + ), ); }); @@ -123,12 +194,48 @@ describe("legacyDiffDeclarativeToMigrations", () => { rmSync(dir, { recursive: true, force: true }); }), ), - Effect.provide(Layer.mergeAll(seam.layer, edge.layer, probe, BunServices.layer)), + Effect.provide( + Layer.mergeAll( + seam.layer, + edge.layer, + probe, + engineLayer(seam.layer, edge.layer), + BunServices.layer, + ), + ), ); }); }); describe("legacyGenerateDeclarativeOutput", () => { + it.effect("propagates debug and no-cache to the selected engine", () => { + const calls: Array<{ readonly debug: boolean; readonly noCache: boolean }> = []; + const engine = Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation: "next", + diffExplicit: () => Effect.die("diffExplicit not used"), + diffDatabase: () => Effect.die("diffDatabase not used"), + exportDeclarativeSchema: (input) => { + calls.push({ debug: input.debug, noCache: input.noCache }); + return Effect.succeed({ files: [] }); + }, + planDeclarativeSchema: () => Effect.die("planDeclarativeSchema not used"), + }), + ); + return legacyGenerateDeclarativeOutput( + { ...ctx("/proj/supabase/database"), debug: true, noCache: true }, + { + kind: "database", + ref: "postgresql://postgres:postgres@127.0.0.1:54322/postgres", + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + ).pipe( + Effect.tap(() => Effect.sync(() => expect(calls).toEqual([{ debug: true, noCache: true }]))), + Effect.provide(engine), + ); + }); + it.effect("diffs the baseline catalog against the live DB and returns files", () => { const seam = mockSeam({ migrations: "m", @@ -141,14 +248,15 @@ describe("legacyGenerateDeclarativeOutput", () => { files: [{ path: "public.sql", order: 0, statements: 1, sql: "create table a();" }], }; const edge = mockEdge(JSON.stringify(payload)); - return legacyGenerateDeclarativeOutput( - ctx("/proj/supabase/database"), - "postgresql://postgres:postgres@127.0.0.1:54322/postgres?connect_timeout=10", - ).pipe( + return legacyGenerateDeclarativeOutput(ctx("/proj/supabase/database"), { + kind: "database", + ref: "postgresql://postgres:postgres@127.0.0.1:54322/postgres?connect_timeout=10", + connectOptions: { isLocal: true, dnsResolver: "native" }, + }).pipe( Effect.tap((output) => Effect.sync(() => { expect(seam.calls).toEqual([{ mode: "baseline", noCache: false }]); - expect(output.files[0]?.path).toBe("public.sql"); + expect(output.files[0]?.name).toBe("public.sql"); // SOURCE = baseline catalog (mapped to /workspace); TARGET = live URL (passthrough). expect(edge.calls[0]!.env["SOURCE"]).toBe("/workspace/supabase/.temp/pgdelta/base.json"); expect(edge.calls[0]!.env["TARGET"]).toBe( @@ -156,7 +264,15 @@ describe("legacyGenerateDeclarativeOutput", () => { ); }), ), - Effect.provide(Layer.mergeAll(seam.layer, edge.layer, probe, BunServices.layer)), + Effect.provide( + Layer.mergeAll( + seam.layer, + edge.layer, + probe, + engineLayer(seam.layer, edge.layer), + BunServices.layer, + ), + ), ); }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 954dc76ec6..522c38e7d9 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -1,101 +1,91 @@ -import { Effect, FileSystem } from "effect"; +import { Effect, FileSystem, Path } from "effect"; +import { legacyFindDropStatements } from "../../../../shared/legacy-sql-split.ts"; +import { + LegacyPgDeltaEngine, + type LegacyPgDeltaDatabaseEndpoint, + type LegacyPgDeltaRenderedFile, +} from "../../shared/legacy-pgdelta-engine.service.ts"; import { - type LegacyPgDeltaContext, - legacyDeclarativeExportPgDelta, - legacyDiffPgDelta, -} from "../../shared/legacy-pgdelta.ts"; + LegacyLoadPgDeltaSqlFiles, + LegacyReadPgDeltaExportManifest, +} from "../../shared/legacy-pgdelta-files.ts"; +import type { LegacyPgDeltaContext } from "../../shared/legacy-pgdelta.ts"; import { LegacyDeclarativeDiffError } from "./declarative.errors.ts"; -import { LegacyDeclarativeSeam } from "../../shared/legacy-pgdelta.seam.service.ts"; -import { legacyFindDropStatements } from "../../../../shared/legacy-sql-split.ts"; /** Ambient inputs shared by the orchestration steps. */ export interface LegacyDeclarativeRunContext { readonly pgDelta: LegacyPgDeltaContext; - /** `experimental.pgdelta.format_options` (trimmed; "" when unset). */ readonly formatOptions: string; - /** Resolved declarative schema dir (workdir-relative, e.g. `supabase/database`). */ readonly declarativeDir: string; readonly schema: ReadonlyArray; readonly noCache: boolean; - /** - * Resolved linked project ref for an explicit `generate --linked`. Threaded into - * the baseline `__catalog` export so the Go config load merges the matching - * `[remotes.]` override into the platform baseline (auth/storage/realtime/api/ - * vault settings), matching Go's `Generate`, which builds the baseline from the - * remote-merged config. `undefined` for local/db-url/smart targets. - */ + readonly debug: boolean; + readonly dnsResolver: "native" | "https"; readonly linkedProjectRef?: string; } /** The output of a declarative-to-migrations diff. Mirrors Go's `SyncResult`. */ export interface LegacyDeclarativeSyncResult { readonly diffSQL: string; + readonly files: ReadonlyArray; readonly sourceRef: string; readonly targetRef: string; readonly dropWarnings: ReadonlyArray; } -/** - * Computes the diff between local migrations state and the declarative schema. - * Mirrors Go's `DiffDeclarativeToMigrations` (`declarative.go:170`): the - * migrations catalog (source) and declarative catalog (target) are provisioned - * via the Go seam (shadow DB + `SetupDatabase` + migrate / apply), then diffed - * natively with pg-delta. - */ +const declarativeError = (message: string) => new LegacyDeclarativeDiffError({ message }); + export const legacyDiffDeclarativeToMigrations = Effect.fnUntraced(function* ( run: LegacyDeclarativeRunContext, ) { const fs = yield* FileSystem.FileSystem; - const seam = yield* LegacyDeclarativeSeam; - + const path = yield* Path.Path; + const engine = yield* LegacyPgDeltaEngine; const exists = yield* fs.exists(run.declarativeDir).pipe(Effect.orElseSucceed(() => false)); if (!exists) { return yield* Effect.fail( - new LegacyDeclarativeDiffError({ - message: - "No declarative schema directory found. Run supabase db schema declarative generate first.", - }), + declarativeError( + "No declarative schema directory found. Run supabase db schema declarative generate first.", + ), ); } - - const sourceRef = yield* seam.exportCatalog({ mode: "migrations", noCache: run.noCache }); - const targetRef = yield* seam.exportCatalog({ mode: "declarative", noCache: run.noCache }); - const diff = yield* legacyDiffPgDelta(run.pgDelta, { - sourceRef, - targetRef, + const files = yield* LegacyLoadPgDeltaSqlFiles(fs, path, run.declarativeDir).pipe( + Effect.mapError((error) => declarativeError(error.message)), + ); + const manifest = yield* LegacyReadPgDeltaExportManifest(fs, path, run.declarativeDir).pipe( + Effect.mapError((error) => declarativeError(error.message)), + ); + const result = yield* engine.planDeclarativeSchema({ + context: run.pgDelta, schema: run.schema, formatOptions: run.formatOptions, + debug: run.debug, + files, + noCache: run.noCache, + ...(manifest !== undefined ? { manifest } : {}), }); return { - diffSQL: diff.sql, - sourceRef, - targetRef, - dropWarnings: legacyFindDropStatements(diff.sql), + diffSQL: result.sql, + files: result.files, + sourceRef: result.sourceRef, + targetRef: result.targetRef, + dropWarnings: legacyFindDropStatements(result.sql), } satisfies LegacyDeclarativeSyncResult; }); -/** - * Exports a live database's schema as declarative file payloads, diffing it - * against the platform-baseline catalog (provisioned via the Go seam). Mirrors - * the catalog half of Go's `Generate` (`declarative.go:110`): the live database - * URL is the target, the baseline is the source. The handler writes the - * returned files after the overwrite prompt. - */ export const legacyGenerateDeclarativeOutput = Effect.fnUntraced(function* ( run: LegacyDeclarativeRunContext, - targetDbUrl: string, + target: LegacyPgDeltaDatabaseEndpoint, ) { - const seam = yield* LegacyDeclarativeSeam; - const baselineRef = yield* seam.exportCatalog({ - mode: "baseline", - noCache: run.noCache, - ...(run.linkedProjectRef !== undefined ? { projectRef: run.linkedProjectRef } : {}), - }); - return yield* legacyDeclarativeExportPgDelta(run.pgDelta, { - sourceRef: baselineRef, - targetRef: targetDbUrl, + const engine = yield* LegacyPgDeltaEngine; + return yield* engine.exportDeclarativeSchema({ + context: run.pgDelta, schema: run.schema, formatOptions: run.formatOptions, + debug: run.debug, + noCache: run.noCache, + ...(run.linkedProjectRef !== undefined ? { projectRef: run.linkedProjectRef } : {}), + target, }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.smart-target.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.smart-target.ts index 2a01c96912..4b6b0b8a7c 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.smart-target.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.smart-target.ts @@ -16,6 +16,7 @@ import { } from "../../../../shared/legacy-db-config.parse.ts"; import { legacyGetHostname } from "../../../../shared/legacy-hostname.ts"; import { legacyToPostgresURL } from "../../../../shared/legacy-postgres-url.ts"; +import type { LegacyPgDeltaDatabaseEndpoint } from "../../shared/legacy-pgdelta-engine.service.ts"; import { LegacyDeclarativeApplyError, LegacyDeclarativeInvalidDbUrlError, @@ -46,30 +47,48 @@ export interface LegacySmartTargetFlags { readonly reset: boolean; } -export const legacyLocalUrl = (local: LegacyLocalConn): string => - legacyToPostgresURL({ - // Go derives the local host from `utils.Config.Hostname` (`GetHostname()`: - // SUPABASE_SERVICES_HOSTNAME → tcp DOCKER_HOST → 127.0.0.1), not a hardcoded - // loopback (`apps/cli-go/internal/utils/misc.go:298-312`). - host: legacyGetHostname(), - port: local.port, - user: "postgres", - password: local.password, - database: "postgres", - }); +const legacyLocalConnection = (local: LegacyLocalConn) => ({ + // Go derives the local host from `utils.Config.Hostname` (`GetHostname()`: + // SUPABASE_SERVICES_HOSTNAME → tcp DOCKER_HOST → 127.0.0.1), not a hardcoded + // loopback (`apps/cli-go/internal/utils/misc.go:298-312`). + host: legacyGetHostname(), + port: local.port, + user: "postgres", + password: local.password, + database: "postgres", +}); + +export const legacyLocalEndpoint = ( + local: LegacyLocalConn, + dnsResolver: "native" | "https", +): LegacyPgDeltaDatabaseEndpoint => { + const connection = legacyLocalConnection(local); + return { + kind: "database", + ref: legacyToPostgresURL(connection), + connection, + connectOptions: { isLocal: true, dnsResolver }, + }; +}; -/** Resolves `--linked` / `--db-url` to a Postgres URL via the shared resolver. */ -export const legacyResolveRemoteUrl = Effect.fnUntraced(function* (flags: LegacySmartTargetFlags) { +/** Resolves a remote target without discarding TLS and connection options. */ +export const legacyResolveRemoteEndpoint = Effect.fnUntraced(function* ( + flags: LegacySmartTargetFlags, +) { const resolver = yield* LegacyDbConfigResolver; const dnsResolver = yield* LegacyDnsResolverFlag; const resolved = yield* resolver.resolve({ dbUrl: flags.dbUrl, - // Remote-only resolution: `--db-url` wins, otherwise the linked project. connType: Option.isSome(flags.dbUrl) ? "db-url" : "linked", dnsResolver, password: flags.password, }); - return legacyToPostgresURL(resolved.conn); + return { + kind: "database", + ref: legacyToPostgresURL(resolved.conn), + connection: resolved.conn, + connectOptions: { isLocal: resolved.isLocal, dnsResolver }, + } satisfies LegacyPgDeltaDatabaseEndpoint; }); /** @@ -78,7 +97,7 @@ export const legacyResolveRemoteUrl = Effect.fnUntraced(function* (flags: Legacy * Shared by `generate` (smart mode) and `sync` (no-declarative-files bootstrap) so * both offer the same local / linked / custom choice and local-reset prompt. */ -export const legacyResolveSmartTargetUrl = Effect.fnUntraced(function* ( +export const legacyResolveSmartTargetEndpoint = Effect.fnUntraced(function* ( flags: LegacySmartTargetFlags, local: LegacyLocalConn, hasMigrations: boolean, @@ -93,7 +112,7 @@ export const legacyResolveSmartTargetUrl = Effect.fnUntraced(function* ( // (db_schema_declarative.go:291), starting a stopped stack. yield* beforeLocalTarget; yield* (yield* LegacyDeclarativeSeam).ensureLocalDatabaseStarted(); - return legacyLocalUrl(local); + return legacyLocalEndpoint(local, yield* LegacyDnsResolverFlag); } const output = yield* Output; @@ -125,7 +144,7 @@ export const legacyResolveSmartTargetUrl = Effect.fnUntraced(function* ( if (choice === "linked") { // Same path as an explicit `--linked` (Go calls `NewDbConfigWithPassword`): // login-role mint + pooler fallback, then `ToPostgresURL`. - return yield* legacyResolveRemoteUrl({ ...flags, linked: Option.some(true) }); + return yield* legacyResolveRemoteEndpoint({ ...flags, linked: Option.some(true) }); } if (choice === "custom") { @@ -151,7 +170,12 @@ export const legacyResolveSmartTargetUrl = Effect.fnUntraced(function* ( }), ); } - return legacyToPostgresURL(conn); + return { + kind: "database", + ref: legacyToPostgresURL(conn), + connection: conn, + connectOptions: { isLocal: false, dnsResolver: yield* LegacyDnsResolverFlag }, + } satisfies LegacyPgDeltaDatabaseEndpoint; } // "Local database" choice: Go runs ensureLocalDatabaseStarted before the reset @@ -194,5 +218,5 @@ export const legacyResolveSmartTargetUrl = Effect.fnUntraced(function* ( ); } } - return legacyLocalUrl(local); + return legacyLocalEndpoint(local, yield* LegacyDnsResolverFlag); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index bd009a4af3..139257200a 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -1,18 +1,39 @@ # `supabase db schema declarative generate` -Generates declarative schema files from a database by diffing a platform-baseline -pg-delta catalog (source) against the target database's catalog (target). +Generates declarative schema files from a database using pg-delta's managed +platform view. + +## Pg-delta implementation and compatibility + +- The default pg-delta engine runs in-process. Pg-delta and pg-topo are bundled + into the CLI binary at build time, so the installed CLI fixes their version and + performs no runtime package download or automatic legacy fallback. +- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy catalog/edge-runtime + implementation. Only that opt-out uses `supabase/.temp/pgdelta-version`, + `PGDELTA_NPM_REGISTRY`, edge-runtime, or legacy catalogs directly below + `supabase/.temp/pgdelta/`. +- `--no-cache` bypasses legacy catalog reuse/warming. The default engine already + extracts live state and has no reusable catalog cache, so the flag does not + change its extraction behavior. +- With `PGDELTA_DEBUG`, default-engine export diagnostics are written below + `supabase/.temp/pgdelta/v2/debug//`; they are never reused as catalogs. +- The default engine refuses an export when extraction reports an error or a + strict coverage gap (`unmodeled_kind` or `unresolved_security_label`). The + refusal names the diagnostic, and debug artifacts are saved first when capture + is enabled. +- Generated SQL bytes and grouping may differ between engines. Reloading the + export to the same managed state is the compatibility contract. ## Files Read | Path | Format | When | | ----------------------------------------------- | ---------- | -------------------------------------------------- | | `/supabase/config.toml` | TOML | always — pg-delta gate, ports, format options | -| `/supabase/.temp/pgdelta-version` | plain text | always — pins the `@supabase/pg-delta` npm version | -| `/supabase/.temp/edge-runtime-version` | plain text | always — pins the edge-runtime image tag | +| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only — edge-runtime image tag | | `/supabase/.temp/postgres-version` | plain text | shadow-DB image resolution (Go seam) | | `/supabase/migrations/*.sql` | SQL | smart mode — detect whether migrations exist | -| `/supabase/.temp/pgdelta/*.json` | JSON | catalog cache (read/written by the Go seam) | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: catalog cache | | `~/.supabase/access-token` | plain text | `--linked` (token resolution) | ## Files Written @@ -20,15 +41,17 @@ pg-delta catalog (source) against the target database's catalog (target). | Path | Format | When | | --------------------------------------------------------------------------------------------------------------------------- | ------ | -------------------------------------------- | | `/supabase/database/**/*.sql` (declarative dir; configurable via `[experimental.pgdelta] declarative_schema_path`) | SQL | always — the entire dir is wiped + rewritten | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | catalog cache (written by the Go seam) | +| `/supabase/database/.pgdelta-export.json` | JSON | default-engine export policy/manifest | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers -| What | When | -| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | -| `supabase-go db schema declarative __catalog --mode baseline --experimental` (hidden seam) — provisions a shadow Postgres + `start.SetupDatabase`, exports the baseline catalog | always | -| Edge-runtime container (`supabase/edge-runtime`) running the pg-delta declarative-export Deno script (host network, deno-cache volume `supabase_edge_runtime_`) | always | -| `supabase-go db reset --local` | smart-mode Local choice when reset is confirmed (or `--reset`) | +| What | When | +| --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| `supabase-go db schema declarative __catalog --mode baseline --experimental` — provisions and exports the legacy baseline catalog | legacy opt-out only | +| Edge-runtime container running the pg-delta declarative-export Deno script | legacy opt-out only | +| `supabase-go db reset --local` | smart-mode Local choice when reset is confirmed (or `--reset`) | ## Environment Variables @@ -36,8 +59,9 @@ pg-delta catalog (source) against the target database's catalog (target). | ---------------------------- | -------------------------------------------------- | --------- | | `SUPABASE_ACCESS_TOKEN` | auth token for `--linked` | no | | `DB_PASSWORD` | password for `--linked` / `--db-url` | no | -| `PGDELTA_NPM_REGISTRY` | private `@supabase` npm registry for pg-delta | no | -| `PGDELTA_DEBUG` | verbose pg-delta diagnostics | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for the legacy edge-runtime engine | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: private npm registry | no | +| `PGDELTA_DEBUG` | structured default-engine debug artifacts | no | | `SUPABASE_GO_BINARY` | override the `supabase-go` seam binary | no | | `SUPABASE_SERVICES_HOSTNAME` | local DB host for `--local` (Go `GetHostname`) | no | | `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | @@ -50,7 +74,7 @@ pg-delta catalog (source) against the target database's catalog (target). | `1` | pg-delta not enabled (no `--experimental` / `[experimental.pgdelta]`) | | `1` | conflicting `--db-url`/`--linked`/`--local` (mutually exclusive) | | `1` | non-interactive mode with no explicit target | -| `1` | shadow-database / edge-runtime / export failure | +| `1` | shadow-database / selected pg-delta engine / export failure | The pg-delta gate and the mutex check are both raised before any side effects run, but the gate wins when both conditions apply simultaneously: Go's @@ -75,10 +99,10 @@ always go to stderr, in every `--output-format`. On success: - Requires `--experimental` or `[experimental.pgdelta] enabled = true`. - `--db-url` / `--linked` / `--local` are mutually exclusive; absent all three, smart mode prompts (existing-files overwrite → Local/Custom choice + reset offer). -- Remote Supabase targets (`--linked` / `--db-url`) get the embedded pg-delta CA - bundle written under `supabase/.temp/pgdelta/` and the URL rewritten to - `sslmode=verify-ca`; local / non-Supabase targets connect without it. -- **Architecture:** the shadow-database platform baseline is provisioned by the - bundled `supabase-go` via the hidden `db schema declarative __catalog` command - (it runs `start.SetupDatabase`'s auth/storage/realtime service migrations). The - rest — orchestration, pg-delta diff/export, file writes, prompts — is native. +- The default engine preserves the shared direct/pooler, DNS, TLS, and client + certificate connection behavior. The legacy opt-out retains its embedded CA + file and `sslmode=verify-ca` URL rewrite. +- **Architecture:** the default engine extracts the target directly using the + bundled Supabase management profile, then renders and writes the export + in-process. Under the opt-out, Go provisions/exports a legacy baseline catalog + and edge-runtime runs the Deno script. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts index def35f12a7..dae9e1e656 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts @@ -1,6 +1,7 @@ import { Effect, FileSystem, Option, Path } from "effect"; import { + LegacyDnsResolverFlag, legacyResolveExperimentalWithProjectEnv, legacyResolveYesWithProjectEnv, } from "../../../../../../shared/legacy/global-flags.ts"; @@ -18,6 +19,11 @@ import { import { LegacyLinkedProjectCache } from "../../../../../telemetry/legacy-linked-project-cache.service.ts"; import { LegacyTelemetryState } from "../../../../../telemetry/legacy-telemetry-state.service.ts"; import { legacyListLocalMigrations } from "../../../shared/legacy-pgdelta.cache.ts"; +import { legacyIsPgDeltaDebugEnabled } from "../../../shared/legacy-pgdelta.ts"; +import { + LegacyPgDeltaEngine, + type LegacyPgDeltaDatabaseEndpoint, +} from "../../../shared/legacy-pgdelta-engine.service.ts"; import { LegacyDeclarativeMutuallyExclusiveFlagsError, LegacyDeclarativeNonInteractiveError, @@ -32,9 +38,9 @@ import { legacyWriteDeclarativeSchemas } from "../../../shared/legacy-pgdelta.wr import type { LegacyDbSchemaDeclarativeGenerateFlags } from "./generate.command.ts"; import { type LegacyLocalConn, - legacyLocalUrl, - legacyResolveRemoteUrl, - legacyResolveSmartTargetUrl, + legacyLocalEndpoint, + legacyResolveRemoteEndpoint, + legacyResolveSmartTargetEndpoint, } from "../declarative.smart-target.ts"; export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.declarative.generate")( @@ -46,6 +52,8 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec const cliConfig = yield* LegacyCliConfig; const telemetryState = yield* LegacyTelemetryState; const linkedProjectCache = yield* LegacyLinkedProjectCache; + const dnsResolver = yield* LegacyDnsResolverFlag; + const engine = yield* LegacyPgDeltaEngine; // Go's `dbDeclarativeCmd.PersistentPreRunE` calls `flags.LoadConfig` — which runs // `loadNestedEnv` and `os.Setenv`s each project-.env key — BEFORE reading // `viper.GetBool("EXPERIMENTAL")` for the gate below (`apps/cli-go/cmd/ @@ -138,13 +146,15 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec declarativeDir, schema: flags.schema, noCache: flags.noCache, + debug: legacyIsPgDeltaDebugEnabled(), + dnsResolver, ...(linkedProjectRef !== undefined ? { linkedProjectRef } : {}), }; const hasExplicitTarget = Option.isSome(flags.local) || Option.isSome(flags.linked) || Option.isSome(flags.dbUrl); - let targetUrl: string; + let target: LegacyPgDeltaDatabaseEndpoint; let overwrite: boolean; if (hasExplicitTarget) { const seam = yield* LegacyDeclarativeSeam; @@ -158,9 +168,9 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec if (Option.getOrElse(flags.local, () => false)) { yield* seam.ensureLocalDatabaseStarted(); } - targetUrl = legacyLocalUrl(local); + target = legacyLocalEndpoint(local, dnsResolver); } else { - targetUrl = yield* legacyResolveRemoteUrl(flags); + target = yield* legacyResolveRemoteEndpoint(flags); } overwrite = flags.overwrite; } else { @@ -216,7 +226,7 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec linkedProjectRef = linkedRef.value; } } - targetUrl = yield* legacyResolveSmartTargetUrl( + target = yield* legacyResolveSmartTargetEndpoint( flags, local, hasMigrations, @@ -229,7 +239,7 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec overwrite = true; } - const result = yield* legacyGenerateDeclarativeOutput(run, targetUrl); + const result = yield* legacyGenerateDeclarativeOutput(run, target); if (!overwrite && (yield* confirmOverwriteHasFiles(fs, declarativeDir))) { // Go's confirmOverwrite goes through Console.PromptYesNo (`internal/db/ @@ -265,7 +275,7 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec // merged config and targets the same dir the handler wrote to (also computed from // the merged `toml`). Go warms against the in-process merged config identically // (`declarative.go:138-154`), so this always runs when `!--no-cache`. - if (!flags.noCache) { + if (!flags.noCache && engine.implementation === "legacy") { yield* (yield* LegacyDeclarativeSeam).exportCatalog({ mode: "declarative", noCache: flags.noCache, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index b8c4e2733d..44e74e8202 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -26,6 +26,8 @@ import { LegacyEdgeRuntimeScript, } from "../../../../../shared/legacy-edge-runtime-script.service.ts"; import { LegacyPgDeltaSslProbe } from "../../../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { legacyPgDeltaLegacyEngineLayer } from "../../../shared/legacy-pgdelta-engine.legacy.layer.ts"; +import { LegacyPgDeltaEngine } from "../../../shared/legacy-pgdelta-engine.service.ts"; import { LegacyDeclarativeShadowDbError } from "../../../shared/legacy-pgdelta.errors.ts"; import { type LegacyCatalogMode, @@ -61,6 +63,7 @@ interface SetupOpts { projectId?: Option.Option; exportFailsForMode?: LegacyCatalogMode; staleLocalImage?: boolean; + engineImplementation?: "legacy" | "next"; } function setup(workdir: string, opts: SetupOpts = {}) { @@ -138,12 +141,39 @@ function setup(workdir: string, opts: SetupOpts = {}) { exec: (args) => Effect.sync(() => void proxyCalls.push(args)), execCapture: () => Effect.succeed(""), }); + const sslProbe = Layer.succeed(LegacyPgDeltaSslProbe, { + requireSsl: () => Effect.succeed(false), + requireSslForHost: () => Effect.succeed(false), + }); + const engine = + opts.engineImplementation === "next" + ? Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation: "next", + diffExplicit: () => Effect.die("diffExplicit not used in generate tests"), + diffDatabase: () => Effect.die("diffDatabase not used in generate tests"), + planDeclarativeSchema: () => + Effect.die("planDeclarativeSchema not used in generate tests"), + exportDeclarativeSchema: () => + Effect.succeed({ + files: [ + { name: "schemas/public/tables/players.sql", sql: "create table players ();" }, + ], + manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, + }), + }), + ) + : legacyPgDeltaLegacyEngineLayer.pipe( + Layer.provide(Layer.mergeAll(seam, edge, sslProbe, BunServices.layer)), + ); const layer = Layer.mergeAll( out.layer, telemetry.layer, cache.layer, seam, edge, + engine, resolver, proxy, mockLegacyCliConfig({ workdir, projectId: opts.projectId ?? Option.some("test") }), @@ -155,10 +185,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { Layer.succeed(LegacyNetworkIdFlag, opts.networkId ?? Option.none()), Layer.succeed(LegacyDnsResolverFlag, "native"), // The remote ref is a non-Supabase host that refuses TLS → no SSL env. - Layer.succeed(LegacyPgDeltaSslProbe, { - requireSsl: () => Effect.succeed(false), - requireSslForHost: () => Effect.succeed(false), - }), + sslProbe, BunServices.layer, ); return { @@ -940,4 +967,21 @@ describe("legacy db schema declarative generate integration", () => { expect(s.edgeCalls[0]!.env["TARGET"]).toContain("@db.example.com:5432/app?connect_timeout="); }).pipe(Effect.provide(s.layer)); }); + + it.effect("next engine writes its manifest and skips legacy catalog warming", () => { + const s = setup(tmp.current, { experimental: true, engineImplementation: "next" }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeGenerate(flags({ local: Option.some(true) })); + const manifest = JSON.parse( + readFileSync(join(tmp.current, "supabase", "database", ".pgdelta-export.json"), "utf8"), + ); + expect(manifest).toMatchObject({ + formatVersion: 1, + redactSecrets: true, + scope: "database", + files: ["schemas/public/tables/players.sql"], + }); + expect(s.seamCalls).toEqual([]); + }).pipe(Effect.provide(s.layer)); + }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts index 6f2429fe57..5eaade6ef5 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts @@ -13,15 +13,19 @@ import { legacyLinkedDbResolverRuntimeLayer } from "../../../../../shared/legacy import { legacyPgDeltaSslProbeLayer } from "../../../../../shared/legacy-pgdelta-ssl-probe.layer.ts"; import { legacyTelemetryStateLayer } from "../../../../../telemetry/legacy-telemetry-state.layer.ts"; import { legacyDeclarativeSeamLayer } from "../../../shared/legacy-pgdelta.seam.layer.ts"; +import { legacyPgDeltaEngineLayer } from "../../../shared/legacy-pgdelta-engine.layer.ts"; +import { legacyPgDeltaNextAdapterLayer } from "../../../shared/legacy-pgdelta-next-adapter.layer.ts"; +import { legacyPgDeltaNextShadowLayer } from "../../../shared/legacy-pgdelta-next-shadow.layer.ts"; /** * Runtime layer for `supabase db schema declarative generate`. * * `Output` / `LegacyGoProxy` / global flags come from the legacy root; the Bun * platform (FileSystem / Path / ChildProcessSpawner / ProcessControl / Tty) from - * `runCli`. This layer adds the declarative-specific services: the edge-runtime - * pg-delta runner and the Go shadow-database seam, plus the db-config resolver - * for `--linked` / `--db-url`. Per the "provide doesn't share to siblings" rule, + * `runCli`. This layer adds both pg-delta implementations and the Go + * shadow-database seam, plus the db-config resolver for `--linked` / `--db-url`. + * The bundled implementation runs in-process by default; edge-runtime is retained + * only for the explicit legacy opt-out. Per the "provide doesn't share to siblings" rule, * `LegacyCliConfig` is provided to every layer that needs it. */ const cliConfig = legacyCliConfigLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); @@ -42,6 +46,15 @@ const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( ); const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); +const nextShadow = legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seam)); +const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(legacyPgDeltaNextAdapterLayer), + Layer.provide(nextShadow), + Layer.provide(edgeRuntime), + Layer.provide(legacyPgDeltaSslProbeLayer), + Layer.provide(seam), + Layer.provide(legacyDebugLoggerLayer), +); export const legacyDbSchemaDeclarativeGenerateRuntimeLayer = Layer.mergeAll( dbConfig, @@ -49,6 +62,7 @@ export const legacyDbSchemaDeclarativeGenerateRuntimeLayer = Layer.mergeAll( edgeRuntime, legacyPgDeltaSslProbeLayer, seam, + pgDeltaEngine, cliConfig, legacyIdentityStitchLayer, legacyTelemetryStateLayer, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index ba4ee2562b..8c98dddee1 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -3,39 +3,61 @@ Diffs local migrations state against declarative schema files and writes the delta as a new timestamped migration. +## Pg-delta implementation and compatibility + +- The default pg-delta and bundled pg-topo run in-process at the versions fixed + when the CLI is built. There is no runtime download or automatic fallback. +- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy catalog/edge-runtime + implementation. `supabase/.temp/pgdelta-version`, `PGDELTA_NPM_REGISTRY`, and + catalogs directly below `supabase/.temp/pgdelta/` are legacy-only. +- `--no-cache` bypasses legacy catalog reuse/warming. The default engine always + extracts current state and maintains no reusable catalog cache. +- With `PGDELTA_DEBUG`, default-engine snapshots, plan, and diagnostics are + written below `supabase/.temp/pgdelta/v2/debug//` and are not reusable. +- The default engine refuses to emit a migration when extraction or declarative + loading reports an error or a strict coverage gap (`unmodeled_kind` or + `unresolved_security_label`). The refusal names the diagnostic, and debug + artifacts are saved first when capture is enabled. +- Default-engine migrations may differ byte-for-byte and may be split into + ordered files to preserve transaction boundaries. Successful execution and an + empty subsequent sync are the compatibility contract. + ## Files Read -| Path | Format | When | -| -------------------------------------------------------- | ---------- | -------------------------------------------------- | -| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | -| `/supabase/.temp/pgdelta-version` | plain text | always — pins the `@supabase/pg-delta` npm version | -| `/supabase/.temp/edge-runtime-version` | plain text | always — pins the edge-runtime image tag | -| `/supabase/database/**/*.sql` (declarative dir) | SQL | always — must exist (else error) | -| `/supabase/migrations/*.sql` | SQL | shadow-DB migrations catalog (Go seam) | -| `/supabase/.temp/pgdelta/*.json` | JSON | catalog cache (read/written by the Go seam) | +| Path | Format | When | +| -------------------------------------------------------- | ---------- | ------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | +| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only — edge-runtime image tag | +| `/supabase/database/**/*.sql` (declarative dir) | SQL | always — must exist (else error) | +| `/supabase/migrations/*.sql` | SQL | default: applied to live shadow; legacy: catalog source | +| `/supabase/database/.pgdelta-export.json` | JSON | default-engine export policy, when present | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: catalog cache | ## Files Written -| Path | Format | When | -| ------------------------------------------------------ | ------ | ----------------------------- | -| `/supabase/migrations/_.sql` | SQL | when schema changes are found | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | catalog cache (Go seam) | +| Path | Format | When | +| ------------------------------------------------------------------ | ------ | ------------------------------------------------- | +| `/supabase/migrations/_[_].sql` | SQL | changes; default engine may emit ordered segments | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers | What | When | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| `supabase-go db schema declarative __catalog --mode migrations --experimental` (seam) — shadow Postgres + `SetupDatabase` + apply migrations → catalog | always | -| `supabase-go db schema declarative __catalog --mode declarative --experimental` (seam) — shadow Postgres + `SetupDatabase` + apply declarative → catalog | always | -| Edge-runtime container running the pg-delta diff Deno script | always | +| `supabase-go db __shadow` / declarative shadow seam — platform baseline plus migrations and clean declarative target | default engine | +| `supabase-go db schema declarative __catalog` migrations/declarative catalog seams | legacy opt-out only | +| Edge-runtime container running the pg-delta diff Deno script | legacy opt-out only | | `supabase-go db reset --local [--network-id ]` (seam) — only on the failed-apply recovery path; `db reset` is still Go-proxied (`wrapped`), so the reset itself shells out to the bundled binary | TTY only, apply failed, and the user confirms "reset and reapply" | ## Environment Variables | Variable | Purpose | Required? | | ---------------------------- | ----------------------------------------------------------- | --------- | -| `PGDELTA_NPM_REGISTRY` | private `@supabase` npm registry for pg-delta | no | -| `PGDELTA_DEBUG` | verbose pg-delta diagnostics | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for the legacy edge-runtime engine | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: private npm registry | no | +| `PGDELTA_DEBUG` | structured default-engine debug artifacts | no | | `SUPABASE_GO_BINARY` | override the `supabase-go` seam binary | no | | `SUPABASE_SERVICES_HOSTNAME` | local DB host for the bootstrap generate (Go `GetHostname`) | no | | `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | @@ -48,7 +70,7 @@ as a new timestamped migration. | `1` | pg-delta not enabled | | `1` | conflicting `--apply`/`--no-apply` (mutually exclusive) | | `1` | no declarative schema files found | -| `1` | shadow-database / edge-runtime / diff failure | +| `1` | shadow-database / selected pg-delta engine / diff failure | | `1` | apply failure (when applied) — propagated from the native migration apply (`applyMigrationToLocal`) | The pg-delta gate and the mutex check are both raised before any side effects run, @@ -62,8 +84,8 @@ surfaces before an `--apply`/`--no-apply` conflict is ever checked. Text mode only. The generated SQL, the created-migration path, drop-statement warnings, and apply status are written to stderr. The no-files bootstrap also prints `Declarative schema written to ` (the relative declarative dir, Go's -`GetDeclarativeDir()`) to stderr after generating, writing, and warming the -catalog cache — on both the interactive-accept and `--yes` paths. +`GetDeclarativeDir()`) to stderr after generation and writing. Under the legacy +opt-out it prints after catalog warming — on both interactive and `--yes` paths. `--no-apply` writes the migration only (never prompts/applies); `--apply` applies without prompting; both override the global `--yes`. `--no-apply` and `--apply` are mutually exclusive. @@ -79,6 +101,6 @@ are mutually exclusive. `supabase/.temp/pgdelta/debug/` and, in a TTY, a reset-and-reapply is offered (the reset itself runs the bundled `supabase-go db reset --local`, since `db reset` is still `wrapped`). -- **Architecture:** the shadow-database platform baseline (migrations / declarative - catalogs) is provisioned by the bundled `supabase-go` via the hidden - `db schema declarative __catalog` seam; the diff is native pg-delta. +- **Architecture:** the bundled `supabase-go` provisions the two shadow databases; + the default engine applies declarative SQL and plans/renders the migration + in-process. The opt-out preserves the hidden legacy catalog seams and Deno script. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index e7f946362b..8f1d21a7d6 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -27,7 +27,10 @@ import { legacyListLocalMigrations, legacyPgDeltaTempPath, } from "../../../shared/legacy-pgdelta.cache.ts"; -import { legacyResolveSmartTargetUrl } from "../declarative.smart-target.ts"; +import { LegacyPgDeltaEngine } from "../../../shared/legacy-pgdelta-engine.service.ts"; +import { legacyIsPgDeltaDebugEnabled } from "../../../shared/legacy-pgdelta.ts"; +import { legacyWritePgDeltaMigrations } from "../../../shared/legacy-pgdelta-migrations.write.ts"; +import { legacyResolveSmartTargetEndpoint } from "../declarative.smart-target.ts"; import { type LegacyDebugBundle, legacyCollectMigrationsList, @@ -92,6 +95,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara const networkId = yield* LegacyNetworkIdFlag; const dnsResolver = yield* LegacyDnsResolverFlag; const seam = yield* LegacyDeclarativeSeam; + const engine = yield* LegacyPgDeltaEngine; const linkedProjectCache = yield* LegacyLinkedProjectCache; // Go's sync bootstrap delegates to `runDeclarativeGenerate`, whose @@ -152,6 +156,8 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara declarativeDir, schema: flags.schema, noCache: flags.noCache, + debug: legacyIsPgDeltaDebugEnabled(), + dnsResolver, }; const ensureLocalPostgresImageCurrent = seam.ensureLocalPostgresImageCurrent(); const declarativeFilesExist = yield* declarativeDirHasFiles(fs, declarativeDir); @@ -225,7 +231,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara } // sync has no target flags (Go passes its target-less `cmd` into generate), // so reset stays interactive (the prompt fires under the local choice). - const targetUrl = yield* legacyResolveSmartTargetUrl( + const target = yield* legacyResolveSmartTargetEndpoint( { dbUrl: Option.none(), linked: Option.none(), password: Option.none(), reset: false }, { port: toml.port, password: toml.password }, hasMigrations, @@ -235,7 +241,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara linkedRef, ensureLocalPostgresImageCurrent, ); - const generated = yield* legacyGenerateDeclarativeOutput(run, targetUrl); + const generated = yield* legacyGenerateDeclarativeOutput(run, target); yield* legacyWriteDeclarativeSchemas(fs, path, declarativeDir, generated); if (!(yield* declarativeDirHasFiles(fs, declarativeDir))) { return yield* Effect.fail( @@ -251,7 +257,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara // catalog / emitting a diff debug bundle, and warming the catalog the following // diff reuses. (sync is target-less and writes to the single toml-resolved dir, // so the generate handler's remote-override dir guard isn't needed here.) - if (!run.noCache) { + if (!run.noCache && engine.implementation === "legacy") { yield* seam.exportCatalog({ mode: "declarative", noCache: run.noCache }); } // Go's delegated `declarative.Generate` prints the written-to line to stderr @@ -308,11 +314,28 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara } // Step 5: write the timestamped migration file. - const timestamp = formatTimestamp(yield* Clock.currentTimeMillis); - const migrationPath = path.join(migrationsDir, `${timestamp}_${migrationName}.sql`); - yield* legacyMakeDir(fs, migrationsDir); - yield* fs.writeFileString(migrationPath, result.diffSQL); - yield* output.raw(`Created new migration at ${legacyBold(migrationPath)}\n`, "stderr"); + const nowMillis = yield* Clock.currentTimeMillis; + let migrationPaths: ReadonlyArray; + if (engine.implementation === "next" && result.files.length > 1) { + const written = yield* legacyWritePgDeltaMigrations(fs, path, { + workdir: cliConfig.workdir, + baseMillis: nowMillis, + name: migrationName, + files: result.files, + }).pipe( + Effect.mapError((error) => new LegacyDeclarativeApplyError({ message: error.message })), + ); + migrationPaths = written.map((migration) => migration.path); + } else { + const timestamp = formatTimestamp(nowMillis); + const migrationPath = path.join(migrationsDir, `${timestamp}_${migrationName}.sql`); + yield* legacyMakeDir(fs, migrationsDir); + yield* fs.writeFileString(migrationPath, result.diffSQL); + migrationPaths = [migrationPath]; + } + for (const migrationPath of migrationPaths) { + yield* output.raw(`Created new migration at ${legacyBold(migrationPath)}\n`, "stderr"); + } // Step 6: drop warnings. if (result.dropWarnings.length > 0) { @@ -346,7 +369,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara yield* ensureLocalPostgresImageCurrent; const applyExit = yield* applyMigrationToLocal( { port: toml.port, password: toml.password, dnsResolver }, - migrationPath, + migrationPaths, ).pipe(Effect.exit); if (Exit.isSuccess(applyExit)) { @@ -460,10 +483,10 @@ const declarativeDirHasFiles = Effect.fnUntraced(function* ( return entries.length > 0; }); -/** Connects to the local database and applies the single migration file (Go's `applyMigrationToLocal`). */ +/** Connects once and applies the ordered migration files (Go's `applyMigrationToLocal`). */ const applyMigrationToLocal = ( local: { port: number; password: string; dnsResolver: "native" | "https" }, - migrationPath: string, + migrationPaths: ReadonlyArray, ) => Effect.gen(function* () { const dbConnection = yield* LegacyDbConnection; @@ -486,11 +509,13 @@ const applyMigrationToLocal = ( .pipe( Effect.mapError((error) => new LegacyDeclarativeApplyError({ message: error.message })), ); - yield* legacyApplyMigrationFile( - session, - fs, - path, - migrationPath, - (message) => new LegacyDeclarativeApplyError({ message }), - ); + for (const migrationPath of migrationPaths) { + yield* legacyApplyMigrationFile( + session, + fs, + path, + migrationPath, + (message) => new LegacyDeclarativeApplyError({ message }), + ); + } }).pipe(Effect.scoped); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index a5acb0655e..684745a8d3 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -26,6 +26,11 @@ import { LegacyEdgeRuntimeScript, } from "../../../../../shared/legacy-edge-runtime-script.service.ts"; import { LegacyPgDeltaSslProbe } from "../../../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { legacyPgDeltaLegacyEngineLayer } from "../../../shared/legacy-pgdelta-engine.legacy.layer.ts"; +import { + LegacyPgDeltaEngine, + type LegacyPgDeltaRenderedFile, +} from "../../../shared/legacy-pgdelta-engine.service.ts"; import { LegacyDeclarativeShadowDbError } from "../../../shared/legacy-pgdelta.errors.ts"; import { LegacyDeclarativeSeam } from "../../../shared/legacy-pgdelta.seam.service.ts"; import type { LegacyDbSchemaDeclarativeSyncFlags } from "./sync.command.ts"; @@ -59,6 +64,8 @@ interface SetupOpts { projectId?: Option.Option; staleLocalImage?: boolean; exportJson?: string; + engineImplementation?: "legacy" | "next"; + renderedFiles?: ReadonlyArray; } function setup(workdir: string, opts: SetupOpts = {}) { @@ -169,12 +176,46 @@ function setup(workdir: string, opts: SetupOpts = {}) { }), resolvePoolerFallback: () => Effect.succeed(Option.none()), }); + const sslProbe = Layer.succeed(LegacyPgDeltaSslProbe, { + requireSsl: () => Effect.succeed(false), + requireSslForHost: () => Effect.succeed(false), + }); + const nextFiles = opts.renderedFiles ?? []; + const engine = + opts.engineImplementation === "next" + ? Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation: "next", + diffExplicit: () => Effect.die("diffExplicit not used in sync tests"), + diffDatabase: () => Effect.die("diffDatabase not used in sync tests"), + exportDeclarativeSchema: () => + Effect.succeed({ + files: [ + { name: "schemas/public/tables/players.sql", sql: "create table players ();" }, + ], + manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, + }), + planDeclarativeSchema: () => + Effect.succeed({ + changes: nextFiles.length > 0, + sql: opts.diffSql ?? nextFiles.map((file) => file.sql).join("\n"), + files: nextFiles, + sourceRef: "migrations", + targetRef: "declarative", + }), + }), + ) + : legacyPgDeltaLegacyEngineLayer.pipe( + Layer.provide(Layer.mergeAll(seam, edge, sslProbe, BunServices.layer)), + ); const layer = Layer.mergeAll( out.layer, telemetry.layer, cache.layer, seam, edge, + engine, dbConn, resolver, mockLegacyCliConfig({ workdir, projectId: opts.projectId ?? Option.some("test") }), @@ -189,10 +230,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { ), Layer.succeed(LegacyDnsResolverFlag, "native"), // Sync diffs against the local DB, which refuses TLS → no SSL env injected. - Layer.succeed(LegacyPgDeltaSslProbe, { - requireSsl: () => Effect.succeed(false), - requireSslForHost: () => Effect.succeed(false), - }), + sslProbe, BunServices.layer, ); return { @@ -896,4 +934,36 @@ describe("legacy db schema declarative sync integration", () => { ]); }).pipe(Effect.provide(s.layer)); }); + + it.effect("next engine preserves ordered migration segments as separate files", () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + experimental: true, + engineImplementation: "next", + renderedFiles: [ + { + sequence: 1, + name: "transactional", + suffix: "_1", + sql: "ALTER TABLE a ADD COLUMN b int;", + transactional: true, + }, + { + sequence: 2, + name: "non_transactional", + suffix: "_2", + sql: "ALTER TYPE mood ADD VALUE 'fine';", + transactional: false, + }, + ], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + const migrations = readdirSync(join(tmp.current, "supabase", "migrations")).sort(); + expect(migrations).toHaveLength(2); + expect(migrations[0]).toMatch(/^\d{14}_declarative_sync_1\.sql$/); + expect(migrations[1]).toMatch(/^\d{14}_declarative_sync_2\.sql$/); + expect(s.exportCatalogCalls).toEqual([]); + }).pipe(Effect.provide(s.layer)); + }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts index 0eb4fc8592..49929b862b 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts @@ -13,6 +13,9 @@ import { legacyLinkedDbResolverRuntimeLayer } from "../../../../../shared/legacy import { legacyPgDeltaSslProbeLayer } from "../../../../../shared/legacy-pgdelta-ssl-probe.layer.ts"; import { legacyTelemetryStateLayer } from "../../../../../telemetry/legacy-telemetry-state.layer.ts"; import { legacyDeclarativeSeamLayer } from "../../../shared/legacy-pgdelta.seam.layer.ts"; +import { legacyPgDeltaEngineLayer } from "../../../shared/legacy-pgdelta-engine.layer.ts"; +import { legacyPgDeltaNextAdapterLayer } from "../../../shared/legacy-pgdelta-next-adapter.layer.ts"; +import { legacyPgDeltaNextShadowLayer } from "../../../shared/legacy-pgdelta-next-shadow.layer.ts"; /** * Runtime layer for `supabase db schema declarative sync`. Sync diffs against the @@ -40,12 +43,22 @@ const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( ); const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); +const nextShadow = legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seam)); +const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(legacyPgDeltaNextAdapterLayer), + Layer.provide(nextShadow), + Layer.provide(edgeRuntime), + Layer.provide(legacyPgDeltaSslProbeLayer), + Layer.provide(seam), + Layer.provide(legacyDebugLoggerLayer), +); export const legacyDbSchemaDeclarativeSyncRuntimeLayer = Layer.mergeAll( dbConfig, edgeRuntime, legacyPgDeltaSslProbeLayer, seam, + pgDeltaEngine, legacyDbConnectionLayer, cliConfig, legacyIdentityStitchLayer, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts new file mode 100644 index 0000000000..df5494aa71 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts @@ -0,0 +1,67 @@ +import { Effect, FileSystem, Layer, Path } from "effect"; + +import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; +import { legacyPgDeltaLegacyEngineLayer } from "./legacy-pgdelta-engine.legacy.layer.ts"; +import { legacyPgDeltaNextEngineLayer } from "./legacy-pgdelta-engine.next.layer.ts"; +import { LegacyPgDeltaEngine } from "./legacy-pgdelta-engine.service.ts"; +import { LegacyPgDeltaNextAdapter } from "./legacy-pgdelta-next-adapter.service.ts"; +import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; +import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; +import { LegacyEdgeRuntimeScript } from "../../../shared/legacy-edge-runtime-script.service.ts"; +import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { legacyResolvePgDeltaImplementation } from "./legacy-pgdelta-next-flag.ts"; + +const FLAG = "SUPABASE_USE_PG_DELTA_NEXT"; + +const resolveAndLog = Effect.fnUntraced(function* (raw: string | undefined) { + const debug = yield* LegacyDebugLogger; + const implementation = legacyResolvePgDeltaImplementation(raw); + yield* debug.debug(`Using pg-delta ${implementation} implementation.`); + return implementation; +}); + +/** + * Selects exactly one implementation layer. There is intentionally no catch or + * retry path between implementations: a selected next-engine failure must + * propagate without invoking the legacy adapter. + */ +export function legacyPgDeltaEngineSelectorLayer( + raw: string | undefined, + layers: { + readonly next: Layer.Layer; + readonly legacy: Layer.Layer; + }, +) { + return Layer.unwrap( + Effect.gen(function* () { + const implementation = yield* resolveAndLog(raw); + return implementation === "next" ? layers.next : layers.legacy; + }), + ); +} + +/** Reads the rollout flag once when the command-scoped layer is constructed. */ +export const legacyPgDeltaEngineLayer = Layer.unwrap( + Effect.gen(function* () { + const raw = process.env[FLAG]; + const implementation = yield* resolveAndLog(raw); + return selectProductionLayer(implementation); + }), +); + +function selectProductionLayer( + implementation: "next" | "legacy", +): Layer.Layer< + LegacyPgDeltaEngine, + never, + | LegacyPgDeltaNextAdapter + | LegacyPgDeltaNextShadow + | LegacyDebugLogger + | LegacyDeclarativeSeam + | LegacyEdgeRuntimeScript + | LegacyPgDeltaSslProbe + | FileSystem.FileSystem + | Path.Path +> { + return implementation === "next" ? legacyPgDeltaNextEngineLayer : legacyPgDeltaLegacyEngineLayer; +} diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts new file mode 100644 index 0000000000..56a8fbcea9 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts @@ -0,0 +1,196 @@ +import { Effect, Exit, Layer } from "effect"; +import * as BunServices from "@effect/platform-bun/BunServices"; +import { it } from "@effect/vitest"; +import { afterEach, describe, expect } from "vitest"; + +import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; +import { LegacyEdgeRuntimeScript } from "../../../shared/legacy-edge-runtime-script.service.ts"; +import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; +import { LegacyPgDeltaNextAdapter } from "./legacy-pgdelta-next-adapter.service.ts"; +import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; +import { + legacyPgDeltaEngineLayer, + legacyPgDeltaEngineSelectorLayer, +} from "./legacy-pgdelta-engine.layer.ts"; +import { LegacyPgDeltaEngine } from "./legacy-pgdelta-engine.service.ts"; + +const FLAG = "SUPABASE_USE_PG_DELTA_NEXT"; + +function debugLayer(messages: Array) { + return Layer.succeed(LegacyDebugLogger, { + debug: (message) => Effect.sync(() => messages.push(message)), + http: () => Effect.void, + }); +} + +function metadataLayer(implementation: "next" | "legacy") { + return Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation, + diffExplicit: () => Effect.die(`${implementation} explicit diff not needed`), + diffDatabase: () => Effect.die(`${implementation} database diff not needed`), + exportDeclarativeSchema: () => Effect.die(`${implementation} export not needed`), + planDeclarativeSchema: () => Effect.die(`${implementation} plan not needed`), + }), + ); +} + +const unusedLegacyRuntime = Layer.mergeAll( + BunServices.layer, + Layer.succeed(LegacyEdgeRuntimeScript, { + run: () => Effect.die("edge runtime not needed"), + }), + Layer.succeed(LegacyPgDeltaSslProbe, { + requireSsl: () => Effect.die("SSL probe not needed"), + requireSslForHost: () => Effect.die("SSL probe not needed"), + }), + Layer.succeed(LegacyDeclarativeSeam, { + exportCatalog: () => Effect.die("catalog not needed"), + execInherit: () => Effect.die("exec not needed"), + ensureLocalDatabaseStarted: () => Effect.die("local start not needed"), + ensureLocalPostgresImageCurrent: () => Effect.die("image check not needed"), + provisionShadow: () => Effect.die("shadow not needed"), + removeShadowContainer: () => Effect.die("cleanup not needed"), + }), + Layer.succeed(LegacyPgDeltaNextAdapter, { + diff: () => Effect.die("adapter not needed"), + exportDeclarativeSchema: () => Effect.die("adapter not needed"), + planDeclarativeSchema: () => Effect.die("adapter not needed"), + captureSnapshot: () => Effect.die("adapter not needed"), + }), + Layer.succeed(LegacyPgDeltaNextShadow, { + provision: () => Effect.die("next shadow not needed"), + }), +); + +describe("legacyPgDeltaEngineSelectorLayer", () => { + it.effect("selects next by default and logs the decision once", () => { + const messages: Array = []; + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + expect(engine.implementation).toBe("next"); + expect(messages).toEqual(["Using pg-delta next implementation."]); + }).pipe( + Effect.provide( + legacyPgDeltaEngineSelectorLayer(undefined, { + next: metadataLayer("next"), + legacy: metadataLayer("legacy"), + }).pipe(Layer.provide(debugLayer(messages))), + ), + ); + }); + + it.effect("selects legacy only for an explicit false value", () => { + const messages: Array = []; + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + expect(engine.implementation).toBe("legacy"); + expect(messages).toEqual(["Using pg-delta legacy implementation."]); + }).pipe( + Effect.provide( + legacyPgDeltaEngineSelectorLayer("false", { + next: metadataLayer("next"), + legacy: metadataLayer("legacy"), + }).pipe(Layer.provide(debugLayer(messages))), + ), + ); + }); + + it.effect("does not invoke legacy after a selected next operation fails", () => { + const messages: Array = []; + let nextCalls = 0; + let legacyCalls = 0; + const next = Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation: "next", + diffExplicit: () => + Effect.sync(() => { + nextCalls += 1; + }).pipe(Effect.andThen(Effect.die("next diff failed"))), + diffDatabase: () => Effect.die("next database diff failed"), + exportDeclarativeSchema: () => Effect.die("next export failed"), + planDeclarativeSchema: () => Effect.die("next plan failed"), + }), + ); + const legacy = Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation: "legacy", + diffExplicit: () => + Effect.sync(() => { + legacyCalls += 1; + return { + changes: false, + sql: "", + files: [], + }; + }), + diffDatabase: () => Effect.die("legacy database diff should not run"), + exportDeclarativeSchema: () => Effect.die("legacy export should not run"), + planDeclarativeSchema: () => Effect.die("legacy plan should not run"), + }), + ); + + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + const exit = yield* engine + .diffExplicit({ + context: { projectId: "test", cwd: "/tmp/test", npmVersion: undefined, denoVersion: 2 }, + source: { + kind: "database", + ref: "postgresql://localhost/source", + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + desired: { + kind: "database", + ref: "postgresql://localhost/desired", + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + schema: [], + formatOptions: "", + debug: false, + }) + .pipe(Effect.exit); + expect(Exit.isFailure(exit)).toBe(true); + expect(nextCalls).toBe(1); + expect(legacyCalls).toBe(0); + }).pipe( + Effect.provide( + legacyPgDeltaEngineSelectorLayer("true", { next, legacy }).pipe( + Layer.provide(debugLayer(messages)), + ), + ), + ); + }); +}); + +describe("legacyPgDeltaEngineLayer", () => { + afterEach(() => { + delete process.env[FLAG]; + }); + + it.effect("reads the environment once for the command-scoped service", () => { + const messages: Array = []; + process.env[FLAG] = "false"; + + return Effect.gen(function* () { + const first = yield* LegacyPgDeltaEngine; + process.env[FLAG] = "true"; + const second = yield* LegacyPgDeltaEngine; + + expect(first).toBe(second); + expect(second.implementation).toBe("legacy"); + expect(messages).toEqual(["Using pg-delta legacy implementation."]); + }).pipe( + Effect.provide( + legacyPgDeltaEngineLayer.pipe( + Layer.provide(unusedLegacyRuntime), + Layer.provide(debugLayer(messages)), + ), + ), + ); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts new file mode 100644 index 0000000000..3505e73c36 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts @@ -0,0 +1,183 @@ +import { Effect, FileSystem, Layer, Path } from "effect"; + +import { LegacyEdgeRuntimeScript } from "../../../shared/legacy-edge-runtime-script.service.ts"; +import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { legacyFindDropStatements } from "../../../shared/legacy-sql-split.ts"; +import { + LegacyPgDeltaEngine, + LegacyPgDeltaEngineError, + type LegacyPgDeltaDiffResult, + type LegacyPgDeltaEndpoint, +} from "./legacy-pgdelta-engine.service.ts"; +import { + legacyDeclarativeExportPgDelta, + legacyDiffPgDelta, + legacyExportCatalogPgDelta, +} from "./legacy-pgdelta.ts"; +import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; + +const mapError = (cause: { readonly message: string }) => + new LegacyPgDeltaEngineError({ message: cause.message, cause }); + +function normalizeDiff( + result: { + readonly sql: string; + readonly stderr: string; + readonly files: ReadonlyArray<{ + readonly order: number; + readonly name: string; + readonly transactionMode: string; + readonly sql: string; + }>; + }, + debug: boolean, +): LegacyPgDeltaDiffResult { + return { + changes: result.sql.trim().length > 0, + sql: result.sql, + files: result.files.map((file) => ({ + sequence: file.order, + name: file.name, + sql: file.sql, + transactional: file.transactionMode !== "non-transactional", + })), + ...(debug ? { debug: { stderr: result.stderr } } : {}), + }; +} + +/** Behavior-preserving adapter for the alpha.33 edge-runtime implementation. */ +export const legacyPgDeltaLegacyEngineLayer = Layer.effect( + LegacyPgDeltaEngine, + Effect.gen(function* () { + const edgeRuntime = yield* LegacyEdgeRuntimeScript; + const sslProbe = yield* LegacyPgDeltaSslProbe; + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const seam = yield* LegacyDeclarativeSeam; + + const provideRuntime = ( + operation: Effect.Effect< + Success, + Error, + LegacyEdgeRuntimeScript | LegacyPgDeltaSslProbe | FileSystem.FileSystem | Path.Path + >, + ) => + operation.pipe( + Effect.provideService(LegacyEdgeRuntimeScript, edgeRuntime), + Effect.provideService(LegacyPgDeltaSslProbe, sslProbe), + Effect.provideService(FileSystem.FileSystem, fs), + Effect.provideService(Path.Path, path), + ); + + const endpointRef = (endpoint: LegacyPgDeltaEndpoint, noCache: boolean) => + endpoint.kind === "database" + ? Effect.succeed(endpoint.ref) + : seam.exportCatalog({ + mode: "migrations", + noCache, + ...(endpoint.projectRef !== undefined ? { projectRef: endpoint.projectRef } : {}), + }); + + return LegacyPgDeltaEngine.of({ + implementation: "legacy", + diffExplicit: (input) => + Effect.gen(function* () { + const sourceRef = yield* endpointRef(input.source, false); + const targetRef = yield* endpointRef(input.desired, false); + const result = yield* provideRuntime( + legacyDiffPgDelta(input.context, { + sourceRef, + targetRef, + schema: input.schema, + formatOptions: input.formatOptions, + }), + ); + return normalizeDiff(result, input.debug); + }).pipe(Effect.mapError(mapError)), + diffDatabase: (input) => + Effect.gen(function* () { + const shadow = yield* seam.provisionShadow({ + mode: "diff", + targetLocal: input.targetLocal, + usePgDelta: true, + schema: input.schema, + ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), + }); + const sourceSnapshot = input.debug + ? yield* provideRuntime( + legacyExportCatalogPgDelta(input.context, { + targetRef: shadow.sourceUrl, + role: "postgres", + }), + ).pipe(Effect.orElseSucceed(() => undefined)) + : undefined; + return yield* provideRuntime( + legacyDiffPgDelta(input.context, { + sourceRef: shadow.sourceUrl, + targetRef: shadow.targetUrlOverride ?? input.target.ref, + schema: input.schema, + formatOptions: input.formatOptions, + }), + ).pipe( + Effect.map((result) => { + const normalized = normalizeDiff(result, input.debug); + return input.debug + ? { + ...normalized, + debug: { + ...(sourceSnapshot !== undefined ? { sourceSnapshot } : {}), + stderr: result.stderr, + }, + } + : normalized; + }), + Effect.ensuring(seam.removeShadowContainer(shadow.container)), + ); + }).pipe(Effect.mapError(mapError)), + exportDeclarativeSchema: (input) => + Effect.gen(function* () { + const baselineRef = yield* seam.exportCatalog({ + mode: "baseline", + noCache: input.noCache, + ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), + }); + const result = yield* provideRuntime( + legacyDeclarativeExportPgDelta(input.context, { + sourceRef: baselineRef, + targetRef: input.target.ref, + schema: input.schema, + formatOptions: input.formatOptions, + }), + ); + return { + files: result.files.map((file) => ({ name: file.path, sql: file.sql })), + }; + }).pipe(Effect.mapError(mapError)), + planDeclarativeSchema: (input) => + Effect.gen(function* () { + const sourceRef = yield* seam.exportCatalog({ + mode: "migrations", + noCache: input.noCache, + }); + const targetRef = yield* seam.exportCatalog({ + mode: "declarative", + noCache: input.noCache, + }); + const result = yield* provideRuntime( + legacyDiffPgDelta(input.context, { + sourceRef, + targetRef, + schema: input.schema, + formatOptions: input.formatOptions, + }), + ); + return { + ...normalizeDiff(result, input.debug), + sourceRef, + targetRef, + dropWarnings: legacyFindDropStatements(result.sql), + }; + }).pipe(Effect.mapError(mapError)), + }); + }), +); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts new file mode 100644 index 0000000000..3250ab029d --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -0,0 +1,368 @@ +import { Clock, Effect, FileSystem, Layer, Path } from "effect"; + +import { parseLegacyConnectionString } from "../../../shared/legacy-db-config.parse.ts"; +import { LegacyDbConnectError } from "../../../shared/legacy-db-connection.errors.ts"; +import { legacyAcquirePgPool } from "../../../shared/legacy-db-connection.sql-pg.layer.ts"; +import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; +import { + LegacyPgDeltaEngine, + LegacyPgDeltaEngineError, + type LegacyPgDeltaDatabaseEndpoint, + type LegacyPgDeltaDiffResult, + type LegacyPgDeltaEndpoint, +} from "./legacy-pgdelta-engine.service.ts"; +import { + LegacyPgDeltaNextAdapter, + type LegacyPgDeltaNextOperation, +} from "./legacy-pgdelta-next-adapter.service.ts"; +import { + legacyFormatPgDeltaNextDebugId, + legacySavePgDeltaNextDebugArtifacts, + type LegacyPgDeltaNextDebugArtifacts, +} from "./legacy-pgdelta-next-artifacts.ts"; +import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; +import { + legacyPgDeltaNextBlockingDiagnostic, + legacyPgDeltaNextBlockingDiagnosticMessage, +} from "./legacy-pgdelta-next-diagnostics.ts"; + +function legacyPgDeltaNextConnectSuggestion(cause: unknown): string | undefined { + if (cause instanceof LegacyDbConnectError) return cause.suggestion; + if (typeof cause !== "object" || cause === null) return undefined; + const nested = Reflect.get(cause, "cause"); + return nested === cause ? undefined : legacyPgDeltaNextConnectSuggestion(nested); +} + +export const legacyPgDeltaNextEngineError = (cause: unknown) => { + if (cause instanceof LegacyPgDeltaEngineError) return cause; + const suggestion = legacyPgDeltaNextConnectSuggestion(cause); + return new LegacyPgDeltaEngineError({ + message: + typeof cause === "object" && + cause !== null && + typeof Reflect.get(cause, "message") === "string" + ? String(Reflect.get(cause, "message")) + : String(cause), + cause, + ...(suggestion !== undefined ? { suggestion } : {}), + }); +}; + +function normalizeNextDiff( + result: { + readonly changes: boolean; + readonly sql: string; + readonly files: ReadonlyArray<{ + readonly sequence: number; + readonly suffix: string | null; + readonly sql: string; + readonly transactional: boolean; + readonly actionCount: number; + }>; + readonly debug?: { + readonly sourceSnapshot?: string; + readonly desiredSnapshot?: string; + readonly plan?: string; + }; + }, + debugDirectory?: string, +): LegacyPgDeltaDiffResult { + return { + changes: result.changes, + sql: result.sql, + files: result.files.map((file) => ({ + sequence: file.sequence, + name: `segment_${file.sequence}`, + suffix: file.suffix, + sql: file.sql, + transactional: file.transactional, + actionCount: file.actionCount, + })), + ...(result.debug !== undefined + ? { + debug: { + ...result.debug, + ...(debugDirectory !== undefined ? { directory: debugDirectory } : {}), + }, + } + : {}), + }; +} + +function parseEndpoint(endpoint: LegacyPgDeltaDatabaseEndpoint) { + if (endpoint.connection !== undefined) return endpoint.connection; + const parsed = parseLegacyConnectionString(endpoint.ref); + if (parsed !== undefined) return parsed; + throw new LegacyPgDeltaEngineError({ + message: "failed to parse Postgres connection string for pg-delta", + cause: endpoint.ref.replace(/:[^:@/]+@/, ":***@"), + }); +} + +/** In-process pg-delta next implementation. Every pool and shadow is scope-owned. */ +export const legacyPgDeltaNextEngineLayer = Layer.effect( + LegacyPgDeltaEngine, + Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + const shadowService = yield* LegacyPgDeltaNextShadow; + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const debugLogger = yield* LegacyDebugLogger; + + const saveDebugArtifacts = ( + workdir: string, + operation: LegacyPgDeltaNextOperation, + artifacts: LegacyPgDeltaNextDebugArtifacts, + ) => + Effect.gen(function* () { + const id = legacyFormatPgDeltaNextDebugId(yield* Clock.currentTimeMillis, operation); + const debugDir = yield* legacySavePgDeltaNextDebugArtifacts( + fs, + path, + workdir, + id, + operation, + artifacts, + ); + yield* debugLogger.debug(`Saved pg-delta next debug artifacts to ${debugDir}.`); + return debugDir; + }).pipe( + Effect.catch((cause) => + debugLogger + .debug( + `Failed to save pg-delta next debug artifacts: ${ + typeof cause === "object" && + cause !== null && + typeof Reflect.get(cause, "message") === "string" + ? String(Reflect.get(cause, "message")) + : String(cause) + }`, + ) + .pipe(Effect.as(undefined)), + ), + ); + + const acquireDatabase = (endpoint: LegacyPgDeltaDatabaseEndpoint) => + legacyAcquirePgPool(parseEndpoint(endpoint), endpoint.connectOptions); + + const rejectBlockingDiagnostic = ( + operation: LegacyPgDeltaNextOperation, + diagnostics: Parameters[0], + ) => { + const blocking = legacyPgDeltaNextBlockingDiagnostic(diagnostics); + return blocking === undefined + ? Effect.void + : Effect.fail( + new LegacyPgDeltaEngineError({ + message: legacyPgDeltaNextBlockingDiagnosticMessage(operation, blocking), + cause: blocking, + }), + ); + }; + + return LegacyPgDeltaEngine.of({ + implementation: "next", + diffExplicit: (input) => + Effect.scoped( + Effect.gen(function* () { + let shadow: { readonly migrationsUrl: string; readonly scratchUrl: string } | undefined; + const migrationsEndpoint = + input.source.kind === "migrations" + ? input.source + : input.desired.kind === "migrations" + ? input.desired + : undefined; + if (migrationsEndpoint !== undefined) { + shadow = yield* shadowService.provision({ + schema: input.schema, + ...(migrationsEndpoint.projectRef !== undefined + ? { projectRef: migrationsEndpoint.projectRef } + : {}), + }); + } + const endpointPool = (endpoint: LegacyPgDeltaEndpoint) => + Effect.gen(function* () { + if (endpoint.kind === "database") return yield* acquireDatabase(endpoint); + if (shadow === undefined) { + return yield* Effect.die("missing pg-delta migrations shadow"); + } + const connection = parseLegacyConnectionString(shadow.migrationsUrl); + if (connection === undefined) { + return yield* Effect.fail( + new LegacyPgDeltaEngineError({ + message: "failed to parse pg-delta migrations shadow URL", + cause: shadow.migrationsUrl.replace(/:[^:@/]+@/, ":***@"), + }), + ); + } + return yield* legacyAcquirePgPool(connection, { + isLocal: true, + dnsResolver: "native", + }); + }); + const [sourcePool, desiredPool] = yield* Effect.all( + [endpointPool(input.source), endpointPool(input.desired)], + { concurrency: 2 }, + ); + const result = yield* adapter.diff({ + sourcePool, + desiredPool, + allowDrops: true, + debug: input.debug, + schema: input.schema, + }); + const debugDirectory = + result.debug !== undefined + ? yield* saveDebugArtifacts(input.context.cwd, "diff", { + ...result.debug, + diagnostics: result.diagnostics, + }) + : undefined; + yield* rejectBlockingDiagnostic("diff", result.diagnostics); + return normalizeNextDiff(result, debugDirectory); + }), + ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), + diffDatabase: (input) => + Effect.scoped( + Effect.gen(function* () { + const shadow = yield* shadowService.provision({ + schema: input.schema, + ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), + }); + const migrations = parseLegacyConnectionString(shadow.migrationsUrl); + const scratch = parseLegacyConnectionString(shadow.scratchUrl); + if (migrations === undefined || scratch === undefined) { + return yield* Effect.fail( + new LegacyPgDeltaEngineError({ + message: "failed to parse pg-delta next shadow database URL", + cause: "invalid password-free shadow output", + }), + ); + } + const migrationsPool = yield* legacyAcquirePgPool(migrations, { + isLocal: true, + dnsResolver: "native", + }); + if (input.declarativeFiles !== undefined) { + const scratchPool = yield* legacyAcquirePgPool(scratch, { + isLocal: true, + dnsResolver: "native", + }); + const result = yield* adapter.planDeclarativeSchema({ + targetPool: migrationsPool, + shadowPool: scratchPool, + files: input.declarativeFiles, + allowDrops: true, + debug: input.debug, + reorder: true, + seedAssumedSchemas: true, + schema: input.schema, + ...(input.declarativeManifest !== undefined + ? { manifest: input.declarativeManifest } + : {}), + }); + const debugDirectory = + result.debug !== undefined + ? yield* saveDebugArtifacts(input.context.cwd, "declarativePlan", { + ...result.debug, + diagnostics: result.diagnostics, + }) + : undefined; + yield* rejectBlockingDiagnostic("declarativePlan", result.diagnostics); + return normalizeNextDiff(result, debugDirectory); + } + const desiredPool = yield* acquireDatabase(input.target); + const result = yield* adapter.diff({ + sourcePool: migrationsPool, + desiredPool, + allowDrops: true, + debug: input.debug, + schema: input.schema, + }); + const debugDirectory = + result.debug !== undefined + ? yield* saveDebugArtifacts(input.context.cwd, "diff", { + ...result.debug, + diagnostics: result.diagnostics, + }) + : undefined; + yield* rejectBlockingDiagnostic("diff", result.diagnostics); + return normalizeNextDiff(result, debugDirectory); + }), + ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), + exportDeclarativeSchema: (input) => + Effect.scoped( + Effect.gen(function* () { + const pool = yield* acquireDatabase(input.target); + const result = yield* adapter.exportDeclarativeSchema({ + pool, + layout: "grouped", + schema: input.schema, + formatOptions: input.formatOptions, + }); + if (input.debug) { + const capture = yield* adapter + .captureSnapshot({ pool, redactSecrets: true }) + .pipe(Effect.orElseSucceed(() => undefined)); + yield* saveDebugArtifacts(input.context.cwd, "declarativeExport", { + ...(capture !== undefined ? { desiredSnapshot: capture.snapshot } : {}), + diagnostics: + capture === undefined + ? result.diagnostics + : [...result.diagnostics, ...capture.diagnostics], + }); + } + yield* rejectBlockingDiagnostic("declarativeExport", result.diagnostics); + return { files: result.files, manifest: result.manifest }; + }), + ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), + planDeclarativeSchema: (input) => + Effect.scoped( + Effect.gen(function* () { + const shadow = yield* shadowService.provision({ schema: input.schema }); + const migrations = parseLegacyConnectionString(shadow.migrationsUrl); + const scratch = parseLegacyConnectionString(shadow.scratchUrl); + if (migrations === undefined || scratch === undefined) { + return yield* Effect.fail( + new LegacyPgDeltaEngineError({ + message: "failed to parse pg-delta next shadow database URL", + cause: "invalid password-free shadow output", + }), + ); + } + const [migrationsPool, scratchPool] = yield* Effect.all( + [ + legacyAcquirePgPool(migrations, { isLocal: true, dnsResolver: "native" }), + legacyAcquirePgPool(scratch, { isLocal: true, dnsResolver: "native" }), + ], + { concurrency: 2 }, + ); + const result = yield* adapter.planDeclarativeSchema({ + targetPool: migrationsPool, + shadowPool: scratchPool, + files: input.files, + allowDrops: true, + debug: input.debug, + reorder: true, + seedAssumedSchemas: true, + schema: input.schema, + ...(input.manifest !== undefined ? { manifest: input.manifest } : {}), + }); + const debugDirectory = + result.debug !== undefined + ? yield* saveDebugArtifacts(input.context.cwd, "declarativePlan", { + ...result.debug, + diagnostics: result.diagnostics, + }) + : undefined; + yield* rejectBlockingDiagnostic("declarativePlan", result.diagnostics); + return { + ...normalizeNextDiff(result, debugDirectory), + sourceRef: "pg-delta-next:migrations", + targetRef: "pg-delta-next:declarative", + }; + }), + ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), + }); + }), +); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts new file mode 100644 index 0000000000..d1db522f46 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts @@ -0,0 +1,44 @@ +import { describe, expect, it } from "vitest"; + +import { LegacyDbConnectError } from "../../../shared/legacy-db-connection.errors.ts"; +import { legacyPgDeltaNextEngineError } from "./legacy-pgdelta-engine.next.layer.ts"; +import { LegacyPgDeltaEngineError } from "./legacy-pgdelta-engine.service.ts"; +import { LegacyPgDeltaNextError } from "./legacy-pgdelta-next-adapter.service.ts"; + +describe("pg-delta next engine errors", () => { + it("preserves database connection suggestions when wrapping failures", () => { + const cause = new LegacyDbConnectError({ + message: "failed to connect to postgres", + suggestion: "Retry with --dns-resolver https.", + }); + + expect(legacyPgDeltaNextEngineError(cause)).toEqual( + new LegacyPgDeltaEngineError({ + message: "failed to connect to postgres", + suggestion: "Retry with --dns-resolver https.", + cause, + }), + ); + }); + + it("finds connection suggestions nested in adapter failures", () => { + const cause = new LegacyDbConnectError({ + message: "failed to connect to postgres", + suggestion: "Retry with --dns-resolver https.", + }); + const adapterError = new LegacyPgDeltaNextError({ + operation: "diff", + message: "Database diff failed", + cause, + }); + + expect(legacyPgDeltaNextEngineError(adapterError).suggestion).toBe( + "Retry with --dns-resolver https.", + ); + }); + + it("does not wrap an existing engine error again", () => { + const error = new LegacyPgDeltaEngineError({ message: "blocked", cause: "diagnostic" }); + expect(legacyPgDeltaNextEngineError(error)).toBe(error); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts new file mode 100644 index 0000000000..a5163f7263 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts @@ -0,0 +1,135 @@ +import { Context, Data, type Effect } from "effect"; + +import type { + LegacyDbConnectOptions, + LegacyPgConnInput, +} from "../../../shared/legacy-db-connection.service.ts"; +import type { LegacyPgDeltaContext } from "./legacy-pgdelta.ts"; +import type { LegacyPgDeltaImplementation } from "./legacy-pgdelta-next-flag.ts"; + +export interface LegacyPgDeltaDatabaseEndpoint { + readonly kind: "database"; + /** URL/reference used by the legacy edge-runtime implementation. */ + readonly ref: string; + /** Full parsed connection, preferred by the next implementation. */ + readonly connection?: LegacyPgConnInput; + readonly connectOptions: LegacyDbConnectOptions; +} + +interface LegacyPgDeltaMigrationsEndpoint { + readonly kind: "migrations"; + readonly projectRef?: string; +} + +export type LegacyPgDeltaEndpoint = LegacyPgDeltaDatabaseEndpoint | LegacyPgDeltaMigrationsEndpoint; + +export interface LegacyPgDeltaSqlFile { + readonly name: string; + readonly sql: string; +} + +export interface LegacyPgDeltaExportManifest { + readonly redactSecrets: boolean; + readonly scope: "database" | "cluster"; + readonly profile?: string; + readonly baselineDigest?: string; + readonly defaultOwner?: string | null; + readonly files?: ReadonlyArray; +} + +export interface LegacyPgDeltaRenderedFile { + readonly sequence: number; + /** Legacy semantic unit name. */ + readonly name: string; + /** Next renderer's exact filename suffix (`null`, `_1`, `_2`, ...). */ + readonly suffix?: string | null; + readonly sql: string; + readonly transactional: boolean; + readonly actionCount?: number; +} + +interface LegacyPgDeltaDebugArtifacts { + readonly sourceSnapshot?: string; + readonly desiredSnapshot?: string; + readonly plan?: string; + readonly stderr?: string; + /** Persisted debug directory, when the selected implementation writes one. */ + readonly directory?: string; +} + +export interface LegacyPgDeltaDiffResult { + readonly changes: boolean; + readonly sql: string; + readonly files: ReadonlyArray; + readonly debug?: LegacyPgDeltaDebugArtifacts; +} + +interface LegacyPgDeltaCommonInput { + readonly context: LegacyPgDeltaContext; + readonly schema: ReadonlyArray; + readonly formatOptions: string; + readonly projectRef?: string; + readonly debug: boolean; +} + +export interface LegacyPgDeltaExplicitDiffInput extends LegacyPgDeltaCommonInput { + readonly source: LegacyPgDeltaEndpoint; + readonly desired: LegacyPgDeltaEndpoint; +} + +export interface LegacyPgDeltaDatabaseDiffInput extends LegacyPgDeltaCommonInput { + readonly target: LegacyPgDeltaDatabaseEndpoint; + readonly targetLocal: boolean; + /** Present when the local desired state is declarative SQL rather than the live DB. */ + readonly declarativeFiles?: ReadonlyArray; + readonly declarativeManifest?: LegacyPgDeltaExportManifest; +} + +interface LegacyPgDeltaDeclarativeExportInput extends LegacyPgDeltaCommonInput { + readonly target: LegacyPgDeltaDatabaseEndpoint; + readonly noCache: boolean; +} + +export interface LegacyPgDeltaDeclarativeExportResult { + readonly files: ReadonlyArray; + readonly manifest?: LegacyPgDeltaExportManifest; +} + +export interface LegacyPgDeltaDeclarativePlanInput extends LegacyPgDeltaCommonInput { + readonly files: ReadonlyArray; + readonly manifest?: LegacyPgDeltaExportManifest; + readonly noCache: boolean; +} + +interface LegacyPgDeltaDeclarativePlanResult extends LegacyPgDeltaDiffResult { + /** Debug labels retained for the legacy apply/reset bundle. */ + readonly sourceRef: string; + readonly targetRef: string; +} + +export class LegacyPgDeltaEngineError extends Data.TaggedError("LegacyPgDeltaEngineError")<{ + readonly message: string; + readonly cause: unknown; + readonly suggestion?: string; +}> {} + +export interface LegacyPgDeltaEngineShape { + readonly implementation: LegacyPgDeltaImplementation; + readonly diffExplicit: ( + input: LegacyPgDeltaExplicitDiffInput, + ) => Effect.Effect; + readonly diffDatabase: ( + input: LegacyPgDeltaDatabaseDiffInput, + ) => Effect.Effect; + readonly exportDeclarativeSchema: ( + input: LegacyPgDeltaDeclarativeExportInput, + ) => Effect.Effect; + readonly planDeclarativeSchema: ( + input: LegacyPgDeltaDeclarativePlanInput, + ) => Effect.Effect; +} + +export class LegacyPgDeltaEngine extends Context.Service< + LegacyPgDeltaEngine, + LegacyPgDeltaEngineShape +>()("supabase/legacy/PgDeltaEngine") {} diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts new file mode 100644 index 0000000000..d24e0562f4 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts @@ -0,0 +1,168 @@ +import { Data, Effect, type FileSystem, Option, type Path } from "effect"; + +import type { + LegacyPgDeltaExportManifest, + LegacyPgDeltaSqlFile, +} from "./legacy-pgdelta-engine.service.ts"; +import { legacyResolveSqlGlobFiles } from "../../../shared/legacy-seed-ops.ts"; + +const EXPORT_MANIFEST_FILE = ".pgdelta-export.json"; + +class LegacyPgDeltaFilesError extends Data.TaggedError("LegacyPgDeltaFilesError")<{ + readonly message: string; +}> {} + +const filesError = (message: string) => new LegacyPgDeltaFilesError({ message }); + +function readManifestValue(doc: object, key: string): unknown { + return Reflect.get(doc, key); +} + +/** Reads a next-engine export manifest from an explicit declarative directory. */ +export const LegacyReadPgDeltaExportManifest = Effect.fnUntraced(function* ( + fs: FileSystem.FileSystem, + path: Path.Path, + directory: string, +) { + const manifestPath = path.join(directory, EXPORT_MANIFEST_FILE); + const exists = yield* fs + .exists(manifestPath) + .pipe( + Effect.mapError((error) => filesError(`cannot inspect export manifest: ${error.message}`)), + ); + if (!exists) return undefined; + + const raw = yield* fs + .readFileString(manifestPath) + .pipe( + Effect.mapError((error) => + filesError(`cannot read export manifest ${manifestPath}: ${error.message}`), + ), + ); + const decoded = yield* Effect.try({ + try: (): unknown => JSON.parse(raw), + catch: (cause) => + filesError( + `malformed export manifest ${manifestPath}: ${cause instanceof Error ? cause.message : String(cause)}`, + ), + }); + if (typeof decoded !== "object" || decoded === null || Array.isArray(decoded)) { + return yield* Effect.fail(filesError(`malformed export manifest ${manifestPath}`)); + } + + const formatVersion = readManifestValue(decoded, "formatVersion"); + const redactSecrets = readManifestValue(decoded, "redactSecrets"); + const scope = readManifestValue(decoded, "scope"); + if ( + (formatVersion !== undefined && formatVersion !== 1) || + typeof redactSecrets !== "boolean" || + (scope !== "database" && scope !== "cluster") + ) { + return yield* Effect.fail( + filesError(`export manifest ${manifestPath} is missing required policy metadata`), + ); + } + + const profile = readManifestValue(decoded, "profile"); + const baselineDigest = readManifestValue(decoded, "baselineDigest"); + const defaultOwner = readManifestValue(decoded, "defaultOwner"); + const files = readManifestValue(decoded, "files"); + return { + redactSecrets, + scope, + ...(typeof profile === "string" ? { profile } : {}), + ...(typeof baselineDigest === "string" ? { baselineDigest } : {}), + ...(typeof defaultOwner === "string" || defaultOwner === null ? { defaultOwner } : {}), + ...(Array.isArray(files) && files.every((file) => typeof file === "string") ? { files } : {}), + } satisfies LegacyPgDeltaExportManifest; +}); + +/** Recursively loads path-safe `.sql` files in stable POSIX-relative order. */ +export const LegacyLoadPgDeltaSqlFiles = Effect.fnUntraced(function* ( + fs: FileSystem.FileSystem, + path: Path.Path, + directory: string, +) { + const pending = [directory]; + const paths: Array<{ readonly full: string; readonly name: string }> = []; + + while (pending.length > 0) { + const current = pending.pop(); + if (current === undefined) break; + const entries = yield* fs + .readDirectory(current) + .pipe( + Effect.mapError((error) => + filesError(`failed to read declarative schema directory: ${error.message}`), + ), + ); + for (const entry of entries) { + const full = path.join(current, entry); + const stat = yield* fs + .stat(full) + .pipe( + Effect.mapError((error) => + filesError(`failed to inspect declarative schema file: ${error.message}`), + ), + ); + if (stat.type === "Directory") { + pending.push(full); + continue; + } + if (path.extname(entry).toLowerCase() !== ".sql") continue; + + const name = path.relative(directory, full).split("\\").join("/"); + const normalized = path.normalize(name); + if (normalized.startsWith("..") || path.isAbsolute(normalized)) { + return yield* Effect.fail(filesError(`unsafe declarative schema path: ${name}`)); + } + paths.push({ full, name }); + } + } + + paths.sort((left, right) => left.name.localeCompare(right.name)); + const files: Array = []; + for (const file of paths) { + const sql = yield* fs + .readFileString(file.full) + .pipe( + Effect.mapError((error) => + filesError(`failed to read declarative schema file: ${error.message}`), + ), + ); + files.push({ name: file.name, sql }); + } + return files; +}); + +/** Loads `[db.migrations].schema_paths` in configured pattern/application order. */ +export const LegacyLoadPgDeltaSqlPaths = Effect.fnUntraced(function* ( + fs: FileSystem.FileSystem, + path: Path.Path, + workdir: string, + patterns: ReadonlyArray, +) { + const resolved = yield* legacyResolveSqlGlobFiles(fs, path, patterns, workdir); + if (resolved.files.length === 0) { + return yield* Effect.fail( + filesError( + Option.isSome(resolved.warning) + ? resolved.warning.value + : "no declarative schema files matched schema_paths", + ), + ); + } + const files: Array = []; + for (const file of resolved.files) { + const full = path.isAbsolute(file) ? file : path.join(workdir, file); + const sql = yield* fs + .readFileString(full) + .pipe( + Effect.mapError((error) => + filesError(`failed to read declarative schema file: ${error.message}`), + ), + ); + files.push({ name: file.split("\\").join("/"), sql }); + } + return files; +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts index e02b5c720f..35d9de726c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts @@ -58,7 +58,11 @@ export const legacyWritePgDeltaMigrations = ( readonly workdir: string; readonly baseMillis: number; readonly name: string; - readonly files: ReadonlyArray<{ readonly name: string; readonly sql: string }>; + readonly files: ReadonlyArray<{ + readonly name: string; + readonly suffix?: string | null; + readonly sql: string; + }>; }, ): Effect.Effect, LegacyPgDeltaMigrationWriteError> => Effect.gen(function* () { @@ -67,7 +71,11 @@ export const legacyWritePgDeltaMigrations = ( const buildSet = (baseMillis: number): Array => files.map((file, i) => { const version = legacyFormatMigrationTimestamp(baseMillis + i * 1000); - const unitName = single ? name : `${name}_${file.name}`; + const unitName = single + ? name + : file.suffix !== undefined && file.suffix !== null + ? `${name}${file.suffix}` + : `${name}_${file.name}`; return { path: legacyGetMigrationPath(pathSvc, workdir, version, unitName), version }; }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts new file mode 100644 index 0000000000..119cca4f9c --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -0,0 +1,615 @@ +import { Effect, Layer } from "effect"; +import type { Pool } from "pg"; +import { serializeSnapshot, encodeId } from "@supabase/pg-delta/core"; +import { buildSchemaExport, planSchemaFiles, renderPlanFiles } from "@supabase/pg-delta/frontends"; +import { + type IntegrationProfile, + resolveProfile, + supabaseProfile, +} from "@supabase/pg-delta/integrations"; +import { plan, serializePlan } from "@supabase/pg-delta/plan"; +import type { Policy } from "@supabase/pg-delta/policy"; +import type { SqlFormatOptions } from "@supabase/pg-delta/sql-format"; + +import { + LegacyPgDeltaNextAdapter, + LegacyPgDeltaNextError, + type LegacyPgDeltaNextAdapterShape, + type LegacyPgDeltaNextDeclarativeExportInput, + type LegacyPgDeltaNextDeclarativeManifestInput, + type LegacyPgDeltaNextDeclarativePlanInput, + type LegacyPgDeltaNextDiagnostic, + type LegacyPgDeltaNextDiagnosticOrigin, + type LegacyPgDeltaNextDiffInput, + type LegacyPgDeltaNextExportManifest, + type LegacyPgDeltaNextRenderedFile, + type LegacyPgDeltaNextSnapshotCaptureInput, + type LegacyPgDeltaNextSqlFile, + type LegacyPgDeltaNextOperation, +} from "./legacy-pgdelta-next-adapter.service.ts"; + +interface LegacyPgDeltaNextLibraryDiagnostic { + readonly code: string; + readonly severity: "error" | "warning" | "info"; + readonly subject?: Subject; + readonly message: string; + readonly context?: Readonly>; +} + +interface LegacyPgDeltaNextLibraryExtractResult { + readonly factBase: FactBase; + readonly pgVersion: string; + readonly diagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[]; +} + +interface LegacyPgDeltaNextResolvedProfile { + readonly id: string; + readonly planOptions: PlanOptions; + readonly extract: ( + pool: Pool, + options?: { readonly redactSecrets?: boolean; readonly statementTimeoutMs?: number }, + ) => Promise>; +} + +interface LegacyPgDeltaNextLibraryRenderedFile { + readonly suffix: string | null; + readonly contents: string; + readonly transactional: boolean; + readonly actionCount: number; +} + +interface LegacyPgDeltaNextLibraryRenderedResult { + readonly changes: boolean; + readonly files: readonly LegacyPgDeltaNextLibraryRenderedFile[]; +} + +interface LegacyPgDeltaNextLibrarySchemaExport { + readonly files: readonly LegacyPgDeltaNextSqlFile[]; + readonly diagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[]; + readonly manifest: LegacyPgDeltaNextExportManifest; +} + +type LegacyPgDeltaNextLibraryExportOptions = ReturnType; + +interface LegacyPgDeltaNextLibrarySchemaPlan { + readonly plan: Plan; + readonly loadDiagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[]; + readonly targetDiagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[]; + readonly skipped: readonly { readonly file: string; readonly stmt: string }[]; +} + +export interface LegacyPgDeltaNextLibraries { + readonly resolveProfile: ( + pool: Pool, + options: { + readonly restrictToApplier?: boolean; + readonly redactSecrets?: boolean; + readonly skipBaseline?: boolean; + }, + schema?: readonly string[], + ) => Promise>; + readonly plan: ( + source: FactBase, + desired: FactBase, + options: PlanOptions & { readonly redactSecrets: boolean }, + ) => Plan; + readonly renderPlanFiles: ( + plan: Plan, + options: { readonly allowDrops: boolean }, + ) => LegacyPgDeltaNextLibraryRenderedResult; + readonly buildSchemaExport: ( + pool: Pool, + input: LegacyPgDeltaNextLibraryExportOptions, + ) => Promise>; + readonly planSchemaFiles: ( + targetPool: Pool, + shadowPool: Pool, + files: readonly LegacyPgDeltaNextSqlFile[], + input: LegacyPgDeltaNextDeclarativePlanInput, + ) => Promise>; + readonly serializeSnapshot: ( + factBase: FactBase, + metadata: { + readonly pgVersion: string; + readonly redactSecrets: boolean; + readonly profile: string; + }, + ) => string; + readonly serializePlan: (plan: Plan) => string; + readonly encodeSubject: (subject: Subject) => string; +} + +function legacyPgDeltaNextMessage(operation: LegacyPgDeltaNextOperation, cause: unknown): string { + const detail = cause instanceof Error ? cause.message : String(cause); + const label = + operation === "declarativeExport" + ? "Declarative schema export" + : operation === "declarativePlan" + ? "Declarative schema planning" + : operation === "snapshotCapture" + ? "Snapshot capture" + : "Database diff"; + return `${label} failed: ${detail}`; +} + +function legacyTryPgDeltaNext( + operation: LegacyPgDeltaNextOperation, + run: () => Promise, +) { + return Effect.tryPromise({ + try: run, + catch: (cause) => + new LegacyPgDeltaNextError({ + operation, + message: legacyPgDeltaNextMessage(operation, cause), + cause, + }), + }); +} + +function legacyNormalizePgDeltaNextDiagnostics( + diagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[], + origin: LegacyPgDeltaNextDiagnosticOrigin, + encodeSubject: (subject: Subject) => string, +): LegacyPgDeltaNextDiagnostic[] { + return diagnostics.map((diagnostic) => ({ + origin, + code: diagnostic.code, + severity: diagnostic.severity, + ...(diagnostic.subject !== undefined ? { subject: encodeSubject(diagnostic.subject) } : {}), + message: diagnostic.message, + ...(diagnostic.context !== undefined ? { context: diagnostic.context } : {}), + })); +} + +function legacyIsPgDeltaNextParameterAclDiagnostic( + diagnostic: LegacyPgDeltaNextLibraryDiagnostic, +): boolean { + return diagnostic.code === "unmodeled_kind" && diagnostic.context?.["kind"] === "parameter ACL"; +} + +/** + * The parameter-ACL catalog is cluster-wide, so a co-located declarative shadow + * observes Supabase platform grants too. Keep strict coverage for every ACL + * other than the exact platform bootstrap grant while removing the aggregate + * diagnostic when that bootstrap grant is the only observed parameter ACL. + */ +export function legacyFilterPgDeltaNextPlatformParameterAclDiagnostics( + diagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[], + userOwnedParameterAcls: readonly string[], +): LegacyPgDeltaNextLibraryDiagnostic[] { + const names = [...new Set(userOwnedParameterAcls)].sort(); + const filtered: LegacyPgDeltaNextLibraryDiagnostic[] = []; + for (const diagnostic of diagnostics) { + if (!legacyIsPgDeltaNextParameterAclDiagnostic(diagnostic)) { + filtered.push(diagnostic); + continue; + } + if (names.length === 0) continue; + const samples = names.slice(0, 5); + const more = names.length > samples.length ? ", …" : ""; + filtered.push({ + ...diagnostic, + message: + `${names.length} unmodeled "parameter ACL" object${names.length === 1 ? "" : "s"} ` + + `not managed by this engine (e.g. ${samples.join(", ")}${more}) — ` + + "v1 detects but does not model this kind", + context: { kind: "parameter ACL", count: names.length, samples }, + }); + } + return filtered; +} + +interface LegacyPgDeltaNextParameterAclGrant { + readonly name: string; + readonly grantee: string; + readonly privilege: string; +} + +// Supabase's platform bootstrap grants these so privileged platform roles can +// manage the setting and the Realtime owner can replay routines whose proconfig +// contains `SET log_min_messages ...`. Parameter ACLs have cluster scope, so +// the grants are also visible from sibling shadow DBs. +const legacyPgDeltaNextPlatformParameterAcls = new Set([ + "log_min_messages\u0000supabase_admin\u0000ALTER SYSTEM", + "log_min_messages\u0000supabase_admin\u0000SET", + "log_min_messages\u0000supabase_realtime_admin\u0000SET", +]); + +function legacyPgDeltaNextParameterAclKey(grant: LegacyPgDeltaNextParameterAclGrant): string { + return `${grant.name}\u0000${grant.grantee}\u0000${grant.privilege}`; +} + +export function legacyPgDeltaNextUserOwnedParameterAcls( + grants: readonly LegacyPgDeltaNextParameterAclGrant[], +): string[] { + return [ + ...new Set( + grants + .filter( + (grant) => + !legacyPgDeltaNextPlatformParameterAcls.has(legacyPgDeltaNextParameterAclKey(grant)), + ) + .map((grant) => grant.name), + ), + ].sort(); +} + +async function legacyFilterPgDeltaNextPlatformDiagnostics( + pool: Pool, + diagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[], +): Promise[]> { + if (!diagnostics.some(legacyIsPgDeltaNextParameterAclDiagnostic)) return [...diagnostics]; + const result = await pool.query( + `SELECT DISTINCT pa.parname AS name, + COALESCE(grantee.rolname, 'PUBLIC') AS grantee, + acl.privilege_type AS privilege + FROM pg_parameter_acl pa + CROSS JOIN LATERAL aclexplode(pa.paracl) acl + LEFT JOIN pg_roles grantee ON grantee.oid = acl.grantee + ORDER BY pa.parname, grantee, privilege`, + ); + return legacyFilterPgDeltaNextPlatformParameterAclDiagnostics( + diagnostics, + legacyPgDeltaNextUserOwnedParameterAcls(result.rows), + ); +} + +function legacyNormalizePgDeltaNextRenderedFiles( + files: readonly LegacyPgDeltaNextLibraryRenderedFile[], +): LegacyPgDeltaNextRenderedFile[] { + return files.map((file, index) => ({ + sequence: index + 1, + suffix: file.suffix, + sql: file.contents, + transactional: file.transactional, + actionCount: file.actionCount, + })); +} + +export function legacyPgDeltaNextProfile( + schema: readonly string[] | undefined, +): IntegrationProfile { + if (schema === undefined || schema.length === 0 || supabaseProfile.policy === undefined) { + return supabaseProfile; + } + const selected = [...schema]; + const policy: Policy = { + id: `supabase-cli-schemas:${selected.join(",")}`, + filter: [ + { + match: { all: [{ schema: "*" }, { not: { schema: selected } }] }, + action: "exclude", + }, + { + match: { all: [{ kind: "schema" }, { not: { name: selected } }] }, + action: "exclude", + }, + { + match: { + all: [{ target: { schema: "*" } }, { not: { target: { schema: selected } } }], + }, + action: "exclude", + }, + ], + extends: [supabaseProfile.policy], + }; + return { ...supabaseProfile, policy }; +} + +function legacyPgDeltaNextFormatOptions(raw: string | undefined): SqlFormatOptions | undefined { + if (raw === undefined || raw.trim().length === 0) return undefined; + const parsed: unknown = JSON.parse(raw); + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) return undefined; + const value = (key: string): unknown => Reflect.get(parsed, key); + const keywordCase = value("keywordCase"); + const commaStyle = value("commaStyle"); + const indent = value("indent"); + const maxWidth = value("maxWidth"); + const alignColumns = value("alignColumns"); + const alignKeyValues = value("alignKeyValues"); + const preserveRoutineBodies = value("preserveRoutineBodies"); + const preserveViewBodies = value("preserveViewBodies"); + const preserveRuleBodies = value("preserveRuleBodies"); + return { + ...(keywordCase === "upper" || keywordCase === "lower" || keywordCase === "preserve" + ? { keywordCase } + : {}), + ...(commaStyle === "trailing" || commaStyle === "leading" ? { commaStyle } : {}), + ...(typeof indent === "number" ? { indent } : {}), + ...(typeof maxWidth === "number" ? { maxWidth } : {}), + ...(typeof alignColumns === "boolean" ? { alignColumns } : {}), + ...(typeof alignKeyValues === "boolean" ? { alignKeyValues } : {}), + ...(typeof preserveRoutineBodies === "boolean" ? { preserveRoutineBodies } : {}), + ...(typeof preserveViewBodies === "boolean" ? { preserveViewBodies } : {}), + ...(typeof preserveRuleBodies === "boolean" ? { preserveRuleBodies } : {}), + }; +} + +function legacyPgDeltaNextExportOptions(input: LegacyPgDeltaNextDeclarativeExportInput) { + const format = legacyPgDeltaNextFormatOptions(input.formatOptions); + return { + profile: legacyPgDeltaNextProfile(input.schema), + ...(input.scope !== undefined ? { scope: input.scope } : {}), + ...(input.redactSecrets !== undefined ? { redactSecrets: input.redactSecrets } : {}), + ...(input.restrictToApplier !== undefined + ? { resolveOptions: { restrictToApplier: input.restrictToApplier } } + : {}), + ...(input.layout !== undefined ? { layout: input.layout } : {}), + ...(input.grouping !== undefined + ? { + grouping: { + ...(input.grouping.mode !== undefined ? { mode: input.grouping.mode } : {}), + ...(input.grouping.groupPatterns !== undefined + ? { groupPatterns: [...input.grouping.groupPatterns] } + : {}), + ...(input.grouping.flatSchemas !== undefined + ? { flatSchemas: [...input.grouping.flatSchemas] } + : {}), + ...(input.grouping.autoGroupPartitions !== undefined + ? { autoGroupPartitions: input.grouping.autoGroupPartitions } + : {}), + }, + } + : {}), + ...(input.defaultOwner !== undefined ? { defaultOwner: input.defaultOwner } : {}), + ...(format !== undefined ? { format } : {}), + ...(input.onWarning !== undefined ? { onWarning: input.onWarning } : {}), + }; +} + +function legacyPgDeltaNextManifest(manifest: LegacyPgDeltaNextDeclarativeManifestInput) { + return { + ...(manifest.redactSecrets !== undefined ? { redactSecrets: manifest.redactSecrets } : {}), + ...(manifest.profile !== undefined ? { profile: manifest.profile } : {}), + ...(manifest.scope !== undefined ? { scope: manifest.scope } : {}), + ...(manifest.baselineDigest !== undefined ? { baselineDigest: manifest.baselineDigest } : {}), + ...(manifest.defaultOwner !== undefined ? { defaultOwner: manifest.defaultOwner } : {}), + ...(manifest.files !== undefined ? { files: [...manifest.files] } : {}), + }; +} + +function legacyPgDeltaNextPlanOptions(input: LegacyPgDeltaNextDeclarativePlanInput) { + return { + profile: legacyPgDeltaNextProfile(input.schema), + ...(input.scope !== undefined ? { scope: input.scope } : {}), + ...(input.manifest !== undefined + ? { manifest: legacyPgDeltaNextManifest(input.manifest) } + : {}), + ...(input.redactSecrets !== undefined ? { redactSecrets: input.redactSecrets } : {}), + ...(input.skipClusterDdl !== undefined ? { skipClusterDdl: input.skipClusterDdl } : {}), + ...(input.isolatedShadow !== undefined ? { isolatedShadow: input.isolatedShadow } : {}), + ...(input.seedAssumedSchemas !== undefined + ? { seedAssumedSchemas: input.seedAssumedSchemas } + : {}), + ...(input.restrictToApplier !== undefined + ? { resolveOptions: { restrictToApplier: input.restrictToApplier } } + : {}), + ...(input.strictFunctionBodies !== undefined + ? { strictFunctionBodies: input.strictFunctionBodies } + : {}), + reorder: input.reorder ?? true, + ...(input.onWarning !== undefined ? { onWarning: input.onWarning } : {}), + }; +} + +function legacyMakePgDeltaNextAdapter( + libraries: LegacyPgDeltaNextLibraries, +): LegacyPgDeltaNextAdapterShape { + return { + diff: (input: LegacyPgDeltaNextDiffInput) => + legacyTryPgDeltaNext("diff", async () => { + const redactSecrets = input.redactSecrets ?? true; + const profile = await libraries.resolveProfile( + input.sourcePool, + { + redactSecrets, + ...(input.restrictToApplier !== undefined + ? { restrictToApplier: input.restrictToApplier } + : {}), + }, + input.schema, + ); + const [source, desired] = await Promise.all([ + profile.extract(input.sourcePool, { redactSecrets }), + profile.extract(input.desiredPool, { redactSecrets }), + ]); + const generatedPlan = libraries.plan(source.factBase, desired.factBase, { + ...profile.planOptions, + redactSecrets, + }); + const rendered = libraries.renderPlanFiles(generatedPlan, { + allowDrops: input.allowDrops, + }); + const diagnostics = [ + ...legacyNormalizePgDeltaNextDiagnostics( + source.diagnostics, + "source", + libraries.encodeSubject, + ), + ...legacyNormalizePgDeltaNextDiagnostics( + desired.diagnostics, + "desired", + libraries.encodeSubject, + ), + ]; + return { + changes: rendered.changes, + sql: rendered.files.map((file) => file.contents).join("\n\n"), + files: legacyNormalizePgDeltaNextRenderedFiles(rendered.files), + diagnostics, + ...(input.debug + ? { + debug: { + sourceSnapshot: libraries.serializeSnapshot(source.factBase, { + pgVersion: source.pgVersion, + redactSecrets, + profile: profile.id, + }), + desiredSnapshot: libraries.serializeSnapshot(desired.factBase, { + pgVersion: desired.pgVersion, + redactSecrets, + profile: profile.id, + }), + plan: libraries.serializePlan(generatedPlan), + }, + } + : {}), + }; + }), + exportDeclarativeSchema: (input: LegacyPgDeltaNextDeclarativeExportInput) => + legacyTryPgDeltaNext("declarativeExport", async () => { + const result = await libraries.buildSchemaExport( + input.pool, + legacyPgDeltaNextExportOptions(input), + ); + return { + files: result.files.map((file) => ({ name: file.name, sql: file.sql })), + manifest: { + ...result.manifest, + files: result.files.map((file) => file.name).sort(), + }, + diagnostics: legacyNormalizePgDeltaNextDiagnostics( + result.diagnostics, + "export", + libraries.encodeSubject, + ), + }; + }), + planDeclarativeSchema: (input: LegacyPgDeltaNextDeclarativePlanInput) => + legacyTryPgDeltaNext("declarativePlan", async () => { + const planningInput = { ...input, reorder: input.reorder ?? true }; + const result = await libraries.planSchemaFiles( + input.targetPool, + input.shadowPool, + input.files, + planningInput, + ); + const rendered = libraries.renderPlanFiles(result.plan, { + allowDrops: input.allowDrops, + }); + return { + changes: rendered.changes, + sql: rendered.files.map((file) => file.contents).join("\n\n"), + files: legacyNormalizePgDeltaNextRenderedFiles(rendered.files), + diagnostics: [ + ...legacyNormalizePgDeltaNextDiagnostics( + result.loadDiagnostics, + "declarativeLoad", + libraries.encodeSubject, + ), + ...legacyNormalizePgDeltaNextDiagnostics( + result.targetDiagnostics, + "declarativeTarget", + libraries.encodeSubject, + ), + ], + skipped: result.skipped.map((skipped) => ({ + file: skipped.file, + statement: skipped.stmt, + })), + ...(input.debug ? { debug: { plan: libraries.serializePlan(result.plan) } } : {}), + }; + }), + captureSnapshot: (input: LegacyPgDeltaNextSnapshotCaptureInput) => + legacyTryPgDeltaNext("snapshotCapture", async () => { + const redactSecrets = input.redactSecrets ?? true; + const profile = await libraries.resolveProfile(input.pool, { + redactSecrets, + skipBaseline: true, + }); + const result = await profile.extract(input.pool, { + redactSecrets, + ...(input.statementTimeoutMs !== undefined + ? { statementTimeoutMs: input.statementTimeoutMs } + : {}), + }); + return { + generation: "v2", + snapshot: libraries.serializeSnapshot(result.factBase, { + pgVersion: result.pgVersion, + redactSecrets, + profile: profile.id, + }), + pgVersion: result.pgVersion, + diagnostics: legacyNormalizePgDeltaNextDiagnostics( + result.diagnostics, + "snapshot", + libraries.encodeSubject, + ), + }; + }), + }; +} + +const legacyPgDeltaNextRealLibraries = { + resolveProfile: async ( + pool: Pool, + options: Parameters[2], + schema?: readonly string[], + ) => { + const resolved = await resolveProfile(pool, legacyPgDeltaNextProfile(schema), options); + return { + ...resolved, + extract: async ( + extractPool: Pool, + extractOptions?: Parameters[1], + ) => { + const result = await resolved.extract(extractPool, extractOptions); + return { + ...result, + diagnostics: await legacyFilterPgDeltaNextPlatformDiagnostics( + extractPool, + result.diagnostics, + ), + }; + }, + }; + }, + plan, + renderPlanFiles, + buildSchemaExport: async (pool: Pool, input: LegacyPgDeltaNextLibraryExportOptions) => { + const result = await buildSchemaExport(pool, input); + return { + ...result, + diagnostics: await legacyFilterPgDeltaNextPlatformDiagnostics(pool, result.diagnostics), + }; + }, + planSchemaFiles: async ( + targetPool: Pool, + shadowPool: Pool, + files: readonly LegacyPgDeltaNextSqlFile[], + input: LegacyPgDeltaNextDeclarativePlanInput, + ) => { + const result = await planSchemaFiles( + targetPool, + shadowPool, + files.map((file) => ({ name: file.name, sql: file.sql })), + legacyPgDeltaNextPlanOptions(input), + ); + const [loadDiagnostics, targetDiagnostics] = await Promise.all([ + legacyFilterPgDeltaNextPlatformDiagnostics(shadowPool, result.loadDiagnostics), + legacyFilterPgDeltaNextPlatformDiagnostics(targetPool, result.targetDiagnostics), + ]); + return { ...result, loadDiagnostics, targetDiagnostics }; + }, + serializeSnapshot, + serializePlan, + encodeSubject: encodeId, +}; + +export function legacyPgDeltaNextAdapterLayerFromLibraries< + FactBase, + PlanOptions extends object, + Plan, + Subject, +>(libraries: LegacyPgDeltaNextLibraries) { + return Layer.succeed( + LegacyPgDeltaNextAdapter, + LegacyPgDeltaNextAdapter.of(legacyMakePgDeltaNextAdapter(libraries)), + ); +} + +export const legacyPgDeltaNextAdapterLayer = legacyPgDeltaNextAdapterLayerFromLibraries( + legacyPgDeltaNextRealLibraries, +); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts new file mode 100644 index 0000000000..80b6cbb130 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts @@ -0,0 +1,189 @@ +import type { Pool } from "pg"; +import { Context, Data, type Effect } from "effect"; + +export type LegacyPgDeltaNextOperation = + | "diff" + | "declarativeExport" + | "declarativePlan" + | "snapshotCapture"; + +export type LegacyPgDeltaNextDiagnosticOrigin = + | "source" + | "desired" + | "export" + | "declarativeLoad" + | "declarativeTarget" + | "snapshot"; + +export interface LegacyPgDeltaNextDiagnostic { + readonly origin: LegacyPgDeltaNextDiagnosticOrigin; + readonly code: string; + readonly severity: "error" | "warning" | "info"; + readonly subject?: string; + readonly message: string; + readonly context?: Readonly>; +} + +export interface LegacyPgDeltaNextRenderedFile { + readonly sequence: number; + readonly suffix: string | null; + readonly sql: string; + readonly transactional: boolean; + readonly actionCount: number; +} + +export interface LegacyPgDeltaNextSqlFile { + readonly name: string; + readonly sql: string; +} + +interface LegacyPgDeltaNextDebugArtifacts { + readonly sourceSnapshot?: string; + readonly desiredSnapshot?: string; + readonly plan?: string; +} + +export interface LegacyPgDeltaNextDiffInput { + /** The live database the rendered migration will be applied to. */ + readonly sourcePool: Pool; + /** The live database whose state is desired. */ + readonly desiredPool: Pool; + readonly allowDrops: boolean; + readonly debug: boolean; + readonly redactSecrets?: boolean; + readonly restrictToApplier?: boolean; + readonly schema?: readonly string[]; +} + +interface LegacyPgDeltaNextDiffResult { + readonly changes: boolean; + readonly sql: string; + readonly files: readonly LegacyPgDeltaNextRenderedFile[]; + readonly diagnostics: readonly LegacyPgDeltaNextDiagnostic[]; + readonly debug?: LegacyPgDeltaNextDebugArtifacts; +} + +type LegacyPgDeltaNextManagementScope = "database" | "cluster"; +type LegacyPgDeltaNextExportLayout = "by-object" | "ordered" | "grouped"; + +interface LegacyPgDeltaNextExportGroupingPattern { + readonly pattern: string; + readonly name: string; +} + +interface LegacyPgDeltaNextExportGrouping { + readonly mode?: "single-file" | "subdirectory"; + readonly groupPatterns?: readonly LegacyPgDeltaNextExportGroupingPattern[]; + readonly flatSchemas?: readonly string[]; + readonly autoGroupPartitions?: boolean; +} + +export interface LegacyPgDeltaNextDeclarativeExportInput { + readonly pool: Pool; + readonly scope?: LegacyPgDeltaNextManagementScope; + readonly redactSecrets?: boolean; + readonly restrictToApplier?: boolean; + readonly layout?: LegacyPgDeltaNextExportLayout; + readonly grouping?: LegacyPgDeltaNextExportGrouping; + readonly defaultOwner?: string | null; + readonly onWarning?: (message: string) => void; + readonly schema?: readonly string[]; + readonly formatOptions?: string; +} + +export interface LegacyPgDeltaNextExportManifest { + readonly redactSecrets: boolean; + readonly scope: LegacyPgDeltaNextManagementScope; + readonly profile?: string; + readonly baselineDigest?: string; + readonly defaultOwner?: string | null; + readonly files?: readonly string[]; +} + +interface LegacyPgDeltaNextDeclarativeExportResult { + readonly files: readonly LegacyPgDeltaNextSqlFile[]; + readonly manifest: LegacyPgDeltaNextExportManifest; + readonly diagnostics: readonly LegacyPgDeltaNextDiagnostic[]; +} + +export interface LegacyPgDeltaNextDeclarativeManifestInput { + readonly redactSecrets?: boolean; + readonly profile?: string; + readonly scope?: LegacyPgDeltaNextManagementScope; + readonly baselineDigest?: string; + readonly defaultOwner?: string | null; + readonly files?: readonly string[]; +} + +export interface LegacyPgDeltaNextDeclarativePlanInput { + readonly targetPool: Pool; + readonly shadowPool: Pool; + readonly files: readonly LegacyPgDeltaNextSqlFile[]; + readonly allowDrops: boolean; + readonly debug: boolean; + readonly scope?: LegacyPgDeltaNextManagementScope; + readonly manifest?: LegacyPgDeltaNextDeclarativeManifestInput; + readonly redactSecrets?: boolean; + readonly skipClusterDdl?: boolean; + readonly isolatedShadow?: boolean; + readonly seedAssumedSchemas?: boolean; + readonly restrictToApplier?: boolean; + readonly strictFunctionBodies?: boolean; + /** Defaults to true, preserving pg-topo statement-level reorder support. */ + readonly reorder?: boolean; + readonly onWarning?: (message: string) => void; + readonly schema?: readonly string[]; +} + +interface LegacyPgDeltaNextSkippedStatement { + readonly file: string; + readonly statement: string; +} + +interface LegacyPgDeltaNextDeclarativePlanResult { + readonly changes: boolean; + readonly sql: string; + readonly files: readonly LegacyPgDeltaNextRenderedFile[]; + readonly diagnostics: readonly LegacyPgDeltaNextDiagnostic[]; + readonly skipped: readonly LegacyPgDeltaNextSkippedStatement[]; + readonly debug?: LegacyPgDeltaNextDebugArtifacts; +} + +export interface LegacyPgDeltaNextSnapshotCaptureInput { + readonly pool: Pool; + readonly redactSecrets?: boolean; + readonly statementTimeoutMs?: number; +} + +interface LegacyPgDeltaNextSnapshotCaptureResult { + readonly generation: "v2"; + readonly snapshot: string; + readonly pgVersion: string; + readonly diagnostics: readonly LegacyPgDeltaNextDiagnostic[]; +} + +export class LegacyPgDeltaNextError extends Data.TaggedError("LegacyPgDeltaNextError")<{ + readonly operation: LegacyPgDeltaNextOperation; + readonly message: string; + readonly cause: unknown; +}> {} + +export interface LegacyPgDeltaNextAdapterShape { + readonly diff: ( + input: LegacyPgDeltaNextDiffInput, + ) => Effect.Effect; + readonly exportDeclarativeSchema: ( + input: LegacyPgDeltaNextDeclarativeExportInput, + ) => Effect.Effect; + readonly planDeclarativeSchema: ( + input: LegacyPgDeltaNextDeclarativePlanInput, + ) => Effect.Effect; + readonly captureSnapshot: ( + input: LegacyPgDeltaNextSnapshotCaptureInput, + ) => Effect.Effect; +} + +export class LegacyPgDeltaNextAdapter extends Context.Service< + LegacyPgDeltaNextAdapter, + LegacyPgDeltaNextAdapterShape +>()("supabase/legacy/PgDeltaNextAdapter") {} diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts new file mode 100644 index 0000000000..3ad7d9d9a2 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -0,0 +1,520 @@ +import { it } from "@effect/vitest"; +import { Effect } from "effect"; +import { Pool } from "pg"; +import { describe, expect } from "vitest"; + +import { + legacyPgDeltaNextAdapterLayer, + legacyPgDeltaNextAdapterLayerFromLibraries, + legacyFilterPgDeltaNextPlatformParameterAclDiagnostics, + legacyPgDeltaNextProfile, + legacyPgDeltaNextUserOwnedParameterAcls, + type LegacyPgDeltaNextLibraries, +} from "./legacy-pgdelta-next-adapter.layer.ts"; +import { + LegacyPgDeltaNextAdapter, + LegacyPgDeltaNextError, +} from "./legacy-pgdelta-next-adapter.service.ts"; + +interface FakeFactBase { + readonly id: string; +} + +interface FakePlanOptions { + readonly managedView: string; +} + +interface FakePlan { + readonly source: string; + readonly desired: string; +} + +interface FakeSubject { + readonly id: string; +} + +function fakeDiagnostic(code: string, subject: string) { + return { + code, + severity: "warning" as const, + subject: { id: subject }, + message: `${code} message`, + context: { detail: code }, + }; +} + +function setupLibraries(sourcePool: Pool, desiredPool: Pool) { + const state = { + resolveCalls: [] as Array<{ + pool: Pool; + options: { + restrictToApplier?: boolean; + redactSecrets?: boolean; + skipBaseline?: boolean; + }; + schema?: readonly string[]; + }>, + extractCalls: [] as Array<{ pool: Pool; options: object | undefined }>, + planCalls: [] as Array<{ + source: FakeFactBase; + desired: FakeFactBase; + options: FakePlanOptions & { redactSecrets: boolean }; + }>, + renderAllowDrops: [] as boolean[], + exportInputs: [] as object[], + declarativeInputs: [] as object[], + snapshotMetadata: [] as object[], + serializedPlans: [] as FakePlan[], + renderChanges: true, + }; + + const extract = async ( + pool: Pool, + options?: { redactSecrets?: boolean; statementTimeoutMs?: number }, + ) => { + state.extractCalls.push({ pool, options }); + const source = pool === sourcePool; + if (!source && pool !== desiredPool) { + throw new Error("unexpected pool passed to fake extractor"); + } + return { + factBase: { id: source ? "source-facts" : "desired-facts" }, + pgVersion: source ? "15.9" : "17.6", + diagnostics: [ + fakeDiagnostic(source ? "source-warning" : "desired-warning", source ? "s" : "d"), + ], + }; + }; + + const libraries: LegacyPgDeltaNextLibraries< + FakeFactBase, + FakePlanOptions, + FakePlan, + FakeSubject + > = { + resolveProfile: async (pool, options, schema) => { + state.resolveCalls.push({ pool, options, ...(schema !== undefined ? { schema } : {}) }); + return { + id: "supabase", + planOptions: { managedView: "shared-profile-options" }, + extract, + }; + }, + plan: (source, desired, options) => { + state.planCalls.push({ source, desired, options }); + return { source: source.id, desired: desired.id }; + }, + renderPlanFiles: (generatedPlan, options) => { + state.renderAllowDrops.push(options.allowDrops); + if (!state.renderChanges) return { changes: false, files: [] }; + return { + changes: true, + files: [ + { + suffix: "_1", + contents: `begin ${generatedPlan.source};\n`, + transactional: true, + actionCount: 2, + }, + { + suffix: "_2", + contents: `alter ${generatedPlan.desired};\n`, + transactional: false, + actionCount: 1, + }, + ], + }; + }, + buildSchemaExport: async (_pool, input) => { + state.exportInputs.push(input); + return { + files: [{ name: "schemas/public/tables/items.sql", sql: "create table items();" }], + diagnostics: [fakeDiagnostic("export-warning", "export")], + manifest: { + redactSecrets: true, + scope: "database", + profile: "supabase", + defaultOwner: "postgres", + }, + }; + }, + planSchemaFiles: async (_targetPool, _shadowPool, _files, input) => { + state.declarativeInputs.push(input); + return { + plan: { source: "target-facts", desired: "loaded-files" }, + loadDiagnostics: [fakeDiagnostic("load-warning", "load")], + targetDiagnostics: [fakeDiagnostic("target-warning", "target")], + skipped: [{ file: "roles.sql", stmt: "create role ignored" }], + }; + }, + serializeSnapshot: (factBase, metadata) => { + state.snapshotMetadata.push(metadata); + return JSON.stringify({ factBase: factBase.id, metadata }); + }, + serializePlan: (generatedPlan) => { + state.serializedPlans.push(generatedPlan); + return JSON.stringify(generatedPlan); + }, + encodeSubject: (subject) => `subject:${subject.id}`, + }; + + return { + state, + layer: legacyPgDeltaNextAdapterLayerFromLibraries(libraries), + }; +} + +describe("LegacyPgDeltaNextAdapter", () => { + it("filters platform parameter ACL coverage without hiding user-owned ACLs", () => { + const diagnostics = [ + { + origin: "declarativeLoad" as const, + code: "unmodeled_kind", + severity: "warning" as const, + message: "2 unmodeled parameter ACLs", + context: { + kind: "parameter ACL", + count: 2, + samples: ["log_min_messages", "work_mem"], + }, + }, + { + origin: "declarativeLoad" as const, + code: "unsupported_extension", + severity: "warning" as const, + message: "extension is externally managed", + }, + ]; + + expect(legacyFilterPgDeltaNextPlatformParameterAclDiagnostics(diagnostics, [])).toEqual([ + diagnostics[1], + ]); + expect( + legacyFilterPgDeltaNextPlatformParameterAclDiagnostics(diagnostics, ["work_mem"]), + ).toEqual([ + { + ...diagnostics[0], + message: + '1 unmodeled "parameter ACL" object not managed by this engine (e.g. work_mem) — v1 detects but does not model this kind', + context: { kind: "parameter ACL", count: 1, samples: ["work_mem"] }, + }, + diagnostics[1], + ]); + }); + + it("recognizes only the exact Supabase platform parameter grant tuples", () => { + expect( + legacyPgDeltaNextUserOwnedParameterAcls([ + { name: "log_min_messages", grantee: "supabase_admin", privilege: "SET" }, + { name: "log_min_messages", grantee: "app_user", privilege: "SET" }, + { name: "work_mem", grantee: "supabase_realtime_admin", privilege: "SET" }, + { name: "work_mem", grantee: "app_user", privilege: "SET" }, + ]), + ).toEqual(["log_min_messages", "work_mem"]); + expect( + legacyPgDeltaNextUserOwnedParameterAcls([ + { name: "log_min_messages", grantee: "supabase_admin", privilege: "ALTER SYSTEM" }, + { name: "log_min_messages", grantee: "supabase_admin", privilege: "SET" }, + { name: "log_min_messages", grantee: "supabase_realtime_admin", privilege: "SET" }, + ]), + ).toEqual([]); + expect( + legacyPgDeltaNextUserOwnedParameterAcls([ + { name: "log_min_messages", grantee: "supabase_realtime_admin", privilege: "ALTER SYSTEM" }, + ]), + ).toEqual(["log_min_messages"]); + }); + + it("composes schema exclusions ahead of the Supabase managed-view policy", () => { + const profile = legacyPgDeltaNextProfile(["public", "tenant"]); + expect(profile.id).toBe("supabase"); + expect(profile.policy?.filter).toEqual([ + { + match: { all: [{ schema: "*" }, { not: { schema: ["public", "tenant"] } }] }, + action: "exclude", + }, + { + match: { + all: [{ kind: "schema" }, { not: { name: ["public", "tenant"] } }], + }, + action: "exclude", + }, + { + match: { + all: [{ target: { schema: "*" } }, { not: { target: { schema: ["public", "tenant"] } } }], + }, + action: "exclude", + }, + ]); + expect(profile.policy?.extends).toHaveLength(1); + }); + + it.effect("constructs the real adapter from supported public pg-delta subpaths", () => + Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + expect(adapter.diff).toBeTypeOf("function"); + expect(adapter.exportDeclarativeSchema).toBeTypeOf("function"); + expect(adapter.planDeclarativeSchema).toBeTypeOf("function"); + expect(adapter.captureSnapshot).toBeTypeOf("function"); + }).pipe(Effect.provide(legacyPgDeltaNextAdapterLayer)), + ); + + it.effect( + "resolves one shared profile for a pool-to-pool diff and emits structured debug data", + () => { + const sourcePool = new Pool(); + const desiredPool = new Pool(); + const { layer, state } = setupLibraries(sourcePool, desiredPool); + + return Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + const result = yield* adapter.diff({ + sourcePool, + desiredPool, + allowDrops: true, + debug: true, + redactSecrets: false, + restrictToApplier: true, + schema: ["public"], + }); + + expect(state.resolveCalls).toEqual([ + { + pool: sourcePool, + options: { redactSecrets: false, restrictToApplier: true }, + schema: ["public"], + }, + ]); + expect(state.extractCalls).toEqual([ + { pool: sourcePool, options: { redactSecrets: false } }, + { pool: desiredPool, options: { redactSecrets: false } }, + ]); + expect(state.planCalls).toEqual([ + { + source: { id: "source-facts" }, + desired: { id: "desired-facts" }, + options: { redactSecrets: false, managedView: "shared-profile-options" }, + }, + ]); + expect(result.files).toEqual([ + { + sequence: 1, + suffix: "_1", + sql: "begin source-facts;\n", + transactional: true, + actionCount: 2, + }, + { + sequence: 2, + suffix: "_2", + sql: "alter desired-facts;\n", + transactional: false, + actionCount: 1, + }, + ]); + expect(result.sql).toBe("begin source-facts;\n\n\nalter desired-facts;\n"); + expect(result.diagnostics).toEqual([ + { + origin: "source", + code: "source-warning", + severity: "warning", + subject: "subject:s", + message: "source-warning message", + context: { detail: "source-warning" }, + }, + { + origin: "desired", + code: "desired-warning", + severity: "warning", + subject: "subject:d", + message: "desired-warning message", + context: { detail: "desired-warning" }, + }, + ]); + expect(result.debug).toEqual({ + sourceSnapshot: expect.stringContaining("source-facts"), + desiredSnapshot: expect.stringContaining("desired-facts"), + plan: JSON.stringify({ source: "source-facts", desired: "desired-facts" }), + }); + expect(state.snapshotMetadata).toEqual([ + { pgVersion: "15.9", redactSecrets: false, profile: "supabase" }, + { pgVersion: "17.6", redactSecrets: false, profile: "supabase" }, + ]); + expect(sourcePool.ending).toBe(false); + expect(sourcePool.ended).toBe(false); + expect(desiredPool.ending).toBe(false); + expect(desiredPool.ended).toBe(false); + yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); + }).pipe(Effect.provide(layer)); + }, + ); + + it.effect("preserves a no-change result without creating debug artifacts", () => { + const sourcePool = new Pool(); + const desiredPool = new Pool(); + const { layer, state } = setupLibraries(sourcePool, desiredPool); + state.renderChanges = false; + + return Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + const result = yield* adapter.diff({ + sourcePool, + desiredPool, + allowDrops: false, + debug: false, + }); + expect(result.changes).toBe(false); + expect(result.sql).toBe(""); + expect(result.files).toEqual([]); + expect(result.debug).toBeUndefined(); + expect(state.snapshotMetadata).toEqual([]); + expect(state.renderAllowDrops).toEqual([false]); + yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); + }).pipe(Effect.provide(layer)); + }); + + it.effect( + "normalizes declarative export and planning results with reorder enabled by default", + () => { + const targetPool = new Pool(); + const shadowPool = new Pool(); + const { layer, state } = setupLibraries(targetPool, shadowPool); + + return Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + const exported = yield* adapter.exportDeclarativeSchema({ + pool: targetPool, + layout: "grouped", + restrictToApplier: true, + formatOptions: + '{"keywordCase":"lower","commaStyle":"leading","indent":4,"maxWidth":100,"alignColumns":true,"alignKeyValues":false,"preserveRoutineBodies":true,"preserveViewBodies":false,"preserveRuleBodies":true,"ignored":"value"}', + }); + expect(exported.files).toEqual([ + { name: "schemas/public/tables/items.sql", sql: "create table items();" }, + ]); + expect(exported.manifest).toEqual({ + redactSecrets: true, + scope: "database", + profile: "supabase", + defaultOwner: "postgres", + files: ["schemas/public/tables/items.sql"], + }); + expect(exported.diagnostics[0]).toMatchObject({ + origin: "export", + subject: "subject:export", + }); + expect(state.exportInputs).toHaveLength(1); + expect(state.exportInputs[0]).toMatchObject({ + layout: "grouped", + resolveOptions: { restrictToApplier: true }, + format: { + keywordCase: "lower", + commaStyle: "leading", + indent: 4, + maxWidth: 100, + alignColumns: true, + alignKeyValues: false, + preserveRoutineBodies: true, + preserveViewBodies: false, + preserveRuleBodies: true, + }, + }); + expect(state.exportInputs[0]).not.toHaveProperty("formatOptions"); + + const planned = yield* adapter.planDeclarativeSchema({ + targetPool, + shadowPool, + files: exported.files, + allowDrops: true, + debug: true, + isolatedShadow: true, + seedAssumedSchemas: true, + }); + expect(state.declarativeInputs).toHaveLength(1); + expect(state.declarativeInputs[0]).toMatchObject({ + reorder: true, + seedAssumedSchemas: true, + }); + expect(planned.diagnostics.map((diagnostic) => diagnostic.origin)).toEqual([ + "declarativeLoad", + "declarativeTarget", + ]); + expect(planned.skipped).toEqual([{ file: "roles.sql", statement: "create role ignored" }]); + expect(planned.debug).toEqual({ + plan: JSON.stringify({ source: "target-facts", desired: "loaded-files" }), + }); + expect(state.renderAllowDrops).toEqual([true]); + yield* Effect.promise(() => Promise.all([targetPool.end(), shadowPool.end()])); + }).pipe(Effect.provide(layer)); + }, + ); + + it.effect("captures a v2 snapshot with a single baseline-free profile resolution", () => { + const pool = new Pool(); + const unusedDesiredPool = new Pool(); + const { layer, state } = setupLibraries(pool, unusedDesiredPool); + + return Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + const result = yield* adapter.captureSnapshot({ + pool, + statementTimeoutMs: 4_000, + }); + expect(result.generation).toBe("v2"); + expect(result.pgVersion).toBe("15.9"); + expect(result.snapshot).toContain("source-facts"); + expect(state.resolveCalls).toEqual([ + { + pool, + options: { redactSecrets: true, skipBaseline: true }, + }, + ]); + expect(state.extractCalls).toEqual([ + { + pool, + options: { redactSecrets: true, statementTimeoutMs: 4_000 }, + }, + ]); + yield* Effect.promise(() => Promise.all([pool.end(), unusedDesiredPool.end()])); + }).pipe(Effect.provide(layer)); + }); + + it.effect("maps library rejections to an actionable typed error", () => { + const sourcePool = new Pool(); + const desiredPool = new Pool(); + const cause = new Error("connection refused for desired database"); + const failingLayer = legacyPgDeltaNextAdapterLayerFromLibraries({ + resolveProfile: async () => { + throw cause; + }, + plan: () => ({ source: "unused", desired: "unused" }), + renderPlanFiles: () => ({ changes: false, files: [] }), + buildSchemaExport: async () => ({ + files: [], + diagnostics: [], + manifest: { redactSecrets: true, scope: "database" }, + }), + planSchemaFiles: async () => ({ + plan: { source: "unused", desired: "unused" }, + loadDiagnostics: [], + targetDiagnostics: [], + skipped: [], + }), + serializeSnapshot: () => "unused", + serializePlan: () => "unused", + encodeSubject: (subject: string) => subject, + }); + + return Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + const error = yield* adapter + .diff({ sourcePool, desiredPool, allowDrops: false, debug: false }) + .pipe(Effect.flip); + expect(error).toBeInstanceOf(LegacyPgDeltaNextError); + expect(error.operation).toBe("diff"); + expect(error.message).toBe("Database diff failed: connection refused for desired database"); + expect(error.cause).toBe(cause); + yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); + }).pipe(Effect.provide(failingLayer)); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts new file mode 100644 index 0000000000..d949442dea --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts @@ -0,0 +1,81 @@ +import { Effect, type FileSystem, type Path } from "effect"; + +import { legacyPgDeltaTempPath } from "./legacy-pgdelta.cache.ts"; +import type { + LegacyPgDeltaNextDiagnostic, + LegacyPgDeltaNextOperation, +} from "./legacy-pgdelta-next-adapter.service.ts"; + +export interface LegacyPgDeltaNextDebugArtifacts { + readonly sourceSnapshot?: string; + readonly desiredSnapshot?: string; + readonly plan?: string; + readonly diagnostics?: ReadonlyArray; +} + +/** Explicit cache/artifact generation for the bundled pg-delta implementation. */ +export function legacyPgDeltaNextTempPath(path: Path.Path, workdir: string): string { + return path.join(legacyPgDeltaTempPath(path, workdir), "v2"); +} + +/** Millisecond-resolution id so multiple operations in one command do not collide. */ +export function legacyFormatPgDeltaNextDebugId( + millis: number, + operation: LegacyPgDeltaNextOperation, +): string { + const digits = new Date(millis).toISOString().replace(/\D/gu, "").slice(0, 17); + return `${digits.slice(0, 8)}-${digits.slice(8, 14)}-${digits.slice(14)}-${operation}`; +} + +interface LegacyPgDeltaNextArtifactMetadata { + readonly version: 1; + readonly generation: "v2"; + readonly implementation: "next"; + readonly operation: LegacyPgDeltaNextOperation; + readonly cacheReusable: false; + readonly files: ReadonlyArray; +} + +/** + * Writes bundled-engine debug data below the v2 generation. These files are + * diagnostics only: they are never considered catalog-cache inputs. + */ +export const legacySavePgDeltaNextDebugArtifacts = Effect.fnUntraced(function* ( + fs: FileSystem.FileSystem, + path: Path.Path, + workdir: string, + id: string, + operation: LegacyPgDeltaNextOperation, + artifacts: LegacyPgDeltaNextDebugArtifacts, +) { + const debugDir = path.join(legacyPgDeltaNextTempPath(path, workdir), "debug", id); + yield* fs.makeDirectory(debugDir, { recursive: true }); + + const files: Array = []; + const write = Effect.fnUntraced(function* (name: string, contents: string | undefined) { + if (contents === undefined || contents.length === 0) return; + yield* fs.writeFileString(path.join(debugDir, name), contents); + files.push(name); + }); + + yield* write("source-snapshot.json", artifacts.sourceSnapshot); + yield* write("desired-snapshot.json", artifacts.desiredSnapshot); + yield* write("plan.json", artifacts.plan); + if (artifacts.diagnostics !== undefined) { + yield* write("diagnostics.json", `${JSON.stringify(artifacts.diagnostics, null, 2)}\n`); + } + + const metadata: LegacyPgDeltaNextArtifactMetadata = { + version: 1, + generation: "v2", + implementation: "next", + operation, + cacheReusable: false, + files: [...files].sort(), + }; + yield* fs.writeFileString( + path.join(debugDir, "metadata.json"), + `${JSON.stringify(metadata, null, 2)}\n`, + ); + return debugDir; +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts new file mode 100644 index 0000000000..3330ed35fd --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts @@ -0,0 +1,74 @@ +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { BunServices } from "@effect/platform-bun"; +import { describe, expect, it } from "@effect/vitest"; +import { Effect, FileSystem, Path } from "effect"; + +import { + legacyFormatPgDeltaNextDebugId, + legacyPgDeltaNextTempPath, + legacySavePgDeltaNextDebugArtifacts, +} from "./legacy-pgdelta-next-artifacts.ts"; +import { legacyPgDeltaTempPath } from "./legacy-pgdelta.cache.ts"; + +describe("pg-delta next artifact generation", () => { + it.effect("isolates v2 artifacts from legacy catalog paths", () => + Effect.gen(function* () { + const path = yield* Path.Path; + expect(legacyPgDeltaTempPath(path, "/project")).toBe( + join("/project", "supabase", ".temp", "pgdelta"), + ); + expect(legacyPgDeltaNextTempPath(path, "/project")).toBe( + join("/project", "supabase", ".temp", "pgdelta", "v2"), + ); + }).pipe(Effect.provide(BunServices.layer)), + ); + + it("uses millisecond-resolution, operation-qualified debug ids", () => { + expect(legacyFormatPgDeltaNextDebugId(Date.UTC(2024, 0, 2, 3, 4, 5, 678), "diff")).toBe( + "20240102-030405-678-diff", + ); + }); + + it.effect("writes structured non-cache artifacts and metadata under v2", () => { + const root = mkdtempSync(join(tmpdir(), "pgdelta-next-artifacts-")); + return Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const debugDir = yield* legacySavePgDeltaNextDebugArtifacts( + fs, + path, + root, + "20240102-030405-678-diff", + "diff", + { + sourceSnapshot: '{"source":true}\n', + desiredSnapshot: '{"desired":true}\n', + plan: '{"plan":true}\n', + diagnostics: [ + { origin: "source", code: "PG001", severity: "warning", message: "warning" }, + ], + }, + ); + + expect(debugDir).toBe( + join(root, "supabase", ".temp", "pgdelta", "v2", "debug", "20240102-030405-678-diff"), + ); + expect(JSON.parse(readFileSync(join(debugDir, "metadata.json"), "utf8"))).toEqual({ + version: 1, + generation: "v2", + implementation: "next", + operation: "diff", + cacheReusable: false, + files: ["desired-snapshot.json", "diagnostics.json", "plan.json", "source-snapshot.json"], + }); + expect(JSON.parse(readFileSync(join(debugDir, "diagnostics.json"), "utf8"))).toEqual([ + { origin: "source", code: "PG001", severity: "warning", message: "warning" }, + ]); + }).pipe( + Effect.provide(BunServices.layer), + Effect.ensuring(Effect.sync(() => rmSync(root, { recursive: true, force: true }))), + ); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts new file mode 100644 index 0000000000..8eef84318d --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts @@ -0,0 +1,30 @@ +import type { + LegacyPgDeltaNextDiagnostic, + LegacyPgDeltaNextOperation, +} from "./legacy-pgdelta-next-adapter.service.ts"; + +const coverageDiagnosticCodes = new Set(["unmodeled_kind", "unresolved_security_label"]); + +export function legacyPgDeltaNextBlockingDiagnostic( + diagnostics: readonly LegacyPgDeltaNextDiagnostic[], +): LegacyPgDeltaNextDiagnostic | undefined { + return diagnostics.find( + (diagnostic) => diagnostic.severity === "error" || coverageDiagnosticCodes.has(diagnostic.code), + ); +} + +export function legacyPgDeltaNextBlockingDiagnosticMessage( + operation: LegacyPgDeltaNextOperation, + diagnostic: LegacyPgDeltaNextDiagnostic, +): string { + const action = + operation === "declarativeExport" + ? "export the declarative schema" + : operation === "declarativePlan" + ? "emit the declarative migration plan" + : operation === "snapshotCapture" + ? "capture the database snapshot" + : "emit the database diff"; + const subject = diagnostic.subject ?? "unknown"; + return `pg-delta next refused to ${action}: origin=${diagnostic.origin} code=${diagnostic.code} subject=${subject} message=${diagnostic.message}`; +} diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts new file mode 100644 index 0000000000..f0cfe77a79 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts @@ -0,0 +1,58 @@ +import { describe, expect, it } from "vitest"; + +import { + legacyPgDeltaNextBlockingDiagnostic, + legacyPgDeltaNextBlockingDiagnosticMessage, +} from "./legacy-pgdelta-next-diagnostics.ts"; + +describe("pg-delta next diagnostic coverage policy", () => { + it("blocks errors and strict coverage gaps while allowing ordinary warnings", () => { + expect( + legacyPgDeltaNextBlockingDiagnostic([ + { + origin: "source", + code: "unsupported_extension", + severity: "warning", + message: "extension is managed externally", + }, + ]), + ).toBeUndefined(); + + expect( + legacyPgDeltaNextBlockingDiagnostic([ + { + origin: "desired", + code: "unmodeled_kind", + severity: "warning", + subject: "object:public.unsupported", + message: "object kind is not modeled", + }, + ]), + ).toMatchObject({ code: "unmodeled_kind" }); + + expect( + legacyPgDeltaNextBlockingDiagnostic([ + { + origin: "export", + code: "extraction_failed", + severity: "error", + message: "catalog query failed", + }, + ]), + ).toMatchObject({ code: "extraction_failed" }); + }); + + it("renders the refused action and complete diagnostic identity", () => { + expect( + legacyPgDeltaNextBlockingDiagnosticMessage("declarativePlan", { + origin: "declarativeLoad", + code: "unresolved_security_label", + severity: "info", + subject: "table:public.accounts", + message: "security label provider was not resolved", + }), + ).toBe( + "pg-delta next refused to emit the declarative migration plan: origin=declarativeLoad code=unresolved_security_label subject=table:public.accounts message=security label provider was not resolved", + ); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-flag.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-flag.ts new file mode 100644 index 0000000000..39f4729dfc --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-flag.ts @@ -0,0 +1,22 @@ +export type LegacyPgDeltaImplementation = "next" | "legacy"; + +/** + * Resolves the pg-delta implementation rollout flag from one raw environment + * value. Defaults to the next implementation when unset or not an explicit + * false; only known false spellings select the legacy implementation. + * + * The caller owns reading `process.env`, allowing the strategy boundary to + * resolve the selection exactly once per command invocation. + */ +export function legacyResolvePgDeltaImplementation( + raw: string | undefined, +): LegacyPgDeltaImplementation { + switch (raw?.toLowerCase()) { + case "0": + case "f": + case "false": + return "legacy"; + default: + return "next"; + } +} diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-flag.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-flag.unit.test.ts new file mode 100644 index 0000000000..1768456b35 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-flag.unit.test.ts @@ -0,0 +1,23 @@ +import { describe, expect, it } from "vitest"; + +import { legacyResolvePgDeltaImplementation } from "./legacy-pgdelta-next-flag.ts"; + +describe("legacyResolvePgDeltaImplementation", () => { + it("defaults to the next implementation when unset", () => { + expect(legacyResolvePgDeltaImplementation(undefined)).toBe("next"); + }); + + it.each(["1", "t", "TRUE", "true", "True", "yes", "on", "", "garbage"])( + "selects the next implementation for %j", + (raw) => { + expect(legacyResolvePgDeltaImplementation(raw)).toBe("next"); + }, + ); + + it.each(["0", "f", "F", "FALSE", "false", "False"])( + "selects the legacy implementation for %s", + (raw) => { + expect(legacyResolvePgDeltaImplementation(raw)).toBe("legacy"); + }, + ); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts new file mode 100644 index 0000000000..aaf8ae08c1 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -0,0 +1,54 @@ +import { Effect, Layer } from "effect"; + +import { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; +import { + LegacyPgDeltaNextShadow, + type LegacyPgDeltaNextShadowDatabases, +} from "./legacy-pgdelta-next-shadow.service.ts"; +import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; + +/** + * Scoped next-engine shadow orchestration over the narrow Go `db __shadow` + * seam. Go creates the migrated target and a dedicated empty same-cluster + * scratch database; declarative SQL remains wholly owned by the TypeScript + * pg-delta next adapter and its `planSchemaFiles` operation. + */ +export const legacyPgDeltaNextShadowLayer = Layer.effect( + LegacyPgDeltaNextShadow, + Effect.gen(function* () { + const seam = yield* LegacyDeclarativeSeam; + + return LegacyPgDeltaNextShadow.of({ + provision: ({ schema, projectRef }) => + Effect.gen(function* () { + // Register cleanup immediately after Go returns the container. URL + // validation happens only after acquireRelease has installed the + // finalizer, so even malformed seam output cannot leak the shadow. + const shadow = yield* Effect.acquireRelease( + seam.provisionShadow({ + mode: "pgdelta-next", + targetLocal: false, + usePgDelta: false, + schema, + ...(projectRef !== undefined ? { projectRef } : {}), + }), + ({ container }) => seam.removeShadowContainer(container).pipe(Effect.ignoreCause), + ); + + if (shadow.targetUrlOverride === undefined) { + return yield* Effect.fail( + new LegacyDeclarativeShadowDbError({ + message: + "failed to provision the pg-delta next shadow database: missing declarative scratch URL.", + }), + ); + } + + return { + migrationsUrl: shadow.sourceUrl, + scratchUrl: shadow.targetUrlOverride, + } satisfies LegacyPgDeltaNextShadowDatabases; + }), + }); + }), +); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts new file mode 100644 index 0000000000..1c34cc8fc8 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts @@ -0,0 +1,32 @@ +import { Context, type Effect, type Scope } from "effect"; + +import type { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; + +/** The two live databases needed to plan with the bundled pg-delta next engine. */ +export interface LegacyPgDeltaNextShadowDatabases { + /** Platform baseline with the project's local migrations applied. */ + readonly migrationsUrl: string; + /** Empty same-cluster database owned by `planSchemaFiles` while loading desired SQL. */ + readonly scratchUrl: string; +} + +interface LegacyPgDeltaNextShadowShape { + /** + * Provisions the next-engine shadow container and owns it for the current + * Effect scope. The container is removed when that scope closes, including + * when URL validation or the caller fails. + */ + readonly provision: (opts: { + readonly schema: ReadonlyArray; + readonly projectRef?: string; + }) => Effect.Effect< + LegacyPgDeltaNextShadowDatabases, + LegacyDeclarativeShadowDbError, + Scope.Scope + >; +} + +export class LegacyPgDeltaNextShadow extends Context.Service< + LegacyPgDeltaNextShadow, + LegacyPgDeltaNextShadowShape +>()("supabase/legacy/PgDeltaNextShadow") {} diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts new file mode 100644 index 0000000000..ef201d34d3 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts @@ -0,0 +1,154 @@ +import { describe, expect, it } from "@effect/vitest"; +import { Data, Effect, Layer } from "effect"; + +import { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; +import { legacyPgDeltaNextShadowLayer } from "./legacy-pgdelta-next-shadow.layer.ts"; +import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; +import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; + +class PrimaryFailure extends Data.TaggedError("PrimaryFailure")<{ + readonly message: string; +}> {} + +function setup( + opts: { + readonly sourceUrl?: string; + readonly scratchUrl?: string; + readonly cleanupDefect?: boolean; + } = {}, +) { + const state = { + provisionCalls: [] as object[], + removedContainers: [] as string[], + legacyMethodCalls: [] as string[], + }; + const seamLayer = Layer.succeed( + LegacyDeclarativeSeam, + LegacyDeclarativeSeam.of({ + exportCatalog: () => + Effect.sync(() => { + state.legacyMethodCalls.push("exportCatalog"); + return "catalog.json"; + }), + execInherit: () => + Effect.sync(() => { + state.legacyMethodCalls.push("execInherit"); + return 0; + }), + ensureLocalDatabaseStarted: () => + Effect.sync(() => { + state.legacyMethodCalls.push("ensureLocalDatabaseStarted"); + }), + ensureLocalPostgresImageCurrent: () => + Effect.sync(() => { + state.legacyMethodCalls.push("ensureLocalPostgresImageCurrent"); + }), + provisionShadow: (input) => + Effect.sync(() => { + state.provisionCalls.push(input); + return { + container: "next-shadow-container", + sourceUrl: opts.sourceUrl ?? "postgresql://postgres@localhost:55432/postgres", + targetUrlOverride: opts.scratchUrl, + }; + }), + removeShadowContainer: (container) => + Effect.gen(function* () { + state.removedContainers.push(container); + if (opts.cleanupDefect === true) { + return yield* Effect.die("cleanup failed"); + } + }), + }), + ); + + return { + state, + layer: legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seamLayer)), + }; +} + +describe("LegacyPgDeltaNextShadow", () => { + it.effect("provisions the exact next mode and exposes the migrated and scratch URLs", () => { + const { layer, state } = setup({ + scratchUrl: "postgresql://postgres@localhost:55432/pgdelta_declarative", + }); + + return Effect.gen(function* () { + const databases = yield* Effect.scoped( + Effect.gen(function* () { + const shadow = yield* LegacyPgDeltaNextShadow; + const acquired = yield* shadow.provision({ + schema: ["public", "extensions"], + projectRef: "linked-project", + }); + expect(state.removedContainers).toEqual([]); + return acquired; + }), + ); + + expect(databases).toEqual({ + migrationsUrl: "postgresql://postgres@localhost:55432/postgres", + scratchUrl: "postgresql://postgres@localhost:55432/pgdelta_declarative", + }); + expect(Object.keys(databases)).toEqual(["migrationsUrl", "scratchUrl"]); + expect(state.provisionCalls).toEqual([ + { + mode: "pgdelta-next", + targetLocal: false, + usePgDelta: false, + schema: ["public", "extensions"], + projectRef: "linked-project", + }, + ]); + expect(state.removedContainers).toEqual(["next-shadow-container"]); + expect(state.legacyMethodCalls).toEqual([]); + }).pipe(Effect.provide(layer)); + }); + + it.effect("cleans up when the caller fails and never lets cleanup mask that failure", () => { + const { layer, state } = setup({ + scratchUrl: "postgresql://postgres@localhost:55432/pgdelta_declarative", + cleanupDefect: true, + }); + const primary = new PrimaryFailure({ message: "caller failed" }); + + return Effect.gen(function* () { + const error = yield* Effect.scoped( + Effect.gen(function* () { + const shadow = yield* LegacyPgDeltaNextShadow; + yield* shadow.provision({ schema: [] }); + return yield* Effect.fail(primary); + }), + ).pipe(Effect.flip); + + expect(error).toEqual(primary); + expect(state.removedContainers).toEqual(["next-shadow-container"]); + }).pipe(Effect.provide(layer)); + }); + + it.effect("cleans up and fails when the declarative scratch URL is missing", () => { + const { layer, state } = setup(); + + return Effect.gen(function* () { + const error = yield* Effect.scoped( + Effect.gen(function* () { + const shadow = yield* LegacyPgDeltaNextShadow; + return yield* shadow.provision({ schema: ["public"] }); + }), + ).pipe(Effect.flip); + + expect(error).toBeInstanceOf(LegacyDeclarativeShadowDbError); + expect(error.message).toContain("missing declarative scratch URL"); + expect(state.removedContainers).toEqual(["next-shadow-container"]); + expect(state.provisionCalls).toEqual([ + { + mode: "pgdelta-next", + targetLocal: false, + usePgDelta: false, + schema: ["public"], + }, + ]); + }).pipe(Effect.provide(layer)); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts new file mode 100644 index 0000000000..c936e1a649 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts @@ -0,0 +1,448 @@ +import { + existsSync, + mkdirSync, + readdirSync, + readFileSync, + renameSync, + writeFileSync, +} from "node:fs"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterAll, beforeAll, expect, test } from "vitest"; + +import { describeDockerLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; + +const COMMAND_TIMEOUT_MS = 280_000; +const SCENARIO_TIMEOUT_MS = 900_000; +const NEXT_ENV = { + PGDELTA_DEBUG: "1", + SUPABASE_USE_PG_DELTA_NEXT: "true", +}; + +const initialDesiredSchema = `create type public.account_state as enum ('pending', 'active'); + +create table public.disposable_note ( + id bigint generated by default as identity primary key, + body text not null +); + +create view public.auth_user_emails as +select id, email +from auth.users; +`; + +const editedDesiredSchema = `create type public.account_state as enum ('pending', 'review', 'active'); + +create view public.auth_user_emails as +select id, email +from auth.users; + +create table public.review_queue ( + id bigint primary key, + state public.account_state not null default 'review' +); +`; + +function commandFailure(result: { stdout: string; stderr: string }): string { + return `stdout:\n${result.stdout}\nstderr:\n${result.stderr}`; +} + +function migrationFiles(projectDir: string): ReadonlyArray { + const migrationsDir = path.join(projectDir, "supabase", "migrations"); + return existsSync(migrationsDir) + ? readdirSync(migrationsDir) + .filter((file) => file.endsWith(".sql")) + .sort() + : []; +} + +function debugBundleDirectories(projectDir: string): ReadonlyArray { + const debugDir = path.join(projectDir, "supabase", ".temp", "pgdelta", "v2", "debug"); + if (!existsSync(debugDir)) return []; + return readdirSync(debugDir, { withFileTypes: true }) + .filter((entry) => entry.isDirectory()) + .map((entry) => path.join(debugDir, entry.name)) + .sort(); +} + +function requireDebugBundle(projectDir: string, operation: "declarativePlan" | "diff"): string { + const bundle = debugBundleDirectories(projectDir) + .filter((dir) => path.basename(dir).endsWith(`-${operation}`)) + .at(-1); + expect(bundle, `missing ${operation} debug bundle`).toBeDefined(); + if (bundle === undefined) throw new Error(`missing ${operation} debug bundle`); + return bundle; +} + +function assertJsonFile(file: string): unknown { + expect(existsSync(file), `missing ${file}`).toBe(true); + return JSON.parse(readFileSync(file, "utf8")); +} + +function localDatabaseUrl(config: string): string { + const dbSection = config.match(/\[db\][\s\S]*?\nport\s*=\s*(\d+)/u); + expect(dbSection?.[1], "db.port missing from generated config.toml").toBeDefined(); + return `postgresql://postgres:postgres@127.0.0.1:${dbSection?.[1]}/postgres?sslmode=disable`; +} + +describeDockerLive("pg-delta next local convergence (live)", () => { + let projectDir = ""; + let desiredSchemaPath = ""; + let databaseUrl = ""; + + beforeAll(async () => { + projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-live-")); + + const init = await runSupabaseLive(["init"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(init.exitCode, commandFailure(init)).toBe(0); + + const configPath = path.join(projectDir, "supabase", "config.toml"); + const config = readFileSync(configPath, "utf8"); + expect(config).toContain("schema_paths = []"); + expect(config).toContain("[experimental.pgdelta]\nenabled = true"); + writeFileSync( + configPath, + config + .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') + .replace( + '# declarative_schema_path = "./database"', + 'declarative_schema_path = "./schemas"', + ), + ); + databaseUrl = localDatabaseUrl(config); + + const schemasDir = path.join(projectDir, "supabase", "schemas"); + mkdirSync(schemasDir, { recursive: true }); + desiredSchemaPath = path.join(schemasDir, "public.sql"); + writeFileSync(desiredSchemaPath, initialDesiredSchema); + + const start = await runSupabaseLive( + [ + "start", + "--exclude", + "studio", + "--exclude", + "logflare", + "--exclude", + "vector", + "--exclude", + "gotrue", + "--exclude", + "realtime", + "--exclude", + "storage-api", + ], + { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + ); + expect(start.exitCode, commandFailure(start)).toBe(0); + }, COMMAND_TIMEOUT_MS); + + afterAll(async () => { + if (projectDir.length === 0) return; + await runSupabaseLive(["stop", "--no-backup"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }).catch(() => undefined); + await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); + }, COMMAND_TIMEOUT_MS); + + test( + "converges declarative state across empty, destructive, enum, URL, and migrations refs", + { timeout: SCENARIO_TIMEOUT_MS }, + async () => { + expect(migrationFiles(projectDir)).toEqual([]); + + const initialDiff = await runSupabaseLive( + ["db", "diff", "--local", "--use-pg-delta", "-f", "initial_declarative"], + { cwd: projectDir, env: NEXT_ENV, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + ); + expect(initialDiff.exitCode, commandFailure(initialDiff)).toBe(0); + + const initialMigrations = migrationFiles(projectDir); + expect(initialMigrations.length).toBeGreaterThan(0); + const initialSql = initialMigrations + .map((file) => readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8")) + .join("\n"); + expect(initialSql).toContain("account_state"); + expect(initialSql).toContain("disposable_note"); + expect(initialSql).toContain("auth_user_emails"); + expect(initialSql).toContain("auth.users"); + expect(initialSql).not.toMatch( + /CREATE\s+(?:SCHEMA|TABLE)\s+(?:IF\s+NOT\s+EXISTS\s+)?["']?(?:auth|storage|realtime)["']?/iu, + ); + + const declarativeBundle = requireDebugBundle(projectDir, "declarativePlan"); + expect(assertJsonFile(path.join(declarativeBundle, "metadata.json"))).toMatchObject({ + version: 1, + generation: "v2", + implementation: "next", + operation: "declarativePlan", + cacheReusable: false, + files: ["diagnostics.json", "plan.json"], + }); + assertJsonFile(path.join(declarativeBundle, "plan.json")); + expect(Array.isArray(assertJsonFile(path.join(declarativeBundle, "diagnostics.json")))).toBe( + true, + ); + + const firstReset = await runSupabaseLive(["db", "reset", "--local", "--no-seed"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(firstReset.exitCode, commandFailure(firstReset)).toBe(0); + + const emptyAfterInitial = await runSupabaseLive(["db", "diff", "--local", "--use-pg-delta"], { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(emptyAfterInitial.exitCode, commandFailure(emptyAfterInitial)).toBe(0); + expect(emptyAfterInitial.stderr).toContain("No schema changes found"); + + writeFileSync(desiredSchemaPath, editedDesiredSchema); + const beforeEdit = new Set(migrationFiles(projectDir)); + const editedDiff = await runSupabaseLive( + ["db", "diff", "--local", "--use-pg-delta", "-f", "enum_and_drop"], + { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(editedDiff.exitCode, commandFailure(editedDiff)).toBe(0); + expect(editedDiff.stderr).toContain("Found drop statements in schema diff"); + + const editedMigrations = migrationFiles(projectDir).filter((file) => !beforeEdit.has(file)); + expect(editedMigrations.length).toBeGreaterThan(1); + const editedMigrationSql = editedMigrations.map((file) => + readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8"), + ); + const editedSql = editedMigrationSql.join("\n"); + expect(editedSql).toMatch(/ALTER\s+TYPE[\s\S]*account_state[\s\S]*ADD\s+VALUE/iu); + expect(editedSql).toMatch(/DROP\s+TABLE[\s\S]*disposable_note/iu); + expect(editedSql).toContain("review_queue"); + + const enumPush = await runSupabaseLive(["db", "push", "--local"], { + cwd: projectDir, + env: { SUPABASE_YES: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(enumPush.exitCode, commandFailure(enumPush)).toBe(0); + + const emptyAfterEdit = await runSupabaseLive(["db", "diff", "--local", "--use-pg-delta"], { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(emptyAfterEdit.exitCode, commandFailure(emptyAfterEdit)).toBe(0); + expect(emptyAfterEdit.stderr).toContain("No schema changes found"); + + const explicit = await runSupabaseLive( + ["db", "diff", "--from", "migrations", "--to", databaseUrl], + { cwd: projectDir, env: NEXT_ENV, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + ); + expect(explicit.exitCode, commandFailure(explicit)).toBe(0); + expect(explicit.stdout.trim()).toBe(""); + + const diffBundle = requireDebugBundle(projectDir, "diff"); + expect(assertJsonFile(path.join(diffBundle, "metadata.json"))).toMatchObject({ + version: 1, + generation: "v2", + implementation: "next", + operation: "diff", + cacheReusable: false, + files: ["desired-snapshot.json", "diagnostics.json", "plan.json", "source-snapshot.json"], + }); + const sourceSnapshot = readFileSync(path.join(diffBundle, "source-snapshot.json"), "utf8"); + const desiredSnapshot = readFileSync(path.join(diffBundle, "desired-snapshot.json"), "utf8"); + expect(sourceSnapshot).toContain("account_state"); + expect(desiredSnapshot).toContain("account_state"); + JSON.parse(sourceSnapshot); + JSON.parse(desiredSnapshot); + assertJsonFile(path.join(diffBundle, "plan.json")); + expect(Array.isArray(assertJsonFile(path.join(diffBundle, "diagnostics.json")))).toBe(true); + + const generated = await runSupabaseLive( + ["db", "schema", "declarative", "generate", "--local", "--overwrite"], + { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(generated.exitCode, commandFailure(generated)).toBe(0); + + const exportedFiles = readdirSync(path.join(projectDir, "supabase", "schemas"), { + recursive: true, + }) + .filter((entry): entry is string => typeof entry === "string" && entry.endsWith(".sql")) + .map((entry) => path.join(projectDir, "supabase", "schemas", entry)) + .sort(); + expect(exportedFiles.length).toBeGreaterThan(0); + expect(existsSync(path.join(projectDir, "supabase", "schemas", ".pgdelta-export.json"))).toBe( + true, + ); + + const migrationsBeforeGeneratedSync = migrationFiles(projectDir); + const emptyGeneratedSync = await runSupabaseLive( + ["db", "schema", "declarative", "sync", "--no-apply"], + { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(emptyGeneratedSync.exitCode, commandFailure(emptyGeneratedSync)).toBe(0); + expect(emptyGeneratedSync.stderr).toContain("No schema changes found"); + expect(migrationFiles(projectDir)).toEqual(migrationsBeforeGeneratedSync); + + const editedExport = exportedFiles[0]; + expect(editedExport).toBeDefined(); + if (editedExport === undefined) throw new Error("declarative export produced no SQL files"); + writeFileSync( + editedExport, + `${readFileSync(editedExport, "utf8")}\ncreate table public.phase6_synced (id bigint primary key);\n`, + ); + + const migrationsBeforeApplySync = new Set(migrationFiles(projectDir)); + const appliedSync = await runSupabaseLive( + ["db", "schema", "declarative", "sync", "--apply", "--name", "phase6_sync"], + { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(appliedSync.exitCode, commandFailure(appliedSync)).toBe(0); + expect(appliedSync.stderr).toContain("Migration applied successfully"); + const appliedSyncMigrations = migrationFiles(projectDir).filter( + (file) => !migrationsBeforeApplySync.has(file), + ); + expect(appliedSyncMigrations.length).toBeGreaterThan(0); + expect( + appliedSyncMigrations + .map((file) => + readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8"), + ) + .join("\n"), + ).toContain("phase6_synced"); + + const emptyAppliedSync = await runSupabaseLive( + ["db", "schema", "declarative", "sync", "--no-apply"], + { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(emptyAppliedSync.exitCode, commandFailure(emptyAppliedSync)).toBe(0); + expect(emptyAppliedSync.stderr).toContain("No schema changes found"); + + const dbOnlyChange = await runSupabaseLive( + ["db", "query", "--local", "create table public.phase6_pulled (id bigint primary key)"], + { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + ); + expect(dbOnlyChange.exitCode, commandFailure(dbOnlyChange)).toBe(0); + + const configPath = path.join(projectDir, "supabase", "config.toml"); + const pullConfig = readFileSync(configPath, "utf8") + .replace('schema_paths = ["./schemas/*.sql"]', "schema_paths = []") + .replace('declarative_schema_path = "./schemas"', 'declarative_schema_path = "./database"'); + writeFileSync(configPath, pullConfig); + renameSync( + path.join(projectDir, "supabase", "schemas"), + path.join(projectDir, "supabase", ".phase6-exported-schemas"), + ); + + const migrationsBeforePull = new Set(migrationFiles(projectDir)); + const pulled = await runSupabaseLive( + ["db", "pull", "phase6_pull", "--db-url", databaseUrl, "--diff-engine", "pg-delta"], + { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true", SUPABASE_YES: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(pulled.exitCode, commandFailure(pulled)).toBe(0); + const pulledMigrations = migrationFiles(projectDir).filter( + (file) => !migrationsBeforePull.has(file), + ); + expect(pulledMigrations.length).toBeGreaterThan(0); + expect( + pulledMigrations + .map((file) => + readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8"), + ) + .join("\n"), + ).toContain("phase6_pulled"); + + const removePulledTable = await runSupabaseLive( + ["db", "query", "--local", "drop table public.phase6_pulled"], + { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(removePulledTable.exitCode, commandFailure(removePulledTable)).toBe(0); + + const pulledVersion = pulledMigrations[0]?.split("_", 1)[0]; + expect(pulledVersion).toMatch(/^\d{14}$/u); + if (pulledVersion === undefined) throw new Error("db pull produced no migration version"); + const markPulledReverted = await runSupabaseLive( + ["migration", "repair", "--local", "--status", "reverted", pulledVersion], + { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(markPulledReverted.exitCode, commandFailure(markPulledReverted)).toBe(0); + + const pullPush = await runSupabaseLive(["db", "push", "--local"], { + cwd: projectDir, + env: { SUPABASE_YES: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(pullPush.exitCode, commandFailure(pullPush)).toBe(0); + + const emptyPull = await runSupabaseLive( + ["db", "pull", "phase6_pull_empty", "--db-url", databaseUrl, "--diff-engine", "pg-delta"], + { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "true", SUPABASE_YES: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(emptyPull.exitCode, commandFailure(emptyPull)).toBe(1); + expect(emptyPull.stderr).toContain("No schema changes found"); + }, + ); + + test( + "keeps the legacy edge-runtime implementation available behind the opt-out", + { timeout: SCENARIO_TIMEOUT_MS }, + async (context) => { + const legacy = await runSupabaseLive( + ["db", "diff", "--from", "migrations", "--to", databaseUrl], + { + cwd: projectDir, + env: { SUPABASE_USE_PG_DELTA_NEXT: "false" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + const output = `${legacy.stdout}\n${legacy.stderr}`; + if ( + legacy.exitCode !== 0 && + /(?:No such image|manifest unknown|pull access denied|edge-runtime: (?:not found|command not found))/iu.test( + output, + ) + ) { + context.skip("legacy edge-runtime image is concretely unavailable on this Docker host"); + } + expect(legacy.exitCode, commandFailure(legacy)).toBe(0); + }, + ); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts index 6a52434c89..c6d8c2b9a8 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts @@ -447,8 +447,10 @@ export const legacyDeclarativeSeamLayer = Layer.effect( offset += chunk.length; } // stdout is three newline-separated lines: container id, source URL, - // and an optional target-override URL (empty unless the local-target - // declarative branch redirected the target to a second shadow db). + // and an optional second-database URL. Legacy diff uses the third URL + // only when its local-target declarative branch redirects the target; + // `pgdelta-next` always returns its empty same-cluster declarative + // scratch database there. That next mode never asks Go to apply SQL. // The URLs arrive WITHOUT a password — the Go seam prints them via // ToPostgresURLWithoutPassword so it never logs a credential to stdout // (CWE-312). The shadow uses the local Postgres password, so we re-inject diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts index 16593f5f75..8fc4958144 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts @@ -11,8 +11,12 @@ export type LegacyCatalogMode = "baseline" | "migrations" | "declarative"; * `db pull` diff source), plus the local-target declarative branch. * - `declarative`: a bare shadow with no baseline/migrations (the `db pull * --declarative` empty export source). + * - `pgdelta-next`: platform baseline + local migrations in `postgres`, plus + * an empty same-cluster `pgdelta_declarative` scratch database. Declarative + * SQL is deliberately not applied by Go in this mode; the TypeScript next + * engine loads it later through `planSchemaFiles`. */ -type LegacyShadowMode = "diff" | "declarative"; +type LegacyShadowMode = "diff" | "declarative" | "pgdelta-next"; /** A live shadow database left running for the caller to diff against and remove. */ export interface LegacyShadowSource { @@ -21,9 +25,10 @@ export interface LegacyShadowSource { /** The diff source Postgres URL (the provisioned shadow). */ readonly sourceUrl: string; /** - * When set, replaces the diff target with a second shadow database - * (`contrib_regression` with declarative schemas applied). Mirrors Go's - * local-target declarative branch, where the user's local DB is not diffed. + * Optional second live database. For legacy diff it replaces the target with + * `contrib_regression` after Go applies declarative schemas. For + * `pgdelta-next` it is the empty declarative scratch database; TypeScript + * loads the declarative files later through `planSchemaFiles`. */ readonly targetUrlOverride: string | undefined; } diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.ts index 93a4504acf..bf2fb2c9bc 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.ts @@ -69,7 +69,7 @@ interface LegacyPgDeltaDiffResult { } /** - * Ambient inputs shared by every pg-delta invocation: the project id (for the + * Ambient inputs retained for the legacy pg-delta adapter: the project id (for the * `supabase_edge_runtime_` Deno-cache volume), the working directory (mounted * at `/workspace`), and the resolved pg-delta npm version (template interpolation). */ diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts index 53dfc3fd0b..3b49c856dd 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts @@ -2,8 +2,16 @@ import { Effect, type FileSystem, type Path } from "effect"; import { legacyBold } from "../../../shared/legacy-colors.ts"; import { LegacyDeclarativeWriteError } from "./legacy-pgdelta.errors.ts"; +import type { + LegacyPgDeltaDeclarativeExportResult, + LegacyPgDeltaExportManifest, +} from "./legacy-pgdelta-engine.service.ts"; import type { LegacyDeclarativeOutput } from "./legacy-pgdelta.ts"; +const EXPORT_MANIFEST_FILE = ".pgdelta-export.json"; + +type LegacyDeclarativeWriteOutput = LegacyDeclarativeOutput | LegacyPgDeltaDeclarativeExportResult; + /** * Go's `declarative.Generate` / `pull.go`'s written-to line, printed by all three * declarative write paths (`generate`, `pull --declarative`, `sync`'s bootstrap). @@ -32,7 +40,7 @@ export const legacyWriteDeclarativeSchemas = Effect.fnUntraced(function* ( fs: FileSystem.FileSystem, path: Path.Path, declarativeDir: string, - output: LegacyDeclarativeOutput, + output: LegacyDeclarativeWriteOutput, ) { yield* fs.remove(declarativeDir, { recursive: true }).pipe( Effect.catchTag("PlatformError", (error) => @@ -48,18 +56,37 @@ export const legacyWriteDeclarativeSchemas = Effect.fnUntraced(function* ( ); yield* fs.makeDirectory(declarativeDir, { recursive: true }); + const writtenFiles: Array = []; for (const file of output.files) { - const rel = path.normalize(file.path); + const name = "name" in file ? file.name : file.path; + const rel = path.normalize(name); if (rel.startsWith("..") || path.isAbsolute(rel)) { return yield* Effect.fail( new LegacyDeclarativeWriteError({ - message: `unsafe declarative export path: ${file.path}`, + message: `unsafe declarative export path: ${name}`, }), ); } const targetPath = path.join(declarativeDir, rel); yield* fs.makeDirectory(path.dirname(targetPath), { recursive: true }); yield* fs.writeFileString(targetPath, file.sql); + writtenFiles.push(name.split("\\").join("/")); + } + + const manifest = "manifest" in output ? output.manifest : undefined; + if (manifest !== undefined) { + const serialized: LegacyPgDeltaExportManifest & { + readonly formatVersion: 1; + readonly files: ReadonlyArray; + } = { + formatVersion: 1, + ...manifest, + files: [...writtenFiles].sort(), + }; + yield* fs.writeFileString( + path.join(declarativeDir, EXPORT_MANIFEST_FILE), + `${JSON.stringify(serialized, null, 2)}\n`, + ); } }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts index ba2684ca1c..1004f0ba20 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts @@ -8,13 +8,17 @@ import { Cause, Effect, Exit, FileSystem, Path } from "effect"; import { legacyBold } from "../../../shared/legacy-colors.ts"; import { LegacyDeclarativeWriteError } from "./legacy-pgdelta.errors.ts"; +import type { LegacyPgDeltaDeclarativeExportResult } from "./legacy-pgdelta-engine.service.ts"; import type { LegacyDeclarativeOutput } from "./legacy-pgdelta.ts"; import { legacyDeclarativeSchemaWrittenLine, legacyWriteDeclarativeSchemas, } from "./legacy-pgdelta.write.ts"; -const write = (declarativeDir: string, output: LegacyDeclarativeOutput) => +const write = ( + declarativeDir: string, + output: LegacyDeclarativeOutput | LegacyPgDeltaDeclarativeExportResult, +) => Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; @@ -41,6 +45,32 @@ describe("legacyWriteDeclarativeSchemas", () => { expect(existsSync(join(declDir, "stale.sql"))).toBe(false); expect(readFileSync(join(declDir, "public.sql"), "utf8")).toBe("create table a();"); expect(readFileSync(join(declDir, "auth", "roles.sql"), "utf8")).toBe("create role app;"); + expect(existsSync(join(declDir, ".pgdelta-export.json"))).toBe(false); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("writes the next export manifest with the generated file list", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); + const declDir = join(dir, "supabase", "database"); + return write(declDir, { + files: [ + { name: "schemas/z.sql", sql: "select 'z';" }, + { name: "schemas/a.sql", sql: "select 'a';" }, + ], + manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, + }).pipe( + Effect.tap(() => + Effect.sync(() => { + expect(JSON.parse(readFileSync(join(declDir, ".pgdelta-export.json"), "utf8"))).toEqual({ + formatVersion: 1, + redactSecrets: true, + scope: "database", + profile: "supabase", + files: ["schemas/a.sql", "schemas/z.sql"], + }); rmSync(dir, { recursive: true, force: true }); }), ), diff --git a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts index 74582272eb..1ad855de06 100644 --- a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts +++ b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts @@ -98,6 +98,8 @@ export interface LegacyDbTomlValues { readonly baseline: LegacyBaselineTomlConfig; /** `[db.migrations] enabled` (default true) — gates `up`/`down` migration apply. */ readonly migrationsEnabled: boolean; + /** `[db.migrations] schema_paths`, resolved relative to `supabase/`. */ + readonly migrationSchemaPaths?: ReadonlyArray; /** `[db.seed]` enabled + supabase-prefixed `sql_paths` globs — used by `down`. */ readonly seed: LegacyDbSeedTomlConfig; /** `[db.vault]` secrets (name → resolved value) — upserted by `up`/`down`. */ @@ -283,6 +285,7 @@ const LEGACY_ENV_OVERRIDABLE_KEYS: ReadonlyArray = [ "db.shadow_port", "db.major_version", "db.migrations.enabled", + "db.migrations.schema_paths", "db.seed.enabled", "db.seed.sql_paths", "auth.enabled", @@ -536,8 +539,8 @@ const DEFAULT_SUPABASE_ENV = "development"; * `process.env` (no project-env map path) and must reflect `supabase/.env`: * `SUPABASE_INTERNAL_IMAGE_REGISTRY` (`legacyGetRegistryImageUrl`) and * `PGDELTA_NPM_REGISTRY` (`legacyPgDeltaNpmRegistryOption`, read straight from - * `process.env` for every pg-delta edge-runtime invocation — diff, declarative - * export/sync, and the push/pull/dump migrations-catalog cache). Go's + * `process.env` for legacy-opt-out pg-delta edge-runtime invocations). The bundled + * next implementation never consults it. Go's * `godotenv.Load` (`loadNestedEnv`) `os.Setenv`s every key from the project * `.env`, so both readers see a `.env`-only value there; omitting either here * would leave that one process.env-only reader blind to a project-`.env`-scoped @@ -1059,10 +1062,11 @@ const readDbTomlCore = Effect.fnUntraced(function* ( .readFileString(poolerUrlPath) .pipe(Effect.map(nonEmptyString), Effect.orElseSucceed(Option.none)); - // Go: `config.go:700-709` — the pg-delta npm version is read from + // Go: `config.go:700-709` — the legacy pg-delta npm version is read from // `.temp/pgdelta-version` (trimmed, non-empty) during Load, never from the // TOML. An absent/empty file leaves it `None` (callers fall back to the - // default via `legacyEffectivePgDeltaNpmVersion`). + // default via `legacyEffectivePgDeltaNpmVersion`). The bundled next engine is + // fixed at CLI build time and ignores this compatibility setting. const pgDeltaVersionPath = path.join(supabaseDir, ".temp", "pgdelta-version"); const pgDeltaNpmVersion = yield* fs.readFileString(pgDeltaVersionPath).pipe( Effect.map((content) => nonEmptyString(content.trim())), @@ -1815,6 +1819,27 @@ const readDbTomlCore = Effect.fnUntraced(function* ( ? undefined : envOverride("SUPABASE_DB_MIGRATIONS_ENABLED"), ); + const rawMigrationSchemaPaths = migrationsRaw?.["schema_paths"]; + const migrationSchemaPathsOverride = remoteOverrideKeys.has("db.migrations.schema_paths") + ? undefined + : envOverride("SUPABASE_DB_MIGRATIONS_SCHEMA_PATHS"); + const splitMigrationSchemaPaths = (value: string): ReadonlyArray => { + const expanded = legacyExpandEnv(value, lookup); + return expanded.length === 0 ? [] : expanded.split(","); + }; + const migrationSchemaPathPatterns = + migrationSchemaPathsOverride !== undefined + ? splitMigrationSchemaPaths(migrationSchemaPathsOverride) + : Array.isArray(rawMigrationSchemaPaths) + ? rawMigrationSchemaPaths + .filter((pattern): pattern is string => typeof pattern === "string") + .map((pattern) => legacyExpandEnv(pattern, lookup)) + : typeof rawMigrationSchemaPaths === "string" + ? splitMigrationSchemaPaths(rawMigrationSchemaPaths) + : []; + const migrationSchemaPaths = migrationSchemaPathPatterns.map((pattern) => + path.isAbsolute(pattern) || pattern.length === 0 ? pattern : path.join("supabase", pattern), + ); // `[db.seed]` — Go defaults enabled true, sql_paths ["seed.sql"]; relative // patterns are supabase-prefixed (`config.go:801-806`). `db.seed.enabled` is @@ -1953,6 +1978,7 @@ const readDbTomlCore = Effect.fnUntraced(function* ( vaultNames, }, migrationsEnabled, + migrationSchemaPaths, seed: { enabled: seedEnabled, sqlPaths: seedSqlPaths }, vault, appliedRemote, diff --git a/apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.integration.test.ts b/apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.integration.test.ts index 0af60b46c9..367dcc7a7b 100644 --- a/apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.integration.test.ts +++ b/apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.integration.test.ts @@ -13,7 +13,10 @@ import { Effect } from "effect"; import { LEGACY_SUGGEST_ENV_VAR, LEGACY_SUGGEST_LOCAL_STACK } from "./legacy-connect-errors.ts"; import type { LegacyDbConnectError, LegacyDbExecError } from "./legacy-db-connection.errors.ts"; import { type LegacyPgConnInput, LegacyDbConnection } from "./legacy-db-connection.service.ts"; -import { legacyDbConnectionSqlPgLayer } from "./legacy-db-connection.sql-pg.layer.ts"; +import { + legacyAcquirePgPool, + legacyDbConnectionSqlPgLayer, +} from "./legacy-db-connection.sql-pg.layer.ts"; const SUGGESTION_CONTEXT = { dashboardUrl: "https://supabase.com/dashboard", @@ -335,3 +338,37 @@ describe("legacyDbConnectionSqlPgLayer exec failures", () => { }), ); }); + +describe("legacyAcquirePgPool", () => { + it.live("returns the winning raw pool and ends it when the caller scope closes", () => + Effect.gen(function* () { + const server = yield* Effect.promise(() => + fakeQueryServer(() => Buffer.concat([commandComplete("SELECT 1"), READY_FOR_QUERY])), + ); + yield* Effect.gen(function* () { + let acquired: import("pg").Pool | undefined; + + yield* Effect.gen(function* () { + const pool = yield* legacyAcquirePgPool( + { + host: "127.0.0.1", + port: server.port, + user: "postgres", + password: SENTINEL_PASSWORD, + database: "postgres", + sslmode: "disable", + }, + { isLocal: true, dnsResolver: "native" }, + ); + acquired = pool; + expect(pool.ending).toBe(false); + expect(pool.ended).toBe(false); + yield* Effect.tryPromise(() => pool.query("select 1")); + }).pipe(Effect.scoped); + + expect(acquired?.ending).toBe(true); + expect(acquired?.ended).toBe(true); + }).pipe(Effect.ensuring(Effect.sync(server.close))); + }), + ); +}); diff --git a/apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts b/apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts index a05da9fd73..1ecdc5bd8d 100644 --- a/apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts +++ b/apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts @@ -574,15 +574,32 @@ export const legacyAcquireProbedPool =

( return pool; }); +/** Map a driver connect failure to the credential-free Go-compatible error. */ +const legacyToConnectError = ( + cfg: LegacyPgConnInput, + isLocal: boolean, + error: unknown, +): LegacyDbConnectError => { + const suggestion = + cfg.suggestionContext === undefined + ? undefined + : legacyConnectSuggestion(error, { ...cfg.suggestionContext, isLocal }); + return new LegacyDbConnectError({ + message: `failed to connect to postgres: ${legacyConnectFailureMessage(cfg, error)}`, + ...(suggestion === undefined ? {} : { suggestion }), + }); +}; + /** - * Default `LegacyDbConnection` layer, backed by `@effect/sql-pg` (pure-JS `pg` - * driver, no native addon — bundles under `bun build --compile`). Each - * `connect` builds a scoped single-client connection that closes on scope exit. + * Acquire the winning raw pool through the full Go-compatible connection attempt + * chain. The pool finalizer is owned by the caller's scope; both the legacy session + * adapter and direct-pool consumers use this one acquisition core so their DNS, + * TLS, fallback, and role behavior cannot drift apart. */ -const connect = ( +const acquirePgPoolConnection = ( cfg: LegacyPgConnInput, { isLocal, dnsResolver }: LegacyDbConnectOptions, -): Effect.Effect => +) => Effect.gen(function* () { // pgconn dials the primary host then each HA fallback in order // (`config.go:326-362`); `cfg.fallbacks` carries the extras parsed from a @@ -616,8 +633,8 @@ const connect = ( // `AfterConnect` hook only on the remote path (`ConnectByConfigStream`, // `connect.go:342-362`), not `ConnectLocalPostgres`, so gate on `!isLocal`. const stepDownRequired = !isLocal && needsRoleStepDown(cfg.user); - // Build the primary connection over a self-managed `pg.Pool` (via - // `PgClient.fromPool`) rather than `PgClient.make`, so we control two pool + // Build the primary connection over a self-managed `pg.Pool` rather than + // `PgClient.make`, so we control two pool // behaviors `PgClient.make` does not expose: `idleTimeoutMillis: 0` (never reap // the single pooled connection — see `legacyBuildPoolConfig`; the fix for the // `db pull` step-down loss) and the per-connection role step-down `verify` hook @@ -626,12 +643,12 @@ const connect = ( // the pool on scope exit AND on every failure/timeout (the leak `PgClient.make` // has). `probe` (below) runs each attempt in a forked scope so a failed fallback // attempt's pool closes immediately, before the next host is dialed. - const makeClient = ( + const makePool = ( dialHost: string, port: number, sslOption: boolean | ConnectionOptions | undefined, - ) => { - const acquire = legacyAcquireProbedPool( + ) => + legacyAcquireProbedPool( () => new Pg.Pool( legacyBuildPoolConfig( @@ -645,8 +662,6 @@ const connect = ( ), connectTimeoutSeconds, ); - return PgClient.fromPool({ acquire }).pipe(Effect.provide(Reactivity.layer)); - }; // Go's `ConnectByUrl` calls `SetConnectSuggestion(err)` on every connect failure // (`connect.go:187`), mapping the driver error to an actionable hint that replaces @@ -657,17 +672,6 @@ const connect = ( // to postgres:` prefix plus the `host=… user=… database=…` identity and the // underlying driver cause — not the bare `SqlError` toString, which drops all // of that detail. - const toConnectError = (error: unknown) => { - const suggestion = - cfg.suggestionContext === undefined - ? undefined - : legacyConnectSuggestion(error, { ...cfg.suggestionContext, isLocal }); - return new LegacyDbConnectError({ - message: `failed to connect to postgres: ${legacyConnectFailureMessage(cfg, error)}`, - ...(suggestion === undefined ? {} : { suggestion }), - }); - }; - // Load the `sslrootcert` CA bundle (pgconn reads it into `RootCAs` at parse // time; a missing/unreadable file aborts). Skipped for local connections, which // never use TLS. pgconn builds TLS per fallback host, so the CA must be loaded @@ -723,7 +727,7 @@ const connect = ( const attempts = dialTargets.flatMap(({ dialHost, port, servername }) => legacySslConfigsFor(cfg.sslmode, isLocal, servername, caCert, dialHost, clientCert).map( (ssl) => ({ - client: makeClient(dialHost, port, ssl), + pool: makePool(dialHost, port, ssl), // pgconn only short-circuits the fallback chain on an auth error when the // failed attempt used TLS (`pgconn.go:182`, gated on `fc.TLSConfig != nil`); // a TLS config is any non-plaintext `ssl` value. @@ -756,9 +760,8 @@ const connect = ( // session and closes with it. const sessionScope = yield* Scope.Scope; const attemptScope = yield* Scope.fork(sessionScope); - return yield* attempt.client.pipe( - Effect.tap((candidate) => candidate`select 1`), - Effect.map((candidate) => ({ candidate, rawConfig: attempt.rawConfig })), + return yield* attempt.pool.pipe( + Effect.map((pool) => ({ pool, rawConfig: attempt.rawConfig })), Scope.provide(attemptScope), Effect.onExit((exit) => Exit.isSuccess(exit) ? Effect.void : Scope.close(attemptScope, exit), @@ -766,7 +769,7 @@ const connect = ( ); }); const lastIndex = attempts.length - 1; - const { candidate: client, rawConfig: winningRawConfig } = yield* attempts + const { pool, rawConfig: winningRawConfig } = yield* attempts .slice(0, lastIndex) .reduceRight( (next, attempt) => @@ -777,26 +780,58 @@ const connect = ( ), probe(attempts[lastIndex]!), ) - .pipe(Effect.mapError(toConnectError)); + .pipe(Effect.mapError((error) => legacyToConnectError(cfg, isLocal, error))); // Step down from the temp/privileged login role before any further SQL — but // only for remote connections: Go installs this hook in `ConnectByConfigStream`, // not `ConnectLocalPostgres`, so a local `--db-url` using `supabase_admin`/ - // `cli_login_*` must not run it. The pool's `"connect"` hook already ran this on - // the physical connection (and on any silent redial); this explicit one-shot is - // the fail-fast path — the hook swallows errors, so a real role-privilege problem - // only surfaces here, as `LegacyDbConnectError: failed to set session role: ...` - // (Go parity). `max: 1` + `idleTimeoutMillis: 0` keep the stepped-down connection + // `cli_login_*` must not run it. The pool's `verify` hook already ran this on + // the physical connection (and runs it on any silent redial); this explicit + // one-shot preserves the fail-fast `LegacyDbConnectError: failed to set session + // role: ...` path. `max: 1` + `idleTimeoutMillis: 0` keep the stepped-down connection // alive so the session-scoped role persists for every later `exec`/`query`. if (stepDownRequired) { - yield* client.unsafe(SET_SESSION_ROLE).pipe( - Effect.asVoid, - Effect.mapError( - (error) => new LegacyDbConnectError({ message: `failed to set session role: ${error}` }), - ), - ); + yield* Effect.tryPromise({ + try: () => pool.query(SET_SESSION_ROLE), + catch: (error) => + new LegacyDbConnectError({ message: `failed to set session role: ${error}` }), + }); } + return { pool, winningRawConfig, stepDownRequired }; + }); + +/** + * Acquire a live `pg.Pool` using the same scoped lifecycle and connection parity + * as `LegacyDbConnection.connect`. The caller owns the surrounding scope; closing + * it ends the winning pool, while every losing fallback attempt is closed before + * the next target is tried. + */ +export const legacyAcquirePgPool = ( + cfg: LegacyPgConnInput, + options: LegacyDbConnectOptions, +): Effect.Effect => + acquirePgPoolConnection(cfg, options).pipe(Effect.map(({ pool }) => pool)); + +/** + * Default `LegacyDbConnection` layer, backed by `@effect/sql-pg` (pure-JS `pg` + * driver, no native addon — bundles under `bun build --compile`). Each + * `connect` builds a scoped single-client connection that closes on scope exit. + */ +const connect = ( + cfg: LegacyPgConnInput, + options: LegacyDbConnectOptions, +): Effect.Effect => + Effect.gen(function* () { + const { pool, winningRawConfig, stepDownRequired } = yield* acquirePgPoolConnection( + cfg, + options, + ); + const client = yield* PgClient.fromPool({ acquire: Effect.succeed(pool) }).pipe( + Effect.provide(Reactivity.layer), + Effect.mapError((error) => legacyToConnectError(cfg, options.isLocal, error)), + ); + // `inspect report` runs ~14 `COPY (...) TO STDOUT` statements. node-postgres' // COPY protocol needs the raw client (which `@effect/sql-pg` does not surface), // so the session opens ONE dedicated raw connection against the winning dial @@ -826,7 +861,7 @@ const connect = ( const fresh = new Pg.Client(winningRawConfig); yield* Effect.tryPromise({ try: () => fresh.connect(), - catch: toConnectError, + catch: (error) => legacyToConnectError(cfg, options.isLocal, error), }); if (stepDownRequired) { yield* Effect.tryPromise({ diff --git a/apps/cli/src/legacy/shared/legacy-db-push-core.ts b/apps/cli/src/legacy/shared/legacy-db-push-core.ts index 31b855bdd4..07e2788004 100644 --- a/apps/cli/src/legacy/shared/legacy-db-push-core.ts +++ b/apps/cli/src/legacy/shared/legacy-db-push-core.ts @@ -9,6 +9,7 @@ import { } from "../commands/db/shared/legacy-pgdelta.cache.ts"; import { type LegacyPgDeltaContext } from "../commands/db/shared/legacy-pgdelta.ts"; import { legacyParseBoolEnv } from "../commands/db/shared/legacy-diff-engine.ts"; +import { legacyResolvePgDeltaImplementation } from "../commands/db/shared/legacy-pgdelta-next-flag.ts"; import { LEGACY_ERR_MISSING_LOCAL, LEGACY_ERR_MISSING_REMOTE, @@ -318,6 +319,9 @@ export const legacyDbPushCore = Effect.fnUntraced(function* (input: LegacyDbPush const cacheEnabled = toml.pgDelta.enabled || legacyParseBoolEnv(toml.envLookup("SUPABASE_EXPERIMENTAL_PG_DELTA")); + const pgDeltaImplementation = legacyResolvePgDeltaImplementation( + toml.envLookup("SUPABASE_USE_PG_DELTA_NEXT"), + ); const pgDeltaCtx: LegacyPgDeltaContext = { // Go's `flags.LoadConfig` seeds `Config.ProjectId = ProjectRef` before // `Config.Load` runs, so an absent config.toml `project_id` retains the @@ -341,7 +345,10 @@ export const legacyDbPushCore = Effect.fnUntraced(function* (input: LegacyDbPush denoVersion: toml.denoVersion, }; yield* legacyTryCacheMigrationsCatalog(fs, path, pgDeltaCtx, { - enabled: cacheEnabled, + // The catalog is an alpha.33-only artifact with no next-engine + // consumer. Default-next commands deliberately skip this obsolete + // warmup so a successful push/bootstrap cannot start edge-runtime. + enabled: cacheEnabled && pgDeltaImplementation === "legacy", targetUrl: legacyToPostgresURL(conn), conn, isLocal, diff --git a/apps/cli/src/legacy/shared/legacy-seed-ops.ts b/apps/cli/src/legacy/shared/legacy-seed-ops.ts index bbea4d4fcc..3419a30df4 100644 --- a/apps/cli/src/legacy/shared/legacy-seed-ops.ts +++ b/apps/cli/src/legacy/shared/legacy-seed-ops.ts @@ -109,6 +109,9 @@ const legacyGlobSeedFiles = Effect.fnUntraced(function* ( } satisfies LegacyGlobResult; }); +/** Shared Go-compatible SQL glob expansion for migration/declarative consumers. */ +export const legacyResolveSqlGlobFiles = legacyGlobSeedFiles; + const toSlash = (p: string): string => p.replaceAll("\\", "/"); /** Splits a forward-slashed path into its directory prefix and final element. */ diff --git a/apps/cli/tests/helpers/live.ts b/apps/cli/tests/helpers/live.ts index 89d8092ba3..327cceacb3 100644 --- a/apps/cli/tests/helpers/live.ts +++ b/apps/cli/tests/helpers/live.ts @@ -1,3 +1,4 @@ +import { execSync } from "node:child_process"; import { describe } from "vitest"; import { runSupabase } from "./cli.ts"; @@ -39,6 +40,23 @@ export { */ export const describeLive = describe.skipIf(!isLiveConfigured()); +function hasDockerDaemon(): boolean { + try { + execSync("docker info", { stdio: "ignore" }); + return true; + } catch { + return false; + } +} + +/** + * `describe` for local-stack live tests that only require a real Docker daemon. + * Unlike `describeLive`, this gate does not require platform credentials or a + * Management API. The synchronous `docker info` probe is read-only and runs once + * when this helper module is collected. + */ +export const describeDockerLive = describe.skipIf(!hasDockerDaemon()); + /** * `describe` for project-scoped live suites: runs only when the live env is * configured AND a project ref is available. On a control-plane-only stack diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b6e5837da1..733db9db68 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -69,6 +69,7 @@ catalogs: overrides: '@effect/platform-node-shared': 4.0.0-beta.97 + '@launchql/protobufjs>@types/node': 24.10.4 importers: @@ -150,6 +151,12 @@ importers: '@supabase/config': specifier: workspace:* version: link:../../packages/config + '@supabase/pg-delta': + specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb + version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb) + '@supabase/pg-topo': + specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb + version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb '@supabase/process-compose': specifier: workspace:* version: link:../../packages/process-compose @@ -715,10 +722,18 @@ packages: resolution: {integrity: sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg==} engines: {node: '>=6.9.0'} + '@babel/traverse@7.28.5': + resolution: {integrity: sha512-TCCj4t55U90khlYkVV/0TfkJkAkUg3jZFA3Neb7unZT8CPok7iiRfaX0F+WnqWqt7OxhOn0uBKXCw4lbL8W0aQ==} + engines: {node: '>=6.9.0'} + '@babel/traverse@7.29.7': resolution: {integrity: sha512-EhlfNQtZ+NK22w5BM61ciuiq1m58ed33Wr1Xan//ZRTy6hgjnwyCffRYwzsGXdASJSUJ1guZILsErh1eQcl+zw==} engines: {node: '>=6.9.0'} + '@babel/types@7.28.5': + resolution: {integrity: sha512-qQ5m48eI/MFLQ5PxQj4PFaprjyCTLI37ElWMmNs0K8Lk3dVeOdNpB3ks8jc7yM5CDmVC73eMVk/trk3fgmrUpA==} + engines: {node: '>=6.9.0'} + '@babel/types@7.29.7': resolution: {integrity: sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==} engines: {node: '>=6.9.0'} @@ -1186,6 +1201,13 @@ packages: '@jridgewell/trace-mapping@0.3.31': resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==} + '@launchql/protobufjs@7.2.6': + resolution: {integrity: sha512-vwi1nG2/heVFsIMHQU1KxTjUp5c757CTtRAZn/jutApCkFlle1iv8tzM/DHlSZJKDldxaYqnNYTg0pTyp8Bbtg==} + engines: {node: '>=12.0.0'} + + '@libpg-query/parser@17.6.10': + resolution: {integrity: sha512-AT/IM9H24/u70HvBhzkYlSBlYQWhJK3Z4CTmTnd3PnMqHU7Ib3o5pk2TEik6IblWsU64D+4GGURn94v2iSRe1A==} + '@mdx-js/mdx@3.1.1': resolution: {integrity: sha512-f6ZO2ifpwAQIpzGWaBQT2TXxPv6z3RBzQKpVftEWN78Vl/YweF1uwussDx8ECAXVtr3Rs89fKyG9YlzUs9DyGQ==} @@ -2298,6 +2320,15 @@ packages: resolution: {integrity: sha512-7FNeNl8NCE7aINx7WXiKQrPYZWC/hvrTsmk6zmxbI7LTXE7hVek/n8AfVgpe2y82zl3w0HvCHN0bVKMBoJcC0w==} engines: {node: '>= 10.0.0'} + '@pgsql/quotes@17.1.0': + resolution: {integrity: sha512-J/H+LcrENBpYgL45WW6aTjb5Yk4tX4+AmB2/k8KZa+Zh3wiCtqmNIag+HZz5HmWaF6EZK9ZGC95NBD1fs+rUvg==} + + '@pgsql/traverse@17.2.6': + resolution: {integrity: sha512-BLOE9DUcvd3y3Ogf56mmpTONPylnMuFCo9PvHQA9SXavcRPhRtvIZ/sRO2ja+bUWK/3KTLJ1Hb61CbbdPkcHoA==} + + '@pgsql/types@17.6.2': + resolution: {integrity: sha512-1UtbELdbqNdyOShhrVfSz3a1gDi0s9XXiQemx+6QqtsrXe62a6zOGU+vjb2GRfG5jeEokI1zBBcfD42enRv0Rw==} + '@pinojs/redact@0.4.0': resolution: {integrity: sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==} @@ -2319,6 +2350,36 @@ packages: '@posthog/types@1.397.1': resolution: {integrity: sha512-W/LpWbKVaaUnfZKuFuHa+Dg03D+fC87cM+PQbG+59JcSPW8F0JcBtSoXmpfrqbpuxUToMo+gktutrUkAb/KQBw==} + '@protobufjs/aspromise@1.1.2': + resolution: {integrity: sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==} + + '@protobufjs/base64@1.1.2': + resolution: {integrity: sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==} + + '@protobufjs/codegen@2.0.5': + resolution: {integrity: sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==} + + '@protobufjs/eventemitter@1.1.1': + resolution: {integrity: sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==} + + '@protobufjs/fetch@1.1.1': + resolution: {integrity: sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==} + + '@protobufjs/float@1.0.2': + resolution: {integrity: sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==} + + '@protobufjs/inquire@1.1.2': + resolution: {integrity: sha512-pa0vFRuws4wkvaXKK1uXZMAwAX4/t8ANaJo45iw/oQHNQ9q5xUzwgFmVJGXiga2BeN+zpX7Vf9vmsiIa2J+MUw==} + + '@protobufjs/path@1.1.2': + resolution: {integrity: sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==} + + '@protobufjs/pool@1.1.0': + resolution: {integrity: sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==} + + '@protobufjs/utf8@1.1.2': + resolution: {integrity: sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug==} + '@radix-ui/number@1.1.2': resolution: {integrity: sha512-ceTwaxc4I5IOi97DgCotl3pqiyRGvffcc0oOsE2dQYaJOFIDsDt4VWG6xEbg1QePv9QWausCEIppud/tJ1wNig==} @@ -2861,6 +2922,21 @@ packages: resolution: {integrity: sha512-megYmexlYEoR/0qlsr4Snh9wtzAodO7MAri3NMevZrXzNvQRKlvmTcSBoKGLQEPDakgDZMqbMdf9DwoZz6qfoA==} engines: {node: '>=22.0.0'} + '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb': + resolution: {integrity: sha512-eWhb8JyODx870aSr2xKr3i81yBBnblAqLjdFqW0MGr6pxDzX/PbJkQGx700u/CESycO6HfW8+2MrDQUE3em53w==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb} + version: 1.0.0-alpha.33 + engines: {node: '>=20.0.0'} + hasBin: true + peerDependencies: + '@supabase/pg-topo': ^1.0.0-alpha.3 + peerDependenciesMeta: + '@supabase/pg-topo': + optional: true + + '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb': + resolution: {integrity: sha512-HoqxATDYB2WygDOV1XQBN/hM/hcfvVUrc7F+R691j6CD89cHumR/nRiRJ+0bh9siZb7Fx81Bp9BAPRfzEkEydA==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb} + version: 1.0.0-alpha.5 + '@supabase/phoenix@0.4.5': resolution: {integrity: sha512-aAn9H9ovVyeApKy11OWOrrOGq8DV68yWeH4ud2lN9fzn4aO8Zb5GLL9m1pUg9nLqIcT+ZDfAcsZe0E/nqdv2lw==} @@ -3035,6 +3111,9 @@ packages: '@types/ms@2.1.0': resolution: {integrity: sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA==} + '@types/node@24.10.4': + resolution: {integrity: sha512-vnDVpYPMzs4wunl27jHrfmwojOGKya0xyM3sH+UE5iv5uPS6vX7UIoh6m+vQc5LGBq52HBKPIn/zcSZVzeDEZg==} + '@types/node@26.1.1': resolution: {integrity: sha512-nxAkRSVkN1Y0JC1W8ky/fTfkGsMmcrRsbx+3XoZE+rMOX71kLYTV7fLXpqud1GpbpP5TuffXFqfX7fH2GgZREw==} @@ -3691,6 +3770,10 @@ packages: caniuse-lite@1.0.30001805: resolution: {integrity: sha512-52noaS3DubycKSXaU30TwPGIp+POyQSUVa5jBEq3vkRkY0kjyb3LQgvhU6WGyCcyXqVLWO0Cw0Q6BSdD0kUfVA==} + case@1.6.3: + resolution: {integrity: sha512-mzDSXIPaFwVDvZAHqZ9VlbyF4yyXRuX6IvB06WvPYkqJVO24kX1PPhv9bfpKNFZyxYFmmgo03HUiD8iklmJYRQ==} + engines: {node: '>= 0.8.0'} + caseless@0.12.0: resolution: {integrity: sha512-4tYFyifaFfGacoiObjJegolkwSU4xQNGbVgUiNYVUxbQ2x2lUsFvY4hVgVzGiIe6WLOPqycWXA40l+PWsxthUw==} @@ -3987,6 +4070,10 @@ packages: resolution: {integrity: sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==} engines: {node: '>=4.0.0'} + deepmerge@4.3.1: + resolution: {integrity: sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A==} + engines: {node: '>=0.10.0'} + defaults@1.0.4: resolution: {integrity: sha512-eFuaLoy/Rxalv2kr+lqMlUnrDWV+3j4pljOIJgLIhI058IQfWJ7vXhyEIHu+HtC738klGALYxOKDO0bQP3tg8A==} @@ -5172,6 +5259,9 @@ packages: resolution: {integrity: sha512-8XPvpAA8uyhfteu8pIvQxpJZ7SYYdpUivZpGy6sFsBuKRY/7rQGavedeB8aK+Zkyq6upMFVL/9AW6vOYzfRyLg==} engines: {node: '>=10'} + long@5.3.2: + resolution: {integrity: sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==} + longest-streak@3.1.0: resolution: {integrity: sha512-9Ri+o0JYgehTaVBBDoMqIl8GXtbWg711O3srftcHhZ0dqnETqLaoIK0x17fUw9rFSlK/0NlsKe0Ahhyl5pXE2g==} @@ -5550,6 +5640,9 @@ packages: nerf-dart@1.0.0: resolution: {integrity: sha512-EZSPZB70jiVsivaBLYDCyntd5eH8NTSMOn3rB+HxwdmKThGELLdYv8qVIMWvZEFy9w8ZZpW9h9OB32l1rGtj7g==} + nested-obj@0.2.2: + resolution: {integrity: sha512-M1etu+T6Ai9Bo06L3K3nWD0ytZWltggBGsrxJlOGvMNGlCA4fokUVlbPKoWzsiiRX+PXq6Cb1xFEn4chiyC7MQ==} + next-themes@0.4.6: resolution: {integrity: sha512-pZvgD5L0IEvX5/9GWyHMf3m8BKiVQwsCMHfoFosXtXBMnaS0ZnIJ9ST4b4NqLVKDEm8QBxoNNGNaBv2JNF6XNA==} peerDependencies: @@ -5948,6 +6041,9 @@ packages: peerDependencies: pg: '>=8.0' + pg-proto-parser@1.30.6: + resolution: {integrity: sha512-2XwPyl9oz5Pest4ebaovRTTJN8MXaa/XvqMQzKq127fFcl4I1POUgV/FtzHzg/p8FjtO5yHsipeW/kAumzNxxw==} + pg-protocol@1.15.0: resolution: {integrity: sha512-cq9sECI5s0+uPUXjbz8ioyPJni6RzsRib0US67i5IoTZKw8fNeYlVE7u8F4dG7vEJJtc5wdD1K189lCCUwqWTQ==} @@ -5971,6 +6067,9 @@ packages: pgpass@1.0.5: resolution: {integrity: sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==} + pgsql-deparser@17.18.5: + resolution: {integrity: sha512-C23etz+aWjp5d09SQwrByisCIV0Zy1dPI0IdBPBaRiMRrDQ2MH8O9txvqpxPyWiXEGRU+MMvZqk48UHxWWbODg==} + picocolors@1.1.1: resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==} @@ -6015,6 +6114,12 @@ packages: resolution: {integrity: sha512-u9mdErTewKSMsr+ceCt8VcNuNP0ro5AXiPXhUVApuEyqr2Zlvt+DdCFBcm+yGWN8mhOdZJ27meIDbnoZgfzpOw==} hasBin: true + plpgsql-deparser@0.7.13: + resolution: {integrity: sha512-vigoLMQL4NdMx4FjP6Q1IEIiThL+mt483ETFtcBoFJJOMxLg8h29k/NMi79XMahqDu3QvQiQnQ8JNK4hPC74Tw==} + + plpgsql-parser@0.5.16: + resolution: {integrity: sha512-zMHt7xLNW//88KzoKSDyhbDvQeEISzllZKYLl5VcpUlKy/v/EA2SnRkBQz2L6Rv+cOMNv/mzckOpyBUdUMjPdA==} + postcss@8.5.10: resolution: {integrity: sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==} engines: {node: ^10 || ^12 || >=14} @@ -6577,6 +6682,9 @@ packages: streamx@2.28.0: resolution: {integrity: sha512-1Yowhzjf0ivGMrTIkY9hav5TxobO9qIVqUE41fiCGMGgc3CLlf4MY+9AHmZqBWgDTue0fY9zWjYFVyf6Diuobw==} + strfy-js@3.2.2: + resolution: {integrity: sha512-hUgJ5k2PR1ivhq4uObxnin5j6GcOr0Y0N1lzi3z6SRhxNqu4rzpDfyoC2ToUAyM8yXNXM0zs6f4KIiqj8NqheQ==} + string-width@4.2.3: resolution: {integrity: sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==} engines: {node: '>=8'} @@ -6842,6 +6950,9 @@ packages: resolution: {integrity: sha512-8gwNZ29+0/3zmXw7ToIHZtg6wK37xnniRUdBt7B27xZxaxfgR5tGMaGHT0t0dLtBV9fXE7zurh0s6Z1DHVjfWg==} engines: {node: '>=14'} + undici-types@7.16.0: + resolution: {integrity: sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw==} + undici-types@8.3.0: resolution: {integrity: sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ==} @@ -7383,6 +7494,18 @@ snapshots: '@babel/parser': 7.29.7 '@babel/types': 7.29.7 + '@babel/traverse@7.28.5': + dependencies: + '@babel/code-frame': 7.29.7 + '@babel/generator': 7.29.7 + '@babel/helper-globals': 7.29.7 + '@babel/parser': 7.29.7 + '@babel/template': 7.29.7 + '@babel/types': 7.29.7 + debug: 4.4.3(supports-color@7.2.0) + transitivePeerDependencies: + - supports-color + '@babel/traverse@7.29.7': dependencies: '@babel/code-frame': 7.29.7 @@ -7395,6 +7518,11 @@ snapshots: transitivePeerDependencies: - supports-color + '@babel/types@7.28.5': + dependencies: + '@babel/helper-string-parser': 7.29.7 + '@babel/helper-validator-identifier': 7.29.7 + '@babel/types@7.29.7': dependencies: '@babel/helper-string-parser': 7.29.7 @@ -7771,6 +7899,26 @@ snapshots: '@jridgewell/resolve-uri': 3.1.2 '@jridgewell/sourcemap-codec': 1.5.5 + '@launchql/protobufjs@7.2.6': + dependencies: + '@protobufjs/aspromise': 1.1.2 + '@protobufjs/base64': 1.1.2 + '@protobufjs/codegen': 2.0.5 + '@protobufjs/eventemitter': 1.1.1 + '@protobufjs/fetch': 1.1.1 + '@protobufjs/float': 1.0.2 + '@protobufjs/inquire': 1.1.2 + '@protobufjs/path': 1.1.2 + '@protobufjs/pool': 1.1.0 + '@protobufjs/utf8': 1.1.2 + '@types/node': 24.10.4 + long: 5.3.2 + + '@libpg-query/parser@17.6.10': + dependencies: + '@launchql/protobufjs': 7.2.6 + '@pgsql/types': 17.6.2 + '@mdx-js/mdx@3.1.1': dependencies: '@types/estree': 1.0.9 @@ -8515,6 +8663,17 @@ snapshots: '@parcel/watcher-win32-arm64': 2.6.0 '@parcel/watcher-win32-x64': 2.6.0 + '@pgsql/quotes@17.1.0': {} + + '@pgsql/traverse@17.2.6': + dependencies: + '@pgsql/types': 17.6.2 + pg-proto-parser: 1.30.6 + transitivePeerDependencies: + - supports-color + + '@pgsql/types@17.6.2': {} + '@pinojs/redact@0.4.0': {} '@pnpm/config.env-replace@1.1.0': {} @@ -8535,6 +8694,28 @@ snapshots: '@posthog/types@1.397.1': {} + '@protobufjs/aspromise@1.1.2': {} + + '@protobufjs/base64@1.1.2': {} + + '@protobufjs/codegen@2.0.5': {} + + '@protobufjs/eventemitter@1.1.1': {} + + '@protobufjs/fetch@1.1.1': + dependencies: + '@protobufjs/aspromise': 1.1.2 + + '@protobufjs/float@1.0.2': {} + + '@protobufjs/inquire@1.1.2': {} + + '@protobufjs/path@1.1.2': {} + + '@protobufjs/pool@1.1.0': {} + + '@protobufjs/utf8@1.1.2': {} + '@radix-ui/number@1.1.2': {} '@radix-ui/primitive@1.1.6': {} @@ -9070,6 +9251,24 @@ snapshots: dependencies: tslib: 2.8.1 + '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb)': + dependencies: + debug: 4.4.3(supports-color@7.2.0) + pg: 8.22.0 + pg-connection-string: 2.14.0 + optionalDependencies: + '@supabase/pg-topo': https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb + transitivePeerDependencies: + - pg-native + - supports-color + + '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb': + dependencies: + '@pgsql/traverse': 17.2.6 + plpgsql-parser: 0.5.16 + transitivePeerDependencies: + - supports-color + '@supabase/phoenix@0.4.5': {} '@supabase/postgrest-js@2.110.7': @@ -9233,6 +9432,10 @@ snapshots: '@types/ms@2.1.0': {} + '@types/node@24.10.4': + dependencies: + undici-types: 7.16.0 + '@types/node@26.1.1': dependencies: undici-types: 8.3.0 @@ -9888,6 +10091,8 @@ snapshots: caniuse-lite@1.0.30001805: {} + case@1.6.3: {} + caseless@0.12.0: {} ccount@2.0.1: {} @@ -10152,6 +10357,8 @@ snapshots: deep-extend@0.6.0: {} + deepmerge@4.3.1: {} + defaults@1.0.4: dependencies: clone: 1.0.4 @@ -11494,6 +11701,8 @@ snapshots: chalk: 4.1.2 is-unicode-supported: 0.1.0 + long@5.3.2: {} + longest-streak@3.1.0: {} lowdb@1.0.0: @@ -12099,6 +12308,8 @@ snapshots: nerf-dart@1.0.0: {} + nested-obj@0.2.2: {} + next-themes@0.4.6(react-dom@19.2.7(react@19.2.7))(react@19.2.7): dependencies: react: 19.2.7 @@ -12610,6 +12821,20 @@ snapshots: dependencies: pg: 8.22.0 + pg-proto-parser@1.30.6: + dependencies: + '@babel/generator': 7.29.7 + '@babel/parser': 7.29.7 + '@babel/traverse': 7.28.5 + '@babel/types': 7.28.5 + '@launchql/protobufjs': 7.2.6 + case: 1.6.3 + deepmerge: 4.3.1 + nested-obj: 0.2.2 + strfy-js: 3.2.2 + transitivePeerDependencies: + - supports-color + pg-protocol@1.15.0: {} pg-types@2.2.0: @@ -12644,6 +12869,11 @@ snapshots: dependencies: split2: 4.2.0 + pgsql-deparser@17.18.5: + dependencies: + '@pgsql/quotes': 17.1.0 + '@pgsql/types': 17.6.2 + picocolors@1.1.1: {} picomatch@2.3.2: {} @@ -12688,6 +12918,21 @@ snapshots: pkg-pr-new@0.0.75: {} + plpgsql-deparser@0.7.13: + dependencies: + '@pgsql/types': 17.6.2 + pgsql-deparser: 17.18.5 + + plpgsql-parser@0.5.16: + dependencies: + '@libpg-query/parser': 17.6.10 + '@pgsql/traverse': 17.2.6 + '@pgsql/types': 17.6.2 + pgsql-deparser: 17.18.5 + plpgsql-deparser: 0.7.13 + transitivePeerDependencies: + - supports-color + postcss@8.5.10: dependencies: nanoid: 3.3.16 @@ -13407,6 +13652,10 @@ snapshots: - bare-abort-controller - react-native-b4a + strfy-js@3.2.2: + dependencies: + minimatch: 10.2.5 + string-width@4.2.3: dependencies: emoji-regex: 8.0.0 @@ -13668,6 +13917,8 @@ snapshots: unbash@4.0.2: {} + undici-types@7.16.0: {} + undici-types@8.3.0: {} undici@6.27.0: {} diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 5cc2f4607d..0318f5d07e 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -37,6 +37,8 @@ blockExoticSubdeps: true overrides: "@effect/platform-node-shared": "4.0.0-beta.97" + # pg-topo's parser chain otherwise resolves bleeding-edge Node globals that conflict with Bun's web types. + "@launchql/protobufjs>@types/node": "24.10.4" minimumReleaseAge: 10200 From 4b697a2e5e0d5090ece6b319e6a1498db91afbcf Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 6 Aug 2026 09:22:53 +0200 Subject: [PATCH 02/82] fix(cli): allow pg-topo parser build script --- pnpm-workspace.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index acb32acc7f..178ee04367 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -5,6 +5,7 @@ packages: allowBuilds: '@parcel/watcher': true + '@launchql/protobufjs': true "@swc/core": true esbuild: true msgpackr-extract: true From c52cf535abd2f8cab0b00027117ebad02b0f27a4 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 6 Aug 2026 16:24:14 +0200 Subject: [PATCH 03/82] fix(cli): embed libpg-query wasm in compiled binary --- .../scripts/build-binary.integration.test.ts | 53 +++++++++++++++++++ .../tests/fixtures/compiled-libpg-query.ts | 21 ++++++++ patches/@libpg-query__parser@17.6.10.patch | 17 ++++++ pnpm-lock.yaml | 7 ++- pnpm-workspace.yaml | 3 ++ 5 files changed, 99 insertions(+), 2 deletions(-) create mode 100644 apps/cli/scripts/build-binary.integration.test.ts create mode 100644 apps/cli/tests/fixtures/compiled-libpg-query.ts create mode 100644 patches/@libpg-query__parser@17.6.10.patch diff --git a/apps/cli/scripts/build-binary.integration.test.ts b/apps/cli/scripts/build-binary.integration.test.ts new file mode 100644 index 0000000000..52dc71ceb3 --- /dev/null +++ b/apps/cli/scripts/build-binary.integration.test.ts @@ -0,0 +1,53 @@ +import { afterEach, describe, expect, test } from "vitest"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const fixturePath = fileURLToPath( + new URL("../tests/fixtures/compiled-libpg-query.ts", import.meta.url), +); +const temporaryDirectories: string[] = []; + +afterEach(async () => { + await Promise.all( + temporaryDirectories.splice(0).map((directory) => rm(directory, { recursive: true })), + ); +}); + +describe("compiled binary assets", () => { + test("embeds and loads libpg-query.wasm", async () => { + const directory = await mkdtemp(path.join(tmpdir(), "supabase-compiled-wasm-")); + temporaryDirectories.push(directory); + const executable = path.join(directory, "parser-probe"); + const bunExecutable = Bun.which("bun"); + if (!bunExecutable) { + throw new Error("Bun executable not found"); + } + + const build = Bun.spawn( + [bunExecutable, "build", fixturePath, "--compile", `--outfile=${executable}`], + { stdout: "pipe", stderr: "pipe" }, + ); + const [buildExitCode, buildStderr] = await Promise.all([ + build.exited, + new Response(build.stderr).text(), + ]); + expect(buildExitCode, buildStderr).toBe(0); + + const probe = Bun.spawn([executable], { + cwd: directory, + env: {}, + stdout: "pipe", + stderr: "pipe", + }); + const [probeExitCode, stdout, stderr] = await Promise.all([ + probe.exited, + new Response(probe.stdout).text(), + new Response(probe.stderr).text(), + ]); + + expect(probeExitCode, stderr).toBe(0); + expect(stdout).toContain("libpg-query.wasm loaded"); + }, 20_000); +}); diff --git a/apps/cli/tests/fixtures/compiled-libpg-query.ts b/apps/cli/tests/fixtures/compiled-libpg-query.ts new file mode 100644 index 0000000000..f367f9e377 --- /dev/null +++ b/apps/cli/tests/fixtures/compiled-libpg-query.ts @@ -0,0 +1,21 @@ +import { validateSqlSyntax } from "@supabase/pg-topo"; +import "@supabase/pg-delta/core"; + +const embeddedParser = Bun.embeddedFiles.find((file) => file.type === "application/wasm"); + +if (!embeddedParser) { + throw new Error("libpg-query.wasm was not embedded in the executable"); +} + +const wasmBytes = new Uint8Array(await embeddedParser.arrayBuffer()); +if ( + wasmBytes[0] !== 0x00 || + wasmBytes[1] !== 0x61 || + wasmBytes[2] !== 0x73 || + wasmBytes[3] !== 0x6d +) { + throw new Error("the embedded libpg-query asset is not WebAssembly"); +} + +await validateSqlSyntax("select 1"); +console.log("libpg-query.wasm loaded"); diff --git a/patches/@libpg-query__parser@17.6.10.patch b/patches/@libpg-query__parser@17.6.10.patch new file mode 100644 index 0000000000..191d73ed04 --- /dev/null +++ b/patches/@libpg-query__parser@17.6.10.patch @@ -0,0 +1,17 @@ +diff --git a/wasm/index.js b/wasm/index.js +index 00caf4f1591549e445b97c5deeed95a9d8dabd8b..ce4a88226d12687644ca76805e08d11a6696b00e 100644 +--- a/wasm/index.js ++++ b/wasm/index.js +@@ -65,10 +65,11 @@ export function formatSqlError(error, query, options = {}) { + } + // @ts-ignore + import PgQueryModule from './libpg-query.js'; ++import libPgQueryWasmPath from './libpg-query.wasm' with { type: 'file' }; + // @ts-ignore + import { pg_query } from '../proto.js'; + let wasmModule; +-const initPromise = PgQueryModule().then((module) => { ++const initPromise = PgQueryModule({ locateFile: () => libPgQueryWasmPath }).then((module) => { + wasmModule = module; + }); + function ensureLoaded() { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index dd27a38bf0..7cba076346 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -71,6 +71,9 @@ overrides: '@effect/platform-node-shared': 4.0.0-beta.103 '@launchql/protobufjs>@types/node': 24.10.4 +patchedDependencies: + '@libpg-query/parser@17.6.10': ed67c0ca88b6ced3ec50fd6862f191d6192a246cf20d9c777d45efdb8373bed3 + importers: .: @@ -7815,7 +7818,7 @@ snapshots: '@types/node': 24.10.4 long: 5.3.2 - '@libpg-query/parser@17.6.10': + '@libpg-query/parser@17.6.10(patch_hash=ed67c0ca88b6ced3ec50fd6862f191d6192a246cf20d9c777d45efdb8373bed3)': dependencies: '@launchql/protobufjs': 7.2.6 '@pgsql/types': 17.6.2 @@ -12728,7 +12731,7 @@ snapshots: plpgsql-parser@0.5.16: dependencies: - '@libpg-query/parser': 17.6.10 + '@libpg-query/parser': 17.6.10(patch_hash=ed67c0ca88b6ced3ec50fd6862f191d6192a246cf20d9c777d45efdb8373bed3) '@pgsql/traverse': 17.2.6 '@pgsql/types': 17.6.2 pgsql-deparser: 17.18.5 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 178ee04367..1a003a6606 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -67,3 +67,6 @@ supportedArchitectures: - darwin - linux - win32 + +patchedDependencies: + '@libpg-query/parser@17.6.10': patches/@libpg-query__parser@17.6.10.patch From 1f82bf9883e4307d45ac1c0ab23477c966539f89 Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 7 Aug 2026 17:16:14 +0200 Subject: [PATCH 04/82] feat(cli): isolate pg-delta next shadow databases --- apps/cli-go/cmd/db.go | 93 ++++- apps/cli-go/cmd/db_shadow_test.go | 112 ++++++ .../internal/db/declarative/declarative.go | 3 + .../db/declarative/declarative_test.go | 22 ++ apps/cli-go/internal/db/diff/diff.go | 41 +++ apps/cli-go/internal/db/diff/diff_test.go | 80 ++++ apps/cli-go/internal/db/diff/shadow.go | 155 +++++--- apps/cli-go/internal/db/diff/shadow_test.go | 199 ++++++---- apps/cli-go/internal/db/reset/reset.go | 3 + apps/cli-go/internal/db/start/start.go | 45 ++- apps/cli-go/internal/db/start/start_test.go | 81 ++++ .../internal/db/start/templates/webhook.sql | 3 - apps/cli-go/internal/utils/edgeruntime.go | 8 +- .../utils/templates/initial_schemas/14.sql | 14 - apps/cli/package.json | 4 +- .../commands/db/diff/diff.integration.test.ts | 1 + .../commands/db/pull/pull.integration.test.ts | 1 + ...eclarative.orchestrate.integration.test.ts | 1 + .../generate/generate.integration.test.ts | 1 + .../declarative/sync/sync.integration.test.ts | 1 + .../legacy-pgdelta-engine.layer.unit.test.ts | 1 + .../legacy-pgdelta-engine.next.layer.ts | 49 ++- ...acy-pgdelta-engine.next.layer.unit.test.ts | 12 + .../legacy-pgdelta-next-adapter.layer.ts | 12 +- .../legacy-pgdelta-next-adapter.unit.test.ts | 57 +++ .../legacy-pgdelta-next-shadow.layer.ts | 37 +- .../legacy-pgdelta-next-shadow.service.ts | 9 +- .../legacy-pgdelta-next-shadow.unit.test.ts | 156 +++----- .../shared/legacy-pgdelta-next.live.test.ts | 347 ++++++++++++++++++ .../db/shared/legacy-pgdelta.cache.ts | 3 + .../shared/legacy-pgdelta.cache.unit.test.ts | 6 +- .../db/shared/legacy-pgdelta.seam.layer.ts | 201 ++++++++-- .../legacy-pgdelta.seam.layer.unit.test.ts | 232 +++++++++++- .../db/shared/legacy-pgdelta.seam.service.ts | 30 +- .../src/legacy/commands/start/lib/db-setup.ts | 34 +- .../commands/start/lib/db-setup.unit.test.ts | 37 ++ .../services/postgres.service.unit.test.ts | 5 + .../templates/db-initial-schema-14.sql.ts | 14 - .../start/templates/db-webhook.sql.ts | 3 - pnpm-lock.yaml | 22 +- 40 files changed, 1752 insertions(+), 383 deletions(-) create mode 100644 apps/cli-go/cmd/db_shadow_test.go create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts diff --git a/apps/cli-go/cmd/db.go b/apps/cli-go/cmd/db.go index fedb35b30c..be733771e6 100644 --- a/apps/cli-go/cmd/db.go +++ b/apps/cli-go/cmd/db.go @@ -1,9 +1,12 @@ package cmd import ( + "bufio" "context" + "encoding/json" "errors" "fmt" + "io" "os" "path" "path/filepath" @@ -31,6 +34,78 @@ import ( "github.com/supabase/cli/pkg/migration" ) +type pgDeltaNextShadowEndpoint struct { + ContainerID string `json:"containerId"` + URL string `json:"url"` +} + +type pgDeltaNextShadowHandoff struct { + Migrations pgDeltaNextShadowEndpoint `json:"migrations"` + Declarative pgDeltaNextShadowEndpoint `json:"declarative"` +} + +// handoffPgDeltaNextShadow transfers cleanup ownership only after the caller +// has received and acknowledged the complete JSON description. Until then Go +// removes both containers on every exit path, including cancellation and I/O +// failure. +func handoffPgDeltaNextShadow(ctx context.Context, shadow diff.PgDeltaNextShadow, in io.Reader, out io.Writer, remove func(string)) error { + transferred := false + defer func() { + if transferred { + return + } + remove(shadow.Migrations.Container) + remove(shadow.Declarative.Container) + }() + + payload := pgDeltaNextShadowHandoff{ + Migrations: pgDeltaNextShadowEndpoint{ + ContainerID: shadow.Migrations.Container, + URL: utils.ToPostgresURLWithoutPassword(shadow.Migrations.Config), + }, + Declarative: pgDeltaNextShadowEndpoint{ + ContainerID: shadow.Declarative.Container, + URL: utils.ToPostgresURLWithoutPassword(shadow.Declarative.Config), + }, + } + if err := json.NewEncoder(out).Encode(payload); err != nil { + return fmt.Errorf("failed to encode pg-delta shadow handoff: %w", err) + } + if flusher, ok := out.(interface{ Flush() error }); ok { + if err := flusher.Flush(); err != nil { + return fmt.Errorf("failed to flush pg-delta shadow handoff: %w", err) + } + } + + type readResult struct { + line string + err error + } + result := make(chan readResult, 1) + go func() { + line, err := bufio.NewReader(in).ReadString('\n') + result <- readResult{line: line, err: err} + }() + + select { + case <-ctx.Done(): + return ctx.Err() + case read := <-result: + if err := ctx.Err(); err != nil { + return err + } + if read.err != nil { + return fmt.Errorf("failed to read pg-delta shadow handoff acknowledgment: %w", read.err) + } + if read.line != "ack\n" { + return fmt.Errorf("unexpected pg-delta shadow handoff acknowledgment %q", read.line) + } + } + + transferred = true + return nil +} + var ( dbCmd = &cobra.Command{ GroupID: groupLocalDev, @@ -208,13 +283,12 @@ var ( shadowProjectRef string // dbShadowCmd is a hidden seam used by the native-TypeScript db diff/pull - // commands to provision the throwaway shadow database that the diff "source" - // runs against, then leave it running so the TS caller can run the differ - // (migra or pg-delta) itself and remove the container afterwards. It prints - // three newline-separated lines to stdout: the container id, the source - // Postgres URL, and an optional target-override URL (empty unless the - // local-target declarative branch redirects the diff target to a second - // shadow database). The URLs are emitted WITHOUT the password + // commands to provision throwaway shadow databases, then leave them running + // so the TS caller can run the differ itself and remove the containers + // afterwards. Legacy modes print three newline-separated lines. pgdelta-next + // emits a JSON object describing its two isolated clusters, then retains + // cleanup ownership until the caller acknowledges receipt. URLs are emitted + // WITHOUT the password // (ToPostgresURLWithoutPassword) so we never log a credential to stdout // (CWE-312); the TS caller re-injects the local Postgres password it already // resolves from config.toml, which is the same value the shadow uses. Shadow @@ -252,10 +326,7 @@ var ( if err != nil { return err } - fmt.Println(nextShadow.Container) - fmt.Println(utils.ToPostgresURLWithoutPassword(nextShadow.Migrated)) - fmt.Println(utils.ToPostgresURLWithoutPassword(nextShadow.Scratch)) - return nil + return handoffPgDeltaNextShadow(cmd.Context(), nextShadow, os.Stdin, os.Stdout, utils.DockerRemove) } var src diff.ShadowSource var err error diff --git a/apps/cli-go/cmd/db_shadow_test.go b/apps/cli-go/cmd/db_shadow_test.go new file mode 100644 index 0000000000..b5df3742af --- /dev/null +++ b/apps/cli-go/cmd/db_shadow_test.go @@ -0,0 +1,112 @@ +package cmd + +import ( + "bytes" + "context" + "errors" + "io" + "strings" + "testing" + + "github.com/jackc/pgconn" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/supabase/cli/internal/db/diff" +) + +func TestHandoffPgDeltaNextShadowTransfersOwnershipAfterAck(t *testing.T) { + shadow := testPgDeltaNextShadow() + var output bytes.Buffer + var removed []string + + err := handoffPgDeltaNextShadow(context.Background(), shadow, strings.NewReader("ack\n"), &output, func(container string) { + removed = append(removed, container) + }) + + require.NoError(t, err) + assert.Equal(t, "{\"migrations\":{\"containerId\":\"migrations-container\",\"url\":\"postgresql://postgres@migrations-host:6543/postgres?connect_timeout=10\"},\"declarative\":{\"containerId\":\"declarative-container\",\"url\":\"postgresql://postgres@declarative-host:7654/postgres?connect_timeout=10\"}}\n", output.String()) + assert.Empty(t, removed) +} + +func TestHandoffPgDeltaNextShadowRetainsOwnershipOnHandshakeFailure(t *testing.T) { + tests := []struct { + name string + input string + wantErr string + }{ + {name: "EOF", wantErr: "failed to read"}, + {name: "ack without newline", input: "ack", wantErr: "failed to read"}, + {name: "bad acknowledgment", input: "nope\n", wantErr: "unexpected"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var removed []string + err := handoffPgDeltaNextShadow(context.Background(), testPgDeltaNextShadow(), strings.NewReader(tt.input), io.Discard, func(container string) { + removed = append(removed, container) + }) + + assert.ErrorContains(t, err, tt.wantErr) + assert.Equal(t, []string{"migrations-container", "declarative-container"}, removed) + }) + } +} + +func TestHandoffPgDeltaNextShadowCleansBothOnEncodingFailure(t *testing.T) { + var removed []string + err := handoffPgDeltaNextShadow(context.Background(), testPgDeltaNextShadow(), strings.NewReader("ack\n"), failingWriter{}, func(container string) { + removed = append(removed, container) + }) + + assert.ErrorContains(t, err, "failed to encode") + assert.Equal(t, []string{"migrations-container", "declarative-container"}, removed) +} + +func TestHandoffPgDeltaNextShadowCleansBothOnCancellation(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + reader, writer := io.Pipe() + cancel() + t.Cleanup(func() { + _ = reader.Close() + _ = writer.Close() + }) + var removed []string + + err := handoffPgDeltaNextShadow(ctx, testPgDeltaNextShadow(), reader, io.Discard, func(container string) { + removed = append(removed, container) + }) + + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, []string{"migrations-container", "declarative-container"}, removed) +} + +func testPgDeltaNextShadow() diff.PgDeltaNextShadow { + return diff.PgDeltaNextShadow{ + Migrations: diff.PgDeltaNextShadowDatabase{ + Container: "migrations-container", + Config: pgconn.Config{ + Host: "migrations-host", + Port: 6543, + User: "postgres", + Password: "must-not-be-emitted", + Database: "postgres", + }, + }, + Declarative: diff.PgDeltaNextShadowDatabase{ + Container: "declarative-container", + Config: pgconn.Config{ + Host: "declarative-host", + Port: 7654, + User: "postgres", + Password: "must-not-be-emitted", + Database: "postgres", + }, + }, + } +} + +type failingWriter struct{} + +func (failingWriter) Write([]byte) (int, error) { + return 0, errors.New("write failed") +} diff --git a/apps/cli-go/internal/db/declarative/declarative.go b/apps/cli-go/internal/db/declarative/declarative.go index b84087bf9f..881db395a5 100644 --- a/apps/cli-go/internal/db/declarative/declarative.go +++ b/apps/cli-go/internal/db/declarative/declarative.go @@ -678,6 +678,7 @@ func baselineVersionToken() string { // // - the Postgres image (initSchema content); // - the service toggles that gate initSchema — auth/storage/realtime; +// - experimental.webhooks.enabled (conditional pg_net installation); // - api.auto_expose_new_tables (ApplyApiPrivileges default ACLs); // - vault secret names (UpsertVaultSecrets); // - supabase/roles.sql (SeedGlobals). @@ -691,6 +692,8 @@ func setupInputsToken(fsys afero.Fs) (string, error) { // initSchema conditionally provisions these service schemas. fmt.Fprintf(h, "auth=%t storage=%t realtime=%t\n", utils.Config.Auth.Enabled, utils.Config.Storage.Enabled, utils.Config.Realtime.Enabled) + webhooksEnabled := utils.Config.Experimental.Webhooks != nil && utils.Config.Experimental.Webhooks.Enabled + fmt.Fprintf(h, "database_webhooks=%t\n", webhooksEnabled) // api.auto_expose_new_tables drives ApplyApiPrivileges (default ACLs). Key on the // effective value, not the raw tri-state: as of the 2026-05-30 flip an unset flag // resolves to the same revoke-by-default baseline as explicit false (see diff --git a/apps/cli-go/internal/db/declarative/declarative_test.go b/apps/cli-go/internal/db/declarative/declarative_test.go index cbd67d29de..50a33bff02 100644 --- a/apps/cli-go/internal/db/declarative/declarative_test.go +++ b/apps/cli-go/internal/db/declarative/declarative_test.go @@ -7,6 +7,7 @@ import ( "path/filepath" "strings" "testing" + "testing/fstest" "github.com/jackc/pgconn" "github.com/jackc/pgx/v4" @@ -468,6 +469,27 @@ func TestBaselineCatalogKeyVariesWithServiceToggles(t *testing.T) { assert.NotEqual(t, on, off, "toggling a service must change the baseline cache key") } +func TestBaselineCatalogKeyVariesWithDatabaseWebhooks(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + fSys := afero.NewMemMapFs() + + disabled := config.NewConfig() + utils.Config = disabled + disabledKey, err := baselineCatalogKey(fSys) + require.NoError(t, err) + + enabled := config.NewConfig() + require.NoError(t, enabled.Load("config.toml", fstest.MapFS{ + "config.toml": &fstest.MapFile{Data: []byte("[experimental.webhooks]\nenabled = true\n")}, + })) + utils.Config = enabled + enabledKey, err := baselineCatalogKey(fSys) + require.NoError(t, err) + + assert.NotEqual(t, disabledKey, enabledKey, "Database Webhooks must change the baseline cache key") +} + func TestDeclarativeCatalogCacheKeyVariesWithSetupInputs(t *testing.T) { // The declarative target is built on the platform baseline, so its cache key // must change when setup inputs change even if the declarative SQL does not. diff --git a/apps/cli-go/internal/db/diff/diff.go b/apps/cli-go/internal/db/diff/diff.go index 32fabb2c21..b06ac78284 100644 --- a/apps/cli-go/internal/db/diff/diff.go +++ b/apps/cli-go/internal/db/diff/diff.go @@ -192,6 +192,47 @@ func SetupShadowDatabase(ctx context.Context, container string, fsys afero.Fs, o return setupShadowConn(ctx, conn, container, fsys) } +var pgDeltaNextDeclarativeExtensionDrops = []struct { + name string + sql string +}{ + {name: "pgcrypto", sql: "DROP EXTENSION IF EXISTS pgcrypto"}, + {name: "uuid-ossp", sql: `DROP EXTENSION IF EXISTS "uuid-ossp"`}, +} + +// SetupPgDeltaNextDeclarativeShadowDatabase provisions cluster B with the +// platform baseline but without activating user-managed extensions. Those +// extensions must come exclusively from declarative SQL so deleting their files +// can produce DROP EXTENSION plans. +func SetupPgDeltaNextDeclarativeShadowDatabase(ctx context.Context, container string, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error { + if utils.Config.Db.MajorVersion != 17 { + return errors.Errorf( + "pg-delta declarative shadow baseline requires Postgres 17 (got major %d, image %q)", + utils.Config.Db.MajorVersion, + utils.Config.Db.Image, + ) + } + conn, err := ConnectShadowDatabase(ctx, 10*time.Second, options...) + if err != nil { + return err + } + defer conn.Close(context.Background()) + if err := start.SetupDatabase(ctx, conn, container[:12], os.Stderr, fsys, start.WithoutUserExtensionActivation()); err != nil { + return err + } + for _, extension := range pgDeltaNextDeclarativeExtensionDrops { + if _, err := conn.Exec(ctx, extension.sql); err != nil { + return errors.Errorf( + "failed to remove user-managed extension %q from pg-delta declarative shadow baseline (image %q): %w", + extension.name, + utils.Config.Db.Image, + err, + ) + } + } + return nil +} + func MigrateShadowDatabase(ctx context.Context, container string, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error { migrations, err := migration.ListLocalMigrations(utils.MigrationsDir, afero.NewIOFS(fsys)) if err != nil { diff --git a/apps/cli-go/internal/db/diff/diff_test.go b/apps/cli-go/internal/db/diff/diff_test.go index aff3242699..7e92f23ef8 100644 --- a/apps/cli-go/internal/db/diff/diff_test.go +++ b/apps/cli-go/internal/db/diff/diff_test.go @@ -8,6 +8,7 @@ import ( "os" "path/filepath" "testing" + stdfs "testing/fstest" "time" "github.com/docker/docker/api/types" @@ -404,6 +405,85 @@ func TestSetupShadowDatabase(t *testing.T) { }) } +func TestSetupPgDeltaNextDeclarativeShadowDatabase(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + + newPg17Config := func(t *testing.T) { + t.Helper() + cfg := pkgconfig.NewConfig() + require.NoError(t, cfg.Load("config.toml", stdfs.MapFS{ + "config.toml": &stdfs.MapFile{Data: []byte("[experimental.webhooks]\nenabled = true\n")}, + })) + cfg.Db.MajorVersion = 17 + cfg.Db.Image = "public.ecr.aws/supabase/postgres:17.6.1.104" + cfg.Db.ShadowPort = 54320 + cfg.Realtime.Enabled = false + cfg.Storage.Enabled = false + cfg.Auth.Enabled = false + utils.Config = cfg + } + + t.Run("provisions PG17 without activating user-managed extensions", func(t *testing.T) { + newPg17Config(t) + conn := pgtest.NewConn() + defer conn.Close(t) + helper.MockApiPrivilegesRevoke(conn). + Query("DROP EXTENSION IF EXISTS pgcrypto"). + Reply("DROP EXTENSION"). + Query(`DROP EXTENSION IF EXISTS "uuid-ossp"`). + Reply("DROP EXTENSION") + + err := SetupPgDeltaNextDeclarativeShadowDatabase( + context.Background(), + "declarative-container", + afero.NewMemMapFs(), + conn.Intercept, + ) + + require.NoError(t, err) + }) + + t.Run("identifies an extension whose non-cascade drop fails", func(t *testing.T) { + newPg17Config(t) + conn := pgtest.NewConn() + defer conn.Close(t) + helper.MockApiPrivilegesRevoke(conn). + Query("DROP EXTENSION IF EXISTS pgcrypto"). + ReplyError(pgerrcode.DependentObjectsStillExist, `cannot drop extension pgcrypto because other objects depend on it`) + + err := SetupPgDeltaNextDeclarativeShadowDatabase( + context.Background(), + "declarative-container", + afero.NewMemMapFs(), + conn.Intercept, + ) + + require.Error(t, err) + assert.ErrorContains(t, err, `user-managed extension "pgcrypto"`) + assert.ErrorContains(t, err, `public.ecr.aws/supabase/postgres:17.6.1.104`) + assert.ErrorContains(t, err, "SQLSTATE 2BP01") + }) + + t.Run("rejects unaudited Postgres majors", func(t *testing.T) { + cfg := pkgconfig.NewConfig() + cfg.Db.MajorVersion = 14 + cfg.Db.Image = "public.ecr.aws/supabase/postgres:14.1.0" + utils.Config = cfg + + err := SetupPgDeltaNextDeclarativeShadowDatabase( + context.Background(), + "declarative-container", + afero.NewMemMapFs(), + ) + + require.Error(t, err) + assert.ErrorContains(t, err, "requires Postgres 17") + assert.ErrorContains(t, err, "major 14") + assert.ErrorContains(t, err, "public.ecr.aws/supabase/postgres:14.1.0") + }) +} + func TestDiffDatabase(t *testing.T) { utils.Config.Db.MajorVersion = 14 utils.Config.Db.ShadowPort = 54320 diff --git a/apps/cli-go/internal/db/diff/shadow.go b/apps/cli-go/internal/db/diff/shadow.go index eba3fb2358..aa32c91e74 100644 --- a/apps/cli-go/internal/db/diff/shadow.go +++ b/apps/cli-go/internal/db/diff/shadow.go @@ -2,11 +2,12 @@ package diff import ( "context" + "fmt" + "math" "time" "github.com/jackc/pgconn" "github.com/jackc/pgx/v4" - "github.com/pkg/errors" "github.com/spf13/afero" "github.com/supabase/cli/internal/db/start" "github.com/supabase/cli/internal/pgdelta" @@ -30,87 +31,135 @@ type ShadowSource struct { TargetOverride *pgconn.Config } -// PgDeltaNextShadow is a provisioned shadow container exposing both database -// states needed by the native pg-delta engine. Migrated contains the platform -// baseline plus local migrations. Scratch is an empty sibling database owned -// by pg-delta's declarative planner while it loads the desired schema files. -type PgDeltaNextShadow struct { - // Container is left running for the caller, which MUST remove it after use. +// PgDeltaNextShadowDatabase is one isolated database state used by pg-delta. +// Container is left running for the caller, which MUST remove it after use. +type PgDeltaNextShadowDatabase struct { Container string - Migrated pgconn.Config - Scratch pgconn.Config + Config pgconn.Config } -type pgDeltaNextShadowDependencies struct { - create func(context.Context, uint16) (string, error) - wait func(context.Context, time.Duration, ...string) error - migrate func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error - createScratch func(context.Context, ...func(*pgx.ConnConfig)) error - remove func(string) +// PgDeltaNextShadow contains the two isolated clusters used by the native +// pg-delta engine. Migrations has the platform baseline plus local migrations; +// Declarative has the same platform baseline and local configuration, ready for +// pg-delta to load declarative SQL into postgres. +type PgDeltaNextShadow struct { + Migrations PgDeltaNextShadowDatabase + Declarative PgDeltaNextShadowDatabase } -const createPgDeltaNextScratch = "CREATE DATABASE pgdelta_declarative TEMPLATE template0" - -// createPgDeltaNextScratchDatabase creates the empty same-cluster database that -// planSchemaFiles owns. Using template0 guarantees it does not inherit the -// platform baseline or local migrations from postgres. -func createPgDeltaNextScratchDatabase(ctx context.Context, options ...func(*pgx.ConnConfig)) error { - conn, err := ConnectShadowDatabase(ctx, 10*time.Second, options...) - if err != nil { - return err - } - defer conn.Close(context.Background()) - if _, err := conn.Exec(ctx, createPgDeltaNextScratch); err != nil { - return errors.Wrap(err, "failed to create pg-delta declarative scratch database") - } - return nil +type pgDeltaNextShadowDependencies struct { + freePort func() (int, error) + create func(context.Context, uint16) (string, error) + wait func(context.Context, time.Duration, ...string) error + migrate func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error + setup func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error + remove func(string) } -// PreparePgDeltaNextShadow provisions the migrated target and an empty live -// sibling database used by the native pg-delta declarative planner. It never -// loads or applies the legacy declarative schemas. On failure, the container is -// removed best-effort without replacing the provisioning error. +// PreparePgDeltaNextShadow provisions isolated migrated and declarative +// clusters. On failure, every container created so far is removed best-effort +// without replacing the provisioning error. func PreparePgDeltaNextShadow(ctx context.Context, fsys afero.Fs, options ...func(*pgx.ConnConfig)) (PgDeltaNextShadow, error) { return preparePgDeltaNextShadow(ctx, fsys, pgDeltaNextShadowDependencies{ - create: CreateShadowDatabase, - wait: start.WaitForHealthyService, - migrate: MigrateShadowDatabase, - createScratch: createPgDeltaNextScratchDatabase, - remove: utils.DockerRemove, + freePort: utils.GetFreeHostPort, + create: CreateShadowDatabase, + wait: start.WaitForHealthyService, + migrate: MigrateShadowDatabase, + setup: SetupPgDeltaNextDeclarativeShadowDatabase, + remove: utils.DockerRemove, }, options...) } func preparePgDeltaNextShadow(ctx context.Context, fsys afero.Fs, dependencies pgDeltaNextShadowDependencies, options ...func(*pgx.ConnConfig)) (PgDeltaNextShadow, error) { - shadow, err := dependencies.create(ctx, utils.Config.Db.ShadowPort) - if err != nil { - return PgDeltaNextShadow{}, err - } + var containers []string ok := false defer func() { if !ok { - dependencies.remove(shadow) + for _, container := range containers { + dependencies.remove(container) + } } }() - if err := dependencies.wait(ctx, utils.Config.Db.HealthTimeout, shadow); err != nil { + + migrationsPort, err := allocatePgDeltaNextPort(dependencies.freePort, 0) + if err != nil { + return PgDeltaNextShadow{}, err + } + migrationsContainer, err := dependencies.create(ctx, migrationsPort) + if migrationsContainer != "" { + containers = append(containers, migrationsContainer) + } + if err != nil { + return PgDeltaNextShadow{}, err + } + if err := dependencies.wait(ctx, utils.Config.Db.HealthTimeout, migrationsContainer); err != nil { return PgDeltaNextShadow{}, err } - if err := dependencies.migrate(ctx, shadow, fsys, options...); err != nil { + if err := dependencies.migrate(ctx, migrationsContainer, fsys, append(options, withShadowPort(migrationsPort))...); err != nil { return PgDeltaNextShadow{}, err } - if err := dependencies.createScratch(ctx, options...); err != nil { + + declarativePort, err := allocatePgDeltaNextPort(dependencies.freePort, migrationsPort) + if err != nil { return PgDeltaNextShadow{}, err } - migrated := pgconn.Config{ + declarativeContainer, err := dependencies.create(ctx, declarativePort) + if declarativeContainer != "" { + containers = append(containers, declarativeContainer) + } + if err != nil { + return PgDeltaNextShadow{}, err + } + if err := dependencies.wait(ctx, utils.Config.Db.HealthTimeout, declarativeContainer); err != nil { + return PgDeltaNextShadow{}, err + } + if err := dependencies.setup(ctx, declarativeContainer, fsys, append(options, withShadowPort(declarativePort))...); err != nil { + return PgDeltaNextShadow{}, err + } + + ok = true + return PgDeltaNextShadow{ + Migrations: PgDeltaNextShadowDatabase{ + Container: migrationsContainer, + Config: pgDeltaNextShadowConfig(migrationsPort), + }, + Declarative: PgDeltaNextShadowDatabase{ + Container: declarativeContainer, + Config: pgDeltaNextShadowConfig(declarativePort), + }, + }, nil +} + +func allocatePgDeltaNextPort(freePort func() (int, error), excluded uint16) (uint16, error) { + for range 10 { + port, err := freePort() + if err != nil { + return 0, err + } + if port <= 0 || port > math.MaxUint16 { + return 0, fmt.Errorf("allocated host port %d is outside the valid range", port) + } + if uint16(port) != excluded { + return uint16(port), nil + } + } + return 0, fmt.Errorf("failed to allocate a host port distinct from %d", excluded) +} + +func withShadowPort(port uint16) func(*pgx.ConnConfig) { + return func(config *pgx.ConnConfig) { + config.Port = port + } +} + +func pgDeltaNextShadowConfig(port uint16) pgconn.Config { + return pgconn.Config{ Host: utils.Config.Hostname, - Port: utils.Config.Db.ShadowPort, + Port: port, User: "postgres", Password: utils.Config.Db.Password, Database: "postgres", } - scratch := migrated - scratch.Database = "pgdelta_declarative" - ok = true - return PgDeltaNextShadow{Container: shadow, Migrated: migrated, Scratch: scratch}, nil } // PrepareShadowSource provisions the shadow database that DiffDatabase diffs diff --git a/apps/cli-go/internal/db/diff/shadow_test.go b/apps/cli-go/internal/db/diff/shadow_test.go index 6cb1d400bd..9d293c70dc 100644 --- a/apps/cli-go/internal/db/diff/shadow_test.go +++ b/apps/cli-go/internal/db/diff/shadow_test.go @@ -17,100 +17,165 @@ func TestPreparePgDeltaNextShadow(t *testing.T) { originalConfig := utils.Config t.Cleanup(func() { utils.Config = originalConfig }) utils.Config.Hostname = "shadow-host" - utils.Config.Db.ShadowPort = 6543 utils.Config.Db.Password = "secret" utils.Config.Db.HealthTimeout = 7 * time.Second - var waitedContainer string - var migratedContainer string - var scratchCreated bool - var removedContainer string + ports := []int{6543, 7654} + var createdPorts []uint16 + var waitedContainers []string + var migratedPort uint16 + var setupPort uint16 + var removedContainers []string dependencies := pgDeltaNextShadowDependencies{ + freePort: func() (int, error) { + port := ports[0] + ports = ports[1:] + return port, nil + }, create: func(_ context.Context, port uint16) (string, error) { - assert.Equal(t, uint16(6543), port) - return "shadow-container", nil + createdPorts = append(createdPorts, port) + if port == 6543 { + return "migrations-container", nil + } + return "declarative-container", nil }, wait: func(_ context.Context, timeout time.Duration, containers ...string) error { assert.Equal(t, 7*time.Second, timeout) require.Len(t, containers, 1) - waitedContainer = containers[0] + waitedContainers = append(waitedContainers, containers[0]) return nil }, - migrate: func(_ context.Context, container string, _ afero.Fs, _ ...func(*pgx.ConnConfig)) error { - migratedContainer = container + migrate: func(_ context.Context, container string, _ afero.Fs, options ...func(*pgx.ConnConfig)) error { + assert.Equal(t, "migrations-container", container) + config := &pgx.ConnConfig{} + for _, option := range options { + option(config) + } + migratedPort = config.Port return nil }, - createScratch: func(_ context.Context, _ ...func(*pgx.ConnConfig)) error { - scratchCreated = true + setup: func(_ context.Context, container string, _ afero.Fs, options ...func(*pgx.ConnConfig)) error { + assert.Equal(t, "declarative-container", container) + config := &pgx.ConnConfig{} + for _, option := range options { + option(config) + } + setupPort = config.Port return nil }, - remove: func(container string) { removedContainer = container }, + remove: func(container string) { removedContainers = append(removedContainers, container) }, } result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) require.NoError(t, err) - assert.Equal(t, "shadow-container", result.Container) - assert.Equal(t, "shadow-container", waitedContainer) - assert.Equal(t, "shadow-container", migratedContainer) - assert.True(t, scratchCreated) - assert.Empty(t, removedContainer) - assert.Equal(t, "shadow-host", result.Migrated.Host) - assert.Equal(t, uint16(6543), result.Migrated.Port) - assert.Equal(t, "postgres", result.Migrated.User) - assert.Equal(t, "secret", result.Migrated.Password) - assert.Equal(t, "postgres", result.Migrated.Database) - assert.Equal(t, result.Migrated.Host, result.Scratch.Host) - assert.Equal(t, result.Migrated.Port, result.Scratch.Port) - assert.Equal(t, result.Migrated.User, result.Scratch.User) - assert.Equal(t, result.Migrated.Password, result.Scratch.Password) - assert.Equal(t, "pgdelta_declarative", result.Scratch.Database) + assert.Equal(t, []uint16{6543, 7654}, createdPorts) + assert.Equal(t, []string{"migrations-container", "declarative-container"}, waitedContainers) + assert.Equal(t, uint16(6543), migratedPort) + assert.Equal(t, uint16(7654), setupPort) + assert.Empty(t, removedContainers) + assert.Equal(t, "migrations-container", result.Migrations.Container) + assert.Equal(t, "declarative-container", result.Declarative.Container) + assert.Equal(t, pgDeltaNextShadowConfig(6543), result.Migrations.Config) + assert.Equal(t, pgDeltaNextShadowConfig(7654), result.Declarative.Config) + assert.Equal(t, "postgres", result.Migrations.Config.Database) + assert.Equal(t, "postgres", result.Declarative.Config.Database) } -func TestPreparePgDeltaNextShadowRemovesContainerAfterFailure(t *testing.T) { - originalConfig := utils.Config - t.Cleanup(func() { utils.Config = originalConfig }) - wantErr := errors.New("migration failed") - var removedContainer string - dependencies := pgDeltaNextShadowDependencies{ - create: func(context.Context, uint16) (string, error) { - return "failed-shadow", nil - }, - wait: func(context.Context, time.Duration, ...string) error { return nil }, - migrate: func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error { - return wantErr - }, - createScratch: func(context.Context, ...func(*pgx.ConnConfig)) error { return nil }, - remove: func(container string) { removedContainer = container }, +func TestPreparePgDeltaNextShadowRemovesEveryCreatedContainerOnFailure(t *testing.T) { + wantErr := errors.New("provisioning failed") + tests := []struct { + name string + failAt string + firstID string + secondID string + wantRemoved []string + }{ + {name: "first port", failAt: "first-port"}, + {name: "first create without id", failAt: "first-create"}, + {name: "first create with id", failAt: "first-create", firstID: "migrations", wantRemoved: []string{"migrations"}}, + {name: "first health", failAt: "first-health", firstID: "migrations", wantRemoved: []string{"migrations"}}, + {name: "migrations", failAt: "migrate", firstID: "migrations", wantRemoved: []string{"migrations"}}, + {name: "second port", failAt: "second-port", firstID: "migrations", wantRemoved: []string{"migrations"}}, + {name: "second create without id", failAt: "second-create", firstID: "migrations", wantRemoved: []string{"migrations"}}, + {name: "second create with id", failAt: "second-create", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"migrations", "declarative"}}, + {name: "second health", failAt: "second-health", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"migrations", "declarative"}}, + {name: "declarative setup", failAt: "setup", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"migrations", "declarative"}}, } - result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + portCalls := 0 + createCalls := 0 + var removed []string + dependencies := pgDeltaNextShadowDependencies{ + freePort: func() (int, error) { + portCalls++ + if (portCalls == 1 && tt.failAt == "first-port") || (portCalls == 2 && tt.failAt == "second-port") { + return 0, wantErr + } + return 6000 + portCalls, nil + }, + create: func(context.Context, uint16) (string, error) { + createCalls++ + if createCalls == 1 { + if tt.failAt == "first-create" { + return tt.firstID, wantErr + } + return tt.firstID, nil + } + if tt.failAt == "second-create" { + return tt.secondID, wantErr + } + return tt.secondID, nil + }, + wait: func(_ context.Context, _ time.Duration, containers ...string) error { + if (containers[0] == tt.firstID && tt.failAt == "first-health") || (containers[0] == tt.secondID && tt.failAt == "second-health") { + return wantErr + } + return nil + }, + migrate: func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error { + if tt.failAt == "migrate" { + return wantErr + } + return nil + }, + setup: func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error { + if tt.failAt == "setup" { + return wantErr + } + return nil + }, + remove: func(container string) { removed = append(removed, container) }, + } - assert.ErrorIs(t, err, wantErr) - assert.Empty(t, result.Container) - assert.Equal(t, "failed-shadow", removedContainer) -} + result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) -func TestPreparePgDeltaNextShadowRemovesContainerAfterScratchFailure(t *testing.T) { - originalConfig := utils.Config - t.Cleanup(func() { utils.Config = originalConfig }) - wantErr := errors.New("scratch creation failed") - var removedContainer string - dependencies := pgDeltaNextShadowDependencies{ - create: func(context.Context, uint16) (string, error) { - return "failed-scratch-shadow", nil - }, - wait: func(context.Context, time.Duration, ...string) error { return nil }, - migrate: func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error { - return nil - }, - createScratch: func(context.Context, ...func(*pgx.ConnConfig)) error { return wantErr }, - remove: func(container string) { removedContainer = container }, + assert.ErrorIs(t, err, wantErr) + assert.Empty(t, result) + assert.Equal(t, tt.wantRemoved, removed) + }) } +} - result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) +func TestAllocatePgDeltaNextPort(t *testing.T) { + t.Run("retries a duplicate port", func(t *testing.T) { + ports := []int{6543, 6544} + port, err := allocatePgDeltaNextPort(func() (int, error) { + result := ports[0] + ports = ports[1:] + return result, nil + }, 6543) - assert.ErrorIs(t, err, wantErr) - assert.Empty(t, result.Container) - assert.Equal(t, "failed-scratch-shadow", removedContainer) + require.NoError(t, err) + assert.Equal(t, uint16(6544), port) + }) + + for _, port := range []int{-1, 0, 65536} { + t.Run("rejects invalid port", func(t *testing.T) { + _, err := allocatePgDeltaNextPort(func() (int, error) { return port, nil }, 0) + assert.ErrorContains(t, err, "outside the valid range") + }) + } } diff --git a/apps/cli-go/internal/db/reset/reset.go b/apps/cli-go/internal/db/reset/reset.go index 7cfe42ff27..eb4709b71c 100644 --- a/apps/cli-go/internal/db/reset/reset.go +++ b/apps/cli-go/internal/db/reset/reset.go @@ -182,6 +182,9 @@ func initDatabase(ctx context.Context, options ...func(*pgx.ConnConfig)) error { if err := start.InitSchema14(ctx, conn); err != nil { return err } + if err := start.ApplyDatabaseWebhooks(ctx, conn); err != nil { + return err + } return start.ApplyApiPrivileges(ctx, conn) } diff --git a/apps/cli-go/internal/db/start/start.go b/apps/cli-go/internal/db/start/start.go index 6cd411791c..13845b1b90 100644 --- a/apps/cli-go/internal/db/start/start.go +++ b/apps/cli-go/internal/db/start/start.go @@ -380,10 +380,36 @@ func SetupLocalDatabase(ctx context.Context, version string, fsys afero.Fs, w io return nil } -func SetupDatabase(ctx context.Context, conn *pgx.Conn, host string, w io.Writer, fsys afero.Fs) error { +type setupDatabaseOptions struct { + activateUserExtensions bool +} + +// SetupDatabaseOption customises platform setup for specialised database +// provisioning paths while keeping the ordinary local setup defaults. +type SetupDatabaseOption func(*setupDatabaseOptions) + +// WithoutUserExtensionActivation keeps platform capabilities such as the +// webhook helpers and event trigger, but leaves activation of user-managed +// extensions such as pg_net to migrations or declarative SQL. +func WithoutUserExtensionActivation() SetupDatabaseOption { + return func(options *setupDatabaseOptions) { + options.activateUserExtensions = false + } +} + +func SetupDatabase(ctx context.Context, conn *pgx.Conn, host string, w io.Writer, fsys afero.Fs, opts ...SetupDatabaseOption) error { + options := setupDatabaseOptions{activateUserExtensions: true} + for _, option := range opts { + option(&options) + } if err := initSchema(ctx, conn, host, w); err != nil { return err } + if options.activateUserExtensions { + if err := ApplyDatabaseWebhooks(ctx, conn); err != nil { + return err + } + } if err := ApplyApiPrivileges(ctx, conn); err != nil { return err } @@ -398,6 +424,23 @@ func SetupDatabase(ctx context.Context, conn *pgx.Conn, host string, w io.Writer return err } +const EnableDatabaseWebhooksSql = `create extension if not exists pg_net schema extensions;` + +// ApplyDatabaseWebhooks installs pg_net only when the Database Webhooks feature is enabled. +// The platform webhook helpers and event trigger are part of the baseline regardless, so an +// explicit CREATE EXTENSION in user migrations/declarative SQL remains supported when disabled. +func ApplyDatabaseWebhooks(ctx context.Context, conn *pgx.Conn) error { + webhooks := utils.Config.Experimental.Webhooks + if webhooks == nil || !webhooks.Enabled { + return nil + } + file, err := migration.NewMigrationFromReader(strings.NewReader(EnableDatabaseWebhooksSql)) + if err != nil { + return err + } + return file.ExecBatch(ctx, conn) +} + // RevokeDefaultDataApiPrivilegesSql matches the SQL that Studio runs at cloud project creation // when the "Default privileges for new entities" toggle is off. It removes the default GRANTs // applied by the initial schema so newly-created entities in `public` owned by `postgres` are diff --git a/apps/cli-go/internal/db/start/start_test.go b/apps/cli-go/internal/db/start/start_test.go index 805691b8ec..ae513d7013 100644 --- a/apps/cli-go/internal/db/start/start_test.go +++ b/apps/cli-go/internal/db/start/start_test.go @@ -7,6 +7,7 @@ import ( "net/http" "os" "testing" + stdfs "testing/fstest" "github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container" @@ -20,6 +21,7 @@ import ( "github.com/supabase/cli/internal/testing/helper" "github.com/supabase/cli/internal/utils" "github.com/supabase/cli/pkg/cast" + "github.com/supabase/cli/pkg/config" "github.com/supabase/cli/pkg/pgtest" ) @@ -379,6 +381,85 @@ func TestSetupDatabase(t *testing.T) { assert.Empty(t, apitest.ListUnmatchedRequests()) }) } + +func TestApplyDatabaseWebhooks(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + + t.Run("does not install pg_net merely because Edge Runtime is enabled", func(t *testing.T) { + cfg := config.NewConfig() + cfg.EdgeRuntime.Enabled = true + utils.Config = cfg + conn := pgtest.NewConn() + defer conn.Close(t) + + require.NoError(t, ApplyDatabaseWebhooks(context.Background(), conn.MockClient(t))) + }) + + t.Run("installs pg_net when Database Webhooks is enabled even without Edge Runtime", func(t *testing.T) { + cfg := config.NewConfig() + require.NoError(t, cfg.Load("config.toml", stdfs.MapFS{ + "config.toml": &stdfs.MapFile{Data: []byte("[experimental.webhooks]\nenabled = true\n")}, + })) + cfg.EdgeRuntime.Enabled = false + utils.Config = cfg + conn := pgtest.NewConn() + defer conn.Close(t) + conn.Query("create extension if not exists pg_net schema extensions").Reply("CREATE EXTENSION") + + require.NoError(t, ApplyDatabaseWebhooks(context.Background(), conn.MockClient(t))) + }) +} + +func TestSetupDatabaseUserExtensionActivation(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + + newWebhookConfig := func(t *testing.T) { + t.Helper() + cfg := config.NewConfig() + require.NoError(t, cfg.Load("config.toml", stdfs.MapFS{ + "config.toml": &stdfs.MapFile{Data: []byte("[experimental.webhooks]\nenabled = true\n")}, + })) + cfg.Db.MajorVersion = 17 + cfg.Realtime.Enabled = false + cfg.Storage.Enabled = false + cfg.Auth.Enabled = false + utils.Config = cfg + } + + t.Run("installs pg_net by default when Database Webhooks is enabled", func(t *testing.T) { + newWebhookConfig(t) + conn := pgtest.NewConn() + defer conn.Close(t) + conn.Query("create extension if not exists pg_net schema extensions"). + Reply("CREATE EXTENSION") + helper.MockApiPrivilegesRevoke(conn) + + err := SetupDatabase(context.Background(), conn.MockClient(t), "postgres-host", io.Discard, afero.NewMemMapFs()) + + require.NoError(t, err) + }) + + t.Run("can leave user extension activation to declarative SQL", func(t *testing.T) { + newWebhookConfig(t) + conn := pgtest.NewConn() + defer conn.Close(t) + helper.MockApiPrivilegesRevoke(conn) + + err := SetupDatabase( + context.Background(), + conn.MockClient(t), + "postgres-host", + io.Discard, + afero.NewMemMapFs(), + WithoutUserExtensionActivation(), + ) + + require.NoError(t, err) + }) +} + func TestStartDatabaseWithCustomSettings(t *testing.T) { t.Run("starts database with custom MaxConnections", func(t *testing.T) { // Setup diff --git a/apps/cli-go/internal/db/start/templates/webhook.sql b/apps/cli-go/internal/db/start/templates/webhook.sql index 52cd097473..6a895256cc 100644 --- a/apps/cli-go/internal/db/start/templates/webhook.sql +++ b/apps/cli-go/internal/db/start/templates/webhook.sql @@ -1,8 +1,5 @@ BEGIN; --- Create pg_net extension -CREATE EXTENSION IF NOT EXISTS pg_net SCHEMA extensions; - -- Create supabase_functions schema CREATE SCHEMA supabase_functions AUTHORIZATION supabase_admin; diff --git a/apps/cli-go/internal/utils/edgeruntime.go b/apps/cli-go/internal/utils/edgeruntime.go index 8e54afa628..da3c3af44f 100644 --- a/apps/cli-go/internal/utils/edgeruntime.go +++ b/apps/cli-go/internal/utils/edgeruntime.go @@ -61,8 +61,10 @@ func WithExtraEnv(entries ...string) EdgeRuntimeOption { } } -// getFreeHostPort asks the OS for an unused TCP port on the host. -func getFreeHostPort() (int, error) { +// GetFreeHostPort asks the OS for an unused TCP port on the host. The listener +// is closed before the port is returned, so callers that need more than one +// port should bind each one before requesting the next. +func GetFreeHostPort() (int, error) { listener, err := net.Listen("tcp", "127.0.0.1:0") if err != nil { return 0, errors.Errorf("failed to allocate free port: %w", err) @@ -80,7 +82,7 @@ func getFreeHostPort() (int, error) { func EdgeRuntimeStartCmd() []string { cmd := []string{"edge-runtime", "start", "--main-service=."} // Skip the flag on the rare allocation failure to preserve prior behavior. - if port, err := getFreeHostPort(); err == nil { + if port, err := GetFreeHostPort(); err == nil { cmd = append(cmd, fmt.Sprintf("--port=%d", port)) } return cmd diff --git a/apps/cli-go/internal/utils/templates/initial_schemas/14.sql b/apps/cli-go/internal/utils/templates/initial_schemas/14.sql index bef44153ec..b0397cbb0b 100644 --- a/apps/cli-go/internal/utils/templates/initial_schemas/14.sql +++ b/apps/cli-go/internal/utils/templates/initial_schemas/14.sql @@ -70,20 +70,6 @@ CREATE SCHEMA IF NOT EXISTS graphql_public; ALTER SCHEMA graphql_public OWNER TO supabase_admin; --- --- Name: pg_net; Type: EXTENSION; Schema: -; Owner: - --- - -CREATE EXTENSION IF NOT EXISTS pg_net WITH SCHEMA extensions; - - --- --- Name: EXTENSION pg_net; Type: COMMENT; Schema: -; Owner: --- - -COMMENT ON EXTENSION pg_net IS 'Async HTTP'; - - -- -- Name: pgbouncer; Type: SCHEMA; Schema: -; Owner: pgbouncer -- diff --git a/apps/cli/package.json b/apps/cli/package.json index 44603081f8..b8171b32f9 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -55,8 +55,8 @@ "@parcel/watcher": "^2.6.0", "@supabase/api": "workspace:*", "@supabase/config": "workspace:*", - "@supabase/pg-delta": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb", - "@supabase/pg-topo": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb", + "@supabase/pg-delta": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb", + "@supabase/pg-topo": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb", "@supabase/process-compose": "workspace:*", "@supabase/stack": "workspace:*", "@tsconfig/bun": "catalog:", diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index f58d482dcc..04a283fd31 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -80,6 +80,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { targetUrlOverride: opts.targetOverride, }); }, + provisionNextShadow: () => Effect.die("provisionNextShadow not used"), removeShadowContainer: (container) => Effect.sync(() => { removedContainers.push(container); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 292e640e8f..dbbdea699a 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -134,6 +134,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { targetUrlOverride: opts.shadowTargetOverride, }); }, + provisionNextShadow: () => Effect.die("provisionNextShadow not used"), removeShadowContainer: (container) => Effect.sync(() => { removedContainers.push(container); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 0828abc62a..2230914cfa 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -36,6 +36,7 @@ function mockSeam(paths: Record) { ensureLocalDatabaseStarted: () => Effect.void, ensureLocalPostgresImageCurrent: () => Effect.void, provisionShadow: () => Effect.die("provisionShadow not used in declarative tests"), + provisionNextShadow: () => Effect.die("provisionNextShadow not used in declarative tests"), removeShadowContainer: () => Effect.void, }); return { layer, calls }; diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index 44e74e8202..e3c0eaa162 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -110,6 +110,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { ), ), provisionShadow: () => Effect.die("provisionShadow not used in declarative tests"), + provisionNextShadow: () => Effect.die("provisionNextShadow not used in declarative tests"), removeShadowContainer: () => Effect.void, }); const edgeCalls: LegacyEdgeRuntimeRunOpts[] = []; diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 684745a8d3..3597b17a80 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -109,6 +109,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { ), ), provisionShadow: () => Effect.die("provisionShadow not used in declarative tests"), + provisionNextShadow: () => Effect.die("provisionNextShadow not used in declarative tests"), removeShadowContainer: () => Effect.void, }); const edge = Layer.succeed(LegacyEdgeRuntimeScript, { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts index 56a8fbcea9..fbda732eed 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts @@ -52,6 +52,7 @@ const unusedLegacyRuntime = Layer.mergeAll( ensureLocalDatabaseStarted: () => Effect.die("local start not needed"), ensureLocalPostgresImageCurrent: () => Effect.die("image check not needed"), provisionShadow: () => Effect.die("shadow not needed"), + provisionNextShadow: () => Effect.die("next shadow not needed"), removeShadowContainer: () => Effect.die("cleanup not needed"), }), Layer.succeed(LegacyPgDeltaNextAdapter, { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 3250ab029d..766a453dfa 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -26,6 +26,12 @@ import { legacyPgDeltaNextBlockingDiagnosticMessage, } from "./legacy-pgdelta-next-diagnostics.ts"; +/** Shared by both declarative planner entrypoints over the full isolated baseline. */ +export const legacyPgDeltaNextIsolatedShadowPlanOptions = { + isolatedShadow: true, + seedAssumedSchemas: false, +} as const; + function legacyPgDeltaNextConnectSuggestion(cause: unknown): string | undefined { if (cause instanceof LegacyDbConnectError) return cause.suggestion; if (typeof cause !== "object" || cause === null) return undefined; @@ -165,7 +171,9 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diffExplicit: (input) => Effect.scoped( Effect.gen(function* () { - let shadow: { readonly migrationsUrl: string; readonly scratchUrl: string } | undefined; + let shadow: + | { readonly migrationsUrl: string; readonly declarativeUrl: string } + | undefined; const migrationsEndpoint = input.source.kind === "migrations" ? input.source @@ -230,8 +238,8 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), }); const migrations = parseLegacyConnectionString(shadow.migrationsUrl); - const scratch = parseLegacyConnectionString(shadow.scratchUrl); - if (migrations === undefined || scratch === undefined) { + const declarative = parseLegacyConnectionString(shadow.declarativeUrl); + if (migrations === undefined || declarative === undefined) { return yield* Effect.fail( new LegacyPgDeltaEngineError({ message: "failed to parse pg-delta next shadow database URL", @@ -239,23 +247,22 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( }), ); } - const migrationsPool = yield* legacyAcquirePgPool(migrations, { - isLocal: true, - dnsResolver: "native", - }); if (input.declarativeFiles !== undefined) { - const scratchPool = yield* legacyAcquirePgPool(scratch, { - isLocal: true, - dnsResolver: "native", - }); + const [migrationsPool, declarativePool] = yield* Effect.all( + [ + legacyAcquirePgPool(migrations, { isLocal: true, dnsResolver: "native" }), + legacyAcquirePgPool(declarative, { isLocal: true, dnsResolver: "native" }), + ], + { concurrency: 2 }, + ); const result = yield* adapter.planDeclarativeSchema({ targetPool: migrationsPool, - shadowPool: scratchPool, + shadowPool: declarativePool, files: input.declarativeFiles, allowDrops: true, debug: input.debug, reorder: true, - seedAssumedSchemas: true, + ...legacyPgDeltaNextIsolatedShadowPlanOptions, schema: input.schema, ...(input.declarativeManifest !== undefined ? { manifest: input.declarativeManifest } @@ -271,6 +278,10 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( yield* rejectBlockingDiagnostic("declarativePlan", result.diagnostics); return normalizeNextDiff(result, debugDirectory); } + const migrationsPool = yield* legacyAcquirePgPool(migrations, { + isLocal: true, + dnsResolver: "native", + }); const desiredPool = yield* acquireDatabase(input.target); const result = yield* adapter.diff({ sourcePool: migrationsPool, @@ -321,8 +332,8 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( Effect.gen(function* () { const shadow = yield* shadowService.provision({ schema: input.schema }); const migrations = parseLegacyConnectionString(shadow.migrationsUrl); - const scratch = parseLegacyConnectionString(shadow.scratchUrl); - if (migrations === undefined || scratch === undefined) { + const declarative = parseLegacyConnectionString(shadow.declarativeUrl); + if (migrations === undefined || declarative === undefined) { return yield* Effect.fail( new LegacyPgDeltaEngineError({ message: "failed to parse pg-delta next shadow database URL", @@ -330,21 +341,21 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( }), ); } - const [migrationsPool, scratchPool] = yield* Effect.all( + const [migrationsPool, declarativePool] = yield* Effect.all( [ legacyAcquirePgPool(migrations, { isLocal: true, dnsResolver: "native" }), - legacyAcquirePgPool(scratch, { isLocal: true, dnsResolver: "native" }), + legacyAcquirePgPool(declarative, { isLocal: true, dnsResolver: "native" }), ], { concurrency: 2 }, ); const result = yield* adapter.planDeclarativeSchema({ targetPool: migrationsPool, - shadowPool: scratchPool, + shadowPool: declarativePool, files: input.files, allowDrops: true, debug: input.debug, reorder: true, - seedAssumedSchemas: true, + ...legacyPgDeltaNextIsolatedShadowPlanOptions, schema: input.schema, ...(input.manifest !== undefined ? { manifest: input.manifest } : {}), }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts new file mode 100644 index 0000000000..7b8921c48e --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts @@ -0,0 +1,12 @@ +import { describe, expect, it } from "vitest"; + +import { legacyPgDeltaNextIsolatedShadowPlanOptions } from "./legacy-pgdelta-engine.next.layer.ts"; + +describe("legacyPgDeltaNextIsolatedShadowPlanOptions", () => { + it("uses the isolated full-baseline mode shared by both declarative planner entrypoints", () => { + expect(legacyPgDeltaNextIsolatedShadowPlanOptions).toEqual({ + isolatedShadow: true, + seedAssumedSchemas: false, + }); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index 119cca4f9c..dd59eee4a8 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -1,7 +1,12 @@ import { Effect, Layer } from "effect"; import type { Pool } from "pg"; import { serializeSnapshot, encodeId } from "@supabase/pg-delta/core"; -import { buildSchemaExport, planSchemaFiles, renderPlanFiles } from "@supabase/pg-delta/frontends"; +import { + buildSchemaExport, + planSchemaFiles, + renderPlanFiles, + ShadowLoadError, +} from "@supabase/pg-delta/frontends"; import { type IntegrationProfile, resolveProfile, @@ -121,6 +126,8 @@ export interface LegacyPgDeltaNextLibraries diagnostic.message) : []; const label = operation === "declarativeExport" ? "Declarative schema export" @@ -129,7 +136,8 @@ function legacyPgDeltaNextMessage(operation: LegacyPgDeltaNextOperation, cause: : operation === "snapshotCapture" ? "Snapshot capture" : "Database diff"; - return `${label} failed: ${detail}`; + const renderedDiagnostics = diagnostics.map((diagnostic) => ` - ${diagnostic}`).join("\n"); + return `${label} failed: ${detail}${renderedDiagnostics === "" ? "" : `\n${renderedDiagnostics}`}`; } function legacyTryPgDeltaNext( diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index 3ad7d9d9a2..5c9d1c51c2 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -1,4 +1,5 @@ import { it } from "@effect/vitest"; +import { ShadowLoadError } from "@supabase/pg-delta/frontends"; import { Effect } from "effect"; import { Pool } from "pg"; import { describe, expect } from "vitest"; @@ -517,4 +518,60 @@ describe("LegacyPgDeltaNextAdapter", () => { yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); }).pipe(Effect.provide(failingLayer)); }); + + it.effect("preserves shadow-load diagnostics in the actionable error", () => { + const targetPool = new Pool(); + const shadowPool = new Pool(); + const cause = new ShadowLoadError("2 files cannot apply", [ + { + code: "stuck_statement", + severity: "error", + message: 'extensions/pg_cron.sql: extension "pg_cron" already exists', + }, + { + code: "stuck_statement", + severity: "error", + message: 'extensions/pg_net.sql: extension "pg_net" already exists', + }, + ]); + const failingLayer = legacyPgDeltaNextAdapterLayerFromLibraries({ + resolveProfile: async () => { + throw new Error("unused"); + }, + plan: () => ({ source: "unused", desired: "unused" }), + renderPlanFiles: () => ({ changes: false, files: [] }), + buildSchemaExport: async () => ({ + files: [], + diagnostics: [], + manifest: { redactSecrets: true, scope: "database" }, + }), + planSchemaFiles: async () => { + throw cause; + }, + serializeSnapshot: () => "unused", + serializePlan: () => "unused", + encodeSubject: (subject: string) => subject, + }); + + return Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + const error = yield* adapter + .planDeclarativeSchema({ + targetPool, + shadowPool, + files: [], + allowDrops: false, + debug: false, + isolatedShadow: true, + seedAssumedSchemas: false, + }) + .pipe(Effect.flip); + expect(error).toBeInstanceOf(LegacyPgDeltaNextError); + expect(error.message).toBe( + 'Declarative schema planning failed: 2 files cannot apply\n - extensions/pg_cron.sql: extension "pg_cron" already exists\n - extensions/pg_net.sql: extension "pg_net" already exists', + ); + expect(error.cause).toBe(cause); + yield* Effect.promise(() => Promise.all([targetPool.end(), shadowPool.end()])); + }).pipe(Effect.provide(failingLayer)); + }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index aaf8ae08c1..63ca24a5e9 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -1,6 +1,5 @@ import { Effect, Layer } from "effect"; -import { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; import { LegacyPgDeltaNextShadow, type LegacyPgDeltaNextShadowDatabases, @@ -9,9 +8,8 @@ import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; /** * Scoped next-engine shadow orchestration over the narrow Go `db __shadow` - * seam. Go creates the migrated target and a dedicated empty same-cluster - * scratch database; declarative SQL remains wholly owned by the TypeScript - * pg-delta next adapter and its `planSchemaFiles` operation. + * seam. Go creates independent migrated and declarative clusters; declarative + * SQL remains wholly owned by the TypeScript pg-delta next adapter. */ export const legacyPgDeltaNextShadowLayer = Layer.effect( LegacyPgDeltaNextShadow, @@ -21,33 +19,10 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( return LegacyPgDeltaNextShadow.of({ provision: ({ schema, projectRef }) => Effect.gen(function* () { - // Register cleanup immediately after Go returns the container. URL - // validation happens only after acquireRelease has installed the - // finalizer, so even malformed seam output cannot leak the shadow. - const shadow = yield* Effect.acquireRelease( - seam.provisionShadow({ - mode: "pgdelta-next", - targetLocal: false, - usePgDelta: false, - schema, - ...(projectRef !== undefined ? { projectRef } : {}), - }), - ({ container }) => seam.removeShadowContainer(container).pipe(Effect.ignoreCause), - ); - - if (shadow.targetUrlOverride === undefined) { - return yield* Effect.fail( - new LegacyDeclarativeShadowDbError({ - message: - "failed to provision the pg-delta next shadow database: missing declarative scratch URL.", - }), - ); - } - - return { - migrationsUrl: shadow.sourceUrl, - scratchUrl: shadow.targetUrlOverride, - } satisfies LegacyPgDeltaNextShadowDatabases; + return (yield* seam.provisionNextShadow({ + schema, + ...(projectRef !== undefined ? { projectRef } : {}), + })) satisfies LegacyPgDeltaNextShadowDatabases; }), }); }), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts index 1c34cc8fc8..24505e8725 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts @@ -6,15 +6,14 @@ import type { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts" export interface LegacyPgDeltaNextShadowDatabases { /** Platform baseline with the project's local migrations applied. */ readonly migrationsUrl: string; - /** Empty same-cluster database owned by `planSchemaFiles` while loading desired SQL. */ - readonly scratchUrl: string; + /** Independent platform baseline owned by `planSchemaFiles` while loading desired SQL. */ + readonly declarativeUrl: string; } interface LegacyPgDeltaNextShadowShape { /** - * Provisions the next-engine shadow container and owns it for the current - * Effect scope. The container is removed when that scope closes, including - * when URL validation or the caller fails. + * Provisions both next-engine shadow containers and owns them for the current + * Effect scope. Both containers are removed when that scope closes. */ readonly provision: (opts: { readonly schema: ReadonlyArray; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts index ef201d34d3..b3c18940a5 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts @@ -1,64 +1,33 @@ import { describe, expect, it } from "@effect/vitest"; -import { Data, Effect, Layer } from "effect"; +import { Effect, Layer } from "effect"; -import { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; import { legacyPgDeltaNextShadowLayer } from "./legacy-pgdelta-next-shadow.layer.ts"; import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; +import { legacyParseNextShadowProtocol } from "./legacy-pgdelta.seam.layer.ts"; import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; -class PrimaryFailure extends Data.TaggedError("PrimaryFailure")<{ - readonly message: string; -}> {} - -function setup( - opts: { - readonly sourceUrl?: string; - readonly scratchUrl?: string; - readonly cleanupDefect?: boolean; - } = {}, -) { +function setup() { const state = { provisionCalls: [] as object[], - removedContainers: [] as string[], legacyMethodCalls: [] as string[], }; const seamLayer = Layer.succeed( LegacyDeclarativeSeam, LegacyDeclarativeSeam.of({ - exportCatalog: () => - Effect.sync(() => { - state.legacyMethodCalls.push("exportCatalog"); - return "catalog.json"; - }), - execInherit: () => - Effect.sync(() => { - state.legacyMethodCalls.push("execInherit"); - return 0; - }), - ensureLocalDatabaseStarted: () => - Effect.sync(() => { - state.legacyMethodCalls.push("ensureLocalDatabaseStarted"); - }), - ensureLocalPostgresImageCurrent: () => - Effect.sync(() => { - state.legacyMethodCalls.push("ensureLocalPostgresImageCurrent"); - }), - provisionShadow: (input) => + exportCatalog: () => Effect.die("exportCatalog not used"), + execInherit: () => Effect.die("execInherit not used"), + ensureLocalDatabaseStarted: () => Effect.die("ensureLocalDatabaseStarted not used"), + ensureLocalPostgresImageCurrent: () => Effect.die("ensureLocalPostgresImageCurrent not used"), + provisionShadow: () => Effect.die("provisionShadow not used"), + provisionNextShadow: (input) => Effect.sync(() => { state.provisionCalls.push(input); return { - container: "next-shadow-container", - sourceUrl: opts.sourceUrl ?? "postgresql://postgres@localhost:55432/postgres", - targetUrlOverride: opts.scratchUrl, + migrationsUrl: "postgresql://postgres:secret@localhost:55432/postgres", + declarativeUrl: "postgresql://postgres:secret@localhost:55433/postgres", }; }), - removeShadowContainer: (container) => - Effect.gen(function* () { - state.removedContainers.push(container); - if (opts.cleanupDefect === true) { - return yield* Effect.die("cleanup failed"); - } - }), + removeShadowContainer: () => Effect.die("removeShadowContainer not used"), }), ); @@ -69,86 +38,69 @@ function setup( } describe("LegacyPgDeltaNextShadow", () => { - it.effect("provisions the exact next mode and exposes the migrated and scratch URLs", () => { - const { layer, state } = setup({ - scratchUrl: "postgresql://postgres@localhost:55432/pgdelta_declarative", + it("validates the dual-shadow JSON protocol structurally", () => { + expect( + legacyParseNextShadowProtocol( + JSON.stringify({ + migrations: { + containerId: "migrations-container", + url: "postgresql://postgres@localhost:55432/postgres", + }, + declarative: { + containerId: "declarative-container", + url: "postgresql://postgres@localhost:55433/postgres", + }, + }), + ), + ).toEqual({ + migrations: { + containerId: "migrations-container", + url: "postgresql://postgres@localhost:55432/postgres", + }, + declarative: { + containerId: "declarative-container", + url: "postgresql://postgres@localhost:55433/postgres", + }, }); + expect(() => legacyParseNextShadowProtocol("not json")).toThrow(); + expect(() => legacyParseNextShadowProtocol('{"migrations":{}}')).toThrow(); + expect(() => + legacyParseNextShadowProtocol( + JSON.stringify({ + migrations: { containerId: "same", url: "postgresql://localhost/postgres" }, + declarative: { containerId: "same", url: "postgresql://localhost/postgres" }, + }), + ), + ).toThrow("next-shadow containers must be distinct"); + }); + + it.effect("delegates to the isolated next-shadow seam and exposes both postgres URLs", () => { + const { layer, state } = setup(); + return Effect.gen(function* () { const databases = yield* Effect.scoped( Effect.gen(function* () { const shadow = yield* LegacyPgDeltaNextShadow; - const acquired = yield* shadow.provision({ + return yield* shadow.provision({ schema: ["public", "extensions"], projectRef: "linked-project", }); - expect(state.removedContainers).toEqual([]); - return acquired; }), ); expect(databases).toEqual({ - migrationsUrl: "postgresql://postgres@localhost:55432/postgres", - scratchUrl: "postgresql://postgres@localhost:55432/pgdelta_declarative", + migrationsUrl: "postgresql://postgres:secret@localhost:55432/postgres", + declarativeUrl: "postgresql://postgres:secret@localhost:55433/postgres", }); - expect(Object.keys(databases)).toEqual(["migrationsUrl", "scratchUrl"]); + expect(Object.keys(databases)).toEqual(["migrationsUrl", "declarativeUrl"]); expect(state.provisionCalls).toEqual([ { - mode: "pgdelta-next", - targetLocal: false, - usePgDelta: false, schema: ["public", "extensions"], projectRef: "linked-project", }, ]); - expect(state.removedContainers).toEqual(["next-shadow-container"]); expect(state.legacyMethodCalls).toEqual([]); }).pipe(Effect.provide(layer)); }); - - it.effect("cleans up when the caller fails and never lets cleanup mask that failure", () => { - const { layer, state } = setup({ - scratchUrl: "postgresql://postgres@localhost:55432/pgdelta_declarative", - cleanupDefect: true, - }); - const primary = new PrimaryFailure({ message: "caller failed" }); - - return Effect.gen(function* () { - const error = yield* Effect.scoped( - Effect.gen(function* () { - const shadow = yield* LegacyPgDeltaNextShadow; - yield* shadow.provision({ schema: [] }); - return yield* Effect.fail(primary); - }), - ).pipe(Effect.flip); - - expect(error).toEqual(primary); - expect(state.removedContainers).toEqual(["next-shadow-container"]); - }).pipe(Effect.provide(layer)); - }); - - it.effect("cleans up and fails when the declarative scratch URL is missing", () => { - const { layer, state } = setup(); - - return Effect.gen(function* () { - const error = yield* Effect.scoped( - Effect.gen(function* () { - const shadow = yield* LegacyPgDeltaNextShadow; - return yield* shadow.provision({ schema: ["public"] }); - }), - ).pipe(Effect.flip); - - expect(error).toBeInstanceOf(LegacyDeclarativeShadowDbError); - expect(error.message).toContain("missing declarative scratch URL"); - expect(state.removedContainers).toEqual(["next-shadow-container"]); - expect(state.provisionCalls).toEqual([ - { - mode: "pgdelta-next", - targetLocal: false, - usePgDelta: false, - schema: ["public"], - }, - ]); - }).pipe(Effect.provide(layer)); - }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts index c936e1a649..8a0f7aafc1 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts @@ -1,3 +1,4 @@ +import { execFileSync } from "node:child_process"; import { existsSync, mkdirSync, @@ -86,6 +87,43 @@ function localDatabaseUrl(config: string): string { return `postgresql://postgres:postgres@127.0.0.1:${dbSection?.[1]}/postgres?sslmode=disable`; } +function projectContainerIds(config: string): ReadonlyArray { + const projectId = config.match(/^project_id\s*=\s*"([^"]+)"/mu)?.[1]; + expect(projectId, "project_id missing from generated config.toml").toBeDefined(); + if (projectId === undefined) throw new Error("project_id missing from generated config.toml"); + const output = execFileSync( + "docker", + ["ps", "-aq", "--filter", `label=com.supabase.cli.project=${projectId}`], + { encoding: "utf8" }, + ); + return output.split(/\r?\n/u).filter(Boolean).sort(); +} + +function findSqlContaining(root: string, needle: string): string { + const match = readdirSync(root, { recursive: true }) + .filter((entry): entry is string => typeof entry === "string" && entry.endsWith(".sql")) + .map((entry) => path.join(root, entry)) + .find((file) => readFileSync(file, "utf8").includes(needle)); + expect(match, `no SQL file under ${root} contains ${needle}`).toBeDefined(); + if (match === undefined) throw new Error(`no SQL file under ${root} contains ${needle}`); + return match; +} + +function findExtensionDeclaration(root: string, extension: string): string { + const escaped = extension.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"); + const declaration = new RegExp( + `\\bCREATE\\s+EXTENSION(?:\\s+IF\\s+NOT\\s+EXISTS)?\\s+(?:"${escaped}"|${escaped})(?=\\s|;)`, + "iu", + ); + const match = readdirSync(root, { recursive: true }) + .filter((entry): entry is string => typeof entry === "string" && entry.endsWith(".sql")) + .map((entry) => path.join(root, entry)) + .find((file) => declaration.test(readFileSync(file, "utf8"))); + expect(match, `no SQL file under ${root} declares extension ${extension}`).toBeDefined(); + if (match === undefined) throw new Error(`no SQL file under ${root} declares ${extension}`); + return match; +} + describeDockerLive("pg-delta next local convergence (live)", () => { let projectDir = ""; let desiredSchemaPath = ""; @@ -446,3 +484,312 @@ describeDockerLive("pg-delta next local convergence (live)", () => { }, ); }); + +describeDockerLive("pg-delta next declarative extension baseline (live)", () => { + let projectDir = ""; + let config = ""; + + beforeAll(async () => { + projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-extensions-live-")); + + const init = await runSupabaseLive(["init"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(init.exitCode, commandFailure(init)).toBe(0); + + const configPath = path.join(projectDir, "supabase", "config.toml"); + const generatedConfig = readFileSync(configPath, "utf8"); + expect(generatedConfig).toContain("major_version = 17"); + expect(generatedConfig).not.toContain("[experimental.webhooks]"); + config = `${generatedConfig + .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') + .replace( + '# declarative_schema_path = "./database"', + 'declarative_schema_path = "./schemas"', + )}\n[experimental.webhooks]\nenabled = true\n`; + writeFileSync(configPath, config); + + const start = await runSupabaseLive( + [ + "start", + "--exclude", + "studio", + "--exclude", + "logflare", + "--exclude", + "vector", + "--exclude", + "gotrue", + "--exclude", + "realtime", + "--exclude", + "storage-api", + ], + { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + ); + expect(start.exitCode, commandFailure(start)).toBe(0); + }, COMMAND_TIMEOUT_MS); + + afterAll(async () => { + if (projectDir.length === 0) return; + await runSupabaseLive(["stop", "--no-backup"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }).catch(() => undefined); + await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); + }, COMMAND_TIMEOUT_MS); + + test( + "loads exported user-managed extensions and plans their removal by file deletion", + { timeout: SCENARIO_TIMEOUT_MS }, + async () => { + const generated = await runSupabaseLive( + ["db", "schema", "declarative", "generate", "--local", "--overwrite"], + { + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(generated.exitCode, commandFailure(generated)).toBe(0); + + const schemasDir = path.join(projectDir, "supabase", "schemas"); + findExtensionDeclaration(schemasDir, "pg_net"); + const pgcryptoFile = findExtensionDeclaration(schemasDir, "pgcrypto"); + findExtensionDeclaration(schemasDir, "uuid-ossp"); + + const containersBeforeEmpty = projectContainerIds(config); + const migrationsBeforeEmpty = migrationFiles(projectDir); + const empty = await runSupabaseLive(["db", "schema", "declarative", "sync", "--no-apply"], { + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(empty.exitCode, commandFailure(empty)).toBe(0); + expect(empty.stderr).toContain("No schema changes found"); + expect(migrationFiles(projectDir)).toEqual(migrationsBeforeEmpty); + expect(projectContainerIds(config)).toEqual(containersBeforeEmpty); + + const pgcryptoSql = readFileSync(pgcryptoFile, "utf8"); + const migrationsBeforeRemoval = new Set(migrationFiles(projectDir)); + await rm(pgcryptoFile); + try { + const containersBeforeRemoval = projectContainerIds(config); + const removal = await runSupabaseLive( + ["db", "schema", "declarative", "sync", "--no-apply", "--name", "drop_pgcrypto"], + { + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(removal.exitCode, commandFailure(removal)).toBe(0); + expect(projectContainerIds(config)).toEqual(containersBeforeRemoval); + + const removalMigrations = migrationFiles(projectDir).filter( + (file) => !migrationsBeforeRemoval.has(file), + ); + expect(removalMigrations.length).toBeGreaterThan(0); + const removalSql = removalMigrations + .map((file) => + readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8"), + ) + .join("\n"); + expect(removalSql).toMatch(/DROP\s+EXTENSION(?:\s+IF\s+EXISTS)?\s+"?pgcrypto"?/iu); + } finally { + writeFileSync(pgcryptoFile, pgcryptoSql); + await Promise.all( + migrationFiles(projectDir) + .filter((file) => !migrationsBeforeRemoval.has(file)) + .map((file) => rm(path.join(projectDir, "supabase", "migrations", file))), + ); + } + }, + ); +}); + +describeDockerLive("pg-delta next isolated cron shadows (live)", () => { + const jobName = "pgdelta_cli_inactive"; + const initialSchedule = "0 0 * * *"; + const changedSchedule = "15 3 * * *"; + let projectDir = ""; + let config = ""; + + beforeAll(async () => { + projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-cron-live-")); + + const init = await runSupabaseLive(["init"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(init.exitCode, commandFailure(init)).toBe(0); + + const configPath = path.join(projectDir, "supabase", "config.toml"); + config = readFileSync(configPath, "utf8") + .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') + .replace('# declarative_schema_path = "./database"', 'declarative_schema_path = "./schemas"'); + writeFileSync(configPath, config); + + const migrationsDir = path.join(projectDir, "supabase", "migrations"); + mkdirSync(migrationsDir, { recursive: true }); + writeFileSync( + path.join(migrationsDir, "20260806000000_cron_inactive.sql"), + `create extension if not exists pg_cron; + +create table public.pgdelta_cron_execution_sentinel ( + executed_at timestamptz not null default now() +); + +select cron.schedule( + '${jobName}', + '${initialSchedule}', + 'insert into public.pgdelta_cron_execution_sentinel default values' +); + +select cron.alter_job( + (select jobid from cron.job where jobname = '${jobName}'), + active := false +); +`, + ); + + const start = await runSupabaseLive( + [ + "start", + "--exclude", + "studio", + "--exclude", + "logflare", + "--exclude", + "vector", + "--exclude", + "gotrue", + "--exclude", + "realtime", + "--exclude", + "storage-api", + ], + { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + ); + expect(start.exitCode, commandFailure(start)).toBe(0); + }, COMMAND_TIMEOUT_MS); + + afterAll(async () => { + if (projectDir.length === 0) return; + await runSupabaseLive(["stop", "--no-backup"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }).catch(() => undefined); + await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); + }, COMMAND_TIMEOUT_MS); + + test( + "keeps an inactive named job converged and replaces only its changed schedule", + { timeout: SCENARIO_TIMEOUT_MS }, + async () => { + const generated = await runSupabaseLive( + ["db", "schema", "declarative", "generate", "--local", "--overwrite"], + { + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(generated.exitCode, commandFailure(generated)).toBe(0); + + const schemasDir = path.join(projectDir, "supabase", "schemas"); + const cronFile = findSqlContaining(schemasDir, `cron.schedule_in_database('${jobName}'`); + const containersBeforeEmpty = projectContainerIds(config); + const empty = await runSupabaseLive(["db", "schema", "declarative", "sync", "--no-apply"], { + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(empty.exitCode, commandFailure(empty)).toBe(0); + expect(empty.stderr).toContain("No schema changes found"); + expect(projectContainerIds(config)).toEqual(containersBeforeEmpty); + + const emptyBundle = requireDebugBundle(projectDir, "declarativePlan"); + expect(assertJsonFile(path.join(emptyBundle, "plan.json"))).toMatchObject({ + deltas: [], + actions: [], + source: { fingerprint: expect.any(String) }, + target: { fingerprint: expect.any(String) }, + }); + + const exportedCron = readFileSync(cronFile, "utf8"); + expect(exportedCron).toContain(`'${initialSchedule}'`); + writeFileSync(cronFile, exportedCron.replace(`'${initialSchedule}'`, `'${changedSchedule}'`)); + + const migrationsBeforeApply = new Set(migrationFiles(projectDir)); + const containersBeforeApply = projectContainerIds(config); + const applied = await runSupabaseLive( + ["db", "schema", "declarative", "sync", "--apply", "--name", "cron_schedule"], + { + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(applied.exitCode, commandFailure(applied)).toBe(0); + expect(applied.stderr).toContain("Migration applied successfully"); + expect(projectContainerIds(config)).toEqual(containersBeforeApply); + + const scheduleMigrations = migrationFiles(projectDir).filter( + (file) => !migrationsBeforeApply.has(file), + ); + expect(scheduleMigrations.length).toBeGreaterThan(0); + const scheduleSql = scheduleMigrations + .map((file) => readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8")) + .join("\n"); + expect(scheduleSql.match(/cron\.unschedule/gu)).toHaveLength(1); + expect(scheduleSql.match(/cron\.schedule_in_database/gu)).toHaveLength(1); + expect(scheduleSql).toContain(`'${changedSchedule}'`); + expect(scheduleSql).not.toMatch( + /\b(?:create|alter|drop)\s+(?:table|schema|function|view|extension|role)\b/iu, + ); + + const job = await runSupabaseLive( + [ + "db", + "query", + "--local", + "-o", + "json", + `select schedule, active from cron.job where jobname = '${jobName}'`, + ], + { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + ); + expect(job.exitCode, commandFailure(job)).toBe(0); + expect(JSON.parse(job.stdout)).toEqual([{ schedule: changedSchedule, active: false }]); + + const executions = await runSupabaseLive( + [ + "db", + "query", + "--local", + "-o", + "json", + "select count(*)::int as executions from public.pgdelta_cron_execution_sentinel", + ], + { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + ); + expect(executions.exitCode, commandFailure(executions)).toBe(0); + expect(JSON.parse(executions.stdout)).toEqual([{ executions: 0 }]); + + const containersBeforeFinal = projectContainerIds(config); + const finalSync = await runSupabaseLive( + ["db", "schema", "declarative", "sync", "--no-apply"], + { + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(finalSync.exitCode, commandFailure(finalSync)).toBe(0); + expect(finalSync.stderr).toContain("No schema changes found"); + expect(projectContainerIds(config)).toEqual(containersBeforeFinal); + }, + ); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.cache.ts index f06b687000..55120a704a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.cache.ts @@ -31,6 +31,8 @@ export interface LegacySetupInputs { readonly authEnabled: boolean; readonly storageEnabled: boolean; readonly realtimeEnabled: boolean; + /** Effective `experimental.webhooks.enabled` (absent → false). */ + readonly webhooksEnabled: boolean; /** Effective `api.auto_expose_new_tables` (unset and false both → false). */ readonly autoExpose: boolean; /** `[db.vault]` secret names (sorted before hashing). */ @@ -90,6 +92,7 @@ export function legacySetupInputsToken(inputs: LegacySetupInputs): string { payload += `auth=${boolToken(inputs.authEnabled)} storage=${boolToken( inputs.storageEnabled, )} realtime=${boolToken(inputs.realtimeEnabled)}\n`; + payload += `database_webhooks=${boolToken(inputs.webhooksEnabled)}\n`; payload += `auto_expose_new_tables=${boolToken(inputs.autoExpose)}\n`; for (const name of [...inputs.vaultNames].sort()) payload += `vault=${name}\n`; payload += inputs.rolesSql; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.cache.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.cache.unit.test.ts index 83535b91c0..4a6df34501 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.cache.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.cache.unit.test.ts @@ -34,6 +34,7 @@ const BASE: LegacySetupInputs = { authEnabled: true, storageEnabled: true, realtimeEnabled: true, + webhooksEnabled: false, autoExpose: false, vaultNames: [], rolesSql: "", @@ -66,7 +67,7 @@ describe("legacyBaselineVersionToken", () => { describe("legacySetupInputsToken", () => { it("byte-matches the Go hash input sequence", () => { const expected = sha12( - "17.6.1.135\nauth=true storage=true realtime=true\nauto_expose_new_tables=false\n", + "17.6.1.135\nauth=true storage=true realtime=true\ndatabase_webhooks=false\nauto_expose_new_tables=false\n", ); expect(legacySetupInputsToken(BASE)).toBe(expected); }); @@ -78,7 +79,7 @@ describe("legacySetupInputsToken", () => { rolesSql: "create role app;", }); const expected = sha12( - "17.6.1.135\nauth=true storage=true realtime=true\nauto_expose_new_tables=false\n" + + "17.6.1.135\nauth=true storage=true realtime=true\ndatabase_webhooks=false\nauto_expose_new_tables=false\n" + "vault=a_secret\nvault=b_secret\ncreate role app;", ); expect(token).toBe(expected); @@ -87,6 +88,7 @@ describe("legacySetupInputsToken", () => { it("self-invalidates when any baseline input changes", () => { const baseToken = legacySetupInputsToken(BASE); expect(legacySetupInputsToken({ ...BASE, authEnabled: false })).not.toBe(baseToken); + expect(legacySetupInputsToken({ ...BASE, webhooksEnabled: true })).not.toBe(baseToken); expect(legacySetupInputsToken({ ...BASE, autoExpose: true })).not.toBe(baseToken); expect(legacySetupInputsToken({ ...BASE, vaultNames: ["x"] })).not.toBe(baseToken); expect(legacySetupInputsToken({ ...BASE, rolesSql: "x" })).not.toBe(baseToken); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts index c6d8c2b9a8..d9e87dda80 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts @@ -1,9 +1,9 @@ -import { Effect, FileSystem, Layer, Option, Path, Stream } from "effect"; +import { Effect, FileSystem, Layer, Option, Path, Scope, Stream } from "effect"; import * as ChildProcess from "effect/unstable/process/ChildProcess"; import { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import { LegacyNetworkIdFlag, LegacyProfileFlag } from "../../../../shared/legacy/global-flags.ts"; -import { resolveBinary } from "../../../../shared/legacy/go-proxy.layer.ts"; +import { type BinaryResolution, resolveBinary } from "../../../../shared/legacy/go-proxy.layer.ts"; import { LegacyCliConfig } from "../../../config/legacy-cli-config.service.ts"; import { containerCliExitCode, spawnContainerCli } from "../../../shared/legacy-container-cli.ts"; import { legacyResolveDbImage } from "../../../shared/legacy-db-image.ts"; @@ -14,7 +14,11 @@ import { localDbContainerId, } from "../../../shared/legacy-docker-ids.ts"; import { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; -import { LegacyDeclarativeSeam, type LegacyShadowSource } from "./legacy-pgdelta.seam.service.ts"; +import { + LegacyDeclarativeSeam, + type LegacyNextShadowSource, + type LegacyShadowSource, +} from "./legacy-pgdelta.seam.service.ts"; import { legacyInjectPostgresPassword } from "./legacy-pgdelta.seam.url.ts"; /** @@ -23,8 +27,7 @@ import { legacyInjectPostgresPassword } from "./legacy-pgdelta.seam.url.ts"; * (the catalog path) and stderr inherited (shadow-DB progress / image pulls). * The Go binary is resolved exactly like `LegacyGoProxy` (`resolveBinary`). */ -export const legacyDeclarativeSeamLayer = Layer.effect( - LegacyDeclarativeSeam, +const makeLegacyDeclarativeSeam = (resolved: BinaryResolution) => Effect.gen(function* () { const cliConfig = yield* LegacyCliConfig; const networkId = yield* LegacyNetworkIdFlag; @@ -39,7 +42,19 @@ export const legacyDeclarativeSeamLayer = Layer.effect( const spawner = yield* ChildProcessSpawner; const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; - const resolved = resolveBinary(); + const removeShadowContainer = (container: string) => + Effect.gen(function* () { + if (container.length === 0) return; + // Best-effort and volume-aware, matching Go's DockerRemove. Each next + // shadow registers this independently so one cleanup defect cannot + // prevent the sibling container from being removed. + yield* containerCliExitCode(spawner, ["rm", "-f", "-v", container], { + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + extendEnv: true, + }).pipe(Effect.ignore); + }); return LegacyDeclarativeSeam.of({ exportCatalog: ({ mode, noCache, projectRef }) => @@ -448,9 +463,7 @@ export const legacyDeclarativeSeamLayer = Layer.effect( } // stdout is three newline-separated lines: container id, source URL, // and an optional second-database URL. Legacy diff uses the third URL - // only when its local-target declarative branch redirects the target; - // `pgdelta-next` always returns its empty same-cluster declarative - // scratch database there. That next mode never asks Go to apply SQL. + // only when its local-target declarative branch redirects the target. // The URLs arrive WITHOUT a password — the Go seam prints them via // ToPostgresURLWithoutPassword so it never logs a credential to stdout // (CWE-312). The shadow uses the local Postgres password, so we re-inject @@ -487,25 +500,126 @@ export const legacyDeclarativeSeamLayer = Layer.effect( } satisfies LegacyShadowSource; }), ), - removeShadowContainer: (container) => + provisionNextShadow: ({ schema, projectRef }) => Effect.gen(function* () { - if (container.length === 0) return; - // Remove the shadow left running by provisionShadow. Best-effort — a - // failure here must never mask the diff result. `-v` removes the - // Postgres anonymous data volume too, matching Go's `DockerRemove` - // (`RemoveOptions{RemoveVolumes: true, Force: true}`, - // `internal/utils/docker.go:330`); without it every shadow leaves a - // dangling volume behind. - yield* containerCliExitCode(spawner, ["rm", "-f", "-v", container], { - stdin: "ignore", - stdout: "ignore", - stderr: "ignore", - extendEnv: true, - }).pipe(Effect.ignore); + if (!("found" in resolved)) { + return yield* Effect.fail( + new LegacyDeclarativeShadowDbError({ + message: + "Could not find the supabase-go binary required to provision the shadow databases.", + }), + ); + } + + // Keep the process in a nested scope. Until `ack\n`, Go owns both + // containers, so parsing/password failures and interruption close the + // child and let its deferred cleanup run. The caller scope receives + // both Docker finalizers before ownership is acknowledged. + const ownerScope = yield* Effect.scope; + return yield* Effect.scoped( + Effect.gen(function* () { + const args = [ + "db", + "__shadow", + "--mode", + "pgdelta-next", + ...(schema.length > 0 ? ["--schema", schema.join(",")] : []), + ...(Option.isSome(networkId) ? ["--network-id", networkId.value] : []), + ...(projectRef !== undefined ? ["--project-ref", projectRef] : []), + ...profileArgs, + ]; + const command = ChildProcess.make(resolved.found, args, { + cwd: cliConfig.workdir, + stdin: "pipe", + stdout: "pipe", + stderr: "inherit", + extendEnv: true, + env: { SUPABASE_TELEMETRY_DISABLED: "1" }, + detached: false, + }); + const handle = yield* spawner.spawn(command).pipe( + Effect.mapError( + () => + new LegacyDeclarativeShadowDbError({ + message: "failed to run the shadow-database provisioner (supabase-go).", + }), + ), + ); + // `runHead` returns as soon as the newline-delimited JSON object is + // emitted; waiting for stdout EOF would deadlock because Go waits + // for the acknowledgment before exiting. + const line = yield* handle.stdout.pipe( + Stream.decodeText, + Stream.splitLines, + Stream.runHead, + Effect.mapError(() => failure()), + ); + if (Option.isNone(line)) { + return yield* Effect.fail(failure()); + } + const protocol = yield* Effect.try({ + try: () => legacyParseNextShadowProtocol(line.value), + catch: () => failure(), + }); + const password = yield* legacyReadDbToml( + fs, + path, + cliConfig.workdir, + projectRef, + ).pipe( + Effect.map((toml) => toml.password), + Effect.mapError( + () => + new LegacyDeclarativeShadowDbError({ + message: + "failed to read the local database password from config.toml to connect to the shadow databases.", + }), + ), + ); + const databases = yield* Effect.try({ + try: () => + ({ + migrationsUrl: legacyInjectPostgresPassword(protocol.migrations.url, password), + declarativeUrl: legacyInjectPostgresPassword( + protocol.declarative.url, + password, + ), + }) satisfies LegacyNextShadowSource, + catch: () => failure(), + }); + + yield* Scope.addFinalizer( + ownerScope, + removeShadowContainer(protocol.migrations.containerId).pipe(Effect.ignoreCause), + ); + yield* Scope.addFinalizer( + ownerScope, + removeShadowContainer(protocol.declarative.containerId).pipe(Effect.ignoreCause), + ); + yield* Stream.make("ack\n").pipe( + Stream.encodeText, + Stream.run(handle.stdin), + Effect.mapError(() => failure()), + ); + const exitCode = yield* handle.exitCode.pipe(Effect.mapError(() => failure())); + if (exitCode !== 0) { + return yield* Effect.fail(failure(exitCode)); + } + return databases; + }), + ); }), + removeShadowContainer, }); - }), -); + }); + +export function makeLegacyDeclarativeSeamLayer(options: { readonly binary?: string } = {}) { + const resolved: BinaryResolution = + options.binary === undefined ? resolveBinary() : { found: options.binary }; + return Layer.effect(LegacyDeclarativeSeam, makeLegacyDeclarativeSeam(resolved)); +} + +export const legacyDeclarativeSeamLayer = makeLegacyDeclarativeSeamLayer(); // Intentionally NOT `LegacyGoChildExitError` (contrast `legacy-db-bootstrap.seam.layer.ts`, // fixed under CLI-1879): this seam's failure is a TS-authored domain summary over noisy @@ -557,3 +671,40 @@ export function legacyResolveContainerInspectImageName(stdout: string): string { function isJsonRecord(value: unknown): value is { readonly [key: string]: unknown } { return typeof value === "object" && value !== null; } + +interface LegacyNextShadowProtocolDatabase { + readonly containerId: string; + readonly url: string; +} + +interface LegacyNextShadowProtocol { + readonly migrations: LegacyNextShadowProtocolDatabase; + readonly declarative: LegacyNextShadowProtocolDatabase; +} + +/** Strict structural validation for the Go next-shadow ownership protocol. */ +export function legacyParseNextShadowProtocol(line: string): LegacyNextShadowProtocol { + const parsed: unknown = JSON.parse(line); + if (!isJsonRecord(parsed)) throw new Error("invalid next-shadow protocol"); + const migrations = parseNextShadowProtocolDatabase(parsed["migrations"]); + const declarative = parseNextShadowProtocolDatabase(parsed["declarative"]); + if (migrations.containerId === declarative.containerId) { + throw new Error("next-shadow containers must be distinct"); + } + return { migrations, declarative }; +} + +function parseNextShadowProtocolDatabase(value: unknown): LegacyNextShadowProtocolDatabase { + if (!isJsonRecord(value)) throw new Error("invalid next-shadow database"); + const containerId = value["containerId"]; + const url = value["url"]; + if ( + typeof containerId !== "string" || + containerId.trim().length === 0 || + typeof url !== "string" || + url.trim().length === 0 + ) { + throw new Error("invalid next-shadow database"); + } + return { containerId: containerId.trim(), url: url.trim() }; +} diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.unit.test.ts index b6b0251a10..004d8ade19 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.unit.test.ts @@ -1,9 +1,239 @@ -import { describe, expect, it } from "vitest"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { BunFileSystem, BunPath } from "@effect/platform-bun"; +import { describe, expect, it } from "@effect/vitest"; +import { Effect, Layer, Option, Sink, Stream } from "effect"; +import * as ChildProcessSpawner from "effect/unstable/process/ChildProcessSpawner"; +import { LegacyNetworkIdFlag, LegacyProfileFlag } from "../../../../shared/legacy/global-flags.ts"; +import { LegacyCliConfig } from "../../../config/legacy-cli-config.service.ts"; import { legacyIsMissingContainerInspectError, legacyResolveContainerInspectImageName, + makeLegacyDeclarativeSeamLayer, } from "./legacy-pgdelta.seam.layer.ts"; +import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; + +const protocol = JSON.stringify({ + migrations: { + containerId: "migrations-container", + url: "postgresql://postgres@localhost:55432/postgres", + }, + declarative: { + containerId: "declarative-container", + url: "postgresql://postgres@localhost:55433/postgres", + }, +}); + +function setup( + options: { + readonly stdout?: string; + readonly interruptOnAck?: boolean; + readonly cleanupDefectContainer?: string; + readonly workdir?: string; + } = {}, +) { + const state = { + commands: [] as Array<{ + readonly command: string; + readonly args: ReadonlyArray; + readonly stdin: unknown; + readonly stdout: unknown; + readonly stderr: unknown; + }>, + stdin: "", + childScopeClosed: 0, + cleanupAttempts: [] as string[], + }; + const spawner = Layer.succeed( + ChildProcessSpawner.ChildProcessSpawner, + ChildProcessSpawner.make((command) => { + if (command._tag !== "StandardCommand") return Effect.die("unexpected pipeline"); + state.commands.push({ + command: command.command, + args: [...command.args], + stdin: command.options.stdin, + stdout: command.options.stdout, + stderr: command.options.stderr, + }); + const isProvisioner = command.command === "/fake/supabase-go"; + if (!isProvisioner) { + const container = command.args.at(-1) ?? ""; + state.cleanupAttempts.push(container); + return Effect.succeed( + ChildProcessSpawner.makeHandle({ + pid: ChildProcessSpawner.ProcessId(2), + exitCode: + container === options.cleanupDefectContainer + ? Effect.die("cleanup defect") + : Effect.succeed(ChildProcessSpawner.ExitCode(0)), + isRunning: Effect.succeed(false), + kill: () => Effect.void, + unref: Effect.succeed(Effect.void), + stdin: Sink.drain, + stdout: Stream.empty, + stderr: Stream.empty, + all: Stream.empty, + getInputFd: () => Sink.drain, + getOutputFd: () => Stream.empty, + }), + ); + } + const handle = ChildProcessSpawner.makeHandle({ + pid: ChildProcessSpawner.ProcessId(1), + exitCode: Effect.sync(() => { + if (state.stdin !== "ack\n") throw new Error("exit awaited before ack"); + return ChildProcessSpawner.ExitCode(0); + }), + isRunning: Effect.succeed(true), + kill: () => Effect.void, + unref: Effect.succeed(Effect.void), + stdin: Sink.forEach((chunk: Uint8Array) => + Effect.sync(() => { + state.stdin += new TextDecoder().decode(chunk); + }).pipe(Effect.andThen(options.interruptOnAck === true ? Effect.interrupt : Effect.void)), + ), + // Never terminate stdout: reading the full stream would deadlock before ack. + stdout: Stream.make(new TextEncoder().encode(`${options.stdout ?? protocol}\n`)).pipe( + Stream.concat(Stream.never), + ), + stderr: Stream.empty, + all: Stream.empty, + getInputFd: () => Sink.drain, + getOutputFd: () => Stream.empty, + }); + return Effect.acquireRelease(Effect.succeed(handle), () => + Effect.sync(() => { + state.childScopeClosed += 1; + }), + ); + }), + ); + const config = Layer.succeed(LegacyCliConfig, { + profile: "supabase", + apiUrl: "https://api.supabase.com", + projectHost: "supabase.co", + poolerHost: "pooler.supabase.com", + dashboardUrl: "https://supabase.com/dashboard", + accessToken: Option.none(), + projectId: Option.none(), + workdir: options.workdir ?? resolve(process.cwd(), "../.."), + userAgent: "test", + }); + const dependencies = Layer.mergeAll( + BunFileSystem.layer, + BunPath.layer, + spawner, + config, + Layer.succeed(LegacyNetworkIdFlag, Option.some("test-network")), + Layer.succeed(LegacyProfileFlag, "snap"), + ); + return { + state, + layer: makeLegacyDeclarativeSeamLayer({ binary: "/fake/supabase-go" }).pipe( + Layer.provide(dependencies), + ), + }; +} + +describe("LegacyDeclarativeSeam next shadow protocol", () => { + it.effect("acks only after acquisition and cleans both containers on caller failure", () => { + const { layer, state } = setup({ cleanupDefectContainer: "declarative-container" }); + return Effect.gen(function* () { + const exit = yield* Effect.scoped( + Effect.gen(function* () { + const seam = yield* LegacyDeclarativeSeam; + const databases = yield* seam.provisionNextShadow({ + schema: ["public", "extensions"], + projectRef: "linked-project", + }); + expect(state.stdin).toBe("ack\n"); + expect(databases).toEqual({ + migrationsUrl: "postgresql://postgres:postgres@localhost:55432/postgres", + declarativeUrl: "postgresql://postgres:postgres@localhost:55433/postgres", + }); + return yield* Effect.fail("caller failed"); + }), + ).pipe(Effect.exit); + + expect(exit._tag).toBe("Failure"); + expect(state.cleanupAttempts).toEqual(["declarative-container", "migrations-container"]); + expect(state.childScopeClosed).toBe(1); + expect(state.commands[0]).toEqual({ + command: "/fake/supabase-go", + args: [ + "db", + "__shadow", + "--mode", + "pgdelta-next", + "--schema", + "public,extensions", + "--network-id", + "test-network", + "--project-ref", + "linked-project", + "--profile", + "snap", + ], + stdin: "pipe", + stdout: "pipe", + stderr: "inherit", + }); + }).pipe(Effect.provide(layer)); + }); + + it.effect("has both cleanup finalizers installed when the ack write is interrupted", () => { + const { layer, state } = setup({ interruptOnAck: true }); + return Effect.gen(function* () { + yield* Effect.scoped( + Effect.gen(function* () { + const seam = yield* LegacyDeclarativeSeam; + yield* seam.provisionNextShadow({ schema: [] }); + }), + ).pipe(Effect.exit); + expect(state.stdin).toBe("ack\n"); + expect(state.cleanupAttempts).toEqual(["declarative-container", "migrations-container"]); + expect(state.childScopeClosed).toBe(1); + }).pipe(Effect.provide(layer)); + }); + + it.effect("does not ack malformed output and leaves cleanup with Go", () => { + const { layer, state } = setup({ stdout: '{"migrations":{}}' }); + return Effect.gen(function* () { + yield* Effect.scoped( + Effect.gen(function* () { + const seam = yield* LegacyDeclarativeSeam; + yield* seam.provisionNextShadow({ schema: [] }); + }), + ).pipe(Effect.exit); + expect(state.stdin).toBe(""); + expect(state.cleanupAttempts).toEqual([]); + expect(state.childScopeClosed).toBe(1); + }).pipe(Effect.provide(layer)); + }); + + it.effect("does not ack when the database password cannot be read", () => { + const workdir = mkdtempSync(join(tmpdir(), "supabase-next-shadow-")); + mkdirSync(join(workdir, "supabase")); + writeFileSync(join(workdir, "supabase", "config.toml"), "[db\ninvalid"); + const { layer, state } = setup({ workdir }); + return Effect.gen(function* () { + yield* Effect.scoped( + Effect.gen(function* () { + const seam = yield* LegacyDeclarativeSeam; + yield* seam.provisionNextShadow({ schema: [] }); + }), + ).pipe(Effect.exit); + expect(state.stdin).toBe(""); + expect(state.cleanupAttempts).toEqual([]); + expect(state.childScopeClosed).toBe(1); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(workdir, { recursive: true, force: true }))), + ); + }); +}); describe("legacyIsMissingContainerInspectError", () => { it("matches Docker and Podman missing-container stderr", () => { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts index 02413bcdd7..847b063dec 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts @@ -1,4 +1,4 @@ -import { Context, type Effect } from "effect"; +import { Context, type Effect, type Scope } from "effect"; import type { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; @@ -11,12 +11,8 @@ export type LegacyCatalogMode = "baseline" | "migrations" | "declarative"; * `db pull` diff source), plus the local-target declarative branch. * - `declarative`: a bare shadow with no baseline/migrations (the `db pull * --declarative` empty export source). - * - `pgdelta-next`: platform baseline + local migrations in `postgres`, plus - * an empty same-cluster `pgdelta_declarative` scratch database. Declarative - * SQL is deliberately not applied by Go in this mode; the TypeScript next - * engine loads it later through `planSchemaFiles`. */ -type LegacyShadowMode = "diff" | "declarative" | "pgdelta-next"; +type LegacyShadowMode = "diff" | "declarative"; /** A live shadow database left running for the caller to diff against and remove. */ export interface LegacyShadowSource { @@ -26,13 +22,19 @@ export interface LegacyShadowSource { readonly sourceUrl: string; /** * Optional second live database. For legacy diff it replaces the target with - * `contrib_regression` after Go applies declarative schemas. For - * `pgdelta-next` it is the empty declarative scratch database; TypeScript - * loads the declarative files later through `planSchemaFiles`. + * `contrib_regression` after Go applies declarative schemas. */ readonly targetUrlOverride: string | undefined; } +/** The independently hosted databases used by the pg-delta next planner. */ +export interface LegacyNextShadowSource { + /** Platform baseline with local configuration and migrations applied. */ + readonly migrationsUrl: string; + /** Platform baseline with local configuration, ready for declarative SQL. */ + readonly declarativeUrl: string; +} + interface LegacyDeclarativeSeamShape { /** * Provisions the shadow-database platform baseline (and, for @@ -116,6 +118,16 @@ interface LegacyDeclarativeSeamShape { */ readonly projectRef?: string; }) => Effect.Effect; + /** + * Provisions the two isolated pg-delta next shadows through the Go seam's + * JSON/ack ownership protocol. Both containers are owned by the current + * Effect scope before the child is acknowledged, and are independently + * removed when that scope closes. + */ + readonly provisionNextShadow: (opts: { + readonly schema: ReadonlyArray; + readonly projectRef?: string; + }) => Effect.Effect; /** * Removes a shadow database container left running by `provisionShadow` * (`docker rm -f `). Best-effort: a failure to remove is swallowed so it diff --git a/apps/cli/src/legacy/commands/start/lib/db-setup.ts b/apps/cli/src/legacy/commands/start/lib/db-setup.ts index c9722c8cdd..a4bca9c0bb 100644 --- a/apps/cli/src/legacy/commands/start/lib/db-setup.ts +++ b/apps/cli/src/legacy/commands/start/lib/db-setup.ts @@ -31,14 +31,18 @@ * `STORAGE_S3_REGION`, no JWKS) — built locally, not reused. * - `initAuthJob` (`start.go:319-332`) — ditto, a minimal env distinct from * `gotrue.service.ts`'s full container builder. - * 2. **`ApplyApiPrivileges`** (`start.go:414-435`) — tri-state on + * 2. **`ApplyDatabaseWebhooks`** — installs `pg_net` only when + * `experimental.webhooks.enabled` is true. The platform webhook helpers/event + * trigger are always present, so user migrations can still create or drop the + * extension explicitly. + * 3. **`ApplyApiPrivileges`** (`start.go:414-435`) — tri-state on * `api.auto_expose_new_tables`: `true` is a no-op (keep the bundled initial-schema * grants); unset/`false` execs {@link LEGACY_START_REVOKE_API_PRIVILEGES_SQL} * (Go's inline `RevokeDefaultDataApiPrivilegesSql` constant, `start.go:405-412`) * via a temp file, same as the schema SQL above. - * 3. **Vault upsert** (`start.go:390-393`) — `legacyUpsertVaultSecrets`, run BEFORE + * 4. **Vault upsert** (`start.go:390-393`) — `legacyUpsertVaultSecrets`, run BEFORE * the custom-roles seed "so roles.sql can reference them" (Go's own comment). - * 4. **Custom-roles seed** (`start.go:394-398` + `pkg/migration/seed.go:84-97`) — + * 5. **Custom-roles seed** (`start.go:394-398` + `pkg/migration/seed.go:84-97`) — * prints "Seeding globals from roles.sql..." UNCONDITIONALLY, BEFORE checking * whether `supabase/roles.sql` even exists (Go's `SeedGlobals` prints first, * then attempts the read), then execs the file via `legacyExecSqlFile` only when @@ -46,7 +50,7 @@ * os.ErrNotExist)` check, reproduced here as an existence check ahead of the read * rather than a caught not-found error — see the call site's own comment for why); * any other read/exec error propagates. - * 5. **`apply.MigrateAndSeed`** (`start.go:368`, via the already-ported + * 6. **`apply.MigrateAndSeed`** (`start.go:368`, via the already-ported * `legacyMigrateAndSeed`) with `version: ""` — every pending migration, matching * `SetupLocalDatabase`'s own call in the `start` context. * @@ -117,6 +121,9 @@ alter default privileges for role postgres in schema public revoke execute on functions from anon, authenticated, service_role; `; +const LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL = + "create extension if not exists pg_net schema extensions;"; + /** * A SQL exec (schema/globals/API-privileges) or one-shot service-migration Docker * job failed, or the scratch temp directory/file could not be created. The Docker @@ -517,6 +524,22 @@ const legacyStartApplyApiPrivileges = Effect.fnUntraced(function* ( ); }); +/** Installs pg_net only for the explicit Database Webhooks feature opt-in. */ +const legacyStartApplyDatabaseWebhooks = Effect.fnUntraced(function* ( + input: LegacyStartSetupLocalDatabaseInput, + tmpDir: string, +) { + if (input.config.experimental.webhooks?.enabled !== true) return; + yield* legacyExecSqlConstant( + input.session, + input.fs, + input.path, + tmpDir, + "enable-database-webhooks.sql", + LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL, + ); +}); + /** * Port of Go's `initCurrentBranch` (`start.go:233-241`): writes * `supabase/.branches/_current_branch` = `"main"` (Go's `CurrBranchPath`, @@ -580,7 +603,7 @@ export const legacyStartSetupLocalDatabase = ( const toml = yield* legacyCheckDbToml(fs, path, workdir); - // SetupDatabase: initSchema -> ApplyApiPrivileges (start.go:383-389). + // SetupDatabase: initSchema -> ApplyDatabaseWebhooks -> ApplyApiPrivileges. yield* Effect.scoped( Effect.gen(function* () { const tmpDir = yield* fs @@ -594,6 +617,7 @@ export const legacyStartSetupLocalDatabase = ( ), ); yield* legacyStartInitSchema(input, tmpDir); + yield* legacyStartApplyDatabaseWebhooks(input, tmpDir); yield* legacyStartApplyApiPrivileges(input, tmpDir, toml.baseline.apiAutoExposeNewTables); }), ); diff --git a/apps/cli/src/legacy/commands/start/lib/db-setup.unit.test.ts b/apps/cli/src/legacy/commands/start/lib/db-setup.unit.test.ts index 8f2decfc6f..45675c1d40 100644 --- a/apps/cli/src/legacy/commands/start/lib/db-setup.unit.test.ts +++ b/apps/cli/src/legacy/commands/start/lib/db-setup.unit.test.ts @@ -43,6 +43,7 @@ const SCHEMA_13_FINGERPRINT = const SCHEMA_14_FINGERPRINT_SUFFIX = "CREATE SCHEMA IF NOT EXISTS graphql"; const REVOKE_PRIVILEGES_FINGERPRINT = "revoke execute on functions from anon, authenticated, service_role"; +const PG_NET_CREATE_FINGERPRINT = "create extension if not exists pg_net schema extensions"; function fakeSession() { const calls: Array<{ kind: "exec" | "query"; sql: string; params?: ReadonlyArray }> = []; @@ -429,6 +430,42 @@ describe("legacyStartSetupLocalDatabase", () => { ); }); + describe("Database Webhooks", () => { + it.effect("does not install pg_net merely because Edge Runtime is enabled", () => { + const workdir = makeWorkdir(); + const { session, calls } = fakeSession(); + const out = mockOutput(); + const docker = mockDockerRun(); + const config = decodeConfig({ edge_runtime: { enabled: true } }); + return run(baseInput(workdir, session, { majorVersion: 14, config }), out, docker).pipe( + Effect.map(() => { + const execSql = calls.filter((c) => c.kind === "exec").map((c) => c.sql); + expect(execSql.some((sql) => sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(false); + rmSync(workdir, { recursive: true, force: true }); + }), + ); + }); + + it.effect("installs pg_net when Database Webhooks is enabled without Edge Runtime", () => { + const workdir = makeWorkdir(); + writeConfigToml(workdir, "[experimental.webhooks]\nenabled = true\n"); + const { session, calls } = fakeSession(); + const out = mockOutput(); + const docker = mockDockerRun(); + const config = decodeConfig({ + edge_runtime: { enabled: false }, + experimental: { webhooks: { enabled: true } }, + }); + return run(baseInput(workdir, session, { majorVersion: 14, config }), out, docker).pipe( + Effect.map(() => { + const execSql = calls.filter((c) => c.kind === "exec").map((c) => c.sql); + expect(execSql.filter((sql) => sql.includes(PG_NET_CREATE_FINGERPRINT))).toHaveLength(1); + rmSync(workdir, { recursive: true, force: true }); + }), + ); + }); + }); + describe("vault upsert + custom-roles seed", () => { it.effect("upserts vault secrets before seeding supabase/roles.sql", () => { const workdir = makeWorkdir(); diff --git a/apps/cli/src/legacy/commands/start/services/postgres.service.unit.test.ts b/apps/cli/src/legacy/commands/start/services/postgres.service.unit.test.ts index b651045860..988123e113 100644 --- a/apps/cli/src/legacy/commands/start/services/postgres.service.unit.test.ts +++ b/apps/cli/src/legacy/commands/start/services/postgres.service.unit.test.ts @@ -85,6 +85,11 @@ describe("legacyBuildPostgresStartContainerSpec", () => { ); expect(script).not.toContain(LEGACY_POSTGRES_DEFAULT_ROOT_KEY); expect(script).not.toContain("pgsodium_root.key"); + expect(LEGACY_START_DB_WEBHOOK_SQL).not.toContain("CREATE EXTENSION IF NOT EXISTS pg_net"); + expect(LEGACY_START_DB_WEBHOOK_SQL).toContain( + "CREATE OR REPLACE FUNCTION extensions.grant_pg_net_access()", + ); + expect(LEGACY_START_DB_WEBHOOK_SQL).toContain("CREATE EVENT TRIGGER issue_pg_net_access"); expect(spec.tmpfs).toBeUndefined(); expect(spec.secretFiles).toEqual([ { diff --git a/apps/cli/src/legacy/commands/start/templates/db-initial-schema-14.sql.ts b/apps/cli/src/legacy/commands/start/templates/db-initial-schema-14.sql.ts index 8199aaad59..b2b506eb09 100644 --- a/apps/cli/src/legacy/commands/start/templates/db-initial-schema-14.sql.ts +++ b/apps/cli/src/legacy/commands/start/templates/db-initial-schema-14.sql.ts @@ -79,20 +79,6 @@ CREATE SCHEMA IF NOT EXISTS graphql_public; ALTER SCHEMA graphql_public OWNER TO supabase_admin; --- --- Name: pg_net; Type: EXTENSION; Schema: -; Owner: - --- - -CREATE EXTENSION IF NOT EXISTS pg_net WITH SCHEMA extensions; - - --- --- Name: EXTENSION pg_net; Type: COMMENT; Schema: -; Owner: --- - -COMMENT ON EXTENSION pg_net IS 'Async HTTP'; - - -- -- Name: pgbouncer; Type: SCHEMA; Schema: -; Owner: pgbouncer -- diff --git a/apps/cli/src/legacy/commands/start/templates/db-webhook.sql.ts b/apps/cli/src/legacy/commands/start/templates/db-webhook.sql.ts index 5aa85e84a5..d71eeff247 100644 --- a/apps/cli/src/legacy/commands/start/templates/db-webhook.sql.ts +++ b/apps/cli/src/legacy/commands/start/templates/db-webhook.sql.ts @@ -7,9 +7,6 @@ */ export const LEGACY_START_DB_WEBHOOK_SQL = `BEGIN; --- Create pg_net extension -CREATE EXTENSION IF NOT EXISTS pg_net SCHEMA extensions; - -- Create supabase_functions schema CREATE SCHEMA supabase_functions AUTHORIZATION supabase_admin; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 7cba076346..69321df5f0 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -155,11 +155,11 @@ importers: specifier: workspace:* version: link:../../packages/config '@supabase/pg-delta': - specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb - version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb) + specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb + version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb) '@supabase/pg-topo': - specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb - version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb + specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb + version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb '@supabase/process-compose': specifier: workspace:* version: link:../../packages/process-compose @@ -2842,8 +2842,8 @@ packages: resolution: {integrity: sha512-RW/OCsd6MO592zU8ifzP8/f8XzxxIdpb+Up5XaOtE26Fw+3zTp475WX7+GuuktiD1WF8pFUDe6khUPbMp77RCw==} engines: {node: '>=22.0.0'} - '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb': - resolution: {integrity: sha512-eWhb8JyODx870aSr2xKr3i81yBBnblAqLjdFqW0MGr6pxDzX/PbJkQGx700u/CESycO6HfW8+2MrDQUE3em53w==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb} + '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb': + resolution: {integrity: sha512-zD/OOjOZaOIaMMjm7VbhlZa23hqeQFDYaWT3mnfT7/6/JgSEp5rge4gwTY+cjy+Q7ObLdwqJe+wDO7ARujJMbA==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb} version: 1.0.0-alpha.33 engines: {node: '>=20.0.0'} hasBin: true @@ -2853,8 +2853,8 @@ packages: '@supabase/pg-topo': optional: true - '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb': - resolution: {integrity: sha512-HoqxATDYB2WygDOV1XQBN/hM/hcfvVUrc7F+R691j6CD89cHumR/nRiRJ+0bh9siZb7Fx81Bp9BAPRfzEkEydA==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb} + '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb': + resolution: {integrity: sha512-HoqxATDYB2WygDOV1XQBN/hM/hcfvVUrc7F+R691j6CD89cHumR/nRiRJ+0bh9siZb7Fx81Bp9BAPRfzEkEydA==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb} version: 1.0.0-alpha.5 '@supabase/phoenix@0.4.5': @@ -9096,18 +9096,18 @@ snapshots: dependencies: tslib: 2.8.1 - '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@951daa9d9b333f5c69c38eb664d8a17847c635eb(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb)': + '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb)': dependencies: debug: 4.4.3(supports-color@7.2.0) pg: 8.22.0 pg-connection-string: 2.14.0 optionalDependencies: - '@supabase/pg-topo': https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb + '@supabase/pg-topo': https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb transitivePeerDependencies: - pg-native - supports-color - '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@951daa9d9b333f5c69c38eb664d8a17847c635eb': + '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb': dependencies: '@pgsql/traverse': 17.2.6 plpgsql-parser: 0.5.16 From 5bffecbfba560944f6b4f0364d6eb2784d87b389 Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 7 Aug 2026 19:05:35 +0200 Subject: [PATCH 05/82] fix(cli): correct diff and migration execution contracts --- apps/cli-go/cmd/db.go | 19 +- apps/cli-go/docs/supabase/db/diff.md | 2 + apps/cli-go/docs/supabase/db/pull.md | 2 + .../db/schema-declarative-generate.md | 2 + .../supabase/db/schema-declarative-sync.md | 2 + apps/cli-go/internal/db/diff/diff.go | 83 +----- apps/cli-go/internal/db/diff/diff_test.go | 248 ++++++------------ apps/cli-go/internal/db/diff/pgdelta.go | 29 +- .../internal/db/diff/pgdelta_migrations.go | 8 + .../db/diff/pgdelta_migrations_test.go | 11 + apps/cli-go/internal/db/diff/pgdelta_test.go | 6 + apps/cli-go/internal/db/diff/shadow.go | 43 +-- .../cli-go/internal/testing/helper/history.go | 16 +- .../internal/testing/helper/privileges.go | 8 +- apps/cli-go/pkg/migration/apply_test.go | 28 +- apps/cli-go/pkg/migration/drop_test.go | 16 +- apps/cli-go/pkg/migration/file.go | 74 +++++- apps/cli-go/pkg/migration/file_test.go | 128 ++++++++- apps/cli-go/pkg/migration/history.go | 22 +- apps/cli-go/pkg/migration/seed_test.go | 24 +- .../create/create.integration.test.ts | 33 ++- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 27 +- .../legacy/commands/db/diff/diff.handler.ts | 64 +---- .../commands/db/diff/diff.integration.test.ts | 114 ++++---- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 5 + .../legacy/commands/db/pull/pull.handler.ts | 62 +---- .../commands/db/pull/pull.integration.test.ts | 34 ++- .../schema/declarative/sync/SIDE_EFFECTS.md | 7 + .../declarative/sync/sync.integration.test.ts | 4 +- .../commands/db/shared/legacy-diff-engine.ts | 3 + .../legacy-pgdelta-engine.legacy.layer.ts | 9 +- .../legacy-pgdelta-engine.next.layer.ts | 38 +-- .../shared/legacy-pgdelta-engine.service.ts | 8 +- .../db/shared/legacy-pgdelta-files.ts | 35 +-- .../shared/legacy-pgdelta-migrations.write.ts | 11 + .../legacy-pgdelta-next-adapter.layer.ts | 2 +- .../legacy-pgdelta-next-adapter.service.ts | 4 +- .../legacy-pgdelta-next-adapter.unit.test.ts | 4 +- .../shared/legacy-pgdelta-next.live.test.ts | 3 + .../shared/legacy-pgdelta.integration.test.ts | 33 +++ .../db/shared/legacy-pgdelta.seam.layer.ts | 13 +- .../db/shared/legacy-pgdelta.seam.service.ts | 11 +- .../commands/db/shared/legacy-pgdelta.ts | 23 +- .../src/legacy/commands/link/link.handler.ts | 17 +- .../commands/link/link.integration.test.ts | 21 ++ .../legacy/shared/legacy-db-config.service.ts | 2 + .../src/legacy/shared/legacy-http-errors.ts | 22 +- .../legacy/shared/legacy-migration-apply.ts | 31 +++ .../legacy-migration-apply.unit.test.ts | 56 ++++ apps/cli/src/legacy/shared/legacy-seed-ops.ts | 3 - packages/api/scripts/generate.ts | 12 + packages/api/scripts/generate.unit.test.ts | 24 +- packages/api/src/effect.ts | 6 +- packages/api/src/generated/contracts.ts | 232 ++++++++-------- packages/api/src/internal/client.ts | 30 ++- packages/api/src/internal/client.unit.test.ts | 76 +++++- 56 files changed, 1046 insertions(+), 804 deletions(-) diff --git a/apps/cli-go/cmd/db.go b/apps/cli-go/cmd/db.go index be733771e6..ec9cb971ca 100644 --- a/apps/cli-go/cmd/db.go +++ b/apps/cli-go/cmd/db.go @@ -276,16 +276,14 @@ var ( }, } - shadowMode string - shadowTargetLocal bool - shadowUsePgDelta bool - shadowSchema []string - shadowProjectRef string + shadowMode string + shadowSchema []string + shadowProjectRef string // dbShadowCmd is a hidden seam used by the native-TypeScript db diff/pull // commands to provision throwaway shadow databases, then leave them running // so the TS caller can run the differ itself and remove the containers - // afterwards. Legacy modes print three newline-separated lines. pgdelta-next + // afterwards. Legacy modes print two newline-separated lines. pgdelta-next // emits a JSON object describing its two isolated clusters, then retains // cleanup ownership until the caller acknowledges receipt. URLs are emitted // WITHOUT the password @@ -334,7 +332,7 @@ var ( case "declarative": src, err = diff.PrepareRawShadow(cmd.Context()) case "diff", "": - src, err = diff.PrepareShadowSource(cmd.Context(), shadowSchema, shadowTargetLocal, shadowUsePgDelta, fsys) + src, err = diff.PrepareShadowSource(cmd.Context(), fsys) default: return fmt.Errorf("unknown shadow mode: %s", shadowMode) } @@ -343,11 +341,6 @@ var ( } fmt.Println(src.Container) fmt.Println(utils.ToPostgresURLWithoutPassword(src.Source)) - if src.TargetOverride != nil { - fmt.Println(utils.ToPostgresURLWithoutPassword(*src.TargetOverride)) - } else { - fmt.Println("") - } return nil }, } @@ -762,8 +755,6 @@ func init() { // Build hidden shadow-provisioning seam command shadowFlags := dbShadowCmd.Flags() shadowFlags.StringVar(&shadowMode, "mode", "diff", "Shadow mode: diff (baseline + migrations), declarative (bare shadow), or pgdelta-next (migrated + empty scratch).") - shadowFlags.BoolVar(&shadowTargetLocal, "target-local", false, "Whether the diff target is the local database (enables the declarative-schema branch).") - shadowFlags.BoolVar(&shadowUsePgDelta, "use-pg-delta", false, "Whether pg-delta is the active diff engine (selects the declarative-apply path).") shadowFlags.StringSliceVarP(&shadowSchema, "schema", "s", []string{}, "Comma separated list of schema to include.") shadowFlags.StringVar(&shadowProjectRef, "project-ref", "", "Linked project ref, so the shadow merges the matching [remotes.] config override.") dbCmd.AddCommand(dbShadowCmd) diff --git a/apps/cli-go/docs/supabase/db/diff.md b/apps/cli-go/docs/supabase/db/diff.md index 497d371a95..0307f41ade 100644 --- a/apps/cli-go/docs/supabase/db/diff.md +++ b/apps/cli-go/docs/supabase/db/diff.md @@ -6,6 +6,8 @@ Requires the local development stack to be running when diffing against the loca Runs [djrobstep/migra](https://github.com/djrobstep/migra) in a container to compare schema differences between the target database and a shadow database. The shadow database is created by applying migrations in local `supabase/migrations` directory in a separate container. Output is written to stdout by default. For convenience, you can also save the schema diff as a new migration file by passing in `-f` flag. +Normal diff mode always compares that migrations shadow with the selected live database. Declarative files under `supabase/database/` and `[db.migrations].schema_paths` do not replace the target. Use `supabase db schema declarative sync` to compare the complete declarative desired state. + By default, all schemas in the target database are diffed. Use the `--schema public,extensions` flag to restrict diffing to a subset of schemas. Projects created by a recent `supabase init` default to the pg-delta diff engine (`[experimental.pgdelta] enabled = true` in `config.toml`). Existing projects are unaffected and keep using migra unless they opt in. To fall back to the legacy migra engine, set `enabled = false` under `[experimental.pgdelta]`, or pass `--use-migra` for a single run. diff --git a/apps/cli-go/docs/supabase/db/pull.md b/apps/cli-go/docs/supabase/db/pull.md index 50c01be1ac..5128f06133 100644 --- a/apps/cli-go/docs/supabase/db/pull.md +++ b/apps/cli-go/docs/supabase/db/pull.md @@ -12,6 +12,8 @@ If no entries exist in the migration history table, the default diff engine uses Pass `--diff-engine pg-delta` to keep the migration-file `db pull` workflow while using pg-delta for the shadow diff step. On initial pull, pg-delta replaces `pg_dump` and produces the full migration from the shadow diff alone. Pass `--declarative` to switch to the declarative pg-delta export workflow instead. +Migration-style pull always compares the local migrations shadow with the selected live database. Declarative files and `[db.migrations].schema_paths` do not replace that target; use `db schema declarative sync` for declarative comparison. + Pg-delta runs in-process by default and is bundled with pg-topo at CLI build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily use the legacy edge-runtime implementation. `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs directly under `supabase/.temp/pgdelta/` affect only that opt-out; the CLI never falls back automatically. pg-delta plans are execution-aware: when a plan crosses a transaction boundary — for example `ALTER TYPE ... ADD VALUE` followed by a statement that uses the new enum value, which cannot run in the same transaction — `db pull` writes one ordered migration file per plan unit instead of a single file (for example `_remote_schema_schema_changes.sql` and `_remote_schema_after_enum_values.sql`), each recorded in the migration history. The common case (a single unit) still produces exactly one `_remote_schema.sql` file. diff --git a/apps/cli-go/docs/supabase/db/schema-declarative-generate.md b/apps/cli-go/docs/supabase/db/schema-declarative-generate.md index d82cae431a..4d82a4b8cf 100644 --- a/apps/cli-go/docs/supabase/db/schema-declarative-generate.md +++ b/apps/cli-go/docs/supabase/db/schema-declarative-generate.md @@ -4,6 +4,8 @@ Generate declarative schema files from a database. Exports the schema of a live database (local, linked, or custom URL) into SQL files under the declarative schema directory. This is the entrypoint for bootstrapping declarative mode. +The generated directory becomes the complete desired state: objects omitted from it are intended removals, including extensions, with or without an export manifest. When upgrading from the legacy workflow, regenerate the directory or add declarations for every extension you intend to retain before syncing, then review destructive-change warnings before applying. + Pg-delta and pg-topo run in-process and are bundled into the CLI at build time. The export includes `.pgdelta-export.json` policy metadata. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy catalog/edge-runtime implementation; `PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs at the `.temp/pgdelta/` root are legacy-only. `--no-cache` bypasses legacy catalog reuse/warming. The bundled engine always extracts live state and has no reusable catalog cache. With `PGDELTA_DEBUG=1`, structured diagnostics are written under `.temp/pgdelta/v2/debug//`. SQL bytes and grouping may differ between engines; reloading the export to the same managed state is the contract. diff --git a/apps/cli-go/docs/supabase/db/schema-declarative-sync.md b/apps/cli-go/docs/supabase/db/schema-declarative-sync.md index a6cf5e5729..867c36076a 100644 --- a/apps/cli-go/docs/supabase/db/schema-declarative-sync.md +++ b/apps/cli-go/docs/supabase/db/schema-declarative-sync.md @@ -4,6 +4,8 @@ Generate a new migration by diffing your declarative schema files against the cu When no declarative schema exists yet, the command offers to run `generate` first. After computing the diff, you can optionally name the migration and apply it to the local database. +The declarative directory is a complete, hand-authored desired state. Missing objects are intended removals, including extensions, regardless of whether the files were generated or whether an export manifest exists. When upgrading from the legacy workflow, regenerate the directory or add declarations for extensions you intend to retain, and review destructive-change warnings before applying. + Pg-delta and pg-topo run in-process and are bundled into the CLI at build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy catalog/edge-runtime implementation; `PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs at the `.temp/pgdelta/` root are legacy-only. `--no-cache` bypasses legacy catalog reuse/warming; the bundled engine extracts current state and has no reusable catalog cache. It may emit multiple ordered migration files to preserve transaction boundaries. SQL bytes may differ from the legacy renderer; successful application followed by an empty sync is the contract. With `PGDELTA_DEBUG=1`, snapshots, the plan, and diagnostics are written under `.temp/pgdelta/v2/debug//`. diff --git a/apps/cli-go/internal/db/diff/diff.go b/apps/cli-go/internal/db/diff/diff.go index b06ac78284..1906f76c41 100644 --- a/apps/cli-go/internal/db/diff/diff.go +++ b/apps/cli-go/internal/db/diff/diff.go @@ -4,11 +4,8 @@ import ( "context" "fmt" "io" - "io/fs" "os" - "path/filepath" "regexp" - "sort" "strconv" "strings" "time" @@ -23,13 +20,14 @@ import ( "github.com/spf13/afero" "github.com/supabase/cli/internal/db/start" "github.com/supabase/cli/internal/utils" - configpkg "github.com/supabase/cli/pkg/config" "github.com/supabase/cli/pkg/migration" "github.com/supabase/cli/pkg/parser" ) type DiffFunc func(context.Context, pgconn.Config, pgconn.Config, []string, ...func(*pgx.ConnConfig)) (string, error) +const schemaPathsTransitionWarning = "WARNING: [db.migrations].schema_paths no longer changes the target of db diff or migration-style db pull. These commands always compare local migrations with the selected database. Use `supabase db schema declarative sync` to compare declarative schema files." + func Run(ctx context.Context, schema []string, file string, config pgconn.Config, differ DiffFunc, usePgDelta bool, fsys afero.Fs, options ...func(*pgx.ConnConfig)) (err error) { result, err := DiffDatabase(ctx, schema, config, os.Stderr, fsys, differ, usePgDelta, options...) if err != nil { @@ -49,75 +47,6 @@ func Run(ctx context.Context, schema []string, file string, config pgconn.Config return nil } -func loadDeclaredSchemas(fsys afero.Fs) ([]string, error) { - if schemas := utils.Config.Db.Migrations.SchemaPaths; len(schemas) > 0 { - return schemas.SQLFiles( - afero.NewIOFS(fsys), - configpkg.WithSkipEmptyGlobs(), - configpkg.WithErrorOnAllSkippedGlobs(), - ) - } - // When pg-delta is enabled, declarative path is the source of truth (config or default). - if utils.IsPgDeltaEnabled() { - declDir := utils.GetDeclarativeDir() - if exists, err := afero.DirExists(fsys, declDir); err == nil && exists { - var declared []string - if err := afero.Walk(fsys, declDir, func(path string, info fs.FileInfo, err error) error { - if err != nil { - return err - } - if info.Mode().IsRegular() && filepath.Ext(info.Name()) == ".sql" { - declared = append(declared, path) - } - return nil - }); err != nil { - return nil, errors.Errorf("failed to walk declarative dir: %w", err) - } - sort.Strings(declared) - return declared, nil - } - } - if exists, err := afero.DirExists(fsys, utils.SchemasDir); err != nil { - return nil, errors.Errorf("failed to check schemas: %w", err) - } else if !exists { - return nil, nil - } - var declared []string - if err := afero.Walk(fsys, utils.SchemasDir, func(path string, info fs.FileInfo, err error) error { - if err != nil { - return err - } - if info.Mode().IsRegular() && filepath.Ext(info.Name()) == ".sql" { - declared = append(declared, path) - } - return nil - }); err != nil { - return nil, errors.Errorf("failed to walk dir: %w", err) - } - // Keep file application order deterministic so diff output stays stable across - // filesystems and operating systems. This is only if no schema paths in config are set. - sort.Strings(declared) - return declared, nil -} - -func shouldApplyDeclarativeWithPgDelta(usePgDelta bool) bool { - if !usePgDelta { - return false - } - schemas := utils.Config.Db.Migrations.SchemaPaths - if len(schemas) == 0 { - return true - } - if len(schemas) != 1 { - return false - } - return cleanSchemaPath(schemas[0]) == cleanSchemaPath(utils.GetDeclarativeDir()) -} - -func cleanSchemaPath(path string) string { - return filepath.ToSlash(filepath.Clean(path)) -} - // https://github.com/djrobstep/migra/blob/master/migra/statements.py#L6 var dropStatementPattern = regexp.MustCompile(`(?i)drop\s+`) @@ -250,16 +179,16 @@ func MigrateShadowDatabase(ctx context.Context, container string, fsys afero.Fs, } func DiffDatabase(ctx context.Context, schema []string, config pgconn.Config, w io.Writer, fsys afero.Fs, differ DiffFunc, usePgDelta bool, options ...func(*pgx.ConnConfig)) (DatabaseDiff, error) { + if len(utils.Config.Db.Migrations.SchemaPaths) > 0 { + fmt.Fprintln(w, schemaPathsTransitionWarning) + } fmt.Fprintln(w, "Creating shadow database...") - shadowSource, err := PrepareShadowSource(ctx, schema, utils.IsLocalDatabase(config), usePgDelta, fsys, options...) + shadowSource, err := PrepareShadowSource(ctx, fsys, options...) if err != nil { return DatabaseDiff{}, err } defer utils.DockerRemove(shadowSource.Container) shadowConfig := shadowSource.Source - if shadowSource.TargetOverride != nil { - config = *shadowSource.TargetOverride - } // Load all user defined schemas if len(schema) > 0 { fmt.Fprintln(w, "Diffing schemas:", strings.Join(schema, ",")) diff --git a/apps/cli-go/internal/db/diff/diff_test.go b/apps/cli-go/internal/db/diff/diff_test.go index 7e92f23ef8..53ee1a9a53 100644 --- a/apps/cli-go/internal/db/diff/diff_test.go +++ b/apps/cli-go/internal/db/diff/diff_test.go @@ -1,6 +1,7 @@ package diff import ( + "bytes" "context" "errors" "io" @@ -38,80 +39,6 @@ var dbConfig = pgconn.Config{ Database: "postgres", } -func TestLoadDeclaredSchemas(t *testing.T) { - t.Run("respects schema_paths order when pg-delta declarative dir exists", func(t *testing.T) { - originalConfig := utils.Config - t.Cleanup(func() { utils.Config = originalConfig }) - utils.Config.Db.Migrations.SchemaPaths = pkgconfig.Glob{ - "supabase/schemas/z_function.sql", - "supabase/schemas/a_table.sql", - } - utils.Config.Experimental.PgDelta = &pkgconfig.PgDeltaConfig{ - Enabled: true, - DeclarativeSchemaPath: utils.SchemasDir, - } - fsys := afero.NewMemMapFs() - require.NoError(t, fsys.MkdirAll(utils.SchemasDir, 0755)) - require.NoError(t, afero.WriteFile(fsys, "supabase/schemas/a_table.sql", []byte("create table a();"), 0644)) - require.NoError(t, afero.WriteFile(fsys, "supabase/schemas/z_function.sql", []byte("create function z() returns void language sql as $$ select 1 $$;"), 0644)) - - declared, err := loadDeclaredSchemas(fsys) - - require.NoError(t, err) - assert.Equal(t, []string{ - "supabase/schemas/z_function.sql", - "supabase/schemas/a_table.sql", - }, declared) - }) - - t.Run("expands schema_paths directory entries deterministically", func(t *testing.T) { - originalConfig := utils.Config - t.Cleanup(func() { utils.Config = originalConfig }) - utils.Config.Db.Migrations.SchemaPaths = pkgconfig.Glob{utils.DeclarativeDir} - fsys := afero.NewMemMapFs() - require.NoError(t, fsys.MkdirAll(filepath.Join(utils.DeclarativeDir, "nested"), 0755)) - require.NoError(t, afero.WriteFile(fsys, filepath.Join(utils.DeclarativeDir, "nested", "b.sql"), []byte("select 2;"), 0644)) - require.NoError(t, afero.WriteFile(fsys, filepath.Join(utils.DeclarativeDir, "a.sql"), []byte("select 1;"), 0644)) - - declared, err := loadDeclaredSchemas(fsys) - - require.NoError(t, err) - assert.Equal(t, []string{ - filepath.Join(utils.DeclarativeDir, "a.sql"), - filepath.Join(utils.DeclarativeDir, "nested", "b.sql"), - }, declared) - }) -} - -func TestShouldApplyDeclarativeWithPgDelta(t *testing.T) { - t.Run("uses pg-delta declarative apply when no schema_paths override is configured", func(t *testing.T) { - originalConfig := utils.Config - t.Cleanup(func() { utils.Config = originalConfig }) - utils.Config.Db.Migrations.SchemaPaths = nil - - assert.True(t, shouldApplyDeclarativeWithPgDelta(true)) - }) - - t.Run("uses pg-delta declarative apply when schema_paths points at the declarative dir", func(t *testing.T) { - originalConfig := utils.Config - t.Cleanup(func() { utils.Config = originalConfig }) - utils.Config.Db.Migrations.SchemaPaths = pkgconfig.Glob{utils.DeclarativeDir + "/"} - - assert.True(t, shouldApplyDeclarativeWithPgDelta(true)) - }) - - t.Run("uses ordered migration apply for explicit schema_paths files", func(t *testing.T) { - originalConfig := utils.Config - t.Cleanup(func() { utils.Config = originalConfig }) - utils.Config.Db.Migrations.SchemaPaths = pkgconfig.Glob{ - "supabase/schemas/z_function.sql", - "supabase/schemas/a_table.sql", - } - - assert.False(t, shouldApplyDeclarativeWithPgDelta(true)) - }) -} - func TestRun(t *testing.T) { t.Run("runs migra diff", func(t *testing.T) { // Setup in-memory fs @@ -174,7 +101,7 @@ func TestRun(t *testing.T) { assert.Equal(t, []byte(diff), contents) }) - t.Run("applies schema_paths in order before saving generated diff", func(t *testing.T) { + t.Run("ignores schema_paths and diffs the selected database", func(t *testing.T) { originalConfig := utils.Config t.Cleanup(func() { utils.Config = originalConfig }) utils.Config.Db.MajorVersion = 14 @@ -212,19 +139,21 @@ func TestRun(t *testing.T) { Reply(http.StatusOK) shadowConn := pgtest.NewConn() defer shadowConn.Close(t) - shadowConn.Query(utils.GlobalsSql). + shadowConn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.GlobalsSql). Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT"). + Query("BEGIN"). + Reply("BEGIN"). Query(utils.InitialSchemaPg14Sql). - Reply("CREATE SCHEMA") + Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT") helper.MockApiPrivilegesRevoke(shadowConn). Query(CREATE_TEMPLATE). Reply("CREATE DATABASE") - declaredConn := pgtest.NewConn() - defer declaredConn.Close(t) - declaredConn.Query(functionSQL). - Reply("CREATE FUNCTION"). - Query(tableSQL). - Reply("CREATE TABLE") // pg-delta bypasses the injected DiffFunc and runs the real edge-runtime // pipeline, so stub the seam DiffDatabase uses (mirrors exportCatalogPgDelta). // The migra differ must never be reached on this path. @@ -233,7 +162,7 @@ func TestRun(t *testing.T) { diffCalled := false diffPgDeltaRefDetailed = func(_ context.Context, _, targetRef string, schema []string, _ string, _ ...func(*pgx.ConnConfig)) (PgDeltaDiffResult, error) { diffCalled = true - assert.Contains(t, targetRef, "contrib_regression") + assert.Contains(t, targetRef, ":54322/postgres") assert.Equal(t, []string{"public"}, schema) return PgDeltaDiffResult{ Files: []PgDeltaPlanFile{{Order: 1, Name: "schema_changes", TransactionMode: "transactional", SQL: generated}}, @@ -252,11 +181,7 @@ func TestRun(t *testing.T) { } err := Run(context.Background(), []string{"public"}, "ordered_schema", localConfig, differ, true, fsys, func(cc *pgx.ConnConfig) { - if cc.Database == "contrib_regression" { - declaredConn.Intercept(cc) - } else { - shadowConn.Intercept(cc) - } + shadowConn.Intercept(cc) }) require.NoError(t, err) @@ -302,20 +227,32 @@ func TestMigrateShadow(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(utils.GlobalsSql). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.GlobalsSql). Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT"). + Query("BEGIN"). + Reply("BEGIN"). Query(utils.InitialSchemaPg14Sql). - Reply("CREATE SCHEMA") + Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn). Query(CREATE_TEMPLATE). Reply("CREATE DATABASE") helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(sql). Reply("CREATE SCHEMA"). Query(migration.INSERT_MIGRATION_VERSION, "0", "test", []string{sql}). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := MigrateShadowDatabase(context.Background(), "test-shadow-db", fsys, conn.Intercept) // Check error @@ -352,8 +289,12 @@ func TestMigrateShadow(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(utils.GlobalsSql). - ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`) + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.GlobalsSql). + ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := MigrateShadowDatabase(context.Background(), "test-shadow-db", fsys, conn.Intercept) // Check error @@ -377,10 +318,18 @@ func TestSetupShadowDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(utils.GlobalsSql). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.GlobalsSql). Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT"). + Query("BEGIN"). + Reply("BEGIN"). Query(utils.InitialSchemaPg14Sql). - Reply("CREATE SCHEMA") + Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn). Query(CREATE_TEMPLATE). Reply("CREATE DATABASE") @@ -396,8 +345,12 @@ func TestSetupShadowDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(utils.GlobalsSql). - ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`) + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.GlobalsSql). + ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := SetupShadowDatabase(context.Background(), "test-shadow-db", afero.NewMemMapFs(), conn.Intercept) // Check error @@ -491,6 +444,8 @@ func TestDiffDatabase(t *testing.T) { utils.InitialSchemaPg14Sql = "create schema private" t.Run("throws error on failure to create shadow", func(t *testing.T) { + utils.Config.Db.Migrations.SchemaPaths = pkgconfig.Glob{"supabase/database/*.sql"} + t.Cleanup(func() { utils.Config.Db.Migrations.SchemaPaths = nil }) errNetwork := errors.New("network error") // Setup in-memory fs fsys := afero.NewMemMapFs() @@ -501,10 +456,12 @@ func TestDiffDatabase(t *testing.T) { Get("/v" + utils.Docker.ClientVersion() + "/images/" + utils.GetRegistryImageUrl(utils.Config.Db.Image) + "/json"). ReplyError(errNetwork) // Run test - result, err := DiffDatabase(context.Background(), []string{"public"}, dbConfig, io.Discard, fsys, DiffSchemaMigra, false) + var output bytes.Buffer + result, err := DiffDatabase(context.Background(), []string{"public"}, dbConfig, &output, fsys, DiffSchemaMigra, false) // Check error assert.Empty(t, result) assert.ErrorIs(t, err, errNetwork) + assert.Contains(t, output.String(), schemaPathsTransitionWarning) assert.Empty(t, apitest.ListUnmatchedRequests()) }) @@ -561,8 +518,12 @@ func TestDiffDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(utils.GlobalsSql). - ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`) + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.GlobalsSql). + ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test result, err := DiffDatabase(context.Background(), []string{"public"}, dbConfig, io.Discard, fsys, DiffSchemaMigra, false, conn.Intercept) // Check error @@ -615,20 +576,32 @@ create schema public`) // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(utils.GlobalsSql). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.GlobalsSql). Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT"). + Query("BEGIN"). + Reply("BEGIN"). Query(utils.InitialSchemaPg14Sql). - Reply("CREATE SCHEMA") + Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn). Query(CREATE_TEMPLATE). Reply("CREATE DATABASE") helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(sql). Reply("CREATE SCHEMA"). Query(migration.INSERT_MIGRATION_VERSION, "0", "test", []string{sql}). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test result, err := DiffDatabase(context.Background(), []string{"public"}, dbConfig, io.Discard, fsys, DiffSchemaMigra, false, func(cc *pgx.ConnConfig) { if cc.Host == dbConfig.Host { @@ -652,70 +625,3 @@ func TestDropStatements(t *testing.T) { drops := findDropStatements("create table t(); drop table t; alter table t drop column c") assert.Equal(t, []string{"drop table t", "alter table t drop column c"}, drops) } - -func TestLoadSchemas(t *testing.T) { - expected := []string{ - filepath.Join(utils.SchemasDir, "comment", "model.sql"), - filepath.Join(utils.SchemasDir, "model.sql"), - filepath.Join(utils.SchemasDir, "reaction", "dislike", "model.sql"), - filepath.Join(utils.SchemasDir, "reaction", "like", "model.sql"), - } - fsys := afero.NewMemMapFs() - for _, fp := range expected { - require.NoError(t, afero.WriteFile(fsys, fp, nil, 0644)) - } - // Run test - schemas, err := loadDeclaredSchemas(fsys) - // Check error - assert.NoError(t, err) - assert.ElementsMatch(t, expected, schemas) -} - -func TestLoadSchemasSkipsEmptySchemaPathGlobs(t *testing.T) { - fsys := afero.NewMemMapFs() - matched := filepath.Join(utils.SupabaseDirPath, "schemas", "tables", "players.sql") - require.NoError(t, afero.WriteFile(fsys, matched, nil, 0644)) - utils.Config.Db.Migrations.SchemaPaths = []string{ - filepath.Join(utils.SupabaseDirPath, "schemas", "tables", "*.sql"), - filepath.Join(utils.SupabaseDirPath, "schemas", "materialized_views", "*.sql"), - } - t.Cleanup(func() { - utils.Config.Db.Migrations.SchemaPaths = nil - }) - - schemas, err := loadDeclaredSchemas(fsys) - - assert.NoError(t, err) - assert.Equal(t, []string{filepath.ToSlash(matched)}, schemas) -} - -func TestLoadSchemasErrorsOnMissingLiteralSchemaPath(t *testing.T) { - fsys := afero.NewMemMapFs() - utils.Config.Db.Migrations.SchemaPaths = []string{ - filepath.Join(utils.SupabaseDirPath, "schemas", "tables", "players.sql"), - } - t.Cleanup(func() { - utils.Config.Db.Migrations.SchemaPaths = nil - }) - - schemas, err := loadDeclaredSchemas(fsys) - - assert.ErrorContains(t, err, "no files matched pattern") - assert.Empty(t, schemas) -} - -func TestLoadSchemasErrorsWhenAllSchemaPathGlobsAreEmpty(t *testing.T) { - fsys := afero.NewMemMapFs() - utils.Config.Db.Migrations.SchemaPaths = []string{ - filepath.Join(utils.SupabaseDirPath, "schemas", "tables", "*.sql"), - filepath.Join(utils.SupabaseDirPath, "schemas", "views", "*.sql"), - } - t.Cleanup(func() { - utils.Config.Db.Migrations.SchemaPaths = nil - }) - - schemas, err := loadDeclaredSchemas(fsys) - - assert.ErrorContains(t, err, "no files matched pattern") - assert.Empty(t, schemas) -} diff --git a/apps/cli-go/internal/db/diff/pgdelta.go b/apps/cli-go/internal/db/diff/pgdelta.go index 5267f0c6dd..f173c0ac32 100644 --- a/apps/cli-go/internal/db/diff/pgdelta.go +++ b/apps/cli-go/internal/db/diff/pgdelta.go @@ -43,14 +43,30 @@ type DeclarativeOutput struct { Files []DeclarativeFile `json:"files"` } +type PgDeltaTransactionMode string + +const ( + PgDeltaTransactionModeTransactional PgDeltaTransactionMode = "transactional" + PgDeltaTransactionModeNone PgDeltaTransactionMode = "none" +) + +func (m PgDeltaTransactionMode) validate() error { + switch m { + case PgDeltaTransactionModeTransactional, PgDeltaTransactionModeNone: + return nil + default: + return errors.Errorf("unknown pg-delta transaction mode %q", m) + } +} + // PgDeltaPlanFile is one execution-aware migration unit rendered by pg-delta's // renderPlanFiles: a numbered SQL file whose header comments record the unit // number, transaction mode and boundary reason. type PgDeltaPlanFile struct { - Order int `json:"order"` - Name string `json:"name"` - TransactionMode string `json:"transactionMode"` - SQL string `json:"sql"` + Order int `json:"order"` + Name string `json:"name"` + TransactionMode PgDeltaTransactionMode `json:"transactionMode"` + SQL string `json:"sql"` } // PgDeltaDiffOutput is the top-level diff envelope emitted by templates/pgdelta.ts. @@ -182,6 +198,11 @@ func parsePgDeltaDiffOutput(stdout, stderr string) (PgDeltaDiffResult, error) { if err := json.Unmarshal([]byte(stdout), &envelope); err != nil { return PgDeltaDiffResult{}, errors.Errorf("failed to parse pg-delta diff output: %w:\n%s", err, stderr) } + for _, file := range envelope.Files { + if err := file.TransactionMode.validate(); err != nil { + return PgDeltaDiffResult{}, err + } + } result.Files = envelope.Files return result, nil } diff --git a/apps/cli-go/internal/db/diff/pgdelta_migrations.go b/apps/cli-go/internal/db/diff/pgdelta_migrations.go index c8f4a8b244..7a743bac6e 100644 --- a/apps/cli-go/internal/db/diff/pgdelta_migrations.go +++ b/apps/cli-go/internal/db/diff/pgdelta_migrations.go @@ -36,6 +36,14 @@ type WrittenMigration struct { // before pre-existing migrations. The resulting ≤N−1s future-dating is inherent to // second-granularity versions and acceptable once uniqueness is enforced. func WritePgDeltaMigrations(files []PgDeltaPlanFile, base time.Time, name string, fsys afero.Fs) (_ []WrittenMigration, err error) { + // Validate the complete plan before touching the filesystem. The CLI supports + // exactly the two pg-delta execution modes; silently treating a future or + // misspelled mode as transactional would write a migration with wrong semantics. + for _, file := range files { + if err := file.TransactionMode.validate(); err != nil { + return nil, err + } + } single := len(files) == 1 buildSet := func(b time.Time) []WrittenMigration { set := make([]WrittenMigration, len(files)) diff --git a/apps/cli-go/internal/db/diff/pgdelta_migrations_test.go b/apps/cli-go/internal/db/diff/pgdelta_migrations_test.go index 16ace1b7ff..df6f82883e 100644 --- a/apps/cli-go/internal/db/diff/pgdelta_migrations_test.go +++ b/apps/cli-go/internal/db/diff/pgdelta_migrations_test.go @@ -49,6 +49,17 @@ func TestWritePgDeltaMigrations(t *testing.T) { assert.Equal(t, "-- unit 1\n\ncreate table a ();\n", string(contents)) }) + t.Run("rejects an unknown transaction mode before creating files", func(t *testing.T) { + fsys := afero.NewMemMapFs() + files := []PgDeltaPlanFile{{Order: 1, Name: "schema_changes", TransactionMode: "future", SQL: "SELECT 1;"}} + written, err := WritePgDeltaMigrations(files, base, "remote_schema", fsys) + require.ErrorContains(t, err, `unknown pg-delta transaction mode "future"`) + assert.Nil(t, written) + entries, readErr := afero.ReadDir(fsys, ".") + require.NoError(t, readErr) + assert.Empty(t, entries) + }) + t.Run("writes one ordered file per unit with strictly increasing versions", func(t *testing.T) { fsys := afero.NewMemMapFs() files := []PgDeltaPlanFile{ diff --git a/apps/cli-go/internal/db/diff/pgdelta_test.go b/apps/cli-go/internal/db/diff/pgdelta_test.go index ad312273c5..a3395ea6ae 100644 --- a/apps/cli-go/internal/db/diff/pgdelta_test.go +++ b/apps/cli-go/internal/db/diff/pgdelta_test.go @@ -68,4 +68,10 @@ func TestParsePgDeltaDiffOutput(t *testing.T) { assert.ErrorContains(t, err, "failed to parse pg-delta diff output") assert.ErrorContains(t, err, "boom on the edge runtime") }) + + t.Run("rejects an unknown transaction mode", func(t *testing.T) { + stdout := `{"version":1,"files":[{"order":1,"name":"schema_changes","transactionMode":"non-transactional","sql":"SELECT 1;"}]}` + _, err := parsePgDeltaDiffOutput(stdout, "") + assert.ErrorContains(t, err, `unknown pg-delta transaction mode "non-transactional"`) + }) } diff --git a/apps/cli-go/internal/db/diff/shadow.go b/apps/cli-go/internal/db/diff/shadow.go index aa32c91e74..a71a4b2c9e 100644 --- a/apps/cli-go/internal/db/diff/shadow.go +++ b/apps/cli-go/internal/db/diff/shadow.go @@ -10,7 +10,6 @@ import ( "github.com/jackc/pgx/v4" "github.com/spf13/afero" "github.com/supabase/cli/internal/db/start" - "github.com/supabase/cli/internal/pgdelta" "github.com/supabase/cli/internal/utils" ) @@ -24,11 +23,6 @@ type ShadowSource struct { // Source is the connection config for the diff source (the shadow with the // platform baseline + local migrations applied). Source pgconn.Config - // TargetOverride, when non-nil, replaces the diff target with a second shadow - // database (contrib_regression with declarative schemas applied). Mirrors - // DiffDatabase's local-target declarative branch, where the user's local - // database is not diffed at all. - TargetOverride *pgconn.Config } // PgDeltaNextShadowDatabase is one isolated database state used by pg-delta. @@ -164,11 +158,8 @@ func pgDeltaNextShadowConfig(port uint16) pgconn.Config { // PrepareShadowSource provisions the shadow database that DiffDatabase diffs // against, but returns it running instead of diffing + removing, so a native -// caller can run the differ itself. targetLocal mirrors -// utils.IsLocalDatabase(config) — the only target-derived input the shadow prep -// needs. usePgDelta selects the declarative-apply engine for the local-declared -// branch, matching DiffDatabase. On error the shadow container is removed. -func PrepareShadowSource(ctx context.Context, schema []string, targetLocal bool, usePgDelta bool, fsys afero.Fs, options ...func(*pgx.ConnConfig)) (ShadowSource, error) { +// caller can run the differ itself. On error the shadow container is removed. +func PrepareShadowSource(ctx context.Context, fsys afero.Fs, options ...func(*pgx.ConnConfig)) (ShadowSource, error) { shadow, err := CreateShadowDatabase(ctx, utils.Config.Db.ShadowPort) if err != nil { return ShadowSource{}, err @@ -192,36 +183,8 @@ func PrepareShadowSource(ctx context.Context, schema []string, targetLocal bool, Password: utils.Config.Db.Password, Database: "postgres", } - var targetOverride *pgconn.Config - if targetLocal { - declared, err := loadDeclaredSchemas(fsys) - if err != nil { - return ShadowSource{}, err - } - if len(declared) > 0 { - override := shadowConfig - override.Database = "contrib_regression" - if shouldApplyDeclarativeWithPgDelta(usePgDelta) { - declDir := utils.GetDeclarativeDir() - if exists, _ := afero.DirExists(fsys, declDir); exists { - if err := pgdelta.ApplyDeclarative(ctx, override, fsys); err != nil { - return ShadowSource{}, err - } - } else { - if err := migrateBaseDatabase(ctx, override, declared, fsys, options...); err != nil { - return ShadowSource{}, err - } - } - } else { - if err := migrateBaseDatabase(ctx, override, declared, fsys, options...); err != nil { - return ShadowSource{}, err - } - } - targetOverride = &override - } - } ok = true - return ShadowSource{Container: shadow, Source: shadowConfig, TargetOverride: targetOverride}, nil + return ShadowSource{Container: shadow, Source: shadowConfig}, nil } // PrepareRawShadow provisions a bare shadow database (created + healthy, with no diff --git a/apps/cli-go/internal/testing/helper/history.go b/apps/cli-go/internal/testing/helper/history.go index 95c846b7ad..594bda9271 100644 --- a/apps/cli-go/internal/testing/helper/history.go +++ b/apps/cli-go/internal/testing/helper/history.go @@ -6,7 +6,9 @@ import ( ) func MockMigrationHistory(conn *pgtest.MockConn) *pgtest.MockConn { - conn.Query(migration.SET_LOCK_TIMEOUT). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.SET_LOCK_TIMEOUT). Query(migration.CREATE_VERSION_SCHEMA). Reply("CREATE SCHEMA"). Query(migration.CREATE_VERSION_TABLE). @@ -14,15 +16,21 @@ func MockMigrationHistory(conn *pgtest.MockConn) *pgtest.MockConn { Query(migration.ADD_STATEMENTS_COLUMN). Reply("ALTER TABLE"). Query(migration.ADD_NAME_COLUMN). - Reply("ALTER TABLE") + Reply("ALTER TABLE"). + Query("COMMIT"). + Reply("COMMIT") return conn } func MockSeedHistory(conn *pgtest.MockConn) *pgtest.MockConn { - conn.Query(migration.SET_LOCK_TIMEOUT). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.SET_LOCK_TIMEOUT). Query(migration.CREATE_VERSION_SCHEMA). Reply("CREATE SCHEMA"). Query(migration.CREATE_SEED_TABLE). - Reply("CREATE TABLE") + Reply("CREATE TABLE"). + Query("COMMIT"). + Reply("COMMIT") return conn } diff --git a/apps/cli-go/internal/testing/helper/privileges.go b/apps/cli-go/internal/testing/helper/privileges.go index 4dcca23f58..7f043f81dc 100644 --- a/apps/cli-go/internal/testing/helper/privileges.go +++ b/apps/cli-go/internal/testing/helper/privileges.go @@ -9,11 +9,15 @@ import "github.com/supabase/cli/pkg/pgtest" // than imported from the start package to avoid an import cycle with that package's own // internal (package start) tests. func MockApiPrivilegesRevoke(conn *pgtest.MockConn) *pgtest.MockConn { - conn.Query("alter default privileges for role postgres in schema public\n revoke select, insert, update, delete on tables from anon, authenticated, service_role"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("alter default privileges for role postgres in schema public\n revoke select, insert, update, delete on tables from anon, authenticated, service_role"). Reply("ALTER DEFAULT PRIVILEGES"). Query("alter default privileges for role postgres in schema public\n revoke usage, select on sequences from anon, authenticated, service_role"). Reply("ALTER DEFAULT PRIVILEGES"). Query("alter default privileges for role postgres in schema public\n revoke execute on functions from anon, authenticated, service_role"). - Reply("ALTER DEFAULT PRIVILEGES") + Reply("ALTER DEFAULT PRIVILEGES"). + Query("COMMIT"). + Reply("COMMIT") return conn } diff --git a/apps/cli-go/pkg/migration/apply_test.go b/apps/cli-go/pkg/migration/apply_test.go index e6df97721f..3b3340524a 100644 --- a/apps/cli-go/pkg/migration/apply_test.go +++ b/apps/cli-go/pkg/migration/apply_test.go @@ -110,10 +110,14 @@ func TestApplyMigrations(t *testing.T) { mockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(testSchema). Reply("CREATE SCHEMA"). Query(INSERT_MIGRATION_VERSION, "0", "schema", []string{testSchema}). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := ApplyMigrations(context.Background(), pending, conn.MockClient(t), testMigrations) // Check error @@ -126,13 +130,17 @@ func TestApplyMigrations(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(SET_LOCK_TIMEOUT). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(SET_LOCK_TIMEOUT). Query(CREATE_VERSION_SCHEMA). Reply("CREATE SCHEMA"). Query(CREATE_VERSION_TABLE). ReplyError(pgerrcode.InsufficientPrivilege, "permission denied for relation supabase_migrations"). Query(ADD_STATEMENTS_COLUMN). - Query(ADD_NAME_COLUMN) + Query(ADD_NAME_COLUMN). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := ApplyMigrations(context.Background(), pending, conn.MockClient(t), fsys) // Check error @@ -161,10 +169,14 @@ func TestApplyMigrations(t *testing.T) { mockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(testSchema). ReplyError(pgerrcode.UndefinedTable, `relation "supabase_migrations.schema_migrations" does not exist`). Query(INSERT_MIGRATION_VERSION, "0", "schema", []string{testSchema}). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := ApplyMigrations(context.Background(), pending, conn.MockClient(t), testMigrations) // Check error @@ -187,7 +199,9 @@ func TestApplyMigrations(t *testing.T) { } func mockMigrationHistory(conn *pgtest.MockConn) *pgtest.MockConn { - conn.Query(SET_LOCK_TIMEOUT). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(SET_LOCK_TIMEOUT). Query(CREATE_VERSION_SCHEMA). Reply("CREATE SCHEMA"). Query(CREATE_VERSION_TABLE). @@ -195,6 +209,8 @@ func mockMigrationHistory(conn *pgtest.MockConn) *pgtest.MockConn { Query(ADD_STATEMENTS_COLUMN). Reply("ALTER TABLE"). Query(ADD_NAME_COLUMN). - Reply("ALTER TABLE") + Reply("ALTER TABLE"). + Query("COMMIT"). + Reply("COMMIT") return conn } diff --git a/apps/cli-go/pkg/migration/drop_test.go b/apps/cli-go/pkg/migration/drop_test.go index 644fb69be0..388ca937b8 100644 --- a/apps/cli-go/pkg/migration/drop_test.go +++ b/apps/cli-go/pkg/migration/drop_test.go @@ -14,8 +14,12 @@ func TestDropSchemas(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(DropObjects). - Reply("INSERT 0") + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(DropObjects). + Reply("INSERT 0"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := DropUserSchemas(context.Background(), conn.MockClient(t)) // Check error @@ -26,8 +30,12 @@ func TestDropSchemas(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(DropObjects). - ReplyError(pgerrcode.InsufficientPrivilege, "permission denied for relation supabase_migrations") + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(DropObjects). + ReplyError(pgerrcode.InsufficientPrivilege, "permission denied for relation supabase_migrations"). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := DropUserSchemas(context.Background(), conn.MockClient(t)) // Check error diff --git a/apps/cli-go/pkg/migration/file.go b/apps/cli-go/pkg/migration/file.go index 83c07f53c7..b2a0ec48bb 100644 --- a/apps/cli-go/pkg/migration/file.go +++ b/apps/cli-go/pkg/migration/file.go @@ -27,13 +27,14 @@ type MigrationFile struct { } var ( - migrateFilePattern = regexp.MustCompile(`^([0-9]+)_(.*)\.sql$`) - typeNamePattern = regexp.MustCompile(`type "([^"]+)" does not exist`) - createIndexPattern = regexp.MustCompile(`^CREATE\s+(UNIQUE\s+)?INDEX\s+CONCURRENTLY(\s|\z)`) - reindexPattern = regexp.MustCompile(`^REINDEX(\s|\().*\sCONCURRENTLY(\s|\z)`) - vacuumPattern = regexp.MustCompile(`^VACUUM(\s|\(|\z)`) - alterSystemPattern = regexp.MustCompile(`^ALTER\s+SYSTEM(\s|\z)`) - clusterPattern = regexp.MustCompile(`^CLUSTER(\s|\z)`) + migrateFilePattern = regexp.MustCompile(`^([0-9]+)_(.*)\.sql$`) + typeNamePattern = regexp.MustCompile(`type "([^"]+)" does not exist`) + createIndexPattern = regexp.MustCompile(`^CREATE\s+(UNIQUE\s+)?INDEX\s+CONCURRENTLY(\s|\z)`) + reindexPattern = regexp.MustCompile(`^REINDEX(\s|\().*\sCONCURRENTLY(\s|\z)`) + vacuumPattern = regexp.MustCompile(`^VACUUM(\s|\(|\z)`) + alterSystemPattern = regexp.MustCompile(`^ALTER\s+SYSTEM(\s|\z)`) + clusterPattern = regexp.MustCompile(`^CLUSTER(\s|\z)`) + transactionControlPattern = regexp.MustCompile(`^(BEGIN|START\s+TRANSACTION|COMMIT|END|ROLLBACK|ABORT|PREPARE\s+TRANSACTION)(\s|\z)`) ) func NewMigrationFromFile(path string, fsys fs.FS) (*MigrationFile, error) { @@ -86,6 +87,14 @@ func isPipelineIncompatible(sql string) bool { clusterPattern.MatchString(upper) } +// hasTransactionControl reports whether a statement controls the transaction +// boundary itself. Files containing these statements must be executed exactly as +// authored: automatically adding BEGIN/COMMIT would nest or otherwise change the +// user's transaction semantics. +func hasTransactionControl(sql string) bool { + return transactionControlPattern.MatchString(strings.ToUpper(trimLeadingSQLComments(sql))) +} + func trimLeadingSQLComments(sql string) string { trimmed := strings.TrimLeftFunc(sql, func(r rune) bool { return r == '\ufeff' || r == ' ' || r == '\t' || r == '\n' || r == '\r' @@ -142,22 +151,67 @@ func (m *MigrationFile) ExecBatch(ctx context.Context, conn *pgx.Conn) error { return errors.Errorf("%w\n%s", err, strings.Join(msg, "\n")) } - flushBatch := func() error { + flushBatch := func(transactional, rollbackOnError bool) error { if batchSize == 0 { return nil } + if transactional { + if _, err := conn.Exec(ctx, "BEGIN"); err != nil { + return errors.Errorf("failed to begin migration transaction: %w", err) + } + } if result, err := conn.PgConn().ExecBatch(ctx, batch).ReadAll(); err != nil { + if rollbackOnError { + _, _ = conn.Exec(ctx, "ROLLBACK") + } return formatError(err, executed+len(result)) } + if transactional { + if _, err := conn.Exec(ctx, "COMMIT"); err != nil { + _, _ = conn.Exec(ctx, "ROLLBACK") + return errors.Errorf("failed to commit migration transaction: %w", err) + } + } executed += batchSize batch = &pgconn.Batch{} batchSize = 0 return nil } + // An authored transaction boundary owns the file's transaction semantics. Do + // not add automatic boundaries around any part of such a file. The history + // insert remains last, after every authored SQL statement has succeeded. + authoredTransaction := false + for _, statement := range m.Statements { + if hasTransactionControl(statement) { + authoredTransaction = true + break + } + } + if authoredTransaction { + for _, line := range m.Statements { + batch.ExecParams(line, nil, nil, nil, nil) + batchSize++ + } + if err := flushBatch(false, true); err != nil { + return err + } + // Queue history only after the authored transaction stream has completed + // successfully. In particular, never pipeline it after an authored COMMIT: + // a preceding failure can turn that COMMIT into ROLLBACK while a later + // history insert would otherwise execute in a fresh implicit transaction. + if len(m.Version) > 0 { + if err := m.insertVersionSQL(conn, batch); err != nil { + return err + } + batchSize++ + } + return flushBatch(false, true) + } + for _, line := range m.Statements { if isPipelineIncompatible(line) { - if err := flushBatch(); err != nil { + if err := flushBatch(true, true); err != nil { return err } if _, err := conn.PgConn().Exec(ctx, line).ReadAll(); err != nil { @@ -178,7 +232,7 @@ func (m *MigrationFile) ExecBatch(ctx context.Context, conn *pgx.Conn) error { batchSize++ } - return flushBatch() + return flushBatch(true, true) } func markError(stat string, pos int) string { diff --git a/apps/cli-go/pkg/migration/file_test.go b/apps/cli-go/pkg/migration/file_test.go index 49fb0f7f68..e560944c96 100644 --- a/apps/cli-go/pkg/migration/file_test.go +++ b/apps/cli-go/pkg/migration/file_test.go @@ -49,10 +49,14 @@ func TestMigrationFile(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.Statements[0]). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.Statements[0]). Reply("CREATE SCHEMA"). Query(INSERT_MIGRATION_VERSION, "0", "", migration.Statements). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := migration.ExecBatch(context.Background(), conn.MockClient(t)) // Check error @@ -72,14 +76,22 @@ func TestMigrationFile(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.Statements[0]). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.Statements[0]). Reply("CREATE TABLE"). + Query("COMMIT"). + Reply("COMMIT"). SimpleQuery(migration.Statements[1]). Reply("CREATE INDEX"). + Query("BEGIN"). + Reply("BEGIN"). Query(migration.Statements[2]). Reply("ALTER TABLE"). Query(INSERT_MIGRATION_VERSION, migration.Version, migration.Name, migration.Statements). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := migration.ExecBatch(context.Background(), conn.MockClient(t)) // Check error @@ -94,14 +106,88 @@ func TestMigrationFile(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(INSERT_MIGRATION_VERSION, migration.Version, migration.Name, migration.Statements). - Reply("INSERT 0 1") + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(INSERT_MIGRATION_VERSION, migration.Version, migration.Name, migration.Statements). + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := migration.ExecBatch(context.Background(), conn.MockClient(t)) // Check error assert.NoError(t, err) }) + t.Run("keeps SET LOCAL and history in the same transaction", func(t *testing.T) { + migration := MigrationFile{ + Statements: []string{ + "SET LOCAL check_function_bodies = off", + "CREATE FUNCTION public.answer() RETURNS int LANGUAGE sql AS 'SELECT 42'", + }, + Version: "20260101000000", + Name: "create_answer", + } + conn := pgtest.NewConn() + defer conn.Close(t) + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.Statements[0]). + Reply("SET"). + Query(migration.Statements[1]). + Reply("CREATE FUNCTION"). + Query(INSERT_MIGRATION_VERSION, migration.Version, migration.Name, migration.Statements). + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") + + err := migration.ExecBatch(context.Background(), conn.MockClient(t)) + assert.NoError(t, err) + }) + + t.Run("preserves user-authored transaction boundaries", func(t *testing.T) { + migration := MigrationFile{ + Statements: []string{"BEGIN", "SET LOCAL check_function_bodies = off", "COMMIT"}, + Version: "20260101000000", + Name: "authored_transaction", + } + conn := pgtest.NewConn() + defer conn.Close(t) + // Exactly the authored BEGIN/COMMIT are sent; no automatic wrapper nests them. + conn.Query(migration.Statements[0]). + Reply("BEGIN"). + Query(migration.Statements[1]). + Reply("SET"). + Query(migration.Statements[2]). + Reply("COMMIT"). + Query(INSERT_MIGRATION_VERSION, migration.Version, migration.Name, migration.Statements). + Reply("INSERT 0 1") + + err := migration.ExecBatch(context.Background(), conn.MockClient(t)) + assert.NoError(t, err) + }) + + t.Run("never records history after an authored transaction fails", func(t *testing.T) { + migration := MigrationFile{ + Statements: []string{"BEGIN", "CREATE TABLE broken (", "COMMIT"}, + Version: "20260101000000", + Name: "broken_authored_transaction", + } + conn := pgtest.NewConn() + defer conn.Close(t) + conn.Query(migration.Statements[0]). + Reply("BEGIN"). + Query(migration.Statements[1]). + ReplyError(pgerrcode.SyntaxError, "syntax error at end of input"). + Query(migration.Statements[2]). + Reply("ROLLBACK"). + Query("ROLLBACK"). + Reply("ROLLBACK") + + err := migration.ExecBatch(context.Background(), conn.MockClient(t)) + assert.ErrorContains(t, err, "syntax error at end of input") + assert.ErrorContains(t, err, "At statement: 1") + }) + t.Run("reports pipeline incompatible statement errors with statement index", func(t *testing.T) { migration := MigrationFile{ Statements: []string{ @@ -115,8 +201,12 @@ func TestMigrationFile(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.Statements[0]). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.Statements[0]). Reply("CREATE TABLE"). + Query("COMMIT"). + Reply("COMMIT"). SimpleQuery(migration.Statements[1]). ReplyError("25001", "CREATE INDEX CONCURRENTLY cannot be executed within a pipeline") // Run test @@ -134,10 +224,14 @@ func TestMigrationFile(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.Statements[0]). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.Statements[0]). ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`). Query(INSERT_MIGRATION_VERSION, "0", "", migration.Statements). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := migration.ExecBatch(context.Background(), conn.MockClient(t)) // Check error @@ -153,10 +247,14 @@ func TestMigrationFile(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.Statements[0]). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.Statements[0]). ReplyError("42704", `type "ltree" does not exist`). Query(INSERT_MIGRATION_VERSION, "0", "", migration.Statements). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := migration.ExecBatch(context.Background(), conn.MockClient(t)) // Check error @@ -175,10 +273,14 @@ func TestMigrationFile(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.Statements[0]). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.Statements[0]). ReplyError("42704", `type "extensions.ltree" does not exist`). Query(INSERT_MIGRATION_VERSION, "0", "", migration.Statements). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := migration.ExecBatch(context.Background(), conn.MockClient(t)) // Check error - should NOT contain hint since type is already schema-qualified diff --git a/apps/cli-go/pkg/migration/history.go b/apps/cli-go/pkg/migration/history.go index 9f156faa4a..4a1f1acf29 100644 --- a/apps/cli-go/pkg/migration/history.go +++ b/apps/cli-go/pkg/migration/history.go @@ -10,7 +10,7 @@ import ( ) const ( - SET_LOCK_TIMEOUT = "SET lock_timeout = '4s'" + SET_LOCK_TIMEOUT = "SET LOCAL lock_timeout = '4s'" CREATE_VERSION_SCHEMA = "CREATE SCHEMA IF NOT EXISTS supabase_migrations" CREATE_VERSION_TABLE = "CREATE TABLE IF NOT EXISTS supabase_migrations.schema_migrations (version text NOT NULL PRIMARY KEY)" ADD_STATEMENTS_COLUMN = "ALTER TABLE supabase_migrations.schema_migrations ADD COLUMN IF NOT EXISTS statements text[]" @@ -30,16 +30,24 @@ const ( // TODO: support overriding `supabase_migrations.schema_migrations` with user defined . func CreateMigrationTable(ctx context.Context, conn *pgx.Conn) error { // This must be run without prepared statements because each statement in the batch depends on - // the previous schema change. The lock timeout will be reset when implicit transaction ends. + // the previous schema change. The explicit transaction makes SET LOCAL effective and non-leaking. batch := pgconn.Batch{} batch.ExecParams(SET_LOCK_TIMEOUT, nil, nil, nil, nil) batch.ExecParams(CREATE_VERSION_SCHEMA, nil, nil, nil, nil) batch.ExecParams(CREATE_VERSION_TABLE, nil, nil, nil, nil) batch.ExecParams(ADD_STATEMENTS_COLUMN, nil, nil, nil, nil) batch.ExecParams(ADD_NAME_COLUMN, nil, nil, nil, nil) + if _, err := conn.Exec(ctx, "BEGIN"); err != nil { + return errors.Errorf("failed to begin migration table transaction: %w", err) + } if _, err := conn.PgConn().ExecBatch(ctx, &batch).ReadAll(); err != nil { + _, _ = conn.Exec(ctx, "ROLLBACK") return errors.Errorf("failed to create migration table: %w", err) } + if _, err := conn.Exec(ctx, "COMMIT"); err != nil { + _, _ = conn.Exec(ctx, "ROLLBACK") + return errors.Errorf("failed to commit migration table transaction: %w", err) + } return nil } @@ -53,14 +61,22 @@ func ReadMigrationTable(ctx context.Context, conn *pgx.Conn) ([]MigrationFile, e func CreateSeedTable(ctx context.Context, conn *pgx.Conn) error { // This must be run without prepared statements because each statement in the batch depends on - // the previous schema change. The lock timeout will be reset when implicit transaction ends. + // the previous schema change. The explicit transaction makes SET LOCAL effective and non-leaking. batch := pgconn.Batch{} batch.ExecParams(SET_LOCK_TIMEOUT, nil, nil, nil, nil) batch.ExecParams(CREATE_VERSION_SCHEMA, nil, nil, nil, nil) batch.ExecParams(CREATE_SEED_TABLE, nil, nil, nil, nil) + if _, err := conn.Exec(ctx, "BEGIN"); err != nil { + return errors.Errorf("failed to begin seed table transaction: %w", err) + } if _, err := conn.PgConn().ExecBatch(ctx, &batch).ReadAll(); err != nil { + _, _ = conn.Exec(ctx, "ROLLBACK") return errors.Errorf("failed to create seed table: %w", err) } + if _, err := conn.Exec(ctx, "COMMIT"); err != nil { + _, _ = conn.Exec(ctx, "ROLLBACK") + return errors.Errorf("failed to commit seed table transaction: %w", err) + } return nil } diff --git a/apps/cli-go/pkg/migration/seed_test.go b/apps/cli-go/pkg/migration/seed_test.go index db4337b54c..e224e43a37 100644 --- a/apps/cli-go/pkg/migration/seed_test.go +++ b/apps/cli-go/pkg/migration/seed_test.go @@ -127,11 +127,15 @@ func TestSeedData(t *testing.T) { } func mockSeedHistory(conn *pgtest.MockConn) *pgtest.MockConn { - conn.Query(SET_LOCK_TIMEOUT). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(SET_LOCK_TIMEOUT). Query(CREATE_VERSION_SCHEMA). Reply("CREATE SCHEMA"). Query(CREATE_SEED_TABLE). - Reply("CREATE TABLE") + Reply("CREATE TABLE"). + Query("COMMIT"). + Reply("COMMIT") return conn } @@ -145,8 +149,12 @@ func TestSeedGlobals(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(testGlobals). - Reply("CREATE ROLE") + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(testGlobals). + Reply("CREATE ROLE"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := SeedGlobals(context.Background(), pending, conn.MockClient(t), testMigrations) // Check error @@ -166,8 +174,12 @@ func TestSeedGlobals(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(testGlobals). - ReplyError(pgerrcode.InvalidCatalogName, `database "postgres" does not exist`) + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(testGlobals). + ReplyError(pgerrcode.InvalidCatalogName, `database "postgres" does not exist`). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := SeedGlobals(context.Background(), pending, conn.MockClient(t), testMigrations) // Check error diff --git a/apps/cli/src/legacy/commands/branches/create/create.integration.test.ts b/apps/cli/src/legacy/commands/branches/create/create.integration.test.ts index d9b7029288..ec677a8341 100644 --- a/apps/cli/src/legacy/commands/branches/create/create.integration.test.ts +++ b/apps/cli/src/legacy/commands/branches/create/create.integration.test.ts @@ -72,7 +72,7 @@ const tempRoot = useLegacyTempWorkdir("supabase-branches-create-int-"); interface SetupOpts { readonly format?: "text" | "json" | "stream-json"; readonly goOutput?: "env" | "pretty" | "json" | "toml" | "yaml"; - readonly response?: CreatedBranch; + readonly response?: unknown; readonly status?: number; readonly network?: "fail"; readonly gated?: boolean; @@ -328,6 +328,37 @@ describe("legacy branches create integration", () => { }).pipe(Effect.provide(layer)); }); + it.live("accepts branch timestamps with an RFC3339 numeric offset", () => { + const { layer, out } = setup({ + response: { + ...CREATED, + created_at: "2026-05-27T03:32:03+02:30", + updated_at: "2026-05-27T03:32:04+02:30", + }, + }); + return Effect.gen(function* () { + yield* legacyBranchesCreate({ ...baseFlags, name: Option.some("feat-x") }); + expect(out.stdoutText).toContain("2026-05-27"); + }).pipe(Effect.provide(layer)); + }); + + it.live("surfaces malformed successful responses as schema errors, not network errors", () => { + const { layer } = setup({ + response: { ...CREATED, created_at: "not-a-timestamp" }, + }); + return Effect.gen(function* () { + const exit = yield* Effect.exit( + legacyBranchesCreate({ ...baseFlags, name: Option.some("feat-x") }), + ); + expect(Exit.isFailure(exit)).toBe(true); + if (Exit.isFailure(exit)) { + const json = JSON.stringify(exit.cause); + expect(json).toContain("LegacyApiResponseSchemaError"); + expect(json).not.toContain("LegacyBranchesCreateNetworkError"); + } + }).pipe(Effect.provide(layer)); + }); + it.live("emits Go-byte-exact indented JSON for --output json", () => { const { layer, out } = setup({ goOutput: "json" }); return Effect.gen(function* () { diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index be1efe65c5..7760be4b2a 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -28,16 +28,15 @@ bundled Go binary. ## Files Read -| Path | Format | When | -| -------------------------------------------------- | ---------- | ----------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | -| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) | -| `/supabase/database/**` (declarative dir) | SQL | local target when declarative schemas exist | -| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution | -| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only: edge-runtime image tag | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: explicit `--from/--to migrations` catalog | +| Path | Format | When | +| ----------------------------------------------- | ---------- | ----------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | +| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) | +| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution | +| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only: edge-runtime image tag | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: explicit `--from/--to migrations` catalog | ## Files Written @@ -91,8 +90,9 @@ bundled Go binary. Progress to stderr (`Creating shadow database...`, `Diffing schemas[: ]`, `Finished supabase db diff on branch .`, drop-statement warning, and the -`--file` write warning). The SQL diff prints to stdout when neither `--file` nor -explicit `--output` is set. +`--file` write warning). A configured `[db.migrations].schema_paths` also prints a +transition warning because it no longer changes the diff target. The SQL diff +prints to stdout when neither `--file` nor explicit `--output` is set. ### `--output-format json` / `stream-json` @@ -108,6 +108,9 @@ Progress strings still go to stderr; stdout carries a single structured envelope binary (their side effects are Go's); the Go child's telemetry is disabled so the single `cli_command_executed` event comes from this TS command. - Explicit `--from`/`--to` mode always uses pg-delta and writes to `--output` (or stdout). +- Normal mode always compares the migrations shadow with the selected live + database. Declarative files and `schema_paths` never replace that target; use + `supabase db schema declarative sync` for declarative comparison. ### `--use-pg-schema` is deprecated (CLI-1960) — keep-in-Go exception diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index bd7ba9b769..fc2b833bb0 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -6,10 +6,7 @@ import { detectGitBranch } from "../../../../shared/git/git-branch.ts"; import { Output } from "../../../../shared/output/output.service.ts"; import { LegacyCliConfig } from "../../../config/legacy-cli-config.service.ts"; import { legacyAqua, legacyYellow } from "../../../shared/legacy-colors.ts"; -import { - legacyReadDbToml, - legacyResolveDeclarativeDir, -} from "../../../shared/legacy-db-config.toml-read.ts"; +import { legacyReadDbToml } from "../../../shared/legacy-db-config.toml-read.ts"; import { LegacyDbConfigResolver } from "../../../shared/legacy-db-config.service.ts"; import type { LegacyDbConnType } from "../../../shared/legacy-db-target-flags.ts"; import { legacyGetHostname } from "../../../shared/legacy-hostname.ts"; @@ -22,6 +19,7 @@ import { LegacyTelemetryState } from "../../../telemetry/legacy-telemetry-state. import { legacyParseBoolEnv, legacyResolveDiffEngine, + legacySchemaPathsTransitionWarning, legacyShouldUsePgDelta, } from "../shared/legacy-diff-engine.ts"; import { @@ -33,14 +31,7 @@ import { LegacyPgDeltaEngine, type LegacyPgDeltaDatabaseEndpoint, type LegacyPgDeltaEndpoint, - type LegacyPgDeltaExportManifest, - type LegacyPgDeltaSqlFile, } from "../shared/legacy-pgdelta-engine.service.ts"; -import { - LegacyLoadPgDeltaSqlFiles, - LegacyLoadPgDeltaSqlPaths, - LegacyReadPgDeltaExportManifest, -} from "../shared/legacy-pgdelta-files.ts"; import { legacyWritePgDeltaMigrations } from "../shared/legacy-pgdelta-migrations.write.ts"; import { legacyIsPgDeltaDebugEnabled, @@ -411,6 +402,9 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy denoVersion: cfg.denoVersion, }; const formatOptions = Option.getOrElse(cfg.pgDelta.formatOptions, () => ""); + if (cfg.migrationSchemaPaths !== undefined && cfg.migrationSchemaPaths.length > 0) { + yield* output.raw(legacySchemaPathsTransitionWarning, "stderr"); + } // Engine resolution (Go's `db.go:110`): the pg-delta env/config/flag gate, // read from the (possibly remote-merged) config. @@ -437,46 +431,6 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // precedes the call because the high-level boundary intentionally exposes // no partially-provisioned resource to the handler. yield* output.raw(diffingMessage, "stderr"); - let declarativeFiles: ReadonlyArray | undefined; - let declarativeManifest: LegacyPgDeltaExportManifest | undefined; - if (pgDelta.implementation === "next" && resolved.isLocal) { - if (cfg.migrationSchemaPaths !== undefined && cfg.migrationSchemaPaths.length > 0) { - declarativeFiles = yield* LegacyLoadPgDeltaSqlPaths( - fs, - path, - cliConfig.workdir, - cfg.migrationSchemaPaths, - ); - } else { - const declarativeDirSetting = legacyResolveDeclarativeDir(path, cfg.pgDelta); - const declarativeDir = path.isAbsolute(declarativeDirSetting) - ? declarativeDirSetting - : path.join(cliConfig.workdir, declarativeDirSetting); - const hasDeclarativeDir = cfg.pgDelta.enabled - ? yield* fs.exists(declarativeDir).pipe(Effect.orElseSucceed(() => false)) - : false; - if (hasDeclarativeDir) { - const loaded = yield* LegacyLoadPgDeltaSqlFiles(fs, path, declarativeDir); - if (loaded.length > 0) { - declarativeFiles = loaded; - declarativeManifest = yield* LegacyReadPgDeltaExportManifest( - fs, - path, - declarativeDir, - ); - } - } else { - const schemasDir = path.join(cliConfig.workdir, "supabase", "schemas"); - const hasSchemasDir = yield* fs - .exists(schemasDir) - .pipe(Effect.orElseSucceed(() => false)); - if (hasSchemasDir) { - const loaded = yield* LegacyLoadPgDeltaSqlFiles(fs, path, schemasDir); - if (loaded.length > 0) declarativeFiles = loaded; - } - } - } - } const result = yield* pgDelta.diffDatabase({ context: ctx, target: { @@ -485,12 +439,9 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy connection: resolved.conn, connectOptions: { isLocal: resolved.isLocal, dnsResolver }, }, - targetLocal: resolved.isLocal, schema: flags.schema, formatOptions, ...(connType === "linked" && linkedRef !== undefined ? { projectRef: linkedRef } : {}), - ...(declarativeFiles !== undefined ? { declarativeFiles } : {}), - ...(declarativeManifest !== undefined ? { declarativeManifest } : {}), debug: legacyIsPgDeltaDebugEnabled(), }); return { sql: result.sql, files: result.files }; @@ -498,8 +449,6 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy : yield* Effect.gen(function* () { const shadow = yield* seam.provisionShadow({ mode: "diff", - targetLocal: resolved.isLocal, - usePgDelta: false, schema: flags.schema, ...(connType === "linked" && linkedRef !== undefined ? { projectRef: linkedRef } : {}), }); @@ -507,7 +456,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy yield* output.raw(diffingMessage, "stderr"); const sql = yield* legacyDiffMigra(ctx, { source: shadow.sourceUrl, - target: shadow.targetUrlOverride ?? targetUrl, + target: targetUrl, schema: flags.schema, connectOptions: { isLocal: resolved.isLocal, dnsResolver }, }); @@ -559,6 +508,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy ? file.suffix.replace(/^_/u, "") : file.name, sql: file.sql, + transactionMode: file.transactionMode, })), }).pipe(Effect.mapError((cause) => new LegacyDbDiffWriteError({ message: cause.message }))); for (const unit of writtenUnits) writtenFiles.push(unit.path); diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 04a283fd31..0d079d186e 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -47,7 +47,6 @@ interface SetupOpts { // Exact suffixes returned by the next renderer, parallel to `diffFiles`. readonly diffSuffixes?: ReadonlyArray; readonly pgDeltaImplementation?: "legacy" | "next"; - readonly targetOverride?: string; readonly oom?: boolean; // edge-runtime OOMs; the bash fallback returns `diffSql` readonly delegateStdout?: string; // stdout returned by a captured Go-delegate run readonly networkId?: string; // --network-id value forwarded to docker runs @@ -62,8 +61,6 @@ function setup(workdir: string, opts: SetupOpts = {}) { const provisionCalls: Array<{ mode: string; - targetLocal: boolean; - usePgDelta: boolean; projectRef?: string; }> = []; const removedContainers: string[] = []; @@ -72,12 +69,11 @@ function setup(workdir: string, opts: SetupOpts = {}) { execInherit: () => Effect.succeed(0), ensureLocalDatabaseStarted: () => Effect.void, ensureLocalPostgresImageCurrent: () => Effect.void, - provisionShadow: ({ mode, targetLocal, usePgDelta, projectRef }) => { - provisionCalls.push({ mode, targetLocal, usePgDelta, projectRef }); + provisionShadow: ({ mode, projectRef }) => { + provisionCalls.push({ mode, projectRef }); return Effect.succeed({ container: "shadow-1", sourceUrl: "postgres://postgres:postgres@127.0.0.1:54320/postgres", - targetUrlOverride: opts.targetOverride, }); }, provisionNextShadow: () => Effect.die("provisionNextShadow not used"), @@ -100,10 +96,17 @@ function setup(workdir: string, opts: SetupOpts = {}) { ? { suffix: opts.diffSuffixes[index] } : {}), sql: file.sql, - transactional: true, + transactionMode: "transactional" as const, })) : sql.length > 0 - ? [{ sequence: 1, name: "schema_changes", sql, transactional: true }] + ? [ + { + sequence: 1, + name: "schema_changes", + sql, + transactionMode: "transactional" as const, + }, + ] : []; return { changes: files.length > 0, @@ -280,7 +283,7 @@ describe("legacy db diff", () => { const s = setup(tmp.current, { diffSql: "create table players ();\n" }); return Effect.gen(function* () { yield* legacyDbDiff(flags()); - expect(s.provisionCalls).toEqual([{ mode: "diff", targetLocal: true, usePgDelta: false }]); + expect(s.provisionCalls).toEqual([{ mode: "diff", projectRef: undefined }]); expect(stdout(s.out)).toBe("create table players ();\n\n"); expect(stderr(s.out)).toContain("Creating shadow database..."); expect(stderr(s.out)).toContain("Diffing schemas..."); @@ -297,7 +300,6 @@ describe("legacy db diff", () => { expect(s.provisionCalls).toEqual([]); expect(s.databaseDiffCalls).toHaveLength(1); expect(s.databaseDiffCalls[0]).toMatchObject({ - targetLocal: true, schema: ["public"], target: { kind: "database", @@ -319,7 +321,7 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("next local diff gives configured schema_paths precedence", () => { + it.effect("next local diff ignores schema_paths and declarative files", () => { mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); writeFileSync( join(tmp.current, "supabase", "config.toml"), @@ -343,55 +345,10 @@ describe("legacy db diff", () => { }); return Effect.gen(function* () { yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); - expect(s.databaseDiffCalls[0]?.declarativeFiles).toEqual([ - { name: "supabase/configured.sql", sql: "create table configured ();\n" }, - ]); - expect(s.databaseDiffCalls[0]?.declarativeManifest).toBeUndefined(); - }).pipe(Effect.provide(s.layer)); - }); - - it.effect("next local diff loads the enabled declarative directory and manifest", () => { - const declarativeDir = join(tmp.current, "supabase", "database"); - mkdirSync(declarativeDir, { recursive: true }); - writeFileSync( - join(tmp.current, "supabase", "config.toml"), - ["[experimental.pgdelta]", "enabled = true", ""].join("\n"), - ); - writeFileSync(join(declarativeDir, "public.sql"), "create table public.t ();\n"); - writeFileSync( - join(declarativeDir, ".pgdelta-export.json"), - JSON.stringify({ formatVersion: 1, redactSecrets: true, scope: "database" }), - ); - const s = setup(tmp.current, { - pgDeltaImplementation: "next", - diffSql: "create table result ();\n", - }); - return Effect.gen(function* () { - yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); - expect(s.databaseDiffCalls[0]?.declarativeFiles).toEqual([ - { name: "public.sql", sql: "create table public.t ();\n" }, - ]); - expect(s.databaseDiffCalls[0]?.declarativeManifest).toEqual({ - redactSecrets: true, - scope: "database", - }); - }).pipe(Effect.provide(s.layer)); - }); - - it.effect("next local diff falls back to supabase/schemas", () => { - const schemasDir = join(tmp.current, "supabase", "schemas"); - mkdirSync(schemasDir, { recursive: true }); - writeFileSync(join(schemasDir, "fallback.sql"), "create table fallback ();\n"); - const s = setup(tmp.current, { - pgDeltaImplementation: "next", - diffSql: "create table result ();\n", - }); - return Effect.gen(function* () { - yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); - expect(s.databaseDiffCalls[0]?.declarativeFiles).toEqual([ - { name: "fallback.sql", sql: "create table fallback ();\n" }, - ]); - expect(s.databaseDiffCalls[0]?.declarativeManifest).toBeUndefined(); + expect(s.databaseDiffCalls[0]).not.toHaveProperty("declarativeFiles"); + expect(s.databaseDiffCalls[0]).not.toHaveProperty("declarativeManifest"); + expect(stderr(s.out)).toContain("schema_paths no longer changes the target"); + expect(stdout(s.out)).toBe("create table result ();\n\n"); }).pipe(Effect.provide(s.layer)); }); @@ -450,7 +407,6 @@ describe("legacy db diff", () => { const s = setup(tmp.current, { diffSql: "create table players ();\n" }); return Effect.gen(function* () { yield* legacyDbDiff(flags()); - expect(s.provisionCalls[0]?.usePgDelta).toBe(false); // The local default never passes a ref, so the shadow uses base config. expect(s.provisionCalls[0]?.projectRef).toBeUndefined(); }).pipe(Effect.provide(s.layer)); @@ -464,16 +420,13 @@ describe("legacy db diff", () => { }); return Effect.gen(function* () { yield* legacyDbDiff(flags({ linked: Option.some(true) })); - expect(s.provisionCalls[0]?.targetLocal).toBe(false); + expect(s.provisionCalls[0]?.projectRef).toBe("abcdefghijklmnopqrst"); expect(s.cache.cached).toBe(true); }).pipe(Effect.provide(s.layer)); }); - it.effect("uses the seam's target override for the local declarative branch", () => { - const s = setup(tmp.current, { - targetOverride: "postgres://postgres:postgres@127.0.0.1:54320/contrib_regression", - diffSql: "create table o ();\n", - }); + it.effect("uses the selected local database as the migra target", () => { + const s = setup(tmp.current, { diffSql: "create table o ();\n" }); return Effect.gen(function* () { yield* legacyDbDiff(flags()); expect(stdout(s.out)).toBe("create table o ();\n\n"); @@ -609,16 +562,37 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("writes a timestamped migration when --file is set instead of printing", () => { - const s = setup(tmp.current, { diffSql: "create table f ();\n" }); + it.effect("writes live-only SQL with --file even when declarative targets are configured", () => { + mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + [ + "[db.migrations]", + 'schema_paths = ["database/*.sql"]', + "", + "[experimental.pgdelta]", + "enabled = true", + "", + ].join("\n"), + ); + writeFileSync( + join(tmp.current, "supabase", "database", "declarative.sql"), + "create table declarative_only ();\n", + ); + const s = setup(tmp.current, { + pgDeltaImplementation: "next", + diffSql: "create table live_only ();\n", + }); return Effect.gen(function* () { - yield* legacyDbDiff(flags({ file: Option.some("my_diff") })); + yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), file: Option.some("my_diff") })); expect(stdout(s.out)).toBe(""); + expect(stderr(s.out)).toContain("schema_paths no longer changes the target"); expect(stderr(s.out)).toContain("WARNING: The diff tool is not foolproof"); const dir = join(tmp.current, "supabase", "migrations"); const files = readdirSync(dir); expect(files).toHaveLength(1); expect(files[0]).toMatch(/^\d{14}_my_diff\.sql$/); + expect(readFileSync(join(dir, files[0]!), "utf8")).toBe("create table live_only ();\n"); }).pipe(Effect.provide(s.layer)); }); diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index a5c49e8a65..fb28ffca76 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -3,6 +3,8 @@ Native Effect port. Pulls the remote schema into either a new timestamped migration (diffing a throwaway shadow against the remote, native pg-delta or migra) or declarative files (`--declarative`, native pg-delta export). The +migration-style path always compares migrations with the selected live database; +declarative files and `[db.migrations].schema_paths` cannot replace its target. initial-migra pull (no local migrations) seeds the migration file with a native `pg_dump` of the remote schema (a Docker `pg_dump` container, with IPv4 transaction-pooler fallback) and then appends the migra diff. `--experimental`'s @@ -120,6 +122,9 @@ written to `. Plus the `--use-pg-delta` deprecation line, the prompt. On success the PostRun line `Finished supabase db pull.` is printed to stdout. +A configured `[db.migrations].schema_paths` prints a transition warning on the +migration path directing users to `supabase db schema declarative sync`. + ### `--output-format json` / `stream-json` Progress strings still go to stderr; stdout carries a single structured envelope diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index baa9830f33..def9a3cf3e 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -42,6 +42,7 @@ import { legacyParseBoolEnv, legacyResolveDeclarativeFromArgs, legacyResolvePullDiffEngine, + legacySchemaPathsTransitionWarning, legacyShouldUsePgDelta, } from "../shared/legacy-diff-engine.ts"; import { legacyDiffMigra } from "../shared/legacy-migra.ts"; @@ -63,14 +64,7 @@ import type { LegacyPgDeltaContext } from "../shared/legacy-pgdelta.ts"; import { LegacyPgDeltaEngine, type LegacyPgDeltaDatabaseEndpoint, - type LegacyPgDeltaExportManifest, - type LegacyPgDeltaSqlFile, } from "../shared/legacy-pgdelta-engine.service.ts"; -import { - LegacyLoadPgDeltaSqlFiles, - LegacyLoadPgDeltaSqlPaths, - LegacyReadPgDeltaExportManifest, -} from "../shared/legacy-pgdelta-files.ts"; import { legacyIsPgDeltaDebugEnabled } from "../shared/legacy-pgdelta.ts"; import { legacySaveEmptyPgDeltaPullDebug } from "./pull.debug.ts"; import { LegacyDeclarativeSeam } from "../shared/legacy-pgdelta.seam.service.ts"; @@ -477,6 +471,14 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy return; } + if ( + !delegatesExperimentalPull && + toml.migrationSchemaPaths !== undefined && + toml.migrationSchemaPaths.length > 0 + ) { + yield* output.raw(legacySchemaPathsTransitionWarning, "stderr"); + } + // Go's `EXPERIMENTAL` structured-dump branch (`pull.go:49-61`) stays // delegated to Go. pg_dump itself is now native (used by the initial-migra // path below), but this branch also calls `format.WriteStructuredSchemas` @@ -639,69 +641,26 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy : "Diffing schemas...\n", "stderr", ); - let declarativeFiles: ReadonlyArray | undefined; - let declarativeManifest: LegacyPgDeltaExportManifest | undefined; - if (usePgDeltaDiff && pgDeltaEngine.implementation === "next" && resolved.isLocal) { - if (toml.migrationSchemaPaths !== undefined && toml.migrationSchemaPaths.length > 0) { - declarativeFiles = yield* LegacyLoadPgDeltaSqlPaths( - fs, - path, - cliConfig.workdir, - toml.migrationSchemaPaths, - ); - } else { - const declarativeDirSetting = legacyResolveDeclarativeDir(path, toml.pgDelta); - const declarativeDir = path.isAbsolute(declarativeDirSetting) - ? declarativeDirSetting - : path.join(cliConfig.workdir, declarativeDirSetting); - const hasDeclarativeDir = toml.pgDelta.enabled - ? yield* fs.exists(declarativeDir).pipe(Effect.orElseSucceed(() => false)) - : false; - if (hasDeclarativeDir) { - const loaded = yield* LegacyLoadPgDeltaSqlFiles(fs, path, declarativeDir); - if (loaded.length > 0) { - declarativeFiles = loaded; - declarativeManifest = yield* LegacyReadPgDeltaExportManifest( - fs, - path, - declarativeDir, - ); - } - } else { - const schemasDir = path.join(cliConfig.workdir, "supabase", "schemas"); - if (yield* fs.exists(schemasDir).pipe(Effect.orElseSucceed(() => false))) { - const loaded = yield* LegacyLoadPgDeltaSqlFiles(fs, path, schemasDir); - if (loaded.length > 0) declarativeFiles = loaded; - } - } - } - } - const diffOutcome = usePgDeltaDiff ? yield* withPoolerFallback(targetEndpoint, (target) => pgDeltaEngine.diffDatabase({ context: ctx, target, - targetLocal: resolved.isLocal, schema: diffSchema, formatOptions, projectRef: connType === "linked" ? linkedRef : undefined, debug: legacyIsPgDeltaDebugEnabled(), - ...(declarativeFiles !== undefined ? { declarativeFiles } : {}), - ...(declarativeManifest !== undefined ? { declarativeManifest } : {}), }), ) : yield* Effect.gen(function* () { const shadow = yield* seam.provisionShadow({ mode: "diff", - targetLocal: resolved.isLocal, - usePgDelta: false, schema: diffSchema, projectRef: connType === "linked" ? linkedRef : undefined, }); return yield* legacyDiffMigra(ctx, { source: shadow.sourceUrl, - target: shadow.targetUrlOverride ?? targetUrl, + target: targetUrl, schema: diffSchema, connectOptions: { isLocal: resolved.isLocal, dnsResolver }, }).pipe( @@ -794,6 +753,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy name: file.name, suffix: file.suffix, sql: file.sql, + transactionMode: file.transactionMode, })), }).pipe( Effect.mapError((cause) => new LegacyDbPullWriteError({ message: cause.message })), diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index dbbdea699a..af8d4f24e5 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -77,7 +77,6 @@ interface SetupOpts { readonly pipedAnswers?: ReadonlyArray; readonly yes?: boolean; readonly experimental?: boolean; - readonly shadowTargetOverride?: string; readonly promptConfirmResponses?: ReadonlyArray; readonly resolvedRef?: string; // Fail the first edge-runtime run with this message (the second succeeds with @@ -116,8 +115,6 @@ function setup(workdir: string, opts: SetupOpts = {}) { const provisionCalls: Array<{ mode: string; - usePgDelta: boolean; - targetLocal: boolean; projectRef?: string; }> = []; const removedContainers: string[] = []; @@ -126,12 +123,11 @@ function setup(workdir: string, opts: SetupOpts = {}) { execInherit: () => Effect.succeed(0), ensureLocalDatabaseStarted: () => Effect.void, ensureLocalPostgresImageCurrent: () => Effect.void, - provisionShadow: ({ mode, usePgDelta, targetLocal, projectRef }) => { - provisionCalls.push({ mode, usePgDelta, targetLocal, projectRef }); + provisionShadow: ({ mode, projectRef }) => { + provisionCalls.push({ mode, projectRef }); return Effect.succeed({ container: "shadow-1", sourceUrl: "postgres://postgres:postgres@127.0.0.1:54320/postgres", - targetUrlOverride: opts.shadowTargetOverride, }); }, provisionNextShadow: () => Effect.die("provisionNextShadow not used"), @@ -145,7 +141,6 @@ function setup(workdir: string, opts: SetupOpts = {}) { operation: "diff" | "export"; targetRef: string; projectRef?: string; - targetLocal?: boolean; }> = []; let engineDiffCount = 0; const pgDeltaEngine = Layer.succeed( @@ -158,7 +153,6 @@ function setup(workdir: string, opts: SetupOpts = {}) { operation: "diff", targetRef: input.target.ref, projectRef: input.projectRef, - targetLocal: input.targetLocal, }); engineDiffCount += 1; if (opts.edgeFailFirstWith !== undefined && engineDiffCount === 1) { @@ -203,11 +197,14 @@ function setup(workdir: string, opts: SetupOpts = {}) { if (typeof sql !== "string" || typeof name !== "string") { throw new Error("invalid file"); } + if (transactionMode !== "transactional" && transactionMode !== "none") { + throw new Error(`unknown transaction mode ${String(transactionMode)}`); + } return { sequence: index + 1, name, sql, - transactional: transactionMode !== "none", + transactionMode, }; }); return Effect.succeed({ @@ -630,8 +627,12 @@ describe("legacy db pull", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("pulls with the default migra engine", () => { + it.effect("pulls with migra and warns that schema_paths cannot replace the target", () => { seedMigration(tmp.current, "20240101000000"); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + ["[db.migrations]", 'schema_paths = ["database/*.sql"]', ""].join("\n"), + ); const s = setup(tmp.current, { remoteVersions: ["20240101000000"], edgeStdout: "create table remote ();\n", @@ -639,8 +640,9 @@ describe("legacy db pull", () => { }); return Effect.gen(function* () { yield* legacyDbPull(flags()); - expect(s.provisionCalls[0]?.usePgDelta).toBe(false); + expect(s.provisionCalls[0]?.mode).toBe("diff"); const err = streamText(s.out, "stderr"); + expect(err).toContain("schema_paths no longer changes the target"); // Go's `ConnectByConfig` prints the Connecting line to stderr before dialing // (`internal/utils/connect.go:348`), ahead of any other pull output. expect(err).toContain("Connecting to remote database...\n"); @@ -835,7 +837,7 @@ describe("legacy db pull", () => { expect(s.dumpCalls[0]?.env["EXTRA_SED"]).toBe("/^--/d"); expect(s.dumpCalls[0]?.env["EXCLUDED_SCHEMAS"]).toContain("auth"); // The diff ran against the shadow with the migra engine (no schema filter). - expect(s.provisionCalls[0]?.usePgDelta).toBe(false); + expect(s.provisionCalls[0]?.mode).toBe("diff"); // The migration file holds the dump output followed by the appended diff. const dir = join(tmp.current, "supabase", "migrations"); const file = readdirSync(dir).find((f) => f.endsWith("_remote_schema.sql")); @@ -1644,20 +1646,16 @@ describe("legacy db pull", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("db pull --local provisions a local-target shadow and uses the target override", () => { - // Go derives the shadow targetLocal from utils.IsLocalDatabase and substitutes - // the declarative contrib_regression target override (diff.go:190,196-197); - // the native handler must pass targetLocal and honor shadow.targetUrlOverride. + it.effect("db pull --local diffs migrations against the selected local database", () => { seedMigration(tmp.current, "20240101000000"); const s = setup(tmp.current, { remoteVersions: ["20240101000000"], edgeStdout: "create table remote ();\n", yes: true, - shadowTargetOverride: "postgres://postgres:postgres@127.0.0.1:54320/contrib_regression", }); return Effect.gen(function* () { yield* legacyDbPull(flags({ local: Option.some(true) })); - expect(s.provisionCalls[0]?.targetLocal).toBe(true); + expect(s.provisionCalls[0]).toEqual({ mode: "diff", projectRef: undefined }); // A local target prints the local wording (Go's `IsLocalDatabase` branch in // `ConnectByConfigStream`, `internal/utils/connect.go:344-346`). expect(streamText(s.out, "stderr")).toContain("Connecting to local database...\n"); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 8c98dddee1..56bea69784 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -93,6 +93,13 @@ are mutually exclusive. ## Notes - Requires `--experimental` or `[experimental.pgdelta] enabled = true`. +- The declarative directory is the complete, hand-authored desired state. An + object omitted from it is intended to be removed, including extensions. This + is deterministic regardless of whether the directory was generated, written + by hand, or has a `.pgdelta-export.json` manifest. +- Projects upgrading from the legacy workflow should regenerate declarations or + add declarations for every extension they intend to retain before syncing. + Review the existing drop-statement warning before applying destructive changes. - `--file` sets the migration filename stem (default `declarative_sync`); `--name` overrides it. In a TTY without `--name`/`--yes`, the name is prompted. - When no declarative files exist, a TTY offers to generate them (from local) first. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 3597b17a80..2adfb3bcac 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -947,14 +947,14 @@ describe("legacy db schema declarative sync integration", () => { name: "transactional", suffix: "_1", sql: "ALTER TABLE a ADD COLUMN b int;", - transactional: true, + transactionMode: "transactional", }, { sequence: 2, name: "non_transactional", suffix: "_2", sql: "ALTER TYPE mood ADD VALUE 'fine';", - transactional: false, + transactionMode: "none", }, ], }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-diff-engine.ts b/apps/cli/src/legacy/commands/db/shared/legacy-diff-engine.ts index 12079b9ab8..ef1528387a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-diff-engine.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-diff-engine.ts @@ -3,6 +3,9 @@ // byte-identical to the Go CLI. No Effect / service dependencies — unit-tested // directly. +export const legacySchemaPathsTransitionWarning = + "WARNING: [db.migrations].schema_paths no longer changes the target of db diff or migration-style db pull. These commands always compare local migrations with the selected database. Use `supabase db schema declarative sync` to compare declarative schema files.\n"; + /** * Whether pg-delta is the active default engine. Mirrors Go's `shouldUsePgDelta` * (`db.go:375-376`): `utils.IsPgDeltaEnabled() || usePgDelta || viper.GetBool("EXPERIMENTAL_PG_DELTA")`. diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts index 3505e73c36..6ad91f2e12 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts @@ -8,6 +8,7 @@ import { LegacyPgDeltaEngineError, type LegacyPgDeltaDiffResult, type LegacyPgDeltaEndpoint, + type LegacyPgDeltaTransactionMode, } from "./legacy-pgdelta-engine.service.ts"; import { legacyDeclarativeExportPgDelta, @@ -26,7 +27,7 @@ function normalizeDiff( readonly files: ReadonlyArray<{ readonly order: number; readonly name: string; - readonly transactionMode: string; + readonly transactionMode: LegacyPgDeltaTransactionMode; readonly sql: string; }>; }, @@ -39,7 +40,7 @@ function normalizeDiff( sequence: file.order, name: file.name, sql: file.sql, - transactional: file.transactionMode !== "non-transactional", + transactionMode: file.transactionMode, })), ...(debug ? { debug: { stderr: result.stderr } } : {}), }; @@ -98,8 +99,6 @@ export const legacyPgDeltaLegacyEngineLayer = Layer.effect( Effect.gen(function* () { const shadow = yield* seam.provisionShadow({ mode: "diff", - targetLocal: input.targetLocal, - usePgDelta: true, schema: input.schema, ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), }); @@ -114,7 +113,7 @@ export const legacyPgDeltaLegacyEngineLayer = Layer.effect( return yield* provideRuntime( legacyDiffPgDelta(input.context, { sourceRef: shadow.sourceUrl, - targetRef: shadow.targetUrlOverride ?? input.target.ref, + targetRef: input.target.ref, schema: input.schema, formatOptions: input.formatOptions, }), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 766a453dfa..886aa020df 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -62,7 +62,7 @@ function normalizeNextDiff( readonly sequence: number; readonly suffix: string | null; readonly sql: string; - readonly transactional: boolean; + readonly transactionMode: "transactional" | "none"; readonly actionCount: number; }>; readonly debug?: { @@ -81,7 +81,7 @@ function normalizeNextDiff( name: `segment_${file.sequence}`, suffix: file.suffix, sql: file.sql, - transactional: file.transactional, + transactionMode: file.transactionMode, actionCount: file.actionCount, })), ...(result.debug !== undefined @@ -238,8 +238,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), }); const migrations = parseLegacyConnectionString(shadow.migrationsUrl); - const declarative = parseLegacyConnectionString(shadow.declarativeUrl); - if (migrations === undefined || declarative === undefined) { + if (migrations === undefined) { return yield* Effect.fail( new LegacyPgDeltaEngineError({ message: "failed to parse pg-delta next shadow database URL", @@ -247,37 +246,6 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( }), ); } - if (input.declarativeFiles !== undefined) { - const [migrationsPool, declarativePool] = yield* Effect.all( - [ - legacyAcquirePgPool(migrations, { isLocal: true, dnsResolver: "native" }), - legacyAcquirePgPool(declarative, { isLocal: true, dnsResolver: "native" }), - ], - { concurrency: 2 }, - ); - const result = yield* adapter.planDeclarativeSchema({ - targetPool: migrationsPool, - shadowPool: declarativePool, - files: input.declarativeFiles, - allowDrops: true, - debug: input.debug, - reorder: true, - ...legacyPgDeltaNextIsolatedShadowPlanOptions, - schema: input.schema, - ...(input.declarativeManifest !== undefined - ? { manifest: input.declarativeManifest } - : {}), - }); - const debugDirectory = - result.debug !== undefined - ? yield* saveDebugArtifacts(input.context.cwd, "declarativePlan", { - ...result.debug, - diagnostics: result.diagnostics, - }) - : undefined; - yield* rejectBlockingDiagnostic("declarativePlan", result.diagnostics); - return normalizeNextDiff(result, debugDirectory); - } const migrationsPool = yield* legacyAcquirePgPool(migrations, { isLocal: true, dnsResolver: "native", diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts index a5163f7263..a88bddbf32 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts @@ -37,6 +37,8 @@ export interface LegacyPgDeltaExportManifest { readonly files?: ReadonlyArray; } +export type LegacyPgDeltaTransactionMode = "transactional" | "none"; + export interface LegacyPgDeltaRenderedFile { readonly sequence: number; /** Legacy semantic unit name. */ @@ -44,7 +46,7 @@ export interface LegacyPgDeltaRenderedFile { /** Next renderer's exact filename suffix (`null`, `_1`, `_2`, ...). */ readonly suffix?: string | null; readonly sql: string; - readonly transactional: boolean; + readonly transactionMode: LegacyPgDeltaTransactionMode; readonly actionCount?: number; } @@ -79,10 +81,6 @@ export interface LegacyPgDeltaExplicitDiffInput extends LegacyPgDeltaCommonInput export interface LegacyPgDeltaDatabaseDiffInput extends LegacyPgDeltaCommonInput { readonly target: LegacyPgDeltaDatabaseEndpoint; - readonly targetLocal: boolean; - /** Present when the local desired state is declarative SQL rather than the live DB. */ - readonly declarativeFiles?: ReadonlyArray; - readonly declarativeManifest?: LegacyPgDeltaExportManifest; } interface LegacyPgDeltaDeclarativeExportInput extends LegacyPgDeltaCommonInput { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts index d24e0562f4..368a8cf20e 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts @@ -1,10 +1,9 @@ -import { Data, Effect, type FileSystem, Option, type Path } from "effect"; +import { Data, Effect, type FileSystem, type Path } from "effect"; import type { LegacyPgDeltaExportManifest, LegacyPgDeltaSqlFile, } from "./legacy-pgdelta-engine.service.ts"; -import { legacyResolveSqlGlobFiles } from "../../../shared/legacy-seed-ops.ts"; const EXPORT_MANIFEST_FILE = ".pgdelta-export.json"; @@ -134,35 +133,3 @@ export const LegacyLoadPgDeltaSqlFiles = Effect.fnUntraced(function* ( } return files; }); - -/** Loads `[db.migrations].schema_paths` in configured pattern/application order. */ -export const LegacyLoadPgDeltaSqlPaths = Effect.fnUntraced(function* ( - fs: FileSystem.FileSystem, - path: Path.Path, - workdir: string, - patterns: ReadonlyArray, -) { - const resolved = yield* legacyResolveSqlGlobFiles(fs, path, patterns, workdir); - if (resolved.files.length === 0) { - return yield* Effect.fail( - filesError( - Option.isSome(resolved.warning) - ? resolved.warning.value - : "no declarative schema files matched schema_paths", - ), - ); - } - const files: Array = []; - for (const file of resolved.files) { - const full = path.isAbsolute(file) ? file : path.join(workdir, file); - const sql = yield* fs - .readFileString(full) - .pipe( - Effect.mapError((error) => - filesError(`failed to read declarative schema file: ${error.message}`), - ), - ); - files.push({ name: file.split("\\").join("/"), sql }); - } - return files; -}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts index 35d9de726c..0546c042f0 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts @@ -5,6 +5,7 @@ import { legacyFormatMigrationTimestamp, legacyGetMigrationPath, } from "../../../shared/legacy-migration-file.ts"; +import type { LegacyPgDeltaTransactionMode } from "./legacy-pgdelta-engine.service.ts"; /** A migration file written by a diff/pull, paired with its history version. */ export interface LegacyWrittenMigration { @@ -62,11 +63,21 @@ export const legacyWritePgDeltaMigrations = ( readonly name: string; readonly suffix?: string | null; readonly sql: string; + readonly transactionMode: LegacyPgDeltaTransactionMode; }>; }, ): Effect.Effect, LegacyPgDeltaMigrationWriteError> => Effect.gen(function* () { const { workdir, name, files } = opts; + for (const file of files) { + if (file.transactionMode !== "transactional" && file.transactionMode !== "none") { + return yield* Effect.fail( + new LegacyPgDeltaMigrationWriteError({ + message: `unknown pg-delta transaction mode ${JSON.stringify(file.transactionMode)}`, + }), + ); + } + } const single = files.length === 1; const buildSet = (baseMillis: number): Array => files.map((file, i) => { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index dd59eee4a8..395dfcbd0b 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -270,7 +270,7 @@ function legacyNormalizePgDeltaNextRenderedFiles( sequence: index + 1, suffix: file.suffix, sql: file.contents, - transactional: file.transactional, + transactionMode: file.transactional ? "transactional" : "none", actionCount: file.actionCount, })); } diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts index 80b6cbb130..0c4e235bcf 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts @@ -1,6 +1,8 @@ import type { Pool } from "pg"; import { Context, Data, type Effect } from "effect"; +import type { LegacyPgDeltaTransactionMode } from "./legacy-pgdelta-engine.service.ts"; + export type LegacyPgDeltaNextOperation = | "diff" | "declarativeExport" @@ -28,7 +30,7 @@ export interface LegacyPgDeltaNextRenderedFile { readonly sequence: number; readonly suffix: string | null; readonly sql: string; - readonly transactional: boolean; + readonly transactionMode: LegacyPgDeltaTransactionMode; readonly actionCount: number; } diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index 5c9d1c51c2..a276e2fec6 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -302,14 +302,14 @@ describe("LegacyPgDeltaNextAdapter", () => { sequence: 1, suffix: "_1", sql: "begin source-facts;\n", - transactional: true, + transactionMode: "transactional", actionCount: 2, }, { sequence: 2, suffix: "_2", sql: "alter desired-facts;\n", - transactional: false, + transactionMode: "none", actionCount: 1, }, ]); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts index 8a0f7aafc1..7de8a4176c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts @@ -558,6 +558,9 @@ describeDockerLive("pg-delta next declarative extension baseline (live)", () => findExtensionDeclaration(schemasDir, "pg_net"); const pgcryptoFile = findExtensionDeclaration(schemasDir, "pgcrypto"); findExtensionDeclaration(schemasDir, "uuid-ossp"); + // The directory itself is the complete desired-state contract. A missing + // manifest must not preserve an extension omitted from the SQL files. + await rm(path.join(schemasDir, ".pgdelta-export.json")); const containersBeforeEmpty = projectContainerIds(config); const migrationsBeforeEmpty = migrationFiles(projectDir); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.integration.test.ts index c64f9f624a..63a7d1169f 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.integration.test.ts @@ -176,6 +176,39 @@ describe("legacyDiffPgDelta", () => { Effect.provide(Layer.mergeAll(edge.layer, probe, BunServices.layer)), ); }); + + it.effect("rejects an unknown transaction mode", () => { + const edge = fakeEdgeRuntime({ + stdout: JSON.stringify({ + version: 1, + files: [ + { + order: 1, + name: "schema_changes", + transactionMode: "non-transactional", + sql: "SELECT 1;", + }, + ], + }), + }); + return legacyDiffPgDelta(CTX, { + targetRef: "postgresql://t", + sourceRef: "", + schema: [], + formatOptions: "", + }).pipe( + Effect.exit, + Effect.tap((exit) => + Effect.sync(() => { + expect(failError(exit)?.constructor.name).toBe("LegacyPgDeltaDiffParseError"); + expect((failError(exit) as { message: string }).message).toContain( + 'unknown pg-delta transaction mode "non-transactional"', + ); + }), + ), + Effect.provide(Layer.mergeAll(edge.layer, probe, BunServices.layer)), + ); + }); }); describe("legacyDeclarativeExportPgDelta", () => { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts index d9e87dda80..5d9ed36c47 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts @@ -394,7 +394,7 @@ const makeLegacyDeclarativeSeam = (resolved: BinaryResolution) => ); }), ), - provisionShadow: ({ mode, targetLocal, usePgDelta, schema, projectRef }) => + provisionShadow: ({ mode, schema, projectRef }) => Effect.scoped( Effect.gen(function* () { if (!("found" in resolved)) { @@ -410,8 +410,6 @@ const makeLegacyDeclarativeSeam = (resolved: BinaryResolution) => "__shadow", "--mode", mode, - ...(targetLocal ? ["--target-local"] : []), - ...(usePgDelta ? ["--use-pg-delta"] : []), ...(schema.length > 0 ? ["--schema", schema.join(",")] : []), ...(Option.isSome(networkId) ? ["--network-id", networkId.value] : []), // Linked path only: pass the resolved ref so the hidden `db __shadow` @@ -461,9 +459,7 @@ const makeLegacyDeclarativeSeam = (resolved: BinaryResolution) => bytes.set(chunk, offset); offset += chunk.length; } - // stdout is three newline-separated lines: container id, source URL, - // and an optional second-database URL. Legacy diff uses the third URL - // only when its local-target declarative branch redirects the target. + // stdout is two newline-separated lines: container id and source URL. // The URLs arrive WITHOUT a password — the Go seam prints them via // ToPostgresURLWithoutPassword so it never logs a credential to stdout // (CWE-312). The shadow uses the local Postgres password, so we re-inject @@ -476,7 +472,6 @@ const makeLegacyDeclarativeSeam = (resolved: BinaryResolution) => const lines = new TextDecoder().decode(bytes).split(/\r?\n/u); const container = (lines[0] ?? "").trim(); const sourceUrl = (lines[1] ?? "").trim(); - const targetOverride = (lines[2] ?? "").trim(); if (container.length === 0 || sourceUrl.length === 0) { return yield* Effect.fail(failure()); } @@ -493,10 +488,6 @@ const makeLegacyDeclarativeSeam = (resolved: BinaryResolution) => return { container, sourceUrl: legacyInjectPostgresPassword(sourceUrl, password), - targetUrlOverride: - targetOverride.length > 0 - ? legacyInjectPostgresPassword(targetOverride, password) - : undefined, } satisfies LegacyShadowSource; }), ), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts index 847b063dec..0f89404d99 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts @@ -20,11 +20,6 @@ export interface LegacyShadowSource { readonly container: string; /** The diff source Postgres URL (the provisioned shadow). */ readonly sourceUrl: string; - /** - * Optional second live database. For legacy diff it replaces the target with - * `contrib_regression` after Go applies declarative schemas. - */ - readonly targetUrlOverride: string | undefined; } /** The independently hosted databases used by the pg-delta next planner. */ @@ -101,14 +96,12 @@ interface LegacyDeclarativeSeamShape { * Provisions a live shadow database via the bundled Go binary's hidden * `db __shadow` command and returns it running (the container is NOT removed — * the caller must call `removeShadowContainer` when the diff completes). This - * is the diff "source" that both the migra and pg-delta engines run against in - * `db diff` / `db pull`, mirroring Go's `DiffDatabase` (`differ(shadow, target)`). + * is the migration-state source that both the migra and pg-delta engines run + * against in `db diff` / `db pull`. * Go's shadow-provisioning progress is teed to stderr. */ readonly provisionShadow: (opts: { readonly mode: LegacyShadowMode; - readonly targetLocal: boolean; - readonly usePgDelta: boolean; readonly schema: ReadonlyArray; /** * Resolved linked project ref, passed ONLY on the `--linked` path so the diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.ts index bf2fb2c9bc..cf341f5384 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.ts @@ -21,6 +21,7 @@ import { LegacyDeclarativeParseOutputError, LegacyPgDeltaDiffParseError, } from "./legacy-pgdelta.errors.ts"; +import type { LegacyPgDeltaTransactionMode } from "./legacy-pgdelta-engine.service.ts"; const PG_DELTA_NPM_REGISTRY_ENV = "PGDELTA_NPM_REGISTRY"; @@ -47,14 +48,18 @@ export interface LegacyDeclarativeOutput { interface LegacyPgDeltaPlanFile { readonly order: number; readonly name: string; - readonly transactionMode: string; + readonly transactionMode: LegacyPgDeltaTransactionMode; readonly sql: string; } /** The pg-delta diff envelope. Mirrors Go's `PgDeltaDiffOutput`. */ interface LegacyPgDeltaDiffOutput { readonly version: number; - readonly files: ReadonlyArray; + readonly files: ReadonlyArray< + Omit & { + readonly transactionMode: string; + } + >; } /** @@ -230,7 +235,19 @@ export const legacyDiffPgDelta = Effect.fnUntraced(function* ( }:\n${result.stderr}`, }), }); - const files = envelope.files ?? []; + const rawFiles = envelope.files ?? []; + const files: Array = []; + for (const file of rawFiles) { + const transactionMode = file.transactionMode; + if (transactionMode !== "transactional" && transactionMode !== "none") { + return yield* Effect.fail( + new LegacyPgDeltaDiffParseError({ + message: `unknown pg-delta transaction mode ${JSON.stringify(transactionMode)}`, + }), + ); + } + files.push({ ...file, transactionMode }); + } // Flatten to one blob for callers that need it; unit header comments keep the // transaction boundaries visible (mirrors Go's `joinPgDeltaFiles`). const sql = files.map((file) => file.sql).join("\n\n"); diff --git a/apps/cli/src/legacy/commands/link/link.handler.ts b/apps/cli/src/legacy/commands/link/link.handler.ts index f1cca7c3e5..eca1beae99 100644 --- a/apps/cli/src/legacy/commands/link/link.handler.ts +++ b/apps/cli/src/legacy/commands/link/link.handler.ts @@ -1,4 +1,4 @@ -import type { ApiClient } from "@supabase/api/effect"; +import { isSupabaseApiResponseSchemaError, type ApiClient } from "@supabase/api/effect"; import { Effect, FileSystem, Option, Path } from "effect"; import type { PlatformError } from "effect/PlatformError"; import * as HttpClientError from "effect/unstable/http/HttpClientError"; @@ -18,7 +18,10 @@ import { } from "../../../shared/telemetry/event-catalog.ts"; import { legacyDashboardUrl } from "../../shared/legacy-profile.ts"; import { legacyMapTenantApiKeysError } from "../../shared/legacy-get-tenant-api-keys.ts"; -import { sanitizeLegacyErrorBody } from "../../shared/legacy-http-errors.ts"; +import { + LegacyApiResponseSchemaError, + sanitizeLegacyErrorBody, +} from "../../shared/legacy-http-errors.ts"; import { legacyLinkServicesCore } from "../../shared/legacy-link-services-core.ts"; import { legacyExtractServiceKeys } from "../../shared/legacy-tenant-keys.ts"; import { legacyTempPaths } from "../../shared/legacy-temp-paths.ts"; @@ -42,8 +45,16 @@ const classifyProjectError = ( cause: unknown, ): Effect.Effect< Option.Option, - LegacyLinkProjectStatusError | LegacyLinkProjectStatusNetworkError + LegacyApiResponseSchemaError | LegacyLinkProjectStatusError | LegacyLinkProjectStatusNetworkError > => { + if (isSupabaseApiResponseSchemaError(cause)) { + return Effect.fail( + new LegacyApiResponseSchemaError({ + operationId: cause.operationId, + message: cause.message, + }), + ); + } if (HttpClientError.isHttpClientError(cause) && cause.response !== undefined) { const status = cause.response.status; if (status === 404) { diff --git a/apps/cli/src/legacy/commands/link/link.integration.test.ts b/apps/cli/src/legacy/commands/link/link.integration.test.ts index 870adb758c..cfe975662f 100644 --- a/apps/cli/src/legacy/commands/link/link.integration.test.ts +++ b/apps/cli/src/legacy/commands/link/link.integration.test.ts @@ -1,6 +1,7 @@ import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { join } from "node:path"; +import { SupabaseApiResponseSchemaError } from "@supabase/api/effect"; import { describe, expect, it } from "@effect/vitest"; import { Effect, Exit, Layer, Option } from "effect"; import * as HttpClient from "effect/unstable/http/HttpClient"; @@ -353,6 +354,26 @@ describe("legacy link integration", () => { }).pipe(Effect.provide(layer)); }); + it.live("surfaces project response schema failures separately from network failures", () => { + const { layer } = setup({ + project: { + fail: new SupabaseApiResponseSchemaError( + "v1GetProject", + new Error("created_at is not RFC3339"), + ), + }, + }); + return Effect.gen(function* () { + const exit = yield* Effect.exit(legacyLink(flags())); + expect(Exit.isFailure(exit)).toBe(true); + if (Exit.isFailure(exit)) { + const json = JSON.stringify(exit.cause); + expect(json).toContain("LegacyApiResponseSchemaError"); + expect(json).not.toContain("LegacyLinkProjectStatusNetworkError"); + } + }).pipe(Effect.provide(layer)); + }); + it.live("fails with auth error when api-keys returns non-200", () => { const { layer } = setup({ apiKeys: { fail: legacyStatusCodeFailure(401) } }); return Effect.gen(function* () { diff --git a/apps/cli/src/legacy/shared/legacy-db-config.service.ts b/apps/cli/src/legacy/shared/legacy-db-config.service.ts index 2b28e4397e..f95597fbce 100644 --- a/apps/cli/src/legacy/shared/legacy-db-config.service.ts +++ b/apps/cli/src/legacy/shared/legacy-db-config.service.ts @@ -7,6 +7,7 @@ import type { } from "../config/legacy-project-ref.errors.ts"; import type { LegacyProjectRefReadError } from "./legacy-temp-paths.ts"; import type { LegacyDbConnectError } from "./legacy-db-connection.errors.ts"; +import type { LegacyApiResponseSchemaError } from "./legacy-http-errors.ts"; import type { LegacyDbConfigConnectTempRoleError, LegacyDbConfigIpv6Error, @@ -41,6 +42,7 @@ export type LegacyDbConfigError = | LegacyDbConfigConnectTempRoleError | LegacyDbConfigPoolerLoginError | LegacyDbConnectError + | LegacyApiResponseSchemaError // The `--linked` path resolves the access token lazily via // `LegacyPlatformApiFactory.make` (only when minting a temp login role), so the // auth-required / invalid-token / api-config errors surface from the resolver diff --git a/apps/cli/src/legacy/shared/legacy-http-errors.ts b/apps/cli/src/legacy/shared/legacy-http-errors.ts index 7639e83f99..80a21bddef 100644 --- a/apps/cli/src/legacy/shared/legacy-http-errors.ts +++ b/apps/cli/src/legacy/shared/legacy-http-errors.ts @@ -1,5 +1,5 @@ -import type { SupabaseApiError } from "@supabase/api/effect"; -import { Effect } from "effect"; +import { isSupabaseApiResponseSchemaError, type SupabaseApiError } from "@supabase/api/effect"; +import { Data, Effect } from "effect"; import * as HttpClientError from "effect/unstable/http/HttpClientError"; // HttpClientError reasons that indicate the server returned an actual response (vs a transport @@ -55,6 +55,12 @@ export type StatusErrorFactory = new (args: { readonly message: string; }) => E; +/** A 2xx Management API response that violates the generated response contract. */ +export class LegacyApiResponseSchemaError extends Data.TaggedError("LegacyApiResponseSchemaError")<{ + readonly operationId: string; + readonly message: string; +}> {} + /** * Build an error mapper that classifies a `SupabaseApiError` into either a typed network * error or a typed unexpected-status error. Pulled out of individual command families so @@ -69,9 +75,17 @@ export function mapLegacyHttpError(opts: { readonly statusError: StatusErrorFactory; readonly networkMessage: (cause: string) => string; readonly statusMessage: (status: number, body: string) => string; -}): (cause: SupabaseApiError) => Effect.Effect { +}): (cause: SupabaseApiError) => Effect.Effect { return (cause) => Effect.gen(function* () { + if (isSupabaseApiResponseSchemaError(cause)) { + return yield* Effect.fail( + new LegacyApiResponseSchemaError({ + operationId: cause.operationId, + message: cause.message, + }), + ); + } if (HttpClientError.isHttpClientError(cause)) { if (RESPONSE_ERROR_TAGS.has(cause.reason._tag) && cause.response !== undefined) { const status = cause.response.status; @@ -92,7 +106,7 @@ export function mapLegacyHttpError(opts: { new opts.networkError({ message: opts.networkMessage(description) }), ); } - // SchemaError or HttpBodyError — treat as transport-level network error. + // Input SchemaError or HttpBodyError — retain their historical mapping. return yield* Effect.fail( new opts.networkError({ message: opts.networkMessage(String(cause)) }), ); diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.ts index a6b3bef1e9..c99b8e5919 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.ts @@ -46,6 +46,8 @@ const REINDEX_CONCURRENTLY_PATTERN = /^REINDEX(?:\s|\().*\sCONCURRENTLY(?:\s|$)/ const VACUUM_PATTERN = /^VACUUM(?:\s|\(|$)/u; const ALTER_SYSTEM_PATTERN = /^ALTER\s+SYSTEM(?:\s|$)/u; const CLUSTER_PATTERN = /^CLUSTER(?:\s|$)/u; +const TRANSACTION_CONTROL_PATTERN = + /^(?:BEGIN|START\s+TRANSACTION|COMMIT|END|ROLLBACK|ABORT|PREPARE\s+TRANSACTION)(?:\s|$)/u; /** * Strips a leading BOM, whitespace, and SQL line (`--`) and block comments from the @@ -92,6 +94,10 @@ export const legacyIsPipelineIncompatible = (sql: string): boolean => { ); }; +/** Whether the statement owns a transaction boundary that must not be nested. */ +export const legacyHasTransactionControl = (sql: string): boolean => + TRANSACTION_CONTROL_PATTERN.test(legacyTrimLeadingSqlComments(sql).toUpperCase()); + /** A buffered statement awaiting the next batch flush; `version` is the history insert. */ type LegacyBatchItem = | { readonly kind: "exec"; readonly sql: string } @@ -194,6 +200,31 @@ const execMigrationBatch = ( return new Error(`${errMessage(e)}\n${msg.join("\n")}`); }; + // A file with authored transaction boundaries owns those semantics. Execute + // the statements exactly as written, clean up a failed authored transaction, + // and only send the history insert after every statement has succeeded. + if (statements.some(legacyHasTransactionControl)) { + const authored = Effect.gen(function* () { + for (const [index, statement] of statements.entries()) { + yield* session + .exec(statement) + .pipe(Effect.mapError((cause) => atStatement(cause, index, statement))); + } + if (version.length > 0) { + yield* session + .query(INSERT_MIGRATION_VERSION, [version, name, statements]) + .pipe( + Effect.mapError((cause) => + atStatement(cause, statements.length, INSERT_MIGRATION_VERSION), + ), + ); + } + }); + return yield* authored.pipe( + Effect.tapError(() => session.exec("ROLLBACK").pipe(Effect.ignore)), + ); + } + // `executed` is the global statement index of the next statement to run, so the // error context stays accurate across flushed batches and standalone statements // (Go threads the same counter through `ExecBatch`). diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts index 749fb9b70b..4ca6249394 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts @@ -9,6 +9,7 @@ import { mockOutput } from "../../../tests/helpers/mocks.ts"; import type { LegacyDbSession } from "./legacy-db-connection.service.ts"; import { legacyApplyMigrationFile, + legacyHasTransactionControl, legacyIsPipelineIncompatible, legacyMarkError, legacySeedGlobals, @@ -193,6 +194,61 @@ describe("legacyApplyMigrationFile", () => { ), ); }); + + it.effect("preserves authored transaction boundaries and records history afterwards", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-apply-")); + const file = join(dir, "20240101120000_authored.sql"); + writeFileSync(file, "BEGIN;\nSET LOCAL check_function_bodies = off;\nCOMMIT;"); + const { session, calls } = fakeSession(); + return run(session, file).pipe( + Effect.tap(() => + Effect.sync(() => { + const execs = calls.filter((call) => call.kind === "exec").map((call) => call.sql); + // One BEGIN/COMMIT belongs to history-table setup; the other pair is + // exactly the authored boundary, with no nested migration wrapper. + expect(execs.filter((sql) => sql === "BEGIN")).toHaveLength(2); + expect(execs.filter((sql) => sql === "COMMIT")).toHaveLength(2); + expect(execs).toContain("SET LOCAL check_function_bodies = off"); + const history = calls.filter((call) => call.kind === "query"); + expect(history).toHaveLength(1); + expect(history[0]?.params?.[0]).toBe("20240101120000"); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("does not record history when an authored transaction fails", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-apply-")); + const file = join(dir, "20240101120000_authored.sql"); + writeFileSync(file, "BEGIN;\nCREATE TABLE broken (;\nCOMMIT;"); + const { session, calls } = fakeSession({ failOn: "CREATE TABLE broken" }); + return run(session, file).pipe( + Effect.exit, + Effect.tap((exit) => + Effect.sync(() => { + expect(Exit.isFailure(exit)).toBe(true); + expect(calls.some((call) => call.kind === "query")).toBe(false); + expect(calls.some((call) => call.kind === "exec" && call.sql === "ROLLBACK")).toBe(true); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); +}); + +describe("legacyHasTransactionControl", () => { + it("recognizes authored boundaries after comments without matching routine bodies", () => { + expect(legacyHasTransactionControl("-- authored\nBEGIN")).toBe(true); + expect(legacyHasTransactionControl("START TRANSACTION ISOLATION LEVEL SERIALIZABLE")).toBe( + true, + ); + expect( + legacyHasTransactionControl( + "CREATE FUNCTION f() RETURNS void AS $$ BEGIN END $$ LANGUAGE plpgsql", + ), + ).toBe(false); + }); }); describe("migration failure rendering (Go ExecBatch parity)", () => { diff --git a/apps/cli/src/legacy/shared/legacy-seed-ops.ts b/apps/cli/src/legacy/shared/legacy-seed-ops.ts index 3419a30df4..bbea4d4fcc 100644 --- a/apps/cli/src/legacy/shared/legacy-seed-ops.ts +++ b/apps/cli/src/legacy/shared/legacy-seed-ops.ts @@ -109,9 +109,6 @@ const legacyGlobSeedFiles = Effect.fnUntraced(function* ( } satisfies LegacyGlobResult; }); -/** Shared Go-compatible SQL glob expansion for migration/declarative consumers. */ -export const legacyResolveSqlGlobFiles = legacyGlobSeedFiles; - const toSlash = (p: string): string => p.replaceAll("\\", "/"); /** Splits a forward-slashed path into its directory prefix and final element. */ diff --git a/packages/api/scripts/generate.ts b/packages/api/scripts/generate.ts index f6ff09526b..6f5b277949 100644 --- a/packages/api/scripts/generate.ts +++ b/packages/api/scripts/generate.ts @@ -215,6 +215,14 @@ function identifier(value: string): string { const UUID_PATTERN = "^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$"; +// The Management API's checked-in OpenAPI currently carries a Z-only pattern +// alongside `format: "date-time"`. OpenAPI date-time is RFC3339, whose time +// offset may be either Z or a numeric `+/-HH:MM` offset. Normalize every such +// node so generated response contracts accept the complete wire format while +// retaining strict calendar/time validation. +const RFC3339_DATE_TIME_PATTERN = + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$"; + // Keys that we want to strip from a schema node because they describe // documentation / example values rather than the value's shape. JSON Schema's // `default` is a primitive (or array/object) literal used for documentation — @@ -290,6 +298,10 @@ export function sanitizeOpenApiSchema( sanitized.pattern = UUID_PATTERN; } + if (sanitized.type === "string" && sanitized.format === "date-time") { + sanitized.pattern = RFC3339_DATE_TIME_PATTERN; + } + return sanitized; } diff --git a/packages/api/scripts/generate.unit.test.ts b/packages/api/scripts/generate.unit.test.ts index 5b4dbe720e..4c6038b210 100644 --- a/packages/api/scripts/generate.unit.test.ts +++ b/packages/api/scripts/generate.unit.test.ts @@ -28,11 +28,31 @@ describe("generate", () => { expect(renderOpenApiSchema({ type: "string", format: "email", nullable: true })).toBe( 'Schema.Union([Schema.String.annotate({ "format": "email" }), Schema.Null])', ); - expect(renderOpenApiSchema({ type: "string", format: "date-time", nullable: true })).toBe( - 'Schema.Union([Schema.String.annotate({ "format": "date-time" }), Schema.Null])', + expect(renderOpenApiSchema({ type: "string", format: "date-time", nullable: true })).toEqual( + expect.stringContaining('Schema.String.annotate({ "format": "date-time" }).check'), ); }); + test("normalizes date-time schemas to strict RFC3339 timestamps with numeric offsets", () => { + const sanitized = sanitizeOpenApiSchema({ + type: "string", + format: "date-time", + pattern: "Z-only-pattern-from-upstream", + }); + expect(typeof sanitized.pattern).toBe("string"); + if (typeof sanitized.pattern !== "string") { + throw new Error("Expected sanitized date-time pattern"); + } + const dateTime = new RegExp(sanitized.pattern); + + expect(dateTime.test("2026-08-07T10:11:12Z")).toBe(true); + expect(dateTime.test("2026-08-07T10:11:12+00:00")).toBe(true); + expect(dateTime.test("2026-08-07T12:41:12+02:30")).toBe(true); + expect(dateTime.test("2026-08-07 10:11:12Z")).toBe(false); + expect(dateTime.test("2026-08-07T10:11:12+25:00")).toBe(false); + expect(dateTime.test("not-a-timestamp")).toBe(false); + }); + test("accepts booleans for string-encoded boolean query parameters", () => { expect( normalizeQueryParameterSchema( diff --git a/packages/api/src/effect.ts b/packages/api/src/effect.ts index 0cb0a4d4fe..64f1459292 100644 --- a/packages/api/src/effect.ts +++ b/packages/api/src/effect.ts @@ -12,7 +12,11 @@ import { } from "./generated/effect-client.ts"; export type { SupabaseApiError, SupabaseApiRetryOptions } from "./internal/client.ts"; -export { SupabaseApiConfigError } from "./internal/client.ts"; +export { + isSupabaseApiResponseSchemaError, + SupabaseApiConfigError, + SupabaseApiResponseSchemaError, +} from "./internal/client.ts"; export type { SupabaseApiClientOptions, SupabaseApiConfig } from "./internal/client.ts"; export { apiConfigLayer, DEFAULT_SUPABASE_API_URL } from "./config/api-config.layer.ts"; export { ApiConfig } from "./config/api-config.service.ts"; diff --git a/packages/api/src/generated/contracts.ts b/packages/api/src/generated/contracts.ts index 589f6895ea..7458f460f7 100644 --- a/packages/api/src/generated/contracts.ts +++ b/packages/api/src/generated/contracts.ts @@ -81,11 +81,11 @@ export const ApiKeyResponse = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -96,11 +96,11 @@ export const ApiKeyResponse = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -197,32 +197,32 @@ export const BranchResponse = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), review_requested_at: Schema.optionalKey( Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -232,11 +232,11 @@ export const BranchResponse = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -1112,32 +1112,32 @@ export const V1CreateABranchOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), review_requested_at: Schema.optionalKey( Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -1147,11 +1147,11 @@ export const V1CreateABranchOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -1553,21 +1553,21 @@ export const V1CreateLegacySigningKeyOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }); @@ -1664,11 +1664,11 @@ export const V1CreateProjectApiKeyOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -1679,11 +1679,11 @@ export const V1CreateProjectApiKeyOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -1880,21 +1880,21 @@ export const V1CreateProjectSigningKeyOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }); @@ -1957,11 +1957,11 @@ export const V1CreateRestorePointOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -2258,11 +2258,11 @@ export const V1DeleteProjectApiKeyOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -2273,11 +2273,11 @@ export const V1DeleteProjectApiKeyOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -2613,32 +2613,32 @@ export const V1GetABranchOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), review_requested_at: Schema.optionalKey( Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -2648,11 +2648,11 @@ export const V1GetABranchOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -3680,11 +3680,11 @@ export const V1GetAuthServiceConfigOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -3881,11 +3881,11 @@ export const V1GetBackupScheduleOutput = Schema.Struct({ }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }); @@ -4183,21 +4183,21 @@ export const V1GetLegacySigningKeyOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }); @@ -4235,11 +4235,11 @@ export const V1GetNetworkRestrictionsOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -4247,11 +4247,11 @@ export const V1GetNetworkRestrictionsOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -5020,11 +5020,11 @@ export const V1GetProjectApiKeyOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -5035,11 +5035,11 @@ export const V1GetProjectApiKeyOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -5193,11 +5193,11 @@ export const V1GetProjectLogsInput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -5205,11 +5205,11 @@ export const V1GetProjectLogsInput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -5251,11 +5251,11 @@ export const V1GetProjectLogsAllInput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -5263,11 +5263,11 @@ export const V1GetProjectLogsAllInput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -5426,21 +5426,21 @@ export const V1GetProjectSigningKeyOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }); @@ -5474,21 +5474,21 @@ export const V1GetProjectSigningKeysOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }), @@ -5560,11 +5560,11 @@ export const V1GetProjectUsageApiCountOutput = Schema.Struct({ timestamp: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), total_auth_requests: Schema.Number.check( @@ -5829,11 +5829,11 @@ export const V1GetRestorePointOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -7057,11 +7057,11 @@ export const V1PatchNetworkRestrictionsOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -7069,11 +7069,11 @@ export const V1PatchNetworkRestrictionsOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -7218,21 +7218,21 @@ export const V1RemoveProjectSigningKeyOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }); @@ -7564,32 +7564,32 @@ export const V1UpdateABranchConfigOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), review_requested_at: Schema.optionalKey( Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -7599,11 +7599,11 @@ export const V1UpdateABranchConfigOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -8365,11 +8365,11 @@ export const V1UpdateAuthServiceConfigInput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -9118,11 +9118,11 @@ export const V1UpdateAuthServiceConfigOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -9228,11 +9228,11 @@ export const V1UpdateBackupScheduleOutput = Schema.Struct({ }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }); @@ -9497,11 +9497,11 @@ export const V1UpdateNetworkRestrictionsOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -9509,11 +9509,11 @@ export const V1UpdateNetworkRestrictionsOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), ), @@ -10149,11 +10149,11 @@ export const V1UpdateProjectApiKeyOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -10164,11 +10164,11 @@ export const V1UpdateProjectApiKeyOutput = Schema.Struct({ Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), Schema.Null, @@ -10227,21 +10227,21 @@ export const V1UpdateProjectSigningKeyOutput = Schema.Struct({ created_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), updated_at: Schema.String.annotate({ format: "date-time" }).check( Schema.isPattern( new RegExp( - "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", ), ).annotate({ expected: - "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$", + "a string matching the RegExp ^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|[+-](?:[01]\\d|2[0-3]):[0-5]\\d)$", }), ), }); diff --git a/packages/api/src/internal/client.ts b/packages/api/src/internal/client.ts index 24dc02ed8b..74b71e83e2 100644 --- a/packages/api/src/internal/client.ts +++ b/packages/api/src/internal/client.ts @@ -47,7 +47,27 @@ export interface SupabaseApiClientOptions { export type SupabaseApiError = | HttpBody.HttpBodyError | HttpClientError.HttpClientError - | SchemaError; + | SchemaError + | SupabaseApiResponseSchemaError; + +/** A successful HTTP response whose JSON body violates the generated output schema. */ +export class SupabaseApiResponseSchemaError extends Error { + readonly _tag = "SupabaseApiResponseSchemaError"; + + constructor( + readonly operationId: OperationId, + override readonly cause: unknown, + ) { + super(`Response schema validation failed for ${operationId}: ${String(cause)}`); + this.name = "SupabaseApiResponseSchemaError"; + } +} + +export function isSupabaseApiResponseSchemaError( + cause: unknown, +): cause is SupabaseApiResponseSchemaError { + return cause instanceof SupabaseApiResponseSchemaError; +} export interface SupabaseApiClientShape { readonly execute: ( @@ -479,7 +499,13 @@ function decodeJsonResponse( definition: OperationDefinition, response: HttpClientResponse.HttpClientResponse, ): Effect.Effect, SupabaseApiError> { - return HttpClientResponse.schemaBodyJson(definition.outputSchema)(response); + return HttpClientResponse.schemaBodyJson(definition.outputSchema)(response).pipe( + Effect.mapError((cause) => + Schema.isSchemaError(cause) + ? new SupabaseApiResponseSchemaError(definition.id, cause) + : cause, + ), + ); } function decodeTextResponse( diff --git a/packages/api/src/internal/client.unit.test.ts b/packages/api/src/internal/client.unit.test.ts index 76515ed47b..952086ef62 100644 --- a/packages/api/src/internal/client.unit.test.ts +++ b/packages/api/src/internal/client.unit.test.ts @@ -1,5 +1,5 @@ import { describe, expect, test } from "vitest"; -import { Effect, Exit, Layer, Option, Redacted } from "effect"; +import { Cause, Effect, Exit, Layer, Option, Redacted } from "effect"; import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientError from "effect/unstable/http/HttpClientError"; import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; @@ -8,7 +8,7 @@ import * as UrlParams from "effect/unstable/http/UrlParams"; import * as Schema from "effect/Schema"; import { operationDefinitions } from "../generated/contracts.ts"; -import { makeSupabaseApiClient } from "./client.ts"; +import { makeSupabaseApiClient, SupabaseApiResponseSchemaError } from "./client.ts"; const textDecoder = new TextDecoder(); @@ -155,6 +155,78 @@ const config = { } as const; describe("makeSupabaseApiClient", () => { + test.each(["2026-08-07T10:11:12Z", "2026-08-07T10:11:12+00:00", "2026-08-07T12:41:12+02:30"])( + "accepts RFC3339 response timestamp %s", + async (createdAt) => { + const result = await Effect.runPromise( + makeSupabaseApiClient(config).pipe( + Effect.flatMap((client) => + client.execute<"v1CreateAProject">(operationDefinitions.v1CreateAProject, { + db_pass: "hunter2", + name: "project-name", + organization_slug: "my-org", + }), + ), + Effect.provide( + httpClientLayer((request) => + Effect.succeed( + jsonResponse(request, 200, { + id: "project-id", + ref: "abcdefghijklmnopqrst", + organization_id: "org-id", + organization_slug: "my-org", + name: "project-name", + region: "us-east-1", + created_at: createdAt, + status: "ACTIVE_HEALTHY", + }), + ), + ), + ), + ), + ); + + expect(result.created_at).toBe(createdAt); + }, + ); + + test("wraps output schema failures separately from input and transport errors", async () => { + const exit = await Effect.runPromise( + makeSupabaseApiClient(config).pipe( + Effect.flatMap((client) => + client.execute<"v1CreateAProject">(operationDefinitions.v1CreateAProject, { + db_pass: "hunter2", + name: "project-name", + organization_slug: "my-org", + }), + ), + Effect.provide( + httpClientLayer((request) => + Effect.succeed( + jsonResponse(request, 200, { + id: "project-id", + ref: "abcdefghijklmnopqrst", + created_at: "malformed", + }), + ), + ), + ), + Effect.exit, + ), + ); + + expect(Exit.isFailure(exit)).toBe(true); + if (Exit.isFailure(exit)) { + const failure = exit.cause.reasons[0]; + expect(failure !== undefined && Cause.isFailReason(failure)).toBe(true); + if (failure !== undefined && Cause.isFailReason(failure)) { + expect(failure.error).toBeInstanceOf(SupabaseApiResponseSchemaError); + if (failure.error instanceof SupabaseApiResponseSchemaError) { + expect(failure.error.operationId).toBe("v1CreateAProject"); + } + } + } + }); test("retries transport errors for POST requests", async () => { let attempts = 0; From 958d36ceb2e3ffde77851dc8f745492c6685c6ac Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 7 Aug 2026 20:33:26 +0200 Subject: [PATCH 06/82] test(cli): update transactional Go mocks --- apps/cli-go/internal/db/push/push_test.go | 18 +++++-- apps/cli-go/internal/db/reset/reset_test.go | 54 +++++++++++++++---- apps/cli-go/internal/db/start/start_test.go | 4 +- .../internal/migration/apply/apply_test.go | 12 ++++- .../internal/migration/down/down_test.go | 36 ++++++++++--- .../internal/migration/squash/squash_test.go | 16 +++++- .../legacy/branch/switch_/switch__test.go | 26 +++++++-- 7 files changed, 133 insertions(+), 33 deletions(-) diff --git a/apps/cli-go/internal/db/push/push_test.go b/apps/cli-go/internal/db/push/push_test.go index b5d0d2c7f3..0357a1875d 100644 --- a/apps/cli-go/internal/db/push/push_test.go +++ b/apps/cli-go/internal/db/push/push_test.go @@ -101,8 +101,12 @@ func TestMigrationPush(t *testing.T) { helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(migration.INSERT_MIGRATION_VERSION, "0", "test", nil). - ReplyError(pgerrcode.NotNullViolation, `null value in column "version" of relation "schema_migrations"`) + ReplyError(pgerrcode.NotNullViolation, `null value in column "version" of relation "schema_migrations"`). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := Run(context.Background(), false, false, false, false, dbConfig, fsys, conn.Intercept) // Check error @@ -125,8 +129,12 @@ func TestPushAll(t *testing.T) { helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(migration.INSERT_MIGRATION_VERSION, "0", "test", nil). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := Run(context.Background(), false, false, true, true, dbConfig, fsys, conn.Intercept) // Check error @@ -185,8 +193,12 @@ func TestPushAll(t *testing.T) { helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(migration.INSERT_MIGRATION_VERSION, "0", "test", nil). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") helper.MockSeedHistory(conn). Query(migration.UPSERT_SEED_FILE, seedPath, digest). ReplyError(pgerrcode.NotNullViolation, `null value in column "hash" of relation "seed_files"`) diff --git a/apps/cli-go/internal/db/reset/reset_test.go b/apps/cli-go/internal/db/reset/reset_test.go index 5d672ec15f..a95377246d 100644 --- a/apps/cli-go/internal/db/reset/reset_test.go +++ b/apps/cli-go/internal/db/reset/reset_test.go @@ -173,8 +173,12 @@ func TestInitDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(utils.InitialSchemaPg14Sql). - Reply("CREATE SCHEMA") + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.InitialSchemaPg14Sql). + Reply("CREATE SCHEMA"). + Query("COMMIT"). + Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn) // Run test assert.NoError(t, initDatabase(context.Background(), conn.Intercept)) @@ -194,8 +198,12 @@ func TestInitDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(utils.InitialSchemaPg14Sql). - ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`) + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(utils.InitialSchemaPg14Sql). + ReplyError(pgerrcode.DuplicateSchema, `schema "public" already exists`). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := initDatabase(context.Background(), conn.Intercept) // Check error @@ -209,14 +217,20 @@ func TestRecreateDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query("ALTER DATABASE _supabase ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query(TERMINATE_BACKENDS). Reply("SELECT 1"). + Query("COMMIT"). + Reply("COMMIT"). Query(COUNT_REPLICATION_SLOTS). Reply("SELECT 1", []any{0}). + Query("BEGIN"). + Reply("BEGIN"). Query("DROP DATABASE IF EXISTS postgres WITH (FORCE)"). Reply("DROP DATABASE"). Query("CREATE DATABASE postgres WITH OWNER postgres"). @@ -224,7 +238,9 @@ func TestRecreateDatabase(t *testing.T) { Query("DROP DATABASE IF EXISTS _supabase WITH (FORCE)"). Reply("DROP DATABASE"). Query("CREATE DATABASE _supabase WITH OWNER postgres"). - Reply("CREATE DATABASE") + Reply("CREATE DATABASE"). + Query("COMMIT"). + Reply("COMMIT") // Run test assert.NoError(t, recreateDatabase(context.Background(), conn.Intercept)) }) @@ -239,11 +255,15 @@ func TestRecreateDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query("ALTER DATABASE _supabase ALLOW_CONNECTIONS false"). ReplyError(pgerrcode.InvalidCatalogName, `database "_supabase" does not exist`). Query(TERMINATE_BACKENDS). + Query("ROLLBACK"). + Reply("ROLLBACK"). Query(COUNT_REPLICATION_SLOTS). ReplyError(pgerrcode.UndefinedTable, `relation "pg_replication_slots" does not exist`) // Run test @@ -257,10 +277,14 @@ func TestRecreateDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). ReplyError(pgerrcode.InvalidParameterValue, `cannot disallow connections for current database`). Query("ALTER DATABASE _supabase ALLOW_CONNECTIONS false"). - Query(TERMINATE_BACKENDS) + Query(TERMINATE_BACKENDS). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := recreateDatabase(context.Background(), conn.Intercept) // Check error @@ -272,21 +296,29 @@ func TestRecreateDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query("ALTER DATABASE _supabase ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query(TERMINATE_BACKENDS). Reply("SELECT 1"). + Query("COMMIT"). + Reply("COMMIT"). Query(COUNT_REPLICATION_SLOTS). Reply("SELECT 1", []any{0}). + Query("BEGIN"). + Reply("BEGIN"). Query("DROP DATABASE IF EXISTS postgres WITH (FORCE)"). ReplyError(pgerrcode.ObjectInUse, `database "postgres" is used by an active logical replication slot`). Query("CREATE DATABASE postgres WITH OWNER postgres"). Query("DROP DATABASE IF EXISTS _supabase WITH (FORCE)"). Reply("DROP DATABASE"). Query("CREATE DATABASE _supabase WITH OWNER postgres"). - Reply("CREATE DATABASE") + Reply("CREATE DATABASE"). + Query("ROLLBACK"). + Reply("ROLLBACK") err := recreateDatabase(context.Background(), conn.Intercept) // Check error assert.ErrorContains(t, err, `ERROR: database "postgres" is used by an active logical replication slot (SQLSTATE 55006)`) diff --git a/apps/cli-go/internal/db/start/start_test.go b/apps/cli-go/internal/db/start/start_test.go index f65d4bea45..04de8b93f6 100644 --- a/apps/cli-go/internal/db/start/start_test.go +++ b/apps/cli-go/internal/db/start/start_test.go @@ -25,8 +25,8 @@ import ( "github.com/supabase/cli/pkg/pgtest" ) -func mockTransactionalStatement(conn *pgtest.MockConn, statement, reply string) *pgtest.MockConn { - return conn.Query("BEGIN"). +func mockTransactionalStatement(conn *pgtest.MockConn, statement, reply string) { + conn.Query("BEGIN"). Reply("BEGIN"). Query(statement). Reply(reply). diff --git a/apps/cli-go/internal/migration/apply/apply_test.go b/apps/cli-go/internal/migration/apply/apply_test.go index f45891cfb4..0f931a7cb0 100644 --- a/apps/cli-go/internal/migration/apply/apply_test.go +++ b/apps/cli-go/internal/migration/apply/apply_test.go @@ -29,10 +29,14 @@ func TestMigrateDatabase(t *testing.T) { helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(sql). Reply("CREATE SCHEMA"). Query(migration.INSERT_MIGRATION_VERSION, "0", "test", []string{sql}). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := MigrateAndSeed(context.Background(), "", conn.MockClient(t), fsys) // Check error @@ -54,10 +58,14 @@ func TestMigrateDatabase(t *testing.T) { helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(sql). Reply("CREATE SCHEMA"). Query(migration.INSERT_MIGRATION_VERSION, "0", "test", []string{sql}). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") utils.Config.Db.Seed.Enabled = false // Run test err := MigrateAndSeed(context.Background(), "", conn.MockClient(t), fsys) diff --git a/apps/cli-go/internal/migration/down/down_test.go b/apps/cli-go/internal/migration/down/down_test.go index 89d9ff66d4..f55d83ee73 100644 --- a/apps/cli-go/internal/migration/down/down_test.go +++ b/apps/cli-go/internal/migration/down/down_test.go @@ -85,13 +85,21 @@ func TestResetRemote(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.DropObjects). - Reply("INSERT 0") + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.DropObjects). + Reply("INSERT 0"). + Query("COMMIT"). + Reply("COMMIT") helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(migration.INSERT_MIGRATION_VERSION, "0", "schema", nil). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := ResetAll(context.Background(), "", conn.MockClient(t), fsys) // Check error @@ -109,13 +117,21 @@ func TestResetRemote(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.DropObjects). - Reply("INSERT 0") + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.DropObjects). + Reply("INSERT 0"). + Query("COMMIT"). + Reply("COMMIT") helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(migration.INSERT_MIGRATION_VERSION, "0", "schema", nil). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") utils.Config.Db.Seed.Enabled = false // Run test err := ResetAll(context.Background(), "", conn.MockClient(t), fsys) @@ -129,8 +145,12 @@ func TestResetRemote(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query(migration.DropObjects). - ReplyError(pgerrcode.InsufficientPrivilege, "permission denied for relation supabase_migrations") + conn.Query("BEGIN"). + Reply("BEGIN"). + Query(migration.DropObjects). + ReplyError(pgerrcode.InsufficientPrivilege, "permission denied for relation supabase_migrations"). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := ResetAll(context.Background(), "", conn.MockClient(t), fsys) // Check error diff --git a/apps/cli-go/internal/migration/squash/squash_test.go b/apps/cli-go/internal/migration/squash/squash_test.go index 91d13066a5..bc44bb9648 100644 --- a/apps/cli-go/internal/migration/squash/squash_test.go +++ b/apps/cli-go/internal/migration/squash/squash_test.go @@ -88,14 +88,22 @@ func TestSquashCommand(t *testing.T) { helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(sql). Reply("CREATE SCHEMA"). Query(migration.INSERT_MIGRATION_VERSION, "0", "init", []string{sql}). Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT"). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(migration.INSERT_MIGRATION_VERSION, "1", "target", nil). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := Run(context.Background(), "", pgconn.Config{ Host: "127.0.0.1", @@ -317,10 +325,14 @@ func TestSquashMigrations(t *testing.T) { helper.MockMigrationHistory(conn). Query("RESET ALL"). Reply("RESET"). + Query("BEGIN"). + Reply("BEGIN"). Query(sql). Reply("CREATE SCHEMA"). Query(migration.INSERT_MIGRATION_VERSION, "0", "init", []string{sql}). - Reply("INSERT 0 1") + Reply("INSERT 0 1"). + Query("COMMIT"). + Reply("COMMIT") // Run test err := squashMigrations(context.Background(), []string{path}, afero.NewReadOnlyFs(fsys), conn.Intercept) // Check error diff --git a/apps/cli-go/legacy/branch/switch_/switch__test.go b/apps/cli-go/legacy/branch/switch_/switch__test.go index f0e51d521e..4ecdc9b5cc 100644 --- a/apps/cli-go/legacy/branch/switch_/switch__test.go +++ b/apps/cli-go/legacy/branch/switch_/switch__test.go @@ -41,12 +41,16 @@ func TestSwitchCommand(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query("ALTER DATABASE _supabase ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query(reset.TERMINATE_BACKENDS). Reply("SELECT 1"). + Query("COMMIT"). + Reply("COMMIT"). Query(reset.COUNT_REPLICATION_SLOTS). Reply("SELECT 1", []any{0}). Query("ALTER DATABASE postgres RENAME TO main;"). @@ -212,10 +216,14 @@ func TestSwitchDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). ReplyError(pgerrcode.InvalidParameterValue, `cannot disallow connections for current database`). Query("ALTER DATABASE _supabase ALLOW_CONNECTIONS false"). - Query(reset.TERMINATE_BACKENDS) + Query(reset.TERMINATE_BACKENDS). + Query("ROLLBACK"). + Reply("ROLLBACK") // Run test err := switchDatabase(context.Background(), "main", "target", conn.Intercept) // Check error @@ -230,12 +238,16 @@ func TestSwitchDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query("ALTER DATABASE _supabase ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query(reset.TERMINATE_BACKENDS). Reply("SELECT 1"). + Query("COMMIT"). + Reply("COMMIT"). Query(reset.COUNT_REPLICATION_SLOTS). Reply("SELECT 1", []any{0}). Query("ALTER DATABASE postgres RENAME TO main;"). @@ -260,12 +272,16 @@ func TestSwitchDatabase(t *testing.T) { // Setup mock postgres conn := pgtest.NewConn() defer conn.Close(t) - conn.Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("ALTER DATABASE postgres ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query("ALTER DATABASE _supabase ALLOW_CONNECTIONS false"). Reply("ALTER DATABASE"). Query(reset.TERMINATE_BACKENDS). Reply("SELECT 1"). + Query("COMMIT"). + Reply("COMMIT"). Query(reset.COUNT_REPLICATION_SLOTS). Reply("SELECT 1", []any{0}). Query("ALTER DATABASE postgres RENAME TO main;"). From d4861957ee293c0a032283a30bd6e3d97112a01e Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 7 Aug 2026 20:40:10 +0200 Subject: [PATCH 07/82] chore(cli): remove unused diff helper --- apps/cli-go/internal/db/diff/diff.go | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/apps/cli-go/internal/db/diff/diff.go b/apps/cli-go/internal/db/diff/diff.go index 1906f76c41..f43581b092 100644 --- a/apps/cli-go/internal/db/diff/diff.go +++ b/apps/cli-go/internal/db/diff/diff.go @@ -227,18 +227,3 @@ func DiffDatabase(ctx context.Context, schema []string, config pgconn.Config, w } return DatabaseDiff{SQL: output}, nil } - -func migrateBaseDatabase(ctx context.Context, config pgconn.Config, migrations []string, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error { - fmt.Fprintln(os.Stderr, "Creating local database from declarative schemas:") - msg := make([]string, len(migrations)) - for i, m := range migrations { - msg[i] = fmt.Sprintf(" • %s", utils.Bold(m)) - } - fmt.Fprintln(os.Stderr, strings.Join(msg, "\n")) - conn, err := utils.ConnectLocalPostgres(ctx, config, options...) - if err != nil { - return err - } - defer conn.Close(context.Background()) - return migration.SeedGlobals(ctx, migrations, conn, afero.NewIOFS(fsys)) -} From 394ecf861d2748c5531c061c3184be2db8f69097 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 12:16:28 +0200 Subject: [PATCH 08/82] fix(cli): isolate pg-delta-next baseline behavior --- .../internal/db/declarative/declarative.go | 7 +- .../db/declarative/declarative_test.go | 12 ++-- apps/cli-go/internal/db/diff/diff.go | 22 +++++-- apps/cli-go/internal/db/diff/diff_test.go | 66 ++++++++++++++++++- apps/cli-go/internal/db/diff/shadow.go | 2 +- apps/cli-go/internal/db/start/start.go | 20 +++++- .../internal/migration/squash/squash.go | 2 +- .../internal/migration/squash/squash_test.go | 12 ++++ ...eclarative.orchestrate.integration.test.ts | 1 - .../shared/legacy-db-config.toml-read.ts | 3 - .../src/legacy/shared/legacy-pgdelta.cache.ts | 6 +- .../shared/legacy-pgdelta.cache.unit.test.ts | 9 +-- 12 files changed, 124 insertions(+), 38 deletions(-) diff --git a/apps/cli-go/internal/db/declarative/declarative.go b/apps/cli-go/internal/db/declarative/declarative.go index 881db395a5..9eecfb6f17 100644 --- a/apps/cli-go/internal/db/declarative/declarative.go +++ b/apps/cli-go/internal/db/declarative/declarative.go @@ -673,12 +673,11 @@ func baselineVersionToken() string { return catalogPrefixRegexp.ReplaceAllString(image, "-") } -// setupInputsToken hashes every project input that start.SetupDatabase consumes -// and that therefore shapes the platform baseline: +// setupInputsToken hashes every project input that shapes the legacy shadow +// baseline produced by start.SetupDatabase with WithLegacyPgNetBaseline: // // - the Postgres image (initSchema content); // - the service toggles that gate initSchema — auth/storage/realtime; -// - experimental.webhooks.enabled (conditional pg_net installation); // - api.auto_expose_new_tables (ApplyApiPrivileges default ACLs); // - vault secret names (UpsertVaultSecrets); // - supabase/roles.sql (SeedGlobals). @@ -692,8 +691,6 @@ func setupInputsToken(fsys afero.Fs) (string, error) { // initSchema conditionally provisions these service schemas. fmt.Fprintf(h, "auth=%t storage=%t realtime=%t\n", utils.Config.Auth.Enabled, utils.Config.Storage.Enabled, utils.Config.Realtime.Enabled) - webhooksEnabled := utils.Config.Experimental.Webhooks != nil && utils.Config.Experimental.Webhooks.Enabled - fmt.Fprintf(h, "database_webhooks=%t\n", webhooksEnabled) // api.auto_expose_new_tables drives ApplyApiPrivileges (default ACLs). Key on the // effective value, not the raw tri-state: as of the 2026-05-30 flip an unset flag // resolves to the same revoke-by-default baseline as explicit false (see diff --git a/apps/cli-go/internal/db/declarative/declarative_test.go b/apps/cli-go/internal/db/declarative/declarative_test.go index 50a33bff02..35a3bd92df 100644 --- a/apps/cli-go/internal/db/declarative/declarative_test.go +++ b/apps/cli-go/internal/db/declarative/declarative_test.go @@ -469,16 +469,11 @@ func TestBaselineCatalogKeyVariesWithServiceToggles(t *testing.T) { assert.NotEqual(t, on, off, "toggling a service must change the baseline cache key") } -func TestBaselineCatalogKeyVariesWithDatabaseWebhooks(t *testing.T) { +func TestBaselineCatalogKeyIgnoresDatabaseWebhooks(t *testing.T) { originalConfig := utils.Config t.Cleanup(func() { utils.Config = originalConfig }) fSys := afero.NewMemMapFs() - disabled := config.NewConfig() - utils.Config = disabled - disabledKey, err := baselineCatalogKey(fSys) - require.NoError(t, err) - enabled := config.NewConfig() require.NoError(t, enabled.Load("config.toml", fstest.MapFS{ "config.toml": &fstest.MapFile{Data: []byte("[experimental.webhooks]\nenabled = true\n")}, @@ -486,8 +481,11 @@ func TestBaselineCatalogKeyVariesWithDatabaseWebhooks(t *testing.T) { utils.Config = enabled enabledKey, err := baselineCatalogKey(fSys) require.NoError(t, err) + utils.Config.Experimental.Webhooks = nil + disabledKey, err := baselineCatalogKey(fSys) + require.NoError(t, err) - assert.NotEqual(t, disabledKey, enabledKey, "Database Webhooks must change the baseline cache key") + assert.Equal(t, disabledKey, enabledKey, "Database Webhooks no longer changes the legacy baseline") } func TestDeclarativeCatalogCacheKeyVariesWithSetupInputs(t *testing.T) { diff --git a/apps/cli-go/internal/db/diff/diff.go b/apps/cli-go/internal/db/diff/diff.go index f43581b092..819929b35c 100644 --- a/apps/cli-go/internal/db/diff/diff.go +++ b/apps/cli-go/internal/db/diff/diff.go @@ -97,8 +97,8 @@ const CREATE_TEMPLATE = "CREATE DATABASE contrib_regression TEMPLATE postgres" // database. It deliberately stops short of applying user migrations so that // callers which only need the platform baseline (declarative apply) share the // exact same starting point as callers that also replay migrations. -func setupShadowConn(ctx context.Context, conn *pgx.Conn, container string, fsys afero.Fs) error { - if err := start.SetupDatabase(ctx, conn, container[:12], os.Stderr, fsys); err != nil { +func setupShadowConn(ctx context.Context, conn *pgx.Conn, container string, fsys afero.Fs, options ...start.SetupDatabaseOption) error { + if err := start.SetupDatabase(ctx, conn, container[:12], os.Stderr, fsys, options...); err != nil { return err } if _, err := conn.Exec(ctx, CREATE_TEMPLATE); err != nil { @@ -118,7 +118,7 @@ func SetupShadowDatabase(ctx context.Context, container string, fsys afero.Fs, o return err } defer conn.Close(context.Background()) - return setupShadowConn(ctx, conn, container, fsys) + return setupShadowConn(ctx, conn, container, fsys, start.WithLegacyPgNetBaseline()) } var pgDeltaNextDeclarativeExtensionDrops = []struct { @@ -162,7 +162,7 @@ func SetupPgDeltaNextDeclarativeShadowDatabase(ctx context.Context, container st return nil } -func MigrateShadowDatabase(ctx context.Context, container string, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error { +func migrateShadowDatabase(ctx context.Context, container string, fsys afero.Fs, setupOptions []start.SetupDatabaseOption, options ...func(*pgx.ConnConfig)) error { migrations, err := migration.ListLocalMigrations(utils.MigrationsDir, afero.NewIOFS(fsys)) if err != nil { return err @@ -172,12 +172,24 @@ func MigrateShadowDatabase(ctx context.Context, container string, fsys afero.Fs, return err } defer conn.Close(context.Background()) - if err := setupShadowConn(ctx, conn, container, fsys); err != nil { + if err := setupShadowConn(ctx, conn, container, fsys, setupOptions...); err != nil { return err } return migration.ApplyMigrations(ctx, migrations, conn, afero.NewIOFS(fsys)) } +// MigrateShadowDatabase preserves the historical platform baseline used by the +// legacy diff engines, including pg_net even when Database Webhooks is disabled. +func MigrateShadowDatabase(ctx context.Context, container string, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error { + return migrateShadowDatabase(ctx, container, fsys, []start.SetupDatabaseOption{start.WithLegacyPgNetBaseline()}, options...) +} + +// MigratePgDeltaNextShadowDatabase provisions the migrations side of the +// isolated pg-delta-next comparison without inheriting legacy baseline behavior. +func MigratePgDeltaNextShadowDatabase(ctx context.Context, container string, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error { + return migrateShadowDatabase(ctx, container, fsys, nil, options...) +} + func DiffDatabase(ctx context.Context, schema []string, config pgconn.Config, w io.Writer, fsys afero.Fs, differ DiffFunc, usePgDelta bool, options ...func(*pgx.ConnConfig)) (DatabaseDiff, error) { if len(utils.Config.Db.Migrations.SchemaPaths) > 0 { fmt.Fprintln(w, schemaPathsTransitionWarning) diff --git a/apps/cli-go/internal/db/diff/diff_test.go b/apps/cli-go/internal/db/diff/diff_test.go index 53ee1a9a53..643cc8181e 100644 --- a/apps/cli-go/internal/db/diff/diff_test.go +++ b/apps/cli-go/internal/db/diff/diff_test.go @@ -72,6 +72,12 @@ func TestRun(t *testing.T) { // Setup mock postgres: with auto_expose_new_tables unset, the shadow database setup // revokes the default Data API GRANTs before creating the regression template. conn := pgtest.NewConn() + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("create extension if not exists pg_net schema extensions"). + Reply("CREATE EXTENSION"). + Query("COMMIT"). + Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn). Query(CREATE_TEMPLATE). Reply("CREATE DATABASE") @@ -150,6 +156,12 @@ func TestRun(t *testing.T) { Query(utils.InitialSchemaPg14Sql). Reply("CREATE SCHEMA"). Query("COMMIT"). + Reply("COMMIT"). + Query("BEGIN"). + Reply("BEGIN"). + Query("create extension if not exists pg_net schema extensions"). + Reply("CREATE EXTENSION"). + Query("COMMIT"). Reply("COMMIT") helper.MockApiPrivilegesRevoke(shadowConn). Query(CREATE_TEMPLATE). @@ -238,6 +250,12 @@ func TestMigrateShadow(t *testing.T) { Query(utils.InitialSchemaPg14Sql). Reply("CREATE SCHEMA"). Query("COMMIT"). + Reply("COMMIT"). + Query("BEGIN"). + Reply("BEGIN"). + Query("create extension if not exists pg_net schema extensions"). + Reply("CREATE EXTENSION"). + Query("COMMIT"). Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn). Query(CREATE_TEMPLATE). @@ -302,7 +320,38 @@ func TestMigrateShadow(t *testing.T) { }) } +func TestMigratePgDeltaNextShadowDatabase(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + utils.Config = pkgconfig.NewConfig() + utils.Config.Db.MajorVersion = 17 + utils.Config.Db.ShadowPort = 54320 + utils.Config.Auth.Enabled = false + utils.Config.Storage.Enabled = false + utils.Config.Realtime.Enabled = false + + conn := pgtest.NewConn() + defer conn.Close(t) + // The config has no Database Webhooks section. Expecting API privileges first + // makes any leaked legacy pg_net activation fail as an unmatched query. + helper.MockApiPrivilegesRevoke(conn). + Query(CREATE_TEMPLATE). + Reply("CREATE DATABASE") + + err := MigratePgDeltaNextShadowDatabase( + context.Background(), + "pg-delta-next-migrations", + afero.NewMemMapFs(), + conn.Intercept, + ) + + require.NoError(t, err) +} + func TestSetupShadowDatabase(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + utils.Config = pkgconfig.NewConfig() utils.Config.Db.MajorVersion = 14 t.Run("sets up platform baseline without applying migrations", func(t *testing.T) { @@ -329,11 +378,18 @@ func TestSetupShadowDatabase(t *testing.T) { Query(utils.InitialSchemaPg14Sql). Reply("CREATE SCHEMA"). Query("COMMIT"). + Reply("COMMIT"). + Query("BEGIN"). + Reply("BEGIN"). + Query("create extension if not exists pg_net schema extensions"). + Reply("CREATE EXTENSION"). + Query("COMMIT"). Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn). Query(CREATE_TEMPLATE). Reply("CREATE DATABASE") - // Run test + // Run with Database Webhooks disabled. Legacy shadows still include pg_net + // because declarative directories historically diffed against that baseline. err := SetupShadowDatabase(context.Background(), "test-shadow-db", fsys, conn.Intercept) // Check error assert.NoError(t, err) @@ -377,7 +433,7 @@ func TestSetupPgDeltaNextDeclarativeShadowDatabase(t *testing.T) { utils.Config = cfg } - t.Run("provisions PG17 without activating user-managed extensions", func(t *testing.T) { + t.Run("keeps pg-delta-next declarative baseline free of pg_net", func(t *testing.T) { newPg17Config(t) conn := pgtest.NewConn() defer conn.Close(t) @@ -587,6 +643,12 @@ create schema public`) Query(utils.InitialSchemaPg14Sql). Reply("CREATE SCHEMA"). Query("COMMIT"). + Reply("COMMIT"). + Query("BEGIN"). + Reply("BEGIN"). + Query("create extension if not exists pg_net schema extensions"). + Reply("CREATE EXTENSION"). + Query("COMMIT"). Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn). Query(CREATE_TEMPLATE). diff --git a/apps/cli-go/internal/db/diff/shadow.go b/apps/cli-go/internal/db/diff/shadow.go index a71a4b2c9e..1092a68175 100644 --- a/apps/cli-go/internal/db/diff/shadow.go +++ b/apps/cli-go/internal/db/diff/shadow.go @@ -58,7 +58,7 @@ func PreparePgDeltaNextShadow(ctx context.Context, fsys afero.Fs, options ...fun freePort: utils.GetFreeHostPort, create: CreateShadowDatabase, wait: start.WaitForHealthyService, - migrate: MigrateShadowDatabase, + migrate: MigratePgDeltaNextShadowDatabase, setup: SetupPgDeltaNextDeclarativeShadowDatabase, remove: utils.DockerRemove, }, options...) diff --git a/apps/cli-go/internal/db/start/start.go b/apps/cli-go/internal/db/start/start.go index 13845b1b90..ed88f03b81 100644 --- a/apps/cli-go/internal/db/start/start.go +++ b/apps/cli-go/internal/db/start/start.go @@ -382,6 +382,7 @@ func SetupLocalDatabase(ctx context.Context, version string, fsys afero.Fs, w io type setupDatabaseOptions struct { activateUserExtensions bool + legacyPgNetBaseline bool } // SetupDatabaseOption customises platform setup for specialised database @@ -397,6 +398,15 @@ func WithoutUserExtensionActivation() SetupDatabaseOption { } } +// WithLegacyPgNetBaseline preserves the historical shadow-database baseline, +// where pg_net was installed independently of the Database Webhooks setting. +// New provisioning paths should use the config-gated default instead. +func WithLegacyPgNetBaseline() SetupDatabaseOption { + return func(options *setupDatabaseOptions) { + options.legacyPgNetBaseline = true + } +} + func SetupDatabase(ctx context.Context, conn *pgx.Conn, host string, w io.Writer, fsys afero.Fs, opts ...SetupDatabaseOption) error { options := setupDatabaseOptions{activateUserExtensions: true} for _, option := range opts { @@ -405,7 +415,11 @@ func SetupDatabase(ctx context.Context, conn *pgx.Conn, host string, w io.Writer if err := initSchema(ctx, conn, host, w); err != nil { return err } - if options.activateUserExtensions { + if options.legacyPgNetBaseline { + if err := enablePgNet(ctx, conn); err != nil { + return err + } + } else if options.activateUserExtensions { if err := ApplyDatabaseWebhooks(ctx, conn); err != nil { return err } @@ -434,6 +448,10 @@ func ApplyDatabaseWebhooks(ctx context.Context, conn *pgx.Conn) error { if webhooks == nil || !webhooks.Enabled { return nil } + return enablePgNet(ctx, conn) +} + +func enablePgNet(ctx context.Context, conn *pgx.Conn) error { file, err := migration.NewMigrationFromReader(strings.NewReader(EnableDatabaseWebhooksSql)) if err != nil { return err diff --git a/apps/cli-go/internal/migration/squash/squash.go b/apps/cli-go/internal/migration/squash/squash.go index afc52c687e..e7d9589642 100644 --- a/apps/cli-go/internal/migration/squash/squash.go +++ b/apps/cli-go/internal/migration/squash/squash.go @@ -93,7 +93,7 @@ func squashMigrations(ctx context.Context, migrations []string, fsys afero.Fs, o return err } defer conn.Close(context.Background()) - if err := start.SetupDatabase(ctx, conn, shadow[:12], os.Stderr, fsys); err != nil { + if err := start.SetupDatabase(ctx, conn, shadow[:12], os.Stderr, fsys, start.WithLegacyPgNetBaseline()); err != nil { return err } // Assuming entities in managed schemas are not altered, we can simply diff the dumps before and after migrations. diff --git a/apps/cli-go/internal/migration/squash/squash_test.go b/apps/cli-go/internal/migration/squash/squash_test.go index bc44bb9648..b694ed9a7d 100644 --- a/apps/cli-go/internal/migration/squash/squash_test.go +++ b/apps/cli-go/internal/migration/squash/squash_test.go @@ -84,6 +84,12 @@ func TestSquashCommand(t *testing.T) { // revokes the default Data API GRANTs before applying migration history. conn := pgtest.NewConn() defer conn.Close(t) + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("create extension if not exists pg_net schema extensions"). + Reply("CREATE EXTENSION"). + Query("COMMIT"). + Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn) helper.MockMigrationHistory(conn). Query("RESET ALL"). @@ -321,6 +327,12 @@ func TestSquashMigrations(t *testing.T) { // revokes the default Data API GRANTs before applying migration history. conn := pgtest.NewConn() defer conn.Close(t) + conn.Query("BEGIN"). + Reply("BEGIN"). + Query("create extension if not exists pg_net schema extensions"). + Reply("CREATE EXTENSION"). + Query("COMMIT"). + Reply("COMMIT") helper.MockApiPrivilegesRevoke(conn) helper.MockMigrationHistory(conn). Query("RESET ALL"). diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index b76f1a39dc..5c88a1cfa1 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -189,7 +189,6 @@ const setupInputs: LegacySetupInputs = { authEnabled: true, storageEnabled: true, realtimeEnabled: true, - webhooksEnabled: false, autoExpose: false, vaultNames: [], rolesSql: "", diff --git a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts index a593fbf64c..5d3d36d0c3 100644 --- a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts +++ b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts @@ -151,8 +151,6 @@ export interface LegacyBaselineTomlConfig { readonly storageEnabled: boolean; /** `[realtime] enabled`, default true. */ readonly realtimeEnabled: boolean; - /** Effective `[experimental.webhooks] enabled` (absent → false). */ - readonly webhooksEnabled: boolean; /** * `[api] auto_expose_new_tables` (tri-state `*bool`). `None` when unset. Drives * `ApplyApiPrivileges`; the cache key folds in the *effective* bool (unset and @@ -2078,7 +2076,6 @@ const readDbTomlCore = Effect.fnUntraced(function* ( true, lookup, ), - webhooksEnabled, apiAutoExposeNewTables, vaultNames, }, diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts index 8eacb8ab6c..39d1ded01a 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts @@ -34,7 +34,7 @@ const MIGRATE_FILE_PATTERN = /^([0-9]+)_(.*)\.sql$/; // `internal/utils/misc.go` — `ProjectHostPattern`, matches a direct `db..supabase.{co,red}` host. const PROJECT_HOST_PATTERN = /^(db\.)([a-z]{20})\.supabase\.(co|red)$/; -/** Inputs to `setupInputsToken` — everything `start.SetupDatabase` consumes. */ +/** Inputs that shape the legacy `WithLegacyPgNetBaseline` shadow setup. */ export interface LegacySetupInputs { /** The resolved Postgres image (`Config.Db.Image`); only its tag is used. */ readonly image: string; @@ -42,8 +42,6 @@ export interface LegacySetupInputs { readonly authEnabled: boolean; readonly storageEnabled: boolean; readonly realtimeEnabled: boolean; - /** Effective `experimental.webhooks.enabled` (absent → false). */ - readonly webhooksEnabled: boolean; /** Effective `api.auto_expose_new_tables` (unset and false both → false). */ readonly autoExpose: boolean; /** `[db.vault]` secret names (sorted before hashing). */ @@ -103,7 +101,6 @@ export function legacySetupInputsToken(inputs: LegacySetupInputs): string { payload += `auth=${boolToken(inputs.authEnabled)} storage=${boolToken( inputs.storageEnabled, )} realtime=${boolToken(inputs.realtimeEnabled)}\n`; - payload += `database_webhooks=${boolToken(inputs.webhooksEnabled)}\n`; payload += `auto_expose_new_tables=${boolToken(inputs.autoExpose)}\n`; for (const name of [...inputs.vaultNames].sort()) payload += `vault=${name}\n`; payload += inputs.rolesSql; @@ -148,7 +145,6 @@ export const legacyResolveSetupInputs = Effect.fnUntraced(function* ( authEnabled: baseline.authEnabled, storageEnabled: baseline.storageEnabled, realtimeEnabled: baseline.realtimeEnabled, - webhooksEnabled: baseline.webhooksEnabled, autoExpose: Option.isSome(baseline.apiAutoExposeNewTables) && baseline.apiAutoExposeNewTables.value, vaultNames: baseline.vaultNames, diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.unit.test.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.unit.test.ts index f67e611a5e..fc32436de8 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.unit.test.ts @@ -37,7 +37,6 @@ const BASE: LegacySetupInputs = { authEnabled: true, storageEnabled: true, realtimeEnabled: true, - webhooksEnabled: false, autoExpose: false, vaultNames: [], rolesSql: "", @@ -70,7 +69,7 @@ describe("legacyBaselineVersionToken", () => { describe("legacySetupInputsToken", () => { it("byte-matches the Go hash input sequence", () => { const expected = sha12( - "17.6.1.135\nauth=true storage=true realtime=true\ndatabase_webhooks=false\nauto_expose_new_tables=false\n", + "17.6.1.135\nauth=true storage=true realtime=true\nauto_expose_new_tables=false\n", ); expect(legacySetupInputsToken(BASE)).toBe(expected); }); @@ -82,7 +81,7 @@ describe("legacySetupInputsToken", () => { rolesSql: "create role app;", }); const expected = sha12( - "17.6.1.135\nauth=true storage=true realtime=true\ndatabase_webhooks=false\nauto_expose_new_tables=false\n" + + "17.6.1.135\nauth=true storage=true realtime=true\nauto_expose_new_tables=false\n" + "vault=a_secret\nvault=b_secret\ncreate role app;", ); expect(token).toBe(expected); @@ -91,7 +90,6 @@ describe("legacySetupInputsToken", () => { it("self-invalidates when any baseline input changes", () => { const baseToken = legacySetupInputsToken(BASE); expect(legacySetupInputsToken({ ...BASE, authEnabled: false })).not.toBe(baseToken); - expect(legacySetupInputsToken({ ...BASE, webhooksEnabled: true })).not.toBe(baseToken); expect(legacySetupInputsToken({ ...BASE, autoExpose: true })).not.toBe(baseToken); expect(legacySetupInputsToken({ ...BASE, vaultNames: ["x"] })).not.toBe(baseToken); expect(legacySetupInputsToken({ ...BASE, rolesSql: "x" })).not.toBe(baseToken); @@ -419,7 +417,6 @@ describe("legacyResolveSetupInputs", () => { authEnabled: true, storageEnabled: false, realtimeEnabled: true, - webhooksEnabled: false, apiAutoExposeNewTables: Option.none(), vaultNames: ["a_secret"], }), @@ -431,7 +428,6 @@ describe("legacyResolveSetupInputs", () => { authEnabled: true, storageEnabled: false, realtimeEnabled: true, - webhooksEnabled: false, autoExpose: false, vaultNames: ["a_secret"], rolesSql: "", @@ -452,7 +448,6 @@ describe("legacyResolveSetupInputs", () => { authEnabled: true, storageEnabled: true, realtimeEnabled: true, - webhooksEnabled: false, apiAutoExposeNewTables: Option.some(true), vaultNames: [], }), From 438a9ac69c67bea4708c2fb38c7303521d6ecdfd Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 12:46:09 +0200 Subject: [PATCH 09/82] fix(cli): warn about manifestless declarative removals --- .../db/schema/declarative/declarative.flow.ts | 42 ++++++++++++ .../declarative/declarative.flow.unit.test.ts | 52 +++++++++++++++ ...eclarative.orchestrate.integration.test.ts | 13 +++- .../declarative/declarative.orchestrate.ts | 5 ++ .../schema/declarative/sync/sync.handler.ts | 10 +++ .../declarative/sync/sync.integration.test.ts | 54 +++++++++++++++ .../legacy-pgdelta-engine.next.layer.ts | 2 + .../shared/legacy-pgdelta-engine.service.ts | 13 ++++ .../legacy-pgdelta-next-adapter.layer.ts | 35 ++++++++++ .../legacy-pgdelta-next-adapter.service.ts | 6 +- .../legacy-pgdelta-next-adapter.unit.test.ts | 66 +++++++++++++++++++ 11 files changed, 296 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts index 008c1e6426..57ba1be344 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts @@ -1,3 +1,6 @@ +import type { LegacyPgDeltaImplementation } from "../../../../shared/legacy-pgdelta-next-flag.ts"; +import type { LegacyPgDeltaRemovalSummary } from "../../shared/legacy-pgdelta-engine.service.ts"; + /** * Pure control-flow helpers ported 1:1 from * `apps/cli-go/cmd/db_schema_declarative.go`. Kept free of Effect/services so @@ -34,3 +37,42 @@ export function legacyResolveDeclarativeSyncApplyDecision(opts: { if (opts.tty) return "prompt"; return "skip"; } + +/** + * Warns when pg-delta next sees semantic removals that a manifest-less, + * potentially legacy-authored declarative tree may simply have omitted. + */ +export function legacyDeclarativeCompatibilityWarning(opts: { + readonly implementation: LegacyPgDeltaImplementation; + readonly manifestPresent: boolean; + readonly removals: LegacyPgDeltaRemovalSummary; +}): string | undefined { + if (opts.implementation !== "next" || opts.manifestPresent) return undefined; + const { extensions, extensionIntents } = opts.removals; + if (extensions.length === 0 && extensionIntents.length === 0) return undefined; + + const cronJobs = extensionIntents + .filter((intent) => intent.extension === "pg_cron" && intent.intentKind === "job") + .map((intent) => intent.key); + const otherIntents = extensionIntents.filter( + (intent) => intent.extension !== "pg_cron" || intent.intentKind !== "job", + ); + const detected = [ + ...(extensions.length > 0 ? [`Extensions: ${extensions.join(", ")}`] : []), + ...(cronJobs.length > 0 ? [`pg_cron jobs: ${cronJobs.join(", ")}`] : []), + ...(otherIntents.length > 0 + ? [ + `Extension intents: ${otherIntents + .map((intent) => `${intent.extension} ${intent.intentKind} ${intent.key}`) + .join(", ")}`, + ] + : []), + ]; + + return [ + "WARNING: This declarative schema has no pg-delta next export manifest and may have been generated by the legacy engine.", + "pg-delta next plans to remove objects that legacy exports may omit:", + ...detected, + "If these removals are unintended, do not apply this migration. Re-export using pg-delta next with `supabase db schema declarative generate --overwrite` for the intended target, or add declarations for the objects you want to keep, then run sync again.", + ].join("\n"); +} diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts index 388c20c475..955ddb88b7 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts @@ -1,10 +1,62 @@ import { describe, expect, it } from "vitest"; import { + legacyDeclarativeCompatibilityWarning, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, } from "./declarative.flow.ts"; +const removals = { + extensions: ["pgcrypto", "uuid-ossp"], + extensionIntents: [ + { extension: "pg_cron", intentKind: "job", key: "refresh download metrics" }, + { extension: "pgmq", intentKind: "queue", key: "emails" }, + ], +}; + +describe("legacyDeclarativeCompatibilityWarning", () => { + it("explains manifest-less next removals and remediation", () => { + const warning = legacyDeclarativeCompatibilityWarning({ + implementation: "next", + manifestPresent: false, + removals, + }); + expect(warning).toContain("may have been generated by the legacy engine"); + expect(warning).toContain("Extensions: pgcrypto, uuid-ossp"); + expect(warning).toContain("pg_cron jobs: refresh download metrics"); + expect(warning).toContain("Extension intents: pgmq queue emails"); + expect(warning).toContain("declarative generate --overwrite"); + expect(warning).toContain("add declarations"); + }); + + it("is suppressed for next exports with a manifest", () => { + expect( + legacyDeclarativeCompatibilityWarning({ + implementation: "next", + manifestPresent: true, + removals, + }), + ).toBeUndefined(); + }); + + it("is suppressed for the legacy engine and irrelevant removals", () => { + expect( + legacyDeclarativeCompatibilityWarning({ + implementation: "legacy", + manifestPresent: false, + removals, + }), + ).toBeUndefined(); + expect( + legacyDeclarativeCompatibilityWarning({ + implementation: "next", + manifestPresent: false, + removals: { extensions: [], extensionIntents: [] }, + }), + ).toBeUndefined(); + }); +}); + describe("legacyResolveDeclarativeMigrationName", () => { it("prefers an explicit --name over --file", () => { expect(legacyResolveDeclarativeMigrationName("my_change", "declarative_sync")).toBe( diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 5c88a1cfa1..33f2c29836 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -155,6 +155,12 @@ describe("legacyDiffDeclarativeToMigrations", () => { files: [], sourceRef: "migrations", targetRef: "declarative", + removals: { + extensions: ["pgcrypto"], + extensionIntents: [ + { extension: "pg_cron", intentKind: "job", key: "refresh metrics" }, + ], + }, }); }, }), @@ -163,7 +169,7 @@ describe("legacyDiffDeclarativeToMigrations", () => { { ...ctx(dir, declDir), debug: true, noCache: true }, setupInputs, ).pipe( - Effect.tap(() => + Effect.tap((result) => Effect.sync(() => { expect(calls[0]?.files).toEqual([ { name: "nested/a.sql", sql: "select 'a';" }, @@ -172,6 +178,11 @@ describe("legacyDiffDeclarativeToMigrations", () => { expect(calls[0]?.manifest).toEqual({ redactSecrets: true, scope: "database" }); expect(calls[0]?.debug).toBe(true); expect(calls[0]?.noCache).toBe(true); + expect(result.manifestPresent).toBe(true); + expect(result.removals).toEqual({ + extensions: ["pgcrypto"], + extensionIntents: [{ extension: "pg_cron", intentKind: "job", key: "refresh metrics" }], + }); rmSync(dir, { recursive: true, force: true }); }), ), diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 8616350f54..2244fd36ac 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -6,6 +6,7 @@ import { legacyFindDropStatements } from "../../../../shared/legacy-sql-split.ts import { LegacyPgDeltaEngine, type LegacyPgDeltaDatabaseEndpoint, + type LegacyPgDeltaRemovalSummary, type LegacyPgDeltaRenderedFile, } from "../../shared/legacy-pgdelta-engine.service.ts"; import { @@ -33,6 +34,8 @@ export interface LegacyDeclarativeSyncResult { readonly sourceRef: string; readonly targetRef: string; readonly dropWarnings: ReadonlyArray; + readonly manifestPresent: boolean; + readonly removals: LegacyPgDeltaRemovalSummary; } const declarativeError = (message: string) => new LegacyDeclarativeDiffError({ message }); @@ -82,6 +85,8 @@ export const legacyDiffDeclarativeToMigrations = Effect.fnUntraced(function* ( sourceRef: result.sourceRef, targetRef: result.targetRef, dropWarnings: legacyFindDropStatements(result.sql), + manifestPresent: manifest !== undefined, + removals: result.removals ?? { extensions: [], extensionIntents: [] }, } satisfies LegacyDeclarativeSyncResult; }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 08637b8412..426561c298 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -46,6 +46,7 @@ import { LegacyDeclarativeNonInteractiveError, } from "../declarative.errors.ts"; import { + legacyDeclarativeCompatibilityWarning, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, } from "../declarative.flow.ts"; @@ -312,6 +313,15 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara yield* output.raw("Generated migration SQL:\n", "stderr"); yield* output.raw(`${result.diffSQL}\n`, "stderr"); + const compatibilityWarning = legacyDeclarativeCompatibilityWarning({ + implementation: engine.implementation, + manifestPresent: result.manifestPresent, + removals: result.removals, + }); + if (compatibilityWarning !== undefined) { + yield* output.raw(`${legacyYellow(compatibilityWarning)}\n`, "stderr"); + } + // Step 4: resolve migration name (prompt in TTY when --name unset). const file = Option.getOrElse(flags.file, () => DEFAULT_SYNC_NAME); const explicitName = Option.getOrElse(flags.name, () => ""); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index ee549f29bc..48aac13a9f 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -47,6 +47,7 @@ import { LegacyPgDeltaSslProbe } from "../../../../../shared/legacy-pgdelta-ssl- import { legacyPgDeltaLegacyEngineLayer } from "../../../shared/legacy-pgdelta-engine.legacy.layer.ts"; import { LegacyPgDeltaEngine, + type LegacyPgDeltaRemovalSummary, type LegacyPgDeltaRenderedFile, } from "../../../shared/legacy-pgdelta-engine.service.ts"; import { LegacyDeclarativeShadowDbError } from "../../../shared/legacy-pgdelta.errors.ts"; @@ -89,6 +90,7 @@ interface SetupOpts { exportJson?: string; engineImplementation?: "legacy" | "next"; renderedFiles?: ReadonlyArray; + removals?: LegacyPgDeltaRemovalSummary; } function setup(workdir: string, opts: SetupOpts = {}) { @@ -248,6 +250,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { files: nextFiles, sourceRef: "migrations", targetRef: "declarative", + removals: opts.removals, }), }), ) @@ -883,6 +886,57 @@ describe("legacy db schema declarative sync integration", () => { }, ); + it.effect( + "warns before writing when next sees legacy-coverage removals without a manifest", + () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + experimental: true, + engineImplementation: "next", + diffSql: + "select cron.unschedule('refresh download metrics');\nDROP EXTENSION \"pgcrypto\";\n", + removals: { + extensions: ["pgcrypto", "uuid-ossp"], + extensionIntents: [ + { extension: "pg_cron", intentKind: "job", key: "refresh download metrics" }, + ], + }, + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + const chunks = s.out.rawChunks.map((chunk) => stripAnsi(chunk.text)); + const warningAt = chunks.findIndex((chunk) => + chunk.includes("may have been generated by the legacy engine"), + ); + const createdAt = chunks.findIndex((chunk) => chunk.includes("Created new migration at")); + expect(warningAt).toBeGreaterThan(-1); + expect(chunks[warningAt]).toContain("Extensions: pgcrypto, uuid-ossp"); + expect(chunks[warningAt]).toContain("pg_cron jobs: refresh download metrics"); + expect(warningAt).toBeLessThan(createdAt); + }).pipe(Effect.provide(s.layer)); + }, + ); + + it.effect("suppresses the compatibility warning when a next export manifest is present", () => { + seedDeclarative(tmp.current); + writeFileSync( + join(tmp.current, "supabase", "database", ".pgdelta-export.json"), + JSON.stringify({ formatVersion: 1, redactSecrets: true, scope: "database" }), + ); + const s = setup(tmp.current, { + experimental: true, + engineImplementation: "next", + diffSql: 'DROP EXTENSION "pgcrypto";\n', + removals: { extensions: ["pgcrypto"], extensionIntents: [] }, + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + const output = stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join("")); + expect(output).not.toContain("may have been generated by the legacy engine"); + expect(output).toContain("Found drop statements"); + }).pipe(Effect.provide(s.layer)); + }); + it.effect( "--apply: applies the migration natively (BEGIN … statements … COMMIT + history)", () => { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 886aa020df..091480a184 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -65,6 +65,7 @@ function normalizeNextDiff( readonly transactionMode: "transactional" | "none"; readonly actionCount: number; }>; + readonly removals?: LegacyPgDeltaDiffResult["removals"]; readonly debug?: { readonly sourceSnapshot?: string; readonly desiredSnapshot?: string; @@ -84,6 +85,7 @@ function normalizeNextDiff( transactionMode: file.transactionMode, actionCount: file.actionCount, })), + ...(result.removals !== undefined ? { removals: result.removals } : {}), ...(result.debug !== undefined ? { debug: { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts index d84fb9a1f6..ece7fe9fc3 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts @@ -51,6 +51,18 @@ export interface LegacyPgDeltaRenderedFile { readonly actionCount?: number; } +interface LegacyPgDeltaExtensionIntentRemoval { + readonly extension: string; + readonly intentKind: string; + readonly key: string; +} + +/** Root object removals retained from a semantic pg-delta plan. */ +export interface LegacyPgDeltaRemovalSummary { + readonly extensions: ReadonlyArray; + readonly extensionIntents: ReadonlyArray; +} + interface LegacyPgDeltaDebugArtifacts { readonly sourceSnapshot?: string; readonly desiredSnapshot?: string; @@ -64,6 +76,7 @@ export interface LegacyPgDeltaDiffResult { readonly changes: boolean; readonly sql: string; readonly files: ReadonlyArray; + readonly removals?: LegacyPgDeltaRemovalSummary; readonly debug?: LegacyPgDeltaDebugArtifacts; } diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index 395dfcbd0b..15ebab0d9a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -13,6 +13,7 @@ import { supabaseProfile, } from "@supabase/pg-delta/integrations"; import { plan, serializePlan } from "@supabase/pg-delta/plan"; +import type { Plan as PgDeltaPlan } from "@supabase/pg-delta/plan"; import type { Policy } from "@supabase/pg-delta/policy"; import type { SqlFormatOptions } from "@supabase/pg-delta/sql-format"; @@ -32,6 +33,7 @@ import { type LegacyPgDeltaNextSqlFile, type LegacyPgDeltaNextOperation, } from "./legacy-pgdelta-next-adapter.service.ts"; +import type { LegacyPgDeltaRemovalSummary } from "./legacy-pgdelta-engine.service.ts"; interface LegacyPgDeltaNextLibraryDiagnostic { readonly code: string; @@ -121,9 +123,40 @@ export interface LegacyPgDeltaNextLibraries string; readonly serializePlan: (plan: Plan) => string; + readonly summarizeRemovals: (plan: Plan) => LegacyPgDeltaRemovalSummary; readonly encodeSubject: (subject: Subject) => string; } +export function legacySummarizePgDeltaNextRemovals( + generatedPlan: Pick, +): LegacyPgDeltaRemovalSummary { + const extensions = new Set(); + const extensionIntents = new Map< + string, + LegacyPgDeltaRemovalSummary["extensionIntents"][number] + >(); + for (const delta of generatedPlan.deltas) { + if (delta.verb !== "remove" || delta.fact.parent !== undefined) continue; + const id = delta.fact.id; + if (id.kind === "extension") { + extensions.add(id.name); + continue; + } + if (id.kind !== "extensionIntent") continue; + const removal = { extension: id.ext, intentKind: id.intentKind, key: id.key }; + extensionIntents.set(`${id.ext}\u0000${id.intentKind}\u0000${id.key}`, removal); + } + return { + extensions: [...extensions].sort(), + extensionIntents: [...extensionIntents.values()].sort( + (left, right) => + left.extension.localeCompare(right.extension) || + left.intentKind.localeCompare(right.intentKind) || + left.key.localeCompare(right.key), + ), + }; +} + function legacyPgDeltaNextMessage(operation: LegacyPgDeltaNextOperation, cause: unknown): string { const detail = cause instanceof Error ? cause.message : String(cause); const diagnostics = @@ -516,6 +549,7 @@ function legacyMakePgDeltaNextAdapter ({ + extensions: ["pgcrypto"], + extensionIntents: [ + { extension: "pg_cron", intentKind: "job", key: "refresh download metrics" }, + ], + }), encodeSubject: (subject) => `subject:${subject.id}`, }; @@ -166,6 +173,57 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { } describe("LegacyPgDeltaNextAdapter", () => { + it("summarizes only root extension and extension-intent removals", () => { + expect( + legacySummarizePgDeltaNextRemovals({ + deltas: [ + { verb: "remove", fact: { id: { kind: "extension", name: "uuid-ossp" }, payload: {} } }, + { verb: "remove", fact: { id: { kind: "extension", name: "pgcrypto" }, payload: {} } }, + { + verb: "remove", + fact: { + id: { kind: "extension", name: "nested-extension" }, + parent: { kind: "schema", name: "extensions" }, + payload: {}, + }, + }, + { + verb: "remove", + fact: { + id: { + kind: "extensionIntent", + ext: "pg_cron", + intentKind: "job", + key: "refresh download metrics", + }, + payload: {}, + }, + }, + { + verb: "remove", + fact: { + id: { kind: "comment", target: { kind: "extension", name: "pgcrypto" } }, + payload: {}, + }, + }, + { + verb: "unlink", + edge: { + from: { kind: "extension", name: "pgcrypto" }, + to: { kind: "schema", name: "extensions" }, + kind: "depends", + }, + }, + ], + }), + ).toEqual({ + extensions: ["pgcrypto", "uuid-ossp"], + extensionIntents: [ + { extension: "pg_cron", intentKind: "job", key: "refresh download metrics" }, + ], + }); + }); + it("filters platform parameter ACL coverage without hiding user-owned ACLs", () => { const diagnostics = [ { @@ -441,6 +499,12 @@ describe("LegacyPgDeltaNextAdapter", () => { "declarativeTarget", ]); expect(planned.skipped).toEqual([{ file: "roles.sql", statement: "create role ignored" }]); + expect(planned.removals).toEqual({ + extensions: ["pgcrypto"], + extensionIntents: [ + { extension: "pg_cron", intentKind: "job", key: "refresh download metrics" }, + ], + }); expect(planned.debug).toEqual({ plan: JSON.stringify({ source: "target-facts", desired: "loaded-files" }), }); @@ -503,6 +567,7 @@ describe("LegacyPgDeltaNextAdapter", () => { }), serializeSnapshot: () => "unused", serializePlan: () => "unused", + summarizeRemovals: () => ({ extensions: [], extensionIntents: [] }), encodeSubject: (subject: string) => subject, }); @@ -550,6 +615,7 @@ describe("LegacyPgDeltaNextAdapter", () => { }, serializeSnapshot: () => "unused", serializePlan: () => "unused", + summarizeRemovals: () => ({ extensions: [], extensionIntents: [] }), encodeSubject: (subject: string) => subject, }); From 0ebe58cd26926dc5333bbb7958240c024c48fb33 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 13:13:36 +0200 Subject: [PATCH 10/82] fix(cli): warn on pg-delta coverage gaps --- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 11 +- .../legacy/commands/db/diff/diff.command.ts | 6 + .../legacy/commands/db/diff/diff.handler.ts | 2 + .../commands/db/diff/diff.integration.test.ts | 6 +- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 11 +- .../legacy/commands/db/pull/pull.command.ts | 6 + .../legacy/commands/db/pull/pull.handler.ts | 2 + .../commands/db/pull/pull.integration.test.ts | 10 +- ...eclarative.orchestrate.integration.test.ts | 31 ++- .../declarative/declarative.orchestrate.ts | 3 + .../schema/declarative/declarative.shared.ts | 5 + .../declarative/generate/SIDE_EFFECTS.md | 9 +- .../declarative/generate/generate.command.ts | 9 +- .../declarative/generate/generate.handler.ts | 1 + .../generate/generate.integration.test.ts | 1 + .../schema/declarative/sync/SIDE_EFFECTS.md | 9 +- .../schema/declarative/sync/sync.command.ts | 8 +- .../schema/declarative/sync/sync.handler.ts | 1 + .../declarative/sync/sync.integration.test.ts | 1 + .../legacy-pgdelta-engine.layer.unit.test.ts | 1 + .../legacy-pgdelta-engine.next.layer.ts | 38 ++-- .../shared/legacy-pgdelta-engine.service.ts | 2 + .../shared/legacy-pgdelta-next-diagnostics.ts | 145 ++++++++++++-- ...gacy-pgdelta-next-diagnostics.unit.test.ts | 189 ++++++++++++++---- 24 files changed, 404 insertions(+), 103 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 7551aaec40..4728ef0fcc 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -19,10 +19,11 @@ bundled Go binary. `metadata.json` and, when available, `source-snapshot.json`, `desired-snapshot.json`, `plan.json`, and `diagnostics.json`. These are diagnostic artifacts, not reusable catalogs. -- The default engine refuses to emit a diff when extraction reports an error or a - strict coverage gap (`unmodeled_kind` or `unresolved_security_label`). The error - identifies the diagnostic origin, code, subject, and message; when debug capture - is enabled, the bundle is saved before the refusal. +- The default engine always refuses extraction errors. Coverage gaps + (`unmodeled_kind` or `unresolved_security_label`) warn and remain unmanaged by + default; `--strict-coverage` turns them into a refusal. Warnings identify the + diagnostic origin and explain that unsupported changes are absent from the diff; + when debug capture is enabled, the bundle is saved before policy evaluation. - SQL text and file segmentation may differ from the legacy renderer. Applicable output and convergence (a subsequent diff is empty) are the compatibility contract. @@ -114,6 +115,8 @@ Progress strings still go to stderr; stdout carries a single structured envelope binary (their side effects are Go's); the Go child's telemetry is disabled so the single `cli_command_executed` event comes from this TS command. - Explicit `--from`/`--to` mode always uses pg-delta and writes to `--output` (or stdout). +- `--strict-coverage` applies to the bundled pg-delta engine and refuses output when + it encounters schema objects it cannot manage. - Normal mode always compares the migrations shadow with the selected live database. Declarative files and `schema_paths` never replace that target; use `supabase db schema declarative sync` for declarative comparison. diff --git a/apps/cli/src/legacy/commands/db/diff/diff.command.ts b/apps/cli/src/legacy/commands/db/diff/diff.command.ts index 0aa0d9b1ff..37c787b137 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.command.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.command.ts @@ -37,6 +37,11 @@ const config = { Flag.withDescription("Use pg-delta to generate schema diff."), Flag.optional, ), + strictCoverage: Flag.boolean("strict-coverage").pipe( + Flag.withDescription( + "Fail when bundled pg-delta finds schema objects it cannot manage instead of leaving them unmanaged.", + ), + ), from: Flag.string("from").pipe( Flag.withDescription("Diff from local, linked, migrations, or a Postgres URL."), Flag.optional, @@ -99,6 +104,7 @@ export const legacyDbDiffCommand = Command.make("diff", config).pipe( "use-pgadmin": flags.usePgAdmin, "use-pg-schema": flags.usePgSchema, "use-pg-delta": flags.usePgDelta, + "strict-coverage": flags.strictCoverage, from: flags.from, to: flags.to, output: flags.output, diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index 20db917027..0941a0a75a 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -287,6 +287,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy schema: flags.schema, formatOptions: Option.getOrElse(cfg.pgDelta.formatOptions, () => ""), debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, }); // Explicit-mode output: `--output` file (Go's `writeOutput`) or stdout // (Go's `fmt.Print`, no trailing newline — pg-delta ends each statement `;\n`). @@ -443,6 +444,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy formatOptions, ...(connType === "linked" && linkedRef !== undefined ? { projectRef: linkedRef } : {}), debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, }); return { sql: result.sql, files: result.files }; }) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index e77c243a3a..6c8c9aa07e 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -250,6 +250,7 @@ const flags = (over: Partial = {}): LegacyDbDiffFlags => ({ usePgAdmin: over.usePgAdmin ?? Option.none(), usePgSchema: over.usePgSchema ?? Option.none(), usePgDelta: over.usePgDelta ?? Option.none(), + strictCoverage: over.strictCoverage ?? false, from: over.from ?? Option.none(), to: over.to ?? Option.none(), output: over.output ?? Option.none(), @@ -295,11 +296,14 @@ describe("legacy db diff", () => { it.effect("diffs local with pgdelta when --use-pg-delta is set", () => { const s = setup(tmp.current, { diffSql: "create table p ();\n" }); return Effect.gen(function* () { - yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), schema: ["public"] })); + yield* legacyDbDiff( + flags({ usePgDelta: Option.some(true), strictCoverage: true, schema: ["public"] }), + ); expect(s.provisionCalls).toEqual([]); expect(s.databaseDiffCalls).toHaveLength(1); expect(s.databaseDiffCalls[0]).toMatchObject({ schema: ["public"], + strictCoverage: true, target: { kind: "database", connection: { diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index fb28ffca76..0a1c342ef0 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -38,10 +38,11 @@ Notes/Delegation section below). `supabase/.temp/pgdelta/v2/debug//` as `metadata.json` plus available snapshot, plan, and diagnostics JSON files. These artifacts are never catalog cache inputs. -- The default engine refuses migration or declarative output when extraction - reports an error or a strict coverage gap (`unmodeled_kind` or - `unresolved_security_label`). The refusal names the diagnostic, and debug - artifacts are saved first when capture is enabled. +- The default engine always refuses extraction errors. Coverage gaps + (`unmodeled_kind` or `unresolved_security_label`) warn and remain unmanaged by + default; `--strict-coverage` turns them into a refusal. Declarative warnings make + clear that unsupported objects are absent from the exported files. Debug + artifacts are saved before policy evaluation when capture is enabled. - New-engine SQL bytes and transaction-split filenames may differ. Successful execution and convergence on a subsequent pull/diff are the contract. @@ -136,6 +137,8 @@ Progress strings still go to stderr; stdout carries a single structured envelope - `--declarative` / deprecated `--use-pg-delta` are mutually exclusive with `--diff-engine`; `--db-url` / `--linked` (default) / `--local` are a target group. - `--use-pg-delta` is hidden and emits the cobra deprecation line to stderr. +- `--strict-coverage` applies to bundled pg-delta diff and declarative-export paths; + it refuses output when pg-delta encounters schema objects it cannot manage. - The initial-migra pull (no local migrations) is native: it streams a `pg_dump` of the remote schema into the migration file, then appends the migra diff. An empty diff after a non-empty dump is swallowed (Go's `swallowInitialInSync`); an empty diff --git a/apps/cli/src/legacy/commands/db/pull/pull.command.ts b/apps/cli/src/legacy/commands/db/pull/pull.command.ts index d024c5e789..64f653150b 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.command.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.command.ts @@ -34,6 +34,11 @@ const config = { Flag.withDescription("Diff engine to use for migration-style db pull."), Flag.optional, ), + strictCoverage: Flag.boolean("strict-coverage").pipe( + Flag.withDescription( + "Fail when bundled pg-delta finds schema objects it cannot manage instead of leaving them unmanaged.", + ), + ), schema: Flag.string("schema").pipe( Flag.withAlias("s"), Flag.withDescription("Comma separated list of schema to include."), @@ -76,6 +81,7 @@ export const legacyDbPullCommand = Command.make("pull", config).pipe( declarative: flags.declarative, "use-pg-delta": flags.usePgDelta, "diff-engine": flags.diffEngine, + "strict-coverage": flags.strictCoverage, schema: flags.schema, "db-url": flags.dbUrl, linked: flags.linked, diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 4382438e6c..10ae535599 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -429,6 +429,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy formatOptions, projectRef: connType === "linked" ? linkedRef : undefined, debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, noCache: false, }), ); @@ -652,6 +653,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy formatOptions, projectRef: connType === "linked" ? linkedRef : undefined, debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, }), ) : yield* Effect.gen(function* () { diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 806e7b5d28..1fe7db5cba 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -140,6 +140,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { operation: "diff" | "export"; targetRef: string; projectRef?: string; + strictCoverage: boolean; }> = []; let engineDiffCount = 0; const pgDeltaEngine = Layer.succeed( @@ -152,6 +153,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { operation: "diff", targetRef: input.target.ref, projectRef: input.projectRef, + strictCoverage: input.strictCoverage, }); engineDiffCount += 1; if (opts.edgeFailFirstWith !== undefined && engineDiffCount === 1) { @@ -225,6 +227,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { operation: "export", targetRef: input.target.ref, projectRef: input.projectRef, + strictCoverage: input.strictCoverage, }); if (opts.edgeFailFirstWith !== undefined && engineCalls.length === 1) { return Effect.fail( @@ -429,6 +432,7 @@ const flags = (over: Partial = {}): LegacyDbPullFlags => ({ declarative: over.declarative ?? Option.none(), usePgDelta: over.usePgDelta ?? Option.none(), diffEngine: over.diffEngine ?? Option.none(), + strictCoverage: over.strictCoverage ?? false, schema: over.schema ?? [], dbUrl: over.dbUrl ?? Option.none(), linked: over.linked ?? Option.none(), @@ -466,7 +470,7 @@ describe("legacy db pull", () => { yes: true, }); return Effect.gen(function* () { - yield* legacyDbPull(flags({ diffEngine: Option.some("pg-delta") })); + yield* legacyDbPull(flags({ diffEngine: Option.some("pg-delta"), strictCoverage: true })); const dir = join(tmp.current, "supabase", "migrations"); expect(existsSync(join(dir, `${"20240101000000"}_local.sql`))).toBe(true); // A single-unit plan keeps the unchanged `_remote_schema.sql` filename. @@ -483,6 +487,7 @@ describe("legacy db pull", () => { expect(s.historyUpserts.length).toBe(1); expect(s.engineCalls).toHaveLength(1); expect(s.engineCalls[0]?.operation).toBe("diff"); + expect(s.engineCalls[0]?.strictCoverage).toBe(true); expect(s.edgeRunCount).toBe(0); expect(streamText(s.out, "stdout")).toContain("Finished supabase db pull."); // The linked ref is pre-loaded (cheap, local-only) before `resolve()` runs, so @@ -658,8 +663,9 @@ describe("legacy db pull", () => { it.effect("pull --declarative exports declarative files (no migration)", () => { const s = setup(tmp.current, { edgeStdout: EXPORT_JSON }); return Effect.gen(function* () { - yield* legacyDbPull(flags({ declarative: Option.some(true) })); + yield* legacyDbPull(flags({ declarative: Option.some(true), strictCoverage: true })); expect(s.engineCalls[0]?.operation).toBe("export"); + expect(s.engineCalls[0]?.strictCoverage).toBe(true); expect(s.edgeRunCount).toBe(0); const err = streamText(s.out, "stderr"); // Go's order: `ConnectByConfig` prints Connecting (`pull.go:40`), then diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 33f2c29836..f85315a3c3 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -115,6 +115,7 @@ const ctx = (cwd: string, declarativeDir: string): LegacyDeclarativeRunContext = schema: [], noCache: false, debug: false, + strictCoverage: false, dnsResolver: "native", }); @@ -166,7 +167,7 @@ describe("legacyDiffDeclarativeToMigrations", () => { }), ); return legacyDiffDeclarativeToMigrations( - { ...ctx(dir, declDir), debug: true, noCache: true }, + { ...ctx(dir, declDir), debug: true, noCache: true, strictCoverage: true }, setupInputs, ).pipe( Effect.tap((result) => @@ -178,6 +179,7 @@ describe("legacyDiffDeclarativeToMigrations", () => { expect(calls[0]?.manifest).toEqual({ redactSecrets: true, scope: "database" }); expect(calls[0]?.debug).toBe(true); expect(calls[0]?.noCache).toBe(true); + expect(calls[0]?.strictCoverage).toBe(true); expect(result.manifestPresent).toBe(true); expect(result.removals).toEqual({ extensions: ["pgcrypto"], @@ -500,8 +502,12 @@ describe("legacyDiffDeclarativeToMigrations", () => { }); describe("legacyGenerateDeclarativeOutput", () => { - it.effect("propagates debug and no-cache to the selected engine", () => { - const calls: Array<{ readonly debug: boolean; readonly noCache: boolean }> = []; + it.effect("propagates debug, no-cache, and strict coverage to the selected engine", () => { + const calls: Array<{ + readonly debug: boolean; + readonly noCache: boolean; + readonly strictCoverage: boolean; + }> = []; const engine = Layer.succeed( LegacyPgDeltaEngine, LegacyPgDeltaEngine.of({ @@ -509,21 +515,34 @@ describe("legacyGenerateDeclarativeOutput", () => { diffExplicit: () => Effect.die("diffExplicit not used"), diffDatabase: () => Effect.die("diffDatabase not used"), exportDeclarativeSchema: (input) => { - calls.push({ debug: input.debug, noCache: input.noCache }); + calls.push({ + debug: input.debug, + noCache: input.noCache, + strictCoverage: input.strictCoverage, + }); return Effect.succeed({ files: [] }); }, planDeclarativeSchema: () => Effect.die("planDeclarativeSchema not used"), }), ); return legacyGenerateDeclarativeOutput( - { ...ctx("/proj", "/proj/supabase/database"), debug: true, noCache: true }, + { + ...ctx("/proj", "/proj/supabase/database"), + debug: true, + noCache: true, + strictCoverage: true, + }, { kind: "database", ref: "postgresql://postgres:postgres@127.0.0.1:54322/postgres", connectOptions: { isLocal: true, dnsResolver: "native" }, }, ).pipe( - Effect.tap(() => Effect.sync(() => expect(calls).toEqual([{ debug: true, noCache: true }]))), + Effect.tap(() => + Effect.sync(() => + expect(calls).toEqual([{ debug: true, noCache: true, strictCoverage: true }]), + ), + ), Effect.provide(engine), ); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 2244fd36ac..e12adcf17a 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -23,6 +23,7 @@ export interface LegacyDeclarativeRunContext { readonly schema: ReadonlyArray; readonly noCache: boolean; readonly debug: boolean; + readonly strictCoverage: boolean; readonly dnsResolver: "native" | "https"; readonly linkedProjectRef?: string; } @@ -73,6 +74,7 @@ export const legacyDiffDeclarativeToMigrations = Effect.fnUntraced(function* ( schema: run.schema, formatOptions: run.formatOptions, debug: run.debug, + strictCoverage: run.strictCoverage, files, noCache: run.noCache, setupInputs, @@ -100,6 +102,7 @@ export const legacyGenerateDeclarativeOutput = Effect.fnUntraced(function* ( schema: run.schema, formatOptions: run.formatOptions, debug: run.debug, + strictCoverage: run.strictCoverage, noCache: run.noCache, ...(run.linkedProjectRef !== undefined ? { projectRef: run.linkedProjectRef } : {}), target, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.shared.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.shared.ts index 1295e979bc..165ca947f2 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.shared.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.shared.ts @@ -16,5 +16,10 @@ export const legacyDbSchemaDeclarativeSharedBase = Command.make("declarative").p noCache: Flag.boolean("no-cache").pipe( Flag.withDescription("Disable catalog cache and force fresh shadow database setup."), ), + strictCoverage: Flag.boolean("strict-coverage").pipe( + Flag.withDescription( + "Fail when bundled pg-delta finds schema objects it cannot manage instead of leaving them unmanaged.", + ), + ), }), ); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index 7c9fe2dfff..f169698a21 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -17,10 +17,11 @@ platform view. change its extraction behavior. - With `PGDELTA_DEBUG`, default-engine export diagnostics are written below `supabase/.temp/pgdelta/v2/debug//`; they are never reused as catalogs. -- The default engine refuses an export when extraction reports an error or a - strict coverage gap (`unmodeled_kind` or `unresolved_security_label`). The - refusal names the diagnostic, and debug artifacts are saved first when capture - is enabled. +- The default engine always refuses extraction errors. Coverage gaps + (`unmodeled_kind` or `unresolved_security_label`) warn by default and explain that + unsupported objects are absent from the generated files; `--strict-coverage` + turns them into a refusal. Debug artifacts are saved before policy evaluation + when capture is enabled. - Generated SQL bytes and grouping may differ between engines. Reloading the export to the same managed state is the compatibility contract. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts index 05214b0f24..42fbd0d43d 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts @@ -61,7 +61,7 @@ const config = { // so the handler input merges it in alongside the leaf's own flags. export type LegacyDbSchemaDeclarativeGenerateFlags = CliCommand.Command.Config.Infer< typeof config -> & { readonly noCache: boolean }; +> & { readonly noCache: boolean; readonly strictCoverage: boolean }; export const legacyDbSchemaDeclarativeGenerateCommand = Command.make("generate", config).pipe( Command.withDescription("Generate declarative schema from a database."), @@ -70,7 +70,11 @@ export const legacyDbSchemaDeclarativeGenerateCommand = Command.make("generate", Effect.gen(function* () { // `--no-cache` is shared on the parent group; read the resolved value there. const shared = yield* legacyDbSchemaDeclarativeSharedBase; - const merged: LegacyDbSchemaDeclarativeGenerateFlags = { ...flags, noCache: shared.noCache }; + const merged: LegacyDbSchemaDeclarativeGenerateFlags = { + ...flags, + noCache: shared.noCache, + strictCoverage: shared.strictCoverage, + }; return yield* legacyDbSchemaDeclarativeGenerate(merged).pipe( // Go's PostRun prints this on success via `fmt.Println` → stdout // (`cmd/db_schema_declarative.go:93`), so keep it on stdout in text mode. In @@ -91,6 +95,7 @@ export const legacyDbSchemaDeclarativeGenerateCommand = Command.make("generate", withLegacyCommandInstrumentation({ flags: { "no-cache": merged.noCache, + "strict-coverage": merged.strictCoverage, overwrite: merged.overwrite, reset: merged.reset, schema: merged.schema, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts index d8c430d0ef..66e71c9e6e 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts @@ -150,6 +150,7 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec schema: flags.schema, noCache: flags.noCache, debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, dnsResolver, ...(linkedProjectRef !== undefined ? { linkedProjectRef } : {}), }; diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index 3374434112..537787d6b6 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -275,6 +275,7 @@ const flags = ( over: Partial = {}, ): LegacyDbSchemaDeclarativeGenerateFlags => ({ noCache: over.noCache ?? false, + strictCoverage: over.strictCoverage ?? false, overwrite: over.overwrite ?? false, reset: over.reset ?? false, schema: over.schema ?? [], diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index b3dafdd1f6..34c8487e17 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -14,10 +14,11 @@ as a new timestamped migration. extracts current state and maintains no reusable catalog cache. - With `PGDELTA_DEBUG`, default-engine snapshots, plan, and diagnostics are written below `supabase/.temp/pgdelta/v2/debug//` and are not reusable. -- The default engine refuses to emit a migration when extraction or declarative - loading reports an error or a strict coverage gap (`unmodeled_kind` or - `unresolved_security_label`). The refusal names the diagnostic, and debug - artifacts are saved first when capture is enabled. +- The default engine always refuses extraction or declarative-loading errors. + Coverage gaps (`unmodeled_kind` or `unresolved_security_label`) warn by default + and explain that unsupported changes are absent from the migration plan; + `--strict-coverage` turns them into a refusal. Debug artifacts are saved before + policy evaluation when capture is enabled. - Default-engine migrations may differ byte-for-byte and may be split into ordered files to preserve transaction boundaries. Successful execution and an empty subsequent sync are the compatibility contract. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts index db9da924de..162c42c05c 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts @@ -52,6 +52,7 @@ const config = { // so the handler input merges it in alongside the leaf's own flags. export type LegacyDbSchemaDeclarativeSyncFlags = CliCommand.Command.Config.Infer & { readonly noCache: boolean; + readonly strictCoverage: boolean; }; export const legacyDbSchemaDeclarativeSyncCommand = Command.make("sync", config).pipe( @@ -61,11 +62,16 @@ export const legacyDbSchemaDeclarativeSyncCommand = Command.make("sync", config) Effect.gen(function* () { // `--no-cache` is shared on the parent group; read the resolved value there. const shared = yield* legacyDbSchemaDeclarativeSharedBase; - const merged: LegacyDbSchemaDeclarativeSyncFlags = { ...flags, noCache: shared.noCache }; + const merged: LegacyDbSchemaDeclarativeSyncFlags = { + ...flags, + noCache: shared.noCache, + strictCoverage: shared.strictCoverage, + }; return yield* legacyDbSchemaDeclarativeSync(merged).pipe( withLegacyCommandInstrumentation({ flags: { "no-cache": merged.noCache, + "strict-coverage": merged.strictCoverage, schema: merged.schema, file: merged.file, name: merged.name, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 426561c298..14ccb9d8fa 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -158,6 +158,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara schema: flags.schema, noCache: flags.noCache, debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, dnsResolver, }; const ensureLocalPostgresImageCurrent = seam.ensureLocalPostgresImageCurrent(); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 48aac13a9f..ed08167276 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -315,6 +315,7 @@ const flags = ( over: Partial = {}, ): LegacyDbSchemaDeclarativeSyncFlags => ({ noCache: over.noCache ?? false, + strictCoverage: over.strictCoverage ?? false, schema: over.schema ?? [], file: over.file ?? Option.none(), name: over.name ?? Option.none(), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts index 2670d7aaa4..ce9fb08d30 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts @@ -154,6 +154,7 @@ describe("legacyPgDeltaEngineSelectorLayer", () => { schema: [], formatOptions: "", debug: false, + strictCoverage: false, }) .pipe(Effect.exit); expect(Exit.isFailure(exit)).toBe(true); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 091480a184..9b7d31fd6e 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -1,5 +1,6 @@ import { Clock, Effect, FileSystem, Layer, Path } from "effect"; +import { Output } from "../../../../shared/output/output.service.ts"; import { parseLegacyConnectionString } from "../../../shared/legacy-db-config.parse.ts"; import { LegacyDbConnectError } from "../../../shared/legacy-db-connection.errors.ts"; import { legacyAcquirePgPool } from "../../../shared/legacy-db-connection.sql-pg.layer.ts"; @@ -22,8 +23,8 @@ import { } from "./legacy-pgdelta-next-artifacts.ts"; import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; import { - legacyPgDeltaNextBlockingDiagnostic, - legacyPgDeltaNextBlockingDiagnosticMessage, + legacyPgDeltaNextDiagnosticReport, + legacyReportPgDeltaNextDiagnostics, } from "./legacy-pgdelta-next-diagnostics.ts"; /** Shared by both declarative planner entrypoints over the full isolated baseline. */ @@ -116,6 +117,8 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; const debugLogger = yield* LegacyDebugLogger; + const output = yield* Output; + let feedbackInvitationShown = false; const saveDebugArtifacts = ( workdir: string, @@ -153,19 +156,20 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( const acquireDatabase = (endpoint: LegacyPgDeltaDatabaseEndpoint) => legacyAcquirePgPool(parseEndpoint(endpoint), endpoint.connectOptions); - const rejectBlockingDiagnostic = ( + const reportDiagnostics = ( operation: LegacyPgDeltaNextOperation, - diagnostics: Parameters[0], + diagnostics: Parameters[1], + strictCoverage: boolean, ) => { - const blocking = legacyPgDeltaNextBlockingDiagnostic(diagnostics); - return blocking === undefined - ? Effect.void - : Effect.fail( - new LegacyPgDeltaEngineError({ - message: legacyPgDeltaNextBlockingDiagnosticMessage(operation, blocking), - cause: blocking, - }), - ); + const report = legacyPgDeltaNextDiagnosticReport(diagnostics, strictCoverage); + const showFeedback = !feedbackInvitationShown && report.unmodeledKinds.length > 0; + if (showFeedback) feedbackInvitationShown = true; + return legacyReportPgDeltaNextDiagnostics( + operation, + diagnostics, + strictCoverage, + showFeedback, + ).pipe(Effect.provideService(Output, output)); }; return LegacyPgDeltaEngine.of({ @@ -228,7 +232,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diagnostics: result.diagnostics, }) : undefined; - yield* rejectBlockingDiagnostic("diff", result.diagnostics); + yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage); return normalizeNextDiff(result, debugDirectory); }), ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), @@ -267,7 +271,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diagnostics: result.diagnostics, }) : undefined; - yield* rejectBlockingDiagnostic("diff", result.diagnostics); + yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage); return normalizeNextDiff(result, debugDirectory); }), ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), @@ -293,7 +297,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( : [...result.diagnostics, ...capture.diagnostics], }); } - yield* rejectBlockingDiagnostic("declarativeExport", result.diagnostics); + yield* reportDiagnostics("declarativeExport", result.diagnostics, input.strictCoverage); return { files: result.files, manifest: result.manifest }; }), ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), @@ -336,7 +340,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diagnostics: result.diagnostics, }) : undefined; - yield* rejectBlockingDiagnostic("declarativePlan", result.diagnostics); + yield* reportDiagnostics("declarativePlan", result.diagnostics, input.strictCoverage); return { ...normalizeNextDiff(result, debugDirectory), sourceRef: "pg-delta-next:migrations", diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts index ece7fe9fc3..c478cd5cd8 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts @@ -86,6 +86,8 @@ interface LegacyPgDeltaCommonInput { readonly formatOptions: string; readonly projectRef?: string; readonly debug: boolean; + /** Refuse coverage-gap diagnostics instead of continuing with those objects unmanaged. */ + readonly strictCoverage: boolean; } export interface LegacyPgDeltaExplicitDiffInput extends LegacyPgDeltaCommonInput { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts index 8eef84318d..453bb3191d 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts @@ -1,3 +1,7 @@ +import { Effect } from "effect"; + +import { Output } from "../../../../shared/output/output.service.ts"; +import { LegacyPgDeltaEngineError } from "./legacy-pgdelta-engine.service.ts"; import type { LegacyPgDeltaNextDiagnostic, LegacyPgDeltaNextOperation, @@ -5,26 +9,135 @@ import type { const coverageDiagnosticCodes = new Set(["unmodeled_kind", "unresolved_security_label"]); -export function legacyPgDeltaNextBlockingDiagnostic( +const operationConsequence: Record = { + diff: "Changes to these objects are omitted from the generated database diff.", + declarativeExport: "These objects are omitted from the exported declarative schema.", + declarativePlan: "Changes to these objects are omitted from the declarative migration plan.", + snapshotCapture: "These objects are omitted from the captured database snapshot.", +}; + +const operationAction: Record = { + diff: "emit the database diff", + declarativeExport: "export the declarative schema", + declarativePlan: "emit the declarative migration plan", + snapshotCapture: "capture the database snapshot", +}; + +export interface LegacyPgDeltaNextDiagnosticReport { + readonly diagnostics: ReadonlyArray; + readonly blocking: ReadonlyArray; + readonly coverage: ReadonlyArray; + readonly unmodeledKinds: ReadonlyArray; +} + +function diagnosticKind(diagnostic: LegacyPgDeltaNextDiagnostic): string | undefined { + if (diagnostic.code !== "unmodeled_kind") return undefined; + const kind = diagnostic.context?.kind; + if (typeof kind !== "string") return undefined; + const normalized = kind.trim().replaceAll(/\s+/gu, " "); + return normalized.length === 0 ? undefined : normalized; +} + +export function legacyPgDeltaNextDiagnosticReport( diagnostics: readonly LegacyPgDeltaNextDiagnostic[], -): LegacyPgDeltaNextDiagnostic | undefined { - return diagnostics.find( - (diagnostic) => diagnostic.severity === "error" || coverageDiagnosticCodes.has(diagnostic.code), + strictCoverage: boolean, +): LegacyPgDeltaNextDiagnosticReport { + const coverage = diagnostics.filter((diagnostic) => coverageDiagnosticCodes.has(diagnostic.code)); + const blocking = diagnostics.filter( + (diagnostic) => + diagnostic.severity === "error" || + (strictCoverage && coverageDiagnosticCodes.has(diagnostic.code)), ); + const unmodeledKinds = [ + ...new Set(diagnostics.map(diagnosticKind).filter((kind) => kind !== undefined)), + ].sort((left, right) => left.localeCompare(right)); + + return { diagnostics: [...diagnostics], blocking, coverage, unmodeledKinds }; } -export function legacyPgDeltaNextBlockingDiagnosticMessage( - operation: LegacyPgDeltaNextOperation, +export function legacyPgDeltaNextDiagnosticMessage( diagnostic: LegacyPgDeltaNextDiagnostic, ): string { - const action = - operation === "declarativeExport" - ? "export the declarative schema" - : operation === "declarativePlan" - ? "emit the declarative migration plan" - : operation === "snapshotCapture" - ? "capture the database snapshot" - : "emit the database diff"; - const subject = diagnostic.subject ?? "unknown"; - return `pg-delta next refused to ${action}: origin=${diagnostic.origin} code=${diagnostic.code} subject=${subject} message=${diagnostic.message}`; + const subject = + diagnostic.subject === undefined || diagnostic.subject === "unknown" + ? "" + : ` subject=${diagnostic.subject}`; + return `pg-delta next diagnostic: origin=${diagnostic.origin} code=${diagnostic.code}${subject} message=${diagnostic.message}`; +} + +function legacyPgDeltaNextCoverageMessage( + operation: LegacyPgDeltaNextOperation, + strictCoverage: boolean, +): string { + const policy = strictCoverage + ? "Strict coverage is enabled, so the operation will stop." + : operationConsequence[operation]; + return `pg-delta found schema objects it does not manage. ${policy}`; +} + +function shellQuote(value: string): string { + return `'${value.replaceAll("'", `'"'"'`)}'`; +} + +export function legacyPgDeltaNextFeedbackInvitation(kinds: readonly string[]): string | undefined { + if (kinds.length === 0) return undefined; + const problem = `pg-delta does not manage these PostgreSQL object kinds: ${kinds.join(", ")}`; + const solution = "Add pg-delta support for these PostgreSQL object kinds."; + return [ + "Request pg-delta support:", + ` supabase issue feature --problem ${shellQuote(problem)} --proposed-solution ${shellQuote(solution)}`, + ].join("\n"); +} + +function legacyPgDeltaNextBlockingDiagnosticMessage( + operation: LegacyPgDeltaNextOperation, + blockedByCoverage: boolean, +): string { + const reason = blockedByCoverage + ? "strict coverage rejected unmanaged schema objects" + : "pg-delta reported an error"; + return `pg-delta next refused to ${operationAction[operation]}: ${reason}`; } + +/** Render every adapter diagnostic and enforce the selected coverage policy. */ +export const legacyReportPgDeltaNextDiagnostics = Effect.fnUntraced(function* ( + operation: LegacyPgDeltaNextOperation, + diagnostics: readonly LegacyPgDeltaNextDiagnostic[], + strictCoverage: boolean, + showFeedback = true, +) { + const output = yield* Output; + const report = legacyPgDeltaNextDiagnosticReport(diagnostics, strictCoverage); + + for (const diagnostic of report.diagnostics) { + const message = legacyPgDeltaNextDiagnosticMessage(diagnostic); + if (diagnostic.severity === "error") { + yield* output.error(message); + } else if (diagnostic.severity === "warning") { + yield* output.warn(message); + } else { + yield* output.info(message); + } + } + + if (report.coverage.length > 0) { + yield* output.warn(legacyPgDeltaNextCoverageMessage(operation, strictCoverage)); + } + + const feedback = showFeedback + ? legacyPgDeltaNextFeedbackInvitation(report.unmodeledKinds) + : undefined; + if (feedback !== undefined) yield* output.info(feedback); + + if (report.blocking.length > 0) { + return yield* Effect.fail( + new LegacyPgDeltaEngineError({ + message: legacyPgDeltaNextBlockingDiagnosticMessage( + operation, + strictCoverage && report.coverage.length > 0, + ), + cause: report.blocking, + }), + ); + } +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts index f0cfe77a79..ce0ea2e371 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts @@ -1,58 +1,163 @@ -import { describe, expect, it } from "vitest"; +import { Effect, Exit } from "effect"; +import { it } from "@effect/vitest"; +import { describe, expect } from "vitest"; +import { mockOutput } from "../../../../../tests/helpers/mocks.ts"; +import type { LegacyPgDeltaNextDiagnostic } from "./legacy-pgdelta-next-adapter.service.ts"; import { - legacyPgDeltaNextBlockingDiagnostic, - legacyPgDeltaNextBlockingDiagnosticMessage, + legacyPgDeltaNextDiagnosticMessage, + legacyPgDeltaNextDiagnosticReport, + legacyPgDeltaNextFeedbackInvitation, + legacyReportPgDeltaNextDiagnostics, } from "./legacy-pgdelta-next-diagnostics.ts"; +const unmodeled = ( + kind: unknown, + overrides: Partial = {}, +): LegacyPgDeltaNextDiagnostic => ({ + origin: "desired", + code: "unmodeled_kind", + severity: "warning", + subject: "object:public.unsupported", + message: "object kind is not modeled", + context: { kind }, + ...overrides, +}); + describe("pg-delta next diagnostic coverage policy", () => { - it("blocks errors and strict coverage gaps while allowing ordinary warnings", () => { - expect( - legacyPgDeltaNextBlockingDiagnostic([ - { - origin: "source", - code: "unsupported_extension", - severity: "warning", - message: "extension is managed externally", - }, - ]), - ).toBeUndefined(); + it("allows coverage gaps by default after rendering diagnostics and one feedback invitation", () => { + const out = mockOutput(); + return Effect.gen(function* () { + yield* legacyReportPgDeltaNextDiagnostics( + "diff", + [unmodeled("text search configuration"), unmodeled("statistics object")], + false, + ); - expect( - legacyPgDeltaNextBlockingDiagnostic([ - { - origin: "desired", - code: "unmodeled_kind", - severity: "warning", - subject: "object:public.unsupported", - message: "object kind is not modeled", - }, - ]), - ).toMatchObject({ code: "unmodeled_kind" }); + expect(out.messages.filter(({ type }) => type === "warn")).toHaveLength(3); + expect(out.messages).toContainEqual({ + type: "warn", + message: + "pg-delta found schema objects it does not manage. Changes to these objects are omitted from the generated database diff.", + }); + const invitations = out.messages.filter(({ message }) => + message.startsWith("Request pg-delta support:"), + ); + expect(invitations).toHaveLength(1); + expect(invitations[0]?.message).toContain("statistics object, text search configuration"); + }).pipe(Effect.provide(out.layer)); + }); - expect( - legacyPgDeltaNextBlockingDiagnostic([ + it("renders coverage diagnostics and then fails in strict mode", () => { + const out = mockOutput(); + return Effect.gen(function* () { + const exit = yield* legacyReportPgDeltaNextDiagnostics( + "declarativePlan", + [unmodeled("text search configuration")], + true, + ).pipe(Effect.exit); + + expect(Exit.isFailure(exit)).toBe(true); + expect(out.messages).toContainEqual({ + type: "warn", + message: + "pg-delta found schema objects it does not manage. Strict coverage is enabled, so the operation will stop.", + }); + expect(out.messages.some(({ message }) => message.includes("supabase issue feature"))).toBe( + true, + ); + }).pipe(Effect.provide(out.layer)); + }); + + it("can suppress a repeated feedback invitation without suppressing warnings", () => { + const out = mockOutput(); + return Effect.gen(function* () { + yield* legacyReportPgDeltaNextDiagnostics( + "declarativePlan", + [unmodeled("text search configuration")], + false, + false, + ); + + expect(out.messages.some(({ message }) => message.includes("supabase issue feature"))).toBe( + false, + ); + expect(out.messages.some(({ type }) => type === "warn")).toBe(true); + }).pipe(Effect.provide(out.layer)); + }); + + it("always renders and fails error diagnostics", () => { + const out = mockOutput(); + return Effect.gen(function* () { + const exit = yield* legacyReportPgDeltaNextDiagnostics( + "declarativeExport", + [ + { + origin: "export", + code: "extraction_failed", + severity: "error", + message: "catalog query failed", + }, + ], + false, + ).pipe(Effect.exit); + + expect(Exit.isFailure(exit)).toBe(true); + expect(out.messages).toContainEqual({ + type: "error", + message: + "pg-delta next diagnostic: origin=export code=extraction_failed message=catalog query failed", + }); + }).pipe(Effect.provide(out.layer)); + }); + + it("classifies both coverage codes and aggregates arbitrary kinds safely", () => { + const report = legacyPgDeltaNextDiagnosticReport( + [ + unmodeled("z future kind"), + unmodeled("a future kind"), + unmodeled("a future kind"), + unmodeled("line\nbreak"), + unmodeled(undefined), + unmodeled(" "), { - origin: "export", - code: "extraction_failed", - severity: "error", - message: "catalog query failed", + origin: "snapshot", + code: "unresolved_security_label", + severity: "info", + message: "provider was not resolved", + context: { kind: 42 }, }, - ]), - ).toMatchObject({ code: "extraction_failed" }); + ], + true, + ); + + expect(report.coverage).toHaveLength(7); + expect(report.blocking).toHaveLength(7); + expect(report.unmodeledKinds).toEqual(["a future kind", "line break", "z future kind"]); }); - it("renders the refused action and complete diagnostic identity", () => { + it("omits an unknown subject and keeps feedback free of diagnostic details", () => { expect( - legacyPgDeltaNextBlockingDiagnosticMessage("declarativePlan", { - origin: "declarativeLoad", - code: "unresolved_security_label", - severity: "info", - subject: "table:public.accounts", - message: "security label provider was not resolved", + legacyPgDeltaNextDiagnosticMessage({ + origin: "source", + code: "unmodeled_kind", + severity: "warning", + subject: "unknown", + message: "private diagnostic message", + context: { kind: "operator class" }, }), - ).toBe( - "pg-delta next refused to emit the declarative migration plan: origin=declarativeLoad code=unresolved_security_label subject=table:public.accounts message=security label provider was not resolved", + ).not.toContain("subject="); + + const invitation = legacyPgDeltaNextFeedbackInvitation(["operator class"]); + expect(invitation).toContain("operator class"); + expect(invitation).not.toContain("private diagnostic message"); + expect(invitation).not.toContain("subject"); + expect(invitation).not.toContain("public."); + }); + + it("shell-quotes future kind names without making feedback kind-specific", () => { + expect(legacyPgDeltaNextFeedbackInvitation(["user's future kind"])).toContain( + `user'"'"'s future kind`, ); }); }); From dcdf7e0adda5fc4caab2cbb732b467566f836c2d Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 16:42:55 +0200 Subject: [PATCH 11/82] fix(cli): format pg-delta SQL by default --- apps/cli-go/docs/supabase/db/diff.md | 2 +- apps/cli-go/docs/supabase/db/pull.md | 2 +- .../db/schema-declarative-generate.md | 2 + .../supabase/db/schema-declarative-sync.md | 2 + apps/cli-go/pkg/config/templates/config.toml | 1 + .../legacy/commands/db/diff/SIDE_EFFECTS.md | 4 ++ .../legacy/commands/db/pull/SIDE_EFFECTS.md | 5 ++ .../declarative/generate/SIDE_EFFECTS.md | 5 ++ .../schema/declarative/sync/SIDE_EFFECTS.md | 4 ++ .../legacy-pgdelta-engine.next.layer.ts | 3 + .../legacy-pgdelta-next-adapter.layer.ts | 45 +++++++++++--- .../legacy-pgdelta-next-adapter.service.ts | 2 + .../legacy-pgdelta-next-adapter.unit.test.ts | 61 ++++++++++++++++--- .../src/shared/init/project-init.templates.ts | 1 + 14 files changed, 120 insertions(+), 19 deletions(-) diff --git a/apps/cli-go/docs/supabase/db/diff.md b/apps/cli-go/docs/supabase/db/diff.md index 0307f41ade..dae49bad5a 100644 --- a/apps/cli-go/docs/supabase/db/diff.md +++ b/apps/cli-go/docs/supabase/db/diff.md @@ -14,7 +14,7 @@ Projects created by a recent `supabase init` default to the pg-delta diff engine The pg-delta engine runs in-process by default and is bundled into the CLI together with pg-topo at build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy edge-runtime implementation. `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs under `supabase/.temp/pgdelta/` affect only that opt-out; there is no automatic fallback. -With the pg-delta engine the diff SQL is formatted by default with the same settings the declarative export uses (uppercase keywords, wrapped at a max width of 180, indented and column-aligned); execution-aware transaction boundaries are preserved as per-unit header comments in the output. Configure overrides with `[experimental.pgdelta] format_options`, or set `format_options = "null"` to emit raw, unformatted statements. +With the pg-delta engine the diff SQL is compacted and formatted by default with pg-delta's human-facing settings (lowercase keywords, wrapped at a max width of 180, indented and column-aligned); execution-aware transaction boundaries are preserved as per-unit header comments in the output. Configure partial overrides with `[experimental.pgdelta] format_options`, or set `format_options = "null"` to emit raw, unformatted statements. Compaction remains enabled in raw mode because it is a separate, semantics-preserving planning step. The bundled and legacy renderers can produce different SQL bytes or file segmentation. The compatibility contract is executable SQL and convergence: after applying the result, a subsequent diff should be empty. With `PGDELTA_DEBUG=1`, bundled-engine snapshots, plans, and diagnostics are stored under `supabase/.temp/pgdelta/v2/debug//`; those files are diagnostic artifacts, not reusable catalogs. diff --git a/apps/cli-go/docs/supabase/db/pull.md b/apps/cli-go/docs/supabase/db/pull.md index 5128f06133..0351a6e519 100644 --- a/apps/cli-go/docs/supabase/db/pull.md +++ b/apps/cli-go/docs/supabase/db/pull.md @@ -18,7 +18,7 @@ Pg-delta runs in-process by default and is bundled with pg-topo at CLI build tim pg-delta plans are execution-aware: when a plan crosses a transaction boundary — for example `ALTER TYPE ... ADD VALUE` followed by a statement that uses the new enum value, which cannot run in the same transaction — `db pull` writes one ordered migration file per plan unit instead of a single file (for example `_remote_schema_schema_changes.sql` and `_remote_schema_after_enum_values.sql`), each recorded in the migration history. The common case (a single unit) still produces exactly one `_remote_schema.sql` file. -By default the emitted SQL is formatted with the same settings the declarative export uses (uppercase keywords, wrapped at a max width of 180, indented and column-aligned). Configure overrides with `[experimental.pgdelta] format_options` in `config.toml`, or set `format_options = "null"` to opt out and emit raw, unformatted statements. +By default the emitted SQL is compacted and formatted with pg-delta's human-facing settings (lowercase keywords, wrapped at a max width of 180, indented and column-aligned). Configure partial overrides with `[experimental.pgdelta] format_options` in `config.toml`, or set `format_options = "null"` to opt out and emit raw, unformatted statements. Compaction remains enabled in raw mode because it is a separate, semantics-preserving planning step. When `[experimental.pgdelta] enabled = true` (the default for projects created by a recent `supabase init`), the migration-file `db pull` workflow uses pg-delta for the shadow diff step by default; it does not switch to declarative output. Existing projects without the section are unaffected and keep using migra. To fall back to the legacy migra engine, set `enabled = false` under `[experimental.pgdelta]`, or pass `--diff-engine migra` for a single run. diff --git a/apps/cli-go/docs/supabase/db/schema-declarative-generate.md b/apps/cli-go/docs/supabase/db/schema-declarative-generate.md index 4d82a4b8cf..1f1a628c44 100644 --- a/apps/cli-go/docs/supabase/db/schema-declarative-generate.md +++ b/apps/cli-go/docs/supabase/db/schema-declarative-generate.md @@ -8,6 +8,8 @@ The generated directory becomes the complete desired state: objects omitted from Pg-delta and pg-topo run in-process and are bundled into the CLI at build time. The export includes `.pgdelta-export.json` policy metadata. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy catalog/edge-runtime implementation; `PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs at the `.temp/pgdelta/` root are legacy-only. +Generated SQL is compacted and formatted by default with pg-delta's human-facing settings (lowercase keywords, a maximum width of 180, indentation, and column alignment). Declarative export safely folds additional constraints that remain separate in executable diff plans. Configure partial formatting overrides with `[experimental.pgdelta] format_options`, or set `format_options = "null"` to emit raw SQL while retaining semantic compaction. + `--no-cache` bypasses legacy catalog reuse/warming. The bundled engine always extracts live state and has no reusable catalog cache. With `PGDELTA_DEBUG=1`, structured diagnostics are written under `.temp/pgdelta/v2/debug//`. SQL bytes and grouping may differ between engines; reloading the export to the same managed state is the contract. Requires `--experimental` flag or `[experimental.pgdelta] enabled = true` in config. diff --git a/apps/cli-go/docs/supabase/db/schema-declarative-sync.md b/apps/cli-go/docs/supabase/db/schema-declarative-sync.md index 867c36076a..8a7318beaf 100644 --- a/apps/cli-go/docs/supabase/db/schema-declarative-sync.md +++ b/apps/cli-go/docs/supabase/db/schema-declarative-sync.md @@ -8,6 +8,8 @@ The declarative directory is a complete, hand-authored desired state. Missing ob Pg-delta and pg-topo run in-process and are bundled into the CLI at build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy catalog/edge-runtime implementation; `PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs at the `.temp/pgdelta/` root are legacy-only. +Generated migrations are compacted and formatted by default with pg-delta's human-facing settings (lowercase keywords, a maximum width of 180, indentation, and column alignment). Configure partial formatting overrides with `[experimental.pgdelta] format_options`, or set `format_options = "null"` to emit raw SQL while retaining semantic compaction. Dependency-sensitive constraints such as foreign keys may remain separate when folding them would be unsafe for migration execution. + `--no-cache` bypasses legacy catalog reuse/warming; the bundled engine extracts current state and has no reusable catalog cache. It may emit multiple ordered migration files to preserve transaction boundaries. SQL bytes may differ from the legacy renderer; successful application followed by an empty sync is the contract. With `PGDELTA_DEBUG=1`, snapshots, the plan, and diagnostics are written under `.temp/pgdelta/v2/debug//`. Requires `--experimental` flag or `[experimental.pgdelta] enabled = true` in config. diff --git a/apps/cli-go/pkg/config/templates/config.toml b/apps/cli-go/pkg/config/templates/config.toml index 98e034f8d8..ec327bc63e 100644 --- a/apps/cli-go/pkg/config/templates/config.toml +++ b/apps/cli-go/pkg/config/templates/config.toml @@ -412,3 +412,4 @@ enabled = {{ .Experimental.PgDeltaInitEnabled }} # declarative_schema_path = "./database" # JSON string passed through to pg-delta SQL formatting. # format_options = "{\"keywordCase\":\"upper\",\"indent\":2,\"maxWidth\":80,\"commaStyle\":\"trailing\"}" +# Set to "null" to disable formatting while retaining plan compaction. diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 4728ef0fcc..b5c37cb235 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -26,6 +26,10 @@ bundled Go binary. when debug capture is enabled, the bundle is saved before policy evaluation. - SQL text and file segmentation may differ from the legacy renderer. Applicable output and convergence (a subsequent diff is empty) are the compatibility contract. +- Default-engine plans retain pg-delta's safe compaction and are formatted with + its human-facing preset (lowercase keywords, max width 180). A JSON object in + `[experimental.pgdelta].format_options` partially overrides that preset; the + JSON literal `null` disables formatting without disabling compaction. ## Files Read diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 0a1c342ef0..d971973ea8 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -45,6 +45,11 @@ Notes/Delegation section below). artifacts are saved before policy evaluation when capture is enabled. - New-engine SQL bytes and transaction-split filenames may differ. Successful execution and convergence on a subsequent pull/diff are the contract. +- Default-engine migration and declarative SQL retains pg-delta's safe compaction + and uses its human-facing formatter (lowercase keywords, max width 180). A JSON + object in `[experimental.pgdelta].format_options` partially overrides the + preset; the JSON literal `null` disables formatting without disabling + compaction. ## Files Read diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index f169698a21..ef442e60e0 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -24,6 +24,11 @@ platform view. when capture is enabled. - Generated SQL bytes and grouping may differ between engines. Reloading the export to the same managed state is the compatibility contract. +- The default engine applies pg-delta's human-facing formatter (lowercase + keywords, max width 180) and export-specific safe constraint folding. A JSON + object in `[experimental.pgdelta].format_options` partially overrides the + formatter; the JSON literal `null` disables formatting without disabling plan + compaction. ## Files Read diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 34c8487e17..03d738e74f 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -22,6 +22,10 @@ as a new timestamped migration. - Default-engine migrations may differ byte-for-byte and may be split into ordered files to preserve transaction boundaries. Successful execution and an empty subsequent sync are the compatibility contract. +- Default-engine migrations use pg-delta's human-facing formatter (lowercase + keywords, max width 180) after safe plan compaction. A JSON object in + `[experimental.pgdelta].format_options` partially overrides the formatter; + the JSON literal `null` disables formatting without disabling compaction. ## Files Read diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 9b7d31fd6e..91b41bd2d0 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -224,6 +224,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( allowDrops: true, debug: input.debug, schema: input.schema, + formatOptions: input.formatOptions, }); const debugDirectory = result.debug !== undefined @@ -263,6 +264,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( allowDrops: true, debug: input.debug, schema: input.schema, + formatOptions: input.formatOptions, }); const debugDirectory = result.debug !== undefined @@ -331,6 +333,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( reorder: true, ...legacyPgDeltaNextIsolatedShadowPlanOptions, schema: input.schema, + formatOptions: input.formatOptions, ...(input.manifest !== undefined ? { manifest: input.manifest } : {}), }); const debugDirectory = diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index 15ebab0d9a..ff1bd0f861 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -15,7 +15,7 @@ import { import { plan, serializePlan } from "@supabase/pg-delta/plan"; import type { Plan as PgDeltaPlan } from "@supabase/pg-delta/plan"; import type { Policy } from "@supabase/pg-delta/policy"; -import type { SqlFormatOptions } from "@supabase/pg-delta/sql-format"; +import { formatSqlStatements, type SqlFormatOptions } from "@supabase/pg-delta/sql-format"; import { LegacyPgDeltaNextAdapter, @@ -338,10 +338,18 @@ export function legacyPgDeltaNextProfile( return { ...supabaseProfile, policy }; } +const legacyPgDeltaNextHumanFormatOptions: SqlFormatOptions = { + keywordCase: "lower", + maxWidth: 180, +}; + function legacyPgDeltaNextFormatOptions(raw: string | undefined): SqlFormatOptions | undefined { - if (raw === undefined || raw.trim().length === 0) return undefined; + if (raw === undefined || raw.trim().length === 0) return legacyPgDeltaNextHumanFormatOptions; const parsed: unknown = JSON.parse(raw); - if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) return undefined; + if (parsed === null) return undefined; + if (typeof parsed !== "object" || Array.isArray(parsed)) { + return legacyPgDeltaNextHumanFormatOptions; + } const value = (key: string): unknown => Reflect.get(parsed, key); const keywordCase = value("keywordCase"); const commaStyle = value("commaStyle"); @@ -353,6 +361,7 @@ function legacyPgDeltaNextFormatOptions(raw: string | undefined): SqlFormatOptio const preserveViewBodies = value("preserveViewBodies"); const preserveRuleBodies = value("preserveRuleBodies"); return { + ...legacyPgDeltaNextHumanFormatOptions, ...(keywordCase === "upper" || keywordCase === "lower" || keywordCase === "preserve" ? { keywordCase } : {}), @@ -367,6 +376,24 @@ function legacyPgDeltaNextFormatOptions(raw: string | undefined): SqlFormatOptio }; } +function legacyTerminatePgDeltaNextStatement(sql: string): string { + const trimmed = sql.trimEnd(); + return trimmed.endsWith(";") ? trimmed : `${trimmed};`; +} + +function legacyFormatPgDeltaNextRenderedFiles( + files: readonly LegacyPgDeltaNextLibraryRenderedFile[], + format: SqlFormatOptions | undefined, +): readonly LegacyPgDeltaNextLibraryRenderedFile[] { + if (format === undefined) return files; + return files.map((file) => ({ + ...file, + contents: `${formatSqlStatements([file.contents], format) + .map(legacyTerminatePgDeltaNextStatement) + .join("\n\n")}\n`, + })); +} + function legacyPgDeltaNextExportOptions(input: LegacyPgDeltaNextDeclarativeExportInput) { const format = legacyPgDeltaNextFormatOptions(input.formatOptions); return { @@ -440,6 +467,7 @@ function legacyMakePgDeltaNextAdapter legacyTryPgDeltaNext("diff", async () => { + const format = legacyPgDeltaNextFormatOptions(input.formatOptions); const redactSecrets = input.redactSecrets ?? true; const profile = await libraries.resolveProfile( input.sourcePool, @@ -462,6 +490,7 @@ function legacyMakePgDeltaNextAdapter file.contents).join("\n\n"), - files: legacyNormalizePgDeltaNextRenderedFiles(rendered.files), + sql: renderedFiles.map((file) => file.contents).join("\n\n"), + files: legacyNormalizePgDeltaNextRenderedFiles(renderedFiles), diagnostics, ...(input.debug ? { @@ -519,6 +548,7 @@ function legacyMakePgDeltaNextAdapter legacyTryPgDeltaNext("declarativePlan", async () => { + const format = legacyPgDeltaNextFormatOptions(input.formatOptions); const planningInput = { ...input, reorder: input.reorder ?? true }; const result = await libraries.planSchemaFiles( input.targetPool, @@ -529,10 +559,11 @@ function legacyMakePgDeltaNextAdapter file.contents).join("\n\n"), - files: legacyNormalizePgDeltaNextRenderedFiles(rendered.files), + sql: renderedFiles.map((file) => file.contents).join("\n\n"), + files: legacyNormalizePgDeltaNextRenderedFiles(renderedFiles), diagnostics: [ ...legacyNormalizePgDeltaNextDiagnostics( result.loadDiagnostics, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts index 26526f8310..aba46a3f66 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts @@ -58,6 +58,7 @@ export interface LegacyPgDeltaNextDiffInput { readonly redactSecrets?: boolean; readonly restrictToApplier?: boolean; readonly schema?: readonly string[]; + readonly formatOptions?: string; } interface LegacyPgDeltaNextDiffResult { @@ -134,6 +135,7 @@ export interface LegacyPgDeltaNextDeclarativePlanInput { readonly seedAssumedSchemas?: boolean; readonly restrictToApplier?: boolean; readonly strictFunctionBodies?: boolean; + readonly formatOptions?: string; /** Defaults to true, preserving pg-topo statement-level reorder support. */ readonly reorder?: boolean; readonly onWarning?: (message: string) => void; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index 31af78f231..98fdaedd9e 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -62,7 +62,7 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { desired: FakeFactBase; options: FakePlanOptions & { redactSecrets: boolean }; }>, - renderAllowDrops: [] as boolean[], + renderOptions: [] as Array<{ allowDrops: boolean }>, exportInputs: [] as object[], declarativeInputs: [] as object[], snapshotMetadata: [] as object[], @@ -106,21 +106,22 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { state.planCalls.push({ source, desired, options }); return { source: source.id, desired: desired.id }; }, - renderPlanFiles: (generatedPlan, options) => { - state.renderAllowDrops.push(options.allowDrops); + renderPlanFiles: (_generatedPlan, options) => { + state.renderOptions.push(options); if (!state.renderChanges) return { changes: false, files: [] }; return { changes: true, files: [ { suffix: "_1", - contents: `begin ${generatedPlan.source};\n`, + contents: "CREATE TABLE public.widgets (id integer, display_name text);\n", transactional: true, actionCount: 2, }, { suffix: "_2", - contents: `alter ${generatedPlan.desired};\n`, + contents: + "-- pg-delta: transaction=false\nSET check_function_bodies = off;\n\nGRANT SELECT ON TABLE public.widgets TO anon;\n\nRESET ALL;\n", transactional: false, actionCount: 1, }, @@ -335,6 +336,7 @@ describe("LegacyPgDeltaNextAdapter", () => { redactSecrets: false, restrictToApplier: true, schema: ["public"], + formatOptions: '{"keywordCase":"upper","indent":4}', }); expect(state.resolveCalls).toEqual([ @@ -355,23 +357,26 @@ describe("LegacyPgDeltaNextAdapter", () => { options: { redactSecrets: false, managedView: "shared-profile-options" }, }, ]); + expect(state.renderOptions).toEqual([{ allowDrops: true }]); expect(result.files).toEqual([ { sequence: 1, suffix: "_1", - sql: "begin source-facts;\n", + sql: "CREATE TABLE public.widgets (\n id integer,\n display_name text\n);\n", transactionMode: "transactional", actionCount: 2, }, { sequence: 2, suffix: "_2", - sql: "alter desired-facts;\n", + sql: "-- pg-delta: transaction=false\nSET check_function_bodies = off;\n\nGRANT SELECT ON TABLE public.widgets TO anon;\n\nRESET ALL;\n", transactionMode: "none", actionCount: 1, }, ]); - expect(result.sql).toBe("begin source-facts;\n\n\nalter desired-facts;\n"); + expect(result.sql).toBe( + "CREATE TABLE public.widgets (\n id integer,\n display_name text\n);\n\n\n-- pg-delta: transaction=false\nSET check_function_bodies = off;\n\nGRANT SELECT ON TABLE public.widgets TO anon;\n\nRESET ALL;\n", + ); expect(result.diagnostics).toEqual([ { origin: "source", @@ -427,7 +432,30 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(result.files).toEqual([]); expect(result.debug).toBeUndefined(); expect(state.snapshotMetadata).toEqual([]); - expect(state.renderAllowDrops).toEqual([false]); + expect(state.renderOptions).toEqual([{ allowDrops: false }]); + yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); + }).pipe(Effect.provide(layer)); + }); + + it.effect("formats rendered migration files with the human-readable defaults", () => { + const sourcePool = new Pool(); + const desiredPool = new Pool(); + const { layer } = setupLibraries(sourcePool, desiredPool); + + return Effect.gen(function* () { + const adapter = yield* LegacyPgDeltaNextAdapter; + const result = yield* adapter.diff({ + sourcePool, + desiredPool, + allowDrops: false, + debug: false, + }); + expect(result.files[0]?.sql).toBe( + "create table public.widgets (\n id integer,\n display_name text\n);\n", + ); + expect(result.files[1]?.sql).toBe( + "-- pg-delta: transaction=false\nset check_function_bodies = off;\n\ngrant select on table public.widgets to anon;\n\nreset all;\n", + ); yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); }).pipe(Effect.provide(layer)); }); @@ -480,6 +508,14 @@ describe("LegacyPgDeltaNextAdapter", () => { }); expect(state.exportInputs[0]).not.toHaveProperty("formatOptions"); + yield* adapter.exportDeclarativeSchema({ + pool: targetPool, + layout: "grouped", + }); + expect(state.exportInputs[1]).toMatchObject({ + format: { keywordCase: "lower", maxWidth: 180 }, + }); + const planned = yield* adapter.planDeclarativeSchema({ targetPool, shadowPool, @@ -488,6 +524,7 @@ describe("LegacyPgDeltaNextAdapter", () => { debug: true, isolatedShadow: true, seedAssumedSchemas: true, + formatOptions: "null", }); expect(state.declarativeInputs).toHaveLength(1); expect(state.declarativeInputs[0]).toMatchObject({ @@ -508,7 +545,11 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(planned.debug).toEqual({ plan: JSON.stringify({ source: "target-facts", desired: "loaded-files" }), }); - expect(state.renderAllowDrops).toEqual([true]); + expect(planned.files.map((file) => file.sql)).toEqual([ + "CREATE TABLE public.widgets (id integer, display_name text);\n", + "-- pg-delta: transaction=false\nSET check_function_bodies = off;\n\nGRANT SELECT ON TABLE public.widgets TO anon;\n\nRESET ALL;\n", + ]); + expect(state.renderOptions).toEqual([{ allowDrops: true }]); yield* Effect.promise(() => Promise.all([targetPool.end(), shadowPool.end()])); }).pipe(Effect.provide(layer)); }, diff --git a/apps/cli/src/shared/init/project-init.templates.ts b/apps/cli/src/shared/init/project-init.templates.ts index 2c6a823579..7acc2f30a0 100644 --- a/apps/cli/src/shared/init/project-init.templates.ts +++ b/apps/cli/src/shared/init/project-init.templates.ts @@ -412,6 +412,7 @@ enabled = true # declarative_schema_path = "./database" # JSON string passed through to pg-delta SQL formatting. # format_options = "{\\"keywordCase\\":\\"upper\\",\\"indent\\":2,\\"maxWidth\\":80,\\"commaStyle\\":\\"trailing\\"}" +# Set to "null" to disable formatting while retaining plan compaction. `; export const INIT_GITIGNORE_TEMPLATE = `# Supabase From 16028da0cde7fc8373306466437763ed894e8be4 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 16:43:56 +0200 Subject: [PATCH 12/82] fix(cli): clarify declarative diff baselines --- apps/cli-go/docs/supabase/db/diff.md | 12 ++- apps/cli-go/docs/supabase/db/pull.md | 4 +- .../db/schema-declarative-generate.md | 2 + .../supabase/db/schema-declarative-sync.md | 2 + .../legacy/commands/db/diff/SIDE_EFFECTS.md | 8 +- .../legacy/commands/db/diff/diff.command.ts | 8 +- .../legacy/commands/db/diff/diff.handler.ts | 49 +++++++++- .../commands/db/diff/diff.integration.test.ts | 98 ++++++++++++++++++- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 3 +- .../legacy/commands/db/pull/pull.command.ts | 6 +- .../legacy/commands/db/pull/pull.handler.ts | 5 +- .../commands/db/pull/pull.integration.test.ts | 2 +- .../declarative/generate/generate.command.ts | 4 +- .../schema/declarative/sync/sync.command.ts | 4 +- .../src/legacy/shared/legacy-diff-engine.ts | 2 +- 15 files changed, 190 insertions(+), 19 deletions(-) diff --git a/apps/cli-go/docs/supabase/db/diff.md b/apps/cli-go/docs/supabase/db/diff.md index dae49bad5a..419d2e6b06 100644 --- a/apps/cli-go/docs/supabase/db/diff.md +++ b/apps/cli-go/docs/supabase/db/diff.md @@ -6,7 +6,17 @@ Requires the local development stack to be running when diffing against the loca Runs [djrobstep/migra](https://github.com/djrobstep/migra) in a container to compare schema differences between the target database and a shadow database. The shadow database is created by applying migrations in local `supabase/migrations` directory in a separate container. Output is written to stdout by default. For convenience, you can also save the schema diff as a new migration file by passing in `-f` flag. -Normal diff mode always compares that migrations shadow with the selected live database. Declarative files under `supabase/database/` and `[db.migrations].schema_paths` do not replace the target. Use `supabase db schema declarative sync` to compare the complete declarative desired state. +`-f dogfood_note` names the generated migration; it does not filter the diff to the `dogfood_note` object. + +| Command | Baseline/source | Compared with/destination | Writes | +| -------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------- | +| `db diff` | `supabase/migrations/` | Live database (`--local` default, `--linked`, or `--db-url`) | stdout, or migration file(s) with `-f` | +| `db pull` | `supabase/migrations/` plus selected database history | Live database (`--linked` default) | migration file(s), then optionally selected database history | +| `db pull --declarative` | Selected live database | `supabase/database/` | replaces the declarative tree; no migration or history update | +| `db schema declarative generate` | Selected live database | `supabase/database/` | replaces declarative files only | +| `db schema declarative sync` | `supabase/migrations/` | `supabase/database/` | migration file(s), optionally applied to the local database | + +Normal diff mode always compares the migrations shadow with the selected live database. Declarative files under `supabase/database/` and `[db.migrations].schema_paths` do not replace the migrations baseline. If migrations are empty or outdated, a saved diff can therefore include objects already represented by declarative files. Use `supabase db schema declarative sync --no-apply` to generate and review a migration from the declarative desired state before making later live changes. By default, all schemas in the target database are diffed. Use the `--schema public,extensions` flag to restrict diffing to a subset of schemas. diff --git a/apps/cli-go/docs/supabase/db/pull.md b/apps/cli-go/docs/supabase/db/pull.md index 0351a6e519..a55b30d545 100644 --- a/apps/cli-go/docs/supabase/db/pull.md +++ b/apps/cli-go/docs/supabase/db/pull.md @@ -10,9 +10,9 @@ Optionally, a new row can be inserted into the migration history table to reflec If no entries exist in the migration history table, the default diff engine uses `pg_dump` to capture all contents of the remote schemas you have created. Otherwise, this command will only diff schema changes against the remote database, similar to running `db diff --linked`. -Pass `--diff-engine pg-delta` to keep the migration-file `db pull` workflow while using pg-delta for the shadow diff step. On initial pull, pg-delta replaces `pg_dump` and produces the full migration from the shadow diff alone. Pass `--declarative` to switch to the declarative pg-delta export workflow instead. +Pass `--diff-engine pg-delta` to keep the migration-file `db pull` workflow while using pg-delta for the shadow diff step. On initial pull, pg-delta replaces `pg_dump` and produces the full migration from the shadow diff alone. Pass `--declarative` to switch to the declarative pg-delta export workflow instead; that mode replaces the declarative tree and does not create migrations or update migration history. -Migration-style pull always compares the local migrations shadow with the selected live database. Declarative files and `[db.migrations].schema_paths` do not replace that target; use `db schema declarative sync` for declarative comparison. +Migration-style pull always compares the local migrations shadow with the selected live database. Declarative files and `[db.migrations].schema_paths` do not replace that migrations baseline; use `db schema declarative sync` for declarative comparison. In non-interactive use, the prompt to record newly pulled migrations in the selected database history takes its default of yes. Pg-delta runs in-process by default and is bundled with pg-topo at CLI build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily use the legacy edge-runtime implementation. `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs directly under `supabase/.temp/pgdelta/` affect only that opt-out; the CLI never falls back automatically. diff --git a/apps/cli-go/docs/supabase/db/schema-declarative-generate.md b/apps/cli-go/docs/supabase/db/schema-declarative-generate.md index 1f1a628c44..8a6001f8ab 100644 --- a/apps/cli-go/docs/supabase/db/schema-declarative-generate.md +++ b/apps/cli-go/docs/supabase/db/schema-declarative-generate.md @@ -4,6 +4,8 @@ Generate declarative schema files from a database. Exports the schema of a live database (local, linked, or custom URL) into SQL files under the declarative schema directory. This is the entrypoint for bootstrapping declarative mode. +Generate replaces the declarative tree only. It does not create migration files or update migration history, so it does not establish the baseline used by `db diff` or migration-style `db pull`. In non-interactive use, pass `--local`, `--linked`, or `--db-url` explicitly. To materialize declarations as a reviewed migration baseline, run `supabase db schema declarative sync --no-apply` before making later live changes. + The generated directory becomes the complete desired state: objects omitted from it are intended removals, including extensions, with or without an export manifest. When upgrading from the legacy workflow, regenerate the directory or add declarations for every extension you intend to retain before syncing, then review destructive-change warnings before applying. Pg-delta and pg-topo run in-process and are bundled into the CLI at build time. The export includes `.pgdelta-export.json` policy metadata. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy catalog/edge-runtime implementation; `PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs at the `.temp/pgdelta/` root are legacy-only. diff --git a/apps/cli-go/docs/supabase/db/schema-declarative-sync.md b/apps/cli-go/docs/supabase/db/schema-declarative-sync.md index 8a7318beaf..0d89d8fff4 100644 --- a/apps/cli-go/docs/supabase/db/schema-declarative-sync.md +++ b/apps/cli-go/docs/supabase/db/schema-declarative-sync.md @@ -4,6 +4,8 @@ Generate a new migration by diffing your declarative schema files against the cu When no declarative schema exists yet, the command offers to run `generate` first. After computing the diff, you can optionally name the migration and apply it to the local database. +For non-interactive generation, pass `--no-apply` explicitly. A non-interactive invocation otherwise skips applying by default, but global `--yes` changes that decision and applies the generated migration to the local database and its migration history. + The declarative directory is a complete, hand-authored desired state. Missing objects are intended removals, including extensions, regardless of whether the files were generated or whether an export manifest exists. When upgrading from the legacy workflow, regenerate the directory or add declarations for extensions you intend to retain, and review destructive-change warnings before applying. Pg-delta and pg-topo run in-process and are bundled into the CLI at build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy catalog/edge-runtime implementation; `PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs at the `.temp/pgdelta/` root are legacy-only. diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index b5c37cb235..b1cf2c33c0 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -108,7 +108,11 @@ prints to stdout when neither `--file` nor explicit `--output` is set. ### `--output-format json` / `stream-json` Progress strings still go to stderr; stdout carries a single structured envelope -`{ diff, file, schemas, engine, dropStatements }` instead of the raw SQL. +`{ diff, file, files, schemas, engine, dropStatements, advisories? }` instead of +the raw SQL. With the default pg-delta implementation, a non-empty `--file` diff +and a non-empty declarative tree add the informational +`DeclarativeSchemaNotUsedAsDiffBaseline` advisory; the same note is written to +stderr. Inspection is best-effort and never changes command success. ## Notes / Delegation @@ -122,7 +126,7 @@ Progress strings still go to stderr; stdout carries a single structured envelope - `--strict-coverage` applies to the bundled pg-delta engine and refuses output when it encounters schema objects it cannot manage. - Normal mode always compares the migrations shadow with the selected live - database. Declarative files and `schema_paths` never replace that target; use + database. Declarative files and `schema_paths` never replace that migrations baseline; use `supabase db schema declarative sync` for declarative comparison. - Under the legacy opt-out, the explicit `migrations` target resolves natively (CLI-1959): a bare diff --git a/apps/cli/src/legacy/commands/db/diff/diff.command.ts b/apps/cli/src/legacy/commands/db/diff/diff.command.ts index 37c787b137..79f1aea427 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.command.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.command.ts @@ -74,7 +74,9 @@ const config = { ), file: Flag.string("file").pipe( Flag.withAlias("f"), - Flag.withDescription("Saves schema diff to a new migration file."), + Flag.withDescription( + "Names and saves the complete schema diff as a new migration; it does not filter objects.", + ), Flag.optional, ), schema: Flag.string("schema").pipe( @@ -94,7 +96,9 @@ const config = { export type LegacyDbDiffFlags = CliCommand.Command.Config.Infer; export const legacyDbDiffCommand = Command.make("diff", config).pipe( - Command.withDescription("Diffs the local database for schema changes."), + Command.withDescription( + "Compares a shadow built from supabase/migrations with a live database (--local by default, --linked, or --db-url). Declarative files under supabase/database are not part of this baseline. Output is printed by default; -f names and saves the complete diff as a migration and does not filter objects.", + ), Command.withShortDescription("Diffs the local database for schema changes"), Command.withHandler((flags) => legacyDbDiff(flags).pipe( diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index 0941a0a75a..3fc065c9f9 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -6,7 +6,10 @@ import { detectGitBranch } from "../../../../shared/git/git-branch.ts"; import { Output } from "../../../../shared/output/output.service.ts"; import { LegacyCliConfig } from "../../../config/legacy-cli-config.service.ts"; import { legacyAqua, legacyYellow } from "../../../shared/legacy-colors.ts"; -import { legacyReadDbToml } from "../../../shared/legacy-db-config.toml-read.ts"; +import { + legacyReadDbToml, + legacyResolveDeclarativeDir, +} from "../../../shared/legacy-db-config.toml-read.ts"; import { LegacyDbConfigResolver } from "../../../shared/legacy-db-config.service.ts"; import type { LegacyDbConnType } from "../../../shared/legacy-db-target-flags.ts"; import { legacyGetHostname } from "../../../shared/legacy-hostname.ts"; @@ -32,6 +35,7 @@ import { type LegacyPgDeltaDatabaseEndpoint, type LegacyPgDeltaEndpoint, } from "../shared/legacy-pgdelta-engine.service.ts"; +import { LegacyLoadPgDeltaSqlFiles } from "../shared/legacy-pgdelta-files.ts"; import { legacyWritePgDeltaMigrations } from "../shared/legacy-pgdelta-migrations.write.ts"; import { legacyIsPgDeltaDebugEnabled, @@ -63,6 +67,20 @@ Run ${legacyAqua("supabase db reset")} to verify that the new migration does not // scope for CLI-1960. const warnPgSchemaDeprecated = `${legacyYellow("WARNING:")} "--use-pg-schema" is deprecated. Use the pg-delta engine ([experimental.pgdelta] enabled = true / --use-pg-delta) or the default migra engine instead.`; +const declarativeBaselineAdvisory = (declarativePath: string | null) => ({ + code: "DeclarativeSchemaNotUsedAsDiffBaseline", + severity: "info", + message: "Declarative schema files were not used as the db diff baseline.", + context: { + baseline: "supabase/migrations", + declarativePath, + fileFlagFiltersObjects: false, + }, +}); + +const declarativeBaselineNote = (displayPath: string) => + `Note: db diff -f uses supabase/migrations as its baseline. Declarative schema files in ${displayPath} are not part of that baseline. If migrations are empty or outdated, the generated migration may include existing declarative objects. -f names the migration; it does not filter objects.\n`; + /** * Rebuilds the `db diff` argv for the pgAdmin / pg-schema delegate path. Flags * stay flags (the Go-proxy channel-parity rule). The explicit `--from`/`--to` and @@ -481,6 +499,32 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy const engine = useDelta ? "pg-delta" : "migra"; const drops = legacyFindDropStatements(out); const writtenFiles: Array = []; + let ignoredDeclarativeAdvisory: ReturnType | undefined; + if ( + out.length >= 2 && + useDelta && + pgDelta.implementation === "next" && + Option.isSome(flags.file) && + flags.file.value.length > 0 + ) { + // This is an informational, best-effort probe only. Declarative files are + // intentionally not inputs to normal db diff, so an unreadable or changing + // directory must never turn a previously successful diff into a failure. + const declarativeDir = legacyResolveDeclarativeDir(path, cfg.pgDelta); + const declarativeDirAbsolute = path.resolve(cliConfig.workdir, declarativeDir); + const hasDeclarativeSql = yield* Effect.gen(function* () { + if (!(yield* fs.exists(declarativeDirAbsolute))) return false; + return (yield* LegacyLoadPgDeltaSqlFiles(fs, path, declarativeDirAbsolute)).length > 0; + }).pipe(Effect.orElseSucceed(() => false)); + if (hasDeclarativeSql) { + const isAbsolute = path.isAbsolute(declarativeDir); + const displayPath = isAbsolute + ? "the configured declarative schema directory" + : declarativeDir.split("\\").join("/"); + ignoredDeclarativeAdvisory = declarativeBaselineAdvisory(isAbsolute ? null : displayPath); + yield* output.raw(declarativeBaselineNote(displayPath), "stderr"); + } + } if (out.length < 2) { yield* output.raw("No schema changes found\n", "stderr"); // Go's `SaveDiff` gates the file write on `len(file) > 0` (`pgadmin.go`), so @@ -550,6 +594,9 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy schemas: flags.schema, engine, dropStatements: drops, + ...(ignoredDeclarativeAdvisory === undefined + ? {} + : { advisories: [ignoredDeclarativeAdvisory] }), }); } }).pipe( diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 6c8c9aa07e..4058370433 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -350,7 +350,8 @@ describe("legacy db diff", () => { yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); expect(s.databaseDiffCalls[0]).not.toHaveProperty("declarativeFiles"); expect(s.databaseDiffCalls[0]).not.toHaveProperty("declarativeManifest"); - expect(stderr(s.out)).toContain("schema_paths no longer changes the target"); + expect(stderr(s.out)).toContain("schema_paths no longer changes the migrations baseline"); + expect(stderr(s.out)).not.toContain("db diff -f uses supabase/migrations"); expect(stdout(s.out)).toBe("create table result ();\n\n"); }).pipe(Effect.provide(s.layer)); }); @@ -589,7 +590,9 @@ describe("legacy db diff", () => { return Effect.gen(function* () { yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), file: Option.some("my_diff") })); expect(stdout(s.out)).toBe(""); - expect(stderr(s.out)).toContain("schema_paths no longer changes the target"); + expect(stderr(s.out)).toContain("schema_paths no longer changes the migrations baseline"); + expect(stderr(s.out)).toContain("db diff -f uses supabase/migrations as its baseline"); + expect(stderr(s.out)).toContain("-f names the migration; it does not filter objects"); expect(stderr(s.out)).toContain("WARNING: The diff tool is not foolproof"); const dir = join(tmp.current, "supabase", "migrations"); const files = readdirSync(dir); @@ -599,6 +602,97 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); + for (const format of ["json", "stream-json"] as const) { + it.effect(`includes the ignored declarative baseline advisory in ${format} output`, () => { + mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "database", "items.sql"), + "create table items ();\n", + ); + const s = setup(tmp.current, { + format, + pgDeltaImplementation: "next", + diffSql: "create table dogfood_note ();\n", + }); + return Effect.gen(function* () { + yield* legacyDbDiff( + flags({ usePgDelta: Option.some(true), file: Option.some("dogfood_note") }), + ); + const success = s.out.messages.find((message) => message.type === "success"); + expect(success?.data).toMatchObject({ + diff: "create table dogfood_note ();\n", + engine: "pg-delta", + advisories: [ + { + code: "DeclarativeSchemaNotUsedAsDiffBaseline", + severity: "info", + context: { + baseline: "supabase/migrations", + declarativePath: "supabase/database", + fileFlagFiltersObjects: false, + }, + }, + ], + }); + expect(stderr(s.out)).toContain("db diff -f uses supabase/migrations as its baseline"); + const written = readdirSync(join(tmp.current, "supabase", "migrations")); + expect(written).toHaveLength(1); + expect(readFileSync(join(tmp.current, "supabase", "migrations", written[0]!), "utf8")).toBe( + "create table dogfood_note ();\n", + ); + }).pipe(Effect.provide(s.layer)); + }); + } + + it.effect("does not emit the advisory for the legacy pg-delta implementation", () => { + mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "database", "items.sql"), + "create table items ();\n", + ); + const s = setup(tmp.current, { + format: "json", + pgDeltaImplementation: "legacy", + diffSql: "create table dogfood_note ();\n", + }); + return Effect.gen(function* () { + yield* legacyDbDiff( + flags({ usePgDelta: Option.some(true), file: Option.some("dogfood_note") }), + ); + const success = s.out.messages.find((message) => message.type === "success"); + expect(success?.data).not.toHaveProperty("advisories"); + expect(stderr(s.out)).not.toContain("db diff -f uses supabase/migrations"); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("ignores declarative inspection errors without changing diff success", () => { + mkdirSync(join(tmp.current, "supabase"), { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + [ + "[experimental.pgdelta]", + "enabled = true", + 'declarative_schema_path = "not-a-directory.sql"', + "", + ].join("\n"), + ); + writeFileSync(join(tmp.current, "supabase", "not-a-directory.sql"), "select 1;\n"); + const s = setup(tmp.current, { + format: "json", + pgDeltaImplementation: "next", + diffSql: "create table dogfood_note ();\n", + }); + return Effect.gen(function* () { + yield* legacyDbDiff( + flags({ usePgDelta: Option.some(true), file: Option.some("dogfood_note") }), + ); + const success = s.out.messages.find((message) => message.type === "success"); + expect(success?.data).not.toHaveProperty("advisories"); + expect(success?.data).toMatchObject({ diff: "create table dogfood_note ();\n" }); + expect(stderr(s.out)).not.toContain("db diff -f uses supabase/migrations"); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("writes one migration file per unit for a multi-unit pg-delta plan", () => { // A pg-delta plan that crosses a transaction boundary yields more than one // ordered unit; writing them into one migration would fail when db push/reset diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index d971973ea8..7130ced6d6 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -4,7 +4,8 @@ Native Effect port. Pulls the remote schema into either a new timestamped migration (diffing a throwaway shadow against the remote, native pg-delta or migra) or declarative files (`--declarative`, native pg-delta export). The migration-style path always compares migrations with the selected live database; -declarative files and `[db.migrations].schema_paths` cannot replace its target. +declarative files and `[db.migrations].schema_paths` cannot replace its migrations +baseline. initial-migra pull (no local migrations) seeds the migration file with a native `pg_dump` of the remote schema (a Docker `pg_dump` container, with IPv4 transaction-pooler fallback) and then appends the migra diff. `--experimental`'s diff --git a/apps/cli/src/legacy/commands/db/pull/pull.command.ts b/apps/cli/src/legacy/commands/db/pull/pull.command.ts index 64f653150b..0878af696a 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.command.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.command.ts @@ -18,7 +18,7 @@ const config = { // pflag `Changed`. declarative: Flag.boolean("declarative").pipe( Flag.withDescription( - "Pull schema as declarative files using pg-delta instead of creating a migration.", + "Replace the declarative schema tree from the selected database instead of creating a migration; migration history is not updated.", ), Flag.optional, ), @@ -72,7 +72,9 @@ const config = { export type LegacyDbPullFlags = CliCommand.Command.Config.Infer; export const legacyDbPullCommand = Command.make("pull", config).pipe( - Command.withDescription("Pull schema from the remote database."), + Command.withDescription( + "Migration mode compares supabase/migrations with the selected live database (--linked by default), writes the complete difference as migration files, and may record them in that database's migration history. --declarative instead replaces the declarative schema tree and does not create migrations or update migration history.", + ), Command.withShortDescription("Pull schema from the remote database"), Command.withHandler((flags) => legacyDbPull(flags).pipe( diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 10ae535599..c3891be574 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -441,8 +441,9 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // (declarative.go:260-268, gated on IsPgDeltaEnabled which reads the config // value). db pull --declarative does not force-enable pg-delta // (cmd/db.go:180-182), so unlike generate/sync this branch is reachable: - // without it, subsequent db reset/db diff keep reading supabase/migrations - // and ignore the files just pulled. + // it preserves the legacy experimental db-reset schema-files workflow. + // Normal db diff and migration-style db pull still use migrations as + // their baseline and ignore this setting. if (!toml.pgDelta.enabled) { yield* legacyUpdateDeclarativeSchemaPathsConfig( fs, diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 1fe7db5cba..80e1e6531f 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -646,7 +646,7 @@ describe("legacy db pull", () => { yield* legacyDbPull(flags()); expect(s.provisionCalls[0]?.mode).toBe("diff"); const err = streamText(s.out, "stderr"); - expect(err).toContain("schema_paths no longer changes the target"); + expect(err).toContain("schema_paths no longer changes the migrations baseline"); // Go's `ConnectByConfig` prints the Connecting line to stderr before dialing // (`internal/utils/connect.go:348`), ahead of any other pull output. expect(err).toContain("Connecting to remote database...\n"); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts index 42fbd0d43d..9455b637d6 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts @@ -64,7 +64,9 @@ export type LegacyDbSchemaDeclarativeGenerateFlags = CliCommand.Command.Config.I > & { readonly noCache: boolean; readonly strictCoverage: boolean }; export const legacyDbSchemaDeclarativeGenerateCommand = Command.make("generate", config).pipe( - Command.withDescription("Generate declarative schema from a database."), + Command.withDescription( + "Exports a live database into the complete declarative schema tree. This replaces declarative files only; it does not create migration files or update migration history. In non-interactive use, pass --local, --linked, or --db-url explicitly.", + ), Command.withShortDescription("Generate declarative schema from a database"), Command.withHandler((flags) => Effect.gen(function* () { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts index 162c42c05c..0501c9a6cc 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts @@ -56,7 +56,9 @@ export type LegacyDbSchemaDeclarativeSyncFlags = CliCommand.Command.Config.Infer }; export const legacyDbSchemaDeclarativeSyncCommand = Command.make("sync", config).pipe( - Command.withDescription("Generate a new migration from declarative schema."), + Command.withDescription( + "Compares the supabase/migrations baseline with the complete declarative schema tree and writes the difference as migration files. Use --no-apply for non-interactive generation without changing the local database; --apply or global --yes applies locally and updates local migration history.", + ), Command.withShortDescription("Generate a new migration from declarative schema"), Command.withHandler((flags) => Effect.gen(function* () { diff --git a/apps/cli/src/legacy/shared/legacy-diff-engine.ts b/apps/cli/src/legacy/shared/legacy-diff-engine.ts index ef1528387a..16c65e0190 100644 --- a/apps/cli/src/legacy/shared/legacy-diff-engine.ts +++ b/apps/cli/src/legacy/shared/legacy-diff-engine.ts @@ -4,7 +4,7 @@ // directly. export const legacySchemaPathsTransitionWarning = - "WARNING: [db.migrations].schema_paths no longer changes the target of db diff or migration-style db pull. These commands always compare local migrations with the selected database. Use `supabase db schema declarative sync` to compare declarative schema files.\n"; + "WARNING: [db.migrations].schema_paths no longer changes the migrations baseline used by db diff or migration-style db pull. These commands always compare local migrations with the selected database. Use `supabase db schema declarative sync` to compare declarative schema files.\n"; /** * Whether pg-delta is the active default engine. Mirrors Go's `shouldUsePgDelta` From b7ad988b755784bd8fee9b8a26197d3e803d3438 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 16:44:11 +0200 Subject: [PATCH 13/82] test(cli): use malformed branch response type --- .../legacy/commands/branches/create/create.integration.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/cli/src/legacy/commands/branches/create/create.integration.test.ts b/apps/cli/src/legacy/commands/branches/create/create.integration.test.ts index ec677a8341..57f4a5becf 100644 --- a/apps/cli/src/legacy/commands/branches/create/create.integration.test.ts +++ b/apps/cli/src/legacy/commands/branches/create/create.integration.test.ts @@ -344,7 +344,7 @@ describe("legacy branches create integration", () => { it.live("surfaces malformed successful responses as schema errors, not network errors", () => { const { layer } = setup({ - response: { ...CREATED, created_at: "not-a-timestamp" }, + response: { ...CREATED, created_at: 42 }, }); return Effect.gen(function* () { const exit = yield* Effect.exit( From 7f6fb326fc1aabaaa0ef992881477b570b65fd6c Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 16:44:21 +0200 Subject: [PATCH 14/82] chore(cli): update pg-delta next preview --- apps/cli/package.json | 4 ++-- pnpm-lock.yaml | 22 +++++++++++----------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/apps/cli/package.json b/apps/cli/package.json index b8171b32f9..4e8d1726b9 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -55,8 +55,8 @@ "@parcel/watcher": "^2.6.0", "@supabase/api": "workspace:*", "@supabase/config": "workspace:*", - "@supabase/pg-delta": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb", - "@supabase/pg-topo": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb", + "@supabase/pg-delta": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86", + "@supabase/pg-topo": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86", "@supabase/process-compose": "workspace:*", "@supabase/stack": "workspace:*", "@tsconfig/bun": "catalog:", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index e55862a752..a593864a86 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -155,11 +155,11 @@ importers: specifier: workspace:* version: link:../../packages/config '@supabase/pg-delta': - specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb - version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb) + specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86 + version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86) '@supabase/pg-topo': - specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb - version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb + specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86 + version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86 '@supabase/process-compose': specifier: workspace:* version: link:../../packages/process-compose @@ -2839,8 +2839,8 @@ packages: resolution: {integrity: sha512-RW/OCsd6MO592zU8ifzP8/f8XzxxIdpb+Up5XaOtE26Fw+3zTp475WX7+GuuktiD1WF8pFUDe6khUPbMp77RCw==} engines: {node: '>=22.0.0'} - '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb': - resolution: {integrity: sha512-zD/OOjOZaOIaMMjm7VbhlZa23hqeQFDYaWT3mnfT7/6/JgSEp5rge4gwTY+cjy+Q7ObLdwqJe+wDO7ARujJMbA==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb} + '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86': + resolution: {integrity: sha512-jEEZhv8uh2vFPIoMeRd4GK1qiTNZh+LiIpY1bjd+oFVlacnb+UIrjS85JdDWStniRT03+NPOUkpjOtGiTspsIg==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86} version: 1.0.0-alpha.33 engines: {node: '>=20.0.0'} hasBin: true @@ -2850,8 +2850,8 @@ packages: '@supabase/pg-topo': optional: true - '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb': - resolution: {integrity: sha512-HoqxATDYB2WygDOV1XQBN/hM/hcfvVUrc7F+R691j6CD89cHumR/nRiRJ+0bh9siZb7Fx81Bp9BAPRfzEkEydA==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb} + '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86': + resolution: {integrity: sha512-HoqxATDYB2WygDOV1XQBN/hM/hcfvVUrc7F+R691j6CD89cHumR/nRiRJ+0bh9siZb7Fx81Bp9BAPRfzEkEydA==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86} version: 1.0.0-alpha.5 '@supabase/phoenix@0.4.5': @@ -9093,18 +9093,18 @@ snapshots: dependencies: tslib: 2.8.1 - '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@ad62ae432865f67bb359a8183a2b3279fa9ebccb(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb)': + '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86)': dependencies: debug: 4.4.3(supports-color@7.2.0) pg: 8.22.0 pg-connection-string: 2.14.0 optionalDependencies: - '@supabase/pg-topo': https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb + '@supabase/pg-topo': https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86 transitivePeerDependencies: - pg-native - supports-color - '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@ad62ae432865f67bb359a8183a2b3279fa9ebccb': + '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86': dependencies: '@pgsql/traverse': 17.2.6 plpgsql-parser: 0.5.16 From f9bd2890b73b6084ac64753accabd34d97454eb3 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 18:32:27 +0200 Subject: [PATCH 15/82] feat(cli): improve declarative schema upgrade flow --- .../schema/declarative/declarative.errors.ts | 7 + .../declarative.extension-repair.ts | 50 +++++++ .../declarative.extension-repair.unit.test.ts | 65 +++++++++ .../db/schema/declarative/declarative.flow.ts | 99 +++++++++----- .../declarative/declarative.flow.unit.test.ts | 62 ++++++--- .../declarative/fixtures/legacy/extension.sql | 2 + .../declarative/generate/SIDE_EFFECTS.md | 16 ++- .../declarative/generate/generate.command.ts | 12 +- .../declarative/generate/generate.handler.ts | 20 +-- .../generate/generate.integration.test.ts | 112 +++++++++++++++ .../schema/declarative/sync/SIDE_EFFECTS.md | 29 +++- .../schema/declarative/sync/sync.command.ts | 2 +- .../schema/declarative/sync/sync.handler.ts | 128 ++++++++++++++---- .../declarative/sync/sync.integration.test.ts | 116 ++++++++++++++-- .../legacy-pgdelta-engine.next.layer.ts | 25 +++- .../shared/legacy-pgdelta-next-diagnostics.ts | 31 ++++- ...gacy-pgdelta-next-diagnostics.unit.test.ts | 110 +++++++++++++-- 17 files changed, 749 insertions(+), 137 deletions(-) create mode 100644 apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.ts create mode 100644 apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/schema/declarative/fixtures/legacy/extension.sql diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts index 4ff6cb9ab5..5c0274556e 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts @@ -71,6 +71,13 @@ export class LegacyDeclarativeDiffError extends Data.TaggedError("LegacyDeclarat readonly message: string; }> {} +/** Sync stopped because a manifest-less legacy schema needs an explicit migration choice. */ +export class LegacyDeclarativeCompatibilityError extends Data.TaggedError( + "LegacyDeclarativeCompatibilityError", +)<{ + readonly message: string; +}> {} + /** * Applying the generated migration to the local database failed. Wraps Go's * `applyMigrationToLocal` error; in interactive mode the handler offers a diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.ts new file mode 100644 index 0000000000..3f5db7862a --- /dev/null +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.ts @@ -0,0 +1,50 @@ +import { Effect, FileSystem, Path } from "effect"; + +import { legacyExtensionDeclaration } from "./declarative.flow.ts"; + +interface LegacyExtensionRepairResult { + readonly path: string; + readonly addedExtensions: ReadonlyArray; + readonly addedDeclarations: ReadonlyArray; +} + +const declaredExtensions = (sql: string): ReadonlySet => { + const extensions = new Set(); + const pattern = + /\bCREATE\s+EXTENSION\s+(?:IF\s+NOT\s+EXISTS\s+)?(?:"([^"]+)"|([a-zA-Z_][\w$-]*))/gi; + for (const match of sql.matchAll(pattern)) { + const extension = match[1] ?? match[2]; + if (extension !== undefined) extensions.add(extension); + } + return extensions; +}; + +/** Appends missing legacy extension declarations without replacing existing SQL. */ +export const legacyAppendExtensionDeclarations = Effect.fnUntraced(function* ( + declarativeDir: string, + extensions: ReadonlyArray, +) { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const extensionPath = path.join(declarativeDir, "extension.sql"); + const exists = yield* fs.exists(extensionPath); + const existing = exists ? yield* fs.readFileString(extensionPath) : ""; + const declared = declaredExtensions(existing); + const addedExtensions = [...new Set(extensions)] + .filter((extension) => !declared.has(extension)) + .sort(); + const addedDeclarations = addedExtensions.map(legacyExtensionDeclaration); + + if (addedDeclarations.length > 0) { + const newline = existing.includes("\r\n") ? "\r\n" : "\n"; + const separator = existing.length === 0 || existing.endsWith("\n") ? "" : newline; + const appended = `${separator}${addedDeclarations.join(newline)}${newline}`; + yield* fs.writeFileString(extensionPath, `${existing}${appended}`); + } + + return { + path: extensionPath, + addedExtensions, + addedDeclarations, + } satisfies LegacyExtensionRepairResult; +}); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.unit.test.ts new file mode 100644 index 0000000000..ab603f9bf9 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.unit.test.ts @@ -0,0 +1,65 @@ +import { readFileSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { BunServices } from "@effect/platform-bun"; +import { describe, expect, it } from "@effect/vitest"; +import { Effect } from "effect"; + +import { useLegacyTempWorkdir } from "../../../../../../tests/helpers/legacy-mocks.ts"; +import { legacyAppendExtensionDeclarations } from "./declarative.extension-repair.ts"; + +describe("legacyAppendExtensionDeclarations", () => { + const tmp = useLegacyTempWorkdir(); + + it.effect("creates root extension.sql with sorted idempotent declarations", () => { + return Effect.gen(function* () { + const result = yield* legacyAppendExtensionDeclarations(tmp.current, [ + "uuid-ossp", + "pgcrypto", + "pgcrypto", + ]); + expect(result.addedExtensions).toEqual(["pgcrypto", "uuid-ossp"]); + expect(readFileSync(join(tmp.current, "extension.sql"), "utf8")).toBe( + [ + 'CREATE EXTENSION IF NOT EXISTS "pgcrypto" WITH SCHEMA "extensions";', + 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";', + "", + ].join("\n"), + ); + + const repeated = yield* legacyAppendExtensionDeclarations(tmp.current, ["uuid-ossp"]); + expect(repeated.addedDeclarations).toEqual([]); + }).pipe(Effect.provide(BunServices.layer)); + }); + + it.effect("preserves existing contents and CRLF newlines", () => { + const extensionPath = join(tmp.current, "extension.sql"); + writeFileSync(extensionPath, 'CREATE EXTENSION "pgcrypto";\r\n-- keep me'); + return Effect.gen(function* () { + const result = yield* legacyAppendExtensionDeclarations(tmp.current, ["pgcrypto", "pg_net"]); + expect(result.addedExtensions).toEqual(["pg_net"]); + expect(readFileSync(extensionPath, "utf8")).toBe( + 'CREATE EXTENSION "pgcrypto";\r\n-- keep me\r\n' + + 'CREATE EXTENSION IF NOT EXISTS "pg_net" WITH SCHEMA "extensions";\r\n', + ); + }).pipe(Effect.provide(BunServices.layer)); + }); + + it.effect("appends to the representative legacy root extension.sql", () => { + const fixture = join( + dirname(fileURLToPath(import.meta.url)), + "fixtures", + "legacy", + "extension.sql", + ); + const extensionPath = join(tmp.current, "extension.sql"); + writeFileSync(extensionPath, readFileSync(fixture, "utf8")); + return Effect.gen(function* () { + yield* legacyAppendExtensionDeclarations(tmp.current, ["uuid-ossp"]); + const updated = readFileSync(extensionPath, "utf8"); + expect(updated).toContain('CREATE EXTENSION IF NOT EXISTS "vector"'); + expect(updated).toContain('CREATE EXTENSION IF NOT EXISTS "uuid-ossp"'); + }).pipe(Effect.provide(BunServices.layer)); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts index 57ba1be344..a2e0a34f9a 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts @@ -1,18 +1,23 @@ import type { LegacyPgDeltaImplementation } from "../../../../shared/legacy-pgdelta-next-flag.ts"; import type { LegacyPgDeltaRemovalSummary } from "../../shared/legacy-pgdelta-engine.service.ts"; -/** - * Pure control-flow helpers ported 1:1 from - * `apps/cli-go/cmd/db_schema_declarative.go`. Kept free of Effect/services so - * the precedence rules are unit-testable in isolation; the handlers run the - * actual TTY prompt for the `"prompt"` decision. - */ +/** Extensions that legacy pg-delta treated as part of its implicit Supabase baseline. */ +const LEGACY_IMPLICIT_EXTENSIONS = ["pg_net", "pgcrypto", "uuid-ossp"] as const; + +type LegacyDeclarativeCompatibilityAction = "none" | "repair-extensions" | "stage-next-export"; + +export interface LegacyDeclarativeCompatibilityGap { + readonly repairableExtensions: ReadonlyArray; + readonly extensionIntents: LegacyPgDeltaRemovalSummary["extensionIntents"]; + readonly ambiguousRemovals: ReadonlyArray; + readonly recommendedAction: LegacyDeclarativeCompatibilityAction; +} /** - * Resolves the migration name. The explicit `--name` wins over `--file` - * (default `declarative_sync`). Mirrors Go's `resolveDeclarativeMigrationName` - * (`:99-104`). + * Pure control-flow helpers ported from the legacy Go implementation and kept + * free of Effect/services so handler decisions remain unit-testable. */ + export function legacyResolveDeclarativeMigrationName(name: string, file: string): string { return name.length > 0 ? name : file; } @@ -20,11 +25,6 @@ export function legacyResolveDeclarativeMigrationName(name: string, file: string /** Whether sync applies the generated migration, prompts, or skips. */ export type LegacyDeclarativeApplyDecision = "apply" | "skip" | "prompt"; -/** - * Decides whether to apply the generated migration to the local database. - * Precedence (Go's `resolveDeclarativeSyncShouldApply`, `:106-124`): - * `--no-apply` > `--apply` > global `--yes` > TTY prompt > non-TTY default (skip). - */ export function legacyResolveDeclarativeSyncApplyDecision(opts: { readonly apply: boolean; readonly noApply: boolean; @@ -38,41 +38,68 @@ export function legacyResolveDeclarativeSyncApplyDecision(opts: { return "skip"; } -/** - * Warns when pg-delta next sees semantic removals that a manifest-less, - * potentially legacy-authored declarative tree may simply have omitted. - */ -export function legacyDeclarativeCompatibilityWarning(opts: { +const emptyCompatibilityGap = (): LegacyDeclarativeCompatibilityGap => ({ + repairableExtensions: [], + extensionIntents: [], + ambiguousRemovals: [], + recommendedAction: "none", +}); + +/** Classifies manifest-less pg-delta next removals without performing any I/O. */ +export function legacyClassifyDeclarativeCompatibilityGap(opts: { readonly implementation: LegacyPgDeltaImplementation; readonly manifestPresent: boolean; readonly removals: LegacyPgDeltaRemovalSummary; -}): string | undefined { - if (opts.implementation !== "next" || opts.manifestPresent) return undefined; - const { extensions, extensionIntents } = opts.removals; - if (extensions.length === 0 && extensionIntents.length === 0) return undefined; +}): LegacyDeclarativeCompatibilityGap { + if (opts.implementation !== "next" || opts.manifestPresent) return emptyCompatibilityGap(); - const cronJobs = extensionIntents - .filter((intent) => intent.extension === "pg_cron" && intent.intentKind === "job") - .map((intent) => intent.key); - const otherIntents = extensionIntents.filter( - (intent) => intent.extension !== "pg_cron" || intent.intentKind !== "job", + const extensions = [...new Set(opts.removals.extensions)].sort(); + const repairableExtensions = extensions.filter((extension) => + LEGACY_IMPLICIT_EXTENSIONS.some((implicit) => implicit === extension), + ); + const ambiguousRemovals = extensions.filter( + (extension) => !LEGACY_IMPLICIT_EXTENSIONS.some((implicit) => implicit === extension), ); + const extensionIntents = opts.removals.extensionIntents; + + if (extensions.length === 0 && extensionIntents.length === 0) return emptyCompatibilityGap(); + const repairable = + repairableExtensions.length > 0 && + ambiguousRemovals.length === 0 && + extensionIntents.length === 0; + return { + repairableExtensions, + extensionIntents, + ambiguousRemovals, + recommendedAction: repairable ? "repair-extensions" : "stage-next-export", + }; +} + +export const legacyExtensionDeclaration = (extension: string): string => + `CREATE EXTENSION IF NOT EXISTS "${extension}" WITH SCHEMA "extensions";`; + +export function legacyFormatStagedExportRecommendation( + gap: LegacyDeclarativeCompatibilityGap, +): string { const detected = [ - ...(extensions.length > 0 ? [`Extensions: ${extensions.join(", ")}`] : []), - ...(cronJobs.length > 0 ? [`pg_cron jobs: ${cronJobs.join(", ")}`] : []), - ...(otherIntents.length > 0 + ...(gap.repairableExtensions.length > 0 + ? [`Legacy-implicit extensions: ${gap.repairableExtensions.join(", ")}`] + : []), + ...(gap.ambiguousRemovals.length > 0 + ? [`Extensions: ${gap.ambiguousRemovals.join(", ")}`] + : []), + ...(gap.extensionIntents.length > 0 ? [ - `Extension intents: ${otherIntents + `Extension-managed objects: ${gap.extensionIntents .map((intent) => `${intent.extension} ${intent.intentKind} ${intent.key}`) .join(", ")}`, ] : []), ]; - return [ - "WARNING: This declarative schema has no pg-delta next export manifest and may have been generated by the legacy engine.", - "pg-delta next plans to remove objects that legacy exports may omit:", + "WARNING: pg-delta next manages schema state that the legacy export did not represent.", ...detected, - "If these removals are unintended, do not apply this migration. Re-export using pg-delta next with `supabase db schema declarative generate --overwrite` for the intended target, or add declarations for the objects you want to keep, then run sync again.", + "Generate a next-compatible schema into a separate directory, review it, and adopt it when ready:", + "supabase db schema declarative generate --output supabase/database-next", ].join("\n"); } diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts index 955ddb88b7..d84756e0a3 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts @@ -1,7 +1,9 @@ import { describe, expect, it } from "vitest"; import { - legacyDeclarativeCompatibilityWarning, + legacyClassifyDeclarativeCompatibilityGap, + legacyExtensionDeclaration, + legacyFormatStagedExportRecommendation, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, } from "./declarative.flow.ts"; @@ -14,46 +16,72 @@ const removals = { ], }; -describe("legacyDeclarativeCompatibilityWarning", () => { - it("explains manifest-less next removals and remediation", () => { - const warning = legacyDeclarativeCompatibilityWarning({ +describe("legacyClassifyDeclarativeCompatibilityGap", () => { + it("repairs only the known legacy-implicit extension set", () => { + const gap = legacyClassifyDeclarativeCompatibilityGap({ + implementation: "next", + manifestPresent: false, + removals: { extensions: ["uuid-ossp", "pgcrypto", "pgcrypto"], extensionIntents: [] }, + }); + expect(gap).toEqual({ + repairableExtensions: ["pgcrypto", "uuid-ossp"], + extensionIntents: [], + ambiguousRemovals: [], + recommendedAction: "repair-extensions", + }); + expect(legacyExtensionDeclaration("uuid-ossp")).toBe( + 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";', + ); + }); + + it("stages a next export for mixed or unknown extension removals", () => { + const gap = legacyClassifyDeclarativeCompatibilityGap({ + implementation: "next", + manifestPresent: false, + removals: { extensions: ["pgcrypto", "postgis"], extensionIntents: [] }, + }); + expect(gap.repairableExtensions).toEqual(["pgcrypto"]); + expect(gap.ambiguousRemovals).toEqual(["postgis"]); + expect(gap.recommendedAction).toBe("stage-next-export"); + }); + + it("stages a next export when extension intents are present", () => { + const gap = legacyClassifyDeclarativeCompatibilityGap({ implementation: "next", manifestPresent: false, removals, }); - expect(warning).toContain("may have been generated by the legacy engine"); - expect(warning).toContain("Extensions: pgcrypto, uuid-ossp"); - expect(warning).toContain("pg_cron jobs: refresh download metrics"); - expect(warning).toContain("Extension intents: pgmq queue emails"); - expect(warning).toContain("declarative generate --overwrite"); - expect(warning).toContain("add declarations"); + expect(gap.recommendedAction).toBe("stage-next-export"); + expect(legacyFormatStagedExportRecommendation(gap)).toContain( + "generate --output supabase/database-next", + ); }); it("is suppressed for next exports with a manifest", () => { expect( - legacyDeclarativeCompatibilityWarning({ + legacyClassifyDeclarativeCompatibilityGap({ implementation: "next", manifestPresent: true, removals, - }), - ).toBeUndefined(); + }).recommendedAction, + ).toBe("none"); }); it("is suppressed for the legacy engine and irrelevant removals", () => { expect( - legacyDeclarativeCompatibilityWarning({ + legacyClassifyDeclarativeCompatibilityGap({ implementation: "legacy", manifestPresent: false, removals, }), - ).toBeUndefined(); + ).toMatchObject({ recommendedAction: "none" }); expect( - legacyDeclarativeCompatibilityWarning({ + legacyClassifyDeclarativeCompatibilityGap({ implementation: "next", manifestPresent: false, removals: { extensions: [], extensionIntents: [] }, }), - ).toBeUndefined(); + ).toMatchObject({ recommendedAction: "none" }); }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/fixtures/legacy/extension.sql b/apps/cli/src/legacy/commands/db/schema/declarative/fixtures/legacy/extension.sql new file mode 100644 index 0000000000..9c5102c4c0 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/schema/declarative/fixtures/legacy/extension.sql @@ -0,0 +1,2 @@ +-- Representative root extension file from a legacy declarative export. +CREATE EXTENSION IF NOT EXISTS "vector" WITH SCHEMA "extensions"; diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index ef442e60e0..9895dac8a4 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -44,12 +44,12 @@ platform view. ## Files Written -| Path | Format | When | -| --------------------------------------------------------------------------------------------------------------------------- | ------ | -------------------------------------------- | -| `/supabase/database/**/*.sql` (declarative dir; configurable via `[experimental.pgdelta] declarative_schema_path`) | SQL | always — the entire dir is wiped + rewritten | -| `/supabase/database/.pgdelta-export.json` | JSON | default-engine export policy/manifest | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------- | +| `/supabase/database/**/*.sql` (declarative dir; configurable via `[experimental.pgdelta] declarative_schema_path`, or invocation-local `--output`) | SQL | the selected destination is wiped + rewritten after overwrite confirmation | +| `/.pgdelta-export.json` | JSON | default-engine export policy/manifest | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers @@ -105,6 +105,10 @@ always go to stderr, in every `--output-format`. On success: - Requires `--experimental` or `[experimental.pgdelta] enabled = true`. - `--db-url` / `--linked` / `--local` are mutually exclusive; absent all three, smart mode prompts (existing-files overwrite → Local/Custom choice + reset offer). +- `--output ` selects a destination for this invocation only. Relative paths + resolve from the project workdir; it does not edit config or activate the output + for later syncs. A non-empty destination still requires confirmation or + `--overwrite`, and the configured declarative tree is left untouched. - The default engine preserves the shared direct/pooler, DNS, TLS, and client certificate connection behavior. The legacy opt-out retains its embedded CA file and `sslmode=verify-ca` URL rewrite. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts index 9455b637d6..f029fe56b4 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts @@ -15,6 +15,13 @@ const config = { overwrite: Flag.boolean("overwrite").pipe( Flag.withDescription("Overwrite declarative schema files without confirmation."), ), + output: Flag.string("output").pipe( + Flag.withAlias("o"), + Flag.withDescription( + "Write the generated declarative schema to this directory without changing the configured declarative schema path.", + ), + Flag.optional, + ), reset: Flag.boolean("reset").pipe( Flag.withDescription("Reset local database before generating (local data will be lost)."), ), @@ -65,7 +72,7 @@ export type LegacyDbSchemaDeclarativeGenerateFlags = CliCommand.Command.Config.I export const legacyDbSchemaDeclarativeGenerateCommand = Command.make("generate", config).pipe( Command.withDescription( - "Exports a live database into the complete declarative schema tree. This replaces declarative files only; it does not create migration files or update migration history. In non-interactive use, pass --local, --linked, or --db-url explicitly.", + "Exports a live database into the complete declarative schema tree. This replaces declarative files only; it does not create migration files or update migration history. Use --output to stage an export without changing the configured declarative path. In non-interactive use, pass --local, --linked, or --db-url explicitly.", ), Command.withShortDescription("Generate declarative schema from a database"), Command.withHandler((flags) => @@ -99,6 +106,7 @@ export const legacyDbSchemaDeclarativeGenerateCommand = Command.make("generate", "no-cache": merged.noCache, "strict-coverage": merged.strictCoverage, overwrite: merged.overwrite, + output: merged.output, reset: merged.reset, schema: merged.schema, "db-url": merged.dbUrl, @@ -113,7 +121,7 @@ export const legacyDbSchemaDeclarativeGenerateCommand = Command.make("generate", // (StringVarP) (`cmd/db_schema_declarative.go:495,500`); telemetry reports // changed flags by canonical `flag.Name` via `pflag.Visit`, so map the // shorthands so `generate -s public -p secret` logs `schema`/`password`. - aliases: { s: "schema", p: "password" }, + aliases: { o: "output", s: "schema", p: "password" }, }), withJsonErrorHandling, ); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts index 66e71c9e6e..46ad222f12 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts @@ -125,13 +125,13 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec } } - // Go prints `utils.GetDeclarativeDir()` verbatim (`declarative.go:156`, - // `db_schema_declarative.go:268`) — the config value, relative unless a user - // configures an absolute `declarative_schema_path` — so user-facing renders use - // `declarativeDirRel`. File I/O needs the resolved dir: `path.resolve` (not - // `path.join`) so an absolute config value is used as-is, matching Go's - // `config.resolve`, which only prefixes the workdir onto a RELATIVE path. - const declarativeDirRel = legacyResolveDeclarativeDir(path, toml.pgDelta); + // Preserve the selected value for user-facing output: invocation-local + // `--output` wins, otherwise use the configured declarative path. File I/O + // resolves relative values from the project workdir while keeping absolute + // values unchanged. + const declarativeDirRel = Option.getOrElse(flags.output, () => + legacyResolveDeclarativeDir(path, toml.pgDelta), + ); const declarativeDir = path.resolve(cliConfig.workdir, declarativeDirRel); const migrationsDir = path.join(cliConfig.workdir, "supabase", "migrations"); const local: LegacyLocalConn = { port: toml.port, password: toml.password }; @@ -279,7 +279,11 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec // merged config and targets the same dir the handler wrote to (also computed from // the merged `toml`). Go warms against the in-process merged config identically // (`declarative.go:138-154`), so this always runs when `!--no-cache`. - if (!flags.noCache && engine.implementation === "legacy") { + // A command-local --output is deliberately not activated in config. The + // legacy catalog seam resolves the configured declarative path itself, so + // warming here would inspect the wrong tree. Skip that optional legacy-only + // cache warm; the generated output remains complete and usable on its own. + if (!flags.noCache && engine.implementation === "legacy" && Option.isNone(flags.output)) { yield* (yield* LegacyDeclarativeSeam).exportCatalog({ mode: "declarative", noCache: flags.noCache, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index 537787d6b6..791784be25 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -277,6 +277,7 @@ const flags = ( noCache: over.noCache ?? false, strictCoverage: over.strictCoverage ?? false, overwrite: over.overwrite ?? false, + output: over.output ?? Option.none(), reset: over.reset ?? false, schema: over.schema ?? [], dbUrl: over.dbUrl ?? Option.none(), @@ -465,6 +466,117 @@ describe("legacy db schema declarative generate integration", () => { }).pipe(Effect.provide(s.layer)); }); + it.effect( + "--output writes a complete next export relative to the project without activating it", + () => { + mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + writeFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "select 1;"); + const configPath = join(tmp.current, "supabase", "config.toml"); + const config = [ + "[experimental.pgdelta]", + "enabled = true", + 'declarative_schema_path = "supabase/database"', + "", + ].join("\n"); + writeFileSync(configPath, config); + const destination = join("supabase", "database-next"); + const s = setup(tmp.current, { experimental: true, engineImplementation: "next" }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeGenerate( + flags({ local: Option.some(true), output: Option.some(destination) }), + ); + + expect( + readFileSync( + join(tmp.current, destination, "schemas", "public", "tables", "players.sql"), + "utf8", + ), + ).toBe("create table players ();"); + expect( + JSON.parse(readFileSync(join(tmp.current, destination, ".pgdelta-export.json"), "utf8")), + ).toMatchObject({ + formatVersion: 1, + profile: "supabase", + files: ["schemas/public/tables/players.sql"], + }); + expect( + readFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "utf8"), + ).toBe("select 1;"); + expect(readFileSync(configPath, "utf8")).toBe(config); + expect( + s.out.rawChunks.map((chunk) => ({ text: stripAnsi(chunk.text), stream: chunk.stream })), + ).toContainEqual({ + text: `Declarative schema written to ${destination}\n`, + stream: "stderr", + }); + }).pipe(Effect.provide(s.layer)); + }, + ); + + it.effect("--output protects a non-empty destination without --overwrite", () => { + const destination = join(tmp.current, "staged-schema"); + mkdirSync(destination, { recursive: true }); + writeFileSync(join(destination, "keep.sql"), "select 'keep';"); + const s = setup(tmp.current, { + experimental: true, + engineImplementation: "next", + promptConfirmResponses: [false], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeGenerate( + flags({ local: Option.some(true), output: Option.some(destination) }), + ); + expect(readFileSync(join(destination, "keep.sql"), "utf8")).toBe("select 'keep';"); + expect(existsSync(join(destination, ".pgdelta-export.json"))).toBe(false); + expect(s.out.rawChunks.some((chunk) => chunk.text.includes("Skipped writing"))).toBe(true); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("--output does not warm the configured legacy declarative tree", () => { + const s = setup(tmp.current, { experimental: true }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeGenerate( + flags({ local: Option.some(true), output: Option.some("staged-schema") }), + ); + expect(s.seamCalls).toEqual(["baseline"]); + expect( + existsSync( + join(tmp.current, "staged-schema", "schemas", "public", "tables", "players.sql"), + ), + ).toBe(true); + expect(existsSync(join(tmp.current, "supabase", "database"))).toBe(false); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("--overwrite replaces only the absolute --output destination", () => { + const destination = mkdtempSync(join(tmpdir(), "legacy-decl-output-")); + mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + writeFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "select 1;"); + writeFileSync(join(destination, "stale.sql"), "select 'stale';"); + const s = setup(tmp.current, { experimental: true, engineImplementation: "next" }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeGenerate( + flags({ + local: Option.some(true), + output: Option.some(destination), + overwrite: true, + }), + ); + expect(existsSync(join(destination, "stale.sql"))).toBe(false); + expect(existsSync(join(destination, ".pgdelta-export.json"))).toBe(true); + expect( + readFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "utf8"), + ).toBe("select 1;"); + expect( + s.out.rawChunks.map((chunk) => ({ text: stripAnsi(chunk.text), stream: chunk.stream })), + ).toContainEqual({ + text: `Declarative schema written to ${destination}\n`, + stream: "stderr", + }); + rmSync(destination, { recursive: true, force: true }); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("explicit --local checks the local Postgres image before generating", () => { const s = setup(tmp.current, { experimental: true, staleLocalImage: true }); return Effect.gen(function* () { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 03d738e74f..8f049bc5ae 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -15,10 +15,10 @@ as a new timestamped migration. - With `PGDELTA_DEBUG`, default-engine snapshots, plan, and diagnostics are written below `supabase/.temp/pgdelta/v2/debug//` and are not reusable. - The default engine always refuses extraction or declarative-loading errors. - Coverage gaps (`unmodeled_kind` or `unresolved_security_label`) warn by default - and explain that unsupported changes are absent from the migration plan; - `--strict-coverage` turns them into a refusal. Debug artifacts are saved before - policy evaluation when capture is enabled. + Fatal diagnostics are always shown. By default, `unmodeled_kind` coverage gaps + are summarized once while nonfatal internal diagnostics remain quiet; + `--strict-coverage` refuses coverage gaps and prints the exact blockers. Debug + mode prints every diagnostic. Artifacts are saved before policy evaluation. - Default-engine migrations may differ byte-for-byte and may be split into ordered files to preserve transaction boundaries. Successful execution and an empty subsequent sync are the compatibility contract. @@ -45,6 +45,7 @@ as a new timestamped migration. | Path | Format | When | | ------------------------------------------------------------------ | ------ | ---------------------------------------------------- | | `/supabase/migrations/_[_].sql` | SQL | changes; default engine may emit ordered segments | +| `/supabase/database/extension.sql` | SQL | interactive, explicit legacy-extension repair only | | `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: native/Go-backed catalog caches | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | @@ -79,6 +80,7 @@ as a new timestamped migration. | `1` | no declarative schema files found | | `1` | shadow-database / selected pg-delta engine / diff failure | | `1` | apply failure (when applied) — propagated from the native migration apply (`applyMigrationToLocal`) | +| `1` | repairable legacy extension omissions in non-interactive mode | The pg-delta gate and the mutex check are both raised before any side effects run, but the gate wins when both conditions apply simultaneously: Go's @@ -97,6 +99,15 @@ opt-out it prints after catalog warming — on both interactive and `--yes` path without prompting; both override the global `--yes`. `--no-apply` and `--apply` are mutually exclusive. +Before writing a migration, a manifest-less legacy tree that would remove only +`pgcrypto`, `uuid-ossp`, or `pg_net` offers three explicit choices: append the +detected declarations to root `extension.sql` and re-plan, continue with the +removals, or cancel. The repair uses `CREATE EXTENSION IF NOT EXISTS ... WITH +SCHEMA "extensions"`, never overwrites existing SQL, never creates a next-export +manifest, and proceeds only when the re-plan removes the compatibility gap. +Non-interactive execution, including global `--yes`, does not modify declarations +and stops with the exact SQL to add. + ## Notes - Requires `--experimental` or `[experimental.pgdelta] enabled = true`. @@ -104,9 +115,13 @@ are mutually exclusive. object omitted from it is intended to be removed, including extensions. This is deterministic regardless of whether the directory was generated, written by hand, or has a `.pgdelta-export.json` manifest. -- Projects upgrading from the legacy workflow should regenerate declarations or - add declarations for every extension they intend to retain before syncing. - Review the existing drop-statement warning before applying destructive changes. +- For gaps involving unknown extensions or extension-managed state such as + `pg_cron` jobs, generate a staged next-compatible tree with + `generate --output supabase/database-next`, review it, and adopt or + merge it explicitly. `--output` neither changes `config.toml` nor activates the + staged tree. +- The targeted `extension.sql` repair preserves detected installed extensions; + it does not certify the legacy tree as a complete pg-delta next export. - `--file` sets the migration filename stem (default `declarative_sync`); `--name` overrides it. In a TTY without `--name`/`--yes`, the name is prompted. - When no declarative files exist, a TTY offers to generate them (from local) first. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts index 0501c9a6cc..5ca71bd824 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.command.ts @@ -57,7 +57,7 @@ export type LegacyDbSchemaDeclarativeSyncFlags = CliCommand.Command.Config.Infer export const legacyDbSchemaDeclarativeSyncCommand = Command.make("sync", config).pipe( Command.withDescription( - "Compares the supabase/migrations baseline with the complete declarative schema tree and writes the difference as migration files. Use --no-apply for non-interactive generation without changing the local database; --apply or global --yes applies locally and updates local migration history.", + "Compares the supabase/migrations baseline with the complete declarative schema tree and writes the difference as migration files. When a legacy export omits known implicit extensions, interactive sync can add declarations and re-plan before writing. Use --no-apply for non-interactive generation without changing the local database; --apply or global --yes applies locally and updates local migration history.", ), Command.withShortDescription("Generate a new migration from declarative schema"), Command.withHandler((flags) => diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 14ccb9d8fa..62c034e8c2 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -41,15 +41,19 @@ import { } from "../../../shared/legacy-debug-bundle.ts"; import { LegacyDeclarativeApplyError, + LegacyDeclarativeCompatibilityError, LegacyDeclarativeMutuallyExclusiveFlagsError, LegacyDeclarativeNoFilesGeneratedError, LegacyDeclarativeNonInteractiveError, } from "../declarative.errors.ts"; import { - legacyDeclarativeCompatibilityWarning, + legacyClassifyDeclarativeCompatibilityGap, + legacyExtensionDeclaration, + legacyFormatStagedExportRecommendation, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, } from "../declarative.flow.ts"; +import { legacyAppendExtensionDeclarations } from "../declarative.extension-repair.ts"; import { legacyRequirePgDelta } from "../declarative.gate.ts"; import { type LegacyDeclarativeRunContext, @@ -283,28 +287,101 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara Option.getOrUndefined(toml.orioledbVersion), toml.baseline, ); - const result: LegacyDeclarativeSyncResult = yield* legacyDiffDeclarativeToMigrations( - run, - setupInputs, - ).pipe( - Effect.tapError((error) => - Effect.gen(function* () { - const migrations = yield* legacyCollectMigrationsList(fs, path, migrationsDir); - yield* legacySaveDebugBundle(fs, path, cliConfig.workdir, tempDir, migrationsDir, { - id: formatDebugId(yield* Clock.currentTimeMillis), - error: error.message, - migrations, - }).pipe( - Effect.matchEffect({ - // Go prints nothing when SaveDebugBundle errors on the diff path - // (`db_schema_declarative.go:337-340`: `if saveErr == nil`). - onFailure: () => Effect.void, - onSuccess: (debugDir) => output.raw(legacyDebugBundleMessage(debugDir), "stderr"), + const planDeclarativeSync = () => + legacyDiffDeclarativeToMigrations(run, setupInputs).pipe( + Effect.tapError((error) => + Effect.gen(function* () { + const migrations = yield* legacyCollectMigrationsList(fs, path, migrationsDir); + yield* legacySaveDebugBundle(fs, path, cliConfig.workdir, tempDir, migrationsDir, { + id: formatDebugId(yield* Clock.currentTimeMillis), + error: error.message, + migrations, + }).pipe( + Effect.matchEffect({ + // Go prints nothing when SaveDebugBundle errors on the diff path + // (`db_schema_declarative.go:337-340`: `if saveErr == nil`). + onFailure: () => Effect.void, + onSuccess: (debugDir) => output.raw(legacyDebugBundleMessage(debugDir), "stderr"), + }), + ); + }), + ), + ); + let result: LegacyDeclarativeSyncResult = yield* planDeclarativeSync(); + + // Resolve manifest-less legacy compatibility before printing or writing a + // migration. A repair is always explicit, even when global --yes is set. + const compatibility = legacyClassifyDeclarativeCompatibilityGap({ + implementation: engine.implementation, + manifestPresent: result.manifestPresent, + removals: result.removals, + }); + if (compatibility.recommendedAction === "repair-extensions") { + const statements = compatibility.repairableExtensions.map(legacyExtensionDeclaration); + const explanation = [ + "This declarative schema appears to use legacy pg-delta behavior. Legacy pg-delta treated these installed extensions as implicit, while pg-delta next treats their omission as removal:", + "", + ...compatibility.repairableExtensions.map((extension) => `- ${extension}`), + ].join("\n"); + if (!tty.stdinIsTty || yes) { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ + message: [ + explanation, + "", + "Non-interactive sync will not modify the declarative schema automatically. Add these statements to extension.sql, then run sync again:", + ...statements, + "", + "Or generate a next-compatible schema into a separate directory:", + "supabase db schema declarative generate --output supabase/database-next", + ].join("\n"), + }), + ); + } + + yield* output.raw(`${legacyYellow(explanation)}\n`, "stderr"); + const choice = yield* output.promptSelect("How would you like to continue?", [ + { + value: "repair", + label: "Add declarations and re-plan", + hint: "recommended", + }, + { value: "continue", label: "Continue with removals" }, + { value: "cancel", label: "Cancel" }, + ]); + if (choice === "cancel") return; + if (choice === "repair") { + const repaired = yield* legacyAppendExtensionDeclarations( + declarativeDir, + compatibility.repairableExtensions, + ); + yield* output.raw( + `Updated ${legacyBold(repaired.path)} with:\n${repaired.addedDeclarations.join("\n")}\n`, + "stderr", + ); + result = yield* planDeclarativeSync(); + const remaining = legacyClassifyDeclarativeCompatibilityGap({ + implementation: engine.implementation, + manifestPresent: result.manifestPresent, + removals: result.removals, + }); + if (remaining.recommendedAction !== "none") { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ + message: [ + "The compatibility removals remain after adding extension declarations.", + legacyFormatStagedExportRecommendation(remaining), + ].join("\n"), }), ); - }), - ), - ); + } + } + } else if (compatibility.recommendedAction === "stage-next-export") { + yield* output.raw( + `${legacyYellow(legacyFormatStagedExportRecommendation(compatibility))}\n`, + "stderr", + ); + } // Step 3: empty diff. if (result.diffSQL.trim().length < 2) { @@ -314,15 +391,6 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara yield* output.raw("Generated migration SQL:\n", "stderr"); yield* output.raw(`${result.diffSQL}\n`, "stderr"); - const compatibilityWarning = legacyDeclarativeCompatibilityWarning({ - implementation: engine.implementation, - manifestPresent: result.manifestPresent, - removals: result.removals, - }); - if (compatibilityWarning !== undefined) { - yield* output.raw(`${legacyYellow(compatibilityWarning)}\n`, "stderr"); - } - // Step 4: resolve migration name (prompt in TTY when --name unset). const file = Option.getOrElse(flags.file, () => DEFAULT_SYNC_NAME); const explicitName = Option.getOrElse(flags.name, () => ""); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index ed08167276..32add16120 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -1,4 +1,4 @@ -import { existsSync, mkdirSync, readdirSync, writeFileSync } from "node:fs"; +import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; @@ -74,6 +74,7 @@ interface SetupOpts { yes?: boolean; stdinIsTty?: boolean; diffSql?: string; + replannedDiffSql?: string; applyFails?: boolean; /** * Makes the recovery reset's `legacyResetLocalDatabase` fail immediately with @@ -243,15 +244,31 @@ function setup(workdir: string, opts: SetupOpts = {}) { ], manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, }), - planDeclarativeSchema: () => - Effect.succeed({ + planDeclarativeSchema: () => { + const extensionPath = join(workdir, "supabase", "database", "extension.sql"); + const extensionSql = existsSync(extensionPath) + ? readFileSync(extensionPath, "utf8") + : ""; + const remainingExtensions = (opts.removals?.extensions ?? []).filter( + (extension) => !extensionSql.includes(`"${extension}"`), + ); + const extensionsRepaired = + remainingExtensions.length < (opts.removals?.extensions.length ?? 0); + return Effect.succeed({ changes: nextFiles.length > 0, - sql: opts.diffSql ?? nextFiles.map((file) => file.sql).join("\n"), + sql: + extensionsRepaired && opts.replannedDiffSql !== undefined + ? opts.replannedDiffSql + : (opts.diffSql ?? nextFiles.map((file) => file.sql).join("\n")), files: nextFiles, sourceRef: "migrations", targetRef: "declarative", - removals: opts.removals, - }), + removals: + opts.removals === undefined + ? undefined + : { ...opts.removals, extensions: remainingExtensions }, + }); + }, }), ) : legacyPgDeltaLegacyEngineLayer.pipe( @@ -888,7 +905,7 @@ describe("legacy db schema declarative sync integration", () => { ); it.effect( - "warns before writing when next sees legacy-coverage removals without a manifest", + "recommends a staged next export before writing for extension-managed legacy gaps", () => { seedDeclarative(tmp.current); const s = setup(tmp.current, { @@ -907,17 +924,96 @@ describe("legacy db schema declarative sync integration", () => { yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); const chunks = s.out.rawChunks.map((chunk) => stripAnsi(chunk.text)); const warningAt = chunks.findIndex((chunk) => - chunk.includes("may have been generated by the legacy engine"), + chunk.includes("legacy export did not represent"), ); const createdAt = chunks.findIndex((chunk) => chunk.includes("Created new migration at")); expect(warningAt).toBeGreaterThan(-1); - expect(chunks[warningAt]).toContain("Extensions: pgcrypto, uuid-ossp"); - expect(chunks[warningAt]).toContain("pg_cron jobs: refresh download metrics"); + expect(chunks[warningAt]).toContain("pg_cron job refresh download metrics"); + expect(chunks[warningAt]).toContain("--output supabase/database-next"); expect(warningAt).toBeLessThan(createdAt); }).pipe(Effect.provide(s.layer)); }, ); + it.effect("adds detected legacy extension declarations and re-plans before writing", () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + diffSql: 'DROP EXTENSION "pg_net";\n', + replannedDiffSql: "", + removals: { extensions: ["pg_net"], extensionIntents: [] }, + promptSelectResponses: ["repair"], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + expect(readFileSync(join(tmp.current, "supabase", "database", "extension.sql"), "utf8")).toBe( + 'CREATE EXTENSION IF NOT EXISTS "pg_net" WITH SCHEMA "extensions";\n', + ); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join(""))).toContain( + "No schema changes found", + ); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect( + "continues with intentional legacy extension removals only after explicit choice", + () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + diffSql: 'DROP EXTENSION "pgcrypto";\n', + removals: { extensions: ["pgcrypto"], extensionIntents: [] }, + promptSelectResponses: ["continue"], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + expect(readdirSync(join(tmp.current, "supabase", "migrations"))).toHaveLength(1); + expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); + }).pipe(Effect.provide(s.layer)); + }, + ); + + it.effect("cancels compatibility resolution without schema or migration writes", () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + diffSql: 'DROP EXTENSION "uuid-ossp";\n', + removals: { extensions: ["uuid-ossp"], extensionIntents: [] }, + promptSelectResponses: ["cancel"], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("fails safely instead of repairing when sync is non-interactive", () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + diffSql: 'DROP EXTENSION "pgcrypto";\n', + removals: { extensions: ["pgcrypto"], extensionIntents: [] }, + }); + return Effect.gen(function* () { + const exit = yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })).pipe( + Effect.exit, + ); + expect(failError(exit)).toMatchObject({ + _tag: "LegacyDeclarativeCompatibilityError", + message: expect.stringContaining( + 'CREATE EXTENSION IF NOT EXISTS "pgcrypto" WITH SCHEMA "extensions";', + ), + }); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("suppresses the compatibility warning when a next export manifest is present", () => { seedDeclarative(tmp.current); writeFileSync( diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 91b41bd2d0..60c727dd93 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -160,6 +160,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( operation: LegacyPgDeltaNextOperation, diagnostics: Parameters[1], strictCoverage: boolean, + verboseDiagnostics: boolean, ) => { const report = legacyPgDeltaNextDiagnosticReport(diagnostics, strictCoverage); const showFeedback = !feedbackInvitationShown && report.unmodeledKinds.length > 0; @@ -169,7 +170,11 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diagnostics, strictCoverage, showFeedback, - ).pipe(Effect.provideService(Output, output)); + verboseDiagnostics, + ).pipe( + Effect.provideService(Output, output), + Effect.provideService(LegacyDebugLogger, debugLogger), + ); }; return LegacyPgDeltaEngine.of({ @@ -233,7 +238,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diagnostics: result.diagnostics, }) : undefined; - yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage); + yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage, input.debug); return normalizeNextDiff(result, debugDirectory); }), ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), @@ -273,7 +278,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diagnostics: result.diagnostics, }) : undefined; - yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage); + yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage, input.debug); return normalizeNextDiff(result, debugDirectory); }), ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), @@ -299,7 +304,12 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( : [...result.diagnostics, ...capture.diagnostics], }); } - yield* reportDiagnostics("declarativeExport", result.diagnostics, input.strictCoverage); + yield* reportDiagnostics( + "declarativeExport", + result.diagnostics, + input.strictCoverage, + input.debug, + ); return { files: result.files, manifest: result.manifest }; }), ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), @@ -343,7 +353,12 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diagnostics: result.diagnostics, }) : undefined; - yield* reportDiagnostics("declarativePlan", result.diagnostics, input.strictCoverage); + yield* reportDiagnostics( + "declarativePlan", + result.diagnostics, + input.strictCoverage, + input.debug, + ); return { ...normalizeNextDiff(result, debugDirectory), sourceRef: "pg-delta-next:migrations", diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts index 453bb3191d..df89b0d79b 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts @@ -1,6 +1,7 @@ import { Effect } from "effect"; import { Output } from "../../../../shared/output/output.service.ts"; +import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; import { LegacyPgDeltaEngineError } from "./legacy-pgdelta-engine.service.ts"; import type { LegacyPgDeltaNextDiagnostic, @@ -65,14 +66,19 @@ export function legacyPgDeltaNextDiagnosticMessage( return `pg-delta next diagnostic: origin=${diagnostic.origin} code=${diagnostic.code}${subject} message=${diagnostic.message}`; } -function legacyPgDeltaNextCoverageMessage( +function legacyPgDeltaNextUnmodeledKindsMessage( operation: LegacyPgDeltaNextOperation, + kinds: readonly string[], strictCoverage: boolean, ): string { const policy = strictCoverage ? "Strict coverage is enabled, so the operation will stop." : operationConsequence[operation]; - return `pg-delta found schema objects it does not manage. ${policy}`; + const summary = + kinds.length === 0 + ? "pg-delta found schema objects it does not manage." + : `pg-delta does not manage these PostgreSQL object kinds: ${kinds.join(", ")}.`; + return `${summary} ${policy}`; } function shellQuote(value: string): string { @@ -99,18 +105,28 @@ function legacyPgDeltaNextBlockingDiagnosticMessage( return `pg-delta next refused to ${operationAction[operation]}: ${reason}`; } -/** Render every adapter diagnostic and enforce the selected coverage policy. */ +/** Render actionable diagnostics, route internal detail to debug, and enforce coverage policy. */ export const legacyReportPgDeltaNextDiagnostics = Effect.fnUntraced(function* ( operation: LegacyPgDeltaNextOperation, diagnostics: readonly LegacyPgDeltaNextDiagnostic[], strictCoverage: boolean, showFeedback = true, + verboseDiagnostics = false, ) { const output = yield* Output; + const debug = yield* LegacyDebugLogger; const report = legacyPgDeltaNextDiagnosticReport(diagnostics, strictCoverage); for (const diagnostic of report.diagnostics) { const message = legacyPgDeltaNextDiagnosticMessage(diagnostic); + const renderDetail = + verboseDiagnostics || + diagnostic.severity === "error" || + (strictCoverage && coverageDiagnosticCodes.has(diagnostic.code)); + if (!renderDetail) { + yield* debug.debug(message); + continue; + } if (diagnostic.severity === "error") { yield* output.error(message); } else if (diagnostic.severity === "warning") { @@ -120,8 +136,13 @@ export const legacyReportPgDeltaNextDiagnostics = Effect.fnUntraced(function* ( } } - if (report.coverage.length > 0) { - yield* output.warn(legacyPgDeltaNextCoverageMessage(operation, strictCoverage)); + const unmodeledCount = report.diagnostics.filter( + (diagnostic) => diagnostic.code === "unmodeled_kind", + ).length; + if (unmodeledCount > 0) { + yield* output.warn( + legacyPgDeltaNextUnmodeledKindsMessage(operation, report.unmodeledKinds, strictCoverage), + ); } const feedback = showFeedback diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts index ce0ea2e371..849b1a9452 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts @@ -1,8 +1,9 @@ -import { Effect, Exit } from "effect"; +import { Effect, Exit, Layer } from "effect"; import { it } from "@effect/vitest"; import { describe, expect } from "vitest"; import { mockOutput } from "../../../../../tests/helpers/mocks.ts"; +import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; import type { LegacyPgDeltaNextDiagnostic } from "./legacy-pgdelta-next-adapter.service.ts"; import { legacyPgDeltaNextDiagnosticMessage, @@ -24,32 +25,73 @@ const unmodeled = ( ...overrides, }); +const debugLayer = (messages: string[]) => + Layer.succeed(LegacyDebugLogger, { + debug: (message) => Effect.sync(() => messages.push(message)), + http: () => Effect.void, + }); + describe("pg-delta next diagnostic coverage policy", () => { - it("allows coverage gaps by default after rendering diagnostics and one feedback invitation", () => { + it("summarizes unmodeled kinds and routes nonfatal diagnostic detail to debug", () => { const out = mockOutput(); + const debugMessages: string[] = []; return Effect.gen(function* () { yield* legacyReportPgDeltaNextDiagnostics( "diff", - [unmodeled("text search configuration"), unmodeled("statistics object")], + [ + unmodeled("text search configuration"), + unmodeled("statistics object"), + { + origin: "source", + code: "dangling_edge", + severity: "warning", + subject: "role:postgres", + message: "edge references a fact not in the base", + }, + { + origin: "declarativeLoad", + code: "invalid_routine_body", + severity: "warning", + message: "routine body failed validation", + }, + { + origin: "snapshot", + code: "unresolved_security_label", + severity: "warning", + message: "provider was not resolved", + }, + ], false, ); - expect(out.messages.filter(({ type }) => type === "warn")).toHaveLength(3); + expect(out.messages.filter(({ type }) => type === "warn")).toHaveLength(1); expect(out.messages).toContainEqual({ type: "warn", message: - "pg-delta found schema objects it does not manage. Changes to these objects are omitted from the generated database diff.", + "pg-delta does not manage these PostgreSQL object kinds: statistics object, text search configuration. Changes to these objects are omitted from the generated database diff.", }); + expect(out.messages.some(({ message }) => message.includes("dangling_edge"))).toBe(false); + expect(out.messages.some(({ message }) => message.includes("invalid_routine_body"))).toBe( + false, + ); + expect( + out.messages.some(({ message }) => message.includes("unresolved_security_label")), + ).toBe(false); + expect(debugMessages).toHaveLength(5); + expect(debugMessages).toContain( + "pg-delta next diagnostic: origin=source code=dangling_edge subject=role:postgres message=edge references a fact not in the base", + ); const invitations = out.messages.filter(({ message }) => message.startsWith("Request pg-delta support:"), ); expect(invitations).toHaveLength(1); expect(invitations[0]?.message).toContain("statistics object, text search configuration"); - }).pipe(Effect.provide(out.layer)); + }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); it("renders coverage diagnostics and then fails in strict mode", () => { const out = mockOutput(); + const debugMessages: string[] = []; return Effect.gen(function* () { const exit = yield* legacyReportPgDeltaNextDiagnostics( "declarativePlan", @@ -61,16 +103,23 @@ describe("pg-delta next diagnostic coverage policy", () => { expect(out.messages).toContainEqual({ type: "warn", message: - "pg-delta found schema objects it does not manage. Strict coverage is enabled, so the operation will stop.", + "pg-delta next diagnostic: origin=desired code=unmodeled_kind subject=object:public.unsupported message=object kind is not modeled", + }); + expect(out.messages).toContainEqual({ + type: "warn", + message: + "pg-delta does not manage these PostgreSQL object kinds: text search configuration. Strict coverage is enabled, so the operation will stop.", }); + expect(debugMessages).toEqual([]); expect(out.messages.some(({ message }) => message.includes("supabase issue feature"))).toBe( true, ); - }).pipe(Effect.provide(out.layer)); + }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); it("can suppress a repeated feedback invitation without suppressing warnings", () => { const out = mockOutput(); + const debugMessages: string[] = []; return Effect.gen(function* () { yield* legacyReportPgDeltaNextDiagnostics( "declarativePlan", @@ -83,11 +132,12 @@ describe("pg-delta next diagnostic coverage policy", () => { false, ); expect(out.messages.some(({ type }) => type === "warn")).toBe(true); - }).pipe(Effect.provide(out.layer)); + }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); it("always renders and fails error diagnostics", () => { const out = mockOutput(); + const debugMessages: string[] = []; return Effect.gen(function* () { const exit = yield* legacyReportPgDeltaNextDiagnostics( "declarativeExport", @@ -108,7 +158,47 @@ describe("pg-delta next diagnostic coverage policy", () => { message: "pg-delta next diagnostic: origin=export code=extraction_failed message=catalog query failed", }); - }).pipe(Effect.provide(out.layer)); + }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); + }); + + it("renders every diagnostic with full detail when pg-delta debug is enabled", () => { + const out = mockOutput(); + const debugMessages: string[] = []; + return Effect.gen(function* () { + yield* legacyReportPgDeltaNextDiagnostics( + "diff", + [ + { + origin: "source", + code: "dangling_edge", + severity: "warning", + subject: "role:postgres", + message: "edge references a fact not in the base", + }, + { + origin: "declarativeLoad", + code: "invalid_routine_body", + severity: "info", + message: "routine body failed validation", + }, + ], + false, + true, + true, + ); + + expect(out.messages).toContainEqual({ + type: "warn", + message: + "pg-delta next diagnostic: origin=source code=dangling_edge subject=role:postgres message=edge references a fact not in the base", + }); + expect(out.messages).toContainEqual({ + type: "info", + message: + "pg-delta next diagnostic: origin=declarativeLoad code=invalid_routine_body message=routine body failed validation", + }); + expect(debugMessages).toEqual([]); + }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); it("classifies both coverage codes and aggregates arbitrary kinds safely", () => { From 46a16147588b301b0a3e416bf66dc0ceffb7f9bf Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 8 Aug 2026 19:48:18 +0200 Subject: [PATCH 16/82] fix(cli): address pg-delta next review findings --- apps/cli-go/cmd/db.go | 65 +++-- apps/cli-go/cmd/db_shadow_test.go | 38 ++- apps/cli-go/internal/db/diff/shadow.go | 111 ++++---- apps/cli-go/internal/db/diff/shadow_test.go | 64 ++++- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 14 +- .../commands/db/diff/diff.integration.test.ts | 3 +- .../legacy/commands/db/diff/diff.layers.ts | 1 + .../legacy/commands/db/pull/SIDE_EFFECTS.md | 9 +- .../commands/db/pull/pull.integration.test.ts | 11 +- .../legacy/commands/db/pull/pull.layers.ts | 1 + .../legacy/commands/db/push/SIDE_EFFECTS.md | 21 +- .../commands/db/push/push.integration.test.ts | 29 ++ .../legacy/commands/db/reset/SIDE_EFFECTS.md | 17 +- .../db/reset/reset.integration.test.ts | 28 ++ ...eclarative.orchestrate.integration.test.ts | 4 +- .../declarative/generate/SIDE_EFFECTS.md | 3 +- .../generate/generate.integration.test.ts | 4 +- .../declarative/generate/generate.layers.ts | 1 + .../schema/declarative/sync/SIDE_EFFECTS.md | 3 +- .../declarative/sync/sync.integration.test.ts | 4 +- .../db/schema/declarative/sync/sync.layers.ts | 1 + .../db/shared/legacy-pgdelta-engine.layer.ts | 12 +- .../legacy-pgdelta-engine.layer.unit.test.ts | 74 ++++- ...elta-engine.next.layer.integration.test.ts | 156 +++++++++++ .../legacy-pgdelta-engine.next.layer.ts | 32 ++- ...acy-pgdelta-engine.next.layer.unit.test.ts | 42 ++- .../legacy-pgdelta-next-shadow.layer.ts | 16 +- .../legacy-pgdelta-next-shadow.service.ts | 24 +- .../legacy-pgdelta-next-shadow.unit.test.ts | 108 ++++--- .../db/shared/legacy-pgdelta.seam.layer.ts | 264 ++++++++++-------- .../legacy-pgdelta.seam.layer.unit.test.ts | 38 ++- .../db/shared/legacy-pgdelta.seam.service.ts | 31 +- .../legacy/shared/legacy-migration-apply.ts | 53 +++- .../legacy-migration-apply.unit.test.ts | 69 +++++ .../legacy/shared/legacy-migration-file.ts | 39 +++ .../shared/legacy-migration-file.unit.test.ts | 45 ++- .../legacy/shared/legacy-migration-history.ts | 5 +- 37 files changed, 1106 insertions(+), 334 deletions(-) create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts diff --git a/apps/cli-go/cmd/db.go b/apps/cli-go/cmd/db.go index 13e6a94b91..cd8e3f7547 100644 --- a/apps/cli-go/cmd/db.go +++ b/apps/cli-go/cmd/db.go @@ -40,34 +40,25 @@ type pgDeltaNextShadowEndpoint struct { } type pgDeltaNextShadowHandoff struct { - Migrations pgDeltaNextShadowEndpoint `json:"migrations"` - Declarative pgDeltaNextShadowEndpoint `json:"declarative"` + Migrations pgDeltaNextShadowEndpoint `json:"migrations"` + Declarative *pgDeltaNextShadowEndpoint `json:"declarative,omitempty"` } // handoffPgDeltaNextShadow transfers cleanup ownership only after the caller // has received and acknowledged the complete JSON description. Until then Go -// removes both containers on every exit path, including cancellation and I/O -// failure. -func handoffPgDeltaNextShadow(ctx context.Context, shadow diff.PgDeltaNextShadow, in io.Reader, out io.Writer, remove func(string)) error { +// removes every described container on exit, including cancellation and I/O failure. +func handoffPgDeltaNextShadow(ctx context.Context, payload pgDeltaNextShadowHandoff, in io.Reader, out io.Writer, remove func(string)) error { transferred := false defer func() { if transferred { return } - remove(shadow.Migrations.Container) - remove(shadow.Declarative.Container) + remove(payload.Migrations.ContainerID) + if payload.Declarative != nil { + remove(payload.Declarative.ContainerID) + } }() - payload := pgDeltaNextShadowHandoff{ - Migrations: pgDeltaNextShadowEndpoint{ - ContainerID: shadow.Migrations.Container, - URL: utils.ToPostgresURLWithoutPassword(shadow.Migrations.Config), - }, - Declarative: pgDeltaNextShadowEndpoint{ - ContainerID: shadow.Declarative.Container, - URL: utils.ToPostgresURLWithoutPassword(shadow.Declarative.Config), - }, - } if err := json.NewEncoder(out).Encode(payload); err != nil { return fmt.Errorf("failed to encode pg-delta shadow handoff: %w", err) } @@ -106,6 +97,29 @@ func handoffPgDeltaNextShadow(ctx context.Context, shadow diff.PgDeltaNextShadow return nil } +func handoffPgDeltaNextMigrationsShadow(ctx context.Context, shadow diff.PgDeltaNextShadowDatabase, in io.Reader, out io.Writer, remove func(string)) error { + return handoffPgDeltaNextShadow(ctx, pgDeltaNextShadowHandoff{ + Migrations: pgDeltaNextShadowEndpoint{ + ContainerID: shadow.Container, + URL: utils.ToPostgresURLWithoutPassword(shadow.Config), + }, + }, in, out, remove) +} + +func handoffPgDeltaNextPlanShadow(ctx context.Context, shadow diff.PgDeltaNextPlanShadow, in io.Reader, out io.Writer, remove func(string)) error { + declarative := pgDeltaNextShadowEndpoint{ + ContainerID: shadow.Declarative.Container, + URL: utils.ToPostgresURLWithoutPassword(shadow.Declarative.Config), + } + return handoffPgDeltaNextShadow(ctx, pgDeltaNextShadowHandoff{ + Migrations: pgDeltaNextShadowEndpoint{ + ContainerID: shadow.Migrations.Container, + URL: utils.ToPostgresURLWithoutPassword(shadow.Migrations.Config), + }, + Declarative: &declarative, + }, in, out, remove) +} + var ( dbCmd = &cobra.Command{ GroupID: groupLocalDev, @@ -284,7 +298,7 @@ var ( // commands to provision throwaway shadow databases, then leave them running // so the TS caller can run the differ itself and remove the containers // afterwards. Legacy modes print two newline-separated lines. pgdelta-next - // emits a JSON object describing its two isolated clusters, then retains + // modes emit a JSON object describing the requested isolated clusters, then retain // cleanup ownership until the caller acknowledges receipt. URLs are emitted // WITHOUT the password // (ToPostgresURLWithoutPassword) so we never log a credential to stdout @@ -319,12 +333,19 @@ var ( if err := flags.LoadConfig(fsys); err != nil { return err } - if shadowMode == "pgdelta-next" { - nextShadow, err := diff.PreparePgDeltaNextShadow(cmd.Context(), fsys) + if shadowMode == "pgdelta-next-migrations" { + nextShadow, err := diff.PreparePgDeltaNextMigrationsShadow(cmd.Context(), fsys) + if err != nil { + return err + } + return handoffPgDeltaNextMigrationsShadow(cmd.Context(), nextShadow, os.Stdin, os.Stdout, utils.DockerRemove) + } + if shadowMode == "pgdelta-next-plan" { + nextShadow, err := diff.PreparePgDeltaNextPlanShadow(cmd.Context(), fsys) if err != nil { return err } - return handoffPgDeltaNextShadow(cmd.Context(), nextShadow, os.Stdin, os.Stdout, utils.DockerRemove) + return handoffPgDeltaNextPlanShadow(cmd.Context(), nextShadow, os.Stdin, os.Stdout, utils.DockerRemove) } var src diff.ShadowSource var err error @@ -689,7 +710,7 @@ func init() { dbCmd.AddCommand(dbPullCmd) // Build hidden shadow-provisioning seam command shadowFlags := dbShadowCmd.Flags() - shadowFlags.StringVar(&shadowMode, "mode", "diff", "Shadow mode: diff (baseline + migrations), declarative (bare shadow), or pgdelta-next (migrated + empty scratch).") + shadowFlags.StringVar(&shadowMode, "mode", "diff", "Shadow mode: diff (baseline + migrations), declarative (bare shadow), pgdelta-next-migrations (migrated), or pgdelta-next-plan (migrated + declarative scratch).") shadowFlags.StringSliceVarP(&shadowSchema, "schema", "s", []string{}, "Comma separated list of schema to include.") shadowFlags.StringVar(&shadowProjectRef, "project-ref", "", "Linked project ref, so the shadow merges the matching [remotes.] config override.") dbCmd.AddCommand(dbShadowCmd) diff --git a/apps/cli-go/cmd/db_shadow_test.go b/apps/cli-go/cmd/db_shadow_test.go index b5df3742af..e2e48f6901 100644 --- a/apps/cli-go/cmd/db_shadow_test.go +++ b/apps/cli-go/cmd/db_shadow_test.go @@ -19,7 +19,7 @@ func TestHandoffPgDeltaNextShadowTransfersOwnershipAfterAck(t *testing.T) { var output bytes.Buffer var removed []string - err := handoffPgDeltaNextShadow(context.Background(), shadow, strings.NewReader("ack\n"), &output, func(container string) { + err := handoffPgDeltaNextPlanShadow(context.Background(), shadow, strings.NewReader("ack\n"), &output, func(container string) { removed = append(removed, container) }) @@ -42,7 +42,7 @@ func TestHandoffPgDeltaNextShadowRetainsOwnershipOnHandshakeFailure(t *testing.T for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { var removed []string - err := handoffPgDeltaNextShadow(context.Background(), testPgDeltaNextShadow(), strings.NewReader(tt.input), io.Discard, func(container string) { + err := handoffPgDeltaNextPlanShadow(context.Background(), testPgDeltaNextShadow(), strings.NewReader(tt.input), io.Discard, func(container string) { removed = append(removed, container) }) @@ -54,7 +54,7 @@ func TestHandoffPgDeltaNextShadowRetainsOwnershipOnHandshakeFailure(t *testing.T func TestHandoffPgDeltaNextShadowCleansBothOnEncodingFailure(t *testing.T) { var removed []string - err := handoffPgDeltaNextShadow(context.Background(), testPgDeltaNextShadow(), strings.NewReader("ack\n"), failingWriter{}, func(container string) { + err := handoffPgDeltaNextPlanShadow(context.Background(), testPgDeltaNextShadow(), strings.NewReader("ack\n"), failingWriter{}, func(container string) { removed = append(removed, container) }) @@ -72,7 +72,7 @@ func TestHandoffPgDeltaNextShadowCleansBothOnCancellation(t *testing.T) { }) var removed []string - err := handoffPgDeltaNextShadow(ctx, testPgDeltaNextShadow(), reader, io.Discard, func(container string) { + err := handoffPgDeltaNextPlanShadow(ctx, testPgDeltaNextShadow(), reader, io.Discard, func(container string) { removed = append(removed, container) }) @@ -80,8 +80,34 @@ func TestHandoffPgDeltaNextShadowCleansBothOnCancellation(t *testing.T) { assert.Equal(t, []string{"migrations-container", "declarative-container"}, removed) } -func testPgDeltaNextShadow() diff.PgDeltaNextShadow { - return diff.PgDeltaNextShadow{ +func TestHandoffPgDeltaNextMigrationsShadowTransfersOneContainer(t *testing.T) { + shadow := testPgDeltaNextShadow().Migrations + var output bytes.Buffer + var removed []string + + err := handoffPgDeltaNextMigrationsShadow(context.Background(), shadow, strings.NewReader("ack\n"), &output, func(container string) { + removed = append(removed, container) + }) + + require.NoError(t, err) + assert.Equal(t, "{\"migrations\":{\"containerId\":\"migrations-container\",\"url\":\"postgresql://postgres@migrations-host:6543/postgres?connect_timeout=10\"}}\n", output.String()) + assert.Empty(t, removed) +} + +func TestHandoffPgDeltaNextMigrationsShadowCleansOneContainerOnFailure(t *testing.T) { + shadow := testPgDeltaNextShadow().Migrations + var removed []string + + err := handoffPgDeltaNextMigrationsShadow(context.Background(), shadow, strings.NewReader("nope\n"), io.Discard, func(container string) { + removed = append(removed, container) + }) + + assert.ErrorContains(t, err, "unexpected") + assert.Equal(t, []string{"migrations-container"}, removed) +} + +func testPgDeltaNextShadow() diff.PgDeltaNextPlanShadow { + return diff.PgDeltaNextPlanShadow{ Migrations: diff.PgDeltaNextShadowDatabase{ Container: "migrations-container", Config: pgconn.Config{ diff --git a/apps/cli-go/internal/db/diff/shadow.go b/apps/cli-go/internal/db/diff/shadow.go index 1092a68175..270d6d6262 100644 --- a/apps/cli-go/internal/db/diff/shadow.go +++ b/apps/cli-go/internal/db/diff/shadow.go @@ -32,11 +32,11 @@ type PgDeltaNextShadowDatabase struct { Config pgconn.Config } -// PgDeltaNextShadow contains the two isolated clusters used by the native +// PgDeltaNextPlanShadow contains the two isolated clusters used by the native // pg-delta engine. Migrations has the platform baseline plus local migrations; // Declarative has the same platform baseline and local configuration, ready for // pg-delta to load declarative SQL into postgres. -type PgDeltaNextShadow struct { +type PgDeltaNextPlanShadow struct { Migrations PgDeltaNextShadowDatabase Declarative PgDeltaNextShadowDatabase } @@ -50,11 +50,11 @@ type pgDeltaNextShadowDependencies struct { remove func(string) } -// PreparePgDeltaNextShadow provisions isolated migrated and declarative -// clusters. On failure, every container created so far is removed best-effort -// without replacing the provisioning error. -func PreparePgDeltaNextShadow(ctx context.Context, fsys afero.Fs, options ...func(*pgx.ConnConfig)) (PgDeltaNextShadow, error) { - return preparePgDeltaNextShadow(ctx, fsys, pgDeltaNextShadowDependencies{ +// PreparePgDeltaNextMigrationsShadow provisions only the migrated cluster used +// by database diffs. On failure, every container created so far is removed +// best-effort without replacing the provisioning error. +func PreparePgDeltaNextMigrationsShadow(ctx context.Context, fsys afero.Fs, options ...func(*pgx.ConnConfig)) (PgDeltaNextShadowDatabase, error) { + return preparePgDeltaNextMigrationsShadow(ctx, fsys, pgDeltaNextShadowDependencies{ freePort: utils.GetFreeHostPort, create: CreateShadowDatabase, wait: start.WaitForHealthyService, @@ -64,63 +64,80 @@ func PreparePgDeltaNextShadow(ctx context.Context, fsys afero.Fs, options ...fun }, options...) } -func preparePgDeltaNextShadow(ctx context.Context, fsys afero.Fs, dependencies pgDeltaNextShadowDependencies, options ...func(*pgx.ConnConfig)) (PgDeltaNextShadow, error) { - var containers []string +// PreparePgDeltaNextPlanShadow provisions isolated migrated and declarative +// clusters for a declarative plan. +func PreparePgDeltaNextPlanShadow(ctx context.Context, fsys afero.Fs, options ...func(*pgx.ConnConfig)) (PgDeltaNextPlanShadow, error) { + return preparePgDeltaNextPlanShadow(ctx, fsys, pgDeltaNextShadowDependencies{ + freePort: utils.GetFreeHostPort, + create: CreateShadowDatabase, + wait: start.WaitForHealthyService, + migrate: MigratePgDeltaNextShadowDatabase, + setup: SetupPgDeltaNextDeclarativeShadowDatabase, + remove: utils.DockerRemove, + }, options...) +} + +func preparePgDeltaNextMigrationsShadow(ctx context.Context, fsys afero.Fs, dependencies pgDeltaNextShadowDependencies, options ...func(*pgx.ConnConfig)) (PgDeltaNextShadowDatabase, error) { + return preparePgDeltaNextShadowDatabase(ctx, fsys, 0, dependencies, dependencies.migrate, options...) +} + +func preparePgDeltaNextPlanShadow(ctx context.Context, fsys afero.Fs, dependencies pgDeltaNextShadowDependencies, options ...func(*pgx.ConnConfig)) (PgDeltaNextPlanShadow, error) { + migrations, err := preparePgDeltaNextMigrationsShadow(ctx, fsys, dependencies, options...) + if err != nil { + return PgDeltaNextPlanShadow{}, err + } ok := false defer func() { if !ok { - for _, container := range containers { - dependencies.remove(container) - } + dependencies.remove(migrations.Container) } }() - migrationsPort, err := allocatePgDeltaNextPort(dependencies.freePort, 0) + declarative, err := preparePgDeltaNextShadowDatabase(ctx, fsys, migrations.Config.Port, dependencies, dependencies.setup, options...) if err != nil { - return PgDeltaNextShadow{}, err - } - migrationsContainer, err := dependencies.create(ctx, migrationsPort) - if migrationsContainer != "" { - containers = append(containers, migrationsContainer) - } - if err != nil { - return PgDeltaNextShadow{}, err - } - if err := dependencies.wait(ctx, utils.Config.Db.HealthTimeout, migrationsContainer); err != nil { - return PgDeltaNextShadow{}, err - } - if err := dependencies.migrate(ctx, migrationsContainer, fsys, append(options, withShadowPort(migrationsPort))...); err != nil { - return PgDeltaNextShadow{}, err + return PgDeltaNextPlanShadow{}, err } - declarativePort, err := allocatePgDeltaNextPort(dependencies.freePort, migrationsPort) + ok = true + return PgDeltaNextPlanShadow{ + Migrations: migrations, + Declarative: declarative, + }, nil +} + +func preparePgDeltaNextShadowDatabase( + ctx context.Context, + fsys afero.Fs, + excludedPort uint16, + dependencies pgDeltaNextShadowDependencies, + initialize func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error, + options ...func(*pgx.ConnConfig), +) (PgDeltaNextShadowDatabase, error) { + port, err := allocatePgDeltaNextPort(dependencies.freePort, excludedPort) if err != nil { - return PgDeltaNextShadow{}, err - } - declarativeContainer, err := dependencies.create(ctx, declarativePort) - if declarativeContainer != "" { - containers = append(containers, declarativeContainer) + return PgDeltaNextShadowDatabase{}, err } + container, err := dependencies.create(ctx, port) + ok := false + defer func() { + if !ok && container != "" { + dependencies.remove(container) + } + }() if err != nil { - return PgDeltaNextShadow{}, err + return PgDeltaNextShadowDatabase{}, err } - if err := dependencies.wait(ctx, utils.Config.Db.HealthTimeout, declarativeContainer); err != nil { - return PgDeltaNextShadow{}, err + if err := dependencies.wait(ctx, utils.Config.Db.HealthTimeout, container); err != nil { + return PgDeltaNextShadowDatabase{}, err } - if err := dependencies.setup(ctx, declarativeContainer, fsys, append(options, withShadowPort(declarativePort))...); err != nil { - return PgDeltaNextShadow{}, err + if err := initialize(ctx, container, fsys, append(options, withShadowPort(port))...); err != nil { + return PgDeltaNextShadowDatabase{}, err } ok = true - return PgDeltaNextShadow{ - Migrations: PgDeltaNextShadowDatabase{ - Container: migrationsContainer, - Config: pgDeltaNextShadowConfig(migrationsPort), - }, - Declarative: PgDeltaNextShadowDatabase{ - Container: declarativeContainer, - Config: pgDeltaNextShadowConfig(declarativePort), - }, + return PgDeltaNextShadowDatabase{ + Container: container, + Config: pgDeltaNextShadowConfig(port), }, nil } diff --git a/apps/cli-go/internal/db/diff/shadow_test.go b/apps/cli-go/internal/db/diff/shadow_test.go index 9d293c70dc..e9b1dc886b 100644 --- a/apps/cli-go/internal/db/diff/shadow_test.go +++ b/apps/cli-go/internal/db/diff/shadow_test.go @@ -13,7 +13,7 @@ import ( "github.com/supabase/cli/internal/utils" ) -func TestPreparePgDeltaNextShadow(t *testing.T) { +func TestPreparePgDeltaNextPlanShadow(t *testing.T) { originalConfig := utils.Config t.Cleanup(func() { utils.Config = originalConfig }) utils.Config.Hostname = "shadow-host" @@ -66,7 +66,7 @@ func TestPreparePgDeltaNextShadow(t *testing.T) { remove: func(container string) { removedContainers = append(removedContainers, container) }, } - result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) + result, err := preparePgDeltaNextPlanShadow(context.Background(), afero.NewMemMapFs(), dependencies) require.NoError(t, err) assert.Equal(t, []uint16{6543, 7654}, createdPorts) @@ -82,7 +82,57 @@ func TestPreparePgDeltaNextShadow(t *testing.T) { assert.Equal(t, "postgres", result.Declarative.Config.Database) } -func TestPreparePgDeltaNextShadowRemovesEveryCreatedContainerOnFailure(t *testing.T) { +func TestPreparePgDeltaNextMigrationsShadowOnlyCreatesMigratedDatabase(t *testing.T) { + originalConfig := utils.Config + t.Cleanup(func() { utils.Config = originalConfig }) + utils.Config.Hostname = "shadow-host" + utils.Config.Db.Password = "secret" + + var portCalls, createCalls, waitCalls, migrateCalls, setupCalls int + dependencies := pgDeltaNextShadowDependencies{ + freePort: func() (int, error) { + portCalls++ + return 6543, nil + }, + create: func(_ context.Context, port uint16) (string, error) { + createCalls++ + assert.Equal(t, uint16(6543), port) + return "migrations-container", nil + }, + wait: func(_ context.Context, _ time.Duration, containers ...string) error { + waitCalls++ + assert.Equal(t, []string{"migrations-container"}, containers) + return nil + }, + migrate: func(_ context.Context, container string, _ afero.Fs, options ...func(*pgx.ConnConfig)) error { + migrateCalls++ + assert.Equal(t, "migrations-container", container) + config := &pgx.ConnConfig{} + for _, option := range options { + option(config) + } + assert.Equal(t, uint16(6543), config.Port) + return nil + }, + setup: func(context.Context, string, afero.Fs, ...func(*pgx.ConnConfig)) error { + setupCalls++ + return nil + }, + remove: func(string) { t.Fatal("successful provision should not remove its container") }, + } + + result, err := preparePgDeltaNextMigrationsShadow(context.Background(), afero.NewMemMapFs(), dependencies) + + require.NoError(t, err) + assert.Equal(t, 1, portCalls) + assert.Equal(t, 1, createCalls) + assert.Equal(t, 1, waitCalls) + assert.Equal(t, 1, migrateCalls) + assert.Zero(t, setupCalls) + assert.Equal(t, "migrations-container", result.Container) +} + +func TestPreparePgDeltaNextPlanShadowRemovesEveryCreatedContainerOnFailure(t *testing.T) { wantErr := errors.New("provisioning failed") tests := []struct { name string @@ -98,9 +148,9 @@ func TestPreparePgDeltaNextShadowRemovesEveryCreatedContainerOnFailure(t *testin {name: "migrations", failAt: "migrate", firstID: "migrations", wantRemoved: []string{"migrations"}}, {name: "second port", failAt: "second-port", firstID: "migrations", wantRemoved: []string{"migrations"}}, {name: "second create without id", failAt: "second-create", firstID: "migrations", wantRemoved: []string{"migrations"}}, - {name: "second create with id", failAt: "second-create", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"migrations", "declarative"}}, - {name: "second health", failAt: "second-health", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"migrations", "declarative"}}, - {name: "declarative setup", failAt: "setup", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"migrations", "declarative"}}, + {name: "second create with id", failAt: "second-create", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"declarative", "migrations"}}, + {name: "second health", failAt: "second-health", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"declarative", "migrations"}}, + {name: "declarative setup", failAt: "setup", firstID: "migrations", secondID: "declarative", wantRemoved: []string{"declarative", "migrations"}}, } for _, tt := range tests { @@ -150,7 +200,7 @@ func TestPreparePgDeltaNextShadowRemovesEveryCreatedContainerOnFailure(t *testin remove: func(container string) { removed = append(removed, container) }, } - result, err := preparePgDeltaNextShadow(context.Background(), afero.NewMemMapFs(), dependencies) + result, err := preparePgDeltaNextPlanShadow(context.Background(), afero.NewMemMapFs(), dependencies) assert.ErrorIs(t, err, wantErr) assert.Empty(t, result) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index b1cf2c33c0..cf78370026 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -11,8 +11,9 @@ bundled Go binary. - The default implementation is the in-process pg-delta engine bundled into the CLI binary together with pg-topo. Its version is fixed when the CLI is built; there is no runtime package download or automatic fallback to the legacy engine. -- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy edge-runtime implementation. - Only that opt-out reads legacy catalogs under `supabase/.temp/pgdelta/`, +- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy edge-runtime implementation + from either the shell or project `supabase/.env` (the shell wins). Only that + opt-out reads legacy catalogs under `supabase/.temp/pgdelta/`, `supabase/.temp/pgdelta-version`, or `PGDELTA_NPM_REGISTRY`. - With `PGDELTA_DEBUG`, default-engine snapshots, plans, and diagnostics are written under `supabase/.temp/pgdelta/v2/debug//`. The directory contains @@ -61,10 +62,11 @@ bundled Go binary. legacy explicit `--from/--to migrations` path also runs the native pg-delta catalog-export script there on a cache miss (CLI-1959; no hidden `__catalog` subprocess). -- Shadow Postgres container(s), provisioned through the Go `db __shadow` seam. - The default engine uses isolated migrations and declarative shadows. The legacy - opt-out provisions a single `mode: "diff"` shadow, including on an explicit - migrations-catalog cache miss, and tears it down after export. +- One shadow Postgres container, provisioned through the Go `db __shadow` seam. + The default engine provisions only its isolated migrations shadow for normal + and explicit migrations diffs. The legacy opt-out provisions a single + `mode: "diff"` shadow, including on an explicit migrations-catalog cache miss, + and tears it down after export. - `supabase/migra` container — the migra OOM bash fallback only. ## API Routes (linked path, via the db-config resolver) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 4058370433..653c13ca20 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -75,7 +75,8 @@ function setup(workdir: string, opts: SetupOpts = {}) { sourceUrl: "postgres://postgres:postgres@127.0.0.1:54320/postgres", }); }, - provisionNextShadow: () => Effect.die("provisionNextShadow not used"), + provisionNextMigrationsShadow: () => Effect.die("next migrations shadow not used"), + provisionNextPlanShadows: () => Effect.die("next plan shadows not used"), removeShadowContainer: (container) => Effect.sync(() => { removedContainers.push(container); diff --git a/apps/cli/src/legacy/commands/db/diff/diff.layers.ts b/apps/cli/src/legacy/commands/db/diff/diff.layers.ts index e53312db93..715e97f993 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.layers.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.layers.ts @@ -49,6 +49,7 @@ const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); const nextShadow = legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seam)); const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(cliConfig), Layer.provide(legacyPgDeltaNextAdapterLayer), Layer.provide(nextShadow), Layer.provide(edgeRuntime), diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 7130ced6d6..2b4d0ea7ff 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -32,7 +32,8 @@ Notes/Delegation section below). - Pg-delta diff and declarative export use the in-process engine bundled into the CLI binary by default. Pg-topo is bundled with it and the version is fixed at CLI build time; the command never downloads it or falls back automatically. -- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy edge-runtime path. +- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy edge-runtime path from + either the shell or project `supabase/.env` (the shell wins). `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs directly below `supabase/.temp/pgdelta/` apply only to that opt-out. - With `PGDELTA_DEBUG`, default-engine diagnostic data is stored under @@ -46,6 +47,9 @@ Notes/Delegation section below). artifacts are saved before policy evaluation when capture is enabled. - New-engine SQL bytes and transaction-split filenames may differ. Successful execution and convergence on a subsequent pull/diff are the contract. +- Nontransactional plan files retain pg-delta's exact first-line + `-- pg-delta: transaction=false` directive. Later push/reset/up commands consume + that durable header to keep the whole file outside a CLI-owned transaction. - Default-engine migration and declarative SQL retains pg-delta's safe compaction and uses its human-facing formatter (lowercase keywords, max width 180). A JSON object in `[experimental.pgdelta].format_options` partially overrides the @@ -81,7 +85,8 @@ Notes/Delegation section below). ## Docker - Edge-runtime container (migra, or pg-delta only under the legacy opt-out). -- Shadow Postgres container (provisioned + torn down via the Go `db __shadow` seam). +- One shadow Postgres container (provisioned + torn down via the Go `db __shadow` + seam); pg-delta next provisions only its migrated shadow for this diff. - `supabase/migra` container — the migra OOM bash fallback only. - `pg_dump` container — the initial-migra pull's native remote-schema dump (`legacyStreamPgDump`, shared with `db dump`). diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 80e1e6531f..d315576c21 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -129,7 +129,8 @@ function setup(workdir: string, opts: SetupOpts = {}) { sourceUrl: "postgres://postgres:postgres@127.0.0.1:54320/postgres", }); }, - provisionNextShadow: () => Effect.die("provisionNextShadow not used"), + provisionNextMigrationsShadow: () => Effect.die("next migrations shadow not used"), + provisionNextPlanShadows: () => Effect.die("next plan shadows not used"), removeShadowContainer: (container) => Effect.sync(() => { removedContainers.push(container); @@ -514,7 +515,7 @@ describe("legacy db pull", () => { { name: "non_transactional", transactionMode: "none", - sql: "-- unit 3\n\ncreate index concurrently i on t (c);", + sql: "-- pg-delta: transaction=false\n-- unit 3\n\ncreate index concurrently i on t (c);", }, ]), yes: true, @@ -533,9 +534,9 @@ describe("legacy db pull", () => { const versions = written.map((f) => f.slice(0, 14)); expect((versions[0] ?? "") < (versions[1] ?? "")).toBe(true); expect((versions[1] ?? "") < (versions[2] ?? "")).toBe(true); - expect(readFileSync(join(dir, written[2] ?? ""), "utf8")).toContain( - "create index concurrently i on t (c);", - ); + const nonTransactional = readFileSync(join(dir, written[2] ?? ""), "utf8"); + expect(nonTransactional.startsWith("-- pg-delta: transaction=false\n")).toBe(true); + expect(nonTransactional).toContain("create index concurrently i on t (c);"); // One "Schema written to" line per unit, each printing the workdir-relative // path (Go's `pull.go:76`), and one history upsert per unit. const err = streamText(s.out, "stderr"); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.layers.ts b/apps/cli/src/legacy/commands/db/pull/pull.layers.ts index 6a9401ac98..a0368dbdae 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.layers.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.layers.ts @@ -41,6 +41,7 @@ const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); const nextShadow = legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seam)); const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(cliConfig), Layer.provide(legacyPgDeltaNextAdapterLayer), Layer.provide(nextShadow), Layer.provide(edgeRuntime), diff --git a/apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md index 58f37c142d..914e12dd2b 100644 --- a/apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md @@ -34,13 +34,13 @@ never read by the default engine. ## Database Mutations -| Statement | When | -| ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -| `RESET ALL` + `BEGIN` … migration statements … `INSERT INTO supabase_migrations.schema_migrations(version, name, statements)` … `COMMIT` | per pending migration (after confirmation); pipeline-incompatible statements run standalone between batches — see Notes | -| `CREATE SCHEMA/TABLE … supabase_migrations.schema_migrations`, `ALTER TABLE … ADD COLUMN …` | once before applying migrations (idempotent) | -| `RESET ALL` + `BEGIN` … roles.sql statements … `COMMIT` (no history row) | per `--include-roles` globals file (after confirmation) | -| `SELECT id, name FROM vault.secrets …`, `SELECT vault.update_secret(...)`, `SELECT vault.create_secret(...)` | when `[db.vault]` has syncable secrets and migrations are applied | -| `CREATE TABLE … supabase_migrations.seed_files`, seed statements, `INSERT … seed_files(path, hash) … ON CONFLICT …` | per pending seed file with `--include-seed` (after confirmation); a dirty seed only refreshes the hash | +| Statement | When | +| ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `RESET ALL` + migration statements + `INSERT INTO supabase_migrations.schema_migrations(version, name, statements)` | per pending migration (after confirmation); normally transaction-batched, except pg-delta `transaction=false` files and standalone-incompatible statements — see Notes | +| `CREATE SCHEMA/TABLE … supabase_migrations.schema_migrations`, `ALTER TABLE … ADD COLUMN …` | once before applying migrations (idempotent) | +| `RESET ALL` + `BEGIN` … roles.sql statements … `COMMIT` (no history row) | per `--include-roles` globals file (after confirmation) | +| `SELECT id, name FROM vault.secrets …`, `SELECT vault.update_secret(...)`, `SELECT vault.create_secret(...)` | when `[db.vault]` has syncable secrets and migrations are applied | +| `CREATE TABLE … supabase_migrations.seed_files`, seed statements, `INSERT … seed_files(path, hash) … ON CONFLICT …` | per pending seed file with `--include-seed` (after confirmation); a dirty seed only refreshes the hash | ## API Routes @@ -125,6 +125,13 @@ stdout is payload-only. A single `result` object is emitted: directly into TS in PR supabase/cli#5671 (landed on develop as `b48fad60`) and back-ported to the pinned `apps/cli-go` oracle under the CLI-1989 parity ruling (2026-07-30). +- **Pg-delta no-transaction files**: an exact first-line + `-- pg-delta: transaction=false` directive is durable execution metadata. Every + statement in that file runs sequentially without a CLI-owned transaction on the + same session, preserving pg-delta's session preamble through its nontransactional + action and generated cleanup. The history row is inserted only after all statements + succeed; failures best-effort `RESET ALL` and leave no history row. Marker-like + comments elsewhere do not change the normal transactional default. - **Migrations catalog cache**: retained only for `SUPABASE_USE_PG_DELTA_NEXT=false` (Go's best-effort `pgcache.TryCacheMigrationsCatalog`). After a successful migration apply, when diff --git a/apps/cli/src/legacy/commands/db/push/push.integration.test.ts b/apps/cli/src/legacy/commands/db/push/push.integration.test.ts index 3abcc35f87..1710b28617 100644 --- a/apps/cli/src/legacy/commands/db/push/push.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/push/push.integration.test.ts @@ -343,6 +343,35 @@ describe("legacy db push", () => { }); }); + it.live("honors pg-delta's no-transaction migration header", () => { + const set = "SET check_function_bodies = off"; + const action = "DROP SUBSCRIPTION app_events"; + const { layer, conn } = setup(tmp.current, { + toml: 'project_id = "test"\n', + files: migrationFile( + "20240101000000", + `-- pg-delta: transaction=false\n${set};\n${action};\nRESET ALL;`, + ), + confirm: [true], + }); + return Effect.gen(function* () { + yield* legacyDbPush(DEFAULT_FLAGS).pipe(Effect.provide(layer)); + const setupCommit = conn.execs.indexOf("COMMIT"); + const setIndex = conn.execs.indexOf(set); + const actionIndex = conn.execs.indexOf(action); + const cleanupIndex = conn.execs.lastIndexOf("RESET ALL"); + + expect(conn.execs.filter((sql) => sql === "BEGIN")).toHaveLength(1); + expect(conn.execs.filter((sql) => sql === "COMMIT")).toHaveLength(1); + expect(setIndex).toBeGreaterThan(setupCommit); + expect(actionIndex).toBeGreaterThan(setIndex); + expect(cleanupIndex).toBeGreaterThan(actionIndex); + expect( + conn.queries.some((query) => query.sql.includes("INSERT INTO supabase_migrations")), + ).toBe(true); + }); + }); + it.live("does not attempt to cache the migrations catalog when pg-delta is disabled", () => { const { layer, out, edgeRunCalls } = setup(tmp.current, { toml: 'project_id = "test"\n', diff --git a/apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md index 8591bad33f..b25b13d2ff 100644 --- a/apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md @@ -72,12 +72,12 @@ remote path produces whatever the delegated Go binary writes. ### Remote path (native, in TS) -| Statement | When | -| ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | -| `drop.sql` `DO` block (drops user schemas/extensions/public objects, truncates auth/migrations) | always, first | -| `SELECT vault.update_secret(...)` / `vault.create_secret(...)` | when `[db.vault]` has syncable secrets | -| migration statements + `schema_migrations` history insert (per file, transactional; pipeline-incompatible statements run standalone — see Notes) | when `[db.migrations].enabled`, for migrations `≤ --version` | -| seed statements + `seed_files` hash upsert | when `[db.seed].enabled` and not `--no-seed` | +| Statement | When | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | +| `drop.sql` `DO` block (drops user schemas/extensions/public objects, truncates auth/migrations) | always, first | +| `SELECT vault.update_secret(...)` / `vault.create_secret(...)` | when `[db.vault]` has syncable secrets | +| migration statements + `schema_migrations` history insert (normally transaction-batched; pg-delta `transaction=false` files run sequentially without a CLI transaction — see Notes) | when `[db.migrations].enabled`, for migrations `≤ --version` | +| seed statements + `seed_files` hash upsert | when `[db.seed].enabled` and not `--no-seed` | ### Local path (native, in TS) @@ -189,6 +189,11 @@ path has no confirmation prompt. standalone outside the per-file transaction batch, with the same non-atomic flush behaviour as `db push` — see `db push`'s SIDE_EFFECTS Notes (supabase/cli#5139, closed Go PR supabase/cli#5156, CLI-1989 parity ruling). +- **Pg-delta no-transaction files** use the same durable first-line + `-- pg-delta: transaction=false` directive as `db push`. The complete file runs + sequentially on one session without a CLI transaction; its history row is written + only after success, and failures best-effort reset the session. See `db push`'s + SIDE_EFFECTS Notes for the shared apply contract. - `--no-seed` forces seeding off (Go sets `Config.Db.Seed.Enabled = false`); on the local path it feeds `legacyResolveResetSeedConfig`, applied on top of the loaded `[db.seed]` config inside the recreate's own `MigrateAndSeed` step (same override diff --git a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts index fd703e747d..0e73206797 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts @@ -1359,6 +1359,34 @@ describe("legacy db reset", () => { }); }); + it.live("honors pg-delta's no-transaction migration header on remote reset", () => { + const set = "SET check_function_bodies = off"; + const action = "DROP SUBSCRIPTION app_events"; + const { layer, conn } = setup(tmp.current, { + toml: 'project_id = "test"\n', + files: migrationFile( + "20240101000000", + `-- pg-delta: transaction=false\n${set};\n${action};\nRESET ALL;`, + ), + confirm: [true], + }); + return Effect.gen(function* () { + yield* legacyDbReset({ ...DEFAULT_FLAGS, linked: true }).pipe(Effect.provide(layer)); + const setupCommit = conn.execs.indexOf("COMMIT"); + const setIndex = conn.execs.indexOf(set); + const actionIndex = conn.execs.indexOf(action); + const cleanupIndex = conn.execs.lastIndexOf("RESET ALL"); + + expect(setIndex).toBeGreaterThan(setupCommit); + expect(actionIndex).toBeGreaterThan(setIndex); + expect(cleanupIndex).toBeGreaterThan(actionIndex); + expect(conn.execs.slice(setIndex, cleanupIndex + 1)).toEqual([set, action, "RESET ALL"]); + expect( + conn.queries.some((query) => query.sql.includes("INSERT INTO supabase_migrations")), + ).toBe(true); + }); + }); + it.live("fails a remote reset before dropping schemas on an undecryptable secret", () => { // Regression: the old point-of-use vault decryption ran AFTER `legacyDropUserSchemas`, // so an undecryptable `encrypted:` secret dropped the schemas before failing. Go runs diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index f85315a3c3..1f417249a1 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -49,7 +49,9 @@ function mockSeam(paths: Record) { }, ensureLocalDatabaseStarted: () => Effect.void, ensureLocalPostgresImageCurrent: () => Effect.void, - provisionNextShadow: () => Effect.die("provisionNextShadow not used in declarative tests"), + provisionNextMigrationsShadow: () => + Effect.die("next migrations shadow not used in declarative tests"), + provisionNextPlanShadows: () => Effect.die("next plan shadows not used in declarative tests"), // The migrations-catalog source now resolves natively (CLI-1959) via // `legacyGetMigrationsCatalogRef`, which provisions its shadow through this // EXISTING `provisionShadow` (Go's unchanged `db __shadow --mode diff`) rather diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index 9895dac8a4..9ed124ed19 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -9,7 +9,8 @@ platform view. into the CLI binary at build time, so the installed CLI fixes their version and performs no runtime package download or automatic legacy fallback. - `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy catalog/edge-runtime - implementation. Only that opt-out uses `supabase/.temp/pgdelta-version`, + implementation from either the shell or project `supabase/.env` (the shell + wins). Only that opt-out uses `supabase/.temp/pgdelta-version`, `PGDELTA_NPM_REGISTRY`, edge-runtime, or legacy catalogs directly below `supabase/.temp/pgdelta/`. - `--no-cache` bypasses legacy catalog reuse/warming. The default engine already diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index 791784be25..fd5603eb91 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -155,7 +155,9 @@ function setup(workdir: string, opts: SetupOpts = {}) { ), ), provisionShadow: () => Effect.die("provisionShadow not used in declarative tests"), - provisionNextShadow: () => Effect.die("provisionNextShadow not used in declarative tests"), + provisionNextMigrationsShadow: () => + Effect.die("next migrations shadow not used in declarative tests"), + provisionNextPlanShadows: () => Effect.die("next plan shadows not used in declarative tests"), removeShadowContainer: () => Effect.void, }); const edgeCalls: LegacyEdgeRuntimeRunOpts[] = []; diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts index 3c0167732f..2e4b21f125 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts @@ -52,6 +52,7 @@ const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); const nextShadow = legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seam)); const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(cliConfig), Layer.provide(legacyPgDeltaNextAdapterLayer), Layer.provide(nextShadow), Layer.provide(edgeRuntime), diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 8f049bc5ae..6d73143879 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -8,7 +8,8 @@ as a new timestamped migration. - The default pg-delta and bundled pg-topo run in-process at the versions fixed when the CLI is built. There is no runtime download or automatic fallback. - `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy catalog/edge-runtime - implementation. `supabase/.temp/pgdelta-version`, `PGDELTA_NPM_REGISTRY`, and + implementation from either the shell or project `supabase/.env` (the shell + wins). `supabase/.temp/pgdelta-version`, `PGDELTA_NPM_REGISTRY`, and catalogs directly below `supabase/.temp/pgdelta/` are legacy-only. - `--no-cache` bypasses legacy catalog reuse/warming. The default engine always extracts current state and maintains no reusable catalog cache. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 32add16120..be1dee6727 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -141,7 +141,9 @@ function setup(workdir: string, opts: SetupOpts = {}) { : Effect.void, ), ), - provisionNextShadow: () => Effect.die("provisionNextShadow not used in declarative tests"), + provisionNextMigrationsShadow: () => + Effect.die("next migrations shadow not used in declarative tests"), + provisionNextPlanShadows: () => Effect.die("next plan shadows not used in declarative tests"), provisionShadow: ({ mode }) => Effect.sync(() => { provisionShadowCalls.push({ mode, rawChunksAt: out.rawChunks.length }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts index e05ef58297..3e19e87dae 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts @@ -50,6 +50,7 @@ const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); const nextShadow = legacyPgDeltaNextShadowLayer.pipe(Layer.provide(seam)); const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(cliConfig), Layer.provide(legacyPgDeltaNextAdapterLayer), Layer.provide(nextShadow), Layer.provide(edgeRuntime), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts index d662e1e7d3..383216398c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts @@ -1,6 +1,8 @@ import { Effect, FileSystem, Layer, Path } from "effect"; import { Output } from "../../../../shared/output/output.service.ts"; +import { LegacyCliConfig } from "../../../config/legacy-cli-config.service.ts"; +import { legacyLoadProjectEnv } from "../../../shared/legacy-db-config.toml-read.ts"; import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; import { legacyPgDeltaLegacyEngineLayer } from "./legacy-pgdelta-engine.legacy.layer.ts"; import { legacyPgDeltaNextEngineLayer } from "./legacy-pgdelta-engine.next.layer.ts"; @@ -41,10 +43,16 @@ export function legacyPgDeltaEngineSelectorLayer( ); } -/** Reads the rollout flag once when the command-scoped layer is constructed. */ +/** Resolves the rollout flag once when the command-scoped layer is constructed. */ export const legacyPgDeltaEngineLayer = Layer.unwrap( Effect.gen(function* () { - const raw = process.env[FLAG]; + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const cliConfig = yield* LegacyCliConfig; + const projectEnv = yield* legacyLoadProjectEnv(fs, path, cliConfig.workdir); + // godotenv.Load never replaces a shell value, including an empty or invalid + // one, so presence in process.env must suppress the project-file fallback. + const raw = process.env[FLAG] ?? projectEnv[FLAG]; const implementation = yield* resolveAndLog(raw); return selectProductionLayer(implementation); }), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts index ce9fb08d30..10955fd28c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts @@ -1,8 +1,14 @@ +import { mkdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; import { Effect, Exit, Layer } from "effect"; import * as BunServices from "@effect/platform-bun/BunServices"; import { it } from "@effect/vitest"; import { afterEach, describe, expect } from "vitest"; +import { + mockLegacyCliConfig, + useLegacyTempWorkdir, +} from "../../../../../tests/helpers/legacy-mocks.ts"; import { mockOutput } from "../../../../../tests/helpers/mocks.ts"; import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; import { LegacyEdgeRuntimeScript } from "../../../shared/legacy-edge-runtime-script.service.ts"; @@ -52,7 +58,8 @@ const unusedLegacyRuntime = Layer.mergeAll( ensureLocalDatabaseStarted: () => Effect.die("local start not needed"), ensureLocalPostgresImageCurrent: () => Effect.die("image check not needed"), provisionShadow: () => Effect.die("shadow not needed"), - provisionNextShadow: () => Effect.die("next shadow not needed"), + provisionNextMigrationsShadow: () => Effect.die("next migrations shadow not needed"), + provisionNextPlanShadows: () => Effect.die("next plan shadows not needed"), removeShadowContainer: () => Effect.die("cleanup not needed"), }), Layer.succeed(LegacyPgDeltaNextAdapter, { @@ -62,7 +69,8 @@ const unusedLegacyRuntime = Layer.mergeAll( captureSnapshot: () => Effect.die("adapter not needed"), }), Layer.succeed(LegacyPgDeltaNextShadow, { - provision: () => Effect.die("next shadow not needed"), + provisionMigrations: () => Effect.die("next migrations shadow not needed"), + provisionPlan: () => Effect.die("next plan shadows not needed"), }), mockOutput().layer, ); @@ -171,10 +179,63 @@ describe("legacyPgDeltaEngineSelectorLayer", () => { }); describe("legacyPgDeltaEngineLayer", () => { + const tmp = useLegacyTempWorkdir("pgdelta-engine-selector-"); + afterEach(() => { delete process.env[FLAG]; }); + const provideProductionSelector = (messages: Array) => + legacyPgDeltaEngineLayer.pipe( + Layer.provide(unusedLegacyRuntime), + Layer.provide(debugLayer(messages)), + Layer.provide(mockLegacyCliConfig({ workdir: tmp.current })), + ); + + const writeProjectFlag = (value: string) => { + mkdirSync(join(tmp.current, "supabase"), { recursive: true }); + writeFileSync(join(tmp.current, "supabase", ".env"), `${FLAG}=${value}\n`); + }; + + it.effect("selects legacy from the project environment when the shell is unset", () => { + const messages: Array = []; + writeProjectFlag("false"); + + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + expect(engine.implementation).toBe("legacy"); + expect(messages).toEqual(["Using pg-delta legacy implementation."]); + }).pipe(Effect.provide(provideProductionSelector(messages))); + }); + + it.effect("prefers a true shell value over a false project value", () => { + const messages: Array = []; + process.env[FLAG] = "true"; + writeProjectFlag("false"); + + return Effect.gen(function* () { + expect((yield* LegacyPgDeltaEngine).implementation).toBe("next"); + }).pipe(Effect.provide(provideProductionSelector(messages))); + }); + + it.effect("prefers a false shell value over a true project value", () => { + const messages: Array = []; + process.env[FLAG] = "false"; + writeProjectFlag("true"); + + return Effect.gen(function* () { + expect((yield* LegacyPgDeltaEngine).implementation).toBe("legacy"); + }).pipe(Effect.provide(provideProductionSelector(messages))); + }); + + it.effect("defaults to next when neither environment defines the flag", () => { + const messages: Array = []; + return Effect.gen(function* () { + expect((yield* LegacyPgDeltaEngine).implementation).toBe("next"); + expect(messages).toEqual(["Using pg-delta next implementation."]); + }).pipe(Effect.provide(provideProductionSelector(messages))); + }); + it.effect("reads the environment once for the command-scoped service", () => { const messages: Array = []; process.env[FLAG] = "false"; @@ -187,13 +248,6 @@ describe("legacyPgDeltaEngineLayer", () => { expect(first).toBe(second); expect(second.implementation).toBe("legacy"); expect(messages).toEqual(["Using pg-delta legacy implementation."]); - }).pipe( - Effect.provide( - legacyPgDeltaEngineLayer.pipe( - Layer.provide(unusedLegacyRuntime), - Layer.provide(debugLayer(messages)), - ), - ), - ); + }).pipe(Effect.provide(provideProductionSelector(messages))); }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts new file mode 100644 index 0000000000..e6ce4dce24 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts @@ -0,0 +1,156 @@ +import * as BunServices from "@effect/platform-bun/BunServices"; +import { describe, expect, it } from "@effect/vitest"; +import { Effect, Layer } from "effect"; + +import { mockOutput } from "../../../../../tests/helpers/mocks.ts"; +import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; +import { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; +import { legacyPgDeltaNextEngineLayer } from "./legacy-pgdelta-engine.next.layer.ts"; +import { LegacyPgDeltaEngine, LegacyPgDeltaEngineError } from "./legacy-pgdelta-engine.service.ts"; +import { LegacyPgDeltaNextAdapter } from "./legacy-pgdelta-next-adapter.service.ts"; +import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; + +const common = { + context: { projectId: "test", cwd: "/tmp/test", npmVersion: undefined, denoVersion: 2 }, + schema: ["public"], + formatOptions: "", + debug: false, + strictCoverage: false, +} as const; + +function setup() { + const state = { migrations: 0, plan: 0 }; + const shadow = Layer.succeed(LegacyPgDeltaNextShadow, { + provisionMigrations: () => + Effect.sync(() => { + state.migrations += 1; + }).pipe( + Effect.andThen( + Effect.fail(new LegacyDeclarativeShadowDbError({ message: "stop after routing" })), + ), + ), + provisionPlan: () => + Effect.sync(() => { + state.plan += 1; + }).pipe( + Effect.andThen( + Effect.fail(new LegacyDeclarativeShadowDbError({ message: "stop after routing" })), + ), + ), + }); + const unusedAdapter = Layer.succeed(LegacyPgDeltaNextAdapter, { + diff: () => Effect.die("adapter not used"), + exportDeclarativeSchema: () => Effect.die("adapter not used"), + planDeclarativeSchema: () => Effect.die("adapter not used"), + captureSnapshot: () => Effect.die("adapter not used"), + }); + const debug = Layer.succeed(LegacyDebugLogger, { + debug: () => Effect.void, + http: () => Effect.void, + }); + const dependencies = Layer.mergeAll( + BunServices.layer, + shadow, + unusedAdapter, + debug, + mockOutput().layer, + ); + return { + state, + layer: legacyPgDeltaNextEngineLayer.pipe(Layer.provide(dependencies)), + }; +} + +describe("pg-delta next shadow selection", () => { + it.effect("uses only the migrated shadow for database diffs", () => { + const { state, layer } = setup(); + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + yield* engine + .diffDatabase({ + ...common, + target: { + kind: "database", + ref: "postgresql://postgres@localhost/postgres", + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + }) + .pipe(Effect.exit); + + expect(state).toEqual({ migrations: 1, plan: 0 }); + }).pipe(Effect.provide(layer)); + }); + + it.effect("uses only the migrated shadow for explicit migrations diffs", () => { + const { state, layer } = setup(); + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + yield* engine + .diffExplicit({ + ...common, + source: { kind: "migrations", projectRef: "linked-project" }, + desired: { + kind: "database", + ref: "postgresql://postgres@localhost/postgres", + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + }) + .pipe(Effect.exit); + + expect(state).toEqual({ migrations: 1, plan: 0 }); + }).pipe(Effect.provide(layer)); + }); + + it.effect("uses both isolated shadows for declarative plans", () => { + const { state, layer } = setup(); + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + yield* engine + .planDeclarativeSchema({ + ...common, + files: [{ name: "schema.sql", sql: "create table example(id int);" }], + noCache: false, + setupInputs: { + image: "postgres:17", + majorVersion: 17, + authEnabled: true, + storageEnabled: true, + realtimeEnabled: true, + autoExpose: false, + vaultNames: [], + rolesSql: "", + }, + }) + .pipe(Effect.exit); + + expect(state).toEqual({ migrations: 0, plan: 1 }); + }).pipe(Effect.provide(layer)); + }); + + it.effect("returns malformed explicit URLs as typed failures rather than defects", () => { + const { state, layer } = setup(); + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + const error = yield* engine + .diffExplicit({ + ...common, + source: { + kind: "database", + ref: "postgresql://postgres:source-secret@[/postgres", + connectOptions: { isLocal: false, dnsResolver: "native" }, + }, + desired: { + kind: "database", + ref: "postgresql://postgres:desired-secret@[/postgres", + connectOptions: { isLocal: false, dnsResolver: "native" }, + }, + }) + .pipe(Effect.flip); + + expect(error).toBeInstanceOf(LegacyPgDeltaEngineError); + expect(String(error.cause)).not.toContain("source-secret"); + expect(String(error.cause)).not.toContain("desired-secret"); + expect(state).toEqual({ migrations: 0, plan: 0 }); + }).pipe(Effect.provide(layer)); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 60c727dd93..af5c220c74 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -98,13 +98,17 @@ function normalizeNextDiff( }; } -function parseEndpoint(endpoint: LegacyPgDeltaDatabaseEndpoint) { - if (endpoint.connection !== undefined) return endpoint.connection; - const parsed = parseLegacyConnectionString(endpoint.ref); - if (parsed !== undefined) return parsed; - throw new LegacyPgDeltaEngineError({ - message: "failed to parse Postgres connection string for pg-delta", - cause: endpoint.ref.replace(/:[^:@/]+@/, ":***@"), +export function legacyParsePgDeltaNextEndpoint(endpoint: LegacyPgDeltaDatabaseEndpoint) { + return Effect.gen(function* () { + if (endpoint.connection !== undefined) return endpoint.connection; + const parsed = parseLegacyConnectionString(endpoint.ref); + if (parsed !== undefined) return parsed; + return yield* Effect.fail( + new LegacyPgDeltaEngineError({ + message: "failed to parse Postgres connection string for pg-delta", + cause: endpoint.ref.replace(/:[^:@/]+@/, ":***@"), + }), + ); }); } @@ -154,7 +158,9 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( ); const acquireDatabase = (endpoint: LegacyPgDeltaDatabaseEndpoint) => - legacyAcquirePgPool(parseEndpoint(endpoint), endpoint.connectOptions); + legacyParsePgDeltaNextEndpoint(endpoint).pipe( + Effect.flatMap((connection) => legacyAcquirePgPool(connection, endpoint.connectOptions)), + ); const reportDiagnostics = ( operation: LegacyPgDeltaNextOperation, @@ -182,9 +188,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diffExplicit: (input) => Effect.scoped( Effect.gen(function* () { - let shadow: - | { readonly migrationsUrl: string; readonly declarativeUrl: string } - | undefined; + let shadow: { readonly migrationsUrl: string } | undefined; const migrationsEndpoint = input.source.kind === "migrations" ? input.source @@ -192,7 +196,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( ? input.desired : undefined; if (migrationsEndpoint !== undefined) { - shadow = yield* shadowService.provision({ + shadow = yield* shadowService.provisionMigrations({ schema: input.schema, ...(migrationsEndpoint.projectRef !== undefined ? { projectRef: migrationsEndpoint.projectRef } @@ -245,7 +249,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( diffDatabase: (input) => Effect.scoped( Effect.gen(function* () { - const shadow = yield* shadowService.provision({ + const shadow = yield* shadowService.provisionMigrations({ schema: input.schema, ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), }); @@ -316,7 +320,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( planDeclarativeSchema: (input) => Effect.scoped( Effect.gen(function* () { - const shadow = yield* shadowService.provision({ schema: input.schema }); + const shadow = yield* shadowService.provisionPlan({ schema: input.schema }); const migrations = parseLegacyConnectionString(shadow.migrationsUrl); const declarative = parseLegacyConnectionString(shadow.declarativeUrl); if (migrations === undefined || declarative === undefined) { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts index 7b8921c48e..20fc56e16b 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts @@ -1,6 +1,12 @@ +import { Effect } from "effect"; import { describe, expect, it } from "vitest"; -import { legacyPgDeltaNextIsolatedShadowPlanOptions } from "./legacy-pgdelta-engine.next.layer.ts"; +import type { LegacyPgDeltaDatabaseEndpoint } from "./legacy-pgdelta-engine.service.ts"; +import { + legacyParsePgDeltaNextEndpoint, + legacyPgDeltaNextIsolatedShadowPlanOptions, +} from "./legacy-pgdelta-engine.next.layer.ts"; +import { LegacyPgDeltaEngineError } from "./legacy-pgdelta-engine.service.ts"; describe("legacyPgDeltaNextIsolatedShadowPlanOptions", () => { it("uses the isolated full-baseline mode shared by both declarative planner entrypoints", () => { @@ -10,3 +16,37 @@ describe("legacyPgDeltaNextIsolatedShadowPlanOptions", () => { }); }); }); + +describe("legacyParsePgDeltaNextEndpoint", () => { + it("fails malformed explicit URLs through the typed error channel and redacts passwords", () => { + const endpoint = { + kind: "database", + ref: "postgresql://postgres:supersecret@[/postgres", + connectOptions: { isLocal: false, dnsResolver: "native" }, + } satisfies LegacyPgDeltaDatabaseEndpoint; + + const error = Effect.runSync(legacyParsePgDeltaNextEndpoint(endpoint).pipe(Effect.flip)); + + expect(error).toBeInstanceOf(LegacyPgDeltaEngineError); + expect(error.message).toBe("failed to parse Postgres connection string for pg-delta"); + expect(error.cause).toBe("postgresql://postgres:***@[/postgres"); + }); + + it("uses a supplied parsed connection without reparsing the display ref", () => { + const connection = { + host: "localhost", + port: 5432, + user: "postgres", + password: "secret", + database: "postgres", + }; + const endpoint = { + kind: "database", + ref: "malformed-display-ref", + connection, + connectOptions: { isLocal: true, dnsResolver: "native" }, + } satisfies LegacyPgDeltaDatabaseEndpoint; + + expect(Effect.runSync(legacyParsePgDeltaNextEndpoint(endpoint))).toBe(connection); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index 63ca24a5e9..b813fd8174 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -2,7 +2,8 @@ import { Effect, Layer } from "effect"; import { LegacyPgDeltaNextShadow, - type LegacyPgDeltaNextShadowDatabases, + type LegacyPgDeltaNextMigrationsShadow, + type LegacyPgDeltaNextPlanShadows, } from "./legacy-pgdelta-next-shadow.service.ts"; import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; @@ -17,12 +18,19 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( const seam = yield* LegacyDeclarativeSeam; return LegacyPgDeltaNextShadow.of({ - provision: ({ schema, projectRef }) => + provisionMigrations: ({ schema, projectRef }) => Effect.gen(function* () { - return (yield* seam.provisionNextShadow({ + return (yield* seam.provisionNextMigrationsShadow({ schema, ...(projectRef !== undefined ? { projectRef } : {}), - })) satisfies LegacyPgDeltaNextShadowDatabases; + })) satisfies LegacyPgDeltaNextMigrationsShadow; + }), + provisionPlan: ({ schema, projectRef }) => + Effect.gen(function* () { + return (yield* seam.provisionNextPlanShadows({ + schema, + ...(projectRef !== undefined ? { projectRef } : {}), + })) satisfies LegacyPgDeltaNextPlanShadows; }), }); }), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts index 24505e8725..c0e5cb5010 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts @@ -2,27 +2,39 @@ import { Context, type Effect, type Scope } from "effect"; import type { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; -/** The two live databases needed to plan with the bundled pg-delta next engine. */ -export interface LegacyPgDeltaNextShadowDatabases { +/** The live migrated database needed by pg-delta next database diffs. */ +export interface LegacyPgDeltaNextMigrationsShadow { /** Platform baseline with the project's local migrations applied. */ readonly migrationsUrl: string; +} + +/** The two live databases needed to plan declarative SQL with pg-delta next. */ +export interface LegacyPgDeltaNextPlanShadows extends LegacyPgDeltaNextMigrationsShadow { /** Independent platform baseline owned by `planSchemaFiles` while loading desired SQL. */ readonly declarativeUrl: string; } interface LegacyPgDeltaNextShadowShape { /** - * Provisions both next-engine shadow containers and owns them for the current - * Effect scope. Both containers are removed when that scope closes. + * Provisions only the migrated next-engine shadow needed by database diffs. + * The container is removed when the current Effect scope closes. */ - readonly provision: (opts: { + readonly provisionMigrations: (opts: { readonly schema: ReadonlyArray; readonly projectRef?: string; }) => Effect.Effect< - LegacyPgDeltaNextShadowDatabases, + LegacyPgDeltaNextMigrationsShadow, LegacyDeclarativeShadowDbError, Scope.Scope >; + /** + * Provisions the independent migrated and declarative shadows needed by a + * declarative plan. Both are removed when the current Effect scope closes. + */ + readonly provisionPlan: (opts: { + readonly schema: ReadonlyArray; + readonly projectRef?: string; + }) => Effect.Effect; } export class LegacyPgDeltaNextShadow extends Context.Service< diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts index 97ff6dba2e..f20f326519 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.unit.test.ts @@ -3,13 +3,16 @@ import { Effect, Layer } from "effect"; import { legacyPgDeltaNextShadowLayer } from "./legacy-pgdelta-next-shadow.layer.ts"; import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; -import { legacyParseNextShadowProtocol } from "./legacy-pgdelta.seam.layer.ts"; +import { + legacyParseNextMigrationsShadowProtocol, + legacyParseNextPlanShadowProtocol, +} from "./legacy-pgdelta.seam.layer.ts"; import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; function setup() { const state = { - provisionCalls: [] as object[], - legacyMethodCalls: [] as string[], + migrationsCalls: [] as object[], + planCalls: [] as object[], }; const seamLayer = Layer.succeed( LegacyDeclarativeSeam, @@ -18,9 +21,16 @@ function setup() { ensureLocalDatabaseStarted: () => Effect.die("ensureLocalDatabaseStarted not used"), ensureLocalPostgresImageCurrent: () => Effect.die("ensureLocalPostgresImageCurrent not used"), provisionShadow: () => Effect.die("provisionShadow not used"), - provisionNextShadow: (input) => + provisionNextMigrationsShadow: (input) => Effect.sync(() => { - state.provisionCalls.push(input); + state.migrationsCalls.push(input); + return { + migrationsUrl: "postgresql://postgres:secret@localhost:55432/postgres", + }; + }), + provisionNextPlanShadows: (input) => + Effect.sync(() => { + state.planCalls.push(input); return { migrationsUrl: "postgresql://postgres:secret@localhost:55432/postgres", declarativeUrl: "postgresql://postgres:secret@localhost:55433/postgres", @@ -37,69 +47,79 @@ function setup() { } describe("LegacyPgDeltaNextShadow", () => { - it("validates the dual-shadow JSON protocol structurally", () => { - expect( - legacyParseNextShadowProtocol( - JSON.stringify({ - migrations: { - containerId: "migrations-container", - url: "postgresql://postgres@localhost:55432/postgres", - }, - declarative: { - containerId: "declarative-container", - url: "postgresql://postgres@localhost:55433/postgres", - }, - }), - ), - ).toEqual({ - migrations: { - containerId: "migrations-container", - url: "postgresql://postgres@localhost:55432/postgres", - }, - declarative: { - containerId: "declarative-container", - url: "postgresql://postgres@localhost:55433/postgres", - }, + it("validates the mode-specific JSON protocols structurally", () => { + const migrations = { + containerId: "migrations-container", + url: "postgresql://postgres@localhost:55432/postgres", + }; + const declarative = { + containerId: "declarative-container", + url: "postgresql://postgres@localhost:55433/postgres", + }; + + expect(legacyParseNextMigrationsShadowProtocol(JSON.stringify({ migrations }))).toEqual({ + migrations, + }); + expect(legacyParseNextPlanShadowProtocol(JSON.stringify({ migrations, declarative }))).toEqual({ + migrations, + declarative, }); - expect(() => legacyParseNextShadowProtocol("not json")).toThrow(); - expect(() => legacyParseNextShadowProtocol('{"migrations":{}}')).toThrow(); + expect(() => legacyParseNextMigrationsShadowProtocol("not json")).toThrow(); + expect(() => legacyParseNextMigrationsShadowProtocol('{"migrations":{}}')).toThrow(); expect(() => - legacyParseNextShadowProtocol( + legacyParseNextMigrationsShadowProtocol(JSON.stringify({ migrations, declarative })), + ).toThrow("unexpected declarative database"); + expect(() => legacyParseNextPlanShadowProtocol(JSON.stringify({ migrations }))).toThrow(); + expect(() => + legacyParseNextPlanShadowProtocol( JSON.stringify({ - migrations: { containerId: "same", url: "postgresql://localhost/postgres" }, - declarative: { containerId: "same", url: "postgresql://localhost/postgres" }, + migrations, + declarative: { ...declarative, containerId: migrations.containerId }, }), ), ).toThrow("next-shadow containers must be distinct"); }); - it.effect("delegates to the isolated next-shadow seam and exposes both postgres URLs", () => { + it.effect("provisions only the migrated database for a database diff", () => { const { layer, state } = setup(); return Effect.gen(function* () { const databases = yield* Effect.scoped( Effect.gen(function* () { const shadow = yield* LegacyPgDeltaNextShadow; - return yield* shadow.provision({ - schema: ["public", "extensions"], + return yield* shadow.provisionMigrations({ + schema: ["public"], projectRef: "linked-project", }); }), ); + expect(databases).toEqual({ + migrationsUrl: "postgresql://postgres:secret@localhost:55432/postgres", + }); + expect(state.migrationsCalls).toEqual([{ schema: ["public"], projectRef: "linked-project" }]); + expect(state.planCalls).toEqual([]); + }).pipe(Effect.provide(layer)); + }); + + it.effect("provisions both isolated databases for a declarative plan", () => { + const { layer, state } = setup(); + + return Effect.gen(function* () { + const databases = yield* Effect.scoped( + Effect.gen(function* () { + const shadow = yield* LegacyPgDeltaNextShadow; + return yield* shadow.provisionPlan({ schema: ["public", "extensions"] }); + }), + ); + expect(databases).toEqual({ migrationsUrl: "postgresql://postgres:secret@localhost:55432/postgres", declarativeUrl: "postgresql://postgres:secret@localhost:55433/postgres", }); - expect(Object.keys(databases)).toEqual(["migrationsUrl", "declarativeUrl"]); - expect(state.provisionCalls).toEqual([ - { - schema: ["public", "extensions"], - projectRef: "linked-project", - }, - ]); - expect(state.legacyMethodCalls).toEqual([]); + expect(state.migrationsCalls).toEqual([]); + expect(state.planCalls).toEqual([{ schema: ["public", "extensions"] }]); }).pipe(Effect.provide(layer)); }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts index 0e29d3f144..13c0cd9e06 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts @@ -16,7 +16,8 @@ import { import { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; import { LegacyDeclarativeSeam, - type LegacyNextShadowSource, + type LegacyNextMigrationsShadowSource, + type LegacyNextPlanShadowSource, type LegacyShadowSource, } from "./legacy-pgdelta.seam.service.ts"; import { legacyInjectPostgresPassword } from "./legacy-pgdelta.seam.url.ts"; @@ -61,6 +62,113 @@ const makeLegacyDeclarativeSeam = (resolved: BinaryResolution) => }).pipe(Effect.ignore); }); + const provisionNextShadow = ( + mode: "pgdelta-next-migrations" | "pgdelta-next-plan", + opts: { readonly schema: ReadonlyArray; readonly projectRef?: string }, + parseProtocol: (line: string) => Protocol, + containerIds: (protocol: Protocol) => ReadonlyArray, + injectPassword: (protocol: Protocol, password: string) => Source, + ) => + Effect.gen(function* () { + if (!("found" in resolved)) { + return yield* Effect.fail( + new LegacyDeclarativeShadowDbError({ + message: + "Could not find the supabase-go binary required to provision the shadow databases.", + }), + ); + } + + // Keep the process in a nested scope. Until `ack\n`, Go owns every + // container, so parsing/password failures and interruption close the + // child and let its deferred cleanup run. The caller scope receives all + // Docker finalizers before ownership is acknowledged. + const ownerScope = yield* Effect.scope; + return yield* Effect.scoped( + Effect.gen(function* () { + const args = [ + "db", + "__shadow", + "--mode", + mode, + ...(opts.schema.length > 0 ? ["--schema", opts.schema.join(",")] : []), + ...(Option.isSome(networkId) ? ["--network-id", networkId.value] : []), + ...(opts.projectRef !== undefined ? ["--project-ref", opts.projectRef] : []), + ...profileArgs, + ]; + const command = ChildProcess.make(resolved.found, args, { + cwd: cliConfig.workdir, + stdin: "pipe", + stdout: "pipe", + stderr: "inherit", + extendEnv: true, + env: { SUPABASE_TELEMETRY_DISABLED: "1" }, + detached: false, + }); + const handle = yield* spawner.spawn(command).pipe( + Effect.mapError( + () => + new LegacyDeclarativeShadowDbError({ + message: "failed to run the shadow-database provisioner (supabase-go).", + }), + ), + ); + // `runHead` returns as soon as the newline-delimited JSON object is + // emitted; waiting for stdout EOF would deadlock because Go waits + // for the acknowledgment before exiting. + const line = yield* handle.stdout.pipe( + Stream.decodeText, + Stream.splitLines, + Stream.runHead, + Effect.mapError(() => failure()), + ); + if (Option.isNone(line)) { + return yield* Effect.fail(failure()); + } + const protocol = yield* Effect.try({ + try: () => parseProtocol(line.value), + catch: () => failure(), + }); + const password = yield* legacyReadDbToml( + fs, + path, + cliConfig.workdir, + opts.projectRef, + ).pipe( + Effect.map((toml) => toml.password), + Effect.mapError( + () => + new LegacyDeclarativeShadowDbError({ + message: + "failed to read the local database password from config.toml to connect to the shadow databases.", + }), + ), + ); + const databases = yield* Effect.try({ + try: () => injectPassword(protocol, password), + catch: () => failure(), + }); + + for (const containerId of containerIds(protocol)) { + yield* Scope.addFinalizer( + ownerScope, + removeShadowContainer(containerId).pipe(Effect.ignoreCause), + ); + } + yield* Stream.make("ack\n").pipe( + Stream.encodeText, + Stream.run(handle.stdin), + Effect.mapError(() => failure()), + ); + const exitCode = yield* handle.exitCode.pipe(Effect.mapError(() => failure())); + if (exitCode !== 0) { + return yield* Effect.fail(failure(exitCode)); + } + return databases; + }), + ); + }); + return LegacyDeclarativeSeam.of({ exportCatalog: ({ mode, noCache, projectRef }) => Effect.scoped( @@ -471,115 +579,29 @@ const makeLegacyDeclarativeSeam = (resolved: BinaryResolution) => } satisfies LegacyShadowSource; }), ), - provisionNextShadow: ({ schema, projectRef }) => - Effect.gen(function* () { - if (!("found" in resolved)) { - return yield* Effect.fail( - new LegacyDeclarativeShadowDbError({ - message: - "Could not find the supabase-go binary required to provision the shadow databases.", - }), - ); - } - - // Keep the process in a nested scope. Until `ack\n`, Go owns both - // containers, so parsing/password failures and interruption close the - // child and let its deferred cleanup run. The caller scope receives - // both Docker finalizers before ownership is acknowledged. - const ownerScope = yield* Effect.scope; - return yield* Effect.scoped( - Effect.gen(function* () { - const args = [ - "db", - "__shadow", - "--mode", - "pgdelta-next", - ...(schema.length > 0 ? ["--schema", schema.join(",")] : []), - ...(Option.isSome(networkId) ? ["--network-id", networkId.value] : []), - ...(projectRef !== undefined ? ["--project-ref", projectRef] : []), - ...profileArgs, - ]; - const command = ChildProcess.make(resolved.found, args, { - cwd: cliConfig.workdir, - stdin: "pipe", - stdout: "pipe", - stderr: "inherit", - extendEnv: true, - env: { SUPABASE_TELEMETRY_DISABLED: "1" }, - detached: false, - }); - const handle = yield* spawner.spawn(command).pipe( - Effect.mapError( - () => - new LegacyDeclarativeShadowDbError({ - message: "failed to run the shadow-database provisioner (supabase-go).", - }), - ), - ); - // `runHead` returns as soon as the newline-delimited JSON object is - // emitted; waiting for stdout EOF would deadlock because Go waits - // for the acknowledgment before exiting. - const line = yield* handle.stdout.pipe( - Stream.decodeText, - Stream.splitLines, - Stream.runHead, - Effect.mapError(() => failure()), - ); - if (Option.isNone(line)) { - return yield* Effect.fail(failure()); - } - const protocol = yield* Effect.try({ - try: () => legacyParseNextShadowProtocol(line.value), - catch: () => failure(), - }); - const password = yield* legacyReadDbToml( - fs, - path, - cliConfig.workdir, - projectRef, - ).pipe( - Effect.map((toml) => toml.password), - Effect.mapError( - () => - new LegacyDeclarativeShadowDbError({ - message: - "failed to read the local database password from config.toml to connect to the shadow databases.", - }), - ), - ); - const databases = yield* Effect.try({ - try: () => - ({ - migrationsUrl: legacyInjectPostgresPassword(protocol.migrations.url, password), - declarativeUrl: legacyInjectPostgresPassword( - protocol.declarative.url, - password, - ), - }) satisfies LegacyNextShadowSource, - catch: () => failure(), - }); - - yield* Scope.addFinalizer( - ownerScope, - removeShadowContainer(protocol.migrations.containerId).pipe(Effect.ignoreCause), - ); - yield* Scope.addFinalizer( - ownerScope, - removeShadowContainer(protocol.declarative.containerId).pipe(Effect.ignoreCause), - ); - yield* Stream.make("ack\n").pipe( - Stream.encodeText, - Stream.run(handle.stdin), - Effect.mapError(() => failure()), - ); - const exitCode = yield* handle.exitCode.pipe(Effect.mapError(() => failure())); - if (exitCode !== 0) { - return yield* Effect.fail(failure(exitCode)); - } - return databases; - }), - ); - }), + provisionNextMigrationsShadow: ({ schema, projectRef }) => + provisionNextShadow( + "pgdelta-next-migrations", + { schema, ...(projectRef !== undefined ? { projectRef } : {}) }, + legacyParseNextMigrationsShadowProtocol, + (protocol) => [protocol.migrations.containerId], + (protocol, password) => + ({ + migrationsUrl: legacyInjectPostgresPassword(protocol.migrations.url, password), + }) satisfies LegacyNextMigrationsShadowSource, + ), + provisionNextPlanShadows: ({ schema, projectRef }) => + provisionNextShadow( + "pgdelta-next-plan", + { schema, ...(projectRef !== undefined ? { projectRef } : {}) }, + legacyParseNextPlanShadowProtocol, + (protocol) => [protocol.migrations.containerId, protocol.declarative.containerId], + (protocol, password) => + ({ + migrationsUrl: legacyInjectPostgresPassword(protocol.migrations.url, password), + declarativeUrl: legacyInjectPostgresPassword(protocol.declarative.url, password), + }) satisfies LegacyNextPlanShadowSource, + ), removeShadowContainer, }); }); @@ -648,13 +670,29 @@ interface LegacyNextShadowProtocolDatabase { readonly url: string; } -interface LegacyNextShadowProtocol { +interface LegacyNextMigrationsShadowProtocol { readonly migrations: LegacyNextShadowProtocolDatabase; +} + +interface LegacyNextPlanShadowProtocol extends LegacyNextMigrationsShadowProtocol { readonly declarative: LegacyNextShadowProtocolDatabase; } -/** Strict structural validation for the Go next-shadow ownership protocol. */ -export function legacyParseNextShadowProtocol(line: string): LegacyNextShadowProtocol { +/** Strict validation for the Go migrations-only next-shadow ownership protocol. */ +export function legacyParseNextMigrationsShadowProtocol( + line: string, +): LegacyNextMigrationsShadowProtocol { + const parsed: unknown = JSON.parse(line); + if (!isJsonRecord(parsed)) throw new Error("invalid next-shadow protocol"); + const migrations = parseNextShadowProtocolDatabase(parsed["migrations"]); + if ("declarative" in parsed) { + throw new Error("unexpected declarative database in migrations-only next-shadow protocol"); + } + return { migrations }; +} + +/** Strict validation for the Go dual-database next-shadow ownership protocol. */ +export function legacyParseNextPlanShadowProtocol(line: string): LegacyNextPlanShadowProtocol { const parsed: unknown = JSON.parse(line); if (!isJsonRecord(parsed)) throw new Error("invalid next-shadow protocol"); const migrations = parseNextShadowProtocolDatabase(parsed["migrations"]); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.unit.test.ts index 004d8ade19..e83fdaa129 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.unit.test.ts @@ -144,7 +144,7 @@ describe("LegacyDeclarativeSeam next shadow protocol", () => { const exit = yield* Effect.scoped( Effect.gen(function* () { const seam = yield* LegacyDeclarativeSeam; - const databases = yield* seam.provisionNextShadow({ + const databases = yield* seam.provisionNextPlanShadows({ schema: ["public", "extensions"], projectRef: "linked-project", }); @@ -166,7 +166,7 @@ describe("LegacyDeclarativeSeam next shadow protocol", () => { "db", "__shadow", "--mode", - "pgdelta-next", + "pgdelta-next-plan", "--schema", "public,extensions", "--network-id", @@ -183,13 +183,41 @@ describe("LegacyDeclarativeSeam next shadow protocol", () => { }).pipe(Effect.provide(layer)); }); + it.effect("owns and cleans only the migrated container for a database diff", () => { + const migrationsProtocol = JSON.stringify({ + migrations: { + containerId: "migrations-container", + url: "postgresql://postgres@localhost:55432/postgres", + }, + }); + const { layer, state } = setup({ stdout: migrationsProtocol }); + return Effect.gen(function* () { + const exit = yield* Effect.scoped( + Effect.gen(function* () { + const seam = yield* LegacyDeclarativeSeam; + const database = yield* seam.provisionNextMigrationsShadow({ schema: ["public"] }); + expect(state.stdin).toBe("ack\n"); + expect(database).toEqual({ + migrationsUrl: "postgresql://postgres:postgres@localhost:55432/postgres", + }); + return yield* Effect.fail("caller failed"); + }), + ).pipe(Effect.exit); + + expect(exit._tag).toBe("Failure"); + expect(state.cleanupAttempts).toEqual(["migrations-container"]); + expect(state.childScopeClosed).toBe(1); + expect(state.commands[0]?.args).toContain("pgdelta-next-migrations"); + }).pipe(Effect.provide(layer)); + }); + it.effect("has both cleanup finalizers installed when the ack write is interrupted", () => { const { layer, state } = setup({ interruptOnAck: true }); return Effect.gen(function* () { yield* Effect.scoped( Effect.gen(function* () { const seam = yield* LegacyDeclarativeSeam; - yield* seam.provisionNextShadow({ schema: [] }); + yield* seam.provisionNextPlanShadows({ schema: [] }); }), ).pipe(Effect.exit); expect(state.stdin).toBe("ack\n"); @@ -204,7 +232,7 @@ describe("LegacyDeclarativeSeam next shadow protocol", () => { yield* Effect.scoped( Effect.gen(function* () { const seam = yield* LegacyDeclarativeSeam; - yield* seam.provisionNextShadow({ schema: [] }); + yield* seam.provisionNextPlanShadows({ schema: [] }); }), ).pipe(Effect.exit); expect(state.stdin).toBe(""); @@ -222,7 +250,7 @@ describe("LegacyDeclarativeSeam next shadow protocol", () => { yield* Effect.scoped( Effect.gen(function* () { const seam = yield* LegacyDeclarativeSeam; - yield* seam.provisionNextShadow({ schema: [] }); + yield* seam.provisionNextPlanShadows({ schema: [] }); }), ).pipe(Effect.exit); expect(state.stdin).toBe(""); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts index 51ff36aab1..e11b428cea 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts @@ -37,10 +37,14 @@ export interface LegacyShadowSource { readonly sourceUrl: string; } -/** The independently hosted databases used by the pg-delta next planner. */ -export interface LegacyNextShadowSource { +/** The independently hosted migrated database used by pg-delta next diffs. */ +export interface LegacyNextMigrationsShadowSource { /** Platform baseline with local configuration and migrations applied. */ readonly migrationsUrl: string; +} + +/** The independently hosted databases used by the pg-delta next planner. */ +export interface LegacyNextPlanShadowSource extends LegacyNextMigrationsShadowSource { /** Platform baseline with local configuration, ready for declarative SQL. */ readonly declarativeUrl: string; } @@ -115,15 +119,26 @@ interface LegacyDeclarativeSeamShape { readonly projectRef?: string; }) => Effect.Effect; /** - * Provisions the two isolated pg-delta next shadows through the Go seam's - * JSON/ack ownership protocol. Both containers are owned by the current - * Effect scope before the child is acknowledged, and are independently - * removed when that scope closes. + * Provisions only the migrated pg-delta next shadow through the Go seam's + * JSON/ack ownership protocol. The container is owned by the current Effect + * scope before the child is acknowledged. + */ + readonly provisionNextMigrationsShadow: (opts: { + readonly schema: ReadonlyArray; + readonly projectRef?: string; + }) => Effect.Effect< + LegacyNextMigrationsShadowSource, + LegacyDeclarativeShadowDbError, + Scope.Scope + >; + /** + * Provisions the migrated and declarative pg-delta next shadows through the + * same ownership protocol. Both are independently removed with the scope. */ - readonly provisionNextShadow: (opts: { + readonly provisionNextPlanShadows: (opts: { readonly schema: ReadonlyArray; readonly projectRef?: string; - }) => Effect.Effect; + }) => Effect.Effect; /** * Removes a shadow database container left running by `provisionShadow` * (`docker rm -f `). Best-effort: a failure to remove is swallowed so it diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.ts index a97a2b078f..70112265cd 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.ts @@ -8,7 +8,7 @@ import { MIGRATE_FILE_PATTERN, legacyCreateMigrationTable, } from "./legacy-migration-history.ts"; -import { legacySplitAndTrim } from "./legacy-sql-split.ts"; +import { legacyParseMigrationContent } from "./legacy-migration-file.ts"; /** * Applying a migration file failed (Go's `ApplyMigrations` / `ExecBatch` error). @@ -157,12 +157,17 @@ const TYPE_NAME_PATTERN = /type "([^"]+)" does not exist/; * statement runs standalone, then batching resumes (supabase/cli#5156). The history * insert goes in the final batch, so the migration is recorded only after every * statement succeeds. A file with no such statements is a single `BEGIN`/`COMMIT`. + * Pg-delta files whose first line is `-- pg-delta: transaction=false` instead run + * every statement sequentially without a CLI-owned transaction. This keeps their + * session preamble, nontransactional action, and cleanup on the same connection. * - * Does NOT create the history table and does NOT `RESET ALL` — Go's `ExecBatch` does - * neither; those are the migration-apply path's responsibility (`ApplyMigrations`, - * apply.go:65-69), so role/globals files (`legacySeedGlobals`) stay reset-free like Go. - * When `forceNoVersion` is set the history insert is skipped regardless of filename - * (Go's `SeedGlobals` clears `Version`). + * Does NOT create the history table and does not unconditionally `RESET ALL` — Go's + * `ExecBatch` does neither; those are the migration-apply path's responsibility + * (`ApplyMigrations`, apply.go:65-69), so ordinary role/globals files + * (`legacySeedGlobals`) stay reset-free like Go. The one exception is best-effort + * cleanup after a failed pg-delta no-transaction file. When `forceNoVersion` is set + * the history insert is skipped regardless of filename (Go's `SeedGlobals` clears + * `Version`). */ const execMigrationBatch = ( session: LegacyDbSession, @@ -174,7 +179,8 @@ const execMigrationBatch = ( ): Effect.Effect => Effect.gen(function* () { const content = yield* fs.readFileString(migrationPath); - const statements = legacySplitAndTrim(content); + const parsed = legacyParseMigrationContent(content); + const { statements, transactionMode } = parsed; const filename = path.basename(migrationPath); const matches = MIGRATE_FILE_PATTERN.exec(filename); const version = forceNoVersion ? "" : (matches?.[1] ?? ""); @@ -205,9 +211,36 @@ const execMigrationBatch = ( return new Error(`${errMessage(e)}\n${msg.join("\n")}`); }; - // A file with authored transaction boundaries owns those semantics. Execute - // the statements exactly as written, clean up a failed authored transaction, - // and only send the history insert after every statement has succeeded. + // The pg-delta directive is file-level execution metadata. Run the complete + // sequence on this session without adding transaction boundaries so session + // settings remain active for the nontransactional action. History is recorded + // only after every statement succeeds. A failed sequence gets a best-effort + // session reset because the generated trailing RESET ALL may not have run yet. + if (transactionMode === "none") { + const nonTransactional = Effect.gen(function* () { + for (const [index, statement] of statements.entries()) { + yield* session + .exec(statement) + .pipe(Effect.mapError((cause) => atStatement(cause, index, statement))); + } + if (version.length > 0) { + yield* session + .query(INSERT_MIGRATION_VERSION, [version, name, statements]) + .pipe( + Effect.mapError((cause) => + atStatement(cause, statements.length, INSERT_MIGRATION_VERSION), + ), + ); + } + }); + return yield* nonTransactional.pipe( + Effect.tapError(() => session.exec("RESET ALL").pipe(Effect.ignore)), + ); + } + + // A headerless file with authored transaction boundaries owns those semantics. + // Execute the statements exactly as written, clean up a failed authored + // transaction, and only send the history insert after every statement succeeds. if (statements.some(legacyHasTransactionControl)) { const authored = Effect.gen(function* () { for (const [index, statement] of statements.entries()) { diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts index 4ca6249394..18f3b78c86 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts @@ -171,6 +171,75 @@ describe("legacyApplyMigrationFile", () => { ); }); + it.effect("honors pg-delta's file-level no-transaction directive", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-apply-")); + const file = join(dir, "20240101120000_drop_subscription.sql"); + writeFileSync( + file, + "-- pg-delta: transaction=false\n" + + "SET check_function_bodies = off;\n" + + "DROP SUBSCRIPTION app_events;\n" + + "RESET ALL;", + ); + const { session, calls } = fakeSession(); + return run(session, file).pipe( + Effect.tap(() => + Effect.sync(() => { + const execs = calls.filter((call) => call.kind === "exec").map((call) => call.sql); + const setupCommit = execs.indexOf("COMMIT"); + const set = execs.indexOf("SET check_function_bodies = off"); + const action = execs.indexOf("DROP SUBSCRIPTION app_events"); + const cleanup = execs.lastIndexOf("RESET ALL"); + + // The history-table setup owns the only CLI transaction. Pg-delta's + // preamble, action, and cleanup then run sequentially on this session. + expect(execs.filter((sql) => sql === "BEGIN")).toHaveLength(1); + expect(execs.filter((sql) => sql === "COMMIT")).toHaveLength(1); + expect(set).toBeGreaterThan(setupCommit); + expect(action).toBeGreaterThan(set); + expect(cleanup).toBeGreaterThan(action); + + const history = calls.filter((call) => call.kind === "query"); + expect(history).toHaveLength(1); + expect(history[0]?.params).toEqual([ + "20240101120000", + "drop_subscription", + ["SET check_function_bodies = off", "DROP SUBSCRIPTION app_events", "RESET ALL"], + ]); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("resets the session and omits history when a no-transaction migration fails", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-apply-")); + const file = join(dir, "20240101120000_drop_subscription.sql"); + writeFileSync( + file, + "-- pg-delta: transaction=false\n" + + "SET check_function_bodies = off;\n" + + "DROP SUBSCRIPTION app_events;\n" + + "RESET ALL;", + ); + const { session, calls } = fakeSession({ failOn: "DROP SUBSCRIPTION" }); + return run(session, file).pipe( + Effect.exit, + Effect.tap((exit) => + Effect.sync(() => { + expect(Exit.isFailure(exit)).toBe(true); + const execs = calls.filter((call) => call.kind === "exec").map((call) => call.sql); + expect(execs.at(-1)).toBe("RESET ALL"); + expect(calls.some((call) => call.kind === "query")).toBe(false); + if (Exit.isFailure(exit)) { + expect(JSON.stringify(exit.cause)).toContain("At statement: 1"); + } + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + it.effect("reports a pipeline-incompatible statement failure with its statement index", () => { const dir = mkdtempSync(join(tmpdir(), "legacy-apply-")); const file = join(dir, "20240101120000_add_index.sql"); diff --git a/apps/cli/src/legacy/shared/legacy-migration-file.ts b/apps/cli/src/legacy/shared/legacy-migration-file.ts index f8a7b7cd6e..6f2d52668e 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-file.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-file.ts @@ -1,5 +1,44 @@ import type { Path } from "effect"; +import { legacySplitAndTrim } from "./legacy-sql-split.ts"; + +type LegacyMigrationTransactionMode = "transactional" | "none"; + +export interface LegacyParsedMigrationContent { + readonly statements: ReadonlyArray; + readonly transactionMode: LegacyMigrationTransactionMode; +} + +const PG_DELTA_NO_TRANSACTION_DIRECTIVE = "-- pg-delta: transaction=false"; + +/** + * Parses the durable execution metadata and SQL statements in a migration file. + * Pg-delta writes its no-transaction directive as the first line because migration + * apply commands only retain the generated file, not the in-memory plan metadata. + * The exact directive may follow a UTF-8 BOM and may end with LF or CRLF. Marker-like + * comments anywhere else remain ordinary SQL comments and preserve the established + * transactional default. + */ +export function legacyParseMigrationContent(content: string): LegacyParsedMigrationContent { + const withoutBom = content.charCodeAt(0) === 0xfeff ? content.slice(1) : content; + const firstNewline = withoutBom.indexOf("\n"); + const rawFirstLine = firstNewline < 0 ? withoutBom : withoutBom.slice(0, firstNewline); + const firstLine = rawFirstLine.endsWith("\r") ? rawFirstLine.slice(0, -1) : rawFirstLine; + + if (firstLine === PG_DELTA_NO_TRANSACTION_DIRECTIVE) { + const sql = firstNewline < 0 ? "" : withoutBom.slice(firstNewline + 1); + return { + statements: legacySplitAndTrim(sql), + transactionMode: "none", + }; + } + + return { + statements: legacySplitAndTrim(content), + transactionMode: "transactional", + }; +} + /** * Go's `GetCurrentTimestamp` (`apps/cli-go/internal/utils/misc.go:130`): the * current time formatted UTC as `YYYYMMDDHHMMSS` (Go's `layoutVersion` diff --git a/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts index e35fc6e311..d3d265007b 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts @@ -1,7 +1,50 @@ import type { Path } from "effect"; import { describe, expect, it } from "vitest"; -import { legacyFormatMigrationTimestamp, legacyGetMigrationPath } from "./legacy-migration-file.ts"; +import { + legacyFormatMigrationTimestamp, + legacyGetMigrationPath, + legacyParseMigrationContent, +} from "./legacy-migration-file.ts"; + +describe("legacyParseMigrationContent", () => { + it.each([ + ["LF", "-- pg-delta: transaction=false\nSET check_function_bodies = off;"], + ["CRLF", "-- pg-delta: transaction=false\r\nSET check_function_bodies = off;"], + ["a UTF-8 BOM", "\uFEFF-- pg-delta: transaction=false\nSET check_function_bodies = off;"], + ])("recognizes the anchored no-transaction directive with %s", (_name, content) => { + expect(legacyParseMigrationContent(content)).toEqual({ + statements: ["SET check_function_bodies = off"], + transactionMode: "none", + }); + }); + + it("defaults an ordinary migration to transactional execution", () => { + expect(legacyParseMigrationContent("CREATE TABLE example (id bigint);")).toEqual({ + statements: ["CREATE TABLE example (id bigint)"], + transactionMode: "transactional", + }); + }); + + it("leaves a later transaction marker as an ordinary transactional comment", () => { + const content = "-- generated migration\n-- pg-delta: transaction=false\nVACUUM;"; + expect(legacyParseMigrationContent(content)).toEqual({ + statements: [content.slice(0, -1)], + transactionMode: "transactional", + }); + }); + + it.each([ + "-- pg-delta: transaction=true\nSELECT 1;", + " -- pg-delta: transaction=false\nSELECT 1;", + "-- pg-delta: transaction=none\nSELECT 1;", + ])("leaves a malformed first-line marker transactional: %s", (content) => { + expect(legacyParseMigrationContent(content)).toEqual({ + statements: [content.trim().slice(0, -1)], + transactionMode: "transactional", + }); + }); +}); describe("legacyFormatMigrationTimestamp", () => { it("formats epoch millis as UTC YYYYMMDDHHMMSS", () => { diff --git a/apps/cli/src/legacy/shared/legacy-migration-history.ts b/apps/cli/src/legacy/shared/legacy-migration-history.ts index 5231def40a..03ff80ebc8 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-history.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-history.ts @@ -9,7 +9,7 @@ import { legacyParseMigrationVersion, } from "./legacy-migration-timestamp.format.ts"; import { LegacyMigrationsReadError } from "./legacy-migration.errors.ts"; -import { legacySplitAndTrim } from "./legacy-sql-split.ts"; +import { legacyParseMigrationContent } from "./legacy-migration-file.ts"; /** * Consolidated `supabase_migrations.schema_migrations` history module — the @@ -470,11 +470,12 @@ export const legacyReadMigrationFile = ( }), ), Effect.map((content) => { + const parsed = legacyParseMigrationContent(content); const match = MIGRATE_FILE_PATTERN.exec(path.basename(migrationPath)); return { version: match?.[1] ?? "", name: match?.[2] ?? "", - statements: legacySplitAndTrim(content), + statements: parsed.statements, }; }), ); From ca55d8a2daf6d9ce477197d459fcc42ae3e590a8 Mon Sep 17 00:00:00 2001 From: avallete Date: Sun, 9 Aug 2026 12:44:45 +0200 Subject: [PATCH 17/82] chore(cli): use published pg-delta alpha --- apps/cli/package.json | 4 ++-- pnpm-lock.yaml | 24 +++++++++++------------- pnpm-workspace.yaml | 2 +- 3 files changed, 14 insertions(+), 16 deletions(-) diff --git a/apps/cli/package.json b/apps/cli/package.json index 4e8d1726b9..29edd7f819 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -55,8 +55,8 @@ "@parcel/watcher": "^2.6.0", "@supabase/api": "workspace:*", "@supabase/config": "workspace:*", - "@supabase/pg-delta": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86", - "@supabase/pg-topo": "https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86", + "@supabase/pg-delta": "1.0.0-alpha.34", + "@supabase/pg-topo": "1.0.0-alpha.5", "@supabase/process-compose": "workspace:*", "@supabase/stack": "workspace:*", "@tsconfig/bun": "catalog:", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a593864a86..f965360f49 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -155,11 +155,11 @@ importers: specifier: workspace:* version: link:../../packages/config '@supabase/pg-delta': - specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86 - version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86) + specifier: 1.0.0-alpha.34 + version: 1.0.0-alpha.34(@supabase/pg-topo@1.0.0-alpha.5) '@supabase/pg-topo': - specifier: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86 - version: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86 + specifier: 1.0.0-alpha.5 + version: 1.0.0-alpha.5 '@supabase/process-compose': specifier: workspace:* version: link:../../packages/process-compose @@ -2839,9 +2839,8 @@ packages: resolution: {integrity: sha512-RW/OCsd6MO592zU8ifzP8/f8XzxxIdpb+Up5XaOtE26Fw+3zTp475WX7+GuuktiD1WF8pFUDe6khUPbMp77RCw==} engines: {node: '>=22.0.0'} - '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86': - resolution: {integrity: sha512-jEEZhv8uh2vFPIoMeRd4GK1qiTNZh+LiIpY1bjd+oFVlacnb+UIrjS85JdDWStniRT03+NPOUkpjOtGiTspsIg==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86} - version: 1.0.0-alpha.33 + '@supabase/pg-delta@1.0.0-alpha.34': + resolution: {integrity: sha512-xjNBdFl4/DXIxZufUK6t52wTywMS4sl1rcXvxTgh3mv1Q50tDeMPkjp8QM5YRjYo/mrdxZJ94kmbf6XAQn0jRg==} engines: {node: '>=20.0.0'} hasBin: true peerDependencies: @@ -2850,9 +2849,8 @@ packages: '@supabase/pg-topo': optional: true - '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86': - resolution: {integrity: sha512-HoqxATDYB2WygDOV1XQBN/hM/hcfvVUrc7F+R691j6CD89cHumR/nRiRJ+0bh9siZb7Fx81Bp9BAPRfzEkEydA==, tarball: https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86} - version: 1.0.0-alpha.5 + '@supabase/pg-topo@1.0.0-alpha.5': + resolution: {integrity: sha512-a34YbUsQhBvS3Of5Gh/M4nXyGebwlID2lI7Od/YQPSA4jzGCR1D8EO/bV+kfAbTb84I/go8Pl1Y91XqFjRaqHg==} '@supabase/phoenix@0.4.5': resolution: {integrity: sha512-aAn9H9ovVyeApKy11OWOrrOGq8DV68yWeH4ud2lN9fzn4aO8Zb5GLL9m1pUg9nLqIcT+ZDfAcsZe0E/nqdv2lw==} @@ -9093,18 +9091,18 @@ snapshots: dependencies: tslib: 2.8.1 - '@supabase/pg-delta@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-delta@2f1d6b677bb44485f0a6874caf288f2c77896f86(@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86)': + '@supabase/pg-delta@1.0.0-alpha.34(@supabase/pg-topo@1.0.0-alpha.5)': dependencies: debug: 4.4.3(supports-color@7.2.0) pg: 8.22.0 pg-connection-string: 2.14.0 optionalDependencies: - '@supabase/pg-topo': https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86 + '@supabase/pg-topo': 1.0.0-alpha.5 transitivePeerDependencies: - pg-native - supports-color - '@supabase/pg-topo@https://pkg.pr.new/supabase/pg-toolbelt/@supabase/pg-topo@2f1d6b677bb44485f0a6874caf288f2c77896f86': + '@supabase/pg-topo@1.0.0-alpha.5': dependencies: '@pgsql/traverse': 17.2.6 plpgsql-parser: 0.5.16 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 1a003a6606..1a70d9f196 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -49,7 +49,7 @@ minimumReleaseAgeExclude: - "@effect/platform-node-shared@4.0.0-beta.103" - "@effect/sql-pg@4.0.0-beta.103" - "@effect/vitest@4.0.0-beta.103" - - "@supabase/pg-delta@1.0.0-alpha.33" + - "@supabase/pg-delta@1.0.0-alpha.34" - "@supabase/pg-topo@1.0.0-alpha.5" - "effect@4.0.0-beta.103" From 9f264e9653bbe259e0a269e7c1c5b74522c486fe Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 11 Aug 2026 10:57:17 +0200 Subject: [PATCH 18/82] fix(cli): address pg-delta next rollout blockers --- .../commands/db/diff/diff.integration.test.ts | 14 ++-- .../schema/declarative/declarative.errors.ts | 6 +- .../db/shared/legacy-pgdelta-files.ts | 13 +++- .../shared/legacy-pgdelta-migrations.write.ts | 77 ++++++++++++++----- .../legacy-pgdelta-next-adapter.service.ts | 11 ++- .../legacy-pgdelta-next-shadow.layer.ts | 7 -- .../db-bootstrap/shadow-database.unit.test.ts | 50 ++++++------ .../src/shared/init/project-init.templates.ts | 1 - 8 files changed, 117 insertions(+), 62 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 333e87fcc8..708df3556a 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -1066,10 +1066,10 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("bumps the version set when a target migration file already exists", () => { - // The full generated set is collision-checked before writing; if any target - // exists the base advances one second so the new files stay strictly ascending - // AND never overwrite the pre-existing migration. + it.effect("bumps the version set when another migration uses the same version", () => { + // Migration identity is the timestamp, not the full filename. If another name + // already uses a generated version, the whole set advances so every new version + // stays strictly ascending and unique. const s = setup(tmp.current, { format: "json", diffFiles: [ @@ -1081,12 +1081,12 @@ describe("legacy db diff", () => { const dir = join(tmp.current, "supabase", "migrations"); mkdirSync(dir, { recursive: true }); // TestClock starts at epoch 0, so the first version the writer tries is - // 19700101000000; pre-seed a colliding file at that version. - const clashing = join(dir, "19700101000000_my_diff_schema_changes.sql"); + // 19700101000000; pre-seed a differently named migration at that version. + const clashing = join(dir, "19700101000000_different_name.sql"); writeFileSync(clashing, "-- pre-existing\n"); yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), file: Option.some("my_diff") })); expect(readdirSync(dir).sort()).toEqual([ - "19700101000000_my_diff_schema_changes.sql", + "19700101000000_different_name.sql", "19700101000001_my_diff_schema_changes.sql", "19700101000002_my_diff_after_enum_values.sql", ]); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts index 484729c4c6..2343134dcf 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts @@ -106,7 +106,11 @@ export class LegacyDeclarativeCompatibilityError extends Data.TaggedError( "LegacyDeclarativeCompatibilityError", )<{ readonly message: string; -}> {} +}> { + get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { + return actionability.dbFinding; + } +} /** * Applying the generated migration to the local database failed. Wraps Go's diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts index 368a8cf20e..9d6b9d02fa 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts @@ -1,5 +1,10 @@ import { Data, Effect, type FileSystem, type Path } from "effect"; +import { + actionability, + type CliErrorActionabilityDeclaration, + ErrorActionabilityId, +} from "../../../../shared/telemetry/error-actionability.ts"; import type { LegacyPgDeltaExportManifest, LegacyPgDeltaSqlFile, @@ -7,9 +12,13 @@ import type { const EXPORT_MANIFEST_FILE = ".pgdelta-export.json"; -class LegacyPgDeltaFilesError extends Data.TaggedError("LegacyPgDeltaFilesError")<{ +export class LegacyPgDeltaFilesError extends Data.TaggedError("LegacyPgDeltaFilesError")<{ readonly message: string; -}> {} +}> { + get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { + return actionability.invalidConfig; + } +} const filesError = (message: string) => new LegacyPgDeltaFilesError({ message }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts index e0f5d0aac2..a6ed31f264 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-migrations.write.ts @@ -47,13 +47,14 @@ const MAX_VERSION_COLLISION_ATTEMPTS = 60; * arithmetic on the base millis, never string increment) so their execution order * and migration-history order stay stable. * - * Before writing anything the FULL set of generated filenames is collision-checked - * against the filesystem: if any target path already exists the base is advanced by - * one second and every version recomputed, so the set stays strictly ascending AND - * unique against pre-existing migrations. The base only ever moves forward — never - * backdated below the caller's wall clock, since backdating could sort a new file - * before pre-existing migrations. The resulting ≤N−1s future-dating is inherent to - * second-granularity versions and acceptable once uniqueness is enforced. + * Before writing anything the FULL set of generated versions is collision-checked + * against the migrations directory: if any version is already used, the base is + * advanced by one second and every version recomputed, so the set stays strictly + * ascending AND unique against pre-existing migrations regardless of their names. + * The base only ever moves forward — never backdated below the caller's wall clock, + * since backdating could sort a new file before pre-existing migrations. The + * resulting ≤N−1s future-dating is inherent to second-granularity versions and + * acceptable once uniqueness is enforced. * * Each file is written with the exclusive `"wx"` flag so a race between the * collision check and the write can still never silently overwrite an existing @@ -88,6 +89,40 @@ export const legacyWritePgDeltaMigrations = ( } } const single = files.length === 1; + const migrationsDir = pathSvc.join(workdir, "supabase", "migrations"); + const migrationEntries = yield* fs.readDirectory(migrationsDir).pipe( + Effect.catchTag("PlatformError", (error) => + error.reason._tag === "NotFound" + ? Effect.succeed([] as ReadonlyArray) + : Effect.fail( + new LegacyPgDeltaMigrationWriteError({ + message: `failed to read migration directory: ${error.message}`, + }), + ), + ), + ); + const usedVersions = new Set(); + for (const entry of migrationEntries) { + const match = /^([0-9]+)_(.+)$/u.exec(entry); + if (match?.[1] === undefined) continue; + if (entry.endsWith(".sql")) { + usedVersions.add(match[1]); + continue; + } + const stat = yield* fs.stat(pathSvc.join(migrationsDir, entry)).pipe( + Effect.mapError( + (cause) => + new LegacyPgDeltaMigrationWriteError({ + message: `failed to inspect migration directory entry: ${cause.message}`, + }), + ), + ); + if (stat.type === "Directory") { + // Nested names such as `snapshots/remote` are stored under a + // `_snapshots/` directory, whose prefix still owns the version. + usedVersions.add(match[1]); + } + } const buildSet = (baseMillis: number): Array => files.map((file, i) => { const version = legacyFormatMigrationTimestamp(baseMillis + i * 1000); @@ -102,19 +137,21 @@ export const legacyWritePgDeltaMigrations = ( let baseMillis = opts.baseMillis; let set = buildSet(baseMillis); for (let attempt = 0; ; attempt++) { - let collision = false; - for (const w of set) { - const exists = yield* fs.exists(w.path).pipe( - Effect.mapError( - (cause) => - new LegacyPgDeltaMigrationWriteError({ - message: `failed to check migration file: ${cause.message}`, - }), - ), - ); - if (exists) { - collision = true; - break; + let collision = set.some((w) => usedVersions.has(w.version)); + if (!collision) { + for (const w of set) { + const exists = yield* fs.exists(w.path).pipe( + Effect.mapError( + (cause) => + new LegacyPgDeltaMigrationWriteError({ + message: `failed to check migration file: ${cause.message}`, + }), + ), + ); + if (exists) { + collision = true; + break; + } } } if (!collision) break; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts index aba46a3f66..74d1629e16 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts @@ -1,6 +1,11 @@ import type { Pool } from "pg"; import { Context, Data, type Effect } from "effect"; +import { + actionability, + type CliErrorActionabilityDeclaration, + ErrorActionabilityId, +} from "../../../../shared/telemetry/error-actionability.ts"; import type { LegacyPgDeltaRemovalSummary, LegacyPgDeltaTransactionMode, @@ -174,7 +179,11 @@ export class LegacyPgDeltaNextError extends Data.TaggedError("LegacyPgDeltaNextE readonly operation: LegacyPgDeltaNextOperation; readonly message: string; readonly cause: unknown; -}> {} +}> { + get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { + return actionability.dbFinding; + } +} export interface LegacyPgDeltaNextAdapterShape { readonly diff: ( diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index 68868d1b97..4207e78410 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -198,13 +198,6 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( const provisionDeclarative = (input: NativeShadowInput) => Effect.gen(function* () { - if (input.localInputs.setup.majorVersion !== 17) { - return yield* Effect.fail( - new LegacyDeclarativeShadowDbError({ - message: `pg-delta declarative shadow baseline requires Postgres 17 (got major ${input.localInputs.setup.majorVersion}, image ${JSON.stringify(input.base.image)})`, - }), - ); - } const handle = yield* acquireShadow(input); yield* legacyWaitForHealthyServices(input.spawner, [handle.containerId], { timeoutSeconds: input.base.healthTimeoutSeconds, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index b3f32546dd..d2c1702575 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -637,7 +637,7 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { }); it.effect( - "does not resolve JWKS on PG14 even when realtime is enabled (Go's initSchema never reaches ResolveJWKS for MajorVersion <= 14)", + "supports the extension-free declarative baseline on PG14 without resolving JWKS", () => { const { session } = fakeSession(); const workdir = tempRoot.current; @@ -646,29 +646,33 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { return Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; - yield* legacySetupShadowDatabase(mock.spawner, { - fs, - path, - workdir, - projectId: "proj", - container: "shadow-container-id-0123456789abcdef", - networkId: "supabase_network_proj", - connConfig: { - host: "127.0.0.1", - port: 54320, - user: "postgres", - password: "postgres", - database: "postgres", - }, - setup: baseShadowSetup({ - majorVersion: 14, - realtimeEnabledForSetup: true, - jwks: Effect.sync(() => { - jwksEvaluated = true; - return '{"keys":[]}'; + yield* legacySetupShadowDatabase( + mock.spawner, + { + fs, + path, + workdir, + projectId: "proj", + container: "shadow-container-id-0123456789abcdef", + networkId: "supabase_network_proj", + connConfig: { + host: "127.0.0.1", + port: 54320, + user: "postgres", + password: "postgres", + database: "postgres", + }, + setup: baseShadowSetup({ + majorVersion: 14, + realtimeEnabledForSetup: true, + jwks: Effect.sync(() => { + jwksEvaluated = true; + return '{"keys":[]}'; + }), }), - }), - }); + }, + { activateUserExtensions: false }, + ); expect(jwksEvaluated).toBe(false); }).pipe( Effect.provide( diff --git a/apps/cli/src/shared/init/project-init.templates.ts b/apps/cli/src/shared/init/project-init.templates.ts index 7acc2f30a0..2c6a823579 100644 --- a/apps/cli/src/shared/init/project-init.templates.ts +++ b/apps/cli/src/shared/init/project-init.templates.ts @@ -412,7 +412,6 @@ enabled = true # declarative_schema_path = "./database" # JSON string passed through to pg-delta SQL formatting. # format_options = "{\\"keywordCase\\":\\"upper\\",\\"indent\\":2,\\"maxWidth\\":80,\\"commaStyle\\":\\"trailing\\"}" -# Set to "null" to disable formatting while retaining plan compaction. `; export const INIT_GITIGNORE_TEMPLATE = `# Supabase From 0879ced132304b65c2f62750f513f858fa5c9f51 Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 11 Aug 2026 11:29:46 +0200 Subject: [PATCH 19/82] fix(cli): add pg-net webhook remediation --- .../db/reset/reset.integration.test.ts | 21 +++ .../schema/declarative/declarative.errors.ts | 2 +- ...eclarative.orchestrate.integration.test.ts | 1 + .../schema/declarative/sync/sync.handler.ts | 17 +++ .../declarative/sync/sync.integration.test.ts | 28 +++- ...elta-engine.next.layer.integration.test.ts | 1 + .../legacy/shared/db-bootstrap/db-setup.ts | 34 +++-- .../db-bootstrap/recreate-local-database.ts | 5 +- .../shared/legacy-db-config.toml-read.ts | 3 + .../legacy-db-config.toml-read.unit.test.ts | 2 + .../legacy/shared/legacy-migrate-and-seed.ts | 34 ++++- .../legacy-migrate-and-seed.unit.test.ts | 142 ++++++++++++++++++ .../legacy/shared/legacy-migration-apply.ts | 53 +++++-- .../legacy/shared/legacy-pg-net-guidance.ts | 21 +++ 14 files changed, 331 insertions(+), 33 deletions(-) create mode 100644 apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts diff --git a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts index c023974bb5..f63a1d9094 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts @@ -1115,6 +1115,27 @@ describe("legacy db reset", () => { }); }); + it.live("installs pg_net before replay when Database Webhooks is enabled (PG14)", () => { + const { layer, conn } = setup(tmp.current, { + toml: `${PG14_TOML}[experimental.webhooks]\nenabled = true\n`, + files: migrationFile( + "20240101000000", + "select net.http_post(url := 'https://example.com');", + ), + args: ["db", "reset", "--local"], + isLocal: true, + }); + return Effect.gen(function* () { + yield* legacyDbReset(DEFAULT_FLAGS).pipe(Effect.provide(layer)); + const pgNetIndex = conn.execs.findIndex((sql) => + sql.includes("create extension if not exists pg_net schema extensions"), + ); + const migrationIndex = conn.execs.findIndex((sql) => sql.includes("https://example.com")); + expect(pgNetIndex).toBeGreaterThanOrEqual(0); + expect(migrationIndex).toBeGreaterThan(pgNetIndex); + }); + }); + it.live( "passes the resolved --version cutoff through to the final MigrateAndSeed step (PG14)", () => { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts index 2343134dcf..f76595a230 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts @@ -108,7 +108,7 @@ export class LegacyDeclarativeCompatibilityError extends Data.TaggedError( readonly message: string; }> { get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { - return actionability.dbFinding; + return actionability.invalidConfig; } } diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index d3a6f8452c..74abb50234 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -278,6 +278,7 @@ const toml: LegacyDbTomlValues = { formatOptions: Option.none(), npmVersion: Option.none(), }, + webhooksEnabled: false, baseline: { authEnabled: true, storageEnabled: true, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 096e062675..26e0c86f22 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -20,6 +20,7 @@ import { } from "../../../../../shared/legacy-db-config.toml-read.ts"; import { legacyMakeDir } from "../../../../../shared/legacy-make-dir.ts"; import { legacyApplyMigrationFile } from "../../../../../shared/legacy-migration-apply.ts"; +import { LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION } from "../../../../../shared/legacy-pg-net-guidance.ts"; import { legacyReadProjectRefFile } from "../../../../../shared/legacy-temp-paths.ts"; import { LegacyLinkedProjectCache } from "../../../../../telemetry/legacy-linked-project-cache.service.ts"; import { LegacyTelemetryState } from "../../../../../telemetry/legacy-telemetry-state.service.ts"; @@ -321,6 +322,22 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara // Resolve manifest-less legacy compatibility before printing or writing a // migration. A repair is always explicit, even when global --yes is set. + if ( + engine.implementation === "next" && + !result.manifestPresent && + !toml.webhooksEnabled && + result.removals.extensions.includes("pg_net") + ) { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ + message: [ + "The migrations state includes pg_net, but Database Webhooks are not enabled in the local project config.", + "", + LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION, + ].join("\n"), + }), + ); + } const compatibility = legacyClassifyDeclarativeCompatibilityGap({ implementation: engine.implementation, manifestPresent: result.manifestPresent, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 39606729df..7f3ac64716 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -1006,20 +1006,42 @@ describe("legacy db schema declarative sync integration", () => { }, ); - it.effect("adds detected legacy extension declarations and re-plans before writing", () => { + it.effect("directs pg_net users to enable Database Webhooks before writing", () => { seedDeclarative(tmp.current); const s = setup(tmp.current, { engineImplementation: "next", stdinIsTty: true, diffSql: 'DROP EXTENSION "pg_net";\n', - replannedDiffSql: "", removals: { extensions: ["pg_net"], extensionIntents: [] }, + }); + return Effect.gen(function* () { + const exit = yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })).pipe( + Effect.exit, + ); + expect(failError(exit)).toMatchObject({ + _tag: "LegacyDeclarativeCompatibilityError", + message: expect.stringContaining("[experimental.webhooks]\nenabled = true"), + }); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); + expect(s.out.promptSelectCalls).toHaveLength(0); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("keeps explicit extension repair for non-config-managed extensions", () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + diffSql: 'DROP EXTENSION "pgcrypto";\n', + replannedDiffSql: "", + removals: { extensions: ["pgcrypto"], extensionIntents: [] }, promptSelectResponses: ["repair"], }); return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); expect(readFileSync(join(tmp.current, "supabase", "database", "extension.sql"), "utf8")).toBe( - 'CREATE EXTENSION IF NOT EXISTS "pg_net" WITH SCHEMA "extensions";\n', + 'CREATE EXTENSION IF NOT EXISTS "pgcrypto" WITH SCHEMA "extensions";\n', ); expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); expect(stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join(""))).toContain( diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts index eac85970cb..394d732fe8 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts @@ -43,6 +43,7 @@ const toml: LegacyDbTomlValues = { formatOptions: Option.none(), npmVersion: Option.none(), }, + webhooksEnabled: false, baseline: { authEnabled: true, storageEnabled: true, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index 15dcb8eee3..4837d35a19 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -915,20 +915,19 @@ export const legacyApplyApiPrivileges = Effect.fnUntraced(function* ( ); }); -const legacyApplyDatabaseWebhooks = Effect.fnUntraced(function* ( - input: LegacySetupDatabaseInput, +/** Installs pg_net for the local Database Webhooks feature when enabled. */ +export const legacyApplyDatabaseWebhooks = Effect.fnUntraced(function* ( + session: LegacyDbSession, + fs: FileSystem.FileSystem, + path: Path.Path, tmpDir: string, - options: LegacySetupDatabaseOptions, + enabled: boolean, ) { - const activateUserExtensions = options.activateUserExtensions ?? true; - const legacyPgNetBaseline = options.legacyPgNetBaseline ?? false; - const userEnabled = - activateUserExtensions && input.config.experimental.webhooks?.enabled === true; - if (!legacyPgNetBaseline && !userEnabled) return; + if (!enabled) return; yield* legacyExecSqlConstant( - input.session, - input.fs, - input.path, + session, + fs, + path, tmpDir, "enable-database-webhooks.sql", LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL, @@ -1021,7 +1020,17 @@ export const legacySetupDatabase = ( ), ); yield* legacyStartInitSchema(spawner, input, tmpDir); - yield* legacyApplyDatabaseWebhooks(input, tmpDir, options); + const activateUserExtensions = options.activateUserExtensions ?? true; + const legacyPgNetBaseline = options.legacyPgNetBaseline ?? false; + const userEnabled = + activateUserExtensions && input.config.experimental.webhooks?.enabled === true; + yield* legacyApplyDatabaseWebhooks( + session, + fs, + path, + tmpDir, + legacyPgNetBaseline || userEnabled, + ); yield* legacyApplyApiPrivileges(session, fs, path, tmpDir, input.apiAutoExposeNewTables); }), ); @@ -1135,6 +1144,7 @@ export const legacyStartSetupLocalDatabase = ( experimental: input.experimental, pgDeltaEnabled: toml.pgDelta.enabled, schemaPaths: toml.schemaPaths, + localDatabaseWebhooksEnabled: toml.webhooksEnabled, }); const output = yield* Output; diff --git a/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts index f3fad9bafb..89ecfe6e78 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts @@ -52,7 +52,7 @@ * ONLY, deliberately WITHOUT globals.sql — see `legacyInitSchema14`'s * own doc comment for why this is NOT the same as `db start`'s PG<=14 path) * + `ApplyApiPrivileges` (the exact same exported function `SetupDatabase` - * also calls). + * also calls), then pg_net activation when Database Webhooks is enabled. * 3. `RestartDatabase` (`reset.go:214-225`) — `Restarting containers...\n` * FIRST, then a REAL `docker restart` of the `db` container itself (NOT * tolerant of "not found" — pg_cron must restart after @@ -120,6 +120,7 @@ import { legacyRunFreshDbSetup, legacyResolveResetSeedConfig, legacyApplyApiPrivileges, + legacyApplyDatabaseWebhooks, legacyInitSchema14, LegacyDbSetupError, type LegacyFreshDbSetupInput, @@ -481,6 +482,7 @@ const legacyRecreateLocalDatabase14 = ( tmpDir, toml.baseline.apiAutoExposeNewTables, ); + yield* legacyApplyDatabaseWebhooks(session, fs, path, tmpDir, toml.webhooksEnabled); }), ); @@ -504,6 +506,7 @@ const legacyRecreateLocalDatabase14 = ( experimental: setup.experimental, pgDeltaEnabled: toml.pgDelta.enabled, schemaPaths: toml.schemaPaths, + localDatabaseWebhooksEnabled: toml.webhooksEnabled, }); }), ); diff --git a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts index a897c0b585..9d9079e1ea 100644 --- a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts +++ b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts @@ -89,6 +89,8 @@ export interface LegacyDbTomlValues { * (`apps/cli-go/pkg/config/config.go:228-234`). */ readonly pgDelta: LegacyPgDeltaTomlConfig; + /** Effective `[experimental.webhooks].enabled`; false when the section is absent. */ + readonly webhooksEnabled: boolean; /** * The subset of config that shapes the shadow-database platform baseline and * therefore the declarative catalog-cache key (Go's `setupInputsToken`, @@ -2748,6 +2750,7 @@ const readDbTomlCore = Effect.fnUntraced(function* ( formatOptions, npmVersion: pgDeltaNpmVersion, }, + webhooksEnabled, baseline: { authEnabled, storageEnabled: yield* resolveBoolOrFail( diff --git a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.unit.test.ts b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.unit.test.ts index 0813b9e416..c90833a091 100644 --- a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.unit.test.ts @@ -1133,6 +1133,7 @@ describe("legacyReadDbToml", () => { Effect.tap((exit) => Effect.sync(() => { expect(Exit.isSuccess(exit)).toBe(true); + if (Exit.isSuccess(exit)) expect(exit.value.webhooksEnabled).toBe(true); if (previous === undefined) delete process.env["SUPABASE_EXPERIMENTAL_WEBHOOKS_ENABLED"]; else process.env["SUPABASE_EXPERIMENTAL_WEBHOOKS_ENABLED"] = previous; @@ -1198,6 +1199,7 @@ describe("legacyReadDbToml", () => { Effect.tap((exit) => Effect.sync(() => { expect(Exit.isSuccess(exit)).toBe(true); + if (Exit.isSuccess(exit)) expect(exit.value.webhooksEnabled).toBe(false); if (previous === undefined) delete process.env["SUPABASE_EXPERIMENTAL_WEBHOOKS_ENABLED"]; else process.env["SUPABASE_EXPERIMENTAL_WEBHOOKS_ENABLED"] = previous; diff --git a/apps/cli/src/legacy/shared/legacy-migrate-and-seed.ts b/apps/cli/src/legacy/shared/legacy-migrate-and-seed.ts index 724765d667..15c268ae93 100644 --- a/apps/cli/src/legacy/shared/legacy-migrate-and-seed.ts +++ b/apps/cli/src/legacy/shared/legacy-migrate-and-seed.ts @@ -2,12 +2,17 @@ import { Effect, type FileSystem, type Path } from "effect"; import { Output } from "../../shared/output/output.service.ts"; import type { LegacyDbSession } from "./legacy-db-connection.service.ts"; +import type { LegacyDbExecError } from "./legacy-db-connection.errors.ts"; import { LegacyMigrationApplyError, legacyApplyMigrationFile, legacyApplySchemaFiles, } from "./legacy-migration-apply.ts"; import { legacyLoadPartialMigrations } from "./legacy-migration-history.ts"; +import { + LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION, + legacyIsPgNetUnavailableError, +} from "./legacy-pg-net-guidance.ts"; import { legacyApplySeedFiles, type LegacySeedConfig } from "./legacy-seed.ts"; /** Config consumed by `legacyMigrateAndSeed`. */ @@ -28,8 +33,29 @@ export interface LegacyMigrateAndSeedConfig { readonly pgDeltaEnabled: boolean; /** `db.migrations.schema_paths` — Go's `Config.Db.Migrations.SchemaPaths`. Only read by the declarative branch above. */ readonly schemaPaths: ReadonlyArray; + /** + * Effective local `[experimental.webhooks].enabled` value. `undefined` means + * this is not a local start/reset replay and disables local-only remediation. + */ + readonly localDatabaseWebhooksEnabled?: boolean; } +const migrationApplyError = ( + message: string, + dbError: LegacyDbExecError | undefined, + localDatabaseWebhooksEnabled: boolean | undefined, +): LegacyMigrationApplyError => { + const pgNetUnavailable = + localDatabaseWebhooksEnabled === false && + dbError !== undefined && + legacyIsPgNetUnavailableError(dbError); + return new LegacyMigrationApplyError({ + message, + suggestion: pgNetUnavailable ? LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION : undefined, + reason: pgNetUnavailable ? "local_pg_net_unavailable" : undefined, + }); +}; + /** * Reapplies local migrations up to `version`, then runs seed files. Port of Go's * `apply.MigrateAndSeed` (`internal/migration/apply/apply.go:16-26`): when `experimental` is @@ -67,12 +93,8 @@ export const legacyMigrateAndSeed = ( ); for (const migrationPath of pending) { yield* output.raw(`Applying migration ${path.basename(migrationPath)}...\n`, "stderr"); - yield* legacyApplyMigrationFile( - session, - fs, - path, - migrationPath, - (message) => new LegacyMigrationApplyError({ message }), + yield* legacyApplyMigrationFile(session, fs, path, migrationPath, (message, dbError) => + migrationApplyError(message, dbError, config.localDatabaseWebhooksEnabled), ); } } diff --git a/apps/cli/src/legacy/shared/legacy-migrate-and-seed.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migrate-and-seed.unit.test.ts index d89c197e81..f5e3ce942b 100644 --- a/apps/cli/src/legacy/shared/legacy-migrate-and-seed.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migrate-and-seed.unit.test.ts @@ -7,6 +7,7 @@ import { Effect, Exit, FileSystem, Layer, Path } from "effect"; import { stripAnsi } from "../../../tests/helpers/ansi.ts"; import { mockOutput } from "../../../tests/helpers/mocks.ts"; +import { actionability, ErrorActionabilityId } from "../../shared/telemetry/error-actionability.ts"; import { LegacyDbExecError } from "./legacy-db-connection.errors.ts"; import type { LegacyDbSession } from "./legacy-db-connection.service.ts"; import { LegacyMigrationApplyError } from "./legacy-migration-apply.ts"; @@ -14,6 +15,10 @@ import { legacyMigrateAndSeed, type LegacyMigrateAndSeedConfig, } from "./legacy-migrate-and-seed.ts"; +import { + LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION, + legacyIsPgNetUnavailableError, +} from "./legacy-pg-net-guidance.ts"; // Root bypasses POSIX permission bits, so chmod 000 wouldn't block readdir() there. const isRoot = typeof process.getuid === "function" && process.getuid() === 0; @@ -52,6 +57,23 @@ function failingExecSession(): { session: LegacyDbSession; execs: Array return { session, execs }; } +function pgNetFailureSession(error: LegacyDbExecError): LegacyDbSession { + return { + exec: (sql) => (sql.includes("net.http_post") ? Effect.fail(error) : Effect.void), + query: () => Effect.succeed([]), + extensionExists: () => Effect.succeed(false), + copyToCsv: () => Effect.succeed(new Uint8Array()), + queryRaw: () => Effect.succeed({ fields: [], rows: [], commandTag: "" }), + }; +} + +function assertMigrationApplyError(error: unknown): asserts error is LegacyMigrationApplyError { + expect(error).toBeInstanceOf(LegacyMigrationApplyError); + if (!(error instanceof LegacyMigrationApplyError)) { + throw new Error("expected LegacyMigrationApplyError"); + } +} + function makeWorkdir(): string { return mkdtempSync(join(tmpdir(), "legacy-migrate-and-seed-")); } @@ -432,3 +454,123 @@ describe("legacyMigrateAndSeed experimental declarative-schema branch", () => { }, ); }); + +describe("legacyMigrateAndSeed local pg_net remediation", () => { + const missingNetSchema = new LegacyDbExecError({ + message: 'ERROR: schema "net" does not exist (SQLSTATE 3F000)', + code: "3F000", + }); + + const setupMigration = (workdir: string) => + writeFile( + workdir, + "supabase/migrations/20240101000000_webhook.sql", + "select net.http_post(url := 'https://example.com');", + ); + + it("classifies only pg_net schema/function errors with their matching SQLSTATE", () => { + expect(legacyIsPgNetUnavailableError(missingNetSchema)).toBe(true); + expect( + legacyIsPgNetUnavailableError({ + message: "ERROR: function net.http_post(unknown, jsonb) does not exist (SQLSTATE 42883)", + code: "42883", + }), + ).toBe(true); + expect( + legacyIsPgNetUnavailableError({ + message: "ERROR: function public.http_post(unknown) does not exist (SQLSTATE 42883)", + code: "42883", + }), + ).toBe(false); + }); + + it.effect("suggests enabling Database Webhooks when local replay cannot find pg_net", () => { + const workdir = makeWorkdir(); + setupMigration(workdir); + const out = mockOutput(); + return run( + workdir, + "", + { ...baseConfig, localDatabaseWebhooksEnabled: false }, + pgNetFailureSession(missingNetSchema), + out, + ).pipe( + Effect.flip, + Effect.tap((error) => + Effect.sync(() => { + assertMigrationApplyError(error); + expect(error.suggestion).toBe(LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION); + expect(error[ErrorActionabilityId]).toEqual(actionability.invalidConfig); + rmSync(workdir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("does not add the local hint when webhooks are enabled", () => { + const workdir = makeWorkdir(); + setupMigration(workdir); + const out = mockOutput(); + return run( + workdir, + "", + { ...baseConfig, localDatabaseWebhooksEnabled: true }, + pgNetFailureSession(missingNetSchema), + out, + ).pipe( + Effect.flip, + Effect.tap((error) => + Effect.sync(() => { + assertMigrationApplyError(error); + expect(error.suggestion).toBeUndefined(); + expect(error[ErrorActionabilityId]).toEqual(actionability.dbFinding); + rmSync(workdir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("does not add the local hint for migration commands without local context", () => { + const workdir = makeWorkdir(); + setupMigration(workdir); + const out = mockOutput(); + return run(workdir, "", baseConfig, pgNetFailureSession(missingNetSchema), out).pipe( + Effect.flip, + Effect.tap((error) => + Effect.sync(() => { + assertMigrationApplyError(error); + expect(error.suggestion).toBeUndefined(); + expect(error[ErrorActionabilityId]).toEqual(actionability.dbFinding); + rmSync(workdir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("requires the matching SQLSTATE instead of classifying by message alone", () => { + const workdir = makeWorkdir(); + setupMigration(workdir); + const out = mockOutput(); + const wrongSqlState = new LegacyDbExecError({ + message: 'ERROR: schema "net" does not exist (SQLSTATE 42P01)', + code: "42P01", + }); + return run( + workdir, + "", + { ...baseConfig, localDatabaseWebhooksEnabled: false }, + pgNetFailureSession(wrongSqlState), + out, + ).pipe( + Effect.flip, + Effect.tap((error) => + Effect.sync(() => { + assertMigrationApplyError(error); + expect(error.suggestion).toBeUndefined(); + expect(error[ErrorActionabilityId]).toEqual(actionability.dbFinding); + rmSync(workdir, { recursive: true, force: true }); + }), + ), + ); + }); +}); diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.ts index 9d323eb4a2..e31242ae12 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.ts @@ -2,7 +2,7 @@ import { Data, Effect, type FileSystem, type Path } from "effect"; import { Output } from "../../shared/output/output.service.ts"; import { legacyBold } from "./legacy-colors.ts"; -import type { LegacyDbExecError } from "./legacy-db-connection.errors.ts"; +import { LegacyDbExecError } from "./legacy-db-connection.errors.ts"; import { actionability, type CliErrorActionabilityDeclaration, @@ -24,16 +24,19 @@ import { legacySplitSqlTokens } from "./legacy-sql-split.ts"; * Used by `migration up` and `migration down`'s migrate-and-seed step. The * declarative sync handler maps its own error type instead. * - * `suggestion` carries Go's `utils.CmdSuggestion` when a caller sets one — currently - * only `legacyApplySchemaFiles`'s "See schema file: " (`apply.go:57`); every other - * caller leaves it unset, matching Go leaving `CmdSuggestion` empty on those paths. + * `suggestion` carries caller remediation. This includes Go's `utils.CmdSuggestion` + * for `legacyApplySchemaFiles` and the local-only pg_net/webhooks remediation added + * when start/reset replay has enough structured context to identify that failure. */ export class LegacyMigrationApplyError extends Data.TaggedError("LegacyMigrationApplyError")<{ readonly message: string; readonly suggestion?: string; + readonly reason?: "local_pg_net_unavailable"; }> { get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { - return actionability.dbFinding; + return this.reason === "local_pg_net_unavailable" + ? actionability.invalidConfig + : actionability.dbFinding; } } @@ -481,7 +484,24 @@ export const legacyFormatExecBatchError = ( msg.push(" Learn more: supabase migration new --help"); } msg.push(`At statement: ${index}`, marked); - return new Error(`${legacyErrorMessage(e)}\n${msg.join("\n")}`); + return formattedExecBatchFailure(`${legacyErrorMessage(e)}\n${msg.join("\n")}`, e); +}; + +/** Retains the server ErrorResponse after adding Go-compatible statement context. */ +const FormattedExecBatchDbErrorId: unique symbol = Symbol("FormattedExecBatchDbError"); +type FormattedExecBatchFailure = Error & { + readonly [FormattedExecBatchDbErrorId]: LegacyDbExecError; +}; +const formattedExecBatchFailure = ( + message: string, + dbError: LegacyDbExecError, +): FormattedExecBatchFailure => + Object.assign(new Error(message), { [FormattedExecBatchDbErrorId]: dbError }); + +const formattedExecBatchDbError = (error: unknown): LegacyDbExecError | undefined => { + if (typeof error !== "object" || error === null) return undefined; + const dbError: unknown = Reflect.get(error, FormattedExecBatchDbErrorId); + return dbError instanceof LegacyDbExecError ? dbError : undefined; }; /** @@ -513,7 +533,7 @@ const execMigrationBatch = ( fs: FileSystem.FileSystem, path: Path.Path, migrationPath: string, - mapError: (message: string, phase: "read" | "exec") => E, + mapError: (message: string, phase: "read" | "exec", dbError?: LegacyDbExecError) => E, forceNoVersion: boolean, displayPath: string = migrationPath, projectEnv: Readonly> = {}, @@ -694,7 +714,11 @@ const execMigrationBatch = ( pending = [...pending, { kind: "version" }]; } yield* flushBatch; - }).pipe(Effect.mapError((error) => mapError(legacyErrorMessage(error), "exec"))); + }).pipe( + Effect.mapError((error) => + mapError(legacyErrorMessage(error), "exec", formattedExecBatchDbError(error)), + ), + ); }); /** @@ -719,20 +743,29 @@ const resetConnectionState = ( * the history table, then run the file's statements + the history insert. * * `mapError` lets the caller tag the failure (e.g. `LegacyPgDeltaDeclarativeApplyError`). + * Statement failures also expose their structured PostgreSQL error so local replay + * can classify precise SQLSTATE/object combinations without parsing formatted context. */ export const legacyApplyMigrationFile = ( session: LegacyDbSession, fs: FileSystem.FileSystem, path: Path.Path, migrationPath: string, - mapError: (message: string) => E, + mapError: (message: string, dbError?: LegacyDbExecError) => E, ): Effect.Effect => Effect.gen(function* () { yield* resetConnectionState(session, mapError); yield* legacyCreateMigrationTable(session).pipe( Effect.mapError((e) => mapError(legacyErrorMessage(e))), ); - yield* execMigrationBatch(session, fs, path, migrationPath, mapError, false); + yield* execMigrationBatch( + session, + fs, + path, + migrationPath, + (message, _phase, dbError) => mapError(message, dbError), + false, + ); }); /** diff --git a/apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts b/apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts new file mode 100644 index 0000000000..e9d9894990 --- /dev/null +++ b/apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts @@ -0,0 +1,21 @@ +import type { LegacyDbExecError } from "./legacy-db-connection.errors.ts"; + +/** Canonical remediation when local migrations need pg_net but webhooks are disabled. */ +export const LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION = + "Add the following to supabase/config.toml and retry:\n\n" + + "[experimental.webhooks]\n" + + "enabled = true"; + +const MISSING_NET_SCHEMA_PATTERN = /schema "net" does not exist/iu; +const MISSING_PG_NET_FUNCTION_PATTERN = /function net\.http_[a-z0-9_]*\([^)]*\) does not exist/iu; + +/** + * Classifies the PostgreSQL failures produced when a migration calls pg_net while + * the extension is unavailable. SQLSTATE keeps similarly worded client errors out; + * the server-reported schema/function identity keeps unrelated undefined objects out. + */ +export const legacyIsPgNetUnavailableError = ( + error: Pick, +): boolean => + (error.code === "3F000" && MISSING_NET_SCHEMA_PATTERN.test(error.message)) || + (error.code === "42883" && MISSING_PG_NET_FUNCTION_PATTERN.test(error.message)); From 3f0428f5d781a1d06431cc19011dc73fc9f6d151 Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 11 Aug 2026 18:05:11 +0200 Subject: [PATCH 20/82] fix(cli): scope pg-delta next schema filters --- .../legacy-pgdelta-next-adapter.layer.ts | 22 +- .../legacy-pgdelta-next-adapter.unit.test.ts | 206 ++++++++++++++++-- 2 files changed, 206 insertions(+), 22 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index ff1bd0f861..b2b1bdb713 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -318,17 +318,21 @@ export function legacyPgDeltaNextProfile( const policy: Policy = { id: `supabase-cli-schemas:${selected.join(",")}`, filter: [ - { - match: { all: [{ schema: "*" }, { not: { schema: selected } }] }, - action: "exclude", - }, - { - match: { all: [{ kind: "schema" }, { not: { name: selected } }] }, - action: "exclude", - }, { match: { - all: [{ target: { schema: "*" } }, { not: { target: { schema: selected } } }], + all: [ + { verb: ["add", "remove", "set", "link", "unlink"] }, + { + not: { + any: [ + { schema: selected }, + { all: [{ kind: "schema" }, { name: selected }] }, + { target: { schema: selected } }, + { target: { kind: "schema", name: selected } }, + ], + }, + }, + ], }, action: "exclude", }, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index 98fdaedd9e..d45956ff66 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -1,5 +1,13 @@ import { it } from "@effect/vitest"; -import { ShadowLoadError } from "@supabase/pg-delta/frontends"; +import { + buildFactBase, + encodeId, + type DependencyEdge, + type Fact, + type StableId, +} from "@supabase/pg-delta/core"; +import { renderPlanFiles, ShadowLoadError } from "@supabase/pg-delta/frontends"; +import { plan } from "@supabase/pg-delta/plan"; import { Effect } from "effect"; import { Pool } from "pg"; import { describe, expect } from "vitest"; @@ -285,23 +293,32 @@ describe("LegacyPgDeltaNextAdapter", () => { ).toEqual(["log_min_messages"]); }); - it("composes schema exclusions ahead of the Supabase managed-view policy", () => { + it("composes the operation-scoped schema complement ahead of the Supabase policy", () => { const profile = legacyPgDeltaNextProfile(["public", "tenant"]); expect(profile.id).toBe("supabase"); expect(profile.policy?.filter).toEqual([ - { - match: { all: [{ schema: "*" }, { not: { schema: ["public", "tenant"] } }] }, - action: "exclude", - }, - { - match: { - all: [{ kind: "schema" }, { not: { name: ["public", "tenant"] } }], - }, - action: "exclude", - }, { match: { - all: [{ target: { schema: "*" } }, { not: { target: { schema: ["public", "tenant"] } } }], + all: [ + { verb: ["add", "remove", "set", "link", "unlink"] }, + { + not: { + any: [ + { schema: ["public", "tenant"] }, + { + all: [{ kind: "schema" }, { name: ["public", "tenant"] }], + }, + { target: { schema: ["public", "tenant"] } }, + { + target: { + kind: "schema", + name: ["public", "tenant"], + }, + }, + ], + }, + }, + ], }, action: "exclude", }, @@ -309,6 +326,169 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(profile.policy?.extends).toHaveLength(1); }); + it("renders only selected-schema state while preserving its metadata and dependencies", () => { + const schemaPublic = { kind: "schema", name: "public" } satisfies StableId; + const schemaAuth = { kind: "schema", name: "auth" } satisfies StableId; + const existingRole = { kind: "role", name: "app_owner" } satisfies StableId; + const existingExtension = { kind: "extension", name: "hstore" } satisfies StableId; + const selectedTable = { + kind: "table", + schema: "public", + name: "selected_items", + } satisfies StableId; + const unselectedSchema = { kind: "schema", name: "private_data" } satisfies StableId; + const unselectedTable = { + kind: "table", + schema: "private_data", + name: "hidden_items", + } satisfies StableId; + const platformTable = { + kind: "table", + schema: "auth", + name: "hidden_platform_table", + } satisfies StableId; + const customRole = { kind: "role", name: "hidden_custom_role" } satisfies StableId; + const customExtension = { + kind: "extension", + name: "hidden_custom_extension", + } satisfies StableId; + const customPublication = { + kind: "publication", + name: "hidden_custom_publication", + } satisfies StableId; + const customFdw = { kind: "fdw", name: "hidden_custom_fdw" } satisfies StableId; + + const fact = (id: StableId, payload: Fact["payload"] = {}, parent?: StableId): Fact => + parent === undefined ? { id, payload } : { id, parent, payload }; + + const sourceFacts: Fact[] = [ + fact(schemaPublic), + fact(schemaAuth), + fact(existingRole, { login: false, config: [] }), + fact(existingExtension, { schema: "public", relocatable: true }), + ]; + const sourceEdges: DependencyEdge[] = [ + { from: existingExtension, to: schemaPublic, kind: "depends" }, + ]; + const desiredFacts: Fact[] = [ + ...sourceFacts, + fact( + selectedTable, + { persistence: "p", partitionBound: null, partitionKey: null, parentTable: null }, + schemaPublic, + ), + fact( + { kind: "comment", target: selectedTable }, + { text: "selected table metadata" }, + selectedTable, + ), + fact( + { kind: "comment", target: schemaPublic }, + { text: "selected schema metadata" }, + schemaPublic, + ), + fact( + { kind: "acl", target: selectedTable, grantee: "PUBLIC" }, + { privileges: ["SELECT"], grantable: [] }, + selectedTable, + ), + fact(unselectedSchema), + fact( + unselectedTable, + { persistence: "p", partitionBound: null, partitionKey: null, parentTable: null }, + unselectedSchema, + ), + fact( + { kind: "comment", target: unselectedTable }, + { text: "unselected metadata" }, + unselectedTable, + ), + fact( + platformTable, + { persistence: "p", partitionBound: null, partitionKey: null, parentTable: null }, + schemaAuth, + ), + fact(customRole, { login: true, config: [] }), + fact(customExtension, { schema: "public", relocatable: true }), + fact(customPublication, { + allTables: false, + publish: ["insert", "update"], + viaRoot: false, + }), + fact(customFdw, { handler: null, validator: null, options: [] }), + ]; + const desiredEdges: DependencyEdge[] = [ + ...sourceEdges, + { from: selectedTable, to: existingExtension, kind: "depends" }, + { from: selectedTable, to: existingRole, kind: "owner" }, + ]; + + const profile = legacyPgDeltaNextProfile(["public", "auth"]); + const generated = plan( + buildFactBase(sourceFacts, sourceEdges), + buildFactBase(desiredFacts, desiredEdges), + { policy: profile.policy }, + ); + const rendered = renderPlanFiles(generated, { allowDrops: true }); + const sql = rendered.files.map((file) => file.contents).join("\n"); + + expect(sql).toContain('CREATE TABLE "public"."selected_items"'); + expect(sql).toContain("selected table metadata"); + expect(sql).toContain("selected schema metadata"); + expect(sql).toContain('GRANT SELECT ON TABLE "public"."selected_items" TO PUBLIC'); + expect(sql).toContain('OWNER TO "app_owner"'); + for (const leakedName of [ + "private_data", + "hidden_items", + "hidden_platform_table", + "hidden_custom_role", + "hidden_custom_extension", + "hidden_custom_publication", + "hidden_custom_fdw", + "unselected metadata", + ]) { + expect(sql).not.toContain(leakedName); + } + + expect(generated.deltas).toContainEqual({ + verb: "link", + edge: { from: selectedTable, to: existingExtension, kind: "depends" }, + }); + expect(generated.deltas).toContainEqual({ + verb: "link", + edge: { from: selectedTable, to: existingRole, kind: "owner" }, + }); + const filtered = generated.filteredDeltas.map((delta) => { + switch (delta.verb) { + case "add": + case "remove": + return encodeId(delta.fact.id); + case "set": + return encodeId(delta.id); + case "link": + case "unlink": + return encodeId(delta.edge.from); + } + }); + expect(filtered).toEqual( + expect.arrayContaining([ + encodeId(unselectedSchema), + encodeId(unselectedTable), + encodeId(customRole), + encodeId(customExtension), + encodeId(customPublication), + encodeId(customFdw), + ]), + ); + expect(filtered).not.toContain(encodeId(platformTable)); + expect( + generated.projectionAudit?.entries.some( + (entry) => + entry.delta.verb === "add" && encodeId(entry.delta.fact.id) === encodeId(platformTable), + ), + ).toBe(true); + }); + it.effect("constructs the real adapter from supported public pg-delta subpaths", () => Effect.gen(function* () { const adapter = yield* LegacyPgDeltaNextAdapter; From 1bc9e6b58e537e63200991e8267df064e194190e Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 11 Aug 2026 18:06:18 +0200 Subject: [PATCH 21/82] fix(cli): diff pg-delta next against live local database --- .../commands/db/diff/diff.integration.test.ts | 14 +++++++++++++- .../legacy/commands/db/pull/pull.handler.ts | 9 +++++---- .../commands/db/pull/pull.integration.test.ts | 19 ++++++++++++++++++- .../db/shared/legacy-shadow-source.ts | 9 +++++---- 4 files changed, 41 insertions(+), 10 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 708df3556a..2842b2ce5d 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -487,6 +487,18 @@ describe("legacy db diff", () => { yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); expect(s.databaseDiffCalls[0]).not.toHaveProperty("declarativeFiles"); expect(s.databaseDiffCalls[0]).not.toHaveProperty("declarativeManifest"); + expect(s.shadowConnectedDatabases).not.toContain("contrib_regression"); + expect(s.databaseDiffCalls[0]?.target.ref).toContain("@127.0.0.1:54322/postgres"); + expect(s.databaseDiffCalls[0]?.target).toMatchObject({ + connection: { + host: "127.0.0.1", + port: 54322, + user: "postgres", + password: "postgres", + database: "postgres", + }, + connectOptions: { isLocal: true, dnsResolver: "native" }, + }); expect(stderr(s.out)).toContain("schema_paths no longer changes the migrations baseline"); expect(stderr(s.out)).not.toContain("db diff -f uses supabase/migrations"); expect(stdout(s.out)).toBe("create table result ();\n\n"); @@ -672,7 +684,7 @@ describe("legacy db diff", () => { ); it.effect( - "provisions a local-target declarative shadow and diffs against the override database", + "migra provisions a local-target declarative shadow and diffs against the override database", () => { // A declarative schema file under supabase/schemas makes `loadDeclaredSchemas` // non-empty, so the native `--target-local` branch redirects the diff target to diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index c1abe4a1d1..e716162b3e 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -818,10 +818,11 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // NOT do — it also resolves after its own banner), would both print nothing on an // image-resolution failure before the banner and skip re-resolving it on retry. const resolvedPullShadowImage = yield* pullLocalInputs.resolvePostgresImage; - // Mirror Go's `DiffDatabase` → `PrepareShadowSource(ctx, schema, - // utils.IsLocalDatabase(config), …)` (`internal/db/diff/diff.go:213`): a - // local target with declarative schema files gets a second - // `contrib_regression` shadow returned as the target override. + // Legacy engines mirror Go's `DiffDatabase` → `PrepareShadowSource(ctx, schema, + // utils.IsLocalDatabase(config), …)` (`internal/db/diff/diff.go:213`): a local + // target with declarative schema files gets a second `contrib_regression` shadow + // returned as the target override. Pg-delta next compares the migrations shadow + // directly to the live target instead. const migrationMode: "legacy" | "pgdelta-next" = usePgDeltaDiff && pgDeltaEngine.implementation === "next" ? "pgdelta-next" : "legacy"; const shadowInput = { diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 9f4db7a90a..f56e3dea04 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -1786,7 +1786,24 @@ describe("legacy db pull", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("db pull --local provisions a local-target shadow and uses the target override", () => { + it.effect("db pull --local with pg-delta-next diffs against the live local database", () => { + seedMigration(tmp.current, "20240101000000"); + mkdirSync(join(tmp.current, "supabase", "schemas"), { recursive: true }); + writeFileSync(join(tmp.current, "supabase", "schemas", "public.sql"), "select 1;\n"); + const s = setup(tmp.current, { + engineImplementation: "next", + remoteVersions: ["20240101000000"], + edgeStdout: pgDeltaDiffEnvelope([{ name: "schema_changes", sql: "create table remote ();" }]), + yes: true, + }); + return Effect.gen(function* () { + yield* legacyDbPull(flags({ local: Option.some(true), diffEngine: Option.some("pg-delta") })); + expect(s.connectedDatabases).not.toContain("contrib_regression"); + expect(s.engineCalls[0]?.targetRef).toContain("@127.0.0.1:5432/postgres"); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("db pull --local with migra uses the declarative target override", () => { // Go derives the shadow targetLocal from utils.IsLocalDatabase and substitutes // the declarative contrib_regression target override (diff.go:190,196-197); a // real declarative schema file makes the native `loadDeclaredSchemas` branch diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts index 19bd81c47a..1d77f9cc3a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts @@ -175,7 +175,7 @@ export interface LegacyPrepareShadowSourceInput extends LegacyShadowConnectio readonly targetLocal: boolean; /** Selects the declarative-apply engine for the local-declared branch, matching `DiffDatabase`. */ readonly usePgDelta: boolean; - /** Selects the historical shadow baseline or pg-delta next's config-gated baseline. */ + /** Selects the shadow baseline and whether a local target may use the legacy declarative override. */ readonly migrationMode?: "legacy" | "pgdelta-next"; /** `db.migrations.schema_paths`, RAW (unresolved) — Go's `Config.Db.Migrations.SchemaPaths` pre-`config.go:976-979`-resolution form. */ readonly schemaPaths: ReadonlyArray; @@ -197,8 +197,9 @@ export type LegacyPrepareShadowSourceError = * Port of Go's `PrepareShadowSource` (`apps/cli-go/internal/db/diff/shadow.go:37-91`): * health-wait against an already-`legacyCreateShadowDatabase`-created shadow -> * `MigrateShadowDatabase` (platform baseline + local migrations + the `contrib_regression` - * template database) -> build the diff-source config -> when `targetLocal`, the - * declarative-schema override branch. + * template database) -> build the diff-source config -> for legacy local targets, the + * declarative-schema override branch. Pg-delta next always compares that migrations shadow + * directly to the live target. * * Deliberately does NOT call `legacyCreateShadowDatabase` (`shadow-database.ts`) itself, and * no longer wraps its own body in `Effect.onError` cleanup — the caller does both, structuring @@ -273,7 +274,7 @@ export const legacyPrepareShadowSource = ( const sourceUrl = legacyToPostgresURL(connConfig); let targetUrlOverride: string | undefined; - if (input.targetLocal) { + if (input.targetLocal && input.migrationMode !== "pgdelta-next") { const declared = yield* legacyLoadDeclaredSchemas( input.fs, input.path, From be38f7b83f3fdb0a9b6f9add3a964ab139b95e18 Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 11 Aug 2026 18:06:33 +0200 Subject: [PATCH 22/82] fix(cli): preserve migration transaction metadata --- .../commands/db/push/push.integration.test.ts | 2 +- .../db/reset/reset.integration.test.ts | 9 ++- .../migration/fetch/fetch.integration.test.ts | 76 ++++++++++++++++++- .../repair/repair.integration.test.ts | 25 ++++++ .../legacy-migration-apply.unit.test.ts | 5 +- .../legacy/shared/legacy-migration-file.ts | 3 +- .../shared/legacy-migration-file.unit.test.ts | 22 ++++-- 7 files changed, 129 insertions(+), 13 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/push/push.integration.test.ts b/apps/cli/src/legacy/commands/db/push/push.integration.test.ts index 9b2223d57f..345c48612d 100644 --- a/apps/cli/src/legacy/commands/db/push/push.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/push/push.integration.test.ts @@ -365,7 +365,7 @@ describe("legacy db push", () => { return Effect.gen(function* () { yield* legacyDbPush(DEFAULT_FLAGS).pipe(Effect.provide(layer)); const setupCommit = conn.execs.indexOf("COMMIT"); - const setIndex = conn.execs.indexOf(set); + const setIndex = conn.execs.indexOf(`-- pg-delta: transaction=false\n${set}`); const actionIndex = conn.execs.indexOf(action); const cleanupIndex = conn.execs.lastIndexOf("RESET ALL"); diff --git a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts index f63a1d9094..417bc959a2 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts @@ -1423,14 +1423,19 @@ describe("legacy db reset", () => { return Effect.gen(function* () { yield* legacyDbReset({ ...DEFAULT_FLAGS, linked: true }).pipe(Effect.provide(layer)); const setupCommit = conn.execs.indexOf("COMMIT"); - const setIndex = conn.execs.indexOf(set); + const firstStatement = `-- pg-delta: transaction=false\n${set}`; + const setIndex = conn.execs.indexOf(firstStatement); const actionIndex = conn.execs.indexOf(action); const cleanupIndex = conn.execs.lastIndexOf("RESET ALL"); expect(setIndex).toBeGreaterThan(setupCommit); expect(actionIndex).toBeGreaterThan(setIndex); expect(cleanupIndex).toBeGreaterThan(actionIndex); - expect(conn.execs.slice(setIndex, cleanupIndex + 1)).toEqual([set, action, "RESET ALL"]); + expect(conn.execs.slice(setIndex, cleanupIndex + 1)).toEqual([ + firstStatement, + action, + "RESET ALL", + ]); expect( conn.queries.some((query) => query.sql.includes("INSERT INTO supabase_migrations")), ).toBe(true); diff --git a/apps/cli/src/legacy/commands/migration/fetch/fetch.integration.test.ts b/apps/cli/src/legacy/commands/migration/fetch/fetch.integration.test.ts index b4c408487d..d66910a38e 100644 --- a/apps/cli/src/legacy/commands/migration/fetch/fetch.integration.test.ts +++ b/apps/cli/src/legacy/commands/migration/fetch/fetch.integration.test.ts @@ -2,7 +2,7 @@ import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from import { join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; -import { Cause, Effect, Exit, Layer, Option } from "effect"; +import { Cause, Effect, Exit, FileSystem, Layer, Option, Path } from "effect"; import { LEGACY_VALID_REF, @@ -22,7 +22,11 @@ import type { LegacyDbConfigFlags, LegacyResolvedDbConfig, } from "../../../shared/legacy-db-config.types.ts"; +import { LegacyDbExecError } from "../../../shared/legacy-db-connection.errors.ts"; import { LegacyDbConnection } from "../../../shared/legacy-db-connection.service.ts"; +import type { LegacyDbSession } from "../../../shared/legacy-db-connection.service.ts"; +import { legacyApplyMigrationFile } from "../../../shared/legacy-migration-apply.ts"; +import { legacyParseMigrationContent } from "../../../shared/legacy-migration-file.ts"; import { legacyMigrationFetch } from "./fetch.handler.ts"; import type { LegacyMigrationFetchFlags } from "./fetch.command.ts"; @@ -133,6 +137,16 @@ const flags = (over: Partial = {}): LegacyMigrationFe const migrationsDir = (workdir: string) => join(workdir, "supabase", "migrations"); const tmp = useLegacyTempWorkdir(); +function stringArray(value: unknown): ReadonlyArray | undefined { + if (!Array.isArray(value)) return undefined; + const values: Array = []; + for (const item of value) { + if (typeof item !== "string") return undefined; + values.push(item); + } + return values; +} + describe("legacy migration fetch", () => { it.live("writes migration files joined with the Go separator when the dir is empty", () => { const { layer, out } = setup(tmp.current, { @@ -155,6 +169,66 @@ describe("legacy migration fetch", () => { }).pipe(Effect.provide(layer)); }); + it.live("preserves no-transaction metadata through apply, history, and fetch", () => { + const rows: Array = []; + const source = join(tmp.current, "20240102000000_drop_subscription.sql"); + writeFileSync( + source, + "\uFEFF-- pg-delta: transaction=false\r\n" + + "SET check_function_bodies = off;\r\n" + + "DROP SUBSCRIPTION app_events;\r\n" + + "RESET ALL;\r\n", + ); + const applySession: LegacyDbSession = { + exec: () => Effect.void, + query: (_sql, params) => + Effect.sync(() => { + const version = params?.[0]; + const name = params?.[1]; + const statements = stringArray(params?.[2]); + if (typeof version === "string" && typeof name === "string" && statements !== undefined) { + rows.push({ version, name, statements }); + } + return []; + }), + extensionExists: () => Effect.succeed(false), + copyToCsv: () => Effect.succeed(new Uint8Array()), + queryRaw: () => Effect.succeed({ fields: [], rows: [], commandTag: "" }), + }; + const { layer } = setup(tmp.current, { rows }); + + return Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + yield* legacyApplyMigrationFile( + applySession, + fs, + path, + source, + (message) => new LegacyDbExecError({ message }), + ); + + const firstStatement = "-- pg-delta: transaction=false\r\nSET check_function_bodies = off"; + expect(rows).toEqual([ + { + version: "20240102000000", + name: "drop_subscription", + statements: [firstStatement, "DROP SUBSCRIPTION app_events", "RESET ALL"], + }, + ]); + + yield* legacyMigrationFetch(flags()); + const fetched = readFileSync( + join(migrationsDir(tmp.current), "20240102000000_drop_subscription.sql"), + "utf8", + ); + expect(legacyParseMigrationContent(fetched)).toEqual({ + statements: [firstStatement, "DROP SUBSCRIPTION app_events", "RESET ALL"], + transactionMode: "none", + }); + }).pipe(Effect.provide(layer)); + }); + it.live("writes a lone separator for a row with no statements (Go parity)", () => { // A `schema_migrations` row can legally have a NULL/empty `statements` array // (older projects, manually-inserted rows). Go does `strings.Join(stmts, ";\n") diff --git a/apps/cli/src/legacy/commands/migration/repair/repair.integration.test.ts b/apps/cli/src/legacy/commands/migration/repair/repair.integration.test.ts index d00cb96677..ea2a790652 100644 --- a/apps/cli/src/legacy/commands/migration/repair/repair.integration.test.ts +++ b/apps/cli/src/legacy/commands/migration/repair/repair.integration.test.ts @@ -159,6 +159,31 @@ describe("legacy migration repair", () => { }).pipe(Effect.provide(layer)); }); + it.live("preserves a no-transaction directive when repairing applied history", () => { + seedMigration( + tmp.current, + "20240102000000_drop_subscription.sql", + "\uFEFF-- pg-delta: transaction=false\r\n" + + "SET check_function_bodies = off;\r\n" + + "DROP SUBSCRIPTION app_events;\r\n" + + "RESET ALL;\r\n", + ); + const { layer, queries } = setup(tmp.current); + return Effect.gen(function* () { + yield* legacyMigrationRepair(input({ versions: ["20240102000000"], status: "applied" })); + const upsert = queries.find((query) => query.sql.includes("ON CONFLICT")); + expect(upsert?.params).toEqual([ + "20240102000000", + "drop_subscription", + [ + "-- pg-delta: transaction=false\r\nSET check_function_bodies = off", + "DROP SUBSCRIPTION app_events", + "RESET ALL", + ], + ]); + }).pipe(Effect.provide(layer)); + }); + it.live("resolves the DB target before parsing positional versions", () => { // Go's cobra order runs ParseDatabaseConfig (PersistentPreRunE, root.go:118) before // repair.Run's strconv.Atoi loop, so an unlinked target error wins over a bad version. diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts index 1b6a5f549a..88faaf272f 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts @@ -224,7 +224,8 @@ describe("legacyApplyMigrationFile", () => { Effect.sync(() => { const execs = calls.filter((call) => call.kind === "exec").map((call) => call.sql); const setupCommit = execs.indexOf("COMMIT"); - const set = execs.indexOf("SET check_function_bodies = off"); + const firstStatement = "-- pg-delta: transaction=false\nSET check_function_bodies = off"; + const set = execs.indexOf(firstStatement); const action = execs.indexOf("DROP SUBSCRIPTION app_events"); const cleanup = execs.lastIndexOf("RESET ALL"); @@ -241,7 +242,7 @@ describe("legacyApplyMigrationFile", () => { expect(history[0]?.params).toEqual([ "20240101120000", "drop_subscription", - ["SET check_function_bodies = off", "DROP SUBSCRIPTION app_events", "RESET ALL"], + [firstStatement, "DROP SUBSCRIPTION app_events", "RESET ALL"], ]); rmSync(dir, { recursive: true, force: true }); }), diff --git a/apps/cli/src/legacy/shared/legacy-migration-file.ts b/apps/cli/src/legacy/shared/legacy-migration-file.ts index 6f2d52668e..6d4b2189e0 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-file.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-file.ts @@ -26,9 +26,8 @@ export function legacyParseMigrationContent(content: string): LegacyParsedMigrat const firstLine = rawFirstLine.endsWith("\r") ? rawFirstLine.slice(0, -1) : rawFirstLine; if (firstLine === PG_DELTA_NO_TRANSACTION_DIRECTIVE) { - const sql = firstNewline < 0 ? "" : withoutBom.slice(firstNewline + 1); return { - statements: legacySplitAndTrim(sql), + statements: legacySplitAndTrim(withoutBom), transactionMode: "none", }; } diff --git a/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts index d3d265007b..7897961665 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts @@ -9,12 +9,24 @@ import { describe("legacyParseMigrationContent", () => { it.each([ - ["LF", "-- pg-delta: transaction=false\nSET check_function_bodies = off;"], - ["CRLF", "-- pg-delta: transaction=false\r\nSET check_function_bodies = off;"], - ["a UTF-8 BOM", "\uFEFF-- pg-delta: transaction=false\nSET check_function_bodies = off;"], - ])("recognizes the anchored no-transaction directive with %s", (_name, content) => { + [ + "LF", + "-- pg-delta: transaction=false\nSET check_function_bodies = off;", + "-- pg-delta: transaction=false\nSET check_function_bodies = off", + ], + [ + "CRLF", + "-- pg-delta: transaction=false\r\nSET check_function_bodies = off;", + "-- pg-delta: transaction=false\r\nSET check_function_bodies = off", + ], + [ + "a UTF-8 BOM", + "\uFEFF-- pg-delta: transaction=false\nSET check_function_bodies = off;", + "-- pg-delta: transaction=false\nSET check_function_bodies = off", + ], + ])("recognizes the anchored no-transaction directive with %s", (_name, content, statement) => { expect(legacyParseMigrationContent(content)).toEqual({ - statements: ["SET check_function_bodies = off"], + statements: [statement], transactionMode: "none", }); }); From 0139811013850ccaded00161871b8698aeb13779 Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 11 Aug 2026 18:06:46 +0200 Subject: [PATCH 23/82] fix(cli): converge database webhooks configuration --- .../db/shared/legacy-shadow-source.ts | 2 + .../legacy/commands/db/start/start.handler.ts | 1 + .../db/start/start.integration.test.ts | 21 ++++- .../legacy/commands/start/start.handler.ts | 12 +-- .../commands/start/start.integration.test.ts | 16 ++++ .../legacy/shared/db-bootstrap/db-setup.ts | 90 ++++++++++++++----- .../shared/db-bootstrap/db-setup.unit.test.ts | 10 ++- .../shared/db-bootstrap/shadow-database.ts | 2 + .../db-bootstrap/shadow-database.unit.test.ts | 45 +++++++++- .../shared/db-bootstrap/start-database.ts | 12 +++ 10 files changed, 180 insertions(+), 31 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts index 1d77f9cc3a..560e488e66 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts @@ -99,6 +99,7 @@ export function legacyShadowRunInputFromLocalContainerInputs( toml: { readonly shadowPort: number; readonly password: string; + readonly webhooksEnabled: boolean; readonly baseline: { readonly apiAutoExposeNewTables: Option.Option }; readonly vault: ReadonlyArray; }, @@ -134,6 +135,7 @@ export function legacyShadowRunInputFromLocalContainerInputs( setup: { majorVersion: localInputs.setup.majorVersion, config: localInputs.setup.config, + webhooksEnabled: toml.webhooksEnabled, // NOT `localInputs.setup.dbUrl` — that carries the REGULAR local container's own // hardcoded-"postgres" password (`legacy-local-config-values.ts`'s `DEFAULT_DB_PASSWORD`), // for a DIFFERENT container. The shadow's own one-shot setup jobs diff --git a/apps/cli/src/legacy/commands/db/start/start.handler.ts b/apps/cli/src/legacy/commands/db/start/start.handler.ts index e032cde37b..aeccb2576a 100644 --- a/apps/cli/src/legacy/commands/db/start/start.handler.ts +++ b/apps/cli/src/legacy/commands/db/start/start.handler.ts @@ -1030,6 +1030,7 @@ export const legacyDbStart = Effect.fn("legacy.db.start")(function* (flags: Lega // (`DockerResolveImageIfNotCached`, `internal/utils/docker.go:363-365`). resolvePostgresImage, dbHealthTimeoutSeconds: bootstrapConfig.dbHealthTimeoutSeconds, + webhooksEnabled: dbTomlValues.webhooksEnabled, // Go's `initSchema15`'s realtime job resolves JWKS itself, LOCALLY, gated on // `Realtime.Enabled` (`internal/db/start/start.go:337-341`) — unlike `supabase // start`'s OWN unconditional, up-front `ResolveJWKS` call (which also feeds the diff --git a/apps/cli/src/legacy/commands/db/start/start.integration.test.ts b/apps/cli/src/legacy/commands/db/start/start.integration.test.ts index caccfcc8e2..3212350572 100644 --- a/apps/cli/src/legacy/commands/db/start/start.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/start/start.integration.test.ts @@ -42,6 +42,7 @@ import { legacyDbStart } from "./start.handler.ts"; import type { LegacyDbStartFlags } from "./start.command.ts"; const DEFAULT_FLAGS: LegacyDbStartFlags = { fromBackup: Option.none() }; +const PG_NET_CREATE_FINGERPRINT = "create extension if not exists pg_net schema extensions"; function flags(fromBackup?: string): LegacyDbStartFlags { return { fromBackup: fromBackup === undefined ? Option.none() : Option.some(fromBackup) }; @@ -589,17 +590,35 @@ describe("legacy db start", () => { it.live( "restarts against an existing volume: skips the SetupLocalDatabase-equivalent pipeline but still writes _current_branch", () => { - const { layer, out, child } = setup(); + const { layer, out, child, dbSession } = setup(); return Effect.gen(function* () { yield* legacyDbStart(DEFAULT_FLAGS).pipe(Effect.provide(layer)); expect(out.stderrText).toContain("Starting database from backup...\n"); expect(out.stderrText).not.toContain("Initialising schema..."); expect(dbSetupJobCalls(child.spawned)).toHaveLength(0); + expect(dbSession.calls).toHaveLength(0); expect(readFileSync(currentBranchPath(tempRoot.current), "utf8")).toBe("main"); }); }, ); + it.live("installs pg_net on an existing volume from effective Webhooks config", () => { + const { layer, out, child, dbSession } = setup({ + configContents: 'project_id = "test"\n[experimental.webhooks]\nenabled = false\n', + projectEnvContents: "SUPABASE_EXPERIMENTAL_WEBHOOKS_ENABLED=true\n", + }); + return Effect.gen(function* () { + yield* legacyDbStart(DEFAULT_FLAGS).pipe(Effect.provide(layer)); + expect(out.stderrText).not.toContain("Initialising schema..."); + expect(dbSetupJobCalls(child.spawned)).toHaveLength(0); + expect( + dbSession.calls.filter( + (call) => call.kind === "exec" && call.sql.includes(PG_NET_CREATE_FINGERPRINT), + ), + ).toHaveLength(1); + }); + }); + it.live( "--from-backup on a fresh volume: uses the restore entrypoint, binds the backup file, and skips the SetupLocalDatabase-equivalent pipeline entirely", () => { diff --git a/apps/cli/src/legacy/commands/start/start.handler.ts b/apps/cli/src/legacy/commands/start/start.handler.ts index b7399f37a0..05d7568b88 100644 --- a/apps/cli/src/legacy/commands/start/start.handler.ts +++ b/apps/cli/src/legacy/commands/start/start.handler.ts @@ -620,11 +620,12 @@ export const legacyStart = Effect.fn("legacy.start")(function* (flags: LegacySta // ran this inside `legacyStartSetupLocalDatabase`, which is itself gated on the DB container's // healthcheck passing AND a fresh volume (Go's `NoBackupVolume` gate) — so a malformed // `SUPABASE_DB_SEED_ENABLED`/an undecryptable `[db.vault]` secret went completely unvalidated - // whenever `start` reused an existing volume. Called here purely for its validation side - // effect and discarded — `legacyStartSetupLocalDatabase`'s own internal call (an already- - // accepted duplicate config-load pass, matching `db start`'s own independent resolution — see - // `../../shared/db-bootstrap/db-setup.ts`'s header) still resolves the real value for its own use when it runs. - yield* legacyCheckDbToml(fs, path, cliConfig.workdir).pipe(Effect.asVoid); + // whenever `start` reused an existing volume. The resolved Webhooks flag is also retained so + // existing volumes can converge `pg_net`; `legacyStartSetupLocalDatabase`'s own internal call + // (an already-accepted duplicate config-load pass, matching `db start`'s own independent + // resolution — see `../../shared/db-bootstrap/db-setup.ts`'s header) still resolves fresh-setup + // values for its own use when it runs. + const dbTomlValues = yield* legacyCheckDbToml(fs, path, cliConfig.workdir); const dbContainerId = localDbContainerId(projectId); const filterValue = legacyCliProjectFilterValue(projectId); @@ -1634,6 +1635,7 @@ export const legacyStart = Effect.fn("legacy.start")(function* (flags: LegacySta // `db start` (see `legacyStartDatabase`'s header for why this is caller-supplied). resolvePostgresImage: Effect.succeed(resolveImage(postgresImage)), dbHealthTimeoutSeconds, + webhooksEnabled: dbTomlValues.webhooksEnabled, setup: { majorVersion, experimental, diff --git a/apps/cli/src/legacy/commands/start/start.integration.test.ts b/apps/cli/src/legacy/commands/start/start.integration.test.ts index 2cace5b741..2dd245cbd6 100644 --- a/apps/cli/src/legacy/commands/start/start.integration.test.ts +++ b/apps/cli/src/legacy/commands/start/start.integration.test.ts @@ -69,6 +69,7 @@ import { * containers. */ const legacyResolveLocalConfigValuesCalls = vi.hoisted(() => ({ count: 0 })); +const PG_NET_CREATE_FINGERPRINT = "create extension if not exists pg_net schema extensions"; vi.mock("../../shared/legacy-local-config-values.ts", async () => { const actual = await vi.importActual( @@ -2603,6 +2604,21 @@ content_path = "./templates/custom_notice.html" }, ); + it.live("installs pg_net when restarting an existing Webhooks-enabled database", () => { + const { layer, out, dbSession } = setup({ + configContents: 'project_id = "demo"\n[experimental.webhooks]\nenabled = true\n', + }); + return Effect.gen(function* () { + yield* legacyStart(flags({ exclude: ["edge-runtime"] })); + expect(out.stderrText).not.toContain("Initialising schema..."); + expect( + dbSession.calls.filter( + (call) => call.kind === "exec" && call.sql.includes(PG_NET_CREATE_FINGERPRINT), + ), + ).toHaveLength(1); + }).pipe(Effect.provide(layer)); + }); + it.live( "still writes supabase/.branches/_current_branch on a restart, even though the fresh-volume DB setup is skipped", () => { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index 4837d35a19..d95e8fac39 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -361,6 +361,8 @@ export interface LegacySetupDatabaseInput { readonly workdir: string; /** The caller's already-resolved, effective config (env overrides already applied). */ readonly config: ProjectConfig; + /** Effective `[experimental.webhooks].enabled`, including supported environment overrides. */ + readonly webhooksEnabled: boolean; /** `db.major_version` (13-17) — Go's `utils.Config.Db.MajorVersion`, resolved by the caller once, ahead of the `db` container's own image tag selection. */ readonly majorVersion: number; /** @@ -459,7 +461,7 @@ export interface LegacySetupDatabaseOptions { /** Input to {@link legacyStartSetupLocalDatabase}. */ export interface LegacyStartSetupLocalDatabaseInput extends Omit< LegacySetupDatabaseInput, - "apiAutoExposeNewTables" | "vault" + "apiAutoExposeNewTables" | "vault" | "webhooksEnabled" > { /** * `--experimental`/`SUPABASE_EXPERIMENTAL`, resolved by the caller (Go's @@ -1022,8 +1024,7 @@ export const legacySetupDatabase = ( yield* legacyStartInitSchema(spawner, input, tmpDir); const activateUserExtensions = options.activateUserExtensions ?? true; const legacyPgNetBaseline = options.legacyPgNetBaseline ?? false; - const userEnabled = - activateUserExtensions && input.config.experimental.webhooks?.enabled === true; + const userEnabled = activateUserExtensions && input.webhooksEnabled; yield* legacyApplyDatabaseWebhooks( session, fs, @@ -1121,6 +1122,7 @@ export const legacyStartSetupLocalDatabase = ( // `apply.MigrateAndSeed` below. yield* legacySetupDatabase(spawner, { ...input, + webhooksEnabled: toml.webhooksEnabled, apiAutoExposeNewTables: toml.baseline.apiAutoExposeNewTables, vault: toml.vault, }); @@ -1261,6 +1263,65 @@ export interface LegacyFreshDbSetupInput { readonly debug: boolean; } +const legacyConnectLocalPostgres = (input: { + readonly hostname: string; + readonly dbPort: number; + readonly password: string; +}) => + Effect.gen(function* () { + const dbConnection = yield* LegacyDbConnection; + return yield* dbConnection + .connect( + { + host: input.hostname, + port: input.dbPort, + user: "postgres", + password: input.password, + database: "postgres", + }, + { isLocal: true, dnsResolver: "native" }, + ) + .pipe( + Effect.retry({ + schedule: Schedule.max([Schedule.spaced("1 seconds"), Schedule.recurs(10)]), + while: (error) => error.retryable === true, + }), + ); + }); + +/** Idempotently converges Database Webhooks on a healthy, existing local database. */ +export const legacyRunDatabaseWebhooksSetup = (input: { + readonly fs: FileSystem.FileSystem; + readonly path: Path.Path; + readonly hostname: string; + readonly dbPort: number; + readonly dbUrl: string; + readonly enabled: boolean; +}) => { + if (!input.enabled) return Effect.void; + return Effect.scoped( + Effect.gen(function* () { + const session = yield* legacyConnectLocalPostgres({ + hostname: input.hostname, + dbPort: input.dbPort, + password: legacyStartInternalDbPassword(input.dbUrl), + }); + const tmpDir = yield* input.fs + .makeTempDirectoryScoped({ prefix: "supabase-start-db-webhooks-" }) + .pipe( + Effect.mapError( + (error) => + new LegacyDbSetupError({ + message: `failed to create temp directory: ${errMessage(error)}`, + reason: "filesystem", + }), + ), + ); + yield* legacyApplyDatabaseWebhooks(session, input.fs, input.path, tmpDir, input.enabled); + }), + ); +}; + /** * Runs {@link legacyStartSetupLocalDatabase} against a freshly-provisioned local * Postgres — the exact sequence BOTH real Go callers run once Postgres's own @@ -1304,7 +1365,6 @@ export const legacyRunFreshDbSetup = ( > => Effect.scoped( Effect.gen(function* () { - const dbConnection = yield* LegacyDbConnection; const { setup } = input; const dbPassword = legacyStartInternalDbPassword(setup.dbUrl); // Go's `SetupLocalDatabase` dials this first host-facing connect exactly @@ -1312,23 +1372,11 @@ export const legacyRunFreshDbSetup = ( // failures: the container's internal health check says nothing about the // HOST side, where Docker Desktop (Windows/WSL2) can publish the port a // few seconds late (#6136). - const session = yield* dbConnection - .connect( - { - host: input.hostname, - port: input.dbPort, - user: "postgres", - password: dbPassword, - database: "postgres", - }, - { isLocal: true, dnsResolver: "native" }, - ) - .pipe( - Effect.retry({ - schedule: Schedule.max([Schedule.spaced("1 seconds"), Schedule.recurs(10)]), - while: (error) => error.retryable === true, - }), - ); + const session = yield* legacyConnectLocalPostgres({ + hostname: input.hostname, + dbPort: input.dbPort, + password: dbPassword, + }); const { jwks, images: dbSetupImages } = yield* legacyResolveDbSetupPrelude(setup); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts index 92d80caec7..f1f356b0af 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts @@ -607,15 +607,19 @@ describe("legacyStartSetupLocalDatabase", () => { ); }); - it.effect("installs pg_net when Database Webhooks is enabled without Edge Runtime", () => { + it.effect("installs pg_net from the effective Database Webhooks environment override", () => { const workdir = makeWorkdir(); - writeConfigToml(workdir, "[experimental.webhooks]\nenabled = true\n"); + writeConfigToml(workdir, "[experimental.webhooks]\nenabled = false\n"); + writeFileSync( + join(workdir, "supabase", ".env"), + "SUPABASE_EXPERIMENTAL_WEBHOOKS_ENABLED=true\n", + ); const { session, calls } = fakeSession(); const out = mockOutput(); const docker = mockDockerRun(); const config = decodeConfig({ edge_runtime: { enabled: false }, - experimental: { webhooks: { enabled: true } }, + experimental: { webhooks: { enabled: false } }, }); return run(baseInput(workdir, session, { majorVersion: 14, config }), out, docker).pipe( Effect.map(() => { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index 055842026f..8fd54c16d6 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -446,6 +446,7 @@ export const legacySetupShadowConn = ( * (`apiAutoExposeNewTables`/`vault`), threaded straight through here rather than re-read. */ export type LegacyShadowDbSetupInput = Omit, "experimental"> & { + readonly webhooksEnabled: LegacySetupDatabaseInput["webhooksEnabled"]; readonly apiAutoExposeNewTables: LegacySetupDatabaseInput["apiAutoExposeNewTables"]; readonly vault: LegacySetupDatabaseInput["vault"]; }; @@ -482,6 +483,7 @@ export const legacyBuildShadowSetupDatabaseInput = ( path: input.path, workdir: input.workdir, config: input.setup.config, + webhooksEnabled: input.setup.webhooksEnabled, majorVersion: input.setup.majorVersion, // Go's `container[:12]` — see this module's own header for why this resolves as a // hostname at all despite the shadow container having no name/alias. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index d2c1702575..954ee9afec 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -363,6 +363,7 @@ function baseSetupDatabaseInput( path, workdir, config: defaultConfig, + webhooksEnabled: false, majorVersion: 17, dbHost: "abcdef012345", projectId: "proj", @@ -419,7 +420,7 @@ describe("legacySetupShadowConn", () => { mock.spawner, { ...input, - config: decodeConfig({ experimental: { webhooks: { enabled: true } } }), + webhooksEnabled: true, }, { activateUserExtensions: false }, ); @@ -438,6 +439,7 @@ function baseShadowSetup( return { majorVersion: 17, config: defaultConfig, + webhooksEnabled: false, dbUrl: "postgresql://postgres:postgrespassword@127.0.0.1:54322/postgres", jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", jwks: Effect.succeed('{"keys":[]}') as Effect.Effect, @@ -636,6 +638,47 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { ); }); + it.effect( + "next migrated shadows install pg_net when effective Webhooks config is enabled", + () => { + const { session, calls } = fakeSession(); + const workdir = tempRoot.current; + const mock = mockSpawner(); + return Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + yield* fs.makeDirectory(path.join(workdir, "supabase", "migrations"), { recursive: true }); + yield* legacyMigrateNextShadowDatabase(mock.spawner, { + fs, + path, + workdir, + projectId: "proj", + container: "shadow-container-id-0123456789abcdef", + networkId: "supabase_network_proj", + connConfig: { + host: "127.0.0.1", + port: 54320, + user: "postgres", + password: "postgres", + database: "postgres", + }, + setup: baseShadowSetup({ webhooksEnabled: true }), + }); + expect(calls.some((call) => call.sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(true); + }).pipe( + Effect.provide( + Layer.mergeAll( + BunServices.layer, + mockOutput().layer, + mockDockerRun(), + mockRuntimeInfo(), + mockDbConnection(session), + ), + ), + ); + }, + ); + it.effect( "supports the extension-free declarative baseline on PG14 without resolving JWKS", () => { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/start-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/start-database.ts index 28be581aaf..19bfb04569 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/start-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/start-database.ts @@ -76,6 +76,7 @@ import { type LegacyVolumeInspectError, } from "./container-lifecycle.ts"; import { + legacyRunDatabaseWebhooksSetup, legacyRunFreshDbSetup, legacyStartInitCurrentBranch, type LegacyFreshDbSetupInput, @@ -155,6 +156,8 @@ export interface LegacyStartDatabaseInput { */ readonly resolvePostgresImage: Effect.Effect; readonly dbHealthTimeoutSeconds: number; + /** Effective `[experimental.webhooks].enabled`, used to converge existing volumes. */ + readonly webhooksEnabled: boolean; readonly setup: LegacyFreshDbSetupInput; /** * Fired synchronously, exactly once, right after the pre-create volume probe resolves — @@ -280,6 +283,15 @@ export const legacyStartDatabase = ( seedFlags: { noSeed: false, sqlPaths: [] }, setup: input.setup, }); + } else if (fromBackup === undefined) { + yield* legacyRunDatabaseWebhooksSetup({ + fs: input.fs, + path: input.path, + hostname: input.hostname, + dbPort: input.dbPort, + dbUrl: input.setup.dbUrl, + enabled: input.webhooksEnabled, + }); } // Go's `initCurrentBranch` (`db/start/start.go:189`) — the LAST line of `StartDatabase`, From ab8423b601aa0a58a83bb893de14d958b0d4a87c Mon Sep 17 00:00:00 2001 From: avallete Date: Wed, 12 Aug 2026 13:11:17 +0200 Subject: [PATCH 24/82] fix(cli): harden pg-delta schema workflows --- .../declarative/generate/generate.handler.ts | 16 +++-- .../generate/generate.integration.test.ts | 14 ++-- .../db/shared/legacy-pgdelta-files.ts | 59 +++++------------ .../shared/legacy-pgdelta-files.unit.test.ts | 65 +++++++++++++++++++ .../legacy-pgdelta-next-shadow.layer.ts | 30 ++++++++- ...acy-pgdelta-next-shadow.layer.unit.test.ts | 44 +++++++++++++ .../shared/legacy-pgdelta-next.live.test.ts | 63 ++++++++++++++++++ .../legacy/shared/db-bootstrap/db-setup.ts | 17 +++++ .../db-bootstrap/shadow-database.unit.test.ts | 14 +++- .../templates/db-initial-schema-14.sql.ts | 14 ++++ .../legacy/shared/legacy-migration-apply.ts | 15 ++++- .../legacy-migration-apply.unit.test.ts | 43 +++++++++++- 12 files changed, 333 insertions(+), 61 deletions(-) create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.unit.test.ts create mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts index 39de18a4b4..878e6723cc 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts @@ -136,15 +136,19 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec const declarativeDirRel = Option.getOrElse(flags.output, () => legacyResolveDeclarativeDir(path, toml.pgDelta), ); - const declarativeDir = path.resolve(cliConfig.workdir, declarativeDirRel); - if ( - declarativeDirRel.trim().length === 0 || - declarativeDir === path.resolve(cliConfig.workdir) - ) { + const workdir = path.resolve(cliConfig.workdir); + const declarativeDir = path.resolve(workdir, declarativeDirRel); + const workdirFromOutput = path.relative(declarativeDir, workdir); + const outputContainsWorkdir = + workdirFromOutput.length === 0 || + (!path.isAbsolute(workdirFromOutput) && + workdirFromOutput !== ".." && + !workdirFromOutput.startsWith(`..${path.sep}`)); + if (declarativeDirRel.trim().length === 0 || outputContainsWorkdir) { return yield* Effect.fail( new LegacyDeclarativeWriteError({ message: - "declarative output directory must not be empty or resolve to the project directory", + "declarative output directory must not be empty, resolve to the project directory, or contain the project directory", }), ); } diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index e616e0fe8d..ae70ea1e18 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -1,6 +1,6 @@ import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { dirname, join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; import { Cause, Effect, Exit, Layer, Option } from "effect"; @@ -556,12 +556,14 @@ describe("legacy db schema declarative generate integration", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("rejects output paths that could overwrite the project directory", () => { - const sentinel = join(tmp.current, "project-sentinel.txt"); + it.effect("rejects output paths that could overwrite the project or an ancestor", () => { + const projectDir = join(tmp.current, "project"); + mkdirSync(projectDir, { recursive: true }); + const sentinel = join(projectDir, "project-sentinel.txt"); writeFileSync(sentinel, "keep"); - const s = setup(tmp.current, { experimental: true, engineImplementation: "next" }); + const s = setup(projectDir, { experimental: true, engineImplementation: "next" }); return Effect.gen(function* () { - for (const output of ["", "."]) { + for (const output of ["", ".", "..", dirname(projectDir)]) { const exit = yield* legacyDbSchemaDeclarativeGenerate( flags({ local: Option.some(true), output: Option.some(output), overwrite: true }), ).pipe(Effect.exit); @@ -569,7 +571,7 @@ describe("legacy db schema declarative generate integration", () => { expect(failError(exit)).toMatchObject({ _tag: "LegacyDeclarativeWriteError", message: - "declarative output directory must not be empty or resolve to the project directory", + "declarative output directory must not be empty, resolve to the project directory, or contain the project directory", }); expect(readFileSync(sentinel, "utf8")).toBe("keep"); } diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts index 9d6b9d02fa..e73738e446 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts @@ -5,6 +5,7 @@ import { type CliErrorActionabilityDeclaration, ErrorActionabilityId, } from "../../../../shared/telemetry/error-actionability.ts"; +import { legacyWalkSqlFiles } from "../../../shared/legacy-glob.ts"; import type { LegacyPgDeltaExportManifest, LegacyPgDeltaSqlFile, @@ -91,54 +92,30 @@ export const LegacyLoadPgDeltaSqlFiles = Effect.fnUntraced(function* ( path: Path.Path, directory: string, ) { - const pending = [directory]; - const paths: Array<{ readonly full: string; readonly name: string }> = []; - - while (pending.length > 0) { - const current = pending.pop(); - if (current === undefined) break; - const entries = yield* fs - .readDirectory(current) - .pipe( - Effect.mapError((error) => - filesError(`failed to read declarative schema directory: ${error.message}`), - ), - ); - for (const entry of entries) { - const full = path.join(current, entry); - const stat = yield* fs - .stat(full) - .pipe( - Effect.mapError((error) => - filesError(`failed to inspect declarative schema file: ${error.message}`), - ), - ); - if (stat.type === "Directory") { - pending.push(full); - continue; - } - if (path.extname(entry).toLowerCase() !== ".sql") continue; - - const name = path.relative(directory, full).split("\\").join("/"); - const normalized = path.normalize(name); - if (normalized.startsWith("..") || path.isAbsolute(normalized)) { - return yield* Effect.fail(filesError(`unsafe declarative schema path: ${name}`)); - } - paths.push({ full, name }); - } - } - - paths.sort((left, right) => left.name.localeCompare(right.name)); + const paths = yield* legacyWalkSqlFiles(fs, directory, "").pipe( + Effect.mapError((error) => + filesError( + error.reason.method === "stat" + ? `failed to inspect declarative schema file: ${error.message}` + : `failed to read declarative schema directory: ${error.message}`, + ), + ), + ); const files: Array = []; - for (const file of paths) { + for (const name of paths) { + const normalized = path.normalize(name); + if (normalized.startsWith("..") || path.isAbsolute(normalized)) { + return yield* Effect.fail(filesError(`unsafe declarative schema path: ${name}`)); + } + const full = path.join(directory, name); const sql = yield* fs - .readFileString(file.full) + .readFileString(full) .pipe( Effect.mapError((error) => filesError(`failed to read declarative schema file: ${error.message}`), ), ); - files.push({ name: file.name, sql }); + files.push({ name, sql }); } return files; }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.unit.test.ts new file mode 100644 index 0000000000..8a61d07532 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.unit.test.ts @@ -0,0 +1,65 @@ +import { mkdirSync, symlinkSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +import { BunServices } from "@effect/platform-bun"; +import { describe, expect, it } from "@effect/vitest"; +import { Effect, FileSystem, Path } from "effect"; + +import { useLegacyTempWorkdir } from "../../../../../tests/helpers/legacy-mocks.ts"; +import { legacyWalkSqlFiles } from "../../../shared/legacy-glob.ts"; +import { LegacyLoadPgDeltaSqlFiles } from "./legacy-pgdelta-files.ts"; + +const load = (directory: string) => + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + return yield* LegacyLoadPgDeltaSqlFiles(fs, path, directory); + }).pipe(Effect.provide(BunServices.layer)); + +describe("LegacyLoadPgDeltaSqlFiles", () => { + const tmp = useLegacyTempWorkdir("legacy-pgdelta-files-"); + + it.effect("does not follow symlinked directories", () => { + const schemas = join(tmp.current, "schemas"); + const outside = join(tmp.current, "outside"); + mkdirSync(schemas); + mkdirSync(outside); + writeFileSync(join(schemas, "kept.sql"), "select 'kept';"); + writeFileSync(join(outside, "hidden.sql"), "select 'hidden';"); + symlinkSync(outside, join(schemas, "linked"), "dir"); + + return Effect.gen(function* () { + const files = yield* load(schemas); + expect(files).toEqual([{ name: "kept.sql", sql: "select 'kept';" }]); + }); + }); + + it.effect("ignores uppercase .SQL files", () => { + const schemas = join(tmp.current, "schemas"); + mkdirSync(schemas); + writeFileSync(join(schemas, "included.sql"), "select 1;"); + writeFileSync(join(schemas, "ignored.SQL"), "select 2;"); + + return Effect.gen(function* () { + const files = yield* load(schemas); + expect(files).toEqual([{ name: "included.sql", sql: "select 1;" }]); + }); + }); + + it.effect("preserves the shared walker's deterministic UTF-8 byte ordering", () => { + const schemas = join(tmp.current, "schemas"); + const privateUse = "a\u{e000}.sql"; + const supplementary = "a\u{1f600}.sql"; + mkdirSync(schemas); + writeFileSync(join(schemas, supplementary), "select 2;"); + writeFileSync(join(schemas, privateUse), "select 1;"); + + return Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const walked = yield* legacyWalkSqlFiles(fs, schemas, ""); + const files = yield* LegacyLoadPgDeltaSqlFiles(fs, yield* Path.Path, schemas); + expect(files.map((file) => file.name)).toEqual(walked); + expect(files.map((file) => file.name)).toEqual([privateUse, supplementary]); + }).pipe(Effect.provide(BunServices.layer)); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index 4207e78410..2e1bcc772d 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -9,7 +9,10 @@ import { } from "../../../../shared/legacy/global-flags.ts"; import { Output } from "../../../../shared/output/output.service.ts"; import { RuntimeInfo } from "../../../../shared/runtime/runtime-info.service.ts"; -import { LegacyDbConnection } from "../../../shared/legacy-db-connection.service.ts"; +import { + LegacyDbConnection, + type LegacyDbSession, +} from "../../../shared/legacy-db-connection.service.ts"; import { LegacyDockerRun } from "../../../shared/legacy-docker-run.service.ts"; import { legacyToPostgresURL } from "../../../shared/legacy-postgres-url.ts"; import { @@ -76,6 +79,25 @@ interface NativeShadowBase { readonly image: string; } +/** + * Removes extensions that the legacy PG14 platform baseline installs implicitly + * so the declarative shadow reflects only extension declarations in schema files. + * `pgjwt` has a hard extension dependency on `pgcrypto`, and `storage.objects.id` + * depends on `uuid-ossp`, so both dependencies must be detached before the + * user-manageable extensions can be dropped with the default RESTRICT behavior. + */ +export const legacyPreparePgDeltaNextDeclarativeBaseline = Effect.fnUntraced(function* ( + session: Pick, + majorVersion: number, +) { + if (majorVersion === 14) { + yield* session.exec("ALTER TABLE storage.objects ALTER COLUMN id DROP DEFAULT"); + yield* session.exec("DROP EXTENSION IF EXISTS pgjwt"); + } + yield* session.exec("DROP EXTENSION IF EXISTS pgcrypto"); + yield* session.exec('DROP EXTENSION IF EXISTS "uuid-ossp"'); +}); + const setupRunInput = (input: NativeShadowInput, handle: LegacyShadowDatabaseHandle) => ({ fs: input.base.fs, path: input.base.path, @@ -209,8 +231,10 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( yield* Effect.scoped( Effect.gen(function* () { const session = yield* legacyConnectShadowDatabase(setup.connConfig); - yield* session.exec("DROP EXTENSION IF EXISTS pgcrypto"); - yield* session.exec('DROP EXTENSION IF EXISTS "uuid-ossp"'); + yield* legacyPreparePgDeltaNextDeclarativeBaseline( + session, + input.base.setup.majorVersion, + ); }), ); return legacyToPostgresURL(setup.connConfig); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts new file mode 100644 index 0000000000..aa40d07057 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts @@ -0,0 +1,44 @@ +import { it } from "@effect/vitest"; +import { Effect } from "effect"; +import { describe, expect } from "vitest"; + +import { legacyPreparePgDeltaNextDeclarativeBaseline } from "./legacy-pgdelta-next-shadow.layer.ts"; + +function recordingSession() { + const statements: string[] = []; + return { + statements, + session: { + exec: (sql: string) => + Effect.sync(() => { + statements.push(sql); + }), + }, + }; +} + +describe("legacyPreparePgDeltaNextDeclarativeBaseline", () => { + it.effect("detaches the PG14 platform dependencies before dropping extensions", () => { + const { session, statements } = recordingSession(); + return Effect.gen(function* () { + yield* legacyPreparePgDeltaNextDeclarativeBaseline(session, 14); + expect(statements).toEqual([ + "ALTER TABLE storage.objects ALTER COLUMN id DROP DEFAULT", + "DROP EXTENSION IF EXISTS pgjwt", + "DROP EXTENSION IF EXISTS pgcrypto", + 'DROP EXTENSION IF EXISTS "uuid-ossp"', + ]); + }); + }); + + it.effect("does not modify PG15+ platform objects before dropping extensions", () => { + const { session, statements } = recordingSession(); + return Effect.gen(function* () { + yield* legacyPreparePgDeltaNextDeclarativeBaseline(session, 17); + expect(statements).toEqual([ + "DROP EXTENSION IF EXISTS pgcrypto", + 'DROP EXTENSION IF EXISTS "uuid-ossp"', + ]); + }); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts index 7de8a4176c..a5692989db 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts @@ -612,6 +612,69 @@ describeDockerLive("pg-delta next declarative extension baseline (live)", () => ); }); +describeDockerLive("pg-delta next PG14 declarative scratch (live)", () => { + let projectDir = ""; + + beforeAll(async () => { + projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-pg14-live-")); + + const init = await runSupabaseLive(["init"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(init.exitCode, commandFailure(init)).toBe(0); + + const configPath = path.join(projectDir, "supabase", "config.toml"); + const generatedConfig = readFileSync(configPath, "utf8"); + expect(generatedConfig).toContain("major_version = 17"); + writeFileSync( + configPath, + generatedConfig + .replace("major_version = 17", "major_version = 14") + .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') + .replace( + '# declarative_schema_path = "./database"', + 'declarative_schema_path = "./schemas"', + ), + ); + + const schemasDir = path.join(projectDir, "supabase", "schemas"); + mkdirSync(schemasDir, { recursive: true }); + writeFileSync( + path.join(schemasDir, "extensions.sql"), + [ + "create extension if not exists pgcrypto with schema extensions;", + "create extension if not exists pgjwt with schema extensions;", + 'create extension if not exists "uuid-ossp" with schema extensions;', + "", + ].join("\n"), + ); + }, COMMAND_TIMEOUT_MS); + + afterAll(async () => { + if (projectDir.length === 0) return; + await runSupabaseLive(["stop", "--no-backup"], { + cwd: projectDir, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }).catch(() => undefined); + await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); + }, COMMAND_TIMEOUT_MS); + + test( + "provisions an extension-free PG14 scratch before loading desired declarations", + { timeout: SCENARIO_TIMEOUT_MS }, + async () => { + const sync = await runSupabaseLive(["db", "schema", "declarative", "sync", "--no-apply"], { + cwd: projectDir, + env: { ...NEXT_ENV, SUPABASE_YES: "true" }, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }); + expect(sync.exitCode, commandFailure(sync)).toBe(0); + expect(sync.stderr).toContain("No schema changes found"); + }, + ); +}); + describeDockerLive("pg-delta next isolated cron shadows (live)", () => { const jobName = "pgdelta_cli_inactive"; const initialSchedule = "0 0 * * *"; diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index d95e8fac39..d9963c7660 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -183,6 +183,12 @@ alter default privileges for role postgres in schema public const LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL = "create extension if not exists pg_net schema extensions;"; +// The historical PG14 dump installs pg_net because later statements grant on +// its schema. Remove it after the dump so the final baseline still follows the +// user's webhooks setting. Enabled projects recreate it after the dump, when +// the bundled event trigger can apply the intended grants. +const LEGACY_START_REMOVE_PG14_DATABASE_WEBHOOKS_SQL = "drop extension if exists pg_net;"; + /** * A SQL exec (schema/globals/API-privileges) or one-shot service-migration Docker * job failed, or the scratch temp directory/file could not be created. The Docker @@ -1021,7 +1027,18 @@ export const legacySetupDatabase = ( }), ), ); + const requiresPg14WebhooksCleanup = input.majorVersion === 14; yield* legacyStartInitSchema(spawner, input, tmpDir); + if (requiresPg14WebhooksCleanup) { + yield* legacyExecSqlConstant( + session, + fs, + path, + tmpDir, + "remove-pg14-database-webhooks.sql", + LEGACY_START_REMOVE_PG14_DATABASE_WEBHOOKS_SQL, + ); + } const activateUserExtensions = options.activateUserExtensions ?? true; const legacyPgNetBaseline = options.legacyPgNetBaseline ?? false; const userEnabled = activateUserExtensions && input.webhooksEnabled; diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index 954ee9afec..60f4205d06 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -682,7 +682,7 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { it.effect( "supports the extension-free declarative baseline on PG14 without resolving JWKS", () => { - const { session } = fakeSession(); + const { session, calls } = fakeSession(); const workdir = tempRoot.current; const mock = mockSpawner(); let jwksEvaluated = false; @@ -717,6 +717,18 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { { activateUserExtensions: false }, ); expect(jwksEvaluated).toBe(false); + const enablePgNet = calls.findIndex((call) => + call.sql.includes("CREATE EXTENSION IF NOT EXISTS pg_net WITH SCHEMA extensions"), + ); + const grantPgNet = calls.findIndex((call) => + call.sql.includes("GRANT USAGE ON SCHEMA net TO supabase_functions_admin"), + ); + const removePgNet = calls.findIndex( + (call) => call.sql === "drop extension if exists pg_net", + ); + expect(enablePgNet).toBeGreaterThanOrEqual(0); + expect(grantPgNet).toBeGreaterThan(enablePgNet); + expect(removePgNet).toBeGreaterThan(grantPgNet); }).pipe( Effect.provide( Layer.mergeAll( diff --git a/apps/cli/src/legacy/shared/db-bootstrap/templates/db-initial-schema-14.sql.ts b/apps/cli/src/legacy/shared/db-bootstrap/templates/db-initial-schema-14.sql.ts index b2b506eb09..b41ee366ff 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/templates/db-initial-schema-14.sql.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/templates/db-initial-schema-14.sql.ts @@ -79,6 +79,20 @@ CREATE SCHEMA IF NOT EXISTS graphql_public; ALTER SCHEMA graphql_public OWNER TO supabase_admin; +-- +-- Name: pg_net; Type: EXTENSION; Schema: -; Owner: - +-- + +CREATE EXTENSION IF NOT EXISTS pg_net WITH SCHEMA extensions; + + +-- +-- Name: EXTENSION pg_net; Type: COMMENT; Schema: -; Owner: +-- + +COMMENT ON EXTENSION pg_net IS 'Async HTTP'; + + -- -- Name: pgbouncer; Type: SCHEMA; Schema: -; Owner: pgbouncer -- diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.ts index e31242ae12..ff8d419b53 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.ts @@ -68,7 +68,7 @@ const VACUUM_PATTERN = /^VACUUM(?:\s|\(|$)/u; const ALTER_SYSTEM_PATTERN = /^ALTER\s+SYSTEM(?:\s|$)/u; const CLUSTER_PATTERN = /^CLUSTER(?:\s|$)/u; const TRANSACTION_CONTROL_PATTERN = - /^(?:BEGIN|START\s+TRANSACTION|COMMIT|END|ROLLBACK|ABORT|PREPARE\s+TRANSACTION)(?:\s|$)/u; + /^(?:BEGIN|START\s+TRANSACTION|COMMIT|END|ABORT|PREPARE\s+TRANSACTION)(?:\s|$)/u; /** * Strips a leading BOM, whitespace, and SQL line (`--`) and block comments from the @@ -116,8 +116,17 @@ export const legacyIsPipelineIncompatible = (sql: string): boolean => { }; /** Whether the statement owns a transaction boundary that must not be nested. */ -export const legacyHasTransactionControl = (sql: string): boolean => - TRANSACTION_CONTROL_PATTERN.test(legacyTrimLeadingSqlComments(sql).toUpperCase()); +export const legacyHasTransactionControl = (sql: string): boolean => { + const upper = legacyTrimLeadingSqlComments(sql).toUpperCase(); + const words = upper.split(/\s+/u); + if (words[0] === "ROLLBACK") { + const toIndex = words[1] === "WORK" || words[1] === "TRANSACTION" ? 2 : 1; + // ROLLBACK [WORK | TRANSACTION] TO [SAVEPOINT] rewinds the current + // transaction without ending it, so it still needs the CLI-managed wrapper. + return words[toIndex] !== "TO"; + } + return TRANSACTION_CONTROL_PATTERN.test(upper); +}; /** A buffered statement awaiting the next batch flush; `version` is the history insert. */ type LegacyBatchItem = diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts index 88faaf272f..02101b8948 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts @@ -325,6 +325,34 @@ describe("legacyApplyMigrationFile", () => { ); }); + it.effect("keeps savepoint rollback inside the managed migration transaction", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-apply-")); + const file = join(dir, "20240101120000_savepoint.sql"); + writeFileSync( + file, + "SAVEPOINT before_change;\n" + + "UPDATE accounts SET active = false;\n" + + "ROLLBACK TO SAVEPOINT before_change;\n" + + "SELECT 1;", + ); + const { session, calls } = fakeSession(); + return run(session, file).pipe( + Effect.tap(() => + Effect.sync(() => { + const execs = calls.filter((call) => call.kind === "exec").map((call) => call.sql); + const setupCommit = execs.indexOf("COMMIT"); + const managedBegin = execs.lastIndexOf("BEGIN"); + const managedCommit = execs.lastIndexOf("COMMIT"); + expect(managedBegin).toBeGreaterThan(setupCommit); + expect(execs.indexOf("SAVEPOINT before_change")).toBeGreaterThan(managedBegin); + expect(execs.indexOf("ROLLBACK TO SAVEPOINT before_change")).toBeLessThan(managedCommit); + expect(calls.filter((call) => call.kind === "query")).toHaveLength(1); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + it.effect("does not record history when an authored transaction fails", () => { const dir = mkdtempSync(join(tmpdir(), "legacy-apply-")); const file = join(dir, "20240101120000_authored.sql"); @@ -350,13 +378,26 @@ describe("legacyHasTransactionControl", () => { expect(legacyHasTransactionControl("START TRANSACTION ISOLATION LEVEL SERIALIZABLE")).toBe( true, ); - expect(legacyHasTransactionControl("ROLLBACK TO SAVEPOINT before_change")).toBe(true); expect( legacyHasTransactionControl( "CREATE FUNCTION f() RETURNS void AS $$ BEGIN END $$ LANGUAGE plpgsql", ), ).toBe(false); }); + + it("distinguishes transaction rollback from savepoint rollback", () => { + for (const sql of ["ROLLBACK", "ROLLBACK WORK", "ROLLBACK TRANSACTION"]) { + expect(legacyHasTransactionControl(sql)).toBe(true); + } + for (const sql of [ + "ROLLBACK TO before_change", + "ROLLBACK TO SAVEPOINT before_change", + "ROLLBACK WORK TO SAVEPOINT before_change", + "ROLLBACK TRANSACTION TO before_change", + ]) { + expect(legacyHasTransactionControl(sql)).toBe(false); + } + }); }); describe("migration failure rendering (Go ExecBatch parity)", () => { From f27874cb9edd838ed1dc380131472984eb6db1d5 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 16:41:21 +0200 Subject: [PATCH 25/82] chore(cli): trim pg-delta PR scope --- .../docs/pg-delta-next-dogfood-playbook.md | 703 ---------------- .../legacy/commands/bootstrap/SIDE_EFFECTS.md | 24 +- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 136 ++-- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 95 +-- .../declarative/generate/SIDE_EFFECTS.md | 100 +-- .../schema/declarative/sync/SIDE_EFFECTS.md | 136 ++-- .../shared/legacy-pgdelta-next.live.test.ts | 761 +----------------- .../0017-schema-first-database-workflow.md | 140 ---- docs/adr/README.md | 1 - docs/cli/schema-workflow-glossary.md | 28 - 10 files changed, 215 insertions(+), 1909 deletions(-) delete mode 100644 apps/cli/docs/pg-delta-next-dogfood-playbook.md delete mode 100644 docs/adr/0017-schema-first-database-workflow.md delete mode 100644 docs/cli/schema-workflow-glossary.md diff --git a/apps/cli/docs/pg-delta-next-dogfood-playbook.md b/apps/cli/docs/pg-delta-next-dogfood-playbook.md deleted file mode 100644 index 25c00acd16..0000000000 --- a/apps/cli/docs/pg-delta-next-dogfood-playbook.md +++ /dev/null @@ -1,703 +0,0 @@ -# Dogfood playbook: pg-delta next on Supabase CLI - -Self-contained instructions for a **fresh agent** (or human) to dogfood the bundled in-process pg-delta “next” engine on the CLI. No prior chat context required. - -**PR:** https://github.com/supabase/cli/pull/6102 - -**Branch:** `feat/upgrade-pg-delta-next` - -**Goal:** Validate that local Docker, staging remote, and real-project (dbdev) workflows work with the **default** next engine; catch regressions; produce a short scorecard + DX failure writeups. - -Do **not** treat this as “run every historical repro forever.” Each pass should (1) confirm previously green paths still green, (2) deeply re-check open failures, (3) report only meaningful deltas. - ---- - -## 1. Success criteria (what “good” means) - -Prefer **behavioral** contracts over byte-identical SQL: - -| Contract | Meaning | -| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Apply + empty follow-up | After generate/sync/pull/push, a second sync/diff/pull reports **no schema changes** (or empty plan). | -| Meaningful drift | Adding one real object produces a migration that contains **that** object (not silent empty, not full-schema noise). No spurious `auth` / `storage` / `realtime` schema creates. | -| Safe upgrade | Legacy declarative trees must not hard-fail next sync, and must not silently write destructive `DROP EXTENSION` / `cron.unschedule` without clear warning / refuse. Compat gate / repair is OK; cryptic crashes are not. | -| Staging API | `link` / `branches` work against staging Management API. | -| Packaging | Preview **pkg.pr.new** binary works (wasm embedded; no CI-path `ENOENT`). | -| DX | Actionable errors, usable empty-state messaging, debug artifacts that help without corrupting SQL stdout. | - -`db push` does **not** run pg-delta (apply only). Prefer `--db-url` to `db..supabase.red:5432` for remote diff/pull (not pooler). Use `POSTGRES_URL_NON_POOLING` + `sslmode=require` for branch URLs. - ---- - -## 2. What you are testing - -Default engine = **pg-delta next** (in-process, bundled). **No automatic fallback** to legacy. - -| Env var | Effect | -| ---------------------------------- | ---------------------------------------------------------------------- | -| unset / `true` | next (default under test) | -| `SUPABASE_USE_PG_DELTA_NEXT=false` | **legacy** edge-runtime pg-delta (upgrade-compat / opt-out cells only) | - -Config gate: `[experimental.pgdelta] enabled = true` and typically `major_version = 17` in `supabase/config.toml`. - -| Surface | Commands | -| --------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| Declarative | `db schema declarative generate`, `db schema declarative sync`, `db pull --declarative` | -| Migration-style | `db diff --use-pg-delta` / `--diff-engine pg-delta`, `db pull --diff-engine pg-delta`, `db push`, `db reset`, `migration list/repair` | -| Live URL | `db diff --from --to <…>` | -| Opt-out | same commands with `SUPABASE_USE_PG_DELTA_NEXT=false` | - -**Baseline reminder:** normal `db diff` / migration-style `db pull` compare **`supabase/migrations` ↔ live DB**. Declarative files are **not** that baseline (`DeclarativeSchemaNotUsedAsDiffBaseline`). Use `db schema declarative sync` to compare migrations ↔ declarative desired state. - ---- - -## 3. Prerequisites - -### Machine - -- macOS with **OrbStack/Docker** healthy (`docker version` shows Server). -- `gh` authenticated to `supabase/cli`. -- `npm` / `pnpm` available. -- Optional CLI worktree / monorepo checkout of the PR — prefer **pkg.pr.new** for dogfood; use local build only if pkg 404 (`dist/supabase` / `SUPABASE_CLI_BINARY_OVERRIDE`). - -### Staging - -- Profile: `supabase-staging` → API `api.supabase.green`, DB hosts `*.supabase.red`. -- Access token: `SUPABASE_ACCESS_TOKEN` matching `sbp_` + 40 hex chars. On this machine, read from macOS keychain profile `supabase-staging` (see setup snippet). **Never** write tokens/passwords into git or report files. -- Throwaway / dogfood project historically used: ref `tcdjmxannewfbyvudoql` (`pgdelta-dogfood-20260806`). Recreate if missing; password may live in an older dogfood dir — ask the user if absent. - -### Corpus (real project) - -- **dbdev** migrations (read-only): `/Users/avallete/Documents/Programming/Supa/dbdev/supabase/migrations` (~54 SQL files). -- **Never mutate** that git checkout. Always `cp` into a temp sandbox. - -### Prior findings (read before inventing new theory) - -| Round / topic | Path | -| --------------------------------- | -------------------------------------------------------------------------- | -| Round 6 complete scorecard | `/Users/avallete/tmp/pgdelta-dogfood6/findings/REPORT6.md` | -| Round 5 + dbdev | `/Users/avallete/tmp/pgdelta-dogfood5/findings/{REPORT5,DBDEV}.md` | -| Failure B deep dive | `/Users/avallete/tmp/pgdelta-dogfood4/findings/FAILURE_B_INVESTIGATION.md` | -| Older repro cookbook | `/Users/avallete/tmp/pgdelta-dogfood/findings/REPRO.md` | -| Session handoff (next-pass focus) | `/tmp/pgdelta-next-dogfood-handoff.md` | -| Older cache-root notes | `~/.cache/supabase-pgdelta-dogfood/` | - -If paths differ on another machine: home user, `Documents/Programming` vs `Programming`. - ---- - -## 4. Known landmines (fix these before declaring FAIL) - -These have already bitten dogfood runs. Check them **before** filing product bugs. - -### 4.1 `~/.supabase/profile` trailing newline - -- **Symptom:** `failed to read profile: Unsupported Config Type ""` during shadow provision (`supabase-go db __shadow`). -- **Cause:** Go `getProfileName` does **not** trim file contents; TS does. `supabase-staging\n` is treated as a YAML path with empty extension. -- **Fix before dogfood:** - -```sh -# Write WITHOUT a trailing newline -printf '%s' 'supabase-staging' > "$HOME/.supabase/profile" -``` - -- Prefer `--profile supabase-staging` on remote commands anyway. - -### 4.2 Linked project storage image pin (staging) - -- **Symptom:** long retries for `storage-api:v1.68.0-queue-bench` (or similar), then `failed to provision the shadow database` on `db diff --linked/--db-url`, migration `db pull`, or even local `declarative sync` in a **linked** workdir. -- **Cause:** `supabase link` writes remote `/storage/v1/version` → `supabase/.temp/storage-version`. Staging may report unpublished tags. -- **Workaround (required after every `link` on affected fleets):** - -```sh -# Dockerfile default is a good public pin (confirm in apps/cli-go/pkg/config/templates/Dockerfile) -printf '%s' 'v1.68.10' > supabase/.temp/storage-version -docker image inspect public.ecr.aws/supabase/storage-api:v1.68.10 >/dev/null \ - || docker pull public.ecr.aws/supabase/storage-api:v1.68.10 -``` - -- Deleting the pin file falls back to the embedded Dockerfile image. - -### 4.3 `PGDELTA_DEBUG=1` can corrupt stdout / flood diagnostics - -- Prior dogfood runs set `PGDELTA_DEBUG=1`, which prints dozens of `invalid_routine_body` / `dangling_edge` lines. That is **debug noise**, not a product failure. -- Redirected `db diff … > patch.sql` can contain clack-framed diagnostics; apply then fails. -- **Rule:** assess “quiet DX” and capture SQL with debug **unset**. Use debug only when investigating. Artifacts (next): `supabase/.temp/pgdelta/v2/debug//`. - -### 4.4 Port conflicts on local `start` - -Default DB port `54322` is often taken. Bump ports in `supabase/config.toml` (e.g. +200) or stop leftover stacks. One Docker project at a time if ports collide. - -### 4.5 Migration history on fresh local + existing remote - -Empty local `migrations/` + remote history → pull blocked with actionable `migration repair --status reverted …` list. That is expected DX, not a next-engine bug. - -### 4.6 Flag mutexes - -- `db pull --declarative` and `--diff-engine` are mutually exclusive. -- `db schema declarative generate` takes **flags only** (use `--output` / configured declarative path; no positional dir). - -### 4.7 Shell hygiene (agents) - -- Use **absolute paths**. Do not set `HOME` to a dogfood subdirectory (breaks path expansion). -- Prefer `--agent no` when you need **raw SQL on stdout** for inspection or apply. -- Agents often JSON-wrap CLI stdout; parse `diff` / structured fields when present. -- zsh: `setopt NULL_GLOB` before `*.sql` globs, or use `find`. - ---- - -## 5. One-time setup for a new pass - -Create a new root (do not overwrite old rounds until you have a report): - -```bash -PASS=7 # increment -ROOT=/Users/avallete/tmp/pgdelta-dogfood${PASS} -mkdir -p "$ROOT/findings" "$ROOT/cli" - -# Resolve PR tip + preview package -OID=$(gh pr view 6102 --json headRefOid --jq .headRefOid) -echo "$OID" > "$ROOT/findings/VERSION.txt" -gh api repos/supabase/cli/issues/6102/comments \ - --jq '[.[] | select(.body|contains("pkg.pr.new"))] | .[-1].body' | head -20 - -npm install --prefix "$ROOT/cli" --engine-strict=false --force \ - "supabase@https://pkg.pr.new/supabase/cli/supabase@${OID}" - -cat > "$ROOT/env.sh" <<'EOF' -#!/usr/bin/env bash -set -euo pipefail -export SUPABASE_PROFILE=supabase-staging -TOKEN_B64=$(security find-generic-password -s "Supabase CLI" -a "supabase-staging" -w | sed 's/^go-keyring-base64://') -export SUPABASE_ACCESS_TOKEN=$(printf '%s' "$TOKEN_B64" | base64 -d) -# Prefer OFF for user-visible DX checks. Turn on only when debugging engine internals. -# export PGDELTA_DEBUG=1 -export SUPABASE_YES=true -export SB_PKG="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cli/node_modules/.bin/supabase" -SB() { "$SB_PKG" "$@"; } -EOF - -source "$ROOT/env.sh" -"$SB_PKG" --version # expect 0.0.0-pr.6102 or similar -printf '%s' 'supabase-staging' > "$HOME/.supabase/profile" - -# Docker -docker version --format '{{.Server.Version}}' || { open -a OrbStack; sleep 5; } -# Stop leftover supabase containers from prior dogfood if ports clash -docker ps --format '{{.Names}}' | rg '^supabase_' || true -``` - -Init / start helpers (use for every sandbox): - -```bash -init_pgdelta_project() { - local W="$1" - mkdir -p "$W" - SB --workdir "$W" init --force - if ! grep -q '\[experimental.pgdelta\]' "$W/supabase/config.toml"; then - cat >> "$W/supabase/config.toml" <<'TOML' - -[experimental.pgdelta] -enabled = true -TOML - fi - perl -pi -e 's/major_version = \d+/major_version = 17/' "$W/supabase/config.toml" -} - -start_trimmed() { - local W="$1" - SB --workdir "$W" start -x gotrue,realtime,storage-api,imgproxy,kong,inbucket,postgrest,postgres-meta,studio,edge-runtime,logflare,vector,supavisor -} -``` - -Direct DB URL pattern (staging): - -```text -postgresql://postgres:@db..supabase.red:5432/postgres?sslmode=require -``` - ---- - -## 6. Complete test matrix (primary pass) - -Run in this order. Log each scenario under `$ROOT/findings/`. Record PASS / FAIL / SKIP with command, exit code, notable stderr, SQL sample (if any), whether re-diff was empty. - -### A. Packaging smoke - -```bash -SB --version -# Any project with supabase/database present: -SB --workdir "$W" db schema declarative sync --no-apply --experimental -``` - -**Fail if:** `ENOENT` … `libpg-query.wasm` or baked CI path `/home/runner/_work/...`. - ---- - -### B. Full dbdev — next happy path (highest value) - -```bash -DBDEV=/Users/avallete/Documents/Programming/Supa/dbdev -W="$ROOT/dbdev-local" -init_pgdelta_project "$W" -rm -rf "$W/supabase/migrations" && mkdir -p "$W/supabase/migrations" -cp -a "$DBDEV/supabase/migrations/"*.sql "$W/supabase/migrations/" -start_trimmed "$W" -# If "Started from backup", prefer a clean apply: -SB --workdir "$W" db reset --local --no-seed - -SB --workdir "$W" db schema declarative generate --local --overwrite --experimental -# Expect .pgdelta-export.json + cluster/misc.sql (cron) + pgcrypto/uuid-ossp files - -SB --workdir "$W" db schema declarative sync --no-apply --experimental -# Expect: No schema changes found -``` - -**Fail if:** non-empty sync for unchanged DB; hard error on cron/pg_cron; missing cron job in export when live has `cron.job`. - ---- - -### C. dbdev — `db diff` with declarative tree present - -```bash -SB --workdir "$W" db query --local "create table public.dogfood_note(id bigint primary key);" -SB --workdir "$W" db diff --local --use-pg-delta -f dogfood_note -``` - -**Expect:** - -- Migration contains `dogfood_note` (and grants). -- Advisory / note: declarative files are **not** the diff baseline (`DeclarativeSchemaNotUsedAsDiffBaseline`). -- Not a silent empty diff. - -**Fail if:** empty diff; or huge recreate of the whole dbdev schema solely because `database/` exists. - -Clean up: drop table + remove the migration file before later steps if needed. - ---- - -### D. Staging — link / branches - -```bash -LW="$ROOT/link-test" -init_pgdelta_project "$LW" -REF= # e.g. tcdjmxannewfbyvudoql -PASS= # from user / prior secure store — do not commit - -SB --workdir "$LW" link --project-ref "$REF" -p "$PASS" -printf '%s' 'v1.68.10' > "$LW/supabase/.temp/storage-version" # AFTER link -SB --workdir "$LW" branches list --project-ref "$REF" -``` - -**Fail if:** Effect `SchemaError` on timestamps (`+00:00` vs `Z`) — historically broken, fixed by round 3+. - ---- - -### E. Staging — pull → empty re-pull (dbdev-shaped remote) - -```bash -PF="$ROOT/pull-fresh" -init_pgdelta_project "$PF" -DB_URL="postgresql://postgres:${PASS}@db.${REF}.supabase.red:5432/postgres" - -# History may conflict with prior dogfood pulls: -SB --workdir "$PF" db pull remote_schema --db-url "$DB_URL" --diff-engine pg-delta -# If conflict: migration repair --status reverted then pull again - -SB --workdir "$PF" db pull again --db-url "$DB_URL" --diff-engine pg-delta -# Expect: No schema changes found (exit 1 / LegacyDbPullInSyncError is OK) -``` - -**Fail if:** shadow apply dies mid-migration (historically function order / `DEFAULT` before callee existed). Inspect pulled SQL if it fails. - ---- - -### F. Mini local — cron + custom schema round-trip - -Separate workdir (stop other stacks if ports collide): - -```bash -M="$ROOT/mini" -init_pgdelta_project "$M" -start_trimmed "$M" - -SB --workdir "$M" db query --local "create table public.items(id bigint primary key, name text not null);" -SB --workdir "$M" db query --local "create type public.item_status as enum ('open','closed');" -SB --workdir "$M" db query --local "alter table public.items add column status public.item_status not null default 'open';" -SB --workdir "$M" db query --local "create extension if not exists pg_cron with schema pg_catalog;" -SB --workdir "$M" db query --local "select cron.schedule('refresh download metrics', '*/30 * * * *', 'select 1;');" -SB --workdir "$M" db query --local "create schema if not exists app;" -SB --workdir "$M" db query --local "create table if not exists app.widgets(id bigint primary key);" - -rm -rf "$M/supabase/database" "$M/supabase/migrations/"*.sql -SB --workdir "$M" db schema declarative generate --local --overwrite --experimental -SB --workdir "$M" db schema declarative sync --no-apply --name baseline --experimental -SB --workdir "$M" db reset --local --no-seed -# Empty sync with debug OFF: -unset PGDELTA_DEBUG -SB --workdir "$M" db schema declarative sync --no-apply --experimental -# Expect: No schema changes found, near-zero diagnostic spam -``` - -**Fail if:** `schema "app" does not exist` on sync; cron job hard-fail (`cron.database_name` / co-located shadow); empty sync non-empty after reset. - ---- - -### G. Failure B — legacy → next upgrade (must deep-check) - -This is the main remaining product gap as of round 6. - -#### G1. As-generated legacy tree (hard-fail path) - -```bash -# On mini or dbdev-local with a live schema already matching migrations/baseline: -rm -rf "$W/supabase/database" -SUPABASE_USE_PG_DELTA_NEXT=false \ - SB --workdir "$W" db schema declarative generate --local --overwrite --experimental - -# Inspect: -find "$W/supabase/database" -type f | sort -cat "$W/supabase/database/cluster/extensions/pg_net.sql" # often: DROP EXTENSION pg_net; -test -f "$W/supabase/database/.pgdelta-export.json" || echo NO_MANIFEST -test -f "$W/supabase/database/cluster/misc.sql" || echo NO_CRON_JOB_FILE - -SUPABASE_USE_PG_DELTA_NEXT=true \ - SB --workdir "$W" db schema declarative sync --no-apply --experimental -``` - -**Historical failure:** shadow load stuck on `pg_net.sql` (`extension "pg_net" does not exist`), sometimes followed by missing `uuid_generate_v4` cascades on dbdev. - -**Success:** no hard-fail; either empty plan, CompatibilityError/refuse under `--yes`, or interactive repair — **without** writing destructive drops by default. - -#### G2. Legacy without `pg_net.sql` (warn / drops path) - -```bash -rm -f "$W/supabase/database/cluster/extensions/pg_net.sql" -SUPABASE_YES=true SUPABASE_USE_PG_DELTA_NEXT=true \ - SB --workdir "$W" db schema declarative sync --no-apply --experimental -``` - -**Historical behavior:** warning about legacy-implicit `pgcrypto` / `uuid-ossp` and cron job; still may write: - -```sql -select cron.unschedule('...'); -drop extension "pgcrypto"; -drop extension "uuid-ossp"; -``` - -**Why (root cause):** next treats those platform extensions + cron jobs as declaratively managed; legacy export omits them. Sync then plans removals. See `FAILURE_B_INVESTIGATION.md`. - -**These drops are destructive if applied/pushed.** Do not apply them as an “upgrade.” - -#### G3. Recommended upgrade path (should stay green) - -```bash -SUPABASE_USE_PG_DELTA_NEXT=true \ - SB --workdir "$W" db schema declarative generate --local --overwrite \ - --output supabase/database-next --experimental - -# Review, then adopt: -rm -rf "$W/supabase/database" -mv "$W/supabase/database-next" "$W/supabase/database" - -unset PGDELTA_DEBUG -SB --workdir "$W" db schema declarative sync --no-apply --experimental -# Expect: No schema changes found -``` - ---- - -### H. Optional / lower priority - -| Test | When | -| --------------------------------------------------------- | ------------------------------------- | -| `SUPABASE_USE_PG_DELTA_NEXT=false` opt-out smoke | Only if packaging/legacy path changed | -| Worktree `bun src/legacy/main.ts` + rebuilt `supabase-go` | pkg.pr.new 404 | -| Preview branch create/delete | Branching UX specifically under test | -| Remote empty `db diff` after push | Blocked until pull/shadow apply green | -| Staging linked migration/declarative loops (§8) | When validating full remote↔local UX | -| Remote ↔ remote branch diffs (§9 matrix C) | When shadow/storage pins are painful | - ---- - -## 7. Extended local / DX cells (when not covered above) - -Use these when the primary matrix does not already prove the behavior, or when chasing a specific DX regression. - -| ID | Scenario | Expect | -| --- | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -| A0 | Declarative SQL **without** `.pgdelta-export.json` and without extension decls → sync `--no-apply` | Non-interactive refusal listing implicit extensions + `CREATE EXTENSION` / generate alternative | -| A3 | Edit declarative (enum value, drop table, new table) → sync → `db push --local` → empty sync | Drop warning; multi-file OK; converges | -| A7 | Bad URL, empty declarative dir, `--strict-coverage` | Actionable errors; artifacts if debug on | -| D1 | `PGDELTA_DEBUG=1` | Debug dir under `.temp/pgdelta/v2/debug/`; **SQL stdout stays clean** (file if still mixed) | -| D2 | Empty `db pull` messaging | Should not look like a hard product bug / issue-tracker funnel when simply up to date | -| D3 | Bad DB password | Clear `28P01` + `SUPABASE_DB_PASSWORD` hint | -| D4 | `--strict-coverage` vs default `unmodeled_kind` summary | Documented, not over-failed | -| D5 | Publications, storage buckets, `security_invoker` views | Known hard cases — document, don’t over-fail | - -Seed schema idea (after extensions exist): - -```sql -create type public.account_state as enum ('pending', 'active'); -create table public.disposable_note ( - id bigint generated by default as identity primary key, - body text not null -); -create view public.auth_user_emails as -select id, email from auth.users; -``` - ---- - -## 8. Staging linked loops (optional depth) - -**Always re-pin storage after `link`.** Prefer `--agent no` for raw SQL. - -### 8.1 Migration remote ↔ local - -```sh -cd "$LOOP" # linked project workdir -printf '%s' 'v1.68.10' > supabase/.temp/storage-version - -"$SB" migration list --linked --profile supabase-staging --agent no -# repair remote-only versions as suggested, then: -"$SB" db pull from_remote_baseline --linked --diff-engine pg-delta \ - --profile supabase-staging --agent no - -"$SB" db diff --linked --use-pg-delta --profile supabase-staging --agent no -# expect: No schema changes found - -# Local → remote -cat > supabase/migrations/$(date -u +%Y%m%d%H%M%S)_add_flag.sql <<'SQL' -alter table public.dogfood_note add column if not exists flag boolean not null default false; -SQL -"$SB" db push --linked --profile supabase-staging --agent no -"$SB" db diff --linked --use-pg-delta --profile supabase-staging --agent no - -# Remote → local -"$SB" db query --linked --profile supabase-staging \ - "alter table public.dogfood_note add column if not exists remote_only_at timestamptz;" -"$SB" db pull remote_only_at --linked --diff-engine pg-delta \ - --profile supabase-staging --agent no -"$SB" db diff --linked --use-pg-delta --profile supabase-staging --agent no -``` - -### 8.2 Declarative remote loop - -```sh -"$SB" db pull --declarative --linked --experimental --profile supabase-staging --agent no -# edit supabase/schemas/... add a table file -"$SB" db schema declarative sync --no-apply --name add_x --experimental --agent no -"$SB" db push --linked --profile supabase-staging --agent no -"$SB" db schema declarative sync --no-apply --experimental --agent no -"$SB" db diff --linked --use-pg-delta --profile supabase-staging --agent no -``` - -### 8.3 Linked matrix checklist - -| ID | Scenario | Expect | -| --- | -------------------------------------------------------- | --------------------------------------------------------- | -| B1 | Empty linked diff after migrations match | Empty | -| B2 | Add migration → push → empty linked diff | Applies; converges | -| B3 | Remote ALTER → pull → empty diff | Single focused migration | -| B4 | `db pull --declarative` or `generate --linked` | Manifest + nested tree; user objects only | -| B5 | Edit declarative → sync → push → empty sync + empty diff | Full loop | -| B6 | Delete declarative table → sync → drop warning → push | Safe, warned | -| B7 | After pulls: `db reset --local --no-seed` | Migration chain applies | -| B8 | Fresh local, remote has history → repair → pull baseline | Actionable repair DX | -| B9 | Repeat B1/B3 with opt-out | Same semantic empty/non-empty; note SSL/image differences | - ---- - -## 9. Staging branches — remote ↔ remote (optional) - -```sh -"$SB" branches create loop-a --project-ref --profile supabase-staging --experimental -"$SB" branches create loop-b --project-ref --profile supabase-staging --experimental -# wait until status != CREATING_PROJECT -"$SB" branches get loop-a --project-ref --profile supabase-staging -o json -# use POSTGRES_URL_NON_POOLING + sslmode=require -``` - -| ID | Scenario | Expect | -| --- | ------------------------------------------------------------------- | ----------------------------------- | -| C1 | Parent → branch `db diff --from/--to` | Meaningful; no managed-schema noise | -| C2 | Branch ↔ branch after diverge | Only intentional objects | -| C3 | Self-diff `--from U --to U` | Empty | -| C4 | Capture clean SQL (`PGDELTA_DEBUG` **off**) → apply → re-diff empty | Convergence | -| C5 | `generate --db-url --overwrite` | Branch-only objects in tree | - -Live `--from/--to` does **not** need local shadow image matching — prefer it when shadow/storage pins are painful. - -```sh -unset PGDELTA_DEBUG -"$SB" db diff --from "$URL_B" --to "$URL_A" --use-pg-delta \ - --profile supabase-staging --agent no > /tmp/b-to-a.sql -# Apply statements on B (db query is single-statement-friendly) -"$SB" db diff --from "$URL_A" --to "$URL_B" --use-pg-delta \ - --profile supabase-staging --agent no -# expect empty -``` - ---- - -## 10. What “good” SQL looks like - -- User objects: tables/types/views/indexes/FKs you changed. -- Supabase role grants (`anon` / `authenticated` / `service_role` / `postgres`) are **normal** under the supabase integration profile — not necessarily noise. -- **Bad:** `CREATE SCHEMA auth|storage|realtime`, recreating managed internals, huge unrelated privilege churn every empty sync. -- Watch for **default-privilege revoke-then-grant** churn when syncing a fresh generate into empty migrations — note it if present. - ---- - -## 11. Known failure catalog (what to watch for) - -| Symptom | Likely cause | Severity | -| ------------------------------------------------------------------------ | ------------------------------- | ---------------------------------------------------- | -| Silent empty `db diff` while local has new tables and `database/` exists | Diff incorrectly ignoring drift | **Critical** (was regressed; fixed) | -| `schema "app" does not exist` on sync/diff | Assumed-schema / shadow seeding | High (fixed with isolate shadow) | -| Cron sync: shadow DB ≠ `cron.database_name` | Co-located shadow | High (fixed with isolate shadow) | -| Legacy → next: `DROP EXTENSION pg_net` load fail | Legacy emits bare DROP file | **Open** as of round 6 | -| Legacy → next: `drop pgcrypto` / `uuid-ossp` / `cron.unschedule` | Exporter coverage mismatch | **Open** (warn improved; still writes under `--yes`) | -| Empty re-pull: function / DEFAULT order on shadow | Pulled SQL not replayable | Was open; fixed by round 5+ | -| Staging `SchemaError` `+00:00` | Management API timestamp decode | Fixed | -| `ENOENT` libpg-query.wasm under `/home/runner/...` | pkg embed missing | Fixed in source; always recheck pkg | -| `SET LOCAL` warning on apply | Migration exec outside txn | Fixed observation | -| Flood of `pg-delta next diagnostic:` lines | Often `PGDELTA_DEBUG=1` | Check debug off before filing | -| `Unsupported Config Type ""` | Profile file trailing newline | Landmine (§4.1) | -| Shadow fails pulling `storage-api:…` | Staging storage-version pin | Landmine (§4.2) | - ---- - -## 12. Opt-out checklist - -For critical cells, re-run once with: - -```sh -export SUPABASE_USE_PG_DELTA_NEXT=false -``` - -Notes: - -- No auto-fallback: next failures must not silently call legacy. -- Proxy/edge-runtime image may be missing → SKIP with reason, not FAIL. -- Flag can also live in `supabase/.env`; shell env wins. Re-check if opt-out appears ignored (layer construction before project env load). - ---- - -## 13. How to report - -Write `$ROOT/findings/REPORT${PASS}.md`: - -1. **Header:** date, PR URL, full commit SHA, binary (`pkg.pr.new` vs worktree), staging project ref, storage pin used. -2. **Verdict:** go / conditional go / no-go — one paragraph. -3. **Scorecard table** vs previous round (PASS / FAIL / improved / SKIP). -4. **Open failures only** as DX user stories: - -```markdown -### Failure X — short title - -**As a user** … -**I want to** … -**so I run:** (exact commands) - -**I expect:** … -**I get:** (verbatim error / SQL) - -**Impact:** … -**Artifacts:** findings/….log -``` - -5. **Green smokes:** one line each. -6. **DX nits** / SQL sample paths. -7. **Cleanup** commands used. -8. **No secrets** in the report. - -### Severity guide - -| Severity | Examples | -| ---------- | ------------------------------------------------------------------------------------------------- | -| Blocker | Linked shadow unusable without obscure pin; data-lossy SQL; non-convergence after apply | -| High | Debug corrupts stdout; cryptic profile errors; opt-out broken; Failure B hard-fail / silent drops | -| Medium | Noisy diagnostics; empty-pull issue footer; privilege churn | -| Low / docs | History repair ceremony; known unmanaged object types | - -Also update `/tmp/pgdelta-next-dogfood-handoff.md` if the next-pass priorities changed. - ---- - -## 14. Agent operating rules - -1. Prefer **pkg.pr.new** at the PR HEAD SHA; record SHA in `VERSION.txt`. -2. Use a **new** `$ROOT` per pass; keep prior rounds for comparison. -3. Copy dbdev migrations; never edit the dbdev repo. -4. Trimmed `supabase start -x …` is enough; full stack not required. -5. One Docker project at a time if ports collide (`supabase stop` the other workdir). -6. `SUPABASE_YES=true` is fine for automation but note it may **skip interactive repair** and still write drop migrations — call that out. -7. Do not apply Failure B drop migrations to staging. -8. Do not commit dogfood sandboxes or passwords. -9. Re-pin storage after every `link` on staging (§4.2). -10. Leave `PGDELTA_DEBUG` unset for quiet DX / SQL capture. -11. Repo coding rules (`AGENTS.md`) apply only if you **change CLI code**; pure dogfood is read/execute/report. - ---- - -## 15. Fast vs complete pass - -**Fast (after a narrow fix):** G1/G2/G3 + B (dbdev empty sync) + E (empty re-pull if pull/SQL touched). - -**Complete (new preview build / large PR update):** A–G all, plus quiet mini empty sync with debug off, plus adopt-path G3. Add §8 / §9 only when the change touches linked shadow, declarative remote loops, or `--from/--to`. - -As of round 6 (`f9bd2890b`), complete pass should still expect **G1 fail**, **G2 warn+drops**, everything else green if no regressions. - ---- - -## 16. Cleanup - -```sh -"$SB" branches delete loop-a --project-ref --profile supabase-staging --experimental --yes -"$SB" branches delete loop-b --project-ref --profile supabase-staging --experimental --yes -# Only if you created a throwaway project this pass: -"$SB" projects delete --profile supabase-staging --yes -"$SB" stop --workdir "$W" --no-backup -"$SB" stop --workdir "$M" --no-backup -``` - -Restore the user’s `~/.supabase/profile` if you changed it (still **no trailing newline**). - ---- - -## 17. Quick triage runbook - -| Symptom | Check | -| ------------------------------------------------ | ----------------------------------------------------------------------- | -| `Unsupported Config Type ""` | Profile file newline / contents (§4.1) | -| Pulling `storage-api:…` fails | `cat supabase/.temp/storage-version`; re-pin; re-link rewrites it | -| Shadow fails after “Initialising schema…” | Storage pin, Docker pull, incomplete local migrations into empty shadow | -| Diff SQL full of diagnostics | `PGDELTA_DEBUG` on → unset and redo | -| Pull blocked on history | `migration list` + suggested `migration repair` | -| Declarative sync wants extensions | Add `extension.sql` or `generate` first (compat gate) | -| `db diff` ignores declarative edits | Use `db schema declarative sync`, not `db diff` | -| Legacy sync hard-fails on `pg_net` | Failure B G1 — see §6.G | -| Sync writes `DROP EXTENSION` / `cron.unschedule` | Failure B G2 — do not apply; use G3 adopt path | -| Opt-out SSL probe fails on staging | Note separately; confirm next path still works | -| `ENOENT` … `libpg-query.wasm` | Packaging regression — fail the pass | - ---- - -## 18. References in this repo - -- PR-facing behavior notes: `src/legacy/commands/db/{diff,pull}/SIDE_EFFECTS.md`, `…/schema/declarative/{sync,generate}/SIDE_EFFECTS.md` -- Live convergence harness: `src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts` -- Engine selector: `src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts` -- Storage pin load: `apps/cli-go/pkg/config/config.go` (`StorageVersionPath`), shadow storage job: `apps/cli-go/internal/db/start/start.go` (`initStorageJob`) -- Profile load trap: `apps/cli-go/internal/utils/profile.go` (`getProfileName`) - ---- - -## 19. Suggested skills (for the next agent) - -- None required for dogfood itself. -- If implementing fixes in the CLI after findings: follow workspace `AGENTS.md` (pnpm, Effect under `.repos/effect/`, `pnpm check:all` in changed workspace). -- Handoff after a pass: Cursor/agent `handoff` skill → OS temp. -- Split fix PRs: Cursor `split-to-prs` skill if the user asks. diff --git a/apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md index a510d61e41..f8408c613b 100644 --- a/apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md @@ -6,11 +6,9 @@ health poll → write `.env` → `db push` → start suggestion. Every step is n including the migration push (`legacyDbPushCore`, shared with the standalone `supabase db push` command — see Notes). -The embedded push step does not warm pg-delta state under the default bundled -engine: no next-engine consumer uses the legacy catalog. Setting -`SUPABASE_USE_PG_DELTA_NEXT=false` retains Go's edge-runtime catalog warmup. -`PGDELTA_NPM_REGISTRY`, `.temp/pgdelta-version`, and catalogs directly under -`.temp/pgdelta/` are meaningful only for that legacy opt-out. +The push step uses the bundled in-process pg-delta engine by default. Set +`SUPABASE_USE_PG_DELTA_NEXT=false` to retain legacy catalog warming; only that +path uses the runtime pg-delta package/edge-runtime settings and catalog cache. ## Files Read @@ -26,8 +24,8 @@ engine: no next-engine consumer uses the legacy catalog. Setting | `/supabase/migrations/*.sql` | SQL | native push step, for each pending migration applied | | seed files from `[db.seed].sql_paths` | SQL | native push step (`--include-seed` is always set; gated on `[db.seed].enabled`) | | `/supabase/roles.sql` | SQL | native push step (`--include-roles` is always set; existence check + apply) | -| `/supabase/.temp/pgdelta-version` | plain text | always read by push config loading for compatibility; affects the legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only: push-step catalog warmup image tag, resolved against the bootstrap workdir | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; used only by the legacy pg-delta opt-out | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's catalog warmup image tag, resolved against the bootstrap workdir | ## Files Written @@ -38,8 +36,8 @@ engine: no next-engine consumer uses the legacy catalog. Setting | `/supabase/.temp/project-ref` | plain text | always (mandatory; fails the command on write error) | | `/supabase/.temp/{pooler-url,rest-version,gotrue-version,storage-version,storage-migration}` | plain text | best-effort, from `link.LinkServices` | | `/.env` | dotenv | best-effort (write failure prints a warning and continues) | -| `/supabase/.temp/pgdelta/catalog--migrations--.json` | JSON | legacy opt-out push step, best-effort after a successful migration apply when pg-delta is enabled; failure only warns | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out catalog export when the target requires SSL | +| `/supabase/.temp/pgdelta/catalog--migrations--.json` | JSON | legacy pg-delta opt-out, best-effort after migration apply (write failure only warns) | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy pg-delta opt-out, when the target requires SSL | | `/supabase/.temp/linked-project.json` | JSON | PersistentPostRun linked-project cache (`Effect.ensuring`); resolves against the bootstrap workdir (the prompted/`--workdir`/env target), not `cliConfig.workdir` | | `~/.supabase/telemetry.json` | JSON | PersistentPostRun telemetry flush (`Effect.ensuring`) | @@ -79,10 +77,10 @@ neither branch ever reaches the temp-login-role/Management-API path a passwordle | `SUPABASE_ACCESS_TOKEN` | auth bypass for ensure-login | no | | `SUPABASE_PROFILE` | profile name/path (env → `~/.supabase/profile` → `supabase`) | no | | `SUPABASE_YES` | auto-confirm the native push step's prompts (Go's viper `YES`), read project-`.env`-aware like the standalone `db push` | no | -| `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the legacy opt-out push cache when `[experimental.pgdelta].enabled` is unset, read project-`.env`-aware | no | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` to retain the legacy push-step catalog warmup, read project-`.env`-aware | no | -| `SUPABASE_INTERNAL_IMAGE_REGISTRY` | legacy opt-out only: overrides the push step's edge-runtime image registry, read project-`.env`-aware | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: overrides the push-step edge-runtime npm registry, read project-`.env`-aware | no | +| `SUPABASE_EXPERIMENTAL_PG_DELTA` | enables the legacy opt-out's catalog cache when `[experimental.pgdelta].enabled` is unset, read project-`.env`-aware | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy catalog warming, read project-`.env`-aware | no | +| `SUPABASE_INTERNAL_IMAGE_REGISTRY` | legacy opt-out's edge-runtime image registry, read project-`.env`-aware | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's edge-runtime npm registry, read project-`.env`-aware | no | ## Exit Codes diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 0260af2cbe..efc2b6a0e6 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -1,80 +1,63 @@ # `supabase db diff` Native Effect port. Diffs the local project's expected schema (a throwaway shadow -database) against a target database (local / linked / `--db-url`). Pg-delta runs -in-process by default, migra runs in Docker via edge-runtime, and pgAdmin runs -natively through the differ container. `--use-pg-schema` is the command's only -remaining Go delegation (a documented keep-in-Go exception, CLI-1960). - -## Pg-delta implementation and compatibility - -- The default implementation is the in-process pg-delta engine bundled into the - CLI binary together with pg-topo. Its version is fixed when the CLI is built; - there is no runtime package download or automatic fallback to the legacy engine. -- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy edge-runtime implementation - from either the shell or project `supabase/.env` (the shell wins). Only that - opt-out reads legacy catalogs under `supabase/.temp/pgdelta/`, - `supabase/.temp/pgdelta-version`, or `PGDELTA_NPM_REGISTRY`. -- With `PGDELTA_DEBUG`, default-engine snapshots, plans, and diagnostics are written - under `supabase/.temp/pgdelta/v2/debug//`. The directory contains - `metadata.json` and, when available, `source-snapshot.json`, - `desired-snapshot.json`, `plan.json`, and `diagnostics.json`. These are diagnostic - artifacts, not reusable catalogs. -- The default engine always refuses extraction errors. Coverage gaps - (`unmodeled_kind` or `unresolved_security_label`) warn and remain unmanaged by - default; `--strict-coverage` turns them into a refusal. Warnings identify the - diagnostic origin and explain that unsupported changes are absent from the diff; - when debug capture is enabled, the bundle is saved before policy evaluation. -- SQL text and file segmentation may differ from the legacy renderer. Applicable - output and convergence (a subsequent diff is empty) are the compatibility contract. -- Default-engine plans retain pg-delta's safe compaction and are formatted with - its human-facing preset (lowercase keywords, max width 180). A JSON object in - `[experimental.pgdelta].format_options` partially overrides that preset; the - JSON literal `null` disables formatting without disabling compaction. +database) against a target database (local / linked / `--db-url`), using one of +three native engines: bundled in-process pg-delta, migra (edge-runtime), or +pgAdmin (CLI-1968 — a native `docker run` of the differ container, no +edge-runtime involved). `--use-pg-schema` is the CLI's sole remaining Go +delegation on this command — a documented keep-in-Go exception (CLI-1960), not a +pending port. + +Set `SUPABASE_USE_PG_DELTA_NEXT=false` to use the legacy edge-runtime pg-delta +implementation and its runtime package/catalog cache. The bundled engine has no +automatic fallback; coverage gaps warn, while `--strict-coverage` makes them fatal, +and `PGDELTA_DEBUG` writes diagnostic JSON under +`supabase/.temp/pgdelta/v2/debug//`. Its SQL and transaction-aware file +splits may differ from legacy output; applicable, convergent SQL is the contract. +The bundled formatter defaults to lowercase SQL at width 180; config overrides +it, and JSON `null` disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | shadow provisioning, including the native pgAdmin path | -| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike; missing file tolerated | -| `[db.migrations].schema_paths` globs / `/supabase/database/**` (pg-delta declarative dir) / `/supabase/schemas/**` | SQL | local target's declarative fallback ladder; pgAdmin does not read it | -| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | linked-ref resolution unless `--project-ref` or `SUPABASE_PROJECT_ID` supplies it | -| `/supabase/.temp/pgdelta-version` | plain text | legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only: edge-runtime image tag | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: explicit `--from/--to migrations` catalog cache | +| Path | Format | When | +| ----------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | +| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | +| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit `--from/--to migrations` catalog cache | ## Files Written -| Path | Format | When | -| ----------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------- | -| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; pg-delta may emit multiple transaction-aware files, while pgAdmin always emits one | -| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: migrations catalog | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out only: Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | -| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| ----------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------ | +| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | +| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker -- Edge-runtime container (migra, or pg-delta only under the legacy opt-out). The - legacy explicit `--from/--to migrations` path also runs the native pg-delta - catalog-export script there on a cache miss (CLI-1959; no hidden `__catalog` - subprocess). +- Edge-runtime container (migra, or pg-delta under the legacy opt-out; also runs the legacy + declarative apply script and the pg-delta + catalog-export script for explicit `--from/--to migrations` on a cache miss — + CLI-1959, native, no longer the hidden Go `__catalog` seam). - Shadow Postgres container — provisioned and torn down natively (`legacyPrepareShadowSource` in `legacy/commands/db/shared/legacy-shadow-source.ts`, over the lower-level primitives in `legacy/shared/db-bootstrap/shadow-database.ts`), no longer via a Go seam. Explicit - `--from/--to migrations` also provisions natively. The default implementation keeps a live, - config-gated migrated shadow; the legacy opt-out uses `legacyResolveMigrationsCatalogRef` -> - `exportViaShadowCatalog` and its historical unconditional platform baseline. Neither uses a - `__catalog`-specific shadow or the retired `mode: "diff"` seam. `--use-pgadmin` provisions its - own shadow via a narrower composition — `legacyCreateShadowDatabase` -> health-wait -> - `legacyMigrateShadowDatabase` + `--from/--to migrations` reuses the SAME native primitives on a cache miss + (`legacyResolveMigrationsCatalogRef` -> `exportViaShadowCatalog`, `legacy-pgdelta.cache.ts`), + called with `targetLocal: false`/`usePgDelta: false` to skip the declarative-schema-override + branch — not a second, `__catalog`-specific shadow, and not a shared `mode: "diff"` parameter + (that seam-era concept no longer exists). `--use-pgadmin` provisions its OWN shadow via a + narrower composition — `legacyCreateShadowDatabase` -> health-wait -> `legacyMigrateShadowDatabase` directly (`diff.handler.ts`'s pgadmin branch) — with no declarative-schema-override branch and no `targetUrlOverride`, matching Go's `pgadmin.go` calling `MigrateShadowDatabase` directly rather than `PrepareShadowSource`. @@ -113,8 +96,8 @@ natively as part of this command's own target resolve, ahead of the differ conta | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta engine | no | | `PGDELTA_DEBUG` | pg-delta debug capture | no | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for the legacy edge-runtime pg-delta implementation | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: scoped `@supabase` npm registry for edge-runtime | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's scoped npm registry | no | | `SUPABASE_SSL_DEBUG` | migra SSL debug logging | no | | `SUPABASE_INTERNAL_IMAGE_REGISTRY` | overrides the differ's / shadow's image registry (shell **or** project `.env`, applied for the run via `legacyApplyProjectEnv`, matching `db push`/`db pull`/`db dump`) | no | @@ -152,20 +135,19 @@ migra/pg-delta-engine-specific). ### `--output-format text` (Go CLI compatible) -For migra and pg-delta, progress goes to stderr (`Creating shadow database...`, `Diffing schemas[: ]`, +Progress to stderr (`Creating shadow database...`, `Diffing schemas[: ]`, `Finished supabase db diff on branch .`, drop-statement warning, and the -`--file` write warning). A configured `[db.migrations].schema_paths` also prints a -transition warning because it no longer changes the diff target. The SQL diff -prints to stdout when neither `--file` nor explicit `--output` is set. +`--file` write warning). A configured `[db.migrations].schema_paths` also warns +that it no longer changes the migrations baseline. The SQL diff prints to stdout +when neither `--file` nor explicit `--output` is set. ### `--output-format json` / `stream-json` Progress strings still go to stderr; stdout carries a single structured envelope `{ diff, file, files, schemas, engine, dropStatements, advisories? }` instead of -the raw SQL. With the default pg-delta implementation, a non-empty `--file` diff -and a non-empty declarative tree add the informational -`DeclarativeSchemaNotUsedAsDiffBaseline` advisory; the same note is written to -stderr. Inspection is best-effort and never changes command success. +the raw SQL. Bundled pg-delta reports the best-effort +`DeclarativeSchemaNotUsedAsDiffBaseline` advisory for a non-empty `--file` diff +when declarative files exist. ### `--use-pgadmin` (CLI-1968) @@ -230,13 +212,9 @@ stderr. Inspection is best-effort and never changes command success. effects are Go's); the Go child's telemetry is disabled so the single `cli_command_executed` event comes from this TS command. - Explicit `--from`/`--to` mode always uses pg-delta and writes to `--output` (or stdout). -- `--strict-coverage` applies to the bundled pg-delta engine and refuses output when - it encounters schema objects it cannot manage. -- Normal mode always compares the migrations shadow with the selected live - database. Declarative files and `schema_paths` never replace that migrations baseline; use - `supabase db schema declarative sync` for declarative comparison. -- Under the legacy opt-out, the explicit `migrations` target resolves natively - (CLI-1959): a bare +- Normal mode always compares the migrations shadow to the selected live database; + declarative files and `schema_paths` do not replace that baseline. +- Under the legacy opt-out, the explicit `migrations` target resolves natively (CLI-1959): a bare migrations-content hash cache lookup (`/supabase/.temp/pgdelta/catalog-local-migrations--.json`, shared with `db push`'s post-apply cache write), and on a miss, a natively-provisioned shadow database (CLI-1956 — `legacyCreateShadowDatabase`/`legacyPrepareShadowSource`, diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index bd211a3c18..a6804d34d7 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -1,11 +1,8 @@ # `supabase db pull` Native Effect port. Pulls the remote schema into either a new timestamped -migration (diffing a throwaway shadow against the remote, native pg-delta or +migration (diffing a throwaway shadow against the remote, bundled pg-delta or migra) or declarative files (`--declarative`, native pg-delta export). The -migration-style path always compares migrations with the selected live database; -declarative files and `[db.migrations].schema_paths` cannot replace its migrations -baseline. initial-migra pull (no local migrations) seeds the migration file with a native `pg_dump` of the remote schema (a Docker `pg_dump` container, with IPv4 transaction-pooler fallback) and then appends the migra diff. `--experimental`'s @@ -13,7 +10,7 @@ structured-dump sub-branch (Go's `format.WriteStructuredSchemas`) stays delegated to the bundled Go binary rather than retired or ported (CLI-1957): it needs a TS PostgreSQL DDL AST parser with no equivalent in this repo. `--declarative` covers the same per-object-files outcome for schema objects via -pg-delta managed-state extraction, though its output tree and cluster-object +pg-delta catalog introspection, though its output tree and cluster-object coverage differ (see Files Written below), so this mode is on a deprecation path — the same DECISION CLI-1960 makes for `db diff --use-pg-schema` (keep delegating, flag for removal), not the same output: Go's own `--use-pg-schema` @@ -27,50 +24,28 @@ Go checks `usePgDelta` before `EXPERIMENTAL`, so that combination never delegates and just runs the declarative export normally (see the Notes/Delegation section below). -## Pg-delta implementation and compatibility - -- Pg-delta diff and declarative export use the in-process engine bundled into the - CLI binary by default. Pg-topo is bundled with it and the version is fixed at - CLI build time; the command never downloads it or falls back automatically. -- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy edge-runtime path from - either the shell or project `supabase/.env` (the shell wins). - `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs - directly below `supabase/.temp/pgdelta/` apply only to that opt-out. -- With `PGDELTA_DEBUG`, default-engine diagnostic data is stored under - `supabase/.temp/pgdelta/v2/debug//` as `metadata.json` plus available - snapshot, plan, and diagnostics JSON files. These artifacts are never catalog - cache inputs. -- The default engine always refuses extraction errors. Coverage gaps - (`unmodeled_kind` or `unresolved_security_label`) warn and remain unmanaged by - default; `--strict-coverage` turns them into a refusal. Declarative warnings make - clear that unsupported objects are absent from the exported files. Debug - artifacts are saved before policy evaluation when capture is enabled. -- New-engine SQL bytes and transaction-split filenames may differ. Successful - execution and convergence on a subsequent pull/diff are the contract. -- Nontransactional plan files retain pg-delta's exact first-line - `-- pg-delta: transaction=false` directive. Later push/reset/up commands consume - that durable header to keep the whole file outside a CLI-owned transaction. -- Default-engine migration and declarative SQL retains pg-delta's safe compaction - and uses its human-facing formatter (lowercase keywords, max width 180). A JSON - object in `[experimental.pgdelta].format_options` partially overrides the - preset; the JSON literal `null` disables formatting without disabling - compaction. +Pg-delta runs in-process by default. Set `SUPABASE_USE_PG_DELTA_NEXT=false` for +the legacy edge-runtime implementation and runtime package/catalog cache; there +is no automatic fallback. Coverage gaps warn; `--strict-coverage` makes them +fatal, while `PGDELTA_DEBUG` writes diagnostic JSON under +`supabase/.temp/pgdelta/v2/debug//`. Bundled output may use different SQL +and transaction-aware file splits but must apply and converge. Its formatter +defaults to lowercase SQL at width 180; config overrides it, and JSON `null` +disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | -| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | -| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | -| `[db.migrations].schema_paths` globs / `/supabase/database/**` (pg-delta declarative dir) / `/supabase/schemas/**` | SQL | migration-style pull against the local target only: 3-source declarative-schema fallback ladder, first non-empty source wins (same as `db diff`) | -| `/supabase/.temp/pgdelta-version` | plain text | legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only: edge-runtime image tag | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: catalog snapshots | +| Path | Format | When | +| ----------------------------------------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | +| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | +| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | ## Files Written @@ -78,23 +53,21 @@ Notes/Delegation section below). | ---------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | | `/supabase/database/**` | SQL | `--declarative` | -| `/supabase/database/.pgdelta-export.json` | JSON | default-engine `--declarative` export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: catalog snapshots | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out only: Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default pg-delta engine with `PGDELTA_DEBUG` | +| `/supabase/database/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | | `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | | `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | | `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker -- Edge-runtime container (migra, or pg-delta only under the legacy opt-out). +- Edge-runtime container (migra, or pg-delta under the legacy opt-out). - Shadow Postgres container — provisioned and torn down natively (`legacyPrepareShadowSource` in `legacy/commands/db/shared/legacy-shadow-source.ts` / `legacyPrepareRawShadow` in `legacy/shared/db-bootstrap/shadow-database.ts`, which also owns the lower-level primitives - both build on), no longer via a Go seam. Migration-style pulls use one for either engine; - declarative export uses the raw shadow only under the legacy opt-out because the bundled - in-process exporter reads the target directly. + both build on), no longer via a Go seam. - `supabase/migra` container — the migra OOM bash fallback only. - `pg_dump` container — the initial-migra pull's native remote-schema dump (`legacyStreamPgDump`, shared with `db dump`). @@ -121,8 +94,8 @@ Notes/Delegation section below). | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta diff engine | no | | `SUPABASE_EXPERIMENTAL` | selects the deprecated structured-dump branch (still delegates to Go, see below) | no | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for the legacy edge-runtime pg-delta implementation | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: scoped npm registry for edge-runtime | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's npm registry | no | ## Exit Codes @@ -147,8 +120,8 @@ written to `. Plus the `--use-pg-delta` deprecation line, the prompt. On success the PostRun line `Finished supabase db pull.` is printed to stdout. -A configured `[db.migrations].schema_paths` prints a transition warning on the -migration path directing users to `supabase db schema declarative sync`. +A configured `[db.migrations].schema_paths` also warns on the migration path +that it no longer replaces the migrations baseline. ### `--output-format json` / `stream-json` @@ -173,8 +146,10 @@ Progress strings still go to stderr; stdout carries a single structured envelope delegated Go child, which would otherwise silently re-resolve the workdir's own linked ref instead. - `--use-pg-delta` is hidden and emits the cobra deprecation line to stderr. -- `--strict-coverage` applies to bundled pg-delta diff and declarative-export paths; - it refuses output when pg-delta encounters schema objects it cannot manage. +- Migration-style pulls always compare migrations with the live target; + declarative files and `schema_paths` do not replace that baseline. +- Bundled nontransactional files begin with + `-- pg-delta: transaction=false`, which later migration commands honor. - The initial-migra pull (no local migrations) is native: it streams a `pg_dump` of the remote schema into the migration file, then appends the migra diff. An empty diff after a non-empty dump is swallowed (Go's `swallowInitialInSync`); an empty diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index 9ed124ed19..657b8ba925 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -3,62 +3,42 @@ Generates declarative schema files from a database using pg-delta's managed platform view. -## Pg-delta implementation and compatibility - -- The default pg-delta engine runs in-process. Pg-delta and pg-topo are bundled - into the CLI binary at build time, so the installed CLI fixes their version and - performs no runtime package download or automatic legacy fallback. -- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy catalog/edge-runtime - implementation from either the shell or project `supabase/.env` (the shell - wins). Only that opt-out uses `supabase/.temp/pgdelta-version`, - `PGDELTA_NPM_REGISTRY`, edge-runtime, or legacy catalogs directly below - `supabase/.temp/pgdelta/`. -- `--no-cache` bypasses legacy catalog reuse/warming. The default engine already - extracts live state and has no reusable catalog cache, so the flag does not - change its extraction behavior. -- With `PGDELTA_DEBUG`, default-engine export diagnostics are written below - `supabase/.temp/pgdelta/v2/debug//`; they are never reused as catalogs. -- The default engine always refuses extraction errors. Coverage gaps - (`unmodeled_kind` or `unresolved_security_label`) warn by default and explain that - unsupported objects are absent from the generated files; `--strict-coverage` - turns them into a refusal. Debug artifacts are saved before policy evaluation - when capture is enabled. -- Generated SQL bytes and grouping may differ between engines. Reloading the - export to the same managed state is the compatibility contract. -- The default engine applies pg-delta's human-facing formatter (lowercase - keywords, max width 180) and export-specific safe constraint folding. A JSON - object in `[experimental.pgdelta].format_options` partially overrides the - formatter; the JSON literal `null` disables formatting without disabling plan - compaction. +Pg-delta runs in-process by default. Set `SUPABASE_USE_PG_DELTA_NEXT=false` for +the legacy catalog/edge-runtime implementation; there is no automatic fallback. +Coverage gaps warn; `--strict-coverage` makes them fatal, and `PGDELTA_DEBUG` +writes diagnostic JSON under `supabase/.temp/pgdelta/v2/debug//`. +`--no-cache` affects only the legacy opt-out. The bundled formatter defaults to +lowercase SQL at width 180; config overrides it, and JSON `null` disables +formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ----------------------------------------------- | ---------- | -------------------------------------------------- | -| `/supabase/config.toml` | TOML | always — pg-delta gate, ports, format options | -| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only — edge-runtime image tag | -| `/supabase/.temp/postgres-version` | plain text | shadow-DB image resolution (Go seam) | -| `/supabase/migrations/*.sql` | SQL | smart mode — detect whether migrations exist | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: catalog cache | -| `~/.supabase/access-token` | plain text | `--linked` (token resolution) | +| Path | Format | When | +| ----------------------------------------------- | ---------- | --------------------------------------------- | +| `/supabase/config.toml` | TOML | always — pg-delta gate, ports, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/.temp/postgres-version` | plain text | shadow-DB image resolution (Go seam) | +| `/supabase/migrations/*.sql` | SQL | smart mode — detect whether migrations exist | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog cache | +| `~/.supabase/access-token` | plain text | `--linked` (token resolution) | ## Files Written -| Path | Format | When | -| ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------- | -| `/supabase/database/**/*.sql` (declarative dir; configurable via `[experimental.pgdelta] declarative_schema_path`, or invocation-local `--output`) | SQL | the selected destination is wiped + rewritten after overwrite confirmation | -| `/.pgdelta-export.json` | JSON | default-engine export policy/manifest | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| --------------------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------ | +| `/supabase/database/**/*.sql` (configured declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | +| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers -| What | When | -| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | -| `supabase-go db schema declarative __catalog --mode baseline --experimental` — provisions and exports the legacy baseline catalog | legacy opt-out only | -| Edge-runtime container running the pg-delta declarative-export Deno script | legacy opt-out only | -| `docker`/`podman` container recreate for the local `db` (+ satellite restarts, Kong reload) — the same primitives `db start`/`db reset` use, via `legacyResetLocalDatabase` | smart-mode Local choice when reset is confirmed (or `--reset`) | +| What | When | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| `supabase-go db schema declarative __catalog --mode baseline --experimental` (hidden seam) — provisions a shadow Postgres + `start.SetupDatabase`, exports the baseline catalog | legacy opt-out only | +| Edge-runtime container (`supabase/edge-runtime`) running the pg-delta declarative-export Deno script (host network, deno-cache volume `supabase_edge_runtime_`) | legacy opt-out only | +| `docker`/`podman` container recreate for the local `db` (+ satellite restarts, Kong reload) — the same primitives `db start`/`db reset` use, via `legacyResetLocalDatabase` | smart-mode Local choice when reset is confirmed (or `--reset`) | ## Environment Variables @@ -66,9 +46,9 @@ platform view. | ---------------------------- | -------------------------------------------------- | --------- | | `SUPABASE_ACCESS_TOKEN` | auth token for `--linked` | no | | `DB_PASSWORD` | password for `--linked` / `--db-url` | no | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for the legacy edge-runtime engine | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: private npm registry | no | -| `PGDELTA_DEBUG` | structured default-engine debug artifacts | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | +| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | | `SUPABASE_GO_BINARY` | override the `supabase-go` seam binary | no | | `SUPABASE_SERVICES_HOSTNAME` | local DB host for `--local` (Go `GetHostname`) | no | | `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | @@ -106,14 +86,12 @@ always go to stderr, in every `--output-format`. On success: - Requires `--experimental` or `[experimental.pgdelta] enabled = true`. - `--db-url` / `--linked` / `--local` are mutually exclusive; absent all three, smart mode prompts (existing-files overwrite → Local/Custom choice + reset offer). -- `--output ` selects a destination for this invocation only. Relative paths - resolve from the project workdir; it does not edit config or activate the output - for later syncs. A non-empty destination still requires confirmation or - `--overwrite`, and the configured declarative tree is left untouched. -- The default engine preserves the shared direct/pooler, DNS, TLS, and client - certificate connection behavior. The legacy opt-out retains its embedded CA - file and `sslmode=verify-ca` URL rewrite. -- **Architecture:** the default engine extracts the target directly using the - bundled Supabase management profile, then renders and writes the export - in-process. Under the opt-out, Go provisions/exports a legacy baseline catalog - and edge-runtime runs the Deno script. +- `--output ` selects a destination for this invocation without changing + config or activating it for later syncs. +- Under the legacy opt-out, remote Supabase targets get the embedded pg-delta CA + bundle written under `supabase/.temp/pgdelta/` and the URL rewritten to + `sslmode=verify-ca`; the bundled engine uses the shared connection/TLS behavior. +- **Architecture:** under the legacy opt-out, the platform baseline is provisioned by the + bundled `supabase-go` via the hidden `db schema declarative __catalog` command + (it runs `start.SetupDatabase`'s auth/storage/realtime service migrations). The + bundled engine extracts and renders the target in-process. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 3b7f3c32a7..c913594e14 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -3,74 +3,58 @@ Diffs local migrations state against declarative schema files and writes the delta as a new timestamped migration. -## Pg-delta implementation and compatibility - -- The default pg-delta and bundled pg-topo run in-process at the versions fixed - when the CLI is built. There is no runtime download or automatic fallback. -- `SUPABASE_USE_PG_DELTA_NEXT=false` selects the legacy catalog/edge-runtime - implementation from either the shell or project `supabase/.env` (the shell - wins). `supabase/.temp/pgdelta-version`, `PGDELTA_NPM_REGISTRY`, and - catalogs directly below `supabase/.temp/pgdelta/` are legacy-only. -- `--no-cache` bypasses legacy catalog reuse/warming. The default engine always - extracts current state and maintains no reusable catalog cache. -- With `PGDELTA_DEBUG`, default-engine snapshots, plan, and diagnostics are - written below `supabase/.temp/pgdelta/v2/debug//` and are not reusable. -- The default engine always refuses extraction or declarative-loading errors. - Fatal diagnostics are always shown. By default, `unmodeled_kind` coverage gaps - are summarized once while nonfatal internal diagnostics remain quiet; - `--strict-coverage` refuses coverage gaps and prints the exact blockers. Debug - mode prints every diagnostic. Artifacts are saved before policy evaluation. -- Default-engine migrations may differ byte-for-byte and may be split into - ordered files to preserve transaction boundaries. Successful execution and an - empty subsequent sync are the compatibility contract. -- Default-engine migrations use pg-delta's human-facing formatter (lowercase - keywords, max width 180) after safe plan compaction. A JSON object in - `[experimental.pgdelta].format_options` partially overrides the formatter; - the JSON literal `null` disables formatting without disabling compaction. +Pg-delta runs in-process by default and uses two scoped shadow databases. Set +`SUPABASE_USE_PG_DELTA_NEXT=false` for the legacy catalog/edge-runtime path; +there is no automatic fallback. Coverage gaps warn; `--strict-coverage` makes +them fatal, while `PGDELTA_DEBUG` writes diagnostic JSON under +`supabase/.temp/pgdelta/v2/debug//`. Bundled output may use different SQL +and ordered transaction-aware files but must apply and converge. `--no-cache` +affects only the legacy opt-out. The bundled formatter defaults to lowercase SQL +at width 180; config overrides it, and JSON `null` disables formatting without +disabling safe compaction. ## Files Read -| Path | Format | When | -| -------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | -| `/supabase/.temp/pgdelta-version` | plain text | always read for compatibility; affects legacy only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out only — edge-runtime image tag | -| `/supabase/database/**/*.sql` (declarative dir) | SQL | always — must exist (else error) | -| `/supabase/migrations/*.sql` | SQL | default: applied to live shadow; legacy: native migrations-catalog resolution/cache | -| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | -| `/supabase/database/.pgdelta-export.json` | JSON | default-engine export policy, when present | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out only: migrations/declarative catalog cache | +| Path | Format | When | +| -------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------ | +| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/database/**/*.sql` (declarative dir) | SQL | always — must exist (else error) | +| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | +| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | +| `/supabase/database/.pgdelta-export.json` | JSON | bundled export metadata, when present | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | ## Files Written -| Path | Format | When | -| ------------------------------------------------------------------ | ------ | ---------------------------------------------------- | -| `/supabase/migrations/_[_].sql` | SQL | changes; default engine may emit ordered segments | -| `/supabase/database/extension.sql` | SQL | interactive, explicit legacy-extension repair only | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out only: native/Go-backed catalog caches | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | default engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| ------------------------------------------------------------------ | ------ | ------------------------------------------------- | +| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | +| `/supabase/database/extension.sql` | SQL | accepted legacy-extension repair | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers -| What | When | -| --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| Two scoped, natively-provisioned shadow Postgres databases: migrated source and declarative target | default engine | -| Natively-provisioned raw shadow used as the declarative export source | no files, bootstrap generation accepted, legacy opt-out | -| Natively-provisioned migrated shadow plus native migration replay and catalog export | legacy opt-out, migrations-catalog cache miss | -| `supabase-go db schema declarative __catalog --mode declarative --experimental` — declarative catalog target | legacy opt-out | -| Edge-runtime container running pg-delta diff/catalog-export scripts | legacy opt-out | -| `docker`/`podman` container recreate for local `db` (+ satellite restarts, Kong reload) via in-process `legacyResetLocalDatabase` | TTY only, apply failed, and the user confirms "reset and reapply" | +| What | When | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| Two natively-provisioned shadows: migrated source and declarative target | bundled engine | +| Natively-provisioned shadow Postgres container (CLI-1956 — `legacyCreateShadowDatabase`/`legacyPrepareShadowSource`) + native migrate/catalog export | legacy opt-out, migrations-catalog cache miss | +| `supabase-go db schema declarative __catalog --mode declarative --experimental` (seam) — shadow Postgres + `SetupDatabase` + apply declarative → catalog | legacy opt-out | +| Edge-runtime container running the pg-delta diff/catalog-export scripts | legacy opt-out | +| `docker`/`podman` container recreate for the local `db` (+ satellite restarts, Kong reload) — the same primitives `db start`/`db reset` use, via `legacyResetLocalDatabase` (CLI-2062: in-process, no `supabase-go` child) — only on the failed-apply recovery path | TTY only, apply failed, and the user confirms "reset and reapply" | ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------- | ------------------------------------------------------- | --------- | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for the legacy edge-runtime engine | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out only: private npm registry | no | -| `PGDELTA_DEBUG` | structured default-engine debug artifacts | no | -| `SUPABASE_GO_BINARY` | override the `supabase-go` seam binary | no | -| `SUPABASE_SERVICES_HOSTNAME` | local DB host for native bootstrap shadow orchestration | no | -| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | +| Variable | Purpose | Required? | +| ---------------------------- | ----------------------------------------------------------- | --------- | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | +| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | +| `SUPABASE_GO_BINARY` | override the `supabase-go` seam binary | no | +| `SUPABASE_SERVICES_HOSTNAME` | local DB host for the bootstrap generate (Go `GetHostname`) | no | +| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | ## Exit Codes @@ -95,47 +79,35 @@ surfaces before an `--apply`/`--no-apply` conflict is ever checked. Text mode only. The generated SQL, the created-migration path, drop-statement warnings, and apply status are written to stderr. The no-files bootstrap also prints `Declarative schema written to ` (the relative declarative dir, Go's -`GetDeclarativeDir()`) to stderr after generation and writing. Under the legacy -opt-out it prints after catalog warming — on both interactive and `--yes` paths. +`GetDeclarativeDir()`) to stderr after generating and writing (and, under the +legacy opt-out, warming the catalog cache) — on both interactive and `--yes` paths. `--no-apply` writes the migration only (never prompts/applies); `--apply` applies without prompting; both override the global `--yes`. `--no-apply` and `--apply` are mutually exclusive. -Before writing a migration, a manifest-less legacy tree that would remove only -`pgcrypto`, `uuid-ossp`, or `pg_net` offers three explicit choices: append the -detected declarations to root `extension.sql` and re-plan, continue with the -removals, or cancel. The repair uses `CREATE EXTENSION IF NOT EXISTS ... WITH -SCHEMA "extensions"`, never overwrites existing SQL, never creates a next-export -manifest, and proceeds only when the re-plan removes the compatibility gap. -Non-interactive execution, including global `--yes`, does not modify declarations -and stops with the exact SQL to add. +Before writing a bundled-engine migration, a manifest-less legacy tree that +would remove only `pgcrypto`, `uuid-ossp`, or `pg_net` offers to append their +declarations to `extension.sql` and re-plan, continue, or cancel. Non-interactive +execution (including `--yes`) stops and prints the SQL instead of modifying the +tree. The repair never overwrites existing SQL or creates an export manifest. ## Notes - Requires `--experimental` or `[experimental.pgdelta] enabled = true`. -- The declarative directory is the complete, hand-authored desired state. An - object omitted from it is intended to be removed, including extensions. This - is deterministic regardless of whether the directory was generated, written - by hand, or has a `.pgdelta-export.json` manifest. -- For gaps involving unknown extensions or extension-managed state such as - `pg_cron` jobs, generate a staged next-compatible tree with - `generate --output supabase/database-next`, review it, and adopt or - merge it explicitly. `--output` neither changes `config.toml` nor activates the - staged tree. -- The targeted `extension.sql` repair preserves detected installed extensions; - it does not certify the legacy tree as a complete pg-delta next export. - `--file` sets the migration filename stem (default `declarative_sync`); `--name` overrides it. In a TTY without `--name`/`--yes`, the name is prompted. - When no declarative files exist, a TTY offers to generate them (from local) first. +- The declarative directory is the complete desired state: omitted objects, + including extensions, are removals. Use `generate --output ` to + review a next-compatible tree without changing config or activating it. - The migration apply is native (connects to the local DB and records migration history). On apply failure a debug bundle is written under `supabase/.temp/pgdelta/debug/` and, in a TTY, a reset-and-reapply is offered (the reset itself is native too — `legacyResetLocalDatabase`, CLI-2062 — run in-process, sharing this command's own telemetry/linked-project-cache finalizer cycle rather than firing a second one from a `supabase-go` child). -- **Architecture:** the default engine uses two scoped live shadow databases and - plans/renders in-process. Under the legacy opt-out, the migrations-catalog diff - source resolves natively (CLI-1959): +- **Architecture:** the bundled engine plans/renders in-process from two live + shadows. Under the legacy opt-out, the migrations-catalog diff source resolves natively (CLI-1959): the setup-inputs-folded cache key, the zero-local-migrations → platform-baseline reuse, and the pg-delta catalog export are all native TS; the shadow-database platform-baseline provisioning + migrations apply is native too now (CLI-1956 — @@ -144,5 +116,5 @@ and stops with the exact SQL to add. declarative-catalog diff target still provisions its shadow-database platform baseline (and applies declarative files) via the hidden `db schema declarative __catalog --mode declarative` seam, since neither a baseline-only shadow nor - `pgdelta.ApplyDeclarative` has a native TS port yet (tracked by CLI-1823). The - legacy opt-out still runs its diff through the edge-runtime Deno script. + `pgdelta.ApplyDeclarative` has a native TS port yet (tracked by CLI-1823). The diff + legacy diff itself runs through the edge-runtime script. diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts index a5692989db..7e55df0c64 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts @@ -1,12 +1,4 @@ -import { execFileSync } from "node:child_process"; -import { - existsSync, - mkdirSync, - readdirSync, - readFileSync, - renameSync, - writeFileSync, -} from "node:fs"; +import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -16,10 +8,7 @@ import { describeDockerLive, runSupabaseLive } from "../../../../../tests/helper const COMMAND_TIMEOUT_MS = 280_000; const SCENARIO_TIMEOUT_MS = 900_000; -const NEXT_ENV = { - PGDELTA_DEBUG: "1", - SUPABASE_USE_PG_DELTA_NEXT: "true", -}; +const NEXT_ENV = { SUPABASE_USE_PG_DELTA_NEXT: "true" }; const initialDesiredSchema = `create type public.account_state as enum ('pending', 'active'); @@ -33,18 +22,6 @@ select id, email from auth.users; `; -const editedDesiredSchema = `create type public.account_state as enum ('pending', 'review', 'active'); - -create view public.auth_user_emails as -select id, email -from auth.users; - -create table public.review_queue ( - id bigint primary key, - state public.account_state not null default 'review' -); -`; - function commandFailure(result: { stdout: string; stderr: string }): string { return `stdout:\n${result.stdout}\nstderr:\n${result.stderr}`; } @@ -58,76 +35,8 @@ function migrationFiles(projectDir: string): ReadonlyArray { : []; } -function debugBundleDirectories(projectDir: string): ReadonlyArray { - const debugDir = path.join(projectDir, "supabase", ".temp", "pgdelta", "v2", "debug"); - if (!existsSync(debugDir)) return []; - return readdirSync(debugDir, { withFileTypes: true }) - .filter((entry) => entry.isDirectory()) - .map((entry) => path.join(debugDir, entry.name)) - .sort(); -} - -function requireDebugBundle(projectDir: string, operation: "declarativePlan" | "diff"): string { - const bundle = debugBundleDirectories(projectDir) - .filter((dir) => path.basename(dir).endsWith(`-${operation}`)) - .at(-1); - expect(bundle, `missing ${operation} debug bundle`).toBeDefined(); - if (bundle === undefined) throw new Error(`missing ${operation} debug bundle`); - return bundle; -} - -function assertJsonFile(file: string): unknown { - expect(existsSync(file), `missing ${file}`).toBe(true); - return JSON.parse(readFileSync(file, "utf8")); -} - -function localDatabaseUrl(config: string): string { - const dbSection = config.match(/\[db\][\s\S]*?\nport\s*=\s*(\d+)/u); - expect(dbSection?.[1], "db.port missing from generated config.toml").toBeDefined(); - return `postgresql://postgres:postgres@127.0.0.1:${dbSection?.[1]}/postgres?sslmode=disable`; -} - -function projectContainerIds(config: string): ReadonlyArray { - const projectId = config.match(/^project_id\s*=\s*"([^"]+)"/mu)?.[1]; - expect(projectId, "project_id missing from generated config.toml").toBeDefined(); - if (projectId === undefined) throw new Error("project_id missing from generated config.toml"); - const output = execFileSync( - "docker", - ["ps", "-aq", "--filter", `label=com.supabase.cli.project=${projectId}`], - { encoding: "utf8" }, - ); - return output.split(/\r?\n/u).filter(Boolean).sort(); -} - -function findSqlContaining(root: string, needle: string): string { - const match = readdirSync(root, { recursive: true }) - .filter((entry): entry is string => typeof entry === "string" && entry.endsWith(".sql")) - .map((entry) => path.join(root, entry)) - .find((file) => readFileSync(file, "utf8").includes(needle)); - expect(match, `no SQL file under ${root} contains ${needle}`).toBeDefined(); - if (match === undefined) throw new Error(`no SQL file under ${root} contains ${needle}`); - return match; -} - -function findExtensionDeclaration(root: string, extension: string): string { - const escaped = extension.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"); - const declaration = new RegExp( - `\\bCREATE\\s+EXTENSION(?:\\s+IF\\s+NOT\\s+EXISTS)?\\s+(?:"${escaped}"|${escaped})(?=\\s|;)`, - "iu", - ); - const match = readdirSync(root, { recursive: true }) - .filter((entry): entry is string => typeof entry === "string" && entry.endsWith(".sql")) - .map((entry) => path.join(root, entry)) - .find((file) => declaration.test(readFileSync(file, "utf8"))); - expect(match, `no SQL file under ${root} declares extension ${extension}`).toBeDefined(); - if (match === undefined) throw new Error(`no SQL file under ${root} declares ${extension}`); - return match; -} - describeDockerLive("pg-delta next local convergence (live)", () => { let projectDir = ""; - let desiredSchemaPath = ""; - let databaseUrl = ""; beforeAll(async () => { projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-live-")); @@ -151,12 +60,10 @@ describeDockerLive("pg-delta next local convergence (live)", () => { 'declarative_schema_path = "./schemas"', ), ); - databaseUrl = localDatabaseUrl(config); const schemasDir = path.join(projectDir, "supabase", "schemas"); mkdirSync(schemasDir, { recursive: true }); - desiredSchemaPath = path.join(schemasDir, "public.sql"); - writeFileSync(desiredSchemaPath, initialDesiredSchema); + writeFileSync(path.join(schemasDir, "public.sql"), initialDesiredSchema); const start = await runSupabaseLive( [ @@ -189,673 +96,43 @@ describeDockerLive("pg-delta next local convergence (live)", () => { }, COMMAND_TIMEOUT_MS); test( - "converges declarative state across empty, destructive, enum, URL, and migrations refs", + "applies a representative declarative schema and converges", { timeout: SCENARIO_TIMEOUT_MS }, async () => { expect(migrationFiles(projectDir)).toEqual([]); - const initialDiff = await runSupabaseLive( + const diff = await runSupabaseLive( ["db", "diff", "--local", "--use-pg-delta", "-f", "initial_declarative"], { cwd: projectDir, env: NEXT_ENV, exitTimeoutMs: COMMAND_TIMEOUT_MS }, ); - expect(initialDiff.exitCode, commandFailure(initialDiff)).toBe(0); + expect(diff.exitCode, commandFailure(diff)).toBe(0); - const initialMigrations = migrationFiles(projectDir); - expect(initialMigrations.length).toBeGreaterThan(0); - const initialSql = initialMigrations + const migrations = migrationFiles(projectDir); + expect(migrations.length).toBeGreaterThan(0); + const sql = migrations .map((file) => readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8")) .join("\n"); - expect(initialSql).toContain("account_state"); - expect(initialSql).toContain("disposable_note"); - expect(initialSql).toContain("auth_user_emails"); - expect(initialSql).toContain("auth.users"); - expect(initialSql).not.toMatch( + expect(sql).toContain("account_state"); + expect(sql).toContain("disposable_note"); + expect(sql).toContain("auth_user_emails"); + expect(sql).toContain("auth.users"); + expect(sql).not.toMatch( /CREATE\s+(?:SCHEMA|TABLE)\s+(?:IF\s+NOT\s+EXISTS\s+)?["']?(?:auth|storage|realtime)["']?/iu, ); - const declarativeBundle = requireDebugBundle(projectDir, "declarativePlan"); - expect(assertJsonFile(path.join(declarativeBundle, "metadata.json"))).toMatchObject({ - version: 1, - generation: "v2", - implementation: "next", - operation: "declarativePlan", - cacheReusable: false, - files: ["diagnostics.json", "plan.json"], - }); - assertJsonFile(path.join(declarativeBundle, "plan.json")); - expect(Array.isArray(assertJsonFile(path.join(declarativeBundle, "diagnostics.json")))).toBe( - true, - ); - - const firstReset = await runSupabaseLive(["db", "reset", "--local", "--no-seed"], { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(firstReset.exitCode, commandFailure(firstReset)).toBe(0); - - const emptyAfterInitial = await runSupabaseLive(["db", "diff", "--local", "--use-pg-delta"], { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(emptyAfterInitial.exitCode, commandFailure(emptyAfterInitial)).toBe(0); - expect(emptyAfterInitial.stderr).toContain("No schema changes found"); - - writeFileSync(desiredSchemaPath, editedDesiredSchema); - const beforeEdit = new Set(migrationFiles(projectDir)); - const editedDiff = await runSupabaseLive( - ["db", "diff", "--local", "--use-pg-delta", "-f", "enum_and_drop"], - { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(editedDiff.exitCode, commandFailure(editedDiff)).toBe(0); - expect(editedDiff.stderr).toContain("Found drop statements in schema diff"); - - const editedMigrations = migrationFiles(projectDir).filter((file) => !beforeEdit.has(file)); - expect(editedMigrations.length).toBeGreaterThan(1); - const editedMigrationSql = editedMigrations.map((file) => - readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8"), - ); - const editedSql = editedMigrationSql.join("\n"); - expect(editedSql).toMatch(/ALTER\s+TYPE[\s\S]*account_state[\s\S]*ADD\s+VALUE/iu); - expect(editedSql).toMatch(/DROP\s+TABLE[\s\S]*disposable_note/iu); - expect(editedSql).toContain("review_queue"); - - const enumPush = await runSupabaseLive(["db", "push", "--local"], { + const reset = await runSupabaseLive(["db", "reset", "--local", "--no-seed"], { cwd: projectDir, - env: { SUPABASE_YES: "true" }, exitTimeoutMs: COMMAND_TIMEOUT_MS, }); - expect(enumPush.exitCode, commandFailure(enumPush)).toBe(0); - - const emptyAfterEdit = await runSupabaseLive(["db", "diff", "--local", "--use-pg-delta"], { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(emptyAfterEdit.exitCode, commandFailure(emptyAfterEdit)).toBe(0); - expect(emptyAfterEdit.stderr).toContain("No schema changes found"); - - const explicit = await runSupabaseLive( - ["db", "diff", "--from", "migrations", "--to", databaseUrl], - { cwd: projectDir, env: NEXT_ENV, exitTimeoutMs: COMMAND_TIMEOUT_MS }, - ); - expect(explicit.exitCode, commandFailure(explicit)).toBe(0); - expect(explicit.stdout.trim()).toBe(""); - - const diffBundle = requireDebugBundle(projectDir, "diff"); - expect(assertJsonFile(path.join(diffBundle, "metadata.json"))).toMatchObject({ - version: 1, - generation: "v2", - implementation: "next", - operation: "diff", - cacheReusable: false, - files: ["desired-snapshot.json", "diagnostics.json", "plan.json", "source-snapshot.json"], - }); - const sourceSnapshot = readFileSync(path.join(diffBundle, "source-snapshot.json"), "utf8"); - const desiredSnapshot = readFileSync(path.join(diffBundle, "desired-snapshot.json"), "utf8"); - expect(sourceSnapshot).toContain("account_state"); - expect(desiredSnapshot).toContain("account_state"); - JSON.parse(sourceSnapshot); - JSON.parse(desiredSnapshot); - assertJsonFile(path.join(diffBundle, "plan.json")); - expect(Array.isArray(assertJsonFile(path.join(diffBundle, "diagnostics.json")))).toBe(true); - - const generated = await runSupabaseLive( - ["db", "schema", "declarative", "generate", "--local", "--overwrite"], - { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(generated.exitCode, commandFailure(generated)).toBe(0); - - const exportedFiles = readdirSync(path.join(projectDir, "supabase", "schemas"), { - recursive: true, - }) - .filter((entry): entry is string => typeof entry === "string" && entry.endsWith(".sql")) - .map((entry) => path.join(projectDir, "supabase", "schemas", entry)) - .sort(); - expect(exportedFiles.length).toBeGreaterThan(0); - expect(existsSync(path.join(projectDir, "supabase", "schemas", ".pgdelta-export.json"))).toBe( - true, - ); - - const migrationsBeforeGeneratedSync = migrationFiles(projectDir); - const emptyGeneratedSync = await runSupabaseLive( - ["db", "schema", "declarative", "sync", "--no-apply"], - { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(emptyGeneratedSync.exitCode, commandFailure(emptyGeneratedSync)).toBe(0); - expect(emptyGeneratedSync.stderr).toContain("No schema changes found"); - expect(migrationFiles(projectDir)).toEqual(migrationsBeforeGeneratedSync); - - const editedExport = exportedFiles[0]; - expect(editedExport).toBeDefined(); - if (editedExport === undefined) throw new Error("declarative export produced no SQL files"); - writeFileSync( - editedExport, - `${readFileSync(editedExport, "utf8")}\ncreate table public.phase6_synced (id bigint primary key);\n`, - ); - - const migrationsBeforeApplySync = new Set(migrationFiles(projectDir)); - const appliedSync = await runSupabaseLive( - ["db", "schema", "declarative", "sync", "--apply", "--name", "phase6_sync"], - { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(appliedSync.exitCode, commandFailure(appliedSync)).toBe(0); - expect(appliedSync.stderr).toContain("Migration applied successfully"); - const appliedSyncMigrations = migrationFiles(projectDir).filter( - (file) => !migrationsBeforeApplySync.has(file), - ); - expect(appliedSyncMigrations.length).toBeGreaterThan(0); - expect( - appliedSyncMigrations - .map((file) => - readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8"), - ) - .join("\n"), - ).toContain("phase6_synced"); - - const emptyAppliedSync = await runSupabaseLive( - ["db", "schema", "declarative", "sync", "--no-apply"], - { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(emptyAppliedSync.exitCode, commandFailure(emptyAppliedSync)).toBe(0); - expect(emptyAppliedSync.stderr).toContain("No schema changes found"); - - const dbOnlyChange = await runSupabaseLive( - ["db", "query", "--local", "create table public.phase6_pulled (id bigint primary key)"], - { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, - ); - expect(dbOnlyChange.exitCode, commandFailure(dbOnlyChange)).toBe(0); - - const configPath = path.join(projectDir, "supabase", "config.toml"); - const pullConfig = readFileSync(configPath, "utf8") - .replace('schema_paths = ["./schemas/*.sql"]', "schema_paths = []") - .replace('declarative_schema_path = "./schemas"', 'declarative_schema_path = "./database"'); - writeFileSync(configPath, pullConfig); - renameSync( - path.join(projectDir, "supabase", "schemas"), - path.join(projectDir, "supabase", ".phase6-exported-schemas"), - ); - - const migrationsBeforePull = new Set(migrationFiles(projectDir)); - const pulled = await runSupabaseLive( - ["db", "pull", "phase6_pull", "--db-url", databaseUrl, "--diff-engine", "pg-delta"], - { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true", SUPABASE_YES: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(pulled.exitCode, commandFailure(pulled)).toBe(0); - const pulledMigrations = migrationFiles(projectDir).filter( - (file) => !migrationsBeforePull.has(file), - ); - expect(pulledMigrations.length).toBeGreaterThan(0); - expect( - pulledMigrations - .map((file) => - readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8"), - ) - .join("\n"), - ).toContain("phase6_pulled"); + expect(reset.exitCode, commandFailure(reset)).toBe(0); - const removePulledTable = await runSupabaseLive( - ["db", "query", "--local", "drop table public.phase6_pulled"], - { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(removePulledTable.exitCode, commandFailure(removePulledTable)).toBe(0); - - const pulledVersion = pulledMigrations[0]?.split("_", 1)[0]; - expect(pulledVersion).toMatch(/^\d{14}$/u); - if (pulledVersion === undefined) throw new Error("db pull produced no migration version"); - const markPulledReverted = await runSupabaseLive( - ["migration", "repair", "--local", "--status", "reverted", pulledVersion], - { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(markPulledReverted.exitCode, commandFailure(markPulledReverted)).toBe(0); - - const pullPush = await runSupabaseLive(["db", "push", "--local"], { - cwd: projectDir, - env: { SUPABASE_YES: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(pullPush.exitCode, commandFailure(pullPush)).toBe(0); - - const emptyPull = await runSupabaseLive( - ["db", "pull", "phase6_pull_empty", "--db-url", databaseUrl, "--diff-engine", "pg-delta"], - { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "true", SUPABASE_YES: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(emptyPull.exitCode, commandFailure(emptyPull)).toBe(1); - expect(emptyPull.stderr).toContain("No schema changes found"); - }, - ); - - test( - "keeps the legacy edge-runtime implementation available behind the opt-out", - { timeout: SCENARIO_TIMEOUT_MS }, - async (context) => { - const legacy = await runSupabaseLive( - ["db", "diff", "--from", "migrations", "--to", databaseUrl], - { - cwd: projectDir, - env: { SUPABASE_USE_PG_DELTA_NEXT: "false" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - const output = `${legacy.stdout}\n${legacy.stderr}`; - if ( - legacy.exitCode !== 0 && - /(?:No such image|manifest unknown|pull access denied|edge-runtime: (?:not found|command not found))/iu.test( - output, - ) - ) { - context.skip("legacy edge-runtime image is concretely unavailable on this Docker host"); - } - expect(legacy.exitCode, commandFailure(legacy)).toBe(0); - }, - ); -}); - -describeDockerLive("pg-delta next declarative extension baseline (live)", () => { - let projectDir = ""; - let config = ""; - - beforeAll(async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-extensions-live-")); - - const init = await runSupabaseLive(["init"], { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(init.exitCode, commandFailure(init)).toBe(0); - - const configPath = path.join(projectDir, "supabase", "config.toml"); - const generatedConfig = readFileSync(configPath, "utf8"); - expect(generatedConfig).toContain("major_version = 17"); - expect(generatedConfig).not.toContain("[experimental.webhooks]"); - config = `${generatedConfig - .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') - .replace( - '# declarative_schema_path = "./database"', - 'declarative_schema_path = "./schemas"', - )}\n[experimental.webhooks]\nenabled = true\n`; - writeFileSync(configPath, config); - - const start = await runSupabaseLive( - [ - "start", - "--exclude", - "studio", - "--exclude", - "logflare", - "--exclude", - "vector", - "--exclude", - "gotrue", - "--exclude", - "realtime", - "--exclude", - "storage-api", - ], - { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, - ); - expect(start.exitCode, commandFailure(start)).toBe(0); - }, COMMAND_TIMEOUT_MS); - - afterAll(async () => { - if (projectDir.length === 0) return; - await runSupabaseLive(["stop", "--no-backup"], { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }).catch(() => undefined); - await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); - }, COMMAND_TIMEOUT_MS); - - test( - "loads exported user-managed extensions and plans their removal by file deletion", - { timeout: SCENARIO_TIMEOUT_MS }, - async () => { - const generated = await runSupabaseLive( - ["db", "schema", "declarative", "generate", "--local", "--overwrite"], - { - cwd: projectDir, - env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(generated.exitCode, commandFailure(generated)).toBe(0); - - const schemasDir = path.join(projectDir, "supabase", "schemas"); - findExtensionDeclaration(schemasDir, "pg_net"); - const pgcryptoFile = findExtensionDeclaration(schemasDir, "pgcrypto"); - findExtensionDeclaration(schemasDir, "uuid-ossp"); - // The directory itself is the complete desired-state contract. A missing - // manifest must not preserve an extension omitted from the SQL files. - await rm(path.join(schemasDir, ".pgdelta-export.json")); - - const containersBeforeEmpty = projectContainerIds(config); - const migrationsBeforeEmpty = migrationFiles(projectDir); - const empty = await runSupabaseLive(["db", "schema", "declarative", "sync", "--no-apply"], { + const converged = await runSupabaseLive(["db", "diff", "--local", "--use-pg-delta"], { cwd: projectDir, env: NEXT_ENV, exitTimeoutMs: COMMAND_TIMEOUT_MS, }); - expect(empty.exitCode, commandFailure(empty)).toBe(0); - expect(empty.stderr).toContain("No schema changes found"); - expect(migrationFiles(projectDir)).toEqual(migrationsBeforeEmpty); - expect(projectContainerIds(config)).toEqual(containersBeforeEmpty); - - const pgcryptoSql = readFileSync(pgcryptoFile, "utf8"); - const migrationsBeforeRemoval = new Set(migrationFiles(projectDir)); - await rm(pgcryptoFile); - try { - const containersBeforeRemoval = projectContainerIds(config); - const removal = await runSupabaseLive( - ["db", "schema", "declarative", "sync", "--no-apply", "--name", "drop_pgcrypto"], - { - cwd: projectDir, - env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(removal.exitCode, commandFailure(removal)).toBe(0); - expect(projectContainerIds(config)).toEqual(containersBeforeRemoval); - - const removalMigrations = migrationFiles(projectDir).filter( - (file) => !migrationsBeforeRemoval.has(file), - ); - expect(removalMigrations.length).toBeGreaterThan(0); - const removalSql = removalMigrations - .map((file) => - readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8"), - ) - .join("\n"); - expect(removalSql).toMatch(/DROP\s+EXTENSION(?:\s+IF\s+EXISTS)?\s+"?pgcrypto"?/iu); - } finally { - writeFileSync(pgcryptoFile, pgcryptoSql); - await Promise.all( - migrationFiles(projectDir) - .filter((file) => !migrationsBeforeRemoval.has(file)) - .map((file) => rm(path.join(projectDir, "supabase", "migrations", file))), - ); - } - }, - ); -}); - -describeDockerLive("pg-delta next PG14 declarative scratch (live)", () => { - let projectDir = ""; - - beforeAll(async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-pg14-live-")); - - const init = await runSupabaseLive(["init"], { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(init.exitCode, commandFailure(init)).toBe(0); - - const configPath = path.join(projectDir, "supabase", "config.toml"); - const generatedConfig = readFileSync(configPath, "utf8"); - expect(generatedConfig).toContain("major_version = 17"); - writeFileSync( - configPath, - generatedConfig - .replace("major_version = 17", "major_version = 14") - .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') - .replace( - '# declarative_schema_path = "./database"', - 'declarative_schema_path = "./schemas"', - ), - ); - - const schemasDir = path.join(projectDir, "supabase", "schemas"); - mkdirSync(schemasDir, { recursive: true }); - writeFileSync( - path.join(schemasDir, "extensions.sql"), - [ - "create extension if not exists pgcrypto with schema extensions;", - "create extension if not exists pgjwt with schema extensions;", - 'create extension if not exists "uuid-ossp" with schema extensions;', - "", - ].join("\n"), - ); - }, COMMAND_TIMEOUT_MS); - - afterAll(async () => { - if (projectDir.length === 0) return; - await runSupabaseLive(["stop", "--no-backup"], { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }).catch(() => undefined); - await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); - }, COMMAND_TIMEOUT_MS); - - test( - "provisions an extension-free PG14 scratch before loading desired declarations", - { timeout: SCENARIO_TIMEOUT_MS }, - async () => { - const sync = await runSupabaseLive(["db", "schema", "declarative", "sync", "--no-apply"], { - cwd: projectDir, - env: { ...NEXT_ENV, SUPABASE_YES: "true" }, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(sync.exitCode, commandFailure(sync)).toBe(0); - expect(sync.stderr).toContain("No schema changes found"); - }, - ); -}); - -describeDockerLive("pg-delta next isolated cron shadows (live)", () => { - const jobName = "pgdelta_cli_inactive"; - const initialSchedule = "0 0 * * *"; - const changedSchedule = "15 3 * * *"; - let projectDir = ""; - let config = ""; - - beforeAll(async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-cron-live-")); - - const init = await runSupabaseLive(["init"], { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(init.exitCode, commandFailure(init)).toBe(0); - - const configPath = path.join(projectDir, "supabase", "config.toml"); - config = readFileSync(configPath, "utf8") - .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') - .replace('# declarative_schema_path = "./database"', 'declarative_schema_path = "./schemas"'); - writeFileSync(configPath, config); - - const migrationsDir = path.join(projectDir, "supabase", "migrations"); - mkdirSync(migrationsDir, { recursive: true }); - writeFileSync( - path.join(migrationsDir, "20260806000000_cron_inactive.sql"), - `create extension if not exists pg_cron; - -create table public.pgdelta_cron_execution_sentinel ( - executed_at timestamptz not null default now() -); - -select cron.schedule( - '${jobName}', - '${initialSchedule}', - 'insert into public.pgdelta_cron_execution_sentinel default values' -); - -select cron.alter_job( - (select jobid from cron.job where jobname = '${jobName}'), - active := false -); -`, - ); - - const start = await runSupabaseLive( - [ - "start", - "--exclude", - "studio", - "--exclude", - "logflare", - "--exclude", - "vector", - "--exclude", - "gotrue", - "--exclude", - "realtime", - "--exclude", - "storage-api", - ], - { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, - ); - expect(start.exitCode, commandFailure(start)).toBe(0); - }, COMMAND_TIMEOUT_MS); - - afterAll(async () => { - if (projectDir.length === 0) return; - await runSupabaseLive(["stop", "--no-backup"], { - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }).catch(() => undefined); - await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); - }, COMMAND_TIMEOUT_MS); - - test( - "keeps an inactive named job converged and replaces only its changed schedule", - { timeout: SCENARIO_TIMEOUT_MS }, - async () => { - const generated = await runSupabaseLive( - ["db", "schema", "declarative", "generate", "--local", "--overwrite"], - { - cwd: projectDir, - env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(generated.exitCode, commandFailure(generated)).toBe(0); - - const schemasDir = path.join(projectDir, "supabase", "schemas"); - const cronFile = findSqlContaining(schemasDir, `cron.schedule_in_database('${jobName}'`); - const containersBeforeEmpty = projectContainerIds(config); - const empty = await runSupabaseLive(["db", "schema", "declarative", "sync", "--no-apply"], { - cwd: projectDir, - env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); - expect(empty.exitCode, commandFailure(empty)).toBe(0); - expect(empty.stderr).toContain("No schema changes found"); - expect(projectContainerIds(config)).toEqual(containersBeforeEmpty); - - const emptyBundle = requireDebugBundle(projectDir, "declarativePlan"); - expect(assertJsonFile(path.join(emptyBundle, "plan.json"))).toMatchObject({ - deltas: [], - actions: [], - source: { fingerprint: expect.any(String) }, - target: { fingerprint: expect.any(String) }, - }); - - const exportedCron = readFileSync(cronFile, "utf8"); - expect(exportedCron).toContain(`'${initialSchedule}'`); - writeFileSync(cronFile, exportedCron.replace(`'${initialSchedule}'`, `'${changedSchedule}'`)); - - const migrationsBeforeApply = new Set(migrationFiles(projectDir)); - const containersBeforeApply = projectContainerIds(config); - const applied = await runSupabaseLive( - ["db", "schema", "declarative", "sync", "--apply", "--name", "cron_schedule"], - { - cwd: projectDir, - env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(applied.exitCode, commandFailure(applied)).toBe(0); - expect(applied.stderr).toContain("Migration applied successfully"); - expect(projectContainerIds(config)).toEqual(containersBeforeApply); - - const scheduleMigrations = migrationFiles(projectDir).filter( - (file) => !migrationsBeforeApply.has(file), - ); - expect(scheduleMigrations.length).toBeGreaterThan(0); - const scheduleSql = scheduleMigrations - .map((file) => readFileSync(path.join(projectDir, "supabase", "migrations", file), "utf8")) - .join("\n"); - expect(scheduleSql.match(/cron\.unschedule/gu)).toHaveLength(1); - expect(scheduleSql.match(/cron\.schedule_in_database/gu)).toHaveLength(1); - expect(scheduleSql).toContain(`'${changedSchedule}'`); - expect(scheduleSql).not.toMatch( - /\b(?:create|alter|drop)\s+(?:table|schema|function|view|extension|role)\b/iu, - ); - - const job = await runSupabaseLive( - [ - "db", - "query", - "--local", - "-o", - "json", - `select schedule, active from cron.job where jobname = '${jobName}'`, - ], - { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, - ); - expect(job.exitCode, commandFailure(job)).toBe(0); - expect(JSON.parse(job.stdout)).toEqual([{ schedule: changedSchedule, active: false }]); - - const executions = await runSupabaseLive( - [ - "db", - "query", - "--local", - "-o", - "json", - "select count(*)::int as executions from public.pgdelta_cron_execution_sentinel", - ], - { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, - ); - expect(executions.exitCode, commandFailure(executions)).toBe(0); - expect(JSON.parse(executions.stdout)).toEqual([{ executions: 0 }]); - - const containersBeforeFinal = projectContainerIds(config); - const finalSync = await runSupabaseLive( - ["db", "schema", "declarative", "sync", "--no-apply"], - { - cwd: projectDir, - env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }, - ); - expect(finalSync.exitCode, commandFailure(finalSync)).toBe(0); - expect(finalSync.stderr).toContain("No schema changes found"); - expect(projectContainerIds(config)).toEqual(containersBeforeFinal); + expect(converged.exitCode, commandFailure(converged)).toBe(0); + expect(converged.stderr).toContain("No schema changes found"); }, ); }); diff --git a/docs/adr/0017-schema-first-database-workflow.md b/docs/adr/0017-schema-first-database-workflow.md deleted file mode 100644 index d47b5008e7..0000000000 --- a/docs/adr/0017-schema-first-database-workflow.md +++ /dev/null @@ -1,140 +0,0 @@ -# 0017. Schema-First Database Workflow - -**Status**: proposed -**Date**: 2026-08-10 - -## Problem Statement - -The alpha CLI needs one coherent workflow for declaring database shape, reviewing the -resulting changes, applying them locally, and synchronizing them with the Supabase -platform. PostgreSQL DDL, migration files, live database state, and platform migration -history are related but distinct forms of state. Without an explicit ownership model, -the CLI can generate migrations that do not match what it applies, overwrite local -intent during a pull, or hide drift between local and remote databases. - -The repository already depends on `@supabase/pg-delta`, which can extract PostgreSQL -state, load declarative SQL through a shadow database, plan changes, render execution- -aware SQL files, apply fingerprint-gated plans, and export declarative SQL. The CLI must -decide which responsibilities belong to pg-delta and which belong to the schema and -migrations workflows. - -## Decisions Already Established - -- `schema` is the primary public command group for database shape changes. -- Declarative schema is the default alpha workflow; migration files are its generated - implementation artifact. -- `schema generate` derives migration files without applying them. -- `schema apply` mutates the local database from declared schema intent. -- `schema push` synchronizes declared schema intent to the platform. -- `schema pull` refreshes the local declarative representation from platform state. -- `migrations` remains the advanced file-level workflow and does not own declarative - schema generation. -- `apply` means local database mutation; `push` and `pull` mean platform synchronization. - -## Proposed Integration Boundary - -Pending interview decisions, the working boundary is: - -- pg-delta is the schema compiler: database/catalog extraction, managed-view policy, - declarative SQL loading, rename analysis, change planning, safety metadata, plan - rendering, and declarative export. -- the CLI is the workflow coordinator: project layout, target resolution, shadow - lifecycle, migration naming and ledger integration, conflict policy, user prompts, - structured output, retries, and composition of schema operations with migrations. -- the migration runner remains responsible for applying and recording concrete - migration files. The CLI must not claim a generated migration was applied unless the - same execution units were recorded in migration history. - -## pg-delta Fit and Gaps - -pg-delta is a strong fit for the schema-compilation portion of the workflow. Its -current library API can load declarations into a real shadow database, extract and -compare managed state, report rename candidates and safety metadata, render one SQL -file per execution segment, apply a fingerprint-gated plan, prove convergence on a -clone, and export a manifest-owned declarative tree. - -It does not provide the workflow around those primitives: - -- no migration versions, applied-history ledger, pending-migration reconciliation, - repair, or append-only file policy; -- no Supabase project/branch resolution, authentication, or production guardrails; -- native shadow lifecycle now exists, but the alpha workflow still needs an explicit - ownership boundary between shared bootstrap and pg-delta-specific isolation; -- no policy for reconciling pulled state with local edits or unpushed migrations; -- no guarantee that a multi-segment plan is globally transactional; -- no stable API or persisted artifact compatibility while the dependency remains alpha. - -Coverage also needs an explicit alpha contract. Entirely unmodeled object kinds are -diagnosed and can be rejected with strict coverage, but known attributes within modeled -families are still invisible to extraction and therefore to pg-delta's own proof. The -Supabase integration profile does not yet ship a complete versioned platform baseline, -and stateful extension intent is only partially represented. Alpha must document and -preflight its supported subset rather than claim complete PostgreSQL round-trip fidelity. - -## Initial Alpha Recommendation - -Subject to the decisions below: - -- use migration replay in a fresh shadow as the observed baseline for generation; -- treat declarations as desired state and append-only migration files/history as the - transition ledger; -- have `schema apply` and `schema push` execute generated files through the migration - runner, never through pg-delta's direct target mutation path; -- export pulls into staging and replace only files owned by the export manifest; -- start with database scope, strict coverage, shape-only SQL, explicit rename acceptance, - and no credential-bearing declarative objects; -- preserve pg-delta execution segments as distinct migration versions; -- keep pg-delta behind a narrow adapter and persist plans only as diagnostic artifacts. - -## Decisions Required - -1. Whether schema commands and direct migration commands may be mixed in one project, - and how the CLI detects stale declarative state when they are mixed. -2. The comparison baseline for `schema generate`: local live state, migration replay, - the last generated checkpoint, or another durable snapshot. -3. Whether `schema apply` must always generate migration files first and then apply - those files, or may apply a pg-delta plan directly. -4. Whether `schema push` generates from local state and pushes migrations, or plans - directly against remote state and then materializes the resulting migration files. -5. Pull conflict behavior when local declarative files or unpushed migrations differ - from the platform. -6. Alpha management scope: database objects only, or cluster-global roles and grants - as well. -7. Alpha safety defaults for destructive changes, rename ambiguity, unmodeled object - diagnostics, table rewrites, lock risk, and declarative files containing data. -8. Whether proof on a disposable clone is required, optional, or deferred for local and - remote workflows. -9. The stability strategy for pg-delta's breaking-change alpha API and artifact formats. - -## Upstream Work or Deferrals - -- Complete the versioned Supabase baseline and Cloud/local default-privilege handling. -- Close modeled-family extraction gaps before advertising broad round-trip fidelity. -- Extend intent handlers for pgmq and pg_partman before making them declarative. -- Improve explicit rename declarations for non-interactive and rename-plus-edit cases. -- Add independent verification for critical workflows rather than treating a proof that - shares the planner's extractor as an independent oracle. - -## Consequences - -### Positive - -- The public workflow stays centered on schema intent while retaining reviewable, - auditable migration history. -- pg-delta's planner, managed-view policy, safety metadata, and export logic are reused - instead of duplicated in the CLI. -- Explicit ownership makes local apply, remote push, and pull conflict behavior testable. - -### Negative - -- The CLI needs orchestration and durable-state machinery beyond pg-delta itself. -- Supporting both declarative and direct migration workflows creates reconciliation - cases that must fail closed when intent is ambiguous. -- pg-delta is currently a breaking-change alpha dependency, so the integration needs a - narrow adapter and compatibility tests. - -## Related Decisions - -- [ADR 0004](0004-cli-design-goals-and-workflows.md): CLI Design Goals & Development Workflows -- [Alpha command structure](../cli/dev-alpha-command-structure.md) -- [Schema workflow glossary](../cli/schema-workflow-glossary.md) diff --git a/docs/adr/README.md b/docs/adr/README.md index 25c3cd0111..887b3e57b5 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -57,7 +57,6 @@ When an ADR becomes outdated, mark it as `deprecated` or reference the supersedi | 0013 | [Live E2E Tests Bypass the Replay Server](0013-live-e2e-bypasses-replay-server.md) | proposed | | 0015 | [Managed Stack Contract Fixtures](0015-managed-stack-contract-fixtures.md) | proposed | | 0016 | [Legacy Port Completion and Go CLI Authority Scope](0016-legacy-port-completion-and-go-cli-authority-scope.md) | proposed | -| 0017 | [Schema-First Database Workflow](0017-schema-first-database-workflow.md) | proposed | ## Template diff --git a/docs/cli/schema-workflow-glossary.md b/docs/cli/schema-workflow-glossary.md deleted file mode 100644 index d8d6e763d8..0000000000 --- a/docs/cli/schema-workflow-glossary.md +++ /dev/null @@ -1,28 +0,0 @@ -# Schema Workflow Glossary - -This glossary defines the terms used while designing the alpha database workflow. It -distinguishes user intent from generated artifacts and live state. - -| Term | Meaning | -| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Declared schema intent | The user-authored SQL files that describe the desired managed database shape. | -| Local schema representation | The declared schema intent plus the metadata needed to interpret it consistently, such as management scope and profile. | -| Managed database shape | The subset of PostgreSQL objects owned by the workflow after applying the selected pg-delta profile, baseline, and scope. | -| Schema compiler | pg-delta's role: turn observed and desired managed database shapes into a plan or export. It does not own the project's migration ledger. | -| Plan | A versioned pg-delta artifact containing ordered actions, fingerprints, transaction boundaries, rename decisions, and safety metadata. | -| Generated migration | One or more concrete SQL migration files rendered from a plan and named for the project's migration runner. | -| Migration ledger | The ordered record of concrete migrations known to the project and recorded as applied by a target database. | -| Schema checkpoint | A possible durable record tying declared schema intent to the migration ledger and observed database fingerprint. Whether alpha needs this is undecided. | -| Shadow database | A disposable PostgreSQL database or isolated cluster used to load declared SQL so pg-delta can observe the desired state. | -| Baseline | A pg-delta snapshot subtracted from both sides to keep platform-provided objects outside the managed shape. | -| Profile | pg-delta policy and integration configuration defining managed objects, assumed platform state, handlers, redaction, and an optional baseline. | -| Database scope | Management of objects local to one database; cluster-global role creation and membership are excluded. | -| Cluster scope | Management that includes cluster-global objects and therefore requires a genuinely isolated shadow cluster. | -| Drift | A difference between an expected managed state and an observed live target that is not represented by the intended workflow transition. | -| Generate | Compile declared schema intent into migration files without mutating a live target. | -| Apply | Mutate the local database and record the concrete migration execution consistently. | -| Push | Synchronize local intent to a platform target. It is never shorthand for local mutation. | -| Pull | Export platform state into the local schema representation. It does not mutate the local database. | -| Destructive change | A planned action that drops an object or that pg-delta marks as destructive data loss. | -| Rewrite risk | A planned action that may rewrite table data even when it is not classified as destructive. | -| Unmodeled object | User-created PostgreSQL state detected by pg-delta but not represented by its fact model. | From 23814e76d703aa775b0d4432e101faeb590a1de7 Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 14 Aug 2026 16:04:05 +0200 Subject: [PATCH 26/82] feat(cli): upgrade pg-delta next to alpha.40 --- apps/cli/package.json | 2 +- .../legacy/commands/db/diff/diff.handler.ts | 14 +- .../commands/db/diff/diff.integration.test.ts | 22 ++ ...eclarative.orchestrate.integration.test.ts | 18 +- .../declarative/declarative.orchestrate.ts | 5 +- .../generate/generate.integration.test.ts | 4 +- .../schema/declarative/sync/sync.handler.ts | 6 +- .../declarative/sync/sync.integration.test.ts | 2 +- .../legacy-pgdelta-engine.next.layer.ts | 3 + ...acy-pgdelta-engine.next.layer.unit.test.ts | 1 + .../shared/legacy-pgdelta-engine.service.ts | 28 ++ .../legacy-pgdelta-next-adapter.layer.ts | 50 +++- .../legacy-pgdelta-next-adapter.service.ts | 14 +- .../legacy-pgdelta-next-adapter.unit.test.ts | 77 +++++- .../shared/legacy-pgdelta-next-diagnostics.ts | 25 +- ...gacy-pgdelta-next-diagnostics.unit.test.ts | 30 +- .../db/shared/legacy-pgdelta.write.ts | 258 ++++++++++++++---- .../shared/legacy-pgdelta.write.unit.test.ts | 100 ++++++- pnpm-lock.yaml | 10 +- pnpm-workspace.yaml | 2 +- 20 files changed, 589 insertions(+), 82 deletions(-) diff --git a/apps/cli/package.json b/apps/cli/package.json index a74a4076de..1560ea3923 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -55,7 +55,7 @@ "@parcel/watcher": "^2.6.0", "@supabase/api": "workspace:*", "@supabase/config": "workspace:*", - "@supabase/pg-delta": "1.0.0-alpha.34", + "@supabase/pg-delta": "1.0.0-alpha.40", "@supabase/pg-topo": "1.0.0-alpha.5", "@supabase/process-compose": "workspace:*", "@supabase/stack": "workspace:*", diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index 3764d0568f..7ad945bd28 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -51,6 +51,7 @@ import { legacyDiffMigra } from "../shared/legacy-migra.ts"; import { LegacyPgDeltaEngine, type LegacyPgDeltaDatabaseEndpoint, + type LegacyPgDeltaDiffResult, type LegacyPgDeltaEndpoint, type LegacyPgDeltaRenderedFile, } from "../shared/legacy-pgdelta-engine.service.ts"; @@ -585,6 +586,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy let diffResult: { readonly sql: string; readonly files: ReadonlyArray | undefined; + readonly hazards?: LegacyPgDeltaDiffResult["hazards"]; }; if (usePgAdmin) { // The running-db check runs AFTER the config load + target resolve above, and — @@ -740,7 +742,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // Keep the per-unit plan files so a multi-unit plan can be written as one // migration file each; `sql` stays the flattened join for stdout review + // machine payloads. - return { sql: result.sql, files: result.files }; + return { sql: result.sql, files: result.files, hazards: result.hazards }; } const sql = yield* legacyDiffMigra(ctx, { source: shadow.sourceUrl, @@ -771,7 +773,11 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // The file-write + drop-statement warning below is bypassed by the pgadmin path. const engine = usePgAdmin ? "pgadmin" : useDelta ? "pg-delta" : "migra"; - const drops: ReadonlyArray = usePgAdmin ? [] : legacyFindDropStatements(out); + const drops: ReadonlyArray = usePgAdmin + ? [] + : diffResult.hazards !== undefined + ? diffResult.hazards.dataLoss.map((action) => action.sql) + : legacyFindDropStatements(out); const writtenFiles: Array = []; let ignoredDeclarativeAdvisory: ReturnType | undefined; if ( @@ -850,7 +856,9 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy } if (drops.length > 0) { yield* output.raw( - "Found drop statements in schema diff. Please double check if these are expected:\n", + diffResult.hazards === undefined + ? "Found drop statements in schema diff. Please double check if these are expected:\n" + : "Found destructive changes in schema diff. Please double check if these are expected:\n", "stderr", ); yield* output.raw(`${legacyYellow(drops.join("\n"))}\n`, "stderr"); diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index ffb0c137d8..ac4afe3443 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -55,6 +55,7 @@ import { LegacyPgDeltaEngine, type LegacyPgDeltaDatabaseDiffInput, type LegacyPgDeltaExplicitDiffInput, + type LegacyPgDeltaHazardReport, } from "../shared/legacy-pgdelta-engine.service.ts"; import type { LegacyDbDiffFlags } from "./diff.command.ts"; import { legacyDbDiff } from "./diff.handler.ts"; @@ -72,6 +73,7 @@ interface SetupOpts { readonly diffFiles?: ReadonlyArray<{ readonly name: string; readonly sql: string }>; // Exact suffixes returned by the next renderer, parallel to `diffFiles`. readonly diffSuffixes?: ReadonlyArray; + readonly hazards?: LegacyPgDeltaHazardReport; readonly pgDeltaImplementation?: "legacy" | "next"; readonly oom?: boolean; // edge-runtime OOMs; the bash fallback returns `diffSql` readonly delegateStdout?: string; // stdout returned by a captured Go-delegate run @@ -201,6 +203,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { changes: files.length > 0, sql: opts.diffFiles !== undefined ? files.map((file) => file.sql).join("\n\n") : sql, files, + ...(opts.hazards !== undefined ? { hazards: opts.hazards } : {}), }; }; const pgDeltaEngine = Layer.succeed( @@ -1926,6 +1929,25 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); + it.effect("warns on semantic data-loss hazards without a DROP statement", () => { + const sql = "ALTER TABLE public.accounts ALTER COLUMN email TYPE text;"; + const s = setup(tmp.current, { + pgDeltaImplementation: "next", + diffSql: sql, + hazards: { + actions: [{ actionIndex: 0, kinds: ["data_loss"] }], + dataLoss: [{ actionIndex: 0, sql }], + coverage: ["data_loss"], + kinds: ["data_loss"], + }, + }); + return Effect.gen(function* () { + yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); + expect(stderr(s.out)).toContain("Found destructive changes in schema diff"); + expect(stderr(s.out)).toContain(sql); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("emits a json envelope with --output-format json (payload-only stdout)", () => { const s = setup(tmp.current, { format: "json", diffSql: "create table j ();\n" }); return Effect.gen(function* () { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index a2e06f1f1a..6c180192cf 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -198,11 +198,22 @@ describe("legacyDiffDeclarativeToMigrations", () => { planDeclarativeSchema: (input) => { calls.push(input); return Effect.succeed({ - changes: false, - sql: "", + changes: true, + sql: "ALTER TABLE public.accounts ALTER COLUMN email TYPE text;", files: [], sourceRef: "migrations", targetRef: "declarative", + hazards: { + actions: [{ actionIndex: 0, kinds: ["data_loss"] }], + dataLoss: [ + { + actionIndex: 0, + sql: "ALTER TABLE public.accounts ALTER COLUMN email TYPE text;", + }, + ], + coverage: ["data_loss"], + kinds: ["data_loss"], + }, removals: { extensions: ["pgcrypto"], extensionIntents: [ @@ -229,6 +240,9 @@ describe("legacyDiffDeclarativeToMigrations", () => { expect(calls[0]?.noCache).toBe(true); expect(calls[0]?.strictCoverage).toBe(true); expect(result.manifestPresent).toBe(true); + expect(result.dropWarnings).toEqual([ + "ALTER TABLE public.accounts ALTER COLUMN email TYPE text;", + ]); expect(result.removals).toEqual({ extensions: ["pgcrypto"], extensionIntents: [{ extension: "pg_cron", intentKind: "job", key: "refresh metrics" }], diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 1a13e0dcb2..83161a80ae 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -102,7 +102,10 @@ export const legacyDiffDeclarativeToMigrations = Effect.fnUntraced(function* ( files: result.files, sourceRef: result.sourceRef, targetRef: result.targetRef, - dropWarnings: legacyFindDropStatements(result.sql), + dropWarnings: + engine.implementation === "next" && result.hazards !== undefined + ? result.hazards.dataLoss.map((action) => action.sql) + : legacyFindDropStatements(result.sql), manifestPresent: manifest !== undefined, removals: result.removals ?? { extensions: [], extensionIntents: [] }, } satisfies LegacyDeclarativeSyncResult; diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index be8948aa0a..377349a320 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -596,7 +596,7 @@ describe("legacy db schema declarative generate integration", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("--overwrite replaces only the absolute --output destination", () => { + it.effect("--overwrite preserves unmanaged files in the absolute --output destination", () => { const destination = mkdtempSync(join(tmpdir(), "legacy-decl-output-")); mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); writeFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "select 1;"); @@ -610,7 +610,7 @@ describe("legacy db schema declarative generate integration", () => { overwrite: true, }), ); - expect(existsSync(join(destination, "stale.sql"))).toBe(false); + expect(readFileSync(join(destination, "stale.sql"), "utf8")).toBe("select 'stale';"); expect(existsSync(join(destination, ".pgdelta-export.json"))).toBe(true); expect( readFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "utf8"), diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index a44f70850c..722d4f8de4 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -458,7 +458,11 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara // Step 6: drop warnings. if (result.dropWarnings.length > 0) { yield* output.raw( - `${legacyYellow("Found drop statements in schema diff. Please double check if these are expected:")}\n`, + `${legacyYellow( + engine.implementation === "next" + ? "Found destructive changes in schema diff. Please double check if these are expected:" + : "Found drop statements in schema diff. Please double check if these are expected:", + )}\n`, "stderr", ); yield* output.raw(`${legacyYellow(result.dropWarnings.join("\n"))}\n`, "stderr"); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 199d5579c7..b8bcda0ab7 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -1124,7 +1124,7 @@ describe("legacy db schema declarative sync integration", () => { yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); const output = stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join("")); expect(output).not.toContain("may have been generated by the legacy engine"); - expect(output).toContain("Found drop statements"); + expect(output).toContain("Found destructive changes"); }).pipe(Effect.provide(s.layer)); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 46755d6f92..c697a91629 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -31,6 +31,7 @@ import { export const legacyPgDeltaNextIsolatedShadowPlanOptions = { isolatedShadow: true, seedAssumedSchemas: false, + strictDataStatements: true, } as const; function legacyPgDeltaNextConnectSuggestion(cause: unknown): string | undefined { @@ -67,6 +68,7 @@ function normalizeNextDiff( readonly actionCount: number; }>; readonly removals?: LegacyPgDeltaDiffResult["removals"]; + readonly hazards: NonNullable; readonly debug?: { readonly sourceSnapshot?: string; readonly desiredSnapshot?: string; @@ -87,6 +89,7 @@ function normalizeNextDiff( actionCount: file.actionCount, })), ...(result.removals !== undefined ? { removals: result.removals } : {}), + hazards: result.hazards, ...(result.debug !== undefined ? { debug: { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts index 20fc56e16b..c16d24fc33 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts @@ -13,6 +13,7 @@ describe("legacyPgDeltaNextIsolatedShadowPlanOptions", () => { expect(legacyPgDeltaNextIsolatedShadowPlanOptions).toEqual({ isolatedShadow: true, seedAssumedSchemas: false, + strictDataStatements: true, }); }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts index bc57107464..9a0724dae4 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts @@ -68,6 +68,33 @@ export interface LegacyPgDeltaRemovalSummary { readonly extensionIntents: ReadonlyArray; } +export type LegacyPgDeltaHazardKind = + | "data_loss" + | "rewrite_risk" + | "non_transactional" + | "access_exclusive_lock" + | "unmodeled_kind" + | "unmodeled_drift" + | "unresolved_security_label"; + +interface LegacyPgDeltaActionHazard { + readonly actionIndex: number; + readonly kinds: ReadonlyArray; +} + +interface LegacyPgDeltaDataLossAction { + readonly actionIndex: number; + readonly sql: string; +} + +/** Semantic safety metadata derived from pg-delta's typed plan actions. */ +export interface LegacyPgDeltaHazardReport { + readonly actions: ReadonlyArray; + readonly dataLoss: ReadonlyArray; + readonly coverage: ReadonlyArray; + readonly kinds: ReadonlyArray; +} + interface LegacyPgDeltaDebugArtifacts { readonly sourceSnapshot?: string; readonly desiredSnapshot?: string; @@ -82,6 +109,7 @@ export interface LegacyPgDeltaDiffResult { readonly sql: string; readonly files: ReadonlyArray; readonly removals?: LegacyPgDeltaRemovalSummary; + readonly hazards?: LegacyPgDeltaHazardReport; readonly debug?: LegacyPgDeltaDebugArtifacts; } diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index b2b1bdb713..425a94a337 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -1,8 +1,13 @@ import { Effect, Layer } from "effect"; import type { Pool } from "pg"; -import { serializeSnapshot, encodeId } from "@supabase/pg-delta/core"; +import { + serializeSnapshot, + encodeId, + type Diagnostic as PgDeltaDiagnostic, +} from "@supabase/pg-delta/core"; import { buildSchemaExport, + dataLossActions, planSchemaFiles, renderPlanFiles, ShadowLoadError, @@ -12,7 +17,7 @@ import { resolveProfile, supabaseProfile, } from "@supabase/pg-delta/integrations"; -import { plan, serializePlan } from "@supabase/pg-delta/plan"; +import { classifyPlanHazards, plan, serializePlan } from "@supabase/pg-delta/plan"; import type { Plan as PgDeltaPlan } from "@supabase/pg-delta/plan"; import type { Policy } from "@supabase/pg-delta/policy"; import { formatSqlStatements, type SqlFormatOptions } from "@supabase/pg-delta/sql-format"; @@ -28,6 +33,7 @@ import { type LegacyPgDeltaNextDiagnosticOrigin, type LegacyPgDeltaNextDiffInput, type LegacyPgDeltaNextExportManifest, + type LegacyPgDeltaNextHazardReport, type LegacyPgDeltaNextRenderedFile, type LegacyPgDeltaNextSnapshotCaptureInput, type LegacyPgDeltaNextSqlFile, @@ -82,6 +88,7 @@ interface LegacyPgDeltaNextLibrarySchemaPlan { readonly plan: Plan; readonly loadDiagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[]; readonly targetDiagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[]; + readonly driftDiagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[]; readonly skipped: readonly { readonly file: string; readonly stmt: string }[]; } @@ -124,6 +131,10 @@ export interface LegacyPgDeltaNextLibraries string; readonly serializePlan: (plan: Plan) => string; readonly summarizeRemovals: (plan: Plan) => LegacyPgDeltaRemovalSummary; + readonly summarizeHazards: ( + plan: Plan, + diagnostics: readonly LegacyPgDeltaNextLibraryDiagnostic[], + ) => LegacyPgDeltaNextHazardReport; readonly encodeSubject: (subject: Subject) => string; } @@ -157,6 +168,22 @@ export function legacySummarizePgDeltaNextRemovals( }; } +export function legacySummarizePgDeltaNextHazards( + generatedPlan: Pick, + diagnostics: readonly PgDeltaDiagnostic[], +): LegacyPgDeltaNextHazardReport { + const classified = classifyPlanHazards(generatedPlan, diagnostics); + return { + actions: classified.actions.map((action) => ({ + actionIndex: action.actionIndex, + kinds: [...action.kinds], + })), + dataLoss: dataLossActions(generatedPlan.actions).map((action) => ({ ...action })), + coverage: [...classified.coverage], + kinds: [...classified.kinds], + }; +} + function legacyPgDeltaNextMessage(operation: LegacyPgDeltaNextOperation, cause: unknown): string { const detail = cause instanceof Error ? cause.message : String(cause); const diagnostics = @@ -460,6 +487,9 @@ function legacyPgDeltaNextPlanOptions(input: LegacyPgDeltaNextDeclarativePlanInp ...(input.strictFunctionBodies !== undefined ? { strictFunctionBodies: input.strictFunctionBodies } : {}), + ...(input.strictDataStatements !== undefined + ? { strictDataStatements: input.strictDataStatements } + : {}), reorder: input.reorder ?? true, ...(input.onWarning !== undefined ? { onWarning: input.onWarning } : {}), }; @@ -512,6 +542,10 @@ function legacyMakePgDeltaNextAdapter file.contents).join("\n\n"), files: legacyNormalizePgDeltaNextRenderedFiles(renderedFiles), diagnostics, + hazards: libraries.summarizeHazards(generatedPlan, [ + ...source.diagnostics, + ...desired.diagnostics, + ]), ...(input.debug ? { debug: { @@ -564,6 +598,11 @@ function legacyMakePgDeltaNextAdapter file.contents).join("\n\n"), @@ -579,7 +618,13 @@ function legacyMakePgDeltaNextAdapter ({ file: skipped.file, statement: skipped.stmt, @@ -673,6 +718,7 @@ const legacyPgDeltaNextRealLibraries = { serializeSnapshot, serializePlan, summarizeRemovals: legacySummarizePgDeltaNextRemovals, + summarizeHazards: legacySummarizePgDeltaNextHazards, encodeSubject: encodeId, }; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts index cc3abfd384..8a470e396a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts @@ -7,7 +7,11 @@ import { ErrorActionabilityId, } from "../../../../shared/telemetry/error-actionability.ts"; import type { LegacyMigrationTransactionMode } from "../../../shared/legacy-migration-file.ts"; -import type { LegacyPgDeltaRemovalSummary } from "./legacy-pgdelta-engine.service.ts"; +import type { + LegacyPgDeltaHazardKind, + LegacyPgDeltaHazardReport, + LegacyPgDeltaRemovalSummary, +} from "./legacy-pgdelta-engine.service.ts"; export type LegacyPgDeltaNextOperation = | "diff" @@ -21,6 +25,7 @@ export type LegacyPgDeltaNextDiagnosticOrigin = | "export" | "declarativeLoad" | "declarativeTarget" + | "declarativeDrift" | "snapshot"; export interface LegacyPgDeltaNextDiagnostic { @@ -40,6 +45,9 @@ export interface LegacyPgDeltaNextRenderedFile { readonly actionCount: number; } +export type LegacyPgDeltaNextHazardKind = LegacyPgDeltaHazardKind; +export type LegacyPgDeltaNextHazardReport = LegacyPgDeltaHazardReport; + export interface LegacyPgDeltaNextSqlFile { readonly name: string; readonly sql: string; @@ -69,6 +77,7 @@ interface LegacyPgDeltaNextDiffResult { readonly sql: string; readonly files: readonly LegacyPgDeltaNextRenderedFile[]; readonly diagnostics: readonly LegacyPgDeltaNextDiagnostic[]; + readonly hazards: LegacyPgDeltaNextHazardReport; readonly debug?: LegacyPgDeltaNextDebugArtifacts; } @@ -138,6 +147,8 @@ export interface LegacyPgDeltaNextDeclarativePlanInput { readonly seedAssumedSchemas?: boolean; readonly restrictToApplier?: boolean; readonly strictFunctionBodies?: boolean; + /** Reject data-changing SQL observed while loading declarative schema files. */ + readonly strictDataStatements?: boolean; readonly formatOptions?: string; /** Defaults to true, preserving pg-topo statement-level reorder support. */ readonly reorder?: boolean; @@ -155,6 +166,7 @@ interface LegacyPgDeltaNextDeclarativePlanResult { readonly sql: string; readonly files: readonly LegacyPgDeltaNextRenderedFile[]; readonly diagnostics: readonly LegacyPgDeltaNextDiagnostic[]; + readonly hazards: LegacyPgDeltaNextHazardReport; readonly skipped: readonly LegacyPgDeltaNextSkippedStatement[]; readonly removals: LegacyPgDeltaRemovalSummary; readonly debug?: LegacyPgDeltaNextDebugArtifacts; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index d45956ff66..08897c35c7 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -7,7 +7,7 @@ import { type StableId, } from "@supabase/pg-delta/core"; import { renderPlanFiles, ShadowLoadError } from "@supabase/pg-delta/frontends"; -import { plan } from "@supabase/pg-delta/plan"; +import { plan, type Action } from "@supabase/pg-delta/plan"; import { Effect } from "effect"; import { Pool } from "pg"; import { describe, expect } from "vitest"; @@ -18,6 +18,7 @@ import { legacyFilterPgDeltaNextPlatformParameterAclDiagnostics, legacyPgDeltaNextProfile, legacyPgDeltaNextUserOwnedParameterAcls, + legacySummarizePgDeltaNextHazards, legacySummarizePgDeltaNextRemovals, type LegacyPgDeltaNextLibraries, } from "./legacy-pgdelta-next-adapter.layer.ts"; @@ -155,6 +156,7 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { plan: { source: "target-facts", desired: "loaded-files" }, loadDiagnostics: [fakeDiagnostic("load-warning", "load")], targetDiagnostics: [fakeDiagnostic("target-warning", "target")], + driftDiagnostics: [fakeDiagnostic("unmodeled_drift", "drift")], skipped: [{ file: "roles.sql", stmt: "create role ignored" }], }; }, @@ -172,6 +174,16 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { { extension: "pg_cron", intentKind: "job", key: "refresh download metrics" }, ], }), + summarizeHazards: (_generatedPlan, diagnostics) => ({ + actions: [{ actionIndex: 0, kinds: ["data_loss"] }], + dataLoss: [{ actionIndex: 0, sql: "TRUNCATE TABLE public.audit_log" }], + coverage: diagnostics.some((diagnostic) => diagnostic.code === "unmodeled_drift") + ? ["unmodeled_drift"] + : [], + kinds: diagnostics.some((diagnostic) => diagnostic.code === "unmodeled_drift") + ? ["data_loss", "unmodeled_drift"] + : ["data_loss"], + }), encodeSubject: (subject) => `subject:${subject.id}`, }; @@ -233,6 +245,42 @@ describe("LegacyPgDeltaNextAdapter", () => { }); }); + it("derives semantic hazards and destructive non-DROP actions from the typed plan", () => { + const destructiveAlter: Action = { + sql: 'ALTER TABLE "public"."items" ALTER COLUMN "quantity" TYPE smallint;', + verb: "alter", + produces: [], + consumes: [], + destroys: [], + releases: [], + transactionality: "transactional", + lockClass: "accessExclusive", + newSegmentBefore: false, + dataLoss: "destructive", + rewriteRisk: true, + }; + + expect( + legacySummarizePgDeltaNextHazards({ actions: [destructiveAlter] }, [ + { + code: "unmodeled_drift", + severity: "warning", + message: "a desired prerequisite is absent from the target", + }, + ]), + ).toEqual({ + actions: [ + { + actionIndex: 0, + kinds: ["data_loss", "rewrite_risk", "access_exclusive_lock"], + }, + ], + dataLoss: [{ actionIndex: 0, sql: destructiveAlter.sql }], + coverage: ["unmodeled_drift"], + kinds: ["data_loss", "rewrite_risk", "access_exclusive_lock", "unmodeled_drift"], + }); + }); + it("filters platform parameter ACL coverage without hiding user-owned ACLs", () => { const diagnostics = [ { @@ -704,17 +752,31 @@ describe("LegacyPgDeltaNextAdapter", () => { debug: true, isolatedShadow: true, seedAssumedSchemas: true, + strictDataStatements: true, formatOptions: "null", }); expect(state.declarativeInputs).toHaveLength(1); expect(state.declarativeInputs[0]).toMatchObject({ reorder: true, seedAssumedSchemas: true, + strictDataStatements: true, }); expect(planned.diagnostics.map((diagnostic) => diagnostic.origin)).toEqual([ "declarativeLoad", "declarativeTarget", + "declarativeDrift", ]); + expect(planned.diagnostics.at(-1)).toMatchObject({ + origin: "declarativeDrift", + code: "unmodeled_drift", + subject: "subject:drift", + }); + expect(planned.hazards).toEqual({ + actions: [{ actionIndex: 0, kinds: ["data_loss"] }], + dataLoss: [{ actionIndex: 0, sql: "TRUNCATE TABLE public.audit_log" }], + coverage: ["unmodeled_drift"], + kinds: ["data_loss", "unmodeled_drift"], + }); expect(planned.skipped).toEqual([{ file: "roles.sql", statement: "create role ignored" }]); expect(planned.removals).toEqual({ extensions: ["pgcrypto"], @@ -784,11 +846,18 @@ describe("LegacyPgDeltaNextAdapter", () => { plan: { source: "unused", desired: "unused" }, loadDiagnostics: [], targetDiagnostics: [], + driftDiagnostics: [], skipped: [], }), serializeSnapshot: () => "unused", serializePlan: () => "unused", summarizeRemovals: () => ({ extensions: [], extensionIntents: [] }), + summarizeHazards: () => ({ + actions: [], + dataLoss: [], + coverage: [], + kinds: [], + }), encodeSubject: (subject: string) => subject, }); @@ -837,6 +906,12 @@ describe("LegacyPgDeltaNextAdapter", () => { serializeSnapshot: () => "unused", serializePlan: () => "unused", summarizeRemovals: () => ({ extensions: [], extensionIntents: [] }), + summarizeHazards: () => ({ + actions: [], + dataLoss: [], + coverage: [], + kinds: [], + }), encodeSubject: (subject: string) => subject, }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts index df89b0d79b..ce0bf8ba86 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts @@ -1,4 +1,5 @@ import { Effect } from "effect"; +import { hasBlockingDiagnostics, STRICT_COVERAGE_CODES } from "@supabase/pg-delta/frontends"; import { Output } from "../../../../shared/output/output.service.ts"; import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; @@ -8,8 +9,6 @@ import type { LegacyPgDeltaNextOperation, } from "./legacy-pgdelta-next-adapter.service.ts"; -const coverageDiagnosticCodes = new Set(["unmodeled_kind", "unresolved_security_label"]); - const operationConsequence: Record = { diff: "Changes to these objects are omitted from the generated database diff.", declarativeExport: "These objects are omitted from the exported declarative schema.", @@ -43,12 +42,20 @@ export function legacyPgDeltaNextDiagnosticReport( diagnostics: readonly LegacyPgDeltaNextDiagnostic[], strictCoverage: boolean, ): LegacyPgDeltaNextDiagnosticReport { - const coverage = diagnostics.filter((diagnostic) => coverageDiagnosticCodes.has(diagnostic.code)); - const blocking = diagnostics.filter( - (diagnostic) => - diagnostic.severity === "error" || - (strictCoverage && coverageDiagnosticCodes.has(diagnostic.code)), - ); + const coverage = diagnostics.filter((diagnostic) => STRICT_COVERAGE_CODES.has(diagnostic.code)); + const libraryDiagnostics = diagnostics.map((diagnostic) => ({ + code: diagnostic.code, + severity: diagnostic.severity, + message: diagnostic.message, + ...(diagnostic.context !== undefined ? { context: { ...diagnostic.context } } : {}), + })); + const blocking = hasBlockingDiagnostics(libraryDiagnostics, { strictCoverage }) + ? diagnostics.filter( + (diagnostic) => + diagnostic.severity === "error" || + (strictCoverage && STRICT_COVERAGE_CODES.has(diagnostic.code)), + ) + : []; const unmodeledKinds = [ ...new Set(diagnostics.map(diagnosticKind).filter((kind) => kind !== undefined)), ].sort((left, right) => left.localeCompare(right)); @@ -122,7 +129,7 @@ export const legacyReportPgDeltaNextDiagnostics = Effect.fnUntraced(function* ( const renderDetail = verboseDiagnostics || diagnostic.severity === "error" || - (strictCoverage && coverageDiagnosticCodes.has(diagnostic.code)); + (strictCoverage && STRICT_COVERAGE_CODES.has(diagnostic.code)); if (!renderDetail) { yield* debug.debug(message); continue; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts index 849b1a9452..fb51bf113e 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts @@ -117,6 +117,24 @@ describe("pg-delta next diagnostic coverage policy", () => { }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); + it("uses the upstream coverage policy to block unmodeled declarative drift", () => { + const report = legacyPgDeltaNextDiagnosticReport( + [ + { + origin: "declarativeDrift", + code: "unmodeled_drift", + severity: "warning", + message: "desired text search configuration is absent from the target", + context: { kind: "text search configuration" }, + }, + ], + true, + ); + + expect(report.coverage).toHaveLength(1); + expect(report.blocking).toEqual(report.coverage); + }); + it("can suppress a repeated feedback invitation without suppressing warnings", () => { const out = mockOutput(); const debugMessages: string[] = []; @@ -201,7 +219,7 @@ describe("pg-delta next diagnostic coverage policy", () => { }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); - it("classifies both coverage codes and aggregates arbitrary kinds safely", () => { + it("classifies all upstream coverage codes and aggregates arbitrary kinds safely", () => { const report = legacyPgDeltaNextDiagnosticReport( [ unmodeled("z future kind"), @@ -217,12 +235,18 @@ describe("pg-delta next diagnostic coverage policy", () => { message: "provider was not resolved", context: { kind: 42 }, }, + { + origin: "declarativeDrift", + code: "unmodeled_drift", + severity: "warning", + message: "desired object is absent from the target", + }, ], true, ); - expect(report.coverage).toHaveLength(7); - expect(report.blocking).toHaveLength(7); + expect(report.coverage).toHaveLength(8); + expect(report.blocking).toHaveLength(8); expect(report.unmodeledKinds).toEqual(["a future kind", "line break", "z future kind"]); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts index e184be77be..21798992d7 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts @@ -1,8 +1,11 @@ import { Effect, type FileSystem, type Path } from "effect"; +import { classifySqlFiles } from "@supabase/pg-delta/frontends"; import { legacyBold } from "../../../shared/legacy-colors.ts"; +import { legacyWalkSqlFiles } from "../../../shared/legacy-glob.ts"; import type { LegacyDeclarativeOutput } from "../../../shared/legacy-pgdelta.ts"; import { LegacyDeclarativeWriteError } from "./legacy-pgdelta.errors.ts"; +import { LegacyReadPgDeltaExportManifest } from "./legacy-pgdelta-files.ts"; import type { LegacyPgDeltaDeclarativeExportResult, LegacyPgDeltaExportManifest, @@ -11,6 +14,205 @@ import type { const EXPORT_MANIFEST_FILE = ".pgdelta-export.json"; type LegacyDeclarativeWriteOutput = LegacyDeclarativeOutput | LegacyPgDeltaDeclarativeExportResult; +type LegacyPgDeltaNextDeclarativeOutput = LegacyPgDeltaDeclarativeExportResult & { + readonly manifest: LegacyPgDeltaExportManifest; +}; + +function legacyDeclarativeWriteError(message: string): LegacyDeclarativeWriteError { + return new LegacyDeclarativeWriteError({ message }); +} + +function isNextDeclarativeOutput( + output: LegacyDeclarativeWriteOutput, +): output is LegacyPgDeltaNextDeclarativeOutput { + return "manifest" in output && output.manifest !== undefined; +} + +function safeDeclarativeExportName(path: Path.Path, name: string): string { + const rel = path.normalize(name.split("\\").join("/")); + if (rel.startsWith("..") || path.isAbsolute(rel)) { + throw legacyDeclarativeWriteError(`unsafe declarative export path: ${name}`); + } + return rel.split("\\").join("/"); +} + +function isCustomDeclarativePath(name: string): boolean { + return name.split("/")[0] === "_custom"; +} + +const readManagedDeclarativeSqlFiles = Effect.fnUntraced(function* ( + fs: FileSystem.FileSystem, + path: Path.Path, + declarativeDir: string, +) { + const names = yield* fs.readDirectory(declarativeDir); + const files: Array<{ readonly name: string; readonly sql: string }> = []; + for (const name of names) { + if (name === "_custom") continue; + const absolute = path.join(declarativeDir, name); + const isSymlink = yield* fs.readLink(absolute).pipe( + Effect.as(true), + Effect.orElseSucceed(() => false), + ); + if (isSymlink) continue; + const info = yield* fs.stat(absolute); + if (info.type === "Directory") { + const nested = yield* legacyWalkSqlFiles(fs, absolute, name); + for (const relative of nested) { + files.push({ + name: relative, + sql: yield* fs.readFileString(path.join(declarativeDir, relative)), + }); + } + } else if (info.type === "File" && name.endsWith(".sql")) { + files.push({ name, sql: yield* fs.readFileString(absolute) }); + } + } + return files; +}); + +const writeLegacyDeclarativeSchemas = Effect.fnUntraced(function* ( + fs: FileSystem.FileSystem, + path: Path.Path, + declarativeDir: string, + output: LegacyDeclarativeWriteOutput, +) { + yield* fs + .remove(declarativeDir, { recursive: true }) + .pipe( + Effect.catchTag("PlatformError", (error) => + error.reason._tag === "NotFound" + ? Effect.void + : Effect.fail( + legacyDeclarativeWriteError( + `failed to clean declarative schema directory: ${error.message}`, + ), + ), + ), + ); + yield* fs.makeDirectory(declarativeDir, { recursive: true }); + + for (const file of output.files) { + const name = "name" in file ? file.name : file.path; + const rel = yield* Effect.try({ + try: () => safeDeclarativeExportName(path, name), + catch: (error) => + error instanceof LegacyDeclarativeWriteError + ? error + : legacyDeclarativeWriteError(String(error)), + }); + const targetPath = path.join(declarativeDir, rel); + yield* fs.makeDirectory(path.dirname(targetPath), { recursive: true }); + yield* fs.writeFileString(targetPath, file.sql); + } +}); + +const writeNextDeclarativeSchemas = Effect.fnUntraced(function* ( + fs: FileSystem.FileSystem, + path: Path.Path, + declarativeDir: string, + output: LegacyPgDeltaNextDeclarativeOutput, +) { + const proposed = yield* Effect.forEach(output.files, (file) => + Effect.try({ + try: () => { + const name = safeDeclarativeExportName(path, file.name); + if (isCustomDeclarativePath(name)) { + throw legacyDeclarativeWriteError( + `refusing to write into reserved declarative schema path: ${file.name}`, + ); + } + return { name, sql: file.sql }; + }, + catch: (error) => + error instanceof LegacyDeclarativeWriteError + ? error + : legacyDeclarativeWriteError(String(error)), + }), + ); + + const exists = yield* fs + .exists(declarativeDir) + .pipe( + Effect.mapError((error) => + legacyDeclarativeWriteError( + `failed to inspect declarative schema directory: ${error.message}`, + ), + ), + ); + const existingFiles = exists + ? yield* readManagedDeclarativeSqlFiles(fs, path, declarativeDir).pipe( + Effect.mapError((error) => + legacyDeclarativeWriteError( + `failed to read managed declarative schema files: ${error.message}`, + ), + ), + ) + : []; + const previousManifest = exists + ? yield* LegacyReadPgDeltaExportManifest(fs, path, declarativeDir).pipe( + Effect.mapError((error) => legacyDeclarativeWriteError(error.message)), + ) + : undefined; + const classification = classifySqlFiles({ + proposed, + existing: new Map(existingFiles.map((file) => [file.name, file.sql])), + ...(previousManifest?.files !== undefined + ? { previouslyOwned: new Set(previousManifest.files) } + : {}), + }); + + yield* fs.makeDirectory(declarativeDir, { recursive: true }); + const changed = new Set([...classification.created, ...classification.updated]); + for (const file of proposed) { + if (!changed.has(file.name)) continue; + const targetPath = path.join(declarativeDir, file.name); + yield* fs.makeDirectory(path.dirname(targetPath), { recursive: true }); + yield* fs.writeFileString(targetPath, file.sql); + } + + for (const name of classification.removed) { + yield* fs + .remove(path.join(declarativeDir, name)) + .pipe( + Effect.mapError((error) => + legacyDeclarativeWriteError( + `failed to remove stale declarative schema file: ${error.message}`, + ), + ), + ); + } + + const manifest: LegacyPgDeltaExportManifest & { + readonly formatVersion: 1; + readonly files: ReadonlyArray; + } = { + formatVersion: 1, + ...output.manifest, + files: proposed.map((file) => file.name).sort(), + }; + const serialized = `${JSON.stringify(manifest, null, 2)}\n`; + const manifestPath = path.join(declarativeDir, EXPORT_MANIFEST_FILE); + const manifestExists = yield* fs + .exists(manifestPath) + .pipe( + Effect.mapError((error) => + legacyDeclarativeWriteError(`failed to inspect export manifest: ${error.message}`), + ), + ); + const previousSerialized = manifestExists + ? yield* fs + .readFileString(manifestPath) + .pipe( + Effect.mapError((error) => + legacyDeclarativeWriteError(`failed to read export manifest: ${error.message}`), + ), + ) + : undefined; + if (previousSerialized !== serialized) { + yield* fs.writeFileString(manifestPath, serialized); + } +}); /** * Go's `declarative.Generate` / `pull.go`'s written-to line, printed by all three @@ -24,8 +226,11 @@ export const legacyDeclarativeSchemaWrittenLine = (dir: string): string => /** * Materializes pg-delta declarative export output under the declarative dir. - * Mirrors Go's `WriteDeclarativeSchemas` (`declarative.go:239`): wipe the dir, - * recreate it, and write each file at its (path-safe) relative path. + * Legacy-engine output keeps Go's wipe-and-rewrite behavior. Next-engine output + * uses pg-delta's manifest ownership and file classification: only stale files + * owned by the previous export are removed, unchanged files are not rewritten, + * unmanaged files are preserved, and the reserved root `_custom/` tree is never + * read as managed output or deleted. * * Go also updates `[db.migrations] schema_paths` afterwards, but only when * pg-delta is *disabled* in config (`if utils.IsPgDeltaEnabled() { return nil }`). @@ -41,52 +246,9 @@ export const legacyWriteDeclarativeSchemas = Effect.fnUntraced(function* ( declarativeDir: string, output: LegacyDeclarativeWriteOutput, ) { - yield* fs.remove(declarativeDir, { recursive: true }).pipe( - Effect.catchTag("PlatformError", (error) => - // Go wraps any failure; a missing dir is fine (we recreate it next). - error.reason._tag === "NotFound" - ? Effect.void - : Effect.fail( - new LegacyDeclarativeWriteError({ - message: `failed to clean declarative schema directory: ${error.message}`, - }), - ), - ), - ); - yield* fs.makeDirectory(declarativeDir, { recursive: true }); - - const writtenFiles: Array = []; - for (const file of output.files) { - const name = "name" in file ? file.name : file.path; - const rel = path.normalize(name); - if (rel.startsWith("..") || path.isAbsolute(rel)) { - return yield* Effect.fail( - new LegacyDeclarativeWriteError({ - message: `unsafe declarative export path: ${name}`, - }), - ); - } - const targetPath = path.join(declarativeDir, rel); - yield* fs.makeDirectory(path.dirname(targetPath), { recursive: true }); - yield* fs.writeFileString(targetPath, file.sql); - writtenFiles.push(name.split("\\").join("/")); - } - - const manifest = "manifest" in output ? output.manifest : undefined; - if (manifest !== undefined) { - const serialized: LegacyPgDeltaExportManifest & { - readonly formatVersion: 1; - readonly files: ReadonlyArray; - } = { - formatVersion: 1, - ...manifest, - files: [...writtenFiles].sort(), - }; - yield* fs.writeFileString( - path.join(declarativeDir, EXPORT_MANIFEST_FILE), - `${JSON.stringify(serialized, null, 2)}\n`, - ); - } + return yield* isNextDeclarativeOutput(output) + ? writeNextDeclarativeSchemas(fs, path, declarativeDir, output) + : writeLegacyDeclarativeSchemas(fs, path, declarativeDir, output); }); // Go's `schemaPathsPattern` (`internal/db/declarative/declarative.go:59`): diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts index 93c9ff5ec5..04edde344b 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts @@ -1,4 +1,12 @@ -import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + statSync, + utimesSync, + writeFileSync, +} from "node:fs"; import { existsSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -77,6 +85,96 @@ describe("legacyWriteDeclarativeSchemas", () => { ); }); + it.effect("preserves custom and unmanaged files while pruning stale owned files", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); + const declDir = join(dir, "supabase", "database"); + mkdirSync(join(declDir, "_custom"), { recursive: true }); + writeFileSync(join(declDir, "_custom", "casts.sql"), "create cast (int as text);"); + writeFileSync(join(declDir, "unmanaged.sql"), "select 'keep me';"); + writeFileSync(join(declDir, "stale.sql"), "select 'remove me';"); + writeFileSync( + join(declDir, ".pgdelta-export.json"), + `${JSON.stringify({ + formatVersion: 1, + redactSecrets: true, + scope: "database", + files: ["stale.sql"], + })}\n`, + ); + + return write(declDir, { + files: [{ name: "schemas/public.sql", sql: "create table public.example(id int);" }], + manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, + }).pipe( + Effect.tap(() => + Effect.sync(() => { + expect(existsSync(join(declDir, "stale.sql"))).toBe(false); + expect(readFileSync(join(declDir, "unmanaged.sql"), "utf8")).toBe("select 'keep me';"); + expect(readFileSync(join(declDir, "_custom", "casts.sql"), "utf8")).toBe( + "create cast (int as text);", + ); + expect( + JSON.parse(readFileSync(join(declDir, ".pgdelta-export.json"), "utf8")).files, + ).toEqual(["schemas/public.sql"]); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("does not rewrite unchanged next-engine files or manifests", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); + const declDir = join(dir, "supabase", "database"); + const schemaPath = join(declDir, "schemas", "public.sql"); + const manifestPath = join(declDir, ".pgdelta-export.json"); + const output: LegacyPgDeltaDeclarativeExportResult = { + files: [{ name: "schemas/public.sql", sql: "create table public.example(id int);" }], + manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, + }; + + return write(declDir, output).pipe( + Effect.flatMap(() => + Effect.sync(() => { + const old = new Date("2020-01-01T00:00:00.000Z"); + utimesSync(schemaPath, old, old); + utimesSync(manifestPath, old, old); + }), + ), + Effect.flatMap(() => write(declDir, output)), + Effect.tap(() => + Effect.sync(() => { + expect(statSync(schemaPath).mtime.toISOString()).toBe("2020-01-01T00:00:00.000Z"); + expect(statSync(manifestPath).mtime.toISOString()).toBe("2020-01-01T00:00:00.000Z"); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("rejects next-engine output targeting the reserved custom directory", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); + const declDir = join(dir, "supabase", "database"); + return write(declDir, { + files: [{ name: "_custom/generated.sql", sql: "select 1;" }], + manifest: { redactSecrets: true, scope: "database" }, + }).pipe( + Effect.exit, + Effect.tap((exit) => + Effect.sync(() => { + expect(Exit.isFailure(exit)).toBe(true); + if (Exit.isFailure(exit)) { + const error = exit.cause.reasons.find(Cause.isFailReason)?.error; + expect(error).toBeInstanceOf(LegacyDeclarativeWriteError); + expect((error as LegacyDeclarativeWriteError).message).toBe( + "refusing to write into reserved declarative schema path: _custom/generated.sql", + ); + } + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + it.effect("creates the declarative dir when absent", () => { const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); const declDir = join(dir, "supabase", "database"); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b45fd20675..229fe866df 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -155,8 +155,8 @@ importers: specifier: workspace:* version: link:../../packages/config '@supabase/pg-delta': - specifier: 1.0.0-alpha.34 - version: 1.0.0-alpha.34(@supabase/pg-topo@1.0.0-alpha.5) + specifier: 1.0.0-alpha.40 + version: 1.0.0-alpha.40(@supabase/pg-topo@1.0.0-alpha.5) '@supabase/pg-topo': specifier: 1.0.0-alpha.5 version: 1.0.0-alpha.5 @@ -2842,8 +2842,8 @@ packages: resolution: {integrity: sha512-RW/OCsd6MO592zU8ifzP8/f8XzxxIdpb+Up5XaOtE26Fw+3zTp475WX7+GuuktiD1WF8pFUDe6khUPbMp77RCw==} engines: {node: '>=22.0.0'} - '@supabase/pg-delta@1.0.0-alpha.34': - resolution: {integrity: sha512-xjNBdFl4/DXIxZufUK6t52wTywMS4sl1rcXvxTgh3mv1Q50tDeMPkjp8QM5YRjYo/mrdxZJ94kmbf6XAQn0jRg==} + '@supabase/pg-delta@1.0.0-alpha.40': + resolution: {integrity: sha512-PL1h0zdg5WQP1pw38P0d/R/k6nB0G1MFFTGENXiEn14fZv3tX3IktS3eDfPx1R7C/eOVzvxlFoR9OGmWGgRJsw==} engines: {node: '>=20.0.0'} hasBin: true peerDependencies: @@ -9094,7 +9094,7 @@ snapshots: dependencies: tslib: 2.8.1 - '@supabase/pg-delta@1.0.0-alpha.34(@supabase/pg-topo@1.0.0-alpha.5)': + '@supabase/pg-delta@1.0.0-alpha.40(@supabase/pg-topo@1.0.0-alpha.5)': dependencies: debug: 4.4.3(supports-color@7.2.0) pg: 8.22.0 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 1a70d9f196..4891efb339 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -49,7 +49,7 @@ minimumReleaseAgeExclude: - "@effect/platform-node-shared@4.0.0-beta.103" - "@effect/sql-pg@4.0.0-beta.103" - "@effect/vitest@4.0.0-beta.103" - - "@supabase/pg-delta@1.0.0-alpha.34" + - "@supabase/pg-delta@1.0.0-alpha.40" - "@supabase/pg-topo@1.0.0-alpha.5" - "effect@4.0.0-beta.103" From a621690aff53134a80ce7f337c0dc87d96bbc84f Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 14 Aug 2026 17:29:58 +0200 Subject: [PATCH 27/82] fix(cli): address pg-delta engine review findings - read the declarative export manifest only under the next engine, so a stale or malformed .pgdelta-export.json can no longer break the SUPABASE_USE_PG_DELTA_NEXT=false escape hatch - surface planSchemaFiles skipped statements as coverage diagnostics: warned by default, blocking under --strict-coverage - warn when a manifest-less declarative directory keeps unmanaged files the next writer cannot prune, advising a clean regenerate - converge pg_net in both directions: PG14 db reset now runs the same drop-before-conditional-create as fresh setup, and start-on-existing- volume drops pg_net when webhooks are disabled unless an applied migration installed it - gate the schema_paths transition warning on the resolved next engine; migra and the legacy opt-out still substitute declared-schema targets - warn once when --strict-coverage is ignored by the legacy engine - redact unparseable connection strings with the shared redactor instead of a naive :password@ regex - record deferred symlink/port-allocation findings in docs/roadmap/pg-delta-next-follow-ups.md Co-Authored-By: Claude Fable 5 --- .../legacy/commands/db/diff/diff.handler.ts | 17 ++- .../commands/db/diff/diff.integration.test.ts | 44 ++++++ .../legacy/commands/db/pull/pull.handler.ts | 26 +++- .../commands/db/pull/pull.integration.test.ts | 44 +++++- .../db/reset/reset.integration.test.ts | 30 ++++ ...eclarative.orchestrate.integration.test.ts | 141 ++++++++++++++++++ .../declarative/declarative.orchestrate.ts | 17 ++- .../declarative/generate/generate.handler.ts | 7 +- .../generate/generate.integration.test.ts | 14 ++ .../schema/declarative/sync/sync.handler.ts | 6 +- .../legacy-pgdelta-engine.legacy.layer.ts | 30 ++++ .../legacy-pgdelta-engine.next.layer.ts | 13 +- ...acy-pgdelta-engine.next.layer.unit.test.ts | 18 ++- .../legacy-pgdelta-next-adapter.layer.ts | 24 +++ .../legacy-pgdelta-next-adapter.unit.test.ts | 15 +- .../shared/legacy-pgdelta-next-diagnostics.ts | 76 ++++++++-- ...gacy-pgdelta-next-diagnostics.unit.test.ts | 71 +++++++++ .../db/shared/legacy-pgdelta.write.ts | 76 +++++++++- .../shared/legacy-pgdelta.write.unit.test.ts | 106 +++++++++++++ .../db/start/start.integration.test.ts | 54 ++++++- .../legacy/shared/db-bootstrap/db-setup.ts | 71 +++++++-- .../shared/db-bootstrap/db-setup.unit.test.ts | 116 +++++++++++++- .../db-bootstrap/recreate-local-database.ts | 9 ++ .../legacy/shared/legacy-pg-net-guidance.ts | 18 +++ .../legacy-pg-net-guidance.unit.test.ts | 56 +++++++ docs/roadmap/pg-delta-next-follow-ups.md | 29 ++++ 26 files changed, 1078 insertions(+), 50 deletions(-) create mode 100644 apps/cli/src/legacy/shared/legacy-pg-net-guidance.unit.test.ts create mode 100644 docs/roadmap/pg-delta-next-follow-ups.md diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index 7ad945bd28..19526fae1d 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -541,9 +541,6 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy projectEnv: cfg.projectEnv, }; const formatOptions = Option.getOrElse(cfg.pgDelta.formatOptions, () => ""); - if (cfg.schemaPaths !== undefined && cfg.schemaPaths.length > 0) { - yield* output.raw(legacySchemaPathsTransitionWarning, "stderr"); - } // Engine resolution: the pg-delta env/config/flag gate, read from the // (possibly remote-merged) config. @@ -558,6 +555,17 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy usePgSchema, pgDeltaDefault, }); + // The bundled next engine is the ONLY mode whose baseline is local migrations + // alone. Every other mode — migra, pgAdmin, and the `SUPABASE_USE_PG_DELTA_NEXT= + // false` legacy pg-delta opt-out — still substitutes the declared-schema + // `contrib_regression` target for a local database (`legacy-shadow-source.ts`'s + // `migrationMode !== "pgdelta-next"` branch), so under those engines + // `schema_paths` genuinely does still shape the output and the transition warning + // would be factually wrong. Gate it on the resolved engine, not on the setting. + const usesPgDeltaNext = useDelta && pgDelta.implementation === "next"; + if (usesPgDeltaNext && cfg.schemaPaths !== undefined && cfg.schemaPaths.length > 0) { + yield* output.raw(legacySchemaPathsTransitionWarning, "stderr"); + } // pgAdmin's own text-mode status lines go to STDOUT, not stderr — unlike the migra/ // pg-delta path's diagnostics below, which always go to stderr. In machine output @@ -670,8 +678,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy diffResult = { sql, files: undefined }; } else { yield* output.raw("Creating shadow database...\n", "stderr"); - const migrationMode: "legacy" | "pgdelta-next" = - useDelta && pgDelta.implementation === "next" ? "pgdelta-next" : "legacy"; + const migrationMode: "legacy" | "pgdelta-next" = usesPgDeltaNext ? "pgdelta-next" : "legacy"; const shadowInput = { ...(yield* resolveShadowRunInput()), targetLocal: resolved.isLocal, diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index ac4afe3443..4a77cfe9d2 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -631,6 +631,50 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); + // The transition warning is only true for the bundled next engine. Every other + // engine still routes a local target with declarative files through the + // declared-schema `contrib_regression` override, so schema_paths DOES still shape + // their output and claiming otherwise would be a lie. + const writeSchemaPathsConfig = (pgDeltaEnabled: boolean) => { + mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + [ + "[db.migrations]", + 'schema_paths = ["configured.sql"]', + "", + "[experimental.pgdelta]", + `enabled = ${pgDeltaEnabled}`, + "", + ].join("\n"), + ); + writeFileSync(join(tmp.current, "supabase", "configured.sql"), "create table configured ();\n"); + }; + + it.effect("legacy pg-delta local diff does not print the schema_paths transition warning", () => { + writeSchemaPathsConfig(true); + const s = setup(tmp.current, { + pgDeltaImplementation: "legacy", + diffSql: "create table result ();\n", + }); + return Effect.gen(function* () { + yield* legacyDbDiff(flags({ usePgDelta: Option.some(true) })); + expect(stderr(s.out)).not.toContain("schema_paths no longer changes the migrations baseline"); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("migra local diff does not print the schema_paths transition warning", () => { + writeSchemaPathsConfig(false); + const s = setup(tmp.current, { + pgDeltaImplementation: "next", + diffSql: "create table result ();\n", + }); + return Effect.gen(function* () { + yield* legacyDbDiff(flags()); + expect(stderr(s.out)).not.toContain("schema_paths no longer changes the migrations baseline"); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("PG14: provisions a shadow via the SQL-exec init path (no PG15+ one-shot jobs)", () => { // This covers the PG14 branch of the `legacySetupDatabase` pipeline, which execs // SQL directly via the session instead of the three one-shot `LegacyDockerRun` diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 9d7b4ee859..5ef424f682 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -50,6 +50,7 @@ import { LegacyLinkedProjectCache } from "../../../telemetry/legacy-linked-proje import { LegacyTelemetryState } from "../../../telemetry/legacy-telemetry-state.service.ts"; import { legacyUpdateDeclarativeSchemaPathsConfig, + legacyWarnPreservedUnmanagedDeclarativeFiles, legacyWriteDeclarativeSchemas, } from "../shared/legacy-pgdelta.write.ts"; import { @@ -460,6 +461,8 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy envEnabled: legacyParseBoolEnv(toml.envLookup("SUPABASE_EXPERIMENTAL_PG_DELTA")), }), }); + /** Whether the migration-style diff runs on the bundled in-process next engine. */ + const usesPgDeltaNext = usePgDeltaDiff && pgDeltaEngine.implementation === "next"; // Runs the Go-delegated `--experimental` structured dump (still delegated, see // `EXPERIMENTAL_STRUCTURED_DUMP_DEPRECATION_LINE` above for why). In machine-output @@ -591,9 +594,17 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy (handle) => legacyRemoveShadowDatabase(spawner, handle.containerId), ); }); - yield* legacyWriteDeclarativeSchemas(fs, path, declarativeDir, exported).pipe( + const written = yield* legacyWriteDeclarativeSchemas( + fs, + path, + declarativeDir, + exported, + ).pipe( Effect.mapError((cause) => new LegacyDbPullWriteError({ message: cause.message })), ); + // Same manifest-less-merge caveat as generate/sync: the next writer only + // prunes manifest-owned files, so name what survived. + yield* legacyWarnPreservedUnmanagedDeclarativeFiles(declarativeDirRel, written); // Go's WriteDeclarativeSchemas also points [db.migrations] schema_paths at // the declarative dir, but only when pg-delta is *disabled* in config // (declarative.go:260-268, gated on IsPgDeltaEnabled which reads the config @@ -633,8 +644,16 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy return; } + // Gated on the resolved engine, not merely on `schema_paths` being set: the + // bundled next engine is the only mode whose baseline is local migrations + // alone. Under migra or the `SUPABASE_USE_PG_DELTA_NEXT=false` legacy pg-delta + // opt-out, `legacy-shadow-source.ts` still substitutes the declared-schema + // `contrib_regression` target for a local database (its `migrationMode !== + // "pgdelta-next"` branch), so `schema_paths` does still shape the output there + // and this warning would be factually wrong. if ( !delegatesExperimentalPull && + usesPgDeltaNext && toml.schemaPaths !== undefined && toml.schemaPaths.length > 0 ) { @@ -828,8 +847,9 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // target with declarative schema files gets a second `contrib_regression` shadow // returned as the target override. Pg-delta next compares the migrations shadow // directly to the live target instead. - const migrationMode: "legacy" | "pgdelta-next" = - usePgDeltaDiff && pgDeltaEngine.implementation === "next" ? "pgdelta-next" : "legacy"; + const migrationMode: "legacy" | "pgdelta-next" = usesPgDeltaNext + ? "pgdelta-next" + : "legacy"; const shadowInput = { ...legacyShadowRunInputFromLocalContainerInputs( pullLocalInputs, diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 4a221ffb78..ef2b874a6e 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -771,7 +771,47 @@ describe("legacy db pull", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("pulls with migra and warns that schema_paths cannot replace the target", () => { + // The transition warning belongs to the bundled next engine only: migra (and the + // legacy pg-delta opt-out) still substitute the declared-schema + // `contrib_regression` target for a local database, so schema_paths does still + // shape their output and the warning would be factually wrong. + it.effect("pulls with the next engine and warns that schema_paths no longer applies", () => { + seedMigration(tmp.current, "20240101000000"); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + [ + "[db.migrations]", + 'schema_paths = ["database/*.sql"]', + "", + "[experimental.pgdelta]", + "enabled = true", + "", + ].join("\n"), + ); + const s = setup(tmp.current, { + remoteVersions: ["20240101000000"], + engineImplementation: "next", + // The next engine's mock parses `edgeStdout` as a rendered-file envelope. + edgeStdout: JSON.stringify({ + files: [ + { + name: "schema_changes", + transactionMode: "transactional", + sql: "create table remote ();\n", + }, + ], + }), + yes: true, + }); + return Effect.gen(function* () { + yield* legacyDbPull(flags()); + expect(streamText(s.out, "stderr")).toContain( + "schema_paths no longer changes the migrations baseline", + ); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("pulls with migra and does not warn about schema_paths", () => { seedMigration(tmp.current, "20240101000000"); writeFileSync( join(tmp.current, "supabase", "config.toml"), @@ -788,7 +828,7 @@ describe("legacy db pull", () => { // (a pg-delta selection would instead try — and fail — to `JSON.parse` it). expect(s.shadowSpawned.filter((call) => call.args[0] === "create")).toHaveLength(1); const err = streamText(s.out, "stderr"); - expect(err).toContain("schema_paths no longer changes the migrations baseline"); + expect(err).not.toContain("schema_paths no longer changes the migrations baseline"); // Go's `ConnectByConfig` prints the Connecting line to stderr before dialing // (`internal/utils/connect.go:348`), ahead of any other pull output. expect(err).toContain("Connecting to remote database...\n"); diff --git a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts index 94c6924346..e78de1e604 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts @@ -1157,6 +1157,36 @@ describe("legacy db reset", () => { const migrationIndex = conn.execs.findIndex((sql) => sql.includes("https://example.com")); expect(pgNetIndex).toBeGreaterThanOrEqual(0); expect(migrationIndex).toBeGreaterThan(pgNetIndex); + // Same drop-then-recreate order as fresh setup: the PG14 dump installs pg_net + // unconditionally, so it is dropped first and only recreated because webhooks + // are enabled. + const dropIndex = conn.execs.findIndex((sql) => + sql.includes("drop extension if exists pg_net"), + ); + expect(dropIndex).toBeGreaterThanOrEqual(0); + expect(pgNetIndex).toBeGreaterThan(dropIndex); + }); + }); + + it.live("drops the PG14 dump's implicit pg_net when Database Webhooks is disabled", () => { + // Fresh setup already removed it here; without the same drop on the reset path a + // PG14 `db reset` left pg_net installed and diverged from `supabase start`, + // surfacing as pg_net drift in the next engine's shadow baseline. + const { layer, conn } = setup(tmp.current, { + toml: PG14_TOML, + args: ["db", "reset", "--local"], + isLocal: true, + }); + return Effect.gen(function* () { + yield* legacyDbReset(DEFAULT_FLAGS).pipe(Effect.provide(layer)); + expect(conn.execs.some((sql) => sql.includes("drop extension if exists pg_net"))).toBe( + true, + ); + expect( + conn.execs.some((sql) => + sql.includes("create extension if not exists pg_net schema extensions"), + ), + ).toBe(false); }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 6c180192cf..835858eeaf 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -253,6 +253,84 @@ describe("legacyDiffDeclarativeToMigrations", () => { Effect.provide(Layer.mergeAll(engine, BunServices.layer)), ); }); + + // The legacy engine's `planDeclarativeSchema` never looks at `input.manifest`, so + // validating the manifest for it turned a stale/hand-edited `.pgdelta-export.json` + // into a hard failure of the documented `SUPABASE_USE_PG_DELTA_NEXT=false` escape + // hatch. The next engine, which does consume it, must still reject it. + const stubEngine = ( + implementation: "legacy" | "next", + calls: LegacyPgDeltaDeclarativePlanInput[], + ) => + Layer.succeed( + LegacyPgDeltaEngine, + LegacyPgDeltaEngine.of({ + implementation, + diffExplicit: () => Effect.die("diffExplicit not used"), + diffDatabase: () => Effect.die("diffDatabase not used"), + exportDeclarativeSchema: () => Effect.die("exportDeclarativeSchema not used"), + planDeclarativeSchema: (input) => { + calls.push(input); + return Effect.succeed({ + changes: true, + sql: "create table public.accounts();", + files: [], + sourceRef: "migrations", + targetRef: "declarative", + }); + }, + }), + ); + + const withCorruptManifest = () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-orch-")); + const declDir = join(dir, "supabase", "database"); + mkdirSync(declDir, { recursive: true }); + writeFileSync(join(declDir, "public.sql"), "create table public.accounts();"); + writeFileSync(join(declDir, ".pgdelta-export.json"), "{ not json at all"); + return { dir, declDir }; + }; + + it.effect("ignores a corrupt export manifest under the legacy engine opt-out", () => { + const { dir, declDir } = withCorruptManifest(); + const calls: LegacyPgDeltaDeclarativePlanInput[] = []; + return legacyDiffDeclarativeToMigrations(ctx(dir, declDir), toml, setupInputs).pipe( + Effect.tap((result) => + Effect.sync(() => { + expect(calls[0]?.files).toEqual([ + { name: "public.sql", sql: "create table public.accounts();" }, + ]); + expect(calls[0]?.manifest).toBeUndefined(); + expect(result.manifestPresent).toBe(false); + expect(result.diffSQL).toBe("create table public.accounts();"); + rmSync(dir, { recursive: true, force: true }); + }), + ), + Effect.provide(Layer.mergeAll(stubEngine("legacy", calls), BunServices.layer)), + ); + }); + + it.effect("still rejects a corrupt export manifest under the next engine", () => { + const { dir, declDir } = withCorruptManifest(); + const calls: LegacyPgDeltaDeclarativePlanInput[] = []; + return legacyDiffDeclarativeToMigrations(ctx(dir, declDir), toml, setupInputs).pipe( + Effect.exit, + Effect.tap((exit) => + Effect.sync(() => { + expect(Exit.isFailure(exit)).toBe(true); + if (Exit.isFailure(exit)) { + const error = exit.cause.reasons.find(Cause.isFailReason)?.error; + expect(String((error as { message?: string } | undefined)?.message)).toContain( + "malformed export manifest", + ); + } + expect(calls).toEqual([]); + rmSync(dir, { recursive: true, force: true }); + }), + ), + Effect.provide(Layer.mergeAll(stubEngine("next", calls), BunServices.layer)), + ); + }); }); // A minimal, valid `LegacySetupInputs` — the exact field values don't matter to @@ -364,6 +442,69 @@ describe("legacyDiffDeclarativeToMigrations", () => { }, ); + // `--strict-coverage` is enforced entirely by the next engine's diagnostic report; + // the legacy engine has no coverage diagnostics, so the flag silently did nothing + // under `SUPABASE_USE_PG_DELTA_NEXT=false`. It must say so instead. + const runWithLegacyEngine = (strictCoverage: boolean) => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-orch-")); + const declDir = join(dir, "supabase", "database"); + mkdirSync(declDir, { recursive: true }); + const seam = mockSeam({ + declarative: "supabase/.temp/pgdelta/decl.json", + baseline: "supabase/.temp/pgdelta/base.json", + }); + const edge = mockEdge("ALTER TABLE x ADD COLUMN y int;\n"); + const out = mockOutput(); + const shadow = mockShadowInfra(); + return { + dir, + out, + effect: legacyDiffDeclarativeToMigrations( + { ...ctx(dir, declDir), strictCoverage }, + toml, + setupInputs, + ).pipe( + Effect.provide( + Layer.mergeAll( + seam.layer, + edge.layer, + probe, + out.layer, + engineLayer(seam.layer, edge.layer, out.layer, shadow.layer), + BunServices.layer, + shadow.layer, + ), + ), + ), + }; + }; + + it.effect("warns that --strict-coverage does nothing on the legacy engine", () => { + const { dir, out, effect } = runWithLegacyEngine(true); + return effect.pipe( + Effect.tap(() => + Effect.sync(() => { + expect(out.stderrText).toContain( + '"--strict-coverage" has no effect with the legacy pg-delta engine.', + ); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("stays silent about --strict-coverage when the flag is unset", () => { + const { dir, out, effect } = runWithLegacyEngine(false); + return effect.pipe( + Effect.tap(() => + Effect.sync(() => { + expect(out.stderrText).not.toContain("--strict-coverage"); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + it.effect( "reuses an already-warmed platform-baseline catalog without provisioning a shadow", () => { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 83161a80ae..3fc73b33a7 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -81,9 +81,20 @@ export const legacyDiffDeclarativeToMigrations = Effect.fnUntraced(function* ( const files = yield* LegacyLoadPgDeltaSqlFiles(fs, path, run.declarativeDir).pipe( Effect.mapError((error) => declarativeError(error.message)), ); - const manifest = yield* LegacyReadPgDeltaExportManifest(fs, path, run.declarativeDir).pipe( - Effect.mapError((error) => declarativeError(error.message)), - ); + // Only the next engine consumes the export manifest (its planner reads ownership + // metadata from it); the legacy engine's `planDeclarativeSchema` ignores + // `input.manifest` entirely. Reading it unconditionally made the strict manifest + // validation (`LegacyReadPgDeltaExportManifest` fails on malformed JSON or missing + // policy metadata) fail a legacy-engine sync over a file the legacy planner never + // looks at, defeating the `SUPABASE_USE_PG_DELTA_NEXT=false` escape hatch. Under + // the legacy engine the manifest is treated as absent, exactly as if the file did + // not exist. + const manifest = + engine.implementation === "next" + ? yield* LegacyReadPgDeltaExportManifest(fs, path, run.declarativeDir).pipe( + Effect.mapError((error) => declarativeError(error.message)), + ) + : undefined; const result = yield* engine.planDeclarativeSchema({ context: run.pgDelta, schema: run.schema, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts index 7d00bfc136..240e74c1f4 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts @@ -40,6 +40,7 @@ import { } from "../declarative.orchestrate.ts"; import { legacyDeclarativeSchemaWrittenLine, + legacyWarnPreservedUnmanagedDeclarativeFiles, legacyWriteDeclarativeSchemas, } from "../../../shared/legacy-pgdelta.write.ts"; import type { LegacyDbSchemaDeclarativeGenerateFlags } from "./generate.command.ts"; @@ -291,7 +292,11 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec } } - yield* legacyWriteDeclarativeSchemas(fs, path, declarativeDir, result); + const written = yield* legacyWriteDeclarativeSchemas(fs, path, declarativeDir, result); + // The overwrite prompts above promise existing files may be deleted, but the + // next writer only prunes what an export manifest claimed — say so when a + // manifest-less directory kept files the export did not replace. + yield* legacyWarnPreservedUnmanagedDeclarativeFiles(declarativeDirRel, written); // Warm the declarative catalog cache after writing the files and before the // success message, gated on `!--no-cache` — Go's `Generate` diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index 377349a320..9001687189 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -612,6 +612,20 @@ describe("legacy db schema declarative generate integration", () => { ); expect(readFileSync(join(destination, "stale.sql"), "utf8")).toBe("select 'stale';"); expect(existsSync(join(destination, ".pgdelta-export.json"))).toBe(true); + // The destination had no manifest, so nothing could be classified as stale and + // `stale.sql` silently survived an "overwrite" the user confirmed. Say so, and + // point at the only instruction that produces a clean tree. + const stderrText = stripAnsi( + s.out.rawChunks + .filter((chunk) => chunk.stream === "stderr") + .map((chunk) => chunk.text) + .join(""), + ); + expect(stderrText).toContain( + "1 existing declarative schema file(s) in " + destination + " are not tracked", + ); + expect(stderrText).toContain("stale.sql"); + expect(stderrText).toContain(`remove ${destination} and re-run`); expect( readFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "utf8"), ).toBe("select 1;"); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 722d4f8de4..3469463a96 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -69,6 +69,7 @@ import { import { LegacyDeclarativeSeam } from "../../../shared/legacy-pgdelta.seam.service.ts"; import { legacyDeclarativeSchemaWrittenLine, + legacyWarnPreservedUnmanagedDeclarativeFiles, legacyWriteDeclarativeSchemas, } from "../../../shared/legacy-pgdelta.write.ts"; import type { LegacyDbSchemaDeclarativeSyncFlags } from "./sync.command.ts"; @@ -261,7 +262,10 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara ensureLocalPostgresImageCurrent, ); const generated = yield* legacyGenerateDeclarativeOutput(run, toml, target); - yield* legacyWriteDeclarativeSchemas(fs, path, declarativeDir, generated); + const written = yield* legacyWriteDeclarativeSchemas(fs, path, declarativeDir, generated); + // A manifest-less directory keeps files the export did not replace, and those + // files go straight into the plan below — warn before diffing against them. + yield* legacyWarnPreservedUnmanagedDeclarativeFiles(declarativeDirRel, written); if (!(yield* declarativeDirHasFiles(fs, declarativeDir))) { return yield* Effect.fail( new LegacyDeclarativeNoFilesGeneratedError({ diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts index c70f3fb58c..a0878b34a8 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.legacy.layer.ts @@ -10,6 +10,7 @@ import { } from "../../../../shared/legacy/global-flags.ts"; import { Output } from "../../../../shared/output/output.service.ts"; import { RuntimeInfo } from "../../../../shared/runtime/runtime-info.service.ts"; +import { legacyYellow } from "../../../shared/legacy-colors.ts"; import { LegacyDbConnection } from "../../../shared/legacy-db-connection.service.ts"; import { LegacyEdgeRuntimeScript } from "../../../shared/legacy-edge-runtime-script.service.ts"; import { LegacyDockerRun } from "../../../shared/legacy-docker-run.service.ts"; @@ -38,6 +39,18 @@ import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; const mapError = (cause: { readonly message: string }) => new LegacyPgDeltaEngineError({ message: cause.message, cause }); +/** + * `--strict-coverage` is enforced entirely by the next engine's diagnostic report + * (`legacy-pgdelta-next-diagnostics.ts`); this adapter has no coverage diagnostics + * to reject, so the flag is a silent no-op under + * `SUPABASE_USE_PG_DELTA_NEXT=false`. Say so once instead, rather than letting a + * user believe an unsupported-object guard is active. Mirrors `db diff`'s + * `warnPgSchemaDeprecated` line shape. + */ +export const legacyStrictCoverageIgnoredWarning = `${legacyYellow( + "WARNING:", +)} "--strict-coverage" has no effect with the legacy pg-delta engine.`; + function normalizeDiff( result: { readonly sql: string; @@ -106,6 +119,19 @@ export const legacyPgDeltaLegacyEngineLayer = Layer.effect( operation: Effect.Effect, ) => operation.pipe(Effect.provide(runtime)); + // Emitted from the engine layer, not from each handler: this is the single place + // where the resolved implementation and the per-operation input meet, so all four + // workflows (`db diff`, `db pull`, and declarative `generate`/`sync`) get the line + // with no per-command wiring. Once per process — `sync` can plan twice (extension + // repair re-plans) and a repeated line adds nothing. + let strictCoverageWarned = false; + const warnStrictCoverageIgnored = (strictCoverage: boolean) => + Effect.suspend(() => { + if (!strictCoverage || strictCoverageWarned) return Effect.void; + strictCoverageWarned = true; + return output.raw(`${legacyStrictCoverageIgnoredWarning}\n`, "stderr"); + }); + const endpointRef = ( context: LegacyPgDeltaContext, endpoint: LegacyPgDeltaEndpoint, @@ -132,6 +158,7 @@ export const legacyPgDeltaLegacyEngineLayer = Layer.effect( implementation: "legacy", diffExplicit: (input) => Effect.gen(function* () { + yield* warnStrictCoverageIgnored(input.strictCoverage); const sourceRef = yield* endpointRef(input.context, input.source, input.toml); const targetRef = yield* endpointRef(input.context, input.desired, input.toml); const result = yield* provideRuntime( @@ -146,6 +173,7 @@ export const legacyPgDeltaLegacyEngineLayer = Layer.effect( }).pipe(Effect.mapError(mapError)), diffDatabase: (input) => Effect.gen(function* () { + yield* warnStrictCoverageIgnored(input.strictCoverage); const sourceSnapshot = input.debug ? yield* provideRuntime( legacyExportCatalogPgDelta(input.context, { @@ -178,6 +206,7 @@ export const legacyPgDeltaLegacyEngineLayer = Layer.effect( }).pipe(Effect.mapError(mapError)), exportDeclarativeSchema: (input) => Effect.gen(function* () { + yield* warnStrictCoverageIgnored(input.strictCoverage); if (input.source === undefined) { return yield* Effect.fail( new LegacyPgDeltaEngineError({ @@ -200,6 +229,7 @@ export const legacyPgDeltaLegacyEngineLayer = Layer.effect( }).pipe(Effect.mapError(mapError)), planDeclarativeSchema: (input) => Effect.gen(function* () { + yield* warnStrictCoverageIgnored(input.strictCoverage); const sourceRef = yield* legacyGetMigrationsCatalogRef( fs, path, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index c697a91629..a122bde2b5 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -1,7 +1,10 @@ import { Clock, Effect, FileSystem, Layer, Path } from "effect"; import { Output } from "../../../../shared/output/output.service.ts"; -import { parseLegacyConnectionString } from "../../../shared/legacy-db-config.parse.ts"; +import { + parseLegacyConnectionString, + redactLegacyConnectionString, +} from "../../../shared/legacy-db-config.parse.ts"; import { LegacyDbConnectError } from "../../../shared/legacy-db-connection.errors.ts"; import { legacyAcquirePgPool } from "../../../shared/legacy-db-connection.sql-pg.layer.ts"; import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; @@ -109,7 +112,11 @@ export function legacyParsePgDeltaNextEndpoint(endpoint: LegacyPgDeltaDatabaseEn return yield* Effect.fail( new LegacyPgDeltaEngineError({ message: "failed to parse Postgres connection string for pg-delta", - cause: endpoint.ref.replace(/:[^:@/]+@/, ":***@"), + // `redactLegacyConnectionString`, not a local `:password@` regex: the input + // reaching here is by definition unparseable, and a hand-typed password + // containing `/`, `@`, or `:` defeats a naive single-character-class match + // (CWE-209). The shared redactor over-redacts instead of leaking. + cause: redactLegacyConnectionString(endpoint.ref), }), ); }); @@ -226,7 +233,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( return yield* Effect.fail( new LegacyPgDeltaEngineError({ message: "failed to parse pg-delta migrations shadow URL", - cause: shadow.migrationsUrl.replace(/:[^:@/]+@/, ":***@"), + cause: redactLegacyConnectionString(shadow.migrationsUrl), }), ); } diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts index c16d24fc33..5cf88b05ac 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts @@ -30,7 +30,23 @@ describe("legacyParsePgDeltaNextEndpoint", () => { expect(error).toBeInstanceOf(LegacyPgDeltaEngineError); expect(error.message).toBe("failed to parse Postgres connection string for pg-delta"); - expect(error.cause).toBe("postgresql://postgres:***@[/postgres"); + expect(error.cause).toBe("postgresql://postgres:[REDACTED]@[/postgres"); + }); + + it("redacts a password containing @, :, and / rather than leaking a fragment", () => { + // The previous inline `/:[^:@/]+@/` regex matched nothing here and surfaced the + // raw URL; the shared redactor anchors on the last `@` before the authority + // terminator and over-redacts instead (CWE-209). + const endpoint = { + kind: "database", + ref: "postgresql://postgres:p@ss:word/x@[/postgres", + connectOptions: { isLocal: false, dnsResolver: "native" }, + } satisfies LegacyPgDeltaDatabaseEndpoint; + + const error = Effect.runSync(legacyParsePgDeltaNextEndpoint(endpoint).pipe(Effect.flip)); + + expect(String(error.cause)).not.toContain("ss:word/x"); + expect(String(error.cause)).toContain("[REDACTED]"); }); it("uses a supplied parsed connection without reparsing the display ref", () => { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index 425a94a337..9fbbe81899 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -40,6 +40,7 @@ import { type LegacyPgDeltaNextOperation, } from "./legacy-pgdelta-next-adapter.service.ts"; import type { LegacyPgDeltaRemovalSummary } from "./legacy-pgdelta-engine.service.ts"; +import { LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE } from "./legacy-pgdelta-next-diagnostics.ts"; interface LegacyPgDeltaNextLibraryDiagnostic { readonly code: string; @@ -230,6 +231,28 @@ function legacyNormalizePgDeltaNextDiagnostics( })); } +/** + * Turns `planSchemaFiles`' skipped statements into coverage diagnostics so they + * travel the ONE diagnostic report path every consumer already renders and + * enforces — warned by default, blocking under `--strict-coverage`. Built here, + * where `skipped` originates, so no consumer has to remember to look at the + * separate `skipped` field (nothing did, and the statements vanished silently). + * The raw statement stays in `context`/detail rather than the aggregate warning, + * since a skipped `CREATE ROLE` can carry a password. + */ +function legacySkippedStatementDiagnostics( + skipped: readonly { readonly file: string; readonly stmt: string }[], +): LegacyPgDeltaNextDiagnostic[] { + return skipped.map((entry) => ({ + origin: "declarativeLoad", + code: LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE, + severity: "warning", + subject: entry.file, + message: `pg-delta could not load a declarative schema statement from ${entry.file}: ${entry.stmt}`, + context: { file: entry.file, statement: entry.stmt }, + })); +} + function legacyIsPgDeltaNextParameterAclDiagnostic( diagnostic: LegacyPgDeltaNextLibraryDiagnostic, ): boolean { @@ -623,6 +646,7 @@ function legacyMakePgDeltaNextAdapter ({ diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index 08897c35c7..6e7a821876 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -765,12 +765,25 @@ describe("LegacyPgDeltaNextAdapter", () => { "declarativeLoad", "declarativeTarget", "declarativeDrift", + "declarativeLoad", ]); - expect(planned.diagnostics.at(-1)).toMatchObject({ + expect(planned.diagnostics.at(-2)).toMatchObject({ origin: "declarativeDrift", code: "unmodeled_drift", subject: "subject:drift", }); + // Statements the loader could not model become coverage diagnostics here, so + // they travel the shared diagnostic report (warn by default, fail under + // `--strict-coverage`) instead of only living in the unread `skipped` field. + expect(planned.diagnostics.at(-1)).toEqual({ + origin: "declarativeLoad", + code: "skipped_statement", + severity: "warning", + subject: "roles.sql", + message: + "pg-delta could not load a declarative schema statement from roles.sql: create role ignored", + context: { file: "roles.sql", statement: "create role ignored" }, + }); expect(planned.hazards).toEqual({ actions: [{ actionIndex: 0, kinds: ["data_loss"] }], dataLoss: [{ actionIndex: 0, sql: "TRUNCATE TABLE public.audit_log" }], diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts index ce0bf8ba86..576ca31bb2 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.ts @@ -23,6 +23,24 @@ const operationAction: Record = { snapshotCapture: "capture the database snapshot", }; +/** + * Declarative statements pg-delta's loader could not model, e.g. `CREATE ROLE` + * (`planSchemaFiles`' `skipped` list). Synthesized by the next adapter rather than + * emitted by pg-delta itself, so it is absent from the library's + * `STRICT_COVERAGE_CODES` — {@link isCoverageDiagnostic} layers it on top so a + * skipped statement warns by default and fails under `--strict-coverage`, exactly + * like an unmodeled object kind. Silently dropping these would let a statement the + * user wrote in a declarative file vanish from the plan with no signal at all. + */ +export const LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE = "skipped_statement"; + +function isCoverageDiagnostic(diagnostic: LegacyPgDeltaNextDiagnostic): boolean { + return ( + STRICT_COVERAGE_CODES.has(diagnostic.code) || + diagnostic.code === LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE + ); +} + export interface LegacyPgDeltaNextDiagnosticReport { readonly diagnostics: ReadonlyArray; readonly blocking: ReadonlyArray; @@ -42,20 +60,23 @@ export function legacyPgDeltaNextDiagnosticReport( diagnostics: readonly LegacyPgDeltaNextDiagnostic[], strictCoverage: boolean, ): LegacyPgDeltaNextDiagnosticReport { - const coverage = diagnostics.filter((diagnostic) => STRICT_COVERAGE_CODES.has(diagnostic.code)); + const coverage = diagnostics.filter(isCoverageDiagnostic); const libraryDiagnostics = diagnostics.map((diagnostic) => ({ code: diagnostic.code, severity: diagnostic.severity, message: diagnostic.message, ...(diagnostic.context !== undefined ? { context: { ...diagnostic.context } } : {}), })); - const blocking = hasBlockingDiagnostics(libraryDiagnostics, { strictCoverage }) - ? diagnostics.filter( - (diagnostic) => - diagnostic.severity === "error" || - (strictCoverage && STRICT_COVERAGE_CODES.has(diagnostic.code)), - ) - : []; + // The library's own policy stays authoritative for library codes; the CLI-owned + // skipped-statement code is OR-ed in so strict coverage blocks on it too. + const blocking = + hasBlockingDiagnostics(libraryDiagnostics, { strictCoverage }) || + (strictCoverage && coverage.length > 0) + ? diagnostics.filter( + (diagnostic) => + diagnostic.severity === "error" || (strictCoverage && isCoverageDiagnostic(diagnostic)), + ) + : []; const unmodeledKinds = [ ...new Set(diagnostics.map(diagnosticKind).filter((kind) => kind !== undefined)), ].sort((left, right) => left.localeCompare(right)); @@ -88,6 +109,34 @@ function legacyPgDeltaNextUnmodeledKindsMessage( return `${summary} ${policy}`; } +/** + * Aggregate line for skipped declarative statements, shaped like the unmodeled-kind + * summary above: it names the files, never the statement SQL (the per-diagnostic + * detail carries that, and only renders under `--debug`/`--strict-coverage`). + */ +function legacyPgDeltaNextSkippedStatementsMessage( + operation: LegacyPgDeltaNextOperation, + skipped: readonly LegacyPgDeltaNextDiagnostic[], + strictCoverage: boolean, +): string { + const policy = strictCoverage + ? "Strict coverage is enabled, so the operation will stop." + : operationConsequence[operation]; + const files = [ + ...new Set( + skipped + .map((diagnostic) => diagnostic.subject) + .filter((subject): subject is string => subject !== undefined && subject !== "unknown"), + ), + ].sort((left, right) => left.localeCompare(right)); + const where = files.length === 0 ? "" : ` in ${files.join(", ")}`; + const count = + skipped.length === 1 + ? "1 declarative schema statement" + : `${skipped.length} declarative schema statements`; + return `pg-delta could not load ${count}${where}. ${policy}`; +} + function shellQuote(value: string): string { return `'${value.replaceAll("'", `'"'"'`)}'`; } @@ -129,7 +178,7 @@ export const legacyReportPgDeltaNextDiagnostics = Effect.fnUntraced(function* ( const renderDetail = verboseDiagnostics || diagnostic.severity === "error" || - (strictCoverage && STRICT_COVERAGE_CODES.has(diagnostic.code)); + (strictCoverage && isCoverageDiagnostic(diagnostic)); if (!renderDetail) { yield* debug.debug(message); continue; @@ -152,6 +201,15 @@ export const legacyReportPgDeltaNextDiagnostics = Effect.fnUntraced(function* ( ); } + const skipped = report.diagnostics.filter( + (diagnostic) => diagnostic.code === LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE, + ); + if (skipped.length > 0) { + yield* output.warn( + legacyPgDeltaNextSkippedStatementsMessage(operation, skipped, strictCoverage), + ); + } + const feedback = showFeedback ? legacyPgDeltaNextFeedbackInvitation(report.unmodeledKinds) : undefined; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts index fb51bf113e..ec2ca53f6e 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts @@ -6,6 +6,7 @@ import { mockOutput } from "../../../../../tests/helpers/mocks.ts"; import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; import type { LegacyPgDeltaNextDiagnostic } from "./legacy-pgdelta-next-adapter.service.ts"; import { + LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE, legacyPgDeltaNextDiagnosticMessage, legacyPgDeltaNextDiagnosticReport, legacyPgDeltaNextFeedbackInvitation, @@ -153,6 +154,76 @@ describe("pg-delta next diagnostic coverage policy", () => { }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); + const skippedStatement = (file: string, statement: string): LegacyPgDeltaNextDiagnostic => ({ + origin: "declarativeLoad", + code: LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE, + severity: "warning", + subject: file, + message: `pg-delta could not load a declarative schema statement from ${file}: ${statement}`, + context: { file, statement }, + }); + + it("warns about skipped declarative statements without leaking their SQL", () => { + const out = mockOutput(); + const debugMessages: string[] = []; + return Effect.gen(function* () { + yield* legacyReportPgDeltaNextDiagnostics( + "declarativePlan", + [ + skippedStatement("roles.sql", "create role app password 's3cret'"), + skippedStatement("roles.sql", "alter role app set search_path = public"), + ], + false, + ); + + expect(out.messages).toContainEqual({ + type: "warn", + message: + "pg-delta could not load 2 declarative schema statements in roles.sql. Changes to these objects are omitted from the declarative migration plan.", + }); + // Statement text stays out of the default-visibility summary; the per-diagnostic + // detail carries it and is routed to debug unless strict/verbose. + expect(out.messages.some(({ message }) => message.includes("s3cret"))).toBe(false); + expect(debugMessages.some((message) => message.includes("s3cret"))).toBe(true); + }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); + }); + + it("fails on skipped declarative statements under strict coverage", () => { + const out = mockOutput(); + const debugMessages: string[] = []; + return Effect.gen(function* () { + const exit = yield* legacyReportPgDeltaNextDiagnostics( + "declarativePlan", + [skippedStatement("roles.sql", "create role app")], + true, + ).pipe(Effect.exit); + + expect(Exit.isFailure(exit)).toBe(true); + expect(out.messages).toContainEqual({ + type: "warn", + message: + "pg-delta could not load 1 declarative schema statement in roles.sql. Strict coverage is enabled, so the operation will stop.", + }); + // Strict mode renders the detail (with the statement) so the user can fix it. + expect(out.messages.some(({ message }) => message.includes("create role app"))).toBe(true); + // No object kinds involved, so no unmodeled-kind summary and no feedback invite. + expect(out.messages.some(({ message }) => message.includes("does not manage"))).toBe(false); + expect(out.messages.some(({ message }) => message.includes("supabase issue feature"))).toBe( + false, + ); + }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); + }); + + it("classifies a skipped statement as a coverage gap only under strict coverage", () => { + const diagnostics = [skippedStatement("roles.sql", "create role app")]; + const lenient = legacyPgDeltaNextDiagnosticReport(diagnostics, false); + expect(lenient.coverage).toHaveLength(1); + expect(lenient.blocking).toEqual([]); + + const strict = legacyPgDeltaNextDiagnosticReport(diagnostics, true); + expect(strict.blocking).toEqual(strict.coverage); + }); + it("always renders and fails error diagnostics", () => { const out = mockOutput(); const debugMessages: string[] = []; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts index 21798992d7..11e8a29e43 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts @@ -1,7 +1,8 @@ import { Effect, type FileSystem, type Path } from "effect"; import { classifySqlFiles } from "@supabase/pg-delta/frontends"; -import { legacyBold } from "../../../shared/legacy-colors.ts"; +import { Output } from "../../../../shared/output/output.service.ts"; +import { legacyBold, legacyYellow } from "../../../shared/legacy-colors.ts"; import { legacyWalkSqlFiles } from "../../../shared/legacy-glob.ts"; import type { LegacyDeclarativeOutput } from "../../../shared/legacy-pgdelta.ts"; import { LegacyDeclarativeWriteError } from "./legacy-pgdelta.errors.ts"; @@ -22,6 +23,22 @@ function legacyDeclarativeWriteError(message: string): LegacyDeclarativeWriteErr return new LegacyDeclarativeWriteError({ message }); } +/** + * What a declarative write left behind, so the calling handler (which owns + * {@link Output}) can tell the user about it. + */ +export interface LegacyDeclarativeWriteResult { + /** + * Pre-existing `.sql` files the next writer preserved because no export + * manifest claimed ownership of them — see + * {@link legacyPreservedUnmanagedDeclarativeFilesWarning}. Always empty for the + * legacy writer, which wipes the directory outright. + */ + readonly preservedUnmanagedFiles: ReadonlyArray; +} + +const NO_PRESERVED_FILES: LegacyDeclarativeWriteResult = { preservedUnmanagedFiles: [] }; + function isNextDeclarativeOutput( output: LegacyDeclarativeWriteOutput, ): output is LegacyPgDeltaNextDeclarativeOutput { @@ -105,6 +122,7 @@ const writeLegacyDeclarativeSchemas = Effect.fnUntraced(function* ( yield* fs.makeDirectory(path.dirname(targetPath), { recursive: true }); yield* fs.writeFileString(targetPath, file.sql); } + return NO_PRESERVED_FILES; }); const writeNextDeclarativeSchemas = Effect.fnUntraced(function* ( @@ -161,6 +179,20 @@ const writeNextDeclarativeSchemas = Effect.fnUntraced(function* ( ? { previouslyOwned: new Set(previousManifest.files) } : {}), }); + // With no manifest there is no ownership record, so nothing can be classified as + // stale: every pre-existing file the export does not itself replace survives. The + // typical producer of such a directory is the OLD legacy full-wipe exporter, whose + // files still feed future plans (`LegacyLoadPgDeltaSqlFiles` walks the whole tree), + // so the result is a silent partial merge behind an "overwrite existing files" + // prompt. Report them and let the handler say so out loud. + const proposedNames = new Set(proposed.map((file) => file.name)); + const preservedUnmanagedFiles = + previousManifest?.files === undefined + ? existingFiles + .map((file) => file.name) + .filter((name) => !proposedNames.has(name)) + .sort() + : []; yield* fs.makeDirectory(declarativeDir, { recursive: true }); const changed = new Set([...classification.created, ...classification.updated]); @@ -212,6 +244,7 @@ const writeNextDeclarativeSchemas = Effect.fnUntraced(function* ( if (previousSerialized !== serialized) { yield* fs.writeFileString(manifestPath, serialized); } + return { preservedUnmanagedFiles } satisfies LegacyDeclarativeWriteResult; }); /** @@ -224,13 +257,52 @@ const writeNextDeclarativeSchemas = Effect.fnUntraced(function* ( export const legacyDeclarativeSchemaWrittenLine = (dir: string): string => `Declarative schema written to ${legacyBold(dir)}\n`; +/** + * The manifest-less-merge warning text. The next writer only prunes files an + * existing `.pgdelta-export.json` claimed, so a directory produced by the old + * legacy full-wipe exporter (no manifest) keeps every file the new export does not + * itself replace — even though the prompt the user just answered said existing + * files may be deleted. Name the survivors and give the one instruction that + * actually produces a clean tree. + */ +export const legacyPreservedUnmanagedDeclarativeFilesWarning = ( + dir: string, + files: ReadonlyArray, +): string => + `${legacyYellow( + `WARNING: ${files.length} existing declarative schema file(s) in ${dir} are not tracked by an export manifest and were preserved: ${files.join( + ", ", + )}`, + )}\n${legacyYellow( + `These files still contribute to future declarative plans. To regenerate the directory cleanly, remove ${dir} and re-run supabase db schema declarative generate.`, + )}\n`; + +/** + * Emits {@link legacyPreservedUnmanagedDeclarativeFilesWarning} when a write + * preserved unmanaged files. Lives next to the writer (which has no `Output`) so + * all three declarative write callers share one warning, and is a no-op otherwise. + */ +export const legacyWarnPreservedUnmanagedDeclarativeFiles = Effect.fnUntraced(function* ( + dir: string, + written: LegacyDeclarativeWriteResult, +) { + if (written.preservedUnmanagedFiles.length === 0) return; + const output = yield* Output; + yield* output.raw( + legacyPreservedUnmanagedDeclarativeFilesWarning(dir, written.preservedUnmanagedFiles), + "stderr", + ); +}); + /** * Materializes pg-delta declarative export output under the declarative dir. * Legacy-engine output keeps Go's wipe-and-rewrite behavior. Next-engine output * uses pg-delta's manifest ownership and file classification: only stale files * owned by the previous export are removed, unchanged files are not rewritten, * unmanaged files are preserved, and the reserved root `_custom/` tree is never - * read as managed output or deleted. + * read as managed output or deleted. Returns which unmanaged files that preservation + * kept, so the caller can warn (see + * {@link legacyWarnPreservedUnmanagedDeclarativeFiles}). * * Go also updates `[db.migrations] schema_paths` afterwards, but only when * pg-delta is *disabled* in config (`if utils.IsPgDeltaEnabled() { return nil }`). diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts index 04edde344b..fa35b261b0 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts @@ -14,12 +14,14 @@ import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; import { Cause, Effect, Exit, FileSystem, Path } from "effect"; +import { mockOutput } from "../../../../../tests/helpers/mocks.ts"; import { legacyBold } from "../../../shared/legacy-colors.ts"; import type { LegacyDeclarativeOutput } from "../../../shared/legacy-pgdelta.ts"; import { LegacyDeclarativeWriteError } from "./legacy-pgdelta.errors.ts"; import type { LegacyPgDeltaDeclarativeExportResult } from "./legacy-pgdelta-engine.service.ts"; import { legacyDeclarativeSchemaWrittenLine, + legacyWarnPreservedUnmanagedDeclarativeFiles, legacyWriteDeclarativeSchemas, } from "./legacy-pgdelta.write.ts"; @@ -122,6 +124,84 @@ describe("legacyWriteDeclarativeSchemas", () => { ); }); + it.effect("reports pre-existing files preserved when no manifest claims ownership", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); + const declDir = join(dir, "supabase", "database"); + // A directory produced by the OLD legacy full-wipe exporter: `.sql` files, no + // `.pgdelta-export.json`. Nothing can be classified as stale, so the next writer + // merges into it — the caller has to tell the user that happened. + mkdirSync(join(declDir, "_custom"), { recursive: true }); + writeFileSync(join(declDir, "_custom", "casts.sql"), "create cast (int as text);"); + writeFileSync(join(declDir, "legacy-b.sql"), "select 'b';"); + writeFileSync(join(declDir, "legacy-a.sql"), "select 'a';"); + writeFileSync(join(declDir, "schemas-public.sql"), "-- replaced by the export below"); + + return write(declDir, { + files: [{ name: "schemas-public.sql", sql: "create table public.example(id int);" }], + manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, + }).pipe( + Effect.tap((written) => + Effect.sync(() => { + // Sorted, excludes the file the export itself replaced and the reserved + // `_custom/` tree (never read as managed output). + expect(written.preservedUnmanagedFiles).toEqual(["legacy-a.sql", "legacy-b.sql"]); + expect(readFileSync(join(declDir, "legacy-a.sql"), "utf8")).toBe("select 'a';"); + expect(readFileSync(join(declDir, "schemas-public.sql"), "utf8")).toBe( + "create table public.example(id int);", + ); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("reports nothing preserved once a manifest owns the directory", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); + const declDir = join(dir, "supabase", "database"); + mkdirSync(declDir, { recursive: true }); + writeFileSync(join(declDir, "unmanaged.sql"), "select 'keep me';"); + writeFileSync( + join(declDir, ".pgdelta-export.json"), + `${JSON.stringify({ + formatVersion: 1, + redactSecrets: true, + scope: "database", + files: ["stale.sql"], + })}\n`, + ); + + return write(declDir, { + files: [{ name: "schemas/public.sql", sql: "create table public.example(id int);" }], + manifest: { redactSecrets: true, scope: "database" }, + }).pipe( + Effect.tap((written) => + Effect.sync(() => { + expect(written.preservedUnmanagedFiles).toEqual([]); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + + it.effect("reports nothing preserved for the legacy full-wipe writer", () => { + const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); + const declDir = join(dir, "supabase", "database"); + mkdirSync(declDir, { recursive: true }); + writeFileSync(join(declDir, "stale.sql"), "-- wiped"); + return write(declDir, { + version: 1, + mode: "declarative", + files: [{ path: "public.sql", order: 0, statements: 0, sql: "select 1;" }], + }).pipe( + Effect.tap((written) => + Effect.sync(() => { + expect(written.preservedUnmanagedFiles).toEqual([]); + rmSync(dir, { recursive: true, force: true }); + }), + ), + ); + }); + it.effect("does not rewrite unchanged next-engine files or manifests", () => { const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); const declDir = join(dir, "supabase", "database"); @@ -225,3 +305,29 @@ describe("legacyDeclarativeSchemaWrittenLine", () => { ); }); }); + +describe("legacyWarnPreservedUnmanagedDeclarativeFiles", () => { + it.effect("names the preserved files and advises a full rewrite", () => { + const out = mockOutput(); + return Effect.gen(function* () { + yield* legacyWarnPreservedUnmanagedDeclarativeFiles("supabase/database", { + preservedUnmanagedFiles: ["legacy-a.sql", "legacy-b.sql"], + }); + const stderr = out.stderrText; + expect(stderr).toContain("2 existing declarative schema file(s) in supabase/database"); + expect(stderr).toContain("legacy-a.sql, legacy-b.sql"); + expect(stderr).toContain("were preserved"); + expect(stderr).toContain("remove supabase/database and re-run"); + }).pipe(Effect.provide(out.layer)); + }); + + it.effect("stays silent when the write preserved nothing", () => { + const out = mockOutput(); + return Effect.gen(function* () { + yield* legacyWarnPreservedUnmanagedDeclarativeFiles("supabase/database", { + preservedUnmanagedFiles: [], + }); + expect(out.stderrText).toBe(""); + }).pipe(Effect.provide(out.layer)); + }); +}); diff --git a/apps/cli/src/legacy/commands/db/start/start.integration.test.ts b/apps/cli/src/legacy/commands/db/start/start.integration.test.ts index a0fd890d14..e0f18b5030 100644 --- a/apps/cli/src/legacy/commands/db/start/start.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/start/start.integration.test.ts @@ -43,6 +43,8 @@ import type { LegacyDbStartFlags } from "./start.command.ts"; const DEFAULT_FLAGS: LegacyDbStartFlags = { fromBackup: Option.none() }; const PG_NET_CREATE_FINGERPRINT = "create extension if not exists pg_net schema extensions"; +const PG_NET_DROP_FINGERPRINT = "drop extension if exists pg_net"; +const GLOBALS_FINGERPRINT = "CREATE ROLE anon"; function flags(fromBackup?: string): LegacyDbStartFlags { return { fromBackup: fromBackup === undefined ? Option.none() : Option.some(fromBackup) }; @@ -234,7 +236,7 @@ const alwaysReadyHttpClientLayer = Layer.succeed( ); /** Mirrors `start.integration.test.ts`'s own `fakeDbSession` — PG15+ (this suite's default) never calls `exec`/`query` (its schema init is three one-shot `LegacyDockerRun` jobs instead). */ -function fakeDbSession() { +function fakeDbSession(appliedMigrationStatements?: ReadonlyArray) { const calls: Array<{ kind: "exec" | "query"; sql: string }> = []; const session: LegacyDbSession = { exec: (sql) => @@ -244,7 +246,18 @@ function fakeDbSession() { query: (sql) => Effect.sync(() => { calls.push({ kind: "query", sql }); - return []; + // The Database Webhooks convergence reads applied-migration statements to + // decide whether pg_net is migration-owned (and must not be dropped). + return appliedMigrationStatements !== undefined && + sql.includes("supabase_migrations.schema_migrations") + ? [ + { + version: "20240101000000", + name: "migration", + statements: [...appliedMigrationStatements], + }, + ] + : []; }), extensionExists: () => Effect.succeed(false), copyToCsv: () => Effect.succeed(new Uint8Array()), @@ -284,6 +297,12 @@ interface SetupOpts { readonly connectFailures?: number; /** Whether the mocked connect failures are dial-level (`retryable`). Defaults to `true`. */ readonly connectFailuresRetryable?: boolean; + /** + * Statements of one recorded row in `supabase_migrations.schema_migrations`, read by + * the existing-volume Database Webhooks convergence to decide whether pg_net is + * migration-owned. Defaults to an empty history. + */ + readonly appliedMigrationStatements?: ReadonlyArray; } function setup(opts: SetupOpts = {}) { @@ -305,7 +324,7 @@ function setup(opts: SetupOpts = {}) { ? runningCheckFailsRoute(baseRoute) : baseRoute; const child = mockContainerCliSpawner(route); - const dbSession = fakeDbSession(); + const dbSession = fakeDbSession(opts.appliedMigrationStatements); const edgeRunCalls: Array = []; const edgeRuntime = Layer.succeed(LegacyEdgeRuntimeScript, { run: (runOpts: LegacyEdgeRuntimeRunOpts) => { @@ -597,12 +616,39 @@ describe("legacy db start", () => { expect(out.stderrText).toContain("Starting database from backup...\n"); expect(out.stderrText).not.toContain("Initialising schema..."); expect(dbSetupJobCalls(child.spawned)).toHaveLength(0); - expect(dbSession.calls).toHaveLength(0); + // No schema/globals/vault/roles SQL — the setup pipeline really is skipped. The + // only SQL on this path is the Database Webhooks convergence, which reads the + // migration history and (webhooks disabled, no migration owning pg_net) drops it. + expect(dbSession.calls.some((call) => call.sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe( + false, + ); + expect(dbSession.calls.some((call) => call.sql.includes(GLOBALS_FINGERPRINT))).toBe(false); + expect(dbSession.calls.some((call) => call.sql.includes(PG_NET_DROP_FINGERPRINT))).toBe( + true, + ); expect(readFileSync(currentBranchPath(tempRoot.current), "utf8")).toBe("main"); }); }, ); + it.live("leaves migration-owned pg_net alone when Webhooks are disabled", () => { + // `supabase start` does not replay migrations on an existing volume, so dropping + // pg_net here would silently break a database whose own migration created it, with + // nothing to put it back. + // Config validation rejects an explicit `enabled = false`, so "disabled" is the + // key being absent — exactly what a user who removes the block ends up with. + const { layer, dbSession } = setup({ + configContents: 'project_id = "test"\n', + appliedMigrationStatements: ["create extension if not exists pg_net with schema extensions"], + }); + return Effect.gen(function* () { + yield* legacyDbStart(DEFAULT_FLAGS).pipe(Effect.provide(layer)); + expect(dbSession.calls.some((call) => call.sql.includes(PG_NET_DROP_FINGERPRINT))).toBe( + false, + ); + }); + }); + it.live("installs pg_net on an existing volume from effective Webhooks config", () => { const { layer, out, child, dbSession } = setup({ configContents: 'project_id = "test"\n[experimental.webhooks]\nenabled = false\n', diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index d9963c7660..5ad6b732ee 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -141,6 +141,8 @@ import { LegacyDockerRun, type LegacyDockerRunOpts } from "../legacy-docker-run. import { LegacyEdgeRuntimeScript } from "../legacy-edge-runtime-script.service.ts"; import { legacyMigrateAndSeed } from "../legacy-migrate-and-seed.ts"; import { LegacyMigrationApplyError, legacyExecSqlFile } from "../legacy-migration-apply.ts"; +import { legacyReadMigrationTable } from "../legacy-migration-history.ts"; +import { legacyStatementInstallsPgNet } from "../legacy-pg-net-guidance.ts"; import { legacyTryCacheMigrationsCatalog } from "../legacy-pgdelta.cache.ts"; import { legacyResolvePgDeltaImplementation } from "../legacy-pgdelta-next-flag.ts"; import type { LegacyPgDeltaContext } from "../legacy-pgdelta.ts"; @@ -187,7 +189,7 @@ const LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL = // its schema. Remove it after the dump so the final baseline still follows the // user's webhooks setting. Enabled projects recreate it after the dump, when // the bundled event trigger can apply the intended grants. -const LEGACY_START_REMOVE_PG14_DATABASE_WEBHOOKS_SQL = "drop extension if exists pg_net;"; +const LEGACY_START_REMOVE_DATABASE_WEBHOOKS_SQL = "drop extension if exists pg_net;"; /** * A SQL exec (schema/globals/API-privileges) or one-shot service-migration Docker @@ -942,6 +944,28 @@ export const legacyApplyDatabaseWebhooks = Effect.fnUntraced(function* ( ); }); +/** + * Drops pg_net. Hoisted so the fresh-setup PG14 dump cleanup, the `db reset` PG14 + * path, and the existing-volume webhooks convergence all run the identical statement + * instead of one of them silently diverging (`db reset` on PG14 with webhooks + * disabled used to leave pg_net installed while fresh setup removed it). + */ +export const legacyRemoveDatabaseWebhooks = Effect.fnUntraced(function* ( + session: LegacyDbSession, + fs: FileSystem.FileSystem, + path: Path.Path, + tmpDir: string, +) { + yield* legacyExecSqlConstant( + session, + fs, + path, + tmpDir, + "remove-database-webhooks.sql", + LEGACY_START_REMOVE_DATABASE_WEBHOOKS_SQL, + ); +}); + /** * Port of Go's `initCurrentBranch` (`start.go:233-241`): writes * `supabase/.branches/_current_branch` = `"main"` (Go's `CurrBranchPath`, @@ -1030,14 +1054,7 @@ export const legacySetupDatabase = ( const requiresPg14WebhooksCleanup = input.majorVersion === 14; yield* legacyStartInitSchema(spawner, input, tmpDir); if (requiresPg14WebhooksCleanup) { - yield* legacyExecSqlConstant( - session, - fs, - path, - tmpDir, - "remove-pg14-database-webhooks.sql", - LEGACY_START_REMOVE_PG14_DATABASE_WEBHOOKS_SQL, - ); + yield* legacyRemoveDatabaseWebhooks(session, fs, path, tmpDir); } const activateUserExtensions = options.activateUserExtensions ?? true; const legacyPgNetBaseline = options.legacyPgNetBaseline ?? false; @@ -1306,7 +1323,22 @@ const legacyConnectLocalPostgres = (input: { ); }); -/** Idempotently converges Database Webhooks on a healthy, existing local database. */ +/** + * Idempotently converges Database Webhooks on a healthy, existing local database — + * in BOTH directions. Installing pg_net when the setting is on was always covered; + * removing it when the setting is turned back off was not, so pg_net silently + * survived on the volume while the next engine's shadow baseline (rebuilt from the + * current config) omitted it, reporting pg_net drift on every `db diff`/`db pull`. + * + * The removal is guarded, because `supabase start` does NOT replay migrations on an + * existing volume: an unconditional drop would delete an extension a user's own + * migration created, with nothing to put it back. So pg_net is dropped only when no + * applied migration in `supabase_migrations.schema_migrations` installs it (see + * {@link legacyStatementInstallsPgNet}), and any failure to read that history — + * missing table, malformed table, insufficient privileges — is treated as + * "migration-owned" and leaves the extension alone. Erring toward not dropping is + * the only safe direction here. + */ export const legacyRunDatabaseWebhooksSetup = (input: { readonly fs: FileSystem.FileSystem; readonly path: Path.Path; @@ -1314,15 +1346,23 @@ export const legacyRunDatabaseWebhooksSetup = (input: { readonly dbPort: number; readonly dbUrl: string; readonly enabled: boolean; -}) => { - if (!input.enabled) return Effect.void; - return Effect.scoped( +}) => + Effect.scoped( Effect.gen(function* () { const session = yield* legacyConnectLocalPostgres({ hostname: input.hostname, dbPort: input.dbPort, password: legacyStartInternalDbPassword(input.dbUrl), }); + if (!input.enabled) { + const pgNetOwnedByMigrations = yield* legacyReadMigrationTable(session).pipe( + Effect.map((migrations) => + migrations.some((migration) => migration.statements.some(legacyStatementInstallsPgNet)), + ), + Effect.orElseSucceed(() => true), + ); + if (pgNetOwnedByMigrations) return; + } const tmpDir = yield* input.fs .makeTempDirectoryScoped({ prefix: "supabase-start-db-webhooks-" }) .pipe( @@ -1334,10 +1374,13 @@ export const legacyRunDatabaseWebhooksSetup = (input: { }), ), ); + if (!input.enabled) { + yield* legacyRemoveDatabaseWebhooks(session, input.fs, input.path, tmpDir); + return; + } yield* legacyApplyDatabaseWebhooks(session, input.fs, input.path, tmpDir, input.enabled); }), ); -}; /** * Runs {@link legacyStartSetupLocalDatabase} against a freshly-provisioned local diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts index f1f356b0af..e171b8a051 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts @@ -9,7 +9,8 @@ import { Deferred, Effect, FileSystem, Layer, Path, Schema, Sink, Stream } from import { ChildProcessSpawner } from "effect/unstable/process"; import { mockOutput, mockRuntimeInfo } from "../../../../tests/helpers/mocks.ts"; -import type { LegacyDbSession } from "../legacy-db-connection.service.ts"; +import { LegacyDbExecError } from "../legacy-db-connection.errors.ts"; +import { LegacyDbConnection, type LegacyDbSession } from "../legacy-db-connection.service.ts"; import { LegacyDockerRun, type LegacyDockerRunOpts } from "../legacy-docker-run.service.ts"; import { LegacyDockerRunError } from "../legacy-docker-run.errors.ts"; import { LegacyEdgeRuntimeScriptError } from "../legacy-edge-runtime-script.errors.ts"; @@ -21,6 +22,7 @@ import { LegacyPgDeltaSslProbe } from "../legacy-pgdelta-ssl-probe.service.ts"; import { LegacyDbSetupError, legacyResolveDbSetupPrelude, + legacyRunDatabaseWebhooksSetup, legacyStartInitCurrentBranch, legacyStartSetupLocalDatabase, type LegacyStartSetupLocalDatabaseInput, @@ -917,6 +919,118 @@ describe("legacyResolveDbSetupPrelude", () => { ); }); +/** + * `supabase start` on an EXISTING volume never replays migrations, so this + * convergence is the only thing that can reconcile the volume's pg_net with the + * current `[experimental.webhooks]` setting — in both directions, and without ever + * dropping an extension a user's own migration created. + */ +describe("legacyRunDatabaseWebhooksSetup", () => { + const PG_NET_DROP_FINGERPRINT = "drop extension if exists pg_net"; + + function fakeWebhooksSession(opts: { + readonly appliedStatements?: ReadonlyArray>; + readonly historyUnavailable?: boolean; + }) { + const execSql: Array = []; + const session: LegacyDbSession = { + exec: (sql) => + Effect.sync(() => { + execSql.push(sql); + }), + query: (sql) => + sql.includes("supabase_migrations.schema_migrations") + ? opts.historyUnavailable === true + ? Effect.fail( + new LegacyDbExecError({ message: 'relation "schema_migrations" does not exist' }), + ) + : Effect.succeed( + (opts.appliedStatements ?? []).map((statements, index) => ({ + version: `2024010100000${index}`, + name: "migration", + statements, + })), + ) + : Effect.succeed([]), + extensionExists: () => Effect.succeed(false), + copyToCsv: () => Effect.succeed(new Uint8Array()), + queryRaw: () => Effect.succeed({ fields: [], rows: [], commandTag: "" }), + }; + return { session, execSql }; + } + + const converge = ( + enabled: boolean, + sessionOpts: Parameters[0] = {}, + ) => { + const { session, execSql } = fakeWebhooksSession(sessionOpts); + const dbConnection = Layer.succeed(LegacyDbConnection, { + connect: () => Effect.succeed(session), + }); + return { + execSql, + effect: Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + yield* legacyRunDatabaseWebhooksSetup({ + fs, + path, + hostname: "127.0.0.1", + dbPort: 54322, + dbUrl: "postgresql://postgres:postgres@127.0.0.1:5432/postgres", + enabled, + }); + }).pipe(Effect.provide(Layer.mergeAll(dbConnection, BunServices.layer))), + }; + }; + + it.effect("installs pg_net when Database Webhooks are enabled", () => { + const { execSql, effect } = converge(true); + return effect.pipe( + Effect.map(() => { + expect(execSql.some((sql) => sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(true); + expect(execSql.some((sql) => sql.includes(PG_NET_DROP_FINGERPRINT))).toBe(false); + }), + ); + }); + + it.effect("drops pg_net when disabled and no applied migration installs it", () => { + const { execSql, effect } = converge(false, { + appliedStatements: [["create table public.items (id int)"]], + }); + return effect.pipe( + Effect.map(() => { + expect(execSql.some((sql) => sql.includes(PG_NET_DROP_FINGERPRINT))).toBe(true); + expect(execSql.some((sql) => sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(false); + }), + ); + }); + + it.effect("preserves pg_net created by an applied migration", () => { + const { execSql, effect } = converge(false, { + appliedStatements: [ + ["create table public.items (id int)"], + ['CREATE EXTENSION IF NOT EXISTS "pg_net" WITH SCHEMA extensions'], + ], + }); + return effect.pipe( + Effect.map(() => { + expect(execSql.some((sql) => sql.includes(PG_NET_DROP_FINGERPRINT))).toBe(false); + }), + ); + }); + + it.effect("preserves pg_net when the migration history cannot be read", () => { + // Erring toward not dropping: an unreadable history is treated as ownership. + const { execSql, effect } = converge(false, { historyUnavailable: true }); + return effect.pipe( + Effect.map(() => { + expect(execSql.some((sql) => sql.includes(PG_NET_DROP_FINGERPRINT))).toBe(false); + }), + ); + }); +}); + describe("legacyStartInitCurrentBranch", () => { it.effect('writes supabase/.branches/_current_branch = "main" when absent', () => { const workdir = makeWorkdir(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts index 89ecfe6e78..c952f43be0 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts @@ -122,6 +122,7 @@ import { legacyApplyApiPrivileges, legacyApplyDatabaseWebhooks, legacyInitSchema14, + legacyRemoveDatabaseWebhooks, LegacyDbSetupError, type LegacyFreshDbSetupInput, type LegacyStartSetupLocalDatabaseError, @@ -475,6 +476,14 @@ const legacyRecreateLocalDatabase14 = ( ), ); yield* legacyInitSchema14(session, fs, path, tmpDir, setup.majorVersion); + // Same drop-then-conditionally-recreate sequence fresh setup runs + // (`db-setup.ts`'s `requiresPg14WebhooksCleanup`): the PG14 dump installs + // pg_net unconditionally because later statements grant on its schema, so + // without this drop a reset with webhooks disabled left pg_net installed and + // diverged from a fresh `supabase start` — visible as pg_net drift in the next + // engine's shadow baseline. `MigrateAndSeed` re-applies every migration below, + // so a user migration that creates pg_net still gets it back. + yield* legacyRemoveDatabaseWebhooks(session, fs, path, tmpDir); yield* legacyApplyApiPrivileges( session, fs, diff --git a/apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts b/apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts index e9d9894990..8fdaf93d94 100644 --- a/apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts +++ b/apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts @@ -19,3 +19,21 @@ export const legacyIsPgNetUnavailableError = ( ): boolean => (error.code === "3F000" && MISSING_NET_SCHEMA_PATTERN.test(error.message)) || (error.code === "42883" && MISSING_PG_NET_FUNCTION_PATTERN.test(error.message)); + +const CREATE_PG_NET_EXTENSION_PATTERN = /\bcreate\s+extension\b[\s\S]*?\bpg_net\b/iu; + +/** + * Whether a recorded `supabase_migrations.schema_migrations` statement installs + * pg_net. + * + * Deliberately a loose, over-matching scan (no SQL parse, no schema/quoting + * awareness) because of the direction the answer is used in: it only ever gates + * AWAY from dropping the extension. A false positive leaves a user's pg_net + * installed, which is harmless; a false negative would drop an extension the user's + * OWN migrations created — and `supabase start` does not replay migrations on an + * existing volume, so nothing would put it back. The constraint this exists to + * enforce: the webhooks-disabled convergence drop must never remove + * migration-owned pg_net. + */ +export const legacyStatementInstallsPgNet = (statement: string): boolean => + CREATE_PG_NET_EXTENSION_PATTERN.test(statement); diff --git a/apps/cli/src/legacy/shared/legacy-pg-net-guidance.unit.test.ts b/apps/cli/src/legacy/shared/legacy-pg-net-guidance.unit.test.ts new file mode 100644 index 0000000000..dddb5f290a --- /dev/null +++ b/apps/cli/src/legacy/shared/legacy-pg-net-guidance.unit.test.ts @@ -0,0 +1,56 @@ +import { describe, expect, it } from "vitest"; + +import { + legacyIsPgNetUnavailableError, + legacyStatementInstallsPgNet, +} from "./legacy-pg-net-guidance.ts"; + +describe("legacyIsPgNetUnavailableError", () => { + it("matches only the pg_net-specific undefined schema and function failures", () => { + expect( + legacyIsPgNetUnavailableError({ code: "3F000", message: 'schema "net" does not exist' }), + ).toBe(true); + expect( + legacyIsPgNetUnavailableError({ + code: "42883", + message: "function net.http_post(url => text) does not exist", + }), + ).toBe(true); + // Right message, wrong SQLSTATE — a client-side echo, not a server verdict. + expect( + legacyIsPgNetUnavailableError({ code: "42P01", message: 'schema "net" does not exist' }), + ).toBe(false); + // Right SQLSTATE, unrelated object. + expect( + legacyIsPgNetUnavailableError({ code: "3F000", message: 'schema "audit" does not exist' }), + ).toBe(false); + expect(legacyIsPgNetUnavailableError({ message: 'schema "net" does not exist' })).toBe(false); + }); +}); + +/** + * This predicate only ever gates AWAY from dropping pg_net, so it must be + * generous: every plausible spelling of an install counts, and no false negative + * is acceptable. + */ +describe("legacyStatementInstallsPgNet", () => { + it.each([ + "create extension pg_net", + "CREATE EXTENSION pg_net", + "create extension if not exists pg_net schema extensions", + 'CREATE EXTENSION IF NOT EXISTS "pg_net" WITH SCHEMA extensions', + "create\n extension if not exists\n pg_net\n with schema extensions", + ])("treats %j as a pg_net install", (statement) => { + expect(legacyStatementInstallsPgNet(statement)).toBe(true); + }); + + it.each([ + "create table public.items (id int)", + "create extension pgcrypto", + "drop extension if exists pg_net", + "select net.http_post(url := 'https://example.com')", + "comment on extension pgcrypto is 'pg_net is not installed here'", + ])("does not treat %j as a pg_net install", (statement) => { + expect(legacyStatementInstallsPgNet(statement)).toBe(false); + }); +}); diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md new file mode 100644 index 0000000000..4bb369e01c --- /dev/null +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -0,0 +1,29 @@ +# pg-delta next: deferred follow-ups + +Findings from the pg-delta-next bundling review (PR #6102) that were triaged as +**explicitly deferred** — understood, judged not worth acting on for this change, and +recorded here so a later reviewer does not have to rediscover them. + +- **The next-engine declarative writer follows symlinked managed directories on write.** + `apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts`'s next write loop + (`writeNextDeclarativeSchemas`) resolves each proposed file's path and writes it with no + containment check, so a symlinked subdirectory under the declarative dir is written + through, while the read path (`readManagedDeclarativeSqlFiles` in the same file) skips + symlinks outright. Asymmetric, but every path written comes from pg-delta's own export + names (already validated by `safeDeclarativeExportName`), so reaching outside the tree + needs the user to have planted the symlink themselves. Deferred — not needed now. + +- **`generate --output` containment is a lexical guard.** + `apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts` + rejects an output dir that resolves to or contains the project dir using `path.resolve` + + `path.relative` only, so an ancestor symlink can make an escaping path look contained. + The catastrophic outcome (a recursive wipe of the resolved dir) only exists on the legacy + full-wipe writer; the next writer never removes anything it does not own. Deferred. + +- **Shadow host-port allocation probes the wrong host.** + The shadow-database port allocator probes `127.0.0.1` inside the CLI process but the + container publishes on the Docker *daemon's* host, so with a remote `DOCKER_HOST` the + probe can report a free port that is taken remotely (or vice versa). Pre-existing latent + pattern, copied verbatim from + `apps/cli/src/legacy/shared/legacy-edge-runtime-script.layer.ts`; not introduced by this + PR and unreachable for the local-Docker default. Deferred. From c3c464315014390ff57ee12faf7d7587b4d7a5a4 Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 14 Aug 2026 17:49:34 +0200 Subject: [PATCH 28/82] fix(cli): recover legacy declarative extension gaps --- .../schema/declarative/declarative.errors.ts | 3 + .../db/schema/declarative/declarative.flow.ts | 176 +++++++++++- .../declarative/declarative.flow.unit.test.ts | 199 ++++++++++++- .../declarative/declarative.orchestrate.ts | 87 +++++- .../schema/declarative/sync/sync.handler.ts | 269 +++++++++++++----- .../declarative/sync/sync.integration.test.ts | 233 ++++++++++++--- .../legacy-pgdelta-engine.next.layer.ts | 3 + .../legacy-pgdelta-engine.next.unit.test.ts | 25 ++ .../shared/legacy-pgdelta-engine.service.ts | 9 + .../legacy-pgdelta-next-adapter.layer.ts | 26 +- .../legacy-pgdelta-next-adapter.service.ts | 2 + .../legacy-pgdelta-next-adapter.unit.test.ts | 15 + 12 files changed, 918 insertions(+), 129 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts index bda24a9be6..c69dd84ae5 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts @@ -5,6 +5,7 @@ import { type CliErrorActionabilityDeclaration, ErrorActionabilityId, } from "../../../../../shared/telemetry/error-actionability.ts"; +import type { LegacyDeclarativeLoadCompatibilityFinding } from "./declarative.flow.ts"; /** * Declarative commands were invoked without `--experimental` and without @@ -108,6 +109,8 @@ export class LegacyDeclarativeCompatibilityError extends Data.TaggedError( "LegacyDeclarativeCompatibilityError", )<{ readonly message: string; + /** Structured only for a known implicit-extension failure during shadow load. */ + readonly loadFindings?: ReadonlyArray; }> { get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { return actionability.invalidConfig; diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts index 35bd7fc755..ec23554341 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts @@ -4,6 +4,28 @@ import type { LegacyPgDeltaRemovalSummary } from "../../shared/legacy-pgdelta-en /** Extensions that legacy pg-delta treated as part of its implicit Supabase baseline. */ const LEGACY_IMPLICIT_EXTENSIONS = ["pg_net", "pgcrypto", "uuid-ossp"] as const; +export type LegacyDeclarativeImplicitExtension = (typeof LEGACY_IMPLICIT_EXTENSIONS)[number]; + +export interface LegacyDeclarativeLoadDiagnostic { + readonly code: string; + readonly severity: string; + readonly message: string; +} + +export interface LegacyDeclarativeSqlFile { + readonly name: string; + readonly sql: string; +} + +export interface LegacyDeclarativeLoadCompatibilityFinding { + readonly extension: LegacyDeclarativeImplicitExtension; + /** Normalized routine or extension signature matched in the load diagnostic. */ + readonly signature: string; + readonly diagnosticMessage: string; + readonly file?: string; + readonly line?: number; +} + type LegacyDeclarativeCompatibilityAction = "none" | "repair-extensions" | "stage-next-export"; export interface LegacyDeclarativeCompatibilityGap { @@ -76,6 +98,153 @@ export function legacyClassifyDeclarativeCompatibilityGap(opts: { }; } +interface LegacyImplicitExtensionMatch { + readonly extension: LegacyDeclarativeImplicitExtension; + readonly signature: string; + readonly sourcePattern: RegExp; +} + +const nonConvergingLoadDiagnosticCodes = new Set(["stuck_statement", "max_rounds_exceeded"]); + +function matchImplicitExtension(message: string): LegacyImplicitExtensionMatch | undefined { + const uuidRoutine = message.match( + /\bfunction\s+extensions\.(uuid_generate_v[a-zA-Z0-9_]*)\s*\([^)]*\)\s+does not exist\b/i, + ); + const uuidFunction = uuidRoutine?.[1]; + if (uuidFunction !== undefined) { + return { + extension: "uuid-ossp", + signature: `extensions.${uuidFunction}()`, + sourcePattern: new RegExp(`\\bextensions\\s*\\.\\s*${uuidFunction}\\s*\\(`, "i"), + }; + } + + const pgcryptoRoutine = message.match( + /\bfunction\s+extensions\.(digest|crypt|gen_random_bytes|pgp_[a-zA-Z0-9_]*)\s*\([^)]*\)\s+does not exist\b/i, + ); + const pgcryptoFunction = pgcryptoRoutine?.[1]; + if (pgcryptoFunction !== undefined) { + return { + extension: "pgcrypto", + signature: `extensions.${pgcryptoFunction}()`, + sourcePattern: new RegExp(`\\bextensions\\s*\\.\\s*${pgcryptoFunction}\\s*\\(`, "i"), + }; + } + + const pgNetRoutine = message.match( + /\bfunction\s+net\.(http_[a-zA-Z0-9_]*)\s*\([^)]*\)\s+does not exist\b/i, + ); + const pgNetFunction = pgNetRoutine?.[1]; + if (pgNetFunction !== undefined) { + return { + extension: "pg_net", + signature: `net.${pgNetFunction}()`, + sourcePattern: new RegExp(`\\bnet\\s*\\.\\s*${pgNetFunction}\\s*\\(`, "i"), + }; + } + + const missingExtension = message.match( + /\bextension\s+"(pg_net|pgcrypto|uuid-ossp)"\s+does not exist\b/i, + )?.[1]; + if (missingExtension === undefined) return undefined; + const extension = LEGACY_IMPLICIT_EXTENSIONS.find( + (implicit) => implicit === missingExtension.toLowerCase(), + ); + if (extension === undefined) return undefined; + return { + extension, + signature: `extension "${extension}"`, + sourcePattern: new RegExp(`(?:"${extension}"|\\b${extension}\\b)`, "i"), + }; +} + +/** + * Masks SQL comments and strings while preserving offsets. Extension declarations + * are DDL, so occurrences inside comments, quoted values, and dollar bodies must + * not suppress compatibility guidance. + */ +function maskSqlNonCode(sql: string): string { + return sql.replaceAll( + /--[^\r\n]*|\/\*[\s\S]*?\*\/|'(?:''|[^'])*'|\$(?:[a-zA-Z_][\w$]*)?\$[\s\S]*?\$(?:[a-zA-Z_][\w$]*)?\$/g, + (matched) => matched.replaceAll(/[^\r\n]/g, " "), + ); +} + +function maskSqlComments(sql: string): string { + return sql.replaceAll(/--[^\r\n]*|\/\*[\s\S]*?\*\//g, (matched) => + matched.replaceAll(/[^\r\n]/g, " "), + ); +} + +function declaredImplicitExtensions( + files: readonly LegacyDeclarativeSqlFile[], +): ReadonlySet { + const declared = new Set(); + const pattern = + /\bCREATE\s+EXTENSION\s+(?:IF\s+NOT\s+EXISTS\s+)?(?:"([^"]+)"|([a-zA-Z_][\w$-]*))/gi; + for (const file of files) { + for (const match of maskSqlNonCode(file.sql).matchAll(pattern)) { + const extensionName = (match[1] ?? match[2])?.toLowerCase(); + const extension = LEGACY_IMPLICIT_EXTENSIONS.find((implicit) => implicit === extensionName); + if (extension !== undefined) declared.add(extension); + } + } + return declared; +} + +function locateSignature( + files: readonly LegacyDeclarativeSqlFile[], + diagnosticMessage: string, + pattern: RegExp, +): Pick { + const diagnosticFile = files.find((file) => diagnosticMessage.startsWith(`${file.name}:`)); + const candidates = diagnosticFile === undefined ? files : [diagnosticFile]; + for (const file of candidates) { + const match = pattern.exec(maskSqlComments(file.sql)); + if (match?.index === undefined) continue; + return { + file: file.name, + line: file.sql.slice(0, match.index).split(/\r\n|\r|\n/).length, + }; + } + return {}; +} + +/** + * Classifies known legacy implicit-extension misses that prevent a manifestless + * declarative tree from loading on pg-delta next's isolated desired shadow. + */ +export function legacyClassifyDeclarativeLoadCompatibility(opts: { + readonly implementation: LegacyPgDeltaImplementation; + readonly manifestPresent: boolean; + readonly diagnostics: readonly LegacyDeclarativeLoadDiagnostic[]; + readonly files: readonly LegacyDeclarativeSqlFile[]; +}): ReadonlyArray { + if (opts.implementation !== "next" || opts.manifestPresent) return []; + + const declared = declaredImplicitExtensions(opts.files); + const findings: LegacyDeclarativeLoadCompatibilityFinding[] = []; + const seen = new Set(); + for (const diagnostic of opts.diagnostics) { + if (diagnostic.severity !== "error" || !nonConvergingLoadDiagnosticCodes.has(diagnostic.code)) { + continue; + } + const match = matchImplicitExtension(diagnostic.message); + if (match === undefined || declared.has(match.extension)) continue; + const location = locateSignature(opts.files, diagnostic.message, match.sourcePattern); + const key = `${match.extension}\0${match.signature}\0${location.file ?? ""}\0${location.line ?? ""}`; + if (seen.has(key)) continue; + seen.add(key); + findings.push({ + extension: match.extension, + signature: match.signature, + diagnosticMessage: diagnostic.message, + ...location, + }); + } + return findings; +} + export const legacyExtensionDeclaration = (extension: string): string => `CREATE EXTENSION IF NOT EXISTS "${extension}" WITH SCHEMA "extensions";`; @@ -101,6 +270,11 @@ export function legacyFormatStagedExportRecommendation( "WARNING: pg-delta next manages schema state that the legacy export did not represent.", ...detected, "Generate a next-compatible schema into a separate directory, review it, and adopt it when ready:", - "supabase db schema declarative generate --output supabase/database-next", + " supabase db schema declarative generate --local --overwrite \\", + " --output supabase/database-next --experimental", + "", + " # review supabase/database-next", + " rm -rf supabase/database && mv supabase/database-next supabase/database", + " supabase db schema declarative sync --no-apply --experimental", ].join("\n"); } diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts index d84756e0a3..f57358b595 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts @@ -2,12 +2,19 @@ import { describe, expect, it } from "vitest"; import { legacyClassifyDeclarativeCompatibilityGap, + legacyClassifyDeclarativeLoadCompatibility, legacyExtensionDeclaration, legacyFormatStagedExportRecommendation, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, } from "./declarative.flow.ts"; +const stuck = (message: string) => ({ + code: "stuck_statement", + severity: "error", + message, +}); + const removals = { extensions: ["pgcrypto", "uuid-ossp"], extensionIntents: [ @@ -53,7 +60,7 @@ describe("legacyClassifyDeclarativeCompatibilityGap", () => { }); expect(gap.recommendedAction).toBe("stage-next-export"); expect(legacyFormatStagedExportRecommendation(gap)).toContain( - "generate --output supabase/database-next", + "generate --local --overwrite \\\n --output supabase/database-next --experimental", ); }); @@ -85,6 +92,196 @@ describe("legacyClassifyDeclarativeCompatibilityGap", () => { }); }); +describe("legacyClassifyDeclarativeLoadCompatibility", () => { + it.each([ + ["extensions.uuid_generate_v4()", "uuid-ossp"], + ["extensions.digest(text, text)", "pgcrypto"], + ["extensions.crypt(text, text)", "pgcrypto"], + ["extensions.gen_random_bytes(integer)", "pgcrypto"], + ["extensions.pgp_sym_encrypt(text, text)", "pgcrypto"], + ["net.http_post(text, jsonb)", "pg_net"], + ])("maps a missing %s routine to %s", (routine, extension) => { + const symbol = routine.slice(0, routine.indexOf("(")); + const findings = legacyClassifyDeclarativeLoadCompatibility({ + implementation: "next", + manifestPresent: false, + diagnostics: [ + stuck( + `schemas/app/tables/members.sql: ERROR: function ${routine} does not exist (failed identically in 6 rounds)`, + ), + ], + files: [ + { + name: "schemas/app/tables/members.sql", + sql: `create table app.members (\n id uuid default ${symbol}()\n);`, + }, + ], + }); + + expect(findings).toEqual([ + { + extension, + signature: `${symbol}()`, + diagnosticMessage: `schemas/app/tables/members.sql: ERROR: function ${routine} does not exist (failed identically in 6 rounds)`, + file: "schemas/app/tables/members.sql", + line: 2, + }, + ]); + }); + + it.each(["uuid-ossp", "pgcrypto", "pg_net"])( + "maps a direct missing %s extension diagnostic", + (extension) => { + expect( + legacyClassifyDeclarativeLoadCompatibility({ + implementation: "next", + manifestPresent: false, + diagnostics: [ + stuck(`cluster/config.sql: ERROR: extension "${extension}" does not exist`), + ], + files: [ + { + name: "cluster/config.sql", + sql: `alter extension "${extension}" update;`, + }, + ], + }), + ).toEqual([ + expect.objectContaining({ + extension, + signature: `extension "${extension}"`, + file: "cluster/config.sql", + line: 1, + }), + ]); + }, + ); + + it("reports the authored line from the diagnostic's file and ignores cascades", () => { + const findings = legacyClassifyDeclarativeLoadCompatibility({ + implementation: "next", + manifestPresent: false, + diagnostics: [ + stuck( + "schemas/app/tables/members.sql: ERROR: function extensions.uuid_generate_v4() does not exist", + ), + stuck('public.views/members.sql: ERROR: relation "app.members" does not exist'), + ], + files: [ + { + name: "other.sql", + sql: "select extensions.uuid_generate_v4();", + }, + { + name: "schemas/app/tables/members.sql", + sql: "-- generated table uses extensions.uuid_generate_v4()\r\n\r\ncreate table app.members (\r\n id uuid default extensions.uuid_generate_v4()\r\n);", + }, + { + name: "public.views/members.sql", + sql: "create view public.members as select * from app.members;", + }, + ], + }); + + expect(findings).toHaveLength(1); + expect(findings[0]).toMatchObject({ + extension: "uuid-ossp", + file: "schemas/app/tables/members.sql", + line: 4, + }); + }); + + it("requires next, no manifest, and an error-level non-converging diagnostic", () => { + const files = [{ name: "members.sql", sql: "select extensions.uuid_generate_v4();" }]; + const diagnostic = stuck("members.sql: function extensions.uuid_generate_v4() does not exist"); + const classify = ( + implementation: "legacy" | "next", + manifestPresent: boolean, + diagnostics: ReadonlyArray<{ code: string; severity: string; message: string }>, + ) => + legacyClassifyDeclarativeLoadCompatibility({ + implementation, + manifestPresent, + diagnostics, + files, + }); + + expect(classify("legacy", false, [diagnostic])).toEqual([]); + expect(classify("next", true, [diagnostic])).toEqual([]); + expect(classify("next", false, [{ ...diagnostic, severity: "warning" }])).toEqual([]); + expect(classify("next", false, [{ ...diagnostic, code: "invalid_routine_body" }])).toEqual([]); + expect(classify("next", false, [{ ...diagnostic, code: "max_rounds_exceeded" }])).toHaveLength( + 1, + ); + }); + + it("does not classify an extension already declared anywhere in the tree", () => { + expect( + legacyClassifyDeclarativeLoadCompatibility({ + implementation: "next", + manifestPresent: false, + diagnostics: [stuck("members.sql: function extensions.uuid_generate_v4() does not exist")], + files: [ + { name: "members.sql", sql: "select extensions.uuid_generate_v4();" }, + { + name: "cluster/extensions/uuid-ossp.sql", + sql: 'create extension if not exists "uuid-ossp" with schema "extensions";', + }, + ], + }), + ).toEqual([]); + }); + + it("does not treat a commented or quoted declaration as an extension declaration", () => { + expect( + legacyClassifyDeclarativeLoadCompatibility({ + implementation: "next", + manifestPresent: false, + diagnostics: [stuck("members.sql: function extensions.uuid_generate_v4() does not exist")], + files: [ + { + name: "members.sql", + sql: [ + '-- create extension "uuid-ossp";', + "select 'create extension uuid-ossp';", + "select extensions.uuid_generate_v4();", + ].join("\n"), + }, + ], + }), + ).toEqual([expect.objectContaining({ extension: "uuid-ossp", file: "members.sql", line: 3 })]); + }); + + it("returns an unlocated finding when the diagnostic has no authored match", () => { + expect( + legacyClassifyDeclarativeLoadCompatibility({ + implementation: "next", + manifestPresent: false, + diagnostics: [stuck('unknown.sql: extension "pg_net" does not exist')], + files: [], + }), + ).toEqual([ + { + extension: "pg_net", + signature: 'extension "pg_net"', + diagnosticMessage: 'unknown.sql: extension "pg_net" does not exist', + }, + ]); + }); + + it("deduplicates identical findings from repeated load diagnostics", () => { + const diagnostic = stuck("members.sql: function extensions.uuid_generate_v4() does not exist"); + expect( + legacyClassifyDeclarativeLoadCompatibility({ + implementation: "next", + manifestPresent: false, + diagnostics: [diagnostic, diagnostic], + files: [{ name: "members.sql", sql: "select extensions.uuid_generate_v4();" }], + }), + ).toHaveLength(1); + }); +}); + describe("legacyResolveDeclarativeMigrationName", () => { it("prefers an explicit --name over --file", () => { expect(legacyResolveDeclarativeMigrationName("my_change", "declarative_sync")).toBe( diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 3fc73b33a7..1ac5234231 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -27,7 +27,15 @@ import { LegacyLoadPgDeltaSqlFiles, LegacyReadPgDeltaExportManifest, } from "../../shared/legacy-pgdelta-files.ts"; -import { LegacyDeclarativeDiffError } from "./declarative.errors.ts"; +import { + LegacyDeclarativeCompatibilityError, + LegacyDeclarativeDiffError, +} from "./declarative.errors.ts"; +import { + legacyClassifyDeclarativeLoadCompatibility, + legacyExtensionDeclaration, + type LegacyDeclarativeLoadCompatibilityFinding, +} from "./declarative.flow.ts"; /** Ambient inputs shared by the orchestration steps. */ export interface LegacyDeclarativeRunContext { @@ -55,6 +63,40 @@ export interface LegacyDeclarativeSyncResult { const declarativeError = (message: string) => new LegacyDeclarativeDiffError({ message }); +const formatImplicitExtensionLoadFailure = ( + findings: ReadonlyArray, +): string => { + const extensions = [...new Set(findings.map((finding) => finding.extension))].sort(); + const detected = findings.map((finding) => { + const location = + finding.file === undefined + ? "A declarative schema file" + : `${finding.file}${finding.line === undefined ? "" : `:${finding.line}`}`; + return `${location} uses ${finding.signature}, but the tree does not declare ${finding.extension}.`; + }); + return [ + "This declarative schema looks like a legacy pg-delta export.", + "", + ...detected, + "", + "pg-delta next loads desired state onto a shadow that only has extensions you declare. Legacy generate omitted platform extensions.", + "", + "Recommended — generate a next-compatible tree, review it, then adopt:", + "", + " supabase db schema declarative generate --local --overwrite \\", + " --output supabase/database-next --experimental", + "", + " # review supabase/database-next", + " rm -rf supabase/database && mv supabase/database-next supabase/database", + " supabase db schema declarative sync --no-apply --experimental", + "", + "Alternative — add the missing extension declarations to extension.sql, then re-plan:", + ...extensions.map((extension) => legacyExtensionDeclaration(extension)), + "", + " supabase db schema declarative sync --no-apply --experimental", + ].join("\n"); +}; + /** * Computes the diff between local migrations state and the declarative schema. * Mirrors Go's `DiffDeclarativeToMigrations` (`declarative.go:170`): the @@ -95,19 +137,36 @@ export const legacyDiffDeclarativeToMigrations = Effect.fnUntraced(function* ( Effect.mapError((error) => declarativeError(error.message)), ) : undefined; - const result = yield* engine.planDeclarativeSchema({ - context: run.pgDelta, - schema: run.schema, - formatOptions: run.formatOptions, - debug: run.debug, - strictCoverage: run.strictCoverage, - files, - noCache: run.noCache, - toml, - setupInputs, - ...(run.linkedProjectRef !== undefined ? { projectRef: run.linkedProjectRef } : {}), - ...(manifest !== undefined ? { manifest } : {}), - }); + const result = yield* engine + .planDeclarativeSchema({ + context: run.pgDelta, + schema: run.schema, + formatOptions: run.formatOptions, + debug: run.debug, + strictCoverage: run.strictCoverage, + files, + noCache: run.noCache, + toml, + setupInputs, + ...(run.linkedProjectRef !== undefined ? { projectRef: run.linkedProjectRef } : {}), + ...(manifest !== undefined ? { manifest } : {}), + }) + .pipe( + Effect.mapError((error) => { + const findings = legacyClassifyDeclarativeLoadCompatibility({ + implementation: engine.implementation, + manifestPresent: manifest !== undefined, + diagnostics: error.diagnostics ?? [], + files, + }); + return findings.length === 0 + ? error + : new LegacyDeclarativeCompatibilityError({ + message: formatImplicitExtensionLoadFailure(findings), + loadFindings: findings, + }); + }), + ); return { diffSQL: result.sql, files: result.files, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 3469463a96..540a5f9405 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -35,7 +35,10 @@ import { legacyResolvePgDeltaProjectId, } from "../../../../../shared/legacy-pgdelta.ts"; import { legacyWritePgDeltaMigrations } from "../../../shared/legacy-pgdelta-migrations.write.ts"; -import { legacyResolveSmartTargetEndpoint } from "../declarative.smart-target.ts"; +import { + legacyLocalEndpoint, + legacyResolveSmartTargetEndpoint, +} from "../declarative.smart-target.ts"; import { type LegacyDebugBundle, legacyCollectMigrationsList, @@ -304,52 +307,192 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara Option.getOrUndefined(toml.orioledbVersion), toml.baseline, ); + const stageNextExport = Effect.fnUntraced(function* () { + const stagedDirRel = "supabase/database-next"; + const stagedDir = path.resolve(cliConfig.workdir, stagedDirRel); + if (stagedDir === declarativeDir) { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ + message: `${stagedDirRel} is the active declarative schema directory; choose a different staging directory.`, + }), + ); + } + const stagedExists = yield* fs.exists(stagedDir).pipe(Effect.orElseSucceed(() => false)); + if (stagedExists) { + const [entries, hasManifest] = yield* Effect.all([ + fs.readDirectory(stagedDir), + fs.exists(path.join(stagedDir, ".pgdelta-export.json")), + ]); + if (entries.length > 0 && !hasManifest) { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ + message: `${stagedDirRel} already contains files without a pg-delta export manifest. Move or remove that directory, then run sync again so the staged export cannot preserve unrelated SQL.`, + }), + ); + } + } + yield* ensureLocalPostgresImageCurrent; + yield* seam.ensureLocalDatabaseStarted(); + const generated = yield* legacyGenerateDeclarativeOutput( + { ...run, declarativeDir: stagedDir }, + toml, + legacyLocalEndpoint({ port: toml.port, password: toml.password }, dnsResolver), + ); + const written = yield* legacyWriteDeclarativeSchemas(fs, path, stagedDir, generated); + yield* legacyWarnPreservedUnmanagedDeclarativeFiles(stagedDirRel, written); + yield* output.raw(legacyDeclarativeSchemaWrittenLine(stagedDirRel), "stderr"); + yield* output.raw( + [ + "Review supabase/database-next, then adopt it:", + " rm -rf supabase/database && mv supabase/database-next supabase/database", + " supabase db schema declarative sync --no-apply --experimental", + "", + ].join("\n"), + "stderr", + ); + }); + const planDeclarativeSync = () => legacyDiffDeclarativeToMigrations(run, toml, setupInputs).pipe( Effect.tapError((error) => - Effect.gen(function* () { - const migrations = yield* legacyCollectMigrationsList(fs, path, migrationsDir); - yield* legacySaveDebugBundle(fs, path, cliConfig.workdir, tempDir, migrationsDir, { - id: formatDebugId(yield* Clock.currentTimeMillis), - error: error.message, - migrations, - }).pipe( - Effect.matchEffect({ - // Go prints nothing when SaveDebugBundle errors on the diff path - // (`db_schema_declarative.go:337-340`: `if saveErr == nil`). - onFailure: () => Effect.void, - onSuccess: (debugDir) => output.raw(legacyDebugBundleMessage(debugDir), "stderr"), + error instanceof LegacyDeclarativeCompatibilityError + ? Effect.void + : Effect.gen(function* () { + const migrations = yield* legacyCollectMigrationsList(fs, path, migrationsDir); + yield* legacySaveDebugBundle( + fs, + path, + cliConfig.workdir, + tempDir, + migrationsDir, + { + id: formatDebugId(yield* Clock.currentTimeMillis), + error: error.message, + migrations, + }, + ).pipe( + Effect.matchEffect({ + // Go prints nothing when SaveDebugBundle errors on the diff path + // (`db_schema_declarative.go:337-340`: `if saveErr == nil`). + onFailure: () => Effect.void, + onSuccess: (debugDir) => + output.raw(legacyDebugBundleMessage(debugDir), "stderr"), + }), + ); }), - ); - }), ), ); - let result: LegacyDeclarativeSyncResult = yield* planDeclarativeSync(); - - // Resolve manifest-less legacy compatibility before printing or writing a - // migration. A repair is always explicit, even when global --yes is set. - if ( - engine.implementation === "next" && - !result.manifestPresent && - !toml.webhooksEnabled && - result.removals.extensions.includes("pg_net") - ) { - return yield* Effect.fail( - new LegacyDeclarativeCompatibilityError({ - message: [ - "The migrations state includes pg_net, but Database Webhooks are not enabled in the local project config.", - "", - LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION, - ].join("\n"), - }), - ); - } - const compatibility = legacyClassifyDeclarativeCompatibilityGap({ - implementation: engine.implementation, - manifestPresent: result.manifestPresent, - removals: result.removals, + + const planWithLoadRecovery = Effect.fnUntraced(function* () { + while (true) { + const attempt = yield* planDeclarativeSync().pipe( + Effect.match({ + onFailure: (error) => ({ error }), + onSuccess: (result) => ({ result }), + }), + ); + if ("result" in attempt) return Option.some(attempt.result); + const error = attempt.error; + if ( + !(error instanceof LegacyDeclarativeCompatibilityError) || + error.loadFindings === undefined + ) { + return yield* Effect.fail(error); + } + + const missingExtensions = [ + ...new Set(error.loadFindings.map((finding) => finding.extension)), + ].sort(); + if (missingExtensions.includes("pg_net") && !toml.webhooksEnabled) { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ + message: [ + "The declarative schema uses pg_net, but Database Webhooks are not enabled in the local project config.", + "", + LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION, + ].join("\n"), + }), + ); + } + if (!tty.stdinIsTty || yes) return yield* Effect.fail(error); + + yield* output.raw(`${legacyYellow(error.message)}\n`, "stderr"); + const choice = yield* output.promptSelect("How would you like to continue?", [ + { + value: "stage", + label: "Generate next export to supabase/database-next", + hint: "recommended", + }, + { value: "repair", label: "Add missing extension declarations and re-plan" }, + { value: "cancel", label: "Cancel" }, + ]); + if (choice === "cancel") return Option.none(); + if (choice === "stage") { + yield* stageNextExport(); + return Option.none(); + } + const repaired = yield* legacyAppendExtensionDeclarations( + declarativeDir, + missingExtensions, + ); + yield* output.raw( + `Updated ${legacyBold(repaired.path)} with:\n${repaired.addedDeclarations.join("\n")}\n`, + "stderr", + ); + } }); - if (compatibility.recommendedAction === "repair-extensions") { + + const initialResult = yield* planWithLoadRecovery(); + if (Option.isNone(initialResult)) return; + let result: LegacyDeclarativeSyncResult = initialResult.value; + + // Resolve successful manifest-less plans too. Repairs re-enter planning so a + // second, broader legacy gap (for example cron intents) cannot fall through to + // migration writing after the first missing extension is declared. + while (true) { + if ( + engine.implementation === "next" && + !result.manifestPresent && + !toml.webhooksEnabled && + result.removals.extensions.includes("pg_net") + ) { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ + message: [ + "The migrations state includes pg_net, but Database Webhooks are not enabled in the local project config.", + "", + LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION, + ].join("\n"), + }), + ); + } + const compatibility = legacyClassifyDeclarativeCompatibilityGap({ + implementation: engine.implementation, + manifestPresent: result.manifestPresent, + removals: result.removals, + }); + if (compatibility.recommendedAction === "none") break; + + if (compatibility.recommendedAction === "stage-next-export") { + const explanation = legacyFormatStagedExportRecommendation(compatibility); + if (!tty.stdinIsTty || yes) { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ message: explanation }), + ); + } + yield* output.raw(`${legacyYellow(explanation)}\n`, "stderr"); + const choice = yield* output.promptSelect("How would you like to continue?", [ + { + value: "stage", + label: "Generate next export to supabase/database-next", + hint: "recommended", + }, + { value: "cancel", label: "Cancel" }, + ]); + if (choice === "stage") yield* stageNextExport(); + return; + } + const statements = compatibility.repairableExtensions.map(legacyExtensionDeclaration); const explanation = [ "This declarative schema appears to use legacy pg-delta behavior. Legacy pg-delta treated these installed extensions as implicit, while pg-delta next treats their omission as removal:", @@ -365,8 +508,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara "Non-interactive sync will not modify the declarative schema automatically. Add these statements to extension.sql, then run sync again:", ...statements, "", - "Or generate a next-compatible schema into a separate directory:", - "supabase db schema declarative generate --output supabase/database-next", + legacyFormatStagedExportRecommendation(compatibility), ].join("\n"), }), ); @@ -374,46 +516,23 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara yield* output.raw(`${legacyYellow(explanation)}\n`, "stderr"); const choice = yield* output.promptSelect("How would you like to continue?", [ - { - value: "repair", - label: "Add declarations and re-plan", - hint: "recommended", - }, + { value: "repair", label: "Add declarations and re-plan", hint: "recommended" }, { value: "continue", label: "Continue with removals" }, { value: "cancel", label: "Cancel" }, ]); if (choice === "cancel") return; - if (choice === "repair") { - const repaired = yield* legacyAppendExtensionDeclarations( - declarativeDir, - compatibility.repairableExtensions, - ); - yield* output.raw( - `Updated ${legacyBold(repaired.path)} with:\n${repaired.addedDeclarations.join("\n")}\n`, - "stderr", - ); - result = yield* planDeclarativeSync(); - const remaining = legacyClassifyDeclarativeCompatibilityGap({ - implementation: engine.implementation, - manifestPresent: result.manifestPresent, - removals: result.removals, - }); - if (remaining.recommendedAction !== "none") { - return yield* Effect.fail( - new LegacyDeclarativeCompatibilityError({ - message: [ - "The compatibility removals remain after adding extension declarations.", - legacyFormatStagedExportRecommendation(remaining), - ].join("\n"), - }), - ); - } - } - } else if (compatibility.recommendedAction === "stage-next-export") { + if (choice === "continue") break; + const repaired = yield* legacyAppendExtensionDeclarations( + declarativeDir, + compatibility.repairableExtensions, + ); yield* output.raw( - `${legacyYellow(legacyFormatStagedExportRecommendation(compatibility))}\n`, + `Updated ${legacyBold(repaired.path)} with:\n${repaired.addedDeclarations.join("\n")}\n`, "stderr", ); + const replanned = yield* planWithLoadRecovery(); + if (Option.isNone(replanned)) return; + result = replanned.value; } // Step 3: empty diff. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index b8bcda0ab7..fc379824eb 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -50,6 +50,7 @@ import { LegacyPgDeltaSslProbe } from "../../../../../shared/legacy-pgdelta-ssl- import { legacyPgDeltaLegacyEngineLayer } from "../../../shared/legacy-pgdelta-engine.legacy.layer.ts"; import { LegacyPgDeltaEngine, + LegacyPgDeltaEngineError, type LegacyPgDeltaRemovalSummary, type LegacyPgDeltaRenderedFile, } from "../../../shared/legacy-pgdelta-engine.service.ts"; @@ -95,6 +96,7 @@ interface SetupOpts { engineImplementation?: "legacy" | "next"; renderedFiles?: ReadonlyArray; removals?: LegacyPgDeltaRemovalSummary; + planErrors?: ReadonlyArray; } function setup(workdir: string, opts: SetupOpts = {}) { @@ -265,6 +267,9 @@ function setup(workdir: string, opts: SetupOpts = {}) { child.layer, ); const nextFiles = opts.renderedFiles ?? []; + const planErrors = [...(opts.planErrors ?? [])]; + let planCalls = 0; + const declarativeExportCalls: Array = []; const engine = opts.engineImplementation === "next" ? Layer.succeed( @@ -274,13 +279,19 @@ function setup(workdir: string, opts: SetupOpts = {}) { diffExplicit: () => Effect.die("diffExplicit not used in sync tests"), diffDatabase: () => Effect.die("diffDatabase not used in sync tests"), exportDeclarativeSchema: () => - Effect.succeed({ - files: [ - { name: "schemas/public/tables/players.sql", sql: "create table players ();" }, - ], - manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, + Effect.sync(() => { + declarativeExportCalls.push(true); + return { + files: [ + { name: "schemas/public/tables/players.sql", sql: "create table players ();" }, + ], + manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, + }; }), planDeclarativeSchema: () => { + planCalls += 1; + const planError = planErrors.shift(); + if (planError !== undefined) return Effect.fail(planError); const extensionPath = join(workdir, "supabase", "database", "extension.sql"); const extensionSql = existsSync(extensionPath) ? readFileSync(extensionPath, "utf8") @@ -352,6 +363,10 @@ function setup(workdir: string, opts: SetupOpts = {}) { telemetry, localPostgresImageChecks, exportCatalogCalls, + declarativeExportCalls, + get planCalls() { + return planCalls; + }, }; } @@ -376,6 +391,45 @@ const seedDeclarative = (workdir: string) => { writeFileSync(join(dir, "public.sql"), "create table a();"); }; +const seedLegacyUuidDeclarative = (workdir: string) => { + const dir = join(workdir, "supabase", "database"); + mkdirSync(join(dir, "schemas", "app", "tables"), { recursive: true }); + mkdirSync(join(dir, "schemas", "public", "views"), { recursive: true }); + writeFileSync( + join(dir, "schemas", "app", "tables", "members.sql"), + [ + "create table app.members (", + " email text not null,", + " id uuid not null default extensions.uuid_generate_v4()", + ");", + ].join("\n"), + ); + writeFileSync( + join(dir, "schemas", "public", "views", "members.sql"), + "create view public.members as select * from app.members;\n", + ); +}; + +const legacyUuidLoadError = () => + new LegacyPgDeltaEngineError({ + message: + "Declarative schema planning failed: shadow load stuck. Tip: split circular REFERENCES clauses.", + cause: new Error("shadow load stuck"), + diagnostics: [ + { + code: "stuck_statement", + severity: "error", + message: + "0001__schemas/app/tables/members.sql: function extensions.uuid_generate_v4() does not exist (failed identically in 6 rounds)", + }, + { + code: "stuck_statement", + severity: "error", + message: '0002__schemas/public/views/members.sql: relation "app.members" does not exist', + }, + ], + }); + describe("legacy db schema declarative sync integration", () => { const tmp = useLegacyTempWorkdir(); @@ -976,36 +1030,147 @@ describe("legacy db schema declarative sync integration", () => { }, ); - it.effect( - "recommends a staged next export before writing for extension-managed legacy gaps", - () => { - seedDeclarative(tmp.current); - const s = setup(tmp.current, { - experimental: true, - engineImplementation: "next", - diffSql: - "select cron.unschedule('refresh download metrics');\nDROP EXTENSION \"pgcrypto\";\n", - removals: { - extensions: ["pgcrypto", "uuid-ossp"], - extensionIntents: [ - { extension: "pg_cron", intentKind: "job", key: "refresh download metrics" }, - ], - }, + it.effect("refuses a known implicit-extension load failure under --yes", () => { + seedLegacyUuidDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + yes: true, + planErrors: [legacyUuidLoadError()], + }); + return Effect.gen(function* () { + const exit = yield* legacyDbSchemaDeclarativeSync(flags()).pipe(Effect.exit); + expect(failError(exit)).toMatchObject({ + _tag: "LegacyDeclarativeCompatibilityError", + message: expect.stringContaining("schemas/app/tables/members.sql:3"), }); - return Effect.gen(function* () { - yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); - const chunks = s.out.rawChunks.map((chunk) => stripAnsi(chunk.text)); - const warningAt = chunks.findIndex((chunk) => - chunk.includes("legacy export did not represent"), - ); - const createdAt = chunks.findIndex((chunk) => chunk.includes("Created new migration at")); - expect(warningAt).toBeGreaterThan(-1); - expect(chunks[warningAt]).toContain("pg_cron job refresh download metrics"); - expect(chunks[warningAt]).toContain("--output supabase/database-next"); - expect(warningAt).toBeLessThan(createdAt); - }).pipe(Effect.provide(s.layer)); - }, - ); + const error = failError(exit); + expect(error).toMatchObject({ + message: expect.stringContaining("uuid-ossp"), + }); + expect(JSON.stringify(error)).toContain( + "supabase db schema declarative generate --local --overwrite", + ); + expect(s.out.promptSelectCalls).toHaveLength(0); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); + const output = stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join("")); + expect(output).not.toContain("pg-toolbelt/issues"); + expect(output).not.toContain("circular REFERENCES"); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("offers adopt, repair, or cancel before a load-failure plan exists", () => { + seedLegacyUuidDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + planErrors: [legacyUuidLoadError()], + promptSelectResponses: ["cancel"], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + expect(s.out.promptSelectCalls[0]?.options).toEqual([ + expect.objectContaining({ value: "stage", hint: "recommended" }), + expect.objectContaining({ value: "repair" }), + expect.objectContaining({ value: "cancel" }), + ]); + expect( + s.out.promptSelectCalls[0]?.options.some((option) => option.value === "continue"), + ).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("adds a missing load-time extension declaration and re-plans", () => { + seedLegacyUuidDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + planErrors: [legacyUuidLoadError()], + promptSelectResponses: ["repair"], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + expect(s.planCalls).toBe(2); + expect(readFileSync(join(tmp.current, "supabase", "database", "extension.sql"), "utf8")).toBe( + 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";\n', + ); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join(""))).toContain( + "No schema changes found", + ); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("stages a complete next export without changing the active tree", () => { + seedLegacyUuidDeclarative(tmp.current); + const activeMember = join( + tmp.current, + "supabase", + "database", + "schemas", + "app", + "tables", + "members.sql", + ); + const before = readFileSync(activeMember, "utf8"); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + planErrors: [legacyUuidLoadError()], + promptSelectResponses: ["stage"], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + expect(s.declarativeExportCalls).toHaveLength(1); + expect(readFileSync(activeMember, "utf8")).toBe(before); + expect( + readFileSync( + join( + tmp.current, + "supabase", + "database-next", + "schemas", + "public", + "tables", + "players.sql", + ), + "utf8", + ), + ).toBe("create table players ();"); + expect( + existsSync(join(tmp.current, "supabase", "database-next", ".pgdelta-export.json")), + ).toBe(true); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("refuses extension-managed legacy gaps under --yes instead of writing drops", () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + experimental: true, + engineImplementation: "next", + yes: true, + diffSql: + "select cron.unschedule('refresh download metrics');\nDROP EXTENSION \"pgcrypto\";\n", + removals: { + extensions: ["pgcrypto", "uuid-ossp"], + extensionIntents: [ + { extension: "pg_cron", intentKind: "job", key: "refresh download metrics" }, + ], + }, + }); + return Effect.gen(function* () { + const exit = yield* legacyDbSchemaDeclarativeSync(flags()).pipe(Effect.exit); + expect(failError(exit)).toMatchObject({ + _tag: "LegacyDeclarativeCompatibilityError", + message: expect.stringContaining("pg_cron job refresh download metrics"), + }); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(s.dbExec).toEqual([]); + }).pipe(Effect.provide(s.layer)); + }); it.effect("directs pg_net users to enable Database Webhooks before writing", () => { seedDeclarative(tmp.current); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index a122bde2b5..03c3d4e01c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -17,6 +17,7 @@ import { } from "./legacy-pgdelta-engine.service.ts"; import { LegacyPgDeltaNextAdapter, + LegacyPgDeltaNextError, type LegacyPgDeltaNextOperation, } from "./legacy-pgdelta-next-adapter.service.ts"; import { @@ -47,6 +48,7 @@ function legacyPgDeltaNextConnectSuggestion(cause: unknown): string | undefined export const legacyPgDeltaNextEngineError = (cause: unknown) => { if (cause instanceof LegacyPgDeltaEngineError) return cause; const suggestion = legacyPgDeltaNextConnectSuggestion(cause); + const diagnostics = cause instanceof LegacyPgDeltaNextError ? cause.diagnostics : undefined; return new LegacyPgDeltaEngineError({ message: typeof cause === "object" && @@ -56,6 +58,7 @@ export const legacyPgDeltaNextEngineError = (cause: unknown) => { : String(cause), cause, ...(suggestion !== undefined ? { suggestion } : {}), + ...(diagnostics !== undefined ? { diagnostics } : {}), }); }; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts index d1db522f46..75a50a6d4c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts @@ -37,6 +37,31 @@ describe("pg-delta next engine errors", () => { ); }); + it("preserves structured diagnostics from adapter failures", () => { + const diagnostics: NonNullable = [ + { + code: "stuck_statement", + severity: "error", + message: "schemas/app/tables/members.sql: function does not exist", + context: { rounds: 6 }, + }, + ]; + const adapterError = new LegacyPgDeltaNextError({ + operation: "declarativePlan", + message: "Declarative schema planning failed", + cause: new Error("shadow load failed"), + diagnostics, + }); + + expect(legacyPgDeltaNextEngineError(adapterError)).toEqual( + new LegacyPgDeltaEngineError({ + message: "Declarative schema planning failed", + cause: adapterError, + diagnostics, + }), + ); + }); + it("does not wrap an existing engine error again", () => { const error = new LegacyPgDeltaEngineError({ message: "blocked", cause: "diagnostic" }); expect(legacyPgDeltaNextEngineError(error)).toBe(error); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts index 9a0724dae4..7f71f6814a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.service.ts @@ -163,10 +163,19 @@ interface LegacyPgDeltaDeclarativePlanResult extends LegacyPgDeltaDiffResult { readonly targetRef: string; } +/** Engine-neutral diagnostic detail retained when an implementation reports a structured failure. */ +export interface LegacyPgDeltaErrorDiagnostic { + readonly code: string; + readonly severity: "error" | "warning" | "info"; + readonly message: string; + readonly context?: Readonly>; +} + export class LegacyPgDeltaEngineError extends Data.TaggedError("LegacyPgDeltaEngineError")<{ readonly message: string; readonly cause: unknown; readonly suggestion?: string; + readonly diagnostics?: readonly LegacyPgDeltaErrorDiagnostic[]; }> { get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { return actionability.dbFinding; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index 9fbbe81899..3e24be26b3 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -39,7 +39,10 @@ import { type LegacyPgDeltaNextSqlFile, type LegacyPgDeltaNextOperation, } from "./legacy-pgdelta-next-adapter.service.ts"; -import type { LegacyPgDeltaRemovalSummary } from "./legacy-pgdelta-engine.service.ts"; +import type { + LegacyPgDeltaErrorDiagnostic, + LegacyPgDeltaRemovalSummary, +} from "./legacy-pgdelta-engine.service.ts"; import { LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE } from "./legacy-pgdelta-next-diagnostics.ts"; interface LegacyPgDeltaNextLibraryDiagnostic { @@ -201,18 +204,33 @@ function legacyPgDeltaNextMessage(operation: LegacyPgDeltaNextOperation, cause: return `${label} failed: ${detail}${renderedDiagnostics === "" ? "" : `\n${renderedDiagnostics}`}`; } +function legacyPgDeltaNextErrorDiagnostics( + cause: unknown, +): readonly LegacyPgDeltaErrorDiagnostic[] | undefined { + if (!(cause instanceof ShadowLoadError)) return undefined; + return cause.details.map((diagnostic) => ({ + code: diagnostic.code, + severity: diagnostic.severity, + message: diagnostic.message, + ...(diagnostic.context !== undefined ? { context: { ...diagnostic.context } } : {}), + })); +} + function legacyTryPgDeltaNext( operation: LegacyPgDeltaNextOperation, run: () => Promise, ) { return Effect.tryPromise({ try: run, - catch: (cause) => - new LegacyPgDeltaNextError({ + catch: (cause) => { + const diagnostics = legacyPgDeltaNextErrorDiagnostics(cause); + return new LegacyPgDeltaNextError({ operation, message: legacyPgDeltaNextMessage(operation, cause), cause, - }), + ...(diagnostics !== undefined ? { diagnostics } : {}), + }); + }, }); } diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts index 8a470e396a..190f864487 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts @@ -8,6 +8,7 @@ import { } from "../../../../shared/telemetry/error-actionability.ts"; import type { LegacyMigrationTransactionMode } from "../../../shared/legacy-migration-file.ts"; import type { + LegacyPgDeltaErrorDiagnostic, LegacyPgDeltaHazardKind, LegacyPgDeltaHazardReport, LegacyPgDeltaRemovalSummary, @@ -189,6 +190,7 @@ export class LegacyPgDeltaNextError extends Data.TaggedError("LegacyPgDeltaNextE readonly operation: LegacyPgDeltaNextOperation; readonly message: string; readonly cause: unknown; + readonly diagnostics?: readonly LegacyPgDeltaErrorDiagnostic[]; }> { get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { return actionability.dbFinding; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index 6e7a821876..ca7b5f2e74 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -883,6 +883,7 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(error.operation).toBe("diff"); expect(error.message).toBe("Database diff failed: connection refused for desired database"); expect(error.cause).toBe(cause); + expect(error.diagnostics).toBeUndefined(); yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); }).pipe(Effect.provide(failingLayer)); }); @@ -895,6 +896,7 @@ describe("LegacyPgDeltaNextAdapter", () => { code: "stuck_statement", severity: "error", message: 'extensions/pg_cron.sql: extension "pg_cron" already exists', + context: { rounds: 6 }, }, { code: "stuck_statement", @@ -945,6 +947,19 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(error.message).toBe( 'Declarative schema planning failed: 2 files cannot apply\n - extensions/pg_cron.sql: extension "pg_cron" already exists\n - extensions/pg_net.sql: extension "pg_net" already exists', ); + expect(error.diagnostics).toEqual([ + { + code: "stuck_statement", + severity: "error", + message: 'extensions/pg_cron.sql: extension "pg_cron" already exists', + context: { rounds: 6 }, + }, + { + code: "stuck_statement", + severity: "error", + message: 'extensions/pg_net.sql: extension "pg_net" already exists', + }, + ]); expect(error.cause).toBe(cause); yield* Effect.promise(() => Promise.all([targetPool.end(), shadowPool.end()])); }).pipe(Effect.provide(failingLayer)); From b2b7cb9ebb91d8ffd71da5eb7f91dc4b21bfa9ff Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 14 Aug 2026 18:25:45 +0200 Subject: [PATCH 29/82] refactor(cli): trim pg-delta next change set --- .../legacy/commands/db/diff/diff.handler.ts | 46 +- .../commands/db/diff/diff.integration.test.ts | 182 +------ .../legacy/commands/db/diff/diff.layers.ts | 85 +--- .../legacy/commands/db/pull/pull.handler.ts | 133 +----- .../commands/db/pull/pull.integration.test.ts | 37 +- .../legacy/commands/db/pull/pull.layers.ts | 70 +-- .../db/reset/reset.integration.test.ts | 33 -- .../declarative.extension-repair.ts | 15 +- .../declarative.extension-repair.unit.test.ts | 20 +- .../db/schema/declarative/declarative.flow.ts | 37 +- .../declarative/declarative.flow.unit.test.ts | 213 ++++----- ...eclarative.orchestrate.integration.test.ts | 18 +- .../declarative/declarative.orchestrate.ts | 8 +- .../generate/generate.integration.test.ts | 62 --- .../declarative/generate/generate.layers.ts | 95 +--- .../declarative/sync/sync.integration.test.ts | 82 ---- .../db/schema/declarative/sync/sync.layers.ts | 94 +--- .../db/shared/legacy-pgdelta-engine.layer.ts | 74 ++- .../legacy-pgdelta-engine.layer.unit.test.ts | 163 +------ ...elta-engine.next.layer.integration.test.ts | 29 +- .../legacy-pgdelta-engine.next.layer.ts | 80 ++-- ...acy-pgdelta-engine.next.layer.unit.test.ts | 15 +- .../legacy-pgdelta-engine.next.unit.test.ts | 28 +- .../shared/legacy-pgdelta-files.unit.test.ts | 65 --- .../legacy-pgdelta-next-adapter.layer.ts | 108 +---- .../legacy-pgdelta-next-adapter.service.ts | 59 +-- .../legacy-pgdelta-next-adapter.unit.test.ts | 401 ++-------------- ...legacy-pgdelta-next-artifacts.unit.test.ts | 46 +- ...gacy-pgdelta-next-diagnostics.unit.test.ts | 103 +--- .../legacy-pgdelta-next-shadow.layer.ts | 4 +- .../shared/legacy-pgdelta.write.unit.test.ts | 359 ++++---------- .../db/start/start.integration.test.ts | 41 +- .../migration/fetch/fetch.integration.test.ts | 76 +-- .../repair/repair.integration.test.ts | 25 - .../commands/start/start.integration.test.ts | 16 - .../db-bootstrap/container-lifecycle.ts | 48 +- .../container-lifecycle.unit.test.ts | 447 ++++++------------ .../legacy/shared/db-bootstrap/db-setup.ts | 28 +- .../shared/db-bootstrap/db-setup.unit.test.ts | 40 -- .../shared/db-bootstrap/docker-create-args.ts | 42 +- .../postgres.service.unit.test.ts | 5 - .../shared/db-bootstrap/shadow-database.ts | 2 +- .../db-bootstrap/shadow-database.unit.test.ts | 155 ++---- .../legacy-migrate-and-seed.unit.test.ts | 48 +- .../legacy/shared/legacy-migration-apply.ts | 44 +- .../shared/legacy-migration-file.unit.test.ts | 11 - .../legacy-pg-net-guidance.unit.test.ts | 4 - docs/roadmap/pg-delta-next-follow-ups.md | 29 -- 48 files changed, 737 insertions(+), 3088 deletions(-) delete mode 100644 apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.unit.test.ts delete mode 100644 docs/roadmap/pg-delta-next-follow-ups.md diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index 19526fae1d..e39942bbec 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -555,13 +555,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy usePgSchema, pgDeltaDefault, }); - // The bundled next engine is the ONLY mode whose baseline is local migrations - // alone. Every other mode — migra, pgAdmin, and the `SUPABASE_USE_PG_DELTA_NEXT= - // false` legacy pg-delta opt-out — still substitutes the declared-schema - // `contrib_regression` target for a local database (`legacy-shadow-source.ts`'s - // `migrationMode !== "pgdelta-next"` branch), so under those engines - // `schema_paths` genuinely does still shape the output and the transition warning - // would be factually wrong. Gate it on the resolved engine, not on the setting. + // Only the next engine ignores schema_paths when building its migrations baseline. const usesPgDeltaNext = useDelta && pgDelta.implementation === "next"; if (usesPgDeltaNext && cfg.schemaPaths !== undefined && cfg.schemaPaths.length > 0) { yield* output.raw(legacySchemaPathsTransitionWarning, "stderr"); @@ -635,12 +629,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy password: shadowBase.password, database: "postgres", }; - // Same `acquireUseRelease` rationale as the migra/pg-delta branch below: `acquire` is - // ONLY container creation (uninterruptible); the health-wait + migrate + diff run - // inside the interruptible `use` phase. `acquire` here is ONLY - // `legacyCreateShadowDatabase` — NOT `legacyPrepareShadowSource` (no `--target-local` - // declarative-schema branch, no `targetUrlOverride`, no pg-delta apply: the pgAdmin - // engine migrates the shadow directly). + // Register cleanup atomically with shadow creation; preparation stays interruptible. const sql = yield* Effect.acquireUseRelease( legacyCreateShadowDatabase(spawner, shadowBase), (handle) => @@ -692,26 +681,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy pgDelta: cfg.pgDelta, ctx, }; - // `Effect.acquireUseRelease`, NOT a separate `yield* legacyCreateShadowDatabase(...)` - // followed by a later `.pipe(Effect.ensuring(...))`: the latter shape leaves a real gap - // between the shadow's successful creation and the `Effect.ensuring` finalizer actually - // being attached — a fiber interrupt landing in that gap (between the two `yield*` - // statements) would skip `legacyRemoveShadowDatabase` entirely, leaking the live shadow - // container and leaving the shadow port occupied. `acquireUseRelease` closes that: - // `acquire` runs inside an `uninterruptibleMask`, and the release finalizer is registered - // in the SAME uninterruptible continuation `acquire` resolves into. This does NOT make - // removal unconditional, though — see `legacyCreateShadowDatabase`'s own doc comment - // (`shadow-database.ts`) for the still-present leak window when `acquire` itself fails - // partway through (a `docker create` success followed by a `docker cp`/`docker start` - // failure). - // - // `acquire` here is ONLY `legacyCreateShadowDatabase` (container creation) — NOT the - // health-wait/migrate/declarative-apply `legacyPrepareShadowSource` performs. Those run - // inside the `use` phase below instead, where a SIGINT can still interrupt them; passing - // all of `legacyPrepareShadowSource` as `acquire` made that whole sequence uninterruptible - // too, since `acquireUseRelease`'s `uninterruptibleMask` has no `restore` around - // `acquire` — see `legacy-shadow-source.ts`'s own doc comment on - // `legacyPrepareShadowSource` for the full rationale. + // Register cleanup atomically with creation; prepare and diff remain interruptible. diffResult = yield* Effect.acquireUseRelease( legacyCreateShadowDatabase(spawner, shadowInput), (handle) => @@ -819,15 +789,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // writing a `_.sql` migration with no name. } else if (Option.isSome(flags.file) && flags.file.value.length > 0) { const fileName = flags.file.value; - // A pg-delta plan that crosses a transaction boundary yields more than one - // ordered unit; writing them into a single migration file would later fail - // when `db push`/`reset` applies it as one transaction. Write one migration - // file per unit in that case via the shared writer: each file appends the - // unit name and gets a strictly increasing timestamp, the full set is - // collision-checked against existing migrations, and every file is written - // exclusively so a pre-existing migration is never overwritten. A - // single-unit plan (and the migra engine) keeps the exact `_.sql` - // file. + // Plans spanning transaction boundaries need one migration per ordered unit. const planFiles = diffResult.files ?? []; if (planFiles.length > 1) { const writtenUnits = yield* legacyWritePgDeltaMigrations(fs, path, { diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 4a77cfe9d2..c17ea1bc58 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -663,18 +663,6 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("migra local diff does not print the schema_paths transition warning", () => { - writeSchemaPathsConfig(false); - const s = setup(tmp.current, { - pgDeltaImplementation: "next", - diffSql: "create table result ();\n", - }); - return Effect.gen(function* () { - yield* legacyDbDiff(flags()); - expect(stderr(s.out)).not.toContain("schema_paths no longer changes the migrations baseline"); - }).pipe(Effect.provide(s.layer)); - }); - it.effect("PG14: provisions a shadow via the SQL-exec init path (no PG15+ one-shot jobs)", () => { // This covers the PG14 branch of the `legacySetupDatabase` pipeline, which execs // SQL directly via the session instead of the three one-shot `LegacyDockerRun` @@ -1423,49 +1411,7 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); - for (const format of ["json", "stream-json"] as const) { - it.effect(`includes the ignored declarative baseline advisory in ${format} output`, () => { - mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); - writeFileSync( - join(tmp.current, "supabase", "database", "items.sql"), - "create table items ();\n", - ); - const s = setup(tmp.current, { - format, - pgDeltaImplementation: "next", - diffSql: "create table dogfood_note ();\n", - }); - return Effect.gen(function* () { - yield* legacyDbDiff( - flags({ usePgDelta: Option.some(true), file: Option.some("dogfood_note") }), - ); - const success = s.out.messages.find((message) => message.type === "success"); - expect(success?.data).toMatchObject({ - diff: "create table dogfood_note ();\n", - engine: "pg-delta", - advisories: [ - { - code: "DeclarativeSchemaNotUsedAsDiffBaseline", - severity: "info", - context: { - baseline: "supabase/migrations", - declarativePath: "supabase/database", - fileFlagFiltersObjects: false, - }, - }, - ], - }); - expect(stderr(s.out)).toContain("db diff -f uses supabase/migrations as its baseline"); - const written = readdirSync(join(tmp.current, "supabase", "migrations")); - expect(written).toHaveLength(1); - expect(readFileSync(join(tmp.current, "supabase", "migrations", written[0]!), "utf8")).toBe( - "create table dogfood_note ();\n", - ); - }).pipe(Effect.provide(s.layer)); - }); - } - - it.effect("does not emit the advisory for the legacy pg-delta implementation", () => { + it.effect("includes the ignored declarative baseline advisory in JSON output", () => { mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); writeFileSync( join(tmp.current, "supabase", "database", "items.sql"), @@ -1473,7 +1419,7 @@ describe("legacy db diff", () => { ); const s = setup(tmp.current, { format: "json", - pgDeltaImplementation: "legacy", + pgDeltaImplementation: "next", diffSql: "create table dogfood_note ();\n", }); return Effect.gen(function* () { @@ -1481,8 +1427,22 @@ describe("legacy db diff", () => { flags({ usePgDelta: Option.some(true), file: Option.some("dogfood_note") }), ); const success = s.out.messages.find((message) => message.type === "success"); - expect(success?.data).not.toHaveProperty("advisories"); - expect(stderr(s.out)).not.toContain("db diff -f uses supabase/migrations"); + expect(success?.data).toMatchObject({ + diff: "create table dogfood_note ();\n", + engine: "pg-delta", + advisories: [ + { + code: "DeclarativeSchemaNotUsedAsDiffBaseline", + severity: "info", + context: { + baseline: "supabase/migrations", + declarativePath: "supabase/database", + fileFlagFiltersObjects: false, + }, + }, + ], + }); + expect(stderr(s.out)).toContain("db diff -f uses supabase/migrations as its baseline"); }).pipe(Effect.provide(s.layer)); }); @@ -1515,53 +1475,29 @@ describe("legacy db diff", () => { }); it.effect("writes one migration file per unit for a multi-unit pg-delta plan", () => { - // A pg-delta plan that crosses a transaction boundary yields more than one - // ordered unit; writing them into one migration would fail when db push/reset - // applies it as a single transaction. Each unit becomes its own file (Go's - // WritePgDeltaMigrations), named `_` with strictly increasing - // timestamps, and the machine payload's `files` lists them all. const s = setup(tmp.current, { format: "json", diffFiles: [ - { name: "schema_changes", sql: "alter type mood add value 'ok';" }, - { name: "after_enum_values", sql: "insert into t values ('ok');" }, + { name: "ignored", sql: "alter type mood add value 'ok';" }, + { name: "ignored", sql: "insert into t values ('ok');" }, ], + diffSuffixes: ["_1", "_2"], }); return Effect.gen(function* () { yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), file: Option.some("my_diff") })); const dir = join(tmp.current, "supabase", "migrations"); const files = readdirSync(dir).sort(); expect(files).toHaveLength(2); - expect(files[0]).toMatch(/^\d{14}_my_diff_schema_changes\.sql$/); - expect(files[1]).toMatch(/^\d{14}_my_diff_after_enum_values\.sql$/); - // Each unit's file carries only that unit's SQL, terminated with a newline. + expect(files[0]).toBe("19700101000000_my_diff_1.sql"); + expect(files[1]).toBe("19700101000001_my_diff_2.sql"); expect(readFileSync(join(dir, files[0]!), "utf8")).toBe("alter type mood add value 'ok';\n"); const success = s.out.messages.find((m) => m.type === "success"); const data = success?.data as { file: string; files: ReadonlyArray }; expect(data.files).toHaveLength(2); - // `file` stays the first written path for released string-field consumers. expect(data.file).toBe(data.files[0]); }).pipe(Effect.provide(s.layer)); }); - it.effect("uses exact next-renderer suffixes for multi-file migration names", () => { - const s = setup(tmp.current, { - diffFiles: [ - { name: "ignored_legacy_name", sql: "a" }, - { name: "ignored_legacy_name", sql: "b" }, - ], - diffSuffixes: ["_1", "_2"], - }); - return Effect.gen(function* () { - yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), file: Option.some("my_diff") })); - const dir = join(tmp.current, "supabase", "migrations"); - expect(readdirSync(dir).sort()).toEqual([ - "19700101000000_my_diff_1.sql", - "19700101000001_my_diff_2.sql", - ]); - }).pipe(Effect.provide(s.layer)); - }); - it.effect("creates nested parent directories for a nested single-unit --file name", () => { // `db diff -f snapshots/remote` must create the `_snapshots/` parent dir // before writing. @@ -1576,78 +1512,6 @@ describe("legacy db diff", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("creates nested parent directories for a nested multi-unit --file name", () => { - const s = setup(tmp.current, { - format: "json", - diffFiles: [ - { name: "schema_changes", sql: "alter type mood add value 'ok';" }, - { name: "after_enum_values", sql: "insert into t values ('ok');" }, - ], - }); - return Effect.gen(function* () { - yield* legacyDbDiff( - flags({ usePgDelta: Option.some(true), file: Option.some("snapshots/remote") }), - ); - const success = s.out.messages.find((m) => m.type === "success"); - const data = success?.data as { files: ReadonlyArray }; - expect(data.files).toHaveLength(2); - for (const written of data.files) expect(existsSync(written)).toBe(true); - expect(data.files[0]).toMatch(/\d{14}_snapshots\/remote_schema_changes\.sql$/u); - expect(data.files[1]).toMatch(/\d{14}_snapshots\/remote_after_enum_values\.sql$/u); - }).pipe(Effect.provide(s.layer)); - }); - - it.effect("bumps the version set when another migration uses the same version", () => { - // Migration identity is the timestamp, not the full filename. If another name - // already uses a generated version, the whole set advances so every new version - // stays strictly ascending and unique. - const s = setup(tmp.current, { - format: "json", - diffFiles: [ - { name: "schema_changes", sql: "a" }, - { name: "after_enum_values", sql: "b" }, - ], - }); - return Effect.gen(function* () { - const dir = join(tmp.current, "supabase", "migrations"); - mkdirSync(dir, { recursive: true }); - // TestClock starts at epoch 0, so the first version the writer tries is - // 19700101000000; pre-seed a differently named migration at that version. - const clashing = join(dir, "19700101000000_different_name.sql"); - writeFileSync(clashing, "-- pre-existing\n"); - yield* legacyDbDiff(flags({ usePgDelta: Option.some(true), file: Option.some("my_diff") })); - expect(readdirSync(dir).sort()).toEqual([ - "19700101000000_different_name.sql", - "19700101000001_my_diff_schema_changes.sql", - "19700101000002_my_diff_after_enum_values.sql", - ]); - // The pre-existing file was never overwritten. - expect(readFileSync(clashing, "utf8")).toBe("-- pre-existing\n"); - }).pipe(Effect.provide(s.layer)); - }); - - it.effect("removes already-written unit files when a later unit write fails", () => { - // A mid-loop write failure best-effort removes every file this invocation - // already wrote, so no partial multi-file migration is left behind. - const s = setup(tmp.current, { - format: "json", - failWriteOnCall: 2, - diffFiles: [ - { name: "schema_changes", sql: "a" }, - { name: "after_enum_values", sql: "b" }, - ], - }); - return Effect.gen(function* () { - const exit = yield* legacyDbDiff( - flags({ usePgDelta: Option.some(true), file: Option.some("my_diff") }), - ).pipe(Effect.exit); - expect(Exit.isFailure(exit)).toBe(true); - const dir = join(tmp.current, "supabase", "migrations"); - const remaining = existsSync(dir) ? readdirSync(dir) : []; - expect(remaining).toEqual([]); - }).pipe(Effect.provide(s.layer)); - }); - it.effect("explicit --from local --to linked prints the diff to stdout", () => { const s = setup(tmp.current, { isLocal: false, diffSql: "create table e ();\n" }); return Effect.gen(function* () { diff --git a/apps/cli/src/legacy/commands/db/diff/diff.layers.ts b/apps/cli/src/legacy/commands/db/diff/diff.layers.ts index ab70a87853..53171388b5 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.layers.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.layers.ts @@ -1,92 +1,19 @@ import { Layer } from "effect"; import { commandRuntimeLayer } from "../../../../shared/runtime/command-runtime.layer.ts"; -import { legacyHttpClientLayer } from "../../../auth/legacy-http-debug.layer.ts"; -import { legacyCliConfigLayer } from "../../../config/legacy-cli-config.layer.ts"; -import { legacyDbConfigLayer } from "../../../shared/legacy-db-config.layer.ts"; -import { legacyDbConnectionLayer } from "../../../shared/legacy-db-connection.layer.ts"; -import { legacyDebugLoggerLayer } from "../../../shared/legacy-debug-logger.layer.ts"; -import { legacyDockerRunLayer } from "../../../shared/legacy-docker-run.layer.ts"; -import { legacyEdgeRuntimeScriptLayer } from "../../../shared/legacy-edge-runtime-script.layer.ts"; import { legacyIdentityStitchLayer } from "../../../shared/legacy-identity-stitch.ts"; import { legacyLinkedDbResolverRuntimeLayer } from "../../../shared/legacy-management-api-runtime.layer.ts"; -import { legacyPgDeltaSslProbeLayer } from "../../../shared/legacy-pgdelta-ssl-probe.layer.ts"; import { legacyTelemetryStateLayer } from "../../../telemetry/legacy-telemetry-state.layer.ts"; -import { legacyDeclarativeSeamLayer } from "../shared/legacy-pgdelta.seam.layer.ts"; -import { legacyPgDeltaEngineLayer } from "../shared/legacy-pgdelta-engine.layer.ts"; -import { legacyPgDeltaNextAdapterLayer } from "../shared/legacy-pgdelta-next-adapter.layer.ts"; -import { legacyPgDeltaNextShadowLayer } from "../shared/legacy-pgdelta-next-shadow.layer.ts"; - -/** - * Runtime layer for `supabase db diff`. - * - * Mirrors `db schema declarative generate` (`generate.layers.ts`): the db-config - * resolver plus the native pg-delta / migra / pgAdmin stack — the edge-runtime - * runner, the SSL probe, and `HttpClient` (the native shadow's health-check wait). - * Shadow provisioning (`db diff`'s own — migra/pg-delta AND pgadmin alike — plus - * the explicit `--from migrations`/`--to migrations` catalog shadow) is fully - * native — see `commands/db/shared/legacy-shadow-source.ts` and - * `shared/legacy-pgdelta.cache.ts` — so no `LegacyDeclarativeSeam` layer is - * needed here. `--use-pg-schema` is the only engine that delegates through - * `LegacyGoProxy`; `--use-pgadmin` uses `LegacyDockerRun` natively instead, the - * same service the migra OOM bash fallback already needed. - * `LegacyDockerRun` is exposed in the merge (not just provided to the - * edge-runtime layer) because both the migra OOM bash fallback and the pgadmin - * differ container run their own container directly. - * Per the "provide doesn't share to siblings" rule, `LegacyCliConfig` is provided - * to every layer that needs it. - */ -const cliConfig = legacyCliConfigLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); - -const dbConfig = legacyDbConfigLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(legacyDebugLoggerLayer), - // The linked db-config resolver snapshots the single `LegacyIdentityStitch`; - // the command runtime must provide it or the bundled binary panics with a - // missing-service error (legacy CLAUDE.md rule 5). - Layer.provide(legacyIdentityStitchLayer), -); - -const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( - Layer.provide(legacyDockerRunLayer), - Layer.provide(cliConfig), -); - -const httpClient = legacyHttpClientLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); -const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); -const nextShadow = legacyPgDeltaNextShadowLayer.pipe( - Layer.provide(legacyDockerRunLayer), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(httpClient), -); -const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyPgDeltaNextAdapterLayer), - Layer.provide(nextShadow), - Layer.provide(edgeRuntime), - Layer.provide(legacyPgDeltaSslProbeLayer), - Layer.provide(seam), - Layer.provide(legacyDockerRunLayer), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(httpClient), - Layer.provide(legacyDebugLoggerLayer), -); +import { + legacyPgDeltaCommandRuntimeLayer, + legacyPgDeltaDbConfigRuntimeLayer, +} from "../shared/legacy-pgdelta-engine.layer.ts"; export const legacyDbDiffRuntimeLayer = Layer.mergeAll( - dbConfig, - legacyDbConnectionLayer, - legacyDockerRunLayer, - edgeRuntime, - legacyPgDeltaSslProbeLayer, - pgDeltaEngine, - httpClient, - cliConfig, + legacyPgDeltaDbConfigRuntimeLayer, + legacyPgDeltaCommandRuntimeLayer, legacyIdentityStitchLayer, legacyTelemetryStateLayer, - // Writes the linked-project cache for `--linked`; this bundle supplies - // `LegacyLinkedProjectCache` (+ the lazy Management-API runtime it needs), - // mirroring `db schema declarative generate`. legacyLinkedDbResolverRuntimeLayer(["db", "diff"]).pipe(Layer.provide(legacyIdentityStitchLayer)), commandRuntimeLayer(["db", "diff"]), ); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 5ef424f682..96c3f68e72 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -331,23 +331,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy const spawner = yield* ChildProcessSpawner.ChildProcessSpawner; const runtimeInfo = yield* RuntimeInfo; const networkIdFlag = yield* LegacyNetworkIdFlag; - // Build (and validate) the shadow's own local container inputs BEFORE `resolver.resolve()` - // below, not after: `legacyBuildLocalDbContainerInputs` -> `legacyResolveLocalConfigValues`/ - // `legacyResolveDbBootstrapConfig` read/validate fields (e.g. enabled API TLS's cert/key - // files) that `toml` above never touches (`legacy-db-config.toml-read.ts` only tracks their - // dotted keys for remote-override gating, it doesn't read the files). This validation must - // run strictly before `resolver.resolve()` (a linked target's temp-role mint over the - // Management API) and `connection.connect()` — otherwise a config broken only in a field - // this build reads (e.g. a missing `api.tls.cert_path` file) would surface after those - // network side effects instead of before them. Skipped for the delegated `--experimental` - // path: that spawns the real Go binary, which performs this exact validation itself — - // building it here too would run (and, for any WARN branch, print) it twice for the same - // invocation. Kept as an `Option`, not built directly into a bare value, so the two - // non-delegate branches below (declarative and migration-file — the exact set - // `delegatesExperimentalPull` excludes) can unwrap it without an `undefined` check; both - // `Option.getOrThrow` call sites document why that unwrap is always `Some` there. Cheap - // either way: image resolution stays lazy (`resolvePostgresImage`), so this doesn't pull the - // shadow's Docker image yet. + // Validate native shadow inputs before target resolution performs remote side effects. const localInputs: Option.Option = delegatesExperimentalPull ? Option.none() : Option.some( @@ -393,17 +377,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy }; const formatOptions = Option.getOrElse(toml.pgDelta.formatOptions, () => ""); - // Container-level pooler fallback (Go's `PoolerFallbackConfig`, - // `internal/db/dump/pooler_fallback.go`, wired into `diffRemoteSchema` and - // `pullDeclarativePgDelta`, `internal/db/pull/pull.go`). A linked pull can reach - // the direct host from the CLI process (so the resolver returned the direct - // conn) yet fail from inside the edge-runtime container on an IPv6-only Docker - // network. When the differ/export error classifies as an IPv6 connectivity - // failure, retry once through the project's IPv4 transaction pooler, reusing the - // same shadow source. Gated to the `--linked` path with a direct - // `db..` connection (Go's `PoolerFallbackEligible` + - // `ProjectRefFromDirectDbHost`). The error message embeds the container stderr - // (edge-runtime/migra errors wrap it), which is what Go classifies. + // A linked direct connection may need the IPv4 transaction pooler from Docker. const targetEndpoint: LegacyPgDeltaDatabaseEndpoint = { kind: "database", ref: targetUrl, @@ -461,7 +435,6 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy envEnabled: legacyParseBoolEnv(toml.envLookup("SUPABASE_EXPERIMENTAL_PG_DELTA")), }), }); - /** Whether the migration-style diff runs on the bundled in-process next engine. */ const usesPgDeltaNext = usePgDeltaDiff && pgDeltaEngine.implementation === "next"; // Runs the Go-delegated `--experimental` structured dump (still delegated, see @@ -537,28 +510,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy strictCoverage: flags.strictCoverage, noCache: false, }); - // Built above, before `resolver.resolve()` — see that build's doc comment. - // `Option.getOrThrow` is safe here: `useDeclarative` is true in this branch, and - // `delegatesExperimentalPull` is defined as `!useDeclarative && (...)`, so - // `localInputs` was always built (never the `Option.none()` delegate case) by the - // time this branch runs. - // `Effect.acquireUseRelease`, NOT a separate `yield* legacyCreateShadowDatabase(...)` - // followed by a later `.pipe(Effect.ensuring(...))` (see this file's migration-path - // call site below, and `diff.handler.ts`'s identical call site, for the full - // rationale): the latter shape leaves a gap between the shadow's successful creation - // and the `Effect.ensuring` finalizer actually being attached, where a fiber interrupt - // would skip `legacyRemoveShadowDatabase` and leak the shadow container. - // `acquireUseRelease` registers the release finalizer in the same uninterruptible - // continuation the acquire resolves into. This does NOT make removal unconditional, - // though — see `legacyCreateShadowDatabase`'s own doc comment (`shadow-database.ts`) - // for the still-present leak window when `acquire` itself fails partway through (a - // `docker create` success followed by a `docker cp`/`docker start` failure). - // - // `acquire` here is ONLY `legacyCreateShadowDatabase` (container creation) — NOT the - // health-wait `legacyPrepareRawShadow` performs; that runs inside the `use` phase - // below instead, so a SIGINT can still interrupt it, matching Go's single cancellable - // `ctx` (see `shadow-database.ts`'s own doc comment on `legacyPrepareRawShadow` for - // the full rationale, review: PRRT_kwDOErm0O86XMrID). + // Legacy export owns an interrupt-safe empty-shadow lifecycle; next reads the target. const exported = pgDeltaEngine.implementation === "next" ? yield* withPoolerFallback(targetEndpoint, (target) => exportSchema(target)) @@ -602,17 +554,8 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy ).pipe( Effect.mapError((cause) => new LegacyDbPullWriteError({ message: cause.message })), ); - // Same manifest-less-merge caveat as generate/sync: the next writer only - // prunes manifest-owned files, so name what survived. yield* legacyWarnPreservedUnmanagedDeclarativeFiles(declarativeDirRel, written); - // Go's WriteDeclarativeSchemas also points [db.migrations] schema_paths at - // the declarative dir, but only when pg-delta is *disabled* in config - // (declarative.go:260-268, gated on IsPgDeltaEnabled which reads the config - // value). db pull --declarative does not force-enable pg-delta - // (cmd/db.go:180-182), so unlike generate/sync this branch is reachable: - // it preserves the legacy experimental db-reset schema-files workflow. - // Normal db diff and migration-style db pull still use migrations as - // their baseline and ignore this setting. + // Preserve the legacy schema_paths workflow only when pg-delta is disabled. if (!toml.pgDelta.enabled) { yield* legacyUpdateDeclarativeSchemaPathsConfig( fs, @@ -644,13 +587,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy return; } - // Gated on the resolved engine, not merely on `schema_paths` being set: the - // bundled next engine is the only mode whose baseline is local migrations - // alone. Under migra or the `SUPABASE_USE_PG_DELTA_NEXT=false` legacy pg-delta - // opt-out, `legacy-shadow-source.ts` still substitutes the declared-schema - // `contrib_regression` target for a local database (its `migrationMode !== - // "pgdelta-next"` branch), so `schema_paths` does still shape the output there - // and this warning would be factually wrong. + // Only next ignores schema_paths in favor of the migrations baseline. if ( !delegatesExperimentalPull && usesPgDeltaNext && @@ -660,15 +597,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy yield* output.raw(legacySchemaPathsTransitionWarning, "stderr"); } - // Go's `EXPERIMENTAL` structured-dump branch (`pull.go:49-61`) stays - // delegated to Go. pg_dump itself is now native (used by the initial-migra - // path below), but this branch also calls `format.WriteStructuredSchemas` - // (`cli-go/internal/migration/format/format.go:99`), which parses every - // dumped statement with a PostgreSQL DDL AST parser (`multigres`, ~50 node - // types) to route objects into structured files. No Postgres DDL parser - // exists in TS yet, and `--declarative` already covers the same per-object - // outcome via pg-delta managed-state extraction, so this path is deprecated - // rather than ported (CLI-1957) — see the deprecation line printed above. + // Structured dump still delegates to Go's PostgreSQL DDL formatter. if (delegatesExperimentalPull) { // The structured-dump path returns before writing a migration or touching // schema_migrations, so no history repair. @@ -820,33 +749,12 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // Native diff: shadow (baseline + local migrations) vs remote → migration SQL. // For the initial pull (no local migrations) the schema filter is ignored. const diffSchema = sync.kind === "missing" ? [] : flags.schema; - // Go's `diffRemoteSchema` retries the ENTIRE `diff.DiffDatabase` call — shadow - // provisioning included — against the pooler config on an IPv6 failure, not - // just the diff step (`internal/db/pull/pull.go:176-190`): `DiffDatabase` - // prints "Creating shadow database..." and runs `PrepareShadowSource` before - // ever touching the remote/target connection (`internal/db/diff/diff.go:211- - // 217`), so a pooler retry re-prints the creation/diff banners and provisions - // + tears down a second, fresh shadow. Mirror that observable behavior by - // wrapping the full prepare-shadow-then-diff operation in the retried - // closure — each attempt gets its own shadow and its own teardown — instead - // of provisioning one shadow and only retrying the diff engine against it. + // Pooler fallback retries the complete shadow-and-diff attempt. const runShadowDiff = (targetEndpoint: LegacyPgDeltaDatabaseEndpoint) => Effect.gen(function* () { - // `legacyPrepareShadowSource` doesn't print its own banner, so the pull - // handler emits it itself to match the migration-style `db pull` output. yield* output.raw("Creating shadow database...\n", "stderr"); - // Resolved AFTER the banner, inside the retried closure, and re-run fresh - // on every pooler-retry attempt (see the comment above). Resolving it - // earlier, outside this closure (as `diff.handler.ts`'s sibling call site - // also resolves after its own banner), would both print nothing on an - // image-resolution failure before the banner and skip re-resolving it on - // retry. const resolvedPullShadowImage = yield* pullLocalInputs.resolvePostgresImage; - // Legacy engines mirror Go's `DiffDatabase` → `PrepareShadowSource(ctx, schema, - // utils.IsLocalDatabase(config), …)` (`internal/db/diff/diff.go:213`): a local - // target with declarative schema files gets a second `contrib_regression` shadow - // returned as the target override. Pg-delta next compares the migrations shadow - // directly to the live target instead. + // Legacy may substitute a declarative target; next always uses the live target. const migrationMode: "legacy" | "pgdelta-next" = usesPgDeltaNext ? "pgdelta-next" : "legacy"; @@ -869,31 +777,12 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy pgDelta: toml.pgDelta, ctx, }; - // `Effect.acquireUseRelease`, NOT a separate `yield* legacyCreateShadowDatabase(...)` - // followed by a later `.pipe(Effect.ensuring(...))` (see `diff.handler.ts`'s - // identical call site for the full rationale): the latter shape leaves a gap - // between the shadow's successful creation and the `Effect.ensuring` finalizer - // actually being attached, where a fiber interrupt would skip - // `legacyRemoveShadowDatabase` and leak the shadow container. `acquireUseRelease` - // registers the release finalizer in the same uninterruptible continuation the - // acquire resolves into. This does NOT make removal unconditional, though — see - // `legacyCreateShadowDatabase`'s own doc comment (`shadow-database.ts`) for the - // still-present leak window when `acquire` itself fails partway through (a - // `docker create` success followed by a `docker cp`/`docker start` failure). - // - // `acquire` here is ONLY `legacyCreateShadowDatabase` (container creation) — NOT - // the health-wait/migrate/declarative-apply `legacyPrepareShadowSource` performs; - // those run inside the `use` phase below instead, so a SIGINT can still interrupt - // them (see `legacy-shadow-source.ts`'s own doc comment on - // `legacyPrepareShadowSource` for the full rationale). + // Register cleanup atomically with shadow acquisition. return yield* Effect.acquireUseRelease( legacyCreateShadowDatabase(spawner, shadowInput), (handle) => Effect.gen(function* () { const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); - // Use the declarative target override when present (Go substitutes it - // for the diff target, `diff.go:219-220`); for remote pulls it's - // undefined, so this is this attempt's resolved target URL. const target = shadow.targetUrlOverride ?? targetEndpoint.ref; yield* output.raw( diffSchema.length > 0 @@ -951,9 +840,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // initial-migra path seeded the file with a pg_dump above, so its empty second // pass is swallowed and falls through to the shared tail below. if (diffEmpty && !seededFromDump) { - // Go saves a pg-delta debug bundle and embeds its path in the in-sync - // error when PGDELTA_DEBUG is set (`internal/db/pull/pull.go:192-201`); a - // bundle-save failure falls through to the plain in-sync error. + // Preserve the legacy empty-diff debug bundle contract. if (pgDeltaEngine.implementation === "legacy" && diffOutcome.debug !== undefined) { const debugDir = yield* legacySaveEmptyPgDeltaPullDebug({ ctx, diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index ef2b874a6e..efe97ee858 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -9,7 +9,6 @@ import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; import { stripAnsi } from "../../../../../tests/helpers/ansi.ts"; import { LEGACY_VALID_REF, - legacyFailWriteStringOnNthCallFsLayer, mockLegacyCliConfig, mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, @@ -120,7 +119,6 @@ interface SetupOpts { // last-occurrence-wins ordering; defaults to empty. readonly args?: ReadonlyArray; // When set, the Nth `writeFileString` fails, exercising cleanup-on-failure. - readonly failWriteOnCall?: number; // `LegacyCliConfig.projectId` (the `SUPABASE_PROJECT_ID` env-only reader). Defaults // to `Option.some("test")`; pass `Option.none()` to exercise the // config.toml/workdir-basename fallback `legacyResolveLocalProjectId` provides for @@ -461,14 +459,8 @@ function setup(workdir: string, opts: SetupOpts = {}) { Layer.succeed(CliArgs, { args: opts.args ?? [] }), mockRuntimeInfo(), ); - // Merged last so its `FileSystem` overrides everything above (last-wins). - const layer = - opts.failWriteOnCall === undefined - ? baseLayer - : Layer.merge(baseLayer, legacyFailWriteStringOnNthCallFsLayer(opts.failWriteOnCall)); - return { - layer, + layer: baseLayer, out, proxyCalls, proxyCaptureCalls, @@ -728,33 +720,6 @@ describe("legacy db pull", () => { }, ); - it.effect("removes already-written unit files when a later pg-delta unit write fails", () => { - // A mid-loop write failure best-effort removes every migration file this - // invocation already wrote, so no partial multi-file pull is left behind. - seedMigration(tmp.current, "20240101000000"); - const s = setup(tmp.current, { - failWriteOnCall: 2, - remoteVersions: ["20240101000000"], - edgeStdout: pgDeltaDiffEnvelope([ - { name: "schema_changes", sql: "alter type mood add value 'ok';" }, - { name: "after_enum_values", sql: "insert into t values ('ok');" }, - ]), - yes: true, - }); - return Effect.gen(function* () { - const exit = yield* legacyDbPull(flags({ diffEngine: Option.some("pg-delta") })).pipe( - Effect.exit, - ); - expect(Exit.isFailure(exit)).toBe(true); - const dir = join(tmp.current, "supabase", "migrations"); - // Only the pre-seeded local migration remains; the first written unit was - // rolled back and the failing second unit never landed. - expect(readdirSync(dir)).toEqual(["20240101000000_local.sql"]); - // No history rows were upserted because the write failed before the prompt. - expect(s.historyUpserts.length).toBe(0); - }).pipe(Effect.provide(s.layer)); - }); - it.effect("a malformed pg-delta diff envelope surfaces a parse error, not 'in sync'", () => { seedMigration(tmp.current, "20240101000000"); const s = setup(tmp.current, { diff --git a/apps/cli/src/legacy/commands/db/pull/pull.layers.ts b/apps/cli/src/legacy/commands/db/pull/pull.layers.ts index 8f837f582a..8faba82d24 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.layers.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.layers.ts @@ -1,76 +1,18 @@ import { Layer } from "effect"; import { commandRuntimeLayer } from "../../../../shared/runtime/command-runtime.layer.ts"; -import { legacyHttpClientLayer } from "../../../auth/legacy-http-debug.layer.ts"; -import { legacyCliConfigLayer } from "../../../config/legacy-cli-config.layer.ts"; -import { legacyDbConfigLayer } from "../../../shared/legacy-db-config.layer.ts"; -import { legacyDbConnectionLayer } from "../../../shared/legacy-db-connection.layer.ts"; -import { legacyDebugLoggerLayer } from "../../../shared/legacy-debug-logger.layer.ts"; -import { legacyDockerRunLayer } from "../../../shared/legacy-docker-run.layer.ts"; -import { legacyEdgeRuntimeScriptLayer } from "../../../shared/legacy-edge-runtime-script.layer.ts"; import { legacyIdentityStitchLayer } from "../../../shared/legacy-identity-stitch.ts"; import { legacyLinkedDbResolverRuntimeLayer } from "../../../shared/legacy-management-api-runtime.layer.ts"; -import { legacyPgDeltaSslProbeLayer } from "../../../shared/legacy-pgdelta-ssl-probe.layer.ts"; import { legacyTelemetryStateLayer } from "../../../telemetry/legacy-telemetry-state.layer.ts"; import { stdinLayer } from "../../../../shared/runtime/stdin.layer.ts"; -import { legacyDeclarativeSeamLayer } from "../shared/legacy-pgdelta.seam.layer.ts"; -import { legacyPgDeltaEngineLayer } from "../shared/legacy-pgdelta-engine.layer.ts"; -import { legacyPgDeltaNextAdapterLayer } from "../shared/legacy-pgdelta-next-adapter.layer.ts"; -import { legacyPgDeltaNextShadowLayer } from "../shared/legacy-pgdelta-next-shadow.layer.ts"; - -/** - * Runtime layer for `supabase db pull`. The db-config resolver, the native pg-delta / migra - * stack (edge-runtime, SSL probe, `HttpClient` for the native shadow's health-check wait — - * shadow provisioning itself is native, see `commands/db/shared/legacy-shadow-source.ts` / - * `shared/db-bootstrap/shadow-database.ts`), `LegacyDbConnection` (remote connect + - * `schema_migrations` reconciliation / history update), and `LegacyDockerRun` for the migra - * fallback. No `LegacyDeclarativeSeam` — neither `db pull` nor `db diff` has a Go-delegate - * branch that needs it any more (native shadow provisioning replaced it entirely); - * `--use-pgadmin`/`--use-pg-schema` delegate through `LegacyGoProxy` instead, not this seam. - */ -const cliConfig = legacyCliConfigLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); - -const dbConfig = legacyDbConfigLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(legacyDebugLoggerLayer), - Layer.provide(legacyIdentityStitchLayer), -); - -const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( - Layer.provide(legacyDockerRunLayer), - Layer.provide(cliConfig), -); - -const httpClient = legacyHttpClientLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); -const seam = legacyDeclarativeSeamLayer.pipe(Layer.provide(cliConfig)); -const nextShadow = legacyPgDeltaNextShadowLayer.pipe( - Layer.provide(legacyDockerRunLayer), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(httpClient), -); -const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyPgDeltaNextAdapterLayer), - Layer.provide(nextShadow), - Layer.provide(edgeRuntime), - Layer.provide(legacyPgDeltaSslProbeLayer), - Layer.provide(seam), - Layer.provide(legacyDockerRunLayer), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(httpClient), - Layer.provide(legacyDebugLoggerLayer), -); +import { + legacyPgDeltaCommandRuntimeLayer, + legacyPgDeltaDbConfigRuntimeLayer, +} from "../shared/legacy-pgdelta-engine.layer.ts"; export const legacyDbPullRuntimeLayer = Layer.mergeAll( - dbConfig, - legacyDbConnectionLayer, - legacyDockerRunLayer, - edgeRuntime, - legacyPgDeltaSslProbeLayer, - pgDeltaEngine, - httpClient, - cliConfig, + legacyPgDeltaDbConfigRuntimeLayer, + legacyPgDeltaCommandRuntimeLayer, legacyIdentityStitchLayer, legacyTelemetryStateLayer, legacyLinkedDbResolverRuntimeLayer(["db", "pull"]).pipe(Layer.provide(legacyIdentityStitchLayer)), diff --git a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts index e78de1e604..8924889d0c 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts @@ -1464,39 +1464,6 @@ describe("legacy db reset", () => { }); }); - it.live("honors pg-delta's no-transaction migration header on remote reset", () => { - const set = "SET check_function_bodies = off"; - const action = "DROP SUBSCRIPTION app_events"; - const { layer, conn } = setup(tmp.current, { - toml: 'project_id = "test"\n', - files: migrationFile( - "20240101000000", - `-- pg-delta: transaction=false\n${set};\n${action};\nRESET ALL;`, - ), - confirm: [true], - }); - return Effect.gen(function* () { - yield* legacyDbReset({ ...DEFAULT_FLAGS, linked: true }).pipe(Effect.provide(layer)); - const setupCommit = conn.execs.indexOf("COMMIT"); - const firstStatement = `-- pg-delta: transaction=false\n${set}`; - const setIndex = conn.execs.indexOf(firstStatement); - const actionIndex = conn.execs.indexOf(action); - const cleanupIndex = conn.execs.lastIndexOf("RESET ALL"); - - expect(setIndex).toBeGreaterThan(setupCommit); - expect(actionIndex).toBeGreaterThan(setIndex); - expect(cleanupIndex).toBeGreaterThan(actionIndex); - expect(conn.execs.slice(setIndex, cleanupIndex + 1)).toEqual([ - firstStatement, - action, - "RESET ALL", - ]); - expect( - conn.queries.some((query) => query.sql.includes("INSERT INTO supabase_migrations")), - ).toBe(true); - }); - }); - it.live("fails a remote reset before dropping schemas on an undecryptable secret", () => { // Regression: the old point-of-use vault decryption ran AFTER `legacyDropUserSchemas`, // so an undecryptable `encrypted:` secret dropped the schemas before failing. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.ts index 3f5db7862a..fda461fec8 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.ts @@ -1,6 +1,6 @@ import { Effect, FileSystem, Path } from "effect"; -import { legacyExtensionDeclaration } from "./declarative.flow.ts"; +import { legacyDeclaredExtensions, legacyExtensionDeclaration } from "./declarative.flow.ts"; interface LegacyExtensionRepairResult { readonly path: string; @@ -8,17 +8,6 @@ interface LegacyExtensionRepairResult { readonly addedDeclarations: ReadonlyArray; } -const declaredExtensions = (sql: string): ReadonlySet => { - const extensions = new Set(); - const pattern = - /\bCREATE\s+EXTENSION\s+(?:IF\s+NOT\s+EXISTS\s+)?(?:"([^"]+)"|([a-zA-Z_][\w$-]*))/gi; - for (const match of sql.matchAll(pattern)) { - const extension = match[1] ?? match[2]; - if (extension !== undefined) extensions.add(extension); - } - return extensions; -}; - /** Appends missing legacy extension declarations without replacing existing SQL. */ export const legacyAppendExtensionDeclarations = Effect.fnUntraced(function* ( declarativeDir: string, @@ -29,7 +18,7 @@ export const legacyAppendExtensionDeclarations = Effect.fnUntraced(function* ( const extensionPath = path.join(declarativeDir, "extension.sql"); const exists = yield* fs.exists(extensionPath); const existing = exists ? yield* fs.readFileString(extensionPath) : ""; - const declared = declaredExtensions(existing); + const declared = legacyDeclaredExtensions([{ name: "extension.sql", sql: existing }]); const addedExtensions = [...new Set(extensions)] .filter((extension) => !declared.has(extension)) .sort(); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.unit.test.ts index ab603f9bf9..bd9af61f6a 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.extension-repair.unit.test.ts @@ -1,6 +1,5 @@ import { readFileSync, writeFileSync } from "node:fs"; -import { dirname, join } from "node:path"; -import { fileURLToPath } from "node:url"; +import { join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; @@ -45,21 +44,4 @@ describe("legacyAppendExtensionDeclarations", () => { ); }).pipe(Effect.provide(BunServices.layer)); }); - - it.effect("appends to the representative legacy root extension.sql", () => { - const fixture = join( - dirname(fileURLToPath(import.meta.url)), - "fixtures", - "legacy", - "extension.sql", - ); - const extensionPath = join(tmp.current, "extension.sql"); - writeFileSync(extensionPath, readFileSync(fixture, "utf8")); - return Effect.gen(function* () { - yield* legacyAppendExtensionDeclarations(tmp.current, ["uuid-ossp"]); - const updated = readFileSync(extensionPath, "utf8"); - expect(updated).toContain('CREATE EXTENSION IF NOT EXISTS "vector"'); - expect(updated).toContain('CREATE EXTENSION IF NOT EXISTS "uuid-ossp"'); - }).pipe(Effect.provide(BunServices.layer)); - }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts index ec23554341..f150a17751 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts @@ -176,22 +176,31 @@ function maskSqlComments(sql: string): string { ); } -function declaredImplicitExtensions( +export function legacyDeclaredExtensions( files: readonly LegacyDeclarativeSqlFile[], -): ReadonlySet { - const declared = new Set(); +): ReadonlySet { + const declared = new Set(); const pattern = /\bCREATE\s+EXTENSION\s+(?:IF\s+NOT\s+EXISTS\s+)?(?:"([^"]+)"|([a-zA-Z_][\w$-]*))/gi; for (const file of files) { for (const match of maskSqlNonCode(file.sql).matchAll(pattern)) { - const extensionName = (match[1] ?? match[2])?.toLowerCase(); - const extension = LEGACY_IMPLICIT_EXTENSIONS.find((implicit) => implicit === extensionName); - if (extension !== undefined) declared.add(extension); + const extension = match[1] ?? match[2]; + if (extension !== undefined) declared.add(extension.toLowerCase()); } } return declared; } +function declaredImplicitExtensions( + files: readonly LegacyDeclarativeSqlFile[], +): ReadonlySet { + const declaredNames = legacyDeclaredExtensions(files); + const declared = new Set( + LEGACY_IMPLICIT_EXTENSIONS.filter((extension) => declaredNames.has(extension)), + ); + return declared; +} + function locateSignature( files: readonly LegacyDeclarativeSqlFile[], diagnosticMessage: string, @@ -248,6 +257,15 @@ export function legacyClassifyDeclarativeLoadCompatibility(opts: { export const legacyExtensionDeclaration = (extension: string): string => `CREATE EXTENSION IF NOT EXISTS "${extension}" WITH SCHEMA "extensions";`; +export const legacyNextExportAdoptionCommands = [ + " supabase db schema declarative generate --local --overwrite \\", + " --output supabase/database-next --experimental", + "", + " # review supabase/database-next", + " rm -rf supabase/database && mv supabase/database-next supabase/database", + " supabase db schema declarative sync --no-apply --experimental", +] as const; + export function legacyFormatStagedExportRecommendation( gap: LegacyDeclarativeCompatibilityGap, ): string { @@ -270,11 +288,6 @@ export function legacyFormatStagedExportRecommendation( "WARNING: pg-delta next manages schema state that the legacy export did not represent.", ...detected, "Generate a next-compatible schema into a separate directory, review it, and adopt it when ready:", - " supabase db schema declarative generate --local --overwrite \\", - " --output supabase/database-next --experimental", - "", - " # review supabase/database-next", - " rm -rf supabase/database && mv supabase/database-next supabase/database", - " supabase db schema declarative sync --no-apply --experimental", + ...legacyNextExportAdoptionCommands, ].join("\n"); } diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts index f57358b595..c365b79ecb 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts @@ -23,73 +23,84 @@ const removals = { ], }; -describe("legacyClassifyDeclarativeCompatibilityGap", () => { - it("repairs only the known legacy-implicit extension set", () => { - const gap = legacyClassifyDeclarativeCompatibilityGap({ - implementation: "next", - manifestPresent: false, - removals: { extensions: ["uuid-ossp", "pgcrypto", "pgcrypto"], extensionIntents: [] }, - }); - expect(gap).toEqual({ - repairableExtensions: ["pgcrypto", "uuid-ossp"], - extensionIntents: [], - ambiguousRemovals: [], - recommendedAction: "repair-extensions", - }); - expect(legacyExtensionDeclaration("uuid-ossp")).toBe( - 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";', - ); +const classifyGap = ( + overrides: Partial[0]> = {}, +) => + legacyClassifyDeclarativeCompatibilityGap({ + implementation: "next", + manifestPresent: false, + removals, + ...overrides, }); - it("stages a next export for mixed or unknown extension removals", () => { - const gap = legacyClassifyDeclarativeCompatibilityGap({ - implementation: "next", - manifestPresent: false, - removals: { extensions: ["pgcrypto", "postgis"], extensionIntents: [] }, - }); - expect(gap.repairableExtensions).toEqual(["pgcrypto"]); - expect(gap.ambiguousRemovals).toEqual(["postgis"]); - expect(gap.recommendedAction).toBe("stage-next-export"); +const classifyLoad = ( + overrides: Partial[0]>, +) => + legacyClassifyDeclarativeLoadCompatibility({ + implementation: "next", + manifestPresent: false, + diagnostics: [], + files: [], + ...overrides, }); - it("stages a next export when extension intents are present", () => { - const gap = legacyClassifyDeclarativeCompatibilityGap({ - implementation: "next", - manifestPresent: false, - removals, - }); - expect(gap.recommendedAction).toBe("stage-next-export"); +describe("legacyClassifyDeclarativeCompatibilityGap", () => { + it.each([ + { + name: "repairs known implicit extensions", + overrides: { + removals: { extensions: ["uuid-ossp", "pgcrypto", "pgcrypto"], extensionIntents: [] }, + }, + expected: { + recommendedAction: "repair-extensions", + repairableExtensions: ["pgcrypto", "uuid-ossp"], + ambiguousRemovals: [], + }, + }, + { + name: "stages mixed known and unknown removals", + overrides: { + removals: { extensions: ["pgcrypto", "postgis"], extensionIntents: [] }, + }, + expected: { + recommendedAction: "stage-next-export", + repairableExtensions: ["pgcrypto"], + ambiguousRemovals: ["postgis"], + }, + }, + { + name: "stages extension intents", + overrides: {}, + expected: { recommendedAction: "stage-next-export" }, + }, + { + name: "trusts a next export manifest", + overrides: { manifestPresent: true }, + expected: { recommendedAction: "none" }, + }, + { + name: "leaves legacy behavior unchanged", + overrides: { implementation: "legacy" as const }, + expected: { recommendedAction: "none" }, + }, + { + name: "ignores an empty removal set", + overrides: { removals: { extensions: [], extensionIntents: [] } }, + expected: { recommendedAction: "none" }, + }, + ])("$name", ({ overrides, expected }) => { + expect(classifyGap(overrides)).toMatchObject(expected); + }); + + it("formats repair and staging instructions", () => { + expect(legacyExtensionDeclaration("uuid-ossp")).toBe( + 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";', + ); + const gap = classifyGap(); expect(legacyFormatStagedExportRecommendation(gap)).toContain( "generate --local --overwrite \\\n --output supabase/database-next --experimental", ); }); - - it("is suppressed for next exports with a manifest", () => { - expect( - legacyClassifyDeclarativeCompatibilityGap({ - implementation: "next", - manifestPresent: true, - removals, - }).recommendedAction, - ).toBe("none"); - }); - - it("is suppressed for the legacy engine and irrelevant removals", () => { - expect( - legacyClassifyDeclarativeCompatibilityGap({ - implementation: "legacy", - manifestPresent: false, - removals, - }), - ).toMatchObject({ recommendedAction: "none" }); - expect( - legacyClassifyDeclarativeCompatibilityGap({ - implementation: "next", - manifestPresent: false, - removals: { extensions: [], extensionIntents: [] }, - }), - ).toMatchObject({ recommendedAction: "none" }); - }); }); describe("legacyClassifyDeclarativeLoadCompatibility", () => { @@ -102,9 +113,7 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { ["net.http_post(text, jsonb)", "pg_net"], ])("maps a missing %s routine to %s", (routine, extension) => { const symbol = routine.slice(0, routine.indexOf("(")); - const findings = legacyClassifyDeclarativeLoadCompatibility({ - implementation: "next", - manifestPresent: false, + const findings = classifyLoad({ diagnostics: [ stuck( `schemas/app/tables/members.sql: ERROR: function ${routine} does not exist (failed identically in 6 rounds)`, @@ -133,9 +142,7 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { "maps a direct missing %s extension diagnostic", (extension) => { expect( - legacyClassifyDeclarativeLoadCompatibility({ - implementation: "next", - manifestPresent: false, + classifyLoad({ diagnostics: [ stuck(`cluster/config.sql: ERROR: extension "${extension}" does not exist`), ], @@ -158,9 +165,7 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { ); it("reports the authored line from the diagnostic's file and ignores cascades", () => { - const findings = legacyClassifyDeclarativeLoadCompatibility({ - implementation: "next", - manifestPresent: false, + const findings = classifyLoad({ diagnostics: [ stuck( "schemas/app/tables/members.sql: ERROR: function extensions.uuid_generate_v4() does not exist", @@ -199,7 +204,7 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { manifestPresent: boolean, diagnostics: ReadonlyArray<{ code: string; severity: string; message: string }>, ) => - legacyClassifyDeclarativeLoadCompatibility({ + classifyLoad({ implementation, manifestPresent, diagnostics, @@ -217,9 +222,7 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { it("does not classify an extension already declared anywhere in the tree", () => { expect( - legacyClassifyDeclarativeLoadCompatibility({ - implementation: "next", - manifestPresent: false, + classifyLoad({ diagnostics: [stuck("members.sql: function extensions.uuid_generate_v4() does not exist")], files: [ { name: "members.sql", sql: "select extensions.uuid_generate_v4();" }, @@ -234,9 +237,7 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { it("does not treat a commented or quoted declaration as an extension declaration", () => { expect( - legacyClassifyDeclarativeLoadCompatibility({ - implementation: "next", - manifestPresent: false, + classifyLoad({ diagnostics: [stuck("members.sql: function extensions.uuid_generate_v4() does not exist")], files: [ { @@ -254,9 +255,7 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { it("returns an unlocated finding when the diagnostic has no authored match", () => { expect( - legacyClassifyDeclarativeLoadCompatibility({ - implementation: "next", - manifestPresent: false, + classifyLoad({ diagnostics: [stuck('unknown.sql: extension "pg_net" does not exist')], files: [], }), @@ -272,9 +271,7 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { it("deduplicates identical findings from repeated load diagnostics", () => { const diagnostic = stuck("members.sql: function extensions.uuid_generate_v4() does not exist"); expect( - legacyClassifyDeclarativeLoadCompatibility({ - implementation: "next", - manifestPresent: false, + classifyLoad({ diagnostics: [diagnostic, diagnostic], files: [{ name: "members.sql", sql: "select extensions.uuid_generate_v4();" }], }), @@ -283,46 +280,26 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { }); describe("legacyResolveDeclarativeMigrationName", () => { - it("prefers an explicit --name over --file", () => { - expect(legacyResolveDeclarativeMigrationName("my_change", "declarative_sync")).toBe( - "my_change", - ); - }); - - it("falls back to --file when --name is empty", () => { - expect(legacyResolveDeclarativeMigrationName("", "declarative_sync")).toBe("declarative_sync"); + it.each([ + ["my_change", "declarative_sync", "my_change"], + ["", "declarative_sync", "declarative_sync"], + ])("resolves name=%j file=%j", (name, file, expected) => { + expect(legacyResolveDeclarativeMigrationName(name, file)).toBe(expected); }); }); describe("legacyResolveDeclarativeSyncApplyDecision", () => { - const base = { apply: false, noApply: false, yes: false, tty: false }; - - it("skips when --no-apply is set, regardless of other flags", () => { - expect( - legacyResolveDeclarativeSyncApplyDecision({ - apply: true, - noApply: true, - yes: true, - tty: true, - }), - ).toBe("skip"); - }); - - it("applies when --apply is set (and --no-apply is not)", () => { - expect( - legacyResolveDeclarativeSyncApplyDecision({ ...base, apply: true, yes: false, tty: false }), - ).toBe("apply"); - }); - - it("applies when global --yes is set", () => { - expect(legacyResolveDeclarativeSyncApplyDecision({ ...base, yes: true })).toBe("apply"); - }); - - it("prompts when on a TTY and no apply flags are set", () => { - expect(legacyResolveDeclarativeSyncApplyDecision({ ...base, tty: true })).toBe("prompt"); - }); - - it("skips in non-interactive mode with no apply flags", () => { - expect(legacyResolveDeclarativeSyncApplyDecision(base)).toBe("skip"); + it.each([ + ["--no-apply wins", { apply: true, noApply: true, yes: true, tty: true }, "skip"], + ["--apply applies", { apply: true, noApply: false, yes: false, tty: false }, "apply"], + ["--yes applies", { apply: false, noApply: false, yes: true, tty: false }, "apply"], + ["TTY prompts", { apply: false, noApply: false, yes: false, tty: true }, "prompt"], + [ + "non-interactive defaults to skip", + { apply: false, noApply: false, yes: false, tty: false }, + "skip", + ], + ] as const)("%s", (_name, options, expected) => { + expect(legacyResolveDeclarativeSyncApplyDecision(options)).toBe(expected); }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 835858eeaf..8ac9215e74 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -445,7 +445,7 @@ describe("legacyDiffDeclarativeToMigrations", () => { // `--strict-coverage` is enforced entirely by the next engine's diagnostic report; // the legacy engine has no coverage diagnostics, so the flag silently did nothing // under `SUPABASE_USE_PG_DELTA_NEXT=false`. It must say so instead. - const runWithLegacyEngine = (strictCoverage: boolean) => { + const runWithStrictCoverageOnLegacyEngine = () => { const dir = mkdtempSync(join(tmpdir(), "legacy-decl-orch-")); const declDir = join(dir, "supabase", "database"); mkdirSync(declDir, { recursive: true }); @@ -460,7 +460,7 @@ describe("legacyDiffDeclarativeToMigrations", () => { dir, out, effect: legacyDiffDeclarativeToMigrations( - { ...ctx(dir, declDir), strictCoverage }, + { ...ctx(dir, declDir), strictCoverage: true }, toml, setupInputs, ).pipe( @@ -480,7 +480,7 @@ describe("legacyDiffDeclarativeToMigrations", () => { }; it.effect("warns that --strict-coverage does nothing on the legacy engine", () => { - const { dir, out, effect } = runWithLegacyEngine(true); + const { dir, out, effect } = runWithStrictCoverageOnLegacyEngine(); return effect.pipe( Effect.tap(() => Effect.sync(() => { @@ -493,18 +493,6 @@ describe("legacyDiffDeclarativeToMigrations", () => { ); }); - it.effect("stays silent about --strict-coverage when the flag is unset", () => { - const { dir, out, effect } = runWithLegacyEngine(false); - return effect.pipe( - Effect.tap(() => - Effect.sync(() => { - expect(out.stderrText).not.toContain("--strict-coverage"); - rmSync(dir, { recursive: true, force: true }); - }), - ), - ); - }); - it.effect( "reuses an already-warmed platform-baseline catalog without provisioning a shadow", () => { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 1ac5234231..271c0289c3 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -34,6 +34,7 @@ import { import { legacyClassifyDeclarativeLoadCompatibility, legacyExtensionDeclaration, + legacyNextExportAdoptionCommands, type LegacyDeclarativeLoadCompatibilityFinding, } from "./declarative.flow.ts"; @@ -83,12 +84,7 @@ const formatImplicitExtensionLoadFailure = ( "", "Recommended — generate a next-compatible tree, review it, then adopt:", "", - " supabase db schema declarative generate --local --overwrite \\", - " --output supabase/database-next --experimental", - "", - " # review supabase/database-next", - " rm -rf supabase/database && mv supabase/database-next supabase/database", - " supabase db schema declarative sync --no-apply --experimental", + ...legacyNextExportAdoptionCommands, "", "Alternative — add the missing extension declarations to extension.sql, then re-plan:", ...extensions.map((extension) => legacyExtensionDeclaration(extension)), diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index 9001687189..0ccd779869 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -596,49 +596,6 @@ describe("legacy db schema declarative generate integration", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("--overwrite preserves unmanaged files in the absolute --output destination", () => { - const destination = mkdtempSync(join(tmpdir(), "legacy-decl-output-")); - mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); - writeFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "select 1;"); - writeFileSync(join(destination, "stale.sql"), "select 'stale';"); - const s = setup(tmp.current, { experimental: true, engineImplementation: "next" }); - return Effect.gen(function* () { - yield* legacyDbSchemaDeclarativeGenerate( - flags({ - local: Option.some(true), - output: Option.some(destination), - overwrite: true, - }), - ); - expect(readFileSync(join(destination, "stale.sql"), "utf8")).toBe("select 'stale';"); - expect(existsSync(join(destination, ".pgdelta-export.json"))).toBe(true); - // The destination had no manifest, so nothing could be classified as stale and - // `stale.sql` silently survived an "overwrite" the user confirmed. Say so, and - // point at the only instruction that produces a clean tree. - const stderrText = stripAnsi( - s.out.rawChunks - .filter((chunk) => chunk.stream === "stderr") - .map((chunk) => chunk.text) - .join(""), - ); - expect(stderrText).toContain( - "1 existing declarative schema file(s) in " + destination + " are not tracked", - ); - expect(stderrText).toContain("stale.sql"); - expect(stderrText).toContain(`remove ${destination} and re-run`); - expect( - readFileSync(join(tmp.current, "supabase", "database", "configured.sql"), "utf8"), - ).toBe("select 1;"); - expect( - s.out.rawChunks.map((chunk) => ({ text: stripAnsi(chunk.text), stream: chunk.stream })), - ).toContainEqual({ - text: `Declarative schema written to ${destination}\n`, - stream: "stderr", - }); - rmSync(destination, { recursive: true, force: true }); - }).pipe(Effect.provide(s.layer)); - }); - it.effect("explicit --local checks the local Postgres image before generating", () => { const s = setup(tmp.current, { experimental: true, staleLocalImage: true }); return Effect.gen(function* () { @@ -808,25 +765,6 @@ describe("legacy db schema declarative generate integration", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("explicit --linked does not route the export source through the Go seam", () => { - const ref = "abcdefghijklmnopqrst"; - const s = setup(tmp.current, { experimental: true, projectId: Option.some(ref) }); - return Effect.gen(function* () { - yield* legacyDbSchemaDeclarativeGenerate(flags({ linked: Option.some(true) })); - expect(s.seamExportCalls.some((call) => call.mode === "baseline")).toBe(false); - }).pipe(Effect.provide(s.layer)); - }); - - it.effect("explicit --local keeps raw-shadow export independent of linked state", () => { - const s = setup(tmp.current, { experimental: true }); - return Effect.gen(function* () { - yield* legacyDbSchemaDeclarativeGenerate(flags({ local: Option.some(true) })); - expect(s.seamExportCalls.some((call) => call.mode === "baseline")).toBe(false); - // No linked ref resolved → no linked-project cache write (Go gates on ProjectRef). - expect(s.cache.cached).toBe(false); - }).pipe(Effect.provide(s.layer)); - }); - it.effect("caches the linked project after generate --linked (Go PersistentPostRun)", () => { const ref = "abcdefghijklmnopqrst"; const s = setup(tmp.current, { experimental: true, projectId: Option.some(ref) }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts index 7429a08638..9780aaa6fb 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts @@ -2,105 +2,22 @@ import { Layer } from "effect"; import { commandRuntimeLayer } from "../../../../../../shared/runtime/command-runtime.layer.ts"; import { stdinLayer } from "../../../../../../shared/runtime/stdin.layer.ts"; -import { legacyHttpClientLayer } from "../../../../../auth/legacy-http-debug.layer.ts"; -import { legacyCliConfigLayer } from "../../../../../config/legacy-cli-config.layer.ts"; -import { legacyDbConfigLayer } from "../../../../../shared/legacy-db-config.layer.ts"; -import { legacyDbConnectionLayer } from "../../../../../shared/legacy-db-connection.layer.ts"; -import { legacyDebugLoggerLayer } from "../../../../../shared/legacy-debug-logger.layer.ts"; -import { legacyDockerRunLayer } from "../../../../../shared/legacy-docker-run.layer.ts"; -import { legacyEdgeRuntimeScriptLayer } from "../../../../../shared/legacy-edge-runtime-script.layer.ts"; import { legacyIdentityStitchLayer } from "../../../../../shared/legacy-identity-stitch.ts"; import { legacyLinkedDbResolverRuntimeLayer } from "../../../../../shared/legacy-management-api-runtime.layer.ts"; -import { legacyPgDeltaSslProbeLayer } from "../../../../../shared/legacy-pgdelta-ssl-probe.layer.ts"; import { legacyTelemetryStateLayer } from "../../../../../telemetry/legacy-telemetry-state.layer.ts"; -import { legacyDeclarativeSeamLayer } from "../../../shared/legacy-pgdelta.seam.layer.ts"; -import { legacyPgDeltaEngineLayer } from "../../../shared/legacy-pgdelta-engine.layer.ts"; -import { legacyPgDeltaNextAdapterLayer } from "../../../shared/legacy-pgdelta-next-adapter.layer.ts"; -import { legacyPgDeltaNextShadowLayer } from "../../../shared/legacy-pgdelta-next-shadow.layer.ts"; - -/** - * Runtime layer for `supabase db schema declarative generate`. - * - * `Output` / global flags come from the legacy root; the Bun - * platform (FileSystem / Path / ChildProcessSpawner / ProcessControl / Tty) from - * `runCli`. This layer adds both pg-delta implementations, the native shadow - * runtime, and the db-config resolver for `--linked` / `--db-url`. - * The bundled implementation runs in-process by default; edge-runtime is retained - * only for the explicit legacy opt-out. Per the "provide doesn't share to siblings" - * rule, `LegacyCliConfig` is provided to every layer that needs it — including - * `seam`, which (as of the fully-native shadow provisioning) also needs `LegacyDbConnection`/ - * `LegacyDockerRun`/`LegacyEdgeRuntimeScript`/`LegacyPgDeltaSslProbe`/`HttpClient` (the native - * shadow's health-check wait, mirroring `db diff`'s own `diff.layers.ts`) explicitly provided, - * not just exposed as sibling entries in the merge below. `legacyDockerRunLayer` is ALSO exposed - * directly (not just provided to `edgeRuntime`/`seam`): the smart-target local-reset prompt now - * calls `legacyResetLocalDatabase` in-process (CLI-2062), whose PG15+ recreate reuses the same - * one-shot migrate jobs `db start`/`db reset` back with this same layer (see those commands' own - * `*.layers.ts`). - */ -const cliConfig = legacyCliConfigLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); - -const dbConfig = legacyDbConfigLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(legacyDebugLoggerLayer), - // The linked db-config resolver snapshots the single `LegacyIdentityStitch` - // (Go's one `sync.Once`); the command runtime must provide it or the bundled - // binary panics with a missing-service error (legacy CLAUDE.md rule 5). - Layer.provide(legacyIdentityStitchLayer), -); - -const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( - Layer.provide(legacyDockerRunLayer), - Layer.provide(cliConfig), -); - -const httpClient = legacyHttpClientLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); -const seam = legacyDeclarativeSeamLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(legacyDockerRunLayer), - Layer.provide(edgeRuntime), - Layer.provide(legacyPgDeltaSslProbeLayer), - Layer.provide(httpClient), -); -const nextShadow = legacyPgDeltaNextShadowLayer.pipe( - Layer.provide(legacyDockerRunLayer), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(httpClient), -); -const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyPgDeltaNextAdapterLayer), - Layer.provide(nextShadow), - Layer.provide(edgeRuntime), - Layer.provide(legacyPgDeltaSslProbeLayer), - Layer.provide(seam), - Layer.provide(legacyDockerRunLayer), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(httpClient), - Layer.provide(legacyDebugLoggerLayer), -); +import { + legacyPgDeltaCommandRuntimeLayer, + legacyPgDeltaDbConfigRuntimeLayer, +} from "../../../shared/legacy-pgdelta-engine.layer.ts"; export const legacyDbSchemaDeclarativeGenerateRuntimeLayer = Layer.mergeAll( - dbConfig, - legacyDbConnectionLayer, - legacyDockerRunLayer, - edgeRuntime, - legacyPgDeltaSslProbeLayer, - httpClient, - seam, - pgDeltaEngine, - cliConfig, + legacyPgDeltaDbConfigRuntimeLayer, + legacyPgDeltaCommandRuntimeLayer, legacyIdentityStitchLayer, legacyTelemetryStateLayer, - // Go's PersistentPostRun writes the linked-project cache for `--linked`; this - // bundle supplies `LegacyLinkedProjectCache` (+ the lazy Management-API runtime - // it needs), mirroring `db query` (`query.layers.ts`). legacyLinkedDbResolverRuntimeLayer(["db", "schema", "declarative", "generate"]).pipe( Layer.provide(legacyIdentityStitchLayer), ), commandRuntimeLayer(["db", "schema", "declarative", "generate"]), - // `stdinLayer`: the confirmation prompts route through `legacyPromptYesNo`, - // whose non-TTY branch reads piped stdin (Go's `Console.ReadLine`). stdinLayer, ); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index fc379824eb..7d4a184f55 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -1050,35 +1050,7 @@ describe("legacy db schema declarative sync integration", () => { expect(JSON.stringify(error)).toContain( "supabase db schema declarative generate --local --overwrite", ); - expect(s.out.promptSelectCalls).toHaveLength(0); expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); - expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); - const output = stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join("")); - expect(output).not.toContain("pg-toolbelt/issues"); - expect(output).not.toContain("circular REFERENCES"); - }).pipe(Effect.provide(s.layer)); - }); - - it.effect("offers adopt, repair, or cancel before a load-failure plan exists", () => { - seedLegacyUuidDeclarative(tmp.current); - const s = setup(tmp.current, { - engineImplementation: "next", - stdinIsTty: true, - planErrors: [legacyUuidLoadError()], - promptSelectResponses: ["cancel"], - }); - return Effect.gen(function* () { - yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); - expect(s.out.promptSelectCalls[0]?.options).toEqual([ - expect.objectContaining({ value: "stage", hint: "recommended" }), - expect.objectContaining({ value: "repair" }), - expect.objectContaining({ value: "cancel" }), - ]); - expect( - s.out.promptSelectCalls[0]?.options.some((option) => option.value === "continue"), - ).toBe(false); - expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); - expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); }).pipe(Effect.provide(s.layer)); }); @@ -1096,10 +1068,6 @@ describe("legacy db schema declarative sync integration", () => { expect(readFileSync(join(tmp.current, "supabase", "database", "extension.sql"), "utf8")).toBe( 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";\n', ); - expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); - expect(stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join(""))).toContain( - "No schema changes found", - ); }).pipe(Effect.provide(s.layer)); }); @@ -1123,7 +1091,6 @@ describe("legacy db schema declarative sync integration", () => { }); return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); - expect(s.declarativeExportCalls).toHaveLength(1); expect(readFileSync(activeMember, "utf8")).toBe(before); expect( readFileSync( @@ -1142,7 +1109,6 @@ describe("legacy db schema declarative sync integration", () => { expect( existsSync(join(tmp.current, "supabase", "database-next", ".pgdelta-export.json")), ).toBe(true); - expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); }).pipe(Effect.provide(s.layer)); }); @@ -1168,7 +1134,6 @@ describe("legacy db schema declarative sync integration", () => { message: expect.stringContaining("pg_cron job refresh download metrics"), }); expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); - expect(s.dbExec).toEqual([]); }).pipe(Effect.provide(s.layer)); }); @@ -1189,30 +1154,6 @@ describe("legacy db schema declarative sync integration", () => { message: expect.stringContaining("[experimental.webhooks]\nenabled = true"), }); expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); - expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); - expect(s.out.promptSelectCalls).toHaveLength(0); - }).pipe(Effect.provide(s.layer)); - }); - - it.effect("keeps explicit extension repair for non-config-managed extensions", () => { - seedDeclarative(tmp.current); - const s = setup(tmp.current, { - engineImplementation: "next", - stdinIsTty: true, - diffSql: 'DROP EXTENSION "pgcrypto";\n', - replannedDiffSql: "", - removals: { extensions: ["pgcrypto"], extensionIntents: [] }, - promptSelectResponses: ["repair"], - }); - return Effect.gen(function* () { - yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); - expect(readFileSync(join(tmp.current, "supabase", "database", "extension.sql"), "utf8")).toBe( - 'CREATE EXTENSION IF NOT EXISTS "pgcrypto" WITH SCHEMA "extensions";\n', - ); - expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); - expect(stripAnsi(s.out.rawChunks.map((chunk) => chunk.text).join(""))).toContain( - "No schema changes found", - ); }).pipe(Effect.provide(s.layer)); }); @@ -1230,7 +1171,6 @@ describe("legacy db schema declarative sync integration", () => { return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); expect(readdirSync(join(tmp.current, "supabase", "migrations"))).toHaveLength(1); - expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); }).pipe(Effect.provide(s.layer)); }, ); @@ -1251,28 +1191,6 @@ describe("legacy db schema declarative sync integration", () => { }).pipe(Effect.provide(s.layer)); }); - it.effect("fails safely instead of repairing when sync is non-interactive", () => { - seedDeclarative(tmp.current); - const s = setup(tmp.current, { - engineImplementation: "next", - diffSql: 'DROP EXTENSION "pgcrypto";\n', - removals: { extensions: ["pgcrypto"], extensionIntents: [] }, - }); - return Effect.gen(function* () { - const exit = yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })).pipe( - Effect.exit, - ); - expect(failError(exit)).toMatchObject({ - _tag: "LegacyDeclarativeCompatibilityError", - message: expect.stringContaining( - 'CREATE EXTENSION IF NOT EXISTS "pgcrypto" WITH SCHEMA "extensions";', - ), - }); - expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); - expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); - }).pipe(Effect.provide(s.layer)); - }); - it.effect("suppresses the compatibility warning when a next export manifest is present", () => { seedDeclarative(tmp.current); writeFileSync( diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts index a64eae44a1..8053094b38 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts @@ -2,104 +2,22 @@ import { Layer } from "effect"; import { commandRuntimeLayer } from "../../../../../../shared/runtime/command-runtime.layer.ts"; import { stdinLayer } from "../../../../../../shared/runtime/stdin.layer.ts"; -import { legacyHttpClientLayer } from "../../../../../auth/legacy-http-debug.layer.ts"; -import { legacyCliConfigLayer } from "../../../../../config/legacy-cli-config.layer.ts"; -import { legacyDbConfigLayer } from "../../../../../shared/legacy-db-config.layer.ts"; -import { legacyDbConnectionLayer } from "../../../../../shared/legacy-db-connection.layer.ts"; -import { legacyDebugLoggerLayer } from "../../../../../shared/legacy-debug-logger.layer.ts"; -import { legacyDockerRunLayer } from "../../../../../shared/legacy-docker-run.layer.ts"; -import { legacyEdgeRuntimeScriptLayer } from "../../../../../shared/legacy-edge-runtime-script.layer.ts"; import { legacyIdentityStitchLayer } from "../../../../../shared/legacy-identity-stitch.ts"; import { legacyLinkedDbResolverRuntimeLayer } from "../../../../../shared/legacy-management-api-runtime.layer.ts"; -import { legacyPgDeltaSslProbeLayer } from "../../../../../shared/legacy-pgdelta-ssl-probe.layer.ts"; import { legacyTelemetryStateLayer } from "../../../../../telemetry/legacy-telemetry-state.layer.ts"; -import { legacyDeclarativeSeamLayer } from "../../../shared/legacy-pgdelta.seam.layer.ts"; -import { legacyPgDeltaEngineLayer } from "../../../shared/legacy-pgdelta-engine.layer.ts"; -import { legacyPgDeltaNextAdapterLayer } from "../../../shared/legacy-pgdelta-next-adapter.layer.ts"; -import { legacyPgDeltaNextShadowLayer } from "../../../shared/legacy-pgdelta-next-shadow.layer.ts"; - -/** - * Runtime layer for `supabase db schema declarative sync`. Sync diffs against the - * local database, but its no-declarative-files bootstrap delegates to the shared - * smart-generate flow, which can target local / linked / custom — so it needs the - * db-config resolver too. `Output` / global flags and the Bun platform come from the - * legacy root / `runCli`. Per the "provide doesn't share to siblings" rule, - * `LegacyCliConfig` is provided to every layer that needs it — including `seam`, - * which (as of the fully-native shadow provisioning) also needs `LegacyDbConnection`/ - * `LegacyDockerRun`/`LegacyEdgeRuntimeScript`/`LegacyPgDeltaSslProbe`/`HttpClient` (the - * native shadow's health-check wait, mirroring `db diff`'s own `diff.layers.ts`) - * explicitly provided, not just exposed as sibling entries in the merge below. - * `legacyDockerRunLayer` is ALSO exposed directly (not just provided to - * `edgeRuntime`/`seam`): both the smart-target bootstrap's local-reset prompt and the - * failed-apply recovery reset now call `legacyResetLocalDatabase` in-process - * (CLI-2062), whose PG15+ recreate reuses the same one-shot migrate jobs `db - * start`/`db reset` back with this same layer (see those commands' own - * `*.layers.ts`). - */ -const cliConfig = legacyCliConfigLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); - -const dbConfig = legacyDbConfigLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(legacyDebugLoggerLayer), - // The linked db-config resolver snapshots the single `LegacyIdentityStitch` - // (Go's one `sync.Once`); the command runtime must provide it or the bundled - // binary panics with a missing-service error (legacy CLAUDE.md rule 5). - Layer.provide(legacyIdentityStitchLayer), -); - -const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( - Layer.provide(legacyDockerRunLayer), - Layer.provide(cliConfig), -); - -const httpClient = legacyHttpClientLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); -const seam = legacyDeclarativeSeamLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(legacyDockerRunLayer), - Layer.provide(edgeRuntime), - Layer.provide(legacyPgDeltaSslProbeLayer), - Layer.provide(httpClient), -); -const nextShadow = legacyPgDeltaNextShadowLayer.pipe( - Layer.provide(legacyDockerRunLayer), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(httpClient), -); -const pgDeltaEngine = legacyPgDeltaEngineLayer.pipe( - Layer.provide(cliConfig), - Layer.provide(legacyPgDeltaNextAdapterLayer), - Layer.provide(nextShadow), - Layer.provide(edgeRuntime), - Layer.provide(legacyPgDeltaSslProbeLayer), - Layer.provide(seam), - Layer.provide(legacyDockerRunLayer), - Layer.provide(legacyDbConnectionLayer), - Layer.provide(httpClient), - Layer.provide(legacyDebugLoggerLayer), -); +import { + legacyPgDeltaCommandRuntimeLayer, + legacyPgDeltaDbConfigRuntimeLayer, +} from "../../../shared/legacy-pgdelta-engine.layer.ts"; export const legacyDbSchemaDeclarativeSyncRuntimeLayer = Layer.mergeAll( - dbConfig, - legacyDockerRunLayer, - edgeRuntime, - legacyPgDeltaSslProbeLayer, - httpClient, - seam, - pgDeltaEngine, - legacyDbConnectionLayer, - cliConfig, + legacyPgDeltaDbConfigRuntimeLayer, + legacyPgDeltaCommandRuntimeLayer, legacyIdentityStitchLayer, legacyTelemetryStateLayer, - // Go's PersistentPostRun writes the linked-project cache when the bootstrap path - // resolved a linked ref; this bundle supplies `LegacyLinkedProjectCache` (+ the - // lazy Management-API runtime it needs), mirroring `generate` (`generate.layers.ts`). legacyLinkedDbResolverRuntimeLayer(["db", "schema", "declarative", "sync"]).pipe( Layer.provide(legacyIdentityStitchLayer), ), commandRuntimeLayer(["db", "schema", "declarative", "sync"]), - // `stdinLayer`: the confirmation prompts route through `legacyPromptYesNo`, - // whose non-TTY branch reads piped stdin (Go's `Console.ReadLine`). stdinLayer, ); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts index 3d5cb85a5f..b01a31f11b 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts @@ -6,23 +6,40 @@ import type { GlobalFlag } from "effect/unstable/cli"; import { CliArgs } from "../../../../shared/cli/cli-args.service.ts"; import { Output } from "../../../../shared/output/output.service.ts"; import { RuntimeInfo } from "../../../../shared/runtime/runtime-info.service.ts"; +import { legacyHttpClientLayer } from "../../../auth/legacy-http-debug.layer.ts"; +import { legacyCliConfigLayer } from "../../../config/legacy-cli-config.layer.ts"; import { LegacyCliConfig } from "../../../config/legacy-cli-config.service.ts"; +import { legacyDbConfigLayer } from "../../../shared/legacy-db-config.layer.ts"; +import { legacyDbConnectionLayer } from "../../../shared/legacy-db-connection.layer.ts"; import { LegacyDbConnection } from "../../../shared/legacy-db-connection.service.ts"; import { legacyLoadProjectEnv } from "../../../shared/legacy-db-config.toml-read.ts"; +import { legacyDebugLoggerLayer } from "../../../shared/legacy-debug-logger.layer.ts"; import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; +import { legacyDockerRunLayer } from "../../../shared/legacy-docker-run.layer.ts"; import { LegacyDockerRun } from "../../../shared/legacy-docker-run.service.ts"; +import { legacyEdgeRuntimeScriptLayer } from "../../../shared/legacy-edge-runtime-script.layer.ts"; import { LegacyEdgeRuntimeScript } from "../../../shared/legacy-edge-runtime-script.service.ts"; +import { legacyIdentityStitchLayer } from "../../../shared/legacy-identity-stitch.ts"; +import { legacyPgDeltaSslProbeLayer } from "../../../shared/legacy-pgdelta-ssl-probe.layer.ts"; import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; +import { legacyResolvePgDeltaImplementation } from "../../../shared/legacy-pgdelta-next-flag.ts"; import { legacyPgDeltaLegacyEngineLayer } from "./legacy-pgdelta-engine.legacy.layer.ts"; import { legacyPgDeltaNextEngineLayer } from "./legacy-pgdelta-engine.next.layer.ts"; import { LegacyPgDeltaEngine } from "./legacy-pgdelta-engine.service.ts"; +import { legacyPgDeltaNextAdapterLayer } from "./legacy-pgdelta-next-adapter.layer.ts"; import { LegacyPgDeltaNextAdapter } from "./legacy-pgdelta-next-adapter.service.ts"; +import { legacyPgDeltaNextShadowLayer } from "./legacy-pgdelta-next-shadow.layer.ts"; import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; +import { legacyDeclarativeSeamLayer } from "./legacy-pgdelta.seam.layer.ts"; import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; -import { legacyResolvePgDeltaImplementation } from "../../../shared/legacy-pgdelta-next-flag.ts"; const FLAG = "SUPABASE_USE_PG_DELTA_NEXT"; +export const legacyPgDeltaImplementationFlag = ( + shellValue: string | undefined, + projectValue: string | undefined, +) => shellValue ?? projectValue; + const resolveAndLog = Effect.fnUntraced(function* (raw: string | undefined) { const debug = yield* LegacyDebugLogger; const implementation = legacyResolvePgDeltaImplementation(raw); @@ -59,7 +76,7 @@ export const legacyPgDeltaEngineLayer = Layer.unwrap( const projectEnv = yield* legacyLoadProjectEnv(fs, path, cliConfig.workdir); // godotenv.Load never replaces a shell value, including an empty or invalid // one, so presence in process.env must suppress the project-file fallback. - const raw = process.env[FLAG] ?? projectEnv[FLAG]; + const raw = legacyPgDeltaImplementationFlag(process.env[FLAG], projectEnv[FLAG]); const implementation = yield* resolveAndLog(raw); return selectProductionLayer(implementation); }), @@ -91,3 +108,56 @@ function selectProductionLayer( > { return implementation === "next" ? legacyPgDeltaNextEngineLayer : legacyPgDeltaLegacyEngineLayer; } + +export const legacyPgDeltaCliConfigRuntimeLayer = legacyCliConfigLayer.pipe( + Layer.provide(legacyDebugLoggerLayer), +); + +export const legacyPgDeltaDbConfigRuntimeLayer = legacyDbConfigLayer.pipe( + Layer.provide(legacyPgDeltaCliConfigRuntimeLayer), + Layer.provide(legacyDbConnectionLayer), + Layer.provide(legacyDebugLoggerLayer), + Layer.provide(legacyIdentityStitchLayer), +); + +const edgeRuntime = legacyEdgeRuntimeScriptLayer.pipe( + Layer.provide(legacyDockerRunLayer), + Layer.provide(legacyPgDeltaCliConfigRuntimeLayer), +); +const httpClient = legacyHttpClientLayer.pipe(Layer.provide(legacyDebugLoggerLayer)); +const seam = legacyDeclarativeSeamLayer.pipe( + Layer.provide(legacyPgDeltaCliConfigRuntimeLayer), + Layer.provide(legacyDbConnectionLayer), + Layer.provide(legacyDockerRunLayer), + Layer.provide(edgeRuntime), + Layer.provide(legacyPgDeltaSslProbeLayer), + Layer.provide(httpClient), +); +const nextShadow = legacyPgDeltaNextShadowLayer.pipe( + Layer.provide(legacyDockerRunLayer), + Layer.provide(legacyDbConnectionLayer), + Layer.provide(httpClient), +); +const engine = legacyPgDeltaEngineLayer.pipe( + Layer.provide(legacyPgDeltaCliConfigRuntimeLayer), + Layer.provide(legacyPgDeltaNextAdapterLayer), + Layer.provide(nextShadow), + Layer.provide(edgeRuntime), + Layer.provide(legacyPgDeltaSslProbeLayer), + Layer.provide(seam), + Layer.provide(legacyDockerRunLayer), + Layer.provide(legacyDbConnectionLayer), + Layer.provide(httpClient), + Layer.provide(legacyDebugLoggerLayer), +); + +export const legacyPgDeltaCommandRuntimeLayer = Layer.mergeAll( + legacyDbConnectionLayer, + legacyDockerRunLayer, + edgeRuntime, + legacyPgDeltaSslProbeLayer, + httpClient, + seam, + engine, + legacyPgDeltaCliConfigRuntimeLayer, +); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts index 202984d1e3..84feb4087f 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts @@ -1,38 +1,14 @@ -import { mkdirSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; -import { Effect, Exit, Layer, Option } from "effect"; -import * as FetchHttpClient from "effect/unstable/http/FetchHttpClient"; -import * as BunServices from "@effect/platform-bun/BunServices"; +import { Effect, Exit, Layer } from "effect"; import { it } from "@effect/vitest"; -import { afterEach, describe, expect } from "vitest"; +import { describe, expect } from "vitest"; -import { - mockLegacyCliConfig, - useLegacyTempWorkdir, -} from "../../../../../tests/helpers/legacy-mocks.ts"; -import { mockOutput, mockRuntimeInfo } from "../../../../../tests/helpers/mocks.ts"; -import { CliArgs } from "../../../../shared/cli/cli-args.service.ts"; -import { - LegacyDebugFlag, - LegacyExperimentalFlag, - LegacyNetworkIdFlag, -} from "../../../../shared/legacy/global-flags.ts"; -import { LegacyDbConnection } from "../../../shared/legacy-db-connection.service.ts"; import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; -import { LegacyDockerRun } from "../../../shared/legacy-docker-run.service.ts"; -import { LegacyEdgeRuntimeScript } from "../../../shared/legacy-edge-runtime-script.service.ts"; -import { LegacyPgDeltaSslProbe } from "../../../shared/legacy-pgdelta-ssl-probe.service.ts"; -import { LegacyDeclarativeSeam } from "./legacy-pgdelta.seam.service.ts"; -import { LegacyPgDeltaNextAdapter } from "./legacy-pgdelta-next-adapter.service.ts"; -import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; import { - legacyPgDeltaEngineLayer, + legacyPgDeltaImplementationFlag, legacyPgDeltaEngineSelectorLayer, } from "./legacy-pgdelta-engine.layer.ts"; import { LegacyPgDeltaEngine } from "./legacy-pgdelta-engine.service.ts"; -const FLAG = "SUPABASE_USE_PG_DELTA_NEXT"; - function debugLayer(messages: Array) { return Layer.succeed(LegacyDebugLogger, { debug: (message) => Effect.sync(() => messages.push(message)), @@ -53,47 +29,6 @@ function metadataLayer(implementation: "next" | "legacy") { ); } -const unusedLegacyRuntime = Layer.mergeAll( - BunServices.layer, - FetchHttpClient.layer, - Layer.succeed(LegacyEdgeRuntimeScript, { - run: () => Effect.die("edge runtime not needed"), - }), - Layer.succeed(LegacyPgDeltaSslProbe, { - requireSsl: () => Effect.die("SSL probe not needed"), - requireSslForHost: () => Effect.die("SSL probe not needed"), - }), - Layer.succeed(LegacyDeclarativeSeam, { - exportCatalog: () => Effect.die("catalog not needed"), - ensureLocalDatabaseStarted: () => Effect.die("local start not needed"), - ensureLocalPostgresImageCurrent: () => Effect.die("image check not needed"), - }), - Layer.succeed(LegacyPgDeltaNextAdapter, { - diff: () => Effect.die("adapter not needed"), - exportDeclarativeSchema: () => Effect.die("adapter not needed"), - planDeclarativeSchema: () => Effect.die("adapter not needed"), - captureSnapshot: () => Effect.die("adapter not needed"), - }), - Layer.succeed(LegacyPgDeltaNextShadow, { - provisionMigrations: () => Effect.die("next migrations shadow not needed"), - provisionPlan: () => Effect.die("next plan shadows not needed"), - }), - Layer.succeed(LegacyDbConnection, { - connect: () => Effect.die("database connection not needed"), - }), - Layer.succeed(LegacyDockerRun, { - run: () => Effect.die("docker run not needed"), - runCapture: () => Effect.die("docker capture not needed"), - runStream: () => Effect.die("docker stream not needed"), - }), - Layer.succeed(CliArgs, { args: [] }), - Layer.succeed(LegacyDebugFlag, false), - Layer.succeed(LegacyExperimentalFlag, false), - Layer.succeed(LegacyNetworkIdFlag, Option.none()), - mockRuntimeInfo(), - mockOutput().layer, -); - describe("legacyPgDeltaEngineSelectorLayer", () => { it.effect("selects next by default and logs the decision once", () => { const messages: Array = []; @@ -111,22 +46,6 @@ describe("legacyPgDeltaEngineSelectorLayer", () => { ); }); - it.effect("selects legacy only for an explicit false value", () => { - const messages: Array = []; - return Effect.gen(function* () { - const engine = yield* LegacyPgDeltaEngine; - expect(engine.implementation).toBe("legacy"); - expect(messages).toEqual(["Using pg-delta legacy implementation."]); - }).pipe( - Effect.provide( - legacyPgDeltaEngineSelectorLayer("false", { - next: metadataLayer("next"), - legacy: metadataLayer("legacy"), - }).pipe(Layer.provide(debugLayer(messages))), - ), - ); - }); - it.effect("does not invoke legacy after a selected next operation fails", () => { const messages: Array = []; let nextCalls = 0; @@ -203,76 +122,10 @@ describe("legacyPgDeltaEngineSelectorLayer", () => { }); }); -describe("legacyPgDeltaEngineLayer", () => { - const tmp = useLegacyTempWorkdir("pgdelta-engine-selector-"); - - afterEach(() => { - delete process.env[FLAG]; - }); - - const provideProductionSelector = (messages: Array) => - legacyPgDeltaEngineLayer.pipe( - Layer.provide(unusedLegacyRuntime), - Layer.provide(debugLayer(messages)), - Layer.provide(mockLegacyCliConfig({ workdir: tmp.current })), - ); - - const writeProjectFlag = (value: string) => { - mkdirSync(join(tmp.current, "supabase"), { recursive: true }); - writeFileSync(join(tmp.current, "supabase", ".env"), `${FLAG}=${value}\n`); - }; - - it.effect("selects legacy from the project environment when the shell is unset", () => { - const messages: Array = []; - writeProjectFlag("false"); - - return Effect.gen(function* () { - const engine = yield* LegacyPgDeltaEngine; - expect(engine.implementation).toBe("legacy"); - expect(messages).toEqual(["Using pg-delta legacy implementation."]); - }).pipe(Effect.provide(provideProductionSelector(messages))); - }); - - it.effect("prefers a true shell value over a false project value", () => { - const messages: Array = []; - process.env[FLAG] = "true"; - writeProjectFlag("false"); - - return Effect.gen(function* () { - expect((yield* LegacyPgDeltaEngine).implementation).toBe("next"); - }).pipe(Effect.provide(provideProductionSelector(messages))); - }); - - it.effect("prefers a false shell value over a true project value", () => { - const messages: Array = []; - process.env[FLAG] = "false"; - writeProjectFlag("true"); - - return Effect.gen(function* () { - expect((yield* LegacyPgDeltaEngine).implementation).toBe("legacy"); - }).pipe(Effect.provide(provideProductionSelector(messages))); - }); - - it.effect("defaults to next when neither environment defines the flag", () => { - const messages: Array = []; - return Effect.gen(function* () { - expect((yield* LegacyPgDeltaEngine).implementation).toBe("next"); - expect(messages).toEqual(["Using pg-delta next implementation."]); - }).pipe(Effect.provide(provideProductionSelector(messages))); - }); - - it.effect("reads the environment once for the command-scoped service", () => { - const messages: Array = []; - process.env[FLAG] = "false"; - - return Effect.gen(function* () { - const first = yield* LegacyPgDeltaEngine; - process.env[FLAG] = "true"; - const second = yield* LegacyPgDeltaEngine; - - expect(first).toBe(second); - expect(second.implementation).toBe("legacy"); - expect(messages).toEqual(["Using pg-delta legacy implementation."]); - }).pipe(Effect.provide(provideProductionSelector(messages))); +describe("legacyPgDeltaImplementationFlag", () => { + it("prefers shell presence and otherwise uses the project value", () => { + expect(legacyPgDeltaImplementationFlag("true", "false")).toBe("true"); + expect(legacyPgDeltaImplementationFlag("", "false")).toBe(""); + expect(legacyPgDeltaImplementationFlag(undefined, "false")).toBe("false"); }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts index 394d732fe8..bddfaf7edd 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts @@ -6,7 +6,7 @@ import { mockOutput } from "../../../../../tests/helpers/mocks.ts"; import { LegacyDebugLogger } from "../../../shared/legacy-debug-logger.service.ts"; import { LegacyDeclarativeShadowDbError } from "./legacy-pgdelta.errors.ts"; import { legacyPgDeltaNextEngineLayer } from "./legacy-pgdelta-engine.next.layer.ts"; -import { LegacyPgDeltaEngine, LegacyPgDeltaEngineError } from "./legacy-pgdelta-engine.service.ts"; +import { LegacyPgDeltaEngine } from "./legacy-pgdelta-engine.service.ts"; import { LegacyPgDeltaNextAdapter } from "./legacy-pgdelta-next-adapter.service.ts"; import { LegacyPgDeltaNextShadow } from "./legacy-pgdelta-next-shadow.service.ts"; import type { LegacyDbTomlValues } from "../../../shared/legacy-db-config.toml-read.ts"; @@ -175,31 +175,4 @@ describe("pg-delta next shadow selection", () => { expect(state).toEqual({ migrations: 0, plan: 1 }); }).pipe(Effect.provide(layer)); }); - - it.effect("returns malformed explicit URLs as typed failures rather than defects", () => { - const { state, layer } = setup(); - return Effect.gen(function* () { - const engine = yield* LegacyPgDeltaEngine; - const error = yield* engine - .diffExplicit({ - ...common, - source: { - kind: "database", - ref: "postgresql://postgres:source-secret@[/postgres", - connectOptions: { isLocal: false, dnsResolver: "native" }, - }, - desired: { - kind: "database", - ref: "postgresql://postgres:desired-secret@[/postgres", - connectOptions: { isLocal: false, dnsResolver: "native" }, - }, - }) - .pipe(Effect.flip); - - expect(error).toBeInstanceOf(LegacyPgDeltaEngineError); - expect(String(error.cause)).not.toContain("source-secret"); - expect(String(error.cause)).not.toContain("desired-secret"); - expect(state).toEqual({ migrations: 0, plan: 0 }); - }).pipe(Effect.provide(layer)); - }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 03c3d4e01c..ee60a9a4f9 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -1,4 +1,5 @@ import { Clock, Effect, FileSystem, Layer, Path } from "effect"; +import type { Pool } from "pg"; import { Output } from "../../../../shared/output/output.service.ts"; import { @@ -31,13 +32,6 @@ import { legacyReportPgDeltaNextDiagnostics, } from "./legacy-pgdelta-next-diagnostics.ts"; -/** Shared by both declarative planner entrypoints over the full isolated baseline. */ -export const legacyPgDeltaNextIsolatedShadowPlanOptions = { - isolatedShadow: true, - seedAssumedSchemas: false, - strictDataStatements: true, -} as const; - function legacyPgDeltaNextConnectSuggestion(cause: unknown): string | undefined { if (cause instanceof LegacyDbConnectError) return cause.suggestion; if (typeof cause !== "object" || cause === null) return undefined; @@ -196,6 +190,37 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( ); }; + const diffPools = ( + input: { + readonly context: { readonly cwd: string }; + readonly schema: ReadonlyArray; + readonly formatOptions: string; + readonly debug: boolean; + readonly strictCoverage: boolean; + }, + sourcePool: Pool, + desiredPool: Pool, + ) => + Effect.gen(function* () { + const result = yield* adapter.diff({ + sourcePool, + desiredPool, + allowDrops: true, + debug: input.debug, + schema: input.schema, + formatOptions: input.formatOptions, + }); + const debugDirectory = + result.debug !== undefined + ? yield* saveDebugArtifacts(input.context.cwd, "diff", { + ...result.debug, + diagnostics: result.diagnostics, + }) + : undefined; + yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage, input.debug); + return normalizeNextDiff(result, debugDirectory); + }); + return LegacyPgDeltaEngine.of({ implementation: "next", diffExplicit: (input) => @@ -249,23 +274,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( [endpointPool(input.source), endpointPool(input.desired)], { concurrency: 2 }, ); - const result = yield* adapter.diff({ - sourcePool, - desiredPool, - allowDrops: true, - debug: input.debug, - schema: input.schema, - formatOptions: input.formatOptions, - }); - const debugDirectory = - result.debug !== undefined - ? yield* saveDebugArtifacts(input.context.cwd, "diff", { - ...result.debug, - diagnostics: result.diagnostics, - }) - : undefined; - yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage, input.debug); - return normalizeNextDiff(result, debugDirectory); + return yield* diffPools(input, sourcePool, desiredPool); }), ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), diffDatabase: (input) => @@ -273,23 +282,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( Effect.gen(function* () { const migrationsPool = yield* acquireDatabase(input.source); const desiredPool = yield* acquireDatabase(input.target); - const result = yield* adapter.diff({ - sourcePool: migrationsPool, - desiredPool, - allowDrops: true, - debug: input.debug, - schema: input.schema, - formatOptions: input.formatOptions, - }); - const debugDirectory = - result.debug !== undefined - ? yield* saveDebugArtifacts(input.context.cwd, "diff", { - ...result.debug, - diagnostics: result.diagnostics, - }) - : undefined; - yield* reportDiagnostics("diff", result.diagnostics, input.strictCoverage, input.debug); - return normalizeNextDiff(result, debugDirectory); + return yield* diffPools(input, migrationsPool, desiredPool); }), ).pipe(Effect.mapError(legacyPgDeltaNextEngineError)), exportDeclarativeSchema: (input) => @@ -298,13 +291,12 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( const pool = yield* acquireDatabase(input.target); const result = yield* adapter.exportDeclarativeSchema({ pool, - layout: "grouped", schema: input.schema, formatOptions: input.formatOptions, }); if (input.debug) { const capture = yield* adapter - .captureSnapshot({ pool, redactSecrets: true }) + .captureSnapshot({ pool }) .pipe(Effect.orElseSucceed(() => undefined)); yield* saveDebugArtifacts(input.context.cwd, "declarativeExport", { ...(capture !== undefined ? { desiredSnapshot: capture.snapshot } : {}), @@ -354,8 +346,6 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( files: input.files, allowDrops: true, debug: input.debug, - reorder: true, - ...legacyPgDeltaNextIsolatedShadowPlanOptions, schema: input.schema, formatOptions: input.formatOptions, ...(input.manifest !== undefined ? { manifest: input.manifest } : {}), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts index 5cf88b05ac..4c44c187df 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts @@ -2,22 +2,9 @@ import { Effect } from "effect"; import { describe, expect, it } from "vitest"; import type { LegacyPgDeltaDatabaseEndpoint } from "./legacy-pgdelta-engine.service.ts"; -import { - legacyParsePgDeltaNextEndpoint, - legacyPgDeltaNextIsolatedShadowPlanOptions, -} from "./legacy-pgdelta-engine.next.layer.ts"; +import { legacyParsePgDeltaNextEndpoint } from "./legacy-pgdelta-engine.next.layer.ts"; import { LegacyPgDeltaEngineError } from "./legacy-pgdelta-engine.service.ts"; -describe("legacyPgDeltaNextIsolatedShadowPlanOptions", () => { - it("uses the isolated full-baseline mode shared by both declarative planner entrypoints", () => { - expect(legacyPgDeltaNextIsolatedShadowPlanOptions).toEqual({ - isolatedShadow: true, - seedAssumedSchemas: false, - strictDataStatements: true, - }); - }); -}); - describe("legacyParsePgDeltaNextEndpoint", () => { it("fails malformed explicit URLs through the typed error channel and redacts passwords", () => { const endpoint = { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts index 75a50a6d4c..866760f72e 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.unit.test.ts @@ -6,21 +6,6 @@ import { LegacyPgDeltaEngineError } from "./legacy-pgdelta-engine.service.ts"; import { LegacyPgDeltaNextError } from "./legacy-pgdelta-next-adapter.service.ts"; describe("pg-delta next engine errors", () => { - it("preserves database connection suggestions when wrapping failures", () => { - const cause = new LegacyDbConnectError({ - message: "failed to connect to postgres", - suggestion: "Retry with --dns-resolver https.", - }); - - expect(legacyPgDeltaNextEngineError(cause)).toEqual( - new LegacyPgDeltaEngineError({ - message: "failed to connect to postgres", - suggestion: "Retry with --dns-resolver https.", - cause, - }), - ); - }); - it("finds connection suggestions nested in adapter failures", () => { const cause = new LegacyDbConnectError({ message: "failed to connect to postgres", @@ -32,9 +17,11 @@ describe("pg-delta next engine errors", () => { cause, }); - expect(legacyPgDeltaNextEngineError(adapterError).suggestion).toBe( - "Retry with --dns-resolver https.", - ); + const error = legacyPgDeltaNextEngineError(adapterError); + expect(error).toBeInstanceOf(LegacyPgDeltaEngineError); + expect(error.message).toBe("Database diff failed"); + expect(error.suggestion).toBe("Retry with --dns-resolver https."); + expect(error.cause).toBe(adapterError); }); it("preserves structured diagnostics from adapter failures", () => { @@ -61,9 +48,4 @@ describe("pg-delta next engine errors", () => { }), ); }); - - it("does not wrap an existing engine error again", () => { - const error = new LegacyPgDeltaEngineError({ message: "blocked", cause: "diagnostic" }); - expect(legacyPgDeltaNextEngineError(error)).toBe(error); - }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.unit.test.ts deleted file mode 100644 index 8a61d07532..0000000000 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.unit.test.ts +++ /dev/null @@ -1,65 +0,0 @@ -import { mkdirSync, symlinkSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; - -import { BunServices } from "@effect/platform-bun"; -import { describe, expect, it } from "@effect/vitest"; -import { Effect, FileSystem, Path } from "effect"; - -import { useLegacyTempWorkdir } from "../../../../../tests/helpers/legacy-mocks.ts"; -import { legacyWalkSqlFiles } from "../../../shared/legacy-glob.ts"; -import { LegacyLoadPgDeltaSqlFiles } from "./legacy-pgdelta-files.ts"; - -const load = (directory: string) => - Effect.gen(function* () { - const fs = yield* FileSystem.FileSystem; - const path = yield* Path.Path; - return yield* LegacyLoadPgDeltaSqlFiles(fs, path, directory); - }).pipe(Effect.provide(BunServices.layer)); - -describe("LegacyLoadPgDeltaSqlFiles", () => { - const tmp = useLegacyTempWorkdir("legacy-pgdelta-files-"); - - it.effect("does not follow symlinked directories", () => { - const schemas = join(tmp.current, "schemas"); - const outside = join(tmp.current, "outside"); - mkdirSync(schemas); - mkdirSync(outside); - writeFileSync(join(schemas, "kept.sql"), "select 'kept';"); - writeFileSync(join(outside, "hidden.sql"), "select 'hidden';"); - symlinkSync(outside, join(schemas, "linked"), "dir"); - - return Effect.gen(function* () { - const files = yield* load(schemas); - expect(files).toEqual([{ name: "kept.sql", sql: "select 'kept';" }]); - }); - }); - - it.effect("ignores uppercase .SQL files", () => { - const schemas = join(tmp.current, "schemas"); - mkdirSync(schemas); - writeFileSync(join(schemas, "included.sql"), "select 1;"); - writeFileSync(join(schemas, "ignored.SQL"), "select 2;"); - - return Effect.gen(function* () { - const files = yield* load(schemas); - expect(files).toEqual([{ name: "included.sql", sql: "select 1;" }]); - }); - }); - - it.effect("preserves the shared walker's deterministic UTF-8 byte ordering", () => { - const schemas = join(tmp.current, "schemas"); - const privateUse = "a\u{e000}.sql"; - const supplementary = "a\u{1f600}.sql"; - mkdirSync(schemas); - writeFileSync(join(schemas, supplementary), "select 2;"); - writeFileSync(join(schemas, privateUse), "select 1;"); - - return Effect.gen(function* () { - const fs = yield* FileSystem.FileSystem; - const walked = yield* legacyWalkSqlFiles(fs, schemas, ""); - const files = yield* LegacyLoadPgDeltaSqlFiles(fs, yield* Path.Path, schemas); - expect(files.map((file) => file.name)).toEqual(walked); - expect(files.map((file) => file.name)).toEqual([privateUse, supplementary]); - }).pipe(Effect.provide(BunServices.layer)); - }); -}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index 3e24be26b3..b9b45b8e5f 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -27,7 +27,6 @@ import { LegacyPgDeltaNextError, type LegacyPgDeltaNextAdapterShape, type LegacyPgDeltaNextDeclarativeExportInput, - type LegacyPgDeltaNextDeclarativeManifestInput, type LegacyPgDeltaNextDeclarativePlanInput, type LegacyPgDeltaNextDiagnostic, type LegacyPgDeltaNextDiagnosticOrigin, @@ -87,6 +86,7 @@ interface LegacyPgDeltaNextLibrarySchemaExport { } type LegacyPgDeltaNextLibraryExportOptions = ReturnType; +type LegacyPgDeltaNextLibraryPlanOptions = ReturnType; interface LegacyPgDeltaNextLibrarySchemaPlan { readonly plan: Plan; @@ -123,7 +123,7 @@ export interface LegacyPgDeltaNextLibraries Promise>; readonly serializeSnapshot: ( factBase: FactBase, @@ -470,69 +470,24 @@ function legacyPgDeltaNextExportOptions(input: LegacyPgDeltaNextDeclarativeExpor const format = legacyPgDeltaNextFormatOptions(input.formatOptions); return { profile: legacyPgDeltaNextProfile(input.schema), - ...(input.scope !== undefined ? { scope: input.scope } : {}), - ...(input.redactSecrets !== undefined ? { redactSecrets: input.redactSecrets } : {}), - ...(input.restrictToApplier !== undefined - ? { resolveOptions: { restrictToApplier: input.restrictToApplier } } - : {}), - ...(input.layout !== undefined ? { layout: input.layout } : {}), - ...(input.grouping !== undefined - ? { - grouping: { - ...(input.grouping.mode !== undefined ? { mode: input.grouping.mode } : {}), - ...(input.grouping.groupPatterns !== undefined - ? { groupPatterns: [...input.grouping.groupPatterns] } - : {}), - ...(input.grouping.flatSchemas !== undefined - ? { flatSchemas: [...input.grouping.flatSchemas] } - : {}), - ...(input.grouping.autoGroupPartitions !== undefined - ? { autoGroupPartitions: input.grouping.autoGroupPartitions } - : {}), - }, - } - : {}), - ...(input.defaultOwner !== undefined ? { defaultOwner: input.defaultOwner } : {}), + layout: "grouped" as const, ...(format !== undefined ? { format } : {}), - ...(input.onWarning !== undefined ? { onWarning: input.onWarning } : {}), - }; -} - -function legacyPgDeltaNextManifest(manifest: LegacyPgDeltaNextDeclarativeManifestInput) { - return { - ...(manifest.redactSecrets !== undefined ? { redactSecrets: manifest.redactSecrets } : {}), - ...(manifest.profile !== undefined ? { profile: manifest.profile } : {}), - ...(manifest.scope !== undefined ? { scope: manifest.scope } : {}), - ...(manifest.baselineDigest !== undefined ? { baselineDigest: manifest.baselineDigest } : {}), - ...(manifest.defaultOwner !== undefined ? { defaultOwner: manifest.defaultOwner } : {}), - ...(manifest.files !== undefined ? { files: [...manifest.files] } : {}), }; } function legacyPgDeltaNextPlanOptions(input: LegacyPgDeltaNextDeclarativePlanInput) { + let manifest; + if (input.manifest !== undefined) { + const { files, ...metadata } = input.manifest; + manifest = { ...metadata, ...(files !== undefined ? { files: [...files] } : {}) }; + } return { profile: legacyPgDeltaNextProfile(input.schema), - ...(input.scope !== undefined ? { scope: input.scope } : {}), - ...(input.manifest !== undefined - ? { manifest: legacyPgDeltaNextManifest(input.manifest) } - : {}), - ...(input.redactSecrets !== undefined ? { redactSecrets: input.redactSecrets } : {}), - ...(input.skipClusterDdl !== undefined ? { skipClusterDdl: input.skipClusterDdl } : {}), - ...(input.isolatedShadow !== undefined ? { isolatedShadow: input.isolatedShadow } : {}), - ...(input.seedAssumedSchemas !== undefined - ? { seedAssumedSchemas: input.seedAssumedSchemas } - : {}), - ...(input.restrictToApplier !== undefined - ? { resolveOptions: { restrictToApplier: input.restrictToApplier } } - : {}), - ...(input.strictFunctionBodies !== undefined - ? { strictFunctionBodies: input.strictFunctionBodies } - : {}), - ...(input.strictDataStatements !== undefined - ? { strictDataStatements: input.strictDataStatements } - : {}), - reorder: input.reorder ?? true, - ...(input.onWarning !== undefined ? { onWarning: input.onWarning } : {}), + ...(manifest !== undefined ? { manifest } : {}), + isolatedShadow: true, + seedAssumedSchemas: false, + strictDataStatements: true, + reorder: true, }; } @@ -543,24 +498,18 @@ function legacyMakePgDeltaNextAdapter legacyTryPgDeltaNext("diff", async () => { const format = legacyPgDeltaNextFormatOptions(input.formatOptions); - const redactSecrets = input.redactSecrets ?? true; const profile = await libraries.resolveProfile( input.sourcePool, - { - redactSecrets, - ...(input.restrictToApplier !== undefined - ? { restrictToApplier: input.restrictToApplier } - : {}), - }, + { redactSecrets: true }, input.schema, ); const [source, desired] = await Promise.all([ - profile.extract(input.sourcePool, { redactSecrets }), - profile.extract(input.desiredPool, { redactSecrets }), + profile.extract(input.sourcePool, { redactSecrets: true }), + profile.extract(input.desiredPool, { redactSecrets: true }), ]); const generatedPlan = libraries.plan(source.factBase, desired.factBase, { ...profile.planOptions, - redactSecrets, + redactSecrets: true, }); const rendered = libraries.renderPlanFiles(generatedPlan, { allowDrops: input.allowDrops, @@ -592,12 +541,12 @@ function legacyMakePgDeltaNextAdapter legacyTryPgDeltaNext("declarativePlan", async () => { const format = legacyPgDeltaNextFormatOptions(input.formatOptions); - const planningInput = { ...input, reorder: input.reorder ?? true }; const result = await libraries.planSchemaFiles( input.targetPool, input.shadowPool, input.files, - planningInput, + legacyPgDeltaNextPlanOptions(input), ); const rendered = libraries.renderPlanFiles(result.plan, { allowDrops: input.allowDrops, @@ -677,22 +625,16 @@ function legacyMakePgDeltaNextAdapter legacyTryPgDeltaNext("snapshotCapture", async () => { - const redactSecrets = input.redactSecrets ?? true; const profile = await libraries.resolveProfile(input.pool, { - redactSecrets, + redactSecrets: true, skipBaseline: true, }); - const result = await profile.extract(input.pool, { - redactSecrets, - ...(input.statementTimeoutMs !== undefined - ? { statementTimeoutMs: input.statementTimeoutMs } - : {}), - }); + const result = await profile.extract(input.pool, { redactSecrets: true }); return { generation: "v2", snapshot: libraries.serializeSnapshot(result.factBase, { pgVersion: result.pgVersion, - redactSecrets, + redactSecrets: true, profile: profile.id, }), pgVersion: result.pgVersion, @@ -743,13 +685,13 @@ const legacyPgDeltaNextRealLibraries = { targetPool: Pool, shadowPool: Pool, files: readonly LegacyPgDeltaNextSqlFile[], - input: LegacyPgDeltaNextDeclarativePlanInput, + input: LegacyPgDeltaNextLibraryPlanOptions, ) => { const result = await planSchemaFiles( targetPool, shadowPool, files.map((file) => ({ name: file.name, sql: file.sql })), - legacyPgDeltaNextPlanOptions(input), + input, ); const [loadDiagnostics, targetDiagnostics] = await Promise.all([ legacyFilterPgDeltaNextPlatformDiagnostics(shadowPool, result.loadDiagnostics), diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts index 190f864487..c2ca976152 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts @@ -9,6 +9,7 @@ import { import type { LegacyMigrationTransactionMode } from "../../../shared/legacy-migration-file.ts"; import type { LegacyPgDeltaErrorDiagnostic, + LegacyPgDeltaExportManifest, LegacyPgDeltaHazardKind, LegacyPgDeltaHazardReport, LegacyPgDeltaRemovalSummary, @@ -67,8 +68,6 @@ export interface LegacyPgDeltaNextDiffInput { readonly desiredPool: Pool; readonly allowDrops: boolean; readonly debug: boolean; - readonly redactSecrets?: boolean; - readonly restrictToApplier?: boolean; readonly schema?: readonly string[]; readonly formatOptions?: string; } @@ -82,42 +81,13 @@ interface LegacyPgDeltaNextDiffResult { readonly debug?: LegacyPgDeltaNextDebugArtifacts; } -type LegacyPgDeltaNextManagementScope = "database" | "cluster"; -type LegacyPgDeltaNextExportLayout = "by-object" | "ordered" | "grouped"; - -interface LegacyPgDeltaNextExportGroupingPattern { - readonly pattern: string; - readonly name: string; -} - -interface LegacyPgDeltaNextExportGrouping { - readonly mode?: "single-file" | "subdirectory"; - readonly groupPatterns?: readonly LegacyPgDeltaNextExportGroupingPattern[]; - readonly flatSchemas?: readonly string[]; - readonly autoGroupPartitions?: boolean; -} - export interface LegacyPgDeltaNextDeclarativeExportInput { readonly pool: Pool; - readonly scope?: LegacyPgDeltaNextManagementScope; - readonly redactSecrets?: boolean; - readonly restrictToApplier?: boolean; - readonly layout?: LegacyPgDeltaNextExportLayout; - readonly grouping?: LegacyPgDeltaNextExportGrouping; - readonly defaultOwner?: string | null; - readonly onWarning?: (message: string) => void; readonly schema?: readonly string[]; readonly formatOptions?: string; } -export interface LegacyPgDeltaNextExportManifest { - readonly redactSecrets: boolean; - readonly scope: LegacyPgDeltaNextManagementScope; - readonly profile?: string; - readonly baselineDigest?: string; - readonly defaultOwner?: string | null; - readonly files?: readonly string[]; -} +export type LegacyPgDeltaNextExportManifest = LegacyPgDeltaExportManifest; interface LegacyPgDeltaNextDeclarativeExportResult { readonly files: readonly LegacyPgDeltaNextSqlFile[]; @@ -125,35 +95,14 @@ interface LegacyPgDeltaNextDeclarativeExportResult { readonly diagnostics: readonly LegacyPgDeltaNextDiagnostic[]; } -export interface LegacyPgDeltaNextDeclarativeManifestInput { - readonly redactSecrets?: boolean; - readonly profile?: string; - readonly scope?: LegacyPgDeltaNextManagementScope; - readonly baselineDigest?: string; - readonly defaultOwner?: string | null; - readonly files?: readonly string[]; -} - export interface LegacyPgDeltaNextDeclarativePlanInput { readonly targetPool: Pool; readonly shadowPool: Pool; readonly files: readonly LegacyPgDeltaNextSqlFile[]; readonly allowDrops: boolean; readonly debug: boolean; - readonly scope?: LegacyPgDeltaNextManagementScope; - readonly manifest?: LegacyPgDeltaNextDeclarativeManifestInput; - readonly redactSecrets?: boolean; - readonly skipClusterDdl?: boolean; - readonly isolatedShadow?: boolean; - readonly seedAssumedSchemas?: boolean; - readonly restrictToApplier?: boolean; - readonly strictFunctionBodies?: boolean; - /** Reject data-changing SQL observed while loading declarative schema files. */ - readonly strictDataStatements?: boolean; + readonly manifest?: LegacyPgDeltaNextExportManifest; readonly formatOptions?: string; - /** Defaults to true, preserving pg-topo statement-level reorder support. */ - readonly reorder?: boolean; - readonly onWarning?: (message: string) => void; readonly schema?: readonly string[]; } @@ -175,8 +124,6 @@ interface LegacyPgDeltaNextDeclarativePlanResult { export interface LegacyPgDeltaNextSnapshotCaptureInput { readonly pool: Pool; - readonly redactSecrets?: boolean; - readonly statementTimeoutMs?: number; } interface LegacyPgDeltaNextSnapshotCaptureResult { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index ca7b5f2e74..1c01317710 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -1,11 +1,5 @@ import { it } from "@effect/vitest"; -import { - buildFactBase, - encodeId, - type DependencyEdge, - type Fact, - type StableId, -} from "@supabase/pg-delta/core"; +import { buildFactBase, type Fact, type StableId } from "@supabase/pg-delta/core"; import { renderPlanFiles, ShadowLoadError } from "@supabase/pg-delta/frontends"; import { plan, type Action } from "@supabase/pg-delta/plan"; import { Effect } from "effect"; @@ -13,7 +7,6 @@ import { Pool } from "pg"; import { describe, expect } from "vitest"; import { - legacyPgDeltaNextAdapterLayer, legacyPgDeltaNextAdapterLayerFromLibraries, legacyFilterPgDeltaNextPlatformParameterAclDiagnostics, legacyPgDeltaNextProfile, @@ -75,8 +68,6 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { exportInputs: [] as object[], declarativeInputs: [] as object[], snapshotMetadata: [] as object[], - serializedPlans: [] as FakePlan[], - renderChanges: true, }; const extract = async ( @@ -117,7 +108,6 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { }, renderPlanFiles: (_generatedPlan, options) => { state.renderOptions.push(options); - if (!state.renderChanges) return { changes: false, files: [] }; return { changes: true, files: [ @@ -165,7 +155,6 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { return JSON.stringify({ factBase: factBase.id, metadata }); }, serializePlan: (generatedPlan) => { - state.serializedPlans.push(generatedPlan); return JSON.stringify(generatedPlan); }, summarizeRemovals: () => ({ @@ -193,6 +182,36 @@ function setupLibraries(sourcePool: Pool, desiredPool: Pool) { }; } +const unusedLibraries: LegacyPgDeltaNextLibraries< + string, + Record, + FakePlan, + string +> = { + resolveProfile: async () => { + throw new Error("unused"); + }, + plan: () => ({ source: "unused", desired: "unused" }), + renderPlanFiles: () => ({ changes: false, files: [] }), + buildSchemaExport: async () => ({ + files: [], + diagnostics: [], + manifest: { redactSecrets: true, scope: "database" }, + }), + planSchemaFiles: async () => ({ + plan: { source: "unused", desired: "unused" }, + loadDiagnostics: [], + targetDiagnostics: [], + driftDiagnostics: [], + skipped: [], + }), + serializeSnapshot: () => "unused", + serializePlan: () => "unused", + summarizeRemovals: () => ({ extensions: [], extensionIntents: [] }), + summarizeHazards: () => ({ actions: [], dataLoss: [], coverage: [], kinds: [] }), + encodeSubject: (subject) => subject, +}; + describe("LegacyPgDeltaNextAdapter", () => { it("summarizes only root extension and extension-intent removals", () => { expect( @@ -341,44 +360,10 @@ describe("LegacyPgDeltaNextAdapter", () => { ).toEqual(["log_min_messages"]); }); - it("composes the operation-scoped schema complement ahead of the Supabase policy", () => { - const profile = legacyPgDeltaNextProfile(["public", "tenant"]); - expect(profile.id).toBe("supabase"); - expect(profile.policy?.filter).toEqual([ - { - match: { - all: [ - { verb: ["add", "remove", "set", "link", "unlink"] }, - { - not: { - any: [ - { schema: ["public", "tenant"] }, - { - all: [{ kind: "schema" }, { name: ["public", "tenant"] }], - }, - { target: { schema: ["public", "tenant"] } }, - { - target: { - kind: "schema", - name: ["public", "tenant"], - }, - }, - ], - }, - }, - ], - }, - action: "exclude", - }, - ]); - expect(profile.policy?.extends).toHaveLength(1); - }); - - it("renders only selected-schema state while preserving its metadata and dependencies", () => { + it("renders selected-schema state without leaking other user or platform objects", () => { const schemaPublic = { kind: "schema", name: "public" } satisfies StableId; const schemaAuth = { kind: "schema", name: "auth" } satisfies StableId; const existingRole = { kind: "role", name: "app_owner" } satisfies StableId; - const existingExtension = { kind: "extension", name: "hstore" } satisfies StableId; const selectedTable = { kind: "table", schema: "public", @@ -396,28 +381,11 @@ describe("LegacyPgDeltaNextAdapter", () => { name: "hidden_platform_table", } satisfies StableId; const customRole = { kind: "role", name: "hidden_custom_role" } satisfies StableId; - const customExtension = { - kind: "extension", - name: "hidden_custom_extension", - } satisfies StableId; - const customPublication = { - kind: "publication", - name: "hidden_custom_publication", - } satisfies StableId; - const customFdw = { kind: "fdw", name: "hidden_custom_fdw" } satisfies StableId; const fact = (id: StableId, payload: Fact["payload"] = {}, parent?: StableId): Fact => parent === undefined ? { id, payload } : { id, parent, payload }; - const sourceFacts: Fact[] = [ - fact(schemaPublic), - fact(schemaAuth), - fact(existingRole, { login: false, config: [] }), - fact(existingExtension, { schema: "public", relocatable: true }), - ]; - const sourceEdges: DependencyEdge[] = [ - { from: existingExtension, to: schemaPublic, kind: "depends" }, - ]; + const sourceFacts: Fact[] = [fact(schemaPublic), fact(schemaAuth), fact(existingRole)]; const desiredFacts: Fact[] = [ ...sourceFacts, fact( @@ -430,51 +398,25 @@ describe("LegacyPgDeltaNextAdapter", () => { { text: "selected table metadata" }, selectedTable, ), - fact( - { kind: "comment", target: schemaPublic }, - { text: "selected schema metadata" }, - schemaPublic, - ), - fact( - { kind: "acl", target: selectedTable, grantee: "PUBLIC" }, - { privileges: ["SELECT"], grantable: [] }, - selectedTable, - ), fact(unselectedSchema), fact( unselectedTable, { persistence: "p", partitionBound: null, partitionKey: null, parentTable: null }, unselectedSchema, ), - fact( - { kind: "comment", target: unselectedTable }, - { text: "unselected metadata" }, - unselectedTable, - ), fact( platformTable, { persistence: "p", partitionBound: null, partitionKey: null, parentTable: null }, schemaAuth, ), fact(customRole, { login: true, config: [] }), - fact(customExtension, { schema: "public", relocatable: true }), - fact(customPublication, { - allTables: false, - publish: ["insert", "update"], - viaRoot: false, - }), - fact(customFdw, { handler: null, validator: null, options: [] }), - ]; - const desiredEdges: DependencyEdge[] = [ - ...sourceEdges, - { from: selectedTable, to: existingExtension, kind: "depends" }, - { from: selectedTable, to: existingRole, kind: "owner" }, ]; + const desiredEdges = [{ from: selectedTable, to: existingRole, kind: "owner" }] as const; const profile = legacyPgDeltaNextProfile(["public", "auth"]); const generated = plan( - buildFactBase(sourceFacts, sourceEdges), - buildFactBase(desiredFacts, desiredEdges), + buildFactBase(sourceFacts, []), + buildFactBase(desiredFacts, [...desiredEdges]), { policy: profile.policy }, ); const rendered = renderPlanFiles(generated, { allowDrops: true }); @@ -482,71 +424,17 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(sql).toContain('CREATE TABLE "public"."selected_items"'); expect(sql).toContain("selected table metadata"); - expect(sql).toContain("selected schema metadata"); - expect(sql).toContain('GRANT SELECT ON TABLE "public"."selected_items" TO PUBLIC'); expect(sql).toContain('OWNER TO "app_owner"'); for (const leakedName of [ "private_data", "hidden_items", "hidden_platform_table", "hidden_custom_role", - "hidden_custom_extension", - "hidden_custom_publication", - "hidden_custom_fdw", - "unselected metadata", ]) { expect(sql).not.toContain(leakedName); } - - expect(generated.deltas).toContainEqual({ - verb: "link", - edge: { from: selectedTable, to: existingExtension, kind: "depends" }, - }); - expect(generated.deltas).toContainEqual({ - verb: "link", - edge: { from: selectedTable, to: existingRole, kind: "owner" }, - }); - const filtered = generated.filteredDeltas.map((delta) => { - switch (delta.verb) { - case "add": - case "remove": - return encodeId(delta.fact.id); - case "set": - return encodeId(delta.id); - case "link": - case "unlink": - return encodeId(delta.edge.from); - } - }); - expect(filtered).toEqual( - expect.arrayContaining([ - encodeId(unselectedSchema), - encodeId(unselectedTable), - encodeId(customRole), - encodeId(customExtension), - encodeId(customPublication), - encodeId(customFdw), - ]), - ); - expect(filtered).not.toContain(encodeId(platformTable)); - expect( - generated.projectionAudit?.entries.some( - (entry) => - entry.delta.verb === "add" && encodeId(entry.delta.fact.id) === encodeId(platformTable), - ), - ).toBe(true); }); - it.effect("constructs the real adapter from supported public pg-delta subpaths", () => - Effect.gen(function* () { - const adapter = yield* LegacyPgDeltaNextAdapter; - expect(adapter.diff).toBeTypeOf("function"); - expect(adapter.exportDeclarativeSchema).toBeTypeOf("function"); - expect(adapter.planDeclarativeSchema).toBeTypeOf("function"); - expect(adapter.captureSnapshot).toBeTypeOf("function"); - }).pipe(Effect.provide(legacyPgDeltaNextAdapterLayer)), - ); - it.effect( "resolves one shared profile for a pool-to-pool diff and emits structured debug data", () => { @@ -561,8 +449,6 @@ describe("LegacyPgDeltaNextAdapter", () => { desiredPool, allowDrops: true, debug: true, - redactSecrets: false, - restrictToApplier: true, schema: ["public"], formatOptions: '{"keywordCase":"upper","indent":4}', }); @@ -570,58 +456,29 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(state.resolveCalls).toEqual([ { pool: sourcePool, - options: { redactSecrets: false, restrictToApplier: true }, + options: { redactSecrets: true }, schema: ["public"], }, ]); expect(state.extractCalls).toEqual([ - { pool: sourcePool, options: { redactSecrets: false } }, - { pool: desiredPool, options: { redactSecrets: false } }, + { pool: sourcePool, options: { redactSecrets: true } }, + { pool: desiredPool, options: { redactSecrets: true } }, ]); expect(state.planCalls).toEqual([ { source: { id: "source-facts" }, desired: { id: "desired-facts" }, - options: { redactSecrets: false, managedView: "shared-profile-options" }, + options: { redactSecrets: true, managedView: "shared-profile-options" }, }, ]); expect(state.renderOptions).toEqual([{ allowDrops: true }]); - expect(result.files).toEqual([ - { - sequence: 1, - suffix: "_1", - sql: "CREATE TABLE public.widgets (\n id integer,\n display_name text\n);\n", - transactionMode: "transactional", - actionCount: 2, - }, - { - sequence: 2, - suffix: "_2", - sql: "-- pg-delta: transaction=false\nSET check_function_bodies = off;\n\nGRANT SELECT ON TABLE public.widgets TO anon;\n\nRESET ALL;\n", - transactionMode: "none", - actionCount: 1, - }, + expect(result.files).toMatchObject([ + { sequence: 1, suffix: "_1", transactionMode: "transactional", actionCount: 2 }, + { sequence: 2, suffix: "_2", transactionMode: "none", actionCount: 1 }, ]); - expect(result.sql).toBe( - "CREATE TABLE public.widgets (\n id integer,\n display_name text\n);\n\n\n-- pg-delta: transaction=false\nSET check_function_bodies = off;\n\nGRANT SELECT ON TABLE public.widgets TO anon;\n\nRESET ALL;\n", - ); - expect(result.diagnostics).toEqual([ - { - origin: "source", - code: "source-warning", - severity: "warning", - subject: "subject:s", - message: "source-warning message", - context: { detail: "source-warning" }, - }, - { - origin: "desired", - code: "desired-warning", - severity: "warning", - subject: "subject:d", - message: "desired-warning message", - context: { detail: "desired-warning" }, - }, + expect(result.diagnostics.map(({ origin, subject }) => ({ origin, subject }))).toEqual([ + { origin: "source", subject: "subject:s" }, + { origin: "desired", subject: "subject:d" }, ]); expect(result.debug).toEqual({ sourceSnapshot: expect.stringContaining("source-facts"), @@ -629,65 +486,14 @@ describe("LegacyPgDeltaNextAdapter", () => { plan: JSON.stringify({ source: "source-facts", desired: "desired-facts" }), }); expect(state.snapshotMetadata).toEqual([ - { pgVersion: "15.9", redactSecrets: false, profile: "supabase" }, - { pgVersion: "17.6", redactSecrets: false, profile: "supabase" }, + { pgVersion: "15.9", redactSecrets: true, profile: "supabase" }, + { pgVersion: "17.6", redactSecrets: true, profile: "supabase" }, ]); - expect(sourcePool.ending).toBe(false); - expect(sourcePool.ended).toBe(false); - expect(desiredPool.ending).toBe(false); - expect(desiredPool.ended).toBe(false); yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); }).pipe(Effect.provide(layer)); }, ); - it.effect("preserves a no-change result without creating debug artifacts", () => { - const sourcePool = new Pool(); - const desiredPool = new Pool(); - const { layer, state } = setupLibraries(sourcePool, desiredPool); - state.renderChanges = false; - - return Effect.gen(function* () { - const adapter = yield* LegacyPgDeltaNextAdapter; - const result = yield* adapter.diff({ - sourcePool, - desiredPool, - allowDrops: false, - debug: false, - }); - expect(result.changes).toBe(false); - expect(result.sql).toBe(""); - expect(result.files).toEqual([]); - expect(result.debug).toBeUndefined(); - expect(state.snapshotMetadata).toEqual([]); - expect(state.renderOptions).toEqual([{ allowDrops: false }]); - yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); - }).pipe(Effect.provide(layer)); - }); - - it.effect("formats rendered migration files with the human-readable defaults", () => { - const sourcePool = new Pool(); - const desiredPool = new Pool(); - const { layer } = setupLibraries(sourcePool, desiredPool); - - return Effect.gen(function* () { - const adapter = yield* LegacyPgDeltaNextAdapter; - const result = yield* adapter.diff({ - sourcePool, - desiredPool, - allowDrops: false, - debug: false, - }); - expect(result.files[0]?.sql).toBe( - "create table public.widgets (\n id integer,\n display_name text\n);\n", - ); - expect(result.files[1]?.sql).toBe( - "-- pg-delta: transaction=false\nset check_function_bodies = off;\n\ngrant select on table public.widgets to anon;\n\nreset all;\n", - ); - yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); - }).pipe(Effect.provide(layer)); - }); - it.effect( "normalizes declarative export and planning results with reorder enabled by default", () => { @@ -699,8 +505,6 @@ describe("LegacyPgDeltaNextAdapter", () => { const adapter = yield* LegacyPgDeltaNextAdapter; const exported = yield* adapter.exportDeclarativeSchema({ pool: targetPool, - layout: "grouped", - restrictToApplier: true, formatOptions: '{"keywordCase":"lower","commaStyle":"leading","indent":4,"maxWidth":100,"alignColumns":true,"alignKeyValues":false,"preserveRoutineBodies":true,"preserveViewBodies":false,"preserveRuleBodies":true,"ignored":"value"}', }); @@ -721,7 +525,6 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(state.exportInputs).toHaveLength(1); expect(state.exportInputs[0]).toMatchObject({ layout: "grouped", - resolveOptions: { restrictToApplier: true }, format: { keywordCase: "lower", commaStyle: "leading", @@ -738,7 +541,6 @@ describe("LegacyPgDeltaNextAdapter", () => { yield* adapter.exportDeclarativeSchema({ pool: targetPool, - layout: "grouped", }); expect(state.exportInputs[1]).toMatchObject({ format: { keywordCase: "lower", maxWidth: 180 }, @@ -750,15 +552,13 @@ describe("LegacyPgDeltaNextAdapter", () => { files: exported.files, allowDrops: true, debug: true, - isolatedShadow: true, - seedAssumedSchemas: true, - strictDataStatements: true, formatOptions: "null", }); expect(state.declarativeInputs).toHaveLength(1); expect(state.declarativeInputs[0]).toMatchObject({ reorder: true, - seedAssumedSchemas: true, + isolatedShadow: true, + seedAssumedSchemas: false, strictDataStatements: true, }); expect(planned.diagnostics.map((diagnostic) => diagnostic.origin)).toEqual([ @@ -810,84 +610,6 @@ describe("LegacyPgDeltaNextAdapter", () => { }, ); - it.effect("captures a v2 snapshot with a single baseline-free profile resolution", () => { - const pool = new Pool(); - const unusedDesiredPool = new Pool(); - const { layer, state } = setupLibraries(pool, unusedDesiredPool); - - return Effect.gen(function* () { - const adapter = yield* LegacyPgDeltaNextAdapter; - const result = yield* adapter.captureSnapshot({ - pool, - statementTimeoutMs: 4_000, - }); - expect(result.generation).toBe("v2"); - expect(result.pgVersion).toBe("15.9"); - expect(result.snapshot).toContain("source-facts"); - expect(state.resolveCalls).toEqual([ - { - pool, - options: { redactSecrets: true, skipBaseline: true }, - }, - ]); - expect(state.extractCalls).toEqual([ - { - pool, - options: { redactSecrets: true, statementTimeoutMs: 4_000 }, - }, - ]); - yield* Effect.promise(() => Promise.all([pool.end(), unusedDesiredPool.end()])); - }).pipe(Effect.provide(layer)); - }); - - it.effect("maps library rejections to an actionable typed error", () => { - const sourcePool = new Pool(); - const desiredPool = new Pool(); - const cause = new Error("connection refused for desired database"); - const failingLayer = legacyPgDeltaNextAdapterLayerFromLibraries({ - resolveProfile: async () => { - throw cause; - }, - plan: () => ({ source: "unused", desired: "unused" }), - renderPlanFiles: () => ({ changes: false, files: [] }), - buildSchemaExport: async () => ({ - files: [], - diagnostics: [], - manifest: { redactSecrets: true, scope: "database" }, - }), - planSchemaFiles: async () => ({ - plan: { source: "unused", desired: "unused" }, - loadDiagnostics: [], - targetDiagnostics: [], - driftDiagnostics: [], - skipped: [], - }), - serializeSnapshot: () => "unused", - serializePlan: () => "unused", - summarizeRemovals: () => ({ extensions: [], extensionIntents: [] }), - summarizeHazards: () => ({ - actions: [], - dataLoss: [], - coverage: [], - kinds: [], - }), - encodeSubject: (subject: string) => subject, - }); - - return Effect.gen(function* () { - const adapter = yield* LegacyPgDeltaNextAdapter; - const error = yield* adapter - .diff({ sourcePool, desiredPool, allowDrops: false, debug: false }) - .pipe(Effect.flip); - expect(error).toBeInstanceOf(LegacyPgDeltaNextError); - expect(error.operation).toBe("diff"); - expect(error.message).toBe("Database diff failed: connection refused for desired database"); - expect(error.cause).toBe(cause); - expect(error.diagnostics).toBeUndefined(); - yield* Effect.promise(() => Promise.all([sourcePool.end(), desiredPool.end()])); - }).pipe(Effect.provide(failingLayer)); - }); - it.effect("preserves shadow-load diagnostics in the actionable error", () => { const targetPool = new Pool(); const shadowPool = new Pool(); @@ -905,29 +627,10 @@ describe("LegacyPgDeltaNextAdapter", () => { }, ]); const failingLayer = legacyPgDeltaNextAdapterLayerFromLibraries({ - resolveProfile: async () => { - throw new Error("unused"); - }, - plan: () => ({ source: "unused", desired: "unused" }), - renderPlanFiles: () => ({ changes: false, files: [] }), - buildSchemaExport: async () => ({ - files: [], - diagnostics: [], - manifest: { redactSecrets: true, scope: "database" }, - }), + ...unusedLibraries, planSchemaFiles: async () => { throw cause; }, - serializeSnapshot: () => "unused", - serializePlan: () => "unused", - summarizeRemovals: () => ({ extensions: [], extensionIntents: [] }), - summarizeHazards: () => ({ - actions: [], - dataLoss: [], - coverage: [], - kinds: [], - }), - encodeSubject: (subject: string) => subject, }); return Effect.gen(function* () { @@ -939,8 +642,6 @@ describe("LegacyPgDeltaNextAdapter", () => { files: [], allowDrops: false, debug: false, - isolatedShadow: true, - seedAssumedSchemas: false, }) .pipe(Effect.flip); expect(error).toBeInstanceOf(LegacyPgDeltaNextError); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts index 567d2ad2f6..0c58965625 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts @@ -13,48 +13,22 @@ import { import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.cache.ts"; describe("pg-delta next artifact generation", () => { - it.effect("isolates v2 artifacts from legacy catalog paths", () => - Effect.gen(function* () { - const path = yield* Path.Path; - expect(legacyPgDeltaTempPath(path, "/project")).toBe( - join("/project", "supabase", ".temp", "pgdelta"), - ); - expect(legacyPgDeltaNextTempPath(path, "/project")).toBe( - join("/project", "supabase", ".temp", "pgdelta", "v2"), - ); - }).pipe(Effect.provide(BunServices.layer)), - ); - - it("uses millisecond-resolution, operation-qualified debug ids", () => { - expect(legacyFormatPgDeltaNextDebugId(Date.UTC(2024, 0, 2, 3, 4, 5, 678), "diff")).toBe( - "20240102-030405-678-diff", - ); - }); - it.effect("writes structured non-cache artifacts and metadata under v2", () => { const root = mkdtempSync(join(tmpdir(), "pgdelta-next-artifacts-")); return Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; - const debugDir = yield* legacySavePgDeltaNextDebugArtifacts( - fs, - path, - root, - "20240102-030405-678-diff", - "diff", - { - sourceSnapshot: '{"source":true}\n', - desiredSnapshot: '{"desired":true}\n', - plan: '{"plan":true}\n', - diagnostics: [ - { origin: "source", code: "PG001", severity: "warning", message: "warning" }, - ], - }, - ); + const debugId = legacyFormatPgDeltaNextDebugId(Date.UTC(2024, 0, 2, 3, 4, 5, 678), "diff"); + const debugDir = yield* legacySavePgDeltaNextDebugArtifacts(fs, path, root, debugId, "diff", { + sourceSnapshot: '{"source":true}\n', + desiredSnapshot: '{"desired":true}\n', + plan: '{"plan":true}\n', + diagnostics: [{ origin: "source", code: "PG001", severity: "warning", message: "warning" }], + }); - expect(debugDir).toBe( - join(root, "supabase", ".temp", "pgdelta", "v2", "debug", "20240102-030405-678-diff"), - ); + expect(debugId).toBe("20240102-030405-678-diff"); + expect(legacyPgDeltaNextTempPath(path, root)).not.toBe(legacyPgDeltaTempPath(path, root)); + expect(debugDir).toBe(join(legacyPgDeltaNextTempPath(path, root), "debug", debugId)); expect(JSON.parse(readFileSync(join(debugDir, "metadata.json"), "utf8"))).toEqual({ version: 1, generation: "v2", diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts index ec2ca53f6e..a0a11f3be0 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-diagnostics.unit.test.ts @@ -49,18 +49,6 @@ describe("pg-delta next diagnostic coverage policy", () => { subject: "role:postgres", message: "edge references a fact not in the base", }, - { - origin: "declarativeLoad", - code: "invalid_routine_body", - severity: "warning", - message: "routine body failed validation", - }, - { - origin: "snapshot", - code: "unresolved_security_label", - severity: "warning", - message: "provider was not resolved", - }, ], false, ); @@ -72,13 +60,7 @@ describe("pg-delta next diagnostic coverage policy", () => { "pg-delta does not manage these PostgreSQL object kinds: statistics object, text search configuration. Changes to these objects are omitted from the generated database diff.", }); expect(out.messages.some(({ message }) => message.includes("dangling_edge"))).toBe(false); - expect(out.messages.some(({ message }) => message.includes("invalid_routine_body"))).toBe( - false, - ); - expect( - out.messages.some(({ message }) => message.includes("unresolved_security_label")), - ).toBe(false); - expect(debugMessages).toHaveLength(5); + expect(debugMessages).toHaveLength(3); expect(debugMessages).toContain( "pg-delta next diagnostic: origin=source code=dangling_edge subject=role:postgres message=edge references a fact not in the base", ); @@ -118,42 +100,6 @@ describe("pg-delta next diagnostic coverage policy", () => { }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); - it("uses the upstream coverage policy to block unmodeled declarative drift", () => { - const report = legacyPgDeltaNextDiagnosticReport( - [ - { - origin: "declarativeDrift", - code: "unmodeled_drift", - severity: "warning", - message: "desired text search configuration is absent from the target", - context: { kind: "text search configuration" }, - }, - ], - true, - ); - - expect(report.coverage).toHaveLength(1); - expect(report.blocking).toEqual(report.coverage); - }); - - it("can suppress a repeated feedback invitation without suppressing warnings", () => { - const out = mockOutput(); - const debugMessages: string[] = []; - return Effect.gen(function* () { - yield* legacyReportPgDeltaNextDiagnostics( - "declarativePlan", - [unmodeled("text search configuration")], - false, - false, - ); - - expect(out.messages.some(({ message }) => message.includes("supabase issue feature"))).toBe( - false, - ); - expect(out.messages.some(({ type }) => type === "warn")).toBe(true); - }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); - }); - const skippedStatement = (file: string, statement: string): LegacyPgDeltaNextDiagnostic => ({ origin: "declarativeLoad", code: LEGACY_PG_DELTA_NEXT_SKIPPED_STATEMENT_CODE, @@ -181,49 +127,11 @@ describe("pg-delta next diagnostic coverage policy", () => { message: "pg-delta could not load 2 declarative schema statements in roles.sql. Changes to these objects are omitted from the declarative migration plan.", }); - // Statement text stays out of the default-visibility summary; the per-diagnostic - // detail carries it and is routed to debug unless strict/verbose. expect(out.messages.some(({ message }) => message.includes("s3cret"))).toBe(false); expect(debugMessages.some((message) => message.includes("s3cret"))).toBe(true); }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); }); - it("fails on skipped declarative statements under strict coverage", () => { - const out = mockOutput(); - const debugMessages: string[] = []; - return Effect.gen(function* () { - const exit = yield* legacyReportPgDeltaNextDiagnostics( - "declarativePlan", - [skippedStatement("roles.sql", "create role app")], - true, - ).pipe(Effect.exit); - - expect(Exit.isFailure(exit)).toBe(true); - expect(out.messages).toContainEqual({ - type: "warn", - message: - "pg-delta could not load 1 declarative schema statement in roles.sql. Strict coverage is enabled, so the operation will stop.", - }); - // Strict mode renders the detail (with the statement) so the user can fix it. - expect(out.messages.some(({ message }) => message.includes("create role app"))).toBe(true); - // No object kinds involved, so no unmodeled-kind summary and no feedback invite. - expect(out.messages.some(({ message }) => message.includes("does not manage"))).toBe(false); - expect(out.messages.some(({ message }) => message.includes("supabase issue feature"))).toBe( - false, - ); - }).pipe(Effect.provide(out.layer), Effect.provide(debugLayer(debugMessages))); - }); - - it("classifies a skipped statement as a coverage gap only under strict coverage", () => { - const diagnostics = [skippedStatement("roles.sql", "create role app")]; - const lenient = legacyPgDeltaNextDiagnosticReport(diagnostics, false); - expect(lenient.coverage).toHaveLength(1); - expect(lenient.blocking).toEqual([]); - - const strict = legacyPgDeltaNextDiagnosticReport(diagnostics, true); - expect(strict.blocking).toEqual(strict.coverage); - }); - it("always renders and fails error diagnostics", () => { const out = mockOutput(); const debugMessages: string[] = []; @@ -297,8 +205,6 @@ describe("pg-delta next diagnostic coverage policy", () => { unmodeled("a future kind"), unmodeled("a future kind"), unmodeled("line\nbreak"), - unmodeled(undefined), - unmodeled(" "), { origin: "snapshot", code: "unresolved_security_label", @@ -316,8 +222,8 @@ describe("pg-delta next diagnostic coverage policy", () => { true, ); - expect(report.coverage).toHaveLength(8); - expect(report.blocking).toHaveLength(8); + expect(report.coverage).toHaveLength(6); + expect(report.blocking).toEqual(report.coverage); expect(report.unmodeledKinds).toEqual(["a future kind", "line break", "z future kind"]); }); @@ -338,9 +244,6 @@ describe("pg-delta next diagnostic coverage policy", () => { expect(invitation).not.toContain("private diagnostic message"); expect(invitation).not.toContain("subject"); expect(invitation).not.toContain("public."); - }); - - it("shell-quotes future kind names without making feedback kind-specific", () => { expect(legacyPgDeltaNextFeedbackInvitation(["user's future kind"])).toContain( `user'"'"'s future kind`, ); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index c49c865188..8319962425 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -225,9 +225,7 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( timeoutSeconds: input.base.healthTimeoutSeconds, }); const setup = setupRunInput(input, handle); - yield* legacySetupShadowDatabase(input.spawner, setup, { - activateUserExtensions: false, - }); + yield* legacySetupShadowDatabase(input.spawner, setup, { webhooks: "disabled" }); yield* Effect.scoped( Effect.gen(function* () { const session = yield* legacyConnectShadowDatabase(setup.connConfig); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts index fa35b261b0..1c26afc67c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts @@ -1,26 +1,16 @@ -import { - mkdirSync, - mkdtempSync, - readFileSync, - rmSync, - statSync, - utimesSync, - writeFileSync, -} from "node:fs"; -import { existsSync } from "node:fs"; -import { tmpdir } from "node:os"; +import { existsSync, mkdirSync, readFileSync, statSync, utimesSync, writeFileSync } from "node:fs"; import { join } from "node:path"; + import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; -import { Cause, Effect, Exit, FileSystem, Path } from "effect"; +import { Effect, FileSystem, Path } from "effect"; +import { useLegacyTempWorkdir } from "../../../../../tests/helpers/legacy-mocks.ts"; import { mockOutput } from "../../../../../tests/helpers/mocks.ts"; -import { legacyBold } from "../../../shared/legacy-colors.ts"; import type { LegacyDeclarativeOutput } from "../../../shared/legacy-pgdelta.ts"; import { LegacyDeclarativeWriteError } from "./legacy-pgdelta.errors.ts"; import type { LegacyPgDeltaDeclarativeExportResult } from "./legacy-pgdelta-engine.service.ts"; import { - legacyDeclarativeSchemaWrittenLine, legacyWarnPreservedUnmanagedDeclarativeFiles, legacyWriteDeclarativeSchemas, } from "./legacy-pgdelta.write.ts"; @@ -35,299 +25,156 @@ const write = ( return yield* legacyWriteDeclarativeSchemas(fs, path, declarativeDir, output); }).pipe(Effect.provide(BunServices.layer)); +const nextOutput = (files: LegacyPgDeltaDeclarativeExportResult["files"]) => ({ + files, + manifest: { redactSecrets: true, scope: "database" as const, profile: "supabase" }, +}); + describe("legacyWriteDeclarativeSchemas", () => { - it.effect("wipes the dir and writes each file at its relative path", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - mkdirSync(declDir, { recursive: true }); - writeFileSync(join(declDir, "stale.sql"), "-- should be removed"); - const output: LegacyDeclarativeOutput = { + const tmp = useLegacyTempWorkdir("legacy-decl-write-"); + const declarativeDir = () => join(tmp.current, "supabase", "database"); + + it.effect("keeps the legacy wipe-and-rewrite behavior", () => { + const dir = declarativeDir(); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, "stale.sql"), "-- should be removed"); + return write(dir, { version: 1, mode: "declarative", files: [ { path: "public.sql", order: 0, statements: 1, sql: "create table a();" }, { path: "auth/roles.sql", order: 1, statements: 1, sql: "create role app;" }, ], - }; - return write(declDir, output).pipe( - Effect.tap(() => - Effect.sync(() => { - expect(existsSync(join(declDir, "stale.sql"))).toBe(false); - expect(readFileSync(join(declDir, "public.sql"), "utf8")).toBe("create table a();"); - expect(readFileSync(join(declDir, "auth", "roles.sql"), "utf8")).toBe("create role app;"); - expect(existsSync(join(declDir, ".pgdelta-export.json"))).toBe(false); - rmSync(dir, { recursive: true, force: true }); - }), - ), - ); - }); - - it.effect("writes the next export manifest with the generated file list", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - return write(declDir, { - files: [ - { name: "schemas/z.sql", sql: "select 'z';" }, - { name: "schemas/a.sql", sql: "select 'a';" }, - ], - manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, - }).pipe( - Effect.tap(() => - Effect.sync(() => { - expect(JSON.parse(readFileSync(join(declDir, ".pgdelta-export.json"), "utf8"))).toEqual({ - formatVersion: 1, - redactSecrets: true, - scope: "database", - profile: "supabase", - files: ["schemas/a.sql", "schemas/z.sql"], - }); - rmSync(dir, { recursive: true, force: true }); - }), - ), - ); - }); - - it.effect("preserves custom and unmanaged files while pruning stale owned files", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - mkdirSync(join(declDir, "_custom"), { recursive: true }); - writeFileSync(join(declDir, "_custom", "casts.sql"), "create cast (int as text);"); - writeFileSync(join(declDir, "unmanaged.sql"), "select 'keep me';"); - writeFileSync(join(declDir, "stale.sql"), "select 'remove me';"); - writeFileSync( - join(declDir, ".pgdelta-export.json"), - `${JSON.stringify({ - formatVersion: 1, - redactSecrets: true, - scope: "database", - files: ["stale.sql"], - })}\n`, - ); - - return write(declDir, { - files: [{ name: "schemas/public.sql", sql: "create table public.example(id int);" }], - manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, - }).pipe( - Effect.tap(() => - Effect.sync(() => { - expect(existsSync(join(declDir, "stale.sql"))).toBe(false); - expect(readFileSync(join(declDir, "unmanaged.sql"), "utf8")).toBe("select 'keep me';"); - expect(readFileSync(join(declDir, "_custom", "casts.sql"), "utf8")).toBe( - "create cast (int as text);", - ); - expect( - JSON.parse(readFileSync(join(declDir, ".pgdelta-export.json"), "utf8")).files, - ).toEqual(["schemas/public.sql"]); - rmSync(dir, { recursive: true, force: true }); - }), - ), - ); - }); - - it.effect("reports pre-existing files preserved when no manifest claims ownership", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - // A directory produced by the OLD legacy full-wipe exporter: `.sql` files, no - // `.pgdelta-export.json`. Nothing can be classified as stale, so the next writer - // merges into it — the caller has to tell the user that happened. - mkdirSync(join(declDir, "_custom"), { recursive: true }); - writeFileSync(join(declDir, "_custom", "casts.sql"), "create cast (int as text);"); - writeFileSync(join(declDir, "legacy-b.sql"), "select 'b';"); - writeFileSync(join(declDir, "legacy-a.sql"), "select 'a';"); - writeFileSync(join(declDir, "schemas-public.sql"), "-- replaced by the export below"); - - return write(declDir, { - files: [{ name: "schemas-public.sql", sql: "create table public.example(id int);" }], - manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, }).pipe( Effect.tap((written) => Effect.sync(() => { - // Sorted, excludes the file the export itself replaced and the reserved - // `_custom/` tree (never read as managed output). - expect(written.preservedUnmanagedFiles).toEqual(["legacy-a.sql", "legacy-b.sql"]); - expect(readFileSync(join(declDir, "legacy-a.sql"), "utf8")).toBe("select 'a';"); - expect(readFileSync(join(declDir, "schemas-public.sql"), "utf8")).toBe( - "create table public.example(id int);", - ); - rmSync(dir, { recursive: true, force: true }); + expect(written.preservedUnmanagedFiles).toEqual([]); + expect(existsSync(join(dir, "stale.sql"))).toBe(false); + expect(readFileSync(join(dir, "public.sql"), "utf8")).toBe("create table a();"); + expect(readFileSync(join(dir, "auth", "roles.sql"), "utf8")).toBe("create role app;"); + expect(existsSync(join(dir, ".pgdelta-export.json"))).toBe(false); }), ), ); }); - it.effect("reports nothing preserved once a manifest owns the directory", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - mkdirSync(declDir, { recursive: true }); - writeFileSync(join(declDir, "unmanaged.sql"), "select 'keep me';"); - writeFileSync( - join(declDir, ".pgdelta-export.json"), - `${JSON.stringify({ + it.effect("tracks next-engine ownership while preserving custom and unmanaged files", () => { + const dir = declarativeDir(); + return Effect.gen(function* () { + yield* write( + dir, + nextOutput([ + { name: "schemas/z.sql", sql: "select 'z';" }, + { name: "stale.sql", sql: "select 'remove later';" }, + ]), + ); + mkdirSync(join(dir, "_custom"), { recursive: true }); + writeFileSync(join(dir, "_custom", "casts.sql"), "create cast (int as text);"); + writeFileSync(join(dir, "unmanaged.sql"), "select 'keep me';"); + + const written = yield* write( + dir, + nextOutput([ + { name: "schemas/z.sql", sql: "select 'z';" }, + { name: "schemas/a.sql", sql: "select 'a';" }, + ]), + ); + + expect(written.preservedUnmanagedFiles).toEqual([]); + expect(existsSync(join(dir, "stale.sql"))).toBe(false); + expect(readFileSync(join(dir, "unmanaged.sql"), "utf8")).toBe("select 'keep me';"); + expect(readFileSync(join(dir, "_custom", "casts.sql"), "utf8")).toBe( + "create cast (int as text);", + ); + expect(JSON.parse(readFileSync(join(dir, ".pgdelta-export.json"), "utf8"))).toEqual({ formatVersion: 1, redactSecrets: true, scope: "database", - files: ["stale.sql"], - })}\n`, - ); - - return write(declDir, { - files: [{ name: "schemas/public.sql", sql: "create table public.example(id int);" }], - manifest: { redactSecrets: true, scope: "database" }, - }).pipe( - Effect.tap((written) => - Effect.sync(() => { - expect(written.preservedUnmanagedFiles).toEqual([]); - rmSync(dir, { recursive: true, force: true }); - }), - ), - ); + profile: "supabase", + files: ["schemas/a.sql", "schemas/z.sql"], + }); + }); }); - it.effect("reports nothing preserved for the legacy full-wipe writer", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - mkdirSync(declDir, { recursive: true }); - writeFileSync(join(declDir, "stale.sql"), "-- wiped"); - return write(declDir, { - version: 1, - mode: "declarative", - files: [{ path: "public.sql", order: 0, statements: 0, sql: "select 1;" }], - }).pipe( + it.effect("reports manifestless files that the next writer preserves", () => { + const dir = declarativeDir(); + mkdirSync(join(dir, "_custom"), { recursive: true }); + writeFileSync(join(dir, "_custom", "casts.sql"), "select 'custom';"); + writeFileSync(join(dir, "legacy-b.sql"), "select 'b';"); + writeFileSync(join(dir, "legacy-a.sql"), "select 'a';"); + writeFileSync(join(dir, "replaced.sql"), "-- old"); + + return write( + dir, + nextOutput([{ name: "replaced.sql", sql: "create table public.example(id int);" }]), + ).pipe( Effect.tap((written) => Effect.sync(() => { - expect(written.preservedUnmanagedFiles).toEqual([]); - rmSync(dir, { recursive: true, force: true }); + expect(written.preservedUnmanagedFiles).toEqual(["legacy-a.sql", "legacy-b.sql"]); + expect(readFileSync(join(dir, "replaced.sql"), "utf8")).toContain("create table"); }), ), ); }); it.effect("does not rewrite unchanged next-engine files or manifests", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - const schemaPath = join(declDir, "schemas", "public.sql"); - const manifestPath = join(declDir, ".pgdelta-export.json"); - const output: LegacyPgDeltaDeclarativeExportResult = { - files: [{ name: "schemas/public.sql", sql: "create table public.example(id int);" }], - manifest: { redactSecrets: true, scope: "database", profile: "supabase" }, - }; - - return write(declDir, output).pipe( - Effect.flatMap(() => + const dir = declarativeDir(); + const schemaPath = join(dir, "schemas", "public.sql"); + const manifestPath = join(dir, ".pgdelta-export.json"); + const output = nextOutput([ + { name: "schemas/public.sql", sql: "create table public.example(id int);" }, + ]); + + return write(dir, output).pipe( + Effect.tap(() => Effect.sync(() => { const old = new Date("2020-01-01T00:00:00.000Z"); utimesSync(schemaPath, old, old); utimesSync(manifestPath, old, old); }), ), - Effect.flatMap(() => write(declDir, output)), + Effect.andThen(write(dir, output)), Effect.tap(() => Effect.sync(() => { expect(statSync(schemaPath).mtime.toISOString()).toBe("2020-01-01T00:00:00.000Z"); expect(statSync(manifestPath).mtime.toISOString()).toBe("2020-01-01T00:00:00.000Z"); - rmSync(dir, { recursive: true, force: true }); }), ), ); }); - it.effect("rejects next-engine output targeting the reserved custom directory", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - return write(declDir, { - files: [{ name: "_custom/generated.sql", sql: "select 1;" }], - manifest: { redactSecrets: true, scope: "database" }, - }).pipe( - Effect.exit, - Effect.tap((exit) => - Effect.sync(() => { - expect(Exit.isFailure(exit)).toBe(true); - if (Exit.isFailure(exit)) { - const error = exit.cause.reasons.find(Cause.isFailReason)?.error; - expect(error).toBeInstanceOf(LegacyDeclarativeWriteError); - expect((error as LegacyDeclarativeWriteError).message).toBe( - "refusing to write into reserved declarative schema path: _custom/generated.sql", - ); - } - rmSync(dir, { recursive: true, force: true }); - }), - ), - ); - }); - - it.effect("creates the declarative dir when absent", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - return write(declDir, { - version: 1, - mode: "declarative", - files: [{ path: "public.sql", order: 0, statements: 0, sql: "select 1;" }], - }).pipe( - Effect.tap(() => - Effect.sync(() => { - expect(readFileSync(join(declDir, "public.sql"), "utf8")).toBe("select 1;"); - rmSync(dir, { recursive: true, force: true }); - }), - ), - ); - }); - - it.effect("rejects an unsafe (path-escaping) export path", () => { - const dir = mkdtempSync(join(tmpdir(), "legacy-decl-write-")); - const declDir = join(dir, "supabase", "database"); - return write(declDir, { - version: 1, - mode: "declarative", - files: [{ path: "../escape.sql", order: 0, statements: 0, sql: "x" }], - }).pipe( - Effect.exit, - Effect.tap((exit) => - Effect.sync(() => { - expect(Exit.isFailure(exit)).toBe(true); - if (Exit.isFailure(exit)) { - const error = exit.cause.reasons.find(Cause.isFailReason)?.error; - expect(error).toBeInstanceOf(LegacyDeclarativeWriteError); - expect((error as LegacyDeclarativeWriteError).message).toBe( - "unsafe declarative export path: ../escape.sql", - ); - } - rmSync(dir, { recursive: true, force: true }); - }), - ), - ); - }); -}); - -describe("legacyDeclarativeSchemaWrittenLine", () => { - it("formats the shared written-to line for the given dir", () => { - expect(legacyDeclarativeSchemaWrittenLine("supabase/database")).toBe( - `Declarative schema written to ${legacyBold("supabase/database")}\n`, - ); - }); + it.effect("rejects reserved and escaping export paths", () => + Effect.gen(function* () { + const reserved = yield* write( + join(tmp.current, "reserved"), + nextOutput([{ name: "_custom/generated.sql", sql: "select 1;" }]), + ).pipe(Effect.flip); + expect(reserved).toBeInstanceOf(LegacyDeclarativeWriteError); + expect(reserved.message).toContain("reserved declarative schema path"); + + const escaping = yield* write(join(tmp.current, "escaping"), { + version: 1, + mode: "declarative", + files: [{ path: "../escape.sql", order: 0, statements: 0, sql: "x" }], + }).pipe(Effect.flip); + expect(escaping).toBeInstanceOf(LegacyDeclarativeWriteError); + expect(escaping.message).toContain("unsafe declarative export path"); + }), + ); }); describe("legacyWarnPreservedUnmanagedDeclarativeFiles", () => { - it.effect("names the preserved files and advises a full rewrite", () => { + it.effect("names preserved files and advises a clean regeneration", () => { const out = mockOutput(); return Effect.gen(function* () { yield* legacyWarnPreservedUnmanagedDeclarativeFiles("supabase/database", { preservedUnmanagedFiles: ["legacy-a.sql", "legacy-b.sql"], }); - const stderr = out.stderrText; - expect(stderr).toContain("2 existing declarative schema file(s) in supabase/database"); - expect(stderr).toContain("legacy-a.sql, legacy-b.sql"); - expect(stderr).toContain("were preserved"); - expect(stderr).toContain("remove supabase/database and re-run"); - }).pipe(Effect.provide(out.layer)); - }); - - it.effect("stays silent when the write preserved nothing", () => { - const out = mockOutput(); - return Effect.gen(function* () { - yield* legacyWarnPreservedUnmanagedDeclarativeFiles("supabase/database", { - preservedUnmanagedFiles: [], - }); - expect(out.stderrText).toBe(""); + expect(out.stderrText).toContain( + "2 existing declarative schema file(s) in supabase/database", + ); + expect(out.stderrText).toContain("legacy-a.sql, legacy-b.sql"); + expect(out.stderrText).toContain("remove supabase/database and re-run"); }).pipe(Effect.provide(out.layer)); }); }); diff --git a/apps/cli/src/legacy/commands/db/start/start.integration.test.ts b/apps/cli/src/legacy/commands/db/start/start.integration.test.ts index e0f18b5030..94ef016d04 100644 --- a/apps/cli/src/legacy/commands/db/start/start.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/start/start.integration.test.ts @@ -236,7 +236,7 @@ const alwaysReadyHttpClientLayer = Layer.succeed( ); /** Mirrors `start.integration.test.ts`'s own `fakeDbSession` — PG15+ (this suite's default) never calls `exec`/`query` (its schema init is three one-shot `LegacyDockerRun` jobs instead). */ -function fakeDbSession(appliedMigrationStatements?: ReadonlyArray) { +function fakeDbSession() { const calls: Array<{ kind: "exec" | "query"; sql: string }> = []; const session: LegacyDbSession = { exec: (sql) => @@ -246,18 +246,7 @@ function fakeDbSession(appliedMigrationStatements?: ReadonlyArray) { query: (sql) => Effect.sync(() => { calls.push({ kind: "query", sql }); - // The Database Webhooks convergence reads applied-migration statements to - // decide whether pg_net is migration-owned (and must not be dropped). - return appliedMigrationStatements !== undefined && - sql.includes("supabase_migrations.schema_migrations") - ? [ - { - version: "20240101000000", - name: "migration", - statements: [...appliedMigrationStatements], - }, - ] - : []; + return []; }), extensionExists: () => Effect.succeed(false), copyToCsv: () => Effect.succeed(new Uint8Array()), @@ -297,12 +286,6 @@ interface SetupOpts { readonly connectFailures?: number; /** Whether the mocked connect failures are dial-level (`retryable`). Defaults to `true`. */ readonly connectFailuresRetryable?: boolean; - /** - * Statements of one recorded row in `supabase_migrations.schema_migrations`, read by - * the existing-volume Database Webhooks convergence to decide whether pg_net is - * migration-owned. Defaults to an empty history. - */ - readonly appliedMigrationStatements?: ReadonlyArray; } function setup(opts: SetupOpts = {}) { @@ -324,7 +307,7 @@ function setup(opts: SetupOpts = {}) { ? runningCheckFailsRoute(baseRoute) : baseRoute; const child = mockContainerCliSpawner(route); - const dbSession = fakeDbSession(opts.appliedMigrationStatements); + const dbSession = fakeDbSession(); const edgeRunCalls: Array = []; const edgeRuntime = Layer.succeed(LegacyEdgeRuntimeScript, { run: (runOpts: LegacyEdgeRuntimeRunOpts) => { @@ -631,24 +614,6 @@ describe("legacy db start", () => { }, ); - it.live("leaves migration-owned pg_net alone when Webhooks are disabled", () => { - // `supabase start` does not replay migrations on an existing volume, so dropping - // pg_net here would silently break a database whose own migration created it, with - // nothing to put it back. - // Config validation rejects an explicit `enabled = false`, so "disabled" is the - // key being absent — exactly what a user who removes the block ends up with. - const { layer, dbSession } = setup({ - configContents: 'project_id = "test"\n', - appliedMigrationStatements: ["create extension if not exists pg_net with schema extensions"], - }); - return Effect.gen(function* () { - yield* legacyDbStart(DEFAULT_FLAGS).pipe(Effect.provide(layer)); - expect(dbSession.calls.some((call) => call.sql.includes(PG_NET_DROP_FINGERPRINT))).toBe( - false, - ); - }); - }); - it.live("installs pg_net on an existing volume from effective Webhooks config", () => { const { layer, out, child, dbSession } = setup({ configContents: 'project_id = "test"\n[experimental.webhooks]\nenabled = false\n', diff --git a/apps/cli/src/legacy/commands/migration/fetch/fetch.integration.test.ts b/apps/cli/src/legacy/commands/migration/fetch/fetch.integration.test.ts index dea2f6adaa..188b419f19 100644 --- a/apps/cli/src/legacy/commands/migration/fetch/fetch.integration.test.ts +++ b/apps/cli/src/legacy/commands/migration/fetch/fetch.integration.test.ts @@ -2,7 +2,7 @@ import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from import { join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; -import { Cause, Effect, Exit, FileSystem, Layer, Option, Path } from "effect"; +import { Cause, Effect, Exit, Layer, Option } from "effect"; import { LEGACY_VALID_REF, @@ -22,11 +22,7 @@ import type { LegacyDbConfigFlags, LegacyResolvedDbConfig, } from "../../../shared/legacy-db-config.types.ts"; -import { LegacyDbExecError } from "../../../shared/legacy-db-connection.errors.ts"; import { LegacyDbConnection } from "../../../shared/legacy-db-connection.service.ts"; -import type { LegacyDbSession } from "../../../shared/legacy-db-connection.service.ts"; -import { legacyApplyMigrationFile } from "../../../shared/legacy-migration-apply.ts"; -import { legacyParseMigrationContent } from "../../../shared/legacy-migration-file.ts"; import { legacyMigrationFetch } from "./fetch.handler.ts"; import type { LegacyMigrationFetchFlags } from "./fetch.command.ts"; @@ -144,16 +140,6 @@ const flags = (over: Partial = {}): LegacyMigrationFe const migrationsDir = (workdir: string) => join(workdir, "supabase", "migrations"); const tmp = useLegacyTempWorkdir(); -function stringArray(value: unknown): ReadonlyArray | undefined { - if (!Array.isArray(value)) return undefined; - const values: Array = []; - for (const item of value) { - if (typeof item !== "string") return undefined; - values.push(item); - } - return values; -} - describe("legacy migration fetch", () => { it.live("writes migration files joined with the Go separator when the dir is empty", () => { const { layer, out } = setup(tmp.current, { @@ -176,66 +162,6 @@ describe("legacy migration fetch", () => { }).pipe(Effect.provide(layer)); }); - it.live("preserves no-transaction metadata through apply, history, and fetch", () => { - const rows: Array = []; - const source = join(tmp.current, "20240102000000_drop_subscription.sql"); - writeFileSync( - source, - "\uFEFF-- pg-delta: transaction=false\r\n" + - "SET check_function_bodies = off;\r\n" + - "DROP SUBSCRIPTION app_events;\r\n" + - "RESET ALL;\r\n", - ); - const applySession: LegacyDbSession = { - exec: () => Effect.void, - query: (_sql, params) => - Effect.sync(() => { - const version = params?.[0]; - const name = params?.[1]; - const statements = stringArray(params?.[2]); - if (typeof version === "string" && typeof name === "string" && statements !== undefined) { - rows.push({ version, name, statements }); - } - return []; - }), - extensionExists: () => Effect.succeed(false), - copyToCsv: () => Effect.succeed(new Uint8Array()), - queryRaw: () => Effect.succeed({ fields: [], rows: [], commandTag: "" }), - }; - const { layer } = setup(tmp.current, { rows }); - - return Effect.gen(function* () { - const fs = yield* FileSystem.FileSystem; - const path = yield* Path.Path; - yield* legacyApplyMigrationFile( - applySession, - fs, - path, - source, - (message) => new LegacyDbExecError({ message }), - ); - - const firstStatement = "-- pg-delta: transaction=false\r\nSET check_function_bodies = off"; - expect(rows).toEqual([ - { - version: "20240102000000", - name: "drop_subscription", - statements: [firstStatement, "DROP SUBSCRIPTION app_events", "RESET ALL"], - }, - ]); - - yield* legacyMigrationFetch(flags()); - const fetched = readFileSync( - join(migrationsDir(tmp.current), "20240102000000_drop_subscription.sql"), - "utf8", - ); - expect(legacyParseMigrationContent(fetched)).toEqual({ - statements: [firstStatement, "DROP SUBSCRIPTION app_events", "RESET ALL"], - transactionMode: "none", - }); - }).pipe(Effect.provide(layer)); - }); - it.live("writes a lone separator for a row with no statements (Go parity)", () => { // A `schema_migrations` row can legally have a NULL/empty `statements` array // (older projects, manually-inserted rows). Joining statements with ";\n" diff --git a/apps/cli/src/legacy/commands/migration/repair/repair.integration.test.ts b/apps/cli/src/legacy/commands/migration/repair/repair.integration.test.ts index 973027c2dd..fa14041e82 100644 --- a/apps/cli/src/legacy/commands/migration/repair/repair.integration.test.ts +++ b/apps/cli/src/legacy/commands/migration/repair/repair.integration.test.ts @@ -166,31 +166,6 @@ describe("legacy migration repair", () => { }).pipe(Effect.provide(layer)); }); - it.live("preserves a no-transaction directive when repairing applied history", () => { - seedMigration( - tmp.current, - "20240102000000_drop_subscription.sql", - "\uFEFF-- pg-delta: transaction=false\r\n" + - "SET check_function_bodies = off;\r\n" + - "DROP SUBSCRIPTION app_events;\r\n" + - "RESET ALL;\r\n", - ); - const { layer, queries } = setup(tmp.current); - return Effect.gen(function* () { - yield* legacyMigrationRepair(input({ versions: ["20240102000000"], status: "applied" })); - const upsert = queries.find((query) => query.sql.includes("ON CONFLICT")); - expect(upsert?.params).toEqual([ - "20240102000000", - "drop_subscription", - [ - "-- pg-delta: transaction=false\r\nSET check_function_bodies = off", - "DROP SUBSCRIPTION app_events", - "RESET ALL", - ], - ]); - }).pipe(Effect.provide(layer)); - }); - it.live("resolves the DB target before parsing positional versions", () => { // The DB config resolves before // version parsing, so an unlinked target error wins over a bad version. diff --git a/apps/cli/src/legacy/commands/start/start.integration.test.ts b/apps/cli/src/legacy/commands/start/start.integration.test.ts index 6b8ce4cf0e..03a3fe826e 100644 --- a/apps/cli/src/legacy/commands/start/start.integration.test.ts +++ b/apps/cli/src/legacy/commands/start/start.integration.test.ts @@ -69,7 +69,6 @@ import { * containers. */ const legacyResolveLocalConfigValuesCalls = vi.hoisted(() => ({ count: 0 })); -const PG_NET_CREATE_FINGERPRINT = "create extension if not exists pg_net schema extensions"; vi.mock("../../shared/legacy-local-config-values.ts", async () => { const actual = await vi.importActual( @@ -2595,21 +2594,6 @@ content_path = "./supabase/templates/custom_notice.html" }, ); - it.live("installs pg_net when restarting an existing Webhooks-enabled database", () => { - const { layer, out, dbSession } = setup({ - configContents: 'project_id = "demo"\n[experimental.webhooks]\nenabled = true\n', - }); - return Effect.gen(function* () { - yield* legacyStart(flags({ exclude: ["edge-runtime"] })); - expect(out.stderrText).not.toContain("Initialising schema..."); - expect( - dbSession.calls.filter( - (call) => call.kind === "exec" && call.sql.includes(PG_NET_CREATE_FINGERPRINT), - ), - ).toHaveLength(1); - }).pipe(Effect.provide(layer)); - }); - it.live( "still writes supabase/.branches/_current_branch on a restart, even though the fresh-volume DB setup is skipped", () => { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts index 763bdf8e9a..b61ab5d870 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts @@ -680,15 +680,7 @@ function legacyDockerStartContainer( } /** - * `docker cp :` — copies an already-written HOST file - * straight into the container's own filesystem over the same Docker CLI/Engine API connection as - * `docker create`/`docker start`, so it works identically against a local or remote - * (`DOCKER_HOST`/Docker-context) daemon. Unlike a bind mount, which Docker resolves against the - * DAEMON's own filesystem - * (https://docs.docker.com/engine/storage/bind-mounts/#considerations-and-constraints), `docker - * cp` never depends on the source path being visible to anything other than the CLI process - * issuing it — see {@link legacyCopyStartSecretFilesIntoContainer}'s doc comment for the full - * rationale (supabase/cli#6022). + * Copies a host file into a created container through the configured Docker connection. */ function legacyDockerCopyIntoContainer( spawner: Spawner, @@ -740,32 +732,9 @@ function legacyDockerCopyIntoContainer( } /** - * Writes one {@link LegacyStartContainerSpec.secretFiles} entry's `content` to a SHORT-LIVED - * local temp file (`fs.mkdtemp`'d under `os.tmpdir()`, one file per entry in its own directory so - * sibling entries never collide), force-`chmod`'d to mode `0644` after writing (a creation-time - * `writeFile({ mode })` is only ever the argument to the underlying `open()`/`creat()` syscall, - * which the kernel ANDs with `~umask` — under a restrictive shell umask like `077`/`027` the file - * would otherwise land at `0600`, unlike `chmod`, which sets the mode unconditionally), then - * `docker cp`s it into `containerId` at `secretFile.containerPath` - * ({@link legacyDockerCopyIntoContainer}). - * - * `0644` (world-readable) still matters here even though this is no longer a bind mount: `docker - * cp`'s underlying tar transfer preserves the source file's permission bits verbatim inside the - * container, exactly like a Linux/Podman bind mount did, and the container reads this file back as - * a NON-ROOT in-container user (Kong's image runs as uid 100 `kong`; Postgres's entrypoint drops - * root and reads `pgsodium_root.key` as the `postgres` user) — a `0600` file would still be - * `EACCES` there. Go's own equivalent (heredoc'd directly into the container by a root-authored - * entrypoint script) already lands at world-readable `0644`, matching this exactly. - * - * The temp file (and its enclosing directory) is removed immediately after this ONE entry's own - * `docker cp` call returns — success, failure, or interruption alike, via `Effect.ensuring` (built - * on `onExit`, so it fires on every exit, not just a `Fail`). Unlike the old host-staged bind - * mount (which had to persist for the container's whole lifetime so a `restartPolicy: - * "unless-stopped"` restart could re-attach it — Go's own heredoc'd-into-`Entrypoint`/`Cmd` - * content survives a restart via dockerd's own persisted container metadata instead), the secret - * content is already INSIDE the container's own filesystem the moment `docker cp` returns, so - * nothing depends on this host temp file surviving a moment longer — a dockerd restart re-attaches - * whatever bind mounts/volumes the container has, but never needs to re-run this copy. + * Stages one secret at mode 0644, copies it into the container, and removes the + * host artifact on success, failure, or interruption. The explicit chmod avoids + * restrictive umasks making the file unreadable to the container's non-root user. */ function legacyCopyStartSecretFileIntoContainer( spawner: Spawner, @@ -787,8 +756,6 @@ function legacyCopyStartSecretFileIntoContainer( return Effect.tryPromise({ try: async () => { await writeFile(hostPath, secretFile.content, { mode: 0o644 }); - // See this function's doc comment — a creation-time `mode` alone isn't enough - // under a restrictive umask. await chmod(hostPath, 0o644); }, catch: (cause) => @@ -813,12 +780,7 @@ function legacyCopyStartSecretFileIntoContainer( } /** - * Delivers every {@link LegacyStartContainerSpec.secretFiles} entry into `containerId` — the - * container `legacyCreateContainer` just created via `docker create`, but has NOT yet started — - * via `docker cp` ({@link legacyCopyStartSecretFileIntoContainer}, one call per entry, run - * concurrently). No Go struct equivalent — see `docker-create-args.ts`'s `secretFiles` doc - * comment for why this port needs it at all, and this function's own doc comment for why `docker - * cp` (not a host bind mount) is how it delivers that content. + * Copies all secrets into the created, not-yet-started container concurrently. */ function legacyCopyStartSecretFilesIntoContainer( spawner: Spawner, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.unit.test.ts index 39e224f6a8..26e7ccfe7b 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.unit.test.ts @@ -363,334 +363,183 @@ describe("legacyCreateContainer", () => { }); describe("legacyCreateContainer secretFiles", () => { - it.live( - "docker cp's a secretFile into the created (not yet started) container, strictly between `docker create` and `docker start`, keeping its content out of every spawned process's own argv, then removes the local temp file", - () => { - let hostPath: string | undefined; - let cpArgs: ReadonlyArray | undefined; - const mock = mockSpawner((args) => { - if (args[0] === "create") return { exitCode: 0, stdout: "container-id-789\n" }; - if (args[0] === "cp") { - cpArgs = args; - hostPath = args[1]; - } - return { exitCode: 0 }; - }); - - const spec: LegacyStartContainerSpec = { - ...baseSpec, - binds: [], - secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], - }; + it.live("copies a mode-0644 secret before start without leaking it to argv or disk", () => { + let hostPath: string | undefined; + let cpArgs: ReadonlyArray | undefined; + let modeAtCopyTime: number | undefined; + const mock = mockSpawner((args) => { + if (args[0] === "create") return { exitCode: 0, stdout: "container-id-umask\n" }; + if (args[0] === "cp") { + cpArgs = args; + hostPath = args[1]; + modeAtCopyTime = statSync(hostPath ?? "").mode & 0o777; + } + return { exitCode: 0 }; + }); - return legacyCreateContainer(mock.spawner, spec, { - projectId: "proj", - isBitbucketPipeline: false, - workdir, - extraHosts: [], - }).pipe( - Effect.map((containerId) => { - expect(containerId).toBe("container-id-789"); + const spec: LegacyStartContainerSpec = { + ...baseSpec, + binds: [], + secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], + }; - expect(cpArgs).toEqual(["cp", hostPath, "container-id-789:/etc/kong/kong.yml"]); - expect(cpArgs?.some((a) => a.includes("super-secret-content"))).toBe(false); - const create = mock.spawned.find((a) => a[0] === "create"); - expect(create?.some((a) => a.includes("super-secret-content"))).toBe(false); + return Effect.sync(() => process.umask(0o077)).pipe( + Effect.flatMap((originalUmask) => + legacyCreateContainer(mock.spawner, spec, { + projectId: "proj", + isBitbucketPipeline: false, + workdir, + extraHosts: [], + }).pipe( + Effect.map(() => { + expect(modeAtCopyTime).toBe(0o644); + expect(cpArgs).toEqual(["cp", hostPath, "container-id-umask:/etc/kong/kong.yml"]); + expect(mock.spawned.map((args) => args[0])).toEqual(["create", "cp", "start"]); + expect(mock.spawned.flat().join(" ")).not.toContain("super-secret-content"); + expect(existsSync(hostPath ?? "")).toBe(false); + }), + Effect.ensuring(Effect.sync(() => process.umask(originalUmask))), + ), + ), + ); + }); - // `docker cp` runs strictly between `docker create` and `docker start` — the - // container must already exist for it to have a target, and must not be running - // yet so its entrypoint never races the copy. - expect(mock.spawned.map((a) => a[0])).toEqual(["create", "cp", "start"]); + it.live("cleans up and never starts when docker cp fails", () => { + let hostPath: string | undefined; + const mock = mockSpawner((args) => { + if (args[0] === "create") return { exitCode: 0, stdout: "container-id-def\n" }; + if (args[0] === "cp") { + hostPath = args[1]; + return { exitCode: 1, stderr: "Error: No such container: container-id-def\n" }; + } + return { exitCode: 0 }; + }); - // Delivered straight into the container — nothing persists on host disk afterward. - expect(hostPath).toBeDefined(); - expect(existsSync(hostPath ?? "")).toBe(false); - }), - ); - }, - ); + const spec: LegacyStartContainerSpec = { + ...baseSpec, + binds: [], + secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], + }; - it.live( - "keeps the copied secretFile at HOST mode 0644 even under a restrictive process umask (writeFile's `mode` is only a creation-time hint ANDed with the umask — without the explicit chmod, a 0077 umask would silently narrow the on-disk mode to 0600, and docker cp's mode-preserving tar transfer would carry that into the container, giving the non-root in-container reader EACCES)", - () => { - let hostPath: string | undefined; - let modeAtCopyTime: number | undefined; - const mock = mockSpawner((args) => { - if (args[0] === "create") return { exitCode: 0, stdout: "container-id-umask\n" }; - if (args[0] === "cp") { - hostPath = args[1]; - modeAtCopyTime = statSync(hostPath ?? "").mode & 0o777; - } - return { exitCode: 0 }; - }); + return legacyCreateContainer(mock.spawner, spec, { + projectId: "proj", + isBitbucketPipeline: false, + workdir, + extraHosts: [], + }).pipe( + Effect.flip, + Effect.map((error) => { + expect(error).toBeInstanceOf(LegacyContainerCreateError); + expect(error.message).toBe( + "failed to create docker container: failed to copy secret file into container: Error: No such container: container-id-def", + ); + expect(hostPath).toBeDefined(); + expect(existsSync(hostPath ?? "")).toBe(false); + expect(mock.spawned.some((args) => args[0] === "start")).toBe(false); + }), + ); + }); - const spec: LegacyStartContainerSpec = { - ...baseSpec, - binds: [], - secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], - }; + it.live("never copies or starts when docker create fails", () => { + const mock = mockSpawner((args) => { + if (args[0] === "create") return { exitCode: 1, stderr: "no such image\n" }; + return { exitCode: 0 }; + }); - // `it.live`'s test function runs inside `Effect.suspend`, so a plain JS try/finally around - // this `return` would restore the umask synchronously right after CONSTRUCTING the effect - // pipeline below, before it actually runs — long before the real write+chmod happens. - // `Effect.ensuring` is the effect-native equivalent: it sequences the restore to run only - // after this effect actually completes, on success, failure, or defect alike. - return Effect.sync(() => process.umask(0o077)).pipe( - Effect.flatMap((originalUmask) => - legacyCreateContainer(mock.spawner, spec, { - projectId: "proj", - isBitbucketPipeline: false, - workdir, - extraHosts: [], - }).pipe( - Effect.map(() => { - expect(hostPath).toBeDefined(); - expect(modeAtCopyTime).toBe(0o644); - }), - Effect.ensuring(Effect.sync(() => process.umask(originalUmask))), - ), - ), - ); - }, - ); + const spec: LegacyStartContainerSpec = { + ...baseSpec, + binds: [], + secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], + }; - it.live( - "removes the local temp file immediately after a successful `docker cp`, even though `docker start` later fails", - () => { - let hostPath: string | undefined; - const mock = mockSpawner((args) => { - if (args[0] === "create") return { exitCode: 0, stdout: "container-id-abc\n" }; - if (args[0] === "cp") { - hostPath = args[1]; - return { exitCode: 0 }; - } - if (args[0] === "start") { - return { exitCode: 1, stderr: "container is already stopped\n" }; - } - return { exitCode: 0 }; - }); + return legacyCreateContainer(mock.spawner, spec, { + projectId: "proj", + isBitbucketPipeline: false, + workdir, + extraHosts: [], + }).pipe( + Effect.flip, + Effect.map((error) => { + expect(error).toBeInstanceOf(LegacyContainerCreateError); + expect(mock.spawned.some((args) => args[0] === "cp")).toBe(false); + expect(mock.spawned.some((args) => args[0] === "start")).toBe(false); + }), + ); + }); - const spec: LegacyStartContainerSpec = { - ...baseSpec, - binds: [], - secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], - }; + it.live("removes the local secret when interrupted during docker cp", () => { + const cpStarted = Deferred.makeUnsafe(); + const hangForever = Deferred.makeUnsafe(); + let hostPath: string | undefined; + const encoder = new TextEncoder(); - return legacyCreateContainer(mock.spawner, spec, { - projectId: "proj", - isBitbucketPipeline: false, - workdir, - extraHosts: [], - }).pipe( - Effect.flip, - Effect.map((error) => { - expect(error).toBeInstanceOf(LegacyContainerStartError); - expect(hostPath).toBeDefined(); - // Already removed right after its own successful `docker cp` — long before `docker - // start` even ran, let alone failed. - expect(existsSync(hostPath ?? "")).toBe(false); - }), - ); - }, - ); + function succeededHandle(stdout = "") { + return Effect.gen(function* () { + const exitDeferred = yield* Deferred.make(); + yield* Deferred.succeed(exitDeferred, ChildProcessSpawner.ExitCode(0)); + return ChildProcessSpawner.makeHandle({ + pid: ChildProcessSpawner.ProcessId(1), + stdout: Stream.fromIterable(stdout.length > 0 ? [encoder.encode(stdout)] : []), + stderr: Stream.empty, + all: Stream.empty, + exitCode: Deferred.await(exitDeferred), + isRunning: Effect.succeed(false), + stdin: Sink.drain, + kill: () => Effect.void, + unref: Effect.succeed(Effect.void), + getInputFd: () => Sink.drain, + getOutputFd: () => Stream.empty, + }); + }); + } - it.live( - "fails with LegacyContainerCreateError when `docker cp` exits non-zero, removes the local temp file, and never invokes `docker start`", - () => { - let hostPath: string | undefined; - const mock = mockSpawner((args) => { - if (args[0] === "create") return { exitCode: 0, stdout: "container-id-def\n" }; + const spawner = ChildProcessSpawner.make((command) => + Effect.gen(function* () { + const args = command._tag === "StandardCommand" ? command.args : []; + if (args[0] === "create") { + return yield* succeededHandle("container-id-sigint\n"); + } if (args[0] === "cp") { hostPath = args[1]; - return { exitCode: 1, stderr: "Error: No such container: container-id-def\n" }; - } - return { exitCode: 0 }; - }); - - const spec: LegacyStartContainerSpec = { - ...baseSpec, - binds: [], - secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], - }; - - return legacyCreateContainer(mock.spawner, spec, { - projectId: "proj", - isBitbucketPipeline: false, - workdir, - extraHosts: [], - }).pipe( - Effect.flip, - Effect.map((error) => { - expect(error).toBeInstanceOf(LegacyContainerCreateError); - expect(error.message).toBe( - "failed to create docker container: failed to copy secret file into container: Error: No such container: container-id-def", - ); - expect(hostPath).toBeDefined(); - expect(existsSync(hostPath ?? "")).toBe(false); - expect(mock.spawned.some((args) => args[0] === "start")).toBe(false); - }), - ); - }, - ); - - it.live( - "never invokes `docker cp` (or writes any local temp file) when `docker create` fails", - () => { - const mock = mockSpawner((args) => { - if (args[0] === "create") return { exitCode: 1, stderr: "no such image\n" }; - return { exitCode: 0 }; - }); - - const spec: LegacyStartContainerSpec = { - ...baseSpec, - binds: [], - secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], - }; - - return legacyCreateContainer(mock.spawner, spec, { - projectId: "proj", - isBitbucketPipeline: false, - workdir, - extraHosts: [], - }).pipe( - Effect.flip, - Effect.map((error) => { - expect(error).toBeInstanceOf(LegacyContainerCreateError); - expect(mock.spawned.some((args) => args[0] === "cp")).toBe(false); - expect(mock.spawned.some((args) => args[0] === "start")).toBe(false); - }), - ); - }, - ); - - it.live( - "removes the local temp file on a SIGINT-style interruption mid-`docker cp`, matching Go's no-orphaned-secrets guarantee", - () => { - // Go never writes these secrets to a host file at all (see - // `legacyCopyStartSecretFileIntoContainer`'s doc comment), so this is judged on its own - // correctness/security merits, not Go parity: a SIGINT landing after the local temp file - // is written but before `docker cp` returns must not leave a plaintext secret file behind - // indefinitely. `Effect.tapError` never sees a pure fiber interrupt — only `Effect.onError`/ - // `Effect.ensuring` (built on `onExit`) do — same class of gap already fixed for the - // top-level bring-up rollback in `start.handler.ts`. - const cpStarted = Deferred.makeUnsafe(); - const hangForever = Deferred.makeUnsafe(); - let hostPath: string | undefined; - const encoder = new TextEncoder(); - - function succeededHandle(stdout = "") { - return Effect.gen(function* () { - const exitDeferred = yield* Deferred.make(); - yield* Deferred.succeed(exitDeferred, ChildProcessSpawner.ExitCode(0)); + yield* Deferred.succeed(cpStarted, undefined); return ChildProcessSpawner.makeHandle({ pid: ChildProcessSpawner.ProcessId(1), - stdout: Stream.fromIterable(stdout.length > 0 ? [encoder.encode(stdout)] : []), + stdout: Stream.empty, stderr: Stream.empty, all: Stream.empty, - exitCode: Deferred.await(exitDeferred), - isRunning: Effect.succeed(false), + exitCode: Deferred.await(hangForever), + isRunning: Effect.succeed(true), stdin: Sink.drain, kill: () => Effect.void, unref: Effect.succeed(Effect.void), getInputFd: () => Sink.drain, getOutputFd: () => Stream.empty, }); - }); - } - - const spawner = ChildProcessSpawner.make((command) => - Effect.gen(function* () { - const args = command._tag === "StandardCommand" ? command.args : []; - if (args[0] === "create") { - return yield* succeededHandle("container-id-sigint\n"); - } - if (args[0] === "cp") { - hostPath = args[1]; - yield* Deferred.succeed(cpStarted, undefined); - // Never resolves on its own — only interruption ends this "process". - return ChildProcessSpawner.makeHandle({ - pid: ChildProcessSpawner.ProcessId(1), - stdout: Stream.empty, - stderr: Stream.empty, - all: Stream.empty, - exitCode: Deferred.await(hangForever), - isRunning: Effect.succeed(true), - stdin: Sink.drain, - kill: () => Effect.void, - unref: Effect.succeed(Effect.void), - getInputFd: () => Sink.drain, - getOutputFd: () => Stream.empty, - }); - } - return yield* succeededHandle(); - }), - ); - - const spec: LegacyStartContainerSpec = { - ...baseSpec, - binds: [], - secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], - }; - - return Effect.gen(function* () { - const fiber = yield* legacyCreateContainer(spawner, spec, { - projectId: "proj", - isBitbucketPipeline: false, - workdir, - extraHosts: [], - }).pipe(Effect.forkChild({ startImmediately: true })); - yield* Deferred.await(cpStarted); - expect(hostPath).toBeDefined(); - expect(existsSync(hostPath ?? "")).toBe(true); - yield* Fiber.interrupt(fiber); - expect(existsSync(hostPath ?? "")).toBe(false); - }); - }, - ); - - it.live( - "maps a local temp-file creation failure to LegacyContainerCreateError, without ever invoking `docker cp` or `docker start`", - () => { - const previousTmpdir = process.env["TMPDIR"]; - // Points `os.tmpdir()` at a path whose PARENT doesn't exist, forcing `fs.mkdtemp` to fail - // deterministically with ENOENT — the only way to exercise this staging try/catch's - // failure branch from a unit test. - process.env["TMPDIR"] = join(workdir, "does-not-exist", "nested"); + } + return yield* succeededHandle(); + }), + ); - const mock = mockSpawner((args) => { - if (args[0] === "create") return { exitCode: 0, stdout: "container-id-tmp\n" }; - return { exitCode: 0 }; - }); - const spec: LegacyStartContainerSpec = { - ...baseSpec, - binds: [], - secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], - }; + const spec: LegacyStartContainerSpec = { + ...baseSpec, + binds: [], + secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }], + }; - return legacyCreateContainer(mock.spawner, spec, { + return Effect.gen(function* () { + const fiber = yield* legacyCreateContainer(spawner, spec, { projectId: "proj", isBitbucketPipeline: false, workdir, extraHosts: [], - }).pipe( - Effect.flip, - Effect.map((error) => { - expect(error).toBeInstanceOf(LegacyContainerCreateError); - expect(error.message).toMatch( - /^failed to create docker container: failed to stage container secret file: /, - ); - expect(mock.spawned.some((args) => args[0] === "cp")).toBe(false); - expect(mock.spawned.some((args) => args[0] === "start")).toBe(false); - }), - Effect.ensuring( - Effect.sync(() => { - if (previousTmpdir === undefined) delete process.env["TMPDIR"]; - else process.env["TMPDIR"] = previousTmpdir; - }), - ), - ); - }, - ); + }).pipe(Effect.forkChild({ startImmediately: true })); + yield* Deferred.await(cpStarted); + expect(hostPath).toBeDefined(); + expect(existsSync(hostPath ?? "")).toBe(true); + yield* Fiber.interrupt(fiber); + expect(existsSync(hostPath ?? "")).toBe(false); + }); + }); }); describe("legacyEnsureNetwork", () => { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index 5ad6b732ee..d96b221d51 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -460,10 +460,7 @@ export interface LegacySetupDatabaseInput { /** Controls the extension side effects of {@link legacySetupDatabase}. */ export interface LegacySetupDatabaseOptions { - /** Apply extensions enabled by project config. Disabled for a declarative desired-state scratch. */ - readonly activateUserExtensions?: boolean; - /** Install `pg_net` as a legacy platform baseline, independently of project config. */ - readonly legacyPgNetBaseline?: boolean; + readonly webhooks?: "config" | "enabled" | "disabled"; } /** Input to {@link legacyStartSetupLocalDatabase}. */ @@ -1056,15 +1053,13 @@ export const legacySetupDatabase = ( if (requiresPg14WebhooksCleanup) { yield* legacyRemoveDatabaseWebhooks(session, fs, path, tmpDir); } - const activateUserExtensions = options.activateUserExtensions ?? true; - const legacyPgNetBaseline = options.legacyPgNetBaseline ?? false; - const userEnabled = activateUserExtensions && input.webhooksEnabled; + const webhooks = options.webhooks ?? "config"; yield* legacyApplyDatabaseWebhooks( session, fs, path, tmpDir, - legacyPgNetBaseline || userEnabled, + webhooks === "enabled" || (webhooks === "config" && input.webhooksEnabled), ); yield* legacyApplyApiPrivileges(session, fs, path, tmpDir, input.apiAutoExposeNewTables); }), @@ -1323,22 +1318,7 @@ const legacyConnectLocalPostgres = (input: { ); }); -/** - * Idempotently converges Database Webhooks on a healthy, existing local database — - * in BOTH directions. Installing pg_net when the setting is on was always covered; - * removing it when the setting is turned back off was not, so pg_net silently - * survived on the volume while the next engine's shadow baseline (rebuilt from the - * current config) omitted it, reporting pg_net drift on every `db diff`/`db pull`. - * - * The removal is guarded, because `supabase start` does NOT replay migrations on an - * existing volume: an unconditional drop would delete an extension a user's own - * migration created, with nothing to put it back. So pg_net is dropped only when no - * applied migration in `supabase_migrations.schema_migrations` installs it (see - * {@link legacyStatementInstallsPgNet}), and any failure to read that history — - * missing table, malformed table, insufficient privileges — is treated as - * "migration-owned" and leaves the extension alone. Erring toward not dropping is - * the only safe direction here. - */ +/** Converges pg_net while preserving extensions installed by user migrations. */ export const legacyRunDatabaseWebhooksSetup = (input: { readonly fs: FileSystem.FileSystem; readonly path: Path.Path; diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts index e171b8a051..f94a45c50a 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts @@ -593,46 +593,6 @@ describe("legacyStartSetupLocalDatabase", () => { ); }); - describe("Database Webhooks", () => { - it.effect("does not install pg_net merely because Edge Runtime is enabled", () => { - const workdir = makeWorkdir(); - const { session, calls } = fakeSession(); - const out = mockOutput(); - const docker = mockDockerRun(); - const config = decodeConfig({ edge_runtime: { enabled: true } }); - return run(baseInput(workdir, session, { majorVersion: 14, config }), out, docker).pipe( - Effect.map(() => { - const execSql = calls.filter((c) => c.kind === "exec").map((c) => c.sql); - expect(execSql.some((sql) => sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(false); - rmSync(workdir, { recursive: true, force: true }); - }), - ); - }); - - it.effect("installs pg_net from the effective Database Webhooks environment override", () => { - const workdir = makeWorkdir(); - writeConfigToml(workdir, "[experimental.webhooks]\nenabled = false\n"); - writeFileSync( - join(workdir, "supabase", ".env"), - "SUPABASE_EXPERIMENTAL_WEBHOOKS_ENABLED=true\n", - ); - const { session, calls } = fakeSession(); - const out = mockOutput(); - const docker = mockDockerRun(); - const config = decodeConfig({ - edge_runtime: { enabled: false }, - experimental: { webhooks: { enabled: false } }, - }); - return run(baseInput(workdir, session, { majorVersion: 14, config }), out, docker).pipe( - Effect.map(() => { - const execSql = calls.filter((c) => c.kind === "exec").map((c) => c.sql); - expect(execSql.filter((sql) => sql.includes(PG_NET_CREATE_FINGERPRINT))).toHaveLength(1); - rmSync(workdir, { recursive: true, force: true }); - }), - ); - }); - }); - describe("vault upsert + custom-roles seed", () => { it.effect("upserts vault secrets before seeding supabase/roles.sql", () => { const workdir = makeWorkdir(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts b/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts index d9332ed0e2..38dbe60f7b 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts @@ -150,46 +150,8 @@ export interface LegacyStartContainerSpec { */ readonly env: Readonly>; /** - * Entrypoint/`Cmd`-script secret content that must land at a specific path - * INSIDE the container without ever appearing in this process's own - * `docker create` argv (`ps aux`/`/proc//cmdline`, CWE-214/522) — the - * entrypoint/`Cmd` analogue of {@link env}'s key-only `-e KEY` protection. - * Kong/Postgres/Supavisor all heredoc or shell-embed secret-bearing content - * (Kong's `kong.yml`/TLS private key, Postgres's pgsodium root key, - * Supavisor's rendered `pooler.exs`) directly into their entrypoint script - * or `Cmd` — safe in Go's Engine-API architecture (never a subprocess's own - * argv) but not in this port's, which shells out to a real `docker create`. - * - * NOT consumed here: {@link legacyBuildStartContainerCreateArgs} stays - * pure/no-I/O and never reads this field. `container-lifecycle.ts`'s - * `legacyCreateContainer` is the sole consumer — once `docker create` returns - * a container id, it writes each entry's `content` to a SHORT-LIVED - * HOST-side temp file (mode `0644` — world-readable, so the non-root - * in-container user reading it (e.g. Kong, Postgres) doesn't hit `EACCES`; - * `docker cp`'s tar transfer preserves the host file's mode verbatim, same - * as a bind mount did — see `legacyCopyStartSecretFileIntoContainer`'s doc - * comment) and `docker cp`s it straight into the (created, not yet started) - * container at `containerPath`, removing the temp file immediately - * afterward — never a host bind mount. Generic by design — any future - * service's spec can set this, not just the three call sites that need it - * today, and it makes no difference whether `containerName` is set: `docker - * cp` addresses the container by the id `docker create` returns, not by - * name, so the shadow database's own unnamed container (`db-bootstrap/ - * shadow-database.ts`) is delivered its pgsodium root key the exact same way. - * - * `docker cp` streams the file's content over the same Docker CLI/Engine - * API connection as `docker create`/`docker start`, so — unlike the - * bind-mount approach this replaced (supabase/cli#6022) — it works - * identically whether `DOCKER_HOST`/Docker-context points at a local or a - * REMOTE daemon: a bind mount's host-side path is resolved by the daemon - * itself - * (https://docs.docker.com/engine/storage/bind-mounts/#considerations-and-constraints), - * so it silently broke against a remote daemon (a scenario this codebase - * otherwise explicitly supports — see `legacy-hostname.ts`'s - * `legacyGetHostname`) even though the daemon itself was reachable. `docker - * cp` has no such requirement, matching Go's own heredoc/`Cmd`-embed - * delivery (the content travels inside the container-create request itself, - * over the Engine API) for that same reason. + * Secret-bearing files copied after create and before start. The lifecycle + * layer keeps their contents out of argv and works with remote Docker daemons. */ readonly secretFiles?: ReadonlyArray; /** diff --git a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts index 440aadfa80..122418f02f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts @@ -89,11 +89,6 @@ describe("legacyBuildPostgresStartContainerSpec", () => { ); expect(script).not.toContain(LEGACY_POSTGRES_DEFAULT_ROOT_KEY); expect(script).not.toContain("pgsodium_root.key"); - expect(LEGACY_START_DB_WEBHOOK_SQL).not.toContain("CREATE EXTENSION IF NOT EXISTS pg_net"); - expect(LEGACY_START_DB_WEBHOOK_SQL).toContain( - "CREATE OR REPLACE FUNCTION extensions.grant_pg_net_access()", - ); - expect(LEGACY_START_DB_WEBHOOK_SQL).toContain("CREATE EVENT TRIGGER issue_pg_net_access"); expect(spec.tmpfs).toBeUndefined(); expect(spec.secretFiles).toEqual([ { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index 992c3be164..a506a5dcd6 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -732,7 +732,7 @@ export const legacyMigrateShadowDatabase = ( void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, Output | LegacyDockerRun | RuntimeInfo | LegacyDbConnection -> => migrateShadowDatabase(spawner, input, { legacyPgNetBaseline: true }); +> => migrateShadowDatabase(spawner, input, { webhooks: "enabled" }); /** * Migrates a shadow for the in-process pg-delta engine. Unlike the legacy engine, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index 60f4205d06..10fe3c93ec 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -407,30 +407,6 @@ describe("legacySetupShadowConn", () => { ), ); }); - - it.effect("can disable config-driven extension activation for a desired-state scratch", () => { - const { session, calls } = fakeSession(); - const workdir = tempRoot.current; - const mock = mockSpawner(); - return Effect.gen(function* () { - const fs = yield* FileSystem.FileSystem; - const path = yield* Path.Path; - const input = baseSetupDatabaseInput(session, fs, path, workdir); - yield* legacySetupShadowConn( - mock.spawner, - { - ...input, - webhooksEnabled: true, - }, - { activateUserExtensions: false }, - ); - expect(calls.some((call) => call.sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(false); - }).pipe( - Effect.provide( - Layer.mergeAll(BunServices.layer, mockOutput().layer, mockDockerRun(), mockRuntimeInfo()), - ), - ); - }); }); function baseShadowSetup( @@ -461,6 +437,44 @@ function baseShadowSetup( }; } +function migrateNextShadow(webhooksEnabled: boolean) { + const { session, calls } = fakeSession(); + const workdir = tempRoot.current; + const mock = mockSpawner(); + const effect = Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + yield* fs.makeDirectory(path.join(workdir, "supabase", "migrations"), { recursive: true }); + yield* legacyMigrateNextShadowDatabase(mock.spawner, { + fs, + path, + workdir, + projectId: "proj", + container: "shadow-container-id-0123456789abcdef", + networkId: "supabase_network_proj", + connConfig: { + host: "127.0.0.1", + port: 54320, + user: "postgres", + password: "postgres", + database: "postgres", + }, + setup: baseShadowSetup({ webhooksEnabled }), + }); + }).pipe( + Effect.provide( + Layer.mergeAll( + BunServices.layer, + mockOutput().layer, + mockDockerRun(), + mockRuntimeInfo(), + mockDbConnection(session), + ), + ), + ); + return { calls, effect }; +} + describe("legacyBuildShadowSetupDatabaseInput", () => { it.effect( "derives dbHost from the container's own 12-char short id and threads every field through", @@ -601,39 +615,12 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { ); it.effect("next migrated shadows keep pg_net activation config-gated", () => { - const { session, calls } = fakeSession(); - const workdir = tempRoot.current; - const mock = mockSpawner(); - return Effect.gen(function* () { - const fs = yield* FileSystem.FileSystem; - const path = yield* Path.Path; - yield* fs.makeDirectory(path.join(workdir, "supabase", "migrations"), { recursive: true }); - yield* legacyMigrateNextShadowDatabase(mock.spawner, { - fs, - path, - workdir, - projectId: "proj", - container: "shadow-container-id-0123456789abcdef", - networkId: "supabase_network_proj", - connConfig: { - host: "127.0.0.1", - port: 54320, - user: "postgres", - password: "postgres", - database: "postgres", - }, - setup: baseShadowSetup(), - }); - expect(calls.some((call) => call.sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(false); - }).pipe( - Effect.provide( - Layer.mergeAll( - BunServices.layer, - mockOutput().layer, - mockDockerRun(), - mockRuntimeInfo(), - mockDbConnection(session), - ), + const { calls, effect } = migrateNextShadow(false); + return effect.pipe( + Effect.tap(() => + Effect.sync(() => { + expect(calls.some((call) => call.sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(false); + }), ), ); }); @@ -641,39 +628,12 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { it.effect( "next migrated shadows install pg_net when effective Webhooks config is enabled", () => { - const { session, calls } = fakeSession(); - const workdir = tempRoot.current; - const mock = mockSpawner(); - return Effect.gen(function* () { - const fs = yield* FileSystem.FileSystem; - const path = yield* Path.Path; - yield* fs.makeDirectory(path.join(workdir, "supabase", "migrations"), { recursive: true }); - yield* legacyMigrateNextShadowDatabase(mock.spawner, { - fs, - path, - workdir, - projectId: "proj", - container: "shadow-container-id-0123456789abcdef", - networkId: "supabase_network_proj", - connConfig: { - host: "127.0.0.1", - port: 54320, - user: "postgres", - password: "postgres", - database: "postgres", - }, - setup: baseShadowSetup({ webhooksEnabled: true }), - }); - expect(calls.some((call) => call.sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(true); - }).pipe( - Effect.provide( - Layer.mergeAll( - BunServices.layer, - mockOutput().layer, - mockDockerRun(), - mockRuntimeInfo(), - mockDbConnection(session), - ), + const { calls, effect } = migrateNextShadow(true); + return effect.pipe( + Effect.tap(() => + Effect.sync(() => { + expect(calls.some((call) => call.sql.includes(PG_NET_CREATE_FINGERPRINT))).toBe(true); + }), ), ); }, @@ -714,21 +674,10 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { }), }), }, - { activateUserExtensions: false }, + { webhooks: "disabled" }, ); expect(jwksEvaluated).toBe(false); - const enablePgNet = calls.findIndex((call) => - call.sql.includes("CREATE EXTENSION IF NOT EXISTS pg_net WITH SCHEMA extensions"), - ); - const grantPgNet = calls.findIndex((call) => - call.sql.includes("GRANT USAGE ON SCHEMA net TO supabase_functions_admin"), - ); - const removePgNet = calls.findIndex( - (call) => call.sql === "drop extension if exists pg_net", - ); - expect(enablePgNet).toBeGreaterThanOrEqual(0); - expect(grantPgNet).toBeGreaterThan(enablePgNet); - expect(removePgNet).toBeGreaterThan(grantPgNet); + expect(calls.some((call) => call.sql === "drop extension if exists pg_net")).toBe(true); }).pipe( Effect.provide( Layer.mergeAll( diff --git a/apps/cli/src/legacy/shared/legacy-migrate-and-seed.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migrate-and-seed.unit.test.ts index c361e46628..5d0b9aca0c 100644 --- a/apps/cli/src/legacy/shared/legacy-migrate-and-seed.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migrate-and-seed.unit.test.ts @@ -15,10 +15,7 @@ import { legacyMigrateAndSeed, type LegacyMigrateAndSeedConfig, } from "./legacy-migrate-and-seed.ts"; -import { - LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION, - legacyIsPgNetUnavailableError, -} from "./legacy-pg-net-guidance.ts"; +import { LEGACY_ENABLE_LOCAL_WEBHOOKS_SUGGESTION } from "./legacy-pg-net-guidance.ts"; // Root bypasses POSIX permission bits, so chmod 000 wouldn't block readdir() there. const isRoot = typeof process.getuid === "function" && process.getuid() === 0; @@ -468,22 +465,6 @@ describe("legacyMigrateAndSeed local pg_net remediation", () => { "select net.http_post(url := 'https://example.com');", ); - it("classifies only pg_net schema/function errors with their matching SQLSTATE", () => { - expect(legacyIsPgNetUnavailableError(missingNetSchema)).toBe(true); - expect( - legacyIsPgNetUnavailableError({ - message: "ERROR: function net.http_post(unknown, jsonb) does not exist (SQLSTATE 42883)", - code: "42883", - }), - ).toBe(true); - expect( - legacyIsPgNetUnavailableError({ - message: "ERROR: function public.http_post(unknown) does not exist (SQLSTATE 42883)", - code: "42883", - }), - ).toBe(false); - }); - it.effect("suggests enabling Database Webhooks when local replay cannot find pg_net", () => { const workdir = makeWorkdir(); setupMigration(workdir); @@ -546,31 +527,4 @@ describe("legacyMigrateAndSeed local pg_net remediation", () => { ), ); }); - - it.effect("requires the matching SQLSTATE instead of classifying by message alone", () => { - const workdir = makeWorkdir(); - setupMigration(workdir); - const out = mockOutput(); - const wrongSqlState = new LegacyDbExecError({ - message: 'ERROR: schema "net" does not exist (SQLSTATE 42P01)', - code: "42P01", - }); - return run( - workdir, - "", - { ...baseConfig, localDatabaseWebhooksEnabled: false }, - pgNetFailureSession(wrongSqlState), - out, - ).pipe( - Effect.flip, - Effect.tap((error) => - Effect.sync(() => { - assertMigrationApplyError(error); - expect(error.suggestion).toBeUndefined(); - expect(error[ErrorActionabilityId]).toEqual(actionability.dbFinding); - rmSync(workdir, { recursive: true, force: true }); - }), - ), - ); - }); }); diff --git a/apps/cli/src/legacy/shared/legacy-migration-apply.ts b/apps/cli/src/legacy/shared/legacy-migration-apply.ts index e2367e6ef3..42c507ea25 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-apply.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-apply.ts @@ -598,13 +598,8 @@ const execMigrationBatch = ( const version = forceNoVersion ? "" : (matches?.[1] ?? ""); const name = matches?.[2] ?? ""; - // The pg-delta directive is file-level execution metadata. Run the complete - // sequence on this session without adding transaction boundaries so session - // settings remain active for the nontransactional action. History is recorded - // only after every statement succeeds. A failed sequence gets a best-effort - // session reset because the generated trailing RESET ALL may not have run yet. - if (transactionMode === "none") { - const nonTransactional = Effect.gen(function* () { + const executeSequentially = (cleanup: string) => + Effect.gen(function* () { for (const [index, statement] of statements.entries()) { yield* session .exec(statement) @@ -621,37 +616,22 @@ const execMigrationBatch = ( ), ); } - }); - return yield* nonTransactional.pipe( - Effect.tapError(() => session.exec("RESET ALL").pipe(Effect.ignore)), - ); + }).pipe(Effect.tapError(() => session.exec(cleanup).pipe(Effect.ignore))); + + // The pg-delta directive is file-level execution metadata. Run the complete + // sequence on this session without adding transaction boundaries so session + // settings remain active for the nontransactional action. History is recorded + // only after every statement succeeds. A failed sequence gets a best-effort + // session reset because the generated trailing RESET ALL may not have run yet. + if (transactionMode === "none") { + return yield* executeSequentially("RESET ALL"); } // A headerless file with authored transaction boundaries owns those semantics. // Execute the statements exactly as written, clean up a failed authored // transaction, and only send the history insert after every statement succeeds. if (statements.some(legacyHasTransactionControl)) { - const authored = Effect.gen(function* () { - for (const [index, statement] of statements.entries()) { - yield* session - .exec(statement) - .pipe( - Effect.mapError((cause) => legacyFormatExecBatchError(cause, index, statement)), - ); - } - if (version.length > 0) { - yield* session - .query(INSERT_MIGRATION_VERSION, [version, name, statements]) - .pipe( - Effect.mapError((cause) => - legacyFormatExecBatchError(cause, statements.length, INSERT_MIGRATION_VERSION), - ), - ); - } - }); - return yield* authored.pipe( - Effect.tapError(() => session.exec("ROLLBACK").pipe(Effect.ignore)), - ); + return yield* executeSequentially("ROLLBACK"); } // `executed` is the global statement index of the next statement to run, so the diff --git a/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts b/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts index 7897961665..1901e7e977 100644 --- a/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-migration-file.unit.test.ts @@ -45,17 +45,6 @@ describe("legacyParseMigrationContent", () => { transactionMode: "transactional", }); }); - - it.each([ - "-- pg-delta: transaction=true\nSELECT 1;", - " -- pg-delta: transaction=false\nSELECT 1;", - "-- pg-delta: transaction=none\nSELECT 1;", - ])("leaves a malformed first-line marker transactional: %s", (content) => { - expect(legacyParseMigrationContent(content)).toEqual({ - statements: [content.trim().slice(0, -1)], - transactionMode: "transactional", - }); - }); }); describe("legacyFormatMigrationTimestamp", () => { diff --git a/apps/cli/src/legacy/shared/legacy-pg-net-guidance.unit.test.ts b/apps/cli/src/legacy/shared/legacy-pg-net-guidance.unit.test.ts index dddb5f290a..53769ab825 100644 --- a/apps/cli/src/legacy/shared/legacy-pg-net-guidance.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-pg-net-guidance.unit.test.ts @@ -36,8 +36,6 @@ describe("legacyIsPgNetUnavailableError", () => { describe("legacyStatementInstallsPgNet", () => { it.each([ "create extension pg_net", - "CREATE EXTENSION pg_net", - "create extension if not exists pg_net schema extensions", 'CREATE EXTENSION IF NOT EXISTS "pg_net" WITH SCHEMA extensions', "create\n extension if not exists\n pg_net\n with schema extensions", ])("treats %j as a pg_net install", (statement) => { @@ -45,10 +43,8 @@ describe("legacyStatementInstallsPgNet", () => { }); it.each([ - "create table public.items (id int)", "create extension pgcrypto", "drop extension if exists pg_net", - "select net.http_post(url := 'https://example.com')", "comment on extension pgcrypto is 'pg_net is not installed here'", ])("does not treat %j as a pg_net install", (statement) => { expect(legacyStatementInstallsPgNet(statement)).toBe(false); diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md deleted file mode 100644 index 4bb369e01c..0000000000 --- a/docs/roadmap/pg-delta-next-follow-ups.md +++ /dev/null @@ -1,29 +0,0 @@ -# pg-delta next: deferred follow-ups - -Findings from the pg-delta-next bundling review (PR #6102) that were triaged as -**explicitly deferred** — understood, judged not worth acting on for this change, and -recorded here so a later reviewer does not have to rediscover them. - -- **The next-engine declarative writer follows symlinked managed directories on write.** - `apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts`'s next write loop - (`writeNextDeclarativeSchemas`) resolves each proposed file's path and writes it with no - containment check, so a symlinked subdirectory under the declarative dir is written - through, while the read path (`readManagedDeclarativeSqlFiles` in the same file) skips - symlinks outright. Asymmetric, but every path written comes from pg-delta's own export - names (already validated by `safeDeclarativeExportName`), so reaching outside the tree - needs the user to have planted the symlink themselves. Deferred — not needed now. - -- **`generate --output` containment is a lexical guard.** - `apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts` - rejects an output dir that resolves to or contains the project dir using `path.resolve` - + `path.relative` only, so an ancestor symlink can make an escaping path look contained. - The catastrophic outcome (a recursive wipe of the resolved dir) only exists on the legacy - full-wipe writer; the next writer never removes anything it does not own. Deferred. - -- **Shadow host-port allocation probes the wrong host.** - The shadow-database port allocator probes `127.0.0.1` inside the CLI process but the - container publishes on the Docker *daemon's* host, so with a remote `DOCKER_HOST` the - probe can report a free port that is taken remotely (or vice versa). Pre-existing latent - pattern, copied verbatim from - `apps/cli/src/legacy/shared/legacy-edge-runtime-script.layer.ts`; not introduced by this - PR and unreachable for the local-Docker default. Deferred. From c3602313b05037e357f03695e9a532e7be115c81 Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 14 Aug 2026 19:25:41 +0200 Subject: [PATCH 30/82] fix(cli): preserve declarative sync context --- .../db/declarative/declarative_test.go | 2 +- apps/cli-go/internal/utils/misc.go | 2 +- apps/cli-go/pkg/config/templates/config.toml | 2 +- apps/cli/docs/templates/examples.yaml | 2 +- .../legacy/commands/db/diff/diff.command.ts | 2 +- .../commands/db/diff/diff.integration.test.ts | 16 ++-- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 4 +- .../legacy/commands/db/pull/pull.handler.ts | 2 +- .../commands/db/pull/pull.integration.test.ts | 16 ++-- .../db/schema/declarative/declarative.flow.ts | 58 +++++++++++-- .../declarative/declarative.flow.unit.test.ts | 35 +++++++- ...eclarative.orchestrate.integration.test.ts | 1 + .../declarative/declarative.orchestrate.ts | 15 +++- .../declarative/generate/SIDE_EFFECTS.md | 12 +-- .../declarative/generate/generate.handler.ts | 1 + .../generate/generate.integration.test.ts | 30 +++---- .../schema/declarative/sync/SIDE_EFFECTS.md | 22 ++--- .../schema/declarative/sync/sync.handler.ts | 28 +++++-- .../declarative/sync/sync.integration.test.ts | 83 +++++++++++++++---- .../shared/legacy-pgdelta-next.live.test.ts | 2 +- .../db/shared/legacy-pgdelta.apply.ts | 4 +- .../legacy-pgdelta.seam.integration.test.ts | 2 +- .../db/shared/legacy-pgdelta.write.ts | 2 +- .../shared/legacy-shadow-source.unit.test.ts | 29 +++++-- .../shared/legacy-db-config.toml-read.ts | 6 +- .../legacy-db-config.toml-read.unit.test.ts | 4 +- .../src/shared/init/project-init.templates.ts | 2 +- apps/docs/public/cli/config.schema.json | 2 +- packages/config/src/experimental.ts | 2 +- 29 files changed, 270 insertions(+), 118 deletions(-) diff --git a/apps/cli-go/internal/db/declarative/declarative_test.go b/apps/cli-go/internal/db/declarative/declarative_test.go index cbd67d29de..093fa6197a 100644 --- a/apps/cli-go/internal/db/declarative/declarative_test.go +++ b/apps/cli-go/internal/db/declarative/declarative_test.go @@ -45,7 +45,7 @@ func TestWriteDeclarativeSchemas(t *testing.T) { cfg, err := afero.ReadFile(fsys, utils.ConfigPath) require.NoError(t, err) - assert.Contains(t, string(cfg), `"database"`) + assert.Contains(t, string(cfg), `"schemas"`) } func TestWriteDeclarativeSchemasSkipsConfigUpdateWhenPgDeltaEnabled(t *testing.T) { diff --git a/apps/cli-go/internal/utils/misc.go b/apps/cli-go/internal/utils/misc.go index 3faf385966..8805544430 100644 --- a/apps/cli-go/internal/utils/misc.go +++ b/apps/cli-go/internal/utils/misc.go @@ -99,7 +99,7 @@ var ( CurrBranchPath = filepath.Join(SupabaseDirPath, ".branches", "_current_branch") // DeclarativeDir is the canonical location for pg-delta declarative schema // files generated or synced by `supabase db schema declarative` commands. - DeclarativeDir = filepath.Join(SupabaseDirPath, "database") + DeclarativeDir = filepath.Join(SupabaseDirPath, "schemas") ClusterDir = filepath.Join(SupabaseDirPath, "cluster") SchemasDir = filepath.Join(SupabaseDirPath, "schemas") MigrationsDir = filepath.Join(SupabaseDirPath, "migrations") diff --git a/apps/cli-go/pkg/config/templates/config.toml b/apps/cli-go/pkg/config/templates/config.toml index 98e034f8d8..fe820ae14b 100644 --- a/apps/cli-go/pkg/config/templates/config.toml +++ b/apps/cli-go/pkg/config/templates/config.toml @@ -409,6 +409,6 @@ s3_secret_key = "env(S3_SECRET_KEY)" [experimental.pgdelta] enabled = {{ .Experimental.PgDeltaInitEnabled }} # Directory under `supabase/` where declarative files are written. -# declarative_schema_path = "./database" +# declarative_schema_path = "./schemas" # JSON string passed through to pg-delta SQL formatting. # format_options = "{\"keywordCase\":\"upper\",\"indent\":2,\"maxWidth\":80,\"commaStyle\":\"trailing\"}" diff --git a/apps/cli/docs/templates/examples.yaml b/apps/cli/docs/templates/examples.yaml index 464f9d30d5..bc451edb14 100644 --- a/apps/cli/docs/templates/examples.yaml +++ b/apps/cli/docs/templates/examples.yaml @@ -310,7 +310,7 @@ supabase-db-schema-declarative-sync: Reset local database to match migrations first? (local data will be lost) [y/N] y Resetting database... ... - Declarative schema written to supabase/database + Declarative schema written to supabase/schemas Finished supabase db schema declarative generate. supabase-test-db: - id: basic-usage diff --git a/apps/cli/src/legacy/commands/db/diff/diff.command.ts b/apps/cli/src/legacy/commands/db/diff/diff.command.ts index 0362a256ce..8d7035652a 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.command.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.command.ts @@ -99,7 +99,7 @@ export type LegacyDbDiffFlags = CliCommand.Command.Config.Infer; export const legacyDbDiffCommand = Command.make("diff", config).pipe( Command.withDescription( - "Compares a shadow built from supabase/migrations with a live database (--local by default, --linked, or --db-url). Declarative files under supabase/database are not part of this baseline. Output is printed by default; -f names and saves the complete diff as a migration and does not filter objects.", + "Compares a shadow built from supabase/migrations with a live database (--local by default, --linked, or --db-url). Declarative files under supabase/schemas are not part of this baseline. Output is printed by default; -f names and saves the complete diff as a migration and does not filter objects.", ), Command.withShortDescription("Diffs the local database for schema changes"), Command.withHandler((flags) => diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index c17ea1bc58..e1879d844c 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -588,7 +588,7 @@ describe("legacy db diff", () => { }); it.effect("next local diff ignores schema_paths and declarative files", () => { - mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + mkdirSync(join(tmp.current, "supabase", "schemas"), { recursive: true }); writeFileSync( join(tmp.current, "supabase", "config.toml"), [ @@ -602,7 +602,7 @@ describe("legacy db diff", () => { ); writeFileSync(join(tmp.current, "supabase", "configured.sql"), "create table configured ();\n"); writeFileSync( - join(tmp.current, "supabase", "database", "ignored.sql"), + join(tmp.current, "supabase", "schemas", "ignored.sql"), "create table ignored ();\n", ); const s = setup(tmp.current, { @@ -1376,12 +1376,12 @@ describe("legacy db diff", () => { }); it.effect("writes live-only SQL with --file even when declarative targets are configured", () => { - mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + mkdirSync(join(tmp.current, "supabase", "schemas"), { recursive: true }); writeFileSync( join(tmp.current, "supabase", "config.toml"), [ "[db.migrations]", - 'schema_paths = ["database/*.sql"]', + 'schema_paths = ["schemas/*.sql"]', "", "[experimental.pgdelta]", "enabled = true", @@ -1389,7 +1389,7 @@ describe("legacy db diff", () => { ].join("\n"), ); writeFileSync( - join(tmp.current, "supabase", "database", "declarative.sql"), + join(tmp.current, "supabase", "schemas", "declarative.sql"), "create table declarative_only ();\n", ); const s = setup(tmp.current, { @@ -1412,9 +1412,9 @@ describe("legacy db diff", () => { }); it.effect("includes the ignored declarative baseline advisory in JSON output", () => { - mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); + mkdirSync(join(tmp.current, "supabase", "schemas"), { recursive: true }); writeFileSync( - join(tmp.current, "supabase", "database", "items.sql"), + join(tmp.current, "supabase", "schemas", "items.sql"), "create table items ();\n", ); const s = setup(tmp.current, { @@ -1436,7 +1436,7 @@ describe("legacy db diff", () => { severity: "info", context: { baseline: "supabase/migrations", - declarativePath: "supabase/database", + declarativePath: "supabase/schemas", fileFlagFiltersObjects: false, }, }, diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 40c7f82404..b5574119d8 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -52,8 +52,8 @@ disables formatting without disabling safe compaction. | Path | Format | When | | ---------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | -| `/supabase/database/**` | SQL | `--declarative` | -| `/supabase/database/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | +| `/supabase/schemas/**` | SQL | `--declarative` | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | | `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | | `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 96c3f68e72..a90be573d9 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -567,7 +567,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy ); } // Prints the config's declarative_schema_path or the relative - // `supabase/database` default — never the resolved absolute directory + // `supabase/schemas` default — never the resolved absolute directory // (established output contract). The json payload below keeps the // absolute path for machine consumers. yield* output.raw( diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index efe97ee858..644cfd12a3 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -856,14 +856,14 @@ describe("legacy db pull", () => { ); // Prints the relative default, not the resolved absolute directory // (established output contract). - expect(err).toContain(`Declarative schema written to ${join("supabase", "database")}\n`); + expect(err).toContain(`Declarative schema written to ${join("supabase", "schemas")}\n`); expect(err).not.toContain(tmp.current); expect( - existsSync(join(tmp.current, "supabase", "database", "schemas", "public", "t.sql")), + existsSync(join(tmp.current, "supabase", "schemas", "schemas", "public", "t.sql")), ).toBe(true); expect( JSON.parse( - readFileSync(join(tmp.current, "supabase", "database", ".pgdelta-export.json"), "utf8"), + readFileSync(join(tmp.current, "supabase", "schemas", ".pgdelta-export.json"), "utf8"), ), ).toMatchObject({ formatVersion: 1, @@ -902,7 +902,7 @@ describe("legacy db pull", () => { yield* legacyDbPull(flags({ declarative: Option.some(true) })); const config = readFileSync(join(tmp.current, "supabase", "config.toml"), "utf8"); expect(config).toContain("[db.migrations]"); - expect(config).toContain('schema_paths = [\n "database",\n]'); + expect(config).toContain('schema_paths = [\n "schemas",\n]'); }).pipe(Effect.provide(s.layer)); }, ); @@ -933,7 +933,7 @@ describe("legacy db pull", () => { return Effect.gen(function* () { yield* legacyDbPull(flags({ declarative: Option.some(true) })); const config = readFileSync(join(tmp.current, "supabase", "config.toml"), "utf8"); - expect(config).toContain('schema_paths = [\n "database",\n]'); + expect(config).toContain('schema_paths = [\n "schemas",\n]'); expect(config).not.toContain("schemas/*.sql"); }).pipe(Effect.provide(s.layer)); }); @@ -946,7 +946,7 @@ describe("legacy db pull", () => { yield* legacyDbPull(flags({ usePgDelta: Option.some(true) })); expect(streamText(s.out, "stderr")).toContain("Flag --use-pg-delta has been deprecated"); expect(streamText(s.out, "stderr")).toContain( - `Declarative schema written to ${join("supabase", "database")}\n`, + `Declarative schema written to ${join("supabase", "schemas")}\n`, ); }).pipe(Effect.provide(s.layer)); }, @@ -1049,7 +1049,7 @@ describe("legacy db pull", () => { // Reaching the declarative write (rather than a migration file / history // upsert) proves the declarative export path ran. expect( - existsSync(join(tmp.current, "supabase", "database", "schemas", "public", "t.sql")), + existsSync(join(tmp.current, "supabase", "schemas", "schemas", "public", "t.sql")), ).toBe(true); expect(s.historyUpserts.length).toBe(0); }).pipe(Effect.provide(s.layer)); @@ -2193,7 +2193,7 @@ describe("legacy db pull", () => { expect(streamText(s.out, "stderr")).toContain("Retrying via the IPv4 connection pooler"); expect(s.engineCalls.filter((call) => call.operation === "export")).toHaveLength(2); expect(streamText(s.out, "stderr")).toContain( - `Declarative schema written to ${join("supabase", "database")}\n`, + `Declarative schema written to ${join("supabase", "schemas")}\n`, ); expect(s.shadowSpawned.filter((c) => c.args[0] === "create")).toHaveLength(1); }).pipe(Effect.provide(s.layer)); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts index f150a17751..0f69a18c17 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts @@ -1,4 +1,5 @@ import type { LegacyPgDeltaImplementation } from "../../../../shared/legacy-pgdelta-next-flag.ts"; +import { legacySchemaToCsvField } from "../../../../shared/legacy-schema-flags.ts"; import type { LegacyPgDeltaRemovalSummary } from "../../shared/legacy-pgdelta-engine.service.ts"; /** Extensions that legacy pg-delta treated as part of its implicit Supabase baseline. */ @@ -257,17 +258,56 @@ export function legacyClassifyDeclarativeLoadCompatibility(opts: { export const legacyExtensionDeclaration = (extension: string): string => `CREATE EXTENSION IF NOT EXISTS "${extension}" WITH SCHEMA "extensions";`; -export const legacyNextExportAdoptionCommands = [ - " supabase db schema declarative generate --local --overwrite \\", - " --output supabase/database-next --experimental", - "", - " # review supabase/database-next", - " rm -rf supabase/database && mv supabase/database-next supabase/database", - " supabase db schema declarative sync --no-apply --experimental", -] as const; +export interface LegacyStagedExportContext { + readonly declarativeDir: string; + readonly schema: ReadonlyArray; +} + +export const legacyResolveStagedDeclarativeDir = (declarativeDir: string): string => + `${declarativeDir}-next`; + +function shellQuoteArgument(value: string): string { + return /^[a-zA-Z0-9_./:@%+=,-]+$/.test(value) ? value : `'${value.replaceAll("'", `'"'"'`)}'`; +} + +function schemaArguments(schema: ReadonlyArray): string { + return schema + .map((name) => ` --schema ${shellQuoteArgument(legacySchemaToCsvField(name))}`) + .join(""); +} + +export const legacyFormatDeclarativeSyncCommand = (schema: ReadonlyArray): string => + ` supabase db schema declarative sync --no-apply${schemaArguments(schema)} --experimental`; + +export function legacyFormatStagedExportAdoption({ + declarativeDir, + schema, +}: LegacyStagedExportContext): ReadonlyArray { + const stagedDir = legacyResolveStagedDeclarativeDir(declarativeDir); + return [ + `Review ${stagedDir}, then adopt it:`, + ` rm -rf ${shellQuoteArgument(declarativeDir)} && mv ${shellQuoteArgument(stagedDir)} ${shellQuoteArgument(declarativeDir)}`, + legacyFormatDeclarativeSyncCommand(schema), + ]; +} + +export function legacyFormatStagedExportCommands( + context: LegacyStagedExportContext, +): ReadonlyArray { + const stagedDir = legacyResolveStagedDeclarativeDir(context.declarativeDir); + return [ + " supabase db schema declarative generate --local --overwrite \\", + ` --output ${shellQuoteArgument(stagedDir)}${schemaArguments(context.schema)} --experimental`, + "", + ...legacyFormatStagedExportAdoption(context).map((line) => + line.startsWith("Review ") ? ` # review ${stagedDir}` : line, + ), + ]; +} export function legacyFormatStagedExportRecommendation( gap: LegacyDeclarativeCompatibilityGap, + context: LegacyStagedExportContext, ): string { const detected = [ ...(gap.repairableExtensions.length > 0 @@ -288,6 +328,6 @@ export function legacyFormatStagedExportRecommendation( "WARNING: pg-delta next manages schema state that the legacy export did not represent.", ...detected, "Generate a next-compatible schema into a separate directory, review it, and adopt it when ready:", - ...legacyNextExportAdoptionCommands, + ...legacyFormatStagedExportCommands(context), ].join("\n"); } diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts index c365b79ecb..a5d87513cb 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts @@ -97,8 +97,39 @@ describe("legacyClassifyDeclarativeCompatibilityGap", () => { 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";', ); const gap = classifyGap(); - expect(legacyFormatStagedExportRecommendation(gap)).toContain( - "generate --local --overwrite \\\n --output supabase/database-next --experimental", + expect( + legacyFormatStagedExportRecommendation(gap, { + declarativeDir: "supabase/schemas", + schema: [], + }), + ).toContain( + "generate --local --overwrite \\\n --output supabase/schemas-next --experimental", + ); + }); + + it("derives staged-export commands from a custom declarative path", () => { + const recommendation = legacyFormatStagedExportRecommendation(classifyGap(), { + declarativeDir: "supabase/custom schema", + schema: [], + }); + + expect(recommendation).toContain("--output 'supabase/custom schema-next'"); + expect(recommendation).toContain( + "rm -rf 'supabase/custom schema' && mv 'supabase/custom schema-next' 'supabase/custom schema'", + ); + }); + + it("preserves schema filters in staged-export and follow-up sync commands", () => { + const recommendation = legacyFormatStagedExportRecommendation(classifyGap(), { + declarativeDir: "supabase/schemas", + schema: ["app", "tenant,one"], + }); + + expect(recommendation).toContain( + `--output supabase/schemas-next --schema app --schema '"tenant,one"' --experimental`, + ); + expect(recommendation).toContain( + `sync --no-apply --schema app --schema '"tenant,one"' --experimental`, ); }); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 8ac9215e74..47788d9346 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -158,6 +158,7 @@ const ctx = (cwd: string, declarativeDir: string): LegacyDeclarativeRunContext = }, formatOptions: "", declarativeDir, + declarativeDirDisplay: declarativeDir, schema: [], noCache: false, debug: false, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 271c0289c3..50740ffdcc 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -34,7 +34,8 @@ import { import { legacyClassifyDeclarativeLoadCompatibility, legacyExtensionDeclaration, - legacyNextExportAdoptionCommands, + legacyFormatDeclarativeSyncCommand, + legacyFormatStagedExportCommands, type LegacyDeclarativeLoadCompatibilityFinding, } from "./declarative.flow.ts"; @@ -43,6 +44,8 @@ export interface LegacyDeclarativeRunContext { readonly pgDelta: LegacyPgDeltaContext; readonly formatOptions: string; readonly declarativeDir: string; + /** User-facing configured/output path, kept separate from the absolute I/O path. */ + readonly declarativeDirDisplay: string; readonly schema: ReadonlyArray; readonly noCache: boolean; readonly debug: boolean; @@ -66,6 +69,7 @@ const declarativeError = (message: string) => new LegacyDeclarativeDiffError({ m const formatImplicitExtensionLoadFailure = ( findings: ReadonlyArray, + run: Pick, ): string => { const extensions = [...new Set(findings.map((finding) => finding.extension))].sort(); const detected = findings.map((finding) => { @@ -84,12 +88,15 @@ const formatImplicitExtensionLoadFailure = ( "", "Recommended — generate a next-compatible tree, review it, then adopt:", "", - ...legacyNextExportAdoptionCommands, + ...legacyFormatStagedExportCommands({ + declarativeDir: run.declarativeDirDisplay, + schema: run.schema, + }), "", "Alternative — add the missing extension declarations to extension.sql, then re-plan:", ...extensions.map((extension) => legacyExtensionDeclaration(extension)), "", - " supabase db schema declarative sync --no-apply --experimental", + legacyFormatDeclarativeSyncCommand(run.schema), ].join("\n"); }; @@ -158,7 +165,7 @@ export const legacyDiffDeclarativeToMigrations = Effect.fnUntraced(function* ( return findings.length === 0 ? error : new LegacyDeclarativeCompatibilityError({ - message: formatImplicitExtensionLoadFailure(findings), + message: formatImplicitExtensionLoadFailure(findings, run), loadFindings: findings, }); }), diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index ae03103514..b78dc090aa 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -25,12 +25,12 @@ formatting without disabling safe compaction. ## Files Written -| Path | Format | When | -| --------------------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------ | -| `/supabase/database/**/*.sql` (configured declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | -| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| ----------------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------ | +| `/supabase/schemas/**/*.sql` (default declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | +| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts index 240e74c1f4..7c6bde1364 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts @@ -176,6 +176,7 @@ export const legacyDbSchemaDeclarativeGenerate = Effect.fn("legacy.db.schema.dec }, formatOptions: Option.getOrElse(toml.pgDelta.formatOptions, () => ""), declarativeDir, + declarativeDirDisplay: declarativeDirRel, schema: flags.schema, noCache: flags.noCache, debug: legacyIsPgDeltaDebugEnabled(), diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts index 0ccd779869..0fd87ba61f 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts @@ -472,7 +472,7 @@ describe("legacy db schema declarative generate integration", () => { ); const written = yield* Effect.promise(async () => (await import("node:fs")).readFileSync( - join(tmp.current, "supabase", "database", "schemas", "public", "tables", "players.sql"), + join(tmp.current, "supabase", "schemas", "schemas", "public", "tables", "players.sql"), "utf8", ), ); @@ -482,7 +482,7 @@ describe("legacy db schema declarative generate integration", () => { expect( s.out.rawChunks.map((c) => ({ text: stripAnsi(c.text), stream: c.stream })), ).toContainEqual({ - text: `Declarative schema written to ${join("supabase", "database")}\n`, + text: `Declarative schema written to ${join("supabase", "schemas")}\n`, stream: "stderr", }); expect(s.out.rawChunks.some((c) => c.text.includes(tmp.current))).toBe(false); @@ -592,7 +592,7 @@ describe("legacy db schema declarative generate integration", () => { join(tmp.current, "staged-schema", "schemas", "public", "tables", "players.sql"), ), ).toBe(true); - expect(existsSync(join(tmp.current, "supabase", "database"))).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "schemas"))).toBe(false); }).pipe(Effect.provide(s.layer)); }); @@ -618,14 +618,14 @@ describe("legacy db schema declarative generate integration", () => { // Go's confirmOverwrite returns true immediately (Console.PromptYesNo); the // handler must skip the prompt and overwrite. No promptConfirmResponses are // queued, so reaching the prompt would error — success proves --yes bypassed it. - mkdirSync(join(tmp.current, "supabase", "database"), { recursive: true }); - writeFileSync(join(tmp.current, "supabase", "database", "existing.sql"), "create table x ();"); + mkdirSync(join(tmp.current, "supabase", "schemas"), { recursive: true }); + writeFileSync(join(tmp.current, "supabase", "schemas", "existing.sql"), "create table x ();"); const s = setup(tmp.current, { experimental: true, yes: true }); return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeGenerate(flags({ local: Option.some(true) })); const written = yield* Effect.promise(async () => (await import("node:fs")).readFileSync( - join(tmp.current, "supabase", "database", "schemas", "public", "tables", "players.sql"), + join(tmp.current, "supabase", "schemas", "schemas", "public", "tables", "players.sql"), "utf8", ), ); @@ -637,10 +637,10 @@ describe("legacy db schema declarative generate integration", () => { // Go's confirmOverwrite returns the ReadDir error and Generate aborts on it // (declarative.go:123-127, 226-229), rather than treating an unreadable existing // dir as empty and letting WriteDeclarativeSchemas wipe/recreate the path. - // Seeding supabase/database as a FILE makes readDirectory fail with ENOTDIR (a + // Seeding supabase/schemas as a FILE makes readDirectory fail with ENOTDIR (a // non-NotFound PlatformError), so the command must fail without writing. mkdirSync(join(tmp.current, "supabase"), { recursive: true }); - writeFileSync(join(tmp.current, "supabase", "database"), "not a directory"); + writeFileSync(join(tmp.current, "supabase", "schemas"), "not a directory"); const s = setup(tmp.current, { experimental: true }); return Effect.gen(function* () { const exit = yield* Effect.exit( @@ -649,7 +649,7 @@ describe("legacy db schema declarative generate integration", () => { expect(Exit.isFailure(exit)).toBe(true); // The declarative path is untouched — still our seeded file, never wiped and // rewritten as a directory of schema files. - expect(readFileSync(join(tmp.current, "supabase", "database"), "utf8")).toBe( + expect(readFileSync(join(tmp.current, "supabase", "schemas"), "utf8")).toBe( "not a directory", ); expect(s.out.rawChunks.some((c) => c.text.includes("Declarative schema written to"))).toBe( @@ -830,7 +830,7 @@ describe("legacy db schema declarative generate integration", () => { }); it.effect("smart mode: existing files + decline regenerate → skips", () => { - const declDir = join(tmp.current, "supabase", "database"); + const declDir = join(tmp.current, "supabase", "schemas"); mkdirSync(declDir, { recursive: true }); writeFileSync(join(declDir, "existing.sql"), "-- existing"); const s = setup(tmp.current, { @@ -852,7 +852,7 @@ describe("legacy db schema declarative generate integration", () => { // under --yes, so existing declarative files are regenerated (not skipped) and // no prompt is shown. No migrations → the smart target resolves to local without // a further prompt. No promptConfirmResponses are queued, so a prompt would throw. - const declDir = join(tmp.current, "supabase", "database"); + const declDir = join(tmp.current, "supabase", "schemas"); mkdirSync(declDir, { recursive: true }); writeFileSync(join(declDir, "existing.sql"), "-- existing"); const s = setup(tmp.current, { experimental: true, stdinIsTty: false, yes: true }); @@ -863,7 +863,7 @@ describe("legacy db schema declarative generate integration", () => { // global YES flag (`console.go:70-72`) — the echo must not be skipped, and // the prompt renders the relative dir (`db_schema_declarative.go:268`). expect(stripAnsi(s.out.stderrText)).toContain( - `Declarative schema already exists at ${join("supabase", "database")}. Regenerate from database? This will overwrite existing files. [y/N] y\n`, + `Declarative schema already exists at ${join("supabase", "schemas")}. Regenerate from database? This will overwrite existing files. [y/N] y\n`, ); expect( s.out.rawChunks.some((c) => c.text.includes("Skipped generating declarative schema")), @@ -874,7 +874,7 @@ describe("legacy db schema declarative generate integration", () => { it.effect("smart mode: SUPABASE_YES=1 regenerates over existing files like --yes", () => { // Go reads `viper.GetBool("YES")`, which `AutomaticEnv` also binds to the // SUPABASE_YES env var — the flag alone is not the whole surface (CLI-1974). - const declDir = join(tmp.current, "supabase", "database"); + const declDir = join(tmp.current, "supabase", "schemas"); mkdirSync(declDir, { recursive: true }); writeFileSync(join(declDir, "existing.sql"), "-- existing"); const prev = process.env["SUPABASE_YES"]; @@ -884,7 +884,7 @@ describe("legacy db schema declarative generate integration", () => { yield* legacyDbSchemaDeclarativeGenerate(flags()); expect(s.seamCalls).toEqual(["declarative"]); expect(stripAnsi(s.out.stderrText)).toContain( - `Declarative schema already exists at ${join("supabase", "database")}. Regenerate from database? This will overwrite existing files. [y/N] y\n`, + `Declarative schema already exists at ${join("supabase", "schemas")}. Regenerate from database? This will overwrite existing files. [y/N] y\n`, ); }).pipe( Effect.ensuring( @@ -1187,7 +1187,7 @@ describe("legacy db schema declarative generate integration", () => { return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeGenerate(flags({ local: Option.some(true) })); const manifest = JSON.parse( - readFileSync(join(tmp.current, "supabase", "database", ".pgdelta-export.json"), "utf8"), + readFileSync(join(tmp.current, "supabase", "schemas", ".pgdelta-export.json"), "utf8"), ); expect(manifest).toMatchObject({ formatVersion: 1, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 616ab7fae2..5b98756199 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -15,23 +15,23 @@ disabling safe compaction. ## Files Read -| Path | Format | When | -| -------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------ | -| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | -| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | -| `/supabase/database/**/*.sql` (declarative dir) | SQL | always — must exist (else error) | -| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | -| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | -| `/supabase/database/.pgdelta-export.json` | JSON | bundled export metadata, when present | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | +| Path | Format | When | +| --------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------ | +| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | +| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | +| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | ## Files Written | Path | Format | When | | ------------------------------------------------------------------ | ------ | ------------------------------------------------- | | `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | -| `/supabase/database/extension.sql` | SQL | accepted legacy-extension repair | +| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | | `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 540a5f9405..250111f75a 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -57,7 +57,9 @@ import { import { legacyClassifyDeclarativeCompatibilityGap, legacyExtensionDeclaration, + legacyFormatStagedExportAdoption, legacyFormatStagedExportRecommendation, + legacyResolveStagedDeclarativeDir, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, } from "../declarative.flow.ts"; @@ -150,7 +152,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara } // Go's `utils.GetDeclarativeDir()` — the config value verbatim (already - // `supabase/`-prefixed when relative) or the relative `supabase/database` + // `supabase/`-prefixed when relative) or the relative `supabase/schemas` // default. Printed verbatim in the bootstrap's written-to line below, exactly // as Go prints it (Go chdirs into the workdir, so its paths stay relative). const declarativeDirRel = legacyResolveDeclarativeDir(path, toml.pgDelta); @@ -158,6 +160,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara // used as-is, matching Go's `config.resolve` (which only prefixes the workdir onto // a relative path). `path.join(workdir, abs)` would mangle the absolute path. const declarativeDir = path.resolve(cliConfig.workdir, declarativeDirRel); + const stagedDirRel = legacyResolveStagedDeclarativeDir(declarativeDirRel); const migrationsDir = path.join(cliConfig.workdir, "supabase", "migrations"); const tempDir = legacyPgDeltaTempPath(path, cliConfig.workdir); const run: LegacyDeclarativeRunContext = { @@ -175,6 +178,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara }, formatOptions: Option.getOrElse(toml.pgDelta.formatOptions, () => ""), declarativeDir, + declarativeDirDisplay: declarativeDirRel, schema: flags.schema, noCache: flags.noCache, debug: legacyIsPgDeltaDebugEnabled(), @@ -308,7 +312,6 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara toml.baseline, ); const stageNextExport = Effect.fnUntraced(function* () { - const stagedDirRel = "supabase/database-next"; const stagedDir = path.resolve(cliConfig.workdir, stagedDirRel); if (stagedDir === declarativeDir) { return yield* Effect.fail( @@ -343,9 +346,10 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara yield* output.raw(legacyDeclarativeSchemaWrittenLine(stagedDirRel), "stderr"); yield* output.raw( [ - "Review supabase/database-next, then adopt it:", - " rm -rf supabase/database && mv supabase/database-next supabase/database", - " supabase db schema declarative sync --no-apply --experimental", + ...legacyFormatStagedExportAdoption({ + declarativeDir: declarativeDirRel, + schema: flags.schema, + }), "", ].join("\n"), "stderr", @@ -420,7 +424,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara const choice = yield* output.promptSelect("How would you like to continue?", [ { value: "stage", - label: "Generate next export to supabase/database-next", + label: `Generate next export to ${stagedDirRel}`, hint: "recommended", }, { value: "repair", label: "Add missing extension declarations and re-plan" }, @@ -474,7 +478,10 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara if (compatibility.recommendedAction === "none") break; if (compatibility.recommendedAction === "stage-next-export") { - const explanation = legacyFormatStagedExportRecommendation(compatibility); + const explanation = legacyFormatStagedExportRecommendation(compatibility, { + declarativeDir: declarativeDirRel, + schema: flags.schema, + }); if (!tty.stdinIsTty || yes) { return yield* Effect.fail( new LegacyDeclarativeCompatibilityError({ message: explanation }), @@ -484,7 +491,7 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara const choice = yield* output.promptSelect("How would you like to continue?", [ { value: "stage", - label: "Generate next export to supabase/database-next", + label: `Generate next export to ${stagedDirRel}`, hint: "recommended", }, { value: "cancel", label: "Cancel" }, @@ -508,7 +515,10 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara "Non-interactive sync will not modify the declarative schema automatically. Add these statements to extension.sql, then run sync again:", ...statements, "", - legacyFormatStagedExportRecommendation(compatibility), + legacyFormatStagedExportRecommendation(compatibility, { + declarativeDir: declarativeDirRel, + schema: flags.schema, + }), ].join("\n"), }), ); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 7d4a184f55..cb2b6cd49b 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -269,7 +269,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { const nextFiles = opts.renderedFiles ?? []; const planErrors = [...(opts.planErrors ?? [])]; let planCalls = 0; - const declarativeExportCalls: Array = []; + const declarativeExportCalls: Array> = []; const engine = opts.engineImplementation === "next" ? Layer.succeed( @@ -278,9 +278,9 @@ function setup(workdir: string, opts: SetupOpts = {}) { implementation: "next", diffExplicit: () => Effect.die("diffExplicit not used in sync tests"), diffDatabase: () => Effect.die("diffDatabase not used in sync tests"), - exportDeclarativeSchema: () => + exportDeclarativeSchema: (input) => Effect.sync(() => { - declarativeExportCalls.push(true); + declarativeExportCalls.push(input.schema); return { files: [ { name: "schemas/public/tables/players.sql", sql: "create table players ();" }, @@ -292,7 +292,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { planCalls += 1; const planError = planErrors.shift(); if (planError !== undefined) return Effect.fail(planError); - const extensionPath = join(workdir, "supabase", "database", "extension.sql"); + const extensionPath = join(workdir, "supabase", "schemas", "extension.sql"); const extensionSql = existsSync(extensionPath) ? readFileSync(extensionPath, "utf8") : ""; @@ -386,13 +386,13 @@ const failError = (exit: Exit.Exit) => Exit.isFailure(exit) ? exit.cause.reasons.find(Cause.isFailReason)?.error : undefined; const seedDeclarative = (workdir: string) => { - const dir = join(workdir, "supabase", "database"); + const dir = join(workdir, "supabase", "schemas"); mkdirSync(dir, { recursive: true }); writeFileSync(join(dir, "public.sql"), "create table a();"); }; -const seedLegacyUuidDeclarative = (workdir: string) => { - const dir = join(workdir, "supabase", "database"); +const seedLegacyUuidDeclarative = (workdir: string, directory = "schemas") => { + const dir = join(workdir, "supabase", directory); mkdirSync(join(dir, "schemas", "app", "tables"), { recursive: true }); mkdirSync(join(dir, "schemas", "public", "views"), { recursive: true }); writeFileSync( @@ -675,7 +675,7 @@ describe("legacy db schema declarative sync integration", () => { // `Declarative schema written to ` to stderr AFTER WriteDeclarativeSchemas // and the catalog warm (`declarative.go:133→138-155→156`), before sync's own // diff (step 2). It prints `utils.GetDeclarativeDir()` — the relative - // `supabase/database` default — never the absolute resolved dir (CLI-1980). + // `supabase/schemas` default — never the absolute resolved dir (CLI-1980). const s = setup(tmp.current, { experimental: true, stdinIsTty: true, @@ -685,7 +685,7 @@ describe("legacy db schema declarative sync integration", () => { }); return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); - const line = `Declarative schema written to ${join("supabase", "database")}\n`; + const line = `Declarative schema written to ${join("supabase", "schemas")}\n`; const written = s.out.rawChunks .map((c, index) => ({ text: stripAnsi(c.text), stream: c.stream, index })) .filter((c) => c.text === line); @@ -708,7 +708,7 @@ describe("legacy db schema declarative sync integration", () => { // The generated files actually landed in the printed (resolved) dir. expect( existsSync( - join(tmp.current, "supabase", "database", "schemas", "public", "tables", "players.sql"), + join(tmp.current, "supabase", "schemas", "schemas", "public", "tables", "players.sql"), ), ).toBe(true); }).pipe(Effect.provide(s.layer)); @@ -729,7 +729,7 @@ describe("legacy db schema declarative sync integration", () => { expect( s.out.rawChunks.map((c) => ({ text: stripAnsi(c.text), stream: c.stream })), ).toContainEqual({ - text: `Declarative schema written to ${join("supabase", "database")}\n`, + text: `Declarative schema written to ${join("supabase", "schemas")}\n`, stream: "stderr", }); }).pipe(Effect.provide(s.layer)); @@ -747,7 +747,7 @@ describe("legacy db schema declarative sync integration", () => { }); return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeSync(flags({ noCache: true, noApply: Option.some(true) })); - const line = `Declarative schema written to ${join("supabase", "database")}\n`; + const line = `Declarative schema written to ${join("supabase", "schemas")}\n`; const written = s.out.rawChunks .map((c, index) => ({ text: stripAnsi(c.text), stream: c.stream, index })) .filter((c) => c.text === line); @@ -1065,7 +1065,7 @@ describe("legacy db schema declarative sync integration", () => { return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); expect(s.planCalls).toBe(2); - expect(readFileSync(join(tmp.current, "supabase", "database", "extension.sql"), "utf8")).toBe( + expect(readFileSync(join(tmp.current, "supabase", "schemas", "extension.sql"), "utf8")).toBe( 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";\n', ); }).pipe(Effect.provide(s.layer)); @@ -1076,7 +1076,7 @@ describe("legacy db schema declarative sync integration", () => { const activeMember = join( tmp.current, "supabase", - "database", + "schemas", "schemas", "app", "tables", @@ -1097,7 +1097,7 @@ describe("legacy db schema declarative sync integration", () => { join( tmp.current, "supabase", - "database-next", + "schemas-next", "schemas", "public", "tables", @@ -1107,11 +1107,58 @@ describe("legacy db schema declarative sync integration", () => { ), ).toBe("create table players ();"); expect( - existsSync(join(tmp.current, "supabase", "database-next", ".pgdelta-export.json")), + existsSync(join(tmp.current, "supabase", "schemas-next", ".pgdelta-export.json")), ).toBe(true); }).pipe(Effect.provide(s.layer)); }); + it.effect("stages beside a custom active path and preserves --schema for adoption", () => { + seedLegacyUuidDeclarative(tmp.current, "custom-declarative"); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + [ + "[experimental.pgdelta]", + "enabled = true", + 'declarative_schema_path = "./custom-declarative"', + "", + ].join("\n"), + ); + const activeMember = join( + tmp.current, + "supabase", + "custom-declarative", + "schemas", + "app", + "tables", + "members.sql", + ); + const before = readFileSync(activeMember, "utf8"); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + planErrors: [legacyUuidLoadError()], + promptSelectResponses: ["stage"], + }); + + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ schema: ["app"], noApply: Option.some(true) })); + + expect(readFileSync(activeMember, "utf8")).toBe(before); + expect( + existsSync( + join(tmp.current, "supabase", "custom-declarative-next", ".pgdelta-export.json"), + ), + ).toBe(true); + expect(s.declarativeExportCalls).toEqual([["app"]]); + expect(stripAnsi(s.out.stderrText)).toContain( + "rm -rf supabase/custom-declarative && mv supabase/custom-declarative-next supabase/custom-declarative", + ); + expect(stripAnsi(s.out.stderrText)).toContain( + "supabase db schema declarative sync --no-apply --schema app --experimental", + ); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("refuses extension-managed legacy gaps under --yes instead of writing drops", () => { seedDeclarative(tmp.current); const s = setup(tmp.current, { @@ -1187,14 +1234,14 @@ describe("legacy db schema declarative sync integration", () => { return Effect.gen(function* () { yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); - expect(existsSync(join(tmp.current, "supabase", "database", "extension.sql"))).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "schemas", "extension.sql"))).toBe(false); }).pipe(Effect.provide(s.layer)); }); it.effect("suppresses the compatibility warning when a next export manifest is present", () => { seedDeclarative(tmp.current); writeFileSync( - join(tmp.current, "supabase", "database", ".pgdelta-export.json"), + join(tmp.current, "supabase", "schemas", ".pgdelta-export.json"), JSON.stringify({ formatVersion: 1, redactSecrets: true, scope: "database" }), ); const s = setup(tmp.current, { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts index 7e55df0c64..d6e5a33c77 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts @@ -56,7 +56,7 @@ describeDockerLive("pg-delta next local convergence (live)", () => { config .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') .replace( - '# declarative_schema_path = "./database"', + '# declarative_schema_path = "./schemas"', 'declarative_schema_path = "./schemas"', ), ); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.apply.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.apply.ts index f9092b4b57..da63c39645 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.apply.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.apply.ts @@ -1,6 +1,6 @@ /** * Port of Go's `pgdelta.ApplyDeclarative` (`apps/cli-go/internal/pgdelta/apply.go:303-354`) — - * CLI-1956's declarative-apply runner: applies `supabase/database` (or the configured + * CLI-1956's declarative-apply runner: applies `supabase/schemas` (or the configured * declarative dir) to the shadow's `contrib_regression` override database via pg-delta's * declarative apply engine, run inside the edge-runtime container. * @@ -884,7 +884,7 @@ export const legacyApplyDeclarativePgDelta = Effect.fnUntraced(function* ( readonly declarativeDirAbs: string; /** * Go's `utils.GetDeclarativeDir()` (`apply.go:304`) — the config value verbatim - * (already `supabase/`-prefixed when relative) or the relative `supabase/database` + * (already `supabase/`-prefixed when relative) or the relative `supabase/schemas` * default. Used ONLY in the not-found error message below: Go interpolates this * relative value, never the `filepath.Abs`-resolved `absDir` it separately computes * for the bind. diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.integration.test.ts index 84068758b3..9171739c15 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.integration.test.ts @@ -208,7 +208,7 @@ describe("legacyDeclarativeSeamLayer.exportCatalog", () => { "writes catalog-nocache-declarative.json on --no-cache, applying the declarative directory first", () => { const dir = mkdtempSync(join(tmpdir(), "legacy-pgdelta-seam-")); - const declDir = join(dir, "supabase", "database"); + const declDir = join(dir, "supabase", "schemas"); mkdirSync(declDir, { recursive: true }); writeFileSync(join(declDir, "public.sql"), "create table t ();"); const { layer, edgeCalls, shadowSpawned } = setup(dir); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts index 11e8a29e43..62912a0cd5 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts @@ -337,7 +337,7 @@ const LEGACY_SCHEMA_PATHS_PATTERN = /\nschema_paths = \[[\s\S]*?\]\n/g; * rather than "doing the right TOML thing". * * `resolvedDeclarativeDir` is the resolved declarative dir (Go's - * `GetDeclarativeDir()`, e.g. `supabase/database`); the leading `supabase/` is + * `GetDeclarativeDir()`, e.g. `supabase/schemas`); the leading `supabase/` is * trimmed for the written value (Go's `strings.TrimPrefix`). */ export const legacyUpdateDeclarativeSchemaPathsConfig = Effect.fnUntraced(function* ( diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.unit.test.ts index 3291e51fc9..4bc35fd47f 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.unit.test.ts @@ -61,7 +61,7 @@ describe("legacyShouldApplyDeclarativeWithPgDelta", () => { () => Effect.gen(function* () { const path = yield* Path.Path; - expect(legacyShouldApplyDeclarativeWithPgDelta(path, true, ["database"], pgDelta())).toBe( + expect(legacyShouldApplyDeclarativeWithPgDelta(path, true, ["schemas"], pgDelta())).toBe( true, ); }).pipe(Effect.provide(BunServices.layer)), @@ -70,7 +70,7 @@ describe("legacyShouldApplyDeclarativeWithPgDelta", () => { it.effect("is false when the single schema_paths entry does not match the declarative dir", () => Effect.gen(function* () { const path = yield* Path.Path; - expect(legacyShouldApplyDeclarativeWithPgDelta(path, true, ["schemas"], pgDelta())).toBe( + expect(legacyShouldApplyDeclarativeWithPgDelta(path, true, ["database"], pgDelta())).toBe( false, ); }).pipe(Effect.provide(BunServices.layer)), @@ -210,7 +210,10 @@ describe("legacyLoadDeclaredSchemas", () => { path, workdir, [], - pgDelta({ enabled: true }), + pgDelta({ + enabled: true, + declarativeSchemaPath: Option.some("supabase/database"), + }), ); expect(result).toEqual(["supabase/database/t.sql"]); rmSync(workdir, { recursive: true, force: true }); @@ -234,7 +237,10 @@ describe("legacyLoadDeclaredSchemas", () => { path, workdir, ["custom/*.sql"], - pgDelta({ enabled: true }), + pgDelta({ + enabled: true, + declarativeSchemaPath: Option.some("supabase/database"), + }), ); expect(result).toEqual(["supabase/custom/x.sql"]); rmSync(workdir, { recursive: true, force: true }); @@ -497,7 +503,10 @@ describe("legacyLoadDeclaredSchemas", () => { path, workdir, [], - pgDelta({ enabled: true }), + pgDelta({ + enabled: true, + declarativeSchemaPath: Option.some("supabase/database"), + }), ); expect(result).toEqual(["supabase/schemas/a.sql"]); rmSync(workdir, { recursive: true, force: true }); @@ -544,7 +553,10 @@ describe("legacyLoadDeclaredSchemas", () => { path, workdir, [], - pgDelta({ enabled: true }), + pgDelta({ + enabled: true, + declarativeSchemaPath: Option.some("supabase/database"), + }), ); expect(result).toEqual([]); rmSync(workdir, { recursive: true, force: true }); @@ -735,7 +747,10 @@ describe("legacyLoadDeclaredSchemas", () => { path, workdir, [], - pgDelta({ enabled: true }), + pgDelta({ + enabled: true, + declarativeSchemaPath: Option.some("supabase/database"), + }), ).pipe(Effect.exit); expect(Exit.isFailure(exit)).toBe(true); if (Exit.isFailure(exit)) { diff --git a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts index 02fcd4858d..cee3825b7d 100644 --- a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts +++ b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts @@ -197,7 +197,7 @@ export interface LegacyPgDeltaTomlConfig { /** * `[experimental.pgdelta] declarative_schema_path`, resolved to a * `supabase/`-prefixed path when relative. `None` → callers use the default - * `supabase/database` (`legacyResolveDeclarativeDir`). + * `supabase/schemas` (`legacyResolveDeclarativeDir`). */ readonly declarativeSchemaPath: Option.Option; /** `[experimental.pgdelta] format_options`, a JSON string passed to pg-delta. */ @@ -215,7 +215,7 @@ const DEFAULT_API_SCHEMAS = ["public", "graphql_public"] as const; const DEFAULT_DENO_VERSION = 2; /** Default declarative schema dir. */ -const DEFAULT_DECLARATIVE_DIR_SEGMENTS = ["supabase", "database"] as const; +const DEFAULT_DECLARATIVE_DIR_SEGMENTS = ["supabase", "schemas"] as const; type RawDoc = { readonly [key: string]: unknown }; @@ -2795,7 +2795,7 @@ export const legacyReadDbToml = ( /** * The effective declarative schema directory: the configured * `declarative_schema_path` (already `supabase/`-prefixed when relative) or the - * default `supabase/database`. Mirrors `utils.GetDeclarativeDir`. + * default `supabase/schemas`. Mirrors `utils.GetDeclarativeDir`. * `path` joins the segments so * the separator matches the host platform, as `filepath.Join` does. */ diff --git a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.unit.test.ts b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.unit.test.ts index af3a2a3bd3..d8c886fa6e 100644 --- a/apps/cli/src/legacy/shared/legacy-db-config.toml-read.unit.test.ts +++ b/apps/cli/src/legacy/shared/legacy-db-config.toml-read.unit.test.ts @@ -2807,7 +2807,7 @@ describe("legacyReadDbToml [experimental.pgdelta]", () => { }); describe("legacyResolveDeclarativeDir", () => { - it.effect("uses the default supabase/database when no path is configured", () => + it.effect("uses the default supabase/schemas when no path is configured", () => Effect.gen(function* () { const path = yield* Path.Path; expect( @@ -2817,7 +2817,7 @@ describe("legacyResolveDeclarativeDir", () => { formatOptions: Option.none(), npmVersion: Option.none(), }), - ).toBe(join("supabase", "database")); + ).toBe(join("supabase", "schemas")); }).pipe(Effect.provide(BunServices.layer)), ); diff --git a/apps/cli/src/shared/init/project-init.templates.ts b/apps/cli/src/shared/init/project-init.templates.ts index 7830b86fb5..576ddf2445 100644 --- a/apps/cli/src/shared/init/project-init.templates.ts +++ b/apps/cli/src/shared/init/project-init.templates.ts @@ -409,7 +409,7 @@ s3_secret_key = "env(S3_SECRET_KEY)" [experimental.pgdelta] enabled = true # Directory under \`supabase/\` where declarative files are written. -# declarative_schema_path = "./database" +# declarative_schema_path = "./schemas" # JSON string passed through to pg-delta SQL formatting. # format_options = "{\\"keywordCase\\":\\"upper\\",\\"indent\\":2,\\"maxWidth\\":80,\\"commaStyle\\":\\"trailing\\"}" `; diff --git a/apps/docs/public/cli/config.schema.json b/apps/docs/public/cli/config.schema.json index ce0c6d3425..5a7d25463b 100644 --- a/apps/docs/public/cli/config.schema.json +++ b/apps/docs/public/cli/config.schema.json @@ -3650,7 +3650,7 @@ "type": "string", "description": "Directory under supabase/ where declarative schema files are written.", "examples": [ - "./database" + "./schemas" ] }, "format_options": { diff --git a/packages/config/src/experimental.ts b/packages/config/src/experimental.ts index 5db11d4ee4..9b6e13b243 100644 --- a/packages/config/src/experimental.ts +++ b/packages/config/src/experimental.ts @@ -86,7 +86,7 @@ export const experimental = Schema.Struct({ declarative_schema_path: Schema.optionalKey( Schema.String.annotate({ description: "Directory under supabase/ where declarative schema files are written.", - examples: ["./database"], + examples: ["./schemas"], tags, }), ), From 2cc84ab0359185f47dba1dc9fb8a5de9e7cd32bf Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 14 Aug 2026 19:25:55 +0200 Subject: [PATCH 31/82] fix(cli): preserve migration-owned pg-net --- .../legacy/shared/db-bootstrap/db-setup.ts | 8 +++++++- .../shared/db-bootstrap/db-setup.unit.test.ts | 19 ++++++++++++++++++- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index d96b221d51..a8fb88343f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -1337,7 +1337,13 @@ export const legacyRunDatabaseWebhooksSetup = (input: { if (!input.enabled) { const pgNetOwnedByMigrations = yield* legacyReadMigrationTable(session).pipe( Effect.map((migrations) => - migrations.some((migration) => migration.statements.some(legacyStatementInstallsPgNet)), + migrations.some( + (migration) => + // NULL/`{}` history rows become `[]`. That is incomplete evidence, + // not proof the migration did not install pg_net — preserve. + migration.statements.length === 0 || + migration.statements.some(legacyStatementInstallsPgNet), + ), ), Effect.orElseSucceed(() => true), ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts index f94a45c50a..2a0535e5b1 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts @@ -889,7 +889,7 @@ describe("legacyRunDatabaseWebhooksSetup", () => { const PG_NET_DROP_FINGERPRINT = "drop extension if exists pg_net"; function fakeWebhooksSession(opts: { - readonly appliedStatements?: ReadonlyArray>; + readonly appliedStatements?: ReadonlyArray | null>; readonly historyUnavailable?: boolean; }) { const execSql: Array = []; @@ -980,6 +980,23 @@ describe("legacyRunDatabaseWebhooksSetup", () => { ); }); + it.effect.each([ + { historyValue: null, description: "NULL" }, + { historyValue: [], description: "an empty array" }, + ])( + "preserves pg_net when an applied history row records $description for statements", + ({ historyValue }) => { + // Older volumes store NULL/`{}` in `schema_migrations.statements`. That is + // incomplete evidence, not proof the migration did not install pg_net. + const { execSql, effect } = converge(false, { appliedStatements: [historyValue] }); + return effect.pipe( + Effect.map(() => { + expect(execSql.some((sql) => sql.includes(PG_NET_DROP_FINGERPRINT))).toBe(false); + }), + ); + }, + ); + it.effect("preserves pg_net when the migration history cannot be read", () => { // Erring toward not dropping: an unreadable history is treated as ownership. const { execSql, effect } = converge(false, { historyUnavailable: true }); From d742d4ae7810b475aab66edb5c1eae712084c626 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 09:10:11 +0200 Subject: [PATCH 32/82] fix(cli): keep staged declarative exports outside the active tree - Normalize trailing separators (and trailing '.' segments) in legacyResolveStagedDeclarativeDir so a configured declarative_schema_path like './schemas/' stages a sibling './schemas-next' instead of nesting '-next' inside the active tree, where the printed rm -rf adoption command would destroy both copies. - Harden the sync staging guard to reject any staged directory that resolves inside the active declarative directory, not just equality. - Gate describeDockerLive on the configured live environment as well as the Docker probe, so live suites stay inert on machines that merely expose Docker; document the gate in AGENTS.md. - Record the TS-only --strict-coverage flag in go-cli-divergences.md. Co-Authored-By: Claude Fable 5 --- apps/cli/AGENTS.md | 3 ++- apps/cli/docs/go-cli-divergences.md | 8 ++++++ .../db/schema/declarative/declarative.flow.ts | 14 ++++++++-- .../declarative/declarative.flow.unit.test.ts | 27 +++++++++++++++++++ .../schema/declarative/sync/sync.handler.ts | 12 +++++++-- apps/cli/tests/helpers/live.ts | 13 +++++---- 6 files changed, 67 insertions(+), 10 deletions(-) diff --git a/apps/cli/AGENTS.md b/apps/cli/AGENTS.md index 950b0b5f49..d9fe136e1d 100644 --- a/apps/cli/AGENTS.md +++ b/apps/cli/AGENTS.md @@ -522,7 +522,8 @@ Live tests are black-box CLI subprocess tests — like `*.e2e.test.ts`, but run - **Where they run:** authored in this repo, but executed by the [`supabase/cli-e2e-ci`](https://github.com/supabase/cli-e2e-ci) harness, which builds this CLI, brings up a full supabox stack (and has a real Docker daemon, since that's how supabox itself runs), and invokes the `live` Vitest project (`nx run-many -t test:live`). They never run as part of the default unit/integration/e2e loop, and locally they no-op unless the live environment is configured (see below) — there is no need to stand up supabox yourself to develop other code. - **Add one whenever you add or change a command whose correctness genuinely depends on a real backend** — a new Management API command, or a change to `start`/`stop`/`status`'s real Docker interaction. Colocate it with the command, same as `*.e2e.test.ts`: `src/legacy/commands//[/].live.test.ts`. - **Gating:** every live suite must be wrapped in one of `tests/helpers/live.ts`'s `describe.skipIf` gates so the file is inert (skipped, not failed) outside the cli-e2e-ci runner: - - `describeLive` — runs whenever `SUPABASE_ACCESS_TOKEN` is set (the live env is configured at all). Reuse this even for commands that don't call the Management API themselves (e.g. `stop`/`status`) — it doubles as the "we're in the full cli-e2e-ci runner, which also has a real Docker daemon" signal, and there is no dedicated Docker-availability gate today. + - `describeLive` — runs whenever `SUPABASE_ACCESS_TOKEN` is set (the live env is configured at all). Reuse this even for commands that don't call the Management API themselves (e.g. `stop`/`status`) — it doubles as the "we're in the full cli-e2e-ci runner, which also has a real Docker daemon" signal. + - `describeDockerLive` — the configured-live gate composed with a `docker info` probe; use for local-stack suites whose scenarios additionally need a reachable Docker daemon at collection time. It never runs on a machine that merely exposes Docker — `SUPABASE_ACCESS_TOKEN` must still be set, so the file stays inert outside the cli-e2e-ci runner like every other live suite. - `describeLiveProject` — additionally requires a provisioned project (`SUPABASE_LIVE_PROJECT_REF`); use for project-scoped Management API commands (branches, functions, project-scoped db). - `describeLiveDataPlane` — additionally requires the project's own Postgres instance to be `ACTIVE_HEALTHY`; use for commands that talk to the project's data plane (migration, db, storage). - **Invocation:** use `runSupabaseLive(args, options?)` (wraps `runSupabase` with the `legacy` entrypoint and the live profile/timeout defaults) rather than calling `runSupabase` directly, so every live test picks up the same environment plumbing. diff --git a/apps/cli/docs/go-cli-divergences.md b/apps/cli/docs/go-cli-divergences.md index 5e0e579e4b..c265fe82e0 100644 --- a/apps/cli/docs/go-cli-divergences.md +++ b/apps/cli/docs/go-cli-divergences.md @@ -20,6 +20,14 @@ These commands exist in the TS CLI today but have no direct top-level equivalent ## Flag divergences from the Go reference +- `db diff`, `db pull`, and `db schema declarative generate`/`sync` have a TS-only + `--strict-coverage` flag (no Go equivalent). It applies only when the bundled + pg-delta next engine is active (the default): coverage gaps that the engine + reports — statements it skipped or objects it could not represent — normally + surface as warnings, and `--strict-coverage` promotes them to hard failures. + Under the `SUPABASE_USE_PG_DELTA_NEXT=false` legacy opt-out the flag is + accepted but has no effect, since the legacy edge-runtime engine does not + emit coverage diagnostics. Default behavior (omitted flag) matches Go. - `db push` has a TS-only `--skip-vault` flag. It applies migrations without resolving or updating `[db.vault]` secrets; default behavior still matches Go. - Every legacy command that resolves a linked project ref for its own database diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts index 0f69a18c17..ad410ebc17 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts @@ -263,8 +263,18 @@ export interface LegacyStagedExportContext { readonly schema: ReadonlyArray; } -export const legacyResolveStagedDeclarativeDir = (declarativeDir: string): string => - `${declarativeDir}-next`; +/** + * Derives the staging directory as a sibling of the declarative directory by + * suffixing its last path segment. Trailing separators (and `/.` segments) in + * the configured `declarative_schema_path` are stripped first — appending to + * `supabase/schemas/` verbatim would nest the staging directory *inside* the + * active tree, so a later sync would load the staged export recursively and + * the printed `rm -rf && mv` adoption command would destroy both copies. + */ +export const legacyResolveStagedDeclarativeDir = (declarativeDir: string): string => { + const trimmed = declarativeDir.replace(/(?:[\\/]+\.?)+$/, ""); + return `${trimmed === "" ? declarativeDir : trimmed}-next`; +}; function shellQuoteArgument(value: string): string { return /^[a-zA-Z0-9_./:@%+=,-]+$/.test(value) ? value : `'${value.replaceAll("'", `'"'"'`)}'`; diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts index a5d87513cb..27d0438637 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts @@ -4,9 +4,11 @@ import { legacyClassifyDeclarativeCompatibilityGap, legacyClassifyDeclarativeLoadCompatibility, legacyExtensionDeclaration, + legacyFormatStagedExportAdoption, legacyFormatStagedExportRecommendation, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, + legacyResolveStagedDeclarativeDir, } from "./declarative.flow.ts"; const stuck = (message: string) => ({ @@ -310,6 +312,31 @@ describe("legacyClassifyDeclarativeLoadCompatibility", () => { }); }); +describe("legacyResolveStagedDeclarativeDir", () => { + it("suffixes the last path segment to produce a sibling directory", () => { + expect(legacyResolveStagedDeclarativeDir("supabase/schemas")).toBe("supabase/schemas-next"); + }); + + it("strips trailing separators so the staged dir cannot nest inside the tree", () => { + expect(legacyResolveStagedDeclarativeDir("./schemas/")).toBe("./schemas-next"); + expect(legacyResolveStagedDeclarativeDir("supabase/schemas//")).toBe("supabase/schemas-next"); + expect(legacyResolveStagedDeclarativeDir("supabase\\schemas\\")).toBe("supabase\\schemas-next"); + }); + + it("strips trailing current-directory segments", () => { + expect(legacyResolveStagedDeclarativeDir("supabase/schemas/.")).toBe("supabase/schemas-next"); + expect(legacyResolveStagedDeclarativeDir("supabase/schemas/./")).toBe("supabase/schemas-next"); + }); + + it("prints adoption commands that target the sibling staged directory", () => { + const lines = legacyFormatStagedExportAdoption({ + declarativeDir: "./schemas/", + schema: [], + }); + expect(lines.join("\n")).toContain("rm -rf ./schemas/ && mv ./schemas-next ./schemas/"); + }); +}); + describe("legacyResolveDeclarativeMigrationName", () => { it.each([ ["my_change", "declarative_sync", "my_change"], diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 250111f75a..4fc84f671d 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -313,10 +313,18 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara ); const stageNextExport = Effect.fnUntraced(function* () { const stagedDir = path.resolve(cliConfig.workdir, stagedDirRel); - if (stagedDir === declarativeDir) { + // Reject the active directory itself AND anything nested under it: a + // staged export inside the declarative tree would be loaded recursively + // by the next sync, and the printed `rm -rf && mv` adoption command + // would delete the staged copy along with the tree. + const stagedRelative = path.relative(declarativeDir, stagedDir); + if ( + stagedRelative === "" || + (!stagedRelative.startsWith("..") && !path.isAbsolute(stagedRelative)) + ) { return yield* Effect.fail( new LegacyDeclarativeCompatibilityError({ - message: `${stagedDirRel} is the active declarative schema directory; choose a different staging directory.`, + message: `${stagedDirRel} is inside the active declarative schema directory; choose a different staging directory.`, }), ); } diff --git a/apps/cli/tests/helpers/live.ts b/apps/cli/tests/helpers/live.ts index 327cceacb3..8123bc96dd 100644 --- a/apps/cli/tests/helpers/live.ts +++ b/apps/cli/tests/helpers/live.ts @@ -50,12 +50,15 @@ function hasDockerDaemon(): boolean { } /** - * `describe` for local-stack live tests that only require a real Docker daemon. - * Unlike `describeLive`, this gate does not require platform credentials or a - * Management API. The synchronous `docker info` probe is read-only and runs once - * when this helper module is collected. + * `describe` for local-stack live tests that additionally require a reachable + * Docker daemon. Composes the configured-live gate (`isLiveConfigured`) with a + * `docker info` probe so these suites stay inert (skipped, not failed) outside + * the cli-e2e-ci runner — a machine that merely exposes Docker must never + * launch a real stack just by collecting the live Vitest project. The + * synchronous read-only probe runs once when this helper module is collected, + * and only when the live environment is configured. */ -export const describeDockerLive = describe.skipIf(!hasDockerDaemon()); +export const describeDockerLive = describe.skipIf(!isLiveConfigured() || !hasDockerDaemon()); /** * `describe` for project-scoped live suites: runs only when the live env is From 27a56e6576b42a96fca7463ceee9d3cf3937acd5 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 09:13:24 +0200 Subject: [PATCH 33/82] fix(cli): replace staged-dir trim regex with a linear scan CodeQL flagged the trailing-separator regex for ambiguous nested quantification (potential exponential backtracking on long separator runs). Trim trailing separators and '.' segments with an explicit linear scan instead; behavior is unchanged and covered by the same unit tests. Co-Authored-By: Claude Fable 5 --- .../db/schema/declarative/declarative.flow.ts | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts index ad410ebc17..3e080c9b3d 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts @@ -272,7 +272,18 @@ export interface LegacyStagedExportContext { * the printed `rm -rf && mv` adoption command would destroy both copies. */ export const legacyResolveStagedDeclarativeDir = (declarativeDir: string): string => { - const trimmed = declarativeDir.replace(/(?:[\\/]+\.?)+$/, ""); + const isSeparator = (ch: string | undefined) => ch === "/" || ch === "\\"; + let end = declarativeDir.length; + while (end > 0) { + if (isSeparator(declarativeDir[end - 1])) { + end -= 1; + } else if (declarativeDir[end - 1] === "." && isSeparator(declarativeDir[end - 2])) { + end -= 1; + } else { + break; + } + } + const trimmed = declarativeDir.slice(0, end); return `${trimmed === "" ? declarativeDir : trimmed}-next`; }; From f7e27f1e5613a40d486e914cc0a775d5d3a3d57d Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 10:04:29 +0200 Subject: [PATCH 34/82] fix(cli): unify declarative compat-gate recovery guidance Dogfooding follow-up for the pg-delta next compatibility gates: - Both gates (shadow-load failure and plan-refuse) now render one shared template: diagnosis + evidence + an explicit do-not-apply hazard line, with the staged-upgrade commands carried on the error's suggestion so Output.fail prints them instead of the generic 'rerun with --debug' footer. A deliberate gate no longer reads as a crash. - Non-interactive runs get exactly one recommended recovery (the staged regenerate). The extension.sql alternative was a false trail there: each hand-added declaration only unlocks the next refusal. - Interactive prompts keep repair as an advanced choice labelled with the full /extension.sql path and a 'may surface another gap' hint, and the repair gate now also offers the staged export as the recommended option. - db pull in-sync keeps Go's message and non-zero exit but replaces the debug footer with an explanatory suggestion; recorded as a deliberate divergence in go-cli-divergences.md. Co-Authored-By: Claude Fable 5 --- apps/cli/docs/go-cli-divergences.md | 7 ++ .../legacy/commands/db/pull/SIDE_EFFECTS.md | 7 +- .../legacy/commands/db/pull/pull.errors.ts | 8 ++ .../legacy/commands/db/pull/pull.handler.ts | 21 +++- .../commands/db/pull/pull.integration.test.ts | 12 ++- .../schema/declarative/declarative.errors.ts | 7 ++ .../db/schema/declarative/declarative.flow.ts | 75 ++++++++++---- .../declarative/declarative.flow.unit.test.ts | 99 +++++++++++++++---- .../declarative/declarative.orchestrate.ts | 56 ++++------- .../schema/declarative/sync/SIDE_EFFECTS.md | 21 +++- .../schema/declarative/sync/sync.handler.ts | 81 +++++++-------- .../declarative/sync/sync.integration.test.ts | 64 +++++++++++- 12 files changed, 332 insertions(+), 126 deletions(-) diff --git a/apps/cli/docs/go-cli-divergences.md b/apps/cli/docs/go-cli-divergences.md index c265fe82e0..00182eb622 100644 --- a/apps/cli/docs/go-cli-divergences.md +++ b/apps/cli/docs/go-cli-divergences.md @@ -159,3 +159,10 @@ These commands exist in the TS CLI today but have no direct top-level equivalent redirect the service-role key to an attacker-controlled host. Intentional TS-only hardening, not a parity bug — see [`services/SIDE_EFFECTS.md`](../src/legacy/commands/services/SIDE_EFFECTS.md). +- `db pull` in-sync (`"No schema changes found"`) keeps Go's message and its non-zero + exit code, but replaces the generic "Try rerunning the command with --debug to + troubleshoot the error." stderr footer with an explanatory suggestion line + ("The remote database is already in sync with your local migrations — nothing to + pull."). An in-sync database is a finding, not a failure to troubleshoot, so the + debug hint sent users chasing a non-existent bug. Message text and exit code — the + parts scripts depend on — are unchanged. diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index b5574119d8..1550e6efbb 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -106,7 +106,12 @@ disables formatting without disabling safe compaction. | `1` | `--project-ref` set with a resolved target other than linked; `--project-ref` combined with the `--experimental` structured-dump pull (see Notes) | > Note: unlike `db diff`, an empty diff (`No schema changes found`) is a **non-zero -> exit** for `db pull`. +> exit** for `db pull`. The message and exit code match Go, but the stderr footer +> does not: instead of Go's generic +> `Try rerunning the command with --debug to troubleshoot the error.`, `db pull` +> prints +> `The remote database is already in sync with your local migrations — nothing to pull.` +> (deliberate divergence — see `docs/go-cli-divergences.md`). ## Output diff --git a/apps/cli/src/legacy/commands/db/pull/pull.errors.ts b/apps/cli/src/legacy/commands/db/pull/pull.errors.ts index 340d8a7c77..568b8ae6a2 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.errors.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.errors.ts @@ -55,6 +55,14 @@ export class LegacyDbPullMigrationConflictError extends Data.TaggedError( */ export class LegacyDbPullInSyncError extends Data.TaggedError("LegacyDbPullInSyncError")<{ readonly message: string; + /** + * Explains the non-zero exit instead of letting `Output.fail` append the + * generic "Try rerunning the command with --debug" footer — an in-sync + * database is a finding, not a failure to troubleshoot. The message and exit + * code stay Go-identical; only the footer diverges (see + * `docs/go-cli-divergences.md`). + */ + readonly suggestion: string; }> { get [ErrorActionabilityId](): CliErrorActionabilityDeclaration { return actionability.dbFinding; diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index a90be573d9..813bc383ea 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -109,6 +109,15 @@ import { legacyUpdateMigrationHistory } from "./pull.sync.ts"; const DEPRECATION_LINE = "Flag --use-pg-delta has been deprecated, use --declarative with [experimental.pgdelta] enabled = true in your config.toml instead."; +/** + * Explains the in-sync non-zero exit. Go prints its generic + * `Try rerunning the command with --debug…` footer here, which reads like a + * crash for what is really a finding; the message and exit code stay Go-identical + * (see `docs/go-cli-divergences.md`). + */ +const IN_SYNC_SUGGESTION = + "The remote database is already in sync with your local migrations — nothing to pull."; + /** Migration-file mode for the initial pg_dump seed. */ const MIGRATION_FILE_MODE = 0o644; @@ -866,6 +875,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy return yield* Effect.fail( new LegacyDbPullInSyncError({ message: `No schema changes found (debug bundle: ${debugDir})`, + suggestion: IN_SYNC_SUGGESTION, }), ); } @@ -878,11 +888,15 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy return yield* Effect.fail( new LegacyDbPullInSyncError({ message: `No schema changes found (debug bundle: ${diffOutcome.debug.directory})`, + suggestion: IN_SYNC_SUGGESTION, }), ); } return yield* Effect.fail( - new LegacyDbPullInSyncError({ message: "No schema changes found" }), + new LegacyDbPullInSyncError({ + message: "No schema changes found", + suggestion: IN_SYNC_SUGGESTION, + }), ); } @@ -960,7 +974,10 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // empty → in sync. if (seededFromDump && !seedWroteBytes && diffEmpty) { return yield* Effect.fail( - new LegacyDbPullInSyncError({ message: "No schema changes found" }), + new LegacyDbPullInSyncError({ + message: "No schema changes found", + suggestion: IN_SYNC_SUGGESTION, + }), ); } writtenMigrations.push({ path: migrationPath, version: timestamp }); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 644cfd12a3..7cd7e431b2 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -1251,8 +1251,16 @@ describe("legacy db pull", () => { seedMigration(tmp.current, "20240101000000"); const s = setup(tmp.current, { remoteVersions: ["20240101000000"], edgeStdout: "" }); return Effect.gen(function* () { - const exit = yield* legacyDbPull(flags()).pipe(Effect.exit); - expect(Exit.isFailure(exit)).toBe(true); + // Go's message and non-zero exit are the contract; the generic + // "rerun with --debug" footer is replaced by an explanation instead + // (docs/go-cli-divergences.md). + const error = yield* legacyDbPull(flags()).pipe(Effect.flip); + expect(error).toMatchObject({ + _tag: "LegacyDbPullInSyncError", + message: "No schema changes found", + suggestion: + "The remote database is already in sync with your local migrations — nothing to pull.", + }); }).pipe(Effect.provide(s.layer)); }); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts index c69dd84ae5..a14f0bb7ca 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.errors.ts @@ -109,6 +109,13 @@ export class LegacyDeclarativeCompatibilityError extends Data.TaggedError( "LegacyDeclarativeCompatibilityError", )<{ readonly message: string; + /** + * Recovery commands, printed bare on stderr by `Output.fail` INSTEAD of the + * generic "Try rerunning the command with --debug" footer. A compatibility + * gate is a deliberate refusal, not a crash, so it must never suggest + * troubleshooting flags (same mechanism as {@link LegacyDeclarativeApplyError}). + */ + readonly suggestion?: string; /** Structured only for a known implicit-extension failure during shadow load. */ readonly loadFindings?: ReadonlyArray; }> { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts index 3e080c9b3d..4f8a4615cd 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts @@ -300,6 +300,9 @@ function schemaArguments(schema: ReadonlyArray): string { export const legacyFormatDeclarativeSyncCommand = (schema: ReadonlyArray): string => ` supabase db schema declarative sync --no-apply${schemaArguments(schema)} --experimental`; +const adoptionCommand = (declarativeDir: string, stagedDir: string): string => + ` rm -rf ${shellQuoteArgument(declarativeDir)} && mv ${shellQuoteArgument(stagedDir)} ${shellQuoteArgument(declarativeDir)}`; + export function legacyFormatStagedExportAdoption({ declarativeDir, schema, @@ -307,30 +310,32 @@ export function legacyFormatStagedExportAdoption({ const stagedDir = legacyResolveStagedDeclarativeDir(declarativeDir); return [ `Review ${stagedDir}, then adopt it:`, - ` rm -rf ${shellQuoteArgument(declarativeDir)} && mv ${shellQuoteArgument(stagedDir)} ${shellQuoteArgument(declarativeDir)}`, + adoptionCommand(declarativeDir, stagedDir), legacyFormatDeclarativeSyncCommand(schema), ]; } -export function legacyFormatStagedExportCommands( - context: LegacyStagedExportContext, -): ReadonlyArray { +/** The staged-upgrade recipe, as a copy-pasteable block of indented shell lines. */ +function stagedExportCommands(context: LegacyStagedExportContext): ReadonlyArray { const stagedDir = legacyResolveStagedDeclarativeDir(context.declarativeDir); return [ " supabase db schema declarative generate --local --overwrite \\", ` --output ${shellQuoteArgument(stagedDir)}${schemaArguments(context.schema)} --experimental`, - "", - ...legacyFormatStagedExportAdoption(context).map((line) => - line.startsWith("Review ") ? ` # review ${stagedDir}` : line, - ), + ` # review ${stagedDir}`, + adoptionCommand(context.declarativeDir, stagedDir), + legacyFormatDeclarativeSyncCommand(context.schema), ]; } -export function legacyFormatStagedExportRecommendation( +/** + * Evidence lines for a plan that succeeded but whose removals reveal the tree is + * a legacy export (the plan-refuse gate). The load-fail gate builds its own + * evidence from the shadow-load diagnostics instead. + */ +export function legacyFormatDeclarativeGapEvidence( gap: LegacyDeclarativeCompatibilityGap, - context: LegacyStagedExportContext, -): string { - const detected = [ +): ReadonlyArray { + return [ ...(gap.repairableExtensions.length > 0 ? [`Legacy-implicit extensions: ${gap.repairableExtensions.join(", ")}`] : []), @@ -345,10 +350,44 @@ export function legacyFormatStagedExportRecommendation( ] : []), ]; - return [ - "WARNING: pg-delta next manages schema state that the legacy export did not represent.", - ...detected, - "Generate a next-compatible schema into a separate directory, review it, and adopt it when ready:", - ...legacyFormatStagedExportCommands(context), - ].join("\n"); +} + +export interface LegacyDeclarativeUpgradeGateText { + readonly message: string; + readonly suggestion: string; +} + +/** + * The single template both compatibility gates render. Both mean the same thing + * ("this declarative tree is a legacy pg-delta export"), so they must read the + * same; only the evidence block differs. The recovery commands live in + * `suggestion` so `Output.fail` prints them instead of the generic + * "rerun with --debug" footer — a deliberate gate is not a crash. + * + * Deliberately offers exactly ONE non-interactive recovery: the staged + * regenerate. Telling a non-interactive user to hand-add an extension + * declaration is a false trail — on a real legacy tree each declaration only + * unlocks the next refusal. Interactive flows still offer the repair as an + * advanced choice. + */ +export function legacyFormatDeclarativeUpgradeGate(opts: { + readonly evidence: ReadonlyArray; + readonly context: LegacyStagedExportContext; +}): LegacyDeclarativeUpgradeGateText { + const { declarativeDir } = opts.context; + return { + message: [ + `This ${declarativeDir} tree looks like a legacy pg-delta export.`, + "pg-delta next only loads extensions the tree declares; legacy exports omitted", + "platform extensions and extension-managed objects like cron jobs.", + ...(opts.evidence.length > 0 ? ["", ...opts.evidence.map((line) => ` ${line}`)] : []), + "", + "Do not apply a sync generated from this tree — it can drop extensions or unschedule jobs.", + ].join("\n"), + suggestion: [ + `Upgrade without changing the active ${declarativeDir} tree:`, + "", + ...stagedExportCommands(opts.context), + ].join("\n"), + }; } diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts index 27d0438637..1a22b7611c 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts @@ -4,8 +4,9 @@ import { legacyClassifyDeclarativeCompatibilityGap, legacyClassifyDeclarativeLoadCompatibility, legacyExtensionDeclaration, + legacyFormatDeclarativeGapEvidence, + legacyFormatDeclarativeUpgradeGate, legacyFormatStagedExportAdoption, - legacyFormatStagedExportRecommendation, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, legacyResolveStagedDeclarativeDir, @@ -98,44 +99,106 @@ describe("legacyClassifyDeclarativeCompatibilityGap", () => { expect(legacyExtensionDeclaration("uuid-ossp")).toBe( 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA "extensions";', ); - const gap = classifyGap(); - expect( - legacyFormatStagedExportRecommendation(gap, { - declarativeDir: "supabase/schemas", - schema: [], - }), - ).toContain( + const { suggestion } = legacyFormatDeclarativeUpgradeGate({ + evidence: legacyFormatDeclarativeGapEvidence(classifyGap()), + context: { declarativeDir: "supabase/schemas", schema: [] }, + }); + expect(suggestion).toContain( "generate --local --overwrite \\\n --output supabase/schemas-next --experimental", ); }); it("derives staged-export commands from a custom declarative path", () => { - const recommendation = legacyFormatStagedExportRecommendation(classifyGap(), { - declarativeDir: "supabase/custom schema", - schema: [], + const { suggestion } = legacyFormatDeclarativeUpgradeGate({ + evidence: legacyFormatDeclarativeGapEvidence(classifyGap()), + context: { declarativeDir: "supabase/custom schema", schema: [] }, }); - expect(recommendation).toContain("--output 'supabase/custom schema-next'"); - expect(recommendation).toContain( + expect(suggestion).toContain("--output 'supabase/custom schema-next'"); + expect(suggestion).toContain( "rm -rf 'supabase/custom schema' && mv 'supabase/custom schema-next' 'supabase/custom schema'", ); }); it("preserves schema filters in staged-export and follow-up sync commands", () => { - const recommendation = legacyFormatStagedExportRecommendation(classifyGap(), { - declarativeDir: "supabase/schemas", - schema: ["app", "tenant,one"], + const { suggestion } = legacyFormatDeclarativeUpgradeGate({ + evidence: legacyFormatDeclarativeGapEvidence(classifyGap()), + context: { declarativeDir: "supabase/schemas", schema: ["app", "tenant,one"] }, }); - expect(recommendation).toContain( + expect(suggestion).toContain( `--output supabase/schemas-next --schema app --schema '"tenant,one"' --experimental`, ); - expect(recommendation).toContain( + expect(suggestion).toContain( `sync --no-apply --schema app --schema '"tenant,one"' --experimental`, ); }); }); +describe("legacyFormatDeclarativeUpgradeGate", () => { + it("renders one template with indented evidence and no --debug-style guidance", () => { + const gate = legacyFormatDeclarativeUpgradeGate({ + evidence: legacyFormatDeclarativeGapEvidence(classifyGap()), + context: { declarativeDir: "supabase/schemas", schema: [] }, + }); + + expect(gate.message).toBe( + [ + "This supabase/schemas tree looks like a legacy pg-delta export.", + "pg-delta next only loads extensions the tree declares; legacy exports omitted", + "platform extensions and extension-managed objects like cron jobs.", + "", + " Legacy-implicit extensions: pgcrypto, uuid-ossp", + " Extension-managed objects: pg_cron job refresh download metrics, pgmq queue emails", + "", + "Do not apply a sync generated from this tree — it can drop extensions or unschedule jobs.", + ].join("\n"), + ); + expect(gate.suggestion).toBe( + [ + "Upgrade without changing the active supabase/schemas tree:", + "", + " supabase db schema declarative generate --local --overwrite \\", + " --output supabase/schemas-next --experimental", + " # review supabase/schemas-next", + " rm -rf supabase/schemas && mv supabase/schemas-next supabase/schemas", + " supabase db schema declarative sync --no-apply --experimental", + ].join("\n"), + ); + }); + + it("offers no extension.sql alternative — the staged upgrade is the only recovery", () => { + const gate = legacyFormatDeclarativeUpgradeGate({ + evidence: [ + "members.sql:3 uses extensions.uuid_generate_v4(), but the tree does not declare uuid-ossp.", + ], + context: { declarativeDir: "supabase/schemas", schema: [] }, + }); + + expect(`${gate.message}\n${gate.suggestion}`).not.toContain("extension.sql"); + expect(gate.message).toContain( + " members.sql:3 uses extensions.uuid_generate_v4(), but the tree does not declare uuid-ossp.", + ); + }); + + it("reports ambiguous extension removals as their own evidence line", () => { + expect( + legacyFormatDeclarativeGapEvidence( + classifyGap({ removals: { extensions: ["postgis"], extensionIntents: [] } }), + ), + ).toEqual(["Extensions: postgis"]); + }); + + it("omits the evidence block entirely when there is nothing to report", () => { + const gate = legacyFormatDeclarativeUpgradeGate({ + evidence: [], + context: { declarativeDir: "supabase/schemas", schema: [] }, + }); + + expect(gate.message).not.toContain("\n\n\n"); + }); +}); + describe("legacyClassifyDeclarativeLoadCompatibility", () => { it.each([ ["extensions.uuid_generate_v4()", "uuid-ossp"], diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts index 50740ffdcc..10a4bccc07 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts @@ -33,10 +33,9 @@ import { } from "./declarative.errors.ts"; import { legacyClassifyDeclarativeLoadCompatibility, - legacyExtensionDeclaration, - legacyFormatDeclarativeSyncCommand, - legacyFormatStagedExportCommands, + legacyFormatDeclarativeUpgradeGate, type LegacyDeclarativeLoadCompatibilityFinding, + type LegacyDeclarativeUpgradeGateText, } from "./declarative.flow.ts"; /** Ambient inputs shared by the orchestration steps. */ @@ -70,35 +69,17 @@ const declarativeError = (message: string) => new LegacyDeclarativeDiffError({ m const formatImplicitExtensionLoadFailure = ( findings: ReadonlyArray, run: Pick, -): string => { - const extensions = [...new Set(findings.map((finding) => finding.extension))].sort(); - const detected = findings.map((finding) => { - const location = - finding.file === undefined - ? "A declarative schema file" - : `${finding.file}${finding.line === undefined ? "" : `:${finding.line}`}`; - return `${location} uses ${finding.signature}, but the tree does not declare ${finding.extension}.`; - }); - return [ - "This declarative schema looks like a legacy pg-delta export.", - "", - ...detected, - "", - "pg-delta next loads desired state onto a shadow that only has extensions you declare. Legacy generate omitted platform extensions.", - "", - "Recommended — generate a next-compatible tree, review it, then adopt:", - "", - ...legacyFormatStagedExportCommands({ - declarativeDir: run.declarativeDirDisplay, - schema: run.schema, +): LegacyDeclarativeUpgradeGateText => + legacyFormatDeclarativeUpgradeGate({ + evidence: findings.map((finding) => { + const location = + finding.file === undefined + ? "A declarative schema file" + : `${finding.file}${finding.line === undefined ? "" : `:${finding.line}`}`; + return `${location} uses ${finding.signature}, but the tree does not declare ${finding.extension}.`; }), - "", - "Alternative — add the missing extension declarations to extension.sql, then re-plan:", - ...extensions.map((extension) => legacyExtensionDeclaration(extension)), - "", - legacyFormatDeclarativeSyncCommand(run.schema), - ].join("\n"); -}; + context: { declarativeDir: run.declarativeDirDisplay, schema: run.schema }, + }); /** * Computes the diff between local migrations state and the declarative schema. @@ -162,12 +143,13 @@ export const legacyDiffDeclarativeToMigrations = Effect.fnUntraced(function* ( diagnostics: error.diagnostics ?? [], files, }); - return findings.length === 0 - ? error - : new LegacyDeclarativeCompatibilityError({ - message: formatImplicitExtensionLoadFailure(findings, run), - loadFindings: findings, - }); + if (findings.length === 0) return error; + const gate = formatImplicitExtensionLoadFailure(findings, run); + return new LegacyDeclarativeCompatibilityError({ + message: gate.message, + suggestion: gate.suggestion, + loadFindings: findings, + }); }), ); return { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 5b98756199..5b14e25304 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -83,11 +83,22 @@ legacy opt-out, warming the catalog cache) — on both interactive and `--yes` p without prompting; both override the global `--yes`. `--no-apply` and `--apply` are mutually exclusive. -Before writing a bundled-engine migration, a manifest-less legacy tree that -would remove only `pgcrypto`, `uuid-ossp`, or `pg_net` offers to append their -declarations to `extension.sql` and re-plan, continue, or cancel. Non-interactive -execution (including `--yes`) stops and prints the SQL instead of modifying the -tree. The repair never overwrites existing SQL or creates an export manifest. +A manifest-less legacy tree is refused by two compatibility gates — one when the +tree fails to load on the bundled engine's shadow, one when the plan's removals +reveal legacy-implicit extensions or extension-managed objects. Both render the +same message (`This tree looks like a legacy pg-delta export.` +plus an indented evidence block) and both carry the staged-upgrade recipe on the +error's suggestion, so the generic `Try rerunning the command with --debug` +footer is **not** printed. Non-interactive execution (including `--yes`) stops +there and modifies nothing; the only recommended recovery is regenerating into +`-next`, reviewing it, and adopting it. + +In a TTY both gates additionally offer to generate that staged export +(recommended), and — when the gap is only `pgcrypto`, `uuid-ossp`, or `pg_net` — +to append those declarations to `/extension.sql` and re-plan, or +to continue with the removals, or cancel. The in-place repair is an advanced +choice (it may surface another gap on the next plan); it never overwrites +existing SQL or creates an export manifest. ## Notes diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 4fc84f671d..8525ef6eaa 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -56,9 +56,9 @@ import { } from "../declarative.errors.ts"; import { legacyClassifyDeclarativeCompatibilityGap, - legacyExtensionDeclaration, + legacyFormatDeclarativeGapEvidence, + legacyFormatDeclarativeUpgradeGate, legacyFormatStagedExportAdoption, - legacyFormatStagedExportRecommendation, legacyResolveStagedDeclarativeDir, legacyResolveDeclarativeMigrationName, legacyResolveDeclarativeSyncApplyDecision, @@ -161,6 +161,9 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara // a relative path). `path.join(workdir, abs)` would mangle the absolute path. const declarativeDir = path.resolve(cliConfig.workdir, declarativeDirRel); const stagedDirRel = legacyResolveStagedDeclarativeDir(declarativeDirRel); + // Repair prompts name the file they would edit by its full configured path — + // a bare `extension.sql` is ambiguous in a tree with nested schema folders. + const extensionSqlRel = path.join(declarativeDirRel, "extension.sql"); const migrationsDir = path.join(cliConfig.workdir, "supabase", "migrations"); const tempDir = legacyPgDeltaTempPath(path, cliConfig.workdir); const run: LegacyDeclarativeRunContext = { @@ -435,7 +438,11 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara label: `Generate next export to ${stagedDirRel}`, hint: "recommended", }, - { value: "repair", label: "Add missing extension declarations and re-plan" }, + { + value: "repair", + label: `Add missing extension declarations to ${extensionSqlRel} and re-plan`, + hint: "may surface another gap", + }, { value: "cancel", label: "Cancel" }, ]); if (choice === "cancel") return Option.none(); @@ -485,17 +492,26 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara }); if (compatibility.recommendedAction === "none") break; + // Both recommended actions mean the same thing to the user — the tree is a + // legacy export — so they render one shared template and differ only in the + // choices offered. Non-interactively there is exactly one recovery: the + // staged regenerate, carried on `suggestion` so `Output.fail` prints it + // instead of the "rerun with --debug" footer. + const gate = legacyFormatDeclarativeUpgradeGate({ + evidence: legacyFormatDeclarativeGapEvidence(compatibility), + context: { declarativeDir: declarativeDirRel, schema: flags.schema }, + }); + if (!tty.stdinIsTty || yes) { + return yield* Effect.fail( + new LegacyDeclarativeCompatibilityError({ + message: gate.message, + suggestion: gate.suggestion, + }), + ); + } + yield* output.raw(`${legacyYellow(gate.message)}\n`, "stderr"); + if (compatibility.recommendedAction === "stage-next-export") { - const explanation = legacyFormatStagedExportRecommendation(compatibility, { - declarativeDir: declarativeDirRel, - schema: flags.schema, - }); - if (!tty.stdinIsTty || yes) { - return yield* Effect.fail( - new LegacyDeclarativeCompatibilityError({ message: explanation }), - ); - } - yield* output.raw(`${legacyYellow(explanation)}\n`, "stderr"); const choice = yield* output.promptSelect("How would you like to continue?", [ { value: "stage", @@ -508,37 +524,24 @@ export const legacyDbSchemaDeclarativeSync = Effect.fn("legacy.db.schema.declara return; } - const statements = compatibility.repairableExtensions.map(legacyExtensionDeclaration); - const explanation = [ - "This declarative schema appears to use legacy pg-delta behavior. Legacy pg-delta treated these installed extensions as implicit, while pg-delta next treats their omission as removal:", - "", - ...compatibility.repairableExtensions.map((extension) => `- ${extension}`), - ].join("\n"); - if (!tty.stdinIsTty || yes) { - return yield* Effect.fail( - new LegacyDeclarativeCompatibilityError({ - message: [ - explanation, - "", - "Non-interactive sync will not modify the declarative schema automatically. Add these statements to extension.sql, then run sync again:", - ...statements, - "", - legacyFormatStagedExportRecommendation(compatibility, { - declarativeDir: declarativeDirRel, - schema: flags.schema, - }), - ].join("\n"), - }), - ); - } - - yield* output.raw(`${legacyYellow(explanation)}\n`, "stderr"); + // Repairing the tree in place is offered only interactively, and only as an + // advanced choice: on a real legacy tree each added declaration tends to + // unlock the next refusal, so it is a false trail for a scripted run. const choice = yield* output.promptSelect("How would you like to continue?", [ - { value: "repair", label: "Add declarations and re-plan", hint: "recommended" }, + { value: "stage", label: `Generate next export to ${stagedDirRel}`, hint: "recommended" }, + { + value: "repair", + label: `Add declarations to ${extensionSqlRel} and re-plan`, + hint: "may surface another gap", + }, { value: "continue", label: "Continue with removals" }, { value: "cancel", label: "Cancel" }, ]); if (choice === "cancel") return; + if (choice === "stage") { + yield* stageNextExport(); + return; + } if (choice === "continue") break; const repaired = yield* legacyAppendExtensionDeclarations( declarativeDir, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index cb2b6cd49b..66f9ec2cb4 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -1046,10 +1046,15 @@ describe("legacy db schema declarative sync integration", () => { const error = failError(exit); expect(error).toMatchObject({ message: expect.stringContaining("uuid-ossp"), + // Recovery commands ride on `suggestion` so `Output.fail` prints them + // instead of the generic "rerun with --debug" footer. + suggestion: expect.stringContaining( + "supabase db schema declarative generate --local --overwrite", + ), }); - expect(JSON.stringify(error)).toContain( - "supabase db schema declarative generate --local --overwrite", - ); + // Hand-editing extension.sql is a false trail non-interactively: each + // declaration only unlocks the next refusal. + expect(JSON.stringify(error)).not.toContain("extension.sql"); expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); }).pipe(Effect.provide(s.layer)); }); @@ -1178,7 +1183,16 @@ describe("legacy db schema declarative sync integration", () => { const exit = yield* legacyDbSchemaDeclarativeSync(flags()).pipe(Effect.exit); expect(failError(exit)).toMatchObject({ _tag: "LegacyDeclarativeCompatibilityError", - message: expect.stringContaining("pg_cron job refresh download metrics"), + // Same unified template as the load-fail gate — only the evidence differs. + message: expect.stringContaining( + "This supabase/schemas tree looks like a legacy pg-delta export.", + ), + suggestion: expect.stringContaining( + "Upgrade without changing the active supabase/schemas tree:", + ), + }); + expect(failError(exit)).toMatchObject({ + message: expect.stringContaining(" Extension-managed objects: pg_cron job refresh"), }); expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); }).pipe(Effect.provide(s.layer)); @@ -1222,6 +1236,48 @@ describe("legacy db schema declarative sync integration", () => { }, ); + it.effect("repairs the active tree in place when the user picks the advanced choice", () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + diffSql: 'DROP EXTENSION "pgcrypto";\n', + replannedDiffSql: "ALTER TABLE a ADD COLUMN b int;\n", + removals: { extensions: ["pgcrypto"], extensionIntents: [] }, + promptSelectResponses: ["repair"], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + expect(readFileSync(join(tmp.current, "supabase", "schemas", "extension.sql"), "utf8")).toBe( + 'CREATE EXTENSION IF NOT EXISTS "pgcrypto" WITH SCHEMA "extensions";\n', + ); + expect(s.planCalls).toBe(2); + expect(readdirSync(join(tmp.current, "supabase", "migrations"))).toHaveLength(1); + }).pipe(Effect.provide(s.layer)); + }); + + it.effect("stages a next export from the repair prompt without touching the tree", () => { + seedDeclarative(tmp.current); + const s = setup(tmp.current, { + engineImplementation: "next", + stdinIsTty: true, + diffSql: 'DROP EXTENSION "pgcrypto";\n', + removals: { extensions: ["pgcrypto"], extensionIntents: [] }, + promptSelectResponses: ["stage"], + }); + return Effect.gen(function* () { + yield* legacyDbSchemaDeclarativeSync(flags({ noApply: Option.some(true) })); + expect( + existsSync(join(tmp.current, "supabase", "schemas-next", ".pgdelta-export.json")), + ).toBe(true); + expect(existsSync(join(tmp.current, "supabase", "schemas", "extension.sql"))).toBe(false); + expect(existsSync(join(tmp.current, "supabase", "migrations"))).toBe(false); + expect(stripAnsi(s.out.stderrText)).toContain( + "rm -rf supabase/schemas && mv supabase/schemas-next supabase/schemas", + ); + }).pipe(Effect.provide(s.layer)); + }); + it.effect("cancels compatibility resolution without schema or migration writes", () => { seedDeclarative(tmp.current); const s = setup(tmp.current, { From 322364607a099827b5a802fbbd98f8f5b14ef0f1 Mon Sep 17 00:00:00 2001 From: Andrew Valleteau Date: Thu, 13 Aug 2026 08:06:14 +0000 Subject: [PATCH 35/82] docs(cli): add shadow-db startup performance plan (readiness gate + warm container cache) Benchmarked the isolated-shadow provisioning pipeline on supabase/postgres:17.6.1.158 (~20.5s cold: ~6.5s healthcheck dead wait, ~10s one-shot service jobs, ~3.5s initdb) and captured the agreed design: - Workstream A: connect-probe readiness for the shadow (saves ~6.5s/provision) - Workstream B: exec the entrypoint so docker stop is ~1s instead of 10s - Workstream C: warm shadow container cache keyed on resolved pg + service image tags and all baseline inputs, with template-DB + role-delta reset and a cold-provision escape hatch CLI-only; pg-delta is explicitly out of scope. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GpcqgbueKJdGMtTQf3Vf6K --- apps/cli/docs/shadow-db-startup-perf-plan.md | 285 +++++++++++++++++++ 1 file changed, 285 insertions(+) create mode 100644 apps/cli/docs/shadow-db-startup-perf-plan.md diff --git a/apps/cli/docs/shadow-db-startup-perf-plan.md b/apps/cli/docs/shadow-db-startup-perf-plan.md new file mode 100644 index 0000000000..43fc03d7a2 --- /dev/null +++ b/apps/cli/docs/shadow-db-startup-perf-plan.md @@ -0,0 +1,285 @@ +# Shadow database startup performance — implementation plan + +Status: **planned, not implemented**. This document is a handoff spec: it contains the +measured baseline, the agreed design decisions, and file-level work items. An implementing +agent should be able to execute it without re-deriving the investigation. + +## Context + +`db diff` / `db pull` / declarative sync provision a **shadow database**: a throwaway +`supabase/postgres` container brought to the platform baseline (init schema + realtime / +storage / auth one-shot migration jobs), then handed to pg-delta as the isolated shadow +(`pgdelta … --isolated-shadow`, loader mode `isolatedCluster`). pg-delta requires this +shadow on a **different Postgres lineage** than the target, so the container is on the hot +path of *every* plan — the migrations-catalog cache cannot make it go away. + +Today the shadow is created cold and destroyed (`docker rm -f -v`) on every run. + +## Measured baseline + +Benchmarked 2026-08-13 on `supabase/postgres:17.6.1.158` (Docker 29, overlayfs, warm image +cache), reproducing the exact container shape `legacyBuildShadowPostgresContainerSpec` +produces (same entrypoint heredoc script, env, healthcheck flags) and the exact one-shot +job env from `db-setup.ts`. Two cold samples, consistent within ~1s: + +| Phase | Time | +|---|---| +| `docker create` + secret `docker cp` + `docker start` | ~0.4s | +| Postgres accepting connections (initdb + bundled init SQL) | ~3.4–4.5s | +| Docker healthcheck reports `healthy` (**the CLI's current gate**) | ~10.2s | +| One-shot realtime job (Elixir boot + tenant seed) | ~6.5s | +| One-shot storage migrate job | ~2.5s | +| One-shot auth (`gotrue migrate`) job | ~0.7s | +| Revoke API privileges + `CREATE DATABASE contrib_regression TEMPLATE postgres` | ~0.3s | +| **Total cold provision** (excl. image pulls, excl. user migrations) | **~20.5s** | + +Additional measurements that motivate the work items: + +- The healthcheck is `interval=10s, timeout=2s, retries=3` with **no + start_period/start_interval** (`postgres.service.ts`), so Docker's first probe runs at + t+10s. Postgres is connectable at ~3.5s → **~6.5s per provision is pure dead wait**. The + same container with `--health-start-period 30s --health-start-interval 1s` reported + healthy at 3.2s. +- `docker start` of an already-initialized shadow container → connectable in **~1.0s** + (also ~1s after a SIGKILL'd stop, via WAL recovery). +- `CREATE DATABASE … TEMPLATE ` on the baseline state: **~0.2s**. `DROP DATABASE … + WITH (FORCE)`: ~0.1s. +- `docker stop` on the current container takes **10.3s**: the entrypoint is `sh -c "… && + docker-entrypoint.sh …"`, so PID 1 is `sh`, which does not forward SIGTERM; Docker waits + the 10s grace period and SIGKILLs. + +Target end state: **~1.5–2s** per plan on a warm cache hit; **~14s** cold (health gate +fixed); cold path otherwise unchanged. + +## Design decisions (already made — do not relitigate) + +1. **This is a CLI-only concern.** pg-delta is not modified and never learns the cache + exists. Its `isolatedCluster` loader mode already supports a pre-provisioned shadow + with a platform baseline and pre-existing rows. The deliverable is "set up the base + `supabase/postgres` container + its owned services faster". +2. **Cache the container, not a volume.** The shadow mounts no volume today (PGDATA in + container fs, `binds: []`), and Docker has no cheap volume clone. Reuse = keep the + initialized container **stopped** between runs, `docker start` it on the next plan, and + prune state with template databases + a role-delta reset. +3. **Invalidate on any input change, never partially refresh.** See cache key below. +4. **Escape hatch everywhere:** any error or anomaly on the warm path (start failure, port + busy, reset error, missing metadata) ⇒ treat as cache miss: `docker rm -f -v` the + container and cold-provision. Worst case is today's behavior; the cache can never + produce a wrong baseline. +5. **Readiness fix uses a direct connect probe, not `--health-start-interval`.** That flag + requires Docker Engine 25+/API 1.44 and is not reliably supported by Podman (the CLI + falls back to Podman via `spawnContainerCli`). A connect probe sidesteps the + runtime-version matrix. (`docker-create-args.ts` already supports + `--health-start-period` if a flag-based variant is ever wanted, but it alone does not + speed up the first probe.) +6. Timing is **not** part of the Go-parity surface (ADR 0016) — these are TS-side + improvements. Do not change stdout text, exit codes, or flag surfaces. + +## Cache key (workstream C) + +Hash (sha256, stable field order) of every input baked into the cluster during cold +provisioning: + +- resolved `supabase/postgres` image tag (full tag, e.g. `17.6.1.158` — *not* major + version) after registry/pin resolution; +- resolved one-shot job image tags — `realtime`, `storage`, `auth` via + `legacyResolvePinnedImage` + `serviceVersionOverrides` — each included **only when its + service is enabled** (a disabled service's job never ran into the baseline); +- service enabled flags themselves (realtime/storage/auth); +- `jwtSecret`, `rootKey`, `[db] password`, `db.settings` (serialized), `jwtExpiry`; +- effective `api.auto_expose_new_tables`; +- `supabase/roles.sql` contents (empty string when absent); +- `[db.vault]` secret **names and values** (values are upserted into the DB — the existing + `setupInputsToken` in `legacy-pgdelta.cache.ts` hashes names only, which is + insufficient here; do not reuse it as-is, but mirror its hashing style); +- `shadowPort` (the stopped container's port binding is fixed at create time); +- `db.major_version` (implied by the image tag in practice, but cheap and explicit). + +Note: the existing `setupInputsToken` also omits the service image tags. That may or may +not matter for the catalog cache (out of scope here — flag it to a human if touched); for +the container cache it definitely matters, because the jobs write versioned schema state +(`auth`, `storage`, `_realtime`) into the cluster. + +## Workstream A — shadow readiness gate (independent, ship first) + +**Problem:** `legacyPrepareRawShadow` and `legacyPrepareShadowSource` gate on +`legacyWaitForHealthyServices` (1s poll of `docker inspect` health), but the container +cannot report healthy before the healthcheck's first 10s-interval probe. ~6.5s dead wait +per provision, including CI and cache-miss paths. + +**Change:** for the **shadow container only** (do not touch the long-running `db` +container's wait), replace the docker-health gate with a readiness probe that polls, on the +same 1s constant backoff and the same `healthTimeoutSeconds` budget: + +1. `legacyInspectContainerState` → still `running`? (preserves the crash-detection the + health gate provided; a dead container fails fast with the same + `LegacyHealthCheckTimeoutError` shape + log dump behavior); then +2. a short-timeout TCP/auth connect attempt (reuse `LegacyDbConnection.connect` the way + `legacyConnectShadowDatabase` does, or `pg_isready` semantics via a cheap connect) — + success ⇒ ready. + +Suggested shape: a `legacyWaitForShadowReady(spawner, containerId, connConfig, opts)` in +`shared/db-bootstrap/health-check.ts` (or a sibling module), used by +`legacyPrepareRawShadow` (`shadow-database.ts`) and `legacyPrepareShadowSource` +(`commands/db/shared/legacy-shadow-source.ts`). Keep the container's healthcheck config +unchanged (other tooling reads it); only the CLI-side wait changes. + +**Files:** + +- `apps/cli/src/legacy/shared/db-bootstrap/health-check.ts` (new probe) +- `apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts` (`legacyPrepareRawShadow`) +- `apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts` +- unit/integration tests colocated per repo convention + +**Tests (RED first, per repo policy):** integration test with a mocked container-state / +connection layer proving the wait resolves as soon as a connect succeeds (does not wait +for docker health), still fails with the timeout error + log dump when the container never +becomes connectable, and fails fast when the container exits. + +**Acceptance:** shadow provision reaches "connected" in roughly `postgres-ready + ≤1s` +(measured ~3.5–5s instead of ~10.5s); error behavior on a broken container unchanged in +shape. + +## Workstream B — clean fast shadow shutdown (small, enables C) + +**Problem:** `sh` as PID 1 swallows SIGTERM → every shadow stop (and workstream C's +cache-release stop) burns the 10s grace period and ends in SIGKILL. + +**Change:** in the entrypoint script builders (`postgres.service.ts`, +`legacyPostgresEntrypointScriptPg15` / `…Pg14`), `exec` the final command: `… && exec +docker-entrypoint.sh postgres -D /etc/postgresql `. Decide scope deliberately: + +- Minimal/safe: apply only in `legacyBuildShadowPostgresContainerSpec`'s script (add a + parameter to the script builders rather than post-processing the string). +- Broader (recommended if reviewers agree): apply to the long-running `db` container too — + it has the same latent 10s-stop cost — but that touches the Go-parity container shape, + so call it out explicitly in the PR rather than folding it in silently. + +**Tests:** unit snapshot of the generated script (existing spec builders have snapshot +coverage patterns); a live test is optional (`stop` timing is observable but flaky to +assert — asserting the script contains `exec` is enough). + +## Workstream C — warm shadow container cache + +New module, suggested `apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts`, exposing +an acquire/release pair that call sites use in place of bare +`legacyCreateShadowDatabase` / `legacyRemoveShadowDatabase`. Gate the whole feature behind +an opt-in env var (e.g. `SUPABASE_SHADOW_CACHE=1`) for the first release; flip the default +once proven. + +### Cold provision (cache miss) + +1. Create + start the shadow as today (no `autoRemove` when caching — the container must + survive), **labeled** with the project labels plus a new + `com.supabase.cli.shadow-cache-key=` label. +2. Run the existing baseline setup (`legacySetupShadowDatabase` / + `legacyMigrateShadowDatabase` path unchanged up to the baseline; user migrations are + *not* part of the cached state — see reset protocol). +3. **Snapshot for reuse**, immediately after the baseline (before migrations/declarative + load): + - `CREATE DATABASE _supabase_shadow_base TEMPLATE postgres` (requires no other + connections to `postgres` — sequence it before pg-delta connects); + - capture cluster-global state: `pg_roles` (name + attribute columns), `pg_auth_members` + (role, member, admin_option), and cluster-wide `pg_db_role_setting` rows + (`setdatabase = 0`); + - persist metadata JSON to `supabase/.temp/pgdelta/shadow-cache-.json`: + `{ key, containerId, createdAt, roleSnapshot, membershipSnapshot, roleSettings }`. +4. Proceed with the run as today (migrations applied to `postgres`, `contrib_regression` + template creation, pg-delta, …). +5. **Release:** instead of `docker rm -f -v` → `docker stop` (fast after workstream B). + Keep release best-effort exactly like `legacyRemoveShadowDatabase` (never mask the + run's own outcome). + +### Warm acquire (cache hit) + +1. Compute the key; look up the metadata file **and** the container (`docker ps -a + --filter label=com.supabase.cli.shadow-cache-key=`). Either missing, or container + already running (another concurrent run owns it) ⇒ miss (concurrent case: fall through + to a one-off uncached cold shadow, do not remove the cached one). +2. `docker start` (~1s), wait via workstream A's probe. +3. **Reset to pristine:** + - reverse the role delta vs. the snapshot: drop roles not in the snapshot (`DROP OWNED + BY` in each affected DB is unnecessary once `postgres` is recreated — drop role after + the DB recreate below to avoid dependency errors; simplest safe order: recreate DBs + first, then `DROP ROLE`), revoke added memberships, re-grant removed ones, delete + cluster-wide `pg_db_role_setting` rows not in the snapshot; + - connect to `_supabase_shadow_base` and `DROP DATABASE postgres WITH (FORCE)` + + `CREATE DATABASE postgres TEMPLATE _supabase_shadow_base` — every downstream consumer + keeps using `postgres` + `contrib_regression` unchanged; + - `DROP DATABASE IF EXISTS contrib_regression WITH (FORCE)` (recreated by the normal + flow). +4. Hand the same `LegacyShadowSourceResult` shape to the caller; the rest of the run is + byte-identical to today. +5. Any step failing ⇒ escape hatch (rm + cold provision + fresh snapshot). + +### Interactions & housekeeping + +- **`supabase stop`** sweeps project-labeled containers — it will delete the cached shadow. + That is acceptable (cache cleared, next run cold-provisions); mention it in the PR, do + not special-case. +- **Metadata/container drift:** container exists but metadata file missing (or vice versa) + ⇒ miss + remove the orphan. Add the metadata file to the same retention/cleanup pass the + catalog cache uses if one exists; otherwise overwrite-on-write is enough (one file per + key, old keys' containers removed on key mismatch — enumerate by label, keep only the + current key's container). +- **Leak window:** unlike today's `--rm` shadow, a crashed CLI leaves a *stopped, labeled* + container. That is the cache working as intended; `supabase stop` and the key-mismatch + sweep both reclaim it. + +### Call sites to convert + +Whatever consumes the create/remove pair around `legacyPrepareShadowSource` / +`legacyPrepareRawShadow` today (as of writing: `db diff`'s handler, `db pull`'s handler, +and `legacy-pgdelta.cache.ts`'s `exportViaShadowCatalog`) — all via +`Effect.acquireUseRelease`, so the seam is narrow: acquire ⇒ `shadow-cache` acquire, +release ⇒ `shadow-cache` release. `migration squash` (if/when it lands on the native +shadow path) picks it up for free through the same primitives. + +### Tests + +- **Unit:** cache-key builder (field order, every input changes the hash, disabled service + excludes its tag); reset-SQL builder (role delta → exact DROP/REVOKE/GRANT statements, + snapshot round-trip). +- **Integration:** acquire/release state machine with mocked spawner + DB session layers — + hit path, each miss reason (no container, no metadata, running container, key mismatch), + and the escape hatch on reset failure (asserts rm + cold fallback ordering). +- **Live (`*.live.test.ts`, gated per repo policy, golden path only):** one scenario — + cold acquire, release, warm acquire on the same key asserts the container id is reused + and a role created between the two runs is gone after reset. + +## Sequencing + +1. **A** — readiness gate (`fix(cli)`, independently shippable, benefits every provision). +2. **B** — entrypoint `exec` (`fix(cli)` or folded into C's PR; required for C's fast + release). +3. **C** — warm cache behind the env-var gate (`feat(cli)`), then a follow-up to default it + on after bake time. + +Each item follows the repo's RED→GREEN rule (failing test first, capture the failure in +the commit/PR). PR titles: conventional commits, `(cli)` scope. Per repo policy, no test +plans in PR descriptions. + +## Non-goals + +- No changes to `supabase/pg-toolbelt` / pg-delta. The isolated shadow contract + (pre-provisioned, baseline rows tolerated, cluster DDL allowed) already accommodates a + reused container. +- No changes to the co-located shadow path (pg-delta's `provisionCoLocatedShadow` is + already sub-second). +- No change to the migrations-catalog cache or its `setupInputsToken` (its service-tag + omission is noted above as a possible separate issue — surface to a human, don't fix + here). +- No healthcheck-config changes on the container itself. + +## Appendix — reproducing the benchmark + +The numbers above came from a throwaway harness (not committed): generate the exact +entrypoint script by importing the `LEGACY_START_DB_*_SQL` template constants and +concatenating them the way `legacyPostgresEntrypointScriptPg15` does (with +`-c max_worker_processes=0`), `docker create` with the spec's env/healthcheck flags, +`docker cp` the pgsodium root key to `/etc/postgresql-custom/pgsodium_root.key`, start, +then poll (a) `pg_isready` via `docker exec`, (b) `docker inspect +'{{.State.Health.Status}}'`. Run the three one-shot jobs with the env built by +`legacyBuildRealtimeEnv` / `legacyStartStorageMigrateEnv` / `legacyStartAuthMigrateEnv` +against the shadow's 12-char container id as `DB_HOST` on the same network. Warm numbers: +`docker stop` + `docker start` the same container and re-poll. From 9d999d263a0fe40ccd46f48e4d989e90acfc051a Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 10:46:51 +0200 Subject: [PATCH 36/82] fix(cli): gate shadow readiness on a direct connect probe (~6.5s/provision) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shadow container's healthcheck runs on a 10s interval with no start_period/start_interval, so the Docker-health gate the CLI polled (legacyWaitForHealthyServices) cannot report healthy before t+10s while Postgres accepts connections at ~3.5s — ~6.5s of pure dead wait on every shadow provision (db diff, db pull, declarative sync, catalog cache miss). Add legacyWaitForShadowReady (health-check.ts): same 1s constant backoff and healthTimeoutSeconds budget, but each round checks the container is still running (exited => fail fast) and then attempts a short scoped connect (2s dial bound); success means ready. Timeout keeps the exact LegacyHealthCheckTimeoutError shape and container-log dump. Used by legacyPrepareRawShadow and legacyPrepareShadowSource only — the long-running db container's Docker-health wait is untouched, and the container's healthcheck config is unchanged. RED first: the four legacyWaitForShadowReady scenarios failed at "legacyWaitForShadowReady is not a function" before the implementation (health-check.unit.test.ts). Follow-ups noted in the plan doc: db diff --use-pgadmin and migration squash still gate their shadows on Docker health. Workstream A of apps/cli/docs/shadow-db-startup-perf-plan.md. Co-Authored-By: Claude Fable 5 --- apps/cli/docs/shadow-db-startup-perf-plan.md | 53 +++-- .../commands/db/diff/diff.integration.test.ts | 62 ++++-- .../commands/db/pull/pull.integration.test.ts | 16 +- .../db/shared/legacy-shadow-source.ts | 16 +- .../shared/db-bootstrap/health-check.ts | 126 +++++++++++ .../db-bootstrap/health-check.unit.test.ts | 203 ++++++++++++++++++ .../shared/db-bootstrap/shadow-database.ts | 38 ++-- 7 files changed, 444 insertions(+), 70 deletions(-) diff --git a/apps/cli/docs/shadow-db-startup-perf-plan.md b/apps/cli/docs/shadow-db-startup-perf-plan.md index 43fc03d7a2..d9ccdc69cc 100644 --- a/apps/cli/docs/shadow-db-startup-perf-plan.md +++ b/apps/cli/docs/shadow-db-startup-perf-plan.md @@ -1,8 +1,8 @@ # Shadow database startup performance — implementation plan -Status: **planned, not implemented**. This document is a handoff spec: it contains the -measured baseline, the agreed design decisions, and file-level work items. An implementing -agent should be able to execute it without re-deriving the investigation. +Status: **workstream A implemented; B and C planned**. This document is a handoff spec: it +contains the measured baseline, the agreed design decisions, and file-level work items. An +implementing agent should be able to execute it without re-deriving the investigation. ## Context @@ -11,7 +11,7 @@ agent should be able to execute it without re-deriving the investigation. storage / auth one-shot migration jobs), then handed to pg-delta as the isolated shadow (`pgdelta … --isolated-shadow`, loader mode `isolatedCluster`). pg-delta requires this shadow on a **different Postgres lineage** than the target, so the container is on the hot -path of *every* plan — the migrations-catalog cache cannot make it go away. +path of _every_ plan — the migrations-catalog cache cannot make it go away. Today the shadow is created cold and destroyed (`docker rm -f -v`) on every run. @@ -22,16 +22,16 @@ cache), reproducing the exact container shape `legacyBuildShadowPostgresContaine produces (same entrypoint heredoc script, env, healthcheck flags) and the exact one-shot job env from `db-setup.ts`. Two cold samples, consistent within ~1s: -| Phase | Time | -|---|---| -| `docker create` + secret `docker cp` + `docker start` | ~0.4s | -| Postgres accepting connections (initdb + bundled init SQL) | ~3.4–4.5s | -| Docker healthcheck reports `healthy` (**the CLI's current gate**) | ~10.2s | -| One-shot realtime job (Elixir boot + tenant seed) | ~6.5s | -| One-shot storage migrate job | ~2.5s | -| One-shot auth (`gotrue migrate`) job | ~0.7s | -| Revoke API privileges + `CREATE DATABASE contrib_regression TEMPLATE postgres` | ~0.3s | -| **Total cold provision** (excl. image pulls, excl. user migrations) | **~20.5s** | +| Phase | Time | +| ------------------------------------------------------------------------------ | ---------- | +| `docker create` + secret `docker cp` + `docker start` | ~0.4s | +| Postgres accepting connections (initdb + bundled init SQL) | ~3.4–4.5s | +| Docker healthcheck reports `healthy` (**the CLI's current gate**) | ~10.2s | +| One-shot realtime job (Elixir boot + tenant seed) | ~6.5s | +| One-shot storage migrate job | ~2.5s | +| One-shot auth (`gotrue migrate`) job | ~0.7s | +| Revoke API privileges + `CREATE DATABASE contrib_regression TEMPLATE postgres` | ~0.3s | +| **Total cold provision** (excl. image pulls, excl. user migrations) | **~20.5s** | Additional measurements that motivate the work items: @@ -43,9 +43,9 @@ Additional measurements that motivate the work items: - `docker start` of an already-initialized shadow container → connectable in **~1.0s** (also ~1s after a SIGKILL'd stop, via WAL recovery). - `CREATE DATABASE … TEMPLATE ` on the baseline state: **~0.2s**. `DROP DATABASE … - WITH (FORCE)`: ~0.1s. +WITH (FORCE)`: ~0.1s. - `docker stop` on the current container takes **10.3s**: the entrypoint is `sh -c "… && - docker-entrypoint.sh …"`, so PID 1 is `sh`, which does not forward SIGTERM; Docker waits +docker-entrypoint.sh …"`, so PID 1 is `sh`, which does not forward SIGTERM; Docker waits the 10s grace period and SIGKILLs. Target end state: **~1.5–2s** per plan on a warm cache hit; **~14s** cold (health gate @@ -80,7 +80,7 @@ fixed); cold path otherwise unchanged. Hash (sha256, stable field order) of every input baked into the cluster during cold provisioning: -- resolved `supabase/postgres` image tag (full tag, e.g. `17.6.1.158` — *not* major +- resolved `supabase/postgres` image tag (full tag, e.g. `17.6.1.158` — _not_ major version) after registry/pin resolution; - resolved one-shot job image tags — `realtime`, `storage`, `auth` via `legacyResolvePinnedImage` + `serviceVersionOverrides` — each included **only when its @@ -100,7 +100,16 @@ not matter for the catalog cache (out of scope here — flag it to a human if to the container cache it definitely matters, because the jobs write versioned schema state (`auth`, `storage`, `_realtime`) into the cluster. -## Workstream A — shadow readiness gate (independent, ship first) +## Workstream A — shadow readiness gate (independent, ship first) — IMPLEMENTED + +Shipped as `legacyWaitForShadowReady` in +`apps/cli/src/legacy/shared/db-bootstrap/health-check.ts`, consumed by +`legacyPrepareRawShadow` (`shadow-database.ts`) and `legacyPrepareShadowSource` +(`commands/db/shared/legacy-shadow-source.ts`). Two shadow health-waits deliberately stayed +on the Docker-health gate and are follow-up candidates: `db diff --use-pgadmin` +(`diff.handler.ts`) and `migration squash` (`squash.handler.ts`) — both provision the same +shadow container and pay the same ~6.5s. The spec below is retained as the record of what +was agreed. **Problem:** `legacyPrepareRawShadow` and `legacyPrepareShadowSource` gate on `legacyWaitForHealthyServices` (1s poll of `docker inspect` health), but the container @@ -174,7 +183,7 @@ once proven. `com.supabase.cli.shadow-cache-key=` label. 2. Run the existing baseline setup (`legacySetupShadowDatabase` / `legacyMigrateShadowDatabase` path unchanged up to the baseline; user migrations are - *not* part of the cached state — see reset protocol). + _not_ part of the cached state — see reset protocol). 3. **Snapshot for reuse**, immediately after the baseline (before migrations/declarative load): - `CREATE DATABASE _supabase_shadow_base TEMPLATE postgres` (requires no other @@ -193,13 +202,13 @@ once proven. ### Warm acquire (cache hit) 1. Compute the key; look up the metadata file **and** the container (`docker ps -a - --filter label=com.supabase.cli.shadow-cache-key=`). Either missing, or container +--filter label=com.supabase.cli.shadow-cache-key=`). Either missing, or container already running (another concurrent run owns it) ⇒ miss (concurrent case: fall through to a one-off uncached cold shadow, do not remove the cached one). 2. `docker start` (~1s), wait via workstream A's probe. 3. **Reset to pristine:** - reverse the role delta vs. the snapshot: drop roles not in the snapshot (`DROP OWNED - BY` in each affected DB is unnecessary once `postgres` is recreated — drop role after +BY` in each affected DB is unnecessary once `postgres` is recreated — drop role after the DB recreate below to avoid dependency errors; simplest safe order: recreate DBs first, then `DROP ROLE`), revoke added memberships, re-grant removed ones, delete cluster-wide `pg_db_role_setting` rows not in the snapshot; @@ -222,7 +231,7 @@ once proven. catalog cache uses if one exists; otherwise overwrite-on-write is enough (one file per key, old keys' containers removed on key mismatch — enumerate by label, keep only the current key's container). -- **Leak window:** unlike today's `--rm` shadow, a crashed CLI leaves a *stopped, labeled* +- **Leak window:** unlike today's `--rm` shadow, a crashed CLI leaves a _stopped, labeled_ container. That is the cache working as intended; `supabase stop` and the key-mismatch sweep both reclaim it. diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index e1879d844c..54d410debe 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -40,6 +40,7 @@ import { type LegacyDbSession, type LegacyPgConnInput, } from "../../../shared/legacy-db-connection.service.ts"; +import { LegacyDbConnectError } from "../../../shared/legacy-db-connection.errors.ts"; import { LegacyDockerRunError } from "../../../shared/legacy-docker-run.errors.ts"; import { LegacyDockerRun, @@ -89,8 +90,14 @@ interface SetupOpts { // When set, the shadow container never reports healthy — for the interrupt-during- // health-wait regression coverage (review: PRRT_kwDOErm0O86XMrID). See // `mockLegacyShadowContainerCliSpawner`'s own doc comment for why this is required - // (not `Effect.never`) to observe a genuinely suspended retry loop. + // (not `Effect.never`) to observe a genuinely suspended retry loop. Only the + // `--use-pgadmin` branch still gates on the Docker healthcheck; the shadow-source + // branch gates on `neverConnectableShadow` below instead. readonly neverHealthyShadow?: boolean; + // When set, every connect to the shadow's own port is refused, so the readiness gate + // (`legacyWaitForShadowReady`) keeps polling — the shadow-source branch's equivalent of + // `neverHealthyShadow`, since that wait no longer consults the Docker healthcheck. + readonly neverConnectableShadow?: boolean; // `LegacyCliConfig.projectId` (the `SUPABASE_PROJECT_ID` env-only reader). Defaults // to `Option.some("test")`; pass `Option.none()` to exercise the // config.toml/workdir-basename fallback `legacyResolveLocalProjectId` provides for @@ -135,14 +142,28 @@ const alwaysReadyHttpClientLayer = Layer.succeed( ), ); -/** Records every `LegacyDbConnection.connect` target's database name, and every `exec`/`query` SQL run against it. */ -function fakeShadowDbConnection() { +/** `[db] shadow_port`'s schema default — the port every connect to the shadow itself dials. */ +const LEGACY_SHADOW_PORT = 54320; + +/** + * Records every `LegacyDbConnection.connect` target's database name, and every `exec`/`query` + * SQL run against it. + * + * `neverConnectableShadow` makes every connect to the SHADOW port fail (leaving the local + * target's own connects untouched) — the shadow's readiness gate is now a direct connect probe + * (`legacyWaitForShadowReady`), so a shadow that never accepts a connection is what keeps a + * provisioning fiber genuinely suspended inside that retry loop. + */ +function fakeShadowDbConnection(opts: { readonly neverConnectableShadow?: boolean } = {}) { const connectedDatabases: Array = []; const execCalls: Array = []; const layer = Layer.succeed(LegacyDbConnection, { connect: (cfg: LegacyPgConnInput) => - Effect.sync(() => { + Effect.gen(function* () { connectedDatabases.push(cfg.database); + if (opts.neverConnectableShadow === true && cfg.port === LEGACY_SHADOW_PORT) { + return yield* Effect.fail(new LegacyDbConnectError({ message: "connection refused" })); + } const session: LegacyDbSession = { exec: (sql) => Effect.sync(() => { @@ -172,7 +193,9 @@ function setup(workdir: string, opts: SetupOpts = {}) { dbNotRunning: opts.dbNotRunning ?? false, dbInspectFailsWith: opts.dbInspectFailsWith, }); - const shadowDbConnection = fakeShadowDbConnection(); + const shadowDbConnection = fakeShadowDbConnection({ + neverConnectableShadow: opts.neverConnectableShadow ?? false, + }); const explicitDiffCalls: LegacyPgDeltaExplicitDiffInput[] = []; const databaseDiffCalls: LegacyPgDeltaDatabaseDiffInput[] = []; @@ -1910,37 +1933,36 @@ describe("legacy db diff", () => { }); it.live( - "removes the shadow container on a SIGINT-style interruption during the health wait, without waiting for the health-check timeout", + "removes the shadow container on a SIGINT-style interruption during the readiness wait, without waiting for the readiness timeout", () => { // Regression test for the acquireUseRelease restructuring (review: // PRRT_kwDOErm0O86XMrID): an earlier shape passed the ENTIRE // `legacyPrepareShadowSource` (create -> health-wait -> migrate -> // declarative-apply) as `acquireUseRelease`'s `acquire`, which Effect's // `uninterruptibleMask` (no `restore` around `acquire`) made completely - // uninterruptible — a SIGINT landing during the health wait (which can run for - // up to 30 real seconds, `LEGACY_HEALTH_CHECK_TIMEOUT_SECONDS`) was silently - // swallowed until the health check gave up on its own. `acquire` is now ONLY - // `legacyCreateShadowDatabase` - // (container creation); the health wait runs inside the interruptible `use` - // phase instead, so a `Fiber.interrupt` here must land promptly. - const s = setup(tmp.current, { neverHealthyShadow: true }); + // uninterruptible — a SIGINT landing during the readiness wait (which can run + // for up to 30 real seconds, `LEGACY_HEALTH_CHECK_TIMEOUT_SECONDS`) was silently + // swallowed until the wait gave up on its own. `acquire` is now ONLY + // `legacyCreateShadowDatabase` (container creation); the readiness wait runs + // inside the interruptible `use` phase instead, so a `Fiber.interrupt` here must + // land promptly. + const s = setup(tmp.current, { neverConnectableShadow: true }); return Effect.gen(function* () { const fiber = yield* legacyDbDiff(flags()).pipe( Effect.provide(s.layer), Effect.forkChild({ startImmediately: true }), ); - // Wait until the shadow's own health check has actually probed the - // never-healthy container at least once — proving the fiber is genuinely - // suspended inside `legacyWaitForHealthyServices`'s retry loop, not merely - // past the `create` call. - while (!s.shadowSpawned.some((c) => c.args[0] === "container" && c.args[1] === "inspect")) { + // Wait until the shadow's readiness gate has actually refused a connect at + // least once — proving the fiber is genuinely suspended inside + // `legacyWaitForShadowReady`'s retry loop, not merely past the `create` call. + while (s.shadowConnectedDatabases.length === 0) { yield* Effect.sleep("5 millis"); } // `Fiber.interrupt` only resolves once the target fiber (and its finalizers, // including `legacyRemoveShadowDatabase`) has fully completed — if `acquire` - // still covered the health wait, this call would hang for up to 30 real + // still covered the readiness wait, this call would hang for up to 30 real // seconds (or until this test's own timeout), instead of resolving as soon - // as the in-flight probe's own subprocess call returns. + // as the in-flight probe returns. yield* Fiber.interrupt(fiber); expect(s.shadowSpawned.filter((c) => c.args[0] === "create")).toHaveLength(1); expect(s.shadowSpawned.filter((c) => c.args[0] === "rm")).toHaveLength(1); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 7cd7e431b2..abc622ea88 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -318,6 +318,8 @@ function setup(workdir: string, opts: SetupOpts = {}) { const execLog: string[] = []; const historyUpserts: ReadonlyArray[] = []; const connectedDatabases: Array = []; + /** Same connects as {@link connectedDatabases}, keeping the port that tells target from shadow apart. */ + const connectTargets: Array<{ readonly database: string; readonly port: number }> = []; // The resolver mock's own target connection always dials port 5432; the native // shadow (platform baseline, `CREATE_TEMPLATE`, migrations, and — on the // declarative branch — the `contrib_regression` override) always dials the @@ -346,6 +348,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { connect: (cfg: { readonly database: string; readonly port: number }) => Effect.sync(() => { connectedDatabases.push(cfg.database); + connectTargets.push({ database: cfg.database, port: cfg.port }); return cfg.port === TARGET_PORT ? targetSession : shadowSession; }), }); @@ -467,6 +470,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { historyUpserts, execLog, connectedDatabases, + connectTargets, poolerFallbackCalls, resolveCalls, dumpCalls, @@ -872,9 +876,15 @@ describe("legacy db pull", () => { files: ["schemas/public/t.sql"], }); // Declarative mode's bare shadow (`legacyPrepareRawShadow`) never connects to set - // up a platform baseline or `contrib_regression` template — the only connect is - // the top-level target connect (`resolved.conn`, database "postgres"). - expect(s.connectedDatabases).toEqual(["postgres"]); + // up a platform baseline or `contrib_regression` template. The only connects are + // the top-level target connect (`resolved.conn`, port 5432, database "postgres") + // and the shadow's own readiness probe on the shadow port — a single short-lived + // connect that is now the provisioning gate (`legacyWaitForShadowReady`) in place + // of waiting on the shadow container's 10s-interval Docker healthcheck. + expect(s.connectTargets).toEqual([ + { database: "postgres", port: 5432 }, + { database: "postgres", port: 54320 }, + ]); expect(s.shadowSpawned.filter((call) => call.args[0] === "create")).toHaveLength(1); expect(s.shadowSpawned.filter((call) => call.args[0] === "rm")).toHaveLength(1); }).pipe(Effect.provide(s.layer)); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts index bfc54450bf..5b144ac1de 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts @@ -42,7 +42,7 @@ import { import type { LegacyDockerRun } from "../../../shared/legacy-docker-run.service.ts"; import type { LegacyImagePrepullError } from "../../../shared/db-bootstrap/image-prepull.ts"; import type { LegacyHealthCheckTimeoutError } from "../../../shared/db-bootstrap/health-check.ts"; -import { legacyWaitForHealthyServices } from "../../../shared/db-bootstrap/health-check.ts"; +import { legacyWaitForShadowReady } from "../../../shared/db-bootstrap/health-check.ts"; import { legacySeedGlobals } from "../../../shared/legacy-migration-apply.ts"; import { LEGACY_BAD_PATTERN_MESSAGE, legacyPathMatch } from "../../../shared/legacy-path-match.ts"; import { legacyToPostgresURL } from "../../../shared/legacy-postgres-url.ts"; @@ -95,7 +95,10 @@ export type LegacyPrepareShadowSourceError = /** * Port of Go's `PrepareShadowSource` (`apps/cli-go/internal/db/diff/shadow.go:37-91`): - * health-wait against an already-`legacyCreateShadowDatabase`-created shadow -> + * readiness-wait against an already-`legacyCreateShadowDatabase`-created shadow (a direct + * connect probe, `legacyWaitForShadowReady` — NOT the Docker-health gate the long-running `db` + * container uses, which the shadow's own 10s-interval healthcheck cannot satisfy until well + * after Postgres is connectable; see that function's own doc comment) -> * `MigrateShadowDatabase` (platform baseline + local migrations + the `contrib_regression` * template database) -> build the diff-source config -> for legacy local targets, the * declarative-schema override branch. Pg-delta next always compares that migrations shadow @@ -145,10 +148,6 @@ export const legacyPrepareShadowSource = ( Effect.gen(function* () { const { containerId } = handle; - yield* legacyWaitForHealthyServices(spawner, [containerId], { - timeoutSeconds: input.healthTimeoutSeconds, - }); - const connConfig: LegacyPgConnInput = { host: input.hostname, port: input.shadowPort, @@ -156,6 +155,11 @@ export const legacyPrepareShadowSource = ( password: input.password, database: "postgres", }; + + yield* legacyWaitForShadowReady(spawner, containerId, connConfig, { + timeoutSeconds: input.healthTimeoutSeconds, + }); + const migrateShadow = input.migrationMode === "pgdelta-next" ? legacyMigrateNextShadowDatabase diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts index ab421387e2..68b86e15da 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts @@ -24,6 +24,7 @@ import { legacySpawnContainerCliWithRuntime, type LegacyContainerRuntime, } from "../legacy-container-cli.ts"; +import { LegacyDbConnection, type LegacyPgConnInput } from "../legacy-db-connection.service.ts"; import { legacyInspectContainerState } from "../legacy-docker-lifecycle.ts"; import { legacyKongAuthHeaders } from "../legacy-kong-auth.ts"; @@ -437,3 +438,128 @@ export function legacyWaitForHealthyServices( ); }); } + +/** + * Bounds a single readiness connect so a dial that hangs (rather than being + * refused outright) cannot swallow the whole poll budget. Explicit rather than + * inherited from the driver's own local default (`legacy-db-connection.sql-pg. + * layer.ts` — `cfg.connectTimeoutSeconds ?? (isLocal ? 2 : 10)`), which happens + * to be the same 2 seconds: this probe's budget is a property of the polling + * loop, not of whichever driver layer answers it. + */ +const LEGACY_SHADOW_READY_CONNECT_TIMEOUT_SECONDS = 2; + +/** + * One round's verdict. `fatal` is what makes an exited container fail fast + * instead of burning the remaining budget: nothing about a dead container can + * change on a later round, whereas a refused connect (or a transient `docker + * container inspect` failure) is just "not ready yet". + */ +interface LegacyShadowReadyFailure { + readonly reason: string; + readonly fatal: boolean; +} + +const legacyShadowNotReady = (reason: string): LegacyShadowReadyFailure => ({ + reason, + fatal: false, +}); + +/** + * A single short-lived connect attempt against the shadow, dialled exactly the + * way `legacyConnectShadowDatabase` (`shadow-database.ts`) dials it — same + * `isLocal`/`dnsResolver` pair, so a config that authenticates for the probe + * authenticates for the real connection too. `Effect.scoped` closes the session + * the moment the probe resolves: the caller opens (and owns) its own connection + * afterwards through `legacyConnectShadowDatabase`. + */ +const legacyProbeShadowConnect = ( + connConfig: LegacyPgConnInput, +): Effect.Effect => + Effect.scoped( + Effect.gen(function* () { + const dbConnection = yield* LegacyDbConnection; + yield* dbConnection.connect( + { ...connConfig, connectTimeoutSeconds: LEGACY_SHADOW_READY_CONNECT_TIMEOUT_SECONDS }, + { isLocal: true, dnsResolver: "native" }, + ); + }), + ).pipe(Effect.mapError((cause) => legacyShadowNotReady(cause.message))); + +export interface LegacyWaitForShadowReadyOptions { + readonly timeoutSeconds?: number; +} + +/** + * The shadow database's readiness gate — {@link legacyWaitForHealthyServices}'s + * counterpart for the ONE container whose Docker healthcheck cannot answer in + * time. The shadow's healthcheck is `interval=10s` with no + * `start_period`/`start_interval` (`postgres.service.ts`, deliberately left + * unchanged — other tooling reads that config), so Docker's very first probe + * runs at t+10s while Postgres has been accepting connections since ~3.5s: + * gating on `Health.Status` spends ~6.5s per provision waiting for a verdict + * that is already knowable. `--health-start-interval` would fix the container + * side, but it needs Docker Engine 25+/API 1.44 and is not reliably supported + * by Podman (which `spawnContainerCli` falls back to), so the CLI-side wait + * asks Postgres directly instead. + * + * Each round, on the same 1-second constant backoff and the same + * `timeoutSeconds` budget as the health gate: + * + * 1. {@link legacyInspectContainerState} — still `running`? This preserves the + * crash detection the health gate provided; an exited container fails + * immediately rather than at the end of the budget. + * 2. a short {@link legacyProbeShadowConnect} — success ⇒ ready, return now. + * + * Failure shape is deliberately identical to the health gate's: the same + * {@link LegacyHealthCheckTimeoutError}, the same `unhealthy` payload, and the + * same `docker logs` dump teed to stderr on the way out, so a broken shadow + * still surfaces exactly what it surfaced before. + */ +export function legacyWaitForShadowReady( + spawner: Spawner, + containerId: string, + connConfig: LegacyPgConnInput, + opts: LegacyWaitForShadowReadyOptions = {}, +): Effect.Effect { + const timeoutSeconds = opts.timeoutSeconds ?? LEGACY_HEALTH_CHECK_TIMEOUT_SECONDS; + + const probe: Effect.Effect = Effect.gen( + function* () { + const state = yield* legacyInspectContainerState(spawner, containerId).pipe( + Effect.mapError((cause) => legacyShadowNotReady(cause.message)), + ); + if (!state.running) { + return yield* Effect.fail({ + reason: `container is not running: ${state.status}`, + fatal: true, + }); + } + yield* legacyProbeShadowConnect(connConfig); + }, + ); + + // Same policy as `legacyWaitForHealthyServices` (Go's + // `NewBackoffPolicy(ctx, timeout)`): a 1-second constant delay, capped at + // `timeoutSeconds` retries after the initial attempt. + const schedule = Schedule.max([Schedule.spaced("1 seconds"), Schedule.recurs(timeoutSeconds)]); + + return probe.pipe( + Effect.retry({ schedule, while: (failure) => !failure.fatal }), + Effect.catch((failure) => + Effect.gen(function* () { + // Go skips this dump on context cancellation (`start.go:215`) — an + // interrupted fiber never reaches this handler, so no separate check. + yield* legacyDumpContainerLogs(spawner, containerId); + return yield* Effect.fail( + new LegacyHealthCheckTimeoutError({ + // The health gate's own ` ` joining, so both waits read + // identically on stderr. + message: `${containerId} ${failure.reason}`, + unhealthy: [{ containerId, reason: failure.reason }], + }), + ); + }), + ), + ); +} diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts index 6e64b732e5..1f1b9a8b88 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts @@ -6,9 +6,16 @@ import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; import * as TestClock from "effect/testing/TestClock"; +import { LegacyDbConnectError } from "../legacy-db-connection.errors.ts"; +import { + LegacyDbConnection, + type LegacyDbSession, + type LegacyPgConnInput, +} from "../legacy-db-connection.service.ts"; import { LegacyHealthCheckTimeoutError, legacyWaitForHealthyServices, + legacyWaitForShadowReady, type LegacyHealthCheckPostgrestGateway, } from "./health-check.ts"; @@ -774,3 +781,199 @@ describe("legacyWaitForHealthyServices", () => { ); }); }); + +const SHADOW_CONTAINER_ID = "abc123456789shadow"; + +const shadowConnConfig: LegacyPgConnInput = { + host: "127.0.0.1", + port: 54320, + user: "postgres", + password: "postgres", + database: "postgres", +}; + +/** + * A `LegacyDbConnection` whose `connect` fails the first `failTimes` calls + * (`Number.POSITIVE_INFINITY` never succeeds), recording every dialled config + * and how many probe sessions were released. `closedSessions` is what proves + * the readiness probe hands nothing back to the caller: the downstream code + * opens its own connection through `legacyConnectShadowDatabase` afterwards. + */ +function mockShadowDbConnection(opts: { readonly failTimes?: number } = {}) { + const failTimes = opts.failTimes ?? 0; + const session: LegacyDbSession = { + exec: () => Effect.void, + query: () => Effect.succeed([]), + extensionExists: () => Effect.succeed(false), + copyToCsv: () => Effect.succeed(new Uint8Array()), + queryRaw: () => Effect.succeed({ fields: [], rows: [], commandTag: "" }), + }; + const attempts: Array = []; + let closedSessions = 0; + const layer = Layer.succeed(LegacyDbConnection, { + connect: (cfg) => + Effect.gen(function* () { + attempts.push(cfg); + if (attempts.length <= failTimes) { + return yield* Effect.fail(new LegacyDbConnectError({ message: "connection refused" })); + } + yield* Effect.addFinalizer(() => + Effect.sync(() => { + closedSessions += 1; + }), + ); + return session; + }), + }); + return { + layer, + attempts, + get closedSessions() { + return closedSessions; + }, + }; +} + +/** The timeout path tees the container's logs to the real stderr — swallow them. */ +function withSilencedStderr(effect: Effect.Effect): Effect.Effect { + return Effect.suspend(() => { + const originalWrite = globalThis.process.stderr.write.bind(globalThis.process.stderr); + globalThis.process.stderr.write = (() => true) as typeof globalThis.process.stderr.write; + return effect.pipe( + Effect.ensuring( + Effect.sync(() => { + globalThis.process.stderr.write = originalWrite; + }), + ), + ); + }); +} + +const inspectCalls = (mock: ReturnType) => + mock.spawned.filter((args) => args[0] === "container" && args[1] === "inspect"); + +describe("legacyWaitForShadowReady", () => { + it.effect( + "resolves on the first successful connect, while Docker still reports the healthcheck as starting", + () => + Effect.gen(function* () { + // The shadow container's healthcheck runs on a 10s interval with no + // start period, so Docker cannot report `healthy` before t+10s even + // though Postgres accepts connections at ~3.5s — the whole point of + // this wait is that the health status is never consulted at all. + const mock = mockHealthSpawner(() => runningStarting); + const db = mockShadowDbConnection(); + + const exit = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 30 }, + ).pipe(Effect.provide(db.layer), Effect.exit); + + expect(Exit.isSuccess(exit)).toBe(true); + expect(inspectCalls(mock)).toHaveLength(1); + expect(db.attempts).toHaveLength(1); + // Bounded, so a hung dial cannot eat a whole poll round's budget. + expect(db.attempts[0]?.connectTimeoutSeconds).toBe(2); + // Released immediately: the caller opens its own connection afterwards. + expect(db.closedSessions).toBe(1); + }), + ); + + it.effect("keeps polling a refused connect on a 1-second backoff until it succeeds", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting); + const db = mockShadowDbConnection({ failTimes: 2 }); + + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 30 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + yield* TestClock.adjust("1 seconds"); + const exit = yield* Fiber.await(fiber); + + expect(Exit.isSuccess(exit)).toBe(true); + // 2 refused attempts, then the 3rd that connects — a refused connect is + // "not ready yet", never an error in its own right. + expect(db.attempts).toHaveLength(3); + expect(inspectCalls(mock)).toHaveLength(3); + expect(db.closedSessions).toBe(1); + }), + ); + + it.effect( + "fails with LegacyHealthCheckTimeoutError and dumps the container's logs when it never becomes connectable", + () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 2 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.unhealthy).toEqual([ + { containerId: SHADOW_CONTAINER_ID, reason: "connection refused" }, + ]); + expect(error.message).toBe(`${SHADOW_CONTAINER_ID} connection refused`); + // 1 initial attempt + `timeoutSeconds` retries, same budget as the + // Docker-health gate this replaces. + expect(db.attempts).toHaveLength(3); + expect( + mock.spawned.some((args) => args[0] === "logs" && args[1] === SHADOW_CONTAINER_ID), + ).toBe(true); + }), + ); + + it.effect("fails fast, well inside the budget, once the container has exited", () => + Effect.gen(function* () { + // Running on the first round, gone on every round after it. + const mock = mockHealthSpawner((_id, callIndex) => + callIndex === 0 ? runningStarting : notRunning, + ); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 30 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.unhealthy).toEqual([ + { containerId: SHADOW_CONTAINER_ID, reason: "container is not running: exited" }, + ]); + // A dead container can never become connectable — one second in, not 30. + expect(inspectCalls(mock)).toHaveLength(2); + expect(db.attempts).toHaveLength(1); + expect( + mock.spawned.some((args) => args[0] === "logs" && args[1] === SHADOW_CONTAINER_ID), + ).toBe(true); + }), + ); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index 0fe542c3e7..8d5bbd339e 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -46,7 +46,6 @@ import { type Path, type Scope, } from "effect"; -import type * as HttpClient from "effect/unstable/http/HttpClient"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import { Output } from "../../../shared/output/output.service.ts"; @@ -77,7 +76,7 @@ import { } from "./container-lifecycle.ts"; import type { LegacyImagePrepullError } from "./image-prepull.ts"; import type { LegacyHealthCheckTimeoutError } from "./health-check.ts"; -import { legacyWaitForHealthyServices } from "./health-check.ts"; +import { legacyWaitForShadowReady } from "./health-check.ts"; import type { LegacyLocalDbContainerInputs } from "./local-container-inputs.ts"; import { legacyListLocalMigrationPaths } from "../legacy-migration-history.ts"; import { legacyToPostgresURL } from "../legacy-postgres-url.ts"; @@ -480,26 +479,31 @@ export function legacyShadowRunInputFromLocalContainerInputs( } /** - * Port of Go's `PrepareRawShadow` (`apps/cli-go/internal/db/diff/shadow.go:93-116`): health-wait - * against an already-{@link legacyCreateShadowDatabase}-created shadow (created + healthy, no - * platform baseline or migrations applied) — used inline (`db pull --declarative`'s empty - * declarative-export source), not the `ok`-sentinel error-path pattern + * Port of Go's `PrepareRawShadow` (`apps/cli-go/internal/db/diff/shadow.go:93-116`): readiness + * wait against an already-{@link legacyCreateShadowDatabase}-created shadow (created + accepting + * connections, no platform baseline or migrations applied) — used inline (`db pull + * --declarative`'s empty declarative-export source), not the `ok`-sentinel error-path pattern * `legacy-shadow-source.ts`'s `legacyPrepareShadowSource` uses, since there is only ONE step - * here that can fail (the health wait) rather than several. Lives here (not + * here that can fail (the readiness wait) rather than several. Lives here (not * `legacy-shadow-source.ts`) because it has zero pg-delta/declarative dependency — see this * module's own header. * + * Gates on {@link legacyWaitForShadowReady}, NOT on the Docker-health + * `legacyWaitForHealthyServices` the long-running `db` container still uses: the shadow's + * own healthcheck cannot report `healthy` before its first 10-second-interval probe, ~6.5s after + * Postgres is already connectable — see that function's own doc comment. + * * Deliberately does NOT call {@link legacyCreateShadowDatabase} itself — the caller does, as the * `acquire` of an `Effect.acquireUseRelease` whose `use` phase is this function (see * `diff.handler.ts`/`pull.handler.ts`'s call sites). Go's `PrepareRawShadow` threads a single - * cancellable `ctx` through both creation and the health wait, so a SIGINT can interrupt either; - * an earlier shape here instead passed the WHOLE create-then-health-wait effect as `acquire`, + * cancellable `ctx` through both creation and the readiness wait, so a SIGINT can interrupt + * either; an earlier shape here instead passed the WHOLE create-then-wait effect as `acquire`, * which Effect's `uninterruptibleMask` (`acquireUseRelease(acquire, use, release) => * uninterruptibleMask(restore => flatMap(acquire, a => onExitPrimitive(restore(use(a)), ...)))`) - * makes entirely uninterruptible — a SIGINT during the health wait (which can run for up to + * makes entirely uninterruptible — a SIGINT during the readiness wait (which can run for up to * `healthTimeoutSeconds`) was silently swallowed until the wait finished or timed out on its * own, unlike Go. Splitting `legacyCreateShadowDatabase` out as the (brief, Docker-API-bound) - * `acquire` and keeping this health-wait as part of the interruptible `use` restores that parity + * `acquire` and keeping this wait as part of the interruptible `use` restores that parity * — a SIGINT here now lands immediately, same as Go's ctx cancellation, while * `legacyRemoveShadowDatabase` still runs as the `release` finalizer regardless of how `use` * exits (review: PRRT_kwDOErm0O86XMrID). @@ -508,16 +512,9 @@ export const legacyPrepareRawShadow = ( spawner: Spawner, handle: LegacyShadowDatabaseHandle, input: LegacyPrepareRawShadowInput, -): Effect.Effect< - LegacyShadowSourceResult, - LegacyHealthCheckTimeoutError, - Output | LegacyDockerRun | RuntimeInfo | HttpClient.HttpClient -> => +): Effect.Effect => Effect.gen(function* () { const { containerId } = handle; - yield* legacyWaitForHealthyServices(spawner, [containerId], { - timeoutSeconds: input.healthTimeoutSeconds, - }); const connConfig: LegacyPgConnInput = { host: input.hostname, port: input.shadowPort, @@ -525,6 +522,9 @@ export const legacyPrepareRawShadow = ( password: input.password, database: "postgres", }; + yield* legacyWaitForShadowReady(spawner, containerId, connConfig, { + timeoutSeconds: input.healthTimeoutSeconds, + }); return { container: containerId, sourceUrl: legacyToPostgresURL(connConfig), From 3c2d4e4b44a7a3eb6a3610203c5ba8c6c5c78ba8 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 10:55:38 +0200 Subject: [PATCH 37/82] fix(cli): exec the postgres entrypoint so containers stop in ~1s MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The generated entrypoint script runs as `sh -c "... && docker-entrypoint.sh postgres ..."`, leaving sh as PID 1. sh does not forward SIGTERM, so every `docker stop` — the shadow database teardown and `supabase stop`'s db-container stop alike — burns the full 10s grace period and ends in SIGKILL. Prefix the final command with `exec` in all three entrypoint builders (PG>=15, PG<=14, --from-backup restore) so Postgres is PID 1, receives the stop signal, and shuts down in ~1s. Deliberate divergence from Go's script, documented at the builders; timing is not part of the Go-parity surface (ADR 0016). Workstream B of apps/cli/docs/shadow-db-startup-perf-plan.md, and a prerequisite for the warm shadow-container cache (workstream C), whose release path is a `docker stop`. Co-Authored-By: Claude Fable 5 --- apps/cli/docs/shadow-db-startup-perf-plan.md | 10 +++++++++- .../legacy/shared/db-bootstrap/postgres.service.ts | 13 ++++++++++--- .../db-bootstrap/postgres.service.unit.test.ts | 10 +++++----- .../db-bootstrap/shadow-database.unit.test.ts | 2 +- 4 files changed, 25 insertions(+), 10 deletions(-) diff --git a/apps/cli/docs/shadow-db-startup-perf-plan.md b/apps/cli/docs/shadow-db-startup-perf-plan.md index d9ccdc69cc..e40e7bd822 100644 --- a/apps/cli/docs/shadow-db-startup-perf-plan.md +++ b/apps/cli/docs/shadow-db-startup-perf-plan.md @@ -149,7 +149,15 @@ becomes connectable, and fails fast when the container exits. (measured ~3.5–5s instead of ~10.5s); error behavior on a broken container unchanged in shape. -## Workstream B — clean fast shadow shutdown (small, enables C) +## Workstream B — clean fast shadow shutdown (small, enables C) — IMPLEMENTED + +Shipped with the **broader scope**: all three entrypoint script builders in +`postgres.service.ts` (`legacyPostgresEntrypointScriptPg15` / `…Pg14` / `…Restore`) now +`exec` the final `docker-entrypoint.sh` command, so the shadow, the long-running `db` +container, and the `--from-backup` restore container all get a PID-1 Postgres that +receives the stop signal — `docker stop` in `supabase stop` benefits too. The divergence +from Go's script is documented at the builders. The spec below is retained as the record +of what was agreed. **Problem:** `sh` as PID 1 swallows SIGTERM → every shadow stop (and workstream C's cache-release stop) burns the 10s grace period and ends in SIGKILL. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts index f3ec4879a4..c94bd73c6e 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts @@ -258,6 +258,13 @@ function legacyPostgresExtraEnv( * {@link legacyBuildPostgresStartContainerSpec}), so it never appears in this * process's own `docker create` argv (CWE-214/522). * + * The final command is `exec`'d — a deliberate divergence from Go's script + * (which leaves `sh` as PID 1, so SIGTERM is never forwarded and every + * `docker stop` burns the full 10s grace period before SIGKILL; with `exec`, + * Postgres is PID 1 and stops in ~1s). Applies to all three entrypoint + * variants below. Timing is not part of the Go-parity surface (ADR 0016); + * see apps/cli/docs/shadow-db-startup-perf-plan.md, workstream B. + * * Otherwise byte-for-byte derived from Go's raw-string concatenation — * `NewContainerConfig(args ...string)` splices `strings.Join(args, " ")` * straight after the literal trailing space following `/etc/postgresql` @@ -276,7 +283,7 @@ function legacyPostgresEntrypointScriptPg15(postgresConfig: string, args = ""): "\n" + "cat <<'EOF' > /etc/postgresql.schema.sql && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - `docker-entrypoint.sh postgres -D /etc/postgresql ${args}\n` + + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${args}\n` + `${LEGACY_START_DB_SCHEMA_SQL}\n` + `${LEGACY_START_DB_WEBHOOK_SQL}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + @@ -301,7 +308,7 @@ function legacyPostgresEntrypointScriptPg14(postgresConfig: string, args = ""): "\n" + "cat <<'EOF' > /docker-entrypoint-initdb.d/supabase_schema.sql && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - `docker-entrypoint.sh postgres -D /etc/postgresql ${args}\n` + + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${args}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + "EOF\n" + `${postgresConfig}\n` + @@ -328,7 +335,7 @@ function legacyPostgresEntrypointScriptRestore(postgresConfig: string): string { "cat <<'EOF' > /etc/postgresql.schema.sql && \\\n" + "cat <<'EOF' > /docker-entrypoint-initdb.d/migrate.sh && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - "docker-entrypoint.sh postgres -D /etc/postgresql\n" + + "exec docker-entrypoint.sh postgres -D /etc/postgresql\n" + `${LEGACY_START_DB_SCHEMA_SQL}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + "EOF\n" + diff --git a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts index 122418f02f..16969daeb7 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.unit.test.ts @@ -79,7 +79,7 @@ describe("legacyBuildPostgresStartContainerSpec", () => { "\n" + "cat <<'EOF' > /etc/postgresql.schema.sql && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - "docker-entrypoint.sh postgres -D /etc/postgresql \n" + + "exec docker-entrypoint.sh postgres -D /etc/postgresql \n" + `${LEGACY_START_DB_SCHEMA_SQL}\n` + `${LEGACY_START_DB_WEBHOOK_SQL}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + @@ -108,7 +108,7 @@ describe("legacyBuildPostgresStartContainerSpec", () => { "\n" + "cat <<'EOF' > /docker-entrypoint-initdb.d/supabase_schema.sql && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - "docker-entrypoint.sh postgres -D /etc/postgresql \n" + + "exec docker-entrypoint.sh postgres -D /etc/postgresql \n" + `${LEGACY_START_DB_SUPABASE_SQL}\n` + "EOF\n" + `${POSTGRES_CONFIG_HEADER}\n` + @@ -252,7 +252,7 @@ describe("legacyBuildPostgresStartContainerSpec", () => { "cat <<'EOF' > /etc/postgresql.schema.sql && \\\n" + "cat <<'EOF' > /docker-entrypoint-initdb.d/migrate.sh && \\\n" + "cat <<'EOF' >> /etc/postgresql/postgresql.conf && \\\n" + - "docker-entrypoint.sh postgres -D /etc/postgresql\n" + + "exec docker-entrypoint.sh postgres -D /etc/postgresql\n" + `${LEGACY_START_DB_SCHEMA_SQL}\n` + `${LEGACY_START_DB_SUPABASE_SQL}\n` + "EOF\n" + @@ -405,7 +405,7 @@ describe("legacyBuildShadowPostgresContainerSpec", () => { ); const script = spec.cmd?.[1]; expect(script).toContain( - `docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}\n`, + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}\n`, ); expect(spec.secretFiles).toEqual([ { @@ -422,7 +422,7 @@ describe("legacyBuildShadowPostgresContainerSpec", () => { ); const script = spec.cmd?.[1]; expect(script).toContain( - `docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}\n`, + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}\n`, ); expect(spec.secretFiles).toBeUndefined(); expect(spec.tmpfs).toEqual({ "/docker-entrypoint-initdb.d": "" }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index 10fe3c93ec..5e3365920a 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -208,7 +208,7 @@ describe("legacyCreateShadowDatabase / legacyRemoveShadowDatabase", () => { // in argv. const script = mock.spawned[createIdx]?.at(-1) ?? ""; expect(script).toContain( - `docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}`, + `exec docker-entrypoint.sh postgres -D /etc/postgresql ${LEGACY_SHADOW_ENTRYPOINT_ARGS}`, ); }), ); From 008256ecf53e0c9ce9a6fd6cee6f4ece68973429 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 16:18:49 +0200 Subject: [PATCH 38/82] feat(cli): cache the shadow baseline as a PGDATA snapshot (default on) Shadow provisioning (db diff / db pull / declarative sync via pg-delta) rebuilds the platform baseline (init schema + realtime/storage/auth one-shot jobs, ~15s+) on every run. Cache that baseline at the DISK level instead: immediately after the baseline and before user migrations, stop the shadow, stream its PGDATA out with the tar form of docker cp, and publish it atomically as supabase/.temp/pgdelta/shadow-baseline-.tar (~50-90MB, one tar per key, stale keys swept). The next run with an identical cache key unpacks that tar into a freshly CREATED container before docker start (preStartArchives on the container spec; the tar-stream form is mandatory - the host-directory form of docker cp resets ownership to root and Postgres refuses the data dir), the entrypoint sees PG_VERSION and skips initdb, and the run applies user migrations directly: measured ~3.3s warm provision vs ~15.5s cold on a default-services PG17 project. Pristine by construction: every run gets a brand-new container with the baseline's exact bytes, so nothing a previous run's migrations did (roles, memberships, cluster settings, extra databases) can survive - there is no reset protocol, no cluster-catalog snapshot, no metadata JSON, no lock file, and no kept container to race over (concurrent cold writers are settled by the atomic rename; readers keep a valid fd). The artifact is a plain file so future native (non-Docker) services can restore the same snapshot into their own data directory. - Default ON; SUPABASE_SHADOW_CACHE=false (or 0) opts out and restores the exact uncached lifecycle. Test helpers pin it off so unrelated suites and e2e stay on the plain lifecycle. - Cache key: sha256 over every input baked into the baseline (resolved postgres + enabled-service job image tags, service flags, jwtSecret, rootKey, db password/settings, jwtExpiry, auto_expose_new_tables, roles.sql contents, vault names AND values, shadowPort, major_version, and the resolved JWKS when realtime is enabled on PG >= 15). - One forced container-shape exception: the cold cache path creates WITHOUT --rm, because Docker destroys an AutoRemove container the moment it stops - including the export's own docker stop (verified). Release still docker rm -f -v's every shadow; only a SIGKILLed CLI leaves a stopped project-labeled container, which supabase stop sweeps. Warm and cache-off paths keep --rm. - Escape hatch everywhere: export failure warns and continues uncached; restore/start/ready failure deletes the suspect tar, removes the container, and cold-provisions. The cache can never fail a run. - SUPABASE_SHADOW_DEBUG=1 prints per-phase stderr timings (ready-attempt/ready-wait, baseline-export, baseline-restore, contrib-regression-create) for diagnosing cache/readiness behavior. - docs/shadow-db-provisioning.md replaces the implementation plan with as-built notes; SIDE_EFFECTS.md updated for db diff, db pull, and declarative sync. db diff --use-pgadmin, db pull --declarative's raw shadow, and migration squash still cold-provision (documented follow-ups). Full unit+integration suite: 449 files / 7730 tests pass; types/lint/ fmt/knip green. Dogfooded on a default-services PG17 project: cold export +4.8s one-time, warm restore 1.8s + ready 1.2s, opt-out byte-identical, correct diff output on local drift. Co-Authored-By: Claude Fable 5 --- apps/cli/docs/shadow-db-provisioning.md | 144 ++++ apps/cli/docs/shadow-db-startup-perf-plan.md | 302 ------- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 44 +- .../legacy/commands/db/diff/diff.handler.ts | 100 +-- .../commands/db/diff/diff.integration.test.ts | 5 + .../legacy/commands/db/pull/SIDE_EFFECTS.md | 32 +- .../src/legacy/commands/db/pull/pull.debug.ts | 2 +- .../legacy/commands/db/pull/pull.handler.ts | 106 +-- .../commands/db/pull/pull.integration.test.ts | 5 + ...eclarative.orchestrate.integration.test.ts | 9 +- .../schema/declarative/sync/SIDE_EFFECTS.md | 57 +- .../schema/declarative/sync/sync.handler.ts | 2 +- .../declarative/sync/sync.integration.test.ts | 5 + .../shared/legacy-pgdelta-next-artifacts.ts | 2 +- ...legacy-pgdelta-next-artifacts.unit.test.ts | 2 +- .../db/shared/legacy-shadow-source.ts | 33 +- .../squash/squash.integration.test.ts | 5 + .../db-bootstrap/container-lifecycle.ts | 62 +- .../shared/db-bootstrap/docker-create-args.ts | 57 +- .../shared/db-bootstrap/health-check.ts | 62 +- .../shared/db-bootstrap/postgres.service.ts | 2 +- .../shadow-cache.integration.test.ts | 666 +++++++++++++++ .../db-bootstrap/shadow-cache.live.test.ts | 188 +++++ .../shared/db-bootstrap/shadow-cache.ts | 785 ++++++++++++++++++ .../db-bootstrap/shadow-cache.unit.test.ts | 232 ++++++ .../shared/db-bootstrap/shadow-database.ts | 142 +++- .../db-bootstrap/shadow-database.unit.test.ts | 7 +- .../shared/db-bootstrap/shadow-debug.ts | 80 ++ .../legacy/shared/legacy-docker-lifecycle.ts | 18 +- .../src/legacy/shared/legacy-pgdelta.cache.ts | 47 +- .../src/legacy/shared/legacy-pgdelta.paths.ts | 16 + apps/cli/tests/helpers/cli.ts | 5 + apps/cli/tests/helpers/legacy-mocks.ts | 27 + 33 files changed, 2755 insertions(+), 496 deletions(-) create mode 100644 apps/cli/docs/shadow-db-provisioning.md delete mode 100644 apps/cli/docs/shadow-db-startup-perf-plan.md create mode 100644 apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts create mode 100644 apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts create mode 100644 apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts create mode 100644 apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts create mode 100644 apps/cli/src/legacy/shared/db-bootstrap/shadow-debug.ts create mode 100644 apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts diff --git a/apps/cli/docs/shadow-db-provisioning.md b/apps/cli/docs/shadow-db-provisioning.md new file mode 100644 index 0000000000..1d3928f6fc --- /dev/null +++ b/apps/cli/docs/shadow-db-provisioning.md @@ -0,0 +1,144 @@ +# Shadow database provisioning + +How `db diff` / `db pull` / `db schema declarative sync` provision the shadow Postgres they diff +against, and why it is shaped the way it is. As-built notes, not a plan. + +## Why the shadow is on every hot path + +These commands hand pg-delta an **isolated shadow** (`pgdelta … --isolated-shadow`, loader mode +`isolatedCluster`): a `supabase/postgres` container brought to the platform baseline — the bundled +init schema plus the PG15+ one-shot realtime / storage / auth migrate jobs — before the user's own +migrations are applied. pg-delta requires that shadow on a different Postgres lineage than the +target, so the container is on the hot path of _every_ plan; the migrations-catalog cache cannot +make it go away. + +Owner: `src/legacy/shared/db-bootstrap/shadow-database.ts` (the primitives) and +`shadow-cache.ts` (the acquire/release seam every call site actually uses, +`legacyWithShadowDatabase`). + +## Readiness: a connect probe, not Docker health + +`legacyWaitForShadowReady` (`health-check.ts`) polls, on a 1s constant backoff: + +1. `docker inspect` — is the container still `running`? (preserves the crash detection the health + gate gave us: a dead container fails fast instead of at the end of the budget); then +2. a short-timeout Postgres connect — success ⇒ ready. + +The shadow's own healthcheck is `interval=10s` with no `start_period`/`start_interval` +(`postgres.service.ts`, deliberately unchanged — other tooling reads that config), so Docker's +first probe runs at t+10s while Postgres has been accepting connections since ~3.5s. Gating on +`Health.Status` spent ~6.5s per provision waiting for an already-knowable verdict. +`--health-start-interval` would fix the container side but needs Docker Engine 25+/API 1.44 and is +not reliably supported by Podman (which `spawnContainerCli` falls back to), so the CLI-side wait +asks Postgres directly instead. The failure shape is identical to the health gate's — same +`LegacyHealthCheckTimeoutError`, same `docker logs` dump. + +## Fast shutdown: the entrypoint `exec`s Postgres + +All three entrypoint builders in `postgres.service.ts` (`…Pg15` / `…Pg14` / `…Restore`) `exec` the +final `docker-entrypoint.sh` command, so PID 1 is Postgres and receives SIGTERM directly. With `sh` +as PID 1 the signal was swallowed and every `docker stop` burned the full 10s grace period before +a SIGKILL. This benefits `supabase stop` too, and it is what makes the cache's own mid-run stop +cheap. The divergence from Go's script is documented at the builders. + +## The PGDATA snapshot cache + +`SUPABASE_SHADOW_CACHE` — **on by default**; set it to `false` or `0` to opt out, in which case +the acquire is exactly `legacyCreateShadowDatabase` and the release exactly +`legacyRemoveShadowDatabase`. + +Everything above the user's migrations is deterministic in the config, so it is cached as a +disk-level snapshot of the initialized data directory: + +- **Cold** (no snapshot for this key): create the shadow and run the baseline as usual, then — at + the baseline/migrations seam, before `contrib_regression` or any user migration — + `docker stop`, stream `docker cp :/var/lib/postgresql/data -` to + `supabase/.temp/pgdelta/shadow-baseline-.tar`, `docker start`, re-await readiness, continue. +- **Warm** (that tar exists): create the shadow, unpack the tar into the created-but-not-yet-started + container (`docker cp - :/var/lib/postgresql`), then start it. `docker-entrypoint.sh` finds a + `PG_VERSION` file and skips `initdb` and the whole baseline; the caller is told + `baselinePresent: true` (`LegacyShadowBaselineState`, `shadow-database.ts`) and goes straight to + `contrib_regression` + user migrations. + +### Cache key + +sha256 (16 hex chars, fixed field order) over every input baked into the cluster during a cold +provision — `legacyShadowCacheKey`: + +- resolved `supabase/postgres` image tag (full tag, _not_ major version) after registry/pin + resolution; +- resolved one-shot job image tags (`realtime`, `storage`, `auth` via `legacyResolvePinnedImage` + + `serviceVersionOverrides`), each included **only when its service is enabled** — a disabled + service's job never ran into the baseline; +- the service enabled flags themselves; +- `jwtSecret`, `rootKey`, `[db] password`, `db.settings` (canonical JSON), `jwtExpiry`; +- effective `api.auto_expose_new_tables` (tri-state: unset ≠ explicit `false`); +- `supabase/roles.sql` contents (empty string when absent); +- `[db.vault]` secret **names and values** — both land in `vault.secrets`. (`setupInputsToken` in + `legacy-pgdelta.cache.ts` hashes names only and omits the job image tags; it is a Go-parity + contract and is deliberately not reused here, only its hashing style is.) +- `shadowPort` and `db.major_version`; +- the resolved JWKS string that realtime's one-shot tenant-seed job bakes in, included **only when + realtime is enabled AND `major_version >= 15`** — the exact compound gate that decides whether + the job is ever reached. An `auth.third_party` change moves this value and nothing else. + +### Invariants + +- **Atomic publish.** The tar is streamed to `..partial` and `rename`d into place, so a + partial tar is never observable under the final name and a reader holding the old inode keeps a + valid fd across a rename-over. +- **No lock file, no coordination.** Every run creates its own container, so no two runs can + contend for one cluster. Two concurrent cold writers on the same key both export and the last + `rename` wins — both tars are equally valid, since the key covers every baked-in input. +- **Retention: current key only.** Publishing a key's tar removes every other + `shadow-baseline-*.tar` in the directory (a snapshot is ~90MB with default services, so a project + holds one, not one per config permutation it has ever used). +- **Escape hatch.** Any warm-path failure (the `docker cp` in, the start, the readiness wait) + removes the container, **deletes the tar** as suspect, and cold-provisions. Any cold-path export + failure warns on stderr and leaves the run uncached; the container is restarted either way. The + cache never fails a user's command and never hands back a wrong baseline — worst case is the + uncached behavior. +- **No container outlives a run.** Cold, warm, and cache-disabled shadows all carry the same two + project labels and are removed with `docker rm -f -v` on release. There is no cache-key Docker + label and nothing extra for `supabase stop` to sweep. +- **One forced divergence:** the cold path drops `--rm`, because Docker destroys an `AutoRemove` + container the moment it exits — `docker stop` included (verified on Docker 29: gone ~1-2s after + the stop returns) — which would leave nothing to restart. The container is still removed on + release; the only visible consequence is that a SIGKILLed CLI leaves a stopped, project-labeled + shadow behind instead of nothing. + +### Why a plain file + +The artifact is a tar under the project's own temp directory, not a Docker object. Nothing about +the mechanism is container-specific, so a future **native** (non-Docker) Postgres service can reuse +the same snapshot by unpacking it into its own data directory — which is why the export is a file +rather than a `docker commit` image or a kept container. + +## Measured + +Benchmarked on `public.ecr.aws/supabase/postgres:17.6.1.158`, Docker 29, warm image cache: + +| Phase | Time | +| -------------------------------------------------------------- | ----------------------------- | +| `docker create` + secret `docker cp` + `docker start` | ~0.4s | +| Postgres accepting connections (initdb + bundled init SQL) | ~8-11s | +| One-shot realtime job (Elixir boot + tenant seed) | ~6.5s | +| One-shot storage migrate job | ~2.5s | +| One-shot auth (`gotrue migrate`) job | ~0.7s | +| **Cold provision total** (excl. image pulls, excl. migrations) | **~30s wall** | +| Snapshot export (`docker cp … -` to file) | ~0.65s (39MB bare cluster) | +| Snapshot restore (`docker cp - …`, before start) | ~0.6s | +| Restored container `docker start` → connectable | ~2s (initdb skipped entirely) | + +Docker's healthcheck reported `healthy` at ~10.2s on the same container, which is the ~6.5s of dead +wait the connect probe removed. + +## Known gaps + +- `db diff --use-pgadmin` and `migration squash` provision the same shadow but keep the + Docker-health gate and do not go through `legacyWithShadowDatabase`, so they are always cold. +- `db pull --declarative`'s bare shadow (`legacyPrepareRawShadow`) runs no platform baseline, so + there is nothing for this cache to snapshot; it stays cold by construction. +- The snapshot is taken **before** user migrations on purpose (that is what makes it reusable + across migration edits). Snapshotting the post-migration state as well, keyed on a migrations + hash, is the obvious next step for repos with large migration histories. diff --git a/apps/cli/docs/shadow-db-startup-perf-plan.md b/apps/cli/docs/shadow-db-startup-perf-plan.md deleted file mode 100644 index e40e7bd822..0000000000 --- a/apps/cli/docs/shadow-db-startup-perf-plan.md +++ /dev/null @@ -1,302 +0,0 @@ -# Shadow database startup performance — implementation plan - -Status: **workstream A implemented; B and C planned**. This document is a handoff spec: it -contains the measured baseline, the agreed design decisions, and file-level work items. An -implementing agent should be able to execute it without re-deriving the investigation. - -## Context - -`db diff` / `db pull` / declarative sync provision a **shadow database**: a throwaway -`supabase/postgres` container brought to the platform baseline (init schema + realtime / -storage / auth one-shot migration jobs), then handed to pg-delta as the isolated shadow -(`pgdelta … --isolated-shadow`, loader mode `isolatedCluster`). pg-delta requires this -shadow on a **different Postgres lineage** than the target, so the container is on the hot -path of _every_ plan — the migrations-catalog cache cannot make it go away. - -Today the shadow is created cold and destroyed (`docker rm -f -v`) on every run. - -## Measured baseline - -Benchmarked 2026-08-13 on `supabase/postgres:17.6.1.158` (Docker 29, overlayfs, warm image -cache), reproducing the exact container shape `legacyBuildShadowPostgresContainerSpec` -produces (same entrypoint heredoc script, env, healthcheck flags) and the exact one-shot -job env from `db-setup.ts`. Two cold samples, consistent within ~1s: - -| Phase | Time | -| ------------------------------------------------------------------------------ | ---------- | -| `docker create` + secret `docker cp` + `docker start` | ~0.4s | -| Postgres accepting connections (initdb + bundled init SQL) | ~3.4–4.5s | -| Docker healthcheck reports `healthy` (**the CLI's current gate**) | ~10.2s | -| One-shot realtime job (Elixir boot + tenant seed) | ~6.5s | -| One-shot storage migrate job | ~2.5s | -| One-shot auth (`gotrue migrate`) job | ~0.7s | -| Revoke API privileges + `CREATE DATABASE contrib_regression TEMPLATE postgres` | ~0.3s | -| **Total cold provision** (excl. image pulls, excl. user migrations) | **~20.5s** | - -Additional measurements that motivate the work items: - -- The healthcheck is `interval=10s, timeout=2s, retries=3` with **no - start_period/start_interval** (`postgres.service.ts`), so Docker's first probe runs at - t+10s. Postgres is connectable at ~3.5s → **~6.5s per provision is pure dead wait**. The - same container with `--health-start-period 30s --health-start-interval 1s` reported - healthy at 3.2s. -- `docker start` of an already-initialized shadow container → connectable in **~1.0s** - (also ~1s after a SIGKILL'd stop, via WAL recovery). -- `CREATE DATABASE … TEMPLATE ` on the baseline state: **~0.2s**. `DROP DATABASE … -WITH (FORCE)`: ~0.1s. -- `docker stop` on the current container takes **10.3s**: the entrypoint is `sh -c "… && -docker-entrypoint.sh …"`, so PID 1 is `sh`, which does not forward SIGTERM; Docker waits - the 10s grace period and SIGKILLs. - -Target end state: **~1.5–2s** per plan on a warm cache hit; **~14s** cold (health gate -fixed); cold path otherwise unchanged. - -## Design decisions (already made — do not relitigate) - -1. **This is a CLI-only concern.** pg-delta is not modified and never learns the cache - exists. Its `isolatedCluster` loader mode already supports a pre-provisioned shadow - with a platform baseline and pre-existing rows. The deliverable is "set up the base - `supabase/postgres` container + its owned services faster". -2. **Cache the container, not a volume.** The shadow mounts no volume today (PGDATA in - container fs, `binds: []`), and Docker has no cheap volume clone. Reuse = keep the - initialized container **stopped** between runs, `docker start` it on the next plan, and - prune state with template databases + a role-delta reset. -3. **Invalidate on any input change, never partially refresh.** See cache key below. -4. **Escape hatch everywhere:** any error or anomaly on the warm path (start failure, port - busy, reset error, missing metadata) ⇒ treat as cache miss: `docker rm -f -v` the - container and cold-provision. Worst case is today's behavior; the cache can never - produce a wrong baseline. -5. **Readiness fix uses a direct connect probe, not `--health-start-interval`.** That flag - requires Docker Engine 25+/API 1.44 and is not reliably supported by Podman (the CLI - falls back to Podman via `spawnContainerCli`). A connect probe sidesteps the - runtime-version matrix. (`docker-create-args.ts` already supports - `--health-start-period` if a flag-based variant is ever wanted, but it alone does not - speed up the first probe.) -6. Timing is **not** part of the Go-parity surface (ADR 0016) — these are TS-side - improvements. Do not change stdout text, exit codes, or flag surfaces. - -## Cache key (workstream C) - -Hash (sha256, stable field order) of every input baked into the cluster during cold -provisioning: - -- resolved `supabase/postgres` image tag (full tag, e.g. `17.6.1.158` — _not_ major - version) after registry/pin resolution; -- resolved one-shot job image tags — `realtime`, `storage`, `auth` via - `legacyResolvePinnedImage` + `serviceVersionOverrides` — each included **only when its - service is enabled** (a disabled service's job never ran into the baseline); -- service enabled flags themselves (realtime/storage/auth); -- `jwtSecret`, `rootKey`, `[db] password`, `db.settings` (serialized), `jwtExpiry`; -- effective `api.auto_expose_new_tables`; -- `supabase/roles.sql` contents (empty string when absent); -- `[db.vault]` secret **names and values** (values are upserted into the DB — the existing - `setupInputsToken` in `legacy-pgdelta.cache.ts` hashes names only, which is - insufficient here; do not reuse it as-is, but mirror its hashing style); -- `shadowPort` (the stopped container's port binding is fixed at create time); -- `db.major_version` (implied by the image tag in practice, but cheap and explicit). - -Note: the existing `setupInputsToken` also omits the service image tags. That may or may -not matter for the catalog cache (out of scope here — flag it to a human if touched); for -the container cache it definitely matters, because the jobs write versioned schema state -(`auth`, `storage`, `_realtime`) into the cluster. - -## Workstream A — shadow readiness gate (independent, ship first) — IMPLEMENTED - -Shipped as `legacyWaitForShadowReady` in -`apps/cli/src/legacy/shared/db-bootstrap/health-check.ts`, consumed by -`legacyPrepareRawShadow` (`shadow-database.ts`) and `legacyPrepareShadowSource` -(`commands/db/shared/legacy-shadow-source.ts`). Two shadow health-waits deliberately stayed -on the Docker-health gate and are follow-up candidates: `db diff --use-pgadmin` -(`diff.handler.ts`) and `migration squash` (`squash.handler.ts`) — both provision the same -shadow container and pay the same ~6.5s. The spec below is retained as the record of what -was agreed. - -**Problem:** `legacyPrepareRawShadow` and `legacyPrepareShadowSource` gate on -`legacyWaitForHealthyServices` (1s poll of `docker inspect` health), but the container -cannot report healthy before the healthcheck's first 10s-interval probe. ~6.5s dead wait -per provision, including CI and cache-miss paths. - -**Change:** for the **shadow container only** (do not touch the long-running `db` -container's wait), replace the docker-health gate with a readiness probe that polls, on the -same 1s constant backoff and the same `healthTimeoutSeconds` budget: - -1. `legacyInspectContainerState` → still `running`? (preserves the crash-detection the - health gate provided; a dead container fails fast with the same - `LegacyHealthCheckTimeoutError` shape + log dump behavior); then -2. a short-timeout TCP/auth connect attempt (reuse `LegacyDbConnection.connect` the way - `legacyConnectShadowDatabase` does, or `pg_isready` semantics via a cheap connect) — - success ⇒ ready. - -Suggested shape: a `legacyWaitForShadowReady(spawner, containerId, connConfig, opts)` in -`shared/db-bootstrap/health-check.ts` (or a sibling module), used by -`legacyPrepareRawShadow` (`shadow-database.ts`) and `legacyPrepareShadowSource` -(`commands/db/shared/legacy-shadow-source.ts`). Keep the container's healthcheck config -unchanged (other tooling reads it); only the CLI-side wait changes. - -**Files:** - -- `apps/cli/src/legacy/shared/db-bootstrap/health-check.ts` (new probe) -- `apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts` (`legacyPrepareRawShadow`) -- `apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts` -- unit/integration tests colocated per repo convention - -**Tests (RED first, per repo policy):** integration test with a mocked container-state / -connection layer proving the wait resolves as soon as a connect succeeds (does not wait -for docker health), still fails with the timeout error + log dump when the container never -becomes connectable, and fails fast when the container exits. - -**Acceptance:** shadow provision reaches "connected" in roughly `postgres-ready + ≤1s` -(measured ~3.5–5s instead of ~10.5s); error behavior on a broken container unchanged in -shape. - -## Workstream B — clean fast shadow shutdown (small, enables C) — IMPLEMENTED - -Shipped with the **broader scope**: all three entrypoint script builders in -`postgres.service.ts` (`legacyPostgresEntrypointScriptPg15` / `…Pg14` / `…Restore`) now -`exec` the final `docker-entrypoint.sh` command, so the shadow, the long-running `db` -container, and the `--from-backup` restore container all get a PID-1 Postgres that -receives the stop signal — `docker stop` in `supabase stop` benefits too. The divergence -from Go's script is documented at the builders. The spec below is retained as the record -of what was agreed. - -**Problem:** `sh` as PID 1 swallows SIGTERM → every shadow stop (and workstream C's -cache-release stop) burns the 10s grace period and ends in SIGKILL. - -**Change:** in the entrypoint script builders (`postgres.service.ts`, -`legacyPostgresEntrypointScriptPg15` / `…Pg14`), `exec` the final command: `… && exec -docker-entrypoint.sh postgres -D /etc/postgresql `. Decide scope deliberately: - -- Minimal/safe: apply only in `legacyBuildShadowPostgresContainerSpec`'s script (add a - parameter to the script builders rather than post-processing the string). -- Broader (recommended if reviewers agree): apply to the long-running `db` container too — - it has the same latent 10s-stop cost — but that touches the Go-parity container shape, - so call it out explicitly in the PR rather than folding it in silently. - -**Tests:** unit snapshot of the generated script (existing spec builders have snapshot -coverage patterns); a live test is optional (`stop` timing is observable but flaky to -assert — asserting the script contains `exec` is enough). - -## Workstream C — warm shadow container cache - -New module, suggested `apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts`, exposing -an acquire/release pair that call sites use in place of bare -`legacyCreateShadowDatabase` / `legacyRemoveShadowDatabase`. Gate the whole feature behind -an opt-in env var (e.g. `SUPABASE_SHADOW_CACHE=1`) for the first release; flip the default -once proven. - -### Cold provision (cache miss) - -1. Create + start the shadow as today (no `autoRemove` when caching — the container must - survive), **labeled** with the project labels plus a new - `com.supabase.cli.shadow-cache-key=` label. -2. Run the existing baseline setup (`legacySetupShadowDatabase` / - `legacyMigrateShadowDatabase` path unchanged up to the baseline; user migrations are - _not_ part of the cached state — see reset protocol). -3. **Snapshot for reuse**, immediately after the baseline (before migrations/declarative - load): - - `CREATE DATABASE _supabase_shadow_base TEMPLATE postgres` (requires no other - connections to `postgres` — sequence it before pg-delta connects); - - capture cluster-global state: `pg_roles` (name + attribute columns), `pg_auth_members` - (role, member, admin_option), and cluster-wide `pg_db_role_setting` rows - (`setdatabase = 0`); - - persist metadata JSON to `supabase/.temp/pgdelta/shadow-cache-.json`: - `{ key, containerId, createdAt, roleSnapshot, membershipSnapshot, roleSettings }`. -4. Proceed with the run as today (migrations applied to `postgres`, `contrib_regression` - template creation, pg-delta, …). -5. **Release:** instead of `docker rm -f -v` → `docker stop` (fast after workstream B). - Keep release best-effort exactly like `legacyRemoveShadowDatabase` (never mask the - run's own outcome). - -### Warm acquire (cache hit) - -1. Compute the key; look up the metadata file **and** the container (`docker ps -a ---filter label=com.supabase.cli.shadow-cache-key=`). Either missing, or container - already running (another concurrent run owns it) ⇒ miss (concurrent case: fall through - to a one-off uncached cold shadow, do not remove the cached one). -2. `docker start` (~1s), wait via workstream A's probe. -3. **Reset to pristine:** - - reverse the role delta vs. the snapshot: drop roles not in the snapshot (`DROP OWNED -BY` in each affected DB is unnecessary once `postgres` is recreated — drop role after - the DB recreate below to avoid dependency errors; simplest safe order: recreate DBs - first, then `DROP ROLE`), revoke added memberships, re-grant removed ones, delete - cluster-wide `pg_db_role_setting` rows not in the snapshot; - - connect to `_supabase_shadow_base` and `DROP DATABASE postgres WITH (FORCE)` + - `CREATE DATABASE postgres TEMPLATE _supabase_shadow_base` — every downstream consumer - keeps using `postgres` + `contrib_regression` unchanged; - - `DROP DATABASE IF EXISTS contrib_regression WITH (FORCE)` (recreated by the normal - flow). -4. Hand the same `LegacyShadowSourceResult` shape to the caller; the rest of the run is - byte-identical to today. -5. Any step failing ⇒ escape hatch (rm + cold provision + fresh snapshot). - -### Interactions & housekeeping - -- **`supabase stop`** sweeps project-labeled containers — it will delete the cached shadow. - That is acceptable (cache cleared, next run cold-provisions); mention it in the PR, do - not special-case. -- **Metadata/container drift:** container exists but metadata file missing (or vice versa) - ⇒ miss + remove the orphan. Add the metadata file to the same retention/cleanup pass the - catalog cache uses if one exists; otherwise overwrite-on-write is enough (one file per - key, old keys' containers removed on key mismatch — enumerate by label, keep only the - current key's container). -- **Leak window:** unlike today's `--rm` shadow, a crashed CLI leaves a _stopped, labeled_ - container. That is the cache working as intended; `supabase stop` and the key-mismatch - sweep both reclaim it. - -### Call sites to convert - -Whatever consumes the create/remove pair around `legacyPrepareShadowSource` / -`legacyPrepareRawShadow` today (as of writing: `db diff`'s handler, `db pull`'s handler, -and `legacy-pgdelta.cache.ts`'s `exportViaShadowCatalog`) — all via -`Effect.acquireUseRelease`, so the seam is narrow: acquire ⇒ `shadow-cache` acquire, -release ⇒ `shadow-cache` release. `migration squash` (if/when it lands on the native -shadow path) picks it up for free through the same primitives. - -### Tests - -- **Unit:** cache-key builder (field order, every input changes the hash, disabled service - excludes its tag); reset-SQL builder (role delta → exact DROP/REVOKE/GRANT statements, - snapshot round-trip). -- **Integration:** acquire/release state machine with mocked spawner + DB session layers — - hit path, each miss reason (no container, no metadata, running container, key mismatch), - and the escape hatch on reset failure (asserts rm + cold fallback ordering). -- **Live (`*.live.test.ts`, gated per repo policy, golden path only):** one scenario — - cold acquire, release, warm acquire on the same key asserts the container id is reused - and a role created between the two runs is gone after reset. - -## Sequencing - -1. **A** — readiness gate (`fix(cli)`, independently shippable, benefits every provision). -2. **B** — entrypoint `exec` (`fix(cli)` or folded into C's PR; required for C's fast - release). -3. **C** — warm cache behind the env-var gate (`feat(cli)`), then a follow-up to default it - on after bake time. - -Each item follows the repo's RED→GREEN rule (failing test first, capture the failure in -the commit/PR). PR titles: conventional commits, `(cli)` scope. Per repo policy, no test -plans in PR descriptions. - -## Non-goals - -- No changes to `supabase/pg-toolbelt` / pg-delta. The isolated shadow contract - (pre-provisioned, baseline rows tolerated, cluster DDL allowed) already accommodates a - reused container. -- No changes to the co-located shadow path (pg-delta's `provisionCoLocatedShadow` is - already sub-second). -- No change to the migrations-catalog cache or its `setupInputsToken` (its service-tag - omission is noted above as a possible separate issue — surface to a human, don't fix - here). -- No healthcheck-config changes on the container itself. - -## Appendix — reproducing the benchmark - -The numbers above came from a throwaway harness (not committed): generate the exact -entrypoint script by importing the `LEGACY_START_DB_*_SQL` template constants and -concatenating them the way `legacyPostgresEntrypointScriptPg15` does (with -`-c max_worker_processes=0`), `docker create` with the spec's env/healthcheck flags, -`docker cp` the pgsodium root key to `/etc/postgresql-custom/pgsodium_root.key`, start, -then poll (a) `pg_isready` via `docker exec`, (b) `docker inspect -'{{.State.Health.Status}}'`. Run the three one-shot jobs with the env built by -`legacyBuildRealtimeEnv` / `legacyStartStorageMigrateEnv` / `legacyStartAuthMigrateEnv` -against the shadow's 12-char container id as `DB_HOST` on the same network. Warm numbers: -`docker stop` + `docker start` the same container and re-poll. diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 23573f2114..7da0f024ee 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -34,15 +34,16 @@ it, and JSON `null` disables formatting without disabling safe compaction. ## Files Written -| Path | Format | When | -| ----------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------ | -| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | -| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| ------------------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------ | +| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | +| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | shadow baseline cache enabled (default) — the shadow's PGDATA snapshot, ~90MB, current key only | +| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker @@ -92,6 +93,7 @@ of this command's own target resolve, ahead of the differ container. | `SUPABASE_DB_MAJOR_VERSION` / `SUPABASE_DB_HEALTH_TIMEOUT` / `SUPABASE_DB_SETTINGS_*` | shadow container-config overrides, same as `db start`/`db reset` | no | | `SUPABASE_PROJECT_ID` | overrides the shadow container's project id/labels, same as `db start`/`db reset` (`utils.DbId`); ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta engine | no | | `PGDELTA_DEBUG` | pg-delta debug capture | no | | `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | @@ -206,6 +208,30 @@ when declarative files exist. no longer the `db __shadow` seam) plus a native pg-delta catalog export. No hidden Go `db schema declarative __catalog` subprocess runs for this path any more. +### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) + +- ON by default; set `SUPABASE_SHADOW_CACHE=false` (or `=0`) to opt out, in which case the shadow + lifecycle is exactly as documented above. +- The cached artifact is a **file**, never a container: `supabase/.temp/pgdelta/shadow-baseline-.tar` + (~90MB), a tar of the shadow's PGDATA directory taken right after the platform baseline and + before `contrib_regression`/any user migration. The key hashes every input baked into the cluster + (images, JWT secret, root key, `[db] password`, `db.settings`, vault secrets, `roles.sql`, + resolved JWKS, shadow port, major version). +- The container lifecycle is otherwise IDENTICAL to the uncached path — same project labels, always + `docker rm -f -v` on release. Nothing is kept between runs, there is no cache-key Docker label, + and no lock file. One forced exception: a COLD cache-enabled run creates the shadow without + `--rm`, because it must `docker stop` the container to take a coherent snapshot and Docker + destroys an `--rm` container the moment it exits. It is still removed on release. +- Cold run: `docker stop` -> `docker cp :/var/lib/postgresql/data -` streamed to the tar (temp + name + atomic rename) -> `docker start` -> readiness wait -> continue. Publishing a new key's tar + deletes every other `shadow-baseline-*.tar` (retention: current key only). +- Warm run: `docker cp - :/var/lib/postgresql` into the created-but-not-yet-started container, + so the entrypoint skips `initdb` and the platform baseline is skipped too. +- Any warm-path failure removes the container, deletes the tar as suspect, and cold-provisions; any + cold export failure only warns on stderr and leaves the run uncached. The cache never fails the + command. +- `--use-pgadmin` is NOT cached — its shadow keeps the plain create/remove lifecycle. + ### `--use-pgadmin` parity quirks and deliberate divergence (CLI-1968) - `source`/`target` are INVERTED relative to the migra/pg-delta path: `source` is the diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index e39942bbec..31da5adf69 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -29,6 +29,7 @@ import { legacyFindDropStatements } from "../../../shared/legacy-sql-split.ts"; import { legacyBuildLocalDbContainerInputs } from "../../../shared/db-bootstrap/local-container-inputs.ts"; import { legacyIsLocalDbRunning } from "../../../shared/db-bootstrap/local-db-running.ts"; import { legacyWaitForHealthyServices } from "../../../shared/db-bootstrap/health-check.ts"; +import { legacyWithShadowDatabase } from "../../../shared/db-bootstrap/shadow-cache.ts"; import { legacyCreateShadowDatabase, legacyMigrateShadowDatabase, @@ -681,57 +682,60 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy pgDelta: cfg.pgDelta, ctx, }; - // Register cleanup atomically with creation; prepare and diff remain interruptible. - diffResult = yield* Effect.acquireUseRelease( - legacyCreateShadowDatabase(spawner, shadowInput), - (handle) => - Effect.gen(function* () { - const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); - const target = shadow.targetUrlOverride ?? targetUrl; - yield* output.raw( - flags.schema.length > 0 - ? `Diffing schemas: ${flags.schema.join(",")}\n` - : "Diffing schemas...\n", - "stderr", - ); - if (useDelta) { - const result = yield* pgDelta.diffDatabase({ - context: ctx, - source: { - kind: "database", - ref: shadow.sourceUrl, - connectOptions: { isLocal: true, dnsResolver: "native" }, - }, - target: { - kind: "database", - ref: target, - ...(shadow.targetUrlOverride === undefined ? { connection: resolved.conn } : {}), - connectOptions: { - isLocal: shadow.targetUrlOverride !== undefined || resolved.isLocal, - dnsResolver, - }, + // `legacyWithShadowDatabase` (`shadow-cache.ts`) rather than a bare + // `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` pair — see its doc comment for + // both halves of the rationale: why the lifecycle is an `Effect.acquireUseRelease` (an + // interrupt must not be able to land between creation and the finalizer being attached) and + // why the cache seam sits here (with `SUPABASE_SHADOW_CACHE` unset it IS today's + // create/remove pair; with it set it restores a key-matching PGDATA snapshot instead of + // rebuilding the platform baseline from scratch). + diffResult = yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => + Effect.gen(function* () { + const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); + const target = shadow.targetUrlOverride ?? targetUrl; + yield* output.raw( + flags.schema.length > 0 + ? `Diffing schemas: ${flags.schema.join(",")}\n` + : "Diffing schemas...\n", + "stderr", + ); + if (useDelta) { + const result = yield* pgDelta.diffDatabase({ + context: ctx, + source: { + kind: "database", + ref: shadow.sourceUrl, + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + target: { + kind: "database", + ref: target, + ...(shadow.targetUrlOverride === undefined ? { connection: resolved.conn } : {}), + connectOptions: { + isLocal: shadow.targetUrlOverride !== undefined || resolved.isLocal, + dnsResolver, }, - schema: flags.schema, - formatOptions, - debug: legacyIsPgDeltaDebugEnabled(), - strictCoverage: flags.strictCoverage, - }); - // Keep the per-unit plan files so a multi-unit plan can be written as one - // migration file each; `sql` stays the flattened join for stdout review + - // machine payloads. - return { sql: result.sql, files: result.files, hazards: result.hazards }; - } - const sql = yield* legacyDiffMigra(ctx, { - source: shadow.sourceUrl, - target, + }, schema: flags.schema, - connectOptions: { isLocal: resolved.isLocal, dnsResolver }, + formatOptions, + debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, }); - // The migra engine has no execution-aware plan units, so it always writes a - // single migration file. - return { sql, files: undefined }; - }), - (handle) => legacyRemoveShadowDatabase(spawner, handle.containerId), + // Keep the per-unit plan files so a multi-unit plan can be written as one + // migration file each; `sql` stays the flattened join for stdout review + + // machine payloads. + return { sql: result.sql, files: result.files, hazards: result.hazards }; + } + const sql = yield* legacyDiffMigra(ctx, { + source: shadow.sourceUrl, + target, + schema: flags.schema, + connectOptions: { isLocal: resolved.isLocal, dnsResolver }, + }); + // The migra engine has no execution-aware plan units, so it always writes a + // single migration file. + return { sql, files: undefined }; + }), ); } const out = diffResult.sql; diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 54d410debe..042fa293af 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -15,6 +15,7 @@ import { mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, } from "../../../../../tests/helpers/legacy-mocks.ts"; import { mockOutput, mockRuntimeInfo } from "../../../../../tests/helpers/mocks.ts"; @@ -509,6 +510,10 @@ const stderr = (out: ReturnType) => ); const tmp = useLegacyTempWorkdir(); +// The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker cp`/ +// `docker start` round trip plus a snapshot tar to every shadow this suite provisions. This suite +// is about the command, not the cache, so it asserts the plain shadow lifecycle. +useLegacyShadowCacheDisabled(); // --- native --use-pgadmin fixtures --- diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 1550e6efbb..70c59135cf 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -58,6 +58,7 @@ disables formatting without disabling safe compaction. | `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | | `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | shadow baseline cache enabled (default) — the shadow's PGDATA snapshot, ~90MB, one file for the current key only | | `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | | `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | @@ -67,11 +68,39 @@ disables formatting without disabling safe compaction. - Shadow Postgres container — provisioned and torn down natively (`legacyPrepareShadowSource` in `legacy/commands/db/shared/legacy-shadow-source.ts` / `legacyPrepareRawShadow` in `legacy/shared/db-bootstrap/shadow-database.ts`, which also owns the lower-level primitives - both build on), no longer via a Go seam. + both build on), no longer via a Go seam. Torn down with `docker rm -f -v` on every run, + cache or no cache — see the shadow baseline cache section below. - `supabase/migra` container — the migra OOM bash fallback only. - `pg_dump` container — the initial-migra pull's native remote-schema dump (`legacyStreamPgDump`, shared with `db dump`). +### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) + +- ON by default; set `SUPABASE_SHADOW_CACHE=false` (or `=0`) to opt out, in which case the shadow + lifecycle is exactly as documented above. +- The cached artifact is a **file**, never a container: `supabase/.temp/pgdelta/shadow-baseline-.tar` + (~90MB), a tar of the shadow's PGDATA directory taken right after the platform baseline and + before `contrib_regression`/any user migration. The key hashes every input baked into the cluster + (images, JWT secret, root key, `[db] password`, `db.settings`, vault secrets, `roles.sql`, + resolved JWKS, shadow port, major version). +- The container lifecycle is otherwise IDENTICAL to the uncached path — same project labels, always + `docker rm -f -v` on release. Nothing is kept between runs, there is no cache-key Docker label, + and no lock file. One forced exception: a COLD cache-enabled run creates the shadow without + `--rm`, because it must `docker stop` the container to take a coherent snapshot and Docker + destroys an `--rm` container the moment it exits. It is still removed on release. +- Cold run: `docker stop` -> `docker cp :/var/lib/postgresql/data -` streamed to the tar (temp + name + atomic rename) -> `docker start` -> readiness wait -> continue. Publishing a new key's tar + deletes every other `shadow-baseline-*.tar` (retention: current key only). +- Warm run: `docker cp - :/var/lib/postgresql` into the created-but-not-yet-started container, + so the entrypoint skips `initdb` and the platform baseline is skipped too. +- Any warm-path failure removes the container, deletes the tar as suspect, and cold-provisions; any + cold export failure only warns on stderr and leaves the run uncached. The cache never fails the + command. +- Each pooler-retry attempt acquires and releases its own shadow; on a warm hit each one restores + the same tar into its own fresh container. +- `--declarative`'s bare shadow runs no platform baseline, so there is nothing to snapshot — it + keeps the plain create/remove lifecycle. + ## API Routes / DB | Method | Path / SQL | Auth | Purpose | @@ -92,6 +121,7 @@ disables formatting without disabling safe compaction. | `SUPABASE_DB_MAJOR_VERSION` / `SUPABASE_DB_HEALTH_TIMEOUT` / `SUPABASE_DB_SETTINGS_*` | shadow container-config overrides, same as `db start`/`db reset` | no | | `SUPABASE_PROJECT_ID` | overrides the shadow container's project id/labels, same as `db start`/`db reset` (`utils.DbId`); ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta diff engine | no | | `SUPABASE_EXPERIMENTAL` | selects the deprecated structured-dump branch (still delegates to Go, see below) | no | | `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | diff --git a/apps/cli/src/legacy/commands/db/pull/pull.debug.ts b/apps/cli/src/legacy/commands/db/pull/pull.debug.ts index 0282f2b3b1..558ae84e6e 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.debug.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.debug.ts @@ -7,7 +7,7 @@ import { legacyDebugBundleMessage, legacySaveDebugBundle, } from "../shared/legacy-debug-bundle.ts"; -import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.cache.ts"; +import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.paths.ts"; import { type LegacyPgDeltaContext, legacyExportCatalogPgDelta, diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 813bc383ea..df1372271b 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -40,6 +40,7 @@ import { legacyBuildLocalDbContainerInputs, type LegacyLocalDbContainerInputs, } from "../../../shared/db-bootstrap/local-container-inputs.ts"; +import { legacyWithShadowDatabase } from "../../../shared/db-bootstrap/shadow-cache.ts"; import { legacyCreateShadowDatabase, legacyPrepareRawShadow, @@ -786,59 +787,62 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy pgDelta: toml.pgDelta, ctx, }; - // Register cleanup atomically with shadow acquisition. - return yield* Effect.acquireUseRelease( - legacyCreateShadowDatabase(spawner, shadowInput), - (handle) => - Effect.gen(function* () { - const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); - const target = shadow.targetUrlOverride ?? targetEndpoint.ref; - yield* output.raw( - diffSchema.length > 0 - ? `Diffing schemas: ${diffSchema.join(",")}\n` - : "Diffing schemas...\n", - "stderr", - ); - if (usePgDeltaDiff) { - return yield* pgDeltaEngine.diffDatabase({ - context: ctx, - source: { - kind: "database", - ref: shadow.sourceUrl, - connectOptions: { isLocal: true, dnsResolver: "native" }, - }, - target: { - kind: "database", - ref: target, - ...(shadow.targetUrlOverride === undefined - ? { - ...(targetEndpoint.connection !== undefined - ? { connection: targetEndpoint.connection } - : {}), - connectOptions: targetEndpoint.connectOptions, - } - : { - connectOptions: { isLocal: true, dnsResolver }, - }), - }, - schema: diffSchema, - formatOptions, - debug: legacyIsPgDeltaDebugEnabled(), - strictCoverage: flags.strictCoverage, - }); - } - const sql = yield* legacyDiffMigra(ctx, { - source: shadow.sourceUrl, - target, + // `legacyWithShadowDatabase` (`shadow-cache.ts`) rather than a bare + // `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` pair — see its doc + // comment for both halves of the rationale: why the lifecycle is an + // `Effect.acquireUseRelease` (an interrupt must not be able to land between creation + // and the finalizer being attached) and why the cache seam sits here. Note each + // pooler-retry attempt still acquires and releases its own shadow — on the warm path + // that is the SAME restored baseline snapshot, sequentially. + return yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => + Effect.gen(function* () { + const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); + const target = shadow.targetUrlOverride ?? targetEndpoint.ref; + yield* output.raw( + diffSchema.length > 0 + ? `Diffing schemas: ${diffSchema.join(",")}\n` + : "Diffing schemas...\n", + "stderr", + ); + if (usePgDeltaDiff) { + return yield* pgDeltaEngine.diffDatabase({ + context: ctx, + source: { + kind: "database", + ref: shadow.sourceUrl, + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + target: { + kind: "database", + ref: target, + ...(shadow.targetUrlOverride === undefined + ? { + ...(targetEndpoint.connection !== undefined + ? { connection: targetEndpoint.connection } + : {}), + connectOptions: targetEndpoint.connectOptions, + } + : { + connectOptions: { isLocal: true, dnsResolver }, + }), + }, schema: diffSchema, - connectOptions: - shadow.targetUrlOverride === undefined - ? targetEndpoint.connectOptions - : { isLocal: true, dnsResolver }, + formatOptions, + debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, }); - return { sql, files: undefined, debug: undefined }; - }), - (handle) => legacyRemoveShadowDatabase(spawner, handle.containerId), + } + const sql = yield* legacyDiffMigra(ctx, { + source: shadow.sourceUrl, + target, + schema: diffSchema, + connectOptions: + shadow.targetUrlOverride === undefined + ? targetEndpoint.connectOptions + : { isLocal: true, dnsResolver }, + }); + return { sql, files: undefined, debug: undefined }; + }), ); }); const diffOutcome = yield* withPoolerFallback(targetEndpoint, runShadowDiff); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index abc622ea88..9fb0bf42b2 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -13,6 +13,7 @@ import { mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, } from "../../../../../tests/helpers/legacy-mocks.ts"; import { @@ -513,6 +514,10 @@ const seedMigration = (workdir: string, version: string) => { }; const tmp = useLegacyTempWorkdir(); +// The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker cp`/ +// `docker start` round trip plus a snapshot tar to every shadow this suite provisions. This suite +// is about the command, not the cache, so it asserts the plain shadow lifecycle. +useLegacyShadowCacheDisabled(); describe("legacy db pull", () => { it.effect("pulls a migration (pgdelta engine) and updates remote history under --yes", () => { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts index 47788d9346..876270f086 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts @@ -5,7 +5,10 @@ import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; import { Cause, Effect, Exit, FileSystem, Layer, Option, Path } from "effect"; -import { mockLegacyShadowContainerCliSpawner } from "../../../../../../tests/helpers/legacy-mocks.ts"; +import { + mockLegacyShadowContainerCliSpawner, + useLegacyShadowCacheDisabled, +} from "../../../../../../tests/helpers/legacy-mocks.ts"; import { alwaysReadyHttpClientLayer } from "../../../../../../tests/helpers/legacy-local-reset.ts"; import { mockOutput, mockRuntimeInfo } from "../../../../../../tests/helpers/mocks.ts"; import { CliArgs } from "../../../../../shared/cli/cli-args.service.ts"; @@ -389,6 +392,10 @@ const toml: LegacyDbTomlValues = { }; describe("legacyDiffDeclarativeToMigrations", () => { + // The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker + // cp`/`docker start` round trip plus a snapshot tar to every shadow this suite provisions. This + // suite is about the orchestration, not the cache, so it asserts the plain shadow lifecycle. + useLegacyShadowCacheDisabled(); it.effect( "resolves the migrations catalog natively and diffs it against the seam-provisioned declarative catalog", () => { diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 5b14e25304..437b57bf2e 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -28,12 +28,13 @@ disabling safe compaction. ## Files Written -| Path | Format | When | -| ------------------------------------------------------------------ | ------ | ------------------------------------------------- | -| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | -| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| ------------------------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------------ | +| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | +| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | shadow baseline cache enabled (default) — the shadow's PGDATA snapshot, ~90MB, current key only | ## Subprocesses / Containers @@ -47,13 +48,14 @@ disabling safe compaction. ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------- | -------------------------------------------------- | --------- | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | -| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | -| `SUPABASE_SERVICES_HOSTNAME` | local DB host for the bootstrap generate | no | -| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | +| Variable | Purpose | Required? | +| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | +| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | +| `SUPABASE_SERVICES_HOSTNAME` | local DB host for the bootstrap generate | no | +| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | ## Exit Codes @@ -119,3 +121,32 @@ existing SQL or creates an export manifest. shadows. Under the legacy opt-out, both catalog shadows are provisioned in-process using the same primitives as `db diff`; catalog export, declarative apply, and diff run through the edge-runtime pg-delta scripts. + +### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) + +The migrations-catalog shadow this command provisions on a cache miss goes through +`legacyGetMigrationsCatalogRef` -> `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), which +uses the same `legacyWithShadowDatabase` seam `db diff`/`db pull` do — so this command reads +`SUPABASE_SHADOW_CACHE` and inherits its whole lifecycle: + +- ON by default; set `SUPABASE_SHADOW_CACHE=false` (or `=0`) to opt out, in which case the shadow + lifecycle is exactly as documented above. +- The cached artifact is a **file**, never a container: `supabase/.temp/pgdelta/shadow-baseline-.tar` + (~90MB), a tar of the shadow's PGDATA directory taken right after the platform baseline and + before `contrib_regression`/any user migration. The key hashes every input baked into the cluster + (images, JWT secret, root key, `[db] password`, `db.settings`, vault secrets, `roles.sql`, + resolved JWKS, shadow port, major version). +- The container lifecycle is otherwise IDENTICAL to the uncached path — same project labels, always + `docker rm -f -v` on release. Nothing is kept between runs, there is no cache-key Docker label, + and no lock file. One forced exception: a COLD cache-enabled run creates the shadow without + `--rm`, because it must `docker stop` the container to take a coherent snapshot and Docker + destroys an `--rm` container the moment it exits. It is still removed on release. +- Cold run: `docker stop` -> `docker cp :/var/lib/postgresql/data -` streamed to the tar (temp + name + atomic rename) -> `docker start` -> readiness wait -> continue. Publishing a new key's tar + deletes every other `shadow-baseline-*.tar` (retention: current key only). +- Warm run: `docker cp - :/var/lib/postgresql` into the created-but-not-yet-started container, + so the entrypoint skips `initdb` and the platform baseline is skipped too. +- Any warm-path failure removes the container, deletes the tar as suspect, and cold-provisions; any + cold export failure only warns on stderr and leaves the run uncached. The cache never fails the + command. +- The declarative-catalog shadow is NOT cached — it is provisioned and torn down per run. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts index 8525ef6eaa..34c7abf1c2 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts @@ -26,9 +26,9 @@ import { LegacyLinkedProjectCache } from "../../../../../telemetry/legacy-linked import { LegacyTelemetryState } from "../../../../../telemetry/legacy-telemetry-state.service.ts"; import { legacyListLocalMigrations, - legacyPgDeltaTempPath, legacyResolveSetupInputs, } from "../../../../../shared/legacy-pgdelta.cache.ts"; +import { legacyPgDeltaTempPath } from "../../../../../shared/legacy-pgdelta.paths.ts"; import { LegacyPgDeltaEngine } from "../../../shared/legacy-pgdelta-engine.service.ts"; import { legacyIsPgDeltaDebugEnabled, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts index 66f9ec2cb4..3018434226 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts @@ -24,6 +24,7 @@ import { mockLegacyLinkedProjectCacheTracked, mockLegacyPlatformApiService, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, } from "../../../../../../../tests/helpers/legacy-mocks.ts"; import { CliArgs } from "../../../../../../shared/cli/cli-args.service.ts"; @@ -432,6 +433,10 @@ const legacyUuidLoadError = () => describe("legacy db schema declarative sync integration", () => { const tmp = useLegacyTempWorkdir(); + // The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker cp`/ + // `docker start` round trip plus a snapshot tar to every shadow this suite provisions. This suite + // is about the command, not the cache, so it asserts the plain shadow lifecycle. + useLegacyShadowCacheDisabled(); it.effect("gate: fails when pg-delta is not enabled", () => { seedDeclarative(tmp.current); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts index ebff9b0a4a..222beda809 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts @@ -1,6 +1,6 @@ import { Effect, type FileSystem, type Path } from "effect"; -import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.cache.ts"; +import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.paths.ts"; import type { LegacyPgDeltaNextDiagnostic, LegacyPgDeltaNextOperation, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts index 0c58965625..40a77a4d32 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts @@ -10,7 +10,7 @@ import { legacyPgDeltaNextTempPath, legacySavePgDeltaNextDebugArtifacts, } from "./legacy-pgdelta-next-artifacts.ts"; -import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.cache.ts"; +import { legacyPgDeltaTempPath } from "../../../shared/legacy-pgdelta.paths.ts"; describe("pg-delta next artifact generation", () => { it.effect("writes structured non-cache artifacts and metadata under v2", () => { diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts index 5b144ac1de..7ff60172bc 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts @@ -46,11 +46,11 @@ import { legacyWaitForShadowReady } from "../../../shared/db-bootstrap/health-ch import { legacySeedGlobals } from "../../../shared/legacy-migration-apply.ts"; import { LEGACY_BAD_PATTERN_MESSAGE, legacyPathMatch } from "../../../shared/legacy-path-match.ts"; import { legacyToPostgresURL } from "../../../shared/legacy-postgres-url.ts"; +import type { LegacyShadowAcquiredHandle } from "../../../shared/db-bootstrap/shadow-cache.ts"; import { legacyMigrateShadowDatabase, legacyMigrateNextShadowDatabase, LegacyShadowDbError, - type LegacyShadowDatabaseHandle, type LegacyShadowSetupInput, type LegacyShadowSourceResult, } from "../../../shared/db-bootstrap/shadow-database.ts"; @@ -126,7 +126,7 @@ export type LegacyPrepareShadowSourceError = */ export const legacyPrepareShadowSource = ( spawner: Spawner, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, input: LegacyPrepareShadowSourceInput, ): Effect.Effect< LegacyShadowSourceResult, @@ -160,20 +160,29 @@ export const legacyPrepareShadowSource = ( timeoutSeconds: input.healthTimeoutSeconds, }); + // `handle` doubles as the baseline state: on a warm shadow-cache hit the cluster it carries + // already holds the platform baseline (so only the template database + user migrations run), + // and on a cache-enabled cold provision it carries the snapshot step that runs between the + // two — see `shadow-cache.ts`/`LegacyShadowBaselineState`. An uncached acquire hands over the + // always-cold state, which reproduces today's sequence exactly. const migrateShadow = input.migrationMode === "pgdelta-next" ? legacyMigrateNextShadowDatabase : legacyMigrateShadowDatabase; - yield* migrateShadow(spawner, { - fs: input.fs, - path: input.path, - workdir: input.workdir, - projectId: input.projectId, - container: containerId, - networkId: input.networkId, - connConfig, - setup: input.setup, - }); + yield* migrateShadow( + spawner, + { + fs: input.fs, + path: input.path, + workdir: input.workdir, + projectId: input.projectId, + container: containerId, + networkId: input.networkId, + connConfig, + setup: input.setup, + }, + handle, + ); const sourceUrl = legacyToPostgresURL(connConfig); diff --git a/apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts b/apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts index cac1d8f1d9..2d8eff9410 100644 --- a/apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts +++ b/apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts @@ -15,6 +15,7 @@ import { mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, mockLegacyTelemetryStateTracked, + useLegacyShadowCacheDisabled, useLegacyTempWorkdir, } from "../../../../../tests/helpers/legacy-mocks.ts"; import { @@ -428,6 +429,10 @@ const failureTag = (exit: Exit.Exit): string | undefined => { }; const tmp = useLegacyTempWorkdir(); +// The shadow baseline cache is ON by default and would otherwise add a `docker stop`/`docker cp`/ +// `docker start` round trip plus a snapshot tar to every shadow this suite provisions. This suite +// is about the command, not the cache, so it asserts the plain shadow lifecycle. +useLegacyShadowCacheDisabled(); describe("legacy migration squash", () => { describe("flag surface & ordering", () => { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts index 2714e8bad2..66bbff6085 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts @@ -45,6 +45,11 @@ import { /** Structural element type of {@link LegacyStartContainerSpec.secretFiles} — not exported from `docker-create-args.ts`, so referenced positionally here. */ type LegacyStartSecretFileSpec = NonNullable[number]; +/** Structural element type of {@link LegacyStartContainerSpec.preStartArchives} — same reasoning as {@link LegacyStartSecretFileSpec}. */ +type LegacyStartPreStartArchiveSpec = NonNullable< + LegacyStartContainerSpec["preStartArchives"] +>[number]; + type Spawner = ChildProcessSpawner["Service"]; /** @@ -765,6 +770,50 @@ function legacyCopyStartSecretFilesIntoContainer( ); } +/** + * `docker cp - :` with one + * {@link LegacyStartContainerSpec.preStartArchives} entry's tar bytes on stdin — the ONE `docker + * cp` form that preserves each archive member's uid/gid inside the container (the host-path form + * rewrites ownership to root, which a restored Postgres data directory cannot survive; see that + * field's own doc comment). + * + * Sequenced by {@link legacyCreateContainer} between `docker create` and `docker start`, for the + * same two reasons the secret-file copies are: the container must exist for `docker cp` to have a + * target, and must not be running yet so its entrypoint never races the copy — which for an + * archive is not merely a race but the whole point, since the entrypoint's behavior depends on + * what it finds already unpacked. + */ +function legacyExtractPreStartArchiveIntoContainer( + spawner: Spawner, + containerId: string, + archive: LegacyStartPreStartArchiveSpec, +): Effect.Effect { + const failure = (detail: string) => + new LegacyContainerCreateError({ + message: `failed to create docker container: failed to restore archive into container${detail}`, + reason: "runtime", + }); + return Effect.scoped( + Effect.gen(function* () { + const child = yield* spawnContainerCli( + spawner, + ["cp", "-", `${containerId}:${archive.containerPath}`], + { stdin: archive.tar, stdout: "ignore", stderr: "pipe" }, + ).pipe(Effect.mapError((cause) => failure(`: ${legacyDescribeContainerCliFailure(cause)}`))); + const [exitCode, stderr] = yield* Effect.all( + [child.exitCode.pipe(Effect.map(Number)), legacyCollectText(child.stderr)], + { concurrency: "unbounded" }, + ).pipe(Effect.mapError(() => failure(""))); + if (exitCode !== 0) { + const message = stderr.trim(); + return yield* Effect.fail( + failure(message.length > 0 ? `: ${message}` : `: exit ${exitCode}`), + ); + } + }), + ); +} + /** * Port of Go's `DockerStart` (`apps/cli-go/internal/utils/docker.go:363-440`), * minus image resolution (already done by `image-prepull.ts`) and network @@ -783,7 +832,11 @@ function legacyCopyStartSecretFilesIntoContainer( * Runs strictly between `docker create` and `docker start`: the container * must already exist for `docker cp` to have a target, and must not be * running yet so its entrypoint never races the copy. - * 6. `docker start`. + * 6. Unpack any `preStartArchives` into the same created-but-unstarted + * container via `docker cp -` (`legacyExtractPreStartArchiveIntoContainer`) + * — also TS-port-only, and for the shadow baseline cache's restored PGDATA + * the "not started yet" half of step 5's ordering is the entire point. + * 7. `docker start`. * * Resolves to the created container's id/name on success. */ @@ -837,6 +890,13 @@ export function legacyCreateContainer( containerId, finalSpec.secretFiles ?? [], ); + // Sequentially, not concurrently like the secret files: two archives could legitimately + // overlap in the container's filesystem, so the spec's own order has to be the applied order. + yield* Effect.forEach( + finalSpec.preStartArchives ?? [], + (archive) => legacyExtractPreStartArchiveIntoContainer(spawner, containerId, archive), + { discard: true }, + ); yield* legacyDockerStartContainer(spawner, containerId, finalSpec); return containerId; }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts b/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts index 7219fe2b9b..99bae326ef 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts @@ -45,14 +45,20 @@ * call site actually needs one; there is no value in modelling Docker surface * this builder never has to reproduce. * - * One field has no Go struct equivalent at all: {@link LegacyStartContainerSpec.secretFiles}. - * It exists purely because this module's own "shell out to `docker create`" - * architecture (unlike Go's direct Engine API calls) has an argv-exposure - * problem `container.Config`/`container.HostConfig` never had — see that - * field's doc comment, and `container-lifecycle.ts`'s `legacyCreateContainer`, - * for the mitigation. + * Two fields have no Go struct equivalent at all: + * {@link LegacyStartContainerSpec.secretFiles} and + * {@link LegacyStartContainerSpec.preStartArchives}. The first exists purely + * because this module's own "shell out to `docker create`" architecture (unlike + * Go's direct Engine API calls) has an argv-exposure problem + * `container.Config`/`container.HostConfig` never had; the second because Go + * destroys its shadow container on every run and so never needs to seed a + * container's filesystem before it starts. See each field's doc comment, and + * `container-lifecycle.ts`'s `legacyCreateContainer`, for how both are + * delivered. */ +import type { PlatformError, Stream } from "effect"; + import { legacyBindMountSpecSource, legacyIsBindMountSource, @@ -123,6 +129,26 @@ interface LegacyStartSecretFileSpec { readonly content: string; } +/** + * One tar archive to extract into the created-but-not-yet-started container — see + * {@link LegacyStartContainerSpec.preStartArchives}'s doc comment for the full contract. Not + * exported on its own: callers reference it structurally through that field. + */ +interface LegacyStartPreStartArchiveSpec { + /** + * The directory INSIDE the container the archive's members are unpacked relative to, i.e. + * `docker cp - :`. Always a POSIX container path, never a host path. + */ + readonly containerPath: string; + /** + * The tar bytes, as a lazily-consumed stream (typically `FileSystem.stream(hostTarPath)`). + * A stream rather than a host path on purpose: it keeps the archive's own storage decisions + * with the producer, and `legacyCreateContainer` needs no `FileSystem` in its own context to + * deliver it. + */ + readonly tar: Stream.Stream; +} + export interface LegacyStartContainerSpec { /** `container.Config.Image` (already resolved/pulled — resolution is out of scope here). */ readonly image: string; @@ -192,6 +218,25 @@ export interface LegacyStartContainerSpec { * over the Engine API) for that same reason. */ readonly secretFiles?: ReadonlyArray; + /** + * Tar archives to unpack into the container's own filesystem AFTER `docker create` and + * strictly BEFORE `docker start` — the shape `docker cp - :` (tar on + * stdin) implements. No Go equivalent: Go's shadow container is destroyed on every run, so + * nothing in the Go CLI ever seeds a container's filesystem ahead of its entrypoint. + * + * The one consumer today is the shadow baseline cache (`shadow-cache.ts`), which restores a + * previously exported PGDATA directory so the `supabase/postgres` entrypoint finds a + * `PG_VERSION` file and skips `initdb` entirely. The delivery form matters and is not + * interchangeable with {@link secretFiles}': `docker cp :` resets the + * copied tree's ownership to the archive-extracting user (root), and Postgres refuses to + * start on a data directory it does not own, whereas the tar-STREAM form preserves each + * member's uid/gid verbatim. + * + * NOT consumed here — {@link legacyBuildStartContainerCreateArgs} stays pure/no-I/O and never + * reads this field, exactly like {@link secretFiles}. `container-lifecycle.ts`'s + * `legacyCreateContainer` is the sole consumer. + */ + readonly preStartArchives?: ReadonlyArray; /** * `container.Config.Entrypoint`'s first element. Docker CLI's `--entrypoint` * only accepts a single executable/script name (unlike the Engine API field, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts index 68b86e15da..656d91ddd8 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts @@ -10,7 +10,7 @@ * and only the final timeout's failures surface to the caller. */ -import { Data, Effect, Schedule, Stream } from "effect"; +import { Clock, Data, Effect, Result, Schedule, Stream } from "effect"; import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientRequest from "effect/unstable/http/HttpClientRequest"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; @@ -27,6 +27,7 @@ import { import { LegacyDbConnection, type LegacyPgConnInput } from "../legacy-db-connection.service.ts"; import { legacyInspectContainerState } from "../legacy-docker-lifecycle.ts"; import { legacyKongAuthHeaders } from "../legacy-kong-auth.ts"; +import { legacyShadowDebugEnabled, legacyShadowDebugTruncate } from "./shadow-debug.ts"; type Spawner = ChildProcessSpawner["Service"]; @@ -523,8 +524,15 @@ export function legacyWaitForShadowReady( opts: LegacyWaitForShadowReadyOptions = {}, ): Effect.Effect { const timeoutSeconds = opts.timeoutSeconds ?? LEGACY_HEALTH_CHECK_TIMEOUT_SECONDS; - - const probe: Effect.Effect = Effect.gen( + // Checked once per call (never cached at module load — a test, or a long-lived process, + // mutates `process.env` and expects the next call to see it) — see `shadow-debug.ts`'s own + // doc comment. `Output` is not in this function's own context (only `LegacyDbConnection` is), + // so debug lines here go straight to stderr via `process.stderr.write` rather than widening + // this function's `R` just for a debug-only concern — the same tradeoff + // `legacyStreamContainerLogsOnce` above already makes for its own log-teeing. + const debug = legacyShadowDebugEnabled(); + + const rawProbe: Effect.Effect = Effect.gen( function* () { const state = yield* legacyInspectContainerState(spawner, containerId).pipe( Effect.mapError((cause) => legacyShadowNotReady(cause.message)), @@ -539,12 +547,40 @@ export function legacyWaitForShadowReady( }, ); + let attempts = 0; + // The most recent attempt's failure reason, kept even once a later attempt succeeds — the + // summary line reports it either way (see the doc comment below on the completion line). + let lastError: string | undefined; + + // Debug-only per-attempt line: `shadow-debug: ready-attempt ms `. + // Exploration 1a's key data point — whether a cold attempt burns the full connect timeout. + const probe: Effect.Effect = !debug + ? rawProbe + : Effect.gen(function* () { + attempts += 1; + const attemptNumber = attempts; + const start = yield* Clock.currentTimeMillis; + const outcome = yield* Effect.result(rawProbe); + const elapsed = (yield* Clock.currentTimeMillis) - start; + if (Result.isFailure(outcome)) lastError = outcome.failure.reason; + yield* Effect.sync(() => { + globalThis.process.stderr.write( + `shadow-debug: ready-attempt ${attemptNumber} ${elapsed}ms ${ + Result.isSuccess(outcome) + ? "ok" + : `error: ${legacyShadowDebugTruncate(outcome.failure.reason)}` + }\n`, + ); + }); + return yield* Effect.fromResult(outcome); + }); + // Same policy as `legacyWaitForHealthyServices` (Go's // `NewBackoffPolicy(ctx, timeout)`): a 1-second constant delay, capped at // `timeoutSeconds` retries after the initial attempt. const schedule = Schedule.max([Schedule.spaced("1 seconds"), Schedule.recurs(timeoutSeconds)]); - return probe.pipe( + const waited: Effect.Effect = probe.pipe( Effect.retry({ schedule, while: (failure) => !failure.fatal }), Effect.catch((failure) => Effect.gen(function* () { @@ -562,4 +598,22 @@ export function legacyWaitForShadowReady( }), ), ); + + if (!debug) return waited; + + // Debug-only completion line, emitted on success OR failure: + // `shadow-debug: ready-wait ms attempts=[ last-error=""]`. + return Effect.gen(function* () { + const start = yield* Clock.currentTimeMillis; + const outcome = yield* Effect.result(waited); + const elapsed = (yield* Clock.currentTimeMillis) - start; + const lastErrorSegment = + lastError === undefined ? "" : ` last-error="${legacyShadowDebugTruncate(lastError)}"`; + yield* Effect.sync(() => { + globalThis.process.stderr.write( + `shadow-debug: ready-wait ${elapsed}ms attempts=${attempts}${lastErrorSegment}\n`, + ); + }); + return yield* Effect.fromResult(outcome); + }); } diff --git a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts index c94bd73c6e..6df84a1346 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts @@ -263,7 +263,7 @@ function legacyPostgresExtraEnv( * `docker stop` burns the full 10s grace period before SIGKILL; with `exec`, * Postgres is PID 1 and stops in ~1s). Applies to all three entrypoint * variants below. Timing is not part of the Go-parity surface (ADR 0016); - * see apps/cli/docs/shadow-db-startup-perf-plan.md, workstream B. + * see apps/cli/docs/shadow-db-provisioning.md. * * Otherwise byte-for-byte derived from Go's raw-string concatenation — * `NewContainerConfig(args ...string)` splices `strings.Join(args, " ")` diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts new file mode 100644 index 0000000000..33d86b4115 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -0,0 +1,666 @@ +/** + * The shadow baseline cache's acquire/export/restore flow, driven end to end against a tiny + * in-test Docker model (create/start/stop/rm all mutate the same container table, and `docker cp` + * really moves bytes in and out of it) plus the REAL filesystem under a per-test temp workdir, so + * the tar artifact, its atomic publish, and its retention rule are exercised for real. + * + * Scenario-oriented on purpose: every test is a sequence of real acquires and releases, and the + * assertions are on the resulting Docker state, the tar on disk, and what the caller was told + * about the baseline — not on internal call ordering, except where the ordering IS the contract + * (the export must stop the container before copying and start it again afterwards). + */ + +import type { ProjectConfig } from "@supabase/config"; +import { ProjectConfigSchema } from "@supabase/config"; +import { BunServices } from "@effect/platform-bun"; +import { describe, expect, it } from "@effect/vitest"; +import { Effect, FileSystem, Layer, Option, Path, Predicate, Schema, Sink, Stream } from "effect"; +import { ChildProcess, ChildProcessSpawner } from "effect/unstable/process"; + +import { useLegacyTempWorkdir } from "../../../../tests/helpers/legacy-mocks.ts"; +import { mockOutput } from "../../../../tests/helpers/mocks.ts"; +import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; +import { LegacyDbConnectError } from "../legacy-db-connection.errors.ts"; +import { + LEGACY_SHADOW_CACHE_ENV, + LEGACY_SHADOW_PGDATA_PARENT_PATH, + LEGACY_SHADOW_PGDATA_PATH, + legacyAcquireShadowDatabase, +} from "./shadow-cache.ts"; +import { LEGACY_SHADOW_DEBUG_ENV } from "./shadow-debug.ts"; +import { legacyRemoveShadowDatabase } from "./shadow-database.ts"; +import type { LegacyShadowDbSetupInput, LegacyShadowSetupInput } from "./shadow-database.ts"; + +const decodeConfig = Schema.decodeUnknownSync(ProjectConfigSchema); +const defaultConfig: ProjectConfig = decodeConfig({}); + +const tempRoot = useLegacyTempWorkdir("legacy-shadow-cache-"); + +/** Sets an env var for the duration of `body`, restoring whatever the host had. */ +const withEnv = ( + name: string, + value: string | undefined, + body: Effect.Effect, +): Effect.Effect => + Effect.acquireUseRelease( + Effect.sync(() => { + const previous = process.env[name]; + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + return previous; + }), + () => body, + (previous) => + Effect.sync(() => { + if (previous === undefined) delete process.env[name]; + else process.env[name] = previous; + }), + ); + +const withShadowCacheEnv = (value: string | undefined, body: Effect.Effect) => + withEnv(LEGACY_SHADOW_CACHE_ENV, value, body); + +const withShadowDebugEnv = (value: string | undefined, body: Effect.Effect) => + withEnv(LEGACY_SHADOW_DEBUG_ENV, value, body); + +/** + * Captures every write `body` makes directly to the real `process.stderr` — the channel + * `legacyWaitForShadowReady`'s own `ready-attempt`/`ready-wait` debug lines use, since that + * function has no `Output` in its context (see `health-check.ts`'s own doc comment). Mirrors + * `health-check.unit.test.ts`'s own capture/restore pattern. + */ +const captureStderr = ( + body: Effect.Effect, +): Effect.Effect<{ readonly result: A; readonly writes: ReadonlyArray }, E, R> => + Effect.gen(function* () { + const writes: Array = []; + const originalWrite = globalThis.process.stderr.write.bind(globalThis.process.stderr); + globalThis.process.stderr.write = ((chunk: string | Uint8Array) => { + writes.push(typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk)); + return true; + }) as typeof globalThis.process.stderr.write; + const result = yield* body.pipe( + Effect.ensuring( + Effect.sync(() => { + globalThis.process.stderr.write = originalWrite; + }), + ), + ); + return { result, writes }; + }); + +// --------------------------------------------------------------------------- +// A minimal, stateful Docker model +// --------------------------------------------------------------------------- + +/** The bytes the fake `docker cp :PGDATA -` emits — stands in for a real ~90MB PGDATA tar. */ +const FAKE_PGDATA_TAR = "data/PG_VERSION\n17\n"; + +interface FakeContainer { + readonly labels: Readonly>; + readonly autoRemove: boolean; + running: boolean; + /** What a previous `docker cp - :` unpacked into this container, if anything. */ + restored: string | undefined; +} + +function fakeDockerDaemon( + opts: { + readonly failStart?: boolean; + readonly failCopyOut?: boolean; + readonly failCopyIn?: boolean; + } = {}, +) { + const containers = new Map(); + const spawned: Array> = []; + const encoder = new TextEncoder(); + const decoder = new TextDecoder(); + let nextId = 0; + + /** + * Drains a `Stream` passed as a command's `stdin` into a string, the way the real + * `NodeChildProcessSpawner` runs a user-supplied stdin stream into the child's stdin sink — a + * fake that ignored it would never notice the restore tar was not actually delivered. + */ + const readStdin = Effect.fnUntraced(function* (command: ChildProcess.Command) { + const configured = command._tag === "StandardCommand" ? command.options.stdin : undefined; + // A caller may pass either a bare `CommandInput` or a `StdinConfig` wrapping one. + const input = Stream.isStream(configured) + ? configured + : Predicate.hasProperty(configured, "stream") + ? configured.stream + : undefined; + if (!Stream.isStream(input)) return ""; + return yield* Stream.runFold( + input, + () => "", + (text: string, chunk) => text + decoder.decode(chunk as Uint8Array, { stream: true }), + ).pipe(Effect.orElseSucceed(() => "")); + }); + + const spawner = ChildProcessSpawner.make((command) => + Effect.gen(function* () { + const args = command._tag === "StandardCommand" ? command.args : []; + spawned.push(args); + + let exitCode = 0; + let stdout = ""; + let stderr = ""; + + if (args[0] === "network" && args[1] === "inspect") { + exitCode = 1; + } else if (args[0] === "create") { + nextId += 1; + const id = `shadowcontainer${String(nextId).padStart(2, "0")}`.padEnd(64, "0"); + const labels: Record = {}; + for (let index = 0; index < args.length; index += 1) { + if (args[index] !== "--label") continue; + const [key = "", ...rest] = (args[index + 1] ?? "").split("="); + labels[key] = rest.join("="); + } + containers.set(id, { + labels, + autoRemove: args.includes("--rm"), + running: false, + restored: undefined, + }); + stdout = id; + } else if (args[0] === "start") { + const container = containers.get(args[1] ?? ""); + if (opts.failStart === true || container === undefined) exitCode = 1; + else container.running = true; + } else if (args[0] === "stop") { + const id = args[1] ?? ""; + const container = containers.get(id); + if (container === undefined) exitCode = 1; + else { + container.running = false; + // Docker destroys an `--rm` container the moment it exits, `docker stop` included — + // verified against Docker 29. This is exactly why the cold export path drops `--rm`. + if (container.autoRemove) containers.delete(id); + } + } else if (args[0] === "rm") { + containers.delete(args[args.length - 1] ?? ""); + } else if (args[0] === "cp" && args[1] === "-") { + // Restore: `docker cp - :`, tar on stdin. + const [id = "", containerPath = ""] = (args[2] ?? "").split(":"); + const container = containers.get(id); + const received = yield* readStdin(command); + if (opts.failCopyIn === true || container === undefined) { + exitCode = 1; + stderr = "no such container"; + } else { + container.restored = `${containerPath}::${received}`; + } + } else if (args[0] === "cp" && args[2] === "-") { + // Export: `docker cp : -`, tar on stdout. + const [id = ""] = (args[1] ?? "").split(":"); + const container = containers.get(id); + if (opts.failCopyOut === true || container === undefined) { + exitCode = 1; + stderr = "no such container"; + } else { + stdout = FAKE_PGDATA_TAR; + } + } else if (args[0] === "container" && args[1] === "inspect") { + const container = containers.get(args[2] ?? ""); + exitCode = container === undefined ? 1 : 0; + stdout = + container === undefined + ? "" + : JSON.stringify({ + Running: container.running, + Status: container.running ? "running" : "exited", + Health: { Status: "healthy" }, + }); + } + + return ChildProcessSpawner.makeHandle({ + pid: ChildProcessSpawner.ProcessId(1), + // `docker cp … -` writes the raw archive, every other call a trailing newline. + stdout: Stream.fromIterable(stdout.length > 0 ? [encoder.encode(stdout)] : []), + stderr: Stream.fromIterable(stderr.length > 0 ? [encoder.encode(stderr)] : []), + all: Stream.empty, + exitCode: Effect.succeed(ChildProcessSpawner.ExitCode(exitCode)), + isRunning: Effect.succeed(false), + stdin: Sink.drain, + kill: () => Effect.void, + unref: Effect.succeed(Effect.void), + getInputFd: () => Sink.drain, + getOutputFd: () => Stream.empty, + }); + }), + ); + + /** + * One readable label per Docker call, so a test can assert the SEQUENCE of meaningful steps + * rather than raw argv. `cp` is split three ways because the shadow issues three different + * copies: the pgsodium root key every shadow gets (`cp-secret`, `container-lifecycle.ts`), the + * baseline export (`cp-out`), and the baseline restore (`cp-in`). + */ + const stepOf = (args: ReadonlyArray): string => { + if (args[0] === "network") return "network"; + if (args[0] === "container" && args[1] === "inspect") return "inspect"; + if (args[0] === "cp") { + if (args[1] === "-") return "cp-in"; + if (args[2] === "-") return "cp-out"; + return "cp-secret"; + } + return args[0] ?? ""; + }; + + return { + spawner, + spawned, + containers, + /** Every argv whose first token is `verb` (`create`/`rm`/`stop`/`start`/`cp`). */ + calls: (verb: string) => spawned.filter((args) => args[0] === verb), + /** Every argv classified as `step` — see {@link stepOf}. */ + stepCalls: (step: string) => spawned.filter((args) => stepOf(args) === step), + /** The full step sequence, with the `network` bookkeeping calls dropped as noise. */ + steps: () => spawned.map(stepOf).filter((step) => step !== "network"), + ids: () => [...containers.keys()], + }; +} + +// --------------------------------------------------------------------------- +// A fake Postgres the readiness probe can connect to +// --------------------------------------------------------------------------- + +function fakeCluster(opts: { readonly failConnect?: boolean } = {}) { + const connected: Array = []; + const layer = Layer.succeed(LegacyDbConnection, { + connect: (cfg) => + Effect.suspend(() => { + connected.push(cfg.database); + return opts.failConnect === true + ? Effect.fail(new LegacyDbConnectError({ message: "connection refused" })) + : Effect.succeed({ + exec: () => Effect.void, + query: () => Effect.succeed([]), + extensionExists: () => Effect.succeed(false), + copyToCsv: () => Effect.succeed(new Uint8Array()), + queryRaw: () => Effect.succeed({ fields: [], rows: [], commandTag: "" }), + }); + }), + }); + return { layer, connected }; +} + +// --------------------------------------------------------------------------- +// Inputs +// --------------------------------------------------------------------------- + +const shadowSetup = (): LegacyShadowDbSetupInput => ({ + majorVersion: 17, + config: defaultConfig, + dbUrl: "postgresql://postgres:postgres@127.0.0.1:54320/postgres", + jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", + jwks: Effect.succeed('{"keys":[]}'), + apiUrl: "http://127.0.0.1:54321", + authExternalUrl: undefined, + siteUrl: defaultConfig.auth.site_url, + anonKey: "anon-key", + serviceRoleKey: "service-role-key", + storageTargetMigration: "", + realtimeEnabledForSetup: false, + storageEnabledForSetup: false, + authEnabledForSetup: false, + serviceVersionOverrides: {}, + projectEnvValues: undefined, + debug: false, + webhooksEnabled: false, + apiAutoExposeNewTables: Option.some(true), + vault: [], +}); + +const shadowInput = ( + fs: FileSystem.FileSystem, + path: Path.Path, + overrides: { readonly shadowPort?: number } = {}, +): LegacyShadowSetupInput => ({ + db: { major_version: 17, settings: {} }, + experimental: defaultConfig.experimental, + jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", + jwtExpiry: 3600, + networkId: "supabase_network_proj", + image: "public.ecr.aws/supabase/postgres:17.4.1.030", + configImage: "supabase/postgres:17.4.1.030", + shadowPort: overrides.shadowPort ?? 54320, + password: "postgres", + projectId: "proj", + isBitbucketPipeline: false, + workdir: tempRoot.current, + extraHosts: [], + fs, + path, + hostname: "127.0.0.1", + healthTimeoutSeconds: 2, + setup: shadowSetup(), +}); + +const pgDeltaTempDir = (path: Path.Path) => + path.join(tempRoot.current, "supabase", ".temp", "pgdelta"); + +/** The one snapshot tar in the temp dir, whatever key it belongs to. */ +const soleTarName = Effect.fnUntraced(function* (fs: FileSystem.FileSystem, path: Path.Path) { + const entries = yield* fs + .readDirectory(pgDeltaTempDir(path)) + .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); + return entries.filter((entry) => entry.endsWith(".tar")); +}); + +/** A full cold run: acquire, export the baseline, release. */ +const coldRun = ( + docker: ReturnType, + input: LegacyShadowSetupInput, +) => + Effect.gen(function* () { + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + yield* handle.snapshotBaseline; + yield* legacyRemoveShadowDatabase(docker.spawner, handle.containerId); + return handle; + }); + +describe("legacyAcquireShadowDatabase", () => { + it.live("is today's bare create when the cache is explicitly disabled", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "0", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(handle.baselinePresent).toBe(false); + + // `--rm` intact, no PGDATA copies either way, and the snapshot step is a no-op. The one + // `cp-secret` is the pgsodium root key every shadow has always been given. + expect(docker.calls("create")[0] ?? []).toContain("--rm"); + yield* handle.snapshotBaseline; + expect(docker.steps()).toEqual(["create", "cp-secret", "start"]); + // Nothing is written to disk at all. + expect(yield* soleTarName(fs, path)).toEqual([]); + + yield* legacyRemoveShadowDatabase(docker.spawner, handle.containerId); + expect(docker.calls("rm")[0]).toEqual(["rm", "-f", "-v", handle.containerId]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("takes the cache path when the env var is unset (default ON)", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + undefined, + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); + yield* handle.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("cold run stops, exports the tar, and starts the container again", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(handle.baselinePresent).toBe(false); + // The cold container must survive its own `docker stop`, so it carries no `--rm`. + expect(docker.calls("create")[0] ?? []).not.toContain("--rm"); + + yield* handle.snapshotBaseline; + + // Ordering IS the contract here: stop before the copy (a live PGDATA is not coherent to + // copy), start plus a readiness probe after it (the caller is about to reconnect). + expect(docker.steps()).toEqual([ + "create", + "cp-secret", + "start", + "stop", + "cp-out", + "start", + "inspect", + ]); + expect(docker.stepCalls("cp-out")[0]).toEqual([ + "cp", + `${handle.containerId}:${LEGACY_SHADOW_PGDATA_PATH}`, + "-", + ]); + expect(docker.containers.get(handle.containerId)?.running).toBe(true); + + // Exactly one tar, published under its final name with the exported bytes intact — no + // `.partial` left behind. + const tars = yield* soleTarName(fs, path); + expect(tars).toHaveLength(1); + expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); + expect(yield* fs.readFileString(path.join(pgDeltaTempDir(path), tars[0] ?? ""))).toBe( + FAKE_PGDATA_TAR, + ); + const leftovers = yield* fs.readDirectory(pgDeltaTempDir(path)); + expect(leftovers.filter((entry) => entry.includes("partial"))).toEqual([]); + + // Release is the uncached removal, same as ever — nothing is kept. + yield* legacyRemoveShadowDatabase(docker.spawner, handle.containerId); + expect(docker.ids()).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("warm run restores the tar into a FRESH container before starting it", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const cold = yield* coldRun(docker, input); + + const warm = yield* legacyAcquireShadowDatabase(docker.spawner, input); + // A brand new container every time — the cache keeps a file, never a container. + expect(warm.containerId).not.toBe(cold.containerId); + expect(warm.baselinePresent).toBe(true); + // Throwaway again: the warm container never gets stopped, so `--rm` is back. + expect(docker.calls("create").at(-1) ?? []).toContain("--rm"); + + // The restore lands BEFORE the start, and carries the exported bytes into PGDATA's parent. + const warmSteps = docker.steps().slice(docker.steps().lastIndexOf("create")); + expect(warmSteps).toEqual(["create", "cp-secret", "cp-in", "start", "inspect"]); + expect(docker.stepCalls("cp-in").at(-1)).toEqual([ + "cp", + "-", + `${warm.containerId}:${LEGACY_SHADOW_PGDATA_PARENT_PATH}`, + ]); + expect(docker.containers.get(warm.containerId)?.restored).toBe( + `${LEGACY_SHADOW_PGDATA_PARENT_PATH}::${FAKE_PGDATA_TAR}`, + ); + + // Nothing more is exported: the baseline is already on disk. + yield* warm.snapshotBaseline; + expect(docker.calls("stop")).toHaveLength(1); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("publishing a new key's tar sweeps every other key's", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + yield* coldRun(docker, shadowInput(fs, path)); + const stale = yield* soleTarName(fs, path); + expect(stale).toHaveLength(1); + + // A changed baseline input (the shadow's own published port) is a different cluster, so + // its snapshot is a different key — and ~90MB each means only the current one is kept. + const rekeyed = yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54399 })); + const fresh = yield* soleTarName(fs, path); + expect(fresh).toHaveLength(1); + expect(fresh[0]).not.toBe(stale[0]); + expect(rekeyed.baselinePresent).toBe(false); + + // An unrelated file in the same shared temp directory is untouched. + yield* fs.writeFileString(path.join(pgDeltaTempDir(path), "catalog-abc.json"), "{}"); + yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54398 })); + expect(yield* fs.exists(path.join(pgDeltaTempDir(path), "catalog-abc.json"))).toBe(true); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a failed export warns, leaves no tar, and still brings the container back up", () => { + const docker = fakeDockerDaemon({ failCopyOut: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); + // The run itself must never fail for a cache problem. + yield* handle.snapshotBaseline; + + expect(out.stderrText).toContain("Warning: shadow baseline not cached"); + // The caller is about to reconnect, so the container is running again regardless. + expect(docker.containers.get(handle.containerId)?.running).toBe(true); + // Neither a published tar nor a half-written temp file survives. + const entries = yield* fs.readDirectory(pgDeltaTempDir(path)); + expect(entries).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a failed warm restore deletes the suspect tar and falls back to a cold run", () => { + const docker = fakeDockerDaemon({ failCopyIn: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + // A cold run first, so a snapshot for this exact key exists to be restored. + const cold = yield* coldRun(docker, input); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(out.stderrText).toContain("cached shadow baseline unusable"); + // Falls all the way back to a cold provision — a fresh container with no baseline. + expect(fallback.baselinePresent).toBe(false); + expect(fallback.containerId).not.toBe(cold.containerId); + // The tar is the suspect and is deleted, so later runs do not retry it forever... + expect(yield* soleTarName(fs, path)).toEqual([]); + // ...and the cold fallback republishes one through its own snapshot step. + yield* fallback.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a restored shadow that never becomes ready is removed before the cold retry", () => { + const docker = fakeDockerDaemon(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const healthy = fakeCluster(); + const cold = yield* coldRun(docker, input).pipe(Effect.provide(healthy.layer)); + const tarBefore = yield* soleTarName(fs, path); + expect(tarBefore).toHaveLength(1); + + // The restored cluster refuses every connection — the restore produced something + // unstartable, so the tar itself is suspect. + const broken = fakeCluster({ failConnect: true }); + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input).pipe( + Effect.provide(broken.layer), + ); + + expect(fallback.baselinePresent).toBe(false); + expect(fallback.containerId).not.toBe(cold.containerId); + // The suspect container is gone before the replacement is created — it holds the shadow's + // published port. + const removeIndex = docker.spawned.findIndex((args) => args[0] === "rm"); + const recreateIndex = docker.spawned.findLastIndex((args) => args[0] === "create"); + expect(removeIndex).toBeGreaterThanOrEqual(0); + expect(removeIndex).toBeLessThan(recreateIndex); + expect(yield* soleTarName(fs, path)).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, fakeCluster().layer))); + }); +}); + +describe("SUPABASE_SHADOW_DEBUG phase-timing instrumentation", () => { + it.live("emits export and restore phase lines when the debug env var is set", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const { writes } = yield* withShadowDebugEnv( + "1", + captureStderr( + Effect.gen(function* () { + yield* coldRun(docker, input); + yield* legacyAcquireShadowDatabase(docker.spawner, input); + }), + ), + ); + + // Routed through the mocked `Output` (both phases run inside shadow-cache.ts, which + // always has `Output` in context). + expect(out.stderrText).toContain("shadow-debug: baseline-export"); + expect(out.stderrText).toContain("shadow-debug: baseline-restore"); + // Written straight to the real `process.stderr` by `legacyWaitForShadowReady` + // (`health-check.ts`), which has no `Output` in its own context. + expect(writes.some((chunk) => chunk.includes("shadow-debug: ready-wait"))).toBe(true); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("emits no shadow-debug lines when the debug env var is unset", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + withShadowDebugEnv( + undefined, + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const { writes } = yield* captureStderr(coldRun(docker, shadowInput(fs, path))); + expect(out.stderrText).not.toContain("shadow-debug:"); + expect(writes.some((chunk) => chunk.includes("shadow-debug:"))).toBe(false); + }), + ), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts new file mode 100644 index 0000000000..133691063c --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts @@ -0,0 +1,188 @@ +/** + * The shadow baseline cache's ONE live scenario (golden path only, per the repo's live-test + * policy): against a real Docker daemon and a real `supabase/postgres` container, a cold acquire + + * export must leave a tar that the next acquire restores into a BRAND NEW container which comes up + * pristine — the facts no mock can prove, since they depend on `docker cp`'s tar stream actually + * preserving PGDATA's ownership, on `docker-entrypoint.sh` actually skipping `initdb` when it finds + * a restored data directory, and on Postgres actually starting on a data directory copied out of a + * stopped container. + * + * Gated with `describeLive` (the cli-e2e-ci signal, which is also the only environment with a real + * Docker daemon). This is deliberately NOT a `runSupabaseLive` subprocess test: the contract under + * test is the acquire pair itself, and driving it directly avoids standing up a full local stack + * for `db diff` just to observe a container's cluster. + * + * The platform baseline itself is out of scope here (its one-shot migrate jobs are exercised by the + * `db diff`/`db pull` suites): the cache snapshots whatever PGDATA contains at the snapshot point, + * so a bare cluster is a faithful stand-in for it. + */ + +import type { ProjectConfig } from "@supabase/config"; +import { ProjectConfigSchema } from "@supabase/config"; +import { BunServices } from "@effect/platform-bun"; +import { expect, it } from "@effect/vitest"; +import { Effect, FileSystem, Layer, Option, Path, Schema } from "effect"; +import { ChildProcessSpawner } from "effect/unstable/process"; + +import { describeLive } from "../../../../tests/helpers/live.ts"; +import { mockOutput } from "../../../../tests/helpers/mocks.ts"; +import { dockerfileServiceImage } from "../../../shared/services/dockerfile-images.ts"; +import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; +import { legacyDbConnectionLayer } from "../legacy-db-connection.layer.ts"; +import { legacyWaitForShadowReady } from "./health-check.ts"; +import { + LEGACY_SHADOW_CACHE_ENV, + legacyAcquireShadowDatabase, + legacyShadowBaselineTarFileName, + legacyShadowCacheEnabled, +} from "./shadow-cache.ts"; +import { legacyRemoveShadowDatabase } from "./shadow-database.ts"; +import type { LegacyShadowDbSetupInput, LegacyShadowSetupInput } from "./shadow-database.ts"; + +const defaultConfig: ProjectConfig = Schema.decodeUnknownSync(ProjectConfigSchema)({}); + +/** A port well clear of the CLI's own defaults, so a live stack on this runner is untouched. */ +const LIVE_SHADOW_PORT = 54399; + +describeLive("shadow baseline cache (live Docker)", () => { + it.live( + "restores a fresh container from the exported snapshot, without the previous run's changes", + () => { + const out = mockOutput(); + return Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const spawner = yield* ChildProcessSpawner.ChildProcessSpawner; + const workdir = yield* fs.makeTempDirectoryScoped({ prefix: "legacy-shadow-cache-live-" }); + yield* fs.makeDirectory(path.join(workdir, "supabase"), { recursive: true }); + + const setup: LegacyShadowDbSetupInput = { + majorVersion: 17, + config: defaultConfig, + dbUrl: `postgresql://postgres:postgres@127.0.0.1:${LIVE_SHADOW_PORT}/postgres`, + jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", + jwks: Effect.succeed('{"keys":[]}'), + apiUrl: "http://127.0.0.1:54321", + authExternalUrl: undefined, + siteUrl: defaultConfig.auth.site_url, + anonKey: "anon-key", + serviceRoleKey: "service-role-key", + storageTargetMigration: "", + realtimeEnabledForSetup: false, + storageEnabledForSetup: false, + authEnabledForSetup: false, + serviceVersionOverrides: {}, + projectEnvValues: undefined, + debug: false, + webhooksEnabled: false, + apiAutoExposeNewTables: Option.some(true), + vault: [], + }; + const input: LegacyShadowSetupInput = { + db: { major_version: 17, settings: {} }, + experimental: defaultConfig.experimental, + jwtSecret: setup.jwtSecret, + jwtExpiry: 3600, + networkId: "supabase_network_shadow_cache_live", + image: dockerfileServiceImage("postgres"), + configImage: dockerfileServiceImage("postgres"), + shadowPort: LIVE_SHADOW_PORT, + password: "postgres", + projectId: "shadow_cache_live", + isBitbucketPipeline: false, + workdir, + extraHosts: [], + fs, + path, + hostname: "127.0.0.1", + healthTimeoutSeconds: 60, + setup, + }; + const connConfig = { + host: input.hostname, + port: input.shadowPort, + user: "postgres", + password: input.password, + database: "postgres", + }; + + process.env[LEGACY_SHADOW_CACHE_ENV] = "1"; + expect(legacyShadowCacheEnabled()).toBe(true); + const connection = yield* LegacyDbConnection; + + // --- Run 1: cold provision, export the pristine cluster, then dirty it. --- + const cold = yield* legacyAcquireShadowDatabase(spawner, input); + yield* Effect.addFinalizer(() => + legacyRemoveShadowDatabase(spawner, cold.containerId).pipe(Effect.ignore), + ); + expect(cold.baselinePresent).toBe(false); + yield* legacyWaitForShadowReady(spawner, cold.containerId, connConfig, { + timeoutSeconds: input.healthTimeoutSeconds, + }); + // The export stops and restarts the container, so nothing may be connected while it runs — + // exactly the contract `legacyMigrateShadowDatabase` honours around this same step. + yield* cold.snapshotBaseline; + yield* Effect.scoped( + Effect.gen(function* () { + const session = yield* connection.connect(connConfig, { + isLocal: true, + dnsResolver: "native", + }); + // Whatever a real run's migrations would do: a cluster-global role plus a database + // object, neither of which may survive into the next run. + yield* session.exec("CREATE ROLE shadow_cache_live_role"); + yield* session.exec("CREATE TABLE shadow_cache_live_table ()"); + }), + ); + yield* legacyRemoveShadowDatabase(spawner, cold.containerId); + + // --- Run 2: the same key restores that snapshot into a NEW container, pristine. --- + const warm = yield* legacyAcquireShadowDatabase(spawner, input); + yield* Effect.addFinalizer(() => + legacyRemoveShadowDatabase(spawner, warm.containerId).pipe(Effect.ignore), + ); + // The cache keeps a file, never a container: run 2 is a brand new container that skipped + // the baseline because its PGDATA arrived pre-initialized. + expect(warm.containerId).not.toBe(cold.containerId); + expect(warm.baselinePresent).toBe(true); + yield* Effect.scoped( + Effect.gen(function* () { + const session = yield* connection.connect(connConfig, { + isLocal: true, + dnsResolver: "native", + }); + const roles = yield* session.query( + "SELECT rolname FROM pg_roles WHERE rolname = 'shadow_cache_live_role'", + ); + expect(roles).toEqual([]); + const tables = yield* session.query( + "SELECT tablename FROM pg_tables WHERE tablename = 'shadow_cache_live_table'", + ); + expect(tables).toEqual([]); + // Restored, not re-initialized: the baseline cluster's own roles are all still there. + const postgres = yield* session.query( + "SELECT rolname FROM pg_roles WHERE rolname = 'postgres'", + ); + expect(postgres).toHaveLength(1); + }), + ); + yield* legacyRemoveShadowDatabase(spawner, warm.containerId); + + // The artifact is a plain file under the project's own temp dir — the property that lets a + // future native (non-Docker) Postgres service consume the same snapshot. + const tempDir = path.join(workdir, "supabase", ".temp", "pgdelta"); + const entries = yield* fs.readDirectory(tempDir); + const tars = entries.filter((entry) => entry.endsWith(".tar")); + expect(tars).toHaveLength(1); + expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); + expect(legacyShadowBaselineTarFileName("0".repeat(16))).toBe( + `shadow-baseline-${"0".repeat(16)}.tar`, + ); + }).pipe( + Effect.scoped, + Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, legacyDbConnectionLayer)), + ); + }, + 300_000, + ); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts new file mode 100644 index 0000000000..0457761a05 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -0,0 +1,785 @@ +/** + * Shadow baseline cache — the acquire/release pair `db diff`/`db pull`/the migrations-catalog + * resolution path use in place of bare `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` + * (`shadow-database.ts`). + * + * The shadow database is on the hot path of every plan: a throwaway `supabase/postgres` container + * brought to the platform baseline (init schema + the PG15+ one-shot realtime/storage/auth migrate + * jobs) and then destroyed. That cold provision measures ~30s wall even with the readiness gate + * fixed, of which ~11s is Postgres's own `initdb` + bundled init SQL and the rest the one-shot + * jobs. None of it depends on the user's migrations, so it is cached — as a **disk-level PGDATA + * snapshot**, not as a kept container: + * + * - **Cold (no snapshot for this key):** create the shadow as an uncached run does (project + * labels only) and run today's baseline unchanged, then — immediately after the baseline and + * BEFORE `contrib_regression`/any user migration — `docker stop` the container, stream `docker + * cp :/var/lib/postgresql/data -` into `supabase/.temp/pgdelta/shadow-baseline-.tar`, + * and `docker start` + re-await readiness. Measured ~0.65s to export a 39MB bare-cluster + * snapshot (~90MB with the default services' one-shot jobs applied) and ~2s back to ready. + * - **Warm (that tar exists):** create the shadow exactly as an uncached run does, but unpack the + * tar into the created-but-not-yet-started container (`docker cp - :/var/lib/postgresql`, + * ~0.6s) so `docker-entrypoint.sh` sees a `PG_VERSION` file and skips `initdb` entirely. The + * container is connectable ~2s after `docker start` with the full baseline cluster in place + * (all roles, `_supabase`, the versioned `auth`/`storage`/`_realtime` schemas), so the caller + * skips the baseline ({@link LegacyShadowBaselineState.baselinePresent}) and applies user + * migrations straight onto the restored `postgres`. + * + * **A plain file artifact on purpose.** The snapshot is a tar under the project's own temp + * directory, not a Docker object, so nothing about the mechanism is Docker-specific: a future + * NATIVE (non-container) Postgres service can restore the same artifact by unpacking it into its + * own data directory. + * + * **No container outlives a run.** Every shadow this module hands out — cold, warm, or + * cache-disabled — carries the same two project labels and is removed with `docker rm -f -v` on + * release. Nothing is kept stopped between runs, there is no cache-key Docker label, and + * `supabase stop` has nothing extra to sweep. The cache's only footprint outside one run is the + * ~90MB tar. + * + * The single container-shape difference is on the cold path, and it is forced: that path drops + * `--rm`, because Docker destroys an `AutoRemove` container the moment it exits — `docker stop` + * included — which would leave nothing to restart after the export. See + * {@link LegacyCreateShadowDatabaseInput.autoRemove} for the full consequence (a SIGKILLed CLI + * leaves a stopped, project-labeled container instead of nothing). + * + * **Concurrency is safe by construction, with no lock file.** Every run creates its own fresh + * container, so no two runs can ever contend for one cluster. The tar is published by writing a + * temp file and `rename`-ing it into place, which is atomic: a reader either sees the previous + * complete tar or the new complete tar, never a partial one, and a reader that already opened the + * old inode keeps a valid fd across a rename-over. Two concurrent cold runs on the same key both + * export and the last `rename` wins — both tars are equally valid, since the key covers every + * input baked into the cluster. + * + * **The cache may never hand back a wrong baseline, and may never fail a run.** Every anomaly on + * the warm path (the `docker cp` in, the start, the readiness wait) removes the suspect container, + * DELETES the tar as suspect, and cold-provisions instead — worst case is exactly today's + * behavior. A cold export that fails only warns on stderr and leaves the run uncached; the + * container is always restarted afterwards, whether the export succeeded or not. + * + * Retention is "current key only": publishing a new key's tar removes every other + * `shadow-baseline-*.tar` in the same directory, so a project holds ~90MB, not ~90MB per config + * permutation it has ever used. + * + * `SUPABASE_SHADOW_CACHE` is ON by default; set it to `false`/`0` to opt out, in which case + * `legacyAcquireShadowDatabase` is `legacyCreateShadowDatabase` and the release is + * `legacyRemoveShadowDatabase` — same argv, same labels, same `--rm`, no extra Docker calls, no + * files written. + */ + +import { createHash } from "node:crypto"; + +import type { ProjectConfig } from "@supabase/config"; +import { Effect, Option, Stream, type FileSystem } from "effect"; +import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; + +import { Output } from "../../../shared/output/output.service.ts"; +import { + containerCliExitCode, + legacyCollectText, + legacyDescribeContainerCliFailure, + spawnContainerCli, +} from "../legacy-container-cli.ts"; +import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; +import type { LegacyPgConnInput } from "../legacy-db-connection.service.ts"; +import { legacyPgDeltaTempPath } from "../legacy-pgdelta.paths.ts"; +import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; +import type { LegacyVaultSecret } from "../legacy-vault.ts"; +import { legacyWaitForShadowReady } from "./health-check.ts"; +import { legacyResolvePinnedImage } from "./pinned-image.ts"; +import { legacyTimeShadowPhase } from "./shadow-debug.ts"; +import { + legacyCreateShadowDatabase, + legacyRemoveShadowDatabase, + type LegacyShadowBaselineState, + type LegacyShadowDbError, + type LegacyShadowSetupInput, +} from "./shadow-database.ts"; + +type Spawner = ChildProcessSpawner["Service"]; + +/** `SUPABASE_SHADOW_CACHE` — the opt-OUT gate for this whole module (default ON). */ +export const LEGACY_SHADOW_CACHE_ENV = "SUPABASE_SHADOW_CACHE"; + +/** + * `PGDATA` in every `supabase/postgres` image — the directory exported on the cold path and + * restored on the warm one. Hardcoded rather than read from the container's own `PGDATA` env: + * the entrypoint scripts this codebase generates (`postgres.service.ts`) never override it, and + * the value is part of the tar's own layout, so a mismatch has to be a deliberate change here. + */ +export const LEGACY_SHADOW_PGDATA_PATH = "/var/lib/postgresql/data"; + +/** + * `docker cp - :` unpacks the archive's members RELATIVE to `dest`, and + * {@link LEGACY_SHADOW_PGDATA_PATH}'s export tar has `data/` as its own top-level member, so the + * restore target is PGDATA's parent. A POSIX constant, not `path.dirname` — this is a container + * path and must not follow the host's separator. + */ +export const LEGACY_SHADOW_PGDATA_PARENT_PATH = "/var/lib/postgresql"; + +/** + * Internal-only "the cache cannot be used" signal. Deliberately NOT a `Data.TaggedError`: it + * never reaches a user or telemetry — every producer is caught by + * {@link legacyAcquireShadowDatabase} (which falls back to a cold provision) or by + * {@link legacyExportShadowBaseline} (which warns and continues uncached), so classifying it as a + * CLI error would be misleading, and would pollute the error-actionability vocabulary with a + * failure the user can neither see nor act on. + */ +interface LegacyShadowCacheUnavailable { + readonly reason: string; +} + +const legacyShadowCacheUnavailable = (reason: string): LegacyShadowCacheUnavailable => ({ reason }); + +/** + * Whether the shadow baseline cache is enabled for this invocation. + * + * Unset (or empty) means ON — this is a default-on optimization, not a feature flag. A value that + * IS set goes through the repo's `viper.GetBool` parser, so `SUPABASE_SHADOW_CACHE=false` and + * `=0` opt out while `=1`/`=true` are explicit opt-ins. + */ +export function legacyShadowCacheEnabled( + env: Readonly> = process.env, +): boolean { + const raw = env[LEGACY_SHADOW_CACHE_ENV]; + if (raw === undefined || raw.length === 0) return true; + return legacyParseBoolEnv(raw); +} + +// --------------------------------------------------------------------------- +// Cache key +// --------------------------------------------------------------------------- + +/** One of the three PG15+ one-shot migrate jobs, as the cache key sees it. */ +export interface LegacyShadowCacheServiceInput { + readonly enabled: boolean; + /** `legacyResolvePinnedImage`'s resolved image — hashed ONLY when {@link enabled}, since a disabled service's job never ran into the baseline. */ + readonly image: string; +} + +/** + * Every input baked into the shadow cluster during a cold provision. Deliberately NOT + * `legacySetupInputsToken`'s shape (`legacy-pgdelta.cache.ts`): that key is a Go-parity + * byte-for-byte contract shared with the Go binary AND hashes vault NAMES only while omitting + * the one-shot job image tags — both fatal for a CLUSTER snapshot, which carries the vault + * secrets' values and the versioned `auth`/`storage`/`_realtime` schema those jobs write. This + * key mirrors that function's hashing STYLE (sha256 over newline-joined formatted fields) without + * reusing it. + */ +export interface LegacyShadowCacheKeyInputs { + /** The resolved, full `supabase/postgres` image (tag included — a major version is not enough). */ + readonly postgresImage: string; + readonly majorVersion: number; + /** + * The shadow's published host port. Not baked into PGDATA itself, but it IS part of the + * container shape a snapshot is restored into, and a plan that changed it is a different plan — + * cheap to include, and it keeps the key a superset of everything the container carries. + */ + readonly shadowPort: number; + readonly jwtSecret: string; + readonly jwtExpiry: number; + readonly rootKey: string; + /** `[db] password` — baked into the cluster as the `postgres` role's password. */ + readonly dbPassword: string; + readonly dbSettings: ProjectConfig["db"]["settings"]; + /** Effective `api.auto_expose_new_tables` tri-state (unset ≠ explicit `false`: only the former keeps the bundled grants). */ + readonly autoExposeNewTables: Option.Option; + /** `supabase/roles.sql`'s contents, `""` when absent. */ + readonly rolesSql: string; + /** `[db.vault]` secrets — names AND values, both of which land in `vault.secrets`. */ + readonly vault: ReadonlyArray; + readonly services: { + readonly realtime: LegacyShadowCacheServiceInput; + readonly storage: LegacyShadowCacheServiceInput; + readonly auth: LegacyShadowCacheServiceInput; + }; + /** + * The resolved JWKS string (`LegacyShadowDbSetupInput.jwks`, an `Effect` the caller resolves + * lazily) that realtime's one-shot tenant-seed job bakes into the cluster + * (`legacyBuildRealtimeEnv`'s `jwks` field, `db-setup.ts`) — a config change under + * `auth.third_party` changes this value without touching anything else in this struct, so it + * needs its own field rather than folding into `services.realtime`. Hashed ONLY when + * `services.realtime` is enabled AND `majorVersion >= 15` — the exact compound gate + * {@link legacyResolveDbSetupPrelude} itself uses to decide whether to resolve (and therefore + * whether the one-shot job ever consumes) this same effect during a real setup: a disabled or + * pre-PG15 realtime never reaches the job that reads it. `""` when excluded, mirroring + * {@link LegacyShadowCacheServiceInput.image}. + */ + readonly jwks: string; +} + +/** JSON with recursively key-sorted objects, so `db.settings`' own property order cannot change the key. */ +function legacyCanonicalJson(value: unknown): string { + if (value === null || typeof value !== "object") return JSON.stringify(value) ?? "null"; + if (Array.isArray(value)) return `[${value.map(legacyCanonicalJson).join(",")}]`; + const entries = Object.entries(value) + .filter(([, entryValue]) => entryValue !== undefined) + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)); + return `{${entries.map(([key, entryValue]) => `${JSON.stringify(key)}:${legacyCanonicalJson(entryValue)}`).join(",")}}`; +} + +const legacyBoolToken = (value: boolean) => (value ? "true" : "false"); + +const legacyTriStateToken = (value: Option.Option) => + Option.isNone(value) ? "unset" : legacyBoolToken(value.value); + +/** + * The cache key: a 16-hex-char (64-bit) sha256 prefix over a fixed field order. 64 bits is + * ample for a per-project local cache whose only cost of a collision would be a wrong baseline + * — and every genuinely divergent input is in the payload, so a collision needs an actual hash + * collision, not a missed field. Short enough to read in a filename. + */ +export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string { + const lines: Array = [ + `postgres_image=${inputs.postgresImage}`, + `major_version=${inputs.majorVersion}`, + `shadow_port=${inputs.shadowPort}`, + `jwt_secret=${inputs.jwtSecret}`, + `jwt_expiry=${inputs.jwtExpiry}`, + `root_key=${inputs.rootKey}`, + `db_password=${inputs.dbPassword}`, + `db_settings=${legacyCanonicalJson(inputs.dbSettings)}`, + `auto_expose_new_tables=${legacyTriStateToken(inputs.autoExposeNewTables)}`, + ]; + for (const name of ["realtime", "storage", "auth"] as const) { + const service = inputs.services[name]; + lines.push( + service.enabled + ? `service=${name} enabled=true image=${service.image}` + : `service=${name} enabled=false`, + ); + } + // Realtime's resolved JWKS — see the field's own doc comment for the compound + // enabled+majorVersion gate (mirrors `service=realtime`'s own `enabled` exclusion above, plus + // the PG15+ gate the one-shot job itself is behind). + lines.push( + inputs.services.realtime.enabled && inputs.majorVersion >= 15 + ? `realtime_jwks=${inputs.jwks}` + : "realtime_jwks=excluded", + ); + for (const secret of [...inputs.vault].sort((left, right) => + left.name < right.name ? -1 : left.name > right.name ? 1 : 0, + )) { + lines.push(`vault=${secret.name}=${secret.value}`); + } + // Last, raw (it can contain anything, including newlines) — mirroring `setupInputsToken`, + // which also appends `roles.sql` verbatim at the end of its own payload. + const payload = `${lines.join("\n")}\nroles_sql=\n${inputs.rolesSql}`; + return createHash("sha256").update(payload, "utf8").digest("hex").slice(0, 16); +} + +/** + * Resolves {@link LegacyShadowCacheKeyInputs} from the same run input the shadow container + * itself is built from, plus `supabase/roles.sql` off disk. The service enabled flags come from + * `setup.config` (NOT the `*EnabledForSetup` fields, which only gate JWKS resolution) because + * `legacySetupDatabase`'s own one-shot job gates read exactly those config fields. + * + * Returns `Option.none` (never a failure) for the one condition that makes the key + * uncomputable — an unreadable `roles.sql` — so this function's OWN error channel carries + * nothing but `E`: the `input.setup.jwks` effect below is `yield*`ed unguarded (only when the + * consuming realtime job is actually reachable — see the field's own doc comment), and a real + * JWKS failure must fail this whole acquire exactly as it would have failed a real setup, never + * be folded into a cache-miss the way {@link LegacyShadowCacheUnavailable} failures are + * elsewhere in this module. Keeping the two failure modes on separate channels (`Option.none` + * vs. a genuine `Effect` failure) is what lets the caller (`legacyAcquireShadowDatabase`) + * degrade the former to an uncached shadow while letting the latter propagate, without an `as` + * cast to tell them apart. + */ +const legacyResolveShadowCacheKeyInputs = ( + input: LegacyShadowSetupInput, +): Effect.Effect, E> => + Effect.gen(function* () { + const rolesPath = input.path.join(input.workdir, "supabase", "roles.sql"); + const rolesSql = yield* input.fs + .readFileString(rolesPath) + .pipe( + Effect.catchTag("PlatformError", (error) => + error.reason._tag === "NotFound" ? Effect.succeed("") : Effect.succeed(undefined), + ), + ); + if (rolesSql === undefined) return Option.none(); + + const overrides = input.setup.serviceVersionOverrides; + // The exact compound gate {@link legacyResolveDbSetupPrelude} uses to decide whether the + // realtime one-shot job's JWKS effect is reached (and therefore run) at all during a real + // setup — see `db-setup.ts`. Gating on anything looser here would resolve (and risk failing + // on) an effect a real cold provision at this same `majorVersion`/`enabled` combination would + // never have touched. + const realtimeConsumesJwks = + input.setup.majorVersion >= 15 && input.setup.config.realtime.enabled; + const jwks = realtimeConsumesJwks ? yield* input.setup.jwks : ""; + return Option.some({ + postgresImage: input.image, + majorVersion: input.db.major_version, + shadowPort: input.shadowPort, + jwtSecret: input.jwtSecret, + jwtExpiry: input.jwtExpiry, + rootKey: input.rootKey ?? "", + dbPassword: input.password, + dbSettings: input.db.settings, + autoExposeNewTables: input.setup.apiAutoExposeNewTables, + rolesSql, + vault: input.setup.vault, + jwks, + services: { + realtime: { + enabled: input.setup.config.realtime.enabled, + image: legacyResolvePinnedImage("realtime", "realtime", overrides), + }, + storage: { + enabled: input.setup.config.storage.enabled, + image: legacyResolvePinnedImage("storage", "storage", overrides), + }, + auth: { + enabled: input.setup.config.auth.enabled, + image: legacyResolvePinnedImage("gotrue", "auth", overrides), + }, + }, + } satisfies LegacyShadowCacheKeyInputs); + }); + +// --------------------------------------------------------------------------- +// The tar artifact +// --------------------------------------------------------------------------- + +/** Filename prefix shared by every key's snapshot — the handle the stale-key sweep enumerates by. */ +const LEGACY_SHADOW_BASELINE_TAR_PREFIX = "shadow-baseline-"; + +const LEGACY_SHADOW_BASELINE_TAR_SUFFIX = ".tar"; + +/** `shadow-baseline-.tar` under `supabase/.temp/pgdelta/` — one ~90MB file per key. */ +export function legacyShadowBaselineTarFileName(key: string): string { + return `${LEGACY_SHADOW_BASELINE_TAR_PREFIX}${key}${LEGACY_SHADOW_BASELINE_TAR_SUFFIX}`; +} + +/** + * Whether `fileName` is a baseline snapshot belonging to some OTHER key — i.e. one the "current + * key only" retention rule removes when a new key's snapshot is published. Pure, so the retention + * rule is unit-testable without a filesystem, and deliberately conservative: only files matching + * this module's own prefix AND suffix are ever candidates, so nothing else in + * `supabase/.temp/pgdelta/` (catalog snapshots, debug bundles) can be swept by accident. + */ +export function legacyIsStaleShadowBaselineTar(fileName: string, key: string): boolean { + return ( + fileName.startsWith(LEGACY_SHADOW_BASELINE_TAR_PREFIX) && + fileName.endsWith(LEGACY_SHADOW_BASELINE_TAR_SUFFIX) && + fileName !== legacyShadowBaselineTarFileName(key) + ); +} + +/** + * The temp name a snapshot is streamed to before being `rename`d over the real one. Carries the + * writing process's pid so two concurrent cold runs of the same key cannot truncate each other's + * in-progress export — they publish independently, and the last `rename` wins (both tars are + * equally valid; see this module's own header). + */ +function legacyShadowBaselineTarTempPath(tarPath: string): string { + return `${tarPath}.${process.pid}.partial`; +} + +/** Best-effort removal — a leftover tar only ever costs disk, never correctness. */ +const legacyForgetShadowBaselineTar = ( + fs: FileSystem.FileSystem, + filePath: string, +): Effect.Effect => fs.remove(filePath).pipe(Effect.orElseSucceed(() => undefined)); + +/** + * Applies the "current key only" retention rule: every `shadow-baseline-*.tar` in the temp + * directory whose key differs from `key` is removed. Best-effort throughout — a snapshot that + * cannot be swept costs ~90MB of disk, so it must never fail the export that just succeeded. + */ +const legacySweepStaleShadowBaselineTars = ( + input: LegacyShadowSetupInput, + key: string, +): Effect.Effect => + Effect.gen(function* () { + const tempDir = legacyPgDeltaTempPath(input.path, input.workdir); + const entries = yield* input.fs + .readDirectory(tempDir) + .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); + yield* Effect.forEach( + entries.filter((entry) => legacyIsStaleShadowBaselineTar(entry, key)), + (entry) => legacyForgetShadowBaselineTar(input.fs, input.path.join(tempDir, entry)), + { discard: true }, + ); + }); + +// --------------------------------------------------------------------------- +// Container primitives the cache adds on top of `shadow-database.ts` +// --------------------------------------------------------------------------- + +/** + * `docker `, resolving to {@link LegacyShadowCacheUnavailable} on anything but a clean + * exit. Both verbs this is used for (`stop`, `start`) are steps the export cannot proceed without, + * so a non-zero exit is an anomaly rather than something to tolerate. + */ +const legacyShadowContainerVerb = ( + spawner: Spawner, + verb: "start" | "stop", + containerId: string, +): Effect.Effect => + containerCliExitCode(spawner, [verb, containerId], { + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + }).pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable( + `failed to ${verb} shadow container: ${legacyDescribeContainerCliFailure(cause)}`, + ), + ), + Effect.flatMap((exitCode) => + exitCode === 0 + ? Effect.void + : Effect.fail(legacyShadowCacheUnavailable(`docker ${verb} exited ${exitCode}`)), + ), + ); + +/** The shadow's own connect target, shared by the readiness waits on both paths. */ +const legacyShadowConnConfig = (input: LegacyShadowSetupInput): LegacyPgConnInput => ({ + host: input.hostname, + port: input.shadowPort, + user: "postgres", + password: input.password, + database: "postgres", +}); + +/** + * `legacyWaitForShadowReady` self-instruments (`ready-attempt`/`ready-wait`, `health-check.ts`) + * whenever `SUPABASE_SHADOW_DEBUG` is on, so neither call site here needs an extra timing wrapper. + */ +const legacyAwaitShadowReady = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + containerId: string, + what: string, +): Effect.Effect => + legacyWaitForShadowReady(spawner, containerId, legacyShadowConnConfig(input), { + timeoutSeconds: input.healthTimeoutSeconds, + }).pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable(`${what} never became ready: ${cause.message}`), + ), + ); + +// --------------------------------------------------------------------------- +// Cold export +// --------------------------------------------------------------------------- + +/** + * Streams `docker cp :/var/lib/postgresql/data -`'s tar straight to a temp file and `rename`s + * it into place. The stream never lands in memory: the child's stdout is piped into + * `FileSystem.sink`, so an ~89MB snapshot costs one buffer's worth of heap. + * + * The `rename` is what publishes the entry, and it is the LAST step for exactly that reason: a + * partially written tar must never be observable under the final name (a reader would restore a + * truncated data directory and get a Postgres that will not start). Any failure removes the temp + * file and reports the cache unavailable; nothing is left behind for the next run to find. + */ +const legacyWriteShadowBaselineTar = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + key: string, + tarPath: string, + containerId: string, +): Effect.Effect => { + const tempPath = legacyShadowBaselineTarTempPath(tarPath); + return Effect.gen(function* () { + const tempDir = legacyPgDeltaTempPath(input.path, input.workdir); + yield* input.fs + .makeDirectory(tempDir, { recursive: true }) + .pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable(`failed to create ${tempDir}: ${cause.message}`), + ), + ); + yield* Effect.scoped( + Effect.gen(function* () { + const child = yield* spawnContainerCli( + spawner, + ["cp", `${containerId}:${LEGACY_SHADOW_PGDATA_PATH}`, "-"], + { stdin: "ignore", stdout: "pipe", stderr: "pipe" }, + ).pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable( + `failed to export ${LEGACY_SHADOW_PGDATA_PATH}: ${legacyDescribeContainerCliFailure(cause)}`, + ), + ), + ); + // stdout is consumed concurrently with awaiting the exit code, not after it: an + // unread pipe would block `docker cp` long before it finished writing 89MB. + const [exitCode, , stderr] = yield* Effect.all( + [ + child.exitCode.pipe(Effect.map(Number)), + Stream.run(child.stdout, input.fs.sink(tempPath, { flag: "w" })), + legacyCollectText(child.stderr), + ], + { concurrency: "unbounded" }, + ).pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable( + `failed to export ${LEGACY_SHADOW_PGDATA_PATH}: ${legacyDescribeContainerCliFailure(cause)}`, + ), + ), + ); + if (exitCode !== 0) { + const message = stderr.trim(); + return yield* Effect.fail( + legacyShadowCacheUnavailable( + `docker cp exited ${exitCode}${message.length > 0 ? `: ${message}` : ""}`, + ), + ); + } + }), + ); + yield* input.fs + .rename(tempPath, tarPath) + .pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable(`failed to publish ${tarPath}: ${cause.message}`), + ), + ); + yield* legacySweepStaleShadowBaselineTars(input, key); + }).pipe(Effect.onError(() => legacyForgetShadowBaselineTar(input.fs, tempPath))); +}; + +/** + * The cold path's snapshot step, run at the baseline/migrations seam — after + * `legacySetupDatabase` and strictly before `contrib_regression` or any user migration, with no + * session open against the shadow ({@link LegacyShadowBaselineState.snapshotBaseline}). + * + * `docker stop` -> export -> `docker start` -> readiness wait. The container is stopped because a + * live Postgres's PGDATA is not a coherent thing to copy; the stop is fast (~1s) because the + * entrypoint `exec`s Postgres, so PID 1 receives the SIGTERM instead of `sh` swallowing it and + * burning the full 10s grace period. + * + * Failure is not the run's problem — it only means this run stays uncached. The ONE thing that + * must happen regardless is bringing the container back up: the caller is about to connect to it + * again. So the restart runs whether the export succeeded or not, and only its own failure (which + * dooms the run anyway, via the caller's next connect) outranks an export failure in the warning. + */ +const legacyExportShadowBaseline = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + key: string, + tarPath: string, + containerId: string, +): Effect.Effect => + legacyTimeShadowPhase( + "baseline-export", + Effect.gen(function* () { + yield* legacyShadowContainerVerb(spawner, "stop", containerId); + // From here the container is DOWN; every exit path below has to start it again. + const written = yield* Effect.result( + legacyWriteShadowBaselineTar(spawner, input, key, tarPath, containerId), + ); + yield* legacyShadowContainerVerb(spawner, "start", containerId); + yield* legacyAwaitShadowReady(spawner, input, containerId, "re-started shadow"); + return yield* Effect.fromResult(written); + }), + ).pipe( + Effect.catch((cause) => + Effect.gen(function* () { + const output = yield* Output; + yield* output.raw(`Warning: shadow baseline not cached: ${cause.reason}\n`, "stderr"); + }), + ), + ); + +// --------------------------------------------------------------------------- +// Acquire / release +// --------------------------------------------------------------------------- + +/** + * What `Effect.acquireUseRelease`'s `acquire` hands the `use` phase: the container, whether its + * cluster already carries the platform baseline, and the snapshot step to run once a fresh + * baseline is in place. Release needs nothing extra — every shadow this module hands out is + * removed the same way an uncached one is. + */ +export interface LegacyShadowAcquiredHandle extends LegacyShadowBaselineState { + readonly containerId: string; +} + +/** A throwaway shadow with no snapshot step — the cache-off path. */ +const legacyUncachedShadow = ( + spawner: Spawner, + input: LegacyShadowSetupInput, +): Effect.Effect => + legacyCreateShadowDatabase(spawner, input).pipe( + Effect.map(({ containerId }) => ({ + containerId, + baselinePresent: false, + snapshotBaseline: Effect.void, + })), + ); + +/** + * A cold, cache-enabled shadow: today's container plus the export step at the baseline seam. + * + * `autoRemove: false` is the one and only container-shape difference the cache introduces, and it + * is forced: the export has to `docker stop` the container and `docker start` it again, and Docker + * destroys an `--rm` container the moment it exits. Release still removes it with `docker rm -f + * -v`, so the container's lifetime is unchanged — see + * {@link LegacyCreateShadowDatabaseInput.autoRemove}. + */ +const legacyColdCachedShadow = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + key: string, + tarPath: string, +): Effect.Effect => + legacyCreateShadowDatabase(spawner, { ...input, autoRemove: false }).pipe( + Effect.map(({ containerId }) => ({ + containerId, + baselinePresent: false, + snapshotBaseline: legacyExportShadowBaseline(spawner, input, key, tarPath, containerId), + })), + ); + +/** + * The warm path proper: create the shadow with the snapshot tar unpacked into it before it starts + * ({@link LegacyCreateShadowDatabaseInput.restoreArchive}), then wait for the restored Postgres. + * Every failure resolves to {@link LegacyShadowCacheUnavailable}, which the caller turns into the + * escape hatch. + * + * The readiness failure removes the container here rather than leaving it to the caller, because + * the caller's fallback creates a REPLACEMENT container and the suspect one must be gone by then + * (it holds the shadow's published port). + */ +const legacyWarmShadow = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + tarPath: string, +): Effect.Effect< + LegacyShadowAcquiredHandle, + LegacyShadowCacheUnavailable, + Output | LegacyDbConnection +> => + Effect.gen(function* () { + const { containerId } = yield* legacyTimeShadowPhase( + "baseline-restore", + legacyCreateShadowDatabase(spawner, { + ...input, + restoreArchive: { + containerPath: LEGACY_SHADOW_PGDATA_PARENT_PATH, + tar: input.fs.stream(tarPath), + }, + }), + ).pipe( + Effect.mapError((cause) => + legacyShadowCacheUnavailable(`failed to restore shadow baseline: ${cause.message}`), + ), + ); + yield* legacyAwaitShadowReady(spawner, input, containerId, "restored shadow").pipe( + Effect.tapError(() => legacyRemoveShadowDatabase(spawner, containerId)), + ); + return { + containerId, + baselinePresent: true, + snapshotBaseline: Effect.void, + } satisfies LegacyShadowAcquiredHandle; + }); + +/** + * `Effect.acquireUseRelease`'s `acquire` for every shadow-provisioning call site that runs the + * platform baseline (`db diff`'s migra/pg-delta branch, `db pull`'s migration diff, + * `legacy-pgdelta.cache.ts`'s catalog export) — see {@link legacyWithShadowDatabase}, which is + * what those call sites actually use. + * + * With {@link LEGACY_SHADOW_CACHE_ENV} set to a falsey value this IS `legacyCreateShadowDatabase`, + * byte for byte. Otherwise it resolves the cache key and either restores this key's snapshot into + * a fresh container (warm) or creates one and arranges for its baseline to be exported (cold). + * Either way the container itself is created identically to an uncached one. + * + * Runs inside `acquireUseRelease`'s uninterruptible `acquire`, same as + * `legacyCreateShadowDatabase` — a warm restore adds ~2s of uninterruptible work (a `docker cp -` + * plus a readiness probe); the multi-second baseline/migration sequence stays in the interruptible + * `use` phase exactly as before. + * + * The `E` in the error channel is {@link legacyResolveShadowCacheKeyInputs}'s own JWKS + * resolution alone (see that function's doc comment): every OTHER failure this function's own + * body can produce while computing the key or restoring the snapshot is caught and degraded to a + * cold provision — a genuine JWKS failure is the one case that must reach the caller instead, + * since a real cold provision at this input would have failed the same way. + */ +export const legacyAcquireShadowDatabase = ( + spawner: Spawner, + input: LegacyShadowSetupInput, +): Effect.Effect< + LegacyShadowAcquiredHandle, + LegacyShadowDbError | E, + Output | LegacyDbConnection +> => + Effect.gen(function* () { + if (!legacyShadowCacheEnabled()) return yield* legacyUncachedShadow(spawner, input); + + const keyInputs = yield* legacyResolveShadowCacheKeyInputs(input); + if (Option.isNone(keyInputs)) return yield* legacyUncachedShadow(spawner, input); + const key = legacyShadowCacheKey(keyInputs.value); + const tarPath = input.path.join( + legacyPgDeltaTempPath(input.path, input.workdir), + legacyShadowBaselineTarFileName(key), + ); + + const cached = yield* input.fs.exists(tarPath).pipe(Effect.orElseSucceed(() => false)); + if (!cached) return yield* legacyColdCachedShadow(spawner, input, key, tarPath); + + return yield* legacyWarmShadow(spawner, input, tarPath).pipe( + Effect.catch((cause) => + Effect.gen(function* () { + const output = yield* Output; + yield* output.raw( + `Warning: cached shadow baseline unusable (${cause.reason}); recreating.\n`, + "stderr", + ); + // The tar is the suspect: a restore that produced an unstartable cluster will produce + // one again on every later run, so it is deleted rather than retried forever. + yield* legacyForgetShadowBaselineTar(input.fs, tarPath); + return yield* legacyColdCachedShadow(spawner, input, key, tarPath); + }), + ), + ); + }); + +/** + * The acquire/use/release triple every shadow-provisioning call site that runs the platform + * baseline uses (`db diff`'s migra/pg-delta branch, `db pull`'s migration diff, + * `legacy-pgdelta.cache.ts`'s catalog export). + * + * `Effect.acquireUseRelease`, NOT a `yield* acquire` followed by a later + * `.pipe(Effect.ensuring(release))`: the latter leaves a real gap between the shadow's successful + * creation and the finalizer actually being attached — a fiber interrupt landing between those two + * statements would skip the release entirely, leaking the live container and leaving the shadow + * port occupied. `acquireUseRelease` closes that: `acquire` runs inside an `uninterruptibleMask` + * and the release finalizer is registered in the SAME uninterruptible continuation `acquire` + * resolves into, matching Go's `defer DockerRemove` immediately after successful creation + * (review: PRRT_kwDOErm0O86XDr4Y). It does NOT make removal unconditional — see + * `legacyCreateShadowDatabase`'s own doc comment (`shadow-database.ts`) for the still-present, + * deliberate-Go-parity leak window when `acquire` itself fails partway through (a `docker create` + * success followed by a `docker cp`/`docker start` failure). + * + * `acquire` is ONLY the container acquisition — NOT the health-wait/migrate/declarative-apply + * `legacyPrepareShadowSource` performs. Those belong in `use`, where a SIGINT can still interrupt + * them (matching Go's single cancellable `ctx` threaded through the equivalent calls); passing all + * of `legacyPrepareShadowSource` as `acquire` made that whole sequence uninterruptible too, since + * `acquireUseRelease`'s `uninterruptibleMask` has no `restore` around `acquire` — see + * `legacy-shadow-source.ts`'s own doc comment on `legacyPrepareShadowSource` for the full + * rationale (review: PRRT_kwDOErm0O86XMrID). + * + * `release` is `legacyRemoveShadowDatabase` unconditionally — the shadow baseline cache keeps a + * file, never a container, so there is nothing here for it to special-case. + * + * The handle `use` receives also carries the baseline state `legacyPrepareShadowSource` needs. + * + * `E` (also present in `use`'s own `E2`, since every real caller's `use` phase already resolves + * the same `input.setup.jwks` effect via `legacyMigrateShadowDatabase`/`legacyResolveDbSetupPrelude`) + * can now ALSO surface straight out of `acquire`: see {@link legacyAcquireShadowDatabase}'s own + * doc comment for why a JWKS failure discovered while computing the cache key must propagate + * rather than degrade to an uncached shadow. + */ +export const legacyWithShadowDatabase = ( + spawner: Spawner, + input: LegacyShadowSetupInput, + use: (handle: LegacyShadowAcquiredHandle) => Effect.Effect, +): Effect.Effect => + Effect.acquireUseRelease(legacyAcquireShadowDatabase(spawner, input), use, (handle) => + legacyRemoveShadowDatabase(spawner, handle.containerId), + ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts new file mode 100644 index 0000000000..0ae4a1081e --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -0,0 +1,232 @@ +import { describe, expect, it } from "@effect/vitest"; +import { Option } from "effect"; + +import { + legacyIsStaleShadowBaselineTar, + legacyShadowBaselineTarFileName, + legacyShadowCacheEnabled, + legacyShadowCacheKey, + type LegacyShadowCacheKeyInputs, +} from "./shadow-cache.ts"; + +const baseKeyInputs = (): LegacyShadowCacheKeyInputs => ({ + postgresImage: "public.ecr.aws/supabase/postgres:17.6.1.158", + majorVersion: 17, + shadowPort: 54320, + jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", + jwtExpiry: 3600, + rootKey: "d4dc5b6d4a1d6a10b2c1e5b6a7c8d9e0", + dbPassword: "postgres", + dbSettings: { effective_cache_size: "128MB", max_connections: 100 }, + autoExposeNewTables: Option.none(), + rolesSql: "create role custom_role;\n", + vault: [{ name: "secret", value: "value", resolved: true }], + jwks: '{"keys":[]}', + services: { + realtime: { enabled: true, image: "supabase/realtime:v2.34.47" }, + storage: { enabled: true, image: "supabase/storage-api:v1.25.7" }, + auth: { enabled: true, image: "supabase/gotrue:v2.177.0" }, + }, +}); + +describe("legacyShadowCacheEnabled", () => { + it("is ON unless the env var explicitly opts out", () => { + // Unset and empty both mean "the user never expressed a preference" — the cache is a + // default-on optimization, not a feature flag. + expect(legacyShadowCacheEnabled({})).toBe(true); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "" })).toBe(true); + // A value that IS set goes through the repo's `viper.GetBool` parser. + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "0" })).toBe(false); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "false" })).toBe(false); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "no" })).toBe(false); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "1" })).toBe(true); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "true" })).toBe(true); + expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "TRUE" })).toBe(true); + }); +}); + +describe("legacyShadowCacheKey", () => { + it("is stable for identical inputs and independent of object key order", () => { + const first = legacyShadowCacheKey(baseKeyInputs()); + expect(legacyShadowCacheKey(baseKeyInputs())).toBe(first); + expect( + legacyShadowCacheKey({ + ...baseKeyInputs(), + dbSettings: { max_connections: 100, effective_cache_size: "128MB" }, + }), + ).toBe(first); + expect(first).toMatch(/^[0-9a-f]{16}$/u); + expect(legacyShadowBaselineTarFileName(first)).toBe(`shadow-baseline-${first}.tar`); + }); + + it("changes when ANY baked-in input changes", () => { + const base = baseKeyInputs(); + const mutations: ReadonlyArray<{ + readonly label: string; + readonly inputs: LegacyShadowCacheKeyInputs; + }> = [ + { label: "postgres image tag", inputs: { ...base, postgresImage: "postgres:17.6.1.159" } }, + { label: "major version", inputs: { ...base, majorVersion: 15 } }, + { label: "shadow port", inputs: { ...base, shadowPort: 54321 } }, + { label: "jwt secret", inputs: { ...base, jwtSecret: "other-secret" } }, + { label: "jwt expiry", inputs: { ...base, jwtExpiry: 7200 } }, + { label: "root key", inputs: { ...base, rootKey: "0000" } }, + { label: "db password", inputs: { ...base, dbPassword: "hunter2" } }, + { label: "db settings", inputs: { ...base, dbSettings: { max_connections: 200 } } }, + { + label: "auto expose new tables", + inputs: { ...base, autoExposeNewTables: Option.some(true) }, + }, + { + label: "auto expose new tables (explicit false vs unset)", + inputs: { ...base, autoExposeNewTables: Option.some(false) }, + }, + { label: "roles.sql", inputs: { ...base, rolesSql: "" } }, + { + label: "jwks (realtime enabled, majorVersion >= 15)", + inputs: { ...base, jwks: '{"keys":["rotated"]}' }, + }, + { + label: "vault secret name", + inputs: { ...base, vault: [{ name: "other", value: "value", resolved: true }] }, + }, + { + label: "vault secret value", + inputs: { ...base, vault: [{ name: "secret", value: "rotated", resolved: true }] }, + }, + { + label: "realtime image", + inputs: { + ...base, + services: { ...base.services, realtime: { enabled: true, image: "realtime:next" } }, + }, + }, + { + label: "storage image", + inputs: { + ...base, + services: { ...base.services, storage: { enabled: true, image: "storage:next" } }, + }, + }, + { + label: "auth image", + inputs: { + ...base, + services: { ...base.services, auth: { enabled: true, image: "gotrue:next" } }, + }, + }, + { + label: "realtime enabled flag", + inputs: { + ...base, + services: { + ...base.services, + realtime: { enabled: false, image: base.services.realtime.image }, + }, + }, + }, + { + label: "storage enabled flag", + inputs: { + ...base, + services: { + ...base.services, + storage: { enabled: false, image: base.services.storage.image }, + }, + }, + }, + { + label: "auth enabled flag", + inputs: { + ...base, + services: { ...base.services, auth: { enabled: false, image: base.services.auth.image } }, + }, + }, + ]; + const baseKey = legacyShadowCacheKey(base); + const seen = new Map([[baseKey, "base"]]); + for (const mutation of mutations) { + const key = legacyShadowCacheKey(mutation.inputs); + const collision = seen.get(key); + expect(collision, `${mutation.label} must change the cache key`).toBeUndefined(); + seen.set(key, mutation.label); + } + }); + + it("excludes a disabled service's image tag entirely", () => { + const base = baseKeyInputs(); + const withRealtimeA: LegacyShadowCacheKeyInputs = { + ...base, + services: { ...base.services, realtime: { enabled: false, image: "supabase/realtime:v1" } }, + }; + const withRealtimeB: LegacyShadowCacheKeyInputs = { + ...base, + services: { ...base.services, realtime: { enabled: false, image: "supabase/realtime:v2" } }, + }; + expect(legacyShadowCacheKey(withRealtimeA)).toBe(legacyShadowCacheKey(withRealtimeB)); + }); + + it("excludes the resolved jwks when realtime is disabled", () => { + const base = baseKeyInputs(); + const disabledRealtime: LegacyShadowCacheKeyInputs = { + ...base, + services: { + ...base.services, + realtime: { ...base.services.realtime, enabled: false }, + }, + }; + const withJwksA: LegacyShadowCacheKeyInputs = { ...disabledRealtime, jwks: '{"keys":["a"]}' }; + const withJwksB: LegacyShadowCacheKeyInputs = { ...disabledRealtime, jwks: '{"keys":["b"]}' }; + expect(legacyShadowCacheKey(withJwksA)).toBe(legacyShadowCacheKey(withJwksB)); + }); + + it("excludes the resolved jwks when majorVersion is below 15, even with realtime enabled", () => { + const base = baseKeyInputs(); + const pre15: LegacyShadowCacheKeyInputs = { ...base, majorVersion: 14 }; + const withJwksA: LegacyShadowCacheKeyInputs = { ...pre15, jwks: '{"keys":["a"]}' }; + const withJwksB: LegacyShadowCacheKeyInputs = { ...pre15, jwks: '{"keys":["b"]}' }; + expect(legacyShadowCacheKey(withJwksA)).toBe(legacyShadowCacheKey(withJwksB)); + }); + + it("hashes vault secrets in a name-stable order", () => { + const base = baseKeyInputs(); + const ascending = legacyShadowCacheKey({ + ...base, + vault: [ + { name: "a", value: "1", resolved: true }, + { name: "b", value: "2", resolved: true }, + ], + }); + const descending = legacyShadowCacheKey({ + ...base, + vault: [ + { name: "b", value: "2", resolved: true }, + { name: "a", value: "1", resolved: true }, + ], + }); + expect(ascending).toBe(descending); + }); +}); +describe("shadow baseline tar retention", () => { + const key = "0123456789abcdef"; + + it("keeps the current key's snapshot and sweeps every other key's", () => { + expect(legacyIsStaleShadowBaselineTar(legacyShadowBaselineTarFileName(key), key)).toBe(false); + expect(legacyIsStaleShadowBaselineTar("shadow-baseline-fedcba9876543210.tar", key)).toBe(true); + }); + + it("never sweeps a file that is not one of this module's own snapshots", () => { + // `supabase/.temp/pgdelta/` is shared with the pg-delta catalog cache and its debug bundles — + // the retention rule must be blind to everything but its own prefix AND suffix, or a `db diff` + // would delete the catalog cache it depends on. + for (const other of [ + "catalog-local-migrations-abc-123.json", + "shadow-baseline.tar", + `shadow-baseline-${key}.tar.4242.partial`, + `shadow-cache-${key}.json`, + "pgdelta-debug.zip", + ]) { + expect(legacyIsStaleShadowBaselineTar(other, key), other).toBe(false); + } + }); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index 8d5bbd339e..f7e18f58ec 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -74,12 +74,14 @@ import { type LegacyContainerError, type LegacyContainerOpts, } from "./container-lifecycle.ts"; +import type { LegacyStartContainerSpec } from "./docker-create-args.ts"; import type { LegacyImagePrepullError } from "./image-prepull.ts"; import type { LegacyHealthCheckTimeoutError } from "./health-check.ts"; import { legacyWaitForShadowReady } from "./health-check.ts"; import type { LegacyLocalDbContainerInputs } from "./local-container-inputs.ts"; import { legacyListLocalMigrationPaths } from "../legacy-migration-history.ts"; import { legacyToPostgresURL } from "../legacy-postgres-url.ts"; +import { legacyTimeShadowPhase } from "./shadow-debug.ts"; import { type LegacyFreshDbSetupInput, type LegacySetupDatabaseInput, @@ -214,6 +216,32 @@ export interface LegacyCreateShadowDatabaseInput extends LegacyShadowPostgresCon readonly isBitbucketPipeline: boolean; readonly workdir: string; readonly extraHosts: ReadonlyArray; + /** + * Set ONLY by the shadow baseline cache's warm path (`shadow-cache.ts`): a previously exported + * PGDATA tar to unpack into the container between `docker create` and `docker start`, so the + * `supabase/postgres` entrypoint finds an initialized data directory and skips `initdb` plus + * the whole platform baseline. Everything else about the container — including `--rm` and the + * project-labels-only label set — is identical to an uncached shadow, so a restored shadow is + * still a throwaway container removed on release. + * + * Delivered as {@link LegacyStartContainerSpec.preStartArchives}; see that field's doc comment + * for why the tar-stream form of `docker cp` is the only one that works here. + */ + readonly restoreArchive?: NonNullable[number]; + /** + * Set ONLY by the shadow baseline cache's COLD path (`shadow-cache.ts`), to `false`. That path + * has to `docker stop` the container mid-run to take a coherent disk-level PGDATA snapshot and + * then `docker start` it again — and Docker removes an `AutoRemove` container the moment it + * exits, `docker stop` included (verified against Docker 29: the container is gone ~1-2s after + * the stop returns), which would leave nothing to restart. + * + * The container is still removed by `docker rm -f -v` on release exactly like every other + * shadow, so the only externally visible difference is what a SIGKILLed CLI leaves behind: a + * stopped shadow container carrying the usual project labels — which `supabase stop` sweeps — + * rather than nothing. Omitted (i.e. Go's `--rm`) on the warm path and whenever the cache is + * off, since neither ever stops the container. + */ + readonly autoRemove?: boolean; } /** Resolved by {@link legacyCreateShadowDatabase} — everything a caller needs to both use and later tear down the shadow. */ @@ -265,7 +293,17 @@ export const legacyCreateShadowDatabase = ( }), ), ); - const spec = legacyBuildShadowPostgresContainerSpec(input); + // Both overrides are the shadow baseline cache's and nobody else's: a warm restore adds one + // `docker cp -` between `docker create` and `docker start` (no argv change at all), and a cold + // cache-enabled provision drops `--rm` so the container survives its own snapshot's `docker + // stop`. The spec builder itself stays on Go's `autoRemove: true` default. See each field's + // doc comment on {@link LegacyCreateShadowDatabaseInput}. + const baseSpec = legacyBuildShadowPostgresContainerSpec(input); + const spec: LegacyStartContainerSpec = { + ...baseSpec, + ...(input.autoRemove === undefined ? {} : { autoRemove: input.autoRemove }), + ...(input.restoreArchive === undefined ? {} : { preStartArchives: [input.restoreArchive] }), + }; // The shadow container has no name (Docker auto-generates one) and no network alias — // see this module's own header for why that's still enough for the shadow's own one-shot // setup jobs to reach it. The pgsodium root key itself (PG15+ only) never touches host @@ -555,7 +593,23 @@ export const legacySetupShadowConn = ( > => Effect.gen(function* () { yield* legacySetupDatabase(spawner, input, options); - yield* input.session.exec(LEGACY_SHADOW_CREATE_TEMPLATE_SQL).pipe( + yield* legacyCreateShadowTemplateDatabase(input.session); + }); + +/** + * {@link legacySetupShadowConn}'s trailing {@link LEGACY_SHADOW_CREATE_TEMPLATE_SQL} step on its + * own (Go's `setupShadowConn`'s second half, `diff.go:178`). Split out because it is the ONE part + * of Go's `setupShadowConn` a warm shadow-cache hit still has to run: the cache's PGDATA snapshot + * is taken strictly BEFORE this statement (`shadow-cache.ts`), so a restored cluster carries the + * platform baseline but no `contrib_regression`, and the template database must be recreated even + * though the baseline itself is skipped. + */ +const legacyCreateShadowTemplateDatabase = ( + session: LegacyDbSession, +): Effect.Effect => + legacyTimeShadowPhase( + "contrib-regression-create", + session.exec(LEGACY_SHADOW_CREATE_TEMPLATE_SQL).pipe( Effect.mapError( (cause) => new LegacyShadowDbError({ @@ -563,8 +617,8 @@ export const legacySetupShadowConn = ( reason: "database", }), ), - ); - }); + ), + ); /** * Shared fields both {@link legacySetupShadowDatabase} and {@link legacyMigrateShadowDatabase} @@ -672,6 +726,43 @@ export const legacySetupShadowDatabase = ( }), ); +/** + * What an `acquire` hands the `use` phase about the shadow cluster's CONTENTS — the seam the warm + * shadow-container cache (`shadow-cache.ts`) needs and nothing else uses. + * + * Deliberately a value the acquire OWNS and returns (alongside the container id), not an + * `afterBaseline` callback threaded down through `legacyPrepareShadowSource`: the cache is the + * only party that knows whether a cluster already carries a baseline and what to do once a fresh + * one exists, so both answers travel together with the container the cache handed over. + * {@link LEGACY_SHADOW_BASELINE_COLD} is what every uncached caller passes. + */ +export interface LegacyShadowBaselineState { + /** + * `true` only on a warm cache hit: the cluster already carries the platform baseline + * (`legacySetupDatabase`'s init schema + API privileges + vault + `roles.sql`), restored from + * the cache's own PGDATA snapshot, so re-running it would be wasted work at best and a + * double-applied baseline at worst. + */ + readonly baselinePresent: boolean; + /** + * Runs immediately after a FRESHLY provisioned baseline and strictly before the template + * database/user migrations — the only point at which `postgres` holds the pristine baseline and + * nothing else. + * + * Takes NO session, and {@link legacyMigrateShadowDatabase} guarantees no session is open + * against the shadow while it runs: the snapshot is a disk-level PGDATA export that has to stop + * the container, which would sever any live backend. `Effect.Effect`: a cache + * that cannot snapshot must degrade silently, never fail the run. + */ + readonly snapshotBaseline: Effect.Effect; +} + +/** The baseline state every uncached caller passes: provision it, snapshot nothing. */ +export const LEGACY_SHADOW_BASELINE_COLD: LegacyShadowBaselineState = { + baselinePresent: false, + snapshotBaseline: Effect.void, +}; + /** * Port of Go's `MigrateShadowDatabase` (`apps/cli-go/internal/db/diff/diff.go:195-209`): * lists local migrations FIRST (Go's `migration.ListLocalMigrations`, fails fast on a bad @@ -684,11 +775,25 @@ export const legacySetupShadowDatabase = ( * (`db-setup.ts`) for the real local `db` container — see {@link legacySetupShadowDatabase}'s * own doc comment for why the ordering matters. Connection closed once this resolves, matching * Go's `defer conn.Close(...)`. + * + * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_COLD}, i.e. exactly the sequence above. + * A warm shadow-cache hit passes a state whose `baselinePresent` is `true`, which skips the + * prelude + `SetupDatabase` steps (the restored cluster already has them) and goes straight to + * the template database and the user migrations; a COLD cache-enabled provision passes the same + * cold sequence plus a `snapshotBaseline` step between the baseline and the template database. + * + * The one structural divergence from Go: on the cold branch the baseline runs in its own scope, so + * its session is CLOSED before {@link LegacyShadowBaselineState.snapshotBaseline}, and the + * template database + migrations then run on a second session. Go uses a single connection for all + * of it, but the disk-level PGDATA snapshot stops the container, which severs any live backend — + * and a session's lifetime ending with the work it was opened for is the right shape anyway. A + * warm hit still uses exactly one session, and the SQL either path issues is unchanged. */ const migrateShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, setupOptions: LegacySetupDatabaseOptions, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, @@ -707,13 +812,24 @@ const migrateShadowDatabase = ( ), ); + if (!baseline.baselinePresent) { + // Own scope: the baseline session must be closed before `snapshotBaseline` — see this + // function's own doc comment. + yield* Effect.scoped( + Effect.gen(function* () { + const setupSession = yield* legacyConnectShadowDatabase(input.connConfig); + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput(input, setupSession, resolved), + setupOptions, + ); + }), + ); + yield* baseline.snapshotBaseline; + } const session = yield* legacyConnectShadowDatabase(input.connConfig); - const resolved = yield* legacyResolveDbSetupPrelude(input.setup); - yield* legacySetupShadowConn( - spawner, - legacyBuildShadowSetupDatabaseInput(input, session, resolved), - setupOptions, - ); + yield* legacyCreateShadowTemplateDatabase(session); yield* legacyApplyMigrations( session, input.fs, @@ -732,11 +848,12 @@ const migrateShadowDatabase = ( export const legacyMigrateShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, Output | LegacyDockerRun | RuntimeInfo | LegacyDbConnection -> => migrateShadowDatabase(spawner, input, { webhooks: "enabled" }); +> => migrateShadowDatabase(spawner, input, { webhooks: "enabled" }, baseline); /** * Migrates a shadow for the in-process pg-delta engine. Unlike the legacy engine, @@ -746,8 +863,9 @@ export const legacyMigrateShadowDatabase = ( export const legacyMigrateNextShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, Output | LegacyDockerRun | RuntimeInfo | LegacyDbConnection -> => migrateShadowDatabase(spawner, input, {}); +> => migrateShadowDatabase(spawner, input, {}, baseline); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index 5e3365920a..81249f6578 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -783,7 +783,12 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { }, setup: baseShadowSetup(), }); - expect(events).toEqual(["list", "connect"]); + // Two connects, not one: on the cold branch the platform baseline runs in its own scope so + // its session is closed before `snapshotBaseline` (which stops the container to take a + // disk-level PGDATA snapshot), and the template database + migrations then run on a second + // session. The ordering under test is unaffected — the migration listing still precedes + // every connect. + expect(events).toEqual(["list", "connect", "connect"]); }).pipe( Effect.provide( Layer.mergeAll( diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-debug.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-debug.ts new file mode 100644 index 0000000000..c7c9248d68 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-debug.ts @@ -0,0 +1,80 @@ +/** + * Debug-only phase-timing instrumentation for the shadow baseline cache + * (`shadow-cache.ts`'s `baseline-export`/`baseline-restore`) and the shadow's readiness gate + * (`health-check.ts`'s `legacyWaitForShadowReady`, which emits `ready-attempt`/`ready-wait`). + * `SUPABASE_SHADOW_DEBUG` is the opt-in gate — with it unset every + * helper here is a pure pass-through (not even a `Clock` read), so leaving instrumentation in + * place costs one boolean check per call site and nothing else. This module changes nothing about + * behavior, stdout, exit codes, or flags: it only ever writes to STDERR, and only when the env + * var is set (Go-parity constraint — this is a TS-only debugging aid with no Go counterpart). + * + * Lives in its own leaf module, not inside `shadow-cache.ts` itself, so `shadow-database.ts` (the + * `contrib_regression` template-database creation) and `health-check.ts` (the readiness gate) can + * use the same timing primitive without an import cycle back through `shadow-cache.ts`, which + * already imports from both of those modules. + * + * Every line has the fixed shape `shadow-debug: ms` — see + * {@link legacyTimeShadowPhase}'s own doc comment for the `detail` contract. + */ + +import { Clock, Effect, Result } from "effect"; + +import { Output } from "../../../shared/output/output.service.ts"; +import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; + +/** `SUPABASE_SHADOW_DEBUG` — the opt-in gate for shadow phase-timing instrumentation (default OFF). */ +export const LEGACY_SHADOW_DEBUG_ENV = "SUPABASE_SHADOW_DEBUG"; + +/** + * Whether shadow phase-timing debug lines are enabled for this invocation. Mirrors + * `legacyShadowCacheEnabled`'s own env-parsing style (`shadow-cache.ts`, itself + * `legacyParseBoolEnv` over `process.env`) — checked fresh on every call, never cached at module + * load, so a test (or a long-lived process) that mutates `process.env` sees the change + * immediately. + */ +export function legacyShadowDebugEnabled( + env: Readonly> = process.env, +): boolean { + return legacyParseBoolEnv(env[LEGACY_SHADOW_DEBUG_ENV]); +} + +/** Truncates a debug-line detail string to `maxLength` (default ~120, matching the brief's per-line budget), appending `…` when cut. */ +export function legacyShadowDebugTruncate(message: string, maxLength = 120): string { + return message.length > maxLength ? `${message.slice(0, maxLength)}…` : message; +} + +/** `shadow-debug: ms\n` — the one line shape every emitter in this module produces. */ +function legacyShadowDebugLine(phase: string, elapsedMs: number, detail: string): string { + return `shadow-debug: ${phase} ${elapsedMs}ms${detail}\n`; +} + +/** + * Times `effect` and, only when {@link legacyShadowDebugEnabled}, emits one + * `shadow-debug: ms` line to stderr via `Output.raw` once `effect` completes — + * on success OR failure, so a failing phase's elapsed time is never silently lost. + * + * `phase` may be a thunk instead of a plain string for a label that is itself non-trivial to + * build (e.g. embedding a SQL statement's first ~60 chars) — the thunk is only ever invoked when + * debug is on, so a disabled run pays neither the label construction nor the `Clock` read. + * + * `detail` receives the effect's own {@link Result.Result} and returns the trailing text (e.g. + * ` attempts=5` or ` error="..."`) appended after the bare `ms`; the default emits nothing + * extra. A complete no-op when debug is off — the input `effect` is returned unchanged, so + * leaving this wrapper in place costs exactly one boolean check per call site. + */ +export const legacyTimeShadowPhase = ( + phase: string | (() => string), + effect: Effect.Effect, + detail: (result: Result.Result) => string = () => "", +): Effect.Effect => { + if (!legacyShadowDebugEnabled()) return effect; + const label = typeof phase === "function" ? phase() : phase; + return Effect.gen(function* () { + const start = yield* Clock.currentTimeMillis; + const outcome = yield* Effect.result(effect); + const elapsed = (yield* Clock.currentTimeMillis) - start; + const output = yield* Output; + yield* output.raw(legacyShadowDebugLine(label, elapsed, detail(outcome)), "stderr"); + return yield* Effect.fromResult(outcome); + }); +}; diff --git a/apps/cli/src/legacy/shared/legacy-docker-lifecycle.ts b/apps/cli/src/legacy/shared/legacy-docker-lifecycle.ts index 77cb13cf4d..1dfe3cb60b 100644 --- a/apps/cli/src/legacy/shared/legacy-docker-lifecycle.ts +++ b/apps/cli/src/legacy/shared/legacy-docker-lifecycle.ts @@ -81,7 +81,7 @@ function splitNonEmptyLines(text: string): ReadonlyArray { } /** - * Shared `docker ps --filter label= [--all] --format + * Shared `docker ps --filter label= [--filter label=…] [--all] --format * ` spawn: one Docker CLI invocation is one underlying `GET * /containers/json` Docker Engine API request regardless of `--format` * (`--format` only controls how the CLI renders the already-returned JSON @@ -89,17 +89,23 @@ function splitNonEmptyLines(text: string): ReadonlyArray { * through here so two differently-formatted needs never accidentally cost * two real requests. See {@link legacyListContainerIdsAndNames}'s doc * comment for why that distinction matters for Go-parity request-log tests. + * + * Multiple `labelFilters` are ANDed by Docker itself (repeating `--filter label=` narrows the + * match), so scoping a listing to a second label costs no extra request either. */ function spawnDockerPsLines( spawner: Spawner, - opts: { readonly projectIdFilter: string; readonly all: boolean; readonly formatArg: string }, + opts: { + readonly labelFilters: ReadonlyArray; + readonly all: boolean; + readonly formatArg: string; + }, ): Effect.Effect, LegacyDockerLifecycleListError> { return Effect.scoped( Effect.gen(function* () { const args = [ "ps", - "--filter", - `label=${opts.projectIdFilter}`, + ...opts.labelFilters.flatMap((filterValue) => ["--filter", `label=${filterValue}`]), ...(opts.all ? ["--all"] : []), "--format", opts.formatArg, @@ -164,7 +170,7 @@ export const legacyListContainersByLabel = ( }, ) => spawnDockerPsLines(spawner, { - projectIdFilter: opts.projectIdFilter, + labelFilters: [opts.projectIdFilter], all: opts.all, formatArg: opts.format === "names" ? "{{.Names}}" : "{{.ID}}", }); @@ -207,7 +213,7 @@ export const legacyListContainerIdsAndNames = ( }, ): Effect.Effect, LegacyDockerLifecycleListError> => spawnDockerPsLines(spawner, { - projectIdFilter: opts.projectIdFilter, + labelFilters: [opts.projectIdFilter], all: opts.all, formatArg: `{{.ID}}\t{{.Names}}\t{{.Label "${LEGACY_CLI_WORKDIR_LABEL}"}}`, }).pipe( diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts index f53595e08d..10fb73e0ee 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts @@ -24,13 +24,15 @@ import { } from "./db-bootstrap/local-container-inputs.ts"; import { legacyWaitForHealthyServices } from "./db-bootstrap/health-check.ts"; import { - legacyCreateShadowDatabase, - legacyRemoveShadowDatabase, + legacyWithShadowDatabase, + type LegacyShadowAcquiredHandle, +} from "./db-bootstrap/shadow-cache.ts"; +import { legacySetupShadowDatabase, legacyShadowRunInputFromLocalContainerInputs, - type LegacyShadowDatabaseHandle, type LegacyShadowSetupInput, } from "./db-bootstrap/shadow-database.ts"; +import { legacyPgDeltaTempPath } from "./legacy-pgdelta.paths.ts"; import { legacyCompareUtf8Bytes } from "./legacy-glob.ts"; import { LegacyMigrationsReadError } from "./legacy-migration.errors.ts"; import { legacyToPostgresURL } from "./legacy-postgres-url.ts"; @@ -227,11 +229,6 @@ export function legacyDeclarativeCatalogFileName( return `catalog-${legacySanitizedCatalogPrefix(prefix)}-declarative-${hash}-${timestampMillis}.json`; } -/** `supabase/.temp/pgdelta` — where catalog snapshots + debug bundles live. */ -export function legacyPgDeltaTempPath(path: Path.Path, workdir: string): string { - return path.join(workdir, "supabase", ".temp", "pgdelta"); -} - /** * Lists local migration file paths under `migrationsDir`. Mirrors Go's * `migration.ListLocalMigrations` (`pkg/migration/list.go:33`): entries are sorted by name — Go's @@ -768,7 +765,7 @@ const exportViaShadowCatalog = ( built: LegacyShadowCatalogInputs, provision: ( spawner: Spawner, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, shadowInput: LegacyShadowSetupInput, ) => Effect.Effect, persist: (snapshot: string) => Effect.Effect, @@ -783,18 +780,20 @@ const exportViaShadowCatalog = ( fs, path, ); - const written = yield* Effect.acquireUseRelease( - legacyCreateShadowDatabase(spawner, shadowInput), - (handle) => - Effect.gen(function* () { - const shadow = yield* provision(spawner, handle, shadowInput); - const snapshot = yield* legacyExportCatalogPgDelta(ctx, { - targetRef: shadow.sourceUrl, - role: "postgres", - }); - return yield* persist(snapshot); - }), - (handle) => legacyRemoveShadowDatabase(spawner, handle.containerId), + // `legacyWithShadowDatabase` (`db-bootstrap/shadow-cache.ts`) rather than a bare + // `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` pair — see its doc comment: with + // `SUPABASE_SHADOW_CACHE` unset it IS that pair (identical Docker argv, identical labels), and + // with it set a catalog cache miss restores a key-matching PGDATA baseline snapshot instead of + // paying the full cold provision — the same swap `db diff`/`db pull`'s own call sites make. + const written = yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => + Effect.gen(function* () { + const shadow = yield* provision(spawner, handle, shadowInput); + const snapshot = yield* legacyExportCatalogPgDelta(ctx, { + targetRef: shadow.sourceUrl, + role: "postgres", + }); + return yield* persist(snapshot); + }), ); return path.relative(ctx.cwd, written); }); @@ -811,7 +810,7 @@ const legacyProvisionMigrationsShadow = ( ctx: LegacyPgDeltaContext, toml: LegacyDbTomlValues, spawner: Spawner, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, shadowInput: LegacyShadowSetupInput, ) => legacyPrepareShadowSource(spawner, handle, { @@ -1050,7 +1049,7 @@ const legacyProvisionBaselineShadow = ( fs: FileSystem.FileSystem, path: Path.Path, ctx: LegacyPgDeltaContext, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, shadowInput: LegacyShadowSetupInput, ) => Effect.gen(function* () { @@ -1092,7 +1091,7 @@ const legacyProvisionDeclarativeShadow = ( ctx: LegacyPgDeltaContext, declarativeDirAbs: string, declarativeDirRel: string, - handle: LegacyShadowDatabaseHandle, + handle: LegacyShadowAcquiredHandle, shadowInput: LegacyShadowSetupInput, ) => Effect.gen(function* () { diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts new file mode 100644 index 0000000000..4649d037f0 --- /dev/null +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts @@ -0,0 +1,16 @@ +/** + * The one on-disk location every pg-delta-adjacent cache/snapshot artefact lives under. + * + * Split out of `legacy-pgdelta.cache.ts` (which owns the catalog cache's keys AND its + * shadow-provisioning resolution path) so `db-bootstrap/shadow-cache.ts` — the warm + * shadow-container cache, which `legacy-pgdelta.cache.ts` itself consumes for its own + * shadow provisioning — can reach the same directory without an import cycle between + * the two. + */ + +import type { Path } from "effect"; + +/** `supabase/.temp/pgdelta` — where catalog snapshots, debug bundles, and the shadow-container cache's metadata live (`declarative.go:44`). */ +export function legacyPgDeltaTempPath(path: Path.Path, workdir: string): string { + return path.join(workdir, "supabase", ".temp", "pgdelta"); +} diff --git a/apps/cli/tests/helpers/cli.ts b/apps/cli/tests/helpers/cli.ts index dea6712740..639a3b9d47 100644 --- a/apps/cli/tests/helpers/cli.ts +++ b/apps/cli/tests/helpers/cli.ts @@ -238,6 +238,11 @@ export function spawnSupabase( SUPABASE_HOME: homeDir, SUPABASE_NO_KEYRING: "1", SUPABASE_TELEMETRY_DISABLED: "1", + // The shadow baseline cache (`db-bootstrap/shadow-cache.ts`) is ON by default. Off here so an + // e2e/live run observes the plain shadow lifecycle and never leaves a ~90MB tar in a temp + // project — a test whose subject IS the cache opts back in through `options.env`, which is + // spread after this. + SUPABASE_SHADOW_CACHE: "0", ...options?.env, }; if (entrypoint === "legacy") { diff --git a/apps/cli/tests/helpers/legacy-mocks.ts b/apps/cli/tests/helpers/legacy-mocks.ts index 71e72f3959..531eb656cb 100644 --- a/apps/cli/tests/helpers/legacy-mocks.ts +++ b/apps/cli/tests/helpers/legacy-mocks.ts @@ -698,6 +698,33 @@ export function useLegacyTempWorkdir(prefix = "supabase-legacy-test-"): { }; } +/** + * Pins `SUPABASE_SHADOW_CACHE=0` for every test in the calling file, restoring whatever the host + * had afterwards. Like {@link useLegacyTempWorkdir} it calls vitest's `beforeEach`/`afterEach` + * internally, so it must be invoked at module scope (or inside the surrounding `describe`). + * + * The shadow baseline cache (`db-bootstrap/shadow-cache.ts`) is ON by default and reads + * `process.env` directly, so ANY suite that provisions a shadow through + * `legacyWithShadowDatabase` with a mocked spawner — `db diff`, `db pull`, declarative sync — now + * exercises the cache path unless it opts out: the cold path adds a `docker stop`/`docker cp`/ + * `docker start` round trip and writes a ~90MB-shaped tar into the test's workdir. Suites whose + * subject is anything OTHER than the cache should call this so they keep asserting the plain + * container lifecycle; the cache's own suites deliberately do not. + */ +export function useLegacyShadowCacheDisabled(): void { + const name = "SUPABASE_SHADOW_CACHE"; + let previous: string | undefined; + beforeEach(() => { + previous = process.env[name]; + process.env[name] = "0"; + }); + afterEach(() => { + if (previous === undefined) delete process.env[name]; + else process.env[name] = previous; + previous = undefined; + }); +} + /** * Ambient isolation for tests that construct the REAL `legacyCliConfigLayer` / * `legacyCredentialsLayer` (directly or inside a command runtime layer) against From 932877546377293690905a4b3f580a767c39c366 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 17:23:39 +0200 Subject: [PATCH 39/82] refactor(cli): extract generic PGDATA snapshot primitives, trim cache docs Review feedback on the shadow baseline cache: - New db-bootstrap/pgdata-snapshot.ts owns the container-agnostic pieces: LEGACY_PGDATA_PATH constants, legacyExportPgDataTar (stream a stopped container's PGDATA to a tar via docker cp, temp file + atomic rename) and legacyPgDataRestoreArchive (the preStartArchives entry that unpacks a tar into a created-but-not-started container). Nothing in it assumes "shadow": the same primitives can savepoint/restore any local Postgres container (e.g. the stack's db container) or, later, a native Postgres data directory. shadow-cache.ts keeps only the cache-specific orchestration and shrinks by 120 lines. - Compressed the shadow-cache module header (64 -> 22 lines) and the three SIDE_EFFECTS.md cache sections (~25 -> ~8-11 lines each, now pointing at shadow-cache.ts for mechanics). - Deleted docs/shadow-db-provisioning.md; the module doc comments carry the design. Repointed the one code-comment reference. No behavior change. Targeted db-bootstrap/diff/pull/schema suites: 29 files / 568 tests pass; types/lint/fmt/knip green. Co-Authored-By: Claude Fable 5 --- apps/cli/docs/shadow-db-provisioning.md | 144 -------------- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 29 +-- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 33 +--- .../schema/declarative/sync/SIDE_EFFECTS.md | 34 +--- .../shared/db-bootstrap/pgdata-snapshot.ts | 143 ++++++++++++++ .../shared/db-bootstrap/postgres.service.ts | 3 +- .../shadow-cache.integration.test.ts | 14 +- .../shared/db-bootstrap/shadow-cache.ts | 182 +++--------------- 8 files changed, 207 insertions(+), 375 deletions(-) delete mode 100644 apps/cli/docs/shadow-db-provisioning.md create mode 100644 apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts diff --git a/apps/cli/docs/shadow-db-provisioning.md b/apps/cli/docs/shadow-db-provisioning.md deleted file mode 100644 index 1d3928f6fc..0000000000 --- a/apps/cli/docs/shadow-db-provisioning.md +++ /dev/null @@ -1,144 +0,0 @@ -# Shadow database provisioning - -How `db diff` / `db pull` / `db schema declarative sync` provision the shadow Postgres they diff -against, and why it is shaped the way it is. As-built notes, not a plan. - -## Why the shadow is on every hot path - -These commands hand pg-delta an **isolated shadow** (`pgdelta … --isolated-shadow`, loader mode -`isolatedCluster`): a `supabase/postgres` container brought to the platform baseline — the bundled -init schema plus the PG15+ one-shot realtime / storage / auth migrate jobs — before the user's own -migrations are applied. pg-delta requires that shadow on a different Postgres lineage than the -target, so the container is on the hot path of _every_ plan; the migrations-catalog cache cannot -make it go away. - -Owner: `src/legacy/shared/db-bootstrap/shadow-database.ts` (the primitives) and -`shadow-cache.ts` (the acquire/release seam every call site actually uses, -`legacyWithShadowDatabase`). - -## Readiness: a connect probe, not Docker health - -`legacyWaitForShadowReady` (`health-check.ts`) polls, on a 1s constant backoff: - -1. `docker inspect` — is the container still `running`? (preserves the crash detection the health - gate gave us: a dead container fails fast instead of at the end of the budget); then -2. a short-timeout Postgres connect — success ⇒ ready. - -The shadow's own healthcheck is `interval=10s` with no `start_period`/`start_interval` -(`postgres.service.ts`, deliberately unchanged — other tooling reads that config), so Docker's -first probe runs at t+10s while Postgres has been accepting connections since ~3.5s. Gating on -`Health.Status` spent ~6.5s per provision waiting for an already-knowable verdict. -`--health-start-interval` would fix the container side but needs Docker Engine 25+/API 1.44 and is -not reliably supported by Podman (which `spawnContainerCli` falls back to), so the CLI-side wait -asks Postgres directly instead. The failure shape is identical to the health gate's — same -`LegacyHealthCheckTimeoutError`, same `docker logs` dump. - -## Fast shutdown: the entrypoint `exec`s Postgres - -All three entrypoint builders in `postgres.service.ts` (`…Pg15` / `…Pg14` / `…Restore`) `exec` the -final `docker-entrypoint.sh` command, so PID 1 is Postgres and receives SIGTERM directly. With `sh` -as PID 1 the signal was swallowed and every `docker stop` burned the full 10s grace period before -a SIGKILL. This benefits `supabase stop` too, and it is what makes the cache's own mid-run stop -cheap. The divergence from Go's script is documented at the builders. - -## The PGDATA snapshot cache - -`SUPABASE_SHADOW_CACHE` — **on by default**; set it to `false` or `0` to opt out, in which case -the acquire is exactly `legacyCreateShadowDatabase` and the release exactly -`legacyRemoveShadowDatabase`. - -Everything above the user's migrations is deterministic in the config, so it is cached as a -disk-level snapshot of the initialized data directory: - -- **Cold** (no snapshot for this key): create the shadow and run the baseline as usual, then — at - the baseline/migrations seam, before `contrib_regression` or any user migration — - `docker stop`, stream `docker cp :/var/lib/postgresql/data -` to - `supabase/.temp/pgdelta/shadow-baseline-.tar`, `docker start`, re-await readiness, continue. -- **Warm** (that tar exists): create the shadow, unpack the tar into the created-but-not-yet-started - container (`docker cp - :/var/lib/postgresql`), then start it. `docker-entrypoint.sh` finds a - `PG_VERSION` file and skips `initdb` and the whole baseline; the caller is told - `baselinePresent: true` (`LegacyShadowBaselineState`, `shadow-database.ts`) and goes straight to - `contrib_regression` + user migrations. - -### Cache key - -sha256 (16 hex chars, fixed field order) over every input baked into the cluster during a cold -provision — `legacyShadowCacheKey`: - -- resolved `supabase/postgres` image tag (full tag, _not_ major version) after registry/pin - resolution; -- resolved one-shot job image tags (`realtime`, `storage`, `auth` via `legacyResolvePinnedImage` + - `serviceVersionOverrides`), each included **only when its service is enabled** — a disabled - service's job never ran into the baseline; -- the service enabled flags themselves; -- `jwtSecret`, `rootKey`, `[db] password`, `db.settings` (canonical JSON), `jwtExpiry`; -- effective `api.auto_expose_new_tables` (tri-state: unset ≠ explicit `false`); -- `supabase/roles.sql` contents (empty string when absent); -- `[db.vault]` secret **names and values** — both land in `vault.secrets`. (`setupInputsToken` in - `legacy-pgdelta.cache.ts` hashes names only and omits the job image tags; it is a Go-parity - contract and is deliberately not reused here, only its hashing style is.) -- `shadowPort` and `db.major_version`; -- the resolved JWKS string that realtime's one-shot tenant-seed job bakes in, included **only when - realtime is enabled AND `major_version >= 15`** — the exact compound gate that decides whether - the job is ever reached. An `auth.third_party` change moves this value and nothing else. - -### Invariants - -- **Atomic publish.** The tar is streamed to `..partial` and `rename`d into place, so a - partial tar is never observable under the final name and a reader holding the old inode keeps a - valid fd across a rename-over. -- **No lock file, no coordination.** Every run creates its own container, so no two runs can - contend for one cluster. Two concurrent cold writers on the same key both export and the last - `rename` wins — both tars are equally valid, since the key covers every baked-in input. -- **Retention: current key only.** Publishing a key's tar removes every other - `shadow-baseline-*.tar` in the directory (a snapshot is ~90MB with default services, so a project - holds one, not one per config permutation it has ever used). -- **Escape hatch.** Any warm-path failure (the `docker cp` in, the start, the readiness wait) - removes the container, **deletes the tar** as suspect, and cold-provisions. Any cold-path export - failure warns on stderr and leaves the run uncached; the container is restarted either way. The - cache never fails a user's command and never hands back a wrong baseline — worst case is the - uncached behavior. -- **No container outlives a run.** Cold, warm, and cache-disabled shadows all carry the same two - project labels and are removed with `docker rm -f -v` on release. There is no cache-key Docker - label and nothing extra for `supabase stop` to sweep. -- **One forced divergence:** the cold path drops `--rm`, because Docker destroys an `AutoRemove` - container the moment it exits — `docker stop` included (verified on Docker 29: gone ~1-2s after - the stop returns) — which would leave nothing to restart. The container is still removed on - release; the only visible consequence is that a SIGKILLed CLI leaves a stopped, project-labeled - shadow behind instead of nothing. - -### Why a plain file - -The artifact is a tar under the project's own temp directory, not a Docker object. Nothing about -the mechanism is container-specific, so a future **native** (non-Docker) Postgres service can reuse -the same snapshot by unpacking it into its own data directory — which is why the export is a file -rather than a `docker commit` image or a kept container. - -## Measured - -Benchmarked on `public.ecr.aws/supabase/postgres:17.6.1.158`, Docker 29, warm image cache: - -| Phase | Time | -| -------------------------------------------------------------- | ----------------------------- | -| `docker create` + secret `docker cp` + `docker start` | ~0.4s | -| Postgres accepting connections (initdb + bundled init SQL) | ~8-11s | -| One-shot realtime job (Elixir boot + tenant seed) | ~6.5s | -| One-shot storage migrate job | ~2.5s | -| One-shot auth (`gotrue migrate`) job | ~0.7s | -| **Cold provision total** (excl. image pulls, excl. migrations) | **~30s wall** | -| Snapshot export (`docker cp … -` to file) | ~0.65s (39MB bare cluster) | -| Snapshot restore (`docker cp - …`, before start) | ~0.6s | -| Restored container `docker start` → connectable | ~2s (initdb skipped entirely) | - -Docker's healthcheck reported `healthy` at ~10.2s on the same container, which is the ~6.5s of dead -wait the connect probe removed. - -## Known gaps - -- `db diff --use-pgadmin` and `migration squash` provision the same shadow but keep the - Docker-health gate and do not go through `legacyWithShadowDatabase`, so they are always cold. -- `db pull --declarative`'s bare shadow (`legacyPrepareRawShadow`) runs no platform baseline, so - there is nothing for this cache to snapshot; it stays cold by construction. -- The snapshot is taken **before** user migrations on purpose (that is what makes it reusable - across migration edits). Snapshotting the post-migration state as well, keyed on a migrations - hash, is the obvious next step for repos with large migration histories. diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 7da0f024ee..6c4aa35a3a 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -210,27 +210,14 @@ when declarative files exist. ### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) -- ON by default; set `SUPABASE_SHADOW_CACHE=false` (or `=0`) to opt out, in which case the shadow - lifecycle is exactly as documented above. -- The cached artifact is a **file**, never a container: `supabase/.temp/pgdelta/shadow-baseline-.tar` - (~90MB), a tar of the shadow's PGDATA directory taken right after the platform baseline and - before `contrib_regression`/any user migration. The key hashes every input baked into the cluster - (images, JWT secret, root key, `[db] password`, `db.settings`, vault secrets, `roles.sql`, - resolved JWKS, shadow port, major version). -- The container lifecycle is otherwise IDENTICAL to the uncached path — same project labels, always - `docker rm -f -v` on release. Nothing is kept between runs, there is no cache-key Docker label, - and no lock file. One forced exception: a COLD cache-enabled run creates the shadow without - `--rm`, because it must `docker stop` the container to take a coherent snapshot and Docker - destroys an `--rm` container the moment it exits. It is still removed on release. -- Cold run: `docker stop` -> `docker cp :/var/lib/postgresql/data -` streamed to the tar (temp - name + atomic rename) -> `docker start` -> readiness wait -> continue. Publishing a new key's tar - deletes every other `shadow-baseline-*.tar` (retention: current key only). -- Warm run: `docker cp - :/var/lib/postgresql` into the created-but-not-yet-started container, - so the entrypoint skips `initdb` and the platform baseline is skipped too. -- Any warm-path failure removes the container, deletes the tar as suspect, and cold-provisions; any - cold export failure only warns on stderr and leaves the run uncached. The cache never fails the - command. -- `--use-pgadmin` is NOT cached — its shadow keeps the plain create/remove lifecycle. +ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out, restoring the documented uncached +lifecycle. Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot +keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. +Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed +on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, +a cold export failure only warns and leaves the run uncached. See `shared/db-bootstrap/ +shadow-cache.ts`'s doc comment for the mechanics. `--use-pgadmin` is NOT cached — its shadow keeps +the plain create/remove lifecycle. ### `--use-pgadmin` parity quirks and deliberate divergence (CLI-1968) diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 70c59135cf..f1138cc809 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -76,30 +76,15 @@ disables formatting without disabling safe compaction. ### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) -- ON by default; set `SUPABASE_SHADOW_CACHE=false` (or `=0`) to opt out, in which case the shadow - lifecycle is exactly as documented above. -- The cached artifact is a **file**, never a container: `supabase/.temp/pgdelta/shadow-baseline-.tar` - (~90MB), a tar of the shadow's PGDATA directory taken right after the platform baseline and - before `contrib_regression`/any user migration. The key hashes every input baked into the cluster - (images, JWT secret, root key, `[db] password`, `db.settings`, vault secrets, `roles.sql`, - resolved JWKS, shadow port, major version). -- The container lifecycle is otherwise IDENTICAL to the uncached path — same project labels, always - `docker rm -f -v` on release. Nothing is kept between runs, there is no cache-key Docker label, - and no lock file. One forced exception: a COLD cache-enabled run creates the shadow without - `--rm`, because it must `docker stop` the container to take a coherent snapshot and Docker - destroys an `--rm` container the moment it exits. It is still removed on release. -- Cold run: `docker stop` -> `docker cp :/var/lib/postgresql/data -` streamed to the tar (temp - name + atomic rename) -> `docker start` -> readiness wait -> continue. Publishing a new key's tar - deletes every other `shadow-baseline-*.tar` (retention: current key only). -- Warm run: `docker cp - :/var/lib/postgresql` into the created-but-not-yet-started container, - so the entrypoint skips `initdb` and the platform baseline is skipped too. -- Any warm-path failure removes the container, deletes the tar as suspect, and cold-provisions; any - cold export failure only warns on stderr and leaves the run uncached. The cache never fails the - command. -- Each pooler-retry attempt acquires and releases its own shadow; on a warm hit each one restores - the same tar into its own fresh container. -- `--declarative`'s bare shadow runs no platform baseline, so there is nothing to snapshot — it - keeps the plain create/remove lifecycle. +ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out, restoring the documented uncached +lifecycle. Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot +keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. +Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed +on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, +a cold export failure only warns and leaves the run uncached. See `shared/db-bootstrap/ +shadow-cache.ts`'s doc comment for the mechanics. Each pooler-retry attempt acquires/releases its +own shadow (a warm hit restores the same tar each time); `--declarative`'s bare shadow runs no +baseline, so it is never cached. ## API Routes / DB diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 437b57bf2e..40f58166bf 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -125,28 +125,12 @@ existing SQL or creates an export manifest. ### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) The migrations-catalog shadow this command provisions on a cache miss goes through -`legacyGetMigrationsCatalogRef` -> `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), which -uses the same `legacyWithShadowDatabase` seam `db diff`/`db pull` do — so this command reads -`SUPABASE_SHADOW_CACHE` and inherits its whole lifecycle: - -- ON by default; set `SUPABASE_SHADOW_CACHE=false` (or `=0`) to opt out, in which case the shadow - lifecycle is exactly as documented above. -- The cached artifact is a **file**, never a container: `supabase/.temp/pgdelta/shadow-baseline-.tar` - (~90MB), a tar of the shadow's PGDATA directory taken right after the platform baseline and - before `contrib_regression`/any user migration. The key hashes every input baked into the cluster - (images, JWT secret, root key, `[db] password`, `db.settings`, vault secrets, `roles.sql`, - resolved JWKS, shadow port, major version). -- The container lifecycle is otherwise IDENTICAL to the uncached path — same project labels, always - `docker rm -f -v` on release. Nothing is kept between runs, there is no cache-key Docker label, - and no lock file. One forced exception: a COLD cache-enabled run creates the shadow without - `--rm`, because it must `docker stop` the container to take a coherent snapshot and Docker - destroys an `--rm` container the moment it exits. It is still removed on release. -- Cold run: `docker stop` -> `docker cp :/var/lib/postgresql/data -` streamed to the tar (temp - name + atomic rename) -> `docker start` -> readiness wait -> continue. Publishing a new key's tar - deletes every other `shadow-baseline-*.tar` (retention: current key only). -- Warm run: `docker cp - :/var/lib/postgresql` into the created-but-not-yet-started container, - so the entrypoint skips `initdb` and the platform baseline is skipped too. -- Any warm-path failure removes the container, deletes the tar as suspect, and cold-provisions; any - cold export failure only warns on stderr and leaves the run uncached. The cache never fails the - command. -- The declarative-catalog shadow is NOT cached — it is provisioned and torn down per run. +`legacyGetMigrationsCatalogRef` -> `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), the same +`legacyWithShadowDatabase` seam `db diff`/`db pull` use, so it inherits the whole lifecycle: ON by +default, `SUPABASE_SHADOW_CACHE=false`/`=0` opts out. Artifact: +`supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot keyed by a hash of +every input baked into the cluster; retention keeps the current key's tar only. Container +lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on +release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a +cold export failure only warns and leaves the run uncached. See `shared/db-bootstrap/ +shadow-cache.ts`'s doc comment for the mechanics. The declarative-catalog shadow is NOT cached. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts new file mode 100644 index 0000000000..0246c300f3 --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts @@ -0,0 +1,143 @@ +/** + * Generic PGDATA snapshot/restore primitives — container-agnostic on purpose. `shadow-cache.ts` + * is the only caller today, but nothing here assumes "shadow": these are the building blocks for + * savepointing ANY local Postgres container and restoring it into a fresh one — the shadow's + * disk-level baseline cache today, a future save/restore for the long-running + * `supabase_db_` stack container tomorrow. + * + * **Coherence contract:** the container must be STOPPED before {@link legacyExportPgDataTar} + * runs — a snapshot of a running Postgres's data directory is not a consistent thing to copy. + * Callers own the stop/start around the export; this module only moves bytes. + * + * **Ownership caveat:** the restore side ({@link legacyPgDataRestoreArchive}) MUST be delivered as + * a tar stream unpacked via `docker cp - :`, never a directory copy — a directory copy + * resets ownership to the extracting user (root) and Postgres refuses to start on a data directory + * it does not own, whereas the tar-stream form preserves each member's uid/gid verbatim. + * + * The artifact is a plain file, so it fits a future NATIVE (non-Docker) Postgres service just as + * well — nothing about the format is container-specific. + */ + +import { Effect, Stream, type FileSystem } from "effect"; +import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; + +import { + legacyCollectText, + legacyDescribeContainerCliFailure, + spawnContainerCli, +} from "../legacy-container-cli.ts"; +import type { LegacyStartContainerSpec } from "./docker-create-args.ts"; + +type Spawner = ChildProcessSpawner["Service"]; + +/** + * `PGDATA` in every `supabase/postgres` image — the directory {@link legacyExportPgDataTar} + * exports and {@link legacyPgDataRestoreArchive} restores. Hardcoded rather than read from the + * container's own `PGDATA` env: the entrypoint scripts this codebase generates + * (`postgres.service.ts`) never override it, and the value is part of the tar's own layout, so a + * mismatch has to be a deliberate change here. + */ +export const LEGACY_PGDATA_PATH = "/var/lib/postgresql/data"; + +/** + * `docker cp - :` unpacks the archive's members RELATIVE to `dest`, and + * {@link LEGACY_PGDATA_PATH}'s export tar has `data/` as its own top-level member, so the restore + * target is PGDATA's parent. A POSIX constant, not `path.dirname` — this is a container path and + * must not follow the host's separator. + */ +export const LEGACY_PGDATA_PARENT_PATH = "/var/lib/postgresql"; + +/** + * Internal-only "the snapshot could not be produced" signal — deliberately NOT a + * `Data.TaggedError`: every caller of {@link legacyExportPgDataTar} decides for itself how to + * degrade (the shadow baseline cache warns and continues uncached), so this must not be mistaken + * for a CLI-facing error. + */ +export interface LegacyPgDataSnapshotUnavailable { + readonly reason: string; +} + +const legacyPgDataSnapshotUnavailable = (reason: string): LegacyPgDataSnapshotUnavailable => ({ + reason, +}); + +/** + * Streams `docker cp :${LEGACY_PGDATA_PATH} -`'s tar straight to a temp file next to + * `tarPath` and `rename`s it into place. The stream never lands in memory: the child's stdout is + * piped into `FileSystem.sink`, so a large snapshot costs one buffer's worth of heap. + * + * The container must already be STOPPED (see this module's own header) — the caller owns the + * stop/start around this call. The `rename` is the LAST step and is what publishes the entry: a + * partially written tar must never be observable under the final name. Any failure removes the + * temp file; nothing is left behind for a later run to find. + */ +export const legacyExportPgDataTar = ( + spawner: Spawner, + containerId: string, + fs: FileSystem.FileSystem, + tarPath: string, +): Effect.Effect => { + const tempPath = `${tarPath}.${process.pid}.partial`; + return Effect.gen(function* () { + yield* Effect.scoped( + Effect.gen(function* () { + const child = yield* spawnContainerCli( + spawner, + ["cp", `${containerId}:${LEGACY_PGDATA_PATH}`, "-"], + { stdin: "ignore", stdout: "pipe", stderr: "pipe" }, + ).pipe( + Effect.mapError((cause) => + legacyPgDataSnapshotUnavailable( + `failed to export ${LEGACY_PGDATA_PATH}: ${legacyDescribeContainerCliFailure(cause)}`, + ), + ), + ); + // stdout is consumed concurrently with awaiting the exit code, not after it: an + // unread pipe would block `docker cp` long before it finished writing the archive. + const [exitCode, , stderr] = yield* Effect.all( + [ + child.exitCode.pipe(Effect.map(Number)), + Stream.run(child.stdout, fs.sink(tempPath, { flag: "w" })), + legacyCollectText(child.stderr), + ], + { concurrency: "unbounded" }, + ).pipe( + Effect.mapError((cause) => + legacyPgDataSnapshotUnavailable( + `failed to export ${LEGACY_PGDATA_PATH}: ${legacyDescribeContainerCliFailure(cause)}`, + ), + ), + ); + if (exitCode !== 0) { + const message = stderr.trim(); + return yield* Effect.fail( + legacyPgDataSnapshotUnavailable( + `docker cp exited ${exitCode}${message.length > 0 ? `: ${message}` : ""}`, + ), + ); + } + }), + ); + yield* fs + .rename(tempPath, tarPath) + .pipe( + Effect.mapError((cause) => + legacyPgDataSnapshotUnavailable(`failed to publish ${tarPath}: ${cause.message}`), + ), + ); + }).pipe(Effect.onError(() => fs.remove(tempPath).pipe(Effect.orElseSucceed(() => undefined)))); +}; + +/** + * Builds the {@link LegacyStartContainerSpec.preStartArchives} entry that restores a + * {@link legacyExportPgDataTar} tar into a container between `docker create` and `docker start`. + * `containerPath` is PGDATA's PARENT, not PGDATA itself — see {@link LEGACY_PGDATA_PARENT_PATH}'s + * own doc comment for why. + */ +export const legacyPgDataRestoreArchive = ( + fs: FileSystem.FileSystem, + tarPath: string, +): NonNullable[number] => ({ + containerPath: LEGACY_PGDATA_PARENT_PATH, + tar: fs.stream(tarPath), +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts index 6df84a1346..461744bf1f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/postgres.service.ts @@ -263,7 +263,8 @@ function legacyPostgresExtraEnv( * `docker stop` burns the full 10s grace period before SIGKILL; with `exec`, * Postgres is PID 1 and stops in ~1s). Applies to all three entrypoint * variants below. Timing is not part of the Go-parity surface (ADR 0016); - * see apps/cli/docs/shadow-db-provisioning.md. + * see `shadow-cache.ts`'s own doc comment for why fast shutdown matters to the shadow baseline + * cache's cold path. * * Otherwise byte-for-byte derived from Go's raw-string concatenation — * `NewContainerConfig(args ...string)` splices `strings.Join(args, " ")` diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 33d86b4115..458a8670da 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -21,12 +21,8 @@ import { useLegacyTempWorkdir } from "../../../../tests/helpers/legacy-mocks.ts" import { mockOutput } from "../../../../tests/helpers/mocks.ts"; import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import { LegacyDbConnectError } from "../legacy-db-connection.errors.ts"; -import { - LEGACY_SHADOW_CACHE_ENV, - LEGACY_SHADOW_PGDATA_PARENT_PATH, - LEGACY_SHADOW_PGDATA_PATH, - legacyAcquireShadowDatabase, -} from "./shadow-cache.ts"; +import { LEGACY_PGDATA_PARENT_PATH, LEGACY_PGDATA_PATH } from "./pgdata-snapshot.ts"; +import { LEGACY_SHADOW_CACHE_ENV, legacyAcquireShadowDatabase } from "./shadow-cache.ts"; import { LEGACY_SHADOW_DEBUG_ENV } from "./shadow-debug.ts"; import { legacyRemoveShadowDatabase } from "./shadow-database.ts"; import type { LegacyShadowDbSetupInput, LegacyShadowSetupInput } from "./shadow-database.ts"; @@ -436,7 +432,7 @@ describe("legacyAcquireShadowDatabase", () => { ]); expect(docker.stepCalls("cp-out")[0]).toEqual([ "cp", - `${handle.containerId}:${LEGACY_SHADOW_PGDATA_PATH}`, + `${handle.containerId}:${LEGACY_PGDATA_PATH}`, "-", ]); expect(docker.containers.get(handle.containerId)?.running).toBe(true); @@ -484,10 +480,10 @@ describe("legacyAcquireShadowDatabase", () => { expect(docker.stepCalls("cp-in").at(-1)).toEqual([ "cp", "-", - `${warm.containerId}:${LEGACY_SHADOW_PGDATA_PARENT_PATH}`, + `${warm.containerId}:${LEGACY_PGDATA_PARENT_PATH}`, ]); expect(docker.containers.get(warm.containerId)?.restored).toBe( - `${LEGACY_SHADOW_PGDATA_PARENT_PATH}::${FAKE_PGDATA_TAR}`, + `${LEGACY_PGDATA_PARENT_PATH}::${FAKE_PGDATA_TAR}`, ); // Nothing more is exported: the baseline is already on disk. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 0457761a05..726dc1ddcc 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -1,82 +1,37 @@ /** * Shadow baseline cache — the acquire/release pair `db diff`/`db pull`/the migrations-catalog * resolution path use in place of bare `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` - * (`shadow-database.ts`). + * (`shadow-database.ts`). Caches the shadow's platform baseline (init schema + the PG15+ one-shot + * realtime/storage/auth jobs) as a disk-level PGDATA snapshot, never a kept container. * - * The shadow database is on the hot path of every plan: a throwaway `supabase/postgres` container - * brought to the platform baseline (init schema + the PG15+ one-shot realtime/storage/auth migrate - * jobs) and then destroyed. That cold provision measures ~30s wall even with the readiness gate - * fixed, of which ~11s is Postgres's own `initdb` + bundled init SQL and the rest the one-shot - * jobs. None of it depends on the user's migrations, so it is cached — as a **disk-level PGDATA - * snapshot**, not as a kept container: + * - **Cold** (no snapshot for this key): provision the shadow as an uncached run does, then — + * right after the baseline and before `contrib_regression`/any user migration — stop the + * container, export its PGDATA via {@link legacyExportPgDataTar} (`pgdata-snapshot.ts`), and + * start it again. + * - **Warm** (that tar exists): create the shadow with the tar unpacked into it before it starts + * ({@link legacyPgDataRestoreArchive}), so the entrypoint skips `initdb` and the whole baseline; + * the caller applies user migrations straight onto the restored `postgres`. * - * - **Cold (no snapshot for this key):** create the shadow as an uncached run does (project - * labels only) and run today's baseline unchanged, then — immediately after the baseline and - * BEFORE `contrib_regression`/any user migration — `docker stop` the container, stream `docker - * cp :/var/lib/postgresql/data -` into `supabase/.temp/pgdelta/shadow-baseline-.tar`, - * and `docker start` + re-await readiness. Measured ~0.65s to export a 39MB bare-cluster - * snapshot (~90MB with the default services' one-shot jobs applied) and ~2s back to ready. - * - **Warm (that tar exists):** create the shadow exactly as an uncached run does, but unpack the - * tar into the created-but-not-yet-started container (`docker cp - :/var/lib/postgresql`, - * ~0.6s) so `docker-entrypoint.sh` sees a `PG_VERSION` file and skips `initdb` entirely. The - * container is connectable ~2s after `docker start` with the full baseline cluster in place - * (all roles, `_supabase`, the versioned `auth`/`storage`/`_realtime` schemas), so the caller - * skips the baseline ({@link LegacyShadowBaselineState.baselinePresent}) and applies user - * migrations straight onto the restored `postgres`. - * - * **A plain file artifact on purpose.** The snapshot is a tar under the project's own temp - * directory, not a Docker object, so nothing about the mechanism is Docker-specific: a future - * NATIVE (non-container) Postgres service can restore the same artifact by unpacking it into its - * own data directory. - * - * **No container outlives a run.** Every shadow this module hands out — cold, warm, or - * cache-disabled — carries the same two project labels and is removed with `docker rm -f -v` on - * release. Nothing is kept stopped between runs, there is no cache-key Docker label, and - * `supabase stop` has nothing extra to sweep. The cache's only footprint outside one run is the - * ~90MB tar. - * - * The single container-shape difference is on the cold path, and it is forced: that path drops - * `--rm`, because Docker destroys an `AutoRemove` container the moment it exits — `docker stop` - * included — which would leave nothing to restart after the export. See - * {@link LegacyCreateShadowDatabaseInput.autoRemove} for the full consequence (a SIGKILLed CLI - * leaves a stopped, project-labeled container instead of nothing). - * - * **Concurrency is safe by construction, with no lock file.** Every run creates its own fresh - * container, so no two runs can ever contend for one cluster. The tar is published by writing a - * temp file and `rename`-ing it into place, which is atomic: a reader either sees the previous - * complete tar or the new complete tar, never a partial one, and a reader that already opened the - * old inode keeps a valid fd across a rename-over. Two concurrent cold runs on the same key both - * export and the last `rename` wins — both tars are equally valid, since the key covers every - * input baked into the cluster. - * - * **The cache may never hand back a wrong baseline, and may never fail a run.** Every anomaly on - * the warm path (the `docker cp` in, the start, the readiness wait) removes the suspect container, - * DELETES the tar as suspect, and cold-provisions instead — worst case is exactly today's - * behavior. A cold export that fails only warns on stderr and leaves the run uncached; the - * container is always restarted afterwards, whether the export succeeded or not. - * - * Retention is "current key only": publishing a new key's tar removes every other - * `shadow-baseline-*.tar` in the same directory, so a project holds ~90MB, not ~90MB per config - * permutation it has ever used. - * - * `SUPABASE_SHADOW_CACHE` is ON by default; set it to `false`/`0` to opt out, in which case - * `legacyAcquireShadowDatabase` is `legacyCreateShadowDatabase` and the release is - * `legacyRemoveShadowDatabase` — same argv, same labels, same `--rm`, no extra Docker calls, no - * files written. + * Invariants: the artifact is a plain file, not a Docker object (native-services friendly — see + * `pgdata-snapshot.ts`'s own header); no container outlives a run — cold, warm, and cache-off + * shadows are all removed with `docker rm -f -v` on release; the cold path alone drops `--rm` (see + * {@link LegacyCreateShadowDatabaseInput.autoRemove} for the consequence); the tar is published by + * an atomic rename, so concurrent writers need no lock file; a cache anomaly never fails the run — + * a warm-path anomaly deletes the tar and cold-provisions, a cold export failure only warns and + * leaves the run uncached; retention keeps the current key's tar only, sweeping every other one on + * publish. `SUPABASE_SHADOW_CACHE` is ON by default; `false`/`0` opts out. */ import { createHash } from "node:crypto"; import type { ProjectConfig } from "@supabase/config"; -import { Effect, Option, Stream, type FileSystem } from "effect"; +import { Effect, Option, type FileSystem } from "effect"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import { Output } from "../../../shared/output/output.service.ts"; import { containerCliExitCode, - legacyCollectText, legacyDescribeContainerCliFailure, - spawnContainerCli, } from "../legacy-container-cli.ts"; import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import type { LegacyPgConnInput } from "../legacy-db-connection.service.ts"; @@ -84,6 +39,8 @@ import { legacyPgDeltaTempPath } from "../legacy-pgdelta.paths.ts"; import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; import type { LegacyVaultSecret } from "../legacy-vault.ts"; import { legacyWaitForShadowReady } from "./health-check.ts"; +import { legacyExportPgDataTar, legacyPgDataRestoreArchive } from "./pgdata-snapshot.ts"; +import type { LegacyPgDataSnapshotUnavailable } from "./pgdata-snapshot.ts"; import { legacyResolvePinnedImage } from "./pinned-image.ts"; import { legacyTimeShadowPhase } from "./shadow-debug.ts"; import { @@ -99,22 +56,6 @@ type Spawner = ChildProcessSpawner["Service"]; /** `SUPABASE_SHADOW_CACHE` — the opt-OUT gate for this whole module (default ON). */ export const LEGACY_SHADOW_CACHE_ENV = "SUPABASE_SHADOW_CACHE"; -/** - * `PGDATA` in every `supabase/postgres` image — the directory exported on the cold path and - * restored on the warm one. Hardcoded rather than read from the container's own `PGDATA` env: - * the entrypoint scripts this codebase generates (`postgres.service.ts`) never override it, and - * the value is part of the tar's own layout, so a mismatch has to be a deliberate change here. - */ -export const LEGACY_SHADOW_PGDATA_PATH = "/var/lib/postgresql/data"; - -/** - * `docker cp - :` unpacks the archive's members RELATIVE to `dest`, and - * {@link LEGACY_SHADOW_PGDATA_PATH}'s export tar has `data/` as its own top-level member, so the - * restore target is PGDATA's parent. A POSIX constant, not `path.dirname` — this is a container - * path and must not follow the host's separator. - */ -export const LEGACY_SHADOW_PGDATA_PARENT_PATH = "/var/lib/postgresql"; - /** * Internal-only "the cache cannot be used" signal. Deliberately NOT a `Data.TaggedError`: it * never reaches a user or telemetry — every producer is caught by @@ -365,16 +306,6 @@ export function legacyIsStaleShadowBaselineTar(fileName: string, key: string): b ); } -/** - * The temp name a snapshot is streamed to before being `rename`d over the real one. Carries the - * writing process's pid so two concurrent cold runs of the same key cannot truncate each other's - * in-progress export — they publish independently, and the last `rename` wins (both tars are - * equally valid; see this module's own header). - */ -function legacyShadowBaselineTarTempPath(tarPath: string): string { - return `${tarPath}.${process.pid}.partial`; -} - /** Best-effort removal — a leftover tar only ever costs disk, never correctness. */ const legacyForgetShadowBaselineTar = ( fs: FileSystem.FileSystem, @@ -465,14 +396,9 @@ const legacyAwaitShadowReady = ( // --------------------------------------------------------------------------- /** - * Streams `docker cp :/var/lib/postgresql/data -`'s tar straight to a temp file and `rename`s - * it into place. The stream never lands in memory: the child's stdout is piped into - * `FileSystem.sink`, so an ~89MB snapshot costs one buffer's worth of heap. - * - * The `rename` is what publishes the entry, and it is the LAST step for exactly that reason: a - * partially written tar must never be observable under the final name (a reader would restore a - * truncated data directory and get a Postgres that will not start). Any failure removes the temp - * file and reports the cache unavailable; nothing is left behind for the next run to find. + * Ensures the tar's temp directory exists, delegates the actual export to + * {@link legacyExportPgDataTar} (`pgdata-snapshot.ts` — see that function's own doc comment for + * the atomic-publish mechanics), then applies the "current key only" retention rule. */ const legacyWriteShadowBaselineTar = ( spawner: Spawner, @@ -480,9 +406,8 @@ const legacyWriteShadowBaselineTar = ( key: string, tarPath: string, containerId: string, -): Effect.Effect => { - const tempPath = legacyShadowBaselineTarTempPath(tarPath); - return Effect.gen(function* () { +): Effect.Effect => + Effect.gen(function* () { const tempDir = legacyPgDeltaTempPath(input.path, input.workdir); yield* input.fs .makeDirectory(tempDir, { recursive: true }) @@ -491,55 +416,13 @@ const legacyWriteShadowBaselineTar = ( legacyShadowCacheUnavailable(`failed to create ${tempDir}: ${cause.message}`), ), ); - yield* Effect.scoped( - Effect.gen(function* () { - const child = yield* spawnContainerCli( - spawner, - ["cp", `${containerId}:${LEGACY_SHADOW_PGDATA_PATH}`, "-"], - { stdin: "ignore", stdout: "pipe", stderr: "pipe" }, - ).pipe( - Effect.mapError((cause) => - legacyShadowCacheUnavailable( - `failed to export ${LEGACY_SHADOW_PGDATA_PATH}: ${legacyDescribeContainerCliFailure(cause)}`, - ), - ), - ); - // stdout is consumed concurrently with awaiting the exit code, not after it: an - // unread pipe would block `docker cp` long before it finished writing 89MB. - const [exitCode, , stderr] = yield* Effect.all( - [ - child.exitCode.pipe(Effect.map(Number)), - Stream.run(child.stdout, input.fs.sink(tempPath, { flag: "w" })), - legacyCollectText(child.stderr), - ], - { concurrency: "unbounded" }, - ).pipe( - Effect.mapError((cause) => - legacyShadowCacheUnavailable( - `failed to export ${LEGACY_SHADOW_PGDATA_PATH}: ${legacyDescribeContainerCliFailure(cause)}`, - ), - ), - ); - if (exitCode !== 0) { - const message = stderr.trim(); - return yield* Effect.fail( - legacyShadowCacheUnavailable( - `docker cp exited ${exitCode}${message.length > 0 ? `: ${message}` : ""}`, - ), - ); - } - }), + yield* legacyExportPgDataTar(spawner, containerId, input.fs, tarPath).pipe( + Effect.mapError((cause: LegacyPgDataSnapshotUnavailable) => + legacyShadowCacheUnavailable(cause.reason), + ), ); - yield* input.fs - .rename(tempPath, tarPath) - .pipe( - Effect.mapError((cause) => - legacyShadowCacheUnavailable(`failed to publish ${tarPath}: ${cause.message}`), - ), - ); yield* legacySweepStaleShadowBaselineTars(input, key); - }).pipe(Effect.onError(() => legacyForgetShadowBaselineTar(input.fs, tempPath))); -}; + }); /** * The cold path's snapshot step, run at the baseline/migrations seam — after @@ -658,10 +541,7 @@ const legacyWarmShadow = ( "baseline-restore", legacyCreateShadowDatabase(spawner, { ...input, - restoreArchive: { - containerPath: LEGACY_SHADOW_PGDATA_PARENT_PATH, - tar: input.fs.stream(tarPath), - }, + restoreArchive: legacyPgDataRestoreArchive(input.fs, tarPath), }), ).pipe( Effect.mapError((cause) => From 8c551230ef762b1229dff58ec7f9386452636d00 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 17:39:32 +0200 Subject: [PATCH 40/82] docs(cli): note the frozen/online hot-save modes on the PGDATA export contract Discussion outcome: the STOPPED coherence contract can later generalize to docker pause (crash-consistent, ~1s stall) or the Postgres backup API (pg_backup_start/stop, zero stall, the only form that fits a native non-container Postgres). Recorded as a TODO at the contract so the future live-stack savepoint feature starts from the right design. Co-Authored-By: Claude Fable 5 --- .../src/legacy/shared/db-bootstrap/pgdata-snapshot.ts | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts index 0246c300f3..471eb26fe1 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts @@ -9,6 +9,16 @@ * runs — a snapshot of a running Postgres's data directory is not a consistent thing to copy. * Callers own the stop/start around the export; this module only moves bytes. * + * TODO(hot-save): the STOPPED contract could generalize to a consistency MODE. `frozen` — + * `docker pause` → copy → `docker unpause` — yields a crash-consistent copy (connections stall + * ~1s instead of dropping; restore boots through normal WAL recovery). `online` — + * `pg_backup_start()` → fuzzy copy → `pg_backup_stop()`, writing the returned `backup_label` + * into the artifact — is the zero-stall, Postgres-native form, and the ONLY one that also works + * for a future NATIVE (non-container) Postgres process, where no freezer exists and recovery + * replays the backup-labeled WAL range instead. Neither is worth the surface for the shadow + * cache (its export runs once per key on an already-cold path); implement when a live-stack + * savepoint feature needs to export without downtime. + * * **Ownership caveat:** the restore side ({@link legacyPgDataRestoreArchive}) MUST be delivered as * a tar stream unpacked via `docker cp - :`, never a directory copy — a directory copy * resets ownership to the extracting user (root) and Postgres refuses to start on a data directory From 88943349df42d1e10860d3f0da3876dcb1a7142c Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 18:09:26 +0200 Subject: [PATCH 41/82] fix(cli): harden the shadow baseline cache against review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four fixes from the depthfirst/Codex review round on #6184, plus stale comment cleanup: - Fold the Storage migration pin (`supabase/.temp/storage-migration`, fed as `DB_MIGRATIONS_FREEZE_AT`) into the cache key, behind the same enabled+PG15 gate its consuming one-shot job is behind — a re-pinned project no longer warm-restores a baseline with the wrong Storage migrations. - Write the snapshot tar 0600: it is a full PGDATA carrying vault secret values, the JWT secret, and role password hashes. - Re-enable interruption around the warm restore's readiness wait (it runs inside acquireUseRelease's uninterruptible acquire and could pin a Ctrl-C for the full health-timeout budget), removing the container on interrupt so nothing leaks. - Remove the created container when a pre-start archive extraction fails, so the warm-anomaly fallback no longer strands one orphaned created container per recovery. Also rewrites the handler/catalog-cache comments still describing the earlier stopped-container-reuse design in terms of the shipped PGDATA snapshot mechanism. Co-Authored-By: Claude Fable 5 --- .../legacy/commands/db/diff/diff.handler.ts | 4 +- .../legacy/commands/db/pull/pull.handler.ts | 3 +- .../db-bootstrap/container-lifecycle.ts | 16 ++++++++ .../shared/db-bootstrap/pgdata-snapshot.ts | 5 ++- .../shadow-cache.integration.test.ts | 3 ++ .../shared/db-bootstrap/shadow-cache.ts | 37 +++++++++++++++++-- .../db-bootstrap/shadow-cache.unit.test.ts | 37 +++++++++++++++++++ .../src/legacy/shared/legacy-pgdelta.cache.ts | 5 ++- .../src/legacy/shared/legacy-pgdelta.paths.ts | 2 +- 9 files changed, 102 insertions(+), 10 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index 31da5adf69..a3b74a1ffa 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -687,8 +687,8 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // both halves of the rationale: why the lifecycle is an `Effect.acquireUseRelease` (an // interrupt must not be able to land between creation and the finalizer being attached) and // why the cache seam sits here (with `SUPABASE_SHADOW_CACHE` unset it IS today's - // create/remove pair; with it set it restores a key-matching PGDATA snapshot instead of - // rebuilding the platform baseline from scratch). + // create/remove pair; otherwise a key-matching PGDATA snapshot is restored into the fresh + // container in a few seconds instead of cold-provisioning the baseline in ~15s). diffResult = yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => Effect.gen(function* () { const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index df1372271b..61bd1d15aa 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -793,7 +793,8 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // `Effect.acquireUseRelease` (an interrupt must not be able to land between creation // and the finalizer being attached) and why the cache seam sits here. Note each // pooler-retry attempt still acquires and releases its own shadow — on the warm path - // that is the SAME restored baseline snapshot, sequentially. + // each attempt restores its own fresh container from the same cached snapshot, + // sequentially. return yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => Effect.gen(function* () { const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts index 66bbff6085..2292b254dc 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts @@ -892,10 +892,26 @@ export function legacyCreateContainer( ); // Sequentially, not concurrently like the secret files: two archives could legitimately // overlap in the container's filesystem, so the spec's own order has to be the applied order. + // + // A failed extraction removes the just-created container (best-effort, `-v` so an anonymous + // volume goes with it) before failing. The secret-file/`docker start` steps deliberately do + // NOT do this (Go-parity leak window — see `legacyCreateShadowDatabase`'s doc comment, + // `shadow-database.ts`), but `preStartArchives` is TS-only with no Go counterpart, and its one + // producer (the shadow baseline cache's warm restore) recovers from this exact failure by + // provisioning a replacement — which must not accumulate an orphaned created container per + // recovery (review: Codex on #6184). yield* Effect.forEach( finalSpec.preStartArchives ?? [], (archive) => legacyExtractPreStartArchiveIntoContainer(spawner, containerId, archive), { discard: true }, + ).pipe( + Effect.tapError(() => + containerCliExitCode(spawner, ["rm", "-f", "-v", containerId], { + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + }).pipe(Effect.orElseSucceed(() => 0)), + ), ); yield* legacyDockerStartContainer(spawner, containerId, finalSpec); return containerId; diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts index 471eb26fe1..4301f0f756 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts @@ -107,7 +107,10 @@ export const legacyExportPgDataTar = ( const [exitCode, , stderr] = yield* Effect.all( [ child.exitCode.pipe(Effect.map(Number)), - Stream.run(child.stdout, fs.sink(tempPath, { flag: "w" })), + // `0o600`: the archive is a full PGDATA — vault secret values, the JWT secret, and + // role password hashes are all in its pages — so it must not be group/world-readable + // on a shared host. `rename` preserves the mode, so the published tar inherits it. + Stream.run(child.stdout, fs.sink(tempPath, { flag: "w", mode: 0o600 })), legacyCollectText(child.stderr), ], { concurrency: "unbounded" }, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 458a8670da..5e8449cb63 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -565,6 +565,9 @@ describe("legacyAcquireShadowDatabase", () => { // Falls all the way back to a cold provision — a fresh container with no baseline. expect(fallback.baselinePresent).toBe(false); expect(fallback.containerId).not.toBe(cold.containerId); + // The container whose restore failed is removed, not orphaned: with the cold run's own + // container already released by `coldRun`, only the fallback's remains. + expect(docker.ids()).toEqual([fallback.containerId]); // The tar is the suspect and is deleted, so later runs do not retry it forever... expect(yield* soleTarName(fs, path)).toEqual([]); // ...and the cold fallback republishes one through its own snapshot step. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 726dc1ddcc..7e75174667 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -120,6 +120,17 @@ export interface LegacyShadowCacheKeyInputs { readonly rootKey: string; /** `[db] password` — baked into the cluster as the `postgres` role's password. */ readonly dbPassword: string; + /** + * The Storage migration pin read from `supabase/.temp/storage-migration` (written by + * `supabase link`), fed as `DB_MIGRATIONS_FREEZE_AT` to the Storage one-shot migrate job + * (`legacyStartStorageMigrateEnv`, `db-setup.ts`) — it decides WHICH Storage migrations the + * baseline carries, independently of the job image's own tag. `""` when absent (unlinked + * project), matching the setup input's own zero value. Hashed ONLY when `services.storage` + * is enabled AND `majorVersion >= 15` — the exact compound gate `legacyStartInitSchema15` + * (`db-setup.ts`) puts the consuming job behind, mirroring {@link jwks}'s treatment + * (review: depthfirst/Codex on #6184). + */ + readonly storageTargetMigration: string; readonly dbSettings: ProjectConfig["db"]["settings"]; /** Effective `api.auto_expose_new_tables` tri-state (unset ≠ explicit `false`: only the former keeps the bundled grants). */ readonly autoExposeNewTables: Option.Option; @@ -196,6 +207,13 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string ? `realtime_jwks=${inputs.jwks}` : "realtime_jwks=excluded", ); + // Storage's migration pin — same compound enabled+majorVersion gate as the JWKS line above, + // because the consuming one-shot job (`legacyStartInitSchema15`) is behind the same gate. + lines.push( + inputs.services.storage.enabled && inputs.majorVersion >= 15 + ? `storage_target_migration=${inputs.storageTargetMigration}` + : "storage_target_migration=excluded", + ); for (const secret of [...inputs.vault].sort((left, right) => left.name < right.name ? -1 : left.name > right.name ? 1 : 0, )) { @@ -256,6 +274,7 @@ const legacyResolveShadowCacheKeyInputs = ( rootKey: input.rootKey ?? "", dbPassword: input.password, dbSettings: input.db.settings, + storageTargetMigration: input.setup.storageTargetMigration, autoExposeNewTables: input.setup.apiAutoExposeNewTables, rolesSql, vault: input.setup.vault, @@ -526,6 +545,15 @@ const legacyColdCachedShadow = ( * The readiness failure removes the container here rather than leaving it to the caller, because * the caller's fallback creates a REPLACEMENT container and the suspect one must be gone by then * (it holds the shadow's published port). + * + * The readiness wait is explicitly `Effect.interruptible`: this whole function runs inside + * `Effect.acquireUseRelease`'s uninterruptible `acquire` ({@link legacyWithShadowDatabase}), and + * while the restore itself is short (~2s of `docker create` + `docker cp -`), a restored container + * that starts but never accepts connections would otherwise pin a Ctrl-C for the full + * `healthTimeoutSeconds` budget — exactly the swallowed-SIGINT shape `legacyPrepareShadowSource`'s + * own doc comment (`legacy-shadow-source.ts`) was restructured to avoid. `Effect.onInterrupt` + * removes the container on that path, so re-enabling interruption cannot leak it (review: Codex + * on #6184). */ const legacyWarmShadow = ( spawner: Spawner, @@ -550,6 +578,8 @@ const legacyWarmShadow = ( ); yield* legacyAwaitShadowReady(spawner, input, containerId, "restored shadow").pipe( Effect.tapError(() => legacyRemoveShadowDatabase(spawner, containerId)), + Effect.onInterrupt(() => legacyRemoveShadowDatabase(spawner, containerId)), + Effect.interruptible, ); return { containerId, @@ -570,9 +600,10 @@ const legacyWarmShadow = ( * Either way the container itself is created identically to an uncached one. * * Runs inside `acquireUseRelease`'s uninterruptible `acquire`, same as - * `legacyCreateShadowDatabase` — a warm restore adds ~2s of uninterruptible work (a `docker cp -` - * plus a readiness probe); the multi-second baseline/migration sequence stays in the interruptible - * `use` phase exactly as before. + * `legacyCreateShadowDatabase` — a warm restore adds ~2s of uninterruptible work (a `docker cp -`); + * its readiness wait re-enables interruption explicitly (see {@link legacyWarmShadow}), and the + * multi-second baseline/migration sequence stays in the interruptible `use` phase exactly as + * before. * * The `E` in the error channel is {@link legacyResolveShadowCacheKeyInputs}'s own JWKS * resolution alone (see that function's doc comment): every OTHER failure this function's own diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index 0ae4a1081e..adddcd2609 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -19,6 +19,7 @@ const baseKeyInputs = (): LegacyShadowCacheKeyInputs => ({ dbPassword: "postgres", dbSettings: { effective_cache_size: "128MB", max_connections: 100 }, autoExposeNewTables: Option.none(), + storageTargetMigration: "20240101000000", rolesSql: "create role custom_role;\n", vault: [{ name: "secret", value: "value", resolved: true }], jwks: '{"keys":[]}', @@ -82,6 +83,14 @@ describe("legacyShadowCacheKey", () => { inputs: { ...base, autoExposeNewTables: Option.some(false) }, }, { label: "roles.sql", inputs: { ...base, rolesSql: "" } }, + { + label: "storage migration pin (storage enabled, majorVersion >= 15)", + inputs: { ...base, storageTargetMigration: "20250607080910" }, + }, + { + label: "storage migration pin (pinned vs unpinned)", + inputs: { ...base, storageTargetMigration: "" }, + }, { label: "jwks (realtime enabled, majorVersion >= 15)", inputs: { ...base, jwks: '{"keys":["rotated"]}' }, @@ -188,6 +197,34 @@ describe("legacyShadowCacheKey", () => { expect(legacyShadowCacheKey(withJwksA)).toBe(legacyShadowCacheKey(withJwksB)); }); + it("excludes the storage migration pin when storage is disabled", () => { + const base = baseKeyInputs(); + const disabledStorage: LegacyShadowCacheKeyInputs = { + ...base, + services: { + ...base.services, + storage: { ...base.services.storage, enabled: false }, + }, + }; + const withPinA: LegacyShadowCacheKeyInputs = { + ...disabledStorage, + storageTargetMigration: "20240101000000", + }; + const withPinB: LegacyShadowCacheKeyInputs = { + ...disabledStorage, + storageTargetMigration: "20250607080910", + }; + expect(legacyShadowCacheKey(withPinA)).toBe(legacyShadowCacheKey(withPinB)); + }); + + it("excludes the storage migration pin when majorVersion is below 15, even with storage enabled", () => { + const base = baseKeyInputs(); + const pre15: LegacyShadowCacheKeyInputs = { ...base, majorVersion: 14 }; + const withPinA: LegacyShadowCacheKeyInputs = { ...pre15, storageTargetMigration: "a" }; + const withPinB: LegacyShadowCacheKeyInputs = { ...pre15, storageTargetMigration: "b" }; + expect(legacyShadowCacheKey(withPinA)).toBe(legacyShadowCacheKey(withPinB)); + }); + it("hashes vault secrets in a name-stable order", () => { const base = baseKeyInputs(); const ascending = legacyShadowCacheKey({ diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts index 10fb73e0ee..1c34355954 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts @@ -783,8 +783,9 @@ const exportViaShadowCatalog = ( // `legacyWithShadowDatabase` (`db-bootstrap/shadow-cache.ts`) rather than a bare // `legacyCreateShadowDatabase`/`legacyRemoveShadowDatabase` pair — see its doc comment: with // `SUPABASE_SHADOW_CACHE` unset it IS that pair (identical Docker argv, identical labels), and - // with it set a catalog cache miss restores a key-matching PGDATA baseline snapshot instead of - // paying the full cold provision — the same swap `db diff`/`db pull`'s own call sites make. + // with it set a catalog cache miss restores a key-matching PGDATA snapshot into the fresh + // shadow instead of paying the full cold provision — the same swap `db diff`/`db pull`'s own + // call sites make. const written = yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => Effect.gen(function* () { const shadow = yield* provision(spawner, handle, shadowInput); diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts index 4649d037f0..58291b6b8a 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts @@ -10,7 +10,7 @@ import type { Path } from "effect"; -/** `supabase/.temp/pgdelta` — where catalog snapshots, debug bundles, and the shadow-container cache's metadata live (`declarative.go:44`). */ +/** `supabase/.temp/pgdelta` — where catalog snapshots, debug bundles, and the shadow baseline cache's PGDATA tars live (`declarative.go:44`). */ export function legacyPgDeltaTempPath(path: Path.Path, workdir: string): string { return path.join(workdir, "supabase", ".temp", "pgdelta"); } From 4520ac940d92bfb29a300f199b438239dd3162fb Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 18:10:38 +0200 Subject: [PATCH 42/82] docs(cli): record deferred shadow-cache review follow-up (init SQL key coverage) Co-Authored-By: Claude Fable 5 --- docs/roadmap/pg-delta-next-follow-ups.md | 25 ++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 docs/roadmap/pg-delta-next-follow-ups.md diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md new file mode 100644 index 0000000000..4992b1ae93 --- /dev/null +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -0,0 +1,25 @@ +# pg-delta / shadow-database follow-ups + +Deferred review findings that are valid but out of scope for the PR that surfaced them. +Each entry names the PR it came out of so the context is recoverable. + +## Shadow baseline cache key does not cover CLI-embedded init SQL (PR #6184) + +The cache key (`legacyShadowCacheKey`, `apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts`) +hashes every *config-derived* input baked into the shadow cluster, but not the CLI-embedded init +SQL the entrypoint heredocs into the cluster at `initdb` time (`LEGACY_START_DB_SCHEMA_SQL`, +`LEGACY_START_DB_WEBHOOK_SQL`, `LEGACY_START_DB_SUPABASE_SQL` — `postgres.service.ts`), nor the +baseline steps `legacySetupDatabase` itself performs (API privileges SQL, vault upsert SQL). If a +CLI release changes any of those without a `supabase/postgres` image bump, a warm tar produced by +the older CLI silently restores the older baseline. + +Realistic frequency is low (these constants track Go's `start.go` templates and change rarely, +usually alongside image bumps), which is why it did not block #6184. Two candidate fixes: + +- Fold the CLI version into the key — over-invalidates once per release (~one 15s cold run per + upgrade), trivially safe, one line. +- Hash the embedded SQL constants themselves — precise, no per-release invalidation, slightly more + surface. + +Either way, add a unit-test mutation case alongside the existing ones in +`shadow-cache.unit.test.ts`. From cbace05755d550953d88f9dbdcf7771bcfdfc395 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 18:55:54 +0200 Subject: [PATCH 43/82] fix(cli): key shadow snapshots by registry-resolved job images, interruptible key resolution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second Codex round on #6184: - Hash the three one-shot job images through `legacyGetRegistryImageUrl` (same projectEnvValues-then-ambient precedence the job's own `legacyEnsureImagesCached` resolve applies), so two values of `SUPABASE_INTERNAL_IMAGE_REGISTRY` no longer share a snapshot under identical tags — matching the postgres image field, which was already the registry-resolved form. - Wrap cache-key input resolution in `Effect.interruptible`: it runs inside acquireUseRelease's uninterruptible acquire but before any container exists, and the JWKS effect inside can be a real third-party discovery request that must not pin a Ctrl-C. Co-Authored-By: Claude Fable 5 --- .../shadow-cache.integration.test.ts | 30 +++++++++++++++++++ .../shared/db-bootstrap/shadow-cache.ts | 29 ++++++++++++++---- 2 files changed, 54 insertions(+), 5 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 5e8449cb63..7c65dc6c9f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -523,6 +523,36 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("a changed internal image registry is a different key, not a warm hit", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + yield* coldRun(docker, input); + const defaultRegistryTar = yield* soleTarName(fs, path); + expect(defaultRegistryTar).toHaveLength(1); + + // The one-shot migrate jobs resolve their images through + // `SUPABASE_INTERNAL_IMAGE_REGISTRY`, so a different registry can bake different + // realtime/storage/auth schema under identical tags — the snapshot must not be shared. + const mirrored = yield* withEnv( + "SUPABASE_INTERNAL_IMAGE_REGISTRY", + "mirror.internal.example", + coldRun(docker, input), + ); + expect(mirrored.baselinePresent).toBe(false); + const mirroredTar = yield* soleTarName(fs, path); + expect(mirroredTar).toHaveLength(1); + expect(mirroredTar[0]).not.toBe(defaultRegistryTar[0]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("a failed export warns, leaves no tar, and still brings the container back up", () => { const docker = fakeDockerDaemon({ failCopyOut: true }); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 7e75174667..7d37de9a7d 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -35,6 +35,7 @@ import { } from "../legacy-container-cli.ts"; import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import type { LegacyPgConnInput } from "../legacy-db-connection.service.ts"; +import { legacyGetRegistryImageUrl } from "../legacy-docker-registry.ts"; import { legacyPgDeltaTempPath } from "../legacy-pgdelta.paths.ts"; import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; import type { LegacyVaultSecret } from "../legacy-vault.ts"; @@ -92,7 +93,15 @@ export function legacyShadowCacheEnabled( /** One of the three PG15+ one-shot migrate jobs, as the cache key sees it. */ export interface LegacyShadowCacheServiceInput { readonly enabled: boolean; - /** `legacyResolvePinnedImage`'s resolved image — hashed ONLY when {@link enabled}, since a disabled service's job never ran into the baseline. */ + /** + * `legacyResolvePinnedImage`'s pinned image passed through `legacyGetRegistryImageUrl` — the + * REGISTRY-RESOLVED ref, not the bare `supabase/:` pin, because that is the identity + * `legacyRunStartMigrateJob`'s own `legacyEnsureImagesCached` resolve actually runs: a changed + * `SUPABASE_INTERNAL_IMAGE_REGISTRY` (ambient or project-`.env`) can serve a different image + * under the same tag, and the postgres image field above is already the registry-resolved form + * (review: Codex on #6184). Hashed ONLY when {@link enabled}, since a disabled service's job + * never ran into the baseline. + */ readonly image: string; } @@ -257,6 +266,11 @@ const legacyResolveShadowCacheKeyInputs = ( if (rolesSql === undefined) return Option.none(); const overrides = input.setup.serviceVersionOverrides; + // The same registry rewrite `legacyRunStartMigrateJob`'s `legacyEnsureImagesCached` applies + // when the job actually runs (same `projectEnvValues`-then-ambient precedence) — see + // {@link LegacyShadowCacheServiceInput.image}. + const resolveJobImage = (image: string): string => + legacyGetRegistryImageUrl(image, input.setup.projectEnvValues); // The exact compound gate {@link legacyResolveDbSetupPrelude} uses to decide whether the // realtime one-shot job's JWKS effect is reached (and therefore run) at all during a real // setup — see `db-setup.ts`. Gating on anything looser here would resolve (and risk failing @@ -282,15 +296,15 @@ const legacyResolveShadowCacheKeyInputs = ( services: { realtime: { enabled: input.setup.config.realtime.enabled, - image: legacyResolvePinnedImage("realtime", "realtime", overrides), + image: resolveJobImage(legacyResolvePinnedImage("realtime", "realtime", overrides)), }, storage: { enabled: input.setup.config.storage.enabled, - image: legacyResolvePinnedImage("storage", "storage", overrides), + image: resolveJobImage(legacyResolvePinnedImage("storage", "storage", overrides)), }, auth: { enabled: input.setup.config.auth.enabled, - image: legacyResolvePinnedImage("gotrue", "auth", overrides), + image: resolveJobImage(legacyResolvePinnedImage("gotrue", "auth", overrides)), }, }, } satisfies LegacyShadowCacheKeyInputs); @@ -622,7 +636,12 @@ export const legacyAcquireShadowDatabase = ( Effect.gen(function* () { if (!legacyShadowCacheEnabled()) return yield* legacyUncachedShadow(spawner, input); - const keyInputs = yield* legacyResolveShadowCacheKeyInputs(input); + // Interruptible: this runs inside `acquireUseRelease`'s uninterruptible `acquire`, but + // nothing has been acquired yet — and the JWKS effect inside can be a real third-party + // discovery request, which must not pin a Ctrl-C for its whole duration (review: Codex on + // #6184). Interruption here simply means no container was ever created, so there is nothing + // for a finalizer to release. + const keyInputs = yield* Effect.interruptible(legacyResolveShadowCacheKeyInputs(input)); if (Option.isNone(keyInputs)) return yield* legacyUncachedShadow(spawner, input); const key = legacyShadowCacheKey(keyInputs.value); const tarPath = input.path.join( From 992533830510d44ed5de3b34d55888dad3271a1a Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 19:07:17 +0200 Subject: [PATCH 44/82] fix(cli): fold the CLI-embedded baseline SQL into the shadow cache key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third review round on #6184 (depthfirst): the key hashed every config-derived input but not the CLI-embedded SQL baked into the cluster — the PG15+ entrypoint's initdb heredocs, the PG<=14 globals + initial schema, and the API privilege revocation. A CLI release changing a grant or schema statement without a postgres image bump would warm-restore the previous release's baseline. The key now folds in a sha256 digest of those constants, computed once at module load; resolves the entry recorded in docs/roadmap/pg-delta-next-follow-ups.md. Co-Authored-By: Claude Fable 5 --- .../legacy/shared/db-bootstrap/db-setup.ts | 6 ++-- .../shared/db-bootstrap/shadow-cache.ts | 34 +++++++++++++++++++ docs/roadmap/pg-delta-next-follow-ups.md | 8 ++++- 3 files changed, 45 insertions(+), 3 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index a8fb88343f..1893a911e6 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -171,9 +171,11 @@ type Spawner = ChildProcessSpawner["Service"]; /** * Go's inline `RevokeDefaultDataApiPrivilegesSql` constant (`start.go:405-412`) — * NOT a `//go:embed` file (unlike the three large SQL templates), so transcribed - * directly here rather than as a sibling `templates/*.sql.ts` module. + * directly here rather than as a sibling `templates/*.sql.ts` module. Exported for the + * shadow baseline cache's embedded-SQL digest (`shadow-cache.ts`), which must re-key + * whenever this text changes across CLI releases. */ -const LEGACY_START_REVOKE_API_PRIVILEGES_SQL = ` +export const LEGACY_START_REVOKE_API_PRIVILEGES_SQL = ` alter default privileges for role postgres in schema public revoke select, insert, update, delete on tables from anon, authenticated, service_role; alter default privileges for role postgres in schema public diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 7d37de9a7d..bc235c08cc 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -38,6 +38,13 @@ import type { LegacyPgConnInput } from "../legacy-db-connection.service.ts"; import { legacyGetRegistryImageUrl } from "../legacy-docker-registry.ts"; import { legacyPgDeltaTempPath } from "../legacy-pgdelta.paths.ts"; import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; +import { LEGACY_START_REVOKE_API_PRIVILEGES_SQL } from "./db-setup.ts"; +import { LEGACY_START_DB_GLOBALS_SQL } from "./templates/db-globals.sql.ts"; +import { LEGACY_START_DB_INITIAL_SCHEMA_13_SQL } from "./templates/db-initial-schema-13.sql.ts"; +import { LEGACY_START_DB_INITIAL_SCHEMA_14_SQL } from "./templates/db-initial-schema-14.sql.ts"; +import { LEGACY_START_DB_SCHEMA_SQL } from "./templates/db-schema.sql.ts"; +import { LEGACY_START_DB_SUPABASE_SQL } from "./templates/db-supabase.sql.ts"; +import { LEGACY_START_DB_WEBHOOK_SQL } from "./templates/db-webhook.sql.ts"; import type { LegacyVaultSecret } from "../legacy-vault.ts"; import { legacyWaitForShadowReady } from "./health-check.ts"; import { legacyExportPgDataTar, legacyPgDataRestoreArchive } from "./pgdata-snapshot.ts"; @@ -167,6 +174,31 @@ export interface LegacyShadowCacheKeyInputs { readonly jwks: string; } +/** + * Digest of every CLI-EMBEDDED SQL text baked into the baseline cluster — the inputs that change + * with a CLI release rather than with the project's config: the PG15+ entrypoint's initdb heredocs + * (schema/webhook/_supabase — `postgres.service.ts`), the PG<=14 setup path's globals + initial + * schema, and the API privilege revocation. Without this line, a CLI upgrade that edits a grant, + * schema statement, or revocation WITHOUT bumping the postgres image would warm-restore the + * previous release's baseline (review: depthfirst on #6184). Computed once at module load — these + * are compile-time constants. When adding a new embedded SQL step to the baseline + * (`legacySetupDatabase`/the entrypoint scripts), add its text here too. + */ +const LEGACY_SHADOW_BASELINE_SQL_DIGEST = createHash("sha256") + .update( + [ + LEGACY_START_DB_SCHEMA_SQL, + LEGACY_START_DB_WEBHOOK_SQL, + LEGACY_START_DB_SUPABASE_SQL, + LEGACY_START_DB_GLOBALS_SQL, + LEGACY_START_DB_INITIAL_SCHEMA_13_SQL, + LEGACY_START_DB_INITIAL_SCHEMA_14_SQL, + LEGACY_START_REVOKE_API_PRIVILEGES_SQL, + ].join("\n--8<--\n"), + "utf8", + ) + .digest("hex"); + /** JSON with recursively key-sorted objects, so `db.settings`' own property order cannot change the key. */ function legacyCanonicalJson(value: unknown): string { if (value === null || typeof value !== "object") return JSON.stringify(value) ?? "null"; @@ -199,6 +231,8 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string `db_password=${inputs.dbPassword}`, `db_settings=${legacyCanonicalJson(inputs.dbSettings)}`, `auto_expose_new_tables=${legacyTriStateToken(inputs.autoExposeNewTables)}`, + // Not a per-run input — see the digest's own doc comment for what it covers and why. + `baseline_sql_digest=${LEGACY_SHADOW_BASELINE_SQL_DIGEST}`, ]; for (const name of ["realtime", "storage", "auth"] as const) { const service = inputs.services[name]; diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md index 4992b1ae93..96944d8d17 100644 --- a/docs/roadmap/pg-delta-next-follow-ups.md +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -3,7 +3,13 @@ Deferred review findings that are valid but out of scope for the PR that surfaced them. Each entry names the PR it came out of so the context is recoverable. -## Shadow baseline cache key does not cover CLI-embedded init SQL (PR #6184) +## ~~Shadow baseline cache key does not cover CLI-embedded init SQL~~ (resolved in PR #6184) + +Resolved on the PR itself after depthfirst independently flagged it: the key now folds in a +digest of the embedded init/privilege SQL constants (`LEGACY_SHADOW_BASELINE_SQL_DIGEST`, +`shadow-cache.ts`). Original write-up kept below for the rationale and the alternative considered. + +## Original: shadow baseline cache key does not cover CLI-embedded init SQL (PR #6184) The cache key (`legacyShadowCacheKey`, `apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts`) hashes every *config-derived* input baked into the shadow cluster, but not the CLI-embedded init From 02e7b3620768c0656631b01516f3bde0f67d7b18 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 19:11:11 +0200 Subject: [PATCH 45/82] fix(cli): resolve the shadow JWKS effect once per run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fourth review round on #6184 (Codex): with the cache enabled, a cold run evaluated `setup.jwks` twice — once for the cache key, once inside `legacyResolveDbSetupPrelude` for the baseline — and third-party JWKS discovery is a real network request, so a transient second failure could break a run and an issuer rotation mid-run could publish a snapshot under a key computed from a different value than the cluster carries. `legacyShadowRunInputFromLocalContainerInputs` now memoizes the effect's first success (failures still re-run), covering every shadow call site. Co-Authored-By: Claude Fable 5 --- .../shared/db-bootstrap/shadow-database.ts | 31 ++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index f7e18f58ec..eb6dedfb83 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -414,6 +414,28 @@ export interface LegacyShadowSetupInput extends LegacyShadowConnectionInput { readonly setup: LegacyShadowDbSetupInput; } +/** + * Memoizes `effect`'s first SUCCESS; failures are never cached, so a retry re-runs the real + * effect. Deliberately not `Effect.cached` (which returns `Effect>` and needs an + * effectful construction site — {@link legacyShadowRunInputFromLocalContainerInputs} is a plain + * function) and not concurrency-guarded: the two consumers of the one field this wraps (`jwks` — + * see its construction inside that function) evaluate sequentially on the same fiber. + */ +function legacyMemoizeSuccess(effect: Effect.Effect): Effect.Effect { + let succeeded: Effect.Effect | undefined; + return Effect.suspend( + () => + succeeded ?? + effect.pipe( + Effect.tap((value) => + Effect.sync(() => { + succeeded = Effect.succeed(value); + }), + ), + ), + ); +} + /** * Adapts {@link LegacyLocalDbContainerInputs} (`local-container-inputs.ts`, the SAME * config/image/JWKS resolution prelude `db start`/`db reset` share) plus the caller's own @@ -497,7 +519,14 @@ export function legacyShadowRunInputFromLocalContainerInputs( database: "postgres", }), jwtSecret: localInputs.setup.jwtSecret, - jwks: localInputs.setup.jwks, + // Memoized: with the shadow baseline cache enabled this effect is evaluated TWICE on a + // cold run — once by `legacyResolveShadowCacheKeyInputs` (`shadow-cache.ts`) for the cache + // key, once by `legacyResolveDbSetupPrelude` for the baseline itself — and third-party + // JWKS discovery can be a real network request. Memoizing the first success keeps the run + // to one request AND guarantees the published snapshot carries the exact value its key was + // computed from, even if the issuer rotates mid-run (review: Codex on #6184). Failures are + // not cached — a transient discovery failure fails the run either way. + jwks: legacyMemoizeSuccess(localInputs.setup.jwks), apiUrl: localInputs.setup.apiUrl, authExternalUrl: localInputs.setup.authExternalUrl, siteUrl: localInputs.setup.siteUrl, From b3bc6527b1eb748f1e3acd169c179dd69e79ccf6 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 19:25:31 +0200 Subject: [PATCH 46/82] fix(cli): honor the shadow cache opt-out from project dotenv, document warm-path output Fifth review round on #6184 (Codex). One fix, one documentation pass, three declines (rationale on the PR threads): - `legacyShadowCacheEnabled` now consults the project's dotenv-merged env before ambient `process.env`, same precedence as the registry override, so `SUPABASE_SHADOW_CACHE=false` in `supabase/.env` is honored. - SIDE_EFFECTS cache sections for db diff/db pull/declarative sync now state that a warm hit skips the `Initialising schema...` line (progress text reflects work actually performed) and that the opt-out works from both env sources. Co-Authored-By: Claude Fable 5 --- .../cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md | 7 +++++-- .../cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md | 7 +++++-- .../db/schema/declarative/sync/SIDE_EFFECTS.md | 4 +++- .../src/legacy/shared/db-bootstrap/shadow-cache.ts | 13 +++++++++++-- .../shared/db-bootstrap/shadow-cache.unit.test.ts | 13 +++++++++++++ 5 files changed, 37 insertions(+), 7 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 6c4aa35a3a..068da79805 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -210,8 +210,11 @@ when declarative files exist. ### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) -ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out, restoring the documented uncached -lifecycle. Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot +ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the ambient env AND the +project's dotenv, e.g. `supabase/.env`), restoring the documented uncached lifecycle. A warm hit +skips the platform baseline, so the `Initialising schema...` progress line does not print — +progress text reflects the work actually performed. +Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index f1138cc809..a090a581da 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -76,8 +76,11 @@ disables formatting without disabling safe compaction. ### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) -ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out, restoring the documented uncached -lifecycle. Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot +ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the ambient env AND the +project's dotenv, e.g. `supabase/.env`), restoring the documented uncached lifecycle. A warm hit +skips the platform baseline, so the `Initialising schema...` progress line does not print — +progress text reflects the work actually performed. +Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 40f58166bf..b836db1a67 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -127,7 +127,9 @@ existing SQL or creates an export manifest. The migrations-catalog shadow this command provisions on a cache miss goes through `legacyGetMigrationsCatalogRef` -> `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), the same `legacyWithShadowDatabase` seam `db diff`/`db pull` use, so it inherits the whole lifecycle: ON by -default, `SUPABASE_SHADOW_CACHE=false`/`=0` opts out. Artifact: +default, `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the ambient env AND the +project's dotenv, e.g. `supabase/.env`); a warm hit skips the platform baseline and therefore the +`Initialising schema...` progress line. Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index bc235c08cc..ef77b45a4e 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -84,11 +84,18 @@ const legacyShadowCacheUnavailable = (reason: string): LegacyShadowCacheUnavaila * Unset (or empty) means ON — this is a default-on optimization, not a feature flag. A value that * IS set goes through the repo's `viper.GetBool` parser, so `SUPABASE_SHADOW_CACHE=false` and * `=0` opt out while `=1`/`=true` are explicit opt-ins. + * + * `projectEnvValues` (the project's dotenv-merged env, ambient-wins — see + * `legacyGetRegistryOverride`'s identical parameter, `legacy-docker-registry.ts`) is consulted + * first so an opt-out set only in `supabase/.env` is honored, matching how Go's `loadNestedEnv` + * makes project dotenv visible to every viper read (review: Codex on #6184). Falls back to `env` + * (ambient) when the record is absent or lacks the key. */ export function legacyShadowCacheEnabled( env: Readonly> = process.env, + projectEnvValues?: Readonly>, ): boolean { - const raw = env[LEGACY_SHADOW_CACHE_ENV]; + const raw = projectEnvValues?.[LEGACY_SHADOW_CACHE_ENV] ?? env[LEGACY_SHADOW_CACHE_ENV]; if (raw === undefined || raw.length === 0) return true; return legacyParseBoolEnv(raw); } @@ -668,7 +675,9 @@ export const legacyAcquireShadowDatabase = ( Output | LegacyDbConnection > => Effect.gen(function* () { - if (!legacyShadowCacheEnabled()) return yield* legacyUncachedShadow(spawner, input); + if (!legacyShadowCacheEnabled(process.env, input.setup.projectEnvValues)) { + return yield* legacyUncachedShadow(spawner, input); + } // Interruptible: this runs inside `acquireUseRelease`'s uninterruptible `acquire`, but // nothing has been acquired yet — and the JWKS effect inside can be a real third-party diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index adddcd2609..f3bb8d5e2c 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -44,6 +44,19 @@ describe("legacyShadowCacheEnabled", () => { expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "true" })).toBe(true); expect(legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "TRUE" })).toBe(true); }); + + it("honors an opt-out set only in the project dotenv values", () => { + // `supabase/.env` is loaded into `projectEnvValues` (ambient-wins merge, upstream), not into + // `process.env` — the gate must consult it, same as the registry override does. + expect(legacyShadowCacheEnabled({}, { SUPABASE_SHADOW_CACHE: "false" })).toBe(false); + expect(legacyShadowCacheEnabled({}, { SUPABASE_SHADOW_CACHE: "1" })).toBe(true); + expect(legacyShadowCacheEnabled({}, {})).toBe(true); + // The record's own value is what counts once present — upstream merging already applied + // ambient-wins precedence when building it. + expect( + legacyShadowCacheEnabled({ SUPABASE_SHADOW_CACHE: "1" }, { SUPABASE_SHADOW_CACHE: "0" }), + ).toBe(false); + }); }); describe("legacyShadowCacheKey", () => { From 7956603c28f3cd031cb8ccaaaf700c3fbbd7ced0 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 19:43:13 +0200 Subject: [PATCH 47/82] fix(cli): make OrioleDB shadows cache-ineligible, collision-proof vault key lines, complete SIDE_EFFECTS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sixth review round on #6184. Two fixes, two doc completions, one decline (rationale on the thread): - An OrioleDB cluster (`experimental.orioledb_version`) now bypasses the cache entirely: that branch runs the shadow with an external S3 storage backend, so a disk-level PGDATA tar is not a coherent snapshot to begin with — stronger than keying the S3 fields. - Vault entries in the key payload are JSON-encoded tuples instead of `name=value`, closing delimiter collisions and newline injection between two unrestricted strings. - SIDE_EFFECTS for db diff/db pull/declarative sync now list the snapshot tar under Files Read (warm hit) and document SUPABASE_SHADOW_DEBUG in the env tables. Co-Authored-By: Claude Fable 5 --- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 2 ++ .../legacy/commands/db/pull/SIDE_EFFECTS.md | 2 ++ .../schema/declarative/sync/SIDE_EFFECTS.md | 2 ++ .../shadow-cache.integration.test.ts | 25 +++++++++++++++++++ .../shared/db-bootstrap/shadow-cache.ts | 19 +++++++++++--- .../db-bootstrap/shadow-cache.unit.test.ts | 15 +++++++++++ 6 files changed, 62 insertions(+), 3 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 068da79805..cabcb70bf1 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -26,6 +26,7 @@ it, and JSON `null` disables formatting without disabling safe compaction. | `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | | `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | | `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit — the snapshot is streamed into the fresh shadow container before it starts | | `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | | `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | | `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | @@ -94,6 +95,7 @@ of this command's own target resolve, ahead of the differ container. | `SUPABASE_PROJECT_ID` | overrides the shadow container's project id/labels, same as `db start`/`db reset` (`utils.DbId`); ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | | `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | +| `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta engine | no | | `PGDELTA_DEBUG` | pg-delta debug capture | no | | `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index a090a581da..9203df67b5 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -42,6 +42,7 @@ disables formatting without disabling safe compaction. | `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | | `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | | `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — snapshot streamed into the fresh shadow container before it starts | | `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | | `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | | `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | @@ -110,6 +111,7 @@ baseline, so it is never cached. | `SUPABASE_PROJECT_ID` | overrides the shadow container's project id/labels, same as `db start`/`db reset` (`utils.DbId`); ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | | `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | +| `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta diff engine | no | | `SUPABASE_EXPERIMENTAL` | selects the deprecated structured-dump branch (still delegates to Go, see below) | no | | `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index b836db1a67..b9bab2bb6c 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -25,6 +25,7 @@ disabling safe compaction. | `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | | `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | | `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit on a migrations-catalog miss — snapshot streamed into the fresh shadow container before it starts | ## Files Written @@ -54,6 +55,7 @@ disabling safe compaction. | `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | | `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | | `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | +| `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | | `SUPABASE_SERVICES_HOSTNAME` | local DB host for the bootstrap generate | no | | `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 7c65dc6c9f..fb02ecfdf0 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -386,6 +386,31 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("stays uncached for an OrioleDB cluster even with the cache enabled", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + // OrioleDB runs the shadow with an external S3 storage backend, so a disk-level PGDATA + // tar is not a coherent snapshot — the acquire must degrade to the bare uncached shadow. + const input = { + ...shadowInput(fs, path), + experimental: { ...defaultConfig.experimental, orioledb_version: "15" }, + }; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(handle.baselinePresent).toBe(false); + expect(docker.calls("create")[0] ?? []).toContain("--rm"); + yield* handle.snapshotBaseline; + expect(docker.calls("stop")).toEqual([]); + expect(yield* soleTarName(fs, path)).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("takes the cache path when the env var is unset (default ON)", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index ef77b45a4e..2caecd9729 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -267,7 +267,10 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string for (const secret of [...inputs.vault].sort((left, right) => left.name < right.name ? -1 : left.name > right.name ? 1 : 0, )) { - lines.push(`vault=${secret.name}=${secret.value}`); + // JSON-encoded tuple, not `name=value`: both halves are unrestricted strings, so a bare + // `=` join would let (`a=b`, `c`) and (`a`, `b=c`) collide — and a value containing a + // newline could forge a whole extra payload line (review: Codex on #6184). + lines.push(`vault=${JSON.stringify([secret.name, secret.value])}`); } // Last, raw (it can contain anything, including newlines) — mirroring `setupInputsToken`, // which also appends `roles.sql` verbatim at the end of its own payload. @@ -281,8 +284,9 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string * `setup.config` (NOT the `*EnabledForSetup` fields, which only gate JWKS resolution) because * `legacySetupDatabase`'s own one-shot job gates read exactly those config fields. * - * Returns `Option.none` (never a failure) for the one condition that makes the key - * uncomputable — an unreadable `roles.sql` — so this function's OWN error channel carries + * Returns `Option.none` (never a failure) for the two conditions that make caching + * unavailable — an OrioleDB cluster (whose state is partly external, see the body's own + * comment) and an unreadable `roles.sql` — so this function's OWN error channel carries * nothing but `E`: the `input.setup.jwks` effect below is `yield*`ed unguarded (only when the * consuming realtime job is actually reachable — see the field's own doc comment), and a real * JWKS failure must fail this whole acquire exactly as it would have failed a real setup, never @@ -296,6 +300,15 @@ const legacyResolveShadowCacheKeyInputs = ( input: LegacyShadowSetupInput, ): Effect.Effect, E> => Effect.gen(function* () { + // OrioleDB (`experimental.orioledb_version`) makes the WHOLE cache ineligible, not just a + // key input: that branch runs the shadow with an external S3 storage backend + // (`S3_ENABLED=true` — `legacyPostgresExtraEnv`, `postgres.service.ts`), so part of the + // cluster's state lives outside PGDATA and a disk-level tar is not a coherent snapshot to + // begin with (review: Codex on #6184). Same `Option.none` degradation as an unreadable + // `roles.sql` below: the run proceeds uncached. + const orioledbVersion = input.experimental.orioledb_version; + if (orioledbVersion !== undefined && orioledbVersion.length > 0) return Option.none(); + const rolesPath = input.path.join(input.workdir, "supabase", "roles.sql"); const rolesSql = yield* input.fs .readFileString(rolesPath) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index f3bb8d5e2c..2057395fed 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -238,6 +238,21 @@ describe("legacyShadowCacheKey", () => { expect(legacyShadowCacheKey(withPinA)).toBe(legacyShadowCacheKey(withPinB)); }); + it("cannot collide vault name/value pairs across the tuple boundary", () => { + const base = baseKeyInputs(); + // `name=a=b, value=c` vs `name=a, value=b=c` — a bare `=`-joined encoding serializes both + // as `vault=a=b=c`. + const left = legacyShadowCacheKey({ + ...base, + vault: [{ name: "a=b", value: "c", resolved: true }], + }); + const right = legacyShadowCacheKey({ + ...base, + vault: [{ name: "a", value: "b=c", resolved: true }], + }); + expect(left).not.toBe(right); + }); + it("hashes vault secrets in a name-stable order", () => { const base = baseKeyInputs(); const ascending = legacyShadowCacheKey({ From 7d033e4dd09a18e03ca47a56776de45f03ded674 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 19:54:03 +0200 Subject: [PATCH 48/82] docs(cli): narrow the snapshot tar's Files Written conditions to cold shadow provisions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seventh review round on #6184 (Codex): the tar rows claimed every cache-enabled invocation writes the snapshot; it is only written by a cache-enabled COLD shadow provision — never --use-pgadmin / --use-pg-schema / pull --declarative / a catalog cache hit / a warm hit. Doc-only. Co-Authored-By: Claude Fable 5 --- apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md | 2 +- apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md | 2 +- .../legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index cabcb70bf1..09f9104604 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -42,7 +42,7 @@ it, and JSON `null` disables formatting without disabling safe compaction. | `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | | `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | shadow baseline cache enabled (default) — the shadow's PGDATA snapshot, ~90MB, current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision only (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`, never a warm hit) — the shadow's PGDATA snapshot, ~90MB, current key only | | `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | | `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 9203df67b5..14bff114cb 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -59,7 +59,7 @@ disables formatting without disabling safe compaction. | `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | | `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | shadow baseline cache enabled (default) — the shadow's PGDATA snapshot, ~90MB, one file for the current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision, migration-style pull only (never `--declarative`'s bare shadow, the delegated `--experimental` path, or a warm hit) — the shadow's PGDATA snapshot, ~90MB, one file for the current key only | | `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | | `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index b9bab2bb6c..e1f3d05482 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -35,7 +35,7 @@ disabling safe compaction. | `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | | `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | shadow baseline cache enabled (default) — the shadow's PGDATA snapshot, ~90MB, current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing) — the shadow's PGDATA snapshot, ~90MB, current key only | ## Subprocesses / Containers From 45c3fb1b7f10e4b0ba5824326b601c74f3189ab2 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 19:55:21 +0200 Subject: [PATCH 49/82] chore(cli): format SIDE_EFFECTS tables Co-Authored-By: Claude Fable 5 --- apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 14bff114cb..409e305272 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -60,8 +60,8 @@ disables formatting without disabling safe compaction. | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | | `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | | `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision, migration-style pull only (never `--declarative`'s bare shadow, the delegated `--experimental` path, or a warm hit) — the shadow's PGDATA snapshot, ~90MB, one file for the current key only | -| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker From cda6d4f80ff4545cc763ee34b4cabbe0db58b219 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 20:11:42 +0200 Subject: [PATCH 50/82] fix(cli): make --no-cache bypass the shadow snapshot, hash only what the cluster carries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eighth review round on #6184 (Codex), all three findings valid: - `db schema declarative sync --no-cache` documents "force fresh shadow database setup", but the new snapshot cache could still warm-restore the baseline under it. `LegacyShadowCacheOpts.bypassCache` now threads the flag through `exportViaShadowCatalog` -> `legacyWithShadowDatabase` -> the acquire, which takes the fully uncached path (no restore, no export). - The key hashes only RESOLVED vault entries — `legacyUpsertVaultSecrets` filters on `resolved`, so an unresolved entry never lands in the cluster and must not affect the key. - Every unrestricted scalar in the key payload is JSON-encoded, closing the newline-forgery collision class for good (rootKey/dbPassword/ jwtSecret/jwks/images/pin — same treatment the vault tuples already got). Co-Authored-By: Claude Fable 5 --- .../schema/declarative/sync/SIDE_EFFECTS.md | 6 +- .../shadow-cache.integration.test.ts | 28 ++++++++++ .../shared/db-bootstrap/shadow-cache.ts | 56 ++++++++++++++----- .../db-bootstrap/shadow-cache.unit.test.ts | 33 +++++++++++ .../src/legacy/shared/legacy-pgdelta.cache.ts | 31 ++++++---- 5 files changed, 129 insertions(+), 25 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index e1f3d05482..aed88cd3f4 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -35,7 +35,7 @@ disabling safe compaction. | `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | | `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing) — the shadow's PGDATA snapshot, ~90MB, current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, current key only | ## Subprocesses / Containers @@ -130,7 +130,9 @@ The migrations-catalog shadow this command provisions on a cache miss goes throu `legacyGetMigrationsCatalogRef` -> `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), the same `legacyWithShadowDatabase` seam `db diff`/`db pull` use, so it inherits the whole lifecycle: ON by default, `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the ambient env AND the -project's dotenv, e.g. `supabase/.env`); a warm hit skips the platform baseline and therefore the +project's dotenv, e.g. `supabase/.env`), and `--no-cache` bypasses it for that invocation (the +flag promises fresh shadow setup, so it disables the snapshot cache along with the catalog +cache); a warm hit skips the platform baseline and therefore the `Initialising schema...` progress line. Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. Container diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index fb02ecfdf0..59c7f519ee 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -386,6 +386,34 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("bypassCache acquires an uncached shadow even when a warm tar exists", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + // A published tar for this exact key — `sync --no-cache`'s bypass must ignore it. + yield* coldRun(docker, input); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input, { + bypassCache: true, + }); + expect(handle.baselinePresent).toBe(false); + // No restore in, no export out: the bypassed run neither reads nor rewrites the tar. + const bypassSteps = docker.steps().slice(docker.steps().lastIndexOf("create")); + expect(bypassSteps).toEqual(["create", "cp-secret", "start"]); + expect(docker.calls("create").at(-1) ?? []).toContain("--rm"); + yield* handle.snapshotBaseline; + expect(docker.stepCalls("cp-out")).toHaveLength(1); // the initial cold run's only + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("stays uncached for an OrioleDB cluster even with the cache enabled", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 2caecd9729..eb3fdfdcd4 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -159,7 +159,11 @@ export interface LegacyShadowCacheKeyInputs { readonly autoExposeNewTables: Option.Option; /** `supabase/roles.sql`'s contents, `""` when absent. */ readonly rolesSql: string; - /** `[db.vault]` secrets — names AND values, both of which land in `vault.secrets`. */ + /** + * `[db.vault]` secrets — names AND values, both of which land in `vault.secrets`. Only + * RESOLVED entries are hashed ({@link legacyShadowCacheKey}), because the upsert skips + * unresolved ones entirely — see the loop's own comment there. + */ readonly vault: ReadonlyArray; readonly services: { readonly realtime: LegacyShadowCacheServiceInput; @@ -228,14 +232,20 @@ const legacyTriStateToken = (value: Option.Option) => * collision, not a missed field. Short enough to read in a filename. */ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string { + // Every UNRESTRICTED string is JSON-encoded before interpolation (`quoted`): a raw newline in + // one field could otherwise forge a whole extra payload line, letting two distinct + // configurations collide (review: Codex on #6184 — same class as the vault tuples below). + // Numbers and closed tokens need no quoting; `rolesSql` stays raw because it is the + // documented LAST field, with nothing after it to forge. + const quoted = (value: string) => JSON.stringify(value); const lines: Array = [ - `postgres_image=${inputs.postgresImage}`, + `postgres_image=${quoted(inputs.postgresImage)}`, `major_version=${inputs.majorVersion}`, `shadow_port=${inputs.shadowPort}`, - `jwt_secret=${inputs.jwtSecret}`, + `jwt_secret=${quoted(inputs.jwtSecret)}`, `jwt_expiry=${inputs.jwtExpiry}`, - `root_key=${inputs.rootKey}`, - `db_password=${inputs.dbPassword}`, + `root_key=${quoted(inputs.rootKey)}`, + `db_password=${quoted(inputs.dbPassword)}`, `db_settings=${legacyCanonicalJson(inputs.dbSettings)}`, `auto_expose_new_tables=${legacyTriStateToken(inputs.autoExposeNewTables)}`, // Not a per-run input — see the digest's own doc comment for what it covers and why. @@ -245,7 +255,7 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string const service = inputs.services[name]; lines.push( service.enabled - ? `service=${name} enabled=true image=${service.image}` + ? `service=${name} enabled=true image=${quoted(service.image)}` : `service=${name} enabled=false`, ); } @@ -254,19 +264,23 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string // the PG15+ gate the one-shot job itself is behind). lines.push( inputs.services.realtime.enabled && inputs.majorVersion >= 15 - ? `realtime_jwks=${inputs.jwks}` + ? `realtime_jwks=${quoted(inputs.jwks)}` : "realtime_jwks=excluded", ); // Storage's migration pin — same compound enabled+majorVersion gate as the JWKS line above, // because the consuming one-shot job (`legacyStartInitSchema15`) is behind the same gate. lines.push( inputs.services.storage.enabled && inputs.majorVersion >= 15 - ? `storage_target_migration=${inputs.storageTargetMigration}` + ? `storage_target_migration=${quoted(inputs.storageTargetMigration)}` : "storage_target_migration=excluded", ); - for (const secret of [...inputs.vault].sort((left, right) => - left.name < right.name ? -1 : left.name > right.name ? 1 : 0, - )) { + // ONLY resolved entries: `legacyUpsertVaultSecrets` (`legacy-vault.ts`) filters on + // `secret.resolved` before touching `vault.secrets`, so an unresolved entry never lands in + // the cluster and must not affect the key — hashing exactly what the upsert processes + // (review: Codex on #6184). + for (const secret of inputs.vault + .filter((secret) => secret.resolved) + .sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0))) { // JSON-encoded tuple, not `name=value`: both halves are unrestricted strings, so a bare // `=` join would let (`a=b`, `c`) and (`a`, `b=c`) collide — and a value containing a // newline could forge a whole extra payload line (review: Codex on #6184). @@ -558,6 +572,17 @@ const legacyExportShadowBaseline = ( // Acquire / release // --------------------------------------------------------------------------- +/** + * Per-invocation cache controls a CALLER passes (as opposed to the user-level + * {@link LEGACY_SHADOW_CACHE_ENV} env gate). `bypassCache` is `db schema declarative sync + * --no-cache`'s hook: that flag documents "force fresh shadow database setup", so a run carrying + * it must neither restore an existing snapshot nor publish a new one — exactly the uncached + * lifecycle, regardless of the env gate (review: Codex on #6184). + */ +export interface LegacyShadowCacheOpts { + readonly bypassCache?: boolean; +} + /** * What `Effect.acquireUseRelease`'s `acquire` hands the `use` phase: the container, whether its * cluster already carries the platform baseline, and the snapshot step to run once a fresh @@ -682,13 +707,17 @@ const legacyWarmShadow = ( export const legacyAcquireShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupInput, + opts: LegacyShadowCacheOpts = {}, ): Effect.Effect< LegacyShadowAcquiredHandle, LegacyShadowDbError | E, Output | LegacyDbConnection > => Effect.gen(function* () { - if (!legacyShadowCacheEnabled(process.env, input.setup.projectEnvValues)) { + if ( + opts.bypassCache === true || + !legacyShadowCacheEnabled(process.env, input.setup.projectEnvValues) + ) { return yield* legacyUncachedShadow(spawner, input); } @@ -765,7 +794,8 @@ export const legacyWithShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupInput, use: (handle: LegacyShadowAcquiredHandle) => Effect.Effect, + opts: LegacyShadowCacheOpts = {}, ): Effect.Effect => - Effect.acquireUseRelease(legacyAcquireShadowDatabase(spawner, input), use, (handle) => + Effect.acquireUseRelease(legacyAcquireShadowDatabase(spawner, input, opts), use, (handle) => legacyRemoveShadowDatabase(spawner, handle.containerId), ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index 2057395fed..8309a9f6c4 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -238,6 +238,39 @@ describe("legacyShadowCacheKey", () => { expect(legacyShadowCacheKey(withPinA)).toBe(legacyShadowCacheKey(withPinB)); }); + it("cannot collide scalar fields across line boundaries", () => { + const base = baseKeyInputs(); + // A newline embedded in one unrestricted scalar must not be able to forge the next + // payload line: rootKey `p\ndb_password="q"` + password `r` vs rootKey `p` + a password + // whose tail mimics the same text. + const left = legacyShadowCacheKey({ + ...base, + rootKey: 'p"\ndb_password="q', + dbPassword: "r", + }); + const right = legacyShadowCacheKey({ + ...base, + rootKey: "p", + dbPassword: 'q"\ndb_password="r', + }); + expect(left).not.toBe(right); + }); + + it("excludes unresolved vault secrets, which the upsert never processes", () => { + const base = baseKeyInputs(); + const withUnresolved = legacyShadowCacheKey({ + ...base, + vault: [...base.vault, { name: "pending", value: "", resolved: false }], + }); + expect(withUnresolved).toBe(legacyShadowCacheKey(base)); + // A RESOLVED empty value does land in the cluster, so it must re-key. + const withResolvedEmpty = legacyShadowCacheKey({ + ...base, + vault: [...base.vault, { name: "pending", value: "", resolved: true }], + }); + expect(withResolvedEmpty).not.toBe(legacyShadowCacheKey(base)); + }); + it("cannot collide vault name/value pairs across the tuple boundary", () => { const base = baseKeyInputs(); // `name=a=b, value=c` vs `name=a, value=b=c` — a bare `=`-joined encoding serializes both diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts index 1c34355954..119e3eabb7 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts @@ -26,6 +26,7 @@ import { legacyWaitForHealthyServices } from "./db-bootstrap/health-check.ts"; import { legacyWithShadowDatabase, type LegacyShadowAcquiredHandle, + type LegacyShadowCacheOpts, } from "./db-bootstrap/shadow-cache.ts"; import { legacySetupShadowDatabase, @@ -769,6 +770,7 @@ const exportViaShadowCatalog = ( shadowInput: LegacyShadowSetupInput, ) => Effect.Effect, persist: (snapshot: string) => Effect.Effect, + shadowCacheOpts: LegacyShadowCacheOpts = {}, ) => Effect.gen(function* () { const { spawner, localInputs } = built; @@ -785,16 +787,21 @@ const exportViaShadowCatalog = ( // `SUPABASE_SHADOW_CACHE` unset it IS that pair (identical Docker argv, identical labels), and // with it set a catalog cache miss restores a key-matching PGDATA snapshot into the fresh // shadow instead of paying the full cold provision — the same swap `db diff`/`db pull`'s own - // call sites make. - const written = yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => - Effect.gen(function* () { - const shadow = yield* provision(spawner, handle, shadowInput); - const snapshot = yield* legacyExportCatalogPgDelta(ctx, { - targetRef: shadow.sourceUrl, - role: "postgres", - }); - return yield* persist(snapshot); - }), + // call sites make. `shadowCacheOpts` carries `sync --no-cache`'s bypass — see + // `LegacyShadowCacheOpts`. + const written = yield* legacyWithShadowDatabase( + spawner, + shadowInput, + (handle) => + Effect.gen(function* () { + const shadow = yield* provision(spawner, handle, shadowInput); + const snapshot = yield* legacyExportCatalogPgDelta(ctx, { + targetRef: shadow.sourceUrl, + role: "postgres", + }); + return yield* persist(snapshot); + }), + shadowCacheOpts, ); return path.relative(ctx.cwd, written); }); @@ -988,6 +995,10 @@ export const legacyGetMigrationsCatalogRef = Effect.fnUntraced(function* ( timestamp, ); }), + // `--no-cache` promises "force fresh shadow database setup" (`declarative.shared.ts`), so it + // must ALSO bypass the shadow baseline snapshot, not just the catalog cache — otherwise a + // warm tar would skip the very setup the flag exists to force (review: Codex on #6184). + { bypassCache: params.noCache }, ); }); From 714a7bd8fa8ec60acf509d8193e2e6142b1ac48b Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 20:26:30 +0200 Subject: [PATCH 51/82] fix(cli): keep uncached shadow runs on one session, sweep abandoned partial snapshots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ninth review round on #6184 (Codex), both findings valid: - The split-session shape (baseline session closed before the snapshot, template + migrations on a second session) is now confined to the cache's own snapshotting cold provision via `LegacyShadowBaselineState.snapshotRequired`. Uncached runs (cache off, --no-cache, OrioleDB) and warm hits use exactly one session, matching Go's single-connection flow — otherwise the reconnect exposes migrations to role-level defaults roles.sql just installed (e.g. ALTER ROLE ... SET statement_timeout), and the opt-outs would not fully restore pre-cache behavior. - A SIGKILLed/crashed cold export leaves a ~90MB shadow-baseline-.tar..partial that nothing ever cleaned (later runs use their own pid; the tar retention sweep ignores .partial names). Every cold export now sweeps partials older than an hour first — mtime-gated so a concurrent writer's live temp file is never touched. Co-Authored-By: Claude Fable 5 --- .../shadow-cache.integration.test.ts | 29 ++++++++++ .../shared/db-bootstrap/shadow-cache.ts | 56 ++++++++++++++++++- .../db-bootstrap/shadow-cache.unit.test.ts | 18 ++++++ .../shared/db-bootstrap/shadow-database.ts | 44 +++++++++++---- .../db-bootstrap/shadow-database.unit.test.ts | 13 +++-- 5 files changed, 143 insertions(+), 17 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 59c7f519ee..2ff562c18e 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -547,6 +547,35 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("a cold export sweeps abandoned partial temp files but never fresh ones", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const tempDir = pgDeltaTempDir(path); + yield* fs.makeDirectory(tempDir, { recursive: true }); + // A SIGKILLed export's leftover (writer long gone — hour-plus-old mtime) and a + // concurrent writer's live temp file (fresh mtime). + const abandoned = path.join(tempDir, "shadow-baseline-0123456789abcdef.tar.99999.partial"); + const live = path.join(tempDir, "shadow-baseline-fedcba9876543210.tar.88888.partial"); + yield* fs.writeFileString(abandoned, "stale"); + yield* fs.writeFileString(live, "in-flight"); + const twoHoursAgo = new Date(Date.now() - 2 * 60 * 60 * 1000); + yield* fs.utimes(abandoned, twoHoursAgo, twoHoursAgo); + + yield* coldRun(docker, shadowInput(fs, path)); + + expect(yield* fs.exists(abandoned)).toBe(false); + expect(yield* fs.exists(live)).toBe(true); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("publishing a new key's tar sweeps every other key's", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index eb3fdfdcd4..b36320f658 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -25,7 +25,7 @@ import { createHash } from "node:crypto"; import type { ProjectConfig } from "@supabase/config"; -import { Effect, Option, type FileSystem } from "effect"; +import { Clock, Effect, Option, type FileSystem } from "effect"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import { Output } from "../../../shared/output/output.service.ts"; @@ -413,6 +413,56 @@ const legacyForgetShadowBaselineTar = ( filePath: string, ): Effect.Effect => fs.remove(filePath).pipe(Effect.orElseSucceed(() => undefined)); +/** + * Whether `fileName` is one of {@link legacyExportPgDataTar}'s in-flight temp files + * (`shadow-baseline-.tar..partial`). Pure name check only — whether it is ABANDONED + * (crashed/SIGKILLed writer, whose `Effect.onError` cleanup never ran) is an mtime question the + * sweep answers separately, so a concurrent writer's live temp file is never a candidate by name + * alone. + */ +export function legacyIsShadowBaselinePartial(fileName: string): boolean { + return /^shadow-baseline-[0-9a-f]{16}\.tar\.\d+\.partial$/u.test(fileName); +} + +/** + * A partial older than this is abandoned: the export itself streams ~90MB in seconds, so an + * hour-old temp file's writer is long gone. Deliberately enormous relative to a real export so a + * slow disk can never get a LIVE temp file swept out from under its writer. + */ +const LEGACY_SHADOW_PARTIAL_ABANDON_MS = 60 * 60 * 1000; + +/** + * Removes abandoned `.partial` temp files (see {@link legacyIsShadowBaselinePartial}) — the one + * artifact a SIGKILLed/crashed cold export leaves behind that nothing else ever cleans: later + * runs use their own pid in the temp name, and the tar retention sweep deliberately ignores + * `.partial` names (review: Codex on #6184). Runs before every cold export (so orphans cannot + * accumulate across repeatedly killed provisions) — best-effort throughout, like every other + * sweep here. + */ +const legacySweepAbandonedShadowBaselinePartials = ( + input: LegacyShadowSetupInput, +): Effect.Effect => + Effect.gen(function* () { + const tempDir = legacyPgDeltaTempPath(input.path, input.workdir); + const entries = yield* input.fs + .readDirectory(tempDir) + .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); + const now = yield* Clock.currentTimeMillis; + yield* Effect.forEach( + entries.filter(legacyIsShadowBaselinePartial), + (entry) => + Effect.gen(function* () { + const filePath = input.path.join(tempDir, entry); + const info = yield* input.fs.stat(filePath); + const mtime = Option.getOrUndefined(info.mtime); + if (mtime !== undefined && now - mtime.getTime() > LEGACY_SHADOW_PARTIAL_ABANDON_MS) { + yield* legacyForgetShadowBaselineTar(input.fs, filePath); + } + }).pipe(Effect.orElseSucceed(() => undefined)), + { discard: true }, + ); + }); + /** * Applies the "current key only" retention rule: every `shadow-baseline-*.tar` in the temp * directory whose key differs from `key` is removed. Best-effort throughout — a snapshot that @@ -517,6 +567,7 @@ const legacyWriteShadowBaselineTar = ( legacyShadowCacheUnavailable(`failed to create ${tempDir}: ${cause.message}`), ), ); + yield* legacySweepAbandonedShadowBaselinePartials(input); yield* legacyExportPgDataTar(spawner, containerId, input.fs, tarPath).pipe( Effect.mapError((cause: LegacyPgDataSnapshotUnavailable) => legacyShadowCacheUnavailable(cause.reason), @@ -602,6 +653,7 @@ const legacyUncachedShadow = ( Effect.map(({ containerId }) => ({ containerId, baselinePresent: false, + snapshotRequired: false, snapshotBaseline: Effect.void, })), ); @@ -625,6 +677,7 @@ const legacyColdCachedShadow = ( Effect.map(({ containerId }) => ({ containerId, baselinePresent: false, + snapshotRequired: true, snapshotBaseline: legacyExportShadowBaseline(spawner, input, key, tarPath, containerId), })), ); @@ -677,6 +730,7 @@ const legacyWarmShadow = ( return { containerId, baselinePresent: true, + snapshotRequired: false, snapshotBaseline: Effect.void, } satisfies LegacyShadowAcquiredHandle; }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index 8309a9f6c4..4108a0dcb0 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -2,6 +2,7 @@ import { describe, expect, it } from "@effect/vitest"; import { Option } from "effect"; import { + legacyIsShadowBaselinePartial, legacyIsStaleShadowBaselineTar, legacyShadowBaselineTarFileName, legacyShadowCacheEnabled, @@ -313,6 +314,23 @@ describe("shadow baseline tar retention", () => { expect(legacyIsStaleShadowBaselineTar("shadow-baseline-fedcba9876543210.tar", key)).toBe(true); }); + it("recognizes only this module's own partial temp files as abandoned-sweep candidates", () => { + expect(legacyIsShadowBaselinePartial(`shadow-baseline-${key}.tar.4242.partial`)).toBe(true); + for (const other of [ + // The published artifact and other keys' artifacts are the tar sweep's business, not this + // predicate's. + legacyShadowBaselineTarFileName(key), + "shadow-baseline-fedcba9876543210.tar", + // Anything not exactly `<16-hex>.tar..partial` is left alone. + "shadow-baseline.tar.4242.partial", + `shadow-baseline-${key}.tar.partial`, + `shadow-baseline-${key}.tar.4242.partial.bak`, + "catalog-local-migrations-abc-123.json", + ]) { + expect(legacyIsShadowBaselinePartial(other), other).toBe(false); + } + }); + it("never sweeps a file that is not one of this module's own snapshots", () => { // `supabase/.temp/pgdelta/` is shared with the pg-delta catalog cache and its debug bundles — // the retention rule must be blind to everything but its own prefix AND suffix, or a `db diff` diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index eb6dedfb83..3a275e7d37 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -773,15 +773,27 @@ export interface LegacyShadowBaselineState { * double-applied baseline at worst. */ readonly baselinePresent: boolean; + /** + * `true` ONLY for a cache-enabled COLD provision — the one state whose + * {@link snapshotBaseline} really stops the container. This is what + * {@link legacyMigrateShadowDatabase} keys its session structure on: the baseline session must + * be closed before a real snapshot (a disk-level export severs any live backend), but when no + * snapshot will run, splitting sessions would be a gratuitous behavior change — a reconnect + * picks up role-level defaults `roles.sql` may have just installed (e.g. `ALTER ROLE postgres + * SET statement_timeout`), which Go's single-connection flow never exposed to migrations + * (review: Codex on #6184). So uncached and warm runs keep exactly one session. + */ + readonly snapshotRequired: boolean; /** * Runs immediately after a FRESHLY provisioned baseline and strictly before the template * database/user migrations — the only point at which `postgres` holds the pristine baseline and * nothing else. * * Takes NO session, and {@link legacyMigrateShadowDatabase} guarantees no session is open - * against the shadow while it runs: the snapshot is a disk-level PGDATA export that has to stop - * the container, which would sever any live backend. `Effect.Effect`: a cache - * that cannot snapshot must degrade silently, never fail the run. + * against the shadow while it runs when {@link snapshotRequired} is set: the snapshot is a + * disk-level PGDATA export that has to stop the container, which would sever any live backend. + * `Effect.Effect`: a cache that cannot snapshot must degrade silently, never + * fail the run. */ readonly snapshotBaseline: Effect.Effect; } @@ -789,6 +801,7 @@ export interface LegacyShadowBaselineState { /** The baseline state every uncached caller passes: provision it, snapshot nothing. */ export const LEGACY_SHADOW_BASELINE_COLD: LegacyShadowBaselineState = { baselinePresent: false, + snapshotRequired: false, snapshotBaseline: Effect.void, }; @@ -811,12 +824,14 @@ export const LEGACY_SHADOW_BASELINE_COLD: LegacyShadowBaselineState = { * the template database and the user migrations; a COLD cache-enabled provision passes the same * cold sequence plus a `snapshotBaseline` step between the baseline and the template database. * - * The one structural divergence from Go: on the cold branch the baseline runs in its own scope, so - * its session is CLOSED before {@link LegacyShadowBaselineState.snapshotBaseline}, and the - * template database + migrations then run on a second session. Go uses a single connection for all - * of it, but the disk-level PGDATA snapshot stops the container, which severs any live backend — - * and a session's lifetime ending with the work it was opened for is the right shape anyway. A - * warm hit still uses exactly one session, and the SQL either path issues is unchanged. + * The one structural divergence from Go is confined to the SNAPSHOTTING cold branch + * (`baseline.snapshotRequired`): there the baseline runs in its own scope, its session is CLOSED + * before {@link LegacyShadowBaselineState.snapshotBaseline} (the disk-level PGDATA snapshot stops + * the container, which severs any live backend), and the template database + migrations run on a + * second session. Every OTHER state — uncached (cache off / `--no-cache` / OrioleDB) and warm — + * uses exactly one session, matching Go's single connection: see + * {@link LegacyShadowBaselineState.snapshotRequired} for why the split must not leak into the + * uncached path. The SQL every path issues is unchanged. */ const migrateShadowDatabase = ( spawner: Spawner, @@ -841,7 +856,7 @@ const migrateShadowDatabase = ( ), ); - if (!baseline.baselinePresent) { + if (!baseline.baselinePresent && baseline.snapshotRequired) { // Own scope: the baseline session must be closed before `snapshotBaseline` — see this // function's own doc comment. yield* Effect.scoped( @@ -858,6 +873,15 @@ const migrateShadowDatabase = ( yield* baseline.snapshotBaseline; } const session = yield* legacyConnectShadowDatabase(input.connConfig); + if (!baseline.baselinePresent && !baseline.snapshotRequired) { + // Go's single-connection flow, verbatim: baseline + template + migrations all on this + // one session — see this function's own doc comment. + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput(input, session, resolved), + ); + } yield* legacyCreateShadowTemplateDatabase(session); yield* legacyApplyMigrations( session, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index 81249f6578..a15f640555 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -783,12 +783,13 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { }, setup: baseShadowSetup(), }); - // Two connects, not one: on the cold branch the platform baseline runs in its own scope so - // its session is closed before `snapshotBaseline` (which stops the container to take a - // disk-level PGDATA snapshot), and the template database + migrations then run on a second - // session. The ordering under test is unaffected — the migration listing still precedes - // every connect. - expect(events).toEqual(["list", "connect", "connect"]); + // ONE connect, matching Go's single-connection flow: the default baseline state + // (`LEGACY_SHADOW_BASELINE_COLD`) requires no snapshot, so baseline + template + + // migrations all share one session — the split-session shape is reserved for the + // cache's own snapshotting cold provision (`snapshotRequired: true`), whose disk-level + // export must close the session before stopping the container. The ordering under test + // is unaffected — the migration listing still precedes the connect. + expect(events).toEqual(["list", "connect"]); }).pipe( Effect.provide( Layer.mergeAll( From d9f7467d24185e4bdb20b5dfcba10f71a0bacffb Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 20:34:12 +0200 Subject: [PATCH 52/82] fix(cli): hash the effective root key into the shadow cache key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tenth review round on #6184 (depthfirst): the shadow container spec falls back to the embedded LEGACY_POSTGRES_DEFAULT_ROOT_KEY when db.root_key is unset, but the cache key hashed "" — so a rotation of that security-sensitive default between CLI releases would not invalidate existing tars. The key now hashes the same effective value the container receives. Co-Authored-By: Claude Fable 5 --- apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index b36320f658..d215993c55 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -38,6 +38,7 @@ import type { LegacyPgConnInput } from "../legacy-db-connection.service.ts"; import { legacyGetRegistryImageUrl } from "../legacy-docker-registry.ts"; import { legacyPgDeltaTempPath } from "../legacy-pgdelta.paths.ts"; import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; +import { LEGACY_POSTGRES_DEFAULT_ROOT_KEY } from "../legacy-local-config-values.ts"; import { LEGACY_START_REVOKE_API_PRIVILEGES_SQL } from "./db-setup.ts"; import { LEGACY_START_DB_GLOBALS_SQL } from "./templates/db-globals.sql.ts"; import { LEGACY_START_DB_INITIAL_SCHEMA_13_SQL } from "./templates/db-initial-schema-13.sql.ts"; @@ -353,7 +354,11 @@ const legacyResolveShadowCacheKeyInputs = ( shadowPort: input.shadowPort, jwtSecret: input.jwtSecret, jwtExpiry: input.jwtExpiry, - rootKey: input.rootKey ?? "", + // The EFFECTIVE value, not the raw input: `legacyBuildShadowPostgresContainerSpec` + // (`postgres.service.ts`) falls back to the embedded default when unset, so hashing `""` + // would fail to re-key if that security-sensitive default ever rotates between CLI + // releases (review: depthfirst on #6184). + rootKey: input.rootKey ?? LEGACY_POSTGRES_DEFAULT_ROOT_KEY, dbPassword: input.password, dbSettings: input.db.settings, storageTargetMigration: input.setup.storageTargetMigration, From a256f4f93d7c5c325ba09ffe33dfce5737b7903b Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 20:44:35 +0200 Subject: [PATCH 53/82] fix(cli): only discard the shadow snapshot when its contents are implicated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eleventh review round on #6184 (Codex): two of three findings taken. - The warm-anomaly fallback deleted the tar on EVERY failure, including a Docker daemon outage or port collision during `docker create` that says nothing about the tar — throwing away a valid baseline that costs ~15s to rebuild. `LegacyShadowCacheUnavailable.tarSuspect` now scopes deletion to the one failure that implicates the contents (a restored cluster that started but never accepted connections); an extraction/infra failure keeps the tar, and a genuinely corrupt one is atomically republished over by the cold fallback's own export in the same run. - SIDE_EFFECTS for db diff/db pull/declarative sync now document the `.tar..partial` in-flight temp file, its rename-on-success / remove-on-failure lifecycle, crash retention, and the hour-gated sweep. Declined (rationale on the thread): hashing the pull-time ECR/GHCR/Docker-Hub fallback winner into the cache key. Co-Authored-By: Claude Fable 5 --- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 1 + .../legacy/commands/db/pull/SIDE_EFFECTS.md | 1 + .../schema/declarative/sync/SIDE_EFFECTS.md | 1 + .../shadow-cache.integration.test.ts | 65 ++++++++++--------- .../shared/db-bootstrap/shadow-cache.ts | 32 +++++++-- 5 files changed, 65 insertions(+), 35 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 09f9104604..9a3eee219d 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -43,6 +43,7 @@ it, and JSON `null` disables formatting without disabling safe compaction. | `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | | `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision only (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`, never a warm hit) — the shadow's PGDATA snapshot, ~90MB, current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | | `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | | `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 409e305272..230fd5f617 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -60,6 +60,7 @@ disables formatting without disabling safe compaction. | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | | `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | | `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision, migration-style pull only (never `--declarative`'s bare shadow, the delegated `--experimental` path, or a warm hit) — the shadow's PGDATA snapshot, ~90MB, one file for the current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | | `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | | `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index aed88cd3f4..0cce764cc1 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -36,6 +36,7 @@ disabling safe compaction. | `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | | `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | ## Subprocesses / Containers diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 2ff562c18e..9ba9dca352 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -658,36 +658,41 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); - it.live("a failed warm restore deletes the suspect tar and falls back to a cold run", () => { - const docker = fakeDockerDaemon({ failCopyIn: true }); - const cluster = fakeCluster(); - const out = mockOutput(); - return withShadowCacheEnv( - "1", - Effect.gen(function* () { - const fs = yield* FileSystem.FileSystem; - const path = yield* Path.Path; - const input = shadowInput(fs, path); - // A cold run first, so a snapshot for this exact key exists to be restored. - const cold = yield* coldRun(docker, input); - expect(yield* soleTarName(fs, path)).toHaveLength(1); - - const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); - expect(out.stderrText).toContain("cached shadow baseline unusable"); - // Falls all the way back to a cold provision — a fresh container with no baseline. - expect(fallback.baselinePresent).toBe(false); - expect(fallback.containerId).not.toBe(cold.containerId); - // The container whose restore failed is removed, not orphaned: with the cold run's own - // container already released by `coldRun`, only the fallback's remains. - expect(docker.ids()).toEqual([fallback.containerId]); - // The tar is the suspect and is deleted, so later runs do not retry it forever... - expect(yield* soleTarName(fs, path)).toEqual([]); - // ...and the cold fallback republishes one through its own snapshot step. - yield* fallback.snapshotBaseline; - expect(yield* soleTarName(fs, path)).toHaveLength(1); - }), - ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); - }); + it.live( + "a failed warm restore falls back cold, keeping the tar until its export replaces it", + () => { + const docker = fakeDockerDaemon({ failCopyIn: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + // A cold run first, so a snapshot for this exact key exists to be restored. + const cold = yield* coldRun(docker, input); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(out.stderrText).toContain("cached shadow baseline unusable"); + // Falls all the way back to a cold provision — a fresh container with no baseline. + expect(fallback.baselinePresent).toBe(false); + expect(fallback.containerId).not.toBe(cold.containerId); + // The container whose restore failed is removed, not orphaned: with the cold run's own + // container already released by `coldRun`, only the fallback's remains. + expect(docker.ids()).toEqual([fallback.containerId]); + // An extraction failure does NOT implicate the tar's contents (it could just as well be + // a daemon hiccup), so the tar survives the fallback decision... + expect(yield* soleTarName(fs, path)).toHaveLength(1); + // ...and the cold fallback's own export atomically republishes over it, so a genuinely + // corrupt tar still self-heals within this one run. + yield* fallback.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toHaveLength(1); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }, + ); it.live("a restored shadow that never becomes ready is removed before the cold retry", () => { const docker = fakeDockerDaemon(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index d215993c55..80610d5dca 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -17,7 +17,8 @@ * shadows are all removed with `docker rm -f -v` on release; the cold path alone drops `--rm` (see * {@link LegacyCreateShadowDatabaseInput.autoRemove} for the consequence); the tar is published by * an atomic rename, so concurrent writers need no lock file; a cache anomaly never fails the run — - * a warm-path anomaly deletes the tar and cold-provisions, a cold export failure only warns and + * a warm-path anomaly cold-provisions (deleting the tar only when its contents are implicated — + * see `LegacyShadowCacheUnavailable.tarSuspect`), a cold export failure only warns and * leaves the run uncached; retention keeps the current key's tar only, sweeping every other one on * publish. `SUPABASE_SHADOW_CACHE` is ON by default; `false`/`0` opts out. */ @@ -75,9 +76,23 @@ export const LEGACY_SHADOW_CACHE_ENV = "SUPABASE_SHADOW_CACHE"; */ interface LegacyShadowCacheUnavailable { readonly reason: string; + /** + * `true` only when the failure implicates the TAR'S CONTENTS — today, exactly one producer: a + * restored cluster that started but never accepted connections ({@link legacyWarmShadow}'s + * readiness wait). Everything else (a `docker create`/`cp`/`start` failure — daemon outage, + * port collision, or even a corrupt archive's failed extraction) leaves the tar in place: an + * infra failure says nothing about the tar, and a genuinely corrupt one is atomically + * REPLACED by the cold fallback's own export in the same run, so deleting up front would only + * throw away a valid ~15s-to-rebuild baseline on transient Docker failures (review: Codex on + * #6184). + */ + readonly tarSuspect?: boolean; } -const legacyShadowCacheUnavailable = (reason: string): LegacyShadowCacheUnavailable => ({ reason }); +const legacyShadowCacheUnavailable = ( + reason: string, + opts: { readonly tarSuspect?: boolean } = {}, +): LegacyShadowCacheUnavailable => ({ reason, ...opts }); /** * Whether the shadow baseline cache is enabled for this invocation. @@ -728,6 +743,9 @@ const legacyWarmShadow = ( ), ); yield* legacyAwaitShadowReady(spawner, input, containerId, "restored shadow").pipe( + // The ONE failure that implicates the tar's contents: the restored cluster started but + // never accepted connections — see {@link LegacyShadowCacheUnavailable.tarSuspect}. + Effect.mapError((cause) => legacyShadowCacheUnavailable(cause.reason, { tarSuspect: true })), Effect.tapError(() => legacyRemoveShadowDatabase(spawner, containerId)), Effect.onInterrupt(() => legacyRemoveShadowDatabase(spawner, containerId)), Effect.interruptible, @@ -804,9 +822,13 @@ export const legacyAcquireShadowDatabase = ( `Warning: cached shadow baseline unusable (${cause.reason}); recreating.\n`, "stderr", ); - // The tar is the suspect: a restore that produced an unstartable cluster will produce - // one again on every later run, so it is deleted rather than retried forever. - yield* legacyForgetShadowBaselineTar(input.fs, tarPath); + // Delete ONLY when the failure implicates the tar's contents (a restored cluster that + // came up broken) — see {@link LegacyShadowCacheUnavailable.tarSuspect} for why an + // infra or extraction failure leaves it in place (the cold fallback's own export + // republishes over a genuinely bad tar anyway). + if (cause.tarSuspect === true) { + yield* legacyForgetShadowBaselineTar(input.fs, tarPath); + } return yield* legacyColdCachedShadow(spawner, input, key, tarPath); }), ), From bcdf030541398261fd8db3f32ffd11517a6a5e45 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 20:53:52 +0200 Subject: [PATCH 54/82] fix(cli): fold the vault upsert SQL into the baseline digest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Twelfth review round on #6184 (Codex): the embedded-SQL digest ended at the privilege-revocation statement and omitted the vault upsert SQL (legacy-vault.ts) that legacySetupDatabase executes into the baseline right after it — a CLI release changing those statements would not have re-keyed existing tars. The three statements are now exported (Legacy_ prefixed per the naming rule) and appended to the digest. Co-Authored-By: Claude Fable 5 --- .../src/legacy/shared/db-bootstrap/shadow-cache.ts | 12 +++++++++++- apps/cli/src/legacy/shared/legacy-vault.ts | 14 ++++++++------ 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 80610d5dca..de177ec640 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -47,7 +47,12 @@ import { LEGACY_START_DB_INITIAL_SCHEMA_14_SQL } from "./templates/db-initial-sc import { LEGACY_START_DB_SCHEMA_SQL } from "./templates/db-schema.sql.ts"; import { LEGACY_START_DB_SUPABASE_SQL } from "./templates/db-supabase.sql.ts"; import { LEGACY_START_DB_WEBHOOK_SQL } from "./templates/db-webhook.sql.ts"; -import type { LegacyVaultSecret } from "../legacy-vault.ts"; +import { + LEGACY_CREATE_VAULT_KV, + LEGACY_READ_VAULT_KV, + LEGACY_UPDATE_VAULT_KV, + type LegacyVaultSecret, +} from "../legacy-vault.ts"; import { legacyWaitForShadowReady } from "./health-check.ts"; import { legacyExportPgDataTar, legacyPgDataRestoreArchive } from "./pgdata-snapshot.ts"; import type { LegacyPgDataSnapshotUnavailable } from "./pgdata-snapshot.ts"; @@ -221,6 +226,11 @@ const LEGACY_SHADOW_BASELINE_SQL_DIGEST = createHash("sha256") LEGACY_START_DB_INITIAL_SCHEMA_13_SQL, LEGACY_START_DB_INITIAL_SCHEMA_14_SQL, LEGACY_START_REVOKE_API_PRIVILEGES_SQL, + // The vault upsert's own SQL (`legacyUpsertVaultSecrets`, `legacy-vault.ts`) runs into the + // baseline right after the privilege pass — same digest rationale as every line above. + LEGACY_READ_VAULT_KV, + LEGACY_UPDATE_VAULT_KV, + LEGACY_CREATE_VAULT_KV, ].join("\n--8<--\n"), "utf8", ) diff --git a/apps/cli/src/legacy/shared/legacy-vault.ts b/apps/cli/src/legacy/shared/legacy-vault.ts index 6b415baf80..6b596ab62d 100644 --- a/apps/cli/src/legacy/shared/legacy-vault.ts +++ b/apps/cli/src/legacy/shared/legacy-vault.ts @@ -24,9 +24,11 @@ export interface LegacyVaultSecret { readonly resolved: boolean; } -const READ_VAULT_KV = "SELECT id, name FROM vault.secrets WHERE name = ANY($1)"; -const UPDATE_VAULT_KV = "SELECT vault.update_secret($1, $2)"; -const CREATE_VAULT_KV = "SELECT vault.create_secret($1, $2)"; +// Exported for the shadow baseline cache's embedded-SQL digest (`shadow-cache.ts`), which must +// re-key whenever the SQL this module bakes into a baseline changes across CLI releases. +export const LEGACY_READ_VAULT_KV = "SELECT id, name FROM vault.secrets WHERE name = ANY($1)"; +export const LEGACY_UPDATE_VAULT_KV = "SELECT vault.update_secret($1, $2)"; +export const LEGACY_CREATE_VAULT_KV = "SELECT vault.create_secret($1, $2)"; /** * Upserts `[db.vault]` secrets into `vault.secrets`. Port of Go's @@ -46,7 +48,7 @@ export const legacyUpsertVaultSecrets = ( yield* output.raw("Updating vault secrets...\n", "stderr"); const existing = yield* session - .query(READ_VAULT_KV, [resolved.map((secret) => secret.name)]) + .query(LEGACY_READ_VAULT_KV, [resolved.map((secret) => secret.name)]) .pipe( Effect.mapError( (cause) => @@ -63,9 +65,9 @@ export const legacyUpsertVaultSecrets = ( for (const secret of resolved) { const id = existingByName.get(secret.name); if (id !== undefined) { - yield* session.query(UPDATE_VAULT_KV, [id, secret.value]); + yield* session.query(LEGACY_UPDATE_VAULT_KV, [id, secret.value]); } else { - yield* session.query(CREATE_VAULT_KV, [secret.value, secret.name]); + yield* session.query(LEGACY_CREATE_VAULT_KV, [secret.value, secret.name]); } } yield* session.exec("COMMIT"); From 1e1b7fa64c28d06648900c7949a0bb373e648a7c Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 22:54:54 +0200 Subject: [PATCH 55/82] docs(cli): record the warm-aware catalog-shadow follow-up from the CLI-1970 merge Co-Authored-By: Claude Fable 5 --- docs/roadmap/pg-delta-next-follow-ups.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md index 96944d8d17..9def9ddc86 100644 --- a/docs/roadmap/pg-delta-next-follow-ups.md +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -29,3 +29,18 @@ usually alongside image bumps), which is why it did not block #6184. Two candida Either way, add a unit-test mutation case alongside the existing ones in `shadow-cache.unit.test.ts`. + +## Make the baseline/declarative catalog shadows warm-aware (PR #6184 × CLI-1970 merge) + +CLI-1970 (#6162) made `legacyExportBaselineCatalogRef`/`legacyExportDeclarativeCatalogRef` +(`legacy-pgdelta.cache.ts`) native, so `db schema declarative sync`/`generate` now provision a +second shadow in-process for the declarative/baseline catalog. Those callers pass an unconditional +`{ bypassCache: true }` to `exportViaShadowCatalog`: their provisions run the platform baseline via +`legacySetupShadowDatabase`, which is not baseline-state-aware, so a warm PGDATA hit would +double-apply the baseline. + +The snapshot seam is a natural fit — the tar IS exactly the post-baseline state these provisions +build (baseline only, no migrations), so a warm hit would need NO further setup at all, saving the +full ~15s per declarative-catalog miss. Requires threading `LegacyShadowBaselineState` through +`legacySetupShadowDatabase` (skip when `baselinePresent`) and swapping their +`legacyWaitForHealthyServices` docker-health wait for `legacyWaitForShadowReady`. From ebe339a3bbe1c538d84efb364c12b0cb7108d2be Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 23:05:20 +0200 Subject: [PATCH 56/82] docs(cli): list the partial-snapshot sweep under Files Read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thirteenth review round on #6184 (Codex): the abandoned-partial sweep enumerates, stats, and conditionally removes .tar..partial files during a later cold export — a filesystem read the Files Read tables did not list. Doc-only, all three commands. Co-Authored-By: Claude Fable 5 --- apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md | 1 + apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md | 1 + .../legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md | 1 + 3 files changed, 3 insertions(+) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 9a3eee219d..4108b38f24 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -27,6 +27,7 @@ it, and JSON `null` disables formatting without disabling safe compaction. | `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | | `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | | `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit — the snapshot is streamed into the fresh shadow container before it starts | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | | `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | | `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | | `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 230fd5f617..9473d904ac 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -43,6 +43,7 @@ disables formatting without disabling safe compaction. | `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | | `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | | `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — snapshot streamed into the fresh shadow container before it starts | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | | `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | | `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | | `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 0cce764cc1..bc6ca3f45a 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -26,6 +26,7 @@ disabling safe compaction. | `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | | `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | | `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit on a migrations-catalog miss — snapshot streamed into the fresh shadow container before it starts | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | ## Files Written From 607107bad20549f4cd5b3cc89715c6d6e0f12d8e Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 23:14:59 +0200 Subject: [PATCH 57/82] fix(cli): exclusive-create the snapshot temp file so its mode can never be inherited MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fourteenth review round on #6184 (depthfirst): flag "w" truncates a pre-created temp file while preserving its permissive mode (mode only governs creation), which the rename would carry to the published tar. The export now best-effort removes any pre-existing file at the temp path and opens with "wx" (O_EXCL) — a recreate in the race window degrades to an uncached run, never to a readable tar. Integration test asserts the published tar is 0600 even with a pre-created 0666 partial. Co-Authored-By: Claude Fable 5 --- .../shared/db-bootstrap/pgdata-snapshot.ts | 11 ++++++- .../shadow-cache.integration.test.ts | 33 +++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts index 4301f0f756..a709f60d0a 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts @@ -89,6 +89,10 @@ export const legacyExportPgDataTar = ( ): Effect.Effect => { const tempPath = `${tarPath}.${process.pid}.partial`; return Effect.gen(function* () { + // Clear any pre-existing file at the temp path (a crashed same-pid predecessor, or an + // adversarially pre-created one on a shared host) so the exclusive-create below starts from + // a genuinely fresh inode — see the sink's own comment. + yield* fs.remove(tempPath).pipe(Effect.orElseSucceed(() => undefined)); yield* Effect.scoped( Effect.gen(function* () { const child = yield* spawnContainerCli( @@ -110,7 +114,12 @@ export const legacyExportPgDataTar = ( // `0o600`: the archive is a full PGDATA — vault secret values, the JWT secret, and // role password hashes are all in its pages — so it must not be group/world-readable // on a shared host. `rename` preserves the mode, so the published tar inherits it. - Stream.run(child.stdout, fs.sink(tempPath, { flag: "w", mode: 0o600 })), + // `wx` (O_EXCL), not `w`: a plain truncating open would inherit an attacker- + // PRE-CREATED file's permissive mode instead of applying `mode` (which only governs + // creation). With the best-effort remove above, `wx` only ever fails if someone + // recreated the path in the race window — and that failure degrades to an uncached + // run, never to a world-readable tar (review: depthfirst on #6184). + Stream.run(child.stdout, fs.sink(tempPath, { flag: "wx", mode: 0o600 })), legacyCollectText(child.stderr), ], { concurrency: "unbounded" }, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 9ba9dca352..24dd228da2 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -547,6 +547,39 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("a pre-created permissive temp file cannot leak into the published tar's mode", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + const tempDir = pgDeltaTempDir(path); + yield* fs.makeDirectory(tempDir, { recursive: true }); + // An adversarially (or crash-) pre-created temp file at THIS process's own temp path, + // world-readable. The export must not inherit its mode: the pre-remove + `wx` + // exclusive-create guarantees a fresh 0600 inode. + yield* coldRun(docker, input); // publish once to learn the tar name + const [tarName = ""] = yield* soleTarName(fs, path); + const tarPath = path.join(tempDir, tarName); + const tempPath = `${tarPath}.${process.pid}.partial`; + yield* fs.remove(tarPath); // force the next run cold + yield* fs.writeFileString(tempPath, "poisoned"); + yield* fs.chmod(tempPath, 0o666); + + yield* coldRun(docker, input); + + const info = yield* fs.stat(tarPath); + // 0o600 exactly — not the pre-created file's 0o666. + expect((Number(info.mode) & 0o777).toString(8)).toBe("600"); + expect(yield* fs.readFileString(tarPath)).toBe(FAKE_PGDATA_TAR); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("a cold export sweeps abandoned partial temp files but never fresh ones", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); From 26147adcfdcb3efa3b61792e130037df233d54e0 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 23:19:07 +0200 Subject: [PATCH 58/82] fix(cli): fail the run when the shadow cannot come back after the baseline snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fifteenth review round on #6184 (Codex): the cold export's broad catch downgraded restart/readiness failures to a "not cached" warning and reported snapshotBaseline as successful over a dead container — the caller's next connect then dialed the shadow port blind, and a different Postgres claiming that port (matching default credentials are common locally) could have received the template + user migrations. The step now splits the two failure classes: stop/export failures still warn and leave the run uncached; the revive (docker start + readiness) propagates as LegacyShadowDbError. The revive runs even when the stop failed (start on a running container is a no-op success). snapshotBaseline's error channel widens from never to LegacyShadowDbError; every caller's channel already carried it. Co-Authored-By: Claude Fable 5 --- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 3 +- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 3 +- .../schema/declarative/sync/SIDE_EFFECTS.md | 9 ++- .../shadow-cache.integration.test.ts | 53 ++++++++++++++- .../shared/db-bootstrap/shadow-cache.ts | 68 ++++++++++++------- .../shared/db-bootstrap/shadow-database.ts | 11 ++- 6 files changed, 114 insertions(+), 33 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 4108b38f24..6478b7a027 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -222,7 +222,8 @@ Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA s keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, -a cold export failure only warns and leaves the run uncached. See `shared/db-bootstrap/ +a cold export failure only warns and leaves the run uncached (one exception: a shadow that +fails to come back up after the snapshot fails the run rather than reporting a false success). See `shared/db-bootstrap/ shadow-cache.ts`'s doc comment for the mechanics. `--use-pgadmin` is NOT cached — its shadow keeps the plain create/remove lifecycle. diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 9473d904ac..413feb575d 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -87,7 +87,8 @@ Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA s keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, -a cold export failure only warns and leaves the run uncached. See `shared/db-bootstrap/ +a cold export failure only warns and leaves the run uncached (one exception: a shadow that +fails to come back up after the snapshot fails the run rather than reporting a false success). See `shared/db-bootstrap/ shadow-cache.ts`'s doc comment for the mechanics. Each pooler-retry attempt acquires/releases its own shadow (a warm hit restores the same tar each time); `--declarative`'s bare shadow runs no baseline, so it is never cached. diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index bc6ca3f45a..1912632ad8 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -140,5 +140,10 @@ cache); a warm hit skips the platform baseline and therefore the every input baked into the cluster; retention keeps the current key's tar only. Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a -cold export failure only warns and leaves the run uncached. See `shared/db-bootstrap/ -shadow-cache.ts`'s doc comment for the mechanics. The declarative-catalog shadow is NOT cached. +cold export failure only warns and leaves the run uncached (one exception: a shadow that fails +to come back up after the snapshot fails the run rather than reporting a false success). See `shared/db-bootstrap/ +shadow-cache.ts`'s doc comment for the mechanics. The DECLARATIVE-catalog shadow (and `generate`'s +baseline-catalog shadow) is NOT cached: its provision runs the platform baseline via +`legacySetupShadowDatabase`, which is not baseline-state-aware, so those callers pass an +unconditional bypass — making them warm-aware is a recorded follow-up +(`docs/roadmap/pg-delta-next-follow-ups.md`). diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 24dd228da2..ecc6d68abe 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -14,7 +14,18 @@ import type { ProjectConfig } from "@supabase/config"; import { ProjectConfigSchema } from "@supabase/config"; import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; -import { Effect, FileSystem, Layer, Option, Path, Predicate, Schema, Sink, Stream } from "effect"; +import { + Effect, + Exit, + FileSystem, + Layer, + Option, + Path, + Predicate, + Schema, + Sink, + Stream, +} from "effect"; import { ChildProcess, ChildProcessSpawner } from "effect/unstable/process"; import { useLegacyTempWorkdir } from "../../../../tests/helpers/legacy-mocks.ts"; @@ -96,6 +107,8 @@ interface FakeContainer { readonly labels: Readonly>; readonly autoRemove: boolean; running: boolean; + /** Whether this container has ever been `docker start`ed — distinguishes a RE-start for `failRestart`. */ + everStarted?: boolean; /** What a previous `docker cp - :` unpacked into this container, if anything. */ restored: string | undefined; } @@ -103,6 +116,8 @@ interface FakeContainer { function fakeDockerDaemon( opts: { readonly failStart?: boolean; + /** Fails only RE-starts (a `docker start` after the container has already run once) — the cold export's revive step. */ + readonly failRestart?: boolean; readonly failCopyOut?: boolean; readonly failCopyIn?: boolean; } = {}, @@ -163,8 +178,16 @@ function fakeDockerDaemon( stdout = id; } else if (args[0] === "start") { const container = containers.get(args[1] ?? ""); - if (opts.failStart === true || container === undefined) exitCode = 1; - else container.running = true; + if ( + opts.failStart === true || + container === undefined || + (opts.failRestart === true && container.everStarted) + ) { + exitCode = 1; + } else { + container.running = true; + container.everStarted = true; + } } else if (args[0] === "stop") { const id = args[1] ?? ""; const container = containers.get(id); @@ -668,6 +691,30 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live( + "a shadow that cannot come back after the snapshot fails the run, not just the cache", + () => { + const docker = fakeDockerDaemon({ failRestart: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); + // The export itself succeeds (stop + copy-out are fine); the revive `docker start` fails. + // Reporting success here would send the caller's next connect to a dead container's port + // — possibly answered by a DIFFERENT Postgres by then — so this must be a failure, not a + // "not cached" warning. + const exit = yield* handle.snapshotBaseline.pipe(Effect.exit); + expect(Exit.isFailure(exit)).toBe(true); + expect(out.stderrText).not.toContain("Warning: shadow baseline not cached"); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }, + ); + it.live("a failed export warns, leaves no tar, and still brings the container back up", () => { const docker = fakeDockerDaemon({ failCopyOut: true }); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index de177ec640..d2e531bd78 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -18,15 +18,17 @@ * {@link LegacyCreateShadowDatabaseInput.autoRemove} for the consequence); the tar is published by * an atomic rename, so concurrent writers need no lock file; a cache anomaly never fails the run — * a warm-path anomaly cold-provisions (deleting the tar only when its contents are implicated — - * see `LegacyShadowCacheUnavailable.tarSuspect`), a cold export failure only warns and - * leaves the run uncached; retention keeps the current key's tar only, sweeping every other one on + * see `LegacyShadowCacheUnavailable.tarSuspect`), a cold export failure only warns and leaves the + * run uncached (with ONE deliberate exception: a shadow that fails to come back up after the + * snapshot fails the run — see `legacyExportShadowBaseline`); retention keeps the current key's + * tar only, sweeping every other one on * publish. `SUPABASE_SHADOW_CACHE` is ON by default; `false`/`0` opts out. */ import { createHash } from "node:crypto"; import type { ProjectConfig } from "@supabase/config"; -import { Clock, Effect, Option, type FileSystem } from "effect"; +import { Clock, Effect, Option, Result, type FileSystem } from "effect"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import { Output } from "../../../shared/output/output.service.ts"; @@ -62,7 +64,7 @@ import { legacyCreateShadowDatabase, legacyRemoveShadowDatabase, type LegacyShadowBaselineState, - type LegacyShadowDbError, + LegacyShadowDbError, type LegacyShadowSetupInput, } from "./shadow-database.ts"; @@ -616,10 +618,15 @@ const legacyWriteShadowBaselineTar = ( * entrypoint `exec`s Postgres, so PID 1 receives the SIGTERM instead of `sh` swallowing it and * burning the full 10s grace period. * - * Failure is not the run's problem — it only means this run stays uncached. The ONE thing that - * must happen regardless is bringing the container back up: the caller is about to connect to it - * again. So the restart runs whether the export succeeded or not, and only its own failure (which - * dooms the run anyway, via the caller's next connect) outranks an export failure in the warning. + * Two failure classes, deliberately NOT one broad catch: a stop/export failure only means this + * run stays uncached — it warns and the run continues. A restart/readiness failure is the RUN'S + * problem: the caller is about to reconnect to the shadow's published port, and reporting success + * over a dead container would make that connect a blind dial — which can even reach a DIFFERENT + * Postgres that claimed the port while the container was down (matching default credentials are + * common locally), applying the template + migrations to the wrong database. So the restart runs + * whether the export succeeded or not (`docker start` on an already-running container — e.g. when + * the stop itself failed — is a no-op success), and its failure PROPAGATES as a + * {@link LegacyShadowDbError} instead of degrading (review: Codex on #6184). */ const legacyExportShadowBaseline = ( spawner: Spawner, @@ -627,26 +634,41 @@ const legacyExportShadowBaseline = ( key: string, tarPath: string, containerId: string, -): Effect.Effect => +): Effect.Effect => legacyTimeShadowPhase( "baseline-export", Effect.gen(function* () { - yield* legacyShadowContainerVerb(spawner, "stop", containerId); - // From here the container is DOWN; every exit path below has to start it again. - const written = yield* Effect.result( - legacyWriteShadowBaselineTar(spawner, input, key, tarPath, containerId), + // Cache-degradable phase: stop + export. A failure here (including a failed stop, after + // which the container is simply still up) only costs this run its snapshot. + const exported = yield* Effect.result( + Effect.gen(function* () { + yield* legacyShadowContainerVerb(spawner, "stop", containerId); + yield* legacyWriteShadowBaselineTar(spawner, input, key, tarPath, containerId); + }), ); - yield* legacyShadowContainerVerb(spawner, "start", containerId); - yield* legacyAwaitShadowReady(spawner, input, containerId, "re-started shadow"); - return yield* Effect.fromResult(written); - }), - ).pipe( - Effect.catch((cause) => - Effect.gen(function* () { + // Run-critical phase: the shadow must be back up and answering before this step reports + // success — see this function's own doc comment for why these failures must propagate. + const revive = Effect.gen(function* () { + yield* legacyShadowContainerVerb(spawner, "start", containerId); + yield* legacyAwaitShadowReady(spawner, input, containerId, "re-started shadow"); + }); + yield* revive.pipe( + Effect.mapError( + (cause) => + new LegacyShadowDbError({ + message: `shadow database did not come back after the baseline snapshot: ${cause.reason}`, + reason: "docker_daemon", + }), + ), + ); + if (Result.isFailure(exported)) { const output = yield* Output; - yield* output.raw(`Warning: shadow baseline not cached: ${cause.reason}\n`, "stderr"); - }), - ), + yield* output.raw( + `Warning: shadow baseline not cached: ${exported.failure.reason}\n`, + "stderr", + ); + } + }), ); // --------------------------------------------------------------------------- diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index 3a275e7d37..b535771099 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -792,10 +792,15 @@ export interface LegacyShadowBaselineState { * Takes NO session, and {@link legacyMigrateShadowDatabase} guarantees no session is open * against the shadow while it runs when {@link snapshotRequired} is set: the snapshot is a * disk-level PGDATA export that has to stop the container, which would sever any live backend. - * `Effect.Effect`: a cache that cannot snapshot must degrade silently, never - * fail the run. + * + * A cache that cannot SNAPSHOT degrades silently (warn + uncached run) — but the error channel + * is {@link LegacyShadowDbError}, not `never`, for the one failure that is the run's problem + * rather than the cache's: a shadow that does not come back up after the export. Reporting + * success there would send the caller's next connect to a dead (or worse, someone else's) + * Postgres on the shadow port — see `legacyExportShadowBaseline`'s doc comment + * (`shadow-cache.ts`). */ - readonly snapshotBaseline: Effect.Effect; + readonly snapshotBaseline: Effect.Effect; } /** The baseline state every uncached caller passes: provision it, snapshot nothing. */ From 08907794fafc19d6ef71cced13241f35300f5731 Mon Sep 17 00:00:00 2001 From: avallete Date: Thu, 13 Aug 2026 23:31:42 +0200 Subject: [PATCH 59/82] fix(cli): make PG<=14 cache-ineligible, sweep partials on warm hits, record the divergence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sixteenth review round on #6184 (Codex), all three findings taken: - PG <= 14 shadows bypass the cache entirely: that setup path executes the bundled globals SQL, whose ALTER ROLE ... SET statements (statement_timeout, postgres's search_path) only affect NEW sessions — Go/uncached runs apply migrations on the same pre-defaults session, while any snapshot boundary forces a reconnect that observes them, so unqualified names in migrations could resolve into different schemas and change the diff. PG15+ moves that SQL into initdb, before any session, so the divergence class does not exist there. - Warm hits now run the abandoned-partial sweep too — a killed concurrent writer's ~90MB leftover would otherwise persist forever once all later runs go warm. - The cache is recorded in docs/go-cli-divergences.md (default-on TS-only behavior, env opt-outs, ineligible configurations, and the honest PG15+ roles.sql session-semantics caveat). Co-Authored-By: Claude Fable 5 --- apps/cli/docs/go-cli-divergences.md | 12 ++++ .../shadow-cache.integration.test.ts | 57 +++++++++++++++++++ .../shared/db-bootstrap/shadow-cache.ts | 16 ++++++ 3 files changed, 85 insertions(+) diff --git a/apps/cli/docs/go-cli-divergences.md b/apps/cli/docs/go-cli-divergences.md index 00182eb622..7a42edfac3 100644 --- a/apps/cli/docs/go-cli-divergences.md +++ b/apps/cli/docs/go-cli-divergences.md @@ -79,6 +79,18 @@ These commands exist in the TS CLI today but have no direct top-level equivalent ## Behavioral divergences from the Go reference +- `db diff`/`db pull`/`db schema declarative sync` shadow baseline cache (#6184): the shadow + database's platform baseline is cached as a PGDATA snapshot under + `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB, current key only) and restored into a + fresh container on later runs, cutting shadow provisioning from ~15s to a few seconds. TS-only, + default ON; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (ambient env or project dotenv), `sync +--no-cache` bypasses it per-invocation, and `SUPABASE_SHADOW_DEBUG=1` prints stderr-only phase + timings. OrioleDB clusters and PG <= 14 are cache-ineligible (external S3 state and mid-session + role-default mutation respectively — see `shadow-cache.ts`). Known session-semantics caveat on + the cached paths: migrations run on a session opened after the baseline, so role-level defaults + a user's `roles.sql` installs (`ALTER ROLE … SET …`) apply to migrations, whereas Go's + single-connection flow ran migrations before those defaults took effect; opting out restores + Go's exact single-session behavior. - `functions serve` per-function env discovery (CLI-2184, #6179): without `--env-file`, each `supabase/functions//.env` overrides matching values from the shared `supabase/functions/.env` for that Function only; an explicit `--env-file` remains the diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index ecc6d68abe..5ccd03d900 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -437,6 +437,63 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("stays uncached on PG14, whose setup mutates role defaults mid-session", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + // PG<=14's globals SQL runs `ALTER ROLE … SET …` on the setup session; a snapshot + // boundary would force migrations onto a fresh session that observes those defaults, + // unlike Go's single-connection flow — so the cache must stand down entirely. + const base = shadowInput(fs, path); + const input = { + ...base, + db: { ...base.db, major_version: 14 }, + setup: { ...base.setup, majorVersion: 14 }, + }; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(handle.baselinePresent).toBe(false); + expect(handle.snapshotRequired).toBe(false); + expect(docker.calls("create")[0] ?? []).toContain("--rm"); + yield* handle.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a warm hit also sweeps abandoned partials left by a killed concurrent writer", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheEnv( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + yield* coldRun(docker, input); + // A concurrent writer that lost the publish race and was SIGKILLed mid-export: its + // partial predates the hour threshold. Every later run is warm, so the warm branch + // must be the one to sweep it. + const abandoned = path.join( + pgDeltaTempDir(path), + "shadow-baseline-0011223344556677.tar.4242.partial", + ); + yield* fs.writeFileString(abandoned, "stale"); + const twoHoursAgo = new Date(Date.now() - 2 * 60 * 60 * 1000); + yield* fs.utimes(abandoned, twoHoursAgo, twoHoursAgo); + + const warm = yield* legacyAcquireShadowDatabase(docker.spawner, input); + expect(warm.baselinePresent).toBe(true); + expect(yield* fs.exists(abandoned)).toBe(false); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("stays uncached for an OrioleDB cluster even with the cache enabled", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index d2e531bd78..1a5dcefb53 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -351,6 +351,16 @@ const legacyResolveShadowCacheKeyInputs = ( const orioledbVersion = input.experimental.orioledb_version; if (orioledbVersion !== undefined && orioledbVersion.length > 0) return Option.none(); + // PG <= 14 is cache-ineligible too: its setup path executes the bundled globals SQL + // (`LEGACY_START_DB_GLOBALS_SQL`, `db-setup.ts`'s pre-15 branch), whose `ALTER ROLE … SET` + // statements (statement_timeout, `postgres`'s search_path) only take effect on NEW sessions. + // Go/uncached runs apply migrations on the SAME session that ran the setup — before those + // defaults exist — while any snapshot boundary forces a reconnect that picks them up, so + // unqualified names in user migrations could resolve into different schemas and change the + // diff (review: Codex on #6184). PG15+ moves that SQL into the entrypoint's initdb heredoc, + // which runs before any CLI session, so no session ever observes a mid-run change there. + if (input.setup.majorVersion <= 14) return Option.none(); + const rolesPath = input.path.join(input.workdir, "supabase", "roles.sql"); const rolesSql = yield* input.fs .readFileString(rolesPath) @@ -846,6 +856,12 @@ export const legacyAcquireShadowDatabase = ( const cached = yield* input.fs.exists(tarPath).pipe(Effect.orElseSucceed(() => false)); if (!cached) return yield* legacyColdCachedShadow(spawner, input, key, tarPath); + // Warm hits sweep abandoned partials too: a killed concurrent writer's leftover would + // otherwise persist indefinitely once every later run goes warm, since the cold export's own + // sweep never runs again (review: Codex on #6184). Best-effort and cheap (one readdir + + // per-candidate stat). + yield* legacySweepAbandonedShadowBaselinePartials(input); + return yield* legacyWarmShadow(spawner, input, tarPath).pipe( Effect.catch((cause) => Effect.gen(function* () { From 502aefcd0265983196591ceb37462632eafb9b73 Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 14 Aug 2026 14:59:42 +0200 Subject: [PATCH 60/82] fix(cli): attach the exec-format recovery hint to shadow readiness failures legacyWaitForShadowReady dumped the container logs on failure but discarded the exec-format scan, so a wrong-architecture shadow postgres image failed with a bare "container is not running" and no recovery steps. The old shadow call sites never passed images to the health gate either, so this closes a pre-existing gap rather than restoring prior behavior: the three shadow readiness call sites now name the resolved postgres image, and the wait attaches the same recovery suggestion the stack health gate produces. Co-Authored-By: Claude Fable 5 --- .../db/shared/legacy-shadow-source.ts | 1 + .../shared/db-bootstrap/health-check.ts | 23 ++++- .../db-bootstrap/health-check.unit.test.ts | 83 +++++++++++++++++++ .../shared/db-bootstrap/shadow-database.ts | 1 + 4 files changed, 106 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts index 7ff60172bc..a79a60307a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts @@ -158,6 +158,7 @@ export const legacyPrepareShadowSource = ( yield* legacyWaitForShadowReady(spawner, containerId, connConfig, { timeoutSeconds: input.healthTimeoutSeconds, + image: input.image, }); // `handle` doubles as the baseline state: on a warm shadow-cache hit the cluster it carries diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts index 656d91ddd8..954f97fc9f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts @@ -489,6 +489,8 @@ const legacyProbeShadowConnect = ( export interface LegacyWaitForShadowReadyOptions { readonly timeoutSeconds?: number; + /** The shadow container's already-resolved postgres image, named in the exec-format recovery hint. */ + readonly image?: string; } /** @@ -515,7 +517,12 @@ export interface LegacyWaitForShadowReadyOptions { * Failure shape is deliberately identical to the health gate's: the same * {@link LegacyHealthCheckTimeoutError}, the same `unhealthy` payload, and the * same `docker logs` dump teed to stderr on the way out, so a broken shadow - * still surfaces exactly what it surfaced before. + * still surfaces exactly what it surfaced before. It additionally attaches the + * exec-format recovery {@link suggestion} when the caller names the shadow's + * image via {@link LegacyWaitForShadowReadyOptions.image} — something the old + * shadow call sites never got, since none of them ever passed `images` to + * {@link legacyWaitForHealthyServices} either. This closes that pre-existing + * gap rather than restoring parity with a behavior that previously existed. */ export function legacyWaitForShadowReady( spawner: Spawner, @@ -586,13 +593,25 @@ export function legacyWaitForShadowReady( Effect.gen(function* () { // Go skips this dump on context cancellation (`start.go:215`) — an // interrupted fiber never reaches this handler, so no separate check. - yield* legacyDumpContainerLogs(spawner, containerId); + const scan = yield* legacyDumpContainerLogs(spawner, containerId); + // Unlike the health gate, the old shadow call sites never named an + // image here at all — so this `opts.image` check is what makes a + // shadow's exec-format failure carry the hint for the first time. + const suggestion = + scan.found && opts.image !== undefined + ? legacyExecFormatRecoveryHint( + [containerId], + new Map([[containerId, opts.image]]), + scan.runtime ?? "docker", + ) + : undefined; return yield* Effect.fail( new LegacyHealthCheckTimeoutError({ // The health gate's own ` ` joining, so both waits read // identically on stderr. message: `${containerId} ${failure.reason}`, unhealthy: [{ containerId, reason: failure.reason }], + ...(suggestion === undefined ? {} : { suggestion }), }), ); }), diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts index 1f1b9a8b88..e1374fb4bf 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts @@ -976,4 +976,87 @@ describe("legacyWaitForShadowReady", () => { ).toBe(true); }), ); + + describe("exec format error recovery advice", () => { + it.effect("names the shadow's image when its dumped logs show exec format error", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting, { + [SHADOW_CONTAINER_ID]: ["exec /docker-entrypoint.sh: exec format error\n"], + }); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 1, image: "public.ecr.aws/supabase/postgres:15.1.0.147" }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.suggestion).toContain( + `${SHADOW_CONTAINER_ID}'s image public.ecr.aws/supabase/postgres:15.1.0.147`, + ); + expect(error.suggestion).toContain( + "supabase stop\n docker image rm -f public.ecr.aws/supabase/postgres:15.1.0.147\n supabase start", + ); + }), + ); + + it.effect("stays silent when no image is named, even if the logs show exec format error", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting, { + [SHADOW_CONTAINER_ID]: ["exec /docker-entrypoint.sh: exec format error\n"], + }); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 1 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.suggestion).toBeUndefined(); + }), + ); + + it.effect("stays silent for a plain not-ready timeout even when an image is named", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting); + const db = mockShadowDbConnection({ failTimes: Number.POSITIVE_INFINITY }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 1, image: "public.ecr.aws/supabase/postgres:15.1.0.147" }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + yield* TestClock.adjust("1 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.suggestion).toBeUndefined(); + }), + ); + }); }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index b535771099..a81d83366f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -591,6 +591,7 @@ export const legacyPrepareRawShadow = ( }; yield* legacyWaitForShadowReady(spawner, containerId, connConfig, { timeoutSeconds: input.healthTimeoutSeconds, + image: input.image, }); return { container: containerId, From 29aa7e3c39aa9b71ed16571fb5c87d2c9f682c88 Mon Sep 17 00:00:00 2001 From: avallete Date: Fri, 14 Aug 2026 14:59:50 +0200 Subject: [PATCH 61/82] refactor(cli): drop PG<=14-only SQL from the shadow baseline digest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LEGACY_START_DB_GLOBALS_SQL and the 13/14 initial-schema templates execute only on the PG<=14 setup branch, and PG<=14 short-circuits to Option.none before any cache key is computed — so they could never influence a cached cluster and only read as load-bearing. The digest doc now records the exclusion and the re-add condition should PG<=14 ever become cache-eligible. Co-Authored-By: Claude Fable 5 --- .../shared/db-bootstrap/shadow-cache.ts | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 1a5dcefb53..ae1f8b3b87 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -43,9 +43,6 @@ import { legacyPgDeltaTempPath } from "../legacy-pgdelta.paths.ts"; import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; import { LEGACY_POSTGRES_DEFAULT_ROOT_KEY } from "../legacy-local-config-values.ts"; import { LEGACY_START_REVOKE_API_PRIVILEGES_SQL } from "./db-setup.ts"; -import { LEGACY_START_DB_GLOBALS_SQL } from "./templates/db-globals.sql.ts"; -import { LEGACY_START_DB_INITIAL_SCHEMA_13_SQL } from "./templates/db-initial-schema-13.sql.ts"; -import { LEGACY_START_DB_INITIAL_SCHEMA_14_SQL } from "./templates/db-initial-schema-14.sql.ts"; import { LEGACY_START_DB_SCHEMA_SQL } from "./templates/db-schema.sql.ts"; import { LEGACY_START_DB_SUPABASE_SQL } from "./templates/db-supabase.sql.ts"; import { LEGACY_START_DB_WEBHOOK_SQL } from "./templates/db-webhook.sql.ts"; @@ -211,12 +208,17 @@ export interface LegacyShadowCacheKeyInputs { /** * Digest of every CLI-EMBEDDED SQL text baked into the baseline cluster — the inputs that change * with a CLI release rather than with the project's config: the PG15+ entrypoint's initdb heredocs - * (schema/webhook/_supabase — `postgres.service.ts`), the PG<=14 setup path's globals + initial - * schema, and the API privilege revocation. Without this line, a CLI upgrade that edits a grant, - * schema statement, or revocation WITHOUT bumping the postgres image would warm-restore the - * previous release's baseline (review: depthfirst on #6184). Computed once at module load — these - * are compile-time constants. When adding a new embedded SQL step to the baseline - * (`legacySetupDatabase`/the entrypoint scripts), add its text here too. + * (schema/webhook/_supabase — `postgres.service.ts`) and the API privilege revocation. Without this + * line, a CLI upgrade that edits a grant, schema statement, or revocation WITHOUT bumping the + * postgres image would warm-restore the previous release's baseline (review: depthfirst on #6184). + * Computed once at module load — these are compile-time constants. When adding a new embedded SQL + * step to the baseline (`legacySetupDatabase`/the entrypoint scripts), add its text here too. + * + * Deliberately EXCLUDES PG<=14's own setup SQL (`LEGACY_START_DB_GLOBALS_SQL`, + * `LEGACY_START_DB_INITIAL_SCHEMA_13_SQL`/`_14_SQL`): PG<=14 is cache-ineligible — + * {@link legacyResolveShadowCacheKeyInputs} returns `Option.none()` for `majorVersion <= 14` before + * any key is computed, so no cluster keyed by this digest can ever have run through that SQL. If + * PG<=14 ever becomes cache-eligible, those templates must be re-added here. */ const LEGACY_SHADOW_BASELINE_SQL_DIGEST = createHash("sha256") .update( @@ -224,9 +226,6 @@ const LEGACY_SHADOW_BASELINE_SQL_DIGEST = createHash("sha256") LEGACY_START_DB_SCHEMA_SQL, LEGACY_START_DB_WEBHOOK_SQL, LEGACY_START_DB_SUPABASE_SQL, - LEGACY_START_DB_GLOBALS_SQL, - LEGACY_START_DB_INITIAL_SCHEMA_13_SQL, - LEGACY_START_DB_INITIAL_SCHEMA_14_SQL, LEGACY_START_REVOKE_API_PRIVILEGES_SQL, // The vault upsert's own SQL (`legacyUpsertVaultSecrets`, `legacy-vault.ts`) runs into the // baseline right after the privilege pass — same digest rationale as every line above. @@ -578,6 +577,7 @@ const legacyAwaitShadowReady = ( ): Effect.Effect => legacyWaitForShadowReady(spawner, containerId, legacyShadowConnConfig(input), { timeoutSeconds: input.healthTimeoutSeconds, + image: input.image, }).pipe( Effect.mapError((cause) => legacyShadowCacheUnavailable(`${what} never became ready: ${cause.message}`), From 0073867b91d85e09c007d7fde78762f7cd986ce2 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 11:56:54 +0200 Subject: [PATCH 62/82] fix(cli): pass shadow setup options on the cold migrate path The snapshot-required branch already forwarded `{ webhooks: "enabled" }` into legacySetupDatabase; the common cold path dropped it, so migra and legacy pg-delta shadows skipped pg_net. Also distinguish streamed secret copies from PGDATA restores in the shadow-cache mock after #6201. --- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 50 +++++++++---------- .../commands/db/diff/diff.integration.test.ts | 21 +++++--- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 50 +++++++++---------- .../schema/declarative/sync/SIDE_EFFECTS.md | 38 +++++++------- .../shadow-cache.integration.test.ts | 15 ++++-- .../shared/db-bootstrap/shadow-database.ts | 1 + apps/cli/tests/helpers/legacy-mocks.ts | 23 +++++++-- 7 files changed, 114 insertions(+), 84 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 6478b7a027..8312e7f97a 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -19,34 +19,34 @@ it, and JSON `null` disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| --------------------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | -| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit — the snapshot is streamed into the fresh shadow container before it starts | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | -| `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | -| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | -| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit `--from/--to migrations` catalog cache | +| Path | Format | When | +| --------------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | +| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit — the snapshot is streamed into the fresh shadow container before it starts | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | +| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit `--from/--to migrations` catalog cache | ## Files Written -| Path | Format | When | -| ------------------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------ | -| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | -| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision only (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`, never a warm hit) — the shadow's PGDATA snapshot, ~90MB, current key only | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | -| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| -------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | +| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision only (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`, never a warm hit) — the shadow's PGDATA snapshot, ~90MB, current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | +| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 042fa293af..9f4b41aa16 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -10,6 +10,7 @@ import { stripAnsi } from "../../../../../tests/helpers/ansi.ts"; import { LEGACY_FAKE_SHADOW_CONTAINER_ID, LEGACY_VALID_REF, + legacyFailWriteStringMatchingFsLayer, legacyFailWriteStringOnNthCallFsLayer, mockLegacyCliConfig, mockLegacyLinkedProjectCacheTracked, @@ -88,6 +89,10 @@ interface SetupOpts { readonly networkId?: string; // --network-id value forwarded to docker runs // When set, the Nth `writeFileString` fails, exercising cleanup-on-failure. readonly failWriteOnCall?: number; + // When set, the first `writeFileString` whose path matches fails. Prefer this + // over `failWriteOnCall` when shadow setup writes extra SQL before the + // command's `--file` migration. + readonly failWriteMatching?: (path: string) => boolean; // When set, the shadow container never reports healthy — for the interrupt-during- // health-wait regression coverage (review: PRRT_kwDOErm0O86XMrID). See // `mockLegacyShadowContainerCliSpawner`'s own doc comment for why this is required @@ -450,10 +455,13 @@ function setup(workdir: string, opts: SetupOpts = {}) { mockRuntimeInfo({ platform: opts.platform ?? "linux" }), ); // Merged last so its `FileSystem` overrides everything above (last-wins). - const layer = - opts.failWriteOnCall === undefined - ? baseLayer - : Layer.merge(baseLayer, legacyFailWriteStringOnNthCallFsLayer(opts.failWriteOnCall)); + const failWriteLayer = + opts.failWriteMatching !== undefined + ? legacyFailWriteStringMatchingFsLayer(opts.failWriteMatching) + : opts.failWriteOnCall !== undefined + ? legacyFailWriteStringOnNthCallFsLayer(opts.failWriteOnCall) + : undefined; + const layer = failWriteLayer === undefined ? baseLayer : Layer.merge(baseLayer, failWriteLayer); return { layer, @@ -2452,12 +2460,9 @@ describe("legacy db diff", () => { it.effect( "fails with LegacyDbDiffWriteError when writing the pgAdmin --file migration fails", () => { - // Shadow setup writes the branch marker and `revoke-api-privileges.sql` - // before the command writes the pgAdmin migration, so call #3 is the - // diff-file write exercised here. const s = setup(tmp.current, { pgadminStdout: [JSON.stringify([pgadminEntry()])], - failWriteOnCall: 3, + failWriteMatching: (path) => path.includes("pgadmin_diff"), }); return Effect.gen(function* () { const error = yield* legacyDbDiff( diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 413feb575d..385986735d 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -35,35 +35,35 @@ disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ----------------------------------------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | -| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — snapshot streamed into the fresh shadow container before it starts | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | -| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | -| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | +| Path | Format | When | +| ----------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | +| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — snapshot streamed into the fresh shadow container before it starts | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | ## Files Written -| Path | Format | When | -| ---------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | -| `/supabase/schemas/**` | SQL | `--declarative` | -| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision, migration-style pull only (never `--declarative`'s bare shadow, the delegated `--experimental` path, or a warm hit) — the shadow's PGDATA snapshot, ~90MB, one file for the current key only | +| Path | Format | When | +| -------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | +| `/supabase/schemas/**` | SQL | `--declarative` | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision, migration-style pull only (never `--declarative`'s bare shadow, the delegated `--experimental` path, or a warm hit) — the shadow's PGDATA snapshot, ~90MB, one file for the current key only | | `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | -| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 1912632ad8..1e5b1d14d1 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -15,29 +15,29 @@ disabling safe compaction. ## Files Read -| Path | Format | When | -| --------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------ | -| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | -| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | -| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | -| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | -| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | -| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit on a migrations-catalog miss — snapshot streamed into the fresh shadow container before it starts | +| Path | Format | When | +| -------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | +| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | +| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit on a migrations-catalog miss — snapshot streamed into the fresh shadow container before it starts | | `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | ## Files Written -| Path | Format | When | -| ------------------------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------------ | -| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | -| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, current key only | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | +| Path | Format | When | +| -------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | +| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, current key only | +| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | ## Subprocesses / Containers diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 5ccd03d900..1ab7da8eea 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -201,14 +201,17 @@ function fakeDockerDaemon( } else if (args[0] === "rm") { containers.delete(args[args.length - 1] ?? ""); } else if (args[0] === "cp" && args[1] === "-") { - // Restore: `docker cp - :`, tar on stdin. + // Secret copy is `docker cp - :/`; restore is `docker cp - :`. + // Both use stdin, so failCopyIn must apply only to the restore — otherwise a warm + // fallback test kills the pgsodium root-key copy and never reaches the archive. const [id = "", containerPath = ""] = (args[2] ?? "").split(":"); const container = containers.get(id); const received = yield* readStdin(command); - if (opts.failCopyIn === true || container === undefined) { + const isSecret = containerPath === "" || containerPath === "/"; + if (container === undefined || (!isSecret && opts.failCopyIn === true)) { exitCode = 1; stderr = "no such container"; - } else { + } else if (!isSecret) { container.restored = `${containerPath}::${received}`; } } else if (args[0] === "cp" && args[2] === "-") { @@ -261,7 +264,11 @@ function fakeDockerDaemon( if (args[0] === "network") return "network"; if (args[0] === "container" && args[1] === "inspect") return "inspect"; if (args[0] === "cp") { - if (args[1] === "-") return "cp-in"; + if (args[1] === "-") { + const dest = args[2] ?? ""; + const containerPath = dest.slice(dest.indexOf(":") + 1); + return containerPath === "" || containerPath === "/" ? "cp-secret" : "cp-in"; + } if (args[2] === "-") return "cp-out"; return "cp-secret"; } diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index a81d83366f..ea3f0f20a9 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -886,6 +886,7 @@ const migrateShadowDatabase = ( yield* legacySetupDatabase( spawner, legacyBuildShadowSetupDatabaseInput(input, session, resolved), + setupOptions, ); } yield* legacyCreateShadowTemplateDatabase(session); diff --git a/apps/cli/tests/helpers/legacy-mocks.ts b/apps/cli/tests/helpers/legacy-mocks.ts index 531eb656cb..11e5c66ffb 100644 --- a/apps/cli/tests/helpers/legacy-mocks.ts +++ b/apps/cli/tests/helpers/legacy-mocks.ts @@ -746,8 +746,8 @@ export function legacyIsolatedHomeLayer( } // --------------------------------------------------------------------------- -// Failing filesystem — wraps the real Bun `FileSystem` and fails the Nth -// `writeFileString` call with a `PlatformError`, so cleanup-on-failure paths +// Failing filesystem — wraps the real Bun `FileSystem` and fails a chosen +// `writeFileString` with a `PlatformError`, so cleanup-on-failure paths // (e.g. the pg-delta multi-file migration writer) can be exercised // deterministically. Every other call delegates to the real filesystem, so // config reads / earlier writes behave normally. Merge this AFTER @@ -757,6 +757,23 @@ export function legacyIsolatedHomeLayer( export function legacyFailWriteStringOnNthCallFsLayer( failOnCall: number, +): Layer.Layer { + return legacyFailWriteStringFsLayer((_, calls) => calls === failOnCall); +} + +/** + * Same as {@link legacyFailWriteStringOnNthCallFsLayer}, but fails the first + * `writeFileString` whose path matches `match`. Prefer this when earlier + * setup writes (shadow SQL, branch markers) make a fixed call index brittle. + */ +export function legacyFailWriteStringMatchingFsLayer( + match: (path: string) => boolean, +): Layer.Layer { + return legacyFailWriteStringFsLayer((path) => match(path)); +} + +function legacyFailWriteStringFsLayer( + shouldFail: (path: string, calls: number) => boolean, ): Layer.Layer { return Layer.effect( FileSystem.FileSystem, @@ -766,7 +783,7 @@ export function legacyFailWriteStringOnNthCallFsLayer( ...real, writeFileString: (path, data, options) => { calls += 1; - if (calls === failOnCall) { + if (shouldFail(path, calls)) { return Effect.fail( new PlatformError( new SystemError({ From 06216665baf2f651a8cff414b7eb87601a225cde Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 14:26:38 +0200 Subject: [PATCH 63/82] fix(cli): store shadow baseline cache under SUPABASE_HOME Share warm PGDATA snapshots across worktrees with the same settings, and replace current-key-only retention with LRU + TTL so the global store cannot grow unbounded. --- apps/cli/docs/go-cli-divergences.md | 4 +- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 32 ++-- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 58 +++---- .../schema/declarative/sync/SIDE_EFFECTS.md | 46 +++--- .../shadow-cache.integration.test.ts | 148 ++++++++++++----- .../db-bootstrap/shadow-cache.live.test.ts | 15 +- .../shared/db-bootstrap/shadow-cache.ts | 151 +++++++++++++----- .../db-bootstrap/shadow-cache.unit.test.ts | 91 ++++++++--- .../src/legacy/shared/legacy-pgdelta.paths.ts | 32 +++- 9 files changed, 398 insertions(+), 179 deletions(-) diff --git a/apps/cli/docs/go-cli-divergences.md b/apps/cli/docs/go-cli-divergences.md index 7a42edfac3..2e0665b74d 100644 --- a/apps/cli/docs/go-cli-divergences.md +++ b/apps/cli/docs/go-cli-divergences.md @@ -81,7 +81,9 @@ These commands exist in the TS CLI today but have no direct top-level equivalent - `db diff`/`db pull`/`db schema declarative sync` shadow baseline cache (#6184): the shadow database's platform baseline is cached as a PGDATA snapshot under - `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB, current key only) and restored into a + `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides + the root; LRU keep-8 + 14-day mtime TTL, shared across worktrees with the same settings) and + restored into a fresh container on later runs, cutting shadow provisioning from ~15s to a few seconds. TS-only, default ON; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (ambient env or project dotenv), `sync --no-cache` bypasses it per-invocation, and `SUPABASE_SHADOW_DEBUG=1` prints stderr-only phase diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 8312e7f97a..6e40120376 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -26,8 +26,8 @@ it, and JSON `null` disables formatting without disabling safe compaction. | `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | | `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | | `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit — the snapshot is streamed into the fresh shadow container before it starts | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — the snapshot is streamed into the fresh shadow container before it starts (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | | `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | | `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | | `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | @@ -36,17 +36,17 @@ it, and JSON `null` disables formatting without disabling safe compaction. ## Files Written -| Path | Format | When | -| -------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | -| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision only (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`, never a warm hit) — the shadow's PGDATA snapshot, ~90MB, current key only | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | -| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | +| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision only (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`, never a warm hit) — the shadow's PGDATA snapshot, ~90MB, LRU keep-8 + 14-day mtime TTL (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker @@ -218,8 +218,10 @@ ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the amb project's dotenv, e.g. `supabase/.env`), restoring the documented uncached lifecycle. A warm hit skips the platform baseline, so the `Initialising schema...` progress line does not print — progress text reflects the work actually performed. -Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot -keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. +Artifact: `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` +overrides the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster; +shared across worktrees with the same settings; retention is LRU (keep 8) + 14-day mtime TTL +(warm hits refresh mtime). Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a cold export failure only warns and leaves the run uncached (one exception: a shadow that diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 385986735d..64e67f4b39 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -35,35 +35,35 @@ disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ----------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | -| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — snapshot streamed into the fresh shadow container before it starts | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | -| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | -| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | +| Path | Format | When | +| ----------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | +| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — snapshot streamed into the fresh shadow container before it starts (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | ## Files Written -| Path | Format | When | -| -------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | -| `/supabase/schemas/**` | SQL | `--declarative` | -| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision, migration-style pull only (never `--declarative`'s bare shadow, the delegated `--experimental` path, or a warm hit) — the shadow's PGDATA snapshot, ~90MB, one file for the current key only | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | -| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | +| `/supabase/schemas/**` | SQL | `--declarative` | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision, migration-style pull only (never `--declarative`'s bare shadow, the delegated `--experimental` path, or a warm hit) — the shadow's PGDATA snapshot, ~90MB, LRU keep-8 + 14-day mtime TTL (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker @@ -83,8 +83,10 @@ ON by default; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the amb project's dotenv, e.g. `supabase/.env`), restoring the documented uncached lifecycle. A warm hit skips the platform baseline, so the `Initialising schema...` progress line does not print — progress text reflects the work actually performed. -Artifact: `supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot -keyed by a hash of every input baked into the cluster; retention keeps the current key's tar only. +Artifact: `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` +overrides the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster; +shared across worktrees with the same settings; retention is LRU (keep 8) + 14-day mtime TTL +(warm hits refresh mtime). Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a cold export failure only warns and leaves the run uncached (one exception: a shadow that diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 1e5b1d14d1..2a00c875b4 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -15,29 +15,29 @@ disabling safe compaction. ## Files Read -| Path | Format | When | -| -------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | -| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | -| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | -| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | -| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | -| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | warm shadow-cache hit on a migrations-catalog miss — snapshot streamed into the fresh shadow container before it starts | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | +| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | +| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit on a migrations-catalog miss — snapshot streamed into the fresh shadow container before it starts (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | ## Files Written -| Path | Format | When | -| -------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | -| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, current key only | -| `/supabase/.temp/pgdelta/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports sweep leftovers older than an hour | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | +| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, LRU keep-8 + 14-day mtime TTL (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | ## Subprocesses / Containers @@ -136,8 +136,10 @@ project's dotenv, e.g. `supabase/.env`), and `--no-cache` bypasses it for that i flag promises fresh shadow setup, so it disables the snapshot cache along with the catalog cache); a warm hit skips the platform baseline and therefore the `Initialising schema...` progress line. Artifact: -`supabase/.temp/pgdelta/shadow-baseline-.tar` (~90MB), a PGDATA snapshot keyed by a hash of -every input baked into the cluster; retention keeps the current key's tar only. Container +`~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides +the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster; shared across +worktrees with the same settings; retention is LRU (keep 8) + 14-day mtime TTL (warm hits refresh +mtime). Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a cold export failure only warns and leaves the run uncached (one exception: a shadow that fails diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 1ab7da8eea..597b0a295f 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -10,6 +10,8 @@ * (the export must stop the container before copying and start it again afterwards). */ +import { join } from "node:path"; + import type { ProjectConfig } from "@supabase/config"; import { ProjectConfigSchema } from "@supabase/config"; import { BunServices } from "@effect/platform-bun"; @@ -32,8 +34,13 @@ import { useLegacyTempWorkdir } from "../../../../tests/helpers/legacy-mocks.ts" import { mockOutput } from "../../../../tests/helpers/mocks.ts"; import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import { LegacyDbConnectError } from "../legacy-db-connection.errors.ts"; +import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; import { LEGACY_PGDATA_PARENT_PATH, LEGACY_PGDATA_PATH } from "./pgdata-snapshot.ts"; -import { LEGACY_SHADOW_CACHE_ENV, legacyAcquireShadowDatabase } from "./shadow-cache.ts"; +import { + LEGACY_SHADOW_BASELINE_KEEP, + LEGACY_SHADOW_CACHE_ENV, + legacyAcquireShadowDatabase, +} from "./shadow-cache.ts"; import { LEGACY_SHADOW_DEBUG_ENV } from "./shadow-debug.ts"; import { legacyRemoveShadowDatabase } from "./shadow-database.ts"; import type { LegacyShadowDbSetupInput, LegacyShadowSetupInput } from "./shadow-database.ts"; @@ -67,6 +74,20 @@ const withEnv = ( const withShadowCacheEnv = (value: string | undefined, body: Effect.Effect) => withEnv(LEGACY_SHADOW_CACHE_ENV, value, body); +/** + * Isolates the global shadow-baseline cache under a per-test `SUPABASE_HOME` so tests never + * write into the developer's real `~/.supabase`. Nested with the opt-in/opt-out gate. + */ +const withShadowCacheHome = ( + value: string | undefined, + body: Effect.Effect, +): Effect.Effect => + withEnv( + "SUPABASE_HOME", + join(tempRoot.current, "_supabase_home"), + withShadowCacheEnv(value, body), + ); + const withShadowDebugEnv = (value: string | undefined, body: Effect.Effect) => withEnv(LEGACY_SHADOW_DEBUG_ENV, value, body); @@ -365,13 +386,12 @@ const shadowInput = ( setup: shadowSetup(), }); -const pgDeltaTempDir = (path: Path.Path) => - path.join(tempRoot.current, "supabase", ".temp", "pgdelta"); +const shadowCacheDir = (path: Path.Path) => legacyShadowBaselineCacheDir(path); -/** The one snapshot tar in the temp dir, whatever key it belongs to. */ +/** The snapshot tars in the global cache dir, whatever keys they belong to. */ const soleTarName = Effect.fnUntraced(function* (fs: FileSystem.FileSystem, path: Path.Path) { const entries = yield* fs - .readDirectory(pgDeltaTempDir(path)) + .readDirectory(shadowCacheDir(path)) .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); return entries.filter((entry) => entry.endsWith(".tar")); }); @@ -393,7 +413,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "0", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -420,7 +440,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -448,7 +468,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -476,7 +496,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -487,7 +507,7 @@ describe("legacyAcquireShadowDatabase", () => { // partial predates the hour threshold. Every later run is warm, so the warm branch // must be the one to sweep it. const abandoned = path.join( - pgDeltaTempDir(path), + shadowCacheDir(path), "shadow-baseline-0011223344556677.tar.4242.partial", ); yield* fs.writeFileString(abandoned, "stale"); @@ -505,7 +525,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -530,7 +550,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( undefined, Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -546,7 +566,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -582,10 +602,10 @@ describe("legacyAcquireShadowDatabase", () => { const tars = yield* soleTarName(fs, path); expect(tars).toHaveLength(1); expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); - expect(yield* fs.readFileString(path.join(pgDeltaTempDir(path), tars[0] ?? ""))).toBe( + expect(yield* fs.readFileString(path.join(shadowCacheDir(path), tars[0] ?? ""))).toBe( FAKE_PGDATA_TAR, ); - const leftovers = yield* fs.readDirectory(pgDeltaTempDir(path)); + const leftovers = yield* fs.readDirectory(shadowCacheDir(path)); expect(leftovers.filter((entry) => entry.includes("partial"))).toEqual([]); // Release is the uncached removal, same as ever — nothing is kept. @@ -599,7 +619,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -638,13 +658,13 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; const input = shadowInput(fs, path); - const tempDir = pgDeltaTempDir(path); + const tempDir = shadowCacheDir(path); yield* fs.makeDirectory(tempDir, { recursive: true }); // An adversarially (or crash-) pre-created temp file at THIS process's own temp path, // world-readable. The export must not inherit its mode: the pre-remove + `wx` @@ -671,12 +691,12 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; - const tempDir = pgDeltaTempDir(path); + const tempDir = shadowCacheDir(path); yield* fs.makeDirectory(tempDir, { recursive: true }); // A SIGKILLed export's leftover (writer long gone — hour-plus-old mtime) and a // concurrent writer's live temp file (fresh mtime). @@ -696,31 +716,71 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); - it.live("publishing a new key's tar sweeps every other key's", () => { + it.live("publishing distinct keys keeps both tars until LRU/TTL eviction", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; const path = yield* Path.Path; yield* coldRun(docker, shadowInput(fs, path)); - const stale = yield* soleTarName(fs, path); - expect(stale).toHaveLength(1); + const first = yield* soleTarName(fs, path); + expect(first).toHaveLength(1); - // A changed baseline input (the shadow's own published port) is a different cluster, so - // its snapshot is a different key — and ~90MB each means only the current one is kept. + // A changed baseline input (the shadow's own published port) is a different cluster / + // different key — both must coexist in the global cache (unlike the old project-local + // current-key-only sweep, which would have deleted the first). const rekeyed = yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54399 })); - const fresh = yield* soleTarName(fs, path); - expect(fresh).toHaveLength(1); - expect(fresh[0]).not.toBe(stale[0]); + const both = yield* soleTarName(fs, path); + expect(both).toHaveLength(2); + expect(both).toContain(first[0]); expect(rekeyed.baselinePresent).toBe(false); - // An unrelated file in the same shared temp directory is untouched. - yield* fs.writeFileString(path.join(pgDeltaTempDir(path), "catalog-abc.json"), "{}"); - yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54398 })); - expect(yield* fs.exists(path.join(pgDeltaTempDir(path), "catalog-abc.json"))).toBe(true); + // An unrelated file in the cache directory is untouched by retention. + const stray = path.join(shadowCacheDir(path), "catalog-abc.json"); + yield* fs.writeFileString(stray, "{}"); + + // Fill to the keep-cap + 1 with more distinct keys; the oldest (first) is evicted. + for (let i = 0; i < LEGACY_SHADOW_BASELINE_KEEP - 1; i++) { + yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54400 + i })); + } + const afterCap = yield* soleTarName(fs, path); + expect(afterCap).toHaveLength(LEGACY_SHADOW_BASELINE_KEEP); + expect(afterCap).not.toContain(first[0]); + expect(yield* fs.exists(stray)).toBe(true); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("worktrees with identical settings share a warm hit from the global cache", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const worktreeA = path.join(tempRoot.current, "worktree-a"); + const worktreeB = path.join(tempRoot.current, "worktree-b"); + yield* fs.makeDirectory(path.join(worktreeA, "supabase"), { recursive: true }); + yield* fs.makeDirectory(path.join(worktreeB, "supabase"), { recursive: true }); + + const cold = yield* coldRun(docker, { ...shadowInput(fs, path), workdir: worktreeA }); + expect(cold.baselinePresent).toBe(false); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + // Same settings, different project path — the second worktree must restore, not re-export. + const warm = yield* legacyAcquireShadowDatabase(docker.spawner, { + ...shadowInput(fs, path), + workdir: worktreeB, + }); + expect(warm.baselinePresent).toBe(true); + expect(warm.containerId).not.toBe(cold.containerId); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + yield* legacyRemoveShadowDatabase(docker.spawner, warm.containerId); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); @@ -729,7 +789,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -749,8 +809,10 @@ describe("legacyAcquireShadowDatabase", () => { ); expect(mirrored.baselinePresent).toBe(false); const mirroredTar = yield* soleTarName(fs, path); - expect(mirroredTar).toHaveLength(1); - expect(mirroredTar[0]).not.toBe(defaultRegistryTar[0]); + // Distinct keys coexist in the global cache — the default-registry tar is not swept. + expect(mirroredTar).toHaveLength(2); + expect(mirroredTar).toContain(defaultRegistryTar[0]); + expect(mirroredTar.some((name) => name !== defaultRegistryTar[0])).toBe(true); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); @@ -761,7 +823,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon({ failRestart: true }); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -783,7 +845,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon({ failCopyOut: true }); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -796,7 +858,7 @@ describe("legacyAcquireShadowDatabase", () => { // The caller is about to reconnect, so the container is running again regardless. expect(docker.containers.get(handle.containerId)?.running).toBe(true); // Neither a published tar nor a half-written temp file survives. - const entries = yield* fs.readDirectory(pgDeltaTempDir(path)); + const entries = yield* fs.readDirectory(shadowCacheDir(path)); expect(entries).toEqual([]); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); @@ -808,7 +870,7 @@ describe("legacyAcquireShadowDatabase", () => { const docker = fakeDockerDaemon({ failCopyIn: true }); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -841,7 +903,7 @@ describe("legacyAcquireShadowDatabase", () => { it.live("a restored shadow that never becomes ready is removed before the cold retry", () => { const docker = fakeDockerDaemon(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -878,7 +940,7 @@ describe("SUPABASE_SHADOW_DEBUG phase-timing instrumentation", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", Effect.gen(function* () { const fs = yield* FileSystem.FileSystem; @@ -909,7 +971,7 @@ describe("SUPABASE_SHADOW_DEBUG phase-timing instrumentation", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); const out = mockOutput(); - return withShadowCacheEnv( + return withShadowCacheHome( "1", withShadowDebugEnv( undefined, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts index 133691063c..74c0b7f70a 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts @@ -17,6 +17,8 @@ * so a bare cluster is a faithful stand-in for it. */ +import { join } from "node:path"; + import type { ProjectConfig } from "@supabase/config"; import { ProjectConfigSchema } from "@supabase/config"; import { BunServices } from "@effect/platform-bun"; @@ -29,6 +31,7 @@ import { mockOutput } from "../../../../tests/helpers/mocks.ts"; import { dockerfileServiceImage } from "../../../shared/services/dockerfile-images.ts"; import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import { legacyDbConnectionLayer } from "../legacy-db-connection.layer.ts"; +import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; import { legacyWaitForShadowReady } from "./health-check.ts"; import { LEGACY_SHADOW_CACHE_ENV, @@ -54,7 +57,11 @@ describeLive("shadow baseline cache (live Docker)", () => { const path = yield* Path.Path; const spawner = yield* ChildProcessSpawner.ChildProcessSpawner; const workdir = yield* fs.makeTempDirectoryScoped({ prefix: "legacy-shadow-cache-live-" }); + const supabaseHome = yield* fs.makeTempDirectoryScoped({ + prefix: "legacy-shadow-cache-home-", + }); yield* fs.makeDirectory(path.join(workdir, "supabase"), { recursive: true }); + process.env.SUPABASE_HOME = supabaseHome; const setup: LegacyShadowDbSetupInput = { majorVersion: 17, @@ -168,13 +175,15 @@ describeLive("shadow baseline cache (live Docker)", () => { ); yield* legacyRemoveShadowDatabase(spawner, warm.containerId); - // The artifact is a plain file under the project's own temp dir — the property that lets a - // future native (non-Docker) Postgres service consume the same snapshot. - const tempDir = path.join(workdir, "supabase", ".temp", "pgdelta"); + // The artifact is a plain file under the global per-settings cache — the property that + // lets worktrees with the same settings share a warm hit, and a future native (non-Docker) + // Postgres service consume the same snapshot. + const tempDir = legacyShadowBaselineCacheDir(path); const entries = yield* fs.readDirectory(tempDir); const tars = entries.filter((entry) => entry.endsWith(".tar")); expect(tars).toHaveLength(1); expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); + expect(tempDir).toBe(join(supabaseHome, "cache", "shadow-baseline")); expect(legacyShadowBaselineTarFileName("0".repeat(16))).toBe( `shadow-baseline-${"0".repeat(16)}.tar`, ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index ae1f8b3b87..8f5ece93e8 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -20,9 +20,10 @@ * a warm-path anomaly cold-provisions (deleting the tar only when its contents are implicated — * see `LegacyShadowCacheUnavailable.tarSuspect`), a cold export failure only warns and leaves the * run uncached (with ONE deliberate exception: a shadow that fails to come back up after the - * snapshot fails the run — see `legacyExportShadowBaseline`); retention keeps the current key's - * tar only, sweeping every other one on - * publish. `SUPABASE_SHADOW_CACHE` is ON by default; `false`/`0` opts out. + * snapshot fails the run — see `legacyExportShadowBaseline`); tars live under the global + * `${SUPABASE_HOME}/cache/shadow-baseline/` (shared across worktrees with the same settings), + * with LRU (keep 8) + 14-day mtime TTL retention. `SUPABASE_SHADOW_CACHE` is ON by default; + * `false`/`0` opts out. */ import { createHash } from "node:crypto"; @@ -39,7 +40,7 @@ import { import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import type { LegacyPgConnInput } from "../legacy-db-connection.service.ts"; import { legacyGetRegistryImageUrl } from "../legacy-docker-registry.ts"; -import { legacyPgDeltaTempPath } from "../legacy-pgdelta.paths.ts"; +import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; import { LEGACY_POSTGRES_DEFAULT_ROOT_KEY } from "../legacy-local-config-values.ts"; import { LEGACY_START_REVOKE_API_PRIVILEGES_SQL } from "./db-setup.ts"; @@ -254,7 +255,7 @@ const legacyTriStateToken = (value: Option.Option) => /** * The cache key: a 16-hex-char (64-bit) sha256 prefix over a fixed field order. 64 bits is - * ample for a per-project local cache whose only cost of a collision would be a wrong baseline + * ample for a per-settings global cache whose only cost of a collision would be a wrong baseline * — and every genuinely divergent input is in the payload, so a collision needs an actual hash * collision, not a missed field. Short enough to read in a filename. */ @@ -423,31 +424,74 @@ const legacyResolveShadowCacheKeyInputs = ( // The tar artifact // --------------------------------------------------------------------------- -/** Filename prefix shared by every key's snapshot — the handle the stale-key sweep enumerates by. */ +/** Filename prefix shared by every key's snapshot — the handle the retention sweep enumerates by. */ const LEGACY_SHADOW_BASELINE_TAR_PREFIX = "shadow-baseline-"; const LEGACY_SHADOW_BASELINE_TAR_SUFFIX = ".tar"; -/** `shadow-baseline-.tar` under `supabase/.temp/pgdelta/` — one ~90MB file per key. */ +/** Cap on published tars in the global cache (~90MB each → ~720MB). */ +export const LEGACY_SHADOW_BASELINE_KEEP = 8; + +/** Drop published tars whose mtime is older than this (warm hits refresh mtime). */ +export const LEGACY_SHADOW_BASELINE_MAX_AGE_MS = 14 * 24 * 60 * 60 * 1000; + +/** + * `shadow-baseline-.tar` under `${SUPABASE_HOME}/cache/shadow-baseline/` — one ~90MB file + * per settings key, shared across worktrees. + */ export function legacyShadowBaselineTarFileName(key: string): string { return `${LEGACY_SHADOW_BASELINE_TAR_PREFIX}${key}${LEGACY_SHADOW_BASELINE_TAR_SUFFIX}`; } /** - * Whether `fileName` is a baseline snapshot belonging to some OTHER key — i.e. one the "current - * key only" retention rule removes when a new key's snapshot is published. Pure, so the retention - * rule is unit-testable without a filesystem, and deliberately conservative: only files matching - * this module's own prefix AND suffix are ever candidates, so nothing else in - * `supabase/.temp/pgdelta/` (catalog snapshots, debug bundles) can be swept by accident. + * Whether `fileName` is a published baseline snapshot (`shadow-baseline-.tar`). Pure and + * deliberately conservative: only this module's own prefix AND suffix, so partials + * (`…tar..partial`) and any unrelated file in the cache dir are never eviction candidates. */ -export function legacyIsStaleShadowBaselineTar(fileName: string, key: string): boolean { +export function legacyIsShadowBaselineTar(fileName: string): boolean { return ( fileName.startsWith(LEGACY_SHADOW_BASELINE_TAR_PREFIX) && fileName.endsWith(LEGACY_SHADOW_BASELINE_TAR_SUFFIX) && - fileName !== legacyShadowBaselineTarFileName(key) + fileName.length === + LEGACY_SHADOW_BASELINE_TAR_PREFIX.length + 16 + LEGACY_SHADOW_BASELINE_TAR_SUFFIX.length && + /^shadow-baseline-[0-9a-f]{16}\.tar$/u.test(fileName) ); } +/** One published tar as the LRU/TTL rule sees it — name + mtime, no filesystem. */ +export interface LegacyShadowBaselineTarEntry { + readonly fileName: string; + readonly mtimeMs: number; +} + +export interface LegacyShadowBaselineRetentionOpts { + readonly keep?: number; + readonly maxAgeMs?: number; +} + +/** + * Pure LRU + age eviction: drop every published tar older than `maxAgeMs`, then among the + * survivors keep the newest `keep` by mtime. Never returns non-tar names (catalogs, partials). + * Unit-testable without a filesystem. + */ +export function legacyShadowBaselineTarsToEvict( + entries: ReadonlyArray, + now: number, + opts: LegacyShadowBaselineRetentionOpts = {}, +): ReadonlyArray { + const keep = opts.keep ?? LEGACY_SHADOW_BASELINE_KEEP; + const maxAgeMs = opts.maxAgeMs ?? LEGACY_SHADOW_BASELINE_MAX_AGE_MS; + const candidates = entries.filter((entry) => legacyIsShadowBaselineTar(entry.fileName)); + const aged = new Set( + candidates.filter((entry) => now - entry.mtimeMs > maxAgeMs).map((entry) => entry.fileName), + ); + const newestFirst = candidates + .filter((entry) => !aged.has(entry.fileName)) + .sort((left, right) => right.mtimeMs - left.mtimeMs); + const overCap = newestFirst.slice(keep).map((entry) => entry.fileName); + return [...aged, ...overCap]; +} + /** Best-effort removal — a leftover tar only ever costs disk, never correctness. */ const legacyForgetShadowBaselineTar = ( fs: FileSystem.FileSystem, @@ -476,24 +520,23 @@ const LEGACY_SHADOW_PARTIAL_ABANDON_MS = 60 * 60 * 1000; * Removes abandoned `.partial` temp files (see {@link legacyIsShadowBaselinePartial}) — the one * artifact a SIGKILLed/crashed cold export leaves behind that nothing else ever cleans: later * runs use their own pid in the temp name, and the tar retention sweep deliberately ignores - * `.partial` names (review: Codex on #6184). Runs before every cold export (so orphans cannot - * accumulate across repeatedly killed provisions) — best-effort throughout, like every other - * sweep here. + * `.partial` names (review: Codex on #6184). Runs before every cold export and on warm hits (so + * orphans cannot accumulate once every later run goes warm) — best-effort throughout. */ const legacySweepAbandonedShadowBaselinePartials = ( input: LegacyShadowSetupInput, ): Effect.Effect => Effect.gen(function* () { - const tempDir = legacyPgDeltaTempPath(input.path, input.workdir); + const cacheDir = legacyShadowBaselineCacheDir(input.path); const entries = yield* input.fs - .readDirectory(tempDir) + .readDirectory(cacheDir) .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); const now = yield* Clock.currentTimeMillis; yield* Effect.forEach( entries.filter(legacyIsShadowBaselinePartial), (entry) => Effect.gen(function* () { - const filePath = input.path.join(tempDir, entry); + const filePath = input.path.join(cacheDir, entry); const info = yield* input.fs.stat(filePath); const mtime = Option.getOrUndefined(info.mtime); if (mtime !== undefined && now - mtime.getTime() > LEGACY_SHADOW_PARTIAL_ABANDON_MS) { @@ -505,26 +548,47 @@ const legacySweepAbandonedShadowBaselinePartials = ( }); /** - * Applies the "current key only" retention rule: every `shadow-baseline-*.tar` in the temp - * directory whose key differs from `key` is removed. Best-effort throughout — a snapshot that - * cannot be swept costs ~90MB of disk, so it must never fail the export that just succeeded. + * Applies the global-cache LRU + TTL retention rule (see {@link legacyShadowBaselineTarsToEvict}). + * Best-effort throughout — a snapshot that cannot be swept costs ~90MB of disk, so it must never + * fail the export or warm hit that just succeeded. */ -const legacySweepStaleShadowBaselineTars = ( +const legacySweepShadowBaselineRetention = ( input: LegacyShadowSetupInput, - key: string, ): Effect.Effect => Effect.gen(function* () { - const tempDir = legacyPgDeltaTempPath(input.path, input.workdir); - const entries = yield* input.fs - .readDirectory(tempDir) + const cacheDir = legacyShadowBaselineCacheDir(input.path); + const names = yield* input.fs + .readDirectory(cacheDir) .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); + const now = yield* Clock.currentTimeMillis; + const entries: Array = []; + for (const fileName of names) { + if (!legacyIsShadowBaselineTar(fileName)) continue; + const info = yield* input.fs + .stat(input.path.join(cacheDir, fileName)) + .pipe(Effect.orElseSucceed(() => undefined)); + if (info === undefined) continue; + const mtime = Option.getOrUndefined(info.mtime); + if (mtime === undefined) continue; + entries.push({ fileName, mtimeMs: mtime.getTime() }); + } yield* Effect.forEach( - entries.filter((entry) => legacyIsStaleShadowBaselineTar(entry, key)), - (entry) => legacyForgetShadowBaselineTar(input.fs, input.path.join(tempDir, entry)), + legacyShadowBaselineTarsToEvict(entries, now), + (fileName) => legacyForgetShadowBaselineTar(input.fs, input.path.join(cacheDir, fileName)), { discard: true }, ); }); +/** Refresh mtime on a warm hit so frequently used keys survive LRU/TTL. Best-effort. */ +const legacyTouchShadowBaselineTar = ( + fs: FileSystem.FileSystem, + tarPath: string, +): Effect.Effect => + Effect.gen(function* () { + const now = new Date(yield* Clock.currentTimeMillis); + yield* fs.utimes(tarPath, now, now); + }).pipe(Effect.orElseSucceed(() => undefined)); + // --------------------------------------------------------------------------- // Container primitives the cache adds on top of `shadow-database.ts` // --------------------------------------------------------------------------- @@ -589,24 +653,23 @@ const legacyAwaitShadowReady = ( // --------------------------------------------------------------------------- /** - * Ensures the tar's temp directory exists, delegates the actual export to + * Ensures the tar's global cache directory exists, delegates the actual export to * {@link legacyExportPgDataTar} (`pgdata-snapshot.ts` — see that function's own doc comment for - * the atomic-publish mechanics), then applies the "current key only" retention rule. + * the atomic-publish mechanics), then applies the LRU + TTL retention rule. */ const legacyWriteShadowBaselineTar = ( spawner: Spawner, input: LegacyShadowSetupInput, - key: string, tarPath: string, containerId: string, ): Effect.Effect => Effect.gen(function* () { - const tempDir = legacyPgDeltaTempPath(input.path, input.workdir); + const cacheDir = legacyShadowBaselineCacheDir(input.path); yield* input.fs - .makeDirectory(tempDir, { recursive: true }) + .makeDirectory(cacheDir, { recursive: true, mode: 0o700 }) .pipe( Effect.mapError((cause) => - legacyShadowCacheUnavailable(`failed to create ${tempDir}: ${cause.message}`), + legacyShadowCacheUnavailable(`failed to create ${cacheDir}: ${cause.message}`), ), ); yield* legacySweepAbandonedShadowBaselinePartials(input); @@ -615,7 +678,7 @@ const legacyWriteShadowBaselineTar = ( legacyShadowCacheUnavailable(cause.reason), ), ); - yield* legacySweepStaleShadowBaselineTars(input, key); + yield* legacySweepShadowBaselineRetention(input); }); /** @@ -653,7 +716,7 @@ const legacyExportShadowBaseline = ( const exported = yield* Effect.result( Effect.gen(function* () { yield* legacyShadowContainerVerb(spawner, "stop", containerId); - yield* legacyWriteShadowBaselineTar(spawner, input, key, tarPath, containerId); + yield* legacyWriteShadowBaselineTar(spawner, input, tarPath, containerId); }), ); // Run-critical phase: the shadow must be back up and answering before this step reports @@ -849,18 +912,20 @@ export const legacyAcquireShadowDatabase = ( if (Option.isNone(keyInputs)) return yield* legacyUncachedShadow(spawner, input); const key = legacyShadowCacheKey(keyInputs.value); const tarPath = input.path.join( - legacyPgDeltaTempPath(input.path, input.workdir), + legacyShadowBaselineCacheDir(input.path), legacyShadowBaselineTarFileName(key), ); const cached = yield* input.fs.exists(tarPath).pipe(Effect.orElseSucceed(() => false)); if (!cached) return yield* legacyColdCachedShadow(spawner, input, key, tarPath); - // Warm hits sweep abandoned partials too: a killed concurrent writer's leftover would - // otherwise persist indefinitely once every later run goes warm, since the cold export's own - // sweep never runs again (review: Codex on #6184). Best-effort and cheap (one readdir + - // per-candidate stat). + // Warm hits refresh mtime (so frequently used keys survive LRU/TTL) and sweep abandoned + // partials — a killed concurrent writer's leftover would otherwise persist indefinitely once + // every later run goes warm, since the cold export's own sweep never runs again (review: + // Codex on #6184). Best-effort and cheap. + yield* legacyTouchShadowBaselineTar(input.fs, tarPath); yield* legacySweepAbandonedShadowBaselinePartials(input); + yield* legacySweepShadowBaselineRetention(input); return yield* legacyWarmShadow(spawner, input, tarPath).pipe( Effect.catch((cause) => diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index 4108a0dcb0..4e5ec8506a 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -2,9 +2,12 @@ import { describe, expect, it } from "@effect/vitest"; import { Option } from "effect"; import { + LEGACY_SHADOW_BASELINE_KEEP, + LEGACY_SHADOW_BASELINE_MAX_AGE_MS, legacyIsShadowBaselinePartial, - legacyIsStaleShadowBaselineTar, + legacyIsShadowBaselineTar, legacyShadowBaselineTarFileName, + legacyShadowBaselineTarsToEvict, legacyShadowCacheEnabled, legacyShadowCacheKey, type LegacyShadowCacheKeyInputs, @@ -308,20 +311,30 @@ describe("legacyShadowCacheKey", () => { }); describe("shadow baseline tar retention", () => { const key = "0123456789abcdef"; + const now = 1_700_000_000_000; - it("keeps the current key's snapshot and sweeps every other key's", () => { - expect(legacyIsStaleShadowBaselineTar(legacyShadowBaselineTarFileName(key), key)).toBe(false); - expect(legacyIsStaleShadowBaselineTar("shadow-baseline-fedcba9876543210.tar", key)).toBe(true); + it("recognizes only this module's own published snapshots", () => { + expect(legacyIsShadowBaselineTar(legacyShadowBaselineTarFileName(key))).toBe(true); + for (const other of [ + "catalog-local-migrations-abc-123.json", + "shadow-baseline.tar", + `shadow-baseline-${key}.tar.4242.partial`, + `shadow-cache-${key}.json`, + "pgdelta-debug.zip", + // Wrong key length / non-hex. + "shadow-baseline-0123456789abcde.tar", + "shadow-baseline-0123456789abcdefg.tar", + "shadow-baseline-0123456789ABCDEF.tar", + ]) { + expect(legacyIsShadowBaselineTar(other), other).toBe(false); + } }); it("recognizes only this module's own partial temp files as abandoned-sweep candidates", () => { expect(legacyIsShadowBaselinePartial(`shadow-baseline-${key}.tar.4242.partial`)).toBe(true); for (const other of [ - // The published artifact and other keys' artifacts are the tar sweep's business, not this - // predicate's. legacyShadowBaselineTarFileName(key), "shadow-baseline-fedcba9876543210.tar", - // Anything not exactly `<16-hex>.tar..partial` is left alone. "shadow-baseline.tar.4242.partial", `shadow-baseline-${key}.tar.partial`, `shadow-baseline-${key}.tar.4242.partial.bak`, @@ -331,18 +344,56 @@ describe("shadow baseline tar retention", () => { } }); - it("never sweeps a file that is not one of this module's own snapshots", () => { - // `supabase/.temp/pgdelta/` is shared with the pg-delta catalog cache and its debug bundles — - // the retention rule must be blind to everything but its own prefix AND suffix, or a `db diff` - // would delete the catalog cache it depends on. - for (const other of [ - "catalog-local-migrations-abc-123.json", - "shadow-baseline.tar", - `shadow-baseline-${key}.tar.4242.partial`, - `shadow-cache-${key}.json`, - "pgdelta-debug.zip", - ]) { - expect(legacyIsStaleShadowBaselineTar(other, key), other).toBe(false); - } + it("evicts aged tars and keeps the newest N among survivors", () => { + const aged = now - LEGACY_SHADOW_BASELINE_MAX_AGE_MS - 1; + const fresh = now - 1_000; + const entries = [ + { fileName: legacyShadowBaselineTarFileName("aaaaaaaaaaaaaaaa"), mtimeMs: aged }, + { fileName: legacyShadowBaselineTarFileName("bbbbbbbbbbbbbbbb"), mtimeMs: fresh }, + { fileName: legacyShadowBaselineTarFileName("cccccccccccccccc"), mtimeMs: fresh - 10 }, + { fileName: "catalog-abc.json", mtimeMs: aged }, + { fileName: `shadow-baseline-${key}.tar.1.partial`, mtimeMs: aged }, + ]; + expect( + legacyShadowBaselineTarsToEvict(entries, now, { + keep: 1, + maxAgeMs: LEGACY_SHADOW_BASELINE_MAX_AGE_MS, + }), + ).toEqual([ + legacyShadowBaselineTarFileName("aaaaaaaaaaaaaaaa"), + legacyShadowBaselineTarFileName("cccccccccccccccc"), + ]); + }); + + it("evicts the oldest beyond the keep cap when all are fresh", () => { + const entries = Array.from({ length: LEGACY_SHADOW_BASELINE_KEEP + 2 }, (_, index) => ({ + fileName: legacyShadowBaselineTarFileName(index.toString(16).padStart(16, "0")), + mtimeMs: now - index * 1_000, + })); + const evicted = legacyShadowBaselineTarsToEvict(entries, now); + expect(evicted).toHaveLength(2); + expect(evicted).toContain( + legacyShadowBaselineTarFileName(LEGACY_SHADOW_BASELINE_KEEP.toString(16).padStart(16, "0")), + ); + expect(evicted).toContain( + legacyShadowBaselineTarFileName( + (LEGACY_SHADOW_BASELINE_KEEP + 1).toString(16).padStart(16, "0"), + ), + ); + }); + + it("never returns a file that is not one of this module's own snapshots", () => { + expect( + legacyShadowBaselineTarsToEvict( + [ + { fileName: "catalog-local-migrations-abc-123.json", mtimeMs: 0 }, + { fileName: "shadow-baseline.tar", mtimeMs: 0 }, + { fileName: `shadow-baseline-${key}.tar.4242.partial`, mtimeMs: 0 }, + { fileName: `shadow-cache-${key}.json`, mtimeMs: 0 }, + { fileName: "pgdelta-debug.zip", mtimeMs: 0 }, + ], + now, + ), + ).toEqual([]); }); }); diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts index 58291b6b8a..fdb2912c92 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.paths.ts @@ -1,16 +1,40 @@ /** - * The one on-disk location every pg-delta-adjacent cache/snapshot artefact lives under. + * On-disk locations for pg-delta-adjacent cache/snapshot artefacts. * * Split out of `legacy-pgdelta.cache.ts` (which owns the catalog cache's keys AND its * shadow-provisioning resolution path) so `db-bootstrap/shadow-cache.ts` — the warm * shadow-container cache, which `legacy-pgdelta.cache.ts` itself consumes for its own - * shadow provisioning — can reach the same directory without an import cycle between - * the two. + * shadow provisioning — can reach path helpers without an import cycle between the two. + * + * Two roots: + * - {@link legacyPgDeltaTempPath}: project-local (`supabase/.temp/pgdelta`) — catalog + * snapshots and debug bundles (Go-shared, workspace-mounted). + * - {@link legacyShadowBaselineCacheDir}: global under `SUPABASE_HOME` — the shadow + * baseline PGDATA tars, shared across worktrees with the same settings. */ +import { homedir } from "node:os"; + import type { Path } from "effect"; -/** `supabase/.temp/pgdelta` — where catalog snapshots, debug bundles, and the shadow baseline cache's PGDATA tars live (`declarative.go:44`). */ +import { resolveSupabaseHome } from "../../shared/config/supabase-home.ts"; + +/** `supabase/.temp/pgdelta` — catalog snapshots and debug bundles (`declarative.go:44`). */ export function legacyPgDeltaTempPath(path: Path.Path, workdir: string): string { return path.join(workdir, "supabase", ".temp", "pgdelta"); } + +/** + * Global shadow-baseline cache directory: + * `${SUPABASE_HOME}/cache/shadow-baseline` (default `~/.supabase/cache/shadow-baseline`). + * + * Pure: callers may pass `env`/`homeDir` for tests; production uses `process.env` and + * `os.homedir()`. + */ +export function legacyShadowBaselineCacheDir( + path: Path.Path, + env: Readonly> = process.env, + homeDir: string = homedir(), +): string { + return path.join(resolveSupabaseHome(env, homeDir), "cache", "shadow-baseline"); +} From 4be4d0c062f4959a4c6e4d08a1bed9e773b70e18 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 14:36:55 +0200 Subject: [PATCH 64/82] fix(cli): warm-cache pg-delta next shadows Wire next's scoped shadows through the baseline snapshot acquire, drop the published host port from the cache key so ephemeral ports still hit, and honor --no-cache on sync. --- apps/cli/docs/go-cli-divergences.md | 5 +- .../schema/declarative/sync/SIDE_EFFECTS.md | 64 ++++++++-------- ...elta-engine.next.layer.integration.test.ts | 43 +++++++++-- .../legacy-pgdelta-engine.next.layer.ts | 1 + .../legacy-pgdelta-next-shadow.layer.ts | 75 +++++++++++++------ .../legacy-pgdelta-next-shadow.service.ts | 5 ++ .../shadow-cache.integration.test.ts | 40 ++++++++-- .../shared/db-bootstrap/shadow-cache.ts | 17 ++--- .../db-bootstrap/shadow-cache.unit.test.ts | 2 - .../shared/db-bootstrap/shadow-database.ts | 35 +++++++-- .../db-bootstrap/shadow-database.unit.test.ts | 58 ++++++++++++++ .../src/legacy/shared/legacy-pgdelta.cache.ts | 50 ++++++++----- docs/roadmap/pg-delta-next-follow-ups.md | 22 +++--- 13 files changed, 300 insertions(+), 117 deletions(-) diff --git a/apps/cli/docs/go-cli-divergences.md b/apps/cli/docs/go-cli-divergences.md index 2e0665b74d..75e0414155 100644 --- a/apps/cli/docs/go-cli-divergences.md +++ b/apps/cli/docs/go-cli-divergences.md @@ -84,7 +84,10 @@ These commands exist in the TS CLI today but have no direct top-level equivalent `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides the root; LRU keep-8 + 14-day mtime TTL, shared across worktrees with the same settings) and restored into a - fresh container on later runs, cutting shadow provisioning from ~15s to a few seconds. TS-only, + fresh container on later runs, cutting shadow provisioning from ~15s to a few seconds. Covers + migra/`db pull` via `legacyWithShadowDatabase`, and the bundled pg-delta next sync/diff + shadows via `legacyAcquireShadowDatabase` (ephemeral host ports are not part of the cache + key — they are not baked into PGDATA). TS-only, default ON; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (ambient env or project dotenv), `sync --no-cache` bypasses it per-invocation, and `SUPABASE_SHADOW_DEBUG=1` prints stderr-only phase timings. OrioleDB clusters and PG <= 14 are cache-ineligible (external S3 state and mid-session diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 2a00c875b4..1dbe51cdd2 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -9,7 +9,8 @@ there is no automatic fallback. Coverage gaps warn; `--strict-coverage` makes them fatal, while `PGDELTA_DEBUG` writes diagnostic JSON under `supabase/.temp/pgdelta/v2/debug//`. Bundled output may use different SQL and ordered transaction-aware files but must apply and converge. `--no-cache` -affects only the legacy opt-out. The bundled formatter defaults to lowercase SQL +bypasses the bundled engine's shadow baseline cache and the legacy opt-out's +catalog + snapshot caches. The bundled formatter defaults to lowercase SQL at width 180; config overrides it, and JSON `null` disables formatting without disabling safe compaction. @@ -25,25 +26,25 @@ disabling safe compaction. | `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | | `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | | `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit on a migrations-catalog miss — snapshot streamed into the fresh shadow container before it starts (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — bundled-engine migrations/declarative shadows, and the legacy opt-out's migrations-catalog miss (`SUPABASE_HOME` overrides the `~/.supabase` root) | | `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | ## Files Written -| Path | Format | When | -| --------------------------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | -| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision on a migrations-catalog cache miss only (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, LRU keep-8 + 14-day mtime TTL (`SUPABASE_HOME` overrides the root) | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | +| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision — bundled-engine migrations/declarative shadows, and the legacy opt-out's migrations-catalog miss (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, LRU keep-8 + 14-day mtime TTL (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | ## Subprocesses / Containers | What | When | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| Two natively-provisioned shadows: migrated source and declarative target | bundled engine | +| Two natively-provisioned shadows (migrated source + declarative target) via `legacyAcquireShadowDatabase` — ephemeral host ports, settings-keyed global baseline cache | bundled engine | | Natively-provisioned shadow Postgres container (`legacyCreateShadowDatabase`/`legacyPrepareShadowSource`) + native migrate; the catalog itself is exported via edge-runtime | legacy opt-out, migrations-catalog cache miss | | Natively-provisioned shadow Postgres container (platform-baseline setup via one-shot auth/storage/realtime migrate jobs, then the declarative directory applied via the pg-delta edge-runtime apply script) → catalog export | legacy opt-out, declarative-catalog cache miss | | Edge-runtime container running the pg-delta diff and, on a catalog cache miss, catalog-export/declarative-apply scripts | legacy opt-out | @@ -128,24 +129,27 @@ existing SQL or creates an export manifest. ### Shadow baseline cache (`SUPABASE_SHADOW_CACHE`, default ON) -The migrations-catalog shadow this command provisions on a cache miss goes through -`legacyGetMigrationsCatalogRef` -> `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), the same -`legacyWithShadowDatabase` seam `db diff`/`db pull` use, so it inherits the whole lifecycle: ON by -default, `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (honored from the ambient env AND the -project's dotenv, e.g. `supabase/.env`), and `--no-cache` bypasses it for that invocation (the -flag promises fresh shadow setup, so it disables the snapshot cache along with the catalog -cache); a warm hit skips the platform baseline and therefore the -`Initialising schema...` progress line. Artifact: +The bundled (pg-delta next) engine provisions both plan shadows through +`legacyAcquireShadowDatabase` (`legacy-pgdelta-next-shadow.layer.ts`): ON by default, +`SUPABASE_SHADOW_CACHE=false`/`=0` opts out (ambient env or project dotenv), and `--no-cache` +bypasses restore and publish for that invocation. Next allocates an ephemeral host port per +shadow; the cache key hashes the cluster recipe, not the published port, so worktrees and +repeated syncs with the same settings share a warm hit. A warm hit skips the platform baseline +on both the migrated and declarative shadows (`legacyMigrateNextShadowDatabase` / +`legacySetupShadowDatabase` are baseline-state-aware). Artifact: `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides -the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster; shared across -worktrees with the same settings; retention is LRU (keep 8) + 14-day mtime TTL (warm hits refresh -mtime). Container -lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on -release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a -cold export failure only warns and leaves the run uncached (one exception: a shadow that fails -to come back up after the snapshot fails the run rather than reporting a false success). See `shared/db-bootstrap/ -shadow-cache.ts`'s doc comment for the mechanics. The DECLARATIVE-catalog shadow (and `generate`'s -baseline-catalog shadow) is NOT cached: its provision runs the platform baseline via -`legacySetupShadowDatabase`, which is not baseline-state-aware, so those callers pass an -unconditional bypass — making them warm-aware is a recorded follow-up +the root); shared across worktrees with the same settings; retention is LRU (keep 8) + 14-day +mtime TTL (warm hits refresh mtime). Container lifecycle is identical to the uncached path +except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the +command — a warm-path anomaly cold-provisions instead, a cold export failure only warns and +leaves the run uncached (one exception: a shadow that fails to come back up after the snapshot +fails the run rather than reporting a false success). See `shared/db-bootstrap/shadow-cache.ts`. + +Under the legacy opt-out, the migrations-catalog shadow on a cache miss goes through +`legacyGetMigrationsCatalogRef` -> `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), the same +`legacyWithShadowDatabase` seam `db diff`/`db pull` use, and `--no-cache` bypasses that snapshot +cache along with the catalog cache. The DECLARATIVE-catalog shadow (and `generate`'s +baseline-catalog shadow) now threads baseline state through `legacySetupShadowDatabase` so a +warm hit does not double-apply the baseline; swapping their `legacyWaitForHealthyServices` +docker-health wait for `legacyWaitForShadowReady` is a recorded follow-up (`docs/roadmap/pg-delta-next-follow-ups.md`). diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts index bddfaf7edd..d73968cf6c 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts @@ -61,7 +61,7 @@ const toml: LegacyDbTomlValues = { }; function setup() { - const state = { migrations: 0, plan: 0 }; + const state = { migrations: 0, plan: 0, planBypassCache: undefined as boolean | undefined }; const shadow = Layer.succeed(LegacyPgDeltaNextShadow, { provisionMigrations: () => Effect.sync(() => { @@ -71,9 +71,10 @@ function setup() { Effect.fail(new LegacyDeclarativeShadowDbError({ message: "stop after routing" })), ), ), - provisionPlan: () => + provisionPlan: (opts) => Effect.sync(() => { state.plan += 1; + state.planBypassCache = opts.bypassCache; }).pipe( Effect.andThen( Effect.fail(new LegacyDeclarativeShadowDbError({ message: "stop after routing" })), @@ -124,7 +125,8 @@ describe("pg-delta next shadow selection", () => { }) .pipe(Effect.exit); - expect(state).toEqual({ migrations: 0, plan: 0 }); + expect(state.migrations).toBe(0); + expect(state.plan).toBe(0); }).pipe(Effect.provide(layer)); }); @@ -145,7 +147,8 @@ describe("pg-delta next shadow selection", () => { }) .pipe(Effect.exit); - expect(state).toEqual({ migrations: 1, plan: 0 }); + expect(state.migrations).toBe(1); + expect(state.plan).toBe(0); }).pipe(Effect.provide(layer)); }); @@ -172,7 +175,37 @@ describe("pg-delta next shadow selection", () => { }) .pipe(Effect.exit); - expect(state).toEqual({ migrations: 0, plan: 1 }); + expect(state.migrations).toBe(0); + expect(state.plan).toBe(1); + expect(state.planBypassCache).toBeUndefined(); + }).pipe(Effect.provide(layer)); + }); + + it.effect("forwards --no-cache as bypassCache on the declarative plan shadows", () => { + const { state, layer } = setup(); + return Effect.gen(function* () { + const engine = yield* LegacyPgDeltaEngine; + yield* engine + .planDeclarativeSchema({ + ...common, + toml, + files: [{ name: "schema.sql", sql: "create table example(id int);" }], + noCache: true, + setupInputs: { + image: "postgres:17", + majorVersion: 17, + authEnabled: true, + storageEnabled: true, + realtimeEnabled: true, + autoExpose: false, + vaultNames: [], + rolesSql: "", + }, + }) + .pipe(Effect.exit); + + expect(state.plan).toBe(1); + expect(state.planBypassCache).toBe(true); }).pipe(Effect.provide(layer)); }); }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index ee60a9a4f9..5628aa04b1 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -322,6 +322,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( context: input.context, toml: input.toml, ...(input.projectRef !== undefined ? { projectRef: input.projectRef } : {}), + ...(input.noCache ? { bypassCache: true } : {}), }); const migrations = parseLegacyConnectionString(shadow.migrationsUrl); const declarative = parseLegacyConnectionString(shadow.declarativeUrl); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index 8319962425..1d71243b2a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -19,15 +19,18 @@ import { legacyBuildLocalDbContainerInputs, type LegacyLocalDbContainerInputs, } from "../../../shared/db-bootstrap/local-container-inputs.ts"; -import { legacyWaitForHealthyServices } from "../../../shared/db-bootstrap/health-check.ts"; +import { legacyWaitForShadowReady } from "../../../shared/db-bootstrap/health-check.ts"; +import { + legacyAcquireShadowDatabase, + type LegacyShadowAcquiredHandle, + type LegacyShadowCacheOpts, +} from "../../../shared/db-bootstrap/shadow-cache.ts"; import { legacyConnectShadowDatabase, - legacyCreateShadowDatabase, legacyMigrateNextShadowDatabase, legacyRemoveShadowDatabase, legacyShadowRunInputFromLocalContainerInputs, legacySetupShadowDatabase, - type LegacyShadowDatabaseHandle, } from "../../../shared/db-bootstrap/shadow-database.ts"; import { ChildProcessSpawner } from "effect/unstable/process"; import type { ChildProcessSpawner as ChildProcessSpawnerType } from "effect/unstable/process/ChildProcessSpawner"; @@ -98,7 +101,7 @@ export const legacyPreparePgDeltaNextDeclarativeBaseline = Effect.fnUntraced(fun yield* session.exec('DROP EXTENSION IF EXISTS "uuid-ossp"'); }); -const setupRunInput = (input: NativeShadowInput, handle: LegacyShadowDatabaseHandle) => ({ +const setupRunInput = (input: NativeShadowInput, handle: LegacyShadowAcquiredHandle) => ({ fs: input.base.fs, path: input.base.path, workdir: input.base.workdir, @@ -198,34 +201,54 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( ), }); - const acquireShadow = (input: NativeShadowInput) => - Effect.acquireRelease(legacyCreateShadowDatabase(input.spawner, input.base), (handle) => - legacyRemoveShadowDatabase(input.spawner, handle.containerId).pipe( - Effect.provideService(Output, output), - ), + /** + * Cache-aware acquire, released when the current scope closes — next returns a URL the + * engine keeps using after provision, so this cannot be `legacyWithShadowDatabase` + * (that wrapper removes the container when `use` returns). + */ + const acquireShadow = (input: NativeShadowInput, opts: LegacyShadowCacheOpts) => + Effect.acquireRelease( + legacyAcquireShadowDatabase(input.spawner, input.base, opts), + (handle) => + legacyRemoveShadowDatabase(input.spawner, handle.containerId).pipe( + Effect.provideService(Output, output), + ), ); - const provisionMigrations = (input: NativeShadowInput) => - Effect.gen(function* () { - const handle = yield* acquireShadow(input); - yield* legacyWaitForHealthyServices(input.spawner, [handle.containerId], { + const awaitShadowReady = (input: NativeShadowInput, handle: LegacyShadowAcquiredHandle) => + legacyWaitForShadowReady( + input.spawner, + handle.containerId, + { + host: input.base.hostname, + port: input.base.shadowPort, + user: "postgres", + password: input.base.password, + database: "postgres", + }, + { timeoutSeconds: input.base.healthTimeoutSeconds, - }); + image: input.base.image, + }, + ); + + const provisionMigrations = (input: NativeShadowInput, opts: LegacyShadowCacheOpts) => + Effect.gen(function* () { + const handle = yield* acquireShadow(input, opts); + yield* awaitShadowReady(input, handle); const setup = setupRunInput(input, handle); - yield* legacyMigrateNextShadowDatabase(input.spawner, setup); + yield* legacyMigrateNextShadowDatabase(input.spawner, setup, handle); return { migrationsUrl: legacyToPostgresURL(setup.connConfig), } satisfies LegacyPgDeltaNextMigrationsShadow; }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); - const provisionDeclarative = (input: NativeShadowInput) => + const provisionDeclarative = (input: NativeShadowInput, opts: LegacyShadowCacheOpts) => Effect.gen(function* () { - const handle = yield* acquireShadow(input); - yield* legacyWaitForHealthyServices(input.spawner, [handle.containerId], { - timeoutSeconds: input.base.healthTimeoutSeconds, - }); + const handle = yield* acquireShadow(input, opts); + yield* awaitShadowReady(input, handle); const setup = setupRunInput(input, handle); - yield* legacySetupShadowDatabase(input.spawner, setup, { webhooks: "disabled" }); + yield* legacySetupShadowDatabase(input.spawner, setup, { webhooks: "disabled" }, handle); yield* Effect.scoped( Effect.gen(function* () { const session = yield* legacyConnectShadowDatabase(setup.connConfig); @@ -238,13 +261,16 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( return legacyToPostgresURL(setup.connConfig); }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); + const cacheOpts = (opts: LegacyPgDeltaNextShadowInput): LegacyShadowCacheOpts => + opts.bypassCache === true ? { bypassCache: true } : {}; + return LegacyPgDeltaNextShadow.of({ provisionMigrations: (opts) => Effect.gen(function* () { const port = yield* nextPort(); const built = yield* buildNativeBase(opts); const input = buildNativeInput(opts, built, port); - return yield* provisionMigrations(input); + return yield* provisionMigrations(input, cacheOpts(opts)); }).pipe(Effect.mapError(nextShadowError)), provisionPlan: (opts) => Effect.gen(function* () { @@ -253,8 +279,9 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( const built = yield* buildNativeBase(opts); const migrationsInput = buildNativeInput(opts, built, migrationsPort); const declarativeInput = buildNativeInput(opts, built, declarativePort); - const migrations = yield* provisionMigrations(migrationsInput); - const declarativeUrl = yield* provisionDeclarative(declarativeInput); + const cache = cacheOpts(opts); + const migrations = yield* provisionMigrations(migrationsInput, cache); + const declarativeUrl = yield* provisionDeclarative(declarativeInput, cache); return { ...migrations, declarativeUrl, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts index c71e08311d..9eed6db23a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts @@ -20,6 +20,11 @@ export interface LegacyPgDeltaNextShadowInput { readonly context: LegacyPgDeltaContext; readonly toml: LegacyDbTomlValues; readonly projectRef?: string; + /** + * `db schema declarative sync --no-cache` (and generate's same flag): force a fresh + * shadow baseline instead of restoring/publishing the global snapshot cache. + */ + readonly bypassCache?: boolean; } interface LegacyPgDeltaNextShadowShape { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 597b0a295f..0ec72308b4 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -364,12 +364,12 @@ const shadowSetup = (): LegacyShadowDbSetupInput => ({ const shadowInput = ( fs: FileSystem.FileSystem, path: Path.Path, - overrides: { readonly shadowPort?: number } = {}, + overrides: { readonly shadowPort?: number; readonly jwtExpiry?: number } = {}, ): LegacyShadowSetupInput => ({ db: { major_version: 17, settings: {} }, experimental: defaultConfig.experimental, jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", - jwtExpiry: 3600, + jwtExpiry: overrides.jwtExpiry ?? 3600, networkId: "supabase_network_proj", image: "public.ecr.aws/supabase/postgres:17.4.1.030", configImage: "supabase/postgres:17.4.1.030", @@ -729,10 +729,10 @@ describe("legacyAcquireShadowDatabase", () => { const first = yield* soleTarName(fs, path); expect(first).toHaveLength(1); - // A changed baseline input (the shadow's own published port) is a different cluster / - // different key — both must coexist in the global cache (unlike the old project-local - // current-key-only sweep, which would have deleted the first). - const rekeyed = yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54399 })); + // A changed baked-in input (jwt expiry) is a different cluster / different key — both + // must coexist in the global cache (unlike the old project-local current-key-only sweep, + // which would have deleted the first). Host publish port is NOT a key input. + const rekeyed = yield* coldRun(docker, shadowInput(fs, path, { jwtExpiry: 7200 })); const both = yield* soleTarName(fs, path); expect(both).toHaveLength(2); expect(both).toContain(first[0]); @@ -744,7 +744,7 @@ describe("legacyAcquireShadowDatabase", () => { // Fill to the keep-cap + 1 with more distinct keys; the oldest (first) is evicted. for (let i = 0; i < LEGACY_SHADOW_BASELINE_KEEP - 1; i++) { - yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54400 + i })); + yield* coldRun(docker, shadowInput(fs, path, { jwtExpiry: 8000 + i })); } const afterCap = yield* soleTarName(fs, path); expect(afterCap).toHaveLength(LEGACY_SHADOW_BASELINE_KEEP); @@ -785,6 +785,32 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("a changed published host port is still a warm hit", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const cold = yield* coldRun(docker, shadowInput(fs, path, { shadowPort: 54320 })); + expect(cold.baselinePresent).toBe(false); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + // pg-delta next allocates an ephemeral host port per shadow; the published port is + // not in PGDATA, so a later run on a different port must restore the same tar. + const warm = yield* legacyAcquireShadowDatabase( + docker.spawner, + shadowInput(fs, path, { shadowPort: 54399 }), + ); + expect(warm.baselinePresent).toBe(true); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + yield* legacyRemoveShadowDatabase(docker.spawner, warm.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("a changed internal image registry is a different key, not a warm hit", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 8f5ece93e8..7f338a98ff 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -153,12 +153,6 @@ export interface LegacyShadowCacheKeyInputs { /** The resolved, full `supabase/postgres` image (tag included — a major version is not enough). */ readonly postgresImage: string; readonly majorVersion: number; - /** - * The shadow's published host port. Not baked into PGDATA itself, but it IS part of the - * container shape a snapshot is restored into, and a plan that changed it is a different plan — - * cheap to include, and it keeps the key a superset of everything the container carries. - */ - readonly shadowPort: number; readonly jwtSecret: string; readonly jwtExpiry: number; readonly rootKey: string; @@ -269,7 +263,9 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string const lines: Array = [ `postgres_image=${quoted(inputs.postgresImage)}`, `major_version=${inputs.majorVersion}`, - `shadow_port=${inputs.shadowPort}`, + // Host publish port is deliberately excluded: it is not baked into PGDATA, and + // pg-delta next allocates an ephemeral port per shadow. Hashing it would miss + // every warm hit on that path. Restore always uses the current run's port. `jwt_secret=${quoted(inputs.jwtSecret)}`, `jwt_expiry=${inputs.jwtExpiry}`, `root_key=${quoted(inputs.rootKey)}`, @@ -388,7 +384,6 @@ const legacyResolveShadowCacheKeyInputs = ( return Option.some({ postgresImage: input.image, majorVersion: input.db.major_version, - shadowPort: input.shadowPort, jwtSecret: input.jwtSecret, jwtExpiry: input.jwtExpiry, // The EFFECTIVE value, not the raw input: `legacyBuildShadowPostgresContainerSpec` @@ -866,8 +861,10 @@ const legacyWarmShadow = ( /** * `Effect.acquireUseRelease`'s `acquire` for every shadow-provisioning call site that runs the * platform baseline (`db diff`'s migra/pg-delta branch, `db pull`'s migration diff, - * `legacy-pgdelta.cache.ts`'s catalog export) — see {@link legacyWithShadowDatabase}, which is - * what those call sites actually use. + * `legacy-pgdelta.cache.ts`'s catalog export, and pg-delta next's scoped shadows) — see + * {@link legacyWithShadowDatabase} for the acquire/use/release wrapper, and + * `legacy-pgdelta-next-shadow.layer.ts` for the scoped `acquireRelease` form next uses so the + * container outlives provision (the engine keeps using the URL after this returns). * * With {@link LEGACY_SHADOW_CACHE_ENV} set to a falsey value this IS `legacyCreateShadowDatabase`, * byte for byte. Otherwise it resolves the cache key and either restores this key's snapshot into diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index 4e5ec8506a..f281514930 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -16,7 +16,6 @@ import { const baseKeyInputs = (): LegacyShadowCacheKeyInputs => ({ postgresImage: "public.ecr.aws/supabase/postgres:17.6.1.158", majorVersion: 17, - shadowPort: 54320, jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", jwtExpiry: 3600, rootKey: "d4dc5b6d4a1d6a10b2c1e5b6a7c8d9e0", @@ -85,7 +84,6 @@ describe("legacyShadowCacheKey", () => { }> = [ { label: "postgres image tag", inputs: { ...base, postgresImage: "postgres:17.6.1.159" } }, { label: "major version", inputs: { ...base, majorVersion: 15 } }, - { label: "shadow port", inputs: { ...base, shadowPort: 54321 } }, { label: "jwt secret", inputs: { ...base, jwtSecret: "other-secret" } }, { label: "jwt expiry", inputs: { ...base, jwtExpiry: 7200 } }, { label: "root key", inputs: { ...base, rootKey: "0000" } }, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts index ea3f0f20a9..c0c640894d 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts @@ -734,11 +734,17 @@ export const legacyBuildShadowSetupDatabaseInput = ( * connect error immediately, not pay for JWKS work first. The connection is closed once this * resolves (Go's `defer conn.Close(...)`), matching `Effect.scoped`'s finalizer running at the * end of this function rather than leaking a `Scope.Scope` requirement to the caller. + * + * `baseline` defaults to {@link LEGACY_SHADOW_BASELINE_COLD}, i.e. exactly the sequence above. + * A warm shadow-cache hit skips the prelude + `SetupDatabase` (the restored cluster already + * has them) and only recreates `contrib_regression`; a cache-enabled COLD provision snapshots + * between the baseline and the template, matching {@link migrateShadowDatabase}. */ export const legacySetupShadowDatabase = ( spawner: Spawner, input: LegacyShadowSetupRunInput, options: LegacySetupDatabaseOptions = {}, + baseline: LegacyShadowBaselineState = LEGACY_SHADOW_BASELINE_COLD, ): Effect.Effect< void, LegacyStartSetupLocalDatabaseError | LegacyShadowDbError | LegacyImagePrepullError | E, @@ -746,13 +752,30 @@ export const legacySetupShadowDatabase = ( > => Effect.scoped( Effect.gen(function* () { + if (!baseline.baselinePresent && baseline.snapshotRequired) { + yield* Effect.scoped( + Effect.gen(function* () { + const setupSession = yield* legacyConnectShadowDatabase(input.connConfig); + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput(input, setupSession, resolved), + options, + ); + }), + ); + yield* baseline.snapshotBaseline; + } const session = yield* legacyConnectShadowDatabase(input.connConfig); - const resolved = yield* legacyResolveDbSetupPrelude(input.setup); - yield* legacySetupShadowConn( - spawner, - legacyBuildShadowSetupDatabaseInput(input, session, resolved), - options, - ); + if (!baseline.baselinePresent && !baseline.snapshotRequired) { + const resolved = yield* legacyResolveDbSetupPrelude(input.setup); + yield* legacySetupDatabase( + spawner, + legacyBuildShadowSetupDatabaseInput(input, session, resolved), + options, + ); + } + yield* legacyCreateShadowTemplateDatabase(session); }), ); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts index a15f640555..755070a546 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-database.unit.test.ts @@ -737,6 +737,64 @@ describe("legacySetupShadowDatabase / legacyMigrateShadowDatabase", () => { ); }); + it.effect( + "skips the platform baseline on a warm cache hit but still creates the template", + () => { + const { session, calls } = fakeSession(); + const workdir = tempRoot.current; + const mock = mockSpawner(); + let jwksEvaluated = false; + return Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + yield* legacySetupShadowDatabase( + mock.spawner, + { + fs, + path, + workdir, + projectId: "proj", + container: "shadow-container-id-0123456789abcdef", + networkId: "supabase_network_proj", + connConfig: { + host: "127.0.0.1", + port: 54320, + user: "postgres", + password: "postgres", + database: "postgres", + }, + setup: baseShadowSetup({ + majorVersion: 17, + realtimeEnabledForSetup: true, + jwks: Effect.sync(() => { + jwksEvaluated = true; + return '{"keys":[]}'; + }), + }), + }, + {}, + { + baselinePresent: true, + snapshotRequired: false, + snapshotBaseline: Effect.void, + }, + ); + expect(jwksEvaluated).toBe(false); + expect(calls.some((c) => c.sql === LEGACY_SHADOW_CREATE_TEMPLATE_SQL)).toBe(true); + }).pipe( + Effect.provide( + Layer.mergeAll( + BunServices.layer, + mockOutput().layer, + mockDockerRun(), + mockRuntimeInfo(), + mockDbConnection(session), + ), + ), + ); + }, + ); + it.effect( "legacyMigrateShadowDatabase lists local migrations BEFORE connecting, tolerating a missing migrations directory as an empty list rather than a failure", () => { diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts index 119e3eabb7..ba25c04969 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts @@ -1075,16 +1075,21 @@ const legacyProvisionBaselineShadow = ( password: shadowInput.password, database: "postgres", }; - yield* legacySetupShadowDatabase(spawner, { - fs, - path, - workdir: ctx.cwd, - projectId: shadowInput.projectId, - container: handle.containerId, - networkId: shadowInput.networkId, - connConfig, - setup: shadowInput.setup, - }); + yield* legacySetupShadowDatabase( + spawner, + { + fs, + path, + workdir: ctx.cwd, + projectId: shadowInput.projectId, + container: handle.containerId, + networkId: shadowInput.networkId, + connConfig, + setup: shadowInput.setup, + }, + {}, + handle, + ); return { sourceUrl: legacyToPostgresURL(connConfig) } satisfies LegacyProvisionedShadow; }); @@ -1117,16 +1122,21 @@ const legacyProvisionDeclarativeShadow = ( password: shadowInput.password, database: "postgres", }; - yield* legacySetupShadowDatabase(spawner, { - fs, - path, - workdir: ctx.cwd, - projectId: shadowInput.projectId, - container: handle.containerId, - networkId: shadowInput.networkId, - connConfig, - setup: shadowInput.setup, - }); + yield* legacySetupShadowDatabase( + spawner, + { + fs, + path, + workdir: ctx.cwd, + projectId: shadowInput.projectId, + container: handle.containerId, + networkId: shadowInput.networkId, + connConfig, + setup: shadowInput.setup, + }, + {}, + handle, + ); const targetUrl = legacyToPostgresURL(connConfig); yield* legacyApplyDeclarativePgDelta(ctx, { fs, diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md index 9def9ddc86..0246da8b82 100644 --- a/docs/roadmap/pg-delta-next-follow-ups.md +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -30,17 +30,15 @@ usually alongside image bumps), which is why it did not block #6184. Two candida Either way, add a unit-test mutation case alongside the existing ones in `shadow-cache.unit.test.ts`. -## Make the baseline/declarative catalog shadows warm-aware (PR #6184 × CLI-1970 merge) +## Make the baseline/declarative catalog shadows use `legacyWaitForShadowReady` (PR #6184 × CLI-1970 merge) CLI-1970 (#6162) made `legacyExportBaselineCatalogRef`/`legacyExportDeclarativeCatalogRef` -(`legacy-pgdelta.cache.ts`) native, so `db schema declarative sync`/`generate` now provision a -second shadow in-process for the declarative/baseline catalog. Those callers pass an unconditional -`{ bypassCache: true }` to `exportViaShadowCatalog`: their provisions run the platform baseline via -`legacySetupShadowDatabase`, which is not baseline-state-aware, so a warm PGDATA hit would -double-apply the baseline. - -The snapshot seam is a natural fit — the tar IS exactly the post-baseline state these provisions -build (baseline only, no migrations), so a warm hit would need NO further setup at all, saving the -full ~15s per declarative-catalog miss. Requires threading `LegacyShadowBaselineState` through -`legacySetupShadowDatabase` (skip when `baselinePresent`) and swapping their -`legacyWaitForHealthyServices` docker-health wait for `legacyWaitForShadowReady`. +(`legacy-pgdelta.cache.ts`) native. `legacySetupShadowDatabase` is now baseline-state-aware +(skip when `baselinePresent`; snapshot on a cache-enabled cold provision), and those catalog +provisions thread the acquire handle through, so a warm hit no longer double-applies the +baseline. They still wait with `legacyWaitForHealthyServices` (docker healthcheck, 10s interval) +instead of `legacyWaitForShadowReady` (container-state + short connect), so a warm restore can +sit until the first healthcheck tick. + +The bundled pg-delta next sync/diff shadows (`legacy-pgdelta-next-shadow.layer.ts`) already use +the cache-aware acquire + `legacyWaitForShadowReady`. From e342263f19dd0cd9be06aec06abfe6e52bec98e2 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 14:56:10 +0200 Subject: [PATCH 65/82] docs(cli): clarify explicit diff output contract --- apps/cli/docs/supabase/db/diff.md | 4 +++- .../legacy/cli/legacy-complete.unit.test.ts | 8 +++++-- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 21 +++++++++++++++++-- .../legacy/commands/db/diff/diff.command.ts | 8 ++++--- 4 files changed, 33 insertions(+), 8 deletions(-) diff --git a/apps/cli/docs/supabase/db/diff.md b/apps/cli/docs/supabase/db/diff.md index 0c0cf05a4d..fb407e4349 100644 --- a/apps/cli/docs/supabase/db/diff.md +++ b/apps/cli/docs/supabase/db/diff.md @@ -6,11 +6,13 @@ Requires the local development stack to be running when diffing against the loca Runs [djrobstep/migra](https://github.com/djrobstep/migra) in a container to compare schema differences between the target database and a shadow database. The shadow database is created by applying migrations in local `supabase/migrations` directory in a separate container. Output is written to stdout by default. For convenience, you can also save the schema diff as a new migration file by passing in `-f` flag. +Explicit `--from`/`--to` mode always uses pg-delta. In this mode, `-f` is ignored and stdout (or `--output`) is a flattened representation for review, not a portable apply script. Do not apply it directly with plain `psql -f`: transactional units can contain `SET LOCAL` preambles that only take effect inside a transaction, while plans that mix transactional and non-transactional units cannot safely be wrapped in one transaction. To create an applicable migration, use normal target mode with `supabase db diff -f `, then apply it through `supabase db reset` locally or `supabase db push` against the linked project. These paths preserve the plan's per-unit transaction semantics. + By default, all schemas in the target database are diffed. Use the `--schema public,extensions` flag to restrict diffing to a subset of schemas. Projects created by a recent `supabase init` default to the pg-delta diff engine (`[experimental.pgdelta] enabled = true` in `config.toml`). Existing projects are unaffected and keep using migra unless they opt in. To fall back to the legacy migra engine, set `enabled = false` under `[experimental.pgdelta]`, or pass `--use-migra` for a single run. -With the pg-delta engine the diff SQL is formatted by default with the same settings the declarative export uses (uppercase keywords, wrapped at a max width of 180, indented and column-aligned); execution-aware transaction boundaries are preserved as per-unit header comments in the output. Configure overrides with `[experimental.pgdelta] format_options`, or set `format_options = "null"` to emit raw, unformatted statements. +With the bundled pg-delta engine, diff SQL defaults to lowercase keywords and a maximum width of 180, matching its declarative export. When `-f` writes migrations, execution-aware transaction semantics are preserved as ordered per-unit files; non-transactional units carry a directive that the CLI apply path honors. Flattened review output retains the rendered SQL and preambles, but not the unit boundaries supplied to a migration runner. Configure overrides with `[experimental.pgdelta] format_options`, or set `format_options = "null"` to emit raw, unformatted statements. While the diff command is able to capture most schema changes, there are cases where it is known to fail. Currently, this could happen if you schema contains: diff --git a/apps/cli/src/legacy/cli/legacy-complete.unit.test.ts b/apps/cli/src/legacy/cli/legacy-complete.unit.test.ts index b81610c922..08cb85509a 100644 --- a/apps/cli/src/legacy/cli/legacy-complete.unit.test.ts +++ b/apps/cli/src/legacy/cli/legacy-complete.unit.test.ts @@ -90,7 +90,9 @@ describe("legacyRespondToComplete", () => { const result = legacyRespondToComplete(legacyRoot, ["__complete", "db", "diff", "--o"]); const outputCandidates = result?.candidates.filter((c) => c.name === "--output"); expect(outputCandidates).toHaveLength(1); - expect(outputCandidates?.[0]?.description).toBe("Write explicit diff output to a file path."); + expect(outputCandidates?.[0]?.description).toBe( + "Write flattened explicit diff SQL to a file for review; this is not a portable apply script.", + ); }); describe("subcommand completion is not blocked by a preceding global flag", () => { @@ -1313,7 +1315,9 @@ describe("legacyCollectInScopeFlags", () => { const flags = legacyCollectInScopeFlags(legacyRoot, commandChain); const outputFlags = flags.filter((flag) => flag.name === "output"); expect(outputFlags).toHaveLength(1); - expect(outputFlags[0]?.description).toBe("Write explicit diff output to a file path."); + expect(outputFlags[0]?.description).toBe( + "Write flattened explicit diff SQL to a file for review; this is not a portable apply script.", + ); }); it("orders flags like cobra's InheritedFlags().VisitAll then NonInheritedFlags().VisitAll — alphabetical within each block, not declaration order (CLI-1965 review)", () => { diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 23573f2114..2f9466e69b 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -37,7 +37,7 @@ it, and JSON `null` disables formatting without disabling safe compaction. | Path | Format | When | | ----------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------ | | `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | -| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | +| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output`; flattened review representation, not a portable apply script | | `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | | `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | | `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | @@ -136,6 +136,17 @@ Progress to stderr (`Creating shadow database...`, `Diffing schemas[: ]`, that it no longer changes the migrations baseline. The SQL diff prints to stdout when neither `--file` nor explicit `--output` is set. +Explicit `--from`/`--to` mode returns before normal `--file` handling, so `-f` is +ignored. It joins the pg-delta plan files with blank lines and writes that same +flattened review representation to stdout or `--output`. File header comments +and preambles remain, but runner-enforced transaction boundaries do not: +transactional units can begin with `SET LOCAL check_function_bodies = off`, +which has no effect under plain `psql -f` without a surrounding transaction. +A plan may also mix transactional and non-transactional units, so wrapping the +whole flattened representation in one transaction is not generally safe. To +create applicable migrations, use normal target mode with `--file`; it persists +the ordered plan units separately for application through `db reset` or `db push`. + ### `--output-format json` / `stream-json` Progress strings still go to stderr; stdout carries a single structured envelope @@ -144,6 +155,10 @@ the raw SQL. Bundled pg-delta reports the best-effort `DeclarativeSchemaNotUsedAsDiffBaseline` advisory for a non-empty `--file` diff when declarative files exist. +In explicit `--from`/`--to` mode, the `diff` field is the same flattened review +representation as text stdout; the machine envelope does not restore the per-unit +transaction metadata. + ### `--use-pgadmin` (CLI-1968) - **Status lines go to STDOUT in text mode, not stderr** — unlike the migra/pg-delta path's @@ -196,7 +211,9 @@ when declarative files exist. - `--use-pg-schema` rebuilds the argv and exec's the bundled Go binary (its side effects are Go's); the Go child's telemetry is disabled so the single `cli_command_executed` event comes from this TS command. -- Explicit `--from`/`--to` mode always uses pg-delta and writes to `--output` (or stdout). +- Explicit `--from`/`--to` mode always uses pg-delta and writes the flattened review + representation to `--output` (or stdout). It ignores `--file`; normal mode retains + per-unit migration files for the CLI apply paths. - Normal mode always compares the migrations shadow to the selected live database; declarative files and `schema_paths` do not replace that baseline. - Under the legacy opt-out, the explicit `migrations` target resolves natively (CLI-1959): a bare diff --git a/apps/cli/src/legacy/commands/db/diff/diff.command.ts b/apps/cli/src/legacy/commands/db/diff/diff.command.ts index 8d7035652a..526ea1129b 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.command.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.command.ts @@ -50,7 +50,9 @@ const config = { ), output: Flag.string("output").pipe( Flag.withAlias("o"), - Flag.withDescription("Write explicit diff output to a file path."), + Flag.withDescription( + "Write flattened explicit diff SQL to a file for review; this is not a portable apply script.", + ), Flag.optional, ), dbUrl: Flag.string("db-url").pipe( @@ -78,7 +80,7 @@ const config = { file: Flag.string("file").pipe( Flag.withAlias("f"), Flag.withDescription( - "Names and saves the complete schema diff as a new migration; it does not filter objects.", + "In normal mode, names and saves the complete schema diff as a new migration; it does not filter objects. Ignored with --from/--to.", ), Flag.optional, ), @@ -99,7 +101,7 @@ export type LegacyDbDiffFlags = CliCommand.Command.Config.Infer; export const legacyDbDiffCommand = Command.make("diff", config).pipe( Command.withDescription( - "Compares a shadow built from supabase/migrations with a live database (--local by default, --linked, or --db-url). Declarative files under supabase/schemas are not part of this baseline. Output is printed by default; -f names and saves the complete diff as a migration and does not filter objects.", + "Compares a shadow built from supabase/migrations with a live database (--local by default, --linked, or --db-url). Declarative files under supabase/schemas are not part of this baseline. Output is printed by default; in normal mode, -f names and saves the complete diff as a migration and does not filter objects. Explicit --from/--to output is flattened review SQL, not a portable apply script.", ), Command.withShortDescription("Diffs the local database for schema changes"), Command.withHandler((flags) => From 7a8fa09b165b786392a5cc226518d18dc365ac3b Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 15:07:05 +0200 Subject: [PATCH 66/82] fix(cli): key shadow snapshots by the effective webhooks policy Legacy migrate, next migrate, and next declarative bake different pg_net states into the same cluster recipe. Hash that policy so they cannot share a snapshot, honor --no-cache on the remaining catalog exports, and document the global cache's LRU/TTL side effects. --- apps/cli/docs/go-cli-divergences.md | 10 +- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 56 +++++----- .../legacy/commands/db/diff/diff.handler.ts | 94 ++++++++-------- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 58 +++++----- .../legacy/commands/db/pull/pull.handler.ts | 100 ++++++++++-------- .../declarative/generate/SIDE_EFFECTS.md | 56 +++++----- .../schema/declarative/sync/SIDE_EFFECTS.md | 69 ++++++------ .../legacy-pgdelta-next-shadow.layer.ts | 19 ++-- .../shadow-cache.integration.test.ts | 39 ++++++- .../shared/db-bootstrap/shadow-cache.ts | 43 +++++++- .../db-bootstrap/shadow-cache.unit.test.ts | 16 +++ .../src/legacy/shared/legacy-pgdelta.cache.ts | 24 +++-- 12 files changed, 360 insertions(+), 224 deletions(-) diff --git a/apps/cli/docs/go-cli-divergences.md b/apps/cli/docs/go-cli-divergences.md index 75e0414155..c27ea4ea82 100644 --- a/apps/cli/docs/go-cli-divergences.md +++ b/apps/cli/docs/go-cli-divergences.md @@ -84,7 +84,9 @@ These commands exist in the TS CLI today but have no direct top-level equivalent `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides the root; LRU keep-8 + 14-day mtime TTL, shared across worktrees with the same settings) and restored into a - fresh container on later runs, cutting shadow provisioning from ~15s to a few seconds. Covers + fresh container on later runs, cutting shadow provisioning from ~15s to a few seconds. The key + includes the effective Webhooks/`pg_net` policy (legacy migrate forces enabled; next migrate + follows config; next declarative forces disabled). Covers migra/`db pull` via `legacyWithShadowDatabase`, and the bundled pg-delta next sync/diff shadows via `legacyAcquireShadowDatabase` (ephemeral host ports are not part of the cache key — they are not baked into PGDATA). TS-only, @@ -96,6 +98,12 @@ These commands exist in the TS CLI today but have no direct top-level equivalent a user's `roles.sql` installs (`ALTER ROLE … SET …`) apply to migrations, whereas Go's single-connection flow ran migrations before those defaults took effect; opting out restores Go's exact single-session behavior. +- Postgres container entrypoint (`postgres.service.ts`): the init script `exec`s + `docker-entrypoint.sh` so Postgres is PID 1. Go leaves `sh` as PID 1, so SIGTERM is never + forwarded and every `docker stop` burns the full 10s grace period. Applies to + `supabase start`, `db start`, `--from-backup`, and shadow containers (the last is why the + shadow baseline cache's cold export can stop/start in ~1s). Timing is not part of the + Go-parity surface (ADR 0016). - `functions serve` per-function env discovery (CLI-2184, #6179): without `--env-file`, each `supabase/functions//.env` overrides matching values from the shared `supabase/functions/.env` for that Function only; an explicit `--env-file` remains the diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 6e40120376..9a803767d0 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -19,34 +19,34 @@ it, and JSON `null` disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| --------------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | -| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — the snapshot is streamed into the fresh shadow container before it starts (`SUPABASE_HOME` overrides the `~/.supabase` root) | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | -| `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | -| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | -| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit `--from/--to migrations` catalog cache | +| Path | Format | When | +| --------------------------------------------------------------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`, deno_version) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | +| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | abandoned-partial sweep on every cache-eligible acquire (warm hit and cold export) — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | +| `~/.supabase/access-token` | plain text | `--linked` / `--db-url` with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | `--linked` ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit `--from/--to migrations` catalog cache | ## Files Written -| Path | Format | When | -| --------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | -| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision only (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`, never a warm hit) — the shadow's PGDATA snapshot, ~90MB, LRU keep-8 + 14-day mtime TTL (`SUPABASE_HOME` overrides the root) | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | -| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_.sql` | SQL | non-empty `--file` diff; bundled pg-delta may emit ordered transaction-aware files, while pgAdmin always emits one | +| `` (from `--output` / `-o`) | SQL | explicit `--from/--to` mode with `--output` | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's explicit migrations catalog | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision creates the current key's snapshot (native diff targets + the explicit `--from/--to migrations` catalog miss; never `--use-pgadmin`/`--use-pg-schema`); a warm hit `touch`es its mtime (LRU); every cache-eligible acquire may delete other keys under LRU keep-8 + 14-day mtime TTL — ~90MB (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| `~/.supabase//linked-project.json` | JSON | `--linked` (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker @@ -96,6 +96,7 @@ of this command's own target resolve, ahead of the differ container. | `SUPABASE_DB_MAJOR_VERSION` / `SUPABASE_DB_HEALTH_TIMEOUT` / `SUPABASE_DB_SETTINGS_*` | shadow container-config overrides, same as `db start`/`db reset` | no | | `SUPABASE_PROJECT_ID` | overrides the shadow container's project id/labels, same as `db start`/`db reset` (`utils.DbId`); ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the shadow baseline cache (and other CLI state) | no | | `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | | `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta engine | no | @@ -219,9 +220,10 @@ project's dotenv, e.g. `supabase/.env`), restoring the documented uncached lifec skips the platform baseline, so the `Initialising schema...` progress line does not print — progress text reflects the work actually performed. Artifact: `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` -overrides the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster; +overrides the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster +(including the effective Webhooks/`pg_net` policy); shared across worktrees with the same settings; retention is LRU (keep 8) + 14-day mtime TTL -(warm hits refresh mtime). +(warm hits refresh mtime; sibling tars may be deleted). Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a cold export failure only warns and leaves the run uncached (one exception: a shadow that diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index a3b74a1ffa..79f3279a1c 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -689,53 +689,59 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // why the cache seam sits here (with `SUPABASE_SHADOW_CACHE` unset it IS today's // create/remove pair; otherwise a key-matching PGDATA snapshot is restored into the fresh // container in a few seconds instead of cold-provisioning the baseline in ~15s). - diffResult = yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => - Effect.gen(function* () { - const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); - const target = shadow.targetUrlOverride ?? targetUrl; - yield* output.raw( - flags.schema.length > 0 - ? `Diffing schemas: ${flags.schema.join(",")}\n` - : "Diffing schemas...\n", - "stderr", - ); - if (useDelta) { - const result = yield* pgDelta.diffDatabase({ - context: ctx, - source: { - kind: "database", - ref: shadow.sourceUrl, - connectOptions: { isLocal: true, dnsResolver: "native" }, - }, - target: { - kind: "database", - ref: target, - ...(shadow.targetUrlOverride === undefined ? { connection: resolved.conn } : {}), - connectOptions: { - isLocal: shadow.targetUrlOverride !== undefined || resolved.isLocal, - dnsResolver, + // `webhooks: "enabled"` matches `legacyMigrateShadowDatabase`'s forced `pg_net` + // baseline — the cache key must not collide with next's config-following migrate. + diffResult = yield* legacyWithShadowDatabase( + spawner, + shadowInput, + (handle) => + Effect.gen(function* () { + const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); + const target = shadow.targetUrlOverride ?? targetUrl; + yield* output.raw( + flags.schema.length > 0 + ? `Diffing schemas: ${flags.schema.join(",")}\n` + : "Diffing schemas...\n", + "stderr", + ); + if (useDelta) { + const result = yield* pgDelta.diffDatabase({ + context: ctx, + source: { + kind: "database", + ref: shadow.sourceUrl, + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + target: { + kind: "database", + ref: target, + ...(shadow.targetUrlOverride === undefined ? { connection: resolved.conn } : {}), + connectOptions: { + isLocal: shadow.targetUrlOverride !== undefined || resolved.isLocal, + dnsResolver, + }, }, - }, + schema: flags.schema, + formatOptions, + debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, + }); + // Keep the per-unit plan files so a multi-unit plan can be written as one + // migration file each; `sql` stays the flattened join for stdout review + + // machine payloads. + return { sql: result.sql, files: result.files, hazards: result.hazards }; + } + const sql = yield* legacyDiffMigra(ctx, { + source: shadow.sourceUrl, + target, schema: flags.schema, - formatOptions, - debug: legacyIsPgDeltaDebugEnabled(), - strictCoverage: flags.strictCoverage, + connectOptions: { isLocal: resolved.isLocal, dnsResolver }, }); - // Keep the per-unit plan files so a multi-unit plan can be written as one - // migration file each; `sql` stays the flattened join for stdout review + - // machine payloads. - return { sql: result.sql, files: result.files, hazards: result.hazards }; - } - const sql = yield* legacyDiffMigra(ctx, { - source: shadow.sourceUrl, - target, - schema: flags.schema, - connectOptions: { isLocal: resolved.isLocal, dnsResolver }, - }); - // The migra engine has no execution-aware plan units, so it always writes a - // single migration file. - return { sql, files: undefined }; - }), + // The migra engine has no execution-aware plan units, so it always writes a + // single migration file. + return { sql, files: undefined }; + }), + { webhooks: "enabled" }, ); } const out = diffResult.sql; diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index 64e67f4b39..d28c298cdd 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -35,35 +35,35 @@ disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ----------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | -| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | -| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | -| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | -| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — snapshot streamed into the fresh shadow container before it starts (`SUPABASE_HOME` overrides the `~/.supabase` root) | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | -| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | -| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | -| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | +| Path | Format | When | +| ----------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always (db port/password, `[experimental.pgdelta]`) | +| `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | +| `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | +| `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | +| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | abandoned-partial sweep on every cache-eligible acquire (warm hit and cold export) — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | +| `/supabase/.temp/project-ref` | plain text | linked ref resolution — skipped when `--project-ref` (or `SUPABASE_PROJECT_ID`) is set | +| `/supabase/.temp/{pgdelta-version,edge-runtime-version}` | plain text | legacy pg-delta opt-out only | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog snapshots | ## Files Written -| Path | Format | When | -| --------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | -| `/supabase/schemas/**` | SQL | `--declarative` | -| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | -| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision, migration-style pull only (never `--declarative`'s bare shadow, the delegated `--experimental` path, or a warm hit) — the shadow's PGDATA snapshot, ~90MB, LRU keep-8 + 14-day mtime TTL (`SUPABASE_HOME` overrides the root) | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | -| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | -| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_.sql` | SQL | migration-style pull (non-empty diff, or the initial-migra `pg_dump` seed) | +| `/supabase/schemas/**` | SQL | `--declarative` | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled `--declarative` export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy pg-delta opt-out catalog snapshots | +| `/supabase/.temp/pgdelta/pgdelta-target-ca.crt` | PEM | legacy opt-out, for a Supabase TLS target | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `/supabase/schemas/**`, `/supabase/cluster/**` | SQL | `--experimental` structured dump (delegated to Go; both dirs are `RemoveAll`'d then rewritten by `format.WriteStructuredSchemas`, not just written to) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision creates the current key's snapshot, migration-style pull only (never `--declarative`'s bare shadow or the delegated `--experimental` path); a warm hit `touch`es its mtime (LRU); every cache-eligible acquire may delete other keys under LRU keep-8 + 14-day mtime TTL — ~90MB (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| `~/.supabase//linked-project.json` | JSON | linked (post-run cache) | +| `~/.supabase/telemetry.json` | JSON | every invocation (post-run) | ## Docker @@ -84,9 +84,10 @@ project's dotenv, e.g. `supabase/.env`), restoring the documented uncached lifec skips the platform baseline, so the `Initialising schema...` progress line does not print — progress text reflects the work actually performed. Artifact: `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` -overrides the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster; +overrides the root), a PGDATA snapshot keyed by a hash of every input baked into the cluster +(including the effective Webhooks/`pg_net` policy); shared across worktrees with the same settings; retention is LRU (keep 8) + 14-day mtime TTL -(warm hits refresh mtime). +(warm hits refresh mtime; sibling tars may be deleted). Container lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a cold export failure only warns and leaves the run uncached (one exception: a shadow that @@ -115,6 +116,7 @@ baseline, so it is never cached. | `SUPABASE_DB_MAJOR_VERSION` / `SUPABASE_DB_HEALTH_TIMEOUT` / `SUPABASE_DB_SETTINGS_*` | shadow container-config overrides, same as `db start`/`db reset` | no | | `SUPABASE_PROJECT_ID` | overrides the shadow container's project id/labels, same as `db start`/`db reset` (`utils.DbId`); ALSO the linked-ref resolution fallback `--project-ref` supersedes — see Notes for the narrower scope of the flag | no | | `SUPABASE_NETWORK_ID` (`--network-id`) | forces the shadow container/network onto an existing Docker network | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the shadow baseline cache (and other CLI state) | no | | `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | | `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | | `SUPABASE_EXPERIMENTAL_PG_DELTA` | force pg-delta diff engine | no | diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 61bd1d15aa..168a0a9f5b 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -795,55 +795,61 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // pooler-retry attempt still acquires and releases its own shadow — on the warm path // each attempt restores its own fresh container from the same cached snapshot, // sequentially. - return yield* legacyWithShadowDatabase(spawner, shadowInput, (handle) => - Effect.gen(function* () { - const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); - const target = shadow.targetUrlOverride ?? targetEndpoint.ref; - yield* output.raw( - diffSchema.length > 0 - ? `Diffing schemas: ${diffSchema.join(",")}\n` - : "Diffing schemas...\n", - "stderr", - ); - if (usePgDeltaDiff) { - return yield* pgDeltaEngine.diffDatabase({ - context: ctx, - source: { - kind: "database", - ref: shadow.sourceUrl, - connectOptions: { isLocal: true, dnsResolver: "native" }, - }, - target: { - kind: "database", - ref: target, - ...(shadow.targetUrlOverride === undefined - ? { - ...(targetEndpoint.connection !== undefined - ? { connection: targetEndpoint.connection } - : {}), - connectOptions: targetEndpoint.connectOptions, - } - : { - connectOptions: { isLocal: true, dnsResolver }, - }), - }, + // `webhooks: "enabled"` matches `legacyMigrateShadowDatabase`'s forced `pg_net` + // baseline — the cache key must not collide with next's config-following migrate. + return yield* legacyWithShadowDatabase( + spawner, + shadowInput, + (handle) => + Effect.gen(function* () { + const shadow = yield* legacyPrepareShadowSource(spawner, handle, shadowInput); + const target = shadow.targetUrlOverride ?? targetEndpoint.ref; + yield* output.raw( + diffSchema.length > 0 + ? `Diffing schemas: ${diffSchema.join(",")}\n` + : "Diffing schemas...\n", + "stderr", + ); + if (usePgDeltaDiff) { + return yield* pgDeltaEngine.diffDatabase({ + context: ctx, + source: { + kind: "database", + ref: shadow.sourceUrl, + connectOptions: { isLocal: true, dnsResolver: "native" }, + }, + target: { + kind: "database", + ref: target, + ...(shadow.targetUrlOverride === undefined + ? { + ...(targetEndpoint.connection !== undefined + ? { connection: targetEndpoint.connection } + : {}), + connectOptions: targetEndpoint.connectOptions, + } + : { + connectOptions: { isLocal: true, dnsResolver }, + }), + }, + schema: diffSchema, + formatOptions, + debug: legacyIsPgDeltaDebugEnabled(), + strictCoverage: flags.strictCoverage, + }); + } + const sql = yield* legacyDiffMigra(ctx, { + source: shadow.sourceUrl, + target, schema: diffSchema, - formatOptions, - debug: legacyIsPgDeltaDebugEnabled(), - strictCoverage: flags.strictCoverage, + connectOptions: + shadow.targetUrlOverride === undefined + ? targetEndpoint.connectOptions + : { isLocal: true, dnsResolver }, }); - } - const sql = yield* legacyDiffMigra(ctx, { - source: shadow.sourceUrl, - target, - schema: diffSchema, - connectOptions: - shadow.targetUrlOverride === undefined - ? targetEndpoint.connectOptions - : { isLocal: true, dnsResolver }, - }); - return { sql, files: undefined, debug: undefined }; - }), + return { sql, files: undefined, debug: undefined }; + }), + { webhooks: "enabled" }, ); }); const diffOutcome = yield* withPoolerFallback(targetEndpoint, runShadowDiff); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index b78dc090aa..6f515903af 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -7,30 +7,34 @@ Pg-delta runs in-process by default. Set `SUPABASE_USE_PG_DELTA_NEXT=false` for the legacy catalog/edge-runtime implementation; there is no automatic fallback. Coverage gaps warn; `--strict-coverage` makes them fatal, and `PGDELTA_DEBUG` writes diagnostic JSON under `supabase/.temp/pgdelta/v2/debug//`. -`--no-cache` affects only the legacy opt-out. The bundled formatter defaults to +`--no-cache` affects only the legacy opt-out (its catalog cache and the shadow +baseline snapshot those catalog exports use). The bundled formatter defaults to lowercase SQL at width 180; config overrides it, and JSON `null` disables formatting without disabling safe compaction. ## Files Read -| Path | Format | When | -| ----------------------------------------------- | ---------- | --------------------------------------------- | -| `/supabase/config.toml` | TOML | always — pg-delta gate, ports, format options | -| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | -| `/supabase/.temp/postgres-version` | plain text | legacy opt-out's shadow-DB image resolution | -| `/supabase/migrations/*.sql` | SQL | smart mode — detect whether migrations exist | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog cache | -| `~/.supabase/access-token` | plain text | `--linked` (token resolution) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always — pg-delta gate, ports, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/.temp/postgres-version` | plain text | legacy opt-out's shadow-DB image resolution | +| `/supabase/migrations/*.sql` | SQL | smart mode — detect whether migrations exist | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog cache | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | legacy opt-out catalog miss, warm shadow-cache hit — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU/TTL (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | legacy opt-out catalog miss — abandoned-partial sweep on every cache-eligible acquire; removed when older than an hour | +| `~/.supabase/access-token` | plain text | `--linked` (token resolution) | ## Files Written -| Path | Format | When | -| ----------------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------ | -| `/supabase/schemas/**/*.sql` (default declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | -| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| ----------------------------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/schemas/**/*.sql` (default declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | +| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | legacy opt-out catalog miss, cache-enabled COLD shadow provision creates the current key's snapshot; a warm hit `touch`es its mtime; LRU/TTL may delete other keys (`SUPABASE_HOME` overrides the root; `--no-cache` neither reads nor writes) | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers @@ -42,15 +46,17 @@ formatting without disabling safe compaction. ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------- | -------------------------------------------------- | --------- | -| `SUPABASE_ACCESS_TOKEN` | auth token for `--linked` | no | -| `DB_PASSWORD` | password for `--linked` / `--db-url` | no | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | -| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | -| `SUPABASE_SERVICES_HOSTNAME` | local DB host for `--local` | no | -| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | +| Variable | Purpose | Required? | +| ---------------------------- | ----------------------------------------------------------------------------------------------------------------- | --------- | +| `SUPABASE_ACCESS_TOKEN` | auth token for `--linked` | no | +| `DB_PASSWORD` | password for `--linked` / `--db-url` | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the legacy opt-out's shadow baseline cache | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache for the legacy opt-out's catalog-miss shadows; ON by default, set to `false`/`0` to opt out | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | +| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | +| `SUPABASE_SERVICES_HOSTNAME` | local DB host for `--local` | no | +| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | ## Exit Codes diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index 1dbe51cdd2..fef7ba82f9 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -16,29 +16,29 @@ disabling safe compaction. ## Files Read -| Path | Format | When | -| --------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | -| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | -| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | -| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | -| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | -| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | -| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | -| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — bundled-engine migrations/declarative shadows, and the legacy opt-out's migrations-catalog miss (`SUPABASE_HOME` overrides the `~/.supabase` root) | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export's abandoned-partial sweep — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | +| Path | Format | When | +| --------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/config.toml` | TOML | always — pg-delta gate, format options | +| `/supabase/.temp/pgdelta-version` | plain text | loaded for compatibility; legacy opt-out only | +| `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | +| `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | +| `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | +| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | +| `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | +| `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — bundled-engine migrations/declarative shadows, and the legacy opt-out's catalog miss; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | abandoned-partial sweep on every cache-eligible acquire (warm hit and cold export) — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | ## Files Written -| Path | Format | When | -| --------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | -| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision — bundled-engine migrations/declarative shadows, and the legacy opt-out's migrations-catalog miss (a catalog hit provisions no shadow; a warm hit rewrites nothing; `--no-cache` bypasses the snapshot cache entirely — neither read nor written) — the shadow's PGDATA snapshot, ~90MB, LRU keep-8 + 14-day mtime TTL (`SUPABASE_HOME` overrides the root) | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| Path | Format | When | +| --------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/migrations/_[_].sql` | SQL | changes; bundled engine may emit ordered segments | +| `/supabase/schemas/extension.sql` | SQL | accepted legacy-extension repair | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | cache-enabled (default) COLD shadow provision creates the current key's snapshot — bundled-engine migrations/declarative shadows, and the legacy opt-out's catalog miss (a catalog hit provisions no shadow; `--no-cache` bypasses the snapshot cache entirely — neither read nor written); a warm hit `touch`es its mtime (LRU); every cache-eligible acquire may delete other keys under LRU keep-8 + 14-day mtime TTL — ~90MB (`SUPABASE_HOME` overrides the root) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | ## Subprocesses / Containers @@ -56,6 +56,7 @@ disabling safe compaction. | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | | `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the shadow baseline cache (and other CLI state) | no | | `SUPABASE_SHADOW_CACHE` | shadow baseline cache; ON by default, set to `false`/`0` to opt out — the shadow's post-baseline PGDATA is snapshotted to a tar and restored into the next run's fresh container (see Notes) | no | | `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | | `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines); never touches stdout/exit codes | no | @@ -133,23 +134,25 @@ The bundled (pg-delta next) engine provisions both plan shadows through `legacyAcquireShadowDatabase` (`legacy-pgdelta-next-shadow.layer.ts`): ON by default, `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (ambient env or project dotenv), and `--no-cache` bypasses restore and publish for that invocation. Next allocates an ephemeral host port per -shadow; the cache key hashes the cluster recipe, not the published port, so worktrees and -repeated syncs with the same settings share a warm hit. A warm hit skips the platform baseline -on both the migrated and declarative shadows (`legacyMigrateNextShadowDatabase` / +shadow; the cache key hashes the cluster recipe (including the effective Webhooks/`pg_net` +policy), not the published port, so worktrees and repeated syncs with the same settings share +a warm hit. The migrations shadow follows project config; the declarative shadow forces +`pg_net` off — those are distinct keys when Webhooks are enabled. A warm hit skips the +platform baseline on both shadows (`legacyMigrateNextShadowDatabase` / `legacySetupShadowDatabase` are baseline-state-aware). Artifact: `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides -the root); shared across worktrees with the same settings; retention is LRU (keep 8) + 14-day -mtime TTL (warm hits refresh mtime). Container lifecycle is identical to the uncached path +the root), keyed by a hash of every input baked into the cluster (including the effective +Webhooks/`pg_net` policy); shared across worktrees with the same settings; retention is LRU +(keep 8) + 14-day mtime TTL (warm hits refresh mtime; sibling tars may be deleted). Container +lifecycle is identical to the uncached path except a cold run drops `--rm` (still removed on release). A cache anomaly never fails the command — a warm-path anomaly cold-provisions instead, a cold export failure only warns and leaves the run uncached (one exception: a shadow that fails to come back up after the snapshot fails the run rather than reporting a false success). See `shared/db-bootstrap/shadow-cache.ts`. -Under the legacy opt-out, the migrations-catalog shadow on a cache miss goes through -`legacyGetMigrationsCatalogRef` -> `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), the same -`legacyWithShadowDatabase` seam `db diff`/`db pull` use, and `--no-cache` bypasses that snapshot -cache along with the catalog cache. The DECLARATIVE-catalog shadow (and `generate`'s -baseline-catalog shadow) now threads baseline state through `legacySetupShadowDatabase` so a -warm hit does not double-apply the baseline; swapping their `legacyWaitForHealthyServices` -docker-health wait for `legacyWaitForShadowReady` is a recorded follow-up -(`docs/roadmap/pg-delta-next-follow-ups.md`). +Under the legacy opt-out, every catalog-miss shadow (migrations, baseline, declarative) goes +through `exportViaShadowCatalog` (`legacy-pgdelta.cache.ts`), the same +`legacyWithShadowDatabase` seam `db diff`/`db pull` use. `--no-cache` bypasses that snapshot +cache along with the catalog cache. Catalog provisioners wait with `legacyWaitForShadowReady` +and thread baseline state through `legacySetupShadowDatabase` so a warm hit does not +double-apply the baseline. diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index 1d71243b2a..71a96172d3 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -261,8 +261,13 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( return legacyToPostgresURL(setup.connConfig); }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); - const cacheOpts = (opts: LegacyPgDeltaNextShadowInput): LegacyShadowCacheOpts => - opts.bypassCache === true ? { bypassCache: true } : {}; + const cacheOpts = ( + opts: LegacyPgDeltaNextShadowInput, + webhooks: NonNullable, + ): LegacyShadowCacheOpts => ({ + webhooks, + ...(opts.bypassCache === true ? { bypassCache: true } : {}), + }); return LegacyPgDeltaNextShadow.of({ provisionMigrations: (opts) => @@ -270,7 +275,7 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( const port = yield* nextPort(); const built = yield* buildNativeBase(opts); const input = buildNativeInput(opts, built, port); - return yield* provisionMigrations(input, cacheOpts(opts)); + return yield* provisionMigrations(input, cacheOpts(opts, "config")); }).pipe(Effect.mapError(nextShadowError)), provisionPlan: (opts) => Effect.gen(function* () { @@ -279,9 +284,11 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( const built = yield* buildNativeBase(opts); const migrationsInput = buildNativeInput(opts, built, migrationsPort); const declarativeInput = buildNativeInput(opts, built, declarativePort); - const cache = cacheOpts(opts); - const migrations = yield* provisionMigrations(migrationsInput, cache); - const declarativeUrl = yield* provisionDeclarative(declarativeInput, cache); + const migrations = yield* provisionMigrations(migrationsInput, cacheOpts(opts, "config")); + const declarativeUrl = yield* provisionDeclarative( + declarativeInput, + cacheOpts(opts, "disabled"), + ); return { ...migrations, declarativeUrl, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 0ec72308b4..ae8f3d00f1 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -40,6 +40,7 @@ import { LEGACY_SHADOW_BASELINE_KEEP, LEGACY_SHADOW_CACHE_ENV, legacyAcquireShadowDatabase, + type LegacyShadowCacheOpts, } from "./shadow-cache.ts"; import { LEGACY_SHADOW_DEBUG_ENV } from "./shadow-debug.ts"; import { legacyRemoveShadowDatabase } from "./shadow-database.ts"; @@ -400,9 +401,10 @@ const soleTarName = Effect.fnUntraced(function* (fs: FileSystem.FileSystem, path const coldRun = ( docker: ReturnType, input: LegacyShadowSetupInput, + opts: LegacyShadowCacheOpts = {}, ) => Effect.gen(function* () { - const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input); + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, input, opts); yield* handle.snapshotBaseline; yield* legacyRemoveShadowDatabase(docker.spawner, handle.containerId); return handle; @@ -811,6 +813,41 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("legacy forced-on webhooks and next config-following webhooks do not share a tar", () => { + const docker = fakeDockerDaemon(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + // `shadowSetup.webhooksEnabled` is false, so `"config"` (next migrate) and + // `"disabled"` (next declarative) bake the same cluster; `"enabled"` (legacy + // migrate) does not and must not restore that tar. + yield* coldRun(docker, input, { webhooks: "config" }); + expect(yield* soleTarName(fs, path)).toHaveLength(1); + + const forcedOn = yield* coldRun(docker, input, { webhooks: "enabled" }); + expect(forcedOn.baselinePresent).toBe(false); + expect(yield* soleTarName(fs, path)).toHaveLength(2); + + const warmConfig = yield* legacyAcquireShadowDatabase(docker.spawner, input, { + webhooks: "config", + }); + expect(warmConfig.baselinePresent).toBe(true); + const warmDisabled = yield* legacyAcquireShadowDatabase(docker.spawner, input, { + webhooks: "disabled", + }); + expect(warmDisabled.baselinePresent).toBe(true); + expect(yield* soleTarName(fs, path)).toHaveLength(2); + yield* legacyRemoveShadowDatabase(docker.spawner, warmConfig.containerId); + yield* legacyRemoveShadowDatabase(docker.spawner, warmDisabled.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("a changed internal image registry is a different key, not a warm hit", () => { const docker = fakeDockerDaemon(); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 7f338a98ff..3a57d9bcf7 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -43,7 +43,10 @@ import { legacyGetRegistryImageUrl } from "../legacy-docker-registry.ts"; import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; import { LEGACY_POSTGRES_DEFAULT_ROOT_KEY } from "../legacy-local-config-values.ts"; -import { LEGACY_START_REVOKE_API_PRIVILEGES_SQL } from "./db-setup.ts"; +import { + LEGACY_START_REVOKE_API_PRIVILEGES_SQL, + type LegacySetupDatabaseOptions, +} from "./db-setup.ts"; import { LEGACY_START_DB_SCHEMA_SQL } from "./templates/db-schema.sql.ts"; import { LEGACY_START_DB_SUPABASE_SQL } from "./templates/db-supabase.sql.ts"; import { LEGACY_START_DB_WEBHOOK_SQL } from "./templates/db-webhook.sql.ts"; @@ -172,6 +175,15 @@ export interface LegacyShadowCacheKeyInputs { readonly dbSettings: ProjectConfig["db"]["settings"]; /** Effective `api.auto_expose_new_tables` tri-state (unset ≠ explicit `false`: only the former keeps the bundled grants). */ readonly autoExposeNewTables: Option.Option; + /** + * Effective Webhooks/`pg_net` policy baked into the cluster — the same boolean + * `legacySetupDatabase` applies (`options.webhooks` resolved against + * `setup.webhooksEnabled`). Distinct from the raw config flag: legacy migrate + * forces enabled, next declarative forces disabled, and next migrate follows + * config. Hashed so those callers cannot share a snapshot (review: Codex/ + * depthfirst on #6184). + */ + readonly webhooksEnabled: boolean; /** `supabase/roles.sql`'s contents, `""` when absent. */ readonly rolesSql: string; /** @@ -272,6 +284,7 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string `db_password=${quoted(inputs.dbPassword)}`, `db_settings=${legacyCanonicalJson(inputs.dbSettings)}`, `auto_expose_new_tables=${legacyTriStateToken(inputs.autoExposeNewTables)}`, + `webhooks_enabled=${legacyBoolToken(inputs.webhooksEnabled)}`, // Not a per-run input — see the digest's own doc comment for what it covers and why. `baseline_sql_digest=${LEGACY_SHADOW_BASELINE_SQL_DIGEST}`, ]; @@ -316,6 +329,19 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string return createHash("sha256").update(payload, "utf8").digest("hex").slice(0, 16); } +/** + * Same resolution `legacySetupDatabase` applies (`db-setup.ts`): `"enabled"` always + * installs `pg_net`, `"disabled"` always removes it, `"config"` (the default) follows + * `setup.webhooksEnabled`. + */ +export function legacyEffectiveShadowWebhooksEnabled( + policy: LegacySetupDatabaseOptions["webhooks"], + webhooksEnabled: boolean, +): boolean { + const webhooks = policy ?? "config"; + return webhooks === "enabled" || (webhooks === "config" && webhooksEnabled); +} + /** * Resolves {@link LegacyShadowCacheKeyInputs} from the same run input the shadow container * itself is built from, plus `supabase/roles.sql` off disk. The service enabled flags come from @@ -334,8 +360,10 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string * degrade the former to an uncached shadow while letting the latter propagate, without an `as` * cast to tell them apart. */ + const legacyResolveShadowCacheKeyInputs = ( input: LegacyShadowSetupInput, + opts: LegacyShadowCacheOpts = {}, ): Effect.Effect, E> => Effect.gen(function* () { // OrioleDB (`experimental.orioledb_version`) makes the WHOLE cache ineligible, not just a @@ -395,6 +423,10 @@ const legacyResolveShadowCacheKeyInputs = ( dbSettings: input.db.settings, storageTargetMigration: input.setup.storageTargetMigration, autoExposeNewTables: input.setup.apiAutoExposeNewTables, + webhooksEnabled: legacyEffectiveShadowWebhooksEnabled( + opts.webhooks, + input.setup.webhooksEnabled, + ), rolesSql, vault: input.setup.vault, jwks, @@ -749,9 +781,16 @@ const legacyExportShadowBaseline = ( * --no-cache`'s hook: that flag documents "force fresh shadow database setup", so a run carrying * it must neither restore an existing snapshot nor publish a new one — exactly the uncached * lifecycle, regardless of the env gate (review: Codex on #6184). + * + * `webhooks` is the same policy the caller will pass to `legacySetupDatabase` / + * `legacySetupShadowDatabase` / `legacyMigrate*ShadowDatabase`. Defaults to `"config"` + * (follow `setup.webhooksEnabled`), matching {@link LegacySetupDatabaseOptions}. Hashed as + * the effective boolean so a forced-on legacy migrate snapshot cannot warm-restore into a + * next path that follows config, or a declarative shadow that forces webhooks off. */ export interface LegacyShadowCacheOpts { readonly bypassCache?: boolean; + readonly webhooks?: LegacySetupDatabaseOptions["webhooks"]; } /** @@ -905,7 +944,7 @@ export const legacyAcquireShadowDatabase = ( // discovery request, which must not pin a Ctrl-C for its whole duration (review: Codex on // #6184). Interruption here simply means no container was ever created, so there is nothing // for a finalizer to release. - const keyInputs = yield* Effect.interruptible(legacyResolveShadowCacheKeyInputs(input)); + const keyInputs = yield* Effect.interruptible(legacyResolveShadowCacheKeyInputs(input, opts)); if (Option.isNone(keyInputs)) return yield* legacyUncachedShadow(spawner, input); const key = legacyShadowCacheKey(keyInputs.value); const tarPath = input.path.join( diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index f281514930..95a5d224bc 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -4,6 +4,7 @@ import { Option } from "effect"; import { LEGACY_SHADOW_BASELINE_KEEP, LEGACY_SHADOW_BASELINE_MAX_AGE_MS, + legacyEffectiveShadowWebhooksEnabled, legacyIsShadowBaselinePartial, legacyIsShadowBaselineTar, legacyShadowBaselineTarFileName, @@ -23,6 +24,7 @@ const baseKeyInputs = (): LegacyShadowCacheKeyInputs => ({ dbSettings: { effective_cache_size: "128MB", max_connections: 100 }, autoExposeNewTables: Option.none(), storageTargetMigration: "20240101000000", + webhooksEnabled: true, rolesSql: "create role custom_role;\n", vault: [{ name: "secret", value: "value", resolved: true }], jwks: '{"keys":[]}', @@ -62,6 +64,19 @@ describe("legacyShadowCacheEnabled", () => { }); }); +describe("legacyEffectiveShadowWebhooksEnabled", () => { + it("matches legacySetupDatabase: enabled/disabled override config, config follows the flag", () => { + expect(legacyEffectiveShadowWebhooksEnabled("enabled", false)).toBe(true); + expect(legacyEffectiveShadowWebhooksEnabled("enabled", true)).toBe(true); + expect(legacyEffectiveShadowWebhooksEnabled("disabled", true)).toBe(false); + expect(legacyEffectiveShadowWebhooksEnabled("disabled", false)).toBe(false); + expect(legacyEffectiveShadowWebhooksEnabled("config", true)).toBe(true); + expect(legacyEffectiveShadowWebhooksEnabled("config", false)).toBe(false); + expect(legacyEffectiveShadowWebhooksEnabled(undefined, true)).toBe(true); + expect(legacyEffectiveShadowWebhooksEnabled(undefined, false)).toBe(false); + }); +}); + describe("legacyShadowCacheKey", () => { it("is stable for identical inputs and independent of object key order", () => { const first = legacyShadowCacheKey(baseKeyInputs()); @@ -97,6 +112,7 @@ describe("legacyShadowCacheKey", () => { label: "auto expose new tables (explicit false vs unset)", inputs: { ...base, autoExposeNewTables: Option.some(false) }, }, + { label: "effective webhooks / pg_net", inputs: { ...base, webhooksEnabled: false } }, { label: "roles.sql", inputs: { ...base, rolesSql: "" } }, { label: "storage migration pin (storage enabled, majorVersion >= 15)", diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts index ba25c04969..4eb27a598a 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts @@ -22,7 +22,7 @@ import { legacyBuildLocalDbContainerInputs, type LegacyLocalDbContainerInputs, } from "./db-bootstrap/local-container-inputs.ts"; -import { legacyWaitForHealthyServices } from "./db-bootstrap/health-check.ts"; +import { legacyWaitForShadowReady } from "./db-bootstrap/health-check.ts"; import { legacyWithShadowDatabase, type LegacyShadowAcquiredHandle, @@ -787,8 +787,8 @@ const exportViaShadowCatalog = ( // `SUPABASE_SHADOW_CACHE` unset it IS that pair (identical Docker argv, identical labels), and // with it set a catalog cache miss restores a key-matching PGDATA snapshot into the fresh // shadow instead of paying the full cold provision — the same swap `db diff`/`db pull`'s own - // call sites make. `shadowCacheOpts` carries `sync --no-cache`'s bypass — see - // `LegacyShadowCacheOpts`. + // call sites make. `shadowCacheOpts` carries `sync --no-cache`'s bypass and the + // caller's effective Webhooks policy — see `LegacyShadowCacheOpts`. const written = yield* legacyWithShadowDatabase( spawner, shadowInput, @@ -998,7 +998,7 @@ export const legacyGetMigrationsCatalogRef = Effect.fnUntraced(function* ( // `--no-cache` promises "force fresh shadow database setup" (`declarative.shared.ts`), so it // must ALSO bypass the shadow baseline snapshot, not just the catalog cache — otherwise a // warm tar would skip the very setup the flag exists to force (review: Codex on #6184). - { bypassCache: params.noCache }, + { bypassCache: params.noCache, webhooks: "enabled" }, ); }); @@ -1065,9 +1065,6 @@ const legacyProvisionBaselineShadow = ( shadowInput: LegacyShadowSetupInput, ) => Effect.gen(function* () { - yield* legacyWaitForHealthyServices(spawner, [handle.containerId], { - timeoutSeconds: shadowInput.healthTimeoutSeconds, - }); const connConfig: LegacyPgConnInput = { host: shadowInput.hostname, port: shadowInput.shadowPort, @@ -1075,6 +1072,10 @@ const legacyProvisionBaselineShadow = ( password: shadowInput.password, database: "postgres", }; + yield* legacyWaitForShadowReady(spawner, handle.containerId, connConfig, { + timeoutSeconds: shadowInput.healthTimeoutSeconds, + image: shadowInput.image, + }); yield* legacySetupShadowDatabase( spawner, { @@ -1112,9 +1113,6 @@ const legacyProvisionDeclarativeShadow = ( shadowInput: LegacyShadowSetupInput, ) => Effect.gen(function* () { - yield* legacyWaitForHealthyServices(spawner, [handle.containerId], { - timeoutSeconds: shadowInput.healthTimeoutSeconds, - }); const connConfig: LegacyPgConnInput = { host: shadowInput.hostname, port: shadowInput.shadowPort, @@ -1122,6 +1120,10 @@ const legacyProvisionDeclarativeShadow = ( password: shadowInput.password, database: "postgres", }; + yield* legacyWaitForShadowReady(spawner, handle.containerId, connConfig, { + timeoutSeconds: shadowInput.healthTimeoutSeconds, + image: shadowInput.image, + }); yield* legacySetupShadowDatabase( spawner, { @@ -1223,6 +1225,7 @@ export const legacyExportBaselineCatalogRef = ( snapshot, ) : legacyWriteCatalogFile(fs, tempDir, cachePath, snapshot), + { bypassCache: params.noCache, webhooks: "config" }, ); }); @@ -1313,5 +1316,6 @@ export const legacyExportDeclarativeCatalogRef = ( timestamp, ); }), + { bypassCache: params.noCache, webhooks: "config" }, ); }); From e912767ba62fbae93c5c498d53a131f2dc1eb513 Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 19:43:20 +0200 Subject: [PATCH 67/82] feat(cli): upgrade pg-delta next to alpha.41 --- apps/cli/package.json | 2 +- .../db/shared/legacy-pgdelta-next-adapter.layer.ts | 1 + .../db/shared/legacy-pgdelta-next-adapter.service.ts | 1 + .../db/shared/legacy-pgdelta-next-adapter.unit.test.ts | 2 ++ pnpm-lock.yaml | 10 +++++----- pnpm-workspace.yaml | 2 +- 6 files changed, 11 insertions(+), 7 deletions(-) diff --git a/apps/cli/package.json b/apps/cli/package.json index 1560ea3923..9ac3b36765 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -55,7 +55,7 @@ "@parcel/watcher": "^2.6.0", "@supabase/api": "workspace:*", "@supabase/config": "workspace:*", - "@supabase/pg-delta": "1.0.0-alpha.40", + "@supabase/pg-delta": "1.0.0-alpha.41", "@supabase/pg-topo": "1.0.0-alpha.5", "@supabase/process-compose": "workspace:*", "@supabase/stack": "workspace:*", diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts index b9b45b8e5f..1bb578fec7 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts @@ -485,6 +485,7 @@ function legacyPgDeltaNextPlanOptions(input: LegacyPgDeltaNextDeclarativePlanInp profile: legacyPgDeltaNextProfile(input.schema), ...(manifest !== undefined ? { manifest } : {}), isolatedShadow: true, + ...(input.allowSameDatabaseIdentity === true ? { allowSameDatabaseIdentity: true } : {}), seedAssumedSchemas: false, strictDataStatements: true, reorder: true, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts index c2ca976152..d2c9cb6846 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.service.ts @@ -100,6 +100,7 @@ export interface LegacyPgDeltaNextDeclarativePlanInput { readonly shadowPool: Pool; readonly files: readonly LegacyPgDeltaNextSqlFile[]; readonly allowDrops: boolean; + readonly allowSameDatabaseIdentity?: boolean; readonly debug: boolean; readonly manifest?: LegacyPgDeltaNextExportManifest; readonly formatOptions?: string; diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts index 1c01317710..c0e89b4827 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts @@ -551,6 +551,7 @@ describe("LegacyPgDeltaNextAdapter", () => { shadowPool, files: exported.files, allowDrops: true, + allowSameDatabaseIdentity: true, debug: true, formatOptions: "null", }); @@ -558,6 +559,7 @@ describe("LegacyPgDeltaNextAdapter", () => { expect(state.declarativeInputs[0]).toMatchObject({ reorder: true, isolatedShadow: true, + allowSameDatabaseIdentity: true, seedAssumedSchemas: false, strictDataStatements: true, }); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 229fe866df..f4bcfbaa7d 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -155,8 +155,8 @@ importers: specifier: workspace:* version: link:../../packages/config '@supabase/pg-delta': - specifier: 1.0.0-alpha.40 - version: 1.0.0-alpha.40(@supabase/pg-topo@1.0.0-alpha.5) + specifier: 1.0.0-alpha.41 + version: 1.0.0-alpha.41(@supabase/pg-topo@1.0.0-alpha.5) '@supabase/pg-topo': specifier: 1.0.0-alpha.5 version: 1.0.0-alpha.5 @@ -2842,8 +2842,8 @@ packages: resolution: {integrity: sha512-RW/OCsd6MO592zU8ifzP8/f8XzxxIdpb+Up5XaOtE26Fw+3zTp475WX7+GuuktiD1WF8pFUDe6khUPbMp77RCw==} engines: {node: '>=22.0.0'} - '@supabase/pg-delta@1.0.0-alpha.40': - resolution: {integrity: sha512-PL1h0zdg5WQP1pw38P0d/R/k6nB0G1MFFTGENXiEn14fZv3tX3IktS3eDfPx1R7C/eOVzvxlFoR9OGmWGgRJsw==} + '@supabase/pg-delta@1.0.0-alpha.41': + resolution: {integrity: sha512-vS3rWejUJ0LYJcox3VI127kFER2yz7s357A6zaLZyGU5ObbpOjPH3gvN/u7yMUwlkE0K9kuQ/de/CuD+he5KYg==} engines: {node: '>=20.0.0'} hasBin: true peerDependencies: @@ -9094,7 +9094,7 @@ snapshots: dependencies: tslib: 2.8.1 - '@supabase/pg-delta@1.0.0-alpha.40(@supabase/pg-topo@1.0.0-alpha.5)': + '@supabase/pg-delta@1.0.0-alpha.41(@supabase/pg-topo@1.0.0-alpha.5)': dependencies: debug: 4.4.3(supports-color@7.2.0) pg: 8.22.0 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 4891efb339..796a20a30b 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -49,7 +49,7 @@ minimumReleaseAgeExclude: - "@effect/platform-node-shared@4.0.0-beta.103" - "@effect/sql-pg@4.0.0-beta.103" - "@effect/vitest@4.0.0-beta.103" - - "@supabase/pg-delta@1.0.0-alpha.40" + - "@supabase/pg-delta@1.0.0-alpha.41" - "@supabase/pg-topo@1.0.0-alpha.5" - "effect@4.0.0-beta.103" From 09d8c6ccc9037cb6639371e20921568a204a599a Mon Sep 17 00:00:00 2001 From: avallete Date: Sat, 15 Aug 2026 19:47:39 +0200 Subject: [PATCH 68/82] fix(cli): allow restored shadows to share database identity --- .../legacy-pgdelta-engine.next.layer.ts | 1 + .../legacy-pgdelta-next-shadow.layer.ts | 34 ++++++++++++++++--- ...acy-pgdelta-next-shadow.layer.unit.test.ts | 26 ++++++++++++-- .../legacy-pgdelta-next-shadow.service.ts | 2 ++ 4 files changed, 56 insertions(+), 7 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts index 5628aa04b1..73386b56bf 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts @@ -346,6 +346,7 @@ export const legacyPgDeltaNextEngineLayer = Layer.effect( shadowPool: declarativePool, files: input.files, allowDrops: true, + ...(shadow.allowSameDatabaseIdentity ? { allowSameDatabaseIdentity: true } : {}), debug: input.debug, schema: input.schema, formatOptions: input.formatOptions, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index 71a96172d3..3f136e4817 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -82,6 +82,22 @@ interface NativeShadowBase { readonly image: string; } +interface ProvisionedMigrationsShadow extends LegacyPgDeltaNextMigrationsShadow { + readonly restoredFromPgDataSnapshot: boolean; +} + +interface ProvisionedDeclarativeShadow { + readonly declarativeUrl: string; + readonly restoredFromPgDataSnapshot: boolean; +} + +export function legacyAllowSameDatabaseIdentityForRestoredShadows( + migrations: Pick, + declarative: Pick, +): boolean { + return migrations.restoredFromPgDataSnapshot && declarative.restoredFromPgDataSnapshot; +} + /** * Removes extensions that the legacy PG14 platform baseline installs implicitly * so the declarative shadow reflects only extension declarations in schema files. @@ -240,7 +256,8 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( yield* legacyMigrateNextShadowDatabase(input.spawner, setup, handle); return { migrationsUrl: legacyToPostgresURL(setup.connConfig), - } satisfies LegacyPgDeltaNextMigrationsShadow; + restoredFromPgDataSnapshot: handle.baselinePresent, + } satisfies ProvisionedMigrationsShadow; }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); const provisionDeclarative = (input: NativeShadowInput, opts: LegacyShadowCacheOpts) => @@ -258,7 +275,10 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( ); }), ); - return legacyToPostgresURL(setup.connConfig); + return { + declarativeUrl: legacyToPostgresURL(setup.connConfig), + restoredFromPgDataSnapshot: handle.baselinePresent, + } satisfies ProvisionedDeclarativeShadow; }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); const cacheOpts = ( @@ -285,13 +305,17 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( const migrationsInput = buildNativeInput(opts, built, migrationsPort); const declarativeInput = buildNativeInput(opts, built, declarativePort); const migrations = yield* provisionMigrations(migrationsInput, cacheOpts(opts, "config")); - const declarativeUrl = yield* provisionDeclarative( + const declarative = yield* provisionDeclarative( declarativeInput, cacheOpts(opts, "disabled"), ); return { - ...migrations, - declarativeUrl, + migrationsUrl: migrations.migrationsUrl, + declarativeUrl: declarative.declarativeUrl, + allowSameDatabaseIdentity: legacyAllowSameDatabaseIdentityForRestoredShadows( + migrations, + declarative, + ), } satisfies LegacyPgDeltaNextPlanShadows; }).pipe(Effect.mapError(nextShadowError)), }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts index aa40d07057..b17848ffb0 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts @@ -1,8 +1,11 @@ import { it } from "@effect/vitest"; import { Effect } from "effect"; -import { describe, expect } from "vitest"; +import { describe, expect, it as vitestIt } from "vitest"; -import { legacyPreparePgDeltaNextDeclarativeBaseline } from "./legacy-pgdelta-next-shadow.layer.ts"; +import { + legacyAllowSameDatabaseIdentityForRestoredShadows, + legacyPreparePgDeltaNextDeclarativeBaseline, +} from "./legacy-pgdelta-next-shadow.layer.ts"; function recordingSession() { const statements: string[] = []; @@ -42,3 +45,22 @@ describe("legacyPreparePgDeltaNextDeclarativeBaseline", () => { }); }); }); + +describe("legacyAllowSameDatabaseIdentityForRestoredShadows", () => { + vitestIt.each([ + { migrations: true, declarative: true, expected: true }, + { migrations: true, declarative: false, expected: false }, + { migrations: false, declarative: true, expected: false }, + { migrations: false, declarative: false, expected: false }, + ])( + "returns $expected for migrations=$migrations and declarative=$declarative", + ({ migrations, declarative, expected }) => { + expect( + legacyAllowSameDatabaseIdentityForRestoredShadows( + { restoredFromPgDataSnapshot: migrations }, + { restoredFromPgDataSnapshot: declarative }, + ), + ).toBe(expected); + }, + ); +}); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts index 9eed6db23a..d2e7e4f75e 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts @@ -14,6 +14,8 @@ export interface LegacyPgDeltaNextMigrationsShadow { export interface LegacyPgDeltaNextPlanShadows extends LegacyPgDeltaNextMigrationsShadow { /** Independent platform baseline owned by `planSchemaFiles` while loading desired SQL. */ readonly declarativeUrl: string; + /** Both databases are separate servers restored from CLI-owned PGDATA snapshots. */ + readonly allowSameDatabaseIdentity: boolean; } export interface LegacyPgDeltaNextShadowInput { From b82b010df92d83ed4af625365abda2782f564138 Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 15:27:04 +0200 Subject: [PATCH 69/82] test(cli): drop duplicated mock engine block from the develop merge Co-Authored-By: Claude Fable 5 --- .../commands/db/diff/diff.integration.test.ts | 53 ------------------- 1 file changed, 53 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index 534b220f04..ff7ef66469 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -256,59 +256,6 @@ function setup(workdir: string, opts: SetupOpts = {}) { }), ); - const explicitDiffCalls: LegacyPgDeltaExplicitDiffInput[] = []; - const databaseDiffCalls: LegacyPgDeltaDatabaseDiffInput[] = []; - const pgDeltaResult = () => { - const sql = opts.diffSql ?? ""; - const files = - opts.diffFiles !== undefined - ? opts.diffFiles.map((file, index) => ({ - sequence: index + 1, - name: file.name, - ...(opts.diffSuffixes?.[index] !== undefined - ? { suffix: opts.diffSuffixes[index] } - : {}), - sql: file.sql, - transactionMode: "transactional" as const, - })) - : sql.length > 0 - ? [ - { - sequence: 1, - name: "schema_changes", - sql, - transactionMode: "transactional" as const, - }, - ] - : []; - return { - changes: files.length > 0, - sql: opts.diffFiles !== undefined ? files.map((file) => file.sql).join("\n\n") : sql, - files, - ...(opts.hazards !== undefined ? { hazards: opts.hazards } : {}), - }; - }; - const pgDeltaEngine = Layer.succeed( - LegacyPgDeltaEngine, - LegacyPgDeltaEngine.of({ - // The handler must route through this strategy even when the selected - // implementation is legacy; the strategy owns edge runtime and shadows. - implementation: opts.pgDeltaImplementation ?? "legacy", - diffExplicit: (input) => - Effect.sync(() => { - explicitDiffCalls.push(input); - return pgDeltaResult(); - }), - diffDatabase: (input) => - Effect.sync(() => { - databaseDiffCalls.push(input); - return pgDeltaResult(); - }), - exportDeclarativeSchema: () => Effect.die("exportDeclarativeSchema unused"), - planDeclarativeSchema: () => Effect.die("planDeclarativeSchema unused"), - }), - ); - const edgeCalls: LegacyEdgeRuntimeRunOpts[] = []; const edge = Layer.succeed(LegacyEdgeRuntimeScript, { run: (runOpts: LegacyEdgeRuntimeRunOpts) => { From e06434e985a542be0aa9fa0c384006a145b0579a Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 15:52:48 +0200 Subject: [PATCH 70/82] fix(cli): harden the shadow cache against review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Bypass pg-delta's same-identity guard by snapshot lineage (same cache key), not by requiring both plan shadows to be warm restores — the first cold plan exports the tar its declarative sibling restores, and the guard rejected that clone (Codex P1). - Fold LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL into the baseline SQL digest so editing it invalidates stale pg_net baselines. - Bound legacyWaitForShadowReady by elapsed time (+ one connect allowance) so hung dials cannot stretch the wait ~3x its budget. Co-Authored-By: Claude Fable 5 --- .../legacy-pgdelta-next-shadow.layer.ts | 43 +++++++++---- ...acy-pgdelta-next-shadow.layer.unit.test.ts | 64 ++++++++++++++----- .../legacy/shared/db-bootstrap/db-setup.ts | 7 +- .../shared/db-bootstrap/health-check.ts | 48 ++++++++++++-- .../db-bootstrap/health-check.unit.test.ts | 48 +++++++++++++- .../shared/db-bootstrap/shadow-cache.ts | 19 +++++- 6 files changed, 193 insertions(+), 36 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts index 3f136e4817..24d6423513 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts @@ -83,19 +83,34 @@ interface NativeShadowBase { } interface ProvisionedMigrationsShadow extends LegacyPgDeltaNextMigrationsShadow { - readonly restoredFromPgDataSnapshot: boolean; + readonly snapshotKey: string | undefined; } interface ProvisionedDeclarativeShadow { readonly declarativeUrl: string; readonly restoredFromPgDataSnapshot: boolean; + readonly snapshotKey: string | undefined; } -export function legacyAllowSameDatabaseIdentityForRestoredShadows( - migrations: Pick, - declarative: Pick, -): boolean { - return migrations.restoredFromPgDataSnapshot && declarative.restoredFromPgDataSnapshot; +/** + * Whether pg-delta's same-database guard must be bypassed for this plan's two shadows — i.e. + * whether they can legitimately report the same PostgreSQL identity (system identifier + + * database OID). That happens exactly when the declarative shadow was physically RESTORED from + * the same snapshot key that also produced the migrations shadow's cluster: same key means same + * tar, and the migrations side is that tar's lineage whether it warm-restored FROM the tar or + * cold-exported it this very run — the baseline handoff, where requiring the migrations handle + * itself to be a warm restore would leave the guard armed against its own clone and fail the + * first cold plan (review: Codex on #6184, P1). A freshly initdb'd declarative shadow always + * carries its own new identity, and different keys mean tars exported from different clusters, + * so both of those stay `false` and keep the guard armed. A `true` alongside identities that + * happen to differ is harmless by design: pg-delta's bypass only takes effect on an exact + * identity match, never on a same-lineage sibling. + */ +export function legacyAllowSameDatabaseIdentityForPlanShadows(opts: { + readonly declarativeRestoredFromPgDataSnapshot: boolean; + readonly sameSnapshotKey: boolean; +}): boolean { + return opts.declarativeRestoredFromPgDataSnapshot && opts.sameSnapshotKey; } /** @@ -256,7 +271,7 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( yield* legacyMigrateNextShadowDatabase(input.spawner, setup, handle); return { migrationsUrl: legacyToPostgresURL(setup.connConfig), - restoredFromPgDataSnapshot: handle.baselinePresent, + snapshotKey: handle.snapshotKey, } satisfies ProvisionedMigrationsShadow; }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); @@ -278,6 +293,7 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( return { declarativeUrl: legacyToPostgresURL(setup.connConfig), restoredFromPgDataSnapshot: handle.baselinePresent, + snapshotKey: handle.snapshotKey, } satisfies ProvisionedDeclarativeShadow; }).pipe(Effect.provide(runtime), Effect.mapError(nextShadowError)); @@ -312,10 +328,15 @@ export const legacyPgDeltaNextShadowLayer = Layer.effect( return { migrationsUrl: migrations.migrationsUrl, declarativeUrl: declarative.declarativeUrl, - allowSameDatabaseIdentity: legacyAllowSameDatabaseIdentityForRestoredShadows( - migrations, - declarative, - ), + // Key equality is what encodes lineage: the declarative shadow restored the very tar + // the migrations side either restored or exported this run, so the two clusters are + // physical clones. An absent key (uncached/bypassed/uncachable) is never lineage. + allowSameDatabaseIdentity: legacyAllowSameDatabaseIdentityForPlanShadows({ + declarativeRestoredFromPgDataSnapshot: declarative.restoredFromPgDataSnapshot, + sameSnapshotKey: + migrations.snapshotKey !== undefined && + migrations.snapshotKey === declarative.snapshotKey, + }), } satisfies LegacyPgDeltaNextPlanShadows; }).pipe(Effect.mapError(nextShadowError)), }); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts index b17848ffb0..9ce2a54731 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts @@ -3,7 +3,7 @@ import { Effect } from "effect"; import { describe, expect, it as vitestIt } from "vitest"; import { - legacyAllowSameDatabaseIdentityForRestoredShadows, + legacyAllowSameDatabaseIdentityForPlanShadows, legacyPreparePgDeltaNextDeclarativeBaseline, } from "./legacy-pgdelta-next-shadow.layer.ts"; @@ -46,21 +46,53 @@ describe("legacyPreparePgDeltaNextDeclarativeBaseline", () => { }); }); -describe("legacyAllowSameDatabaseIdentityForRestoredShadows", () => { +describe("legacyAllowSameDatabaseIdentityForPlanShadows", () => { vitestIt.each([ - { migrations: true, declarative: true, expected: true }, - { migrations: true, declarative: false, expected: false }, - { migrations: false, declarative: true, expected: false }, - { migrations: false, declarative: false, expected: false }, - ])( - "returns $expected for migrations=$migrations and declarative=$declarative", - ({ migrations, declarative, expected }) => { - expect( - legacyAllowSameDatabaseIdentityForRestoredShadows( - { restoredFromPgDataSnapshot: migrations }, - { restoredFromPgDataSnapshot: declarative }, - ), - ).toBe(expected); + { + // The baseline handoff on a cold cache: the migrations shadow exported the tar this very + // run and the declarative shadow warm-restored that same key — an exact physical clone, + // so the guard has to be bypassed even though the migrations handle is not a restore. + scenario: "the declarative shadow restored the tar the migrations shadow just exported", + restored: true, + sameKey: true, + expected: true, }, - ); + { + // Both sides warm off the same tar on a later run — same lineage, same conclusion. + scenario: "both shadows warm-restored the same key", + restored: true, + sameKey: true, + expected: true, + }, + { + // A freshly initdb'd declarative shadow always carries a brand-new identity, so the guard + // stays armed no matter what the migrations side did. + scenario: "the declarative shadow was cold-provisioned", + restored: false, + sameKey: true, + expected: false, + }, + { + // Restored from a DIFFERENT key's tar: a different originating cluster, own identity. This + // also covers an absent key on either side (uncached/bypassed/uncachable acquisitions), + // which the caller folds into `sameSnapshotKey: false`. + scenario: "the shadows carry different or absent snapshot keys", + restored: true, + sameKey: false, + expected: false, + }, + { + scenario: "neither shadow came from a snapshot", + restored: false, + sameKey: false, + expected: false, + }, + ])("returns $expected when $scenario", ({ restored, sameKey, expected }) => { + expect( + legacyAllowSameDatabaseIdentityForPlanShadows({ + declarativeRestoredFromPgDataSnapshot: restored, + sameSnapshotKey: sameKey, + }), + ).toBe(expected); + }); }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts index 8688326a92..81934cde58 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts @@ -188,7 +188,12 @@ alter default privileges for role postgres in schema public revoke execute on functions from anon, authenticated, service_role; `; -const LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL = +/** + * Exported for the shadow baseline cache's embedded-SQL digest (`shadow-cache.ts`), same as + * {@link LEGACY_START_REVOKE_API_PRIVILEGES_SQL}: a webhooks-enabled baseline bakes this + * statement into PGDATA, so the digest must re-key whenever this text changes across releases. + */ +export const LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL = "create extension if not exists pg_net schema extensions;"; // The historical PG14 dump installs pg_net because later statements grant on diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts index 954f97fc9f..f82c85e373 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts @@ -10,7 +10,7 @@ * and only the final timeout's failures surface to the caller. */ -import { Clock, Data, Effect, Result, Schedule, Stream } from "effect"; +import { Clock, Data, Duration, Effect, Result, Schedule, Stream } from "effect"; import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientRequest from "effect/unstable/http/HttpClientRequest"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; @@ -514,6 +514,10 @@ export interface LegacyWaitForShadowReadyOptions { * immediately rather than at the end of the budget. * 2. a short {@link legacyProbeShadowConnect} — success ⇒ ready, return now. * + * The round count is not the only bound: the whole wait also carries a wall-clock cap, since a + * round that hangs costs its own connect timeout on top of the 1-second delay — see + * `boundSeconds` below. + * * Failure shape is deliberately identical to the health gate's: the same * {@link LegacyHealthCheckTimeoutError}, the same `unhealthy` payload, and the * same `docker logs` dump teed to stderr on the way out, so a broken shadow @@ -539,6 +543,11 @@ export function legacyWaitForShadowReady( // `legacyStreamContainerLogsOnce` above already makes for its own log-teeing. const debug = legacyShadowDebugEnabled(); + // The most recent attempt's failure, kept even once a later attempt succeeds — the debug + // summary line reports it either way (see the doc comment below on the completion line), and + // the elapsed-time bound below replays it so an expiry fails exactly like an exhausted retry. + let lastFailure: LegacyShadowReadyFailure | undefined; + const rawProbe: Effect.Effect = Effect.gen( function* () { const state = yield* legacyInspectContainerState(spawner, containerId).pipe( @@ -552,12 +561,15 @@ export function legacyWaitForShadowReady( } yield* legacyProbeShadowConnect(connConfig); }, + ).pipe( + Effect.tapError((failure) => + Effect.sync(() => { + lastFailure = failure; + }), + ), ); let attempts = 0; - // The most recent attempt's failure reason, kept even once a later attempt succeeds — the - // summary line reports it either way (see the doc comment below on the completion line). - let lastError: string | undefined; // Debug-only per-attempt line: `shadow-debug: ready-attempt ms `. // Exploration 1a's key data point — whether a cold attempt burns the full connect timeout. @@ -569,7 +581,6 @@ export function legacyWaitForShadowReady( const start = yield* Clock.currentTimeMillis; const outcome = yield* Effect.result(rawProbe); const elapsed = (yield* Clock.currentTimeMillis) - start; - if (Result.isFailure(outcome)) lastError = outcome.failure.reason; yield* Effect.sync(() => { globalThis.process.stderr.write( `shadow-debug: ready-attempt ${attemptNumber} ${elapsed}ms ${ @@ -587,8 +598,31 @@ export function legacyWaitForShadowReady( // `timeoutSeconds` retries after the initial attempt. const schedule = Schedule.max([Schedule.spaced("1 seconds"), Schedule.recurs(timeoutSeconds)]); + /** + * The wall-clock cap over the WHOLE wait. `Schedule.recurs` counts attempts, not seconds, and + * an attempt is not free: a dial that hangs burns its full + * {@link LEGACY_SHADOW_READY_CONNECT_TIMEOUT_SECONDS} before the next 1-second delay even + * starts, so a nominal 30-second budget could run ~90 seconds of real time (review: Codex on + * #6184). The cap is the schedule's own total spacing PLUS one attempt's connect allowance, so + * it lands strictly after a clean retries-exhausted finish: the retry count still decides every + * ordinary run (the last round's probe is never cut short), and this only fires when attempts + * themselves overrun. + */ + const boundSeconds = timeoutSeconds + LEGACY_SHADOW_READY_CONNECT_TIMEOUT_SECONDS; + const waited: Effect.Effect = probe.pipe( Effect.retry({ schedule, while: (failure) => !failure.fatal }), + // Expiry re-raises the last attempt's own failure (or, if nothing has failed yet, the first + // attempt is still hanging) so it flows into the same handler below — a timed-out wait + // reports the same error, the same log dump, and the same stderr text as an exhausted one. + Effect.timeoutOrElse({ + duration: Duration.seconds(boundSeconds), + orElse: () => + Effect.fail( + lastFailure ?? + legacyShadowNotReady(`not ready after ${boundSeconds}s: connection attempt hung`), + ), + }), Effect.catch((failure) => Effect.gen(function* () { // Go skips this dump on context cancellation (`start.go:215`) — an @@ -627,7 +661,9 @@ export function legacyWaitForShadowReady( const outcome = yield* Effect.result(waited); const elapsed = (yield* Clock.currentTimeMillis) - start; const lastErrorSegment = - lastError === undefined ? "" : ` last-error="${legacyShadowDebugTruncate(lastError)}"`; + lastFailure === undefined + ? "" + : ` last-error="${legacyShadowDebugTruncate(lastFailure.reason)}"`; yield* Effect.sync(() => { globalThis.process.stderr.write( `shadow-debug: ready-wait ${elapsed}ms attempts=${attempts}${lastErrorSegment}\n`, diff --git a/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts index e1374fb4bf..48cda1fdd2 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/health-check.unit.test.ts @@ -799,7 +799,9 @@ const shadowConnConfig: LegacyPgConnInput = { * the readiness probe hands nothing back to the caller: the downstream code * opens its own connection through `legacyConnectShadowDatabase` afterwards. */ -function mockShadowDbConnection(opts: { readonly failTimes?: number } = {}) { +function mockShadowDbConnection( + opts: { readonly failTimes?: number; readonly connectMillis?: number } = {}, +) { const failTimes = opts.failTimes ?? 0; const session: LegacyDbSession = { exec: () => Effect.void, @@ -814,6 +816,8 @@ function mockShadowDbConnection(opts: { readonly failTimes?: number } = {}) { connect: (cfg) => Effect.gen(function* () { attempts.push(cfg); + // A dial that hangs before answering — how a real attempt burns its own connect timeout. + if (opts.connectMillis !== undefined) yield* Effect.sleep(opts.connectMillis); if (attempts.length <= failTimes) { return yield* Effect.fail(new LegacyDbConnectError({ message: "connection refused" })); } @@ -942,6 +946,48 @@ describe("legacyWaitForShadowReady", () => { }), ); + it.effect("stops on elapsed time, not on the retry count, when every attempt hangs", () => + Effect.gen(function* () { + const mock = mockHealthSpawner(() => runningStarting); + // 1.2s per dial: the round trip is now 2.2s (dial + the 1-second backoff), so the + // `timeoutSeconds` retry count alone would keep polling until t=7.8s. + const db = mockShadowDbConnection({ + failTimes: Number.POSITIVE_INFINITY, + connectMillis: 1200, + }); + + const error = yield* withSilencedStderr( + Effect.gen(function* () { + const fiber = yield* legacyWaitForShadowReady( + mock.spawner, + SHADOW_CONTAINER_ID, + shadowConnConfig, + { timeoutSeconds: 3 }, + ).pipe(Effect.provide(db.layer), Effect.forkChild({ startImmediately: true })); + + // The whole wait's wall-clock cap: `timeoutSeconds` of backoff plus one dial's + // 2-second connect allowance. Joining here would hang if the wait were still counting + // rounds instead of seconds. + yield* TestClock.adjust("5 seconds"); + return yield* Fiber.join(fiber).pipe(Effect.flip); + }), + ); + + // Cut off mid-way through the third dial — the 4-attempt retry budget never ran out. + expect(db.attempts).toHaveLength(3); + // Same failure the exhausted path produces: the last completed attempt's own reason, + // the same ` ` message, and the same log dump. + expect(error).toBeInstanceOf(LegacyHealthCheckTimeoutError); + expect(error.message).toBe(`${SHADOW_CONTAINER_ID} connection refused`); + expect(error.unhealthy).toEqual([ + { containerId: SHADOW_CONTAINER_ID, reason: "connection refused" }, + ]); + expect( + mock.spawned.some((args) => args[0] === "logs" && args[1] === SHADOW_CONTAINER_ID), + ).toBe(true); + }), + ); + it.effect("fails fast, well inside the budget, once the container has exited", () => Effect.gen(function* () { // Running on the first round, gone on every round after it. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 3a57d9bcf7..c47e065da4 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -44,6 +44,7 @@ import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; import { legacyParseBoolEnv } from "../legacy-diff-engine.ts"; import { LEGACY_POSTGRES_DEFAULT_ROOT_KEY } from "../legacy-local-config-values.ts"; import { + LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL, LEGACY_START_REVOKE_API_PRIVILEGES_SQL, type LegacySetupDatabaseOptions, } from "./db-setup.ts"; @@ -234,6 +235,10 @@ const LEGACY_SHADOW_BASELINE_SQL_DIGEST = createHash("sha256") LEGACY_START_DB_WEBHOOK_SQL, LEGACY_START_DB_SUPABASE_SQL, LEGACY_START_REVOKE_API_PRIVILEGES_SQL, + // The webhooks-enable statement `legacySetupDatabase` runs for a webhooks-enabled baseline + // (`db-setup.ts`). `webhooksEnabled` above only says WHETHER it ran; this line covers the + // text it ran, so editing the statement re-keys those tars too (review: Codex on #6184). + LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL, // The vault upsert's own SQL (`legacyUpsertVaultSecrets`, `legacy-vault.ts`) runs into the // baseline right after the privilege pass — same digest rationale as every line above. LEGACY_READ_VAULT_KV, @@ -801,6 +806,15 @@ export interface LegacyShadowCacheOpts { */ export interface LegacyShadowAcquiredHandle extends LegacyShadowBaselineState { readonly containerId: string; + /** + * The resolved shadow-baseline cache key this handle's cluster is keyed under — present + * exactly when the acquisition was cache-eligible (a cold export or a warm restore), absent + * for an uncached, `bypassCache`d, or uncachable one. Two handles carrying the SAME key share + * the same tar's lineage: one either restored it or exported it this run, so their clusters + * are physical clones of each other. `legacy-pgdelta-next-shadow.layer.ts` reads it to decide + * whether pg-delta's same-database-identity guard must be bypassed for a plan's two shadows. + */ + readonly snapshotKey?: string; } /** A throwaway shadow with no snapshot step — the cache-off path. */ @@ -835,6 +849,7 @@ const legacyColdCachedShadow = ( legacyCreateShadowDatabase(spawner, { ...input, autoRemove: false }).pipe( Effect.map(({ containerId }) => ({ containerId, + snapshotKey: key, baselinePresent: false, snapshotRequired: true, snapshotBaseline: legacyExportShadowBaseline(spawner, input, key, tarPath, containerId), @@ -863,6 +878,7 @@ const legacyColdCachedShadow = ( const legacyWarmShadow = ( spawner: Spawner, input: LegacyShadowSetupInput, + key: string, tarPath: string, ): Effect.Effect< LegacyShadowAcquiredHandle, @@ -891,6 +907,7 @@ const legacyWarmShadow = ( ); return { containerId, + snapshotKey: key, baselinePresent: true, snapshotRequired: false, snapshotBaseline: Effect.void, @@ -963,7 +980,7 @@ export const legacyAcquireShadowDatabase = ( yield* legacySweepAbandonedShadowBaselinePartials(input); yield* legacySweepShadowBaselineRetention(input); - return yield* legacyWarmShadow(spawner, input, tarPath).pipe( + return yield* legacyWarmShadow(spawner, input, key, tarPath).pipe( Effect.catch((cause) => Effect.gen(function* () { const output = yield* Output; From b057c0e8e673992ad79981ff6d4feb833e60721a Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 15:52:53 +0200 Subject: [PATCH 71/82] docs(cli): complete generate's shadow-cache checklist, log deferred review findings Co-Authored-By: Claude Fable 5 --- .../declarative/generate/SIDE_EFFECTS.md | 38 +++++++++-------- docs/roadmap/pg-delta-next-follow-ups.md | 41 ++++++++++++++++++- 2 files changed, 60 insertions(+), 19 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index 57285e8136..03b586a01d 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -28,13 +28,14 @@ formatting without disabling safe compaction. ## Files Written -| Path | Format | When | -| ----------------------------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/supabase/schemas/**/*.sql` (default declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | -| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | -| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | -| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | legacy opt-out catalog miss, cache-enabled COLD shadow provision creates the current key's snapshot; a warm hit `touch`es its mtime; LRU/TTL may delete other keys (`SUPABASE_HOME` overrides the root; `--no-cache` neither reads nor writes) | -| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | +| Path | Format | When | +| ----------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/supabase/schemas/**/*.sql` (default declarative dir, or invocation-local `--output`) | SQL | selected destination is wiped + rewritten after confirmation | +| `/.pgdelta-export.json` | JSON | bundled-engine export metadata | +| `/supabase/.temp/pgdelta/catalog-*.json` | JSON | legacy opt-out's catalog cache | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | legacy opt-out catalog miss, cache-enabled COLD shadow provision creates the current key's snapshot; a warm hit `touch`es its mtime; LRU/TTL may delete other keys (`SUPABASE_HOME` overrides the root; `--no-cache` neither reads nor writes) | +| `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | legacy opt-out catalog miss, during a cold export — the in-flight temp file, `rename`d into the tar above on success and removed on failure; only a crash/SIGKILL leaves it behind, and later cold exports / warm hits sweep leftovers older than an hour | +| `/supabase/.temp/pgdelta/v2/debug//*.json` | JSON | bundled engine with `PGDELTA_DEBUG` | ## Subprocesses / Containers @@ -46,17 +47,18 @@ formatting without disabling safe compaction. ## Environment Variables -| Variable | Purpose | Required? | -| ---------------------------- | ----------------------------------------------------------------------------------------------------------------- | --------- | -| `SUPABASE_ACCESS_TOKEN` | auth token for `--linked` | no | -| `DB_PASSWORD` | password for `--linked` / `--db-url` | no | -| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the legacy opt-out's shadow baseline cache | no | -| `SUPABASE_SHADOW_CACHE` | shadow baseline cache for the legacy opt-out's catalog-miss shadows; ON by default, set to `false`/`0` to opt out | no | -| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | -| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | -| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | -| `SUPABASE_SERVICES_HOSTNAME` | local DB host for `--local` | no | -| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | +| Variable | Purpose | Required? | +| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | +| `SUPABASE_ACCESS_TOKEN` | auth token for `--linked` | no | +| `DB_PASSWORD` | password for `--linked` / `--db-url` | no | +| `SUPABASE_HOME` | overrides the `~/.supabase` root used for the legacy opt-out's shadow baseline cache | no | +| `SUPABASE_SHADOW_CACHE` | shadow baseline cache for the legacy opt-out's catalog-miss shadows; ON by default, set to `false`/`0` to opt out | no | +| `SUPABASE_SHADOW_DEBUG` | opt-in (default off) shadow phase-timing diagnostics on stderr (`shadow-debug:` lines) for the legacy opt-out's catalog-miss shadows; never touches stdout/exit codes | no | +| `SUPABASE_USE_PG_DELTA_NEXT` | set to `false` for legacy edge-runtime pg-delta | no | +| `PGDELTA_NPM_REGISTRY` | legacy opt-out's private npm registry | no | +| `PGDELTA_DEBUG` | bundled-engine debug artifacts | no | +| `SUPABASE_SERVICES_HOSTNAME` | local DB host for `--local` | no | +| `DOCKER_HOST` | tcp daemon host used as the local DB host fallback | no | ## Exit Codes diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md index 0246da8b82..40a884425c 100644 --- a/docs/roadmap/pg-delta-next-follow-ups.md +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -30,7 +30,12 @@ usually alongside image bumps), which is why it did not block #6184. Two candida Either way, add a unit-test mutation case alongside the existing ones in `shadow-cache.unit.test.ts`. -## Make the baseline/declarative catalog shadows use `legacyWaitForShadowReady` (PR #6184 × CLI-1970 merge) +## ~~Make the baseline/declarative catalog shadows use `legacyWaitForShadowReady`~~ (resolved in PR #6184) + +Resolved on the PR itself: both catalog provisioners now wait with `legacyWaitForShadowReady` +(`legacy-pgdelta.cache.ts`). Original write-up kept below. + +## Original: make the baseline/declarative catalog shadows use `legacyWaitForShadowReady` (PR #6184 × CLI-1970 merge) CLI-1970 (#6162) made `legacyExportBaselineCatalogRef`/`legacyExportDeclarativeCatalogRef` (`legacy-pgdelta.cache.ts`) native. `legacySetupShadowDatabase` is now baseline-state-aware @@ -42,3 +47,37 @@ sit until the first healthcheck tick. The bundled pg-delta next sync/diff shadows (`legacy-pgdelta-next-shadow.layer.ts`) already use the cache-aware acquire + `legacyWaitForShadowReady`. + +## Shadow-cache robustness follow-ups from the #6184 review (Codex, deferred) + +Five valid-but-deferred findings from the #6184 review rounds. None block the feature: each is an +edge on an already-degraded path (infra failures, races) where the cache's fail-open design keeps +the command correct, at worst at cold-provision speed. + +- **Warm readiness failures always mark the tar suspect** (`shadow-cache.ts`, + `legacyWarmShadow`): a Docker inspect/daemon hiccup during the restored container's readiness + wait reaches the unconditional `tarSuspect: true` mapping, so a valid tar can be deleted on a + pure infra blip (the cold fallback then re-exports it, so the cost is one cold run). Fix: only + set `tarSuspect` when the container stayed inspectable/running and Postgres itself failed + readiness. (Codex comment 3786040107.) +- **Snapshot-revive failures all report `reason: "docker_daemon"`** (`shadow-cache.ts`, + `legacyExportShadowBaseline`): the post-snapshot `docker start` and the follow-up readiness + wait share one error mapping, so a Postgres-side failure after a successful start is + fingerprinted as a daemon outage in telemetry. Fix: map the two stages separately. + (Codex comment 3786040112.) +- **A failed warm restore can leak the created container** (`container-lifecycle.ts` + + `shadow-cache.ts`): `legacyCreateContainer`'s post-create cleanup covers a failed archive + extraction, but a failure in the secret-file copy or the final `docker start` leaves the + created container behind while the warm→cold fallback retries with a replacement. At most one + stopped project-labeled container; `supabase stop` sweeps it. Fix: extend the tapError cleanup + to every post-create step. (Codex comment 3789148842.) +- **Cache-key JWKS resolution runs before the `Initialising schema...` banner** + (`shadow-cache.ts`, `legacyResolveShadowCacheKeyInputs`): a third-party JWKS discovery failure + on a cache-eligible acquire surfaces before the banner that the uncached flow prints first + (CLI-1956 shape). Fix: defer the key's JWKS resolution until after the prelude has printed, + reusing the resolved value for setup. (Codex comment 3789363067.) +- **`roles.sql` TOCTOU between key hash and execution** (`shadow-cache.ts` + + `db-setup.ts`): the key hashes `roles.sql` once, and `legacySetupDatabase` re-reads the path + later; an edit in that window publishes a tar under a key describing the old contents. Fix: + thread the captured contents into setup, or re-hash before publishing. (Codex comment + 3789363070.) From 0430203364a77e55dcc2ab4338e49598f2de9bfe Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 15:55:28 +0200 Subject: [PATCH 72/82] docs(cli): log the Realtime seeded-host warm-restore follow-up Co-Authored-By: Claude Fable 5 --- docs/roadmap/pg-delta-next-follow-ups.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md index 40a884425c..cfd42cf6fb 100644 --- a/docs/roadmap/pg-delta-next-follow-ups.md +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -81,3 +81,9 @@ the command correct, at worst at cold-provision speed. later; an edit in that window publishes a tar under a key describing the old contents. Fix: thread the captured contents into setup, or re-hash before publishing. (Codex comment 3789363070.) +- **Warm restores keep the cold container's id as Realtime's seeded `DB_HOST`** + (`shadow-database.ts` + `realtime-env.ts`): the cold one-shot Realtime job seeds + `_realtime.extensions` with the exporting container's 12-char id, and a warm restore (different + container, job skipped) leaves that dead id in the encrypted settings. Only migrations that + inspect or act on Realtime tenant configuration can observe the difference. Fix: normalize the + seeded host before snapshotting, or refresh it after restore. (Codex comment 3789481478.) From 40dfa4839373442ff1a90c9bc72d678640733253 Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 16:15:03 +0200 Subject: [PATCH 73/82] fix(cli): key migra-path shadow snapshots by the migrate the mode will run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit db diff/db pull hardcoded webhooks: "enabled" into the cache key even when pgdelta-next's config-following migrate would run, and db diff's explicit migrations-catalog path omitted the policy entirely while its provisioner forces pg_net on — either mismatch lets the two engines restore each other's tars. Derive the key from migrationMode at the diff/pull seams, declare "enabled" at the explicit catalog call, and drop exportViaShadowCatalog's opts default so callers must state their policy. Co-Authored-By: Claude Fable 5 --- apps/cli/src/legacy/commands/db/diff/diff.handler.ts | 9 ++++++--- apps/cli/src/legacy/commands/db/pull/pull.handler.ts | 9 ++++++--- apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts | 8 +++++++- 3 files changed, 19 insertions(+), 7 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts index 6342a2dfd1..c063c10c1e 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.handler.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.handler.ts @@ -698,8 +698,11 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // why the cache seam sits here (with `SUPABASE_SHADOW_CACHE` unset it IS today's // create/remove pair; otherwise a key-matching PGDATA snapshot is restored into the fresh // container in a few seconds instead of cold-provisioning the baseline in ~15s). - // `webhooks: "enabled"` matches `legacyMigrateShadowDatabase`'s forced `pg_net` - // baseline — the cache key must not collide with next's config-following migrate. + // The key's webhooks policy mirrors the migrate `legacyPrepareShadowSource` will actually + // select for this mode: legacy's `legacyMigrateShadowDatabase` forces `pg_net` on, next's + // `legacyMigrateNextShadowDatabase` follows project config — a key that said "enabled" for + // a config-following baseline would let the two engines restore each other's tars + // (review: Codex on #6184). diffResult = yield* legacyWithShadowDatabase( spawner, shadowInput, @@ -750,7 +753,7 @@ export const legacyDbDiff = Effect.fn("legacy.db.diff")(function* (flags: Legacy // single migration file. return { sql, files: undefined }; }), - { webhooks: "enabled" }, + { webhooks: migrationMode === "pgdelta-next" ? "config" : "enabled" }, ); } const out = diffResult.sql; diff --git a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts index 168a0a9f5b..d3266d8c0d 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.handler.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.handler.ts @@ -795,8 +795,11 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy // pooler-retry attempt still acquires and releases its own shadow — on the warm path // each attempt restores its own fresh container from the same cached snapshot, // sequentially. - // `webhooks: "enabled"` matches `legacyMigrateShadowDatabase`'s forced `pg_net` - // baseline — the cache key must not collide with next's config-following migrate. + // The key's webhooks policy mirrors the migrate `legacyPrepareShadowSource` will + // actually select for this mode: legacy's `legacyMigrateShadowDatabase` forces + // `pg_net` on, next's `legacyMigrateNextShadowDatabase` follows project config — + // a key that said "enabled" for a config-following baseline would let the two + // engines restore each other's tars (review: Codex on #6184). return yield* legacyWithShadowDatabase( spawner, shadowInput, @@ -849,7 +852,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy }); return { sql, files: undefined, debug: undefined }; }), - { webhooks: "enabled" }, + { webhooks: migrationMode === "pgdelta-next" ? "config" : "enabled" }, ); }); const diffOutcome = yield* withPoolerFallback(targetEndpoint, runShadowDiff); diff --git a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts index 4eb27a598a..11b4349e42 100644 --- a/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts +++ b/apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts @@ -770,7 +770,9 @@ const exportViaShadowCatalog = ( shadowInput: LegacyShadowSetupInput, ) => Effect.Effect, persist: (snapshot: string) => Effect.Effect, - shadowCacheOpts: LegacyShadowCacheOpts = {}, + // No default: every caller must declare its provisioner's effective webhooks policy, or the + // key records config-following while the baseline forced `pg_net` on (review: Codex on #6184). + shadowCacheOpts: LegacyShadowCacheOpts, ) => Effect.gen(function* () { const { spawner, localInputs } = built; @@ -888,6 +890,10 @@ export const legacyResolveMigrationsCatalogRef = Effect.fnUntraced(function* ( timestamp, ); }), + // `legacyProvisionMigrationsShadow` migrates via `legacyMigrateShadowDatabase`, which forces + // `pg_net` on — the key must record that, not the config-following default (no `bypassCache`: + // `db diff` has no `--no-cache` on this path). + { webhooks: "enabled" }, ); }); From 04dd1004d7347ca6e30aefbb627e9e65592957b2 Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 16:52:38 +0200 Subject: [PATCH 74/82] test(cli): pin the LRU victim's mtime in the retention test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rapid-fire publishes can land within the filesystem's timestamp granularity, making the "oldest" tar ambiguous and the eviction pick arbitrary (CI-only flake). Age the first tar explicitly — the test asserts the keep-cap, not tie-breaking. Co-Authored-By: Claude Fable 5 --- .../shared/db-bootstrap/shadow-cache.integration.test.ts | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index ae8f3d00f1..a6db3feb5d 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -744,6 +744,13 @@ describe("legacyAcquireShadowDatabase", () => { const stray = path.join(shadowCacheDir(path), "catalog-abc.json"); yield* fs.writeFileString(stray, "{}"); + // mtime is the LRU ordinal, and the rapid-fire publishes below can land within the + // filesystem's timestamp granularity — an mtime tie makes "oldest" ambiguous and the + // eviction pick arbitrary (observed as a CI-only failure). Age the first tar explicitly: + // this test asserts the keep-cap behavior, not tie-breaking. + const anHourAgo = new Date(Date.now() - 60 * 60 * 1000); + yield* fs.utimes(path.join(shadowCacheDir(path), first[0] ?? ""), anHourAgo, anHourAgo); + // Fill to the keep-cap + 1 with more distinct keys; the oldest (first) is evicted. for (let i = 0; i < LEGACY_SHADOW_BASELINE_KEEP - 1; i++) { yield* coldRun(docker, shadowInput(fs, path, { jwtExpiry: 8000 + i })); From 9970ce432ddac681f8c6aca899d353aae64da965 Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 17:42:50 +0200 Subject: [PATCH 75/82] test(cli): pin the handlers' mode-matching shadow-cache webhooks policy db diff and db pull each gain a cache-enabled scenario proving a legacy-engine baseline tar is never restored into a pg-delta-next run (and vice versa each publishes its own key). Mutation-verified: reverting either call site to a hardcoded webhooks: "enabled" fails the new test at the warm-restore assertion. The shadow-cache suite's stateful Docker model is hoisted into tests/helpers for reuse; the stateless shadow spawner mock stays byte-identical for existing callers. Co-Authored-By: Claude Fable 5 --- .../commands/db/diff/diff.integration.test.ts | 72 ++++- .../commands/db/pull/pull.integration.test.ts | 79 ++++- .../shadow-cache.integration.test.ts | 293 +++--------------- apps/cli/tests/helpers/legacy-mocks.ts | 240 +++++++++++++- 4 files changed, 424 insertions(+), 260 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts index ff7ef66469..394309e36b 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts @@ -12,7 +12,9 @@ import { LEGACY_VALID_REF, legacyFailWriteStringMatchingFsLayer, legacyFailWriteStringOnNthCallFsLayer, + legacyWithEnv, mockLegacyCliConfig, + mockLegacyDockerDaemonCliSpawner, mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, mockLegacyTelemetryStateTracked, @@ -139,6 +141,10 @@ interface SetupOpts { // host-gateway` (Linux-only). Defaults to `"linux"` (every other test's implicit // baseline); pass `"darwin"`/`"win32"` to exercise the no-add-host branch. readonly platform?: NodeJS.Platform; + // Swaps the stateless shadow spawner for the stateful Docker model, whose + // `stop`/`cp`/`start` really move bytes. Required by (and only by) the tests that + // enable the shadow BASELINE CACHE — see `mockLegacyDockerDaemonCliSpawner`. + readonly statefulDocker?: boolean; } const alwaysReadyHttpClientLayer = Layer.succeed( @@ -199,6 +205,11 @@ function setup(workdir: string, opts: SetupOpts = {}) { dbNotRunning: opts.dbNotRunning ?? false, dbInspectFailsWith: opts.dbInspectFailsWith, }); + // The shadow baseline cache's cold export and warm restore only mean anything against a + // daemon that actually holds container state and carries `docker cp` bytes, so the cache + // tests below opt into the stateful model instead. + const dockerDaemon = + opts.statefulDocker === true ? mockLegacyDockerDaemonCliSpawner() : undefined; const shadowDbConnection = fakeShadowDbConnection({ neverConnectableShadow: opts.neverConnectableShadow ?? false, }); @@ -434,7 +445,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { edge, docker, shadowDbConnection.layer, - shadowSpawner.layer, + dockerDaemon?.layer ?? shadowSpawner.layer, alwaysReadyHttpClientLayer, resolver, projectRefResolver, @@ -480,6 +491,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { differRegistryEnvAtCall, shadowSetupJobCalls, shadowSpawned: shadowSpawner.spawned, + dockerDaemon, shadowConnectedDatabases: shadowDbConnection.connectedDatabases, shadowExecCalls: shadowDbConnection.execCalls, }; @@ -2580,4 +2592,62 @@ describe("legacy db diff", () => { }, ); }); + + describe("shadow baseline cache", () => { + /** The `.tar` files published under the per-test `SUPABASE_HOME` this block pins. */ + const publishedTars = () => { + const dir = join(tmp.current, "_supabase_home", "cache", "shadow-baseline"); + return existsSync(dir) ? readdirSync(dir).filter((entry) => entry.endsWith(".tar")) : []; + }; + + /** + * Runs `db diff` with the shadow baseline cache ENABLED (this file pins it off for every + * other test) and its artifacts isolated under the workdir, against the stateful Docker + * model the export/restore round trip needs. + */ + const runCached = (implementation: "legacy" | "next") => { + const s = setup(tmp.current, { + statefulDocker: true, + pgDeltaImplementation: implementation, + diffSql: "create table t ();\n", + }); + return legacyWithEnv( + "SUPABASE_HOME", + join(tmp.current, "_supabase_home"), + legacyWithEnv( + "SUPABASE_SHADOW_CACHE", + "1", + legacyDbDiff(flags({ usePgDelta: Option.some(true) })).pipe(Effect.provide(s.layer)), + ), + ).pipe(Effect.as(s)); + }; + + // Regression: both migrate paths used to pass a hardcoded `{ webhooks: "enabled" }`, so the + // legacy run's forced-`pg_net` baseline and the next run's config-following baseline keyed + // to the SAME tar and silently restored each other's cluster. The handler now forks the + // policy on `migrationMode`; `shadow-cache.integration.test.ts` covers the cache's half of + // the contract, this covers `db diff`'s call site. + it.live("a legacy-engine baseline is never restored into a pg-delta-next run", () => { + mkdirSync(join(tmp.current, "supabase"), { recursive: true }); + writeFileSync( + join(tmp.current, "supabase", "config.toml"), + "[experimental.pgdelta]\nenabled = true\n", + ); + return Effect.gen(function* () { + // Legacy migrate forces `pg_net` on regardless of config, and publishes that baseline. + const legacyRun = yield* runCached("legacy"); + expect(legacyRun.dockerDaemon?.stepCalls("cp-out")).toHaveLength(1); + const legacyTars = publishedTars(); + expect(legacyTars).toHaveLength(1); + + // pg-delta next follows the config (webhooks are off here), so it must cold-provision + // and publish its OWN baseline rather than restore the forced-on one above. + const nextRun = yield* runCached("next"); + expect(nextRun.dockerDaemon?.stepCalls("cp-in")).toHaveLength(0); + expect(nextRun.dockerDaemon?.stepCalls("cp-out")).toHaveLength(1); + expect(publishedTars()).toHaveLength(2); + expect(publishedTars()).toEqual(expect.arrayContaining(legacyTars)); + }); + }); + }); }); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts index 8d479b471e..11cadf9de3 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts @@ -9,7 +9,9 @@ import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; import { stripAnsi } from "../../../../../tests/helpers/ansi.ts"; import { LEGACY_VALID_REF, + legacyWithEnv, mockLegacyCliConfig, + mockLegacyDockerDaemonCliSpawner, mockLegacyLinkedProjectCacheTracked, mockLegacyShadowContainerCliSpawner, mockLegacyTelemetryStateTracked, @@ -129,6 +131,10 @@ interface SetupOpts { // `LegacyProjectNotLinkedError` absent an explicit `--project-ref` flag, // instead of silently falling back to `opts.resolvedRef ?? LEGACY_VALID_REF`. readonly linkedFails?: boolean; + // Swaps the stateless shadow spawner for the stateful Docker model, whose + // `stop`/`cp`/`start` really move bytes. Required by (and only by) the tests that + // enable the shadow BASELINE CACHE — see `mockLegacyDockerDaemonCliSpawner`. + readonly statefulDocker?: boolean; } function setup(workdir: string, opts: SetupOpts = {}) { @@ -141,6 +147,11 @@ function setup(workdir: string, opts: SetupOpts = {}) { // A real docker-spawner fake backs container create/start/health-inspect/cleanup. const shadowSpawner = mockLegacyShadowContainerCliSpawner(); + // The shadow baseline cache's cold export and warm restore only mean anything against a + // daemon that actually holds container state and carries `docker cp` bytes, so the cache + // tests below opt into the stateful model instead. + const dockerDaemon = + opts.statefulDocker === true ? mockLegacyDockerDaemonCliSpawner() : undefined; const engineCalls: Array<{ operation: "diff" | "export"; @@ -440,7 +451,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { edge, docker, dbConnection, - shadowSpawner.layer, + dockerDaemon?.layer ?? shadowSpawner.layer, alwaysReadyHttpClientLayer, resolver, projectRefResolver, @@ -477,6 +488,7 @@ function setup(workdir: string, opts: SetupOpts = {}) { dumpCalls, engineCalls, shadowSpawned: shadowSpawner.spawned, + dockerDaemon, get edgeRunCount() { return edgeRunCount; }, @@ -2267,4 +2279,69 @@ describe("legacy db pull", () => { expect(Exit.isFailure(exit)).toBe(true); }).pipe(Effect.provide(s.layer)); }); + + describe("shadow baseline cache", () => { + /** The `.tar` files published under the per-test `SUPABASE_HOME` this block pins. */ + const publishedTars = () => { + const dir = join(tmp.current, "_supabase_home", "cache", "shadow-baseline"); + return existsSync(dir) ? readdirSync(dir).filter((entry) => entry.endsWith(".tar")) : []; + }; + + /** + * Runs `db pull` with the shadow baseline cache ENABLED (this file pins it off for every + * other test) and its artifacts isolated under the temp root, against the stateful Docker + * model the export/restore round trip needs. + * + * Each run gets its OWN workdir so the migration file the previous pull wrote cannot shift + * the second run's behaviour — the cache key is global and deliberately workdir-independent, + * so two worktrees with identical settings still collide on the same tar. + */ + const runCached = (implementation: "legacy" | "next") => { + const workdir = join(tmp.current, `${implementation}-worktree`); + seedMigration(workdir, "20240101000000"); + writeFileSync( + join(workdir, "supabase", "config.toml"), + "[experimental.pgdelta]\nenabled = true\n", + ); + const s = setup(workdir, { + statefulDocker: true, + engineImplementation: implementation, + remoteVersions: ["20240101000000"], + edgeStdout: pgDeltaDiffEnvelope([{ name: "schema_changes", sql: "create table t ();" }]), + yes: true, + }); + return legacyWithEnv( + "SUPABASE_HOME", + join(tmp.current, "_supabase_home"), + legacyWithEnv( + "SUPABASE_SHADOW_CACHE", + "1", + legacyDbPull(flags()).pipe(Effect.provide(s.layer)), + ), + ).pipe(Effect.as(s)); + }; + + // Regression: both migrate paths used to pass a hardcoded `{ webhooks: "enabled" }`, so the + // legacy run's forced-`pg_net` baseline and the next run's config-following baseline keyed + // to the SAME tar and silently restored each other's cluster. The handler now forks the + // policy on `migrationMode`; `shadow-cache.integration.test.ts` covers the cache's half of + // the contract, this covers `db pull`'s call site. + it.live("a legacy-engine baseline is never restored into a pg-delta-next run", () => { + return Effect.gen(function* () { + // Legacy migrate forces `pg_net` on regardless of config, and publishes that baseline. + const legacyRun = yield* runCached("legacy"); + expect(legacyRun.dockerDaemon?.stepCalls("cp-out")).toHaveLength(1); + const legacyTars = publishedTars(); + expect(legacyTars).toHaveLength(1); + + // pg-delta next follows the config (webhooks are off here), so it must cold-provision + // and publish its OWN baseline rather than restore the forced-on one above. + const nextRun = yield* runCached("next"); + expect(nextRun.dockerDaemon?.stepCalls("cp-in")).toHaveLength(0); + expect(nextRun.dockerDaemon?.stepCalls("cp-out")).toHaveLength(1); + expect(publishedTars()).toHaveLength(2); + expect(publishedTars()).toEqual(expect.arrayContaining(legacyTars)); + }); + }); + }); }); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index a6db3feb5d..c2d964c15b 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -16,21 +16,14 @@ import type { ProjectConfig } from "@supabase/config"; import { ProjectConfigSchema } from "@supabase/config"; import { BunServices } from "@effect/platform-bun"; import { describe, expect, it } from "@effect/vitest"; +import { Effect, Exit, FileSystem, Layer, Option, Path, Schema } from "effect"; + import { - Effect, - Exit, - FileSystem, - Layer, - Option, - Path, - Predicate, - Schema, - Sink, - Stream, -} from "effect"; -import { ChildProcess, ChildProcessSpawner } from "effect/unstable/process"; - -import { useLegacyTempWorkdir } from "../../../../tests/helpers/legacy-mocks.ts"; + LEGACY_FAKE_PGDATA_TAR, + legacyWithEnv, + mockLegacyDockerDaemonCliSpawner, + useLegacyTempWorkdir, +} from "../../../../tests/helpers/legacy-mocks.ts"; import { mockOutput } from "../../../../tests/helpers/mocks.ts"; import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import { LegacyDbConnectError } from "../legacy-db-connection.errors.ts"; @@ -51,29 +44,8 @@ const defaultConfig: ProjectConfig = decodeConfig({}); const tempRoot = useLegacyTempWorkdir("legacy-shadow-cache-"); -/** Sets an env var for the duration of `body`, restoring whatever the host had. */ -const withEnv = ( - name: string, - value: string | undefined, - body: Effect.Effect, -): Effect.Effect => - Effect.acquireUseRelease( - Effect.sync(() => { - const previous = process.env[name]; - if (value === undefined) delete process.env[name]; - else process.env[name] = value; - return previous; - }), - () => body, - (previous) => - Effect.sync(() => { - if (previous === undefined) delete process.env[name]; - else process.env[name] = previous; - }), - ); - const withShadowCacheEnv = (value: string | undefined, body: Effect.Effect) => - withEnv(LEGACY_SHADOW_CACHE_ENV, value, body); + legacyWithEnv(LEGACY_SHADOW_CACHE_ENV, value, body); /** * Isolates the global shadow-baseline cache under a per-test `SUPABASE_HOME` so tests never @@ -83,14 +55,14 @@ const withShadowCacheHome = ( value: string | undefined, body: Effect.Effect, ): Effect.Effect => - withEnv( + legacyWithEnv( "SUPABASE_HOME", join(tempRoot.current, "_supabase_home"), withShadowCacheEnv(value, body), ); const withShadowDebugEnv = (value: string | undefined, body: Effect.Effect) => - withEnv(LEGACY_SHADOW_DEBUG_ENV, value, body); + legacyWithEnv(LEGACY_SHADOW_DEBUG_ENV, value, body); /** * Captures every write `body` makes directly to the real `process.stderr` — the channel @@ -118,199 +90,6 @@ const captureStderr = ( return { result, writes }; }); -// --------------------------------------------------------------------------- -// A minimal, stateful Docker model -// --------------------------------------------------------------------------- - -/** The bytes the fake `docker cp :PGDATA -` emits — stands in for a real ~90MB PGDATA tar. */ -const FAKE_PGDATA_TAR = "data/PG_VERSION\n17\n"; - -interface FakeContainer { - readonly labels: Readonly>; - readonly autoRemove: boolean; - running: boolean; - /** Whether this container has ever been `docker start`ed — distinguishes a RE-start for `failRestart`. */ - everStarted?: boolean; - /** What a previous `docker cp - :` unpacked into this container, if anything. */ - restored: string | undefined; -} - -function fakeDockerDaemon( - opts: { - readonly failStart?: boolean; - /** Fails only RE-starts (a `docker start` after the container has already run once) — the cold export's revive step. */ - readonly failRestart?: boolean; - readonly failCopyOut?: boolean; - readonly failCopyIn?: boolean; - } = {}, -) { - const containers = new Map(); - const spawned: Array> = []; - const encoder = new TextEncoder(); - const decoder = new TextDecoder(); - let nextId = 0; - - /** - * Drains a `Stream` passed as a command's `stdin` into a string, the way the real - * `NodeChildProcessSpawner` runs a user-supplied stdin stream into the child's stdin sink — a - * fake that ignored it would never notice the restore tar was not actually delivered. - */ - const readStdin = Effect.fnUntraced(function* (command: ChildProcess.Command) { - const configured = command._tag === "StandardCommand" ? command.options.stdin : undefined; - // A caller may pass either a bare `CommandInput` or a `StdinConfig` wrapping one. - const input = Stream.isStream(configured) - ? configured - : Predicate.hasProperty(configured, "stream") - ? configured.stream - : undefined; - if (!Stream.isStream(input)) return ""; - return yield* Stream.runFold( - input, - () => "", - (text: string, chunk) => text + decoder.decode(chunk as Uint8Array, { stream: true }), - ).pipe(Effect.orElseSucceed(() => "")); - }); - - const spawner = ChildProcessSpawner.make((command) => - Effect.gen(function* () { - const args = command._tag === "StandardCommand" ? command.args : []; - spawned.push(args); - - let exitCode = 0; - let stdout = ""; - let stderr = ""; - - if (args[0] === "network" && args[1] === "inspect") { - exitCode = 1; - } else if (args[0] === "create") { - nextId += 1; - const id = `shadowcontainer${String(nextId).padStart(2, "0")}`.padEnd(64, "0"); - const labels: Record = {}; - for (let index = 0; index < args.length; index += 1) { - if (args[index] !== "--label") continue; - const [key = "", ...rest] = (args[index + 1] ?? "").split("="); - labels[key] = rest.join("="); - } - containers.set(id, { - labels, - autoRemove: args.includes("--rm"), - running: false, - restored: undefined, - }); - stdout = id; - } else if (args[0] === "start") { - const container = containers.get(args[1] ?? ""); - if ( - opts.failStart === true || - container === undefined || - (opts.failRestart === true && container.everStarted) - ) { - exitCode = 1; - } else { - container.running = true; - container.everStarted = true; - } - } else if (args[0] === "stop") { - const id = args[1] ?? ""; - const container = containers.get(id); - if (container === undefined) exitCode = 1; - else { - container.running = false; - // Docker destroys an `--rm` container the moment it exits, `docker stop` included — - // verified against Docker 29. This is exactly why the cold export path drops `--rm`. - if (container.autoRemove) containers.delete(id); - } - } else if (args[0] === "rm") { - containers.delete(args[args.length - 1] ?? ""); - } else if (args[0] === "cp" && args[1] === "-") { - // Secret copy is `docker cp - :/`; restore is `docker cp - :`. - // Both use stdin, so failCopyIn must apply only to the restore — otherwise a warm - // fallback test kills the pgsodium root-key copy and never reaches the archive. - const [id = "", containerPath = ""] = (args[2] ?? "").split(":"); - const container = containers.get(id); - const received = yield* readStdin(command); - const isSecret = containerPath === "" || containerPath === "/"; - if (container === undefined || (!isSecret && opts.failCopyIn === true)) { - exitCode = 1; - stderr = "no such container"; - } else if (!isSecret) { - container.restored = `${containerPath}::${received}`; - } - } else if (args[0] === "cp" && args[2] === "-") { - // Export: `docker cp : -`, tar on stdout. - const [id = ""] = (args[1] ?? "").split(":"); - const container = containers.get(id); - if (opts.failCopyOut === true || container === undefined) { - exitCode = 1; - stderr = "no such container"; - } else { - stdout = FAKE_PGDATA_TAR; - } - } else if (args[0] === "container" && args[1] === "inspect") { - const container = containers.get(args[2] ?? ""); - exitCode = container === undefined ? 1 : 0; - stdout = - container === undefined - ? "" - : JSON.stringify({ - Running: container.running, - Status: container.running ? "running" : "exited", - Health: { Status: "healthy" }, - }); - } - - return ChildProcessSpawner.makeHandle({ - pid: ChildProcessSpawner.ProcessId(1), - // `docker cp … -` writes the raw archive, every other call a trailing newline. - stdout: Stream.fromIterable(stdout.length > 0 ? [encoder.encode(stdout)] : []), - stderr: Stream.fromIterable(stderr.length > 0 ? [encoder.encode(stderr)] : []), - all: Stream.empty, - exitCode: Effect.succeed(ChildProcessSpawner.ExitCode(exitCode)), - isRunning: Effect.succeed(false), - stdin: Sink.drain, - kill: () => Effect.void, - unref: Effect.succeed(Effect.void), - getInputFd: () => Sink.drain, - getOutputFd: () => Stream.empty, - }); - }), - ); - - /** - * One readable label per Docker call, so a test can assert the SEQUENCE of meaningful steps - * rather than raw argv. `cp` is split three ways because the shadow issues three different - * copies: the pgsodium root key every shadow gets (`cp-secret`, `container-lifecycle.ts`), the - * baseline export (`cp-out`), and the baseline restore (`cp-in`). - */ - const stepOf = (args: ReadonlyArray): string => { - if (args[0] === "network") return "network"; - if (args[0] === "container" && args[1] === "inspect") return "inspect"; - if (args[0] === "cp") { - if (args[1] === "-") { - const dest = args[2] ?? ""; - const containerPath = dest.slice(dest.indexOf(":") + 1); - return containerPath === "" || containerPath === "/" ? "cp-secret" : "cp-in"; - } - if (args[2] === "-") return "cp-out"; - return "cp-secret"; - } - return args[0] ?? ""; - }; - - return { - spawner, - spawned, - containers, - /** Every argv whose first token is `verb` (`create`/`rm`/`stop`/`start`/`cp`). */ - calls: (verb: string) => spawned.filter((args) => args[0] === verb), - /** Every argv classified as `step` — see {@link stepOf}. */ - stepCalls: (step: string) => spawned.filter((args) => stepOf(args) === step), - /** The full step sequence, with the `network` bookkeeping calls dropped as noise. */ - steps: () => spawned.map(stepOf).filter((step) => step !== "network"), - ids: () => [...containers.keys()], - }; -} - // --------------------------------------------------------------------------- // A fake Postgres the readiness probe can connect to // --------------------------------------------------------------------------- @@ -399,7 +178,7 @@ const soleTarName = Effect.fnUntraced(function* (fs: FileSystem.FileSystem, path /** A full cold run: acquire, export the baseline, release. */ const coldRun = ( - docker: ReturnType, + docker: ReturnType, input: LegacyShadowSetupInput, opts: LegacyShadowCacheOpts = {}, ) => @@ -412,7 +191,7 @@ const coldRun = ( describe("legacyAcquireShadowDatabase", () => { it.live("is today's bare create when the cache is explicitly disabled", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -439,7 +218,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("bypassCache acquires an uncached shadow even when a warm tar exists", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -467,7 +246,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("stays uncached on PG14, whose setup mutates role defaults mid-session", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -495,7 +274,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("a warm hit also sweeps abandoned partials left by a killed concurrent writer", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -524,7 +303,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("stays uncached for an OrioleDB cluster even with the cache enabled", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -549,7 +328,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("takes the cache path when the env var is unset (default ON)", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -565,7 +344,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("cold run stops, exports the tar, and starts the container again", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -605,7 +384,7 @@ describe("legacyAcquireShadowDatabase", () => { expect(tars).toHaveLength(1); expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); expect(yield* fs.readFileString(path.join(shadowCacheDir(path), tars[0] ?? ""))).toBe( - FAKE_PGDATA_TAR, + LEGACY_FAKE_PGDATA_TAR, ); const leftovers = yield* fs.readDirectory(shadowCacheDir(path)); expect(leftovers.filter((entry) => entry.includes("partial"))).toEqual([]); @@ -618,7 +397,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("warm run restores the tar into a FRESH container before starting it", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -645,7 +424,7 @@ describe("legacyAcquireShadowDatabase", () => { `${warm.containerId}:${LEGACY_PGDATA_PARENT_PATH}`, ]); expect(docker.containers.get(warm.containerId)?.restored).toBe( - `${LEGACY_PGDATA_PARENT_PATH}::${FAKE_PGDATA_TAR}`, + `${LEGACY_PGDATA_PARENT_PATH}::${LEGACY_FAKE_PGDATA_TAR}`, ); // Nothing more is exported: the baseline is already on disk. @@ -657,7 +436,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("a pre-created permissive temp file cannot leak into the published tar's mode", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -684,13 +463,13 @@ describe("legacyAcquireShadowDatabase", () => { const info = yield* fs.stat(tarPath); // 0o600 exactly — not the pre-created file's 0o666. expect((Number(info.mode) & 0o777).toString(8)).toBe("600"); - expect(yield* fs.readFileString(tarPath)).toBe(FAKE_PGDATA_TAR); + expect(yield* fs.readFileString(tarPath)).toBe(LEGACY_FAKE_PGDATA_TAR); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); it.live("a cold export sweeps abandoned partial temp files but never fresh ones", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -719,7 +498,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("publishing distinct keys keeps both tars until LRU/TTL eviction", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -764,7 +543,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("worktrees with identical settings share a warm hit from the global cache", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -795,7 +574,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("a changed published host port is still a warm hit", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -821,7 +600,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("legacy forced-on webhooks and next config-following webhooks do not share a tar", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -856,7 +635,7 @@ describe("legacyAcquireShadowDatabase", () => { }); it.live("a changed internal image registry is a different key, not a warm hit", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -872,7 +651,7 @@ describe("legacyAcquireShadowDatabase", () => { // The one-shot migrate jobs resolve their images through // `SUPABASE_INTERNAL_IMAGE_REGISTRY`, so a different registry can bake different // realtime/storage/auth schema under identical tags — the snapshot must not be shared. - const mirrored = yield* withEnv( + const mirrored = yield* legacyWithEnv( "SUPABASE_INTERNAL_IMAGE_REGISTRY", "mirror.internal.example", coldRun(docker, input), @@ -890,7 +669,7 @@ describe("legacyAcquireShadowDatabase", () => { it.live( "a shadow that cannot come back after the snapshot fails the run, not just the cache", () => { - const docker = fakeDockerDaemon({ failRestart: true }); + const docker = mockLegacyDockerDaemonCliSpawner({ failRestart: true }); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -912,7 +691,7 @@ describe("legacyAcquireShadowDatabase", () => { ); it.live("a failed export warns, leaves no tar, and still brings the container back up", () => { - const docker = fakeDockerDaemon({ failCopyOut: true }); + const docker = mockLegacyDockerDaemonCliSpawner({ failCopyOut: true }); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -937,7 +716,7 @@ describe("legacyAcquireShadowDatabase", () => { it.live( "a failed warm restore falls back cold, keeping the tar until its export replaces it", () => { - const docker = fakeDockerDaemon({ failCopyIn: true }); + const docker = mockLegacyDockerDaemonCliSpawner({ failCopyIn: true }); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -971,7 +750,7 @@ describe("legacyAcquireShadowDatabase", () => { ); it.live("a restored shadow that never becomes ready is removed before the cold retry", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const out = mockOutput(); return withShadowCacheHome( "1", @@ -1007,7 +786,7 @@ describe("legacyAcquireShadowDatabase", () => { describe("SUPABASE_SHADOW_DEBUG phase-timing instrumentation", () => { it.live("emits export and restore phase lines when the debug env var is set", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( @@ -1038,7 +817,7 @@ describe("SUPABASE_SHADOW_DEBUG phase-timing instrumentation", () => { }); it.live("emits no shadow-debug lines when the debug env var is unset", () => { - const docker = fakeDockerDaemon(); + const docker = mockLegacyDockerDaemonCliSpawner(); const cluster = fakeCluster(); const out = mockOutput(); return withShadowCacheHome( diff --git a/apps/cli/tests/helpers/legacy-mocks.ts b/apps/cli/tests/helpers/legacy-mocks.ts index 11e5c66ffb..34391ac032 100644 --- a/apps/cli/tests/helpers/legacy-mocks.ts +++ b/apps/cli/tests/helpers/legacy-mocks.ts @@ -4,8 +4,9 @@ import { join } from "node:path"; import { BunServices } from "@effect/platform-bun"; import { type ApiClient, makeApiClient, type SupabaseApiConfigError } from "@supabase/api/effect"; -import { Effect, FileSystem, Layer, Option, Redacted, Sink, Stream } from "effect"; +import { Effect, FileSystem, Layer, Option, Predicate, Redacted, Sink, Stream } from "effect"; import { PlatformError, SystemError } from "effect/PlatformError"; +import type { ChildProcess } from "effect/unstable/process"; import { ChildProcessSpawner } from "effect/unstable/process"; import * as HttpClient from "effect/unstable/http/HttpClient"; import * as HttpClientError from "effect/unstable/http/HttpClientError"; @@ -698,6 +699,33 @@ export function useLegacyTempWorkdir(prefix = "supabase-legacy-test-"): { }; } +/** + * Sets `name` to `value` (or unsets it when `value` is `undefined`) for the duration of `body`, + * restoring whatever was there before — including whatever a surrounding `beforeEach` such as + * {@link useLegacyShadowCacheDisabled} put there. Scoping the override to the effect rather than + * to a nested `describe` keeps it independent of vitest's hook ordering, so a single test can + * opt back INTO a variable its file pins off. + */ +export const legacyWithEnv = ( + name: string, + value: string | undefined, + body: Effect.Effect, +): Effect.Effect => + Effect.acquireUseRelease( + Effect.sync(() => { + const previous = process.env[name]; + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + return previous; + }), + () => body, + (previous) => + Effect.sync(() => { + if (previous === undefined) delete process.env[name]; + else process.env[name] = previous; + }), + ); + /** * Pins `SUPABASE_SHADOW_CACHE=0` for every test in the calling file, restoring whatever the host * had afterwards. Like {@link useLegacyTempWorkdir} it calls vitest's `beforeEach`/`afterEach` @@ -966,6 +994,216 @@ export function mockLegacyShadowContainerCliSpawner( return { layer, spawned }; } +// --------------------------------------------------------------------------- +// A minimal, stateful Docker model — the shadow BASELINE CACHE's round trip +// (`docker stop` -> `docker cp :PGDATA -` -> `docker start`, and the warm +// `docker cp - :` restore) really moves bytes, which +// `mockLegacyShadowContainerCliSpawner` above deliberately does not model. +// --------------------------------------------------------------------------- + +/** The bytes the fake `docker cp :PGDATA -` emits — stands in for a real ~90MB PGDATA tar. */ +export const LEGACY_FAKE_PGDATA_TAR = "data/PG_VERSION\n17\n"; + +interface LegacyFakeContainer { + readonly labels: Readonly>; + readonly autoRemove: boolean; + running: boolean; + /** Whether this container has ever been `docker start`ed — distinguishes a RE-start for `failRestart`. */ + everStarted?: boolean; + /** What a previous `docker cp - :` unpacked into this container, if anything. */ + restored: string | undefined; +} + +/** + * Every `docker` call a shadow provision issues, backed by ONE stateful container table: + * `create`/`start`/`stop`/`rm` mutate it, and `docker cp` really carries bytes in and out. Use + * this instead of {@link mockLegacyShadowContainerCliSpawner} whenever the shadow BASELINE CACHE + * is enabled for the test — the cold export's stop/copy-out/start round trip and the warm + * restore's copy-in have no meaning against a stateless spawner, and a failed export is + * fail-open (a warning, no tar), so a stateless fake makes cache assertions pass vacuously. + * + * `container inspect supabase_db_` reports "no such container" here (the table only + * holds shadows), i.e. the local stack is DOWN — fine for every non-`--use-pgadmin` path, which + * never issues that probe. + */ +export function mockLegacyDockerDaemonCliSpawner( + opts: { + readonly failStart?: boolean; + /** Fails only RE-starts (a `docker start` after the container has already run once) — the cold export's revive step. */ + readonly failRestart?: boolean; + readonly failCopyOut?: boolean; + readonly failCopyIn?: boolean; + } = {}, +) { + const containers = new Map(); + const spawned: Array> = []; + const encoder = new TextEncoder(); + const decoder = new TextDecoder(); + let nextId = 0; + + /** + * Drains a `Stream` passed as a command's `stdin` into a string, the way the real + * `NodeChildProcessSpawner` runs a user-supplied stdin stream into the child's stdin sink — a + * fake that ignored it would never notice the restore tar was not actually delivered. + */ + const readStdin = Effect.fnUntraced(function* (command: ChildProcess.Command) { + const configured = command._tag === "StandardCommand" ? command.options.stdin : undefined; + // A caller may pass either a bare `CommandInput` or a `StdinConfig` wrapping one. + const input = Stream.isStream(configured) + ? configured + : Predicate.hasProperty(configured, "stream") + ? configured.stream + : undefined; + if (!Stream.isStream(input)) return ""; + return yield* Stream.runFold( + input, + () => "", + (text: string, chunk) => text + decoder.decode(chunk as Uint8Array, { stream: true }), + ).pipe(Effect.orElseSucceed(() => "")); + }); + + const spawner = ChildProcessSpawner.make((command) => + Effect.gen(function* () { + const args = command._tag === "StandardCommand" ? command.args : []; + spawned.push(args); + + let exitCode = 0; + let stdout = ""; + let stderr = ""; + + if (args[0] === "network" && args[1] === "inspect") { + exitCode = 1; + } else if (args[0] === "create") { + nextId += 1; + const id = `shadowcontainer${String(nextId).padStart(2, "0")}`.padEnd(64, "0"); + const labels: Record = {}; + for (let index = 0; index < args.length; index += 1) { + if (args[index] !== "--label") continue; + const [key = "", ...rest] = (args[index + 1] ?? "").split("="); + labels[key] = rest.join("="); + } + containers.set(id, { + labels, + autoRemove: args.includes("--rm"), + running: false, + restored: undefined, + }); + stdout = id; + } else if (args[0] === "start") { + const container = containers.get(args[1] ?? ""); + if ( + opts.failStart === true || + container === undefined || + (opts.failRestart === true && container.everStarted) + ) { + exitCode = 1; + } else { + container.running = true; + container.everStarted = true; + } + } else if (args[0] === "stop") { + const id = args[1] ?? ""; + const container = containers.get(id); + if (container === undefined) exitCode = 1; + else { + container.running = false; + // Docker destroys an `--rm` container the moment it exits, `docker stop` included — + // verified against Docker 29. This is exactly why the cold export path drops `--rm`. + if (container.autoRemove) containers.delete(id); + } + } else if (args[0] === "rm") { + containers.delete(args[args.length - 1] ?? ""); + } else if (args[0] === "cp" && args[1] === "-") { + // Secret copy is `docker cp - :/`; restore is `docker cp - :`. + // Both use stdin, so failCopyIn must apply only to the restore — otherwise a warm + // fallback test kills the pgsodium root-key copy and never reaches the archive. + const [id = "", containerPath = ""] = (args[2] ?? "").split(":"); + const container = containers.get(id); + const received = yield* readStdin(command); + const isSecret = containerPath === "" || containerPath === "/"; + if (container === undefined || (!isSecret && opts.failCopyIn === true)) { + exitCode = 1; + stderr = "no such container"; + } else if (!isSecret) { + container.restored = `${containerPath}::${received}`; + } + } else if (args[0] === "cp" && args[2] === "-") { + // Export: `docker cp : -`, tar on stdout. + const [id = ""] = (args[1] ?? "").split(":"); + const container = containers.get(id); + if (opts.failCopyOut === true || container === undefined) { + exitCode = 1; + stderr = "no such container"; + } else { + stdout = LEGACY_FAKE_PGDATA_TAR; + } + } else if (args[0] === "container" && args[1] === "inspect") { + const container = containers.get(args[2] ?? ""); + exitCode = container === undefined ? 1 : 0; + stdout = + container === undefined + ? "" + : JSON.stringify({ + Running: container.running, + Status: container.running ? "running" : "exited", + Health: { Status: "healthy" }, + }); + } + + return ChildProcessSpawner.makeHandle({ + pid: ChildProcessSpawner.ProcessId(1), + // `docker cp … -` writes the raw archive, every other call a trailing newline. + stdout: Stream.fromIterable(stdout.length > 0 ? [encoder.encode(stdout)] : []), + stderr: Stream.fromIterable(stderr.length > 0 ? [encoder.encode(stderr)] : []), + all: Stream.empty, + exitCode: Effect.succeed(ChildProcessSpawner.ExitCode(exitCode)), + isRunning: Effect.succeed(false), + stdin: Sink.drain, + kill: () => Effect.void, + unref: Effect.succeed(Effect.void), + getInputFd: () => Sink.drain, + getOutputFd: () => Stream.empty, + }); + }), + ); + + /** + * One readable label per Docker call, so a test can assert the SEQUENCE of meaningful steps + * rather than raw argv. `cp` is split three ways because the shadow issues three different + * copies: the pgsodium root key every shadow gets (`cp-secret`, `container-lifecycle.ts`), the + * baseline export (`cp-out`), and the baseline restore (`cp-in`). + */ + const stepOf = (args: ReadonlyArray): string => { + if (args[0] === "network") return "network"; + if (args[0] === "container" && args[1] === "inspect") return "inspect"; + if (args[0] === "cp") { + if (args[1] === "-") { + const dest = args[2] ?? ""; + const containerPath = dest.slice(dest.indexOf(":") + 1); + return containerPath === "" || containerPath === "/" ? "cp-secret" : "cp-in"; + } + if (args[2] === "-") return "cp-out"; + return "cp-secret"; + } + return args[0] ?? ""; + }; + + return { + /** Ready to merge into a test runtime; `spawner` is the bare handler for direct callers. */ + layer: Layer.succeed(ChildProcessSpawner.ChildProcessSpawner, spawner), + spawner, + spawned, + containers, + /** Every argv whose first token is `verb` (`create`/`rm`/`stop`/`start`/`cp`). */ + calls: (verb: string) => spawned.filter((args) => args[0] === verb), + /** Every argv classified as `step` — see {@link stepOf}. */ + stepCalls: (step: string) => spawned.filter((args) => stepOf(args) === step), + /** The full step sequence, with the `network` bookkeeping calls dropped as noise. */ + steps: () => spawned.map(stepOf).filter((step) => step !== "network"), + ids: () => [...containers.keys()], + }; +} + // --------------------------------------------------------------------------- // Runtime composition — bundles the entire Layer.mergeAll(...) graph that // every native-port integration test re-builds, including the easy-to-mis-wire From 9baed89bbc083d9e3f0c18ea3e4dd0fd90df56be Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 18:09:48 +0200 Subject: [PATCH 76/82] fix(cli): validate restored shadow tars, canonicalize the API-grants key token MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - A warm hit now scans the tar's entry headers for the cluster marker (data/PG_VERSION) before restoring: an empty or foreign-but-valid tar no longer initdbs a bare cluster that reports baselinePresent and skips setup — it is discarded as suspect and the run cold-provisions. - The cache key hashes the EFFECTIVE api.auto_expose_new_tables behavior (grants kept vs revoked): unset and explicit false execute identical revoke SQL, so they no longer force separate ~90MB tars. - Docs: generate joins the cache divergence entry; roles.sql cache-key reads listed across diff/pull/sync/generate SIDE_EFFECTS. Co-Authored-By: Claude Fable 5 --- apps/cli/docs/go-cli-divergences.md | 9 +- .../legacy/commands/db/diff/SIDE_EFFECTS.md | 2 +- .../legacy/commands/db/pull/SIDE_EFFECTS.md | 2 +- .../declarative/generate/SIDE_EFFECTS.md | 1 + .../schema/declarative/sync/SIDE_EFFECTS.md | 2 +- .../shared/db-bootstrap/pgdata-snapshot.ts | 223 +++++++++++++++++- .../db-bootstrap/pgdata-snapshot.unit.test.ts | 131 ++++++++++ .../shadow-cache.integration.test.ts | 40 ++++ .../shared/db-bootstrap/shadow-cache.ts | 60 ++++- .../db-bootstrap/shadow-cache.unit.test.ts | 19 +- apps/cli/tests/helpers/legacy-mocks.ts | 58 ++++- 11 files changed, 522 insertions(+), 25 deletions(-) create mode 100644 apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts diff --git a/apps/cli/docs/go-cli-divergences.md b/apps/cli/docs/go-cli-divergences.md index bc442bc17e..b7784563cf 100644 --- a/apps/cli/docs/go-cli-divergences.md +++ b/apps/cli/docs/go-cli-divergences.md @@ -86,7 +86,8 @@ These commands exist in the TS CLI today but have no direct top-level equivalent ## Behavioral divergences from the Go reference -- `db diff`/`db pull`/`db schema declarative sync` shadow baseline cache (#6184): the shadow +- `db diff`/`db pull`/`db schema declarative sync`/`db schema declarative generate` shadow + baseline cache (#6184): the shadow database's platform baseline is cached as a PGDATA snapshot under `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` (~90MB; `SUPABASE_HOME` overrides the root; LRU keep-8 + 14-day mtime TTL, shared across worktrees with the same settings) and @@ -94,8 +95,10 @@ These commands exist in the TS CLI today but have no direct top-level equivalent fresh container on later runs, cutting shadow provisioning from ~15s to a few seconds. The key includes the effective Webhooks/`pg_net` policy (legacy migrate forces enabled; next migrate follows config; next declarative forces disabled). Covers - migra/`db pull` via `legacyWithShadowDatabase`, and the bundled pg-delta next sync/diff - shadows via `legacyAcquireShadowDatabase` (ephemeral host ports are not part of the cache + migra/`db pull` via `legacyWithShadowDatabase`, the bundled pg-delta next sync/diff + shadows via `legacyAcquireShadowDatabase`, and `generate`/`sync`/`diff`'s legacy pg-delta + opt-out (`SUPABASE_USE_PG_DELTA_NEXT=false`), whose catalog exports provision a shadow through + the same acquire on a catalog cache miss (ephemeral host ports are not part of the cache key — they are not baked into PGDATA). TS-only, default ON; `SUPABASE_SHADOW_CACHE=false`/`=0` opts out (ambient env or project dotenv), `sync --no-cache` bypasses it per-invocation, and `SUPABASE_SHADOW_DEBUG=1` prints stderr-only phase diff --git a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md index 8f3d5d76ad..f075c41d17 100644 --- a/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md @@ -25,7 +25,7 @@ it, and JSON `null` disables formatting without disabling safe compaction. | `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (all native targets, and the explicit `--from/--to migrations` cache miss) | | `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | | `/supabase/migrations/*.sql` | SQL | shadow provisioning (applied to the shadow source) — `--use-pgadmin` too, via the SAME `legacyMigrateShadowDatabase` | -| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); missing file tolerated | +| `/supabase/roles.sql` | SQL | shadow provisioning, PG14 and PG15 alike (unlike `db reset`'s PG15-only local path); also hashed into the shadow-baseline cache key on every cache-eligible acquire, warm hits included (where no baseline is applied at all); missing file tolerated | | `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | | `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | abandoned-partial sweep on every cache-eligible acquire (warm hit and cold export) — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | | `[db.migrations].schema_paths` globs / `/supabase/database/**` / `/supabase/schemas/**` | SQL | legacy engines only, for the local-target declarative-schema fallback; pg-delta next always compares the migrations baseline directly to the live target | diff --git a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md index d28c298cdd..1bddbe510c 100644 --- a/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md @@ -41,7 +41,7 @@ disables formatting without disabling safe compaction. | `/supabase/.env`, `.env.local`, project-root/`SUPABASE_ENV`-selected dotenv file | dotenv | shadow provisioning (`--declarative` and migration-style pull; not the delegated `--experimental` structured-dump path) | | `api.tls.cert_path` / `api.tls.key_path` (under `/supabase/`) | PEM | shadow provisioning, when `api.enabled && api.tls.enabled` | | `/supabase/migrations/*.sql` | SQL | history reconciliation + shadow provisioning | -| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); missing file tolerated | +| `/supabase/roles.sql` | SQL | migration-style pull only (`--declarative`'s bare shadow skips `SetupDatabase`); also hashed into the shadow-baseline cache key on every cache-eligible acquire, warm hits included (where no baseline is applied at all); missing file tolerated | | `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit (migration-style pull) — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | | `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | abandoned-partial sweep on every cache-eligible acquire (warm hit and cold export) — enumerated and `stat`ed, and removed when older than an hour (a crashed/SIGKILLed earlier export's leftover) | | `~/.supabase/access-token` | plain text | linked target with no `SUPABASE_ACCESS_TOKEN` | diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md index 03b586a01d..bf322f47ee 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md @@ -21,6 +21,7 @@ formatting without disabling safe compaction. | `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | | `/supabase/.temp/postgres-version` | plain text | legacy opt-out's shadow-DB image resolution | | `/supabase/migrations/*.sql` | SQL | smart mode — detect whether migrations exist | +| `/supabase/roles.sql` | SQL | legacy opt-out — hashed into the catalog cache key, and on a catalog miss also into the shadow-baseline cache key (on warm hits too, not just cold ones) and applied to a cold shadow's baseline; missing file tolerated (hashed as empty) | | `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's catalog cache | | `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | legacy opt-out catalog miss, warm shadow-cache hit — the matching snapshot is streamed into the fresh shadow; every cache-eligible acquire also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU/TTL (`SUPABASE_HOME` overrides the root) | | `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar..partial` | tar | legacy opt-out catalog miss — abandoned-partial sweep on every cache-eligible acquire; removed when older than an hour | diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md index b428c3b32f..edb37e6a17 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md @@ -23,7 +23,7 @@ disabling safe compaction. | `/supabase/.temp/edge-runtime-version` | plain text | legacy opt-out's edge-runtime image tag | | `/supabase/schemas/**/*.sql` (default declarative dir) | SQL | always — must exist (else error) | | `/supabase/migrations/*.sql` | SQL | bundled engine applies them to a live shadow; legacy opt-out resolves a migrations catalog | -| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent) | +| `/supabase/roles.sql` | SQL | legacy migrations-catalog cache key (empty when absent); separately hashed into the shadow-baseline cache key on every cache-eligible acquire — bundled-engine shadows and the legacy opt-out's catalog miss alike, warm hits included — and applied to a cold shadow's baseline | | `/supabase/schemas/.pgdelta-export.json` | JSON | bundled export metadata, when present | | `/supabase/.temp/pgdelta/*.json` | JSON | legacy opt-out's migrations/declarative catalog cache | | `~/.supabase/cache/shadow-baseline/shadow-baseline-.tar` | tar | warm shadow-cache hit — bundled-engine migrations/declarative shadows, and the legacy opt-out's catalog miss; every cache-eligible acquire (warm hit and successful cold export) also enumerates and `stat`s every `shadow-baseline-*.tar` for LRU keep-8 + 14-day mtime TTL and may delete other keys (`SUPABASE_HOME` overrides the `~/.supabase` root) | diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts index a709f60d0a..8dbcc172b6 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts @@ -28,7 +28,7 @@ * well — nothing about the format is container-specific. */ -import { Effect, Stream, type FileSystem } from "effect"; +import { Effect, Option, Stream, type FileSystem } from "effect"; import type { ChildProcessSpawner } from "effect/unstable/process/ChildProcessSpawner"; import { @@ -150,6 +150,227 @@ export const legacyExportPgDataTar = ( }).pipe(Effect.onError(() => fs.remove(tempPath).pipe(Effect.orElseSucceed(() => undefined)))); }; +// --------------------------------------------------------------------------- +// Archive validation +// --------------------------------------------------------------------------- + +/** + * PGDATA's own directory name — `docker cp : -` names its members after the source + * BASENAME, so `data/` is the export tar's top-level entry (see {@link LEGACY_PGDATA_PARENT_PATH}). + */ +const LEGACY_PGDATA_DIR_NAME = LEGACY_PGDATA_PATH.slice(LEGACY_PGDATA_PARENT_PATH.length + 1); + +/** + * The one entry whose presence proves an archive really carries an exported cluster: every + * Postgres data directory has a `PG_VERSION` file at its root, and `initdb` writes it first. An + * archive that unpacks cleanly but lacks it is the corruption a restore cannot otherwise notice — + * see {@link legacyPgDataArchiveHasCluster}. + */ +export const LEGACY_PGDATA_MARKER_ENTRY = `${LEGACY_PGDATA_DIR_NAME}/PG_VERSION`; + +/** POSIX tar's fixed block size: headers, file content, and the end marker are all multiples of it. */ +const LEGACY_TAR_BLOCK_SIZE = 512; + +const LEGACY_TAR_NO_BYTES = new Uint8Array(0); + +const legacyTarDecoder = new TextDecoder(); + +/** + * {@link legacyScanTarChunkForEntry}'s carry-over state — everything needed to resume a header + * walk at an arbitrary chunk boundary, and nothing else. `carry` holds the bytes of a header block + * a chunk ended in the middle of (always `< 512`); `skip` counts the file-content bytes still to be + * STEPPED OVER without buffering, which is what keeps a ~90MB archive off the heap. + */ +export interface LegacyTarScanState { + readonly carry: Uint8Array; + readonly skip: number; + /** Consecutive all-zero blocks seen; two in a row is tar's end-of-archive marker. */ + readonly zeroBlocks: number; + readonly found: boolean; + readonly ended: boolean; + /** A block that is neither zero nor a checksum-valid header: not a tar (or a truncated one). */ + readonly malformed: boolean; +} + +export const legacyInitialTarScanState: LegacyTarScanState = { + carry: LEGACY_TAR_NO_BYTES, + skip: 0, + zeroBlocks: 0, + found: false, + ended: false, + malformed: false, +}; + +/** Whether the scan has reached a verdict — nothing later in the archive can change it. */ +export const legacyTarScanSettled = (state: LegacyTarScanState): boolean => + state.found || state.ended || state.malformed; + +/** A NUL-terminated text field of a tar header block. */ +const legacyTarTextField = (block: Uint8Array, offset: number, length: number): string => { + const raw = block.subarray(offset, offset + length); + const end = raw.indexOf(0); + return legacyTarDecoder.decode(end === -1 ? raw : raw.subarray(0, end)); +}; + +/** + * A numeric header field: NUL/space-padded octal, or GNU's base-256 form (high bit of the first + * byte) for sizes past what 11 octal digits can hold. `undefined` when neither parses. + */ +const legacyTarNumericField = ( + block: Uint8Array, + offset: number, + length: number, +): number | undefined => { + const first = block[offset] ?? 0; + if ((first & 0x80) !== 0) { + let value = first & 0x7f; + for (let index = offset + 1; index < offset + length; index += 1) { + value = value * 256 + (block[index] ?? 0); + } + return value; + } + const text = legacyTarTextField(block, offset, length).trim(); + if (text.length === 0) return 0; + if (!/^[0-7]+$/u.test(text)) return undefined; + return Number.parseInt(text, 8); +}; + +/** + * Tar's own integrity check on a header block: the stored checksum is the sum of all 512 bytes + * with the checksum field itself read as spaces. Both the unsigned and the (historical) signed + * summation are accepted, as every tar reader does. This is what tells a genuine header apart from + * arbitrary bytes, so a non-tar file cannot be walked as if it were one. + */ +const legacyTarChecksumValid = (block: Uint8Array): boolean => { + const stored = legacyTarNumericField(block, 148, 8); + if (stored === undefined) return false; + let unsigned = 0; + let signed = 0; + for (let index = 0; index < LEGACY_TAR_BLOCK_SIZE; index += 1) { + const byte = index >= 148 && index < 156 ? 0x20 : (block[index] ?? 0); + unsigned += byte; + signed += byte > 127 ? byte - 256 : byte; + } + return stored === unsigned || stored === signed; +}; + +/** The header's full member path: ustar's `prefix` field rejoined, with a leading `./` dropped. */ +const legacyTarEntryName = (block: Uint8Array): string => { + const name = legacyTarTextField(block, 0, 100); + const prefix = legacyTarTextField(block, 345, 155); + const joined = prefix.length > 0 ? `${prefix}/${name}` : name; + return joined.startsWith("./") ? joined.slice(2) : joined; +}; + +const legacyTarBlockIsZero = (block: Uint8Array): boolean => block.every((byte) => byte === 0); + +/** + * Folds one stream chunk into a tar HEADER walk looking for `entryName`. Pure and + * chunk-boundary-agnostic: file content is stepped over by byte count rather than buffered, so the + * whole scan costs one partial header block of memory no matter how large the archive is. Stops + * (and stays stopped) at the first of: the entry found, the end-of-archive marker, or a block that + * is not a valid header. + */ +export const legacyScanTarChunkForEntry = ( + state: LegacyTarScanState, + chunk: Uint8Array, + entryName: string, +): LegacyTarScanState => { + if (legacyTarScanSettled(state)) return state; + const settle = ( + verdict: Pick, + ): LegacyTarScanState => ({ + ...legacyInitialTarScanState, + ...verdict, + }); + let carry = state.carry; + let skip = state.skip; + let zeroBlocks = state.zeroBlocks; + // Content bytes carried over from the previous chunk come first — they are not headers. + let offset = Math.min(skip, chunk.length); + skip -= offset; + while (offset < chunk.length) { + const available = chunk.length - offset; + let block: Uint8Array; + if (carry.length > 0) { + const take = Math.min(LEGACY_TAR_BLOCK_SIZE - carry.length, available); + const merged = new Uint8Array(carry.length + take); + merged.set(carry); + merged.set(chunk.subarray(offset, offset + take), carry.length); + offset += take; + if (merged.length < LEGACY_TAR_BLOCK_SIZE) { + carry = merged; + break; + } + carry = LEGACY_TAR_NO_BYTES; + block = merged; + } else if (available < LEGACY_TAR_BLOCK_SIZE) { + carry = chunk.slice(offset); + break; + } else { + block = chunk.subarray(offset, offset + LEGACY_TAR_BLOCK_SIZE); + offset += LEGACY_TAR_BLOCK_SIZE; + } + + if (legacyTarBlockIsZero(block)) { + zeroBlocks += 1; + if (zeroBlocks >= 2) return settle({ found: false, ended: true, malformed: false }); + continue; + } + zeroBlocks = 0; + if (!legacyTarChecksumValid(block)) { + return settle({ found: false, ended: false, malformed: true }); + } + if (legacyTarEntryName(block) === entryName) { + return settle({ found: true, ended: false, malformed: false }); + } + const size = legacyTarNumericField(block, 124, 12); + if (size === undefined || size < 0) { + return settle({ found: false, ended: false, malformed: true }); + } + // Content is padded up to the next block boundary; directories and links carry size 0. + const content = Math.ceil(size / LEGACY_TAR_BLOCK_SIZE) * LEGACY_TAR_BLOCK_SIZE; + const stepped = Math.min(content, chunk.length - offset); + offset += stepped; + skip = content - stepped; + } + return { carry, skip, zeroBlocks, found: false, ended: false, malformed: false }; +}; + +/** + * Whether `tarPath` really is a {@link legacyExportPgDataTar} archive — i.e. whether its member + * list contains {@link LEGACY_PGDATA_MARKER_ENTRY}. + * + * This exists because an archive that is syntactically fine but carries no cluster (an EMPTY tar + * qualifies) restores SILENTLY: `docker cp -` extracts nothing, the Postgres entrypoint finds an + * empty PGDATA and runs a fresh `initdb`, readiness passes, and the caller is handed a bare + * cluster it believes carries the platform baseline. Validating the header stream up front is the + * only place that difference is observable, so callers must check BEFORE restoring. + * + * Reads the file locally — no Docker, no extraction — and stops at the marker, so a warm hit + * normally touches only the archive's first blocks. Only a genuine read failure fails; a valid tar + * without the marker simply resolves `false`. + */ +export const legacyPgDataArchiveHasCluster = ( + fs: FileSystem.FileSystem, + tarPath: string, +): Effect.Effect => + fs.stream(tarPath).pipe( + Stream.mapAccum( + () => legacyInitialTarScanState, + (state: LegacyTarScanState, chunk: Uint8Array) => { + const next = legacyScanTarChunkForEntry(state, chunk, LEGACY_PGDATA_MARKER_ENTRY); + return [next, [next]] as const; + }, + ), + Stream.takeUntil(legacyTarScanSettled), + Stream.runLast, + Effect.map((last) => Option.isSome(last) && last.value.found), + Effect.mapError((cause) => + legacyPgDataSnapshotUnavailable(`failed to read ${tarPath}: ${cause.message}`), + ), + ); + /** * Builds the {@link LegacyStartContainerSpec.preStartArchives} entry that restores a * {@link legacyExportPgDataTar} tar into a container between `docker create` and `docker start`. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts new file mode 100644 index 0000000000..74128bc09b --- /dev/null +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts @@ -0,0 +1,131 @@ +/** + * The pure tar-header walk behind {@link legacyPgDataArchiveHasCluster}'s pre-restore check. Unit + * tests rather than integration ones because the interesting cases are all in the format handling: + * chunk boundaries landing mid-header, content that must be stepped over rather than parsed, and + * bytes that are not a tar at all. + */ + +import { describe, expect, it } from "@effect/vitest"; + +import { + LEGACY_PGDATA_MARKER_ENTRY, + legacyInitialTarScanState, + legacyScanTarChunkForEntry, + legacyTarScanSettled, + type LegacyTarScanState, +} from "./pgdata-snapshot.ts"; + +const BLOCK = 512; + +const encoder = new TextEncoder(); + +const tarBlock = (fields: ReadonlyArray): Uint8Array => { + const block = new Uint8Array(BLOCK); + for (const [offset, text] of fields) block.set(encoder.encode(text), offset); + return block; +}; + +const octal = (value: number, width: number) => `${value.toString(8).padStart(width, "0")}\0`; + +/** One ustar member: a checksummed header block plus its NUL-padded content blocks. */ +const tarEntry = (name: string, content: string, typeFlag = "0"): Uint8Array => { + const header = tarBlock([ + [0, name], + [100, octal(0o600, 7)], + [108, octal(0, 7)], + [116, octal(0, 7)], + [124, octal(content.length, 11)], + [136, octal(0, 11)], + [148, " "], + [156, typeFlag], + [257, "ustar\0"], + [263, "00"], + ]); + let checksum = 0; + for (const byte of header) checksum += byte; + header.set(encoder.encode(`${octal(checksum, 6)} `), 148); + const padded = new Uint8Array(Math.ceil(content.length / BLOCK) * BLOCK); + padded.set(encoder.encode(content)); + const out = new Uint8Array(header.length + padded.length); + out.set(header); + out.set(padded, header.length); + return out; +}; + +const TAR_END = new Uint8Array(2 * BLOCK); + +const concat = (...parts: ReadonlyArray): Uint8Array => { + const total = parts.reduce((sum, part) => sum + part.length, 0); + const out = new Uint8Array(total); + let offset = 0; + for (const part of parts) { + out.set(part, offset); + offset += part.length; + } + return out; +}; + +/** Feeds `tar` through the scanner in fixed-size chunks, stopping as the real stream would. */ +const scan = (tar: Uint8Array, chunkSize: number, entry = LEGACY_PGDATA_MARKER_ENTRY) => { + let state: LegacyTarScanState = legacyInitialTarScanState; + for (let offset = 0; offset < tar.length; offset += chunkSize) { + state = legacyScanTarChunkForEntry(state, tar.subarray(offset, offset + chunkSize), entry); + if (legacyTarScanSettled(state)) break; + } + return state; +}; + +describe("legacyScanTarChunkForEntry", () => { + const pgdataTar = concat( + tarEntry("data/", "", "5"), + tarEntry("data/postgresql.conf", "listen_addresses = '*'\n"), + tarEntry(LEGACY_PGDATA_MARKER_ENTRY, "17\n"), + TAR_END, + ); + + it("finds the cluster marker regardless of where the chunk boundaries fall", () => { + // 7 and 513 both split header blocks; the marker is the third member, behind a directory and + // a file whose content must be stepped over rather than mistaken for a header. + for (const chunkSize of [1, 7, 100, 512, 513, 4096, pgdataTar.length]) { + const state = scan(pgdataTar, chunkSize); + expect(state.found, `chunk size ${chunkSize}`).toBe(true); + expect(state.malformed).toBe(false); + } + }); + + it("stops at the marker without walking the rest of the archive", () => { + // Everything after the marker is garbage: reaching it would settle `malformed` instead. + const trailing = concat( + tarEntry(LEGACY_PGDATA_MARKER_ENTRY, "17\n"), + encoder.encode("x".repeat(2048)), + ); + expect(scan(trailing, 512).found).toBe(true); + }); + + it("reports a valid but cluster-less archive as not found", () => { + expect(scan(TAR_END, 512)).toMatchObject({ found: false, ended: true, malformed: false }); + const otherEntries = concat(tarEntry("data/base/1/2345", "rows"), TAR_END); + expect(scan(otherEntries, 512)).toMatchObject({ found: false, ended: true }); + }); + + it("never mistakes file content for a header", () => { + // A member whose CONTENT is itself a valid `data/PG_VERSION` header — only the real member + // list counts, so this archive must come back cluster-less. + const decoy = new TextDecoder().decode(tarEntry(LEGACY_PGDATA_MARKER_ENTRY, "17\n")); + expect(scan(concat(tarEntry("data/decoy", decoy), TAR_END), 512).found).toBe(false); + }); + + it("settles as malformed on bytes that are not a tar", () => { + expect(scan(encoder.encode("x".repeat(4096)), 512).malformed).toBe(true); + // A header whose checksum does not add up (a flipped byte in an otherwise real archive). + const corrupted = concat(tarEntry("data/PG_VERSION", "17\n"), TAR_END); + corrupted[5] = 0x41; + expect(scan(corrupted, 512).malformed).toBe(true); + }); + + it("treats a truncated trailing block as the end of what it can read", () => { + // No end marker and a half header: nothing found, but nothing claimed either. + const truncated = concat(tarEntry("data/base/1/2345", "rows")).subarray(0, BLOCK + 100); + expect(scan(truncated, 512)).toMatchObject({ found: false, ended: false, malformed: false }); + }); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index c2d964c15b..f9800156bf 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -19,6 +19,7 @@ import { describe, expect, it } from "@effect/vitest"; import { Effect, Exit, FileSystem, Layer, Option, Path, Schema } from "effect"; import { + LEGACY_FAKE_EMPTY_TAR, LEGACY_FAKE_PGDATA_TAR, legacyWithEnv, mockLegacyDockerDaemonCliSpawner, @@ -749,6 +750,45 @@ describe("legacyAcquireShadowDatabase", () => { }, ); + it.live("a published tar carrying no cluster is discarded instead of restored", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + yield* coldRun(docker, input); + const [tarName = ""] = yield* soleTarName(fs, path); + const tarPath = path.join(shadowCacheDir(path), tarName); + + // The published artifact is replaced by a tar that is perfectly well-formed but carries no + // PGDATA: `docker cp -` would extract nothing, the entrypoint would `initdb` a fresh + // cluster, readiness would pass, and the caller would diff against a BARE database while + // being told the platform baseline was present. + yield* fs.writeFileString(tarPath, LEGACY_FAKE_EMPTY_TAR); + const stepsBefore = docker.steps().length; + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); + + expect(out.stderrText).toContain("cached shadow baseline unusable"); + expect(out.stderrText).toContain("data/PG_VERSION"); + expect(fallback.baselinePresent).toBe(false); + // Caught before any container was created, so nothing was ever restored. + expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); + // The contents ARE the problem, so the tar goes — and the cold fallback republishes a + // good one within the same run, which is what keeps this fail-open. + expect(yield* soleTarName(fs, path)).toEqual([]); + yield* fallback.snapshotBaseline; + expect(yield* soleTarName(fs, path)).toHaveLength(1); + expect(yield* fs.readFileString(tarPath)).toBe(LEGACY_FAKE_PGDATA_TAR); + yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("a restored shadow that never becomes ready is removed before the cold retry", () => { const docker = mockLegacyDockerDaemonCliSpawner(); const out = mockOutput(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index c47e065da4..2d2d8df5ce 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -58,7 +58,12 @@ import { type LegacyVaultSecret, } from "../legacy-vault.ts"; import { legacyWaitForShadowReady } from "./health-check.ts"; -import { legacyExportPgDataTar, legacyPgDataRestoreArchive } from "./pgdata-snapshot.ts"; +import { + LEGACY_PGDATA_MARKER_ENTRY, + legacyExportPgDataTar, + legacyPgDataArchiveHasCluster, + legacyPgDataRestoreArchive, +} from "./pgdata-snapshot.ts"; import type { LegacyPgDataSnapshotUnavailable } from "./pgdata-snapshot.ts"; import { legacyResolvePinnedImage } from "./pinned-image.ts"; import { legacyTimeShadowPhase } from "./shadow-debug.ts"; @@ -86,9 +91,11 @@ export const LEGACY_SHADOW_CACHE_ENV = "SUPABASE_SHADOW_CACHE"; interface LegacyShadowCacheUnavailable { readonly reason: string; /** - * `true` only when the failure implicates the TAR'S CONTENTS — today, exactly one producer: a - * restored cluster that started but never accepted connections ({@link legacyWarmShadow}'s - * readiness wait). Everything else (a `docker create`/`cp`/`start` failure — daemon outage, + * `true` only when the failure implicates the TAR'S CONTENTS — today, exactly two producers, + * both in {@link legacyWarmShadow}: an archive whose header stream carries no PGDATA cluster + * (checked before any container is created), and a restored cluster that started but never + * accepted connections (the readiness wait). Everything else (a `docker create`/`cp`/`start` + * failure — daemon outage, * port collision, or even a corrupt archive's failed extraction) leaves the tar in place: an * infra failure says nothing about the tar, and a genuinely corrupt one is atomically * REPLACED by the cold fallback's own export in the same run, so deleting up front would only @@ -174,7 +181,10 @@ export interface LegacyShadowCacheKeyInputs { */ readonly storageTargetMigration: string; readonly dbSettings: ProjectConfig["db"]["settings"]; - /** Effective `api.auto_expose_new_tables` tri-state (unset ≠ explicit `false`: only the former keeps the bundled grants). */ + /** + * `api.auto_expose_new_tables` as config carries it. Hashed as the EFFECTIVE two-state behavior, + * not the raw tri-state — see {@link legacyEffectiveShadowApiGrantsKept}. + */ readonly autoExposeNewTables: Option.Option; /** * Effective Webhooks/`pg_net` policy baked into the cluster — the same boolean @@ -261,8 +271,16 @@ function legacyCanonicalJson(value: unknown): string { const legacyBoolToken = (value: boolean) => (value ? "true" : "false"); -const legacyTriStateToken = (value: Option.Option) => - Option.isNone(value) ? "unset" : legacyBoolToken(value.value); +/** + * The two-state behavior `legacyApplyApiPrivileges` (`db-setup.ts`) actually derives from + * `api.auto_expose_new_tables`' tri-state: it returns early ONLY for an explicit `true`, so unset + * and explicit `false` both exec {@link LEGACY_START_REVOKE_API_PRIVILEGES_SQL} and bake the exact + * same cluster. Hashing the raw tri-state would split those two into different keys and force a + * spurious ~90MB re-snapshot for a config edit that changes nothing on disk (review: Codex on + * #6184). + */ +const legacyEffectiveShadowApiGrantsKept = (value: Option.Option): boolean => + Option.getOrElse(value, () => false); /** * The cache key: a 16-hex-char (64-bit) sha256 prefix over a fixed field order. 64 bits is @@ -288,7 +306,7 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string `root_key=${quoted(inputs.rootKey)}`, `db_password=${quoted(inputs.dbPassword)}`, `db_settings=${legacyCanonicalJson(inputs.dbSettings)}`, - `auto_expose_new_tables=${legacyTriStateToken(inputs.autoExposeNewTables)}`, + `api_grants_kept=${legacyBoolToken(legacyEffectiveShadowApiGrantsKept(inputs.autoExposeNewTables))}`, `webhooks_enabled=${legacyBoolToken(inputs.webhooksEnabled)}`, // Not a per-run input — see the digest's own doc comment for what it covers and why. `baseline_sql_digest=${LEGACY_SHADOW_BASELINE_SQL_DIGEST}`, @@ -857,10 +875,10 @@ const legacyColdCachedShadow = ( ); /** - * The warm path proper: create the shadow with the snapshot tar unpacked into it before it starts - * ({@link LegacyCreateShadowDatabaseInput.restoreArchive}), then wait for the restored Postgres. - * Every failure resolves to {@link LegacyShadowCacheUnavailable}, which the caller turns into the - * escape hatch. + * The warm path proper: verify the snapshot tar really contains a cluster, create the shadow with + * it unpacked into it before it starts ({@link LegacyCreateShadowDatabaseInput.restoreArchive}), + * then wait for the restored Postgres. Every failure resolves to + * {@link LegacyShadowCacheUnavailable}, which the caller turns into the escape hatch. * * The readiness failure removes the container here rather than leaving it to the caller, because * the caller's fallback creates a REPLACEMENT container and the suspect one must be gone by then @@ -886,6 +904,24 @@ const legacyWarmShadow = ( Output | LegacyDbConnection > => Effect.gen(function* () { + // An archive that unpacks cleanly but carries no cluster is the ONE corruption the restore + // itself cannot report: `docker cp -` extracts nothing, the entrypoint runs a fresh `initdb` + // into the empty PGDATA, readiness passes, and this function would hand back + // `baselinePresent: true` for a BARE cluster — the caller then skips `legacySetupDatabase` and + // diffs against it, silently producing wrong SQL. So the tar's own headers are scanned for the + // cluster marker BEFORE anything is created (locally, no Docker — see + // {@link legacyPgDataArchiveHasCluster}). A read failure is infra and leaves the tar in place; + // a missing marker implicates its CONTENTS (review: Codex on #6184). + const hasCluster = yield* legacyPgDataArchiveHasCluster(input.fs, tarPath).pipe( + Effect.mapError((cause) => legacyShadowCacheUnavailable(cause.reason)), + ); + if (!hasCluster) { + return yield* Effect.fail( + legacyShadowCacheUnavailable(`snapshot has no ${LEGACY_PGDATA_MARKER_ENTRY} entry`, { + tarSuspect: true, + }), + ); + } const { containerId } = yield* legacyTimeShadowPhase( "baseline-restore", legacyCreateShadowDatabase(spawner, { diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts index 95a5d224bc..7ec071700a 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.unit.test.ts @@ -108,10 +108,6 @@ describe("legacyShadowCacheKey", () => { label: "auto expose new tables", inputs: { ...base, autoExposeNewTables: Option.some(true) }, }, - { - label: "auto expose new tables (explicit false vs unset)", - inputs: { ...base, autoExposeNewTables: Option.some(false) }, - }, { label: "effective webhooks / pg_net", inputs: { ...base, webhooksEnabled: false } }, { label: "roles.sql", inputs: { ...base, rolesSql: "" } }, { @@ -193,6 +189,21 @@ describe("legacyShadowCacheKey", () => { } }); + it("collapses auto_expose_new_tables to the behavior legacyApplyApiPrivileges actually takes", () => { + // `legacyApplyApiPrivileges` (`db-setup.ts`) returns early ONLY for an explicit `true`; unset + // and explicit `false` both exec the revoke SQL, so they bake the identical cluster and must + // share a snapshot instead of forcing a ~90MB re-export. + const base = baseKeyInputs(); + const unset = legacyShadowCacheKey({ ...base, autoExposeNewTables: Option.none() }); + const explicitFalse = legacyShadowCacheKey({ + ...base, + autoExposeNewTables: Option.some(false), + }); + const explicitTrue = legacyShadowCacheKey({ ...base, autoExposeNewTables: Option.some(true) }); + expect(explicitFalse).toBe(unset); + expect(explicitTrue).not.toBe(unset); + }); + it("excludes a disabled service's image tag entirely", () => { const base = baseKeyInputs(); const withRealtimeA: LegacyShadowCacheKeyInputs = { diff --git a/apps/cli/tests/helpers/legacy-mocks.ts b/apps/cli/tests/helpers/legacy-mocks.ts index 34391ac032..c99af51a15 100644 --- a/apps/cli/tests/helpers/legacy-mocks.ts +++ b/apps/cli/tests/helpers/legacy-mocks.ts @@ -1001,8 +1001,62 @@ export function mockLegacyShadowContainerCliSpawner( // `mockLegacyShadowContainerCliSpawner` above deliberately does not model. // --------------------------------------------------------------------------- -/** The bytes the fake `docker cp :PGDATA -` emits — stands in for a real ~90MB PGDATA tar. */ -export const LEGACY_FAKE_PGDATA_TAR = "data/PG_VERSION\n17\n"; +/** + * A real (if tiny) POSIX tar, byte for byte — `legacyPgDataArchiveHasCluster` + * (`db-bootstrap/pgdata-snapshot.ts`) walks the header stream and checksum-validates every block + * before a warm restore, so the fake export has to be a genuine archive rather than a stand-in + * string. Every byte stays ASCII (padding is NUL), which is why the spawner's `TextEncoder` and + * the tests' `readFileString` round-trip it unchanged. + */ +const legacyFakeTarBlock = (fields: ReadonlyArray): string => { + const block = Array.from({ length: 512 }, () => "\0"); + for (const [offset, text] of fields) { + for (let index = 0; index < text.length; index += 1) { + block[offset + index] = text[index] ?? "\0"; + } + } + return block.join(""); +}; + +const legacyFakeTarOctal = (value: number, width: number) => + `${value.toString(8).padStart(width, "0")}\0`; + +/** One ustar member: a checksummed 512-byte header plus its NUL-padded content blocks. */ +const legacyFakeTarEntry = (name: string, content: string, typeFlag: "0" | "5"): string => { + const header = legacyFakeTarBlock([ + [0, name], + [100, legacyFakeTarOctal(typeFlag === "5" ? 0o755 : 0o600, 7)], + [108, legacyFakeTarOctal(0, 7)], + [116, legacyFakeTarOctal(0, 7)], + [124, legacyFakeTarOctal(content.length, 11)], + [136, legacyFakeTarOctal(0, 11)], + // The checksum is computed over the header with this field read as 8 spaces. + [148, " "], + [156, typeFlag], + [257, "ustar\0"], + [263, "00"], + ]); + let checksum = 0; + for (let index = 0; index < header.length; index += 1) checksum += header.charCodeAt(index); + const padding = "\0".repeat((512 - (content.length % 512)) % 512); + return `${header.slice(0, 148)}${legacyFakeTarOctal(checksum, 6)} ${header.slice(156)}${content}${padding}`; +}; + +/** Tar's end-of-archive marker: two all-zero blocks. */ +const LEGACY_FAKE_TAR_END = "\0".repeat(1024); + +/** + * The bytes the fake `docker cp :PGDATA -` emits — stands in for a real ~90MB PGDATA tar, + * with the same top-level `data/` member and `data/PG_VERSION` marker a real export carries. + */ +export const LEGACY_FAKE_PGDATA_TAR = `${legacyFakeTarEntry("data/", "", "5")}${legacyFakeTarEntry("data/PG_VERSION", "17\n", "0")}${LEGACY_FAKE_TAR_END}`; + +/** + * A syntactically VALID tar carrying no members at all — what a replaced or truncated cache + * artifact looks like to `docker cp -`, which extracts it happily and lets the entrypoint `initdb` + * a fresh cluster over the top. The pre-restore marker check is what catches it. + */ +export const LEGACY_FAKE_EMPTY_TAR = LEGACY_FAKE_TAR_END; interface LegacyFakeContainer { readonly labels: Readonly>; From 4d494f4d96c0051846c273eac24209635e488e62 Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 18:36:22 +0200 Subject: [PATCH 77/82] fix(cli): stamp and require a baseline marker in shadow snapshots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The export now writes data/SUPABASE_BASELINE into the stopped shadow's PGDATA right before the outbound docker cp, and warm-hit validation requires it alongside data/PG_VERSION — a valid bare-cluster tar is discarded as suspect instead of restoring into baselinePresent: true and skipping setup. Only snapshotBaseline stamps, strictly after the platform baseline, so a snapshot taken too early can never carry the marker (regression guard). Uppercase name keeps the entry at the front of docker cp's sorted tar so warm validations stay a first-blocks read. Co-Authored-By: Claude Fable 5 --- .../shared/db-bootstrap/pgdata-snapshot.ts | 247 ++++++++++++++---- .../db-bootstrap/pgdata-snapshot.unit.test.ts | 88 +++++-- .../shadow-cache.integration.test.ts | 95 ++++++- .../shared/db-bootstrap/shadow-cache.ts | 48 ++-- apps/cli/tests/helpers/legacy-mocks.ts | 61 ++++- 5 files changed, 430 insertions(+), 109 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts index 8dbcc172b6..ff91168b83 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts @@ -57,6 +57,66 @@ export const LEGACY_PGDATA_PATH = "/var/lib/postgresql/data"; */ export const LEGACY_PGDATA_PARENT_PATH = "/var/lib/postgresql"; +// --------------------------------------------------------------------------- +// What a valid snapshot must contain +// --------------------------------------------------------------------------- + +/** + * PGDATA's own directory name — `docker cp : -` names its members after the source + * BASENAME, so `data/` is the export tar's top-level entry (see {@link LEGACY_PGDATA_PARENT_PATH}). + */ +const LEGACY_PGDATA_DIR_NAME = LEGACY_PGDATA_PATH.slice(LEGACY_PGDATA_PARENT_PATH.length + 1); + +/** + * The entry whose presence proves an archive really carries an exported cluster: every Postgres + * data directory has a `PG_VERSION` file at its root, and `initdb` writes it first. An archive that + * unpacks cleanly but lacks it is the corruption a restore cannot otherwise notice. + * + * On its own it proves only "*a* PostgreSQL cluster", which is strictly weaker than what a cache + * key promises — hence {@link LEGACY_PGDATA_BASELINE_MARKER_ENTRY}. + */ +export const LEGACY_PGDATA_CLUSTER_ENTRY = `${LEGACY_PGDATA_DIR_NAME}/PG_VERSION`; + +/** + * A file this module writes into PGDATA's ROOT ({@link legacyStampPgDataBaselineMarker}) as the + * last step before the export copies the directory out. Postgres ignores unknown regular files at + * the data directory's root (`pg_upgrade` and friends routinely leave some there), and a restored + * container simply carries it along, so the cost of the stamp is one 512-byte tar member. + * + * SCREAMING_SNAKE on purpose, and not by taste: `docker cp` tars a directory through Go's + * `filepath.Walk`, which visits each level in sorted order, so an uppercase root file lands + * immediately next to `PG_VERSION` — near the front of a ~90MB archive rather than behind every + * `base/` page. That is a PERFORMANCE hint for {@link legacyPgDataArchiveMissingEntries} only: + * the scan is correct at any position, and settles late (not wrongly) if a Docker release ever + * reorders its walk. + */ +export const LEGACY_PGDATA_BASELINE_MARKER_NAME = "SUPABASE_BASELINE"; + +/** + * The marker's tar entry — what {@link legacyPgDataArchiveMissingEntries} looks for, and the reason + * a cached snapshot means "the Supabase platform baseline this key promises" rather than merely "a + * PostgreSQL cluster". + * + * Its whole value is WHEN it is written: {@link legacyStampPgDataBaselineMarker} is called from the + * export step alone, after the caller's own baseline has completed and immediately before the + * copy-out. So an archive produced before the baseline ran — a wiring regression that moves the + * snapshot earlier, or a hand-placed bare PGDATA tar dropped into the cache directory — cannot + * carry it, and is rejected before anything is restored. + */ +export const LEGACY_PGDATA_BASELINE_MARKER_ENTRY = `${LEGACY_PGDATA_DIR_NAME}/${LEGACY_PGDATA_BASELINE_MARKER_NAME}`; + +/** + * Only PRESENCE is the contract — nothing reads these bytes back, so the content can grow into + * provenance metadata later without breaking any reader. + */ +const LEGACY_PGDATA_BASELINE_MARKER_CONTENT = "1\n"; + +/** Every entry {@link legacyPgDataArchiveMissingEntries} requires of a restorable snapshot. */ +export const LEGACY_PGDATA_REQUIRED_ENTRIES: ReadonlyArray = [ + LEGACY_PGDATA_CLUSTER_ENTRY, + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, +]; + /** * Internal-only "the snapshot could not be produced" signal — deliberately NOT a * `Data.TaggedError`: every caller of {@link legacyExportPgDataTar} decides for itself how to @@ -71,6 +131,74 @@ const legacyPgDataSnapshotUnavailable = (reason: string): LegacyPgDataSnapshotUn reason, }); +/** + * The one-member tar {@link legacyStampPgDataBaselineMarker} pushes into the container: + * `SUPABASE_BASELINE` relative to the `docker cp` destination, which is PGDATA itself. + * + * Exported for the unit test that round-trips it through this module's own scanner — the stamp and + * the check have to agree on the entry name, and nothing else proves that they do. + */ +export const legacyPgDataBaselineMarkerTar = (): Effect.Effect< + Uint8Array, + LegacyPgDataSnapshotUnavailable +> => + Effect.tryPromise({ + try: () => + new Bun.Archive({ + [LEGACY_PGDATA_BASELINE_MARKER_NAME]: LEGACY_PGDATA_BASELINE_MARKER_CONTENT, + }).bytes(), + catch: (cause) => + legacyPgDataSnapshotUnavailable( + `failed to build the baseline marker archive: ${cause instanceof Error ? cause.message : String(cause)}`, + ), + }); + +/** + * Writes {@link LEGACY_PGDATA_BASELINE_MARKER_ENTRY} into the container's PGDATA, so the export + * that follows carries it and {@link legacyPgDataArchiveMissingEntries} can tell a snapshot of a + * COMPLETED baseline apart from any other cluster. + * + * Delivered as a stdin tar through `docker cp -`, the same form the restore side uses + * (`legacyExtractPreStartArchiveIntoContainer`, `container-lifecycle.ts`) and for the same reason: + * it needs no daemon-visible host path, so it works against local, remote-context, and confined + * Docker clients alike. `docker cp` into a STOPPED container is fully supported — which is exactly + * the state this module's coherence contract already requires of the export. + * + * The caller owns the ORDERING that gives the marker its meaning: this must be the last mutation + * before {@link legacyExportPgDataTar}, and must run only once whatever the snapshot is supposed to + * capture is genuinely in place. + */ +export const legacyStampPgDataBaselineMarker = ( + spawner: Spawner, + containerId: string, +): Effect.Effect => + Effect.gen(function* () { + const tar = yield* legacyPgDataBaselineMarkerTar(); + const failure = (detail: string) => + legacyPgDataSnapshotUnavailable( + `failed to stamp ${LEGACY_PGDATA_BASELINE_MARKER_ENTRY}: ${detail}`, + ); + yield* Effect.scoped( + Effect.gen(function* () { + const child = yield* spawnContainerCli( + spawner, + ["cp", "-", `${containerId}:${LEGACY_PGDATA_PATH}`], + { stdin: Stream.make(tar), stdout: "ignore", stderr: "pipe" }, + ).pipe(Effect.mapError((cause) => failure(legacyDescribeContainerCliFailure(cause)))); + const [exitCode, stderr] = yield* Effect.all( + [child.exitCode.pipe(Effect.map(Number)), legacyCollectText(child.stderr)], + { concurrency: "unbounded" }, + ).pipe(Effect.mapError((cause) => failure(legacyDescribeContainerCliFailure(cause)))); + if (exitCode !== 0) { + const message = stderr.trim(); + return yield* Effect.fail( + failure(`docker cp exited ${exitCode}${message.length > 0 ? `: ${message}` : ""}`), + ); + } + }), + ); + }); + /** * Streams `docker cp :${LEGACY_PGDATA_PATH} -`'s tar straight to a temp file next to * `tarPath` and `rename`s it into place. The stream never lands in memory: the child's stdout is @@ -154,20 +282,6 @@ export const legacyExportPgDataTar = ( // Archive validation // --------------------------------------------------------------------------- -/** - * PGDATA's own directory name — `docker cp : -` names its members after the source - * BASENAME, so `data/` is the export tar's top-level entry (see {@link LEGACY_PGDATA_PARENT_PATH}). - */ -const LEGACY_PGDATA_DIR_NAME = LEGACY_PGDATA_PATH.slice(LEGACY_PGDATA_PARENT_PATH.length + 1); - -/** - * The one entry whose presence proves an archive really carries an exported cluster: every - * Postgres data directory has a `PG_VERSION` file at its root, and `initdb` writes it first. An - * archive that unpacks cleanly but lacks it is the corruption a restore cannot otherwise notice — - * see {@link legacyPgDataArchiveHasCluster}. - */ -export const LEGACY_PGDATA_MARKER_ENTRY = `${LEGACY_PGDATA_DIR_NAME}/PG_VERSION`; - /** POSIX tar's fixed block size: headers, file content, and the end marker are all multiples of it. */ const LEGACY_TAR_BLOCK_SIZE = 512; @@ -176,7 +290,7 @@ const LEGACY_TAR_NO_BYTES = new Uint8Array(0); const legacyTarDecoder = new TextDecoder(); /** - * {@link legacyScanTarChunkForEntry}'s carry-over state — everything needed to resume a header + * {@link legacyScanTarChunkForEntries}'s carry-over state — everything needed to resume a header * walk at an arbitrary chunk boundary, and nothing else. `carry` holds the bytes of a header block * a chunk ended in the middle of (always `< 512`); `skip` counts the file-content bytes still to be * STEPPED OVER without buffering, which is what keeps a ~90MB archive off the heap. @@ -186,24 +300,32 @@ export interface LegacyTarScanState { readonly skip: number; /** Consecutive all-zero blocks seen; two in a row is tar's end-of-archive marker. */ readonly zeroBlocks: number; - readonly found: boolean; + /** + * The required entries not seen yet. Empty means every one of them was found; whatever is left + * once the scan settles is what the archive is missing, which is what the caller reports. + */ + readonly missing: ReadonlySet; readonly ended: boolean; /** A block that is neither zero nor a checksum-valid header: not a tar (or a truncated one). */ readonly malformed: boolean; } -export const legacyInitialTarScanState: LegacyTarScanState = { +/** A fresh walk looking for `required` — every entry of which must appear for the scan to pass. */ +export const legacyInitialTarScanState = (required: Iterable): LegacyTarScanState => ({ carry: LEGACY_TAR_NO_BYTES, skip: 0, zeroBlocks: 0, - found: false, + missing: new Set(required), ended: false, malformed: false, -}; +}); + +/** Whether every required entry has been seen. */ +export const legacyTarScanFound = (state: LegacyTarScanState): boolean => state.missing.size === 0; /** Whether the scan has reached a verdict — nothing later in the archive can change it. */ export const legacyTarScanSettled = (state: LegacyTarScanState): boolean => - state.found || state.ended || state.malformed; + legacyTarScanFound(state) || state.ended || state.malformed; /** A NUL-terminated text field of a tar header block. */ const legacyTarTextField = (block: Uint8Array, offset: number, length: number): string => { @@ -265,27 +387,31 @@ const legacyTarEntryName = (block: Uint8Array): string => { const legacyTarBlockIsZero = (block: Uint8Array): boolean => block.every((byte) => byte === 0); /** - * Folds one stream chunk into a tar HEADER walk looking for `entryName`. Pure and - * chunk-boundary-agnostic: file content is stepped over by byte count rather than buffered, so the - * whole scan costs one partial header block of memory no matter how large the archive is. Stops - * (and stays stopped) at the first of: the entry found, the end-of-archive marker, or a block that - * is not a valid header. + * Folds one stream chunk into a tar HEADER walk looking for every entry still in `state.missing`. + * Pure and chunk-boundary-agnostic: file content is stepped over by byte count rather than + * buffered, so the whole scan costs one partial header block of memory no matter how large the + * archive is. Stops (and stays stopped) at the first of: the last required entry found, the + * end-of-archive marker, or a block that is not a valid header. */ -export const legacyScanTarChunkForEntry = ( +export const legacyScanTarChunkForEntries = ( state: LegacyTarScanState, chunk: Uint8Array, - entryName: string, ): LegacyTarScanState => { if (legacyTarScanSettled(state)) return state; + let carry = state.carry; + let skip = state.skip; + let zeroBlocks = state.zeroBlocks; + let missing = state.missing; + // A verdict keeps `missing` (it is the report) and drops the walk's resumption state. const settle = ( - verdict: Pick, + verdict: Pick, ): LegacyTarScanState => ({ - ...legacyInitialTarScanState, + carry: LEGACY_TAR_NO_BYTES, + skip: 0, + zeroBlocks: 0, + missing, ...verdict, }); - let carry = state.carry; - let skip = state.skip; - let zeroBlocks = state.zeroBlocks; // Content bytes carried over from the previous chunk come first — they are not headers. let offset = Math.min(skip, chunk.length); skip -= offset; @@ -314,19 +440,23 @@ export const legacyScanTarChunkForEntry = ( if (legacyTarBlockIsZero(block)) { zeroBlocks += 1; - if (zeroBlocks >= 2) return settle({ found: false, ended: true, malformed: false }); + if (zeroBlocks >= 2) return settle({ ended: true, malformed: false }); continue; } zeroBlocks = 0; if (!legacyTarChecksumValid(block)) { - return settle({ found: false, ended: false, malformed: true }); + return settle({ ended: false, malformed: true }); } - if (legacyTarEntryName(block) === entryName) { - return settle({ found: true, ended: false, malformed: false }); + const name = legacyTarEntryName(block); + if (missing.has(name)) { + const remaining = new Set(missing); + remaining.delete(name); + missing = remaining; + if (missing.size === 0) return settle({ ended: false, malformed: false }); } const size = legacyTarNumericField(block, 124, 12); if (size === undefined || size < 0) { - return settle({ found: false, ended: false, malformed: true }); + return settle({ ended: false, malformed: true }); } // Content is padded up to the next block boundary; directories and links carry size 0. const content = Math.ceil(size / LEGACY_TAR_BLOCK_SIZE) * LEGACY_TAR_BLOCK_SIZE; @@ -334,38 +464,49 @@ export const legacyScanTarChunkForEntry = ( offset += stepped; skip = content - stepped; } - return { carry, skip, zeroBlocks, found: false, ended: false, malformed: false }; + return { carry, skip, zeroBlocks, missing, ended: false, malformed: false }; }; /** - * Whether `tarPath` really is a {@link legacyExportPgDataTar} archive — i.e. whether its member - * list contains {@link LEGACY_PGDATA_MARKER_ENTRY}. + * Which of {@link LEGACY_PGDATA_REQUIRED_ENTRIES} `tarPath`'s member list does NOT contain — empty + * for a snapshot that is safe to restore. * - * This exists because an archive that is syntactically fine but carries no cluster (an EMPTY tar - * qualifies) restores SILENTLY: `docker cp -` extracts nothing, the Postgres entrypoint finds an - * empty PGDATA and runs a fresh `initdb`, readiness passes, and the caller is handed a bare - * cluster it believes carries the platform baseline. Validating the header stream up front is the - * only place that difference is observable, so callers must check BEFORE restoring. + * Both entries are load-bearing, and for different failures. Without `PG_VERSION`, an archive that + * is syntactically fine but carries no cluster (an EMPTY tar qualifies) restores SILENTLY: + * `docker cp -` extracts nothing, the Postgres entrypoint finds an empty PGDATA and runs a fresh + * `initdb`, readiness passes, and the caller is handed a bare cluster it believes carries the + * platform baseline. Without the baseline marker, that same silent-success shape survives one level + * up: a REAL but bare PGDATA tar (dropped into the cache directory by hand, or produced by a future + * regression that exports before the baseline runs) restores, starts, and answers — and only the + * resulting diff would ever show it. Validating the header stream up front is the only place either + * difference is observable, so callers must check BEFORE restoring. * - * Reads the file locally — no Docker, no extraction — and stops at the marker, so a warm hit - * normally touches only the archive's first blocks. Only a genuine read failure fails; a valid tar - * without the marker simply resolves `false`. + * Reads the file locally — no Docker, no extraction — and stops as soon as both entries have been + * seen; see {@link LEGACY_PGDATA_BASELINE_MARKER_NAME} for why that is normally within the + * archive's first blocks. Even a full walk only parses HEADERS (content is stepped over by byte + * count), so the cost is one sequential read with O(1) memory. Only a genuine read failure fails; a + * valid tar missing an entry simply reports it. */ -export const legacyPgDataArchiveHasCluster = ( +export const legacyPgDataArchiveMissingEntries = ( fs: FileSystem.FileSystem, tarPath: string, -): Effect.Effect => +): Effect.Effect, LegacyPgDataSnapshotUnavailable> => fs.stream(tarPath).pipe( Stream.mapAccum( - () => legacyInitialTarScanState, + () => legacyInitialTarScanState(LEGACY_PGDATA_REQUIRED_ENTRIES), (state: LegacyTarScanState, chunk: Uint8Array) => { - const next = legacyScanTarChunkForEntry(state, chunk, LEGACY_PGDATA_MARKER_ENTRY); + const next = legacyScanTarChunkForEntries(state, chunk); return [next, [next]] as const; }, ), Stream.takeUntil(legacyTarScanSettled), Stream.runLast, - Effect.map((last) => Option.isSome(last) && last.value.found), + // An empty file yields no chunks at all, so `None` means nothing was found: everything missing. + Effect.map((last) => + Option.isSome(last) + ? LEGACY_PGDATA_REQUIRED_ENTRIES.filter((entry) => last.value.missing.has(entry)) + : LEGACY_PGDATA_REQUIRED_ENTRIES, + ), Effect.mapError((cause) => legacyPgDataSnapshotUnavailable(`failed to read ${tarPath}: ${cause.message}`), ), diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts index 74128bc09b..bed6e1c064 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts @@ -1,16 +1,22 @@ /** - * The pure tar-header walk behind {@link legacyPgDataArchiveHasCluster}'s pre-restore check. Unit - * tests rather than integration ones because the interesting cases are all in the format handling: - * chunk boundaries landing mid-header, content that must be stepped over rather than parsed, and - * bytes that are not a tar at all. + * The pure tar-header walk behind {@link legacyPgDataArchiveMissingEntries}'s pre-restore check. + * Unit tests rather than integration ones because the interesting cases are all in the format + * handling: chunk boundaries landing mid-header, content that must be stepped over rather than + * parsed, and bytes that are not a tar at all. */ import { describe, expect, it } from "@effect/vitest"; +import { Effect } from "effect"; import { - LEGACY_PGDATA_MARKER_ENTRY, + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + LEGACY_PGDATA_BASELINE_MARKER_NAME, + LEGACY_PGDATA_CLUSTER_ENTRY, + LEGACY_PGDATA_REQUIRED_ENTRIES, legacyInitialTarScanState, - legacyScanTarChunkForEntry, + legacyPgDataBaselineMarkerTar, + legacyScanTarChunkForEntries, + legacyTarScanFound, legacyTarScanSettled, type LegacyTarScanState, } from "./pgdata-snapshot.ts"; @@ -66,26 +72,33 @@ const concat = (...parts: ReadonlyArray): Uint8Array => { }; /** Feeds `tar` through the scanner in fixed-size chunks, stopping as the real stream would. */ -const scan = (tar: Uint8Array, chunkSize: number, entry = LEGACY_PGDATA_MARKER_ENTRY) => { - let state: LegacyTarScanState = legacyInitialTarScanState; +const scan = ( + tar: Uint8Array, + chunkSize: number, + required: ReadonlyArray = LEGACY_PGDATA_REQUIRED_ENTRIES, +) => { + let state: LegacyTarScanState = legacyInitialTarScanState(required); for (let offset = 0; offset < tar.length; offset += chunkSize) { - state = legacyScanTarChunkForEntry(state, tar.subarray(offset, offset + chunkSize), entry); + state = legacyScanTarChunkForEntries(state, tar.subarray(offset, offset + chunkSize)); if (legacyTarScanSettled(state)) break; } - return state; + return { ...state, found: legacyTarScanFound(state) }; }; -describe("legacyScanTarChunkForEntry", () => { +const MARKER_ENTRY = tarEntry(LEGACY_PGDATA_BASELINE_MARKER_ENTRY, "1\n"); + +describe("legacyScanTarChunkForEntries", () => { const pgdataTar = concat( tarEntry("data/", "", "5"), tarEntry("data/postgresql.conf", "listen_addresses = '*'\n"), - tarEntry(LEGACY_PGDATA_MARKER_ENTRY, "17\n"), + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + MARKER_ENTRY, TAR_END, ); - it("finds the cluster marker regardless of where the chunk boundaries fall", () => { - // 7 and 513 both split header blocks; the marker is the third member, behind a directory and - // a file whose content must be stepped over rather than mistaken for a header. + it("finds every required entry regardless of where the chunk boundaries fall", () => { + // 7 and 513 both split header blocks; the required entries are the last two members, behind a + // directory and a file whose content must be stepped over rather than mistaken for a header. for (const chunkSize of [1, 7, 100, 512, 513, 4096, pgdataTar.length]) { const state = scan(pgdataTar, chunkSize); expect(state.found, `chunk size ${chunkSize}`).toBe(true); @@ -93,17 +106,36 @@ describe("legacyScanTarChunkForEntry", () => { } }); - it("stops at the marker without walking the rest of the archive", () => { - // Everything after the marker is garbage: reaching it would settle `malformed` instead. + it("stops at the last required entry without walking the rest of the archive", () => { + // Everything after them is garbage: reaching it would settle `malformed` instead. const trailing = concat( - tarEntry(LEGACY_PGDATA_MARKER_ENTRY, "17\n"), + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + MARKER_ENTRY, encoder.encode("x".repeat(2048)), ); expect(scan(trailing, 512).found).toBe(true); }); + it("keeps looking when only some of the required entries have been seen", () => { + // A REAL cluster with no baseline marker — the shape a hand-placed PGDATA tar, or a snapshot + // exported before the platform baseline ran, would have. It must not pass. + const bare = concat( + tarEntry("data/", "", "5"), + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + TAR_END, + ); + const state = scan(bare, 512); + expect(state.found).toBe(false); + expect(state.ended).toBe(true); + expect([...state.missing]).toEqual([LEGACY_PGDATA_BASELINE_MARKER_ENTRY]); + // ...and the same bytes DO pass once the marker is there, so nothing else about them is wrong. + const stamped = concat(bare.subarray(0, bare.length - TAR_END.length), MARKER_ENTRY, TAR_END); + expect(scan(stamped, 512).found).toBe(true); + }); + it("reports a valid but cluster-less archive as not found", () => { expect(scan(TAR_END, 512)).toMatchObject({ found: false, ended: true, malformed: false }); + expect([...scan(TAR_END, 512).missing]).toEqual([...LEGACY_PGDATA_REQUIRED_ENTRIES]); const otherEntries = concat(tarEntry("data/base/1/2345", "rows"), TAR_END); expect(scan(otherEntries, 512)).toMatchObject({ found: false, ended: true }); }); @@ -111,14 +143,14 @@ describe("legacyScanTarChunkForEntry", () => { it("never mistakes file content for a header", () => { // A member whose CONTENT is itself a valid `data/PG_VERSION` header — only the real member // list counts, so this archive must come back cluster-less. - const decoy = new TextDecoder().decode(tarEntry(LEGACY_PGDATA_MARKER_ENTRY, "17\n")); + const decoy = new TextDecoder().decode(tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n")); expect(scan(concat(tarEntry("data/decoy", decoy), TAR_END), 512).found).toBe(false); }); it("settles as malformed on bytes that are not a tar", () => { expect(scan(encoder.encode("x".repeat(4096)), 512).malformed).toBe(true); // A header whose checksum does not add up (a flipped byte in an otherwise real archive). - const corrupted = concat(tarEntry("data/PG_VERSION", "17\n"), TAR_END); + const corrupted = concat(tarEntry("data/PG_VERSION", "17\n"), MARKER_ENTRY, TAR_END); corrupted[5] = 0x41; expect(scan(corrupted, 512).malformed).toBe(true); }); @@ -129,3 +161,19 @@ describe("legacyScanTarChunkForEntry", () => { expect(scan(truncated, 512)).toMatchObject({ found: false, ended: false, malformed: false }); }); }); + +describe("legacyPgDataBaselineMarkerTar", () => { + it("stamps the very entry the pre-restore scan requires", () => + Effect.runPromise( + Effect.gen(function* () { + // The stamp and the check are two halves of one contract, and only a round trip proves + // they agree: `docker cp - :` unpacks this archive's members RELATIVE to + // PGDATA, so its bare `SUPABASE_BASELINE` member is what a later export tars up as + // `data/SUPABASE_BASELINE`. + const bytes = yield* legacyPgDataBaselineMarkerTar(); + expect(scan(bytes, 512, [LEGACY_PGDATA_BASELINE_MARKER_NAME]).found).toBe(true); + // Nothing else rides along — one member, so the stamp costs a couple of blocks. + expect(scan(bytes, 512, [LEGACY_PGDATA_BASELINE_MARKER_ENTRY]).found).toBe(false); + }), + )); +}); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index f9800156bf..158707ce1b 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -21,6 +21,7 @@ import { Effect, Exit, FileSystem, Layer, Option, Path, Schema } from "effect"; import { LEGACY_FAKE_EMPTY_TAR, LEGACY_FAKE_PGDATA_TAR, + LEGACY_FAKE_UNSTAMPED_PGDATA_TAR, legacyWithEnv, mockLegacyDockerDaemonCliSpawner, useLegacyTempWorkdir, @@ -29,7 +30,12 @@ import { mockOutput } from "../../../../tests/helpers/mocks.ts"; import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import { LegacyDbConnectError } from "../legacy-db-connection.errors.ts"; import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; -import { LEGACY_PGDATA_PARENT_PATH, LEGACY_PGDATA_PATH } from "./pgdata-snapshot.ts"; +import { + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + LEGACY_PGDATA_BASELINE_MARKER_NAME, + LEGACY_PGDATA_PARENT_PATH, + LEGACY_PGDATA_PATH, +} from "./pgdata-snapshot.ts"; import { LEGACY_SHADOW_BASELINE_KEEP, LEGACY_SHADOW_CACHE_ENV, @@ -362,16 +368,29 @@ describe("legacyAcquireShadowDatabase", () => { yield* handle.snapshotBaseline; // Ordering IS the contract here: stop before the copy (a live PGDATA is not coherent to - // copy), start plus a readiness probe after it (the caller is about to reconnect). + // copy), the baseline marker stamped in between (it must be the LAST thing written to + // PGDATA, so nothing after the platform baseline can be missing from what it vouches + // for), start plus a readiness probe after it (the caller is about to reconnect). expect(docker.steps()).toEqual([ "create", "cp-secret", "start", "stop", + "cp-stamp", "cp-out", "start", "inspect", ]); + expect(docker.stepCalls("cp-stamp")[0]).toEqual([ + "cp", + "-", + `${handle.containerId}:${LEGACY_PGDATA_PATH}`, + ]); + // The stamp really carries the marker file, delivered as a tar so `docker cp` unpacks it + // relative to PGDATA rather than rewriting the directory's ownership. + expect(docker.containers.get(handle.containerId)?.stamp).toContain( + LEGACY_PGDATA_BASELINE_MARKER_NAME, + ); expect(docker.stepCalls("cp-out")[0]).toEqual([ "cp", `${handle.containerId}:${LEGACY_PGDATA_PATH}`, @@ -384,9 +403,11 @@ describe("legacyAcquireShadowDatabase", () => { const tars = yield* soleTarName(fs, path); expect(tars).toHaveLength(1); expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); - expect(yield* fs.readFileString(path.join(shadowCacheDir(path), tars[0] ?? ""))).toBe( - LEGACY_FAKE_PGDATA_TAR, - ); + const published = yield* fs.readFileString(path.join(shadowCacheDir(path), tars[0] ?? "")); + expect(published).toBe(LEGACY_FAKE_PGDATA_TAR); + // The stamp made it all the way into the artifact — this is the entry the next run's + // pre-restore scan requires, so a cold export that skipped it would never warm anything. + expect(published).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); const leftovers = yield* fs.readDirectory(shadowCacheDir(path)); expect(leftovers.filter((entry) => entry.includes("partial"))).toEqual([]); @@ -789,6 +810,70 @@ describe("legacyAcquireShadowDatabase", () => { ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live("a published tar carrying a bare cluster is discarded instead of restored", () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const input = shadowInput(fs, path); + yield* coldRun(docker, input); + const [tarName = ""] = yield* soleTarName(fs, path); + const tarPath = path.join(shadowCacheDir(path), tarName); + + // A REAL, perfectly restorable PGDATA — but one that never ran the platform baseline. This + // is the failure `data/PG_VERSION` alone cannot see: `docker cp -` extracts a genuine + // cluster, the entrypoint SKIPS `initdb`, readiness passes, and the caller would be told + // the baseline is present while diffing against a bare database. Only the missing marker + // separates it from a usable snapshot. + yield* fs.writeFileString(tarPath, LEGACY_FAKE_UNSTAMPED_PGDATA_TAR); + const stepsBefore = docker.steps().length; + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, input); + + expect(out.stderrText).toContain("cached shadow baseline unusable"); + expect(out.stderrText).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); + expect(fallback.baselinePresent).toBe(false); + // Caught before any container was created, so nothing was ever restored. + expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); + // The contents ARE the problem, so the tar goes — and the cold fallback republishes a + // marked one within the same run. + expect(yield* soleTarName(fs, path)).toEqual([]); + yield* fallback.snapshotBaseline; + expect(yield* fs.readFileString(tarPath)).toBe(LEGACY_FAKE_PGDATA_TAR); + yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + + it.live("a failed baseline stamp leaves the run uncached rather than publishing a tar", () => { + const docker = mockLegacyDockerDaemonCliSpawner({ failStamp: true }); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const handle = yield* legacyAcquireShadowDatabase(docker.spawner, shadowInput(fs, path)); + // Same fail-open contract as every other export failure: the run itself never fails... + yield* handle.snapshotBaseline; + + expect(out.stderrText).toContain("Warning: shadow baseline not cached"); + expect(out.stderrText).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); + // ...the shadow is back up for the caller to reconnect to... + expect(docker.containers.get(handle.containerId)?.running).toBe(true); + // ...and nothing is published, because an UNMARKED tar would only be thrown away on the + // next run anyway. The export never even runs. + expect(docker.stepCalls("cp-out")).toHaveLength(0); + expect(yield* soleTarName(fs, path)).toEqual([]); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }); + it.live("a restored shadow that never becomes ready is removed before the cold retry", () => { const docker = mockLegacyDockerDaemonCliSpawner(); const out = mockOutput(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 2d2d8df5ce..0640dc5130 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -59,10 +59,10 @@ import { } from "../legacy-vault.ts"; import { legacyWaitForShadowReady } from "./health-check.ts"; import { - LEGACY_PGDATA_MARKER_ENTRY, legacyExportPgDataTar, - legacyPgDataArchiveHasCluster, + legacyPgDataArchiveMissingEntries, legacyPgDataRestoreArchive, + legacyStampPgDataBaselineMarker, } from "./pgdata-snapshot.ts"; import type { LegacyPgDataSnapshotUnavailable } from "./pgdata-snapshot.ts"; import { legacyResolvePinnedImage } from "./pinned-image.ts"; @@ -92,8 +92,9 @@ interface LegacyShadowCacheUnavailable { readonly reason: string; /** * `true` only when the failure implicates the TAR'S CONTENTS — today, exactly two producers, - * both in {@link legacyWarmShadow}: an archive whose header stream carries no PGDATA cluster - * (checked before any container is created), and a restored cluster that started but never + * both in {@link legacyWarmShadow}: an archive whose header stream is missing a required entry — + * the PGDATA cluster file or the baseline marker — (checked before any container is created), + * and a restored cluster that started but never * accepted connections (the readiness wait). Everything else (a `docker create`/`cp`/`start` * failure — daemon outage, * port collision, or even a corrupt archive's failed extraction) leaves the tar in place: an @@ -766,6 +767,18 @@ const legacyExportShadowBaseline = ( const exported = yield* Effect.result( Effect.gen(function* () { yield* legacyShadowContainerVerb(spawner, "stop", containerId); + // The stamp is what makes the published tar mean "the baseline this key promises" rather + // than "some PostgreSQL cluster", and its ONLY guarantee is this position in the + // sequence: this whole step runs from `snapshotBaseline`, which `legacySetupShadowDatabase` + // invokes strictly after `legacySetupDatabase` returns (`shadow-database.ts`), and the + // stamp is the last mutation before the copy-out. A future regression that snapshots + // EARLIER therefore cannot produce a marked tar — it just stays uncached, instead of + // silently publishing a bare cluster under a baseline key. + yield* legacyStampPgDataBaselineMarker(spawner, containerId).pipe( + Effect.mapError((cause: LegacyPgDataSnapshotUnavailable) => + legacyShadowCacheUnavailable(cause.reason), + ), + ); yield* legacyWriteShadowBaselineTar(spawner, input, tarPath, containerId); }), ); @@ -875,8 +888,8 @@ const legacyColdCachedShadow = ( ); /** - * The warm path proper: verify the snapshot tar really contains a cluster, create the shadow with - * it unpacked into it before it starts ({@link LegacyCreateShadowDatabaseInput.restoreArchive}), + * The warm path proper: verify the snapshot tar really carries a baselined cluster, create the + * shadow with it unpacked into it before it starts ({@link LegacyCreateShadowDatabaseInput.restoreArchive}), * then wait for the restored Postgres. Every failure resolves to * {@link LegacyShadowCacheUnavailable}, which the caller turns into the escape hatch. * @@ -904,20 +917,21 @@ const legacyWarmShadow = ( Output | LegacyDbConnection > => Effect.gen(function* () { - // An archive that unpacks cleanly but carries no cluster is the ONE corruption the restore - // itself cannot report: `docker cp -` extracts nothing, the entrypoint runs a fresh `initdb` - // into the empty PGDATA, readiness passes, and this function would hand back - // `baselinePresent: true` for a BARE cluster — the caller then skips `legacySetupDatabase` and - // diffs against it, silently producing wrong SQL. So the tar's own headers are scanned for the - // cluster marker BEFORE anything is created (locally, no Docker — see - // {@link legacyPgDataArchiveHasCluster}). A read failure is infra and leaves the tar in place; - // a missing marker implicates its CONTENTS (review: Codex on #6184). - const hasCluster = yield* legacyPgDataArchiveHasCluster(input.fs, tarPath).pipe( + // An archive that unpacks cleanly but carries the wrong thing is the ONE corruption the restore + // itself cannot report: `docker cp -` extracts whatever it is given, the entrypoint skips + // `initdb` (or runs one over an empty PGDATA), readiness passes, and this function would hand + // back `baselinePresent: true` for a cluster that never saw `legacySetupDatabase` — the caller + // then skips it too and diffs against a BARE database, silently producing wrong SQL. So the + // tar's own headers are scanned BEFORE anything is created (locally, no Docker) for both the + // cluster file AND the baseline marker this module stamps immediately before every export — + // see {@link legacyPgDataArchiveMissingEntries}. A read failure is infra and leaves the tar in + // place; a missing entry implicates its CONTENTS (review: Codex on #6184). + const missing = yield* legacyPgDataArchiveMissingEntries(input.fs, tarPath).pipe( Effect.mapError((cause) => legacyShadowCacheUnavailable(cause.reason)), ); - if (!hasCluster) { + if (missing.length > 0) { return yield* Effect.fail( - legacyShadowCacheUnavailable(`snapshot has no ${LEGACY_PGDATA_MARKER_ENTRY} entry`, { + legacyShadowCacheUnavailable(`snapshot has no ${missing.join(" or ")} entry`, { tarSuspect: true, }), ); diff --git a/apps/cli/tests/helpers/legacy-mocks.ts b/apps/cli/tests/helpers/legacy-mocks.ts index c99af51a15..dc2c55cd4d 100644 --- a/apps/cli/tests/helpers/legacy-mocks.ts +++ b/apps/cli/tests/helpers/legacy-mocks.ts @@ -36,6 +36,7 @@ import { LegacyLoginVerificationError, } from "../../src/legacy/commands/login/login.errors.ts"; import { LegacyCliConfig } from "../../src/legacy/config/legacy-cli-config.service.ts"; +import { LEGACY_PGDATA_PATH } from "../../src/legacy/shared/db-bootstrap/pgdata-snapshot.ts"; import { legacyProjectRefLayer } from "../../src/legacy/config/legacy-project-ref.layer.ts"; import { LegacyLinkedProjectCache } from "../../src/legacy/telemetry/legacy-linked-project-cache.service.ts"; import { LegacyTelemetryState } from "../../src/legacy/telemetry/legacy-telemetry-state.service.ts"; @@ -996,7 +997,8 @@ export function mockLegacyShadowContainerCliSpawner( // --------------------------------------------------------------------------- // A minimal, stateful Docker model — the shadow BASELINE CACHE's round trip -// (`docker stop` -> `docker cp :PGDATA -` -> `docker start`, and the warm +// (`docker stop` -> `docker cp - :PGDATA` (the baseline stamp) -> +// `docker cp :PGDATA -` -> `docker start`, and the warm // `docker cp - :` restore) really moves bytes, which // `mockLegacyShadowContainerCliSpawner` above deliberately does not model. // --------------------------------------------------------------------------- @@ -1046,10 +1048,19 @@ const legacyFakeTarEntry = (name: string, content: string, typeFlag: "0" | "5"): const LEGACY_FAKE_TAR_END = "\0".repeat(1024); /** - * The bytes the fake `docker cp :PGDATA -` emits — stands in for a real ~90MB PGDATA tar, - * with the same top-level `data/` member and `data/PG_VERSION` marker a real export carries. + * What the fake `docker cp :PGDATA -` emits for a container the export path has NOT stamped — + * a real cluster (`data/` + `data/PG_VERSION`) and nothing more. Stands in both for a hand-placed + * bare PGDATA archive and for a snapshot taken before the platform baseline ran: it restores, + * starts, and answers, so only the missing marker tells it apart from a usable baseline. */ -export const LEGACY_FAKE_PGDATA_TAR = `${legacyFakeTarEntry("data/", "", "5")}${legacyFakeTarEntry("data/PG_VERSION", "17\n", "0")}${LEGACY_FAKE_TAR_END}`; +export const LEGACY_FAKE_UNSTAMPED_PGDATA_TAR = `${legacyFakeTarEntry("data/", "", "5")}${legacyFakeTarEntry("data/PG_VERSION", "17\n", "0")}${LEGACY_FAKE_TAR_END}`; + +/** + * The bytes the fake `docker cp :PGDATA -` emits once the export path has stamped the + * container — stands in for a real ~90MB PGDATA tar, with the same top-level `data/` member, + * `data/PG_VERSION` cluster file, and `data/SUPABASE_BASELINE` marker a real export carries. + */ +export const LEGACY_FAKE_PGDATA_TAR = `${legacyFakeTarEntry("data/", "", "5")}${legacyFakeTarEntry("data/PG_VERSION", "17\n", "0")}${legacyFakeTarEntry("data/SUPABASE_BASELINE", "1\n", "0")}${LEGACY_FAKE_TAR_END}`; /** * A syntactically VALID tar carrying no members at all — what a replaced or truncated cache @@ -1066,6 +1077,12 @@ interface LegacyFakeContainer { everStarted?: boolean; /** What a previous `docker cp - :` unpacked into this container, if anything. */ restored: string | undefined; + /** + * What a `docker cp - :` delivered — the export path's baseline stamp. Its presence + * is what makes the fake's copy-OUT emit a marked archive, so a production regression that stops + * stamping publishes {@link LEGACY_FAKE_UNSTAMPED_PGDATA_TAR} and every warm-path test notices. + */ + stamp: string | undefined; } /** @@ -1087,6 +1104,8 @@ export function mockLegacyDockerDaemonCliSpawner( readonly failRestart?: boolean; readonly failCopyOut?: boolean; readonly failCopyIn?: boolean; + /** Fails the export path's baseline stamp (`docker cp - :`) only. */ + readonly failStamp?: boolean; } = {}, ) { const containers = new Map(); @@ -1141,6 +1160,7 @@ export function mockLegacyDockerDaemonCliSpawner( autoRemove: args.includes("--rm"), running: false, restored: undefined, + stamp: undefined, }); stdout = id; } else if (args[0] === "start") { @@ -1168,28 +1188,39 @@ export function mockLegacyDockerDaemonCliSpawner( } else if (args[0] === "rm") { containers.delete(args[args.length - 1] ?? ""); } else if (args[0] === "cp" && args[1] === "-") { - // Secret copy is `docker cp - :/`; restore is `docker cp - :`. - // Both use stdin, so failCopyIn must apply only to the restore — otherwise a warm - // fallback test kills the pgsodium root-key copy and never reaches the archive. + // Three different stdin copies share this form, so each failure switch has to pick out + // exactly one: the pgsodium root key goes to `:/`, the export path's baseline stamp to + // `:`, and the warm restore to `:`. Otherwise a warm + // fallback test would kill the root-key copy and never reach the archive. const [id = "", containerPath = ""] = (args[2] ?? "").split(":"); const container = containers.get(id); const received = yield* readStdin(command); const isSecret = containerPath === "" || containerPath === "/"; - if (container === undefined || (!isSecret && opts.failCopyIn === true)) { + const isStamp = containerPath === LEGACY_PGDATA_PATH; + const failed = + (isStamp && opts.failStamp === true) || + (!isSecret && !isStamp && opts.failCopyIn === true); + if (container === undefined || failed) { exitCode = 1; stderr = "no such container"; + } else if (isStamp) { + container.stamp = received; } else if (!isSecret) { container.restored = `${containerPath}::${received}`; } } else if (args[0] === "cp" && args[2] === "-") { - // Export: `docker cp : -`, tar on stdout. + // Export: `docker cp : -`, tar on stdout. What comes out depends on + // whether the container was stamped first — that is the whole point of the marker. const [id = ""] = (args[1] ?? "").split(":"); const container = containers.get(id); if (opts.failCopyOut === true || container === undefined) { exitCode = 1; stderr = "no such container"; } else { - stdout = LEGACY_FAKE_PGDATA_TAR; + stdout = + container.stamp === undefined + ? LEGACY_FAKE_UNSTAMPED_PGDATA_TAR + : LEGACY_FAKE_PGDATA_TAR; } } else if (args[0] === "container" && args[1] === "inspect") { const container = containers.get(args[2] ?? ""); @@ -1223,9 +1254,10 @@ export function mockLegacyDockerDaemonCliSpawner( /** * One readable label per Docker call, so a test can assert the SEQUENCE of meaningful steps - * rather than raw argv. `cp` is split three ways because the shadow issues three different - * copies: the pgsodium root key every shadow gets (`cp-secret`, `container-lifecycle.ts`), the - * baseline export (`cp-out`), and the baseline restore (`cp-in`). + * rather than raw argv. `cp` is split four ways because the shadow issues four different copies: + * the pgsodium root key every shadow gets (`cp-secret`, `container-lifecycle.ts`), the baseline + * marker stamped into PGDATA just before an export (`cp-stamp`), the baseline export (`cp-out`), + * and the baseline restore (`cp-in`). */ const stepOf = (args: ReadonlyArray): string => { if (args[0] === "network") return "network"; @@ -1234,7 +1266,8 @@ export function mockLegacyDockerDaemonCliSpawner( if (args[1] === "-") { const dest = args[2] ?? ""; const containerPath = dest.slice(dest.indexOf(":") + 1); - return containerPath === "" || containerPath === "/" ? "cp-secret" : "cp-in"; + if (containerPath === "" || containerPath === "/") return "cp-secret"; + return containerPath === LEGACY_PGDATA_PATH ? "cp-stamp" : "cp-in"; } if (args[2] === "-") return "cp-out"; return "cp-secret"; From b382f949048fec6509057755b59d73282068c2ad Mon Sep 17 00:00:00 2001 From: avallete Date: Mon, 17 Aug 2026 19:09:17 +0200 Subject: [PATCH 78/82] fix(cli): key-bind the snapshot marker, digest the Realtime seed constants The baseline marker's content is now the snapshot's own cache key and warm validation verifies it, so a valid snapshot copied over another key's filename is discarded (wrong-key verdict, distinct from a missing marker) instead of restoring a mismatched baseline. The baseline digest (renamed baseline_embedded_digest) now also hashes the Realtime seed constants persisted by the one-shot job (tenant id, encryption key, db user/name/port), so editing them without an image bump invalidates stale tars. Co-Authored-By: Claude Fable 5 --- .../shared/db-bootstrap/pgdata-snapshot.ts | 230 ++++++++++++++---- .../db-bootstrap/pgdata-snapshot.unit.test.ts | 90 ++++++- .../shared/db-bootstrap/realtime-env.ts | 4 +- .../shadow-cache.integration.test.ts | 89 ++++++- .../shared/db-bootstrap/shadow-cache.ts | 121 ++++++--- apps/cli/tests/helpers/legacy-mocks.ts | 47 +++- 6 files changed, 478 insertions(+), 103 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts index ff91168b83..6974cc5996 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.ts @@ -86,15 +86,15 @@ export const LEGACY_PGDATA_CLUSTER_ENTRY = `${LEGACY_PGDATA_DIR_NAME}/PG_VERSION * SCREAMING_SNAKE on purpose, and not by taste: `docker cp` tars a directory through Go's * `filepath.Walk`, which visits each level in sorted order, so an uppercase root file lands * immediately next to `PG_VERSION` — near the front of a ~90MB archive rather than behind every - * `base/` page. That is a PERFORMANCE hint for {@link legacyPgDataArchiveMissingEntries} only: + * `base/` page. That is a PERFORMANCE hint for {@link legacyValidatePgDataArchive} only: * the scan is correct at any position, and settles late (not wrongly) if a Docker release ever * reorders its walk. */ export const LEGACY_PGDATA_BASELINE_MARKER_NAME = "SUPABASE_BASELINE"; /** - * The marker's tar entry — what {@link legacyPgDataArchiveMissingEntries} looks for, and the reason - * a cached snapshot means "the Supabase platform baseline this key promises" rather than merely "a + * The marker's tar entry — what {@link legacyValidatePgDataArchive} looks for, and the reason a + * cached snapshot means "the Supabase platform baseline this key promises" rather than merely "a * PostgreSQL cluster". * * Its whole value is WHEN it is written: {@link legacyStampPgDataBaselineMarker} is called from the @@ -106,12 +106,23 @@ export const LEGACY_PGDATA_BASELINE_MARKER_NAME = "SUPABASE_BASELINE"; export const LEGACY_PGDATA_BASELINE_MARKER_ENTRY = `${LEGACY_PGDATA_DIR_NAME}/${LEGACY_PGDATA_BASELINE_MARKER_NAME}`; /** - * Only PRESENCE is the contract — nothing reads these bytes back, so the content can grow into - * provenance metadata later without breaking any reader. + * The marker's CONTENT: the caller's own identity token for what the snapshot carries — for the + * shadow baseline cache, the cache key the archive is published under, which is also its + * filename's stem (`legacyShadowBaselineTarFileName`, `shadow-cache.ts`). + * + * Presence alone is strictly weaker than a snapshot's own filename claims: a perfectly valid + * archive COPIED or RENAMED over another key's cache file passes a name-only check and warm-restores + * a baseline built from different roles/vault values/service versions. Binding the marker to the + * key — stamped at export, compared at validation ({@link legacyValidatePgDataArchive}) — is what + * makes an archive vouch for the filename it is stored under, not merely for "some baseline". + * + * The trailing newline is the canonical form on BOTH sides: it makes the stamped file a normal + * one-line text file (`cat`-able while debugging a cache directory) and both halves of the contract + * go through this one function, so the two can never drift. */ -const LEGACY_PGDATA_BASELINE_MARKER_CONTENT = "1\n"; +export const legacyPgDataBaselineMarkerContent = (key: string): string => `${key}\n`; -/** Every entry {@link legacyPgDataArchiveMissingEntries} requires of a restorable snapshot. */ +/** Every entry {@link legacyValidatePgDataArchive} requires of a restorable snapshot. */ export const LEGACY_PGDATA_REQUIRED_ENTRIES: ReadonlyArray = [ LEGACY_PGDATA_CLUSTER_ENTRY, LEGACY_PGDATA_BASELINE_MARKER_ENTRY, @@ -133,19 +144,20 @@ const legacyPgDataSnapshotUnavailable = (reason: string): LegacyPgDataSnapshotUn /** * The one-member tar {@link legacyStampPgDataBaselineMarker} pushes into the container: - * `SUPABASE_BASELINE` relative to the `docker cp` destination, which is PGDATA itself. + * `SUPABASE_BASELINE` relative to the `docker cp` destination, which is PGDATA itself, carrying + * {@link legacyPgDataBaselineMarkerContent}'s token for `key`. * * Exported for the unit test that round-trips it through this module's own scanner — the stamp and - * the check have to agree on the entry name, and nothing else proves that they do. + * the check have to agree on the entry name AND on the content encoding, and nothing else proves + * that they do. */ -export const legacyPgDataBaselineMarkerTar = (): Effect.Effect< - Uint8Array, - LegacyPgDataSnapshotUnavailable -> => +export const legacyPgDataBaselineMarkerTar = ( + key: string, +): Effect.Effect => Effect.tryPromise({ try: () => new Bun.Archive({ - [LEGACY_PGDATA_BASELINE_MARKER_NAME]: LEGACY_PGDATA_BASELINE_MARKER_CONTENT, + [LEGACY_PGDATA_BASELINE_MARKER_NAME]: legacyPgDataBaselineMarkerContent(key), }).bytes(), catch: (cause) => legacyPgDataSnapshotUnavailable( @@ -155,8 +167,9 @@ export const legacyPgDataBaselineMarkerTar = (): Effect.Effect< /** * Writes {@link LEGACY_PGDATA_BASELINE_MARKER_ENTRY} into the container's PGDATA, so the export - * that follows carries it and {@link legacyPgDataArchiveMissingEntries} can tell a snapshot of a - * COMPLETED baseline apart from any other cluster. + * that follows carries it and {@link legacyValidatePgDataArchive} can tell a snapshot of a + * COMPLETED baseline for `key` apart from any other cluster — including a valid snapshot of a + * DIFFERENT key that was copied over this key's cache file. * * Delivered as a stdin tar through `docker cp -`, the same form the restore side uses * (`legacyExtractPreStartArchiveIntoContainer`, `container-lifecycle.ts`) and for the same reason: @@ -171,9 +184,10 @@ export const legacyPgDataBaselineMarkerTar = (): Effect.Effect< export const legacyStampPgDataBaselineMarker = ( spawner: Spawner, containerId: string, + key: string, ): Effect.Effect => Effect.gen(function* () { - const tar = yield* legacyPgDataBaselineMarkerTar(); + const tar = yield* legacyPgDataBaselineMarkerTar(key); const failure = (detail: string) => legacyPgDataSnapshotUnavailable( `failed to stamp ${LEGACY_PGDATA_BASELINE_MARKER_ENTRY}: ${detail}`, @@ -289,6 +303,16 @@ const LEGACY_TAR_NO_BYTES = new Uint8Array(0); const legacyTarDecoder = new TextDecoder(); +/** + * The most content {@link legacyScanTarChunkForEntries} will ever buffer for `captureEntry`. The + * only entry any caller captures is the baseline marker, whose content is one short identity token, + * so a larger member cannot be a marker this module wrote: it is left UNCAPTURED (`captured` stays + * `undefined`, which every reader treats as "does not match"), rather than being read into memory + * on the word of an untrusted archive's own size field. Keeps the scan's O(1) memory property + * intact whatever a hand-placed tar in the cache directory claims. + */ +const LEGACY_TAR_CAPTURE_MAX_BYTES = 1024; + /** * {@link legacyScanTarChunkForEntries}'s carry-over state — everything needed to resume a header * walk at an arbitrary chunk boundary, and nothing else. `carry` holds the bytes of a header block @@ -308,25 +332,53 @@ export interface LegacyTarScanState { readonly ended: boolean; /** A block that is neither zero nor a checksum-valid header: not a tar (or a truncated one). */ readonly malformed: boolean; + /** + * The one entry whose CONTENT the walk reads rather than steps over — the baseline marker, whose + * bytes say which key the archive belongs to. `undefined` for a presence-only walk. + */ + readonly captureEntry: string | undefined; + /** + * {@link captureEntry}'s content bytes. `undefined` until its header is seen, and STILL undefined + * afterwards when the entry was too large to capture ({@link LEGACY_TAR_CAPTURE_MAX_BYTES}) — + * both mean "no content to compare against", which is the safe verdict either way. + */ + readonly captured: Uint8Array | undefined; + /** Content bytes of {@link captureEntry} still to be captured; `0` once complete or not capturing. */ + readonly capturePending: number; } -/** A fresh walk looking for `required` — every entry of which must appear for the scan to pass. */ -export const legacyInitialTarScanState = (required: Iterable): LegacyTarScanState => ({ +/** + * A fresh walk looking for `required` — every entry of which must appear for the scan to pass — + * capturing `captureEntry`'s content along the way when given (it must be one of `required`, so + * that "found everything" also means "the captured entry's header was seen"). + */ +export const legacyInitialTarScanState = ( + required: Iterable, + captureEntry?: string, +): LegacyTarScanState => ({ carry: LEGACY_TAR_NO_BYTES, skip: 0, zeroBlocks: 0, missing: new Set(required), ended: false, malformed: false, + captureEntry, + captured: undefined, + capturePending: 0, }); -/** Whether every required entry has been seen. */ -export const legacyTarScanFound = (state: LegacyTarScanState): boolean => state.missing.size === 0; +/** Whether every required entry has been seen AND its captured content (if any) is complete. */ +export const legacyTarScanFound = (state: LegacyTarScanState): boolean => + state.missing.size === 0 && state.capturePending === 0; /** Whether the scan has reached a verdict — nothing later in the archive can change it. */ export const legacyTarScanSettled = (state: LegacyTarScanState): boolean => legacyTarScanFound(state) || state.ended || state.malformed; +/** {@link LegacyTarScanState.captured} decoded, or `undefined` when there is nothing to compare. */ +export const legacyTarScanCapturedText = (state: LegacyTarScanState): string | undefined => + state.captured === undefined ? undefined : legacyTarDecoder.decode(state.captured); + /** A NUL-terminated text field of a tar header block. */ const legacyTarTextField = (block: Uint8Array, offset: number, length: number): string => { const raw = block.subarray(offset, offset + length); @@ -387,10 +439,12 @@ const legacyTarEntryName = (block: Uint8Array): string => { const legacyTarBlockIsZero = (block: Uint8Array): boolean => block.every((byte) => byte === 0); /** - * Folds one stream chunk into a tar HEADER walk looking for every entry still in `state.missing`. - * Pure and chunk-boundary-agnostic: file content is stepped over by byte count rather than - * buffered, so the whole scan costs one partial header block of memory no matter how large the - * archive is. Stops (and stays stopped) at the first of: the last required entry found, the + * Folds one stream chunk into a tar HEADER walk looking for every entry still in `state.missing`, + * capturing `state.captureEntry`'s content if it has one. Pure and chunk-boundary-agnostic: every + * other member's content is stepped over by byte count rather than buffered, so the whole scan + * costs one partial header block plus (at most) + * {@link LEGACY_TAR_CAPTURE_MAX_BYTES} of memory no matter how large the archive is. Stops (and + * stays stopped) at the first of: the last required entry found (with its capture complete), the * end-of-archive marker, or a block that is not a valid header. */ export const legacyScanTarChunkForEntries = ( @@ -402,7 +456,25 @@ export const legacyScanTarChunkForEntries = ( let skip = state.skip; let zeroBlocks = state.zeroBlocks; let missing = state.missing; - // A verdict keeps `missing` (it is the report) and drops the walk's resumption state. + let captured = state.captured; + let capturePending = state.capturePending; + const { captureEntry } = state; + /** + * Appends the leading `capturePending` bytes of a content run being consumed. Content bytes are + * always consumed front-to-back, and `capturePending` never exceeds what is left of the capture + * entry's own content, so this is correct whether the run is the tail carried over from the + * previous chunk or a fresh member's content in this one. + */ + const takeCapture = (bytes: Uint8Array): void => { + if (capturePending <= 0 || captured === undefined) return; + const take = Math.min(capturePending, bytes.length); + const merged = new Uint8Array(captured.length + take); + merged.set(captured); + merged.set(bytes.subarray(0, take), captured.length); + captured = merged; + capturePending -= take; + }; + // A verdict keeps `missing`/`captured` (they are the report) and drops the walk's resumption state. const settle = ( verdict: Pick, ): LegacyTarScanState => ({ @@ -410,11 +482,17 @@ export const legacyScanTarChunkForEntries = ( skip: 0, zeroBlocks: 0, missing, + captureEntry, + captured, + capturePending, ...verdict, }); // Content bytes carried over from the previous chunk come first — they are not headers. let offset = Math.min(skip, chunk.length); + takeCapture(chunk.subarray(0, offset)); skip -= offset; + // The capture may have been the only thing outstanding when the previous chunk ran out. + if (missing.size === 0 && capturePending === 0) return settle({ ended: false, malformed: false }); while (offset < chunk.length) { const available = chunk.length - offset; let block: Uint8Array; @@ -448,52 +526,99 @@ export const legacyScanTarChunkForEntries = ( return settle({ ended: false, malformed: true }); } const name = legacyTarEntryName(block); - if (missing.has(name)) { + const wasMissing = missing.has(name); + if (wasMissing) { const remaining = new Set(missing); remaining.delete(name); missing = remaining; - if (missing.size === 0) return settle({ ended: false, malformed: false }); } const size = legacyTarNumericField(block, 124, 12); if (size === undefined || size < 0) { return settle({ ended: false, malformed: true }); } + // Arm the capture on the capture entry's FIRST occurrence only (`wasMissing`), so a duplicate + // member later in the archive cannot overwrite what the real one said. An oversized entry is + // left uncaptured — see {@link LEGACY_TAR_CAPTURE_MAX_BYTES}. + if (wasMissing && name === captureEntry && size <= LEGACY_TAR_CAPTURE_MAX_BYTES) { + captured = LEGACY_TAR_NO_BYTES; + capturePending = size; + } // Content is padded up to the next block boundary; directories and links carry size 0. const content = Math.ceil(size / LEGACY_TAR_BLOCK_SIZE) * LEGACY_TAR_BLOCK_SIZE; const stepped = Math.min(content, chunk.length - offset); + takeCapture(chunk.subarray(offset, offset + stepped)); offset += stepped; skip = content - stepped; + // Only now, with the capture (if any) armed and fed: settling at the header would have + // discarded the very bytes the marker check needs. + if (missing.size === 0 && capturePending === 0) { + return settle({ ended: false, malformed: false }); + } } - return { carry, skip, zeroBlocks, missing, ended: false, malformed: false }; + return { + carry, + skip, + zeroBlocks, + missing, + ended: false, + malformed: false, + captureEntry, + captured, + capturePending, + }; }; /** - * Which of {@link LEGACY_PGDATA_REQUIRED_ENTRIES} `tarPath`'s member list does NOT contain — empty - * for a snapshot that is safe to restore. + * Why an archive at `tarPath` must not be restored under `key` — the two distinguishable ways + * {@link legacyValidatePgDataArchive} can reject one. Both implicate the FILE, never the + * infrastructure, so both are safe for a caller to act on by discarding it. + */ +export type LegacyPgDataArchiveProblem = + /** The header stream never carried one of {@link LEGACY_PGDATA_REQUIRED_ENTRIES}. */ + | { readonly _tag: "missing-entries"; readonly entries: ReadonlyArray } + /** + * Every entry is there, but the marker vouches for a DIFFERENT key than the one this archive is + * stored under. `found` is the marker's own token (trimmed), or `undefined` when it carried none + * that could be read (an oversized or truncated marker member). + */ + | { readonly _tag: "wrong-key"; readonly expected: string; readonly found: string | undefined }; + +/** + * Whether `tarPath` is a snapshot that may be restored as `key`'s baseline — `Option.none()` when + * it is, the reason it is not otherwise. * - * Both entries are load-bearing, and for different failures. Without `PG_VERSION`, an archive that - * is syntactically fine but carries no cluster (an EMPTY tar qualifies) restores SILENTLY: + * Three failures, each invisible to the restore itself. Without `PG_VERSION`, an archive that is + * syntactically fine but carries no cluster (an EMPTY tar qualifies) restores SILENTLY: * `docker cp -` extracts nothing, the Postgres entrypoint finds an empty PGDATA and runs a fresh * `initdb`, readiness passes, and the caller is handed a bare cluster it believes carries the * platform baseline. Without the baseline marker, that same silent-success shape survives one level * up: a REAL but bare PGDATA tar (dropped into the cache directory by hand, or produced by a future * regression that exports before the baseline runs) restores, starts, and answers — and only the - * resulting diff would ever show it. Validating the header stream up front is the only place either - * difference is observable, so callers must check BEFORE restoring. + * resulting diff would ever show it. And without the marker's CONTENT, it survives one level up + * again: a genuine, fully baselined snapshot of ANOTHER key, copied or renamed over this key's + * cache file, passes every name-only check while carrying different roles, vault values, and + * service-version schema — see {@link legacyPgDataBaselineMarkerContent}. Validating the header + * stream up front is the only place any of the three is observable, so callers must check BEFORE + * restoring. * * Reads the file locally — no Docker, no extraction — and stops as soon as both entries have been - * seen; see {@link LEGACY_PGDATA_BASELINE_MARKER_NAME} for why that is normally within the - * archive's first blocks. Even a full walk only parses HEADERS (content is stepped over by byte - * count), so the cost is one sequential read with O(1) memory. Only a genuine read failure fails; a - * valid tar missing an entry simply reports it. + * seen and the marker's few bytes read; see {@link LEGACY_PGDATA_BASELINE_MARKER_NAME} for why that + * is normally within the archive's first blocks. Even a full walk only parses HEADERS (every other + * member's content is stepped over by byte count), so the cost is one sequential read with O(1) + * memory. Only a genuine read failure fails; a valid tar that does not qualify simply reports why. */ -export const legacyPgDataArchiveMissingEntries = ( +export const legacyValidatePgDataArchive = ( fs: FileSystem.FileSystem, tarPath: string, -): Effect.Effect, LegacyPgDataSnapshotUnavailable> => + key: string, +): Effect.Effect, LegacyPgDataSnapshotUnavailable> => fs.stream(tarPath).pipe( Stream.mapAccum( - () => legacyInitialTarScanState(LEGACY_PGDATA_REQUIRED_ENTRIES), + () => + legacyInitialTarScanState( + LEGACY_PGDATA_REQUIRED_ENTRIES, + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + ), (state: LegacyTarScanState, chunk: Uint8Array) => { const next = legacyScanTarChunkForEntries(state, chunk); return [next, [next]] as const; @@ -504,14 +629,31 @@ export const legacyPgDataArchiveMissingEntries = ( // An empty file yields no chunks at all, so `None` means nothing was found: everything missing. Effect.map((last) => Option.isSome(last) - ? LEGACY_PGDATA_REQUIRED_ENTRIES.filter((entry) => last.value.missing.has(entry)) - : LEGACY_PGDATA_REQUIRED_ENTRIES, + ? legacyPgDataArchiveProblem(last.value, key) + : Option.some({ + _tag: "missing-entries", + entries: LEGACY_PGDATA_REQUIRED_ENTRIES, + }), ), Effect.mapError((cause) => legacyPgDataSnapshotUnavailable(`failed to read ${tarPath}: ${cause.message}`), ), ); +/** {@link legacyValidatePgDataArchive}'s verdict, split out so it is pure and directly testable. */ +const legacyPgDataArchiveProblem = ( + state: LegacyTarScanState, + key: string, +): Option.Option => { + const entries = LEGACY_PGDATA_REQUIRED_ENTRIES.filter((entry) => state.missing.has(entry)); + if (entries.length > 0) return Option.some({ _tag: "missing-entries", entries }); + const stamped = legacyTarScanCapturedText(state); + if (stamped === legacyPgDataBaselineMarkerContent(key)) return Option.none(); + // Trimmed for the report only — the comparison above is on the exact canonical form, so a marker + // padded with whitespace is a mismatch rather than something to normalize into a match. + return Option.some({ _tag: "wrong-key", expected: key, found: stamped?.trim() }); +}; + /** * Builds the {@link LegacyStartContainerSpec.preStartArchives} entry that restores a * {@link legacyExportPgDataTar} tar into a container between `docker create` and `docker start`. diff --git a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts index bed6e1c064..bb13014a08 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/pgdata-snapshot.unit.test.ts @@ -1,8 +1,8 @@ /** - * The pure tar-header walk behind {@link legacyPgDataArchiveMissingEntries}'s pre-restore check. + * The pure tar-header walk behind {@link legacyValidatePgDataArchive}'s pre-restore check. * Unit tests rather than integration ones because the interesting cases are all in the format - * handling: chunk boundaries landing mid-header, content that must be stepped over rather than - * parsed, and bytes that are not a tar at all. + * handling: chunk boundaries landing mid-header (or mid-MARKER-CONTENT), content that must be + * stepped over rather than parsed, and bytes that are not a tar at all. */ import { describe, expect, it } from "@effect/vitest"; @@ -14,8 +14,10 @@ import { LEGACY_PGDATA_CLUSTER_ENTRY, LEGACY_PGDATA_REQUIRED_ENTRIES, legacyInitialTarScanState, + legacyPgDataBaselineMarkerContent, legacyPgDataBaselineMarkerTar, legacyScanTarChunkForEntries, + legacyTarScanCapturedText, legacyTarScanFound, legacyTarScanSettled, type LegacyTarScanState, @@ -76,16 +78,29 @@ const scan = ( tar: Uint8Array, chunkSize: number, required: ReadonlyArray = LEGACY_PGDATA_REQUIRED_ENTRIES, + captureEntry: string | undefined = LEGACY_PGDATA_BASELINE_MARKER_ENTRY, ) => { - let state: LegacyTarScanState = legacyInitialTarScanState(required); + let state: LegacyTarScanState = legacyInitialTarScanState( + required, + required.includes(captureEntry ?? "") ? captureEntry : undefined, + ); for (let offset = 0; offset < tar.length; offset += chunkSize) { state = legacyScanTarChunkForEntries(state, tar.subarray(offset, offset + chunkSize)); if (legacyTarScanSettled(state)) break; } - return { ...state, found: legacyTarScanFound(state) }; + return { + ...state, + found: legacyTarScanFound(state), + capturedText: legacyTarScanCapturedText(state), + }; }; -const MARKER_ENTRY = tarEntry(LEGACY_PGDATA_BASELINE_MARKER_ENTRY, "1\n"); +const KEY = "0011223344556677"; + +const MARKER_ENTRY = tarEntry( + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + legacyPgDataBaselineMarkerContent(KEY), +); describe("legacyScanTarChunkForEntries", () => { const pgdataTar = concat( @@ -99,13 +114,45 @@ describe("legacyScanTarChunkForEntries", () => { it("finds every required entry regardless of where the chunk boundaries fall", () => { // 7 and 513 both split header blocks; the required entries are the last two members, behind a // directory and a file whose content must be stepped over rather than mistaken for a header. + // The marker's own CONTENT is read rather than stepped over, so those same boundaries also + // land mid-token — the capture has to survive being fed one byte at a time. for (const chunkSize of [1, 7, 100, 512, 513, 4096, pgdataTar.length]) { const state = scan(pgdataTar, chunkSize); expect(state.found, `chunk size ${chunkSize}`).toBe(true); expect(state.malformed).toBe(false); + expect(state.capturedText, `chunk size ${chunkSize}`).toBe( + legacyPgDataBaselineMarkerContent(KEY), + ); } }); + it("captures the marker's own key, not a same-named entry's, and not an oversized one", () => { + // The whole point of the content: two archives with identical member LISTS, telling apart the + // key each one actually vouches for. + const other = concat( + tarEntry("data/", "", "5"), + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + tarEntry( + LEGACY_PGDATA_BASELINE_MARKER_ENTRY, + legacyPgDataBaselineMarkerContent("ffffffff00000000"), + ), + TAR_END, + ); + expect(scan(other, 512).found).toBe(true); + expect(scan(other, 512).capturedText).toBe("ffffffff00000000\n"); + + // A marker member larger than the cap is left UNCAPTURED rather than buffered on the say-so of + // an untrusted archive's own size field — "no content" is the safe verdict, never a match. + const oversized = concat( + tarEntry(LEGACY_PGDATA_CLUSTER_ENTRY, "17\n"), + tarEntry(LEGACY_PGDATA_BASELINE_MARKER_ENTRY, "x".repeat(2048)), + TAR_END, + ); + const state = scan(oversized, 512); + expect(state.found).toBe(true); + expect(state.capturedText).toBeUndefined(); + }); + it("stops at the last required entry without walking the rest of the archive", () => { // Everything after them is garbage: reaching it would settle `malformed` instead. const trailing = concat( @@ -163,15 +210,34 @@ describe("legacyScanTarChunkForEntries", () => { }); describe("legacyPgDataBaselineMarkerTar", () => { - it("stamps the very entry the pre-restore scan requires", () => + it("stamps the very entry the pre-restore scan requires, carrying its key", () => Effect.runPromise( Effect.gen(function* () { // The stamp and the check are two halves of one contract, and only a round trip proves - // they agree: `docker cp - :` unpacks this archive's members RELATIVE to - // PGDATA, so its bare `SUPABASE_BASELINE` member is what a later export tars up as - // `data/SUPABASE_BASELINE`. - const bytes = yield* legacyPgDataBaselineMarkerTar(); - expect(scan(bytes, 512, [LEGACY_PGDATA_BASELINE_MARKER_NAME]).found).toBe(true); + // they agree — on the entry name AND on the content encoding: `docker cp - :` + // unpacks this archive's members RELATIVE to PGDATA, so its bare `SUPABASE_BASELINE` + // member is what a later export tars up as `data/SUPABASE_BASELINE`. + const bytes = yield* legacyPgDataBaselineMarkerTar(KEY); + const stamped = scan( + bytes, + 512, + [LEGACY_PGDATA_BASELINE_MARKER_NAME], + LEGACY_PGDATA_BASELINE_MARKER_NAME, + ); + expect(stamped.found).toBe(true); + // What the warm path compares against: the key this snapshot is published under, in the + // one canonical form both halves go through. + expect(stamped.capturedText).toBe(legacyPgDataBaselineMarkerContent(KEY)); + // ...so a snapshot stamped with ANOTHER key does not read as this one's. + const otherKey = yield* legacyPgDataBaselineMarkerTar("ffffffff00000000"); + expect( + scan( + otherKey, + 512, + [LEGACY_PGDATA_BASELINE_MARKER_NAME], + LEGACY_PGDATA_BASELINE_MARKER_NAME, + ).capturedText, + ).not.toBe(legacyPgDataBaselineMarkerContent(KEY)); // Nothing else rides along — one member, so the stamp costs a couple of blocks. expect(scan(bytes, 512, [LEGACY_PGDATA_BASELINE_MARKER_ENTRY]).found).toBe(false); }), diff --git a/apps/cli/src/legacy/shared/db-bootstrap/realtime-env.ts b/apps/cli/src/legacy/shared/db-bootstrap/realtime-env.ts index 8210ec20b4..9d31d45676 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/realtime-env.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/realtime-env.ts @@ -24,7 +24,7 @@ import { * `authenticator`, Storage's `supabase_storage_admin`), so it is not hoisted * alongside `legacyStartInternalDbUrl`. */ -const LEGACY_REALTIME_DB_USER = "supabase_admin"; +export const LEGACY_REALTIME_DB_USER = "supabase_admin"; /** * Go's `realtime.TenantId` default (`pkg/config/config.go:481`) — `toml:"-"` @@ -37,7 +37,7 @@ const LEGACY_REALTIME_DB_USER = "supabase_admin"; export const LEGACY_REALTIME_TENANT_ID = "realtime-dev"; /** Go's `realtime.EncryptionKey` default (`pkg/config/config.go:482`) — `toml:"-"`, never configurable. */ -const LEGACY_REALTIME_ENCRYPTION_KEY = "supabaserealtime"; +export const LEGACY_REALTIME_ENCRYPTION_KEY = "supabaserealtime"; /** Go's `realtime.SecretKeyBase` default (`pkg/config/config.go:483`) — `toml:"-"`, never configurable. */ const LEGACY_REALTIME_SECRET_KEY_BASE = diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts index 158707ce1b..6153f811b9 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.integration.test.ts @@ -20,8 +20,8 @@ import { Effect, Exit, FileSystem, Layer, Option, Path, Schema } from "effect"; import { LEGACY_FAKE_EMPTY_TAR, - LEGACY_FAKE_PGDATA_TAR, LEGACY_FAKE_UNSTAMPED_PGDATA_TAR, + legacyFakePgDataTar, legacyWithEnv, mockLegacyDockerDaemonCliSpawner, useLegacyTempWorkdir, @@ -35,6 +35,7 @@ import { LEGACY_PGDATA_BASELINE_MARKER_NAME, LEGACY_PGDATA_PARENT_PATH, LEGACY_PGDATA_PATH, + legacyPgDataBaselineMarkerContent, } from "./pgdata-snapshot.ts"; import { LEGACY_SHADOW_BASELINE_KEEP, @@ -183,6 +184,17 @@ const soleTarName = Effect.fnUntraced(function* (fs: FileSystem.FileSystem, path return entries.filter((entry) => entry.endsWith(".tar")); }); +/** The cache key a published snapshot's filename (`shadow-baseline-.tar`) is stored under. */ +const keyOf = (tarName: string) => tarName.slice("shadow-baseline-".length, -".tar".length); + +/** + * What a correct cold export publishes under `tarName`: the fake PGDATA archive carrying the + * baseline marker stamped with THAT filename's own key. Derived from the name rather than hardcoded + * so the assertion fails if the export ever stamps a different key than it publishes under. + */ +const expectedTarFor = (tarName: string) => + legacyFakePgDataTar(legacyPgDataBaselineMarkerContent(keyOf(tarName))); + /** A full cold run: acquire, export the baseline, release. */ const coldRun = ( docker: ReturnType, @@ -387,10 +399,11 @@ describe("legacyAcquireShadowDatabase", () => { `${handle.containerId}:${LEGACY_PGDATA_PATH}`, ]); // The stamp really carries the marker file, delivered as a tar so `docker cp` unpacks it - // relative to PGDATA rather than rewriting the directory's ownership. - expect(docker.containers.get(handle.containerId)?.stamp).toContain( - LEGACY_PGDATA_BASELINE_MARKER_NAME, - ); + // relative to PGDATA rather than rewriting the directory's ownership — and its content is + // this run's own cache key, which is what binds the artifact to the name it is filed under. + const stamp = docker.containers.get(handle.containerId)?.stamp ?? ""; + expect(stamp).toContain(LEGACY_PGDATA_BASELINE_MARKER_NAME); + expect(stamp).toContain(legacyPgDataBaselineMarkerContent(handle.snapshotKey ?? "")); expect(docker.stepCalls("cp-out")[0]).toEqual([ "cp", `${handle.containerId}:${LEGACY_PGDATA_PATH}`, @@ -404,7 +417,8 @@ describe("legacyAcquireShadowDatabase", () => { expect(tars).toHaveLength(1); expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); const published = yield* fs.readFileString(path.join(shadowCacheDir(path), tars[0] ?? "")); - expect(published).toBe(LEGACY_FAKE_PGDATA_TAR); + expect(published).toBe(expectedTarFor(tars[0] ?? "")); + expect(keyOf(tars[0] ?? "")).toBe(handle.snapshotKey); // The stamp made it all the way into the artifact — this is the entry the next run's // pre-restore scan requires, so a cold export that skipped it would never warm anything. expect(published).toContain(LEGACY_PGDATA_BASELINE_MARKER_ENTRY); @@ -445,8 +459,9 @@ describe("legacyAcquireShadowDatabase", () => { "-", `${warm.containerId}:${LEGACY_PGDATA_PARENT_PATH}`, ]); + const [warmTarName = ""] = yield* soleTarName(fs, path); expect(docker.containers.get(warm.containerId)?.restored).toBe( - `${LEGACY_PGDATA_PARENT_PATH}::${LEGACY_FAKE_PGDATA_TAR}`, + `${LEGACY_PGDATA_PARENT_PATH}::${expectedTarFor(warmTarName)}`, ); // Nothing more is exported: the baseline is already on disk. @@ -485,7 +500,7 @@ describe("legacyAcquireShadowDatabase", () => { const info = yield* fs.stat(tarPath); // 0o600 exactly — not the pre-created file's 0o666. expect((Number(info.mode) & 0o777).toString(8)).toBe("600"); - expect(yield* fs.readFileString(tarPath)).toBe(LEGACY_FAKE_PGDATA_TAR); + expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); @@ -804,7 +819,7 @@ describe("legacyAcquireShadowDatabase", () => { expect(yield* soleTarName(fs, path)).toEqual([]); yield* fallback.snapshotBaseline; expect(yield* soleTarName(fs, path)).toHaveLength(1); - expect(yield* fs.readFileString(tarPath)).toBe(LEGACY_FAKE_PGDATA_TAR); + expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); @@ -843,12 +858,66 @@ describe("legacyAcquireShadowDatabase", () => { // marked one within the same run. expect(yield* soleTarName(fs, path)).toEqual([]); yield* fallback.snapshotBaseline; - expect(yield* fs.readFileString(tarPath)).toBe(LEGACY_FAKE_PGDATA_TAR); + expect(yield* fs.readFileString(tarPath)).toBe(expectedTarFor(tarName)); yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); }), ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); }); + it.live( + "another key's snapshot copied over this key's filename is discarded, not restored", + () => { + const docker = mockLegacyDockerDaemonCliSpawner(); + const cluster = fakeCluster(); + const out = mockOutput(); + return withShadowCacheHome( + "1", + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const inputA = shadowInput(fs, path, { jwtExpiry: 3600 }); + const inputB = shadowInput(fs, path, { jwtExpiry: 7200 }); + // Two genuinely different configurations, each with its own published snapshot. + const coldA = yield* coldRun(docker, inputA); + const coldB = yield* coldRun(docker, inputB); + const keyA = coldA.snapshotKey ?? ""; + const keyB = coldB.snapshotKey ?? ""; + expect(keyA).not.toBe(keyB); + const tarPathB = path.join(shadowCacheDir(path), `shadow-baseline-${keyB}.tar`); + const tarPathA = path.join(shadowCacheDir(path), `shadow-baseline-${keyA}.tar`); + + // A's snapshot copied over B's cache file — the shape a copied `~/.supabase/cache` + // directory, a restored backup, or a hand-renamed tar produces. Every entry the + // presence check requires is there (it IS a real, fully baselined cluster), so only the + // marker's key separates it from B's own baseline: restoring it would silently diff + // against A's roles, vault values and service schema. + yield* fs.writeFileString(tarPathB, yield* fs.readFileString(tarPathA)); + const stepsBefore = docker.steps().length; + + const fallback = yield* legacyAcquireShadowDatabase(docker.spawner, inputB); + + expect(out.stderrText).toContain("cached shadow baseline unusable"); + expect(out.stderrText).toContain(`snapshot is stamped with key ${keyA}, not ${keyB}`); + expect(fallback.baselinePresent).toBe(false); + // Caught before any container was created, so nothing was ever restored. + expect(docker.steps().slice(stepsBefore)).not.toContain("cp-in"); + // Only the MISNAMED COPY goes: nothing else can ever be filed under B's name, while A's + // own tar — still correctly named — is left completely alone. + expect(yield* fs.exists(tarPathB)).toBe(false); + expect(yield* fs.readFileString(tarPathA)).toBe( + expectedTarFor(`shadow-baseline-${keyA}.tar`), + ); + // ...and the cold fallback republishes B's real baseline within the same run. + yield* fallback.snapshotBaseline; + expect(yield* fs.readFileString(tarPathB)).toBe( + expectedTarFor(`shadow-baseline-${keyB}.tar`), + ); + yield* legacyRemoveShadowDatabase(docker.spawner, fallback.containerId); + }), + ).pipe(Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, cluster.layer))); + }, + ); + it.live("a failed baseline stamp leaves the run uncached rather than publishing a tar", () => { const docker = mockLegacyDockerDaemonCliSpawner({ failStamp: true }); const cluster = fakeCluster(); diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts index 0640dc5130..c4a852c4c1 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts @@ -48,6 +48,15 @@ import { LEGACY_START_REVOKE_API_PRIVILEGES_SQL, type LegacySetupDatabaseOptions, } from "./db-setup.ts"; +import { + LEGACY_START_INTERNAL_DB_NAME, + LEGACY_START_INTERNAL_DB_PORT, +} from "./internal-db-connection.ts"; +import { + LEGACY_REALTIME_DB_USER, + LEGACY_REALTIME_ENCRYPTION_KEY, + LEGACY_REALTIME_TENANT_ID, +} from "./realtime-env.ts"; import { LEGACY_START_DB_SCHEMA_SQL } from "./templates/db-schema.sql.ts"; import { LEGACY_START_DB_SUPABASE_SQL } from "./templates/db-supabase.sql.ts"; import { LEGACY_START_DB_WEBHOOK_SQL } from "./templates/db-webhook.sql.ts"; @@ -60,11 +69,14 @@ import { import { legacyWaitForShadowReady } from "./health-check.ts"; import { legacyExportPgDataTar, - legacyPgDataArchiveMissingEntries, legacyPgDataRestoreArchive, legacyStampPgDataBaselineMarker, + legacyValidatePgDataArchive, +} from "./pgdata-snapshot.ts"; +import type { + LegacyPgDataArchiveProblem, + LegacyPgDataSnapshotUnavailable, } from "./pgdata-snapshot.ts"; -import type { LegacyPgDataSnapshotUnavailable } from "./pgdata-snapshot.ts"; import { legacyResolvePinnedImage } from "./pinned-image.ts"; import { legacyTimeShadowPhase } from "./shadow-debug.ts"; import { @@ -91,11 +103,15 @@ export const LEGACY_SHADOW_CACHE_ENV = "SUPABASE_SHADOW_CACHE"; interface LegacyShadowCacheUnavailable { readonly reason: string; /** - * `true` only when the failure implicates the TAR'S CONTENTS — today, exactly two producers, - * both in {@link legacyWarmShadow}: an archive whose header stream is missing a required entry — - * the PGDATA cluster file or the baseline marker — (checked before any container is created), - * and a restored cluster that started but never - * accepted connections (the readiness wait). Everything else (a `docker create`/`cp`/`start` + * `true` only when the failure implicates the TAR'S CONTENTS — today, exactly three producers, + * all in {@link legacyWarmShadow}: an archive whose header stream is missing a required entry — + * the PGDATA cluster file or the baseline marker; one whose marker vouches for a DIFFERENT cache + * key than the filename it is stored under (both checked before any container is created); and a + * restored cluster that started but never + * accepted connections (the readiness wait). The wrong-key case is the one where the bytes may + * be a perfectly good snapshot — of another key — so what is discarded is only the MISNAMED + * COPY, which is exactly right: nothing else can be keyed by this filename. + * Everything else (a `docker create`/`cp`/`start` * failure — daemon outage, * port collision, or even a corrupt archive's failed extraction) leaves the tar in place: an * infra failure says nothing about the tar, and a genuinely corrupt one is atomically @@ -225,13 +241,15 @@ export interface LegacyShadowCacheKeyInputs { } /** - * Digest of every CLI-EMBEDDED SQL text baked into the baseline cluster — the inputs that change + * Digest of every CLI-EMBEDDED literal baked into the baseline cluster — the inputs that change * with a CLI release rather than with the project's config: the PG15+ entrypoint's initdb heredocs - * (schema/webhook/_supabase — `postgres.service.ts`) and the API privilege revocation. Without this - * line, a CLI upgrade that edits a grant, schema statement, or revocation WITHOUT bumping the - * postgres image would warm-restore the previous release's baseline (review: depthfirst on #6184). - * Computed once at module load — these are compile-time constants. When adding a new embedded SQL - * step to the baseline (`legacySetupDatabase`/the entrypoint scripts), add its text here too. + * (schema/webhook/_supabase — `postgres.service.ts`), the API privilege revocation, and the + * Realtime one-shot job's seed constants. Without this line, a CLI upgrade that edits a grant, + * schema statement, revocation, or seeded literal WITHOUT bumping the corresponding image would + * warm-restore the previous release's baseline (review: depthfirst/Codex on #6184). Computed once + * at module load — these are compile-time constants. When adding a new embedded step to the + * baseline (`legacySetupDatabase`/the entrypoint scripts/a one-shot job's env), add its text here + * too. * * Deliberately EXCLUDES PG<=14's own setup SQL (`LEGACY_START_DB_GLOBALS_SQL`, * `LEGACY_START_DB_INITIAL_SCHEMA_13_SQL`/`_14_SQL`): PG<=14 is cache-ineligible — @@ -239,7 +257,7 @@ export interface LegacyShadowCacheKeyInputs { * any key is computed, so no cluster keyed by this digest can ever have run through that SQL. If * PG<=14 ever becomes cache-eligible, those templates must be re-added here. */ -const LEGACY_SHADOW_BASELINE_SQL_DIGEST = createHash("sha256") +const LEGACY_SHADOW_BASELINE_EMBEDDED_DIGEST = createHash("sha256") .update( [ LEGACY_START_DB_SCHEMA_SQL, @@ -255,6 +273,20 @@ const LEGACY_SHADOW_BASELINE_SQL_DIGEST = createHash("sha256") LEGACY_READ_VAULT_KV, LEGACY_UPDATE_VAULT_KV, LEGACY_CREATE_VAULT_KV, + // The Realtime one-shot job's CLI-embedded seed literals. `SEED_SELF_HOST=true` + // (`legacyBuildRealtimeEnv`, `realtime-env.ts`) makes that job PERSIST a tenant plus its + // `postgres_cdc_rls` extension settings into `_realtime`, encrypted with `DB_ENC_KEY` — so + // these values are baked into the snapshot exactly like the SQL above, and every one of them + // is a `toml:"-"`/hardcoded literal a CLI release can edit. `services.realtime.image` only + // re-keys when the IMAGE moves, so without these lines such an edit would warm-restore the + // old tenant identity and encryption key (review: Codex on #6184). Only the CONSTANTS + // belong here: the job's per-run env (the shadow's own short container id as `DB_HOST`, its + // password, the resolved JWKS) is either already a key field or deliberately excluded. + LEGACY_REALTIME_TENANT_ID, + LEGACY_REALTIME_ENCRYPTION_KEY, + LEGACY_REALTIME_DB_USER, + LEGACY_START_INTERNAL_DB_NAME, + String(LEGACY_START_INTERNAL_DB_PORT), ].join("\n--8<--\n"), "utf8", ) @@ -310,7 +342,7 @@ export function legacyShadowCacheKey(inputs: LegacyShadowCacheKeyInputs): string `api_grants_kept=${legacyBoolToken(legacyEffectiveShadowApiGrantsKept(inputs.autoExposeNewTables))}`, `webhooks_enabled=${legacyBoolToken(inputs.webhooksEnabled)}`, // Not a per-run input — see the digest's own doc comment for what it covers and why. - `baseline_sql_digest=${LEGACY_SHADOW_BASELINE_SQL_DIGEST}`, + `baseline_embedded_digest=${LEGACY_SHADOW_BASELINE_EMBEDDED_DIGEST}`, ]; for (const name of ["realtime", "storage", "auth"] as const) { const service = inputs.services[name]; @@ -767,14 +799,17 @@ const legacyExportShadowBaseline = ( const exported = yield* Effect.result( Effect.gen(function* () { yield* legacyShadowContainerVerb(spawner, "stop", containerId); - // The stamp is what makes the published tar mean "the baseline this key promises" rather - // than "some PostgreSQL cluster", and its ONLY guarantee is this position in the + // The stamp is what makes the published tar mean "the baseline THIS key promises" rather + // than "some PostgreSQL cluster". Two things give it that meaning. Its POSITION in the // sequence: this whole step runs from `snapshotBaseline`, which `legacySetupShadowDatabase` // invokes strictly after `legacySetupDatabase` returns (`shadow-database.ts`), and the - // stamp is the last mutation before the copy-out. A future regression that snapshots - // EARLIER therefore cannot produce a marked tar — it just stays uncached, instead of - // silently publishing a bare cluster under a baseline key. - yield* legacyStampPgDataBaselineMarker(spawner, containerId).pipe( + // stamp is the last mutation before the copy-out — so a future regression that snapshots + // EARLIER cannot produce a marked tar, it just stays uncached instead of silently + // publishing a bare cluster under a baseline key. And its CONTENT: `key` itself, which + // `legacyWarmShadow` compares against the key it resolved this run, so a valid snapshot + // of a DIFFERENT key that was copied over this filename is rejected too (review: Codex + // on #6184). + yield* legacyStampPgDataBaselineMarker(spawner, containerId, key).pipe( Effect.mapError((cause: LegacyPgDataSnapshotUnavailable) => legacyShadowCacheUnavailable(cause.reason), ), @@ -887,6 +922,31 @@ const legacyColdCachedShadow = ( })), ); +/** A cache key as {@link legacyShadowCacheKey} produces it — 16 hex chars, nothing else. */ +const LEGACY_SHADOW_CACHE_KEY_PATTERN = /^[0-9a-f]{16}$/u; + +/** + * The warm-path warning's wording for a rejected snapshot. Names WHICH of the two content failures + * happened, because they mean different things to whoever reads the line: a missing entry is a + * broken or hand-placed artifact, while a wrong key is a real snapshot of another configuration + * sitting under this one's filename (a copied cache directory, a renamed file) — and only the + * misnamed copy is being discarded, not that other key's own tar. + * + * The marker's own bytes are NOT echoed verbatim: they come from a file this run did not write, + * capped at a KiB but otherwise arbitrary, and stderr is not the place to render them. Only a token + * that is shaped like a cache key is shown. + */ +const legacyDescribeShadowArchiveProblem = (problem: LegacyPgDataArchiveProblem): string => { + if (problem._tag === "missing-entries") { + return `snapshot has no ${problem.entries.join(" or ")} entry`; + } + const found = + problem.found !== undefined && LEGACY_SHADOW_CACHE_KEY_PATTERN.test(problem.found) + ? `key ${problem.found}` + : "an unreadable key"; + return `snapshot is stamped with ${found}, not ${problem.expected}`; +}; + /** * The warm path proper: verify the snapshot tar really carries a baselined cluster, create the * shadow with it unpacked into it before it starts ({@link LegacyCreateShadowDatabaseInput.restoreArchive}), @@ -921,17 +981,20 @@ const legacyWarmShadow = ( // itself cannot report: `docker cp -` extracts whatever it is given, the entrypoint skips // `initdb` (or runs one over an empty PGDATA), readiness passes, and this function would hand // back `baselinePresent: true` for a cluster that never saw `legacySetupDatabase` — the caller - // then skips it too and diffs against a BARE database, silently producing wrong SQL. So the - // tar's own headers are scanned BEFORE anything is created (locally, no Docker) for both the - // cluster file AND the baseline marker this module stamps immediately before every export — - // see {@link legacyPgDataArchiveMissingEntries}. A read failure is infra and leaves the tar in - // place; a missing entry implicates its CONTENTS (review: Codex on #6184). - const missing = yield* legacyPgDataArchiveMissingEntries(input.fs, tarPath).pipe( + // then skips it too and diffs against a BARE database, silently producing wrong SQL. The same + // shape hides a second lie: a fully baselined snapshot of ANOTHER key, copied over this key's + // filename, restores just as cleanly while carrying different roles/vault values/service + // schema. So the tar's own headers are scanned BEFORE anything is created (locally, no Docker) + // for the cluster file, for the baseline marker this module stamps immediately before every + // export, AND for that marker's key — see {@link legacyValidatePgDataArchive}. A read failure + // is infra and leaves the tar in place; either verdict implicates its CONTENTS (review: Codex + // on #6184). + const problem = yield* legacyValidatePgDataArchive(input.fs, tarPath, key).pipe( Effect.mapError((cause) => legacyShadowCacheUnavailable(cause.reason)), ); - if (missing.length > 0) { + if (Option.isSome(problem)) { return yield* Effect.fail( - legacyShadowCacheUnavailable(`snapshot has no ${missing.join(" or ")} entry`, { + legacyShadowCacheUnavailable(legacyDescribeShadowArchiveProblem(problem.value), { tarSuspect: true, }), ); diff --git a/apps/cli/tests/helpers/legacy-mocks.ts b/apps/cli/tests/helpers/legacy-mocks.ts index dc2c55cd4d..7ada5b7431 100644 --- a/apps/cli/tests/helpers/legacy-mocks.ts +++ b/apps/cli/tests/helpers/legacy-mocks.ts @@ -36,7 +36,10 @@ import { LegacyLoginVerificationError, } from "../../src/legacy/commands/login/login.errors.ts"; import { LegacyCliConfig } from "../../src/legacy/config/legacy-cli-config.service.ts"; -import { LEGACY_PGDATA_PATH } from "../../src/legacy/shared/db-bootstrap/pgdata-snapshot.ts"; +import { + LEGACY_PGDATA_BASELINE_MARKER_NAME, + LEGACY_PGDATA_PATH, +} from "../../src/legacy/shared/db-bootstrap/pgdata-snapshot.ts"; import { legacyProjectRefLayer } from "../../src/legacy/config/legacy-project-ref.layer.ts"; import { LegacyLinkedProjectCache } from "../../src/legacy/telemetry/legacy-linked-project-cache.service.ts"; import { LegacyTelemetryState } from "../../src/legacy/telemetry/legacy-telemetry-state.service.ts"; @@ -1004,7 +1007,7 @@ export function mockLegacyShadowContainerCliSpawner( // --------------------------------------------------------------------------- /** - * A real (if tiny) POSIX tar, byte for byte — `legacyPgDataArchiveHasCluster` + * A real (if tiny) POSIX tar, byte for byte — `legacyValidatePgDataArchive` * (`db-bootstrap/pgdata-snapshot.ts`) walks the header stream and checksum-validates every block * before a warm restore, so the fake export has to be a genuine archive rather than a stand-in * string. Every byte stays ASCII (padding is NUL), which is why the spawner's `TextEncoder` and @@ -1059,8 +1062,35 @@ export const LEGACY_FAKE_UNSTAMPED_PGDATA_TAR = `${legacyFakeTarEntry("data/", " * The bytes the fake `docker cp :PGDATA -` emits once the export path has stamped the * container — stands in for a real ~90MB PGDATA tar, with the same top-level `data/` member, * `data/PG_VERSION` cluster file, and `data/SUPABASE_BASELINE` marker a real export carries. + * + * `markerContent` is whatever the export ACTUALLY stamped, carried through rather than fixed: the + * marker binds a snapshot to its cache key, so a fake that always emitted the same marker would + * make every key-binding assertion pass vacuously — a production regression stamping the wrong key + * (or a fixed one) has to show up as a warm-path mismatch here too. */ -export const LEGACY_FAKE_PGDATA_TAR = `${legacyFakeTarEntry("data/", "", "5")}${legacyFakeTarEntry("data/PG_VERSION", "17\n", "0")}${legacyFakeTarEntry("data/SUPABASE_BASELINE", "1\n", "0")}${LEGACY_FAKE_TAR_END}`; +export const legacyFakePgDataTar = (markerContent: string): string => + `${legacyFakeTarEntry("data/", "", "5")}${legacyFakeTarEntry("data/PG_VERSION", "17\n", "0")}${legacyFakeTarEntry("data/SUPABASE_BASELINE", markerContent, "0")}${LEGACY_FAKE_TAR_END}`; + +/** + * The `SUPABASE_BASELINE` member's content inside a `docker cp - :` stamp payload — a + * plain 512-block walk over the archive `legacyPgDataBaselineMarkerTar` built, so the fake export + * below can carry the real stamp forward. `undefined` when the payload has no such member, which + * is the fake's "this container was never really stamped" signal. + */ +const legacyFakeStampedMarkerContent = (stamp: string): string | undefined => { + let offset = 0; + while (offset + 512 <= stamp.length) { + const header = stamp.slice(offset, offset + 512); + offset += 512; + const name = (header.slice(0, 100).split("\0")[0] ?? "").replace(/^\.\//u, ""); + if (name.length === 0) return undefined; // the end-of-archive marker + const size = Number.parseInt((header.slice(124, 136).split("\0")[0] ?? "").trim(), 8); + if (!Number.isInteger(size) || size < 0) return undefined; + if (name === LEGACY_PGDATA_BASELINE_MARKER_NAME) return stamp.slice(offset, offset + size); + offset += Math.ceil(size / 512) * 512; + } + return undefined; +}; /** * A syntactically VALID tar carrying no members at all — what a replaced or truncated cache @@ -1217,10 +1247,15 @@ export function mockLegacyDockerDaemonCliSpawner( exitCode = 1; stderr = "no such container"; } else { - stdout = + // The marker the stamp really delivered rides along into the exported archive; a + // container that was never stamped (or whose stamp carried no marker member) exports the + // bare cluster the warm path must reject. + const marker = container.stamp === undefined - ? LEGACY_FAKE_UNSTAMPED_PGDATA_TAR - : LEGACY_FAKE_PGDATA_TAR; + ? undefined + : legacyFakeStampedMarkerContent(container.stamp); + stdout = + marker === undefined ? LEGACY_FAKE_UNSTAMPED_PGDATA_TAR : legacyFakePgDataTar(marker); } } else if (args[0] === "container" && args[1] === "inspect") { const container = containers.get(args[2] ?? ""); From 5ca12af03248f3e0e6583f43eafac059f76e28a3 Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 18 Aug 2026 10:54:11 +0200 Subject: [PATCH 79/82] test(cli): harden the shadow-cache live suite Docker-aware gate (describeDockerLive), ephemeral shadow port instead of a fixed one, best-effort network removal in the scope finalizer, and scoped SUPABASE_HOME/SUPABASE_SHADOW_CACHE overrides restored on every outcome. Co-Authored-By: Claude Fable 5 --- .../db-bootstrap/shadow-cache.live.test.ts | 201 +++++++++++------- 1 file changed, 120 insertions(+), 81 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts index 74c0b7f70a..683f3ce285 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts @@ -7,10 +7,10 @@ * a restored data directory, and on Postgres actually starting on a data directory copied out of a * stopped container. * - * Gated with `describeLive` (the cli-e2e-ci signal, which is also the only environment with a real - * Docker daemon). This is deliberately NOT a `runSupabaseLive` subprocess test: the contract under - * test is the acquire pair itself, and driving it directly avoids standing up a full local stack - * for `db diff` just to observe a container's cluster. + * Gated with `describeDockerLive` (the cli-e2e-ci signal composed with a `docker info` probe, since + * this is a Docker-only local-stack suite). This is deliberately NOT a `runSupabaseLive` subprocess + * test: the contract under test is the acquire pair itself, and driving it directly avoids standing + * up a full local stack for `db diff` just to observe a container's cluster. * * The platform baseline itself is out of scope here (its one-shot migrate jobs are exercised by the * `db diff`/`db pull` suites): the cache snapshots whatever PGDATA contains at the snapshot point, @@ -18,6 +18,7 @@ */ import { join } from "node:path"; +import * as net from "node:net"; import type { ProjectConfig } from "@supabase/config"; import { ProjectConfigSchema } from "@supabase/config"; @@ -26,9 +27,11 @@ import { expect, it } from "@effect/vitest"; import { Effect, FileSystem, Layer, Option, Path, Schema } from "effect"; import { ChildProcessSpawner } from "effect/unstable/process"; -import { describeLive } from "../../../../tests/helpers/live.ts"; +import { describeDockerLive } from "../../../../tests/helpers/live.ts"; +import { legacyWithEnv } from "../../../../tests/helpers/legacy-mocks.ts"; import { mockOutput } from "../../../../tests/helpers/mocks.ts"; import { dockerfileServiceImage } from "../../../shared/services/dockerfile-images.ts"; +import { containerCliExitCode } from "../legacy-container-cli.ts"; import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; import { legacyDbConnectionLayer } from "../legacy-db-connection.layer.ts"; import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; @@ -44,10 +47,24 @@ import type { LegacyShadowDbSetupInput, LegacyShadowSetupInput } from "./shadow- const defaultConfig: ProjectConfig = Schema.decodeUnknownSync(ProjectConfigSchema)({}); -/** A port well clear of the CLI's own defaults, so a live stack on this runner is untouched. */ -const LIVE_SHADOW_PORT = 54399; +const SHADOW_CACHE_LIVE_NETWORK_ID = "supabase_network_shadow_cache_live"; -describeLive("shadow baseline cache (live Docker)", () => { +/** + * Binds an ephemeral port and releases it immediately, mirroring + * `legacy-db-connection.sql-pg.integration.test.ts`'s `acquireClosedPort` idiom: a fixed port + * would collide with a concurrent live file (or a stray local process) holding it, so each run + * claims a free one from the OS instead. + */ +const acquireEphemeralPort = (): Promise => + new Promise((resolve, reject) => { + const server = net.createServer(); + server.listen(0, "127.0.0.1", () => { + const address = server.address() as net.AddressInfo; + server.close((error) => (error === undefined ? resolve(address.port) : reject(error))); + }); + }); + +describeDockerLive("shadow baseline cache (live Docker)", () => { it.live( "restores a fresh container from the exported snapshot, without the previous run's changes", () => { @@ -61,12 +78,12 @@ describeLive("shadow baseline cache (live Docker)", () => { prefix: "legacy-shadow-cache-home-", }); yield* fs.makeDirectory(path.join(workdir, "supabase"), { recursive: true }); - process.env.SUPABASE_HOME = supabaseHome; + const shadowPort = yield* Effect.promise(() => acquireEphemeralPort()); const setup: LegacyShadowDbSetupInput = { majorVersion: 17, config: defaultConfig, - dbUrl: `postgresql://postgres:postgres@127.0.0.1:${LIVE_SHADOW_PORT}/postgres`, + dbUrl: `postgresql://postgres:postgres@127.0.0.1:${shadowPort}/postgres`, jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", jwks: Effect.succeed('{"keys":[]}'), apiUrl: "http://127.0.0.1:54321", @@ -90,10 +107,10 @@ describeLive("shadow baseline cache (live Docker)", () => { experimental: defaultConfig.experimental, jwtSecret: setup.jwtSecret, jwtExpiry: 3600, - networkId: "supabase_network_shadow_cache_live", + networkId: SHADOW_CACHE_LIVE_NETWORK_ID, image: dockerfileServiceImage("postgres"), configImage: dockerfileServiceImage("postgres"), - shadowPort: LIVE_SHADOW_PORT, + shadowPort, password: "postgres", projectId: "shadow_cache_live", isBitbucketPipeline: false, @@ -113,79 +130,101 @@ describeLive("shadow baseline cache (live Docker)", () => { database: "postgres", }; - process.env[LEGACY_SHADOW_CACHE_ENV] = "1"; - expect(legacyShadowCacheEnabled()).toBe(true); - const connection = yield* LegacyDbConnection; - - // --- Run 1: cold provision, export the pristine cluster, then dirty it. --- - const cold = yield* legacyAcquireShadowDatabase(spawner, input); + // Owned by this run's `legacyEnsureNetwork` call inside `legacyAcquireShadowDatabase`: + // registered before either container's own finalizer below, so it removes the network + // LAST (finalizers run LIFO), after both containers have already been torn down. yield* Effect.addFinalizer(() => - legacyRemoveShadowDatabase(spawner, cold.containerId).pipe(Effect.ignore), - ); - expect(cold.baselinePresent).toBe(false); - yield* legacyWaitForShadowReady(spawner, cold.containerId, connConfig, { - timeoutSeconds: input.healthTimeoutSeconds, - }); - // The export stops and restarts the container, so nothing may be connected while it runs — - // exactly the contract `legacyMigrateShadowDatabase` honours around this same step. - yield* cold.snapshotBaseline; - yield* Effect.scoped( - Effect.gen(function* () { - const session = yield* connection.connect(connConfig, { - isLocal: true, - dnsResolver: "native", - }); - // Whatever a real run's migrations would do: a cluster-global role plus a database - // object, neither of which may survive into the next run. - yield* session.exec("CREATE ROLE shadow_cache_live_role"); - yield* session.exec("CREATE TABLE shadow_cache_live_table ()"); - }), + containerCliExitCode(spawner, ["network", "rm", SHADOW_CACHE_LIVE_NETWORK_ID], { + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + }).pipe(Effect.ignore), ); - yield* legacyRemoveShadowDatabase(spawner, cold.containerId); - // --- Run 2: the same key restores that snapshot into a NEW container, pristine. --- - const warm = yield* legacyAcquireShadowDatabase(spawner, input); - yield* Effect.addFinalizer(() => - legacyRemoveShadowDatabase(spawner, warm.containerId).pipe(Effect.ignore), - ); - // The cache keeps a file, never a container: run 2 is a brand new container that skipped - // the baseline because its PGDATA arrived pre-initialized. - expect(warm.containerId).not.toBe(cold.containerId); - expect(warm.baselinePresent).toBe(true); - yield* Effect.scoped( - Effect.gen(function* () { - const session = yield* connection.connect(connConfig, { - isLocal: true, - dnsResolver: "native", - }); - const roles = yield* session.query( - "SELECT rolname FROM pg_roles WHERE rolname = 'shadow_cache_live_role'", - ); - expect(roles).toEqual([]); - const tables = yield* session.query( - "SELECT tablename FROM pg_tables WHERE tablename = 'shadow_cache_live_table'", - ); - expect(tables).toEqual([]); - // Restored, not re-initialized: the baseline cluster's own roles are all still there. - const postgres = yield* session.query( - "SELECT rolname FROM pg_roles WHERE rolname = 'postgres'", - ); - expect(postgres).toHaveLength(1); - }), - ); - yield* legacyRemoveShadowDatabase(spawner, warm.containerId); + yield* legacyWithEnv( + "SUPABASE_HOME", + supabaseHome, + legacyWithEnv( + LEGACY_SHADOW_CACHE_ENV, + "1", + Effect.gen(function* () { + expect(legacyShadowCacheEnabled()).toBe(true); + const connection = yield* LegacyDbConnection; + + // --- Run 1: cold provision, export the pristine cluster, then dirty it. --- + const cold = yield* legacyAcquireShadowDatabase(spawner, input); + yield* Effect.addFinalizer(() => + legacyRemoveShadowDatabase(spawner, cold.containerId).pipe(Effect.ignore), + ); + expect(cold.baselinePresent).toBe(false); + yield* legacyWaitForShadowReady(spawner, cold.containerId, connConfig, { + timeoutSeconds: input.healthTimeoutSeconds, + }); + // The export stops and restarts the container, so nothing may be connected while it + // runs — exactly the contract `legacyMigrateShadowDatabase` honours around this same + // step. + yield* cold.snapshotBaseline; + yield* Effect.scoped( + Effect.gen(function* () { + const session = yield* connection.connect(connConfig, { + isLocal: true, + dnsResolver: "native", + }); + // Whatever a real run's migrations would do: a cluster-global role plus a + // database object, neither of which may survive into the next run. + yield* session.exec("CREATE ROLE shadow_cache_live_role"); + yield* session.exec("CREATE TABLE shadow_cache_live_table ()"); + }), + ); + yield* legacyRemoveShadowDatabase(spawner, cold.containerId); + + // --- Run 2: the same key restores that snapshot into a NEW container, pristine. --- + const warm = yield* legacyAcquireShadowDatabase(spawner, input); + yield* Effect.addFinalizer(() => + legacyRemoveShadowDatabase(spawner, warm.containerId).pipe(Effect.ignore), + ); + // The cache keeps a file, never a container: run 2 is a brand new container that + // skipped the baseline because its PGDATA arrived pre-initialized. + expect(warm.containerId).not.toBe(cold.containerId); + expect(warm.baselinePresent).toBe(true); + yield* Effect.scoped( + Effect.gen(function* () { + const session = yield* connection.connect(connConfig, { + isLocal: true, + dnsResolver: "native", + }); + const roles = yield* session.query( + "SELECT rolname FROM pg_roles WHERE rolname = 'shadow_cache_live_role'", + ); + expect(roles).toEqual([]); + const tables = yield* session.query( + "SELECT tablename FROM pg_tables WHERE tablename = 'shadow_cache_live_table'", + ); + expect(tables).toEqual([]); + // Restored, not re-initialized: the baseline cluster's own roles are all still + // there. + const postgres = yield* session.query( + "SELECT rolname FROM pg_roles WHERE rolname = 'postgres'", + ); + expect(postgres).toHaveLength(1); + }), + ); + yield* legacyRemoveShadowDatabase(spawner, warm.containerId); - // The artifact is a plain file under the global per-settings cache — the property that - // lets worktrees with the same settings share a warm hit, and a future native (non-Docker) - // Postgres service consume the same snapshot. - const tempDir = legacyShadowBaselineCacheDir(path); - const entries = yield* fs.readDirectory(tempDir); - const tars = entries.filter((entry) => entry.endsWith(".tar")); - expect(tars).toHaveLength(1); - expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); - expect(tempDir).toBe(join(supabaseHome, "cache", "shadow-baseline")); - expect(legacyShadowBaselineTarFileName("0".repeat(16))).toBe( - `shadow-baseline-${"0".repeat(16)}.tar`, + // The artifact is a plain file under the global per-settings cache — the property + // that lets worktrees with the same settings share a warm hit, and a future native + // (non-Docker) Postgres service consume the same snapshot. + const tempDir = legacyShadowBaselineCacheDir(path); + const entries = yield* fs.readDirectory(tempDir); + const tars = entries.filter((entry) => entry.endsWith(".tar")); + expect(tars).toHaveLength(1); + expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); + expect(tempDir).toBe(join(supabaseHome, "cache", "shadow-baseline")); + expect(legacyShadowBaselineTarFileName("0".repeat(16))).toBe( + `shadow-baseline-${"0".repeat(16)}.tar`, + ); + }), + ), ); }).pipe( Effect.scoped, From 4ba00ac24952cb7ec7090103697ded4e5c9f28ae Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 18 Aug 2026 11:14:02 +0200 Subject: [PATCH 80/82] test(cli): make the shadow-cache live suite a black-box CLI scenario MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One runSupabaseLive golden path: init + minimal start, then db diff --local --use-pg-delta twice against the same SUPABASE_HOME — the cold run must export the tar (shadow-debug: baseline-export), the warm run must restore it (baseline-restore, refreshed mtime, identical stdout). Replaces the in-process legacyAcquireShadowDatabase calls, which could stay green while the command wiring or env propagation broke; mechanics coverage lives in the integration suite. Shadow port comes from SUPABASE_DB_SHADOW_PORT with a bounded retry on bind conflicts — true reservation is impossible since Docker must bind the port itself. Co-Authored-By: Claude Fable 5 --- .../db-bootstrap/shadow-cache.live.test.ts | 427 +++++++++--------- 1 file changed, 208 insertions(+), 219 deletions(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts index 683f3ce285..aa053ba8e1 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts @@ -1,236 +1,225 @@ /** * The shadow baseline cache's ONE live scenario (golden path only, per the repo's live-test - * policy): against a real Docker daemon and a real `supabase/postgres` container, a cold acquire + - * export must leave a tar that the next acquire restores into a BRAND NEW container which comes up - * pristine — the facts no mock can prove, since they depend on `docker cp`'s tar stream actually - * preserving PGDATA's ownership, on `docker-entrypoint.sh` actually skipping `initdb` when it finds - * a restored data directory, and on Postgres actually starting on a data directory copied out of a - * stopped container. + * policy): the SAME `db diff` invocation run TWICE against a real local stack must cold-publish a + * `shadow-baseline-.tar` on the first run, warm-restore that exact tar on the second, and + * produce byte-identical diff output either way. + * + * A black-box `runSupabaseLive` subprocess test, like every other `*.live.test.ts` in this + * workspace: the facts it is here to prove are the ones only the real wiring can — that `db diff` + * actually routes through `legacyAcquireShadowDatabase`, that the cache's env gates + * (`SUPABASE_SHADOW_CACHE`/`SUPABASE_SHADOW_DEBUG`) and its `${SUPABASE_HOME}/cache/shadow-baseline` + * artifact location survive a real process boundary, that the cache key is STABLE across two + * separate CLI processes (an in-process test computes it once), and that a warm-restored cluster + * yields the same migration SQL as a cold-provisioned one. It replaces an earlier in-process + * version of this file that called `legacyAcquireShadowDatabase` directly with a synthetic layer + * graph — that shape could stay green while the `db diff` wiring, the env propagation, or the cache + * enablement was broken. + * + * The acquire/export/restore MECHANICS (cold export, warm restore, tar validation and rejection, + * retention/LRU, cache-off and bypass paths) are covered exhaustively by + * `shadow-cache.integration.test.ts` against its in-test Docker model plus a real filesystem, and + * the pure key/retention logic by `shadow-cache.unit.test.ts`. Nothing branch-shaped belongs here. * * Gated with `describeDockerLive` (the cli-e2e-ci signal composed with a `docker info` probe, since - * this is a Docker-only local-stack suite). This is deliberately NOT a `runSupabaseLive` subprocess - * test: the contract under test is the acquire pair itself, and driving it directly avoids standing - * up a full local stack for `db diff` just to observe a container's cluster. + * this is a Docker-only local-stack suite). + */ + +import { mkdtemp, readdir, rm, stat } from "node:fs/promises"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, expect, test } from "vitest"; + +import { makeTempHome } from "../../../../tests/helpers/cli.ts"; +import { describeDockerLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; + +const START_TIMEOUT_MS = 280_000; +const DIFF_TIMEOUT_MS = 180_000; +// One full `start` plus the cold/warm `db diff` pair, with lifecycle overhead for `init`, the +// filesystem inspection between runs, and the fast-failing port-conflict retries below (a +// conflicting publish fails in `docker create`/`start`, i.e. seconds, never a whole +// `DIFF_TIMEOUT_MS`) — same "budget each subprocess separately" shape as `diff.live.test.ts`. +const LIFECYCLE_OVERHEAD_MS = 90_000; + +/** + * `db diff`'s shadow port, published on the host by the shadow container. Docker itself has to + * bind it, so a test CANNOT truly reserve it up front: binding a listener and releasing it proves + * nothing about the window between the release and the container's own bind. The honest mitigation + * is therefore two-part — pick ports far from the `[db] shadow_port` default (54320) that a stray + * local stack or a neighbouring suite would be holding, and retry the scenario on the next + * candidate when the CLI reports a real bind conflict. * - * The platform baseline itself is out of scope here (its one-shot migrate jobs are exercised by the - * `db diff`/`db pull` suites): the cache snapshots whatever PGDATA contains at the snapshot point, - * so a bare cluster is a faithful stand-in for it. + * Fed through `SUPABASE_DB_SHADOW_PORT` (`legacy-db-config.toml-read.ts`'s `envOverride`) rather + * than by rewriting the generated `config.toml`, so the `init` template stays exactly as a user's + * would be. The port is deliberately NOT part of the cache key (see `legacyShadowCacheKey`), so + * retrying on a different one cannot change which tar the run looks for. */ +const SHADOW_PORT_CANDIDATES = [54987, 54988] as const; -import { join } from "node:path"; -import * as net from "node:net"; - -import type { ProjectConfig } from "@supabase/config"; -import { ProjectConfigSchema } from "@supabase/config"; -import { BunServices } from "@effect/platform-bun"; -import { expect, it } from "@effect/vitest"; -import { Effect, FileSystem, Layer, Option, Path, Schema } from "effect"; -import { ChildProcessSpawner } from "effect/unstable/process"; - -import { describeDockerLive } from "../../../../tests/helpers/live.ts"; -import { legacyWithEnv } from "../../../../tests/helpers/legacy-mocks.ts"; -import { mockOutput } from "../../../../tests/helpers/mocks.ts"; -import { dockerfileServiceImage } from "../../../shared/services/dockerfile-images.ts"; -import { containerCliExitCode } from "../legacy-container-cli.ts"; -import { LegacyDbConnection } from "../legacy-db-connection.service.ts"; -import { legacyDbConnectionLayer } from "../legacy-db-connection.layer.ts"; -import { legacyShadowBaselineCacheDir } from "../legacy-pgdelta.paths.ts"; -import { legacyWaitForShadowReady } from "./health-check.ts"; -import { - LEGACY_SHADOW_CACHE_ENV, - legacyAcquireShadowDatabase, - legacyShadowBaselineTarFileName, - legacyShadowCacheEnabled, -} from "./shadow-cache.ts"; -import { legacyRemoveShadowDatabase } from "./shadow-database.ts"; -import type { LegacyShadowDbSetupInput, LegacyShadowSetupInput } from "./shadow-database.ts"; - -const defaultConfig: ProjectConfig = Schema.decodeUnknownSync(ProjectConfigSchema)({}); - -const SHADOW_CACHE_LIVE_NETWORK_ID = "supabase_network_shadow_cache_live"; +const DIFF_ARGS = ["db", "diff", "--local", "--use-pg-delta"] as const; + +/** `shadow-cache.ts`'s published artifact name — `shadow-baseline-<16 hex key>.tar`. */ +const BASELINE_TAR_PATTERN = /^shadow-baseline-[0-9a-f]{16}\.tar$/u; /** - * Binds an ephemeral port and releases it immediately, mirroring - * `legacy-db-connection.sql-pg.integration.test.ts`'s `acquireClosedPort` idiom: a fixed port - * would collide with a concurrent live file (or a stray local process) holding it, so each run - * claims a free one from the OS instead. + * Docker's own bind-conflict wording, as it reaches stderr through the shadow's + * `docker create`/`docker start` failure. Only used to decide whether to retry on another + * candidate port — never asserted on. */ -const acquireEphemeralPort = (): Promise => - new Promise((resolve, reject) => { - const server = net.createServer(); - server.listen(0, "127.0.0.1", () => { - const address = server.address() as net.AddressInfo; - server.close((error) => (error === undefined ? resolve(address.port) : reject(error))); - }); - }); +function isShadowPortConflict(stderr: string): boolean { + return /port is already allocated|address already in use|Bind for \S+ failed/iu.test(stderr); +} + +async function baselineTars(cacheDir: string): Promise> { + const entries = await readdir(cacheDir).catch(() => [] as Array); + return entries.filter((entry) => entry.endsWith(".tar")).sort(); +} describeDockerLive("shadow baseline cache (live Docker)", () => { - it.live( - "restores a fresh container from the exported snapshot, without the previous run's changes", - () => { - const out = mockOutput(); - return Effect.gen(function* () { - const fs = yield* FileSystem.FileSystem; - const path = yield* Path.Path; - const spawner = yield* ChildProcessSpawner.ChildProcessSpawner; - const workdir = yield* fs.makeTempDirectoryScoped({ prefix: "legacy-shadow-cache-live-" }); - const supabaseHome = yield* fs.makeTempDirectoryScoped({ - prefix: "legacy-shadow-cache-home-", - }); - yield* fs.makeDirectory(path.join(workdir, "supabase"), { recursive: true }); - const shadowPort = yield* Effect.promise(() => acquireEphemeralPort()); - - const setup: LegacyShadowDbSetupInput = { - majorVersion: 17, - config: defaultConfig, - dbUrl: `postgresql://postgres:postgres@127.0.0.1:${shadowPort}/postgres`, - jwtSecret: "super-secret-jwt-token-with-at-least-32-characters-long", - jwks: Effect.succeed('{"keys":[]}'), - apiUrl: "http://127.0.0.1:54321", - authExternalUrl: undefined, - siteUrl: defaultConfig.auth.site_url, - anonKey: "anon-key", - serviceRoleKey: "service-role-key", - storageTargetMigration: "", - realtimeEnabledForSetup: false, - storageEnabledForSetup: false, - authEnabledForSetup: false, - serviceVersionOverrides: {}, - projectEnvValues: undefined, - debug: false, - webhooksEnabled: false, - apiAutoExposeNewTables: Option.some(true), - vault: [], - }; - const input: LegacyShadowSetupInput = { - db: { major_version: 17, settings: {} }, - experimental: defaultConfig.experimental, - jwtSecret: setup.jwtSecret, - jwtExpiry: 3600, - networkId: SHADOW_CACHE_LIVE_NETWORK_ID, - image: dockerfileServiceImage("postgres"), - configImage: dockerfileServiceImage("postgres"), - shadowPort, - password: "postgres", - projectId: "shadow_cache_live", - isBitbucketPipeline: false, - workdir, - extraHosts: [], - fs, - path, - hostname: "127.0.0.1", - healthTimeoutSeconds: 60, - setup, - }; - const connConfig = { - host: input.hostname, - port: input.shadowPort, - user: "postgres", - password: input.password, - database: "postgres", - }; + let projectDir: string | undefined; + let home: ReturnType | undefined; - // Owned by this run's `legacyEnsureNetwork` call inside `legacyAcquireShadowDatabase`: - // registered before either container's own finalizer below, so it removes the network - // LAST (finalizers run LIFO), after both containers have already been torn down. - yield* Effect.addFinalizer(() => - containerCliExitCode(spawner, ["network", "rm", SHADOW_CACHE_LIVE_NETWORK_ID], { - stdin: "ignore", - stdout: "ignore", - stderr: "ignore", - }).pipe(Effect.ignore), - ); - - yield* legacyWithEnv( - "SUPABASE_HOME", - supabaseHome, - legacyWithEnv( - LEGACY_SHADOW_CACHE_ENV, - "1", - Effect.gen(function* () { - expect(legacyShadowCacheEnabled()).toBe(true); - const connection = yield* LegacyDbConnection; - - // --- Run 1: cold provision, export the pristine cluster, then dirty it. --- - const cold = yield* legacyAcquireShadowDatabase(spawner, input); - yield* Effect.addFinalizer(() => - legacyRemoveShadowDatabase(spawner, cold.containerId).pipe(Effect.ignore), - ); - expect(cold.baselinePresent).toBe(false); - yield* legacyWaitForShadowReady(spawner, cold.containerId, connConfig, { - timeoutSeconds: input.healthTimeoutSeconds, - }); - // The export stops and restarts the container, so nothing may be connected while it - // runs — exactly the contract `legacyMigrateShadowDatabase` honours around this same - // step. - yield* cold.snapshotBaseline; - yield* Effect.scoped( - Effect.gen(function* () { - const session = yield* connection.connect(connConfig, { - isLocal: true, - dnsResolver: "native", - }); - // Whatever a real run's migrations would do: a cluster-global role plus a - // database object, neither of which may survive into the next run. - yield* session.exec("CREATE ROLE shadow_cache_live_role"); - yield* session.exec("CREATE TABLE shadow_cache_live_table ()"); - }), - ); - yield* legacyRemoveShadowDatabase(spawner, cold.containerId); - - // --- Run 2: the same key restores that snapshot into a NEW container, pristine. --- - const warm = yield* legacyAcquireShadowDatabase(spawner, input); - yield* Effect.addFinalizer(() => - legacyRemoveShadowDatabase(spawner, warm.containerId).pipe(Effect.ignore), - ); - // The cache keeps a file, never a container: run 2 is a brand new container that - // skipped the baseline because its PGDATA arrived pre-initialized. - expect(warm.containerId).not.toBe(cold.containerId); - expect(warm.baselinePresent).toBe(true); - yield* Effect.scoped( - Effect.gen(function* () { - const session = yield* connection.connect(connConfig, { - isLocal: true, - dnsResolver: "native", - }); - const roles = yield* session.query( - "SELECT rolname FROM pg_roles WHERE rolname = 'shadow_cache_live_role'", - ); - expect(roles).toEqual([]); - const tables = yield* session.query( - "SELECT tablename FROM pg_tables WHERE tablename = 'shadow_cache_live_table'", - ); - expect(tables).toEqual([]); - // Restored, not re-initialized: the baseline cluster's own roles are all still - // there. - const postgres = yield* session.query( - "SELECT rolname FROM pg_roles WHERE rolname = 'postgres'", - ); - expect(postgres).toHaveLength(1); - }), - ); - yield* legacyRemoveShadowDatabase(spawner, warm.containerId); - - // The artifact is a plain file under the global per-settings cache — the property - // that lets worktrees with the same settings share a warm hit, and a future native - // (non-Docker) Postgres service consume the same snapshot. - const tempDir = legacyShadowBaselineCacheDir(path); - const entries = yield* fs.readDirectory(tempDir); - const tars = entries.filter((entry) => entry.endsWith(".tar")); - expect(tars).toHaveLength(1); - expect(tars[0]).toMatch(/^shadow-baseline-[0-9a-f]{16}\.tar$/u); - expect(tempDir).toBe(join(supabaseHome, "cache", "shadow-baseline")); - expect(legacyShadowBaselineTarFileName("0".repeat(16))).toBe( - `shadow-baseline-${"0".repeat(16)}.tar`, - ); - }), - ), - ); - }).pipe( - Effect.scoped, - Effect.provide(Layer.mergeAll(BunServices.layer, out.layer, legacyDbConnectionLayer)), + afterEach(async () => { + if (projectDir !== undefined) { + // Best-effort cleanup even if an assertion above failed mid-lifecycle — a leaked local + // stack would otherwise pollute the CI runner for later jobs. + await runSupabaseLive(["stop", "--no-backup"], { + cwd: projectDir, + ...(home === undefined ? {} : { home: home.dir }), + }).catch(() => undefined); + await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); + } + // Disposes the temp `SUPABASE_HOME`, and with it the ~90MB baseline tar this suite published. + home?.[Symbol.dispose](); + projectDir = undefined; + home = undefined; + }); + + test( + "publishes a baseline snapshot on the first db diff, then restores it on the second with identical output", + { timeout: START_TIMEOUT_MS + 2 * DIFF_TIMEOUT_MS + LIFECYCLE_OVERHEAD_MS }, + async () => { + projectDir = await mkdtemp(path.join(tmpdir(), "sb-shadow-cache-live-")); + // One temp `SUPABASE_HOME` for every run in this test, so the two `db diff` processes share + // the global `${SUPABASE_HOME}/cache/shadow-baseline` directory the cache publishes into — + // `runSupabase` otherwise mints (and disposes) a fresh home per invocation, which would make + // every run a cold one. + home = makeTempHome(); + const cacheDir = path.join(home.dir, "cache", "shadow-baseline"); + + const init = await runSupabaseLive(["init"], { cwd: projectDir, home: home.dir }); + expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + + // Same declarative setup as `db/diff/diff.live.test.ts`: point `[db.migrations] + // schema_paths` at a schema directory so `db diff --local` has real, deterministic SQL to + // produce — the payload whose byte-identity across the cold and warm runs is the actual + // user-visible contract here. Paths are relative to `supabase/`. + const configPath = path.join(projectDir, "supabase", "config.toml"); + const config = readFileSync(configPath, "utf8"); + expect(config).toContain("schema_paths = []"); + writeFileSync( + configPath, + config.replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]'), + ); + const schemasDir = path.join(projectDir, "supabase", "schemas"); + mkdirSync(schemasDir, { recursive: true }); + writeFileSync( + path.join(schemasDir, "01_probe_fn.sql"), + `create function public.probe_fn() +returns void +language sql +as $$ select 1; $$; +`, ); + + // Exclude the heaviest, least relevant services — `db diff` only needs the local Postgres + // container reachable, same rationale as stop/status/diff. + const start = await runSupabaseLive( + ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], + { cwd: projectDir, home: home.dir, exitTimeoutMs: START_TIMEOUT_MS }, + ); + expect(start.exitCode, `stdout:\n${start.stdout}\nstderr:\n${start.stderr}`).toBe(0); + + let cold: Awaited> | undefined; + let warm: Awaited> | undefined; + let coldTars: ReadonlyArray = []; + let coldMtimeMs = 0; + + for (const [index, shadowPort] of SHADOW_PORT_CANDIDATES.entries()) { + const canRetry = index < SHADOW_PORT_CANDIDATES.length - 1; + // Each attempt must start from an empty cache, or the previous attempt's tar would make + // this attempt's first run a warm one. + await rm(cacheDir, { recursive: true, force: true }); + const diffOptions = { + cwd: projectDir, + home: home.dir, + exitTimeoutMs: DIFF_TIMEOUT_MS, + env: { + // The e2e/live harness pins this to "0" by default so ordinary suites never leave a + // ~90MB tar behind; this suite's subject IS the cache, so it opts back in. + SUPABASE_SHADOW_CACHE: "1", + // Turns on `shadow-debug.ts`'s stderr phase lines, which name the path actually taken + // (`baseline-export` cold, `baseline-restore` warm). + SUPABASE_SHADOW_DEBUG: "1", + SUPABASE_DB_SHADOW_PORT: String(shadowPort), + }, + }; + + const first = await runSupabaseLive([...DIFF_ARGS], diffOptions); + if (first.exitCode !== 0 && canRetry && isShadowPortConflict(first.stderr)) continue; + cold = first; + coldTars = await baselineTars(cacheDir); + if (coldTars.length === 1) { + coldMtimeMs = (await stat(path.join(cacheDir, coldTars[0]!))).mtimeMs; + } + // A genuine cold-run failure is reported below rather than spending another full + // `DIFF_TIMEOUT_MS` on a warm run that has no snapshot to restore. + if (first.exitCode !== 0) break; + + const second = await runSupabaseLive([...DIFF_ARGS], diffOptions); + if (second.exitCode !== 0 && canRetry && isShadowPortConflict(second.stderr)) { + cold = undefined; + continue; + } + warm = second; + break; + } + + // --- Run 1: cold. The baseline was provisioned and exported as one keyed tar. --- + expect(cold, "every candidate shadow port reported a bind conflict").toBeDefined(); + if (cold === undefined) return; + expect(cold.exitCode, `stdout:\n${cold.stdout}\nstderr:\n${cold.stderr}`).toBe(0); + expect(cold.stderr).toContain("shadow-debug: baseline-export"); + expect(cold.stderr).not.toContain("shadow-debug: baseline-restore"); + // The artifact is a plain file under the global per-settings cache — the property that lets + // worktrees with the same settings share a warm hit, and a future native (non-Docker) + // Postgres service consume the same snapshot. + expect(coldTars, `cache dir: ${cacheDir}\nstderr:\n${cold.stderr}`).toHaveLength(1); + expect(coldTars[0]).toMatch(BASELINE_TAR_PATTERN); + + // --- Run 2: warm. The same key restored that snapshot instead of rebuilding it. --- + expect(warm).toBeDefined(); + if (warm === undefined) return; + expect(warm.exitCode, `stdout:\n${warm.stdout}\nstderr:\n${warm.stderr}`).toBe(0); + expect(warm.stderr).toContain("shadow-debug: baseline-restore"); + expect(warm.stderr).not.toContain("shadow-debug: baseline-export"); + // Neither degradation path may have engaged — both warn on stderr before falling back to a + // cold provision, and either would otherwise hide a broken warm path behind a passing run. + expect(warm.stderr).not.toContain("cached shadow baseline unusable"); + expect(warm.stderr).not.toContain("shadow baseline not cached"); + // Same single tar, same filename: the key is reproducible across processes, and the warm run + // published nothing of its own. + const warmTars = await baselineTars(cacheDir); + expect(warmTars).toEqual(coldTars); + // Warm hits refresh mtime so a frequently used key survives LRU/TTL retention. + const warmMtimeMs = (await stat(path.join(cacheDir, warmTars[0]!))).mtimeMs; + expect(warmMtimeMs).toBeGreaterThan(coldMtimeMs); + + // The user-visible contract is unchanged by which path ran: stdout carries the migration SQL + // (no `-f`, so `db diff` prints it), and a restored cluster must diff to exactly the same + // statements as a freshly baselined one. + expect(cold.stdout).toContain("CREATE FUNCTION public.probe_fn()"); + expect(warm.stdout).toBe(cold.stdout); }, - 300_000, ); }); From 9d277539bbbfb421c3a2ff6d4a5082a023a7aad4 Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 18 Aug 2026 11:23:01 +0200 Subject: [PATCH 81/82] test(cli): derive the live suite's shadow-port candidates from the run's pid Eight candidates from a pid-seeded base in the IANA dynamic range replace the fixed 54987/54988 pair, so independently concurrent runs start from different bases and an occupied port costs one retry step. Co-Authored-By: Claude Fable 5 --- .../shared/db-bootstrap/shadow-cache.live.test.ts | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts index aa053ba8e1..c76445b069 100644 --- a/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts +++ b/apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.live.test.ts @@ -53,8 +53,18 @@ const LIFECYCLE_OVERHEAD_MS = 90_000; * than by rewriting the generated `config.toml`, so the `init` template stays exactly as a user's * would be. The port is deliberately NOT part of the cache key (see `legacyShadowCacheKey`), so * retrying on a different one cannot change which tar the run looks for. + * + * The candidate sequence is derived from this process's own pid, so two independently concurrent + * runs of this suite start from different bases instead of racing for one shared pair, and a + * locally-occupied port only costs one retry step. The base stays inside the IANA dynamic range + * (49152-65535) with room for every candidate below its ceiling. */ -const SHADOW_PORT_CANDIDATES = [54987, 54988] as const; +const SHADOW_PORT_CANDIDATE_COUNT = 8; +const SHADOW_PORT_BASE = 49152 + ((process.pid * 37) % (16384 - SHADOW_PORT_CANDIDATE_COUNT)); +const SHADOW_PORT_CANDIDATES: ReadonlyArray = Array.from( + { length: SHADOW_PORT_CANDIDATE_COUNT }, + (_, index) => SHADOW_PORT_BASE + index, +); const DIFF_ARGS = ["db", "diff", "--local", "--use-pg-delta"] as const; From f3dc6094f336d82df94fcfea9ed02655ae571b3d Mon Sep 17 00:00:00 2001 From: avallete Date: Tue, 18 Aug 2026 15:59:11 +0200 Subject: [PATCH 82/82] docs(cli): log the image-tag-vs-digest cache-key accepted risk Co-Authored-By: Claude Fable 5 --- docs/roadmap/pg-delta-next-follow-ups.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/roadmap/pg-delta-next-follow-ups.md b/docs/roadmap/pg-delta-next-follow-ups.md index cfd42cf6fb..760e1df470 100644 --- a/docs/roadmap/pg-delta-next-follow-ups.md +++ b/docs/roadmap/pg-delta-next-follow-ups.md @@ -87,3 +87,14 @@ the command correct, at worst at cold-provision speed. container, job skipped) leaves that dead id in the encrypted settings. Only migrations that inspect or act on Realtime tenant configuration can observe the difference. Fix: normalize the seeded host before snapshotting, or refresh it after restore. (Codex comment 3789481478.) +- **Cache key hashes image tags, not immutable digests** (`shadow-cache.ts`, + `legacyShadowCacheKey`): a registry tag republished with different bytes (or a locally retagged + image) keeps the same key, so a warm hit would restore the previous image's baseline and skip + the updated service migrations. Accepted risk: every keyed image is an exact pinned version tag + the release pipeline treats as immutable, a changed `SUPABASE_INTERNAL_IMAGE_REGISTRY` is + already in the key, and the 14-day TTL bounds staleness. Hashing digests is structurally + costly — the key is computed before images are pulled, so `docker image inspect` would have to + either move pulls ahead of the cache decision or fall back to tags for unpulled images (making + the first published tar never warm-hit), and it adds per-acquire inspect round-trips to the + warm path. Revisit only if the image-resolution pipeline ever produces mutable tags (`latest`, + branch tags) — those should become cache-ineligible at that point. (Codex comment 3802804637.)