From d7a4a4058f705ac18c68c18ce41fcf1fddccae5b Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 15:19:54 +0200 Subject: [PATCH 01/23] docs(cli-e2e): document live coverage strategy --- CONTRIBUTING.md | 124 +++++++++++++++++++++++------------------ apps/cli-e2e/AGENTS.md | 5 +- docs/adr/README.md | 34 +++++------ 3 files changed, 91 insertions(+), 72 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4ea9a7eac9..6e2834c4fd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -52,13 +52,13 @@ mise install `mise install` resolves the versions this repo expects from a handful of files, rather than hardcoding them all in one place: -| Tool | Version source | -| --- | --- | -| Bun | `.bun-version` | -| Node.js | `devEngines.runtime` field in `package.json` | -| pnpm | `packageManager` field in `package.json` | -| Go | `mise.toml` | -| golangci-lint | `mise.toml` | +| Tool | Version source | +| ------------- | -------------------------------------------- | +| Bun | `.bun-version` | +| Node.js | `devEngines.runtime` field in `package.json` | +| pnpm | `packageManager` field in `package.json` | +| Go | `mise.toml` | +| golangci-lint | `mise.toml` | The Go and golangci-lint entries in `mise.toml` are intentionally temporary while the Go CLI remains in the repo. The canonical Go module metadata still lives in `apps/cli-go/go.mod`; keep the `mise.toml` entries aligned only until the Go code is removed. @@ -105,28 +105,28 @@ That pulls `.repos/effect/`, which is the local source of truth for Effect v4 AP ## Apps -| Workspace | Purpose | -| --- | --- | -| `apps/cli` | Main `supabase` package. Contains command handlers, runtime services, auth, output, telemetry, and docs generation scripts. | +| Workspace | Purpose | +| -------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| `apps/cli` | Main `supabase` package. Contains command handlers, runtime services, auth, output, telemetry, and docs generation scripts. | | `apps/cli-e2e` | Compatibility e2e test suite. Record-and-replay harness for testing the TS Legacy port against real Supabase Management API responses. | -| `apps/docs` | Internal docs site built with Next.js and generated from the CLI docs sources. | +| `apps/docs` | Internal docs site built with Next.js and generated from the CLI docs sources. | ## Packages -| Workspace | Purpose | -| --- | --- | -| `packages/api` | Auto-generated TypeScript client for the Supabase Management API. | -| `packages/cli-test-helpers` | CLI test harness library — `createHarness`/`exec` API for spawning TS Legacy and TS Next CLI subprocesses in tests. | -| `packages/config` | JSON Schema and generated TypeScript types for Supabase configuration. | -| `packages/process-compose` | TypeScript/Bun port of `process-compose` used for multi-service orchestration. | -| `packages/stack` | Programmatic local Supabase stack used by the CLI and other tooling. | -| `packages/cli-darwin-arm64` | Published native CLI binary wrapper for macOS arm64. | -| `packages/cli-darwin-x64` | Published native CLI binary wrapper for macOS x64. | -| `packages/cli-linux-arm64` | Published native CLI binary wrapper for Linux arm64 (glibc). | -| `packages/cli-linux-arm64-musl` | Published native CLI binary wrapper for Linux arm64 (musl). | -| `packages/cli-linux-x64` | Published native CLI binary wrapper for Linux x64 (glibc). | -| `packages/cli-linux-x64-musl` | Published native CLI binary wrapper for Linux x64 (musl). | -| `packages/cli-windows-x64` | Published native CLI binary wrapper for Windows x64. | +| Workspace | Purpose | +| ------------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `packages/api` | Auto-generated TypeScript client for the Supabase Management API. | +| `packages/cli-test-helpers` | CLI test harness library — `createHarness`/`exec` API for spawning TS Legacy and TS Next CLI subprocesses in tests. | +| `packages/config` | JSON Schema and generated TypeScript types for Supabase configuration. | +| `packages/process-compose` | TypeScript/Bun port of `process-compose` used for multi-service orchestration. | +| `packages/stack` | Programmatic local Supabase stack used by the CLI and other tooling. | +| `packages/cli-darwin-arm64` | Published native CLI binary wrapper for macOS arm64. | +| `packages/cli-darwin-x64` | Published native CLI binary wrapper for macOS x64. | +| `packages/cli-linux-arm64` | Published native CLI binary wrapper for Linux arm64 (glibc). | +| `packages/cli-linux-arm64-musl` | Published native CLI binary wrapper for Linux arm64 (musl). | +| `packages/cli-linux-x64` | Published native CLI binary wrapper for Linux x64 (glibc). | +| `packages/cli-linux-x64-musl` | Published native CLI binary wrapper for Linux x64 (musl). | +| `packages/cli-windows-x64` | Published native CLI binary wrapper for Windows x64. | ## Working In The Monorepo @@ -143,22 +143,22 @@ pnpm run fix:all # run all fixers across every project All standard TypeScript workspaces (`apps/cli`, `packages/api`, `packages/config`, `packages/process-compose`, `packages/stack`) expose the following scripts: -| Script | What it does | -|--------|--------------| -| `test` | Run the full test suite (unit + integration + e2e) | -| `test:core` | Run unit and integration tests | -| `test:unit` | Run unit tests _(inferred by Nx plugin)_ | -| `test:integration` | Run integration tests _(inferred by Nx plugin)_ | -| `test:e2e` | Run end-to-end tests _(inferred by Nx plugin)_ | -| `check:all` | Run all check targets for this project | -| `fix:all` | Run all fix targets for this project | -| `types:check` | Type-check with `tsc --noEmit` _(inferred by Nx plugin)_ | -| `lint:check` | Check for lint errors with `oxlint` _(inferred by Nx plugin)_ | -| `lint:fix` | Auto-fix lint errors _(inferred by Nx plugin)_ | -| `fmt:check` | Check formatting with `oxfmt --check` _(inferred by Nx plugin)_ | -| `fmt:fix` | Auto-fix formatting _(inferred by Nx plugin)_ | -| `knip:check` | Find unused exports and dependencies with `knip-bun` _(inferred by Nx plugin)_ | -| `knip:fix` | Auto-remove unused exports and dependencies _(inferred by Nx plugin)_ | +| Script | What it does | +| ------------------ | ------------------------------------------------------------------------------ | +| `test` | Run the full test suite (unit + integration + e2e) | +| `test:core` | Run unit and integration tests | +| `test:unit` | Run unit tests _(inferred by Nx plugin)_ | +| `test:integration` | Run integration tests _(inferred by Nx plugin)_ | +| `test:e2e` | Run end-to-end tests _(inferred by Nx plugin)_ | +| `check:all` | Run all check targets for this project | +| `fix:all` | Run all fix targets for this project | +| `types:check` | Type-check with `tsc --noEmit` _(inferred by Nx plugin)_ | +| `lint:check` | Check for lint errors with `oxlint` _(inferred by Nx plugin)_ | +| `lint:fix` | Auto-fix lint errors _(inferred by Nx plugin)_ | +| `fmt:check` | Check formatting with `oxfmt --check` _(inferred by Nx plugin)_ | +| `fmt:fix` | Auto-fix formatting _(inferred by Nx plugin)_ | +| `knip:check` | Find unused exports and dependencies with `knip-bun` _(inferred by Nx plugin)_ | +| `knip:fix` | Auto-remove unused exports and dependencies _(inferred by Nx plugin)_ | The inferred scripts (`test:unit`, `test:integration`, `test:e2e`, `types:check`, `lint:*`, `fmt:*`, `knip:*`) are not declared in `package.json` — they are injected by local Nx plugins in `tools/nx-plugins/`. They are fully cached and can be discovered via `nx show project `. @@ -176,18 +176,36 @@ pnpm run check:all ## E2E Compatibility Test Suite -`apps/cli-e2e` implements a record-and-replay test harness for testing the TypeScript Legacy CLI (`ts-legacy`, the only shipped CLI shell) against real Supabase Management API responses without hitting staging on every run. It still shells out to the bundled Go binary for the handful of commands the TS port proxies (`db diff`, `db pull`, `db branch *`, `db remote *`, `gen keys`, `functions download`), so `apps/cli-go/` is built alongside the TS CLI for this suite, but the suite itself no longer compares Go and TS output — that go-target parity harness was retired once the legacy port and the CLI-1970 Go binary trim landed. +`apps/cli-e2e` implements a record-and-replay test harness for testing the TypeScript Legacy CLI (`ts-legacy`, the only shipped CLI shell) against real Supabase Management API responses without hitting staging on every run, plus a separate live suite for smoke-testing real remote behavior. It still shells out to the bundled Go binary for the handful of commands the TS port proxies (`db diff`, `db pull`, `db branch *`, `db remote *`, `gen keys`, `functions download`), so `apps/cli-go/` is built alongside the TS CLI for this suite, but the suite itself no longer compares Go and TS output — that go-target parity harness was retired once the legacy port and the CLI-1970 Go binary trim landed. ### Architecture -Fixtures are recorded by running `ts-legacy` against the real Supabase staging API and capturing the request/response pairs. Every other run replays those committed fixtures against the same CLI, so tests are fast and deterministic with no network access. +Replay fixtures are recorded by running `ts-legacy` against the real Supabase staging API and capturing request/response pairs. Replay runs serve those committed fixtures back to the same CLI, so compatibility tests are fast and deterministic with no network access; live mode is described below. -The harness works in two modes: +The harness works in three modes: -| Mode | When | What it does | -|------|------|-------------| +| Mode | When | What it does | +| -------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Replay** (default) | Every PR / local dev | Loads committed fixtures; serves recorded responses to the CLI subprocess. Fast and deterministic — no network access. | -| **Record** | `RECORD=true` | Proxies CLI traffic to staging and captures request/response pairs as fixture files. | +| **Record** | `RECORD=true` | Proxies CLI traffic to staging and captures request/response pairs as fixture files. | +| **Live** | `CLI_E2E_MODE=live` | Runs selected tests directly against a real staging project and Docker daemon. | + +### Live remote-project coverage + +The live suite runs the real `ts-legacy` CLI directly against the staging Management API — it does not use the replay server. Each serial run provisions one ephemeral project, waits for it to become `ACTIVE_HEALTHY`, uses the real Docker daemon for operations such as function deployment, and deletes the project during teardown (including after failures). Tests live in [`apps/cli-e2e/src/tests/live/**/*.live.e2e.test.ts`](apps/cli-e2e/src/tests/live), use the `testLive` fixtures (`run`, `invoke`, `workspace`, `projectRef`, `anonKey`, `functionsUrl`, `dbUrl`, `storageBucket`, and related values), and make **outcome-based assertions** on exit codes, output, HTTP status, and response bodies. + +Live coverage is smoke coverage, not an exhaustive command matrix. Grow it toward one representative golden-path test for every user-facing CLI command: choose one realistic, common success workflow and let integration tests cover validation, formatting, fallbacks, errors, and other matrices unless the real remote/runtime boundary is the behavior under test. See [ADR 0013](docs/adr/0013-live-e2e-bypasses-replay-server.md) for the architecture and [`apps/cli-e2e/.env.example`](apps/cli-e2e/.env.example) for environment settings. + +To run the live suite locally, build the Go binary first, then provide a staging token: + +```sh +cd apps/cli-go && go build -o /tmp/supabase-test-binary . && cd - +SUPABASE_GO_BINARY=/tmp/supabase-test-binary \ + SUPABASE_ACCESS_TOKEN=sbp_... \ + pnpm --filter @supabase/cli-e2e test:e2e:live +``` + +Live CI is manual or daily scheduled and is not PR-blocking; run it manually on a PR branch when you need pre-merge remote coverage. ### Running the tests @@ -303,13 +321,13 @@ supabase --version ### Troubleshooting -| Problem | Fix | -|---------|-----| -| `Error: Something is already running on port 4873` | Kill the leftover Verdaccio process (`lsof -ti:4873 \| xargs kill`) and retry | -| `go not found in PATH` (legacy only) | Install Go from https://go.dev/dl/ | -| `Error: Go CLI source not found` (legacy only) | Run `pnpm repos:install` to clone `apps/cli-go` | +| Problem | Fix | +| ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Error: Something is already running on port 4873` | Kill the leftover Verdaccio process (`lsof -ti:4873 \| xargs kill`) and retry | +| `go not found in PATH` (legacy only) | Install Go from https://go.dev/dl/ | +| `Error: Go CLI source not found` (legacy only) | Run `pnpm repos:install` to clone `apps/cli-go` | | `npm` / `pnpm` tries to fetch from `localhost:4873` when no registry is running | Stale global registry override left behind by an older version of `local-registry.ts` (the current script never modifies global config). Run `npm config delete registry` and `pnpm config delete registry`. Note that pnpm stores the override in its own global config (`~/Library/Preferences/pnpm/auth.ini` on macOS, `~/.config/pnpm/` on Linux), not `~/.npmrc` — check there if the delete command fails | -| `npx` resolves from npm instead of local | Pass `--registry http://localhost:4873` explicitly to `npx` / `npm install` | +| `npx` resolves from npm instead of local | Pass `--registry http://localhost:4873` explicitly to `npx` / `npm install` | ## Using Nx diff --git a/apps/cli-e2e/AGENTS.md b/apps/cli-e2e/AGENTS.md index b309111a1f..6a7a9d5497 100644 --- a/apps/cli-e2e/AGENTS.md +++ b/apps/cli-e2e/AGENTS.md @@ -146,12 +146,13 @@ The pre-recording cleanup deletes projects named `cli-e2e-test`, `my-project`, a `live` is a third mode (`CLI_E2E_MODE=live`) that, unlike replay/record, **does not use the replay server**. The harness is wired straight at the real Management API (`CLI_E2E_API_URL`) and the real Docker socket; tests assert on **real outcomes**. - Live tests are `src/tests/live/**/*.live.e2e.test.ts`, run only via `vitest.live.config.ts` (the default config excludes them). They `skipIf(!isLive)`, so they are inert on the replay suite. -- Global setup (`tests/live-setup.ts`) provisions **one ephemeral project per run** (`cli-e2e-live-{target}-{runId}-{short}`), waits for `ACTIVE_HEALTHY`, resolves the anon JWT, the IPv4 **session-pooler `dbUrl`** (for `--db-url` DB commands), the functions URL, and a seeded storage bucket, exposing them via `inject()`. It deletes the project on teardown (even on failure). Setup is intentionally **dumb** — no provisioning retry; the CI job re-runs the step on flake. +- Global setup (`tests/live-setup.ts`) provisions **one ephemeral project per run** (`cli-e2e-live-{target}-{runId}-{short}`), waits for `ACTIVE_HEALTHY`, resolves the anon JWT, the IPv4 **session-pooler `dbUrl`** (for `--db-url` DB commands), the functions URL, and a seeded storage bucket, exposing them via `inject()`. It deletes the project on teardown (even on failure). Setup is intentionally **dumb** — no provisioning retry; the CI step retries the live suite on flake. - Use `testLive` from `src/tests/live/live-context.ts`: `run(cmd)` (direct-wired CLI), `invoke(slug)` (direct HTTP call sending the **anon JWT** in both `Authorization: Bearer` and `apikey`), plus `workspace` (a fresh `supabase init` config so golden paths exercise a generated config), `projectRef`, `anonKey`, `functionsUrl`, `dbUrl`, `storageBucket`. The functions deploy tests call `seedFunctions(workspace.path)` to layer the `deploy-e2e-*` fixtures + their `[functions.*]` config onto the init'd config. - **Assertion style:** outcome-based — assert `exitCode`/`stdout` substrings and the function's HTTP status + JSON body. This is ID-agnostic, so **no normalization/snapshots by default**. If the CLI's own diagnostic output is ever the assertion target, add a scoped normalizer for that one test — do not make normalization the default. +- **Coverage policy:** live tests are smoke coverage, not an exhaustive matrix. Grow toward one representative golden-path test for every user-facing CLI command, choosing one common, realistic success workflow that exercises the real remote/runtime boundary. Keep validation, formatting, fallback, error, and matrix details in integration tests unless the remote boundary itself is what needs verification; check for an existing command's golden path before adding another live variant. - **Authoring/CI target is `ts-legacy`** — the only shipped CLI shell. It still shells out to the Go binary for the handful of commands the TS port proxies (`db diff`, `db pull`, `db branch *`, `db remote *`, `gen keys`, `functions download`), so `SUPABASE_GO_BINARY` must point at a built Go binary for those to resolve. - Retargeting to another env (e.g. `supabox`) is an env swap only: `CLI_E2E_TARGET_ENV` + `CLI_E2E_API_URL` + `CLI_E2E_PROJECT_HOST` + token. Tests assert on function output, not hostnames. -- **CI triggers** (`.github/workflows/live-e2e.yml`): `workflow_dispatch` (manual; the Actions branch picker selects the ref — no free-form `ref` input, so the staging token never reaches arbitrary code) and an hourly `schedule`. There is **no `pull_request` trigger** — run it manually on a PR branch for pre-merge coverage. The scheduled run exercises the `@beta` channel: `develop` is the default branch and the beta release source, so it builds from `develop` source and runs the `ts-legacy` job. A `gate` job skips the run unless the published `supabase@beta` version changed since the last green run (an `actions/cache` marker keyed on the version, written by `finalize` only after the job passes), so a staging project is spent only when there is a new beta to test. Because the marker is written only on a green run, a chronically-failing `@beta` keeps re-running every hour until it goes green or a newer beta supersedes it (intended — the failure stays visible). +- **CI triggers** (`.github/workflows/live-e2e.yml`): `workflow_dispatch` (manual; the Actions branch picker selects the ref — no free-form `ref` input, so the staging token never reaches arbitrary code) and a daily `schedule`. There is **no `pull_request` trigger** — run it manually on a PR branch for pre-merge coverage. The scheduled run exercises the `@beta` channel: `develop` is the default branch and the beta release source, so it builds from `develop` source and runs the `ts-legacy` job. A `gate` job skips the run unless the published `supabase@beta` version changed since the last green run (an `actions/cache` marker keyed on the version, written by `finalize` only after the job passes), so a staging project is spent only when there is a new beta to test. Because the marker is written only on a green run, a chronically-failing `@beta` keeps re-running daily until it goes green or a newer beta supersedes it (intended — the failure stays visible). The job retries the live suite up to three times per run; setup itself has no provisioning retry. ## Running the suite diff --git a/docs/adr/README.md b/docs/adr/README.md index c5ab90ea00..c0bc1cf7b5 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -41,23 +41,23 @@ When an ADR becomes outdated, mark it as `deprecated` or reference the supersedi ## ADR index -| ID | Title | Status | -| ---- | ------------------------------------------------------------------------------------------ | -------- | -| 0000 | [Use ADR to Record Decisions](0000-use-adr-to-record-decisions.md) | accepted | -| 0001 | [CLI DX Architecture: The 7 Pillars](0001-cli-dx-architecture-pillars.md) | accepted | -| 0002 | [CLI Product Metrics](0002-cli-product-metrics.md) | accepted | -| 0003 | [Self-Documenting CLI & Documentation Strategy](0003-self-documenting-cli.md) | accepted | -| 0004 | [CLI Design Goals & Development Workflows](0004-cli-design-goals-and-workflows.md) | accepted | -| 0005 | [OpenAPI-Driven Code Generation for CRUD Commands](0005-openapi-driven-code-generation.md) | proposed | -| 0006 | [Environment Management & Variable Resolution](0006-environment-management.md) | proposed | -| 0007 | [Real-time Progress in Command Handlers](0007-realtime-progress-in-command-handlers.md) | proposed | -| 0008 | [Authentication & Token Management](0008-authentication-and-token-management.md) | proposed | -| 0009 | [Configuration Schema & Validation](0009-configuration-schema-and-validation.md) | proposed | -| 0011 | [CLI Release & Distribution Strategy](0011-cli-release-and-distribution-strategy.md) | proposed | -| 0013 | [Live E2E Tests Bypass the Replay Server](0013-live-e2e-bypasses-replay-server.md) | proposed | -| 0015 | [Managed Stack Contract Fixtures](0015-managed-stack-contract-fixtures.md) | superseded | -| 0016 | [Legacy Port Completion and Go CLI Authority Scope](0016-legacy-port-completion-and-go-cli-authority-scope.md) | proposed | -| 0017 | [Simplified Managed Stack Architecture](0017-simplified-managed-stack-architecture.md) | accepted | +| ID | Title | Status | +| ---- | -------------------------------------------------------------------------------------------------------------- | ---------- | +| 0000 | [Use ADR to Record Decisions](0000-use-adr-to-record-decisions.md) | accepted | +| 0001 | [CLI DX Architecture: The 7 Pillars](0001-cli-dx-architecture-pillars.md) | accepted | +| 0002 | [CLI Product Metrics](0002-cli-product-metrics.md) | accepted | +| 0003 | [Self-Documenting CLI & Documentation Strategy](0003-self-documenting-cli.md) | accepted | +| 0004 | [CLI Design Goals & Development Workflows](0004-cli-design-goals-and-workflows.md) | accepted | +| 0005 | [OpenAPI-Driven Code Generation for CRUD Commands](0005-openapi-driven-code-generation.md) | proposed | +| 0006 | [Environment Management & Variable Resolution](0006-environment-management.md) | proposed | +| 0007 | [Real-time Progress in Command Handlers](0007-realtime-progress-in-command-handlers.md) | proposed | +| 0008 | [Authentication & Token Management](0008-authentication-and-token-management.md) | proposed | +| 0009 | [Configuration Schema & Validation](0009-configuration-schema-and-validation.md) | proposed | +| 0011 | [CLI Release & Distribution Strategy](0011-cli-release-and-distribution-strategy.md) | proposed | +| 0013 | [Live E2E Tests Bypass the Replay Server](0013-live-e2e-bypasses-replay-server.md) | accepted | +| 0015 | [Managed Stack Contract Fixtures](0015-managed-stack-contract-fixtures.md) | superseded | +| 0016 | [Legacy Port Completion and Go CLI Authority Scope](0016-legacy-port-completion-and-go-cli-authority-scope.md) | proposed | +| 0017 | [Simplified Managed Stack Architecture](0017-simplified-managed-stack-architecture.md) | accepted | ## Template From 5863068da0ccf1833bc302fe6d40b0c03f99f39f Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 17:02:02 +0200 Subject: [PATCH 02/23] test(cli): consolidate live e2e harness --- apps/cli/package.json | 1 + apps/cli/tests/helpers/live-context.ts | 178 ++++++++++ apps/cli/tests/helpers/live-env.ts | 68 +++- apps/cli/tests/helpers/live-project.ts | 318 ++++++++++++++++++ .../tests/helpers/live-provided-context.ts | 20 ++ apps/cli/tests/helpers/live.ts | 10 +- apps/cli/tests/live-global-setup.ts | 103 ++++-- 7 files changed, 662 insertions(+), 36 deletions(-) create mode 100644 apps/cli/tests/helpers/live-context.ts create mode 100644 apps/cli/tests/helpers/live-project.ts create mode 100644 apps/cli/tests/helpers/live-provided-context.ts diff --git a/apps/cli/package.json b/apps/cli/package.json index 96706fd08e..ad8894dbf6 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -34,6 +34,7 @@ "dev:legacy": "pnpm exec bun src/legacy/main.ts", "test": "nx run-many -t test:core test:e2e --projects=$npm_package_name", "test:core": "nx run-many -t test:unit test:integration --projects=$npm_package_name --coverage.enabled", + "test:live": "bun --bun vitest run --project live", "test:smoke": "bun run tests/smoke-test.ts", "check:all": "nx run-many -t types:check lint:check fmt:check knip:check --projects=$npm_package_name", "fix:all": "nx run-many -t lint:fix fmt:fix knip:fix --projects=$npm_package_name" diff --git a/apps/cli/tests/helpers/live-context.ts b/apps/cli/tests/helpers/live-context.ts new file mode 100644 index 0000000000..b2750e5cdd --- /dev/null +++ b/apps/cli/tests/helpers/live-context.ts @@ -0,0 +1,178 @@ +import { mkdtempSync, readFileSync, rmSync, cpSync, appendFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import { inject, test } from "vitest"; + +import { makeTempHome } from "./cli.ts"; +import { runSupabaseLive } from "./live.ts"; +import { isLiveConfigured, liveProjectRef } from "./live-env.ts"; + +type RunOptions = NonNullable[1]>; +type RunResult = Awaited>; +type TempHome = ReturnType; + +export interface LiveWorkspace { + readonly path: string; +} + +export interface InvokeResult { + readonly status: number; + readonly body: unknown; + readonly text: string; +} + +export interface LiveFixtures { + readonly projectRef: string; + readonly anonKey: string; + readonly functionsUrl: string; + readonly dbUrl: string; + readonly dbPassword: string; + readonly storageBucket: string; + readonly home: TempHome; + readonly workspace: LiveWorkspace; + readonly run: (args: string[], options?: RunOptions) => Promise; + readonly invoke: ( + slug: string, + options?: { anonKey?: string; payload?: unknown }, + ) => Promise; +} + +const base = test.extend({ + // eslint-disable-next-line no-empty-pattern + projectRef: async ({}, use) => { + await use(inject("projectRef")); + }, + + // eslint-disable-next-line no-empty-pattern + anonKey: async ({}, use) => { + await use(inject("anonKey")); + }, + + // eslint-disable-next-line no-empty-pattern + functionsUrl: async ({}, use) => { + await use(inject("functionsUrl")); + }, + + // eslint-disable-next-line no-empty-pattern + dbUrl: async ({}, use) => { + await use(inject("dbUrl")); + }, + + // eslint-disable-next-line no-empty-pattern + dbPassword: async ({}, use) => { + await use(inject("dbPassword")); + }, + + // eslint-disable-next-line no-empty-pattern + storageBucket: async ({}, use) => { + await use(inject("storageBucket")); + }, + + home: async (_fixtures, use) => { + const home = makeTempHome(); + try { + await use(home); + } finally { + home[Symbol.dispose](); + } + }, + + workspace: async ({ task, home }, use) => { + const suffix = task.name.replace(/[^a-z0-9-]+/giu, "-").slice(0, 40); + const directory = mkdtempSync(path.join(tmpdir(), `supabase-live-${suffix || "test"}-`)); + try { + const initialized = await runSupabaseLive(["init"], { + cwd: directory, + home: home.dir, + }); + if (initialized.exitCode !== 0) { + throw new Error( + `supabase init failed (exit ${initialized.exitCode})\n${initialized.stderr || initialized.stdout}`, + ); + } + await use({ path: directory }); + } finally { + rmSync(directory, { recursive: true, force: true }); + } + }, + + run: async ({ workspace, home }, use) => { + await use((args, options) => + runSupabaseLive(args, { + ...options, + cwd: options?.cwd ?? workspace.path, + // A test may use a subdirectory as cwd, but must share this HOME so + // setup/command/teardown observe the same link and config state. + home: home.dir, + }), + ); + }, + + invoke: async ({ functionsUrl, anonKey }, use) => { + await use(async (slug, options) => { + const key = options?.anonKey ?? anonKey; + const headers: Record = { "Content-Type": "application/json" }; + if (key.length > 0) { + headers["Authorization"] = `Bearer ${key}`; + headers["apikey"] = key; + } + const response = await fetch(`${functionsUrl}/${slug}`, { + method: "POST", + headers, + body: JSON.stringify(options?.payload ?? {}), + }); + const text = await response.text(); + let body: unknown; + try { + body = JSON.parse(text); + } catch { + body = text; + } + return { status: response.status, body, text }; + }); + }, +}); + +/** Live subprocess fixture. The global setup owns the shared platform project; + * this fixture owns only one isolated workspace and HOME per test. */ +export const testLive = base.skipIf(!isLiveConfigured()); + +/** Fixture for scenarios that require a project ref from managed or attached setup. */ +export const testLiveProject = base.skipIf(!isLiveConfigured() || !liveProjectRef()); + +/** Layer deploy-e2e function fixtures onto the generated workspace config. */ +export function seedFunctions( + workspacePath: string, + sourceDirectory: string, + configSnippet: string, +): void { + const supabaseDirectory = path.join(workspacePath, "supabase"); + cpSync(sourceDirectory, supabaseDirectory, { recursive: true }); + appendFileSync( + path.join(supabaseDirectory, "config.toml"), + `\n${readFileSync(configSnippet, "utf8")}`, + ); +} + +export function expectFunctionOk( + result: InvokeResult, + slug: string, + extra?: Record, +): void { + if (result.status !== 200) { + throw new Error( + `Expected function ${slug} to return 200, got ${result.status}: ${result.text}`, + ); + } + if (typeof result.body !== "object" || result.body === null) { + throw new Error(`Expected function ${slug} to return JSON: ${result.text}`); + } + const body = result.body as Record; + if (body.case !== slug || body.ok !== true) { + throw new Error(`Unexpected response from ${slug}: ${result.text}`); + } + for (const [key, value] of Object.entries(extra ?? {})) { + if (body[key] !== value) throw new Error(`Unexpected ${key} from ${slug}: ${result.text}`); + } +} diff --git a/apps/cli/tests/helpers/live-env.ts b/apps/cli/tests/helpers/live-env.ts index f34cd1734d..bd74c29369 100644 --- a/apps/cli/tests/helpers/live-env.ts +++ b/apps/cli/tests/helpers/live-env.ts @@ -12,8 +12,9 @@ * - `SUPABASE_PROFILE` — selects the API base URL; defaults to `supabase-local` * (→ `http://localhost:8080`, `project_host: supabase.red`). Note the cli does * NOT honor `SUPABASE_API_URL` (Go parity) — the profile is the override. - * - `SUPABASE_LIVE_API_URL` — base URL the readiness check probes; defaults to - * `http://localhost:8080`. + * - `SUPABASE_LIVE_API_URL` — base URL the readiness check probes; attached + * mode defaults to `http://localhost:8080`, managed staging defaults to + * `https://api.supabase.green`. * - `SUPABASE_LIVE_PROJECT_REF` — a provisioned project; gates project-scoped * suites (functions, branches, db, storage). * - `NODE_EXTRA_CA_CERTS` — trusts the supabox CA for `*.supabase.red` TLS; @@ -23,6 +24,33 @@ /** Default profile for the host runner: api_url → localhost:8080, project_host → supabase.red. */ export const LIVE_DEFAULT_PROFILE = "supabase-local"; +export type LiveMode = "attached" | "managed"; + +/** + * Selects how the run-scoped live environment is obtained. + * + * The default is deliberately attached: a token in a developer's environment + * must never cause the test suite to create or delete a project. Managed mode + * is an explicit opt-in for staging runs and is selected with + * `SUPABASE_LIVE_MODE=managed`. + */ +export function liveMode(): LiveMode { + const value = process.env["SUPABASE_LIVE_MODE"]; + if (value === undefined || value === "" || value === "attached") { + return "attached"; + } + if (value === "managed") { + return "managed"; + } + throw new Error( + `Unsupported SUPABASE_LIVE_MODE ${JSON.stringify(value)}; expected "attached" or "managed"`, + ); +} + +export function isManagedLive(): boolean { + return liveMode() === "managed"; +} + /** * Default subprocess exit timeout for live runs. `runSupabase` otherwise caps at * 60s, which would kill a slow-but-valid supabox call before the live tests' @@ -33,7 +61,41 @@ export const LIVE_EXIT_TIMEOUT_MS = 240_000; /** Management API base URL probed by the live readiness check. */ export function liveApiBaseUrl(): string { - return process.env["SUPABASE_LIVE_API_URL"] ?? "http://localhost:8080"; + return ( + process.env["SUPABASE_LIVE_API_URL"] ?? + (isManagedLive() ? "https://api.supabase.green" : "http://localhost:8080") + ); +} + +/** Profile used by the CLI subprocess. Attached runs preserve the historical + * local Supabox default; managed staging runs use the built-in staging profile + * unless the caller explicitly selected another profile. */ +export function liveProfile(): string { + return ( + process.env["SUPABASE_PROFILE"] ?? (isManagedLive() ? "supabase-staging" : LIVE_DEFAULT_PROFILE) + ); +} + +/** Host used for project-scoped endpoints such as Edge Functions and Storage. */ +export function liveProjectHost(): string { + if (process.env["SUPABASE_LIVE_PROJECT_HOST"] !== undefined) { + return process.env["SUPABASE_LIVE_PROJECT_HOST"]!; + } + + switch (liveProfile()) { + case "supabase": + return "supabase.co"; + case "supabase-staging": + case "supabase-local": + return "supabase.red"; + default: + return "supabase.red"; + } +} + +/** Keep an explicitly managed project alive for debugging instead of deleting it. */ +export function keepLiveProject(): boolean { + return process.env["SUPABASE_LIVE_KEEP_PROJECT"] === "1"; } /** diff --git a/apps/cli/tests/helpers/live-project.ts b/apps/cli/tests/helpers/live-project.ts new file mode 100644 index 0000000000..8a23c3f80e --- /dev/null +++ b/apps/cli/tests/helpers/live-project.ts @@ -0,0 +1,318 @@ +import { randomBytes, randomUUID } from "node:crypto"; + +import { runSupabase } from "./cli.ts"; +import { keepLiveProject, liveApiBaseUrl, liveProfile, liveProjectHost } from "./live-env.ts"; + +const PROJECT_REF_RE = /^[a-z]{20}$/u; +const TERMINAL_BAD_STATUSES = new Set(["INIT_FAILED", "RESTORE_FAILED", "REMOVED"]); +const DEFAULT_STORAGE_BUCKET = "supabase-cli-live-bucket"; + +export interface LiveProjectEnvironment { + readonly projectRef: string; + readonly anonKey: string; + readonly functionsUrl: string; + readonly dbUrl: string; + readonly dbPassword: string; + readonly storageBucket: string; + readonly projectName: string; + readonly owned: true; +} + +function accessToken(): string { + const token = process.env["SUPABASE_ACCESS_TOKEN"]; + if (token === undefined || token.length === 0) { + throw new Error( + "Managed live mode requires SUPABASE_ACCESS_TOKEN; refusing to provision with an empty token.", + ); + } + return token; +} + +function apiBaseUrl(): string { + return liveApiBaseUrl().replace(/\/+$/u, ""); +} + +function cliEnv(): Record { + return { SUPABASE_PROFILE: liveProfile() }; +} + +async function managementCommand(args: string[]) { + return runSupabase(args, { + entrypoint: "legacy", + env: cliEnv(), + exitTimeoutMs: 240_000, + }); +} + +function jsonError( + result: { exitCode: number; stderr: string; stdout: string }, + command: string, +): Error { + return new Error( + `${command} failed (exit ${result.exitCode}): ${result.stderr || result.stdout}`, + ); +} + +async function resolveOrgId(): Promise { + const override = process.env["SUPABASE_LIVE_ORG_ID"]; + if (override !== undefined && override.length > 0) return override; + + const result = await managementCommand(["orgs", "list", "--output", "json"]); + if (result.exitCode !== 0) throw jsonError(result, "orgs list"); + const rows = JSON.parse(result.stdout) as Array<{ id?: string }>; + const id = rows[0]?.id; + if (id === undefined || id.length === 0) { + throw new Error("No organizations found; cannot create the managed live project"); + } + return id; +} + +function generateDbPassword(): string { + return ( + process.env["SUPABASE_LIVE_DB_PASSWORD"] ?? + `supabase-cli-live-${randomBytes(12).toString("hex")}` + ); +} + +async function createProject(name: string, orgId: string, dbPassword: string): Promise { + const region = process.env["SUPABASE_LIVE_REGION"] ?? "us-east-1"; + const result = await managementCommand([ + "projects", + "create", + name, + "--org-id", + orgId, + "--db-password", + dbPassword, + "--region", + region, + "--output", + "json", + ]); + if (result.exitCode !== 0) throw jsonError(result, "projects create"); + + const project = JSON.parse(result.stdout) as { id?: string; ref?: string }; + const ref = project.ref ?? project.id; + if (ref === undefined || !PROJECT_REF_RE.test(ref)) { + throw new Error(`Unexpected project ref from projects create: ${result.stdout}`); + } + return ref; +} + +async function deleteProject(projectRef: string): Promise { + const result = await managementCommand(["projects", "delete", projectRef, "--yes"]); + if (result.exitCode !== 0) throw jsonError(result, `projects delete ${projectRef}`); +} + +async function waitForProjectReady(projectRef: string, timeoutMs = 300_000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + const response = await fetch(`${apiBaseUrl()}/v1/projects/${projectRef}`, { + headers: { Authorization: `Bearer ${accessToken()}` }, + }); + if (response.ok) { + const project = (await response.json()) as { status?: string }; + if (project.status === "ACTIVE_HEALTHY") return; + if (project.status !== undefined && TERMINAL_BAD_STATUSES.has(project.status)) { + throw new Error( + `Project ${projectRef} entered terminal status ${project.status} during provisioning`, + ); + } + } else { + await response.body?.cancel(); + } + const remaining = deadline - Date.now(); + if (remaining <= 0) break; + await new Promise((resolve) => setTimeout(resolve, Math.min(5_000, remaining))); + } + throw new Error(`Project ${projectRef} did not become ACTIVE_HEALTHY within ${timeoutMs}ms`); +} + +interface ApiKey { + readonly name?: string; + readonly api_key?: string; +} + +/** + * API keys are eventually consistent after project readiness. Polling is an + * intrinsic part of provisioning because the Management API exposes no + * readiness event; bound it by wall-clock time rather than attempt count. + */ +async function getProjectKeys(projectRef: string, timeoutMs = 180_000): Promise { + const deadline = Date.now() + timeoutMs; + let lastStatus = "unknown"; + while (Date.now() < deadline) { + const response = await fetch(`${apiBaseUrl()}/v1/projects/${projectRef}/api-keys`, { + headers: { Authorization: `Bearer ${accessToken()}` }, + }); + lastStatus = String(response.status); + if (response.ok) { + const keys = (await response.json()) as ApiKey[]; + if (keys.length > 0) return keys; + } else { + await response.body?.cancel(); + } + const remaining = deadline - Date.now(); + if (remaining <= 0) break; + await new Promise((resolve) => setTimeout(resolve, Math.min(10_000, remaining))); + } + throw new Error( + `Failed to resolve API keys for ${projectRef} within ${timeoutMs}ms (${lastStatus})`, + ); +} + +async function getAnonKey(projectRef: string): Promise { + const keys = await getProjectKeys(projectRef); + const anon = keys.find((key) => key.name === "anon" && key.api_key)?.api_key; + if (anon !== undefined) return anon; + throw new Error(`Project ${projectRef} returned no legacy anon JWT for function invokes`); +} + +async function getServiceRoleKey(projectRef: string): Promise { + const keys = await getProjectKeys(projectRef); + const serviceRole = + keys.find((key) => key.name === "service_role" && key.api_key)?.api_key ?? + keys.find((key) => key.api_key?.startsWith("sb_secret_"))?.api_key; + if (serviceRole !== undefined) return serviceRole; + throw new Error(`Project ${projectRef} returned no service-role key`); +} + +interface PoolerConfig { + readonly database_type?: string; + readonly connection_string?: string; +} + +async function getPoolerSessionUrl( + projectRef: string, + password: string, + timeoutMs = 180_000, +): Promise { + const deadline = Date.now() + timeoutMs; + let lastStatus = "unknown"; + while (Date.now() < deadline) { + const response = await fetch( + `${apiBaseUrl()}/v1/projects/${projectRef}/config/database/pooler`, + { headers: { Authorization: `Bearer ${accessToken()}` } }, + ); + lastStatus = String(response.status); + if (response.ok) { + const raw = (await response.json()) as PoolerConfig | PoolerConfig[]; + const configs = Array.isArray(raw) ? raw : [raw]; + const primary = configs.find((config) => config.database_type === "PRIMARY") ?? configs[0]; + if (primary?.connection_string !== undefined) { + const url = new URL(primary.connection_string); + url.password = password; + url.port = "5432"; + if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); + return url.toString(); + } + } else { + await response.body?.cancel(); + } + const remaining = deadline - Date.now(); + if (remaining <= 0) break; + await new Promise((resolve) => setTimeout(resolve, Math.min(10_000, remaining))); + } + throw new Error( + `Failed to resolve pooler config for ${projectRef} within ${timeoutMs}ms (${lastStatus})`, + ); +} + +async function createStorageBucket( + projectRef: string, + serviceRoleKey: string, + bucket: string, +): Promise { + const response = await fetch(`https://${projectRef}.${liveProjectHost()}/storage/v1/bucket`, { + method: "POST", + headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, + body: JSON.stringify({ id: bucket, name: bucket, public: false }), + }); + if (!response.ok && response.status !== 409) { + throw new Error( + `Failed to create storage bucket ${bucket}: ${response.status} ${await response.text()}`, + ); + } +} + +/** Provision one unique staging project and all values needed by project live tests. */ +export async function provisionManagedLiveProject(): Promise { + const orgId = await resolveOrgId(); + const baseName = process.env["SUPABASE_LIVE_PROJECT_NAME"] ?? "supabase-cli-live"; + const runId = process.env["GITHUB_RUN_ID"] ?? process.env["CI_JOB_ID"] ?? String(Date.now()); + const projectName = `${baseName}-${runId}-${randomUUID().slice(0, 8)}`; + const dbPassword = generateDbPassword(); + const projectRef = await createProject(projectName, orgId, dbPassword); + + try { + await waitForProjectReady(projectRef); + const anonKey = await getAnonKey(projectRef); + const serviceRoleKey = await getServiceRoleKey(projectRef); + const dbUrl = await getPoolerSessionUrl(projectRef, dbPassword); + const storageBucket = process.env["SUPABASE_LIVE_STORAGE_BUCKET"] ?? DEFAULT_STORAGE_BUCKET; + await createStorageBucket(projectRef, serviceRoleKey, storageBucket); + + return { + projectRef, + anonKey, + functionsUrl: `https://${projectRef}.${liveProjectHost()}/functions/v1`, + dbUrl, + dbPassword, + storageBucket, + projectName, + owned: true, + }; + } catch (error) { + if (!keepLiveProject()) { + await deleteProject(projectRef).catch((cleanupError) => { + console.error(`Failed to delete managed live project ${projectRef}:`, cleanupError); + }); + } + throw error; + } +} + +export async function deleteManagedLiveProject(projectRef: string): Promise { + if (keepLiveProject()) { + console.log(`SUPABASE_LIVE_KEEP_PROJECT=1 — leaving managed live project ${projectRef} alive`); + return; + } + await deleteProject(projectRef); +} + +/** Read-only API values supplied by an attached Supabox/local harness. */ +export function attachedLiveValues( + projectRef: string | undefined, +): Omit { + const anonKey = process.env["SUPABASE_LIVE_ANON_KEY"] ?? ""; + const dbUrl = process.env["SUPABASE_LIVE_DB_URL"] ?? ""; + const dbPassword = process.env["SUPABASE_LIVE_DB_PASSWORD"] ?? ""; + const storageBucket = process.env["SUPABASE_LIVE_STORAGE_BUCKET"] ?? DEFAULT_STORAGE_BUCKET; + return { + projectRef: projectRef ?? "", + anonKey, + functionsUrl: + process.env["SUPABASE_LIVE_FUNCTIONS_URL"] ?? + (projectRef === undefined ? "" : `https://${projectRef}.${liveProjectHost()}/functions/v1`), + dbUrl, + dbPassword, + storageBucket, + }; +} + +/** Resolve the attached environment without ever creating or deleting a project. */ +export async function assertAttachedLiveReachable(): Promise { + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), 30_000); + try { + await fetch(`${apiBaseUrl()}/v1/organizations`, { signal: controller.signal }); + } catch (error) { + const reason = error instanceof Error ? error.message : String(error); + throw new Error( + `Live platform is not reachable at ${apiBaseUrl()}/v1/organizations: ${reason}.\n` + + "Ensure the Supabox/local API platform is running and reachable.", + ); + } finally { + clearTimeout(timeout); + } +} diff --git a/apps/cli/tests/helpers/live-provided-context.ts b/apps/cli/tests/helpers/live-provided-context.ts new file mode 100644 index 0000000000..9a177d8d48 --- /dev/null +++ b/apps/cli/tests/helpers/live-provided-context.ts @@ -0,0 +1,20 @@ +// Vitest evaluates global setup separately from test modules. Keep the +// `ProvidedContext` augmentation in a side-effect-free module so global setup +// can import it without loading Vitest's test APIs. +export {}; + +declare module "vitest" { + export interface ProvidedContext { + /** Environment selected by the live global setup. */ + liveMode: "attached" | "managed"; + /** Shared project wiring. Empty strings mean the attached harness did not + * provide a project-scoped value and the corresponding suite should use a + * project/data-plane gate. */ + projectRef: string; + anonKey: string; + functionsUrl: string; + dbUrl: string; + dbPassword: string; + storageBucket: string; + } +} diff --git a/apps/cli/tests/helpers/live.ts b/apps/cli/tests/helpers/live.ts index 8123bc96dd..3ebb5118fe 100644 --- a/apps/cli/tests/helpers/live.ts +++ b/apps/cli/tests/helpers/live.ts @@ -4,9 +4,9 @@ import { describe } from "vitest"; import { runSupabase } from "./cli.ts"; import { isLiveConfigured, - LIVE_DEFAULT_PROFILE, LIVE_EXIT_TIMEOUT_MS, liveProjectDataPlaneReady, + liveProfile, liveProjectRef, } from "./live-env.ts"; @@ -28,8 +28,14 @@ export { LIVE_DEFAULT_PROFILE, LIVE_EXIT_TIMEOUT_MS, liveApiBaseUrl, + isManagedLive, + keepLiveProject, liveProjectDataPlaneReady, + liveMode, + liveProfile, + liveProjectHost, liveProjectRef, + type LiveMode, requireLiveProjectRef, } from "./live-env.ts"; @@ -93,7 +99,7 @@ export function runSupabaseLive( ...options, exitTimeoutMs: options?.exitTimeoutMs ?? LIVE_EXIT_TIMEOUT_MS, env: { - SUPABASE_PROFILE: process.env["SUPABASE_PROFILE"] ?? LIVE_DEFAULT_PROFILE, + SUPABASE_PROFILE: liveProfile(), ...options?.env, }, }); diff --git a/apps/cli/tests/live-global-setup.ts b/apps/cli/tests/live-global-setup.ts index d7584e757a..97d4b5ffbb 100644 --- a/apps/cli/tests/live-global-setup.ts +++ b/apps/cli/tests/live-global-setup.ts @@ -1,39 +1,80 @@ -// Import from the Vitest-free env module — globalSetup runs in a context where -// importing Vitest test APIs (which `helpers/live.ts` pulls in) is not valid. -import { isLiveConfigured, liveApiBaseUrl } from "./helpers/live-env.ts"; +import type { ProvidedContext } from "vitest"; + +import "./helpers/live-provided-context.ts"; +import { + assertAttachedLiveReachable, + attachedLiveValues, + deleteManagedLiveProject, + provisionManagedLiveProject, +} from "./helpers/live-project.ts"; +import { isLiveConfigured, liveMode, liveProfile, liveProjectRef } from "./helpers/live-env.ts"; + +type LiveSetupContext = { + provide: (key: K, value: ProvidedContext[K]) => void; +}; + +function provideEnvironment( + provide: LiveSetupContext["provide"], + values: { + projectRef: string; + anonKey: string; + functionsUrl: string; + dbUrl: string; + dbPassword: string; + storageBucket: string; + }, + mode: "attached" | "managed", +): void { + provide("liveMode", mode); + provide("projectRef", values.projectRef); + provide("anonKey", values.anonKey); + provide("functionsUrl", values.functionsUrl); + provide("dbUrl", values.dbUrl); + provide("dbPassword", values.dbPassword); + provide("storageBucket", values.storageBucket); +} /** - * Global setup for the `live` Vitest project. When the live environment is not - * configured the suite is skipped (via `describeLive`) and this is a no-op. + * Own one live environment for the entire Vitest run. * - * When it IS configured (the cli-e2e-ci runner sets `SUPABASE_ACCESS_TOKEN`), - * fail fast with a clear message if the platform is unreachable, so a - * misconfigured stack surfaces as a setup error rather than dozens of opaque - * per-test timeouts. + * Attached mode is the historical Supabox/local contract: it only probes the + * configured platform and never mutates or deletes a project. Managed mode is + * an explicit staging opt-in (`SUPABASE_LIVE_MODE=managed`) and creates one + * uniquely named project, provides its wiring to workers, then deletes exactly + * that project during global teardown unless `SUPABASE_LIVE_KEEP_PROJECT=1`. */ -export async function setup(): Promise { - if (!isLiveConfigured()) { - return; +export async function setup({ provide }: LiveSetupContext): Promise<() => Promise> { + const mode = liveMode(); + + if (mode === "managed") { + if (!isLiveConfigured()) { + throw new Error( + "SUPABASE_LIVE_MODE=managed requires SUPABASE_ACCESS_TOKEN; refusing to provision with an ambient empty token.", + ); + } + + // Make the managed profile/ref visible to existing collection-time gates in + // addition to providing the values to fixtures through Vitest's context. + if (process.env["SUPABASE_PROFILE"] === undefined) { + process.env["SUPABASE_PROFILE"] = liveProfile(); + } + const environment = await provisionManagedLiveProject(); + process.env["SUPABASE_LIVE_PROJECT_REF"] = environment.projectRef; + provideEnvironment(provide, environment, mode); + + return async () => { + await deleteManagedLiveProject(environment.projectRef); + }; } - // Reachability gate only. Any HTTP response — including 401/404 — proves the - // Management API is up and routing, which is all this probe needs to assert. - // supabox's mgmt-api requires auth on every route and exposes no public health - // endpoint (`/v1/health` 404s; an unauthenticated request is rejected by the - // auth middleware with 401), so we deliberately do NOT require a 2xx here. - // Functional and auth coverage is the live tests' job (e.g. `orgs list`). - const probeUrl = `${liveApiBaseUrl()}/v1/organizations`; - const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), 30_000); - try { - await fetch(probeUrl, { signal: controller.signal }); - } catch (error) { - const reason = error instanceof Error ? error.message : String(error); - throw new Error( - `Live platform is not reachable at ${probeUrl}: ${reason}.\n` + - "Ensure the supabox stack is up and the host can reach mgmt-api (see cli-e2e-ci).", - ); - } finally { - clearTimeout(timeout); + if (!isLiveConfigured()) { + return async () => {}; } + + await assertAttachedLiveReachable(); + const values = attachedLiveValues(liveProjectRef()); + provideEnvironment(provide, values, mode); + return async () => {}; } + +export default setup; From 37ea3e8eb9ba1c40e566804e573821bf613b7c26 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 17:09:07 +0200 Subject: [PATCH 03/23] fix(test): bound live provisioning requests --- apps/cli/tests/helpers/live-project.ts | 153 +++++++++++++++++-------- 1 file changed, 108 insertions(+), 45 deletions(-) diff --git a/apps/cli/tests/helpers/live-project.ts b/apps/cli/tests/helpers/live-project.ts index 8a23c3f80e..dbd6720e67 100644 --- a/apps/cli/tests/helpers/live-project.ts +++ b/apps/cli/tests/helpers/live-project.ts @@ -32,6 +32,38 @@ function apiBaseUrl(): string { return liveApiBaseUrl().replace(/\/+$/u, ""); } +/** + * Keep both connection establishment and response-body consumption inside the + * caller's wall-clock deadline. A fetch can resolve its headers while a body + * read remains hung, so the timer intentionally surrounds `consume` too. + */ +async function fetchWithinDeadline( + operation: string, + url: string, + init: RequestInit, + deadline: number, + consume: (response: Response) => Promise, +): Promise { + const remaining = deadline - Date.now(); + if (remaining <= 0) { + throw new Error(`${operation} exceeded its provisioning deadline before starting: ${url}`); + } + + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), remaining); + try { + const response = await fetch(url, { ...init, signal: controller.signal }); + return await consume(response); + } catch (error) { + if (controller.signal.aborted) { + throw new Error(`${operation} exceeded its provisioning deadline: ${url}`, { cause: error }); + } + throw error; + } finally { + clearTimeout(timeout); + } +} + function cliEnv(): Record { return { SUPABASE_PROFILE: liveProfile() }; } @@ -107,19 +139,32 @@ async function deleteProject(projectRef: string): Promise { async function waitForProjectReady(projectRef: string, timeoutMs = 300_000): Promise { const deadline = Date.now() + timeoutMs; while (Date.now() < deadline) { - const response = await fetch(`${apiBaseUrl()}/v1/projects/${projectRef}`, { - headers: { Authorization: `Bearer ${accessToken()}` }, - }); - if (response.ok) { - const project = (await response.json()) as { status?: string }; - if (project.status === "ACTIVE_HEALTHY") return; - if (project.status !== undefined && TERMINAL_BAD_STATUSES.has(project.status)) { + const result = await fetchWithinDeadline( + `Waiting for project ${projectRef} readiness`, + `${apiBaseUrl()}/v1/projects/${projectRef}`, + { headers: { Authorization: `Bearer ${accessToken()}` } }, + deadline, + async (response) => { + if (response.ok) { + return { + ok: true, + project: (await response.json()) as { status?: string }, + }; + } + await response.body?.cancel(); + return { ok: false, project: undefined }; + }, + ); + if (result.ok) { + if (result.project?.status === "ACTIVE_HEALTHY") return; + if ( + result.project?.status !== undefined && + TERMINAL_BAD_STATUSES.has(result.project.status) + ) { throw new Error( - `Project ${projectRef} entered terminal status ${project.status} during provisioning`, + `Project ${projectRef} entered terminal status ${result.project.status} during provisioning`, ); } - } else { - await response.body?.cancel(); } const remaining = deadline - Date.now(); if (remaining <= 0) break; @@ -142,16 +187,21 @@ async function getProjectKeys(projectRef: string, timeoutMs = 180_000): Promise< const deadline = Date.now() + timeoutMs; let lastStatus = "unknown"; while (Date.now() < deadline) { - const response = await fetch(`${apiBaseUrl()}/v1/projects/${projectRef}/api-keys`, { - headers: { Authorization: `Bearer ${accessToken()}` }, - }); - lastStatus = String(response.status); - if (response.ok) { - const keys = (await response.json()) as ApiKey[]; - if (keys.length > 0) return keys; - } else { - await response.body?.cancel(); - } + const result = await fetchWithinDeadline( + `Resolving API keys for project ${projectRef}`, + `${apiBaseUrl()}/v1/projects/${projectRef}/api-keys`, + { headers: { Authorization: `Bearer ${accessToken()}` } }, + deadline, + async (response) => { + if (response.ok) { + return { status: response.status, keys: (await response.json()) as ApiKey[] }; + } + await response.body?.cancel(); + return { status: response.status, keys: undefined }; + }, + ); + lastStatus = String(result.status); + if (result.keys !== undefined && result.keys.length > 0) return result.keys; const remaining = deadline - Date.now(); if (remaining <= 0) break; await new Promise((resolve) => setTimeout(resolve, Math.min(10_000, remaining))); @@ -190,24 +240,29 @@ async function getPoolerSessionUrl( const deadline = Date.now() + timeoutMs; let lastStatus = "unknown"; while (Date.now() < deadline) { - const response = await fetch( + const result = await fetchWithinDeadline( + `Resolving pooler config for project ${projectRef}`, `${apiBaseUrl()}/v1/projects/${projectRef}/config/database/pooler`, { headers: { Authorization: `Bearer ${accessToken()}` } }, + deadline, + async (response) => { + if (!response.ok) { + await response.body?.cancel(); + return { status: response.status, connectionString: undefined }; + } + const raw = (await response.json()) as PoolerConfig | PoolerConfig[]; + const configs = Array.isArray(raw) ? raw : [raw]; + const primary = configs.find((config) => config.database_type === "PRIMARY") ?? configs[0]; + return { status: response.status, connectionString: primary?.connection_string }; + }, ); - lastStatus = String(response.status); - if (response.ok) { - const raw = (await response.json()) as PoolerConfig | PoolerConfig[]; - const configs = Array.isArray(raw) ? raw : [raw]; - const primary = configs.find((config) => config.database_type === "PRIMARY") ?? configs[0]; - if (primary?.connection_string !== undefined) { - const url = new URL(primary.connection_string); - url.password = password; - url.port = "5432"; - if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); - return url.toString(); - } - } else { - await response.body?.cancel(); + lastStatus = String(result.status); + if (result.connectionString !== undefined) { + const url = new URL(result.connectionString); + url.password = password; + url.port = "5432"; + if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); + return url.toString(); } const remaining = deadline - Date.now(); if (remaining <= 0) break; @@ -223,16 +278,24 @@ async function createStorageBucket( serviceRoleKey: string, bucket: string, ): Promise { - const response = await fetch(`https://${projectRef}.${liveProjectHost()}/storage/v1/bucket`, { - method: "POST", - headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, - body: JSON.stringify({ id: bucket, name: bucket, public: false }), - }); - if (!response.ok && response.status !== 409) { - throw new Error( - `Failed to create storage bucket ${bucket}: ${response.status} ${await response.text()}`, - ); - } + const url = `https://${projectRef}.${liveProjectHost()}/storage/v1/bucket`; + await fetchWithinDeadline( + `Creating storage bucket ${bucket}`, + url, + { + method: "POST", + headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, + body: JSON.stringify({ id: bucket, name: bucket, public: false }), + }, + Date.now() + 60_000, + async (response) => { + if (!response.ok && response.status !== 409) { + throw new Error( + `Failed to create storage bucket ${bucket}: ${response.status} ${await response.text()}`, + ); + } + }, + ); } /** Provision one unique staging project and all values needed by project live tests. */ From f121882093df25c57e817f2d18b7be7cecc789e8 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 17:32:03 +0200 Subject: [PATCH 04/23] test(cli): collocate live command coverage --- .../branches/create/create.live.test.ts | 37 ++++++++ .../branches/delete/delete.live.test.ts | 29 ++++++ .../commands/branches/list/list.live.test.ts | 48 +++++----- .../legacy/commands/db/dump/dump.live.test.ts | 52 ++--------- .../legacy/commands/db/pull/pull.live.test.ts | 93 ++++++------------- .../legacy/commands/db/push/push.live.test.ts | 31 +++++++ .../commands/db/reset/reset.live.test.ts | 31 +++++++ .../commands/db/start/start.live.test.ts | 22 +++++ .../functions/delete/delete.live.test.ts | 35 +++++++ .../functions/deploy/deploy.live.test.ts | 42 +++++++++ .../commands/gen/types/types.live.test.ts | 9 ++ .../inspect/db/db-stats/db-stats.live.test.ts | 9 ++ .../legacy/commands/link/link.live.test.ts | 15 +++ .../commands/migration/list/list.live.test.ts | 60 +++--------- .../projects/api-keys/api-keys.live.test.ts | 19 ++++ .../commands/projects/list/list.live.test.ts | 44 +++------ .../commands/secrets/list/list.live.test.ts | 32 +++++++ .../commands/secrets/set/set.live.test.ts | 26 ++++++ .../commands/secrets/unset/unset.live.test.ts | 29 ++++++ .../commands/storage/cp/cp.live.test.ts | 42 +++++++++ .../commands/storage/ls/ls.live.test.ts | 43 +++++++++ .../commands/storage/rm/rm.live.test.ts | 44 +++++++++ apps/cli/tests/helpers/live-context.ts | 45 ++++++++- 23 files changed, 624 insertions(+), 213 deletions(-) create mode 100644 apps/cli/src/legacy/commands/branches/create/create.live.test.ts create mode 100644 apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts create mode 100644 apps/cli/src/legacy/commands/db/push/push.live.test.ts create mode 100644 apps/cli/src/legacy/commands/db/reset/reset.live.test.ts create mode 100644 apps/cli/src/legacy/commands/db/start/start.live.test.ts create mode 100644 apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts create mode 100644 apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts create mode 100644 apps/cli/src/legacy/commands/gen/types/types.live.test.ts create mode 100644 apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts create mode 100644 apps/cli/src/legacy/commands/link/link.live.test.ts create mode 100644 apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts create mode 100644 apps/cli/src/legacy/commands/secrets/list/list.live.test.ts create mode 100644 apps/cli/src/legacy/commands/secrets/set/set.live.test.ts create mode 100644 apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts create mode 100644 apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts create mode 100644 apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts create mode 100644 apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts diff --git a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts new file mode 100644 index 0000000000..18a6c602ba --- /dev/null +++ b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts @@ -0,0 +1,37 @@ +import { randomUUID } from "node:crypto"; +import { expect } from "vitest"; + +import { testLiveProject } from "../../../../../tests/helpers/live-context.ts"; + +async function deleteBranch( + run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + name: string, + projectRef: string, +): Promise { + const deleted = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); + if (deleted.exitCode !== 0) { + throw new Error( + `branches delete cleanup failed (exit ${deleted.exitCode})\n${deleted.stdout}\n${deleted.stderr}`, + ); + } +} + +testLiveProject("creates a preview branch", async ({ run, projectRef, skip }) => { + const name = `cli-e2e-create-${randomUUID().slice(0, 8)}`; + const result = await run(["branches", "create", name, "--project-ref", projectRef]); + + // Branching is plan-gated. A free Supabox org lacks the optional fixture for + // this golden path, so skip only the recognized plan-gate response. + if (result.exitCode !== 0) { + if (/paid plan|upgrade|not.*support/i.test(`${result.stdout}\n${result.stderr}`)) { + skip("Preview branches require a paid plan"); + } + expect(result.exitCode, result.stderr).toBe(0); + } + + try { + expect(result.stdout).toContain("Created preview branch"); + } finally { + await deleteBranch(run, name, projectRef); + } +}); diff --git a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts new file mode 100644 index 0000000000..876b5f3bb8 --- /dev/null +++ b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts @@ -0,0 +1,29 @@ +import { randomUUID } from "node:crypto"; +import { expect } from "vitest"; + +import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; + +testLiveProject("deletes a preview branch", async ({ run, projectRef, skip }) => { + const name = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; + const created = await run(["branches", "create", name, "--project-ref", projectRef]); + if ( + created.exitCode !== 0 && + /paid plan|upgrade|not.*support/i.test(`${created.stdout}\n${created.stderr}`) + ) { + skip("Preview branches require a paid plan"); + } + requireLiveSuccess(created, "branches create"); + + let deleted = false; + try { + const removed = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); + expect(removed.exitCode, removed.stderr).toBe(0); + expect(removed.stdout).toContain("Deleted preview branch"); + deleted = true; + } finally { + if (!deleted) { + const cleanup = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); + requireLiveSuccess(cleanup, "branches delete cleanup"); + } + } +}); diff --git a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts index 65422ad51b..85e647635a 100644 --- a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts @@ -1,30 +1,26 @@ -import { expect, test } from "vitest"; +import { randomUUID } from "node:crypto"; +import { expect } from "vitest"; -import { - describeLiveProject, - requireLiveProjectRef, - runSupabaseLive, -} from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; -const LIVE_TIMEOUT_MS = 120_000; +testLiveProject("lists a preview branch for the project", async ({ run, projectRef, skip }) => { + const name = `cli-e2e-list-${randomUUID().slice(0, 8)}`; + const created = await run(["branches", "create", name, "--project-ref", projectRef]); + if ( + created.exitCode !== 0 && + /paid plan|upgrade|not.*support/i.test(`${created.stdout}\n${created.stderr}`) + ) { + skip("Preview branches require a paid plan"); + } + requireLiveSuccess(created, "branches create setup"); -// Project-scoped read-only scenario. Skipped unless SUPABASE_LIVE_PROJECT_REF is -// set — i.e. a project has been provisioned on the stack (the cli-e2e-ci runner -// does this; a control-plane-only stack, like local macOS, skips it). -// -// Entry point for the branching lifecycle tracked in CLI-1834 -// (create / switch / delete) — extend here once a provisioned project is -// available on the full stack. -describeLiveProject("supabase branches list (live)", () => { - test("lists branches for the project", { timeout: LIVE_TIMEOUT_MS }, async () => { - const ref = requireLiveProjectRef(); - const { exitCode, stdout, stderr } = await runSupabaseLive([ - "branches", - "list", - "--project-ref", - ref, - ]); - expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); - expect(exitCode).toBe(0); - }); + try { + const result = await run(["branches", "list", "--output", "json", "--project-ref", projectRef]); + expect(result.exitCode, result.stderr).toBe(0); + const branches = JSON.parse(result.stdout) as Array<{ name?: string }>; + expect(branches.map((branch) => branch.name)).toContain(name); + } finally { + const deleted = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); + requireLiveSuccess(deleted, "branches delete cleanup"); + } }); diff --git a/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts b/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts index 8ebaaa8d63..d99a6d75a1 100644 --- a/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts +++ b/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts @@ -1,48 +1,12 @@ -import { existsSync, mkdtempSync, rmSync } from "node:fs"; -import { tmpdir } from "node:os"; +import { existsSync } from "node:fs"; import { join } from "node:path"; -import { expect, test } from "vitest"; +import { expect } from "vitest"; -import { - describeLiveDataPlane, - requireLiveProjectRef, - runSupabaseLive, -} from "../../../../../tests/helpers/live.ts"; +import { testLiveDataPlane } from "../../../../../tests/helpers/live-context.ts"; -const LIVE_TIMEOUT_MS = 300_000; - -// A fresh, isolated temp workdir so the CLI writes the dump there and never touches -// the repo tree. The provisioned project ref is supplied to `--linked` via the -// `SUPABASE_PROJECT_ID` env var — that is the `--linked` resolver chain (flag → -// `SUPABASE_PROJECT_ID` → `supabase/.temp/project-ref`); `config.toml`'s -// `project_id` is NOT consulted for `--linked`. -function tempWorkdir(): string { - return mkdtempSync(join(tmpdir(), "sb-db-dump-live-")); -} - -// Data-plane: needs a provisioned project whose database is routable (the -// cli-e2e-ci Linux runner). `describeLiveDataPlane` runs this only when the project -// instance is ACTIVE_HEALTHY, so a control-plane-only stack (ref set but the DB -// unreachable, e.g. local macOS or the current cli-e2e-ci control-plane case) is -// skipped rather than timing out on pg_dump. -describeLiveDataPlane("supabase db dump (live)", () => { - test("dumps the linked project's schema to a file", { timeout: LIVE_TIMEOUT_MS }, async () => { - const ref = requireLiveProjectRef(); - const dir = tempWorkdir(); - try { - const outFile = join(dir, "schema.sql"); - const { exitCode, stdout, stderr } = await runSupabaseLive( - ["db", "dump", "--linked", "-f", outFile], - { cwd: dir, env: { SUPABASE_PROJECT_ID: ref }, exitTimeoutMs: LIVE_TIMEOUT_MS - 20_000 }, - ); - expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); - expect(exitCode).toBe(0); - // The native pg_dump container (shared `legacyStreamPgDump`) opened + wrote - // the dump file. A fresh project's public schema may be near-empty, so assert - // the file was created rather than its size. - expect(existsSync(outFile)).toBe(true); - } finally { - rmSync(dir, { recursive: true, force: true }); - } - }); +testLiveDataPlane("dumps the remote schema to a file", async ({ run, dbUrl, workspace }) => { + const outFile = join(workspace.path, "schema.sql"); + const result = await run(["db", "dump", "--db-url", dbUrl, "-f", outFile]); + expect(result.exitCode, result.stderr).toBe(0); + expect(existsSync(outFile)).toBe(true); }); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts index 320d274d60..133d32f219 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts @@ -1,69 +1,36 @@ -import { existsSync, mkdtempSync, readdirSync, rmSync } from "node:fs"; -import { tmpdir } from "node:os"; +import { mkdir, unlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; -import { expect, test } from "vitest"; +import { expect } from "vitest"; import { - describeLiveDataPlane, - requireLiveProjectRef, - runSupabaseLive, -} from "../../../../../tests/helpers/live.ts"; + requireLiveSuccess, + testLiveDataPlane, +} from "../../../../../tests/helpers/live-context.ts"; -const LIVE_TIMEOUT_MS = 300_000; +testLiveDataPlane( + "pulls the remote schema after a local migration is applied", + async ({ run, dbUrl, workspace }) => { + const version = `${Date.now()}${Math.floor(Math.random() * 10_000) + .toString() + .padStart(4, "0")}`; + const migrations = join(workspace.path, "supabase", "migrations"); + await mkdir(migrations, { recursive: true }); + const migrationFile = join(migrations, `${version}_e2e_pull.sql`); + await writeFile(migrationFile, `create table if not exists e2e_pull_${version} (id int);\n`); -// A fresh, isolated temp workdir so the CLI writes migrations there and never -// touches the repo tree. The provisioned project ref is supplied to `--linked` via -// the `SUPABASE_PROJECT_ID` env var — that is the `--linked` resolver chain in both -// Go and the legacy port (flag → `SUPABASE_PROJECT_ID` → `supabase/.temp/project-ref`); -// `config.toml`'s `project_id` is NOT consulted for `--linked`. -function tempWorkdir(): string { - return mkdtempSync(join(tmpdir(), "sb-db-pull-live-")); -} + try { + const pushed = await run(["db", "push", "--db-url", dbUrl, "--yes"]); + requireLiveSuccess(pushed, "db push setup"); -// Data-plane: needs a provisioned project whose database is routable (the -// cli-e2e-ci Linux runner). `describeLiveDataPlane` runs this only when the project -// instance is ACTIVE_HEALTHY, so a control-plane-only stack (ref set but the DB -// unreachable, e.g. local macOS or the current cli-e2e-ci control-plane case) is -// skipped rather than timing out on the pg_dump seed. -describeLiveDataPlane("supabase db pull (live)", () => { - test( - "initial pull from the linked project (native pg_dump seed + migra diff)", - { timeout: LIVE_TIMEOUT_MS }, - async () => { - const ref = requireLiveProjectRef(); - const dir = tempWorkdir(); - try { - const { stdout, stderr, exitCode } = await runSupabaseLive(["db", "pull", "--linked"], { - cwd: dir, - env: { SUPABASE_PROJECT_ID: ref }, - exitTimeoutMs: LIVE_TIMEOUT_MS - 20_000, - // Decline the "Update remote migration history table?" prompt with a piped - // `n`: this project ref is shared across live runs, and writing a - // `schema_migrations` row here would make a later run see it as an extra - // remote migration and fail with a history conflict before pulling. The - // piped answer also exercises the native prompt's stdin scanning end to end. - stdin: "n\n", - }); - const combined = `${stdout}${stderr}`; - expect(combined).not.toContain("Unauthorized"); - // No local migrations → the native initial-migra path runs: pg_dump the remote - // schema, then append the migra diff. Assert on the durable side effect: a - // provisioned project with schema writes a `_remote_schema.sql` - // migration; a fresh empty schema reports "No schema changes found". Either - // proves the path ran end to end against the real database without hanging. - const migDir = join(dir, "supabase", "migrations"); - const wroteMigration = - existsSync(migDir) && readdirSync(migDir).some((f) => f.endsWith("_remote_schema.sql")); - expect(wroteMigration || combined.includes("No schema changes found")).toBe(true); - // The native path creates the migration file BEFORE pg_dump runs, so a failed - // dump/diff could leave a stray file behind — a written migration is only - // meaningful if the command actually succeeded. - if (wroteMigration) { - expect(exitCode).toBe(0); - } - } finally { - rmSync(dir, { recursive: true, force: true }); - } - }, - ); -}); + const result = await run(["db", "pull", "--db-url", dbUrl, "--yes"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(`${result.stdout}${result.stderr}`).not.toMatch( + /dial|no route|connection refused|could not connect|server closed the connection|i\/o timeout/i, + ); + } finally { + await unlink(migrationFile).catch(() => undefined); + const reset = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); + requireLiveSuccess(reset, "db reset cleanup after db pull"); + } + }, +); diff --git a/apps/cli/src/legacy/commands/db/push/push.live.test.ts b/apps/cli/src/legacy/commands/db/push/push.live.test.ts new file mode 100644 index 0000000000..7f2751cb0d --- /dev/null +++ b/apps/cli/src/legacy/commands/db/push/push.live.test.ts @@ -0,0 +1,31 @@ +import { mkdir, unlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { + requireLiveSuccess, + testLiveDataPlane, +} from "../../../../../tests/helpers/live-context.ts"; + +testLiveDataPlane( + "pushes a local migration to the remote database", + async ({ run, dbUrl, workspace }) => { + const version = `${Date.now()}${Math.floor(Math.random() * 10_000) + .toString() + .padStart(4, "0")}`; + const migrations = join(workspace.path, "supabase", "migrations"); + await mkdir(migrations, { recursive: true }); + const migrationFile = join(migrations, `${version}_e2e_push.sql`); + await writeFile(migrationFile, `create table if not exists e2e_push_${version} (id int);\n`); + + try { + const result = await run(["db", "push", "--db-url", dbUrl, "--yes"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Finished supabase db push"); + } finally { + await unlink(migrationFile).catch(() => undefined); + const reset = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); + requireLiveSuccess(reset, "db reset cleanup after db push"); + } + }, +); diff --git a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts new file mode 100644 index 0000000000..ee11b3c9f1 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts @@ -0,0 +1,31 @@ +import { mkdir, unlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { + requireLiveSuccess, + testLiveDataPlane, +} from "../../../../../tests/helpers/live-context.ts"; + +testLiveDataPlane( + "resets the remote database with local migrations", + async ({ run, dbUrl, workspace }) => { + const version = `${Date.now()}${Math.floor(Math.random() * 10_000) + .toString() + .padStart(4, "0")}`; + const migrations = join(workspace.path, "supabase", "migrations"); + await mkdir(migrations, { recursive: true }); + const migrationFile = join(migrations, `${version}_e2e_reset.sql`); + await writeFile(migrationFile, `create table if not exists e2e_reset_${version} (id int);\n`); + + try { + const result = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(`${result.stdout}${result.stderr}`).toContain("Resetting remote database"); + } finally { + await unlink(migrationFile).catch(() => undefined); + const reset = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); + requireLiveSuccess(reset, "db reset cleanup"); + } + }, +); diff --git a/apps/cli/src/legacy/commands/db/start/start.live.test.ts b/apps/cli/src/legacy/commands/db/start/start.live.test.ts new file mode 100644 index 0000000000..699d1209cd --- /dev/null +++ b/apps/cli/src/legacy/commands/db/start/start.live.test.ts @@ -0,0 +1,22 @@ +import { expect } from "vitest"; + +import { describeDockerLive } from "../../../../../tests/helpers/live.ts"; +import { testLive } from "../../../../../tests/helpers/live-context.ts"; + +describeDockerLive("supabase db start (live)", () => { + testLive( + "boots the local database", + async ({ run }) => { + try { + const started = await run(["db", "start"]); + expect(started.exitCode, started.stderr).toBe(0); + expect(`${started.stdout}${started.stderr}`).toMatch( + /Starting database|Initialising schema/i, + ); + } finally { + await run(["stop", "--no-backup"]).catch(() => undefined); + } + }, + 600_000, + ); +}); diff --git a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts new file mode 100644 index 0000000000..c5de9dbb27 --- /dev/null +++ b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts @@ -0,0 +1,35 @@ +import { randomUUID } from "node:crypto"; +import { mkdir, writeFile } from "node:fs/promises"; +import { expect } from "vitest"; + +import { describeDockerLive } from "../../../../../tests/helpers/live.ts"; +import { + requireLiveSuccess, + testLiveFunctions, +} from "../../../../../tests/helpers/live-context.ts"; + +describeDockerLive("functions delete (live)", () => { + testLiveFunctions("deletes a deployed function", async ({ run, projectRef, workspace }) => { + const slug = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; + const directory = `${workspace.path}/supabase/functions/${slug}`; + await mkdir(directory, { recursive: true }); + await writeFile(`${directory}/index.ts`, "Deno.serve(() => Response.json({ ok: true }));\n"); + await writeFile(`${directory}/deno.json`, '{\n "imports": {}\n}\n'); + + const deployed = await run(["functions", "deploy", slug, "--project-ref", projectRef]); + requireLiveSuccess(deployed, "functions deploy setup"); + + let deleted = false; + try { + const result = await run(["functions", "delete", slug, "--project-ref", projectRef]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Deleted Function"); + deleted = true; + } finally { + if (!deleted) { + const cleanup = await run(["functions", "delete", slug, "--project-ref", projectRef]); + requireLiveSuccess(cleanup, "functions delete cleanup"); + } + } + }); +}); diff --git a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts new file mode 100644 index 0000000000..7bcee15460 --- /dev/null +++ b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts @@ -0,0 +1,42 @@ +import { randomUUID } from "node:crypto"; +import { mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { describeDockerLive } from "../../../../../tests/helpers/live.ts"; +import { + expectFunctionOk, + requireLiveSuccess, + testLiveFunctions, +} from "../../../../../tests/helpers/live-context.ts"; + +describeDockerLive("functions deploy (live)", () => { + testLiveFunctions( + "deploys a function that responds over HTTP", + async ({ run, invoke, projectRef, workspace }) => { + const slug = `cli-e2e-deploy-${randomUUID().slice(0, 8)}`; + const directory = join(workspace.path, "supabase", "functions", slug); + await mkdir(directory, { recursive: true }); + await writeFile( + join(directory, "index.ts"), + `Deno.serve(() => Response.json({ case: ${JSON.stringify(slug)}, ok: true }));\n`, + ); + await writeFile(join(directory, "deno.json"), '{\n "imports": {}\n}\n'); + + let deployed = false; + try { + const result = await run(["functions", "deploy", slug, "--project-ref", projectRef]); + expect(result.exitCode, result.stderr).toBe(0); + deployed = true; + expect(result.stdout).toMatch(/Deployed Function/i); + + expectFunctionOk(await invoke(slug), slug); + } finally { + if (deployed) { + const deleted = await run(["functions", "delete", slug, "--project-ref", projectRef]); + requireLiveSuccess(deleted, "functions delete cleanup"); + } + } + }, + ); +}); diff --git a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts new file mode 100644 index 0000000000..4654d921ff --- /dev/null +++ b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts @@ -0,0 +1,9 @@ +import { expect } from "vitest"; + +import { testLiveDataPlane } from "../../../../../tests/helpers/live-context.ts"; + +testLiveDataPlane("generates TypeScript types from the remote schema", async ({ run, dbUrl }) => { + const result = await run(["gen", "types", "--db-url", dbUrl, "--lang", "typescript"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toMatch(/export type (Database|Json)/); +}); diff --git a/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts b/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts new file mode 100644 index 0000000000..7ea0a19489 --- /dev/null +++ b/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts @@ -0,0 +1,9 @@ +import { expect } from "vitest"; + +import { testLiveDataPlane } from "../../../../../../tests/helpers/live-context.ts"; + +testLiveDataPlane("reports statistics from the remote database", async ({ run, dbUrl }) => { + const result = await run(["inspect", "db", "db-stats", "--db-url", dbUrl]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Database Size"); +}); diff --git a/apps/cli/src/legacy/commands/link/link.live.test.ts b/apps/cli/src/legacy/commands/link/link.live.test.ts new file mode 100644 index 0000000000..b5b790a760 --- /dev/null +++ b/apps/cli/src/legacy/commands/link/link.live.test.ts @@ -0,0 +1,15 @@ +import { existsSync } from "node:fs"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { testLiveProject } from "../../../../tests/helpers/live-context.ts"; + +testLiveProject( + "links a project and writes its workspace cache", + async ({ run, projectRef, workspace }) => { + const result = await run(["link", "--project-ref", projectRef, "--skip-pooler"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Finished supabase link"); + expect(existsSync(join(workspace.path, "supabase", ".temp", "linked-project.json"))).toBe(true); + }, +); diff --git a/apps/cli/src/legacy/commands/migration/list/list.live.test.ts b/apps/cli/src/legacy/commands/migration/list/list.live.test.ts index 358baff4d2..cdb6778717 100644 --- a/apps/cli/src/legacy/commands/migration/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/list/list.live.test.ts @@ -1,53 +1,15 @@ -import { expect, test } from "vitest"; +import { expect } from "vitest"; -import { - describeLiveDataPlane, - requireLiveProjectRef, - runSupabaseLive, -} from "../../../../../tests/helpers/live.ts"; +import { testLiveDataPlane } from "../../../../../tests/helpers/live-context.ts"; -const LIVE_TIMEOUT_MS = 120_000; - -// Data-plane scenario: unlike `functions`/`branches` list (Management-API -// reads), `migration list` connects to the project's *Postgres* over the pooler. -// `describeLiveDataPlane` runs this only when the project instance is -// ACTIVE_HEALTHY — i.e. the full stack with supabase-postgres-17. The current -// cli-e2e-ci CI omits it (CLI-1825), so the project record exists but its DB is -// unreachable, and this suite SKIPS there rather than failing (see the gate's -// note). It activates automatically once the data-plane is provisioned. -// -// The `--linked` default mints a temp login role via the Management API, then -// reads `supabase_migrations.schema_migrations`. On a freshly provisioned -// project the history table is absent, which the handler maps to an empty list -// (an undefined-table error), so the command still exits 0. The ref is -// supplied via SUPABASE_PROJECT_ID (migration commands resolve the linked ref -// from env / config.toml / ref-file, not a `--project-ref` flag). -describeLiveDataPlane("supabase migration list (live)", () => { - test( - "lists migrations on the linked project's database", - { timeout: LIVE_TIMEOUT_MS }, - async () => { - const ref = requireLiveProjectRef(); - const { exitCode, stdout, stderr } = await runSupabaseLive(["migration", "list"], { - env: { SUPABASE_PROJECT_ID: ref }, - }); - expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); - expect(exitCode, `stdout:\n${stdout}\nstderr:\n${stderr}`).toBe(0); - }, - ); +testLiveDataPlane("lists migrations from the remote database", async ({ run, dbUrl }) => { + const result = await run(["migration", "list", "--db-url", dbUrl]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).not.toContain("Unauthorized"); +}); - test( - "emits machine-readable JSON with --output-format json", - { timeout: LIVE_TIMEOUT_MS }, - async () => { - const ref = requireLiveProjectRef(); - const { exitCode, stdout, stderr } = await runSupabaseLive( - ["migration", "list", "--output-format", "json"], - { env: { SUPABASE_PROJECT_ID: ref } }, - ); - expect(exitCode, `stdout:\n${stdout}\nstderr:\n${stderr}`).toBe(0); - // stdout must be payload-only valid JSON in json mode (no spinner/log noise). - expect(() => JSON.parse(stdout)).not.toThrow(); - }, - ); +testLiveDataPlane("emits migration list as JSON", async ({ run, dbUrl }) => { + const result = await run(["migration", "list", "--db-url", dbUrl, "--output-format", "json"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(() => JSON.parse(result.stdout)).not.toThrow(); }); diff --git a/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts b/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts new file mode 100644 index 0000000000..08cdfbb250 --- /dev/null +++ b/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts @@ -0,0 +1,19 @@ +import { expect } from "vitest"; + +import { testLiveProject } from "../../../../../tests/helpers/live-context.ts"; + +testLiveProject("lists API keys for a project", async ({ run, projectRef }) => { + const result = await run([ + "projects", + "api-keys", + "--project-ref", + projectRef, + "--output", + "json", + ]); + expect(result.exitCode, result.stderr).toBe(0); + const rows = JSON.parse(result.stdout) as Array<{ name?: string; api_key?: string }>; + expect( + rows.some((key) => key.name === "anon" || key.api_key?.startsWith("sb_publishable_")), + ).toBe(true); +}); diff --git a/apps/cli/src/legacy/commands/projects/list/list.live.test.ts b/apps/cli/src/legacy/commands/projects/list/list.live.test.ts index 8c4ca20f33..a34f257bfa 100644 --- a/apps/cli/src/legacy/commands/projects/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/projects/list/list.live.test.ts @@ -1,33 +1,19 @@ -import { expect, test } from "vitest"; +import { expect } from "vitest"; -import { describeLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; +import { testLive, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; -const LIVE_TIMEOUT_MS = 60_000; +testLiveProject( + "lists the live project for the authenticated token", + async ({ run, projectRef }) => { + const result = await run(["projects", "list", "--output-format", "json"]); + expect(result.exitCode, result.stderr).toBe(0); + const projects = JSON.parse(result.stdout) as Array<{ id?: string; ref?: string }>; + expect(projects.map((project) => project.ref ?? project.id)).toContain(projectRef); + }, +); -// Account-level read-only live scenario, alongside `orgs list`. Lists every -// project the authenticated token can access — no project ref required, so it -// runs against just the control plane (no provisioned project instance needed). -// Safe to run repeatedly; creates nothing. -describeLive("supabase projects list (live)", () => { - test("lists projects for the authenticated token", { timeout: LIVE_TIMEOUT_MS }, async () => { - const { exitCode, stdout, stderr } = await runSupabaseLive(["projects", "list"]); - expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); - expect(exitCode).toBe(0); - }); - - test( - "emits machine-readable JSON with --output-format json", - { timeout: LIVE_TIMEOUT_MS }, - async () => { - const { exitCode, stdout } = await runSupabaseLive([ - "projects", - "list", - "--output-format", - "json", - ]); - expect(exitCode).toBe(0); - // stdout must be payload-only valid JSON in json mode (no spinner/log noise). - expect(() => JSON.parse(stdout)).not.toThrow(); - }, - ); +testLive("emits projects as JSON for an account-level read", async ({ run }) => { + const result = await run(["projects", "list", "--output-format", "json"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(() => JSON.parse(result.stdout)).not.toThrow(); }); diff --git a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts new file mode 100644 index 0000000000..ebca088bf3 --- /dev/null +++ b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts @@ -0,0 +1,32 @@ +import { randomUUID } from "node:crypto"; +import { expect } from "vitest"; + +import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; + +async function unsetSecret( + run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + name: string, + projectRef: string, +): Promise { + const cleanup = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); + if (cleanup.exitCode !== 0) { + throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); + } +} + +testLiveProject("lists a secret created on the remote project", async ({ run, projectRef }) => { + const name = `CLI_E2E_LIST_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; + const created = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); + requireLiveSuccess(created, "secrets set setup"); + + try { + const result = await run(["secrets", "list", "--output", "json", "--project-ref", projectRef]); + expect(result.exitCode, result.stderr).toBe(0); + const names = (JSON.parse(result.stdout) as Array<{ name: string }>).map( + (secret) => secret.name, + ); + expect(names).toContain(name); + } finally { + await unsetSecret(run, name, projectRef); + } +}); diff --git a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts new file mode 100644 index 0000000000..fc24a48e53 --- /dev/null +++ b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts @@ -0,0 +1,26 @@ +import { randomUUID } from "node:crypto"; +import { expect } from "vitest"; + +import { testLiveProject } from "../../../../../tests/helpers/live-context.ts"; + +async function unsetSecret( + run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + name: string, + projectRef: string, +): Promise { + const cleanup = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); + if (cleanup.exitCode !== 0) { + throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); + } +} + +testLiveProject("sets a secret on the remote project", async ({ run, projectRef }) => { + const name = `CLI_E2E_SET_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; + const result = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); + try { + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Finished"); + } finally { + await unsetSecret(run, name, projectRef); + } +}); diff --git a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts new file mode 100644 index 0000000000..9c2165aaa9 --- /dev/null +++ b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts @@ -0,0 +1,29 @@ +import { randomUUID } from "node:crypto"; +import { expect } from "vitest"; + +import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; + +async function unsetSecret( + run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + name: string, + projectRef: string, +): Promise { + const cleanup = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); + if (cleanup.exitCode !== 0) { + throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); + } +} + +testLiveProject("unsets a secret from the remote project", async ({ run, projectRef }) => { + const name = `CLI_E2E_UNSET_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; + const created = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); + requireLiveSuccess(created, "secrets set setup"); + + try { + const result = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Finished"); + } finally { + await unsetSecret(run, name, projectRef); + } +}); diff --git a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts new file mode 100644 index 0000000000..9350b6050e --- /dev/null +++ b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts @@ -0,0 +1,42 @@ +import { randomUUID } from "node:crypto"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { requireLiveSuccess, testLiveStorage } from "../../../../../tests/helpers/live-context.ts"; + +const STORAGE_FLAGS = ["--linked", "--experimental"]; + +async function removeObject( + run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + remote: string, +): Promise { + const removed = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + if (removed.exitCode !== 0) { + throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); + } +} + +testLiveStorage( + "copies a local file to the remote bucket", + async ({ run, projectRef, dbPassword, storageBucket, workspace }) => { + const suffix = randomUUID().slice(0, 8); + const local = join(workspace.path, `upload-${suffix}.txt`); + const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + await writeFile(local, "live-e2e storage payload\n"); + + const linked = await run(["link", "--project-ref", projectRef], { + env: { SUPABASE_DB_PASSWORD: dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage cp"); + + let uploaded = false; + try { + const result = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + expect(result.exitCode, result.stderr).toBe(0); + uploaded = true; + } finally { + if (uploaded) await removeObject(run, remote); + } + }, +); diff --git a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts new file mode 100644 index 0000000000..d0147d611e --- /dev/null +++ b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts @@ -0,0 +1,43 @@ +import { randomUUID } from "node:crypto"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { requireLiveSuccess, testLiveStorage } from "../../../../../tests/helpers/live-context.ts"; + +const STORAGE_FLAGS = ["--linked", "--experimental"]; + +async function removeObject( + run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + remote: string, +): Promise { + const removed = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + if (removed.exitCode !== 0) { + throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); + } +} + +testLiveStorage( + "lists an uploaded object", + async ({ run, projectRef, dbPassword, storageBucket, workspace }) => { + const suffix = randomUUID().slice(0, 8); + const local = join(workspace.path, `upload-${suffix}.txt`); + const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + await writeFile(local, "live-e2e storage payload\n"); + + const linked = await run(["link", "--project-ref", projectRef], { + env: { SUPABASE_DB_PASSWORD: dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage ls"); + const uploaded = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + requireLiveSuccess(uploaded, "storage cp setup for storage ls"); + + try { + const result = await run(["storage", "ls", `ss:///${storageBucket}/`, ...STORAGE_FLAGS]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain(`upload-${suffix}.txt`); + } finally { + await removeObject(run, remote); + } + }, +); diff --git a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts new file mode 100644 index 0000000000..b287750c4c --- /dev/null +++ b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts @@ -0,0 +1,44 @@ +import { randomUUID } from "node:crypto"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { requireLiveSuccess, testLiveStorage } from "../../../../../tests/helpers/live-context.ts"; + +const STORAGE_FLAGS = ["--linked", "--experimental"]; + +async function removeObject( + run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + remote: string, +): Promise { + const removed = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + if (removed.exitCode !== 0) { + throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); + } +} + +testLiveStorage( + "removes an uploaded object", + async ({ run, projectRef, dbPassword, storageBucket, workspace }) => { + const suffix = randomUUID().slice(0, 8); + const local = join(workspace.path, `upload-${suffix}.txt`); + const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + await writeFile(local, "live-e2e storage payload\n"); + + const linked = await run(["link", "--project-ref", projectRef], { + env: { SUPABASE_DB_PASSWORD: dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage rm"); + const uploaded = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + requireLiveSuccess(uploaded, "storage cp setup for storage rm"); + + let removed = false; + try { + const result = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + expect(result.exitCode, result.stderr).toBe(0); + removed = true; + } finally { + if (!removed) await removeObject(run, remote); + } + }, +); diff --git a/apps/cli/tests/helpers/live-context.ts b/apps/cli/tests/helpers/live-context.ts index b2750e5cdd..cb423e92b4 100644 --- a/apps/cli/tests/helpers/live-context.ts +++ b/apps/cli/tests/helpers/live-context.ts @@ -6,7 +6,12 @@ import { inject, test } from "vitest"; import { makeTempHome } from "./cli.ts"; import { runSupabaseLive } from "./live.ts"; -import { isLiveConfigured, liveProjectRef } from "./live-env.ts"; +import { + isLiveConfigured, + isManagedLive, + liveProjectDataPlaneReady, + liveProjectRef, +} from "./live-env.ts"; type RunOptions = NonNullable[1]>; type RunResult = Awaited>; @@ -69,7 +74,7 @@ const base = test.extend({ await use(inject("storageBucket")); }, - home: async (_fixtures, use) => { + home: async ({ task: _task }, use) => { const home = makeTempHome(); try { await use(home); @@ -141,6 +146,42 @@ export const testLive = base.skipIf(!isLiveConfigured()); /** Fixture for scenarios that require a project ref from managed or attached setup. */ export const testLiveProject = base.skipIf(!isLiveConfigured() || !liveProjectRef()); +/** Fixture for Edge Function deploy/invoke scenarios. */ +export const testLiveFunctions = base.skipIf( + !isLiveConfigured() || + !liveProjectRef() || + (!isManagedLive() && + ((process.env["SUPABASE_LIVE_ANON_KEY"] ?? "").length === 0 || + (process.env["SUPABASE_LIVE_FUNCTIONS_URL"] ?? "").length === 0)), +); + +/** Fixture for scenarios that require the project's Postgres data plane. */ +export const testLiveDataPlane = base.skipIf( + !(await liveProjectDataPlaneReady()) || + (!isManagedLive() && (process.env["SUPABASE_LIVE_DB_URL"] ?? "").length === 0), +); + +/** Fixture for Storage scenarios requiring a linked database password. */ +export const testLiveStorage = base.skipIf( + !(await liveProjectDataPlaneReady()) || + !liveProjectRef() || + (!isManagedLive() && + ((process.env["SUPABASE_LIVE_DB_PASSWORD"] ?? "").length === 0 || + (process.env["SUPABASE_LIVE_STORAGE_BUCKET"] ?? "").length === 0)), +); + +/** Throw with command diagnostics when a setup/teardown command fails. */ +export function requireLiveSuccess( + result: { readonly exitCode: number; readonly stdout: string; readonly stderr: string }, + command: string, +): void { + if (result.exitCode !== 0) { + throw new Error( + `${command} failed (exit ${result.exitCode})\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`, + ); + } +} + /** Layer deploy-e2e function fixtures onto the generated workspace config. */ export function seedFunctions( workspacePath: string, From 8063c001eb1633fb64c3a526ba9baae88ed14f9d Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 17:37:01 +0200 Subject: [PATCH 05/23] test(cli): cover bulk function deployment --- .../functions/delete/delete.live.test.ts | 53 ++++++++++--------- .../functions/deploy/deploy.live.test.ts | 2 +- 2 files changed, 28 insertions(+), 27 deletions(-) diff --git a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts index c5de9dbb27..b4eb173da2 100644 --- a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts @@ -2,34 +2,35 @@ import { randomUUID } from "node:crypto"; import { mkdir, writeFile } from "node:fs/promises"; import { expect } from "vitest"; -import { describeDockerLive } from "../../../../../tests/helpers/live.ts"; -import { - requireLiveSuccess, - testLiveFunctions, -} from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; -describeDockerLive("functions delete (live)", () => { - testLiveFunctions("deletes a deployed function", async ({ run, projectRef, workspace }) => { - const slug = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; - const directory = `${workspace.path}/supabase/functions/${slug}`; - await mkdir(directory, { recursive: true }); - await writeFile(`${directory}/index.ts`, "Deno.serve(() => Response.json({ ok: true }));\n"); - await writeFile(`${directory}/deno.json`, '{\n "imports": {}\n}\n'); +testLiveProject("deletes a deployed function", async ({ run, projectRef, workspace }) => { + const slug = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; + const directory = `${workspace.path}/supabase/functions/${slug}`; + await mkdir(directory, { recursive: true }); + await writeFile(`${directory}/index.ts`, "Deno.serve(() => Response.json({ ok: true }));\n"); + await writeFile(`${directory}/deno.json`, '{\n "imports": {}\n}\n'); - const deployed = await run(["functions", "deploy", slug, "--project-ref", projectRef]); - requireLiveSuccess(deployed, "functions deploy setup"); + const deployed = await run([ + "functions", + "deploy", + slug, + "--project-ref", + projectRef, + "--use-api", + ]); + requireLiveSuccess(deployed, "functions deploy setup"); - let deleted = false; - try { - const result = await run(["functions", "delete", slug, "--project-ref", projectRef]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).toContain("Deleted Function"); - deleted = true; - } finally { - if (!deleted) { - const cleanup = await run(["functions", "delete", slug, "--project-ref", projectRef]); - requireLiveSuccess(cleanup, "functions delete cleanup"); - } + let deleted = false; + try { + const result = await run(["functions", "delete", slug, "--project-ref", projectRef]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Deleted Function"); + deleted = true; + } finally { + if (!deleted) { + const cleanup = await run(["functions", "delete", slug, "--project-ref", projectRef]); + requireLiveSuccess(cleanup, "functions delete cleanup"); } - }); + } }); diff --git a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts index 7bcee15460..6b2079ceb4 100644 --- a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts @@ -25,7 +25,7 @@ describeDockerLive("functions deploy (live)", () => { let deployed = false; try { - const result = await run(["functions", "deploy", slug, "--project-ref", projectRef]); + const result = await run(["functions", "deploy", "--project-ref", projectRef]); expect(result.exitCode, result.stderr).toBe(0); deployed = true; expect(result.stdout).toMatch(/Deployed Function/i); From 25b3aada3573215ef589834826c8482b8488026a Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 17:52:35 +0200 Subject: [PATCH 06/23] test(cli): move live suite ownership into cli --- .github/workflows/dispatch-cli-e2e-ci.yml | 9 +- .github/workflows/live-e2e.yml | 21 ++- CONTRIBUTING.md | 24 +-- apps/cli-e2e/.env.example | 25 +-- apps/cli-e2e/AGENTS.md | 26 +-- .../fixtures/live/functions-config.toml | 19 -- .../live/functions-project/assets/badge.svg | 3 - .../functions/_shared/greet.ts | 1 - .../functions/deploy-e2e-basic/deno.json | 3 - .../functions/deploy-e2e-basic/index.ts | 1 - .../deploy-e2e-custom-entry/deno.json | 3 - .../deploy-e2e-custom-entry/handler.ts | 3 - .../deploy-e2e-deno-jsonc/deno.jsonc | 6 - .../functions/deploy-e2e-deno-jsonc/index.ts | 5 - .../deploy-e2e-deprecated-map/import_map.json | 5 - .../deploy-e2e-deprecated-map/index.ts | 5 - .../deploy-e2e-dynamic-import/deno.json | 3 - .../deploy-e2e-dynamic-import/index.ts | 4 - .../deploy-e2e-dynamic-import/lazy.ts | 1 - .../functions/deploy-e2e-jsr/deno.json | 3 - .../functions/deploy-e2e-jsr/index.ts | 5 - .../deploy-e2e-jwt-required/deno.json | 3 - .../deploy-e2e-jwt-required/index.ts | 1 - .../deploy-e2e-local-imports/deno.json | 3 - .../deploy-e2e-local-imports/helpers.ts | 1 - .../deploy-e2e-local-imports/index.ts | 6 - .../functions/deploy-e2e-mode-api/deno.json | 3 - .../functions/deploy-e2e-mode-api/index.ts | 1 - .../deploy-e2e-mode-default/deno.json | 3 - .../deploy-e2e-mode-default/index.ts | 1 - .../deploy-e2e-mode-docker/deno.json | 3 - .../functions/deploy-e2e-mode-docker/index.ts | 1 - .../functions/deploy-e2e-no-jwt/deno.json | 3 - .../functions/deploy-e2e-no-jwt/index.ts | 1 - .../functions/deploy-e2e-npm/deno.json | 3 - .../functions/deploy-e2e-npm/index.ts | 10 - .../deploy-e2e-package-json/index.ts | 1 - .../deploy-e2e-package-json/package.json | 4 - .../deploy-e2e-remote-only/deno.json | 3 - .../functions/deploy-e2e-remote-only/index.ts | 1 - .../functions/deploy-e2e-root-map/deno.json | 3 - .../functions/deploy-e2e-root-map/index.ts | 5 - .../functions/deploy-e2e-scoped-map/deno.json | 5 - .../functions/deploy-e2e-scoped-map/index.ts | 5 - .../deploy-e2e-static-asset/assets/badge.svg | 3 - .../deploy-e2e-static-asset/deno.json | 3 - .../deploy-e2e-static-asset/index.ts | 10 - .../deploy-e2e-static-in-fn/deno.json | 3 - .../deploy-e2e-static-in-fn/index.ts | 8 - .../deploy-e2e-static-in-fn/static/note.txt | 1 - .../live/functions-project/import_map.json | 5 - apps/cli-e2e/package.json | 4 +- apps/cli-e2e/src/tests/env.ts | 54 +----- .../src/tests/live/branches.live.e2e.test.ts | 49 ----- .../src/tests/live/database.live.e2e.test.ts | 32 ---- .../live/db-reset-start.live.e2e.test.ts | 88 --------- .../src/tests/live/db-sync.live.e2e.test.ts | 47 ----- .../live/functions-deploy.live.e2e.test.ts | 66 ------- .../live/functions-lifecycle.live.e2e.test.ts | 73 -------- .../src/tests/live/gen-types.live.e2e.test.ts | 13 -- apps/cli-e2e/src/tests/live/invoke.ts | 47 ----- .../src/tests/live/link.live.e2e.test.ts | 21 --- apps/cli-e2e/src/tests/live/live-context.ts | 122 ------------ .../src/tests/live/projects.live.e2e.test.ts | 37 ---- .../src/tests/live/secrets.live.e2e.test.ts | 48 ----- .../src/tests/live/storage.live.e2e.test.ts | 45 ----- apps/cli-e2e/tests/live-setup.ts | 106 ----------- apps/cli-e2e/tests/provided-context.ts | 20 +- apps/cli-e2e/tests/staging-project.ts | 175 +----------------- apps/cli-e2e/vitest.config.ts | 5 +- apps/cli-e2e/vitest.live.config.ts | 21 --- apps/cli/AGENTS.md | 10 +- apps/cli/live.env.example | 37 ++++ .../cli}/scripts/sweep-live-projects.sh | 8 +- .../legacy/commands/start/start.live.test.ts | 2 +- apps/cli/tests/helpers/live-context.ts | 16 +- apps/cli/tests/helpers/live-env.ts | 19 +- apps/cli/tests/helpers/live.ts | 14 +- .../0013-live-e2e-bypasses-replay-server.md | 60 +++--- 79 files changed, 162 insertions(+), 1355 deletions(-) delete mode 100644 apps/cli-e2e/fixtures/live/functions-config.toml delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/assets/badge.svg delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/_shared/greet.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-basic/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-basic/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-custom-entry/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-custom-entry/handler.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deno-jsonc/deno.jsonc delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deno-jsonc/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deprecated-map/import_map.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deprecated-map/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/lazy.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jsr/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jsr/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jwt-required/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jwt-required/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/helpers.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-api/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-api/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-default/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-default/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-docker/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-docker/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-no-jwt/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-no-jwt/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-npm/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-npm/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-package-json/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-package-json/package.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-remote-only/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-remote-only/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-root-map/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-root-map/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-scoped-map/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-scoped-map/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/assets/badge.svg delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-in-fn/deno.json delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-in-fn/index.ts delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-in-fn/static/note.txt delete mode 100644 apps/cli-e2e/fixtures/live/functions-project/import_map.json delete mode 100644 apps/cli-e2e/src/tests/live/branches.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/database.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/db-reset-start.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/db-sync.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/functions-deploy.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/functions-lifecycle.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/gen-types.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/invoke.ts delete mode 100644 apps/cli-e2e/src/tests/live/link.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/live-context.ts delete mode 100644 apps/cli-e2e/src/tests/live/projects.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/secrets.live.e2e.test.ts delete mode 100644 apps/cli-e2e/src/tests/live/storage.live.e2e.test.ts delete mode 100644 apps/cli-e2e/tests/live-setup.ts delete mode 100644 apps/cli-e2e/vitest.live.config.ts create mode 100644 apps/cli/live.env.example rename {.github => apps/cli}/scripts/sweep-live-projects.sh (80%) diff --git a/.github/workflows/dispatch-cli-e2e-ci.yml b/.github/workflows/dispatch-cli-e2e-ci.yml index 4dfecad569..109b85cf49 100644 --- a/.github/workflows/dispatch-cli-e2e-ci.yml +++ b/.github/workflows/dispatch-cli-e2e-ci.yml @@ -3,10 +3,11 @@ name: Dispatch cli-e2e-ci # Asks the supabase/cli-e2e-ci harness to run the cli `test:live` suite against # a full supabox stack, built from THIS PR's head commit (CLI-1825 / CLI-1831). # -# This is distinct from `live-e2e.yml`, which runs the cli-e2e package against -# real staging (api.supabase.green). Here the suite runs against a local supabox -# stack stood up inside the private cli-e2e-ci repo; we only fire the trigger and -# pass our head SHA — cli-e2e-ci checks that SHA out into its `cli` submodule. +# This is distinct from `live-e2e.yml`, which runs the collocated live suite +# against managed staging (api.supabase.green). Here the same `apps/cli` suite +# runs against a local Supabox stack stood up inside the private cli-e2e-ci repo; +# we only fire the trigger and pass our head SHA — cli-e2e-ci checks that SHA out +# into its `cli` submodule. # # Opt-in by label to keep the expensive full-stack run off every PR: add the # `run-live-e2e-ci` label (re-dispatches on each subsequent push while labeled). diff --git a/.github/workflows/live-e2e.yml b/.github/workflows/live-e2e.yml index 2d1aff3a99..3b2bc4c6ca 100644 --- a/.github/workflows/live-e2e.yml +++ b/.github/workflows/live-e2e.yml @@ -1,7 +1,7 @@ name: Live E2E -# Live e2e suite (ADR-0013). Runs the real CLI against the real staging -# Management API + Docker bundler, then invokes the deployed functions over HTTP. +# Live e2e suite. Runs the collocated `apps/cli` tests against the real staging +# Management API + Docker bundler, then invokes deployed functions over HTTP. # # Non-blocking by construction: this is a standalone workflow, NOT part of the # required-checks set, and it never runs on the default PR path of test.yml. @@ -99,10 +99,11 @@ jobs: # Non-secret config is job-level; the staging token is scoped to only the two # steps that need it (run + cleanup) so build/checkout/docker never see it. env: - CLI_E2E_MODE: live - CLI_E2E_TARGET_ENV: staging - CLI_E2E_API_URL: https://api.supabase.green - CLI_E2E_PROJECT_HOST: supabase.red + SUPABASE_LIVE_MODE: managed + SUPABASE_PROFILE: supabase-staging + SUPABASE_LIVE_API_URL: https://api.supabase.green + SUPABASE_LIVE_PROJECT_HOST: supabase.red + SUPABASE_LIVE_PROJECT_NAME: supabase-cli-live-${{ matrix.target }} CLI_HARNESS_TARGET: ${{ matrix.target }} steps: - name: Checkout @@ -135,15 +136,15 @@ jobs: env: SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} run: | - PREFIX="cli-e2e-live-${CLI_HARNESS_TARGET}-${GITHUB_RUN_ID}-" + PREFIX="supabase-cli-live-${CLI_HARNESS_TARGET}-${GITHUB_RUN_ID}-" # GitHub runs this step as `bash -e`; use `if cmd; then` (errexit-exempt) # so a failing attempt does not abort the step before the retry. for attempt in 1 2 3; do echo "::group::live e2e attempt ${attempt}" if [ "$attempt" -gt 1 ]; then - bash .github/scripts/sweep-live-projects.sh "$PREFIX" || true + bash apps/cli/scripts/sweep-live-projects.sh "$PREFIX" || true fi - if pnpm --filter @supabase/cli-e2e test:e2e:live; then + if pnpm --filter supabase test:live; then echo "::endgroup::" exit 0 fi @@ -158,7 +159,7 @@ jobs: if: always() env: SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} - run: bash .github/scripts/sweep-live-projects.sh "cli-e2e-live-${CLI_HARNESS_TARGET}-${GITHUB_RUN_ID}-" + run: bash apps/cli/scripts/sweep-live-projects.sh "supabase-cli-live-${CLI_HARNESS_TARGET}-${GITHUB_RUN_ID}-" # Record that this beta tested green so the next scheduled run skips it. Needs # the whole matrix: the marker is saved only if the ts-legacy leg passed (a diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6e2834c4fd..c63d7d1671 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -176,35 +176,37 @@ pnpm run check:all ## E2E Compatibility Test Suite -`apps/cli-e2e` implements a record-and-replay test harness for testing the TypeScript Legacy CLI (`ts-legacy`, the only shipped CLI shell) against real Supabase Management API responses without hitting staging on every run, plus a separate live suite for smoke-testing real remote behavior. It still shells out to the bundled Go binary for the handful of commands the TS port proxies (`db diff`, `db pull`, `db branch *`, `db remote *`, `gen keys`, `functions download`), so `apps/cli-go/` is built alongside the TS CLI for this suite, but the suite itself no longer compares Go and TS output — that go-target parity harness was retired once the legacy port and the CLI-1970 Go binary trim landed. +`apps/cli-e2e` implements the replay-and-record compatibility harness for the TypeScript Legacy CLI (`ts-legacy`, the only shipped CLI shell). Live tests are owned by `apps/cli` and run from the command they cover. The CLI still shells out to the bundled Go binary for the handful of commands the TS port proxies (`db diff`, `db pull`, `db branch *`, `db remote *`, `gen keys`, `functions download`), so `apps/cli-go/` is built alongside the TS CLI for these suites, but there is no Go-vs-TypeScript parity runner. ### Architecture -Replay fixtures are recorded by running `ts-legacy` against the real Supabase staging API and capturing request/response pairs. Replay runs serve those committed fixtures back to the same CLI, so compatibility tests are fast and deterministic with no network access; live mode is described below. +Replay fixtures are recorded by running `ts-legacy` against the real Supabase staging API and capturing request/response pairs. Replay runs serve those committed fixtures back to the same CLI, so compatibility tests are fast and deterministic with no network access. The replay/record suite remains entirely under `apps/cli-e2e`. -The harness works in three modes: +The replay/record harness has two modes: | Mode | When | What it does | | -------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Replay** (default) | Every PR / local dev | Loads committed fixtures; serves recorded responses to the CLI subprocess. Fast and deterministic — no network access. | | **Record** | `RECORD=true` | Proxies CLI traffic to staging and captures request/response pairs as fixture files. | -| **Live** | `CLI_E2E_MODE=live` | Runs selected tests directly against a real staging project and Docker daemon. | ### Live remote-project coverage -The live suite runs the real `ts-legacy` CLI directly against the staging Management API — it does not use the replay server. Each serial run provisions one ephemeral project, waits for it to become `ACTIVE_HEALTHY`, uses the real Docker daemon for operations such as function deployment, and deletes the project during teardown (including after failures). Tests live in [`apps/cli-e2e/src/tests/live/**/*.live.e2e.test.ts`](apps/cli-e2e/src/tests/live), use the `testLive` fixtures (`run`, `invoke`, `workspace`, `projectRef`, `anonKey`, `functionsUrl`, `dbUrl`, `storageBucket`, and related values), and make **outcome-based assertions** on exit codes, output, HTTP status, and response bodies. +The live suite lives in `apps/cli/src/**` as collocated `*.live.test.ts` files and runs in the CLI package's `live` Vitest project. Global setup provisions one shared environment per run. Attached mode (the default) preserves the local Supabox contract: `SUPABASE_PROFILE` selects the platform, `SUPABASE_LIVE_API_URL` is used for readiness, and `SUPABASE_LIVE_PROJECT_REF` identifies the existing project; attached teardown never deletes it. Managed mode is explicit (`SUPABASE_LIVE_MODE=managed`) and provisions one uniquely named staging project, shares it across all live tests, then deletes exactly that project during teardown unless `SUPABASE_LIVE_KEEP_PROJECT=1` is set. -Live coverage is smoke coverage, not an exhaustive command matrix. Grow it toward one representative golden-path test for every user-facing CLI command: choose one realistic, common success workflow and let integration tests cover validation, formatting, fallbacks, errors, and other matrices unless the real remote/runtime boundary is the behavior under test. See [ADR 0013](docs/adr/0013-live-e2e-bypasses-replay-server.md) for the architecture and [`apps/cli-e2e/.env.example`](apps/cli-e2e/.env.example) for environment settings. +Live coverage is smoke coverage, not an exhaustive command matrix. Add one representative golden-path test for each user-facing command, colocated beside that command. A live test should assert one target command; setup and teardown may invoke other commands when they prepare or clean up state, but those commands are not asserted in that test. Keep validation, formatting, fallback, error, and matrix details in integration tests unless the remote/runtime boundary itself is the behavior under test. See [ADR 0013](docs/adr/0013-live-e2e-bypasses-replay-server.md) and [`apps/cli/live.env.example`](apps/cli/live.env.example). -To run the live suite locally, build the Go binary first, then provide a staging token: +To run the live suite locally against an attached Supabox/local platform, copy [`apps/cli/live.env.example`](apps/cli/live.env.example), fill in the platform values, and run: ```sh -cd apps/cli-go && go build -o /tmp/supabase-test-binary . && cd - -SUPABASE_GO_BINARY=/tmp/supabase-test-binary \ - SUPABASE_ACCESS_TOKEN=sbp_... \ - pnpm --filter @supabase/cli-e2e test:e2e:live +cd apps/cli +pnpm test:live ``` +For an explicit managed staging run, set `SUPABASE_LIVE_MODE=managed`, +`SUPABASE_PROFILE=supabase-staging`, `SUPABASE_LIVE_API_URL=https://api.supabase.green`, +and `SUPABASE_ACCESS_TOKEN` before invoking the same command. The suite's global setup +handles project provisioning and teardown. + Live CI is manual or daily scheduled and is not PR-blocking; run it manually on a PR branch when you need pre-merge remote coverage. ### Running the tests diff --git a/apps/cli-e2e/.env.example b/apps/cli-e2e/.env.example index 24e2ff10e9..77de786a27 100644 --- a/apps/cli-e2e/.env.example +++ b/apps/cli-e2e/.env.example @@ -1,18 +1,14 @@ -# cli-e2e environment — copy to `.env.local` (gitignored) and fill in. -# Only the live/record modes need real values; replay mode (the default) needs none. +# cli-e2e replay/record environment — copy to `.env.local` (gitignored) and fill in. +# Replay mode (the default) needs no environment variables. -# Mode: replay (default, no creds) | record (capture fixtures) | live (ADR-0013). -CLI_E2E_MODE=live - -# Backend the live/record suite targets. Only `staging` is wired today. -CLI_E2E_TARGET_ENV=staging +# Set RECORD=true (or CLI_E2E_MODE=record) to capture fixtures from staging. +CLI_E2E_MODE=record # CLI target under test: ts-legacy (the shipped shell, default) | ts-next. # (The `go` target was retired when the Go CLI was trimmed to the proxied subset.) CLI_HARNESS_TARGET=ts-legacy -# Staging Management API token. Either name works (the suite also reads -# SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN). Required in record/live mode. +# Staging Management API token. Required in record mode. SUPABASE_ACCESS_TOKEN=sbp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # ts-legacy shells out to the bundled Go binary for the proxied commands @@ -21,14 +17,9 @@ SUPABASE_ACCESS_TOKEN=sbp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # cd apps/cli-go && go build -o /tmp/supabase-test-binary . SUPABASE_GO_BINARY=/tmp/supabase-test-binary -# --- Optional overrides (sensible defaults in src/tests/env.ts) --- -# Management API base + per-project host (default to staging: api.supabase.green / supabase.red). +# --- Optional record overrides (sensible defaults in src/tests/env.ts) --- +# Management API base (also accepted as SUPABASE_STAGING_URL). # CLI_E2E_API_URL=https://api.supabase.green -# CLI_E2E_PROJECT_HOST=supabase.red -# DB password for the ephemeral project (default: random per run). +# DB password for the recording project (default: random per run). # CLI_E2E_DB_PASSWORD= -# Skip org resolution / region / pick a specific org. -# CLI_E2E_ORG_ID= # CLI_E2E_REGION=us-east-1 -# Leave the ephemeral live project alive after the run (debugging). -# CLI_E2E_KEEP_PROJECT=1 diff --git a/apps/cli-e2e/AGENTS.md b/apps/cli-e2e/AGENTS.md index 6a7a9d5497..d66cc8f665 100644 --- a/apps/cli-e2e/AGENTS.md +++ b/apps/cli-e2e/AGENTS.md @@ -141,19 +141,6 @@ In **record mode**: global setup resolves the org, deletes any orphaned test pro The pre-recording cleanup deletes projects named `cli-e2e-test`, `my-project`, and `to-delete` so re-recording never hits a 409 name-conflict. Do not add tests that rely on pre-existing named projects existing on staging. -## Live mode (ADR-0013) - -`live` is a third mode (`CLI_E2E_MODE=live`) that, unlike replay/record, **does not use the replay server**. The harness is wired straight at the real Management API (`CLI_E2E_API_URL`) and the real Docker socket; tests assert on **real outcomes**. - -- Live tests are `src/tests/live/**/*.live.e2e.test.ts`, run only via `vitest.live.config.ts` (the default config excludes them). They `skipIf(!isLive)`, so they are inert on the replay suite. -- Global setup (`tests/live-setup.ts`) provisions **one ephemeral project per run** (`cli-e2e-live-{target}-{runId}-{short}`), waits for `ACTIVE_HEALTHY`, resolves the anon JWT, the IPv4 **session-pooler `dbUrl`** (for `--db-url` DB commands), the functions URL, and a seeded storage bucket, exposing them via `inject()`. It deletes the project on teardown (even on failure). Setup is intentionally **dumb** — no provisioning retry; the CI step retries the live suite on flake. -- Use `testLive` from `src/tests/live/live-context.ts`: `run(cmd)` (direct-wired CLI), `invoke(slug)` (direct HTTP call sending the **anon JWT** in both `Authorization: Bearer` and `apikey`), plus `workspace` (a fresh `supabase init` config so golden paths exercise a generated config), `projectRef`, `anonKey`, `functionsUrl`, `dbUrl`, `storageBucket`. The functions deploy tests call `seedFunctions(workspace.path)` to layer the `deploy-e2e-*` fixtures + their `[functions.*]` config onto the init'd config. -- **Assertion style:** outcome-based — assert `exitCode`/`stdout` substrings and the function's HTTP status + JSON body. This is ID-agnostic, so **no normalization/snapshots by default**. If the CLI's own diagnostic output is ever the assertion target, add a scoped normalizer for that one test — do not make normalization the default. -- **Coverage policy:** live tests are smoke coverage, not an exhaustive matrix. Grow toward one representative golden-path test for every user-facing CLI command, choosing one common, realistic success workflow that exercises the real remote/runtime boundary. Keep validation, formatting, fallback, error, and matrix details in integration tests unless the remote boundary itself is what needs verification; check for an existing command's golden path before adding another live variant. -- **Authoring/CI target is `ts-legacy`** — the only shipped CLI shell. It still shells out to the Go binary for the handful of commands the TS port proxies (`db diff`, `db pull`, `db branch *`, `db remote *`, `gen keys`, `functions download`), so `SUPABASE_GO_BINARY` must point at a built Go binary for those to resolve. -- Retargeting to another env (e.g. `supabox`) is an env swap only: `CLI_E2E_TARGET_ENV` + `CLI_E2E_API_URL` + `CLI_E2E_PROJECT_HOST` + token. Tests assert on function output, not hostnames. -- **CI triggers** (`.github/workflows/live-e2e.yml`): `workflow_dispatch` (manual; the Actions branch picker selects the ref — no free-form `ref` input, so the staging token never reaches arbitrary code) and a daily `schedule`. There is **no `pull_request` trigger** — run it manually on a PR branch for pre-merge coverage. The scheduled run exercises the `@beta` channel: `develop` is the default branch and the beta release source, so it builds from `develop` source and runs the `ts-legacy` job. A `gate` job skips the run unless the published `supabase@beta` version changed since the last green run (an `actions/cache` marker keyed on the version, written by `finalize` only after the job passes), so a staging project is spent only when there is a new beta to test. Because the marker is written only on a green run, a chronically-failing `@beta` keeps re-running daily until it goes green or a newer beta supersedes it (intended — the failure stays visible). The job retries the live suite up to three times per run; setup itself has no provisioning retry. - ## Running the suite ```sh @@ -163,18 +150,11 @@ pnpm nx run @supabase/cli-e2e:test:legacy # ts-legacy target # Record (requires staging access) SUPABASE_ACCESS_TOKEN=sbp_... SUPABASE_STAGING_URL=https://api.supabase.green \ pnpm nx run @supabase/cli-e2e:record - -# Live (requires staging access; creates + deletes a real project; needs Docker). -# Build the Go binary first so newly-added proxy commands resolve (the system -# `supabase` may be stale) — mirrors what CI does. -cd apps/cli-go && go build -o /tmp/supabase-test-binary . && cd - -SUPABASE_GO_BINARY=/tmp/supabase-test-binary \ - SUPABASE_ACCESS_TOKEN=sbp_... \ - pnpm --filter @supabase/cli-e2e test:e2e:live ``` -See `apps/cli-e2e/.env.example` for the full set of live/record env vars (copy to -a gitignored `.env.local`). +See `apps/cli-e2e/.env.example` for replay/record env vars (copy to a gitignored +`.env.local`). Live environment setup is documented in `apps/cli/AGENTS.md` and +`apps/cli/live.env.example`. After recording, replay must pass with no changes between the two commands. diff --git a/apps/cli-e2e/fixtures/live/functions-config.toml b/apps/cli-e2e/fixtures/live/functions-config.toml deleted file mode 100644 index d210d7800e..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-config.toml +++ /dev/null @@ -1,19 +0,0 @@ -# Per-function config appended onto the `supabase init`-generated config.toml by -# seedFunctions() for the functions deploy tests (the import-map, custom -# entrypoint, static-file, and no-jwt fixtures need these). Everything else runs -# against the bare generated config. - -[functions."deploy-e2e-root-map"] -import_map = "./import_map.json" - -[functions."deploy-e2e-custom-entry"] -entrypoint = "./functions/deploy-e2e-custom-entry/handler.ts" - -[functions."deploy-e2e-static-in-fn"] -static_files = ["./functions/deploy-e2e-static-in-fn/static/*.txt"] - -[functions."deploy-e2e-static-asset"] -static_files = ["./assets/*.svg", "./functions/deploy-e2e-static-asset/assets/*.svg"] - -[functions."deploy-e2e-no-jwt"] -verify_jwt = false diff --git a/apps/cli-e2e/fixtures/live/functions-project/assets/badge.svg b/apps/cli-e2e/fixtures/live/functions-project/assets/badge.svg deleted file mode 100644 index 914f94e2e0..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/assets/badge.svg +++ /dev/null @@ -1,3 +0,0 @@ - - outside-static - diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/_shared/greet.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/_shared/greet.ts deleted file mode 100644 index d901eb79d4..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/_shared/greet.ts +++ /dev/null @@ -1 +0,0 @@ -export const greet = () => "hello"; diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-basic/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-basic/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-basic/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-basic/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-basic/index.ts deleted file mode 100644 index cc000c3fc7..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-basic/index.ts +++ /dev/null @@ -1 +0,0 @@ -Deno.serve(() => Response.json({ case: "deploy-e2e-basic", ok: true })); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-custom-entry/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-custom-entry/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-custom-entry/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-custom-entry/handler.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-custom-entry/handler.ts deleted file mode 100644 index ff43ad2065..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-custom-entry/handler.ts +++ /dev/null @@ -1,3 +0,0 @@ -Deno.serve(() => - Response.json({ case: "deploy-e2e-custom-entry", ok: true, entry: "handler.ts" }) -); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deno-jsonc/deno.jsonc b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deno-jsonc/deno.jsonc deleted file mode 100644 index 6f14fbcc6e..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deno-jsonc/deno.jsonc +++ /dev/null @@ -1,6 +0,0 @@ -{ - // scoped alias with comments - "imports": { - "@shared/": "../_shared/" - } -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deno-jsonc/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deno-jsonc/index.ts deleted file mode 100644 index 8b1ba2da96..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deno-jsonc/index.ts +++ /dev/null @@ -1,5 +0,0 @@ -import { greet } from "@shared/greet.ts"; - -Deno.serve(() => - Response.json({ case: "deploy-e2e-deno-jsonc", ok: true, message: greet() }) -); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deprecated-map/import_map.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deprecated-map/import_map.json deleted file mode 100644 index 4e99a415b5..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deprecated-map/import_map.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "imports": { - "@shared/": "../_shared/" - } -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deprecated-map/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deprecated-map/index.ts deleted file mode 100644 index 21231dc870..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-deprecated-map/index.ts +++ /dev/null @@ -1,5 +0,0 @@ -import { greet } from "@shared/greet.ts"; - -Deno.serve(() => - Response.json({ case: "deploy-e2e-deprecated-map", ok: true, message: greet() }) -); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/index.ts deleted file mode 100644 index 41a2055f44..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/index.ts +++ /dev/null @@ -1,4 +0,0 @@ -Deno.serve(async () => { - const { value } = await import("./lazy.ts"); - return Response.json({ case: "deploy-e2e-dynamic-import", ok: true, value }); -}); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/lazy.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/lazy.ts deleted file mode 100644 index 636afa7830..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-dynamic-import/lazy.ts +++ /dev/null @@ -1 +0,0 @@ -export const value = "lazy-ok"; diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jsr/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jsr/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jsr/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jsr/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jsr/index.ts deleted file mode 100644 index b136d09c48..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jsr/index.ts +++ /dev/null @@ -1,5 +0,0 @@ -import "jsr:@supabase/functions-js/edge-runtime.d.ts"; - -Deno.serve((req) => - Response.json({ case: "deploy-e2e-jsr", ok: true, method: req.method }) -); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jwt-required/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jwt-required/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jwt-required/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jwt-required/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jwt-required/index.ts deleted file mode 100644 index 81648d03de..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-jwt-required/index.ts +++ /dev/null @@ -1 +0,0 @@ -Deno.serve(() => Response.json({ case: "deploy-e2e-jwt-required", ok: true })); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/helpers.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/helpers.ts deleted file mode 100644 index 16e3e308e4..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/helpers.ts +++ /dev/null @@ -1 +0,0 @@ -export const suffix = "-imports"; diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/index.ts deleted file mode 100644 index fb7ea13f9d..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-local-imports/index.ts +++ /dev/null @@ -1,6 +0,0 @@ -import { greet } from "../_shared/greet.ts"; -import { suffix } from "./helpers.ts"; - -Deno.serve(() => - Response.json({ case: "deploy-e2e-local-imports", ok: true, message: greet() + suffix }) -); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-api/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-api/deno.json deleted file mode 100644 index f6ca8454c5..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-api/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-api/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-api/index.ts deleted file mode 100644 index e344e16514..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-api/index.ts +++ /dev/null @@ -1 +0,0 @@ -Deno.serve(() => Response.json({ case: "deploy-e2e-mode-api", ok: true })); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-default/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-default/deno.json deleted file mode 100644 index f6ca8454c5..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-default/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-default/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-default/index.ts deleted file mode 100644 index dbdfe144ff..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-default/index.ts +++ /dev/null @@ -1 +0,0 @@ -Deno.serve(() => Response.json({ case: "deploy-e2e-mode-default", ok: true })); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-docker/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-docker/deno.json deleted file mode 100644 index f6ca8454c5..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-docker/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-docker/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-docker/index.ts deleted file mode 100644 index fcd8ea060a..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-mode-docker/index.ts +++ /dev/null @@ -1 +0,0 @@ -Deno.serve(() => Response.json({ case: "deploy-e2e-mode-docker", ok: true })); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-no-jwt/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-no-jwt/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-no-jwt/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-no-jwt/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-no-jwt/index.ts deleted file mode 100644 index 1697305182..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-no-jwt/index.ts +++ /dev/null @@ -1 +0,0 @@ -Deno.serve(() => Response.json({ case: "deploy-e2e-no-jwt", ok: true })); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-npm/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-npm/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-npm/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-npm/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-npm/index.ts deleted file mode 100644 index 76b0dbb54a..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-npm/index.ts +++ /dev/null @@ -1,10 +0,0 @@ -import { createClient } from "npm:@supabase/supabase-js@2"; - -Deno.serve(() => { - const client = createClient("https://example.supabase.co", "anon-key"); - return Response.json({ - case: "deploy-e2e-npm", - ok: true, - hasClient: typeof client.from === "function", - }); -}); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-package-json/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-package-json/index.ts deleted file mode 100644 index c2671f20ac..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-package-json/index.ts +++ /dev/null @@ -1 +0,0 @@ -Deno.serve(() => Response.json({ case: "deploy-e2e-package-json", ok: true })); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-package-json/package.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-package-json/package.json deleted file mode 100644 index b667d153ab..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-package-json/package.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "type": "module", - "dependencies": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-remote-only/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-remote-only/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-remote-only/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-remote-only/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-remote-only/index.ts deleted file mode 100644 index b911f4475e..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-remote-only/index.ts +++ /dev/null @@ -1 +0,0 @@ -Deno.serve(() => Response.json({ case: "deploy-e2e-remote-only", ok: true })); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-root-map/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-root-map/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-root-map/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-root-map/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-root-map/index.ts deleted file mode 100644 index fd1cd5a53f..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-root-map/index.ts +++ /dev/null @@ -1,5 +0,0 @@ -import { greet } from "@root/greet.ts"; - -Deno.serve(() => - Response.json({ case: "deploy-e2e-root-map", ok: true, message: greet() }) -); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-scoped-map/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-scoped-map/deno.json deleted file mode 100644 index 4e99a415b5..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-scoped-map/deno.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "imports": { - "@shared/": "../_shared/" - } -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-scoped-map/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-scoped-map/index.ts deleted file mode 100644 index 783b8506d6..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-scoped-map/index.ts +++ /dev/null @@ -1,5 +0,0 @@ -import { greet } from "@shared/greet.ts"; - -Deno.serve(() => - Response.json({ case: "deploy-e2e-scoped-map", ok: true, message: greet() }) -); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/assets/badge.svg b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/assets/badge.svg deleted file mode 100644 index 914f94e2e0..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/assets/badge.svg +++ /dev/null @@ -1,3 +0,0 @@ - - outside-static - diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/deno.json b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/deno.json deleted file mode 100644 index 80c4e4a920..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/deno.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "imports": {} -} diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/index.ts b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/index.ts deleted file mode 100644 index 9a598ec2c9..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-asset/index.ts +++ /dev/null @@ -1,10 +0,0 @@ -// static_files bundles supabase/assets/*.svg (outside functions/) plus the function-local -// assets/ copy used at runtime (same pattern as deploy-e2e-static-in-fn). -Deno.serve(async () => { - const svg = await Deno.readTextFile(new URL("./assets/badge.svg", import.meta.url)); - return Response.json({ - case: "deploy-e2e-static-asset", - ok: true, - static: svg.includes("outside-static") || svg.includes(" { - const text = await Deno.readTextFile(new URL("./static/note.txt", import.meta.url)); - return Response.json({ - case: "deploy-e2e-static-in-fn", - ok: true, - static: text.trim(), - }); -}); diff --git a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-in-fn/static/note.txt b/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-in-fn/static/note.txt deleted file mode 100644 index 99337dc661..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/functions/deploy-e2e-static-in-fn/static/note.txt +++ /dev/null @@ -1 +0,0 @@ -in-fn-static diff --git a/apps/cli-e2e/fixtures/live/functions-project/import_map.json b/apps/cli-e2e/fixtures/live/functions-project/import_map.json deleted file mode 100644 index c84d752202..0000000000 --- a/apps/cli-e2e/fixtures/live/functions-project/import_map.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "imports": { - "@root/": "./functions/_shared/" - } -} diff --git a/apps/cli-e2e/package.json b/apps/cli-e2e/package.json index 84261f76a4..3d34d2ca2b 100644 --- a/apps/cli-e2e/package.json +++ b/apps/cli-e2e/package.json @@ -8,7 +8,6 @@ "test:e2e": "bun --bun vitest run", "test:legacy": "CLI_HARNESS_TARGET=ts-legacy bun --bun vitest run", "test:next": "CLI_HARNESS_TARGET=ts-next bun --bun vitest run", - "test:e2e:live": "CLI_E2E_MODE=live CLI_E2E_TARGET_ENV=staging bun --bun vitest run --config vitest.live.config.ts", "record": "RECORD=true CLI_HARNESS_TARGET=ts-legacy bun --bun vitest run", "check:all": "nx run-many -t types:check lint:check fmt:check knip:check --projects=$npm_package_name", "fix:all": "nx run-many -t lint:fix fmt:fix knip:fix --projects=$npm_package_name" @@ -30,9 +29,8 @@ "knip": { "entry": [ "src/**/*.e2e.test.ts", - "src/**/*.live.e2e.test.ts", "tests/**/*.ts", - "vitest.live.config.ts" + "vitest.config.ts" ], "ignore": [ "fixtures/**" diff --git a/apps/cli-e2e/src/tests/env.ts b/apps/cli-e2e/src/tests/env.ts index 57c1af2cc7..50f70ddbc4 100644 --- a/apps/cli-e2e/src/tests/env.ts +++ b/apps/cli-e2e/src/tests/env.ts @@ -1,18 +1,15 @@ import type { CLITarget } from "@supabase/cli-test-helpers"; -type CliE2eMode = "replay" | "record" | "live"; -type CliE2eTargetEnv = "staging" | "supabox"; +type CliE2eMode = "replay" | "record"; // Runtime mode. `replay` (default) serves recorded fixtures; `record` proxies to -// staging and captures fixtures; `live` (ADR-0013) bypasses the replay server and -// wires the CLI straight at the real Management API + Docker socket. +// staging and captures fixtures. // Back-compat: RECORD=true still maps to `record`. const MODE: CliE2eMode = (process.env["CLI_E2E_MODE"] as CliE2eMode | undefined) ?? (process.env["RECORD"] === "true" ? "record" : "replay"); export const isRecording = MODE === "record"; -export const isLive = MODE === "live"; // The replay server + tests/setup.ts key recording off the RECORD env var // directly. Keep RECORD in sync with MODE in BOTH directions so an explicit @@ -31,42 +28,14 @@ if (isRecording && !process.env["SUPABASE_STAGING_URL"] && process.env["CLI_E2E_ process.env["SUPABASE_STAGING_URL"] = process.env["CLI_E2E_API_URL"]; } -// Which backend the live/record suite targets. Only `staging` is wired today; -// `supabox` is a later env swap (CLI_E2E_API_URL + CLI_E2E_PROJECT_HOST + token). -const TARGET_ENV: CliE2eTargetEnv = - (process.env["CLI_E2E_TARGET_ENV"] as CliE2eTargetEnv | undefined) ?? "staging"; - -// Base Management API URL for record/live modes (the real API). In live mode the -// harness apiUrl is wired here directly — there is no replay server in front. -// Replay mode never reads this. -export const TARGET_API_URL = - process.env["CLI_E2E_API_URL"] ?? - process.env["SUPABASE_STAGING_URL"] ?? - "https://api.supabase.green"; - -// Host used to build the deployed-function invoke URL: -// https://{ref}.{PROJECT_HOST}/functions/v1 -// Environment-specific (staging is not supabase.co), so it is configurable. -export const PROJECT_HOST = - process.env["CLI_E2E_PROJECT_HOST"] ?? (TARGET_ENV === "staging" ? "supabase.red" : ""); - -// In replay mode the token never reaches a real API, but the Go CLI validates -// the format before making any request (must match sbp_[a-f0-9]{40}). -// In record/live mode it must be a valid token for the target env. Falls back to -// the live staging secret name so a local `.env.local` works without remapping. +// In replay mode the token never reaches a real API, but the CLI validates the +// format before making any request (must match sbp_[a-f0-9]{40}). In record mode +// it must be a valid token for the staging API. export const ACCESS_TOKEN = - process.env["SUPABASE_ACCESS_TOKEN"] ?? - process.env["SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN"] ?? - "sbp_0000000000000000000000000000000000000000"; - -// Whether a real token was supplied (vs the replay placeholder above). Live mode -// must fail fast on a missing token instead of letting every API call 401. -export const isAccessTokenProvided = Boolean( - process.env["SUPABASE_ACCESS_TOKEN"] ?? process.env["SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN"], -); + process.env["SUPABASE_ACCESS_TOKEN"] ?? "sbp_0000000000000000000000000000000000000000"; // Which target to run. Defaults to "ts-legacy" — the only shipped CLI shell and -// therefore the authoritative target for both recording and live tests. Validated +// therefore the authoritative target for replay and recording. Validated // eagerly so a stale value (e.g. the retired "go" target) fails with a clear error // instead of an undefined-command crash inside the harness. const VALID_TARGETS: ReadonlyArray = ["ts-legacy", "ts-next"]; @@ -80,16 +49,9 @@ if (matchedTarget === undefined) { } export const TARGET = matchedTarget; -// Optional org for the fresh live project. When unset, live-setup resolves it via -// `orgs list` (which also exercises that command against the real API). -export const ORG_ID_OVERRIDE = process.env["CLI_E2E_ORG_ID"]; - -// Region for the fresh live project. +// Region for the fresh recording project. export const REGION = process.env["CLI_E2E_REGION"] ?? "us-east-1"; -// Skip live-project teardown for debugging. -export const KEEP_PROJECT = process.env["CLI_E2E_KEEP_PROJECT"] === "1"; - // In replay mode any 20-char lowercase alpha string normalises to __PROJECT_REF__ // in the fixture key. In record mode supply a real project ref via env. export const PROJECT_REF = process.env["SUPABASE_TEST_PROJECT_REF"] ?? "aaaaaaaaaaaaaaaaaaaa"; diff --git a/apps/cli-e2e/src/tests/live/branches.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/branches.live.e2e.test.ts deleted file mode 100644 index 5c88ce20ce..0000000000 --- a/apps/cli-e2e/src/tests/live/branches.live.e2e.test.ts +++ /dev/null @@ -1,49 +0,0 @@ -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -// Preview branches (workflow 3). `branches create` provisions a real branch and -// requires a paid plan; the cli-e2e test org may be on the free plan, in which -// case the CLI must surface the plan requirement rather than crash. Handle both: -// on a paid org, create → list → delete; on a free org, assert the plan-gate. -describe("branches (live)", () => { - testLive("create + list + delete (or surface the plan gate)", async ({ run, projectRef }) => { - // Unique per attempt so a retry (vitest retry:2) after a post-create flake - // can't collide on the name; a finally guarantees cleanup either way. - const name = `e2e-branch-${Date.now()}`; - const created = await run(["branches", "create", name, "--project-ref", projectRef]); - - if (created.exitCode !== 0) { - // Free-plan org: the command must clearly report that branching needs a - // paid plan (not fail opaquely). - expect(created.stderr, created.stderr).toMatch(/paid plan|upgrade|not.*support/i); - return; - } - - let branchDeleted = false; - try { - expect(created.stdout).toContain("Created preview branch"); - - const listed = await run([ - "branches", - "list", - "--output", - "json", - "--project-ref", - projectRef, - ]); - expect(listed.exitCode, listed.stderr).toBe(0); - const names = (JSON.parse(listed.stdout) as Array<{ name?: string }>).map((b) => b.name); - expect(names).toContain(name); - - const deleted = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); - expect(deleted.exitCode, deleted.stderr).toBe(0); - branchDeleted = true; - } finally { - // Retry/leak safety: clean up only if the in-try delete didn't already - // succeed (e.g. an earlier assertion threw). Tolerates a not-found branch. - if (!branchDeleted) { - await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); - } - } - }); -}); diff --git a/apps/cli-e2e/src/tests/live/database.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/database.live.e2e.test.ts deleted file mode 100644 index 6a99f7131f..0000000000 --- a/apps/cli-e2e/src/tests/live/database.live.e2e.test.ts +++ /dev/null @@ -1,32 +0,0 @@ -import { existsSync, readFileSync } from "node:fs"; -import { join } from "node:path"; -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -// DB-connectivity commands against the fresh project's Postgres via the IPv4 -// session-mode Supavisor pooler (`dbUrl` from live-setup). The direct host -// (db..supabase.red) is IPv6-only and unreachable from IPv4-only CI -// runners; the pooler is IPv4, and session mode is required for pg_dump. -// A non-zero exit here means the connection itself failed. -describe("database (live, session pooler --db-url)", () => { - testLive("inspect db db-stats connects and reports stats", async ({ run, dbUrl }) => { - const res = await run(["inspect", "db", "db-stats", "--db-url", dbUrl]); - expect(res.exitCode, res.stderr).toBe(0); - expect(res.stdout).toContain("Database Size"); - }); - - testLive("migration list connects to the remote migration history", async ({ run, dbUrl }) => { - const res = await run(["migration", "list", "--db-url", dbUrl]); - // Fresh project has no migrations, but exit 0 proves it connected and - // queried the remote history table. - expect(res.exitCode, res.stderr).toBe(0); - }); - - testLive("db dump exports the remote schema", async ({ run, dbUrl, workspace }) => { - const file = join(workspace.path, "dump.sql"); - const res = await run(["db", "dump", "--db-url", dbUrl, "-f", file]); - expect(res.exitCode, res.stderr).toBe(0); - expect(existsSync(file)).toBe(true); - expect(readFileSync(file, "utf8")).toMatch(/CREATE|PostgreSQL database dump|SCHEMA/i); - }); -}); diff --git a/apps/cli-e2e/src/tests/live/db-reset-start.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/db-reset-start.live.e2e.test.ts deleted file mode 100644 index d3fba9d7e7..0000000000 --- a/apps/cli-e2e/src/tests/live/db-reset-start.live.e2e.test.ts +++ /dev/null @@ -1,88 +0,0 @@ -import { mkdirSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; -import { describe, expect } from "vitest"; -import { TARGET } from "../env.ts"; -import { testLive } from "./live-context.ts"; - -// Real-backend live coverage for the native `db start` / `db reset` ports. -// -// `db start` / `db reset` live only in the `go` reference and the `ts-legacy` -// port (the `next` shell has no `db` group), so skip the `ts-next` target. -// -// The live suite runs serially (`fileParallelism: false`, `maxWorkers: 1`), so the -// destructive remote reset below is safe against the throwaway per-run project. - -// --- Local leg: db start + db reset --local against the real Docker socket ----- -// Exercises `db start`'s native container-bootstrap sequence (network/volume/container -// bring-up, health wait, the fresh-volume SetupLocalDatabase-equivalent pipeline, and -// `_current_branch`) and `db reset --local`'s container-recreate flow end-to-end — the -// real-Docker boundary the in-process integration suites mock. Both are fully native TS -// now: `db reset --local`'s hidden Go `db __db-bootstrap` seam (`--mode recreate`/ -// `--mode await-storage`) was removed in CLI-1955 (see -// `commands/db/reset/reset.handler.ts` / `shared/db-bootstrap/recreate-local-database.ts`), -// the same way `db start`'s own seam usage was removed in CLI-1954 (see -// `commands/db/start/start.handler.ts`). The start → already-running → reset cycle runs -// in one test so it shares a single booted stack, and `finally` stops it (legacy proxies -// `stop` to Go) so the run never leaves containers behind. -describe.skipIf(TARGET === "ts-next")("db start / db reset --local (live, local Docker)", () => { - testLive( - "db start boots, is idempotent, and db reset --local recreates", - { timeout: 600_000 }, - async ({ run }) => { - try { - const start = await run(["db", "start"]); - expect(start.exitCode, start.stderr).toBe(0); - // Bootstrap progress goes to stderr on every target (Go, and native TS since CLI-1954). - expect(`${start.stdout}${start.stderr}`).toMatch(/Starting database|Initialising schema/i); - - // Second start is a no-op: the db is already running, exit 0. - const again = await run(["db", "start"]); - expect(again.exitCode, again.stderr).toBe(0); - expect(`${again.stdout}${again.stderr}`).toMatch(/already[\s-]running/i); - - // Local reset recreates the container and prints the git-branch line. - const reset = await run(["db", "reset", "--local"]); - expect(reset.exitCode, reset.stderr).toBe(0); - expect(reset.stderr).toContain("on branch "); - } finally { - await run(["stop", "--no-backup"]).catch(() => undefined); - } - }, - ); -}); - -// --- Remote leg: db reset against the staging project over the session pooler --- -// Exercises the native remote reset path (drop user schemas → apply local -// migrations → seed) against a real Postgres, no Docker. `--yes` auto-accepts the -// confirmation prompt (the non-interactive default is decline). Mutates the -// throwaway project's schema — deleted on teardown. The IPv4 session pooler -// `dbUrl` is used because the direct host is IPv6-only and unreachable from -// IPv4-only CI runners. -describe.skipIf(TARGET === "ts-next")("db reset (live, remote session pooler)", () => { - testLive( - "resets the remote schema and re-applies a local migration", - { timeout: 600_000 }, - async ({ run, dbUrl, workspace }) => { - const migrations = join(workspace.path, "supabase", "migrations"); - mkdirSync(migrations, { recursive: true }); - writeFileSync( - join(migrations, "20240101000000_e2e_reset.sql"), - "create table if not exists e2e_reset (id int);\n", - ); - - const reset = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); - expect(reset.exitCode, reset.stderr).toBe(0); - expect(reset.stderr).toContain("Resetting remote database"); - // A real connection failure must never be mistaken for a benign outcome. - expect(`${reset.stdout}${reset.stderr}`, "db reset hit a connection error").not.toMatch( - /dial|no route|connection refused|could not connect|server closed the connection|i\/o timeout/i, - ); - - // The migration history shows the re-applied version → proves the drop + - // migrate ran against the remote database. - const listed = await run(["migration", "list", "--db-url", dbUrl]); - expect(listed.exitCode, listed.stderr).toBe(0); - expect(listed.stdout).toContain("20240101000000"); - }, - ); -}); diff --git a/apps/cli-e2e/src/tests/live/db-sync.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/db-sync.live.e2e.test.ts deleted file mode 100644 index 4780b56537..0000000000 --- a/apps/cli-e2e/src/tests/live/db-sync.live.e2e.test.ts +++ /dev/null @@ -1,47 +0,0 @@ -import { mkdirSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -// Local↔remote schema sync (workflows 1-2) over the IPv4 session pooler. Done as -// one round-trip in a single workspace: pushing first makes the local migration -// history match the remote, so the subsequent pull's consistency check passes -// (a separate fresh-workspace pull would see a history mismatch on the shared -// per-run project). db push/pull confirm via a prompt that only auto-accepts -// with --yes. Mutates the throwaway project's schema — deleted on teardown. -describe("db push + pull (live, session pooler)", () => { - testLive( - "pushes a local migration and pulls the remote schema back", - async ({ run, dbUrl, workspace }) => { - const migrations = join(workspace.path, "supabase", "migrations"); - mkdirSync(migrations, { recursive: true }); - writeFileSync( - join(migrations, "20240101000000_e2e_push.sql"), - "create table if not exists e2e_push (id int);\n", - ); - - const pushed = await run(["db", "push", "--db-url", dbUrl, "--yes"]); - expect(pushed.exitCode, pushed.stderr).toBe(0); - - const listed = await run(["migration", "list", "--db-url", dbUrl]); - expect(listed.exitCode, listed.stderr).toBe(0); - expect(listed.stdout).toContain("20240101000000"); - - // Local history now matches remote, so pull connects and runs the diff. - // It either finds a remote-only change (exit 0, writes a migration) or - // reports no changes — both prove connectivity; only a real connection - // failure would surface a different error. - const pulled = await run(["db", "pull", "--db-url", dbUrl, "--yes"]); - const pullOutput = `${pulled.stdout}${pulled.stderr}`; - // The point of this test is connectivity over the pooler: a real connection - // failure must never be mistaken for a benign "no changes" outcome. - expect(pullOutput, "db pull hit a connection error").not.toMatch( - /dial|no route|connection refused|could not connect|server closed the connection|i\/o timeout/i, - ); - expect( - pulled.exitCode === 0 || /No schema changes found/i.test(pullOutput), - pulled.stderr, - ).toBe(true); - }, - ); -}); diff --git a/apps/cli-e2e/src/tests/live/functions-deploy.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/functions-deploy.live.e2e.test.ts deleted file mode 100644 index e9101cc1be..0000000000 --- a/apps/cli-e2e/src/tests/live/functions-deploy.live.e2e.test.ts +++ /dev/null @@ -1,66 +0,0 @@ -import { readdirSync } from "node:fs"; -import { join } from "node:path"; -import { describe, expect } from "vitest"; -import { expectFunctionOk } from "./invoke.ts"; -import { seedFunctions, testLive } from "./live-context.ts"; - -// Pilot (ADR-0013): deploy with the real CLI across the three bundler paths, -// then invoke the deployed function over HTTP and assert the body it returns. -// Each mode deploys a DISTINCT slug so the invoke proves THAT mode's deploy -// produced a running function — the shared project means a single slug could -// otherwise be served by an earlier mode's deploy. Negative/arg-validation -// cases live in apps/cli integration tests. -const MODES = [ - { name: "default", slug: "deploy-e2e-mode-default", flags: [] as string[] }, - { name: "use-api", slug: "deploy-e2e-mode-api", flags: ["--use-api"] }, - { name: "use-docker", slug: "deploy-e2e-mode-docker", flags: ["--use-docker"] }, -] as const; - -describe.each(MODES)("functions deploy ($name)", ({ slug, flags }) => { - testLive("deploys and the function responds", async ({ run, invoke, workspace, projectRef }) => { - seedFunctions(workspace.path); - const deployed = await run([ - "functions", - "deploy", - slug, - "--project-ref", - projectRef, - ...flags, - ]); - expect(deployed.exitCode, deployed.stderr).toBe(0); - expect(deployed.stdout).toContain("Deployed Functions"); - - const res = await invoke(slug); - expectFunctionOk(res, slug); - }); -}); - -// No slug → the CLI walks every function declared under supabase/functions and -// deploys them all. Assert each declared function appears in the deploy output, -// then smoke-invoke a representative one. -testLive( - "deploys every declared function when no slug is given", - async ({ run, invoke, workspace, projectRef }) => { - seedFunctions(workspace.path); - const declared = readdirSync(join(workspace.path, "supabase", "functions"), { - withFileTypes: true, - }) - .filter((e) => e.isDirectory() && !e.name.startsWith("_")) - .map((e) => e.name); - expect(declared.length).toBeGreaterThan(1); - - const deployed = await run(["functions", "deploy", "--project-ref", projectRef]); - expect(deployed.exitCode, deployed.stderr).toBe(0); - expect(deployed.stdout).toContain("Deployed Functions"); - - // Each declared function must be listed in the deploy output AND respond - // with its own {case: slug, ok: true}. A handler returns that marker only if - // it actually executed — and for the npm/jsr/local-imports/scoped-map - // fixtures only if their imports resolved at runtime — so this proves the - // feature ran end-to-end, not merely that the function deployed and booted. - for (const slug of declared) { - expect(deployed.stdout, `expected "${slug}" in deploy output`).toContain(slug); - expectFunctionOk(await invoke(slug), slug); - } - }, -); diff --git a/apps/cli-e2e/src/tests/live/functions-lifecycle.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/functions-lifecycle.live.e2e.test.ts deleted file mode 100644 index b800b8bda2..0000000000 --- a/apps/cli-e2e/src/tests/live/functions-lifecycle.live.e2e.test.ts +++ /dev/null @@ -1,73 +0,0 @@ -import { mkdirSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -// Write a throwaway Edge Function into the test workspace so the lifecycle tests -// own a dedicated slug (the shared per-run project is cleaned up on teardown). -function writeFunction(workspacePath: string, slug: string, jsonBody: string): void { - const dir = join(workspacePath, "supabase", "functions", slug); - mkdirSync(dir, { recursive: true }); - writeFileSync(join(dir, "index.ts"), `Deno.serve(() => Response.json(${jsonBody}));\n`); - writeFileSync(join(dir, "deno.json"), `{\n "imports": {}\n}\n`); -} - -// Active (non-REMOVED) function slugs. The Management API can keep deleted -// functions in the list with status REMOVED (the Go prune path skips them), so a -// successful delete may leave a REMOVED row — filter those out. -function activeSlugs(stdout: string): string[] { - return (JSON.parse(stdout) as Array<{ slug?: string; name?: string; status?: string }>) - .filter((f) => (f.status ?? "").toUpperCase() !== "REMOVED") - .map((f) => f.slug ?? f.name ?? ""); -} - -describe("functions update + delete (live)", () => { - // There is no dedicated `functions update` command — re-deploying a slug - // upserts it. Verify the second deploy replaces the running code. - testLive( - "re-deploying a function updates the running code", - async ({ run, invoke, workspace, projectRef }) => { - const slug = "deploy-e2e-update"; - - writeFunction(workspace.path, slug, `{ case: "${slug}", version: 1 }`); - expect((await run(["functions", "deploy", slug, "--project-ref", projectRef])).exitCode).toBe( - 0, - ); - expect((await invoke(slug)).body).toMatchObject({ case: slug, version: 1 }); - - writeFunction(workspace.path, slug, `{ case: "${slug}", version: 2 }`); - expect((await run(["functions", "deploy", slug, "--project-ref", projectRef])).exitCode).toBe( - 0, - ); - expect((await invoke(slug)).body).toMatchObject({ case: slug, version: 2 }); - }, - ); - - testLive("delete removes a deployed function", async ({ run, workspace, projectRef }) => { - const slug = "deploy-e2e-delete"; - - writeFunction(workspace.path, slug, `{ case: "${slug}", ok: true }`); - expect((await run(["functions", "deploy", slug, "--project-ref", projectRef])).exitCode).toBe( - 0, - ); - - const before = await run([ - "functions", - "list", - "--output", - "json", - "--project-ref", - projectRef, - ]); - expect(before.exitCode, before.stderr).toBe(0); - expect(activeSlugs(before.stdout)).toContain(slug); - - const del = await run(["functions", "delete", slug, "--project-ref", projectRef]); - expect(del.exitCode, del.stderr).toBe(0); - expect(del.stdout).toContain("Deleted Function"); - - const after = await run(["functions", "list", "--output", "json", "--project-ref", projectRef]); - expect(after.exitCode, after.stderr).toBe(0); - expect(activeSlugs(after.stdout)).not.toContain(slug); - }); -}); diff --git a/apps/cli-e2e/src/tests/live/gen-types.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/gen-types.live.e2e.test.ts deleted file mode 100644 index e75455989b..0000000000 --- a/apps/cli-e2e/src/tests/live/gen-types.live.e2e.test.ts +++ /dev/null @@ -1,13 +0,0 @@ -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -// gen types introspects the remote schema over the IPv4 session pooler and emits -// TypeScript types. It pulls the postgres-meta Docker image, so it needs Docker -// (present in the CI live job alongside the --use-docker bundler cell). -describe("gen types (live, session pooler)", () => { - testLive("generates TypeScript types from the remote schema", async ({ run, dbUrl }) => { - const res = await run(["gen", "types", "--db-url", dbUrl, "--lang", "typescript"]); - expect(res.exitCode, res.stderr).toBe(0); - expect(res.stdout).toMatch(/export type (Database|Json)/); - }); -}); diff --git a/apps/cli-e2e/src/tests/live/invoke.ts b/apps/cli-e2e/src/tests/live/invoke.ts deleted file mode 100644 index a5efcb3e58..0000000000 --- a/apps/cli-e2e/src/tests/live/invoke.ts +++ /dev/null @@ -1,47 +0,0 @@ -import { expect } from "vitest"; - -export interface InvokeResult { - status: number; - body: unknown; - text: string; -} - -/** Direct HTTP-invoke a deployed Edge Function and return status + parsed body. - * The replay server is not involved (ADR-0013) — this is a real call to the - * deployed function. Staging expects the publishable/anon key in BOTH the - * Authorization Bearer header and the apikey header. */ -export async function invokeFunction(opts: { - functionsUrl: string; - slug: string; - anonKey?: string; - payload?: unknown; -}): Promise { - const headers: Record = { "Content-Type": "application/json" }; - if (opts.anonKey) { - headers["Authorization"] = `Bearer ${opts.anonKey}`; - headers["apikey"] = opts.anonKey; - } - const res = await fetch(`${opts.functionsUrl}/${opts.slug}`, { - method: "POST", - headers, - body: JSON.stringify(opts.payload ?? {}), - }); - const text = await res.text(); - let body: unknown; - try { - body = JSON.parse(text); - } catch { - body = text; - } - return { status: res.status, body, text }; -} - -/** Assert the playbook's default per-slug expectation: 200 + `{case: slug, ok: true}`. */ -export function expectFunctionOk( - result: InvokeResult, - slug: string, - extra?: Record, -): void { - expect(result.status, result.text).toBe(200); - expect(result.body).toMatchObject({ case: slug, ok: true, ...extra }); -} diff --git a/apps/cli-e2e/src/tests/live/link.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/link.live.e2e.test.ts deleted file mode 100644 index f0c75b5e7b..0000000000 --- a/apps/cli-e2e/src/tests/live/link.live.e2e.test.ts +++ /dev/null @@ -1,21 +0,0 @@ -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -// `link` is the backbone of workflows 1-3. --skip-pooler keeps it -// Management-API-only (no IPv6-only DB connection): it validates the ref and -// writes the linked-project cache into the workspace's supabase/.temp. -describe("link (live)", () => { - testLive("links the project so ref-less commands resolve it", async ({ run, projectRef }) => { - const linked = await run(["link", "--project-ref", projectRef, "--skip-pooler"]); - expect(linked.exitCode, linked.stderr).toBe(0); - expect(linked.stdout).toContain("Finished supabase link"); - - // No --project-ref and no SUPABASE_PROJECT_ID env: a remote command must now - // resolve the ref from the link written above. - const listed = await run(["secrets", "list", "--output", "json"], { - env: { SUPABASE_PROJECT_ID: "" }, - }); - expect(listed.exitCode, listed.stderr).toBe(0); - expect(Array.isArray(JSON.parse(listed.stdout))).toBe(true); - }); -}); diff --git a/apps/cli-e2e/src/tests/live/live-context.ts b/apps/cli-e2e/src/tests/live/live-context.ts deleted file mode 100644 index 2cabe016e3..0000000000 --- a/apps/cli-e2e/src/tests/live/live-context.ts +++ /dev/null @@ -1,122 +0,0 @@ -import { appendFileSync, cpSync, readFileSync } from "node:fs"; -import { join } from "node:path"; -import { inject, test } from "vitest"; -import { - createHarness, - exec, - makeTempDir, - type CLIResult, - type TempDir, -} from "@supabase/cli-test-helpers"; -import { ACCESS_TOKEN, isLive, PROJECT_HOST, TARGET, TARGET_API_URL } from "../env.ts"; -import { invokeFunction, type InvokeResult } from "./invoke.ts"; - -type ExecOptions = NonNullable[2]>; - -// deploy-e2e-* function files (functions/, import_map.json, assets/) + the -// [functions.*] config snippet, layered onto an init-generated config by -// seedFunctions() for the functions deploy tests. -const FUNCTIONS_PROJECT_DIR = new URL("../../../fixtures/live/functions-project", import.meta.url) - .pathname; -const FUNCTIONS_CONFIG_SNIPPET = new URL( - "../../../fixtures/live/functions-config.toml", - import.meta.url, -).pathname; - -function liveHarness(cwd: string) { - return createHarness(TARGET, { - apiUrl: TARGET_API_URL, - accessToken: ACCESS_TOKEN, - cwd, - projectId: inject("projectRef"), - // Real host so host-derived commands (storage --linked → .) reach - // the live endpoint instead of localhost. - projectHost: PROJECT_HOST, - }); -} - -/** Layer the deploy-e2e-* function files + their [functions.*] config onto an - * init-generated workspace. Used by the functions deploy tests; every other - * test runs against the bare `supabase init` config. */ -export function seedFunctions(workspacePath: string): void { - const supabaseDir = join(workspacePath, "supabase"); - cpSync(FUNCTIONS_PROJECT_DIR, supabaseDir, { recursive: true }); - appendFileSync( - join(supabaseDir, "config.toml"), - `\n${readFileSync(FUNCTIONS_CONFIG_SNIPPET, "utf8")}`, - ); -} - -interface LiveFixtures { - projectRef: string; - anonKey: string; - functionsUrl: string; - dbUrl: string; - dbPassword: string; - storageBucket: string; - workspace: TempDir; - run: (cmd: string[], execOpts?: ExecOptions) => Promise; - invoke: (slug: string, opts?: { anonKey?: string; payload?: unknown }) => Promise; -} - -const base = test.extend({ - // eslint-disable-next-line no-empty-pattern - projectRef: async ({}, use) => { - await use(inject("projectRef")); - }, - - // eslint-disable-next-line no-empty-pattern - anonKey: async ({}, use) => { - await use(inject("anonKey")); - }, - - // eslint-disable-next-line no-empty-pattern - functionsUrl: async ({}, use) => { - await use(inject("functionsUrl")); - }, - - // eslint-disable-next-line no-empty-pattern - dbUrl: async ({}, use) => { - await use(inject("dbUrl")); - }, - - // eslint-disable-next-line no-empty-pattern - dbPassword: async ({}, use) => { - await use(inject("dbPassword")); - }, - - // eslint-disable-next-line no-empty-pattern - storageBucket: async ({}, use) => { - await use(inject("storageBucket")); - }, - - workspace: async ({ task }, use) => { - const dir = makeTempDir(`cli-e2e-live-${task.name.slice(0, 30)}-`); - // Generate config.toml via `supabase init` so the golden paths run against a - // freshly-generated config (functions tests add functions via seedFunctions). - const init = await exec(liveHarness(dir.path), ["init"]); - if (init.exitCode !== 0) throw new Error(`supabase init failed: ${init.stderr}`); - await use(dir); - dir[Symbol.dispose](); - }, - - run: async ({ workspace }, use) => { - const harness = liveHarness(workspace.path); - await use((cmd, execOpts) => exec(harness, cmd, execOpts)); - }, - - invoke: async ({ functionsUrl, anonKey }, use) => { - await use((slug, opts) => - invokeFunction({ - functionsUrl, - slug, - anonKey: opts && "anonKey" in opts ? opts.anonKey : anonKey, - payload: opts?.payload, - }), - ); - }, -}); - -/** Live test API — skipped unless CLI_E2E_MODE=live, so files are inert on - * replay/PR runs (and globalSetup provisions nothing). */ -export const testLive = base.skipIf(!isLive); diff --git a/apps/cli-e2e/src/tests/live/projects.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/projects.live.e2e.test.ts deleted file mode 100644 index 1b17aad672..0000000000 --- a/apps/cli-e2e/src/tests/live/projects.live.e2e.test.ts +++ /dev/null @@ -1,37 +0,0 @@ -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -// projects create/delete are exercised implicitly by live-setup (it provisions -// and tears down the per-run project). Here we cover the read paths against the -// real Management API: the fresh project shows up in `projects list`, and -// `projects api-keys` returns its keys. -describe("projects (live)", () => { - testLive( - "list includes the project and api-keys returns the anon key", - async ({ run, projectRef }) => { - const listed = await run(["projects", "list", "--output", "json"]); - expect(listed.exitCode, listed.stderr).toBe(0); - const refs = (JSON.parse(listed.stdout) as Array<{ id?: string; ref?: string }>).map( - (p) => p.ref ?? p.id, - ); - expect(refs).toContain(projectRef); - - const keys = await run([ - "projects", - "api-keys", - "--project-ref", - projectRef, - "--output", - "json", - ]); - expect(keys.exitCode, keys.stderr).toBe(0); - // Accept either a legacy anon JWT or a new-style publishable key — projects - // that only issue new keys still return a usable key. - const rows = JSON.parse(keys.stdout) as Array<{ name?: string; api_key?: string }>; - const hasUsableKey = rows.some( - (k) => k.name === "anon" || k.api_key?.startsWith("sb_publishable_"), - ); - expect(hasUsableKey, "expected an anon or publishable key").toBe(true); - }, - ); -}); diff --git a/apps/cli-e2e/src/tests/live/secrets.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/secrets.live.e2e.test.ts deleted file mode 100644 index b5c2170b68..0000000000 --- a/apps/cli-e2e/src/tests/live/secrets.live.e2e.test.ts +++ /dev/null @@ -1,48 +0,0 @@ -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -interface SecretRow { - name: string; -} - -// Live secrets flow (Management API only — no Docker, no DB). The fresh per-run -// project isolates the secret; the unset at the end cleans it up. Asserts on the -// real remote outcome: the key appears in `secrets list` after set and is gone -// after unset. -describe("secrets", () => { - testLive("set surfaces the key in list, unset removes it", async ({ run, projectRef }) => { - const key = "LIVE_E2E_SECRET"; - - const set = await run(["secrets", "set", `${key}=live-value`, "--project-ref", projectRef]); - expect(set.exitCode, set.stderr).toBe(0); - expect(set.stdout).toContain("Finished"); - - const afterSet = await run([ - "secrets", - "list", - "--output", - "json", - "--project-ref", - projectRef, - ]); - expect(afterSet.exitCode, afterSet.stderr).toBe(0); - const setNames = (JSON.parse(afterSet.stdout) as SecretRow[]).map((s) => s.name); - expect(setNames).toContain(key); - - const unset = await run(["secrets", "unset", key, "--project-ref", projectRef, "--yes"]); - expect(unset.exitCode, unset.stderr).toBe(0); - expect(unset.stdout).toContain("Finished"); - - const afterUnset = await run([ - "secrets", - "list", - "--output", - "json", - "--project-ref", - projectRef, - ]); - expect(afterUnset.exitCode, afterUnset.stderr).toBe(0); - const unsetNames = (JSON.parse(afterUnset.stdout) as SecretRow[]).map((s) => s.name); - expect(unsetNames).not.toContain(key); - }); -}); diff --git a/apps/cli-e2e/src/tests/live/storage.live.e2e.test.ts b/apps/cli-e2e/src/tests/live/storage.live.e2e.test.ts deleted file mode 100644 index 2d705f690d..0000000000 --- a/apps/cli-e2e/src/tests/live/storage.live.e2e.test.ts +++ /dev/null @@ -1,45 +0,0 @@ -import { writeFileSync } from "node:fs"; -import { join } from "node:path"; -import { describe, expect } from "vitest"; -import { testLive } from "./live-context.ts"; - -// Storage object round-trip against the project's real Storage API. `storage -// --linked` opens a DB connection to resolve storage config; the direct host is -// IPv6-only (unreachable from IPv4-only CI), so we `link` first (with the db -// password) to persist the IPv4 pooler connection that storage then reuses. -// The bucket is pre-seeded by live-setup; storage is gated behind --experimental. -const STORAGE_FLAGS = ["--linked", "--experimental"]; -describe("storage (live --linked)", () => { - testLive( - "uploads, lists, and removes an object", - async ({ run, workspace, projectRef, storageBucket, dbPassword }) => { - const linked = await run(["link", "--project-ref", projectRef], { - env: { SUPABASE_DB_PASSWORD: dbPassword }, - }); - expect(linked.exitCode, linked.stderr).toBe(0); - - const local = join(workspace.path, "upload.txt"); - writeFileSync(local, "live-e2e storage payload\n"); - const remote = `ss:///${storageBucket}/upload.txt`; - - const cp = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); - expect(cp.exitCode, cp.stderr).toBe(0); - - // Trailing slash lists the bucket's contents (without it, ls returns the - // bucket entry itself). - const ls = await run(["storage", "ls", `ss:///${storageBucket}/`, ...STORAGE_FLAGS]); - expect(ls.exitCode, ls.stderr).toBe(0); - expect(ls.stdout).toContain("upload.txt"); - - // --yes: rm prompts (default No) and would otherwise skip deletion in the - // non-TTY harness yet still exit 0. - const rm = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); - expect(rm.exitCode, rm.stderr).toBe(0); - - // Confirm the object is actually gone (guards against a no-op delete). - const after = await run(["storage", "ls", `ss:///${storageBucket}/`, ...STORAGE_FLAGS]); - expect(after.exitCode, after.stderr).toBe(0); - expect(after.stdout).not.toContain("upload.txt"); - }, - ); -}); diff --git a/apps/cli-e2e/tests/live-setup.ts b/apps/cli-e2e/tests/live-setup.ts deleted file mode 100644 index 4672cf8a2e..0000000000 --- a/apps/cli-e2e/tests/live-setup.ts +++ /dev/null @@ -1,106 +0,0 @@ -import { randomUUID } from "node:crypto"; -import type { ProvidedContext } from "vitest"; -import { - isAccessTokenProvided, - isLive, - KEEP_PROJECT, - ORG_ID_OVERRIDE, - PROJECT_HOST, - TARGET, - TARGET_API_URL, -} from "../src/tests/env.ts"; -import { - createStorageBucket, - createTestProject, - deleteTestProject, - generateDbPassword, - getAnonKey, - getPoolerSessionUrl, - getServiceRoleKey, - resolveOrgId, - waitForProjectReady, -} from "./staging-project.ts"; -import "./provided-context.ts"; // centralized `inject()` key augmentation - -const STORAGE_BUCKET = "cli-e2e-live-bucket"; - -// Live e2e global setup (ADR-0013). Provisions ONE ephemeral project per run, -// wired straight at the real Management API — no replay server. Intentionally -// dumb: no provisioning retry (the CI job re-runs the whole step on flake). -export async function setup({ - provide, -}: { - provide: (key: K, value: ProvidedContext[K]) => void; -}) { - if (!isLive) { - // The live config was invoked without CLI_E2E_MODE=live. Every test is - // skipIf(!isLive), so provision nothing. - return () => {}; - } - if (!isAccessTokenProvided) { - throw new Error( - "Live mode requires a staging access token: set SUPABASE_ACCESS_TOKEN " + - "(or SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN). Refusing to provision against an empty token.", - ); - } - if (!PROJECT_HOST) { - throw new Error("CLI_E2E_PROJECT_HOST is required in live mode (function invoke host)"); - } - - // Resolving the org via `orgs list` also exercises that command against the - // real API; CLI_E2E_ORG_ID short-circuits it when set. - const orgId = ORG_ID_OVERRIDE ?? (await resolveOrgId(TARGET_API_URL)); - - // Per-job, per-run unique name so the CI cleanup can target only this job's - // project (never a sibling matrix job's). - const runId = process.env["GITHUB_RUN_ID"] ?? String(Date.now()); - const name = `cli-e2e-live-${TARGET}-${runId}-${randomUUID().slice(0, 8)}`; - - // Generated here (not a shared export) and routed through provide() so the - // password reaches tests only via inject(), never an importable module const. - const dbPassword = generateDbPassword(); - const projectRef = await createTestProject(TARGET_API_URL, orgId, name, dbPassword); - - // Once the project exists, any later setup failure must still delete it — - // setup returns before the teardown closure, so Vitest cannot clean up. - let anonKey: string; - let functionsUrl: string; - let dbUrl: string; - try { - await waitForProjectReady(TARGET_API_URL, projectRef); - anonKey = await getAnonKey(TARGET_API_URL, projectRef); - functionsUrl = `https://${projectRef}.${PROJECT_HOST}/functions/v1`; - // IPv4 session-mode pooler — the direct host is IPv6-only (unreachable from - // IPv4-only CI runners); the pooler is IPv4 and session mode supports pg_dump. - dbUrl = await getPoolerSessionUrl(TARGET_API_URL, projectRef, dbPassword); - // Seed a private bucket via the Storage API so the storage live tests have - // something to cp/ls/rm against (cleaned up with the project on teardown). - const serviceRoleKey = await getServiceRoleKey(TARGET_API_URL, projectRef); - await createStorageBucket(PROJECT_HOST, projectRef, serviceRoleKey, STORAGE_BUCKET); - } catch (err) { - // Delete the half-provisioned project, but never mask the original failure. - if (!KEEP_PROJECT) { - await deleteTestProject(TARGET_API_URL, projectRef, { throwOnError: true }).catch( - (cleanupErr) => console.error("Failed to delete project after setup failure:", cleanupErr), - ); - } - throw err; - } - - provide("projectRef", projectRef); - provide("anonKey", anonKey); - provide("functionsUrl", functionsUrl); - provide("dbUrl", dbUrl); - provide("dbPassword", dbPassword); - provide("storageBucket", STORAGE_BUCKET); - - return async () => { - if (KEEP_PROJECT) { - console.log(`CLI_E2E_KEEP_PROJECT set — leaving project ${projectRef} (${name}) alive`); - return; - } - // Surface a failed teardown so a leaked staging project is visible locally - // (CI also has the always() sweep as a backstop). - await deleteTestProject(TARGET_API_URL, projectRef, { throwOnError: true }); - }; -} diff --git a/apps/cli-e2e/tests/provided-context.ts b/apps/cli-e2e/tests/provided-context.ts index a02d045184..a498890840 100644 --- a/apps/cli-e2e/tests/provided-context.ts +++ b/apps/cli-e2e/tests/provided-context.ts @@ -1,15 +1,14 @@ -// Single source of truth for Vitest's `inject()` keys across all three modes -// (replay/record use the replay-server keys; live uses the staging-project keys). -// Both global setups import this module so the augmentation is always in the -// build and `inject("…")` is typed without `as` casts. +// Single source of truth for Vitest's `inject()` keys used by the replay/record +// harness. The global setup imports this module so the augmentation is always +// in the build and `inject("…")` is typed without `as` casts. export {}; declare module "vitest" { export interface ProvidedContext { - // Shared by every mode. + // Shared by replay and record. projectRef: string; storageBucket: string; - // Replay/record only (replay server + pg/docker mocks). + // Replay/record (replay server + pg/docker mocks). replayServerUrl: string; orgId: string; pgMockPort: number; @@ -17,14 +16,5 @@ declare module "vitest" { * In record mode the relay forwards to the real Docker socket; in replay * mode it serves recorded Docker API fixtures. */ dockerHostUrl: string; - // Live only (ADR-0013): real ephemeral project wiring. - /** Legacy anon JWT for invoking deployed functions over HTTP. */ - anonKey: string; - /** https://{ref}.{CLI_E2E_PROJECT_HOST}/functions/v1 */ - functionsUrl: string; - /** IPv4 session-pooler Postgres URL for --db-url DB commands. */ - dbUrl: string; - /** DB password of the ephemeral project (for `link` → persisted pooler config). */ - dbPassword: string; } } diff --git a/apps/cli-e2e/tests/staging-project.ts b/apps/cli-e2e/tests/staging-project.ts index e0d54017fb..d30d0c8bd8 100644 --- a/apps/cli-e2e/tests/staging-project.ts +++ b/apps/cli-e2e/tests/staging-project.ts @@ -2,28 +2,20 @@ import { randomBytes } from "node:crypto"; import { createHarness, exec } from "@supabase/cli-test-helpers"; import { ACCESS_TOKEN, REGION, TARGET } from "../src/tests/env.ts"; -// Shared staging-project helpers used by both record setup (tests/setup.ts) and -// live setup (tests/live-setup.ts). -// -// `apiUrl` is whatever the CLI talks to: in record mode that is the replay -// server (so calls are captured); in live mode it is the real Management API -// (CLI_E2E_API_URL). The harness target + token come from env. +// Shared staging-project helpers used by record setup (tests/setup.ts). +// `apiUrl` is the replay server URL, which proxies calls to staging while +// recording. The harness target + token come from env. function harness(apiUrl: string) { return createHarness(TARGET, { apiUrl, accessToken: ACCESS_TOKEN }); } const PROJECT_REF_RE = /^[a-z]{20}$/; - -// Project statuses from which provisioning never recovers — fast-fail instead of -// polling to the timeout. const TERMINAL_BAD_STATUSES = new Set(["INIT_FAILED", "RESTORE_FAILED", "REMOVED"]); -/** A DB password for a throwaway project, used at creation and to build the live - * --db-url. Randomised per call (overridable via CLI_E2E_DB_PASSWORD) so no - * static credential is committed — the project is deleted on teardown anyway. - * Each setup generates its own and routes it through provide()/inject() rather - * than sharing a module-level export. */ +/** A DB password for a throwaway recording project. Randomised per call + * (overridable via CLI_E2E_DB_PASSWORD) so no static credential is committed — + * the project is deleted on teardown anyway. */ export function generateDbPassword(): string { return process.env["CLI_E2E_DB_PASSWORD"] ?? `cli-e2e-${randomBytes(12).toString("hex")}`; } @@ -64,8 +56,8 @@ export async function createTestProject( return ref; } -// `throwOnError` surfaces a failed deletion (live teardown uses it so a leaked -// staging project fails the run loudly; record setup keeps the lenient default). +// `throwOnError` surfaces deletion failures when a caller needs to fail loudly; +// record setup keeps the lenient default. export async function deleteTestProject( apiUrl: string, projectRef: string, @@ -100,8 +92,7 @@ export async function cleanupProjectsByName(apiUrl: string, names: string[]): Pr } } -/** Poll the real Management API until the project is ACTIVE_HEALTHY. Hits the API - * directly (not via any proxy) — this is setup-only and must not be recorded. */ +/** Poll the Management API until the recording project is ACTIVE_HEALTHY. */ export async function waitForProjectReady( apiBaseUrl: string, projectRef: string, @@ -121,155 +112,9 @@ export async function waitForProjectReady( ); } } else { - await res.body?.cancel(); // free the socket before sleeping + await res.body?.cancel(); } await new Promise((r) => setTimeout(r, 5_000)); } throw new Error(`Project ${projectRef} did not become ACTIVE_HEALTHY within ${timeoutMs}ms`); } - -interface ApiKey { - name?: string; - api_key?: string; -} - -/** Resolve a key for invoking the project's deployed functions over HTTP. - * Prefers the legacy `anon` JWT: Edge Functions default to verify_jwt=true and - * a publishable (sb_publishable_) key is NOT a JWT, so it fails the platform - * JWT check on a verified function. Falls back to the publishable key for - * projects that only issue new-style keys. Even after ACTIVE_HEALTHY the - * api-keys endpoint can briefly 4xx, so retry. */ -export async function getAnonKey( - apiBaseUrl: string, - projectRef: string, - attempts = 12, -): Promise { - for (let attempt = 1; attempt <= attempts; attempt++) { - const res = await fetch(`${apiBaseUrl}/v1/projects/${projectRef}/api-keys`, { - headers: { Authorization: `Bearer ${ACCESS_TOKEN}` }, - }); - if (res.ok) { - const keys = (await res.json()) as ApiKey[]; - const anonJwt = keys.find((k) => k.name === "anon" && k.api_key)?.api_key; - if (anonJwt) return anonJwt; - // Keys present but no legacy anon JWT. A publishable (sb_publishable_) key - // is NOT a JWT and 401s on the default verify_jwt=true functions, so fail - // loudly rather than proceed with a key that can't authenticate verified - // invokes (the suite would need to deploy with --no-verify-jwt instead). - if (keys.length > 0) { - throw new Error( - `Project ${projectRef} returned no anon JWT (only new-style keys); verified-function invokes require a JWT`, - ); - } - } else if (attempt < attempts) { - await res.body?.cancel(); // free the socket before sleeping - } - if (attempt === attempts) { - const detail = res.bodyUsed ? res.status : await res.text().catch(() => res.status); - throw new Error( - `Failed to resolve anon key for ${projectRef} after ${attempts} attempts: ${detail}`, - ); - } - await new Promise((r) => setTimeout(r, 10_000)); - } - // Unreachable — the loop either returns a key or throws on the last attempt. - throw new Error(`Failed to resolve anon key for ${projectRef}`); -} - -/** Service-role / secret key, used to seed a storage bucket for the live storage - * tests (the same way record setup does). Retries like getAnonKey. */ -export async function getServiceRoleKey( - apiBaseUrl: string, - projectRef: string, - attempts = 12, -): Promise { - for (let attempt = 1; attempt <= attempts; attempt++) { - const res = await fetch(`${apiBaseUrl}/v1/projects/${projectRef}/api-keys`, { - headers: { Authorization: `Bearer ${ACCESS_TOKEN}` }, - }); - if (res.ok) { - const keys = (await res.json()) as ApiKey[]; - const secret = - keys.find((k) => k.name === "service_role" && k.api_key)?.api_key ?? - keys.find((k) => k.api_key?.startsWith("sb_secret_"))?.api_key; - if (secret) return secret; - } else { - await res.body?.cancel(); // free the socket before sleeping - } - if (attempt === attempts) { - throw new Error(`Failed to resolve service-role key for ${projectRef}`); - } - await new Promise((r) => setTimeout(r, 10_000)); - } - throw new Error(`Failed to resolve service-role key for ${projectRef}`); -} - -/** Create a private storage bucket via the project's Storage API (host derived - * from projectHost, IPv4-reachable). Idempotent — treats an existing bucket as - * success. */ -export async function createStorageBucket( - projectHost: string, - projectRef: string, - serviceRoleKey: string, - bucket: string, -): Promise { - const res = await fetch(`https://${projectRef}.${projectHost}/storage/v1/bucket`, { - method: "POST", - headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, - body: JSON.stringify({ id: bucket, name: bucket, public: false }), - }); - if (!res.ok && res.status !== 409) { - throw new Error(`Failed to create bucket ${bucket}: ${res.status} ${await res.text()}`); - } -} - -interface PoolerConfig { - database_type?: string; - connection_string?: string; -} - -/** Build a SESSION-mode (port 5432) Supavisor pooler connection string for the - * project's Postgres. The direct host (db....) is IPv6-only and unreachable - * from IPv4-only CI runners, so DB commands go through the pooler, which is IPv4. - * Session mode (not the API's default transaction 6543) is required for pg_dump - * (`db dump`). - * - * Reuses the Management API's `connection_string` verbatim — it carries tenant - * routing (e.g. options=reference=... query params) that a field-reconstructed - * URL would drop — and only swaps in our password and the session port. Mirrors - * the Go connector by selecting the PRIMARY pooler config. */ -export async function getPoolerSessionUrl( - apiBaseUrl: string, - projectRef: string, - password: string, - attempts = 12, -): Promise { - for (let attempt = 1; attempt <= attempts; attempt++) { - const res = await fetch(`${apiBaseUrl}/v1/projects/${projectRef}/config/database/pooler`, { - headers: { Authorization: `Bearer ${ACCESS_TOKEN}` }, - }); - if (res.ok) { - const raw = (await res.json()) as PoolerConfig | PoolerConfig[]; - const configs = Array.isArray(raw) ? raw : [raw]; - const primary = configs.find((c) => c.database_type === "PRIMARY") ?? configs[0]; - if (primary?.connection_string) { - const url = new URL(primary.connection_string); - url.password = password; // overwrites the [YOUR-PASSWORD] placeholder (URL-encoded) - url.port = "5432"; // session mode (API returns the 6543 transaction port) - if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); - return url.toString(); - } - } else if (attempt < attempts) { - await res.body?.cancel(); // free the socket before sleeping - } - if (attempt === attempts) { - const detail = res.bodyUsed ? res.status : await res.text().catch(() => res.status); - throw new Error( - `Failed to resolve pooler config for ${projectRef} after ${attempts} attempts: ${detail}`, - ); - } - await new Promise((r) => setTimeout(r, 10_000)); - } - // Unreachable — the loop either returns a URL or throws on the last attempt. - throw new Error(`Failed to resolve pooler config for ${projectRef}`); -} diff --git a/apps/cli-e2e/vitest.config.ts b/apps/cli-e2e/vitest.config.ts index 74c9117ecc..bb87f884aa 100644 --- a/apps/cli-e2e/vitest.config.ts +++ b/apps/cli-e2e/vitest.config.ts @@ -5,10 +5,7 @@ export default defineConfig({ test: { passWithNoTests: true, include: ["**/*.e2e.test.ts"], - // Live tests are *.live.e2e.test.ts and run only via vitest.live.config.ts. - // They also match the include glob, so exclude them here to keep the - // PR-blocking replay suite from globbing them. - exclude: ["**/node_modules/**", "**/*.live.e2e.test.ts"], + exclude: ["**/node_modules/**"], fileParallelism: false, maxWorkers: 1, globalSetup: ["tests/setup.ts"], diff --git a/apps/cli-e2e/vitest.live.config.ts b/apps/cli-e2e/vitest.live.config.ts deleted file mode 100644 index 5a6ea689ff..0000000000 --- a/apps/cli-e2e/vitest.live.config.ts +++ /dev/null @@ -1,21 +0,0 @@ -import { defineConfig } from "vitest/config"; - -// Live e2e project (ADR-0013): runs *.live.e2e.test.ts against a real backend. -// Separate from vitest.config.ts so the PR-blocking replay suite never globs -// live tests. The replay server is NOT started here — live-setup wires the CLI -// straight at the real Management API + Docker socket. -export default defineConfig({ - test: { - passWithNoTests: true, - include: ["**/*.live.e2e.test.ts"], - fileParallelism: false, - maxWorkers: 1, - globalSetup: ["tests/live-setup.ts"], - // Real provisioning + Docker bundling are slow; give each test plenty of room. - testTimeout: 600_000, - hookTimeout: 600_000, - // Per-test flake (a single invoke/deploy blip) retries here; provisioning / - // setup flake is handled by the CI job re-running the whole step. - retry: 2, - }, -}); diff --git a/apps/cli/AGENTS.md b/apps/cli/AGENTS.md index 4b32693b00..8a9f7834fd 100644 --- a/apps/cli/AGENTS.md +++ b/apps/cli/AGENTS.md @@ -491,14 +491,16 @@ Read https://www.effect.solutions/testing for Effect testing patterns. Note that Live tests are black-box CLI subprocess tests — like `*.e2e.test.ts`, but run against a **real backend** instead of local fakes/mocks: either the real Management API (a full [supabox](https://github.com/supabase/supabox) platform stack) or a real local Docker dev stack (`supabase start`'s actual containers). They are the highest-fidelity, most expensive tier — reserved for the small set of behaviors that only a genuinely running backend can prove (auth round-trips, real Docker label filtering, real container lifecycle), not for anything an integration test can already cover with mocks. -- **Where they run:** authored in this repo, but executed by the [`supabase/cli-e2e-ci`](https://github.com/supabase/cli-e2e-ci) harness, which builds this CLI, brings up a full supabox stack (and has a real Docker daemon, since that's how supabox itself runs), and invokes the `live` Vitest project (`nx run-many -t test:live`). They never run as part of the default unit/integration/e2e loop, and locally they no-op unless the live environment is configured (see below) — there is no need to stand up supabox yourself to develop other code. +- **Where they run:** authored and executed from this workspace's `live` Vitest project (`pnpm test:live`). CI may run the same files from the [`supabase/cli-e2e-ci`](https://github.com/supabase/cli-e2e-ci) Supabox harness or the managed-staging workflow. They never run as part of the default unit/integration/e2e loop, and locally they no-op unless the live environment is configured (see below). - **Add one whenever you add or change a command whose correctness genuinely depends on a real backend** — a new Management API command, or a change to `start`/`stop`/`status`'s real Docker interaction. Colocate it with the command, same as `*.e2e.test.ts`: `src/legacy/commands//[/].live.test.ts`. -- **Gating:** every live suite must be wrapped in one of `tests/helpers/live.ts`'s `describe.skipIf` gates so the file is inert (skipped, not failed) outside the cli-e2e-ci runner: - - `describeLive` — runs whenever `SUPABASE_ACCESS_TOKEN` is set (the live env is configured at all). Reuse this even for commands that don't call the Management API themselves (e.g. `stop`/`status`) — it doubles as the "we're in the full cli-e2e-ci runner, which also has a real Docker daemon" signal. - - `describeDockerLive` — the configured-live gate composed with a `docker info` probe; use for local-stack suites whose scenarios additionally need a reachable Docker daemon at collection time. It never runs on a machine that merely exposes Docker — `SUPABASE_ACCESS_TOKEN` must still be set, so the file stays inert outside the cli-e2e-ci runner like every other live suite. +- **Gating:** every live suite must be wrapped in one of `tests/helpers/live.ts`'s `describe.skipIf` gates so the file is inert (skipped, not failed) outside a configured live runner: + - `describeLive` — runs whenever `SUPABASE_ACCESS_TOKEN` is set (the live env is configured at all). Reuse this even for commands that don't call the Management API themselves (e.g. `stop`/`status`) — it doubles as the "we're in the configured live runner" signal. + - `describeDockerLive` — the configured-live gate composed with a `docker info` probe; use for local-stack suites whose scenarios additionally need a reachable Docker daemon at collection time. It never runs on a machine that merely exposes Docker — `SUPABASE_ACCESS_TOKEN` must still be set, so the file stays inert outside a configured live runner like every other live suite. - `describeLiveProject` — additionally requires a provisioned project (`SUPABASE_LIVE_PROJECT_REF`); use for project-scoped Management API commands (branches, functions, project-scoped db). - `describeLiveDataPlane` — additionally requires the project's own Postgres instance to be `ACTIVE_HEALTHY`; use for commands that talk to the project's data plane (migration, db, storage). - **Invocation:** use `runSupabaseLive(args, options?)` (wraps `runSupabase` with the `legacy` entrypoint and the live profile/timeout defaults) rather than calling `runSupabase` directly, so every live test picks up the same environment plumbing. + - **Environment:** attached mode is the default and preserves the local Supabox contract (`SUPABASE_PROFILE=supabase-local`, `SUPABASE_LIVE_API_URL=http://localhost:8080`, `SUPABASE_LIVE_PROJECT_REF`, and optional data-plane keys). It only reads the existing project and never deletes it. Set `SUPABASE_LIVE_MODE=managed` for an explicit staging run; global setup then provisions one uniquely named project, shares it across files, and deletes exactly that project at teardown. See `live.env.example` for the full contract. +- **Assertion scope:** each file should assert one command's golden path. Setup and teardown may call other commands to prepare or clean state, but those calls are not assertions. Keep the suite focused on one common success workflow per command; use integration tests for exhaustive branches, validation, formatting, errors, and matrices. - **Local-dev-stack live tests** (`start`/`stop`/`status`, and anything else that manages real Docker containers rather than calling the Management API) follow the same file/gating convention but don't need `SUPABASE_PROFILE`/project-ref machinery. Pattern: `mkdtemp` a project dir, `runSupabaseLive(["init"], { cwd })` to generate a real `config.toml`, `runSupabaseLive(["start", ...])` to bring up (a lightweight subset of) the real stack, exercise the command under test, then clean up in `afterEach` (best-effort `stop --no-backup` + `rm` the temp dir) so a failed assertion never leaks containers onto the CI runner. See `commands/stop/stop.live.test.ts` and `commands/status/status.live.test.ts` for the canonical example. - **Keep the suite small and golden-path only** — same philosophy as `*.e2e.test.ts`, but even more so given the cost of a real backend. One or two scenarios per command is normal; branch-by-branch coverage belongs in `*.integration.test.ts`. - Timeouts are generous by default (`testTimeout`/`hookTimeout: 300_000` for the whole `live` project) because real platform/Docker operations are slow — pass an explicit per-`test()` timeout when a scenario needs less (or, for a real local-stack `start`, close to the full budget). diff --git a/apps/cli/live.env.example b/apps/cli/live.env.example new file mode 100644 index 0000000000..4b83949a5f --- /dev/null +++ b/apps/cli/live.env.example @@ -0,0 +1,37 @@ +# Live CLI e2e environment. The suite is in the `live` Vitest project and is +# intentionally separate from the replay/record harness in apps/cli-e2e. + +# Required in both modes. The default attached mode never creates or deletes a +# project; it uses the project supplied by the local Supabox/stack harness. +SUPABASE_ACCESS_TOKEN=sbp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +SUPABASE_LIVE_PROJECT_REF=abcdefghijklmnopqrst + +# Attached mode (default): point these at a running Supabox/local platform. +SUPABASE_LIVE_MODE=attached +SUPABASE_PROFILE=supabase-local +SUPABASE_LIVE_API_URL=http://localhost:8080 +SUPABASE_LIVE_PROJECT_HOST=supabase.red +NODE_EXTRA_CA_CERTS=/path/to/supabox/ca.pem + +# Optional attached data-plane values. Project-scoped tests skip when a value +# they need is unavailable. +# SUPABASE_LIVE_ANON_KEY=eyJ... +# SUPABASE_LIVE_FUNCTIONS_URL=https://abcdefghijklmnopqrst.supabase.red/functions/v1 +# SUPABASE_LIVE_DB_URL=postgres://... +# SUPABASE_LIVE_DB_PASSWORD=... +# SUPABASE_LIVE_STORAGE_BUCKET=supabase-cli-live-bucket + +# Managed mode (explicit opt-in): provision one staging project for the whole +# Vitest run, then delete exactly that project during global teardown. +# SUPABASE_LIVE_MODE=managed +# SUPABASE_PROFILE=supabase-staging +# SUPABASE_LIVE_API_URL=https://api.supabase.green +# SUPABASE_LIVE_PROJECT_HOST=supabase.red +# SUPABASE_LIVE_ORG_ID=... +# SUPABASE_LIVE_REGION=us-east-1 +# SUPABASE_LIVE_PROJECT_NAME=supabase-cli-live +# SUPABASE_LIVE_DB_PASSWORD= # optional; generated randomly when unset +# SUPABASE_LIVE_KEEP_PROJECT=1 # debugging only; skips managed teardown + +# Run with no external harness: +# pnpm test:live diff --git a/.github/scripts/sweep-live-projects.sh b/apps/cli/scripts/sweep-live-projects.sh similarity index 80% rename from .github/scripts/sweep-live-projects.sh rename to apps/cli/scripts/sweep-live-projects.sh index 19456281ed..39774ff381 100755 --- a/.github/scripts/sweep-live-projects.sh +++ b/apps/cli/scripts/sweep-live-projects.sh @@ -3,18 +3,18 @@ # e2e job's per-run prefix). Shared by the in-run retry sweep (called best-effort # with `|| true`) and the always() cleanup step (which propagates the exit code). # -# Reads SUPABASE_ACCESS_TOKEN + CLI_E2E_API_URL from the environment. Exits +# Reads SUPABASE_ACCESS_TOKEN + SUPABASE_LIVE_API_URL from the environment. Exits # non-zero if any DELETE failed; a failed *listing* also exits non-zero (pipefail). set -o pipefail PREFIX="${1:?usage: sweep-live-projects.sh PREFIX}" : "${SUPABASE_ACCESS_TOKEN:?SUPABASE_ACCESS_TOKEN required}" -: "${CLI_E2E_API_URL:?CLI_E2E_API_URL required}" +: "${SUPABASE_LIVE_API_URL:?SUPABASE_LIVE_API_URL required}" # Capture the list in a var (not a pipe-to-while subshell) so a failed delete is # recorded in $failed; a failed listing aborts here via pipefail. refs=$(curl -fsS -H "Authorization: Bearer ${SUPABASE_ACCESS_TOKEN}" \ - "${CLI_E2E_API_URL}/v1/projects" \ + "${SUPABASE_LIVE_API_URL}/v1/projects" \ | jq -r --arg p "$PREFIX" '.[] | select(.name|startswith($p)) | .ref // .id') failed=0 @@ -22,7 +22,7 @@ for ref in $refs; do [ -n "$ref" ] || continue echo "deleting leftover project $ref" if ! curl -fsS -X DELETE -H "Authorization: Bearer ${SUPABASE_ACCESS_TOKEN}" \ - "${CLI_E2E_API_URL}/v1/projects/${ref}" >/dev/null; then + "${SUPABASE_LIVE_API_URL}/v1/projects/${ref}" >/dev/null; then echo "::error::failed to delete leftover project $ref" failed=1 fi diff --git a/apps/cli/src/legacy/commands/start/start.live.test.ts b/apps/cli/src/legacy/commands/start/start.live.test.ts index fcfd71f2a0..b369b0cc41 100644 --- a/apps/cli/src/legacy/commands/start/start.live.test.ts +++ b/apps/cli/src/legacy/commands/start/start.live.test.ts @@ -61,7 +61,7 @@ function splitNonEmptyLines(text: string): ReadonlyArray { // `start` is the one local-dev-stack command whose correctness genuinely // depends on a real Docker daemon — real label filtering and real container // lifecycle, not just CLI exit codes. `describeLive` is reused purely as the -// "we're in the full cli-e2e-ci runner" signal (see stop.live.test.ts's own +// "we're in a configured live runner" signal (see stop.live.test.ts's own // comment for why this, not a Management-API gate, is correct here). See // AGENTS.md's "Live tests" section for the full convention. describeLive("supabase start (live)", () => { diff --git a/apps/cli/tests/helpers/live-context.ts b/apps/cli/tests/helpers/live-context.ts index cb423e92b4..880c200487 100644 --- a/apps/cli/tests/helpers/live-context.ts +++ b/apps/cli/tests/helpers/live-context.ts @@ -1,4 +1,4 @@ -import { mkdtempSync, readFileSync, rmSync, cpSync, appendFileSync } from "node:fs"; +import { mkdtempSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -182,20 +182,6 @@ export function requireLiveSuccess( } } -/** Layer deploy-e2e function fixtures onto the generated workspace config. */ -export function seedFunctions( - workspacePath: string, - sourceDirectory: string, - configSnippet: string, -): void { - const supabaseDirectory = path.join(workspacePath, "supabase"); - cpSync(sourceDirectory, supabaseDirectory, { recursive: true }); - appendFileSync( - path.join(supabaseDirectory, "config.toml"), - `\n${readFileSync(configSnippet, "utf8")}`, - ); -} - export function expectFunctionOk( result: InvokeResult, slug: string, diff --git a/apps/cli/tests/helpers/live-env.ts b/apps/cli/tests/helpers/live-env.ts index bd74c29369..1db8fd5a50 100644 --- a/apps/cli/tests/helpers/live-env.ts +++ b/apps/cli/tests/helpers/live-env.ts @@ -6,7 +6,8 @@ * the test-facing pieces (`describeLive`, `runSupabaseLive`, …) live in * `live.ts` and re-export these. * - * Environment contract (provided by the cli-e2e-ci runner): + * Environment contract (provided by the managed-staging workflow or an + * attached Supabox/local runner): * - `SUPABASE_ACCESS_TOKEN` — required; the platform PAT (supabox seeds a * deterministic `sbp_…` token into its mgmt-api database). * - `SUPABASE_PROFILE` — selects the API base URL; defaults to `supabase-local` @@ -15,8 +16,8 @@ * - `SUPABASE_LIVE_API_URL` — base URL the readiness check probes; attached * mode defaults to `http://localhost:8080`, managed staging defaults to * `https://api.supabase.green`. - * - `SUPABASE_LIVE_PROJECT_REF` — a provisioned project; gates project-scoped - * suites (functions, branches, db, storage). + * - `SUPABASE_LIVE_PROJECT_REF` — the shared project; gates project-scoped + * suites (functions, branches, db, storage). Managed setup populates it. * - `NODE_EXTRA_CA_CERTS` — trusts the supabox CA for `*.supabase.red` TLS; * inherited by the subprocess via the parent environment. */ @@ -128,7 +129,7 @@ export function requireLiveProjectRef(): string { if (!ref) { throw new Error( "SUPABASE_LIVE_PROJECT_REF must be set for project-scoped live tests " + - "(the cli-e2e-ci runner sets it after provisioning a project).", + "(managed global setup sets it after provisioning a project).", ); } return ref; @@ -136,13 +137,9 @@ export function requireLiveProjectRef(): string { /** * Whether the live project's *data-plane* — its own Postgres instance — is up - * and healthy. This is a stronger gate than `liveProjectRef()`: cli-e2e-ci - * currently builds the stack WITHOUT `supabase-postgres-17` (CLI-1825), so a - * provisioned project's *record* exists — Management-API reads (orgs / projects - * / functions / branches list) work — but the instance never reaches - * `ACTIVE_HEALTHY` and its database is unreachable. Commands that talk to the - * project Postgres (migration, db, storage) gate on this and SKIP until the full - * stack lands, then activate automatically. + * and healthy. A control-plane-only platform may expose a project record without + * a reachable database; commands that talk to project Postgres (migration, db, + * storage) gate on this and SKIP until the data plane is ready. * * Probes `GET /v1/projects` (already proven reachable by `projects list`) and * matches the live ref. Any failure or missing prerequisite returns `false` — diff --git a/apps/cli/tests/helpers/live.ts b/apps/cli/tests/helpers/live.ts index 3ebb5118fe..dd591a84be 100644 --- a/apps/cli/tests/helpers/live.ts +++ b/apps/cli/tests/helpers/live.ts @@ -13,7 +13,7 @@ import { /** * Test-facing helpers for the `live` Vitest project (`*.live.test.ts`): * black-box CLI subprocess tests that run against a *real* Supabase platform — - * in CI a local supabox stack (see the `supabase/cli-e2e-ci` harness). + * in CI a local Supabox stack or a managed staging project. * * This module imports Vitest test APIs (`describe`), so it must NOT be imported * from `globalSetup` (Vitest evaluates that in a different context). The @@ -41,8 +41,8 @@ export { /** * `describe` that runs only when the live environment is configured. Use this - * for every live suite so the file is inert (skipped, not failed) outside the - * cli-e2e-ci runner. + * for every live suite so the file is inert (skipped, not failed) outside a + * configured live environment. */ export const describeLive = describe.skipIf(!isLiveConfigured()); @@ -59,7 +59,7 @@ function hasDockerDaemon(): boolean { * `describe` for local-stack live tests that additionally require a reachable * Docker daemon. Composes the configured-live gate (`isLiveConfigured`) with a * `docker info` probe so these suites stay inert (skipped, not failed) outside - * the cli-e2e-ci runner — a machine that merely exposes Docker must never + * a configured live runner — a machine that merely exposes Docker must never * launch a real stack just by collecting the live Vitest project. The * synchronous read-only probe runs once when this helper module is collected, * and only when the live environment is configured. @@ -69,16 +69,14 @@ export const describeDockerLive = describe.skipIf(!isLiveConfigured() || !hasDoc /** * `describe` for project-scoped live suites: runs only when the live env is * configured AND a project ref is available. On a control-plane-only stack - * (e.g. local macOS where project instances can't be built) these skip rather - * than fail. See `requireLiveProjectRef`. + * these skip rather than fail. See `requireLiveProjectRef`. */ export const describeLiveProject = describe.skipIf(!isLiveConfigured() || !liveProjectRef()); /** * `describe` for data-plane live suites (migration / db / storage): runs only * when the live env is configured AND the project's own Postgres instance is - * `ACTIVE_HEALTHY`. On a control-plane-only stack — including the current - * cli-e2e-ci CI, which omits `supabase-postgres-17` (CLI-1825) — the project DB + * `ACTIVE_HEALTHY`. On a control-plane-only stack where the project DB * is unreachable, so these SKIP rather than fail. They activate automatically * once the full data-plane is provisioned. The readiness probe runs once at * collection time (top-level await); see `liveProjectDataPlaneReady`. diff --git a/docs/adr/0013-live-e2e-bypasses-replay-server.md b/docs/adr/0013-live-e2e-bypasses-replay-server.md index 9b0c758818..49d6b24e15 100644 --- a/docs/adr/0013-live-e2e-bypasses-replay-server.md +++ b/docs/adr/0013-live-e2e-bypasses-replay-server.md @@ -14,9 +14,9 @@ Management API calls and the real Docker bundler (e.g. `functions deploy`) — i therefore untested. [CLI-1630](https://linear.app/supabase/issue/CLI-1630/set-up-proper-live-e2e-tests-for-the-cli) -adds a structured Vitest **live** suite that runs the real CLI against a real -backend (staging today, the dockerized `supabox` stack later) as a non-blocking -smoke test before a stable deploy. +adds a structured Vitest **live** suite under `apps/cli` that runs the real CLI +against a real backend (managed staging or an attached Supabox/local stack) as +a non-blocking smoke test before a stable deploy. The open architectural question was *how* live mode should reach the backend. The first instinct was to add a third runtime mode inside `replay-server.ts` @@ -29,16 +29,21 @@ existing Docker and storage proxies "for free." Live mode **does not route through the replay server**. It is a harness-wiring mode, not a `replay-server.ts` branch. -- Live tests reuse `createHarness`/`exec` from `@supabase/cli-test-helpers`, but - the harness is wired **directly**: `apiUrl = CLI_E2E_API_URL` (the real - Management API) and `DOCKER_HOST` points at the **real Docker socket**. +- Live tests are collocated with their command under `apps/cli/src/**` as + `*.live.test.ts` files and use `runSupabaseLive`, which wires the CLI directly + to the selected `SUPABASE_PROFILE` and real Docker socket. +- Global setup provisions one shared environment per Vitest run. Attached mode + (the default) probes the existing Supabox/local platform and never mutates or + deletes its project. Explicit managed mode (`SUPABASE_LIVE_MODE=managed`) + provisions one uniquely named staging project, shares it across tests, and + deletes exactly that project during teardown unless explicitly kept. - `replay-server.ts` is untouched — no `live` branch, no live Docker or storage proxy. - Assertions are **outcome-based**, modeled on the manual deploy playbook: 1. run the real CLI (`run([...])`) and assert `exitCode` / `stdout`; 2. **invoke the deployed function over HTTP directly** and assert HTTP status + the JSON body the function itself returns (e.g. `{case, ok:true}`). - The invoke is a direct HTTP call to `https://{ref}.{CLI_E2E_PROJECT_HOST}/functions/v1`, + The invoke is a direct HTTP call to `https://{ref}.{SUPABASE_LIVE_PROJECT_HOST}/functions/v1`, not a proxied call — the replay server is nowhere in the assertion path. - Because the assertion target is the function's own deterministic response (plus exit codes / stdout substrings), the suite is **ID-agnostic** — no response @@ -46,11 +51,10 @@ mode, not a `replay-server.ts` branch. anon key are resolved at setup from the freshly created project (anon key via `GET /v1/projects/{ref}/api-keys`). -The CLI target is a CI **matrix axis** (`CLI_HARNESS_TARGET`): each target runs -as its own job with `fail-fast: false`, so each implementation is independently -green/red. The pilot covers `go` (raw Go binary) and `ts-legacy` (the TS rewrite -that shells out to Go for most commands and runs native TS logic for ported -ones); `ts-next` is a later axis. +The live CI currently targets the shipped `ts-legacy` shell. A live file asserts +one command's golden path; setup and teardown may invoke other commands without +asserting them. This keeps cross-command scenarios for a later phase while the +suite grows toward one representative live test per command. ## Rationale @@ -75,9 +79,9 @@ later live command genuinely needs host rewriting (e.g. storage on a different host than the Management API), a scoped passthrough can be introduced *then* for that command — YAGNI until a concrete need exists. -The per-target matrix exists because `go` and `ts-legacy` are different code -paths reaching the same backend; running them as separate jobs gives two -independent green signals instead of one averaged result. +The attached/managed environment contract keeps local Supabox and staging +runs interchangeable: switch `SUPABASE_PROFILE` and the `SUPABASE_LIVE_*` +values without changing test code. ## Consequences @@ -87,11 +91,12 @@ independent green signals instead of one averaged result. guards. The Docker bundler talks to the real daemon as users' machines do. - `replay-server.ts` and the replay/record contract are unchanged, so the PR-blocking `e2e` suite is unaffected. -- Tests are trivial to add: drop a `deploy-e2e-foo` fixture function returning a - known body, add one `testLive` that runs deploy → invoke → asserts body. -- Retargeting from staging to `supabox` is genuinely an env swap - (`CLI_E2E_TARGET_ENV` + `CLI_E2E_API_URL` + `CLI_E2E_PROJECT_HOST` + token), - because assertions key off function output, not hostnames. +- Tests are trivial to add: colocate one `*.live.test.ts` next to the command, + prepare any state with unasserted setup commands, then assert the target + command's real outcome. +- Retargeting from staging to Supabox/local is an environment swap + (`SUPABASE_PROFILE` + `SUPABASE_LIVE_API_URL` + `SUPABASE_LIVE_PROJECT_HOST` + + token), because assertions key off behavior, not hostnames. ### Negative @@ -114,12 +119,13 @@ independent green signals instead of one averaged result. assertions on function bodies are naturally ID-agnostic; a scoped normalizer is added only if a future case makes CLI diagnostic output itself the assertion target. -3. **Single CLI target**: rejected. `go` and `ts-legacy` are distinct - implementations of the same commands; one job would hide a regression in - whichever target was not chosen. +3. **Keep live tests in `apps/cli-e2e`**: rejected. The replay/record harness and + its fixtures stay isolated there, while command live tests belong with the + command implementation and share the CLI package's global setup. 4. **One shared long-lived staging project**: rejected. State would leak between - runs and overlapping runs would collide; ephemeral per-job projects with - scoped teardown keep runs isolated. + runs and overlapping runs would collide; managed mode owns one project per + run with scoped teardown, while attached mode leaves the caller's project + untouched. ## Related Decisions @@ -130,4 +136,6 @@ independent green signals instead of one averaged result. ## See Also -- [cli-e2e harness](../../apps/cli-e2e/AGENTS.md) +- [CLI live-test guidance](../../apps/cli/AGENTS.md) +- [Live environment example](../../apps/cli/live.env.example) +- [Replay/record harness](../../apps/cli-e2e/AGENTS.md) From dc56c30f491684733b7f3300be5f7bb3cd1569f2 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 18:03:46 +0200 Subject: [PATCH 07/23] test(cli): harden live environment gates --- apps/cli/AGENTS.md | 6 ++-- apps/cli/live.env.example | 9 ++++-- .../legacy/commands/db/diff/diff.live.test.ts | 9 +++--- .../legacy/commands/db/dump/dump.live.test.ts | 26 ++++++++++++----- .../legacy/commands/db/pull/pull.live.test.ts | 28 +++++++++++++++---- .../legacy/commands/db/push/push.live.test.ts | 21 ++++++++++---- .../commands/db/reset/reset.live.test.ts | 21 ++++++++++---- .../shared/legacy-pgdelta-next.live.test.ts | 7 +++-- .../commands/gen/types/types.live.test.ts | 24 ++++++++++++---- .../inspect/db/db-stats/db-stats.live.test.ts | 23 +++++++++++---- .../commands/migration/list/list.live.test.ts | 28 +++++++++++++------ .../legacy/commands/start/start.live.test.ts | 9 +++--- .../commands/status/status.live.test.ts | 5 ++-- .../legacy/commands/stop/stop.live.test.ts | 11 ++++---- apps/cli/tests/helpers/live-context.ts | 20 +++++++++---- .../0013-live-e2e-bypasses-replay-server.md | 8 ++++-- 16 files changed, 180 insertions(+), 75 deletions(-) diff --git a/apps/cli/AGENTS.md b/apps/cli/AGENTS.md index 8a9f7834fd..e56cb60e12 100644 --- a/apps/cli/AGENTS.md +++ b/apps/cli/AGENTS.md @@ -497,11 +497,11 @@ Live tests are black-box CLI subprocess tests — like `*.e2e.test.ts`, but run - `describeLive` — runs whenever `SUPABASE_ACCESS_TOKEN` is set (the live env is configured at all). Reuse this even for commands that don't call the Management API themselves (e.g. `stop`/`status`) — it doubles as the "we're in the configured live runner" signal. - `describeDockerLive` — the configured-live gate composed with a `docker info` probe; use for local-stack suites whose scenarios additionally need a reachable Docker daemon at collection time. It never runs on a machine that merely exposes Docker — `SUPABASE_ACCESS_TOKEN` must still be set, so the file stays inert outside a configured live runner like every other live suite. - `describeLiveProject` — additionally requires a provisioned project (`SUPABASE_LIVE_PROJECT_REF`); use for project-scoped Management API commands (branches, functions, project-scoped db). - - `describeLiveDataPlane` — additionally requires the project's own Postgres instance to be `ACTIVE_HEALTHY`; use for commands that talk to the project's data plane (migration, db, storage). + - `testLiveDataPlane` / `testLiveDestructiveDataPlane` — fixture gates for data-plane commands; the latter additionally requires `SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1` in attached mode because its setup/cleanup resets remote state. - **Invocation:** use `runSupabaseLive(args, options?)` (wraps `runSupabase` with the `legacy` entrypoint and the live profile/timeout defaults) rather than calling `runSupabase` directly, so every live test picks up the same environment plumbing. - - **Environment:** attached mode is the default and preserves the local Supabox contract (`SUPABASE_PROFILE=supabase-local`, `SUPABASE_LIVE_API_URL=http://localhost:8080`, `SUPABASE_LIVE_PROJECT_REF`, and optional data-plane keys). It only reads the existing project and never deletes it. Set `SUPABASE_LIVE_MODE=managed` for an explicit staging run; global setup then provisions one uniquely named project, shares it across files, and deletes exactly that project at teardown. See `live.env.example` for the full contract. + - **Environment:** attached mode is the default and preserves the local Supabox contract (`SUPABASE_PROFILE=supabase-local`, `SUPABASE_LIVE_API_URL=http://localhost:8080`, `SUPABASE_LIVE_PROJECT_REF`, and optional data-plane keys). It only reads the existing project and never deletes it. Destructive remote DB tests are opt-in with `SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1`. Set `SUPABASE_LIVE_MODE=managed` for an explicit staging run; global setup then provisions one uniquely named project, shares it across files, and deletes exactly that project at teardown. See `live.env.example` for the full contract. - **Assertion scope:** each file should assert one command's golden path. Setup and teardown may call other commands to prepare or clean state, but those calls are not assertions. Keep the suite focused on one common success workflow per command; use integration tests for exhaustive branches, validation, formatting, errors, and matrices. -- **Local-dev-stack live tests** (`start`/`stop`/`status`, and anything else that manages real Docker containers rather than calling the Management API) follow the same file/gating convention but don't need `SUPABASE_PROFILE`/project-ref machinery. Pattern: `mkdtemp` a project dir, `runSupabaseLive(["init"], { cwd })` to generate a real `config.toml`, `runSupabaseLive(["start", ...])` to bring up (a lightweight subset of) the real stack, exercise the command under test, then clean up in `afterEach` (best-effort `stop --no-backup` + `rm` the temp dir) so a failed assertion never leaks containers onto the CI runner. See `commands/stop/stop.live.test.ts` and `commands/status/status.live.test.ts` for the canonical example. +- **Local-dev-stack live tests** (`start`/`stop`/`status`, and anything else that manages real Docker containers rather than calling the Management API) follow the same file/gating convention but don't need `SUPABASE_PROFILE`/project-ref machinery. Pattern: `mkdtemp` a project dir, run `init` and any bootstrap `start`/`status` commands with diagnostic failures but no Vitest assertions, exercise the command under test, then clean up in `afterEach` (best-effort `stop --no-backup` + `rm` the temp dir) so a failed assertion never leaks containers onto the CI runner. See `commands/stop/stop.live.test.ts` and `commands/status/status.live.test.ts` for the canonical example. - **Keep the suite small and golden-path only** — same philosophy as `*.e2e.test.ts`, but even more so given the cost of a real backend. One or two scenarios per command is normal; branch-by-branch coverage belongs in `*.integration.test.ts`. - Timeouts are generous by default (`testTimeout`/`hookTimeout: 300_000` for the whole `live` project) because real platform/Docker operations are slow — pass an explicit per-`test()` timeout when a scenario needs less (or, for a real local-stack `start`, close to the full budget). diff --git a/apps/cli/live.env.example b/apps/cli/live.env.example index 4b83949a5f..41312fff05 100644 --- a/apps/cli/live.env.example +++ b/apps/cli/live.env.example @@ -13,13 +13,16 @@ SUPABASE_LIVE_API_URL=http://localhost:8080 SUPABASE_LIVE_PROJECT_HOST=supabase.red NODE_EXTRA_CA_CERTS=/path/to/supabox/ca.pem -# Optional attached data-plane values. Project-scoped tests skip when a value -# they need is unavailable. +# Optional attached data-plane values. Read-only database tests fall back to +# `--linked --project-ref` when DB_URL is omitted. Destructive remote database +# tests additionally require SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1. # SUPABASE_LIVE_ANON_KEY=eyJ... -# SUPABASE_LIVE_FUNCTIONS_URL=https://abcdefghijklmnopqrst.supabase.red/functions/v1 +# SUPABASE_LIVE_FUNCTIONS_URL=https://abcdefghijklmnopqrst.supabase.red/functions/v1 # optional; derived from project ref when omitted # SUPABASE_LIVE_DB_URL=postgres://... # SUPABASE_LIVE_DB_PASSWORD=... # SUPABASE_LIVE_STORAGE_BUCKET=supabase-cli-live-bucket +# Destructive remote DB tests (db push/pull/reset) are opt-in when attached. +# SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1 # Managed mode (explicit opt-in): provision one staging project for the whole # Vitest run, then delete exactly that project during global teardown. diff --git a/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts index 78614d9aa7..28a6dea73c 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts @@ -7,6 +7,7 @@ import { promisify } from "node:util"; import { afterEach, expect, test } from "vitest"; import { describeLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess } from "../../../../../tests/helpers/live-context.ts"; const execFileAsync = promisify(execFile); @@ -46,7 +47,7 @@ describeLive("supabase db diff (live, pg-delta declarative privileges)", () => { projectDir = await mkdtemp(path.join(tmpdir(), "sb-db-diff-live-")); const init = await runSupabaseLive(["init"], { cwd: projectDir }); - expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + requireLiveSuccess(init, "init setup"); // `init`'s template already enables pg-delta by default (CLI-1877/#5511), but // point `[db.migrations] schema_paths` at a declarative schema directory so @@ -83,7 +84,7 @@ revoke execute on function public.probe_fn() from public; ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - expect(start.exitCode, `stdout:\n${start.stdout}\nstderr:\n${start.stderr}`).toBe(0); + requireLiveSuccess(start, "start setup"); const diff = await runSupabaseLive( ["db", "diff", "--local", "--use-pg-delta", "-f", "revoke_public_execute"], @@ -161,7 +162,7 @@ describeLive("supabase db diff (live, --use-pgadmin native differ container)", ( projectId = path.basename(projectDir); const init = await runSupabaseLive(["init"], { cwd: projectDir }); - expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + requireLiveSuccess(init, "init setup"); // Exclude the heaviest, least relevant services — `db diff --use-pgadmin` only // needs the local Postgres container reachable, same rationale as stop/status. @@ -169,7 +170,7 @@ describeLive("supabase db diff (live, --use-pgadmin native differ container)", ( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - expect(start.exitCode, `stdout:\n${start.stdout}\nstderr:\n${start.stderr}`).toBe(0); + requireLiveSuccess(start, "start setup"); const diff = await runSupabaseLive(["db", "diff", "--use-pgadmin"], { cwd: projectDir, diff --git a/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts b/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts index d99a6d75a1..121f0d5978 100644 --- a/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts +++ b/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts @@ -2,11 +2,23 @@ import { existsSync } from "node:fs"; import { join } from "node:path"; import { expect } from "vitest"; -import { testLiveDataPlane } from "../../../../../tests/helpers/live-context.ts"; +import { + liveDatabaseTargetArgs, + testLiveDataPlane, +} from "../../../../../tests/helpers/live-context.ts"; -testLiveDataPlane("dumps the remote schema to a file", async ({ run, dbUrl, workspace }) => { - const outFile = join(workspace.path, "schema.sql"); - const result = await run(["db", "dump", "--db-url", dbUrl, "-f", outFile]); - expect(result.exitCode, result.stderr).toBe(0); - expect(existsSync(outFile)).toBe(true); -}); +testLiveDataPlane( + "dumps the remote schema to a file", + async ({ run, dbUrl, projectRef, workspace }) => { + const outFile = join(workspace.path, "schema.sql"); + const result = await run([ + "db", + "dump", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "-f", + outFile, + ]); + expect(result.exitCode, result.stderr).toBe(0); + expect(existsSync(outFile)).toBe(true); + }, +); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts index 133d32f219..13de8d74c4 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts @@ -3,13 +3,14 @@ import { join } from "node:path"; import { expect } from "vitest"; import { + liveDatabaseTargetArgs, requireLiveSuccess, - testLiveDataPlane, + testLiveDestructiveDataPlane, } from "../../../../../tests/helpers/live-context.ts"; -testLiveDataPlane( +testLiveDestructiveDataPlane( "pulls the remote schema after a local migration is applied", - async ({ run, dbUrl, workspace }) => { + async ({ run, dbUrl, projectRef, workspace }) => { const version = `${Date.now()}${Math.floor(Math.random() * 10_000) .toString() .padStart(4, "0")}`; @@ -19,17 +20,32 @@ testLiveDataPlane( await writeFile(migrationFile, `create table if not exists e2e_pull_${version} (id int);\n`); try { - const pushed = await run(["db", "push", "--db-url", dbUrl, "--yes"]); + const pushed = await run([ + "db", + "push", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--yes", + ]); requireLiveSuccess(pushed, "db push setup"); - const result = await run(["db", "pull", "--db-url", dbUrl, "--yes"]); + const result = await run([ + "db", + "pull", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--yes", + ]); expect(result.exitCode, result.stderr).toBe(0); expect(`${result.stdout}${result.stderr}`).not.toMatch( /dial|no route|connection refused|could not connect|server closed the connection|i\/o timeout/i, ); } finally { await unlink(migrationFile).catch(() => undefined); - const reset = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); + const reset = await run([ + "db", + "reset", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--yes", + ]); requireLiveSuccess(reset, "db reset cleanup after db pull"); } }, diff --git a/apps/cli/src/legacy/commands/db/push/push.live.test.ts b/apps/cli/src/legacy/commands/db/push/push.live.test.ts index 7f2751cb0d..d6f846deda 100644 --- a/apps/cli/src/legacy/commands/db/push/push.live.test.ts +++ b/apps/cli/src/legacy/commands/db/push/push.live.test.ts @@ -3,13 +3,14 @@ import { join } from "node:path"; import { expect } from "vitest"; import { + liveDatabaseTargetArgs, requireLiveSuccess, - testLiveDataPlane, + testLiveDestructiveDataPlane, } from "../../../../../tests/helpers/live-context.ts"; -testLiveDataPlane( +testLiveDestructiveDataPlane( "pushes a local migration to the remote database", - async ({ run, dbUrl, workspace }) => { + async ({ run, dbUrl, projectRef, workspace }) => { const version = `${Date.now()}${Math.floor(Math.random() * 10_000) .toString() .padStart(4, "0")}`; @@ -19,12 +20,22 @@ testLiveDataPlane( await writeFile(migrationFile, `create table if not exists e2e_push_${version} (id int);\n`); try { - const result = await run(["db", "push", "--db-url", dbUrl, "--yes"]); + const result = await run([ + "db", + "push", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--yes", + ]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Finished supabase db push"); } finally { await unlink(migrationFile).catch(() => undefined); - const reset = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); + const reset = await run([ + "db", + "reset", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--yes", + ]); requireLiveSuccess(reset, "db reset cleanup after db push"); } }, diff --git a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts index ee11b3c9f1..34beb90397 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts @@ -3,13 +3,14 @@ import { join } from "node:path"; import { expect } from "vitest"; import { + liveDatabaseTargetArgs, requireLiveSuccess, - testLiveDataPlane, + testLiveDestructiveDataPlane, } from "../../../../../tests/helpers/live-context.ts"; -testLiveDataPlane( +testLiveDestructiveDataPlane( "resets the remote database with local migrations", - async ({ run, dbUrl, workspace }) => { + async ({ run, dbUrl, projectRef, workspace }) => { const version = `${Date.now()}${Math.floor(Math.random() * 10_000) .toString() .padStart(4, "0")}`; @@ -19,12 +20,22 @@ testLiveDataPlane( await writeFile(migrationFile, `create table if not exists e2e_reset_${version} (id int);\n`); try { - const result = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); + const result = await run([ + "db", + "reset", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--yes", + ]); expect(result.exitCode, result.stderr).toBe(0); expect(`${result.stdout}${result.stderr}`).toContain("Resetting remote database"); } finally { await unlink(migrationFile).catch(() => undefined); - const reset = await run(["db", "reset", "--db-url", dbUrl, "--yes"]); + const reset = await run([ + "db", + "reset", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--yes", + ]); requireLiveSuccess(reset, "db reset cleanup"); } }, diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts index d6e5a33c77..3d661eb87a 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts +++ b/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts @@ -5,6 +5,7 @@ import path from "node:path"; import { afterAll, beforeAll, expect, test } from "vitest"; import { describeDockerLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess } from "../../../../../tests/helpers/live-context.ts"; const COMMAND_TIMEOUT_MS = 280_000; const SCENARIO_TIMEOUT_MS = 900_000; @@ -45,7 +46,7 @@ describeDockerLive("pg-delta next local convergence (live)", () => { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS, }); - expect(init.exitCode, commandFailure(init)).toBe(0); + requireLiveSuccess(init, "init setup"); const configPath = path.join(projectDir, "supabase", "config.toml"); const config = readFileSync(configPath, "utf8"); @@ -83,7 +84,7 @@ describeDockerLive("pg-delta next local convergence (live)", () => { ], { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, ); - expect(start.exitCode, commandFailure(start)).toBe(0); + requireLiveSuccess(start, "start setup"); }, COMMAND_TIMEOUT_MS); afterAll(async () => { @@ -124,7 +125,7 @@ describeDockerLive("pg-delta next local convergence (live)", () => { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS, }); - expect(reset.exitCode, commandFailure(reset)).toBe(0); + requireLiveSuccess(reset, "db reset setup"); const converged = await runSupabaseLive(["db", "diff", "--local", "--use-pg-delta"], { cwd: projectDir, diff --git a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts index 4654d921ff..f950c15b30 100644 --- a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts +++ b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts @@ -1,9 +1,21 @@ import { expect } from "vitest"; -import { testLiveDataPlane } from "../../../../../tests/helpers/live-context.ts"; +import { + liveDatabaseTargetArgs, + testLiveDataPlane, +} from "../../../../../tests/helpers/live-context.ts"; -testLiveDataPlane("generates TypeScript types from the remote schema", async ({ run, dbUrl }) => { - const result = await run(["gen", "types", "--db-url", dbUrl, "--lang", "typescript"]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).toMatch(/export type (Database|Json)/); -}); +testLiveDataPlane( + "generates TypeScript types from the remote schema", + async ({ run, dbUrl, projectRef }) => { + const result = await run([ + "gen", + "types", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--lang", + "typescript", + ]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toMatch(/export type (Database|Json)/); + }, +); diff --git a/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts b/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts index 7ea0a19489..3a27a75b65 100644 --- a/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts +++ b/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts @@ -1,9 +1,20 @@ import { expect } from "vitest"; -import { testLiveDataPlane } from "../../../../../../tests/helpers/live-context.ts"; +import { + liveDatabaseTargetArgs, + testLiveDataPlane, +} from "../../../../../../tests/helpers/live-context.ts"; -testLiveDataPlane("reports statistics from the remote database", async ({ run, dbUrl }) => { - const result = await run(["inspect", "db", "db-stats", "--db-url", dbUrl]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).toContain("Database Size"); -}); +testLiveDataPlane( + "reports statistics from the remote database", + async ({ run, dbUrl, projectRef }) => { + const result = await run([ + "inspect", + "db", + "db-stats", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + ]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Database Size"); + }, +); diff --git a/apps/cli/src/legacy/commands/migration/list/list.live.test.ts b/apps/cli/src/legacy/commands/migration/list/list.live.test.ts index cdb6778717..2a29c2ee53 100644 --- a/apps/cli/src/legacy/commands/migration/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/list/list.live.test.ts @@ -1,15 +1,27 @@ import { expect } from "vitest"; -import { testLiveDataPlane } from "../../../../../tests/helpers/live-context.ts"; +import { + liveDatabaseTargetArgs, + testLiveDataPlane, +} from "../../../../../tests/helpers/live-context.ts"; -testLiveDataPlane("lists migrations from the remote database", async ({ run, dbUrl }) => { - const result = await run(["migration", "list", "--db-url", dbUrl]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).not.toContain("Unauthorized"); -}); +testLiveDataPlane( + "lists migrations from the remote database", + async ({ run, dbUrl, projectRef }) => { + const result = await run(["migration", "list", ...liveDatabaseTargetArgs(dbUrl, projectRef)]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).not.toContain("Unauthorized"); + }, +); -testLiveDataPlane("emits migration list as JSON", async ({ run, dbUrl }) => { - const result = await run(["migration", "list", "--db-url", dbUrl, "--output-format", "json"]); +testLiveDataPlane("emits migration list as JSON", async ({ run, dbUrl, projectRef }) => { + const result = await run([ + "migration", + "list", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--output-format", + "json", + ]); expect(result.exitCode, result.stderr).toBe(0); expect(() => JSON.parse(result.stdout)).not.toThrow(); }); diff --git a/apps/cli/src/legacy/commands/start/start.live.test.ts b/apps/cli/src/legacy/commands/start/start.live.test.ts index b369b0cc41..0e647733cf 100644 --- a/apps/cli/src/legacy/commands/start/start.live.test.ts +++ b/apps/cli/src/legacy/commands/start/start.live.test.ts @@ -8,6 +8,7 @@ import { promisify } from "node:util"; import { afterEach, expect, test } from "vitest"; import { describeLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; import { legacySanitizeProjectId, legacyServiceContainerName, @@ -102,7 +103,7 @@ describeLive("supabase start (live)", () => { cwd: projectDir, exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, }); - expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + requireLiveSuccess(init, "init setup"); const start = await runSupabaseLive(startArgs, { cwd: projectDir, @@ -199,7 +200,7 @@ describeLive("supabase start (live)", () => { cwd: projectDir, exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, }); - expect(status.exitCode, `stdout:\n${status.stdout}\nstderr:\n${status.stderr}`).toBe(0); + requireLiveSuccess(status, "status setup"); }, ); @@ -213,7 +214,7 @@ describeLive("supabase start (live)", () => { cwd: projectDir, exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, }); - expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + requireLiveSuccess(init, "init setup"); let proxyConnections = 0; const proxy = createServer((socket) => { @@ -284,7 +285,7 @@ describeLive("supabase start (live)", () => { cwd: projectDir, exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, }); - expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + requireLiveSuccess(init, "init setup"); // A `scratch` image whose entrypoint is not an executable binary — the // kernel refuses it with exactly the "exec format error" this diagnoses. diff --git a/apps/cli/src/legacy/commands/status/status.live.test.ts b/apps/cli/src/legacy/commands/status/status.live.test.ts index 64569c118c..1948715d14 100644 --- a/apps/cli/src/legacy/commands/status/status.live.test.ts +++ b/apps/cli/src/legacy/commands/status/status.live.test.ts @@ -4,6 +4,7 @@ import path from "node:path"; import { afterEach, expect, test } from "vitest"; import { describeLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; const START_TIMEOUT_MS = 280_000; @@ -28,13 +29,13 @@ describeLive("supabase status (live)", () => { projectDir = await mkdtemp(path.join(tmpdir(), "sb-status-live-")); const init = await runSupabaseLive(["init"], { cwd: projectDir }); - expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + requireLiveSuccess(init, "init setup"); const start = await runSupabaseLive( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - expect(start.exitCode, `stdout:\n${start.stdout}\nstderr:\n${start.stderr}`).toBe(0); + requireLiveSuccess(start, "start setup"); const pretty = await runSupabaseLive(["status"], { cwd: projectDir }); expect(pretty.exitCode, `stdout:\n${pretty.stdout}\nstderr:\n${pretty.stderr}`).toBe(0); diff --git a/apps/cli/src/legacy/commands/stop/stop.live.test.ts b/apps/cli/src/legacy/commands/stop/stop.live.test.ts index 12d71ed169..0d1f1fc463 100644 --- a/apps/cli/src/legacy/commands/stop/stop.live.test.ts +++ b/apps/cli/src/legacy/commands/stop/stop.live.test.ts @@ -6,6 +6,7 @@ import { promisify } from "node:util"; import { afterEach, expect, test } from "vitest"; import { describeLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; import { legacySanitizeProjectId } from "../../shared/legacy-docker-ids.ts"; const execFileAsync = promisify(execFile); @@ -42,7 +43,7 @@ describeLive("supabase stop (live)", () => { projectId = path.basename(projectDir); const init = await runSupabaseLive(["init"], { cwd: projectDir }); - expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + requireLiveSuccess(init, "init setup"); // Exclude the heaviest, least relevant services (Next.js Studio build, the // logging pipeline) — `stop`'s Docker label-filtering logic doesn't care @@ -52,11 +53,11 @@ describeLive("supabase stop (live)", () => { ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - expect(start.exitCode, `stdout:\n${start.stdout}\nstderr:\n${start.stderr}`).toBe(0); + requireLiveSuccess(start, "start setup"); // Sanity: confirm the stack is actually up before testing `stop` against it. const before = await runSupabaseLive(["status"], { cwd: projectDir }); - expect(before.exitCode, `stdout:\n${before.stdout}\nstderr:\n${before.stderr}`).toBe(0); + requireLiveSuccess(before, "status setup"); const stop = await runSupabaseLive(["stop"], { cwd: projectDir }); expect(stop.exitCode, `stdout:\n${stop.stdout}\nstderr:\n${stop.stderr}`).toBe(0); @@ -88,13 +89,13 @@ describeLive("supabase stop (live)", () => { projectId = legacySanitizeProjectId(path.basename(projectDir)); const init = await runSupabaseLive(["init"], { cwd: projectDir }); - expect(init.exitCode, `stdout:\n${init.stdout}\nstderr:\n${init.stderr}`).toBe(0); + requireLiveSuccess(init, "init setup"); const start = await runSupabaseLive( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - expect(start.exitCode, `stdout:\n${start.stdout}\nstderr:\n${start.stderr}`).toBe(0); + requireLiveSuccess(start, "start setup"); // `--no-backup` exercises the volume-prune branch; `--debug` turns on // the `Pruned …:` stderr reports, which are diff --git a/apps/cli/tests/helpers/live-context.ts b/apps/cli/tests/helpers/live-context.ts index 880c200487..6c861839f3 100644 --- a/apps/cli/tests/helpers/live-context.ts +++ b/apps/cli/tests/helpers/live-context.ts @@ -150,15 +150,16 @@ export const testLiveProject = base.skipIf(!isLiveConfigured() || !liveProjectRe export const testLiveFunctions = base.skipIf( !isLiveConfigured() || !liveProjectRef() || - (!isManagedLive() && - ((process.env["SUPABASE_LIVE_ANON_KEY"] ?? "").length === 0 || - (process.env["SUPABASE_LIVE_FUNCTIONS_URL"] ?? "").length === 0)), + (!isManagedLive() && (process.env["SUPABASE_LIVE_ANON_KEY"] ?? "").length === 0), ); /** Fixture for scenarios that require the project's Postgres data plane. */ -export const testLiveDataPlane = base.skipIf( +export const testLiveDataPlane = base.skipIf(!(await liveProjectDataPlaneReady())); + +/** Fixture for remote database tests that create/drop transient schema state. */ +export const testLiveDestructiveDataPlane = base.skipIf( !(await liveProjectDataPlaneReady()) || - (!isManagedLive() && (process.env["SUPABASE_LIVE_DB_URL"] ?? "").length === 0), + (!isManagedLive() && process.env["SUPABASE_LIVE_ALLOW_DESTRUCTIVE"] !== "1"), ); /** Fixture for Storage scenarios requiring a linked database password. */ @@ -182,6 +183,15 @@ export function requireLiveSuccess( } } +/** Resolve a remote database target without requiring staging-only pooler data. */ +export function liveDatabaseTargetArgs(dbUrl: string, projectRef: string): string[] { + if (dbUrl.length > 0) return ["--db-url", dbUrl]; + if (projectRef.length === 0) { + throw new Error("A project ref is required when SUPABASE_LIVE_DB_URL is unavailable"); + } + return ["--linked", "--project-ref", projectRef]; +} + export function expectFunctionOk( result: InvokeResult, slug: string, diff --git a/docs/adr/0013-live-e2e-bypasses-replay-server.md b/docs/adr/0013-live-e2e-bypasses-replay-server.md index 49d6b24e15..186e57034c 100644 --- a/docs/adr/0013-live-e2e-bypasses-replay-server.md +++ b/docs/adr/0013-live-e2e-bypasses-replay-server.md @@ -103,9 +103,11 @@ values without changing test code. - Live mode requires a working Docker daemon on the runner (enforced by a `docker info` preflight) — unlike the replay suite, which served Docker fixtures and needed no daemon. -- Each live run provisions and tears down a real staging project, so the suite is - inherently slower and subject to provisioning flake. Mitigated by a CI-level - re-run (up to 3×) rather than in-setup retry. +- Explicit managed runs provision and tear down one real staging project, so those + runs are inherently slower and subject to provisioning flake. The default attached + mode uses the caller-provided Supabox/local project; global setup never deletes it. + Managed provisioning is mitigated by a CI-level re-run (up to 3×) rather than + in-setup retry. - A second wiring path now exists for the same harness (replay-via-server vs live-direct); contributors must know which mode wires the CLI how. From 7b7e06edc319100fb7988d657f45d30cb75fb92b Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 18:07:10 +0200 Subject: [PATCH 08/23] test(cli): use supported gen types target --- .../legacy/commands/gen/types/types.live.test.ts | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts index f950c15b30..cb7cd2fd2a 100644 --- a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts +++ b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts @@ -1,20 +1,12 @@ import { expect } from "vitest"; -import { - liveDatabaseTargetArgs, - testLiveDataPlane, -} from "../../../../../tests/helpers/live-context.ts"; +import { testLiveDataPlane } from "../../../../../tests/helpers/live-context.ts"; testLiveDataPlane( "generates TypeScript types from the remote schema", async ({ run, dbUrl, projectRef }) => { - const result = await run([ - "gen", - "types", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--lang", - "typescript", - ]); + const targetArgs = dbUrl.length > 0 ? ["--db-url", dbUrl] : ["--project-id", projectRef]; + const result = await run(["gen", "types", ...targetArgs, "--lang", "typescript"]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toMatch(/export type (Database|Json)/); }, From 8145ed0fa3dcc200f6317c8113b790c5a2ac46b5 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 19:53:30 +0200 Subject: [PATCH 09/23] test(cli): scope managed live e2e runs --- .github/workflows/live-e2e.yml | 22 +-- apps/cli/AGENTS.md | 8 +- apps/cli/live.env.example | 5 + .../branches/delete/delete.live.test.ts | 27 ++- .../legacy/commands/db/diff/diff.live.test.ts | 176 +++--------------- .../legacy/commands/db/pull/pull.live.test.ts | 40 +++- .../declarative/sync/sync.live.test.ts} | 56 +++--- .../commands/db/start/start.live.test.ts | 4 +- .../migration/fetch/fetch.live.test.ts | 141 +++++++------- .../commands/projects/list/list.live.test.ts | 19 +- .../commands/secrets/unset/unset.live.test.ts | 18 +- .../legacy/commands/start/start.live.test.ts | 6 +- .../commands/status/status.live.test.ts | 6 +- .../legacy/commands/stop/stop.live.test.ts | 6 +- .../commands/functions/dev/dev.live.test.ts | 4 +- .../next/commands/start/start.live.test.ts | 17 +- apps/cli/tests/helpers/live-env.ts | 25 +++ apps/cli/tests/helpers/live-env.unit.test.ts | 52 ++++++ apps/cli/tests/helpers/live.ts | 11 ++ .../0013-live-e2e-bypasses-replay-server.md | 9 +- 20 files changed, 352 insertions(+), 300 deletions(-) rename apps/cli/src/legacy/commands/db/{shared/legacy-pgdelta-next.live.test.ts => schema/declarative/sync/sync.live.test.ts} (75%) create mode 100644 apps/cli/tests/helpers/live-env.unit.test.ts diff --git a/.github/workflows/live-e2e.yml b/.github/workflows/live-e2e.yml index 3b2bc4c6ca..bc996f08f7 100644 --- a/.github/workflows/live-e2e.yml +++ b/.github/workflows/live-e2e.yml @@ -100,6 +100,7 @@ jobs: # steps that need it (run + cleanup) so build/checkout/docker never see it. env: SUPABASE_LIVE_MODE: managed + SUPABASE_LIVE_LOCAL_STACK: "0" SUPABASE_PROFILE: supabase-staging SUPABASE_LIVE_API_URL: https://api.supabase.green SUPABASE_LIVE_PROJECT_HOST: supabase.red @@ -132,26 +133,11 @@ jobs: - name: Docker preflight run: docker info - - name: Run live e2e (retry up to 3x) + - name: Run live e2e + timeout-minutes: 20 env: SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} - run: | - PREFIX="supabase-cli-live-${CLI_HARNESS_TARGET}-${GITHUB_RUN_ID}-" - # GitHub runs this step as `bash -e`; use `if cmd; then` (errexit-exempt) - # so a failing attempt does not abort the step before the retry. - for attempt in 1 2 3; do - echo "::group::live e2e attempt ${attempt}" - if [ "$attempt" -gt 1 ]; then - bash apps/cli/scripts/sweep-live-projects.sh "$PREFIX" || true - fi - if pnpm --filter supabase test:live; then - echo "::endgroup::" - exit 0 - fi - echo "::endgroup::" - echo "attempt ${attempt} failed" - done - exit 1 + run: pnpm --filter supabase test:live # Backstop: delete any project this job created that survived a crash. # The script exits non-zero (failing this step) if any delete failed. diff --git a/apps/cli/AGENTS.md b/apps/cli/AGENTS.md index e56cb60e12..9dc8611f42 100644 --- a/apps/cli/AGENTS.md +++ b/apps/cli/AGENTS.md @@ -494,14 +494,16 @@ Live tests are black-box CLI subprocess tests — like `*.e2e.test.ts`, but run - **Where they run:** authored and executed from this workspace's `live` Vitest project (`pnpm test:live`). CI may run the same files from the [`supabase/cli-e2e-ci`](https://github.com/supabase/cli-e2e-ci) Supabox harness or the managed-staging workflow. They never run as part of the default unit/integration/e2e loop, and locally they no-op unless the live environment is configured (see below). - **Add one whenever you add or change a command whose correctness genuinely depends on a real backend** — a new Management API command, or a change to `start`/`stop`/`status`'s real Docker interaction. Colocate it with the command, same as `*.e2e.test.ts`: `src/legacy/commands//[/].live.test.ts`. - **Gating:** every live suite must be wrapped in one of `tests/helpers/live.ts`'s `describe.skipIf` gates so the file is inert (skipped, not failed) outside a configured live runner: - - `describeLive` — runs whenever `SUPABASE_ACCESS_TOKEN` is set (the live env is configured at all). Reuse this even for commands that don't call the Management API themselves (e.g. `stop`/`status`) — it doubles as the "we're in the configured live runner" signal. - - `describeDockerLive` — the configured-live gate composed with a `docker info` probe; use for local-stack suites whose scenarios additionally need a reachable Docker daemon at collection time. It never runs on a machine that merely exposes Docker — `SUPABASE_ACCESS_TOKEN` must still be set, so the file stays inert outside a configured live runner like every other live suite. + - `describeLive` — runs whenever `SUPABASE_ACCESS_TOKEN` is set (the live environment is configured at all). Use it for account-level and platform/control-plane scenarios that do not require a provisioned project or a local Docker stack. Commands that manage the local dev stack must use `describeLocalStackLive` instead. + - `describeDockerLive` — the configured-live gate composed with a `docker info` probe; use for Docker-dependent remote scenarios such as `functions deploy` that still run under managed staging. It never runs on a machine that merely exposes Docker — `SUPABASE_ACCESS_TOKEN` must still be set, so the file stays inert outside a configured live runner like every other live suite. + - `describeLocalStackLive` — additionally requires `SUPABASE_LIVE_LOCAL_STACK` to be enabled and a reachable Docker daemon. Attached Supabox/local runs default to enabled; managed staging defaults to disabled so it does not launch unrelated local stacks. Set `SUPABASE_LIVE_LOCAL_STACK=1` to opt into those suites, or `0` to disable them explicitly. Invalid values fail loudly. - `describeLiveProject` — additionally requires a provisioned project (`SUPABASE_LIVE_PROJECT_REF`); use for project-scoped Management API commands (branches, functions, project-scoped db). - `testLiveDataPlane` / `testLiveDestructiveDataPlane` — fixture gates for data-plane commands; the latter additionally requires `SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1` in attached mode because its setup/cleanup resets remote state. - **Invocation:** use `runSupabaseLive(args, options?)` (wraps `runSupabase` with the `legacy` entrypoint and the live profile/timeout defaults) rather than calling `runSupabase` directly, so every live test picks up the same environment plumbing. - **Environment:** attached mode is the default and preserves the local Supabox contract (`SUPABASE_PROFILE=supabase-local`, `SUPABASE_LIVE_API_URL=http://localhost:8080`, `SUPABASE_LIVE_PROJECT_REF`, and optional data-plane keys). It only reads the existing project and never deletes it. Destructive remote DB tests are opt-in with `SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1`. Set `SUPABASE_LIVE_MODE=managed` for an explicit staging run; global setup then provisions one uniquely named project, shares it across files, and deletes exactly that project at teardown. See `live.env.example` for the full contract. - **Assertion scope:** each file should assert one command's golden path. Setup and teardown may call other commands to prepare or clean state, but those calls are not assertions. Keep the suite focused on one common success workflow per command; use integration tests for exhaustive branches, validation, formatting, errors, and matrices. -- **Local-dev-stack live tests** (`start`/`stop`/`status`, and anything else that manages real Docker containers rather than calling the Management API) follow the same file/gating convention but don't need `SUPABASE_PROFILE`/project-ref machinery. Pattern: `mkdtemp` a project dir, run `init` and any bootstrap `start`/`status` commands with diagnostic failures but no Vitest assertions, exercise the command under test, then clean up in `afterEach` (best-effort `stop --no-backup` + `rm` the temp dir) so a failed assertion never leaks containers onto the CI runner. See `commands/stop/stop.live.test.ts` and `commands/status/status.live.test.ts` for the canonical example. +- **Local-dev-stack live tests** (`start`/`stop`/`status`, and anything else that manages real Docker containers rather than calling the Management API) use `describeLocalStackLive` and don't need `SUPABASE_PROFILE`/project-ref machinery. Pattern: `mkdtemp` a project dir, run `init` and any bootstrap `start`/`status` commands with diagnostic failures but no Vitest assertions, exercise the command under test, then clean up in `afterEach` (best-effort `stop --no-backup` + `rm` the temp dir) so a failed assertion never leaks containers onto the CI runner. See `commands/stop/stop.live.test.ts` and `commands/status/status.live.test.ts` for the canonical example. +- **Managed CI execution** provisions one shared project per run and invokes `pnpm --filter supabase test:live` once with a 20-minute step timeout. It does not retry the whole suite; deterministic failures should be diagnosed from the first run, while the separate cleanup step removes any leftover managed project. - **Keep the suite small and golden-path only** — same philosophy as `*.e2e.test.ts`, but even more so given the cost of a real backend. One or two scenarios per command is normal; branch-by-branch coverage belongs in `*.integration.test.ts`. - Timeouts are generous by default (`testTimeout`/`hookTimeout: 300_000` for the whole `live` project) because real platform/Docker operations are slow — pass an explicit per-`test()` timeout when a scenario needs less (or, for a real local-stack `start`, close to the full budget). diff --git a/apps/cli/live.env.example b/apps/cli/live.env.example index 41312fff05..c0ea5c4988 100644 --- a/apps/cli/live.env.example +++ b/apps/cli/live.env.example @@ -5,6 +5,9 @@ # project; it uses the project supplied by the local Supabox/stack harness. SUPABASE_ACCESS_TOKEN=sbp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx SUPABASE_LIVE_PROJECT_REF=abcdefghijklmnopqrst +# Local Docker-stack lifecycle suites default on in attached mode and off in +# managed staging. Set explicitly to override the mode default (only 0 or 1). +# SUPABASE_LIVE_LOCAL_STACK=1 # Attached mode (default): point these at a running Supabox/local platform. SUPABASE_LIVE_MODE=attached @@ -35,6 +38,8 @@ NODE_EXTRA_CA_CERTS=/path/to/supabox/ca.pem # SUPABASE_LIVE_PROJECT_NAME=supabase-cli-live # SUPABASE_LIVE_DB_PASSWORD= # optional; generated randomly when unset # SUPABASE_LIVE_KEEP_PROJECT=1 # debugging only; skips managed teardown +# Managed runs do not retry the entire live suite; failures stop the job and +# leave the separate scoped cleanup step to remove any provisioned project. # Run with no external harness: # pnpm test:live diff --git a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts index 876b5f3bb8..182fd7fc35 100644 --- a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts @@ -15,15 +15,36 @@ testLiveProject("deletes a preview branch", async ({ run, projectRef, skip }) => requireLiveSuccess(created, "branches create"); let deleted = false; + let targetError: unknown; + let cleanupError: unknown; try { const removed = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); expect(removed.exitCode, removed.stderr).toBe(0); - expect(removed.stdout).toContain("Deleted preview branch"); deleted = true; + expect(removed.stderr).toContain("Deleted preview branch"); + } catch (error) { + targetError = error; } finally { if (!deleted) { - const cleanup = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); - requireLiveSuccess(cleanup, "branches delete cleanup"); + try { + const cleanup = await run([ + "branches", + "delete", + name, + "--project-ref", + projectRef, + "--yes", + ]); + if (cleanup.exitCode !== 0 && !/not found/i.test(`${cleanup.stdout}\n${cleanup.stderr}`)) { + cleanupError = new Error( + `branches delete cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`, + ); + } + } catch (error) { + cleanupError = error; + } } } + if (targetError !== undefined) throw targetError; + if (cleanupError !== undefined) throw cleanupError; }); diff --git a/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts index 28a6dea73c..0f9ea564ab 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts @@ -1,22 +1,13 @@ -import { execFile } from "node:child_process"; -import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; +import { existsSync, readdirSync, readFileSync } from "node:fs"; import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; -import { promisify } from "node:util"; import { afterEach, expect, test } from "vitest"; -import { describeLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; +import { describeLocalStackLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; import { requireLiveSuccess } from "../../../../../tests/helpers/live-context.ts"; -const execFileAsync = promisify(execFile); - const START_TIMEOUT_MS = 280_000; -// Lifecycle allowance for scenarios that run TWO full-budget subprocesses (`start` -// then the command under test) plus init/inspection overhead — same shape as -// `start.live.test.ts`. A single shared `START_TIMEOUT_MS` test budget would let a -// slow-but-valid `start` starve the command under test before it ever runs. -const LIFECYCLE_OVERHEAD_MS = 90_000; // CLI-1947 regression: pg-delta's `filterPublicBuiltInDefaults()` unconditionally // treated PUBLIC's implicit built-in privilege as a no-op on both sides of a diff, @@ -25,10 +16,10 @@ const LIFECYCLE_OVERHEAD_MS = 90_000; // statement. Fixed upstream in @supabase/pg-delta@1.0.0-alpha.33 // (supabase/pg-toolbelt#357). Verified directly against this repo's build: with // the pre-fix pin (1.0.0-alpha.32) the migration below contains only the CREATE -// FUNCTION statement; the REVOKE is silently absent. `describeLive` is reused as -// the "real local Docker stack is available" signal, same as stop/status — this -// never calls the Management API. See AGENTS.md's "Live tests" section. -describeLive("supabase db diff (live, pg-delta declarative privileges)", () => { +// FUNCTION statement; the REVOKE is silently absent. This suite uses the local +// Docker-stack gate and never calls the Management API. See AGENTS.md's "Live +// tests" section. +describeLocalStackLive("supabase db diff (live, pg-delta declarative privileges)", () => { let projectDir: string | undefined; afterEach(async () => { @@ -49,42 +40,32 @@ describeLive("supabase db diff (live, pg-delta declarative privileges)", () => { const init = await runSupabaseLive(["init"], { cwd: projectDir }); requireLiveSuccess(init, "init setup"); - // `init`'s template already enables pg-delta by default (CLI-1877/#5511), but - // point `[db.migrations] schema_paths` at a declarative schema directory so - // `db diff --local` diffs against it instead of the (empty) local migration - // history. Paths are relative to `supabase/`. - const configPath = path.join(projectDir, "supabase", "config.toml"); - const config = readFileSync(configPath, "utf8"); - expect(config).toContain("schema_paths = []"); - writeFileSync( - configPath, - config.replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]'), + // Exclude the heaviest, least relevant services — `db diff` only needs the + // local Postgres container reachable, same rationale as stop/status. + const start = await runSupabaseLive( + ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], + { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); + requireLiveSuccess(start, "start setup"); - // Minimal, deterministic repro: a fresh function's implicit PUBLIC EXECUTE - // grant, explicitly revoked. Verified empirically against this build: pre-fix - // (pg-delta 1.0.0-alpha.32) the generated migration contains only the CREATE - // FUNCTION statement; the REVOKE is silently dropped. - const schemasDir = path.join(projectDir, "supabase", "schemas"); - mkdirSync(schemasDir, { recursive: true }); - writeFileSync( - path.join(schemasDir, "01_probe_fn.sql"), - `create function public.probe_fn() + // Minimal, deterministic repro: execute a fresh function's implicit PUBLIC + // EXECUTE grant, explicitly revoked, directly against the local database. + // `db query` is setup only; the command under test remains `db diff`. + const query = await runSupabaseLive( + [ + "db", + "query", + `create function public.probe_fn() returns void language sql as $$ select 1; $$; -revoke execute on function public.probe_fn() from public; -`, - ); - - // Exclude the heaviest, least relevant services — `db diff` only needs the - // local Postgres container reachable, same rationale as stop/status. - const start = await runSupabaseLive( - ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], +revoke execute on function public.probe_fn() from public;`, + "--local", + ], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - requireLiveSuccess(start, "start setup"); + requireLiveSuccess(query, "db query setup"); const diff = await runSupabaseLive( ["db", "diff", "--local", "--use-pg-delta", "-f", "revoke_public_execute"], @@ -111,112 +92,3 @@ revoke execute on function public.probe_fn() from public; }, ); }); - -// `--use-pgadmin` is a native `docker run` of the differ container, no -// edge-runtime and no Go delegation involved. Golden-path smoke coverage only — the -// pure filtering/progress logic and the docker-run argv are covered exhaustively by -// `legacy-pgadmin-diff.unit.test.ts` and `diff.integration.test.ts`; this just proves -// the real container actually runs against a real local stack and cleans up after -// itself either way. -// -// The real, reachable outcome here is a FAILURE, not a golden diff, by design: the -// differ container joins the project's own bridge network -// (`supabase_network_`), and both diff endpoints are hardcoded loopback -// URLs from that container's own point of view — `source` (resolving to `127.0.0.1` -// for a local target) and `target` -// (`postgresql://postgres:postgres@127.0.0.1:/postgres`). Inside a -// bridge-attached container, `127.0.0.1` is the container's OWN loopback, not the -// host's — so neither the local db nor the shadow is reachable from inside the -// differ, and the container exits non-zero. See `SIDE_EFFECTS.md`'s "Network -// reachability" entry for the full static ruling. (The historical value-receiver bug -// documented there — always reporting "No schema changes found" regardless of the -// differ's actual output — only ever engages when the differ container exits 0; it -// plays no role in this failure path.) Note that a plain `--network-id host` does NOT -// rescue a golden run here: it also rewires the SHADOW container onto host -// networking, discarding its own `54320->5432` port publish that `target` depends on -// — so `source` would become reachable but `target` would not, still failing the -// diff. This suite therefore verifies the real, always-reachable failure mode -// end-to-end, plus that both the differ AND the shadow container it provisions are -// still cleaned up. -describeLive("supabase db diff (live, --use-pgadmin native differ container)", () => { - let projectDir: string | undefined; - let projectId: string | undefined; - - afterEach(async () => { - if (projectDir === undefined) return; - // Best-effort cleanup even if an assertion above failed mid-lifecycle — a - // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabaseLive(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); - await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); - projectDir = undefined; - projectId = undefined; - }); - - test( - "runs the native differ container against the real stack, surfaces Go's error running container failure, and leaves no differ container behind", - { timeout: START_TIMEOUT_MS * 2 + LIFECYCLE_OVERHEAD_MS }, - async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-db-diff-pgadmin-live-")); - // No `project_id` override, so the cli resolves it from the workdir basename - // (see legacy-docker-ids.ts), same as `stop.live.test.ts`. - projectId = path.basename(projectDir); - - const init = await runSupabaseLive(["init"], { cwd: projectDir }); - requireLiveSuccess(init, "init setup"); - - // Exclude the heaviest, least relevant services — `db diff --use-pgadmin` only - // needs the local Postgres container reachable, same rationale as stop/status. - const start = await runSupabaseLive( - ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, - ); - requireLiveSuccess(start, "start setup"); - - const diff = await runSupabaseLive(["db", "diff", "--use-pgadmin"], { - cwd: projectDir, - exitTimeoutMs: START_TIMEOUT_MS, - }); - // Both hardcoded loopback endpoints are unreachable from inside the - // bridge-attached differ container (see this suite's own header comment for the - // full, static ruling) — the differ exits non-zero and the CLI surfaces its own - // wrapper message. The differ's own exit code isn't pinned: only that the differ - // ran and failed, not the shadow/connection machinery around it. - expect(diff.exitCode, `stdout:\n${diff.stdout}\nstderr:\n${diff.stderr}`).toBe(1); - expect(diff.stderr).toContain("error running container: exit "); - - // The differ is a one-shot `docker run --rm` — real Docker must agree that no - // container survives it, the same "the daemon must agree" check - // `stop.live.test.ts` runs against `com.supabase.cli.project`. - const { stdout: remainingDiffer } = await execFileAsync("docker", [ - "ps", - "-a", - "--filter", - "ancestor=supabase/pgadmin-schema-diff:cli-0.0.5", - "--format", - "{{.ID}}", - ]); - expect(remainingDiffer.trim()).toBe(""); - - // This failure path exercises the shadow's `acquireUseRelease` teardown for - // real (the differ error propagates out of the `use` phase after the shadow was - // already created) — the shadow itself is created with no `--name` (Docker - // auto-generates one), unlike every real stack container, which is always named - // `supabase__`. So a leaked shadow shows up as a - // project-labeled container whose name does NOT carry that fixed prefix. - const { stdout: projectContainers } = await execFileAsync("docker", [ - "ps", - "-a", - "--filter", - `label=com.supabase.cli.project=${projectId}`, - "--format", - "{{.Names}}", - ]); - const names = projectContainers - .trim() - .split("\n") - .filter((name) => name.length > 0); - expect(names.length).toBeGreaterThan(0); - expect(names.every((name) => name.startsWith("supabase_"))).toBe(true); - }, - ); -}); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts index 13de8d74c4..ce40e7c12b 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts @@ -1,4 +1,4 @@ -import { mkdir, unlink, writeFile } from "node:fs/promises"; +import { mkdir, readdir, unlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; @@ -16,9 +16,11 @@ testLiveDestructiveDataPlane( .padStart(4, "0")}`; const migrations = join(workspace.path, "supabase", "migrations"); await mkdir(migrations, { recursive: true }); + const existingMigrations = new Set(await readdir(migrations)); const migrationFile = join(migrations, `${version}_e2e_pull.sql`); await writeFile(migrationFile, `create table if not exists e2e_pull_${version} (id int);\n`); + let targetError: unknown; try { const pushed = await run([ "db", @@ -38,8 +40,31 @@ testLiveDestructiveDataPlane( expect(`${result.stdout}${result.stderr}`).not.toMatch( /dial|no route|connection refused|could not connect|server closed the connection|i\/o timeout/i, ); - } finally { - await unlink(migrationFile).catch(() => undefined); + } catch (error) { + targetError = error; + } + + let cleanupError: unknown; + try { + // Remove all migrations created by this test before resetting. This + // includes both the seed migration and the migration generated by + // `db pull`; resetting with only the generated grant statements left + // behind can reference a table that no longer exists. + const currentMigrations = await readdir(migrations).catch(() => []); + for (const file of currentMigrations.filter( + (candidate) => !existingMigrations.has(candidate), + )) { + try { + await unlink(join(migrations, file)); + } catch (error) { + throw new Error( + `db pull cleanup could not remove test migration ${join(migrations, file)}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + } + const reset = await run([ "db", "reset", @@ -47,6 +72,15 @@ testLiveDestructiveDataPlane( "--yes", ]); requireLiveSuccess(reset, "db reset cleanup after db pull"); + } catch (error) { + cleanupError = error; + } + + if (targetError !== undefined) { + throw targetError; + } + if (cleanupError !== undefined) { + throw cleanupError; } }, ); diff --git a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.live.test.ts similarity index 75% rename from apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts rename to apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.live.test.ts index 3d661eb87a..153690cdd0 100644 --- a/apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next.live.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.live.test.ts @@ -4,8 +4,11 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { afterAll, beforeAll, expect, test } from "vitest"; -import { describeDockerLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; -import { requireLiveSuccess } from "../../../../../tests/helpers/live-context.ts"; +import { + describeLocalStackLive, + runSupabaseLive, +} from "../../../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess } from "../../../../../../../tests/helpers/live-context.ts"; const COMMAND_TIMEOUT_MS = 280_000; const SCENARIO_TIMEOUT_MS = 900_000; @@ -36,7 +39,7 @@ function migrationFiles(projectDir: string): ReadonlyArray { : []; } -describeDockerLive("pg-delta next local convergence (live)", () => { +describeLocalStackLive("db schema declarative sync (live)", () => { let projectDir = ""; beforeAll(async () => { @@ -50,16 +53,15 @@ describeDockerLive("pg-delta next local convergence (live)", () => { const configPath = path.join(projectDir, "supabase", "config.toml"); const config = readFileSync(configPath, "utf8"); - expect(config).toContain("schema_paths = []"); - expect(config).toContain("[experimental.pgdelta]\nenabled = true"); + if (!config.includes("[experimental.pgdelta]\nenabled = true")) { + throw new Error("init setup did not enable experimental pg-delta in config.toml"); + } writeFileSync( configPath, - config - .replace("schema_paths = []", 'schema_paths = ["./schemas/*.sql"]') - .replace( - '# declarative_schema_path = "./schemas"', - 'declarative_schema_path = "./schemas"', - ), + config.replace( + '# declarative_schema_path = "./schemas"', + 'declarative_schema_path = "./schemas"', + ), ); const schemasDir = path.join(projectDir, "supabase", "schemas"); @@ -100,13 +102,20 @@ describeDockerLive("pg-delta next local convergence (live)", () => { "applies a representative declarative schema and converges", { timeout: SCENARIO_TIMEOUT_MS }, async () => { - expect(migrationFiles(projectDir)).toEqual([]); - - const diff = await runSupabaseLive( - ["db", "diff", "--local", "--use-pg-delta", "-f", "initial_declarative"], + const sync = await runSupabaseLive( + [ + "db", + "schema", + "declarative", + "sync", + "--no-apply", + "--name", + "initial_declarative", + "--experimental", + ], { cwd: projectDir, env: NEXT_ENV, exitTimeoutMs: COMMAND_TIMEOUT_MS }, ); - expect(diff.exitCode, commandFailure(diff)).toBe(0); + expect(sync.exitCode, commandFailure(sync)).toBe(0); const migrations = migrationFiles(projectDir); expect(migrations.length).toBeGreaterThan(0); @@ -127,13 +136,16 @@ describeDockerLive("pg-delta next local convergence (live)", () => { }); requireLiveSuccess(reset, "db reset setup"); - const converged = await runSupabaseLive(["db", "diff", "--local", "--use-pg-delta"], { - cwd: projectDir, - env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }); + const converged = await runSupabaseLive( + ["db", "schema", "declarative", "sync", "--no-apply", "--experimental"], + { + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, + ); expect(converged.exitCode, commandFailure(converged)).toBe(0); - expect(converged.stderr).toContain("No schema changes found"); + expect(`${converged.stdout}${converged.stderr}`).toContain("No schema changes found"); }, ); }); diff --git a/apps/cli/src/legacy/commands/db/start/start.live.test.ts b/apps/cli/src/legacy/commands/db/start/start.live.test.ts index 699d1209cd..00d2fa6ab3 100644 --- a/apps/cli/src/legacy/commands/db/start/start.live.test.ts +++ b/apps/cli/src/legacy/commands/db/start/start.live.test.ts @@ -1,9 +1,9 @@ import { expect } from "vitest"; -import { describeDockerLive } from "../../../../../tests/helpers/live.ts"; +import { describeLocalStackLive } from "../../../../../tests/helpers/live.ts"; import { testLive } from "../../../../../tests/helpers/live-context.ts"; -describeDockerLive("supabase db start (live)", () => { +describeLocalStackLive("supabase db start (live)", () => { testLive( "boots the local database", async ({ run }) => { diff --git a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts index 377ca8c033..0845367046 100644 --- a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts @@ -1,29 +1,29 @@ import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; -import { expect, test } from "vitest"; +import { expect } from "vitest"; import { - describeLiveDataPlane, - requireLiveProjectRef, - runSupabaseLive, -} from "../../../../../tests/helpers/live.ts"; + liveDatabaseTargetArgs, + requireLiveSuccess, + testLiveDestructiveDataPlane, +} from "../../../../../tests/helpers/live-context.ts"; const LIVE_TIMEOUT_MS = 120_000; -// A deterministic migration to seed into the remote history and fetch back. -const VERSION = "20240101000000"; -const NAME = "cli_live_roundtrip"; -const MIGRATION_FILE = `${VERSION}_${NAME}.sql`; +// A uniquely named migration to seed into the remote history and fetch back. +const NAME = "cli_live_fetch"; -// Data-plane scenario (Postgres over the pooler) — see the note in -// `../list/list.live.test.ts`. `describeLiveDataPlane` runs this only when the -// project instance is ACTIVE_HEALTHY (the full stack with supabase-postgres-17); -// it SKIPS on the control-plane-only CI that omits it (CLI-1825). +function liveMigrationVersion(): string { + return new Date().toISOString().replace(/\D/gu, "").slice(0, 14); +} + +// Destructive data-plane scenario (Postgres over the pooler) — the setup repairs +// remote migration history and the teardown reverts that exact row. The fixture +// keeps this opt-in for attached runs and waits for an ACTIVE_HEALTHY project. // -// Round-trip: `migration fetch` reads the remote `schema_migrations` history and -// writes each row to `supabase/migrations/_.sql`; `migration list` -// then reads those files back as the Local column. +// Golden path: `migration fetch` reads the remote `schema_migrations` history and +// writes each row to `supabase/migrations/_.sql`. // // Unlike `migration list`, `migration fetch` does NOT tolerate a missing history // table: reading the migration table has no undefined-table fallback (only @@ -32,60 +32,65 @@ const MIGRATION_FILE = `${VERSION}_${NAME}.sql`; // (`relation … does not exist`). So we first SEED one migration into the remote // history via `migration repair --status applied` (which creates the migration // table then upserts the version from the local file), establishing -// the table + a row for `fetch` to read back. The ref is supplied via -// SUPABASE_PROJECT_ID. The seed is idempotent (upsert) and the supabox stack is torn -// down per run, so it leaves no shared state behind. -describeLiveDataPlane("supabase migration fetch (live)", () => { - test( - "seeds remote history, fetches it back, and lists it (round-trip)", - { timeout: LIVE_TIMEOUT_MS }, - async () => { - const ref = requireLiveProjectRef(); - const seedDir = await mkdtemp(path.join(tmpdir(), "sb-migration-seed-live-")); - const fetchDir = await mkdtemp(path.join(tmpdir(), "sb-migration-fetch-live-")); - try { - // Seed: record one migration in the remote history. `repair --status applied` - // reads the local file for the version's name/statements, so write it first. - await mkdir(path.join(seedDir, "supabase", "migrations"), { recursive: true }); - await writeFile( - path.join(seedDir, "supabase", "migrations", MIGRATION_FILE), - "create table if not exists public.cli_live_roundtrip (id int);\n", - ); - const repaired = await runSupabaseLive( - ["migration", "repair", VERSION, "--status", "applied"], - { cwd: seedDir, env: { SUPABASE_PROJECT_ID: ref } }, - ); - expect(`${repaired.stdout}${repaired.stderr}`).not.toContain("Unauthorized"); - expect(repaired.exitCode, `stdout:\n${repaired.stdout}\nstderr:\n${repaired.stderr}`).toBe( - 0, - ); - - // Fetch into a fresh (empty) dir so no overwrite prompt fires; it reads the - // remote history and writes _.sql. - const fetched = await runSupabaseLive(["migration", "fetch"], { - cwd: fetchDir, - env: { SUPABASE_PROJECT_ID: ref }, - }); - expect(`${fetched.stdout}${fetched.stderr}`).not.toContain("Unauthorized"); - expect(fetched.exitCode, `stdout:\n${fetched.stdout}\nstderr:\n${fetched.stderr}`).toBe(0); +// the table + a row for `fetch` to read back. The shared fixture's pooler URL is +// passed explicitly so the test does not fall back to a direct IPv6 host. +testLiveDestructiveDataPlane( + "fetches a seeded remote migration into the local migrations directory", + { timeout: LIVE_TIMEOUT_MS }, + async ({ run, dbUrl, projectRef }) => { + const targetArgs = liveDatabaseTargetArgs(dbUrl, projectRef); + const version = liveMigrationVersion(); + const migrationFile = `${version}_${NAME}.sql`; + const seedDir = await mkdtemp(path.join(tmpdir(), "sb-migration-seed-live-")); + const fetchDir = await mkdtemp(path.join(tmpdir(), "sb-migration-fetch-live-")); + let repaired = false; + let targetError: unknown; + let cleanupError: unknown; + try { + // Seed: record one migration in the remote history. `repair --status applied` + // reads the local file for the version's name/statements, so write it first. + await mkdir(path.join(seedDir, "supabase", "migrations"), { recursive: true }); + await writeFile( + path.join(seedDir, "supabase", "migrations", migrationFile), + "create table if not exists public.cli_live_roundtrip (id int);\n", + ); + const repairResult = await run( + ["migration", "repair", version, "--status", "applied", ...targetArgs], + { cwd: seedDir }, + ); + requireLiveSuccess(repairResult, "migration repair setup"); + repaired = true; - // fetch wrote the seeded migration back, under its established filename format. - const files = await readdir(path.join(fetchDir, "supabase", "migrations")); - expect(files).toContain(MIGRATION_FILE); + // Fetch into a fresh (empty) dir so no overwrite prompt fires; it reads the + // remote history and writes _.sql. + const fetched = await run(["migration", "fetch", ...targetArgs], { cwd: fetchDir }); + expect(fetched.exitCode, `stdout:\n${fetched.stdout}\nstderr:\n${fetched.stderr}`).toBe(0); - // The same dir feeds `migration list` as the Local column — exit 0 and the - // fetched version is reflected back. - const listed = await runSupabaseLive(["migration", "list"], { - cwd: fetchDir, - env: { SUPABASE_PROJECT_ID: ref }, - }); - expect(`${listed.stdout}${listed.stderr}`).not.toContain("Unauthorized"); - expect(listed.exitCode, `stdout:\n${listed.stdout}\nstderr:\n${listed.stderr}`).toBe(0); - expect(listed.stdout).toContain(VERSION); - } finally { + // fetch wrote the seeded migration back, under its established filename format. + const files = await readdir(path.join(fetchDir, "supabase", "migrations")); + expect(files).toContain(migrationFile); + } catch (error) { + targetError = error; + } finally { + if (repaired) { + try { + const reverted = await run( + ["migration", "repair", version, "--status", "reverted", ...targetArgs], + { cwd: seedDir }, + ); + requireLiveSuccess(reverted, "migration repair cleanup"); + } catch (error) { + cleanupError = error; + } + } + try { await rm(seedDir, { recursive: true, force: true }); await rm(fetchDir, { recursive: true, force: true }); + } catch (error) { + cleanupError ??= error; } - }, - ); -}); + } + if (targetError !== undefined) throw targetError; + if (cleanupError !== undefined) throw cleanupError; + }, +); diff --git a/apps/cli/src/legacy/commands/projects/list/list.live.test.ts b/apps/cli/src/legacy/commands/projects/list/list.live.test.ts index a34f257bfa..8737bc5315 100644 --- a/apps/cli/src/legacy/commands/projects/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/projects/list/list.live.test.ts @@ -7,8 +7,23 @@ testLiveProject( async ({ run, projectRef }) => { const result = await run(["projects", "list", "--output-format", "json"]); expect(result.exitCode, result.stderr).toBe(0); - const projects = JSON.parse(result.stdout) as Array<{ id?: string; ref?: string }>; - expect(projects.map((project) => project.ref ?? project.id)).toContain(projectRef); + const parsed: unknown = JSON.parse(result.stdout); + expect(parsed).toEqual(expect.objectContaining({ projects: expect.any(Array) })); + if ( + parsed === null || + typeof parsed !== "object" || + !("projects" in parsed) || + !Array.isArray(parsed.projects) + ) { + throw new Error("projects list JSON response did not contain a projects array"); + } + const refs = parsed.projects.flatMap((project) => { + if (project === null || typeof project !== "object") return []; + if ("ref" in project && typeof project.ref === "string") return [project.ref]; + if ("id" in project && typeof project.id === "string") return [project.id]; + return []; + }); + expect(refs).toContain(projectRef); }, ); diff --git a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts index 9c2165aaa9..314e265a15 100644 --- a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts @@ -9,7 +9,7 @@ async function unsetSecret( projectRef: string, ): Promise { const cleanup = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); - if (cleanup.exitCode !== 0) { + if (cleanup.exitCode !== 0 && !/not found/i.test(`${cleanup.stdout}\n${cleanup.stderr}`)) { throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); } } @@ -19,11 +19,25 @@ testLiveProject("unsets a secret from the remote project", async ({ run, project const created = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); requireLiveSuccess(created, "secrets set setup"); + let deleted = false; + let targetError: unknown; + let cleanupError: unknown; try { const result = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); expect(result.exitCode, result.stderr).toBe(0); + deleted = true; expect(result.stdout).toContain("Finished"); + } catch (error) { + targetError = error; } finally { - await unsetSecret(run, name, projectRef); + if (!deleted) { + try { + await unsetSecret(run, name, projectRef); + } catch (error) { + cleanupError = error; + } + } } + if (targetError !== undefined) throw targetError; + if (cleanupError !== undefined) throw cleanupError; }); diff --git a/apps/cli/src/legacy/commands/start/start.live.test.ts b/apps/cli/src/legacy/commands/start/start.live.test.ts index 0e647733cf..fc664f4487 100644 --- a/apps/cli/src/legacy/commands/start/start.live.test.ts +++ b/apps/cli/src/legacy/commands/start/start.live.test.ts @@ -7,7 +7,7 @@ import path from "node:path"; import { promisify } from "node:util"; import { afterEach, expect, test } from "vitest"; -import { describeLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { describeLocalStackLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; import { legacySanitizeProjectId, @@ -61,11 +61,11 @@ function splitNonEmptyLines(text: string): ReadonlyArray { // `start` is the one local-dev-stack command whose correctness genuinely // depends on a real Docker daemon — real label filtering and real container -// lifecycle, not just CLI exit codes. `describeLive` is reused purely as the +// lifecycle, not just CLI exit codes. `describeLocalStackLive` gates the // "we're in a configured live runner" signal (see stop.live.test.ts's own // comment for why this, not a Management-API gate, is correct here). See // AGENTS.md's "Live tests" section for the full convention. -describeLive("supabase start (live)", () => { +describeLocalStackLive("supabase start (live)", () => { let projectDir: string | undefined; afterEach(async () => { diff --git a/apps/cli/src/legacy/commands/status/status.live.test.ts b/apps/cli/src/legacy/commands/status/status.live.test.ts index 1948715d14..c97c2827e7 100644 --- a/apps/cli/src/legacy/commands/status/status.live.test.ts +++ b/apps/cli/src/legacy/commands/status/status.live.test.ts @@ -3,16 +3,16 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { afterEach, expect, test } from "vitest"; -import { describeLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { describeLocalStackLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; const START_TIMEOUT_MS = 280_000; -// See stop.live.test.ts for why `describeLive` (not a Management-API gate) is +// See stop.live.test.ts for why `describeLocalStackLive` (not a Management-API gate) is // the right reuse here: `status` never calls the Management API, only the real // Docker daemon the cli-e2e-ci runner provides. See AGENTS.md's "Live tests" // section for the full convention. -describeLive("supabase status (live)", () => { +describeLocalStackLive("supabase status (live)", () => { let projectDir: string | undefined; afterEach(async () => { diff --git a/apps/cli/src/legacy/commands/stop/stop.live.test.ts b/apps/cli/src/legacy/commands/stop/stop.live.test.ts index 0d1f1fc463..0a1d2a1c4c 100644 --- a/apps/cli/src/legacy/commands/stop/stop.live.test.ts +++ b/apps/cli/src/legacy/commands/stop/stop.live.test.ts @@ -5,7 +5,7 @@ import path from "node:path"; import { promisify } from "node:util"; import { afterEach, expect, test } from "vitest"; -import { describeLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { describeLocalStackLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; import { legacySanitizeProjectId } from "../../shared/legacy-docker-ids.ts"; @@ -14,12 +14,12 @@ const execFileAsync = promisify(execFile); const START_TIMEOUT_MS = 280_000; // `stop` never calls the Management API — it talks directly to the real local -// Docker stack `start` creates. `describeLive` is reused +// Docker stack `start` creates. `describeLocalStackLive` gates // purely as the "we're in the full cli-e2e-ci runner" signal (it also has a // real Docker daemon, since that's how supabox itself runs); the // SUPABASE_ACCESS_TOKEN it gates on is otherwise irrelevant here. See // AGENTS.md's "Live tests" section for the full convention. -describeLive("supabase stop (live)", () => { +describeLocalStackLive("supabase stop (live)", () => { let projectDir: string | undefined; let projectId: string | undefined; diff --git a/apps/cli/src/next/commands/functions/dev/dev.live.test.ts b/apps/cli/src/next/commands/functions/dev/dev.live.test.ts index ddb5670c82..0277ad9ebc 100644 --- a/apps/cli/src/next/commands/functions/dev/dev.live.test.ts +++ b/apps/cli/src/next/commands/functions/dev/dev.live.test.ts @@ -7,7 +7,7 @@ import { runSupabase, spawnSupabase, } from "../../../../../tests/helpers/cli.ts"; -import { describeLive } from "../../../../../tests/helpers/live.ts"; +import { describeLocalStackLive } from "../../../../../tests/helpers/live.ts"; import { cleanupRegisteredStackProjects } from "../../../../../tests/helpers/stack-e2e-cleanup.ts"; const FUNCTIONS_DEV_STARTUP_TIMEOUT_MS = 60_000; @@ -74,7 +74,7 @@ async function waitForFunctionResponse( // This crosses the compiled CLI, detached supervisor, full local stack, file // watcher, and HTTP runtime boundaries. Keep the one golden path in the // opt-in live suite instead of slowing and destabilizing ordinary e2e shards. -describeLive("supabase functions dev (live)", () => { +describeLocalStackLive("supabase functions dev (live)", () => { afterEach(cleanupRegisteredStackProjects); test( diff --git a/apps/cli/src/next/commands/start/start.live.test.ts b/apps/cli/src/next/commands/start/start.live.test.ts index f9183d5f16..bf0283023a 100644 --- a/apps/cli/src/next/commands/start/start.live.test.ts +++ b/apps/cli/src/next/commands/start/start.live.test.ts @@ -1,6 +1,7 @@ import { afterEach, expect, test } from "vitest"; import { makeTempHome, makeTempStackProject } from "../../../../tests/helpers/cli.ts"; -import { describeLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { describeLocalStackLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { cleanupRegisteredStackProjects } from "../../../../tests/helpers/stack-e2e-cleanup.ts"; const START_TIMEOUT_MS = 180_000; const COMMAND_OPTIONS = { entrypoint: "next" as const }; @@ -31,20 +32,12 @@ const LIGHTWEIGHT_DOCKER_ARGS = [ // Lazy service activation crosses the real proxy, daemon, Docker network, and // container lifecycle boundaries, so keep one gated golden-path live test. -describeLive("supabase start lazy lifecycle (live)", () => { +describeLocalStackLive("supabase start lazy lifecycle (live)", () => { let project: Awaited> | undefined; let home: ReturnType | undefined; afterEach(async () => { - if (project !== undefined && home !== undefined) { - await runSupabaseLive(["stop", "--no-backup"], { - ...COMMAND_OPTIONS, - cwd: project.dir, - home: home.dir, - }).catch(() => undefined); - } - await project?.cleanup(); - home?.[Symbol.dispose](); + await cleanupRegisteredStackProjects(); project = undefined; home = undefined; }); @@ -70,7 +63,7 @@ describeLive("supabase start lazy lifecycle (live)", () => { home: home.dir, }); expect(before.exitCode, `stdout:\n${before.stdout}\nstderr:\n${before.stderr}`).toBe(0); - expect(before.stdout).toContain("auth: Pending"); + expect(before.stdout).toContain("auth: Dormant"); const response = await fetch(`http://127.0.0.1:${project.ports.apiPort}/auth/v1/health`, { signal: AbortSignal.timeout(60_000), diff --git a/apps/cli/tests/helpers/live-env.ts b/apps/cli/tests/helpers/live-env.ts index 1db8fd5a50..76ed269359 100644 --- a/apps/cli/tests/helpers/live-env.ts +++ b/apps/cli/tests/helpers/live-env.ts @@ -18,6 +18,9 @@ * `https://api.supabase.green`. * - `SUPABASE_LIVE_PROJECT_REF` — the shared project; gates project-scoped * suites (functions, branches, db, storage). Managed setup populates it. + * - `SUPABASE_LIVE_LOCAL_STACK` — whether local Docker-stack lifecycle suites + * run. Attached runs default to `1`; managed staging defaults to `0`. + * Set explicitly to `0` or `1` to override the mode default. * - `NODE_EXTRA_CA_CERTS` — trusts the supabox CA for `*.supabase.red` TLS; * inherited by the subprocess via the parent environment. */ @@ -52,6 +55,28 @@ export function isManagedLive(): boolean { return liveMode() === "managed"; } +/** + * Whether suites that own a local Docker development stack should run. + * Attached runners preserve the historical default; managed project runners + * only exercise the remote project unless explicitly opted into local-stack + * coverage. + */ +export function localStackLiveEnabled(): boolean { + const value = process.env["SUPABASE_LIVE_LOCAL_STACK"]; + if (value === undefined) { + return !isManagedLive(); + } + if (value === "0") { + return false; + } + if (value === "1") { + return true; + } + throw new Error( + `Unsupported SUPABASE_LIVE_LOCAL_STACK ${JSON.stringify(value)}; expected "0" or "1"`, + ); +} + /** * Default subprocess exit timeout for live runs. `runSupabase` otherwise caps at * 60s, which would kill a slow-but-valid supabox call before the live tests' diff --git a/apps/cli/tests/helpers/live-env.unit.test.ts b/apps/cli/tests/helpers/live-env.unit.test.ts new file mode 100644 index 0000000000..96b9249062 --- /dev/null +++ b/apps/cli/tests/helpers/live-env.unit.test.ts @@ -0,0 +1,52 @@ +import { afterEach, describe, expect, it } from "vitest"; + +import { localStackLiveEnabled } from "./live-env.ts"; + +const originalMode = process.env["SUPABASE_LIVE_MODE"]; +const originalLocalStack = process.env["SUPABASE_LIVE_LOCAL_STACK"]; + +afterEach(() => { + if (originalMode === undefined) { + delete process.env["SUPABASE_LIVE_MODE"]; + } else { + process.env["SUPABASE_LIVE_MODE"] = originalMode; + } + if (originalLocalStack === undefined) { + delete process.env["SUPABASE_LIVE_LOCAL_STACK"]; + } else { + process.env["SUPABASE_LIVE_LOCAL_STACK"] = originalLocalStack; + } +}); + +describe("localStackLiveEnabled", () => { + it("defaults to enabled for attached live runs", () => { + delete process.env["SUPABASE_LIVE_MODE"]; + delete process.env["SUPABASE_LIVE_LOCAL_STACK"]; + + expect(localStackLiveEnabled()).toBe(true); + }); + + it("defaults to disabled for managed live runs", () => { + process.env["SUPABASE_LIVE_MODE"] = "managed"; + delete process.env["SUPABASE_LIVE_LOCAL_STACK"]; + + expect(localStackLiveEnabled()).toBe(false); + }); + + it("honors an explicit enabled or disabled value", () => { + process.env["SUPABASE_LIVE_MODE"] = "managed"; + process.env["SUPABASE_LIVE_LOCAL_STACK"] = "1"; + expect(localStackLiveEnabled()).toBe(true); + + process.env["SUPABASE_LIVE_LOCAL_STACK"] = "0"; + expect(localStackLiveEnabled()).toBe(false); + }); + + it("rejects unsupported values", () => { + process.env["SUPABASE_LIVE_LOCAL_STACK"] = "yes"; + + expect(() => localStackLiveEnabled()).toThrow( + 'Unsupported SUPABASE_LIVE_LOCAL_STACK "yes"; expected "0" or "1"', + ); + }); +}); diff --git a/apps/cli/tests/helpers/live.ts b/apps/cli/tests/helpers/live.ts index dd591a84be..9ad2fd96d8 100644 --- a/apps/cli/tests/helpers/live.ts +++ b/apps/cli/tests/helpers/live.ts @@ -5,6 +5,7 @@ import { runSupabase } from "./cli.ts"; import { isLiveConfigured, LIVE_EXIT_TIMEOUT_MS, + localStackLiveEnabled, liveProjectDataPlaneReady, liveProfile, liveProjectRef, @@ -29,6 +30,7 @@ export { LIVE_EXIT_TIMEOUT_MS, liveApiBaseUrl, isManagedLive, + localStackLiveEnabled, keepLiveProject, liveProjectDataPlaneReady, liveMode, @@ -66,6 +68,15 @@ function hasDockerDaemon(): boolean { */ export const describeDockerLive = describe.skipIf(!isLiveConfigured() || !hasDockerDaemon()); +/** + * `describe` for suites that own a local Docker development stack. Attached + * Supabox/local runners enable these by default; managed staging only runs + * them when `SUPABASE_LIVE_LOCAL_STACK=1` is explicitly set. + */ +export const describeLocalStackLive = describe.skipIf( + !isLiveConfigured() || !localStackLiveEnabled() || !hasDockerDaemon(), +); + /** * `describe` for project-scoped live suites: runs only when the live env is * configured AND a project ref is available. On a control-plane-only stack diff --git a/docs/adr/0013-live-e2e-bypasses-replay-server.md b/docs/adr/0013-live-e2e-bypasses-replay-server.md index 186e57034c..cf8ce50f08 100644 --- a/docs/adr/0013-live-e2e-bypasses-replay-server.md +++ b/docs/adr/0013-live-e2e-bypasses-replay-server.md @@ -39,6 +39,10 @@ mode, not a `replay-server.ts` branch. deletes exactly that project during teardown unless explicitly kept. - `replay-server.ts` is untouched — no `live` branch, no live Docker or storage proxy. +- Local Docker-stack lifecycle suites are gated separately by + `SUPABASE_LIVE_LOCAL_STACK`: attached Supabox/local runs default to enabled, + while managed staging defaults to disabled. Docker-dependent remote scenarios + such as `functions deploy` retain the broader Docker gate. - Assertions are **outcome-based**, modeled on the manual deploy playbook: 1. run the real CLI (`run([...])`) and assert `exitCode` / `stdout`; 2. **invoke the deployed function over HTTP directly** and assert HTTP status + @@ -106,8 +110,9 @@ values without changing test code. - Explicit managed runs provision and tear down one real staging project, so those runs are inherently slower and subject to provisioning flake. The default attached mode uses the caller-provided Supabox/local project; global setup never deletes it. - Managed provisioning is mitigated by a CI-level re-run (up to 3×) rather than - in-setup retry. + Managed CI runs the live suite once with a 20-minute step timeout and always runs + the separate scoped project-cleanup step; it does not amplify deterministic failures + by retrying the entire suite. - A second wiring path now exists for the same harness (replay-via-server vs live-direct); contributors must know which mode wires the CLI how. From a7b533c994e87fb818624baef9f91a441f08513f Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 21:56:52 +0200 Subject: [PATCH 10/23] test(cli): simplify live e2e environment --- .github/workflows/live-e2e.yml | 4 - apps/cli/AGENTS.md | 54 +- apps/cli/live.env.example | 46 +- .../branches/create/create.live.test.ts | 4 +- .../branches/delete/delete.live.test.ts | 4 +- .../commands/branches/list/list.live.test.ts | 4 +- ...f.live.test.ts => diff.docker.e2e.test.ts} | 22 +- .../legacy/commands/db/dump/dump.live.test.ts | 32 +- .../legacy/commands/db/pull/pull.live.test.ts | 131 ++--- .../legacy/commands/db/push/push.live.test.ts | 58 +- .../commands/db/reset/reset.live.test.ts | 63 +- ...c.live.test.ts => sync.docker.e2e.test.ts} | 27 +- .../db/start/start.docker.e2e.test.ts | 17 + .../commands/db/start/start.live.test.ts | 22 - .../functions/delete/delete.live.test.ts | 4 +- .../functions/deploy/deploy.live.test.ts | 62 +- .../commands/functions/list/list.live.test.ts | 72 ++- .../commands/gen/types/types.live.test.ts | 17 +- .../inspect/db/db-stats/db-stats.live.test.ts | 28 +- .../legacy/commands/link/link.live.test.ts | 17 +- .../migration/fetch/fetch.live.test.ts | 6 +- .../commands/migration/list/list.live.test.ts | 20 +- .../commands/orgs/list/list.live.test.ts | 39 +- .../projects/api-keys/api-keys.live.test.ts | 4 +- .../commands/projects/list/list.live.test.ts | 49 +- .../commands/secrets/list/list.live.test.ts | 4 +- .../commands/secrets/set/set.live.test.ts | 4 +- .../commands/secrets/unset/unset.live.test.ts | 4 +- ....live.test.ts => start.docker.e2e.test.ts} | 45 +- ...live.test.ts => status.docker.e2e.test.ts} | 24 +- ...p.live.test.ts => stop.docker.e2e.test.ts} | 39 +- .../commands/storage/cp/cp.live.test.ts | 47 +- .../commands/storage/ls/ls.live.test.ts | 49 +- .../commands/storage/rm/rm.live.test.ts | 51 +- ...ev.live.test.ts => dev.docker.e2e.test.ts} | 7 +- ....live.test.ts => start.docker.e2e.test.ts} | 13 +- apps/cli/tests/helpers/cli.ts | 11 + apps/cli/tests/helpers/live-context.ts | 215 ------- apps/cli/tests/helpers/live-env.ts | 230 ++------ apps/cli/tests/helpers/live-env.unit.test.ts | 66 +-- apps/cli/tests/helpers/live-project.ts | 547 +++++++----------- .../tests/helpers/live-provided-context.ts | 26 +- apps/cli/tests/helpers/live.ts | 273 +++++---- apps/cli/tests/live-global-setup.ts | 76 +-- apps/cli/vitest.config.ts | 6 +- .../0013-live-e2e-bypasses-replay-server.md | 178 ++---- 46 files changed, 1048 insertions(+), 1673 deletions(-) rename apps/cli/src/legacy/commands/db/diff/{diff.live.test.ts => diff.docker.e2e.test.ts} (83%) rename apps/cli/src/legacy/commands/db/schema/declarative/sync/{sync.live.test.ts => sync.docker.e2e.test.ts} (85%) create mode 100644 apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts delete mode 100644 apps/cli/src/legacy/commands/db/start/start.live.test.ts rename apps/cli/src/legacy/commands/start/{start.live.test.ts => start.docker.e2e.test.ts} (88%) rename apps/cli/src/legacy/commands/status/{status.live.test.ts => status.docker.e2e.test.ts} (66%) rename apps/cli/src/legacy/commands/stop/{stop.live.test.ts => stop.docker.e2e.test.ts} (79%) rename apps/cli/src/next/commands/functions/dev/{dev.live.test.ts => dev.docker.e2e.test.ts} (95%) rename apps/cli/src/next/commands/start/{start.live.test.ts => start.docker.e2e.test.ts} (81%) delete mode 100644 apps/cli/tests/helpers/live-context.ts diff --git a/.github/workflows/live-e2e.yml b/.github/workflows/live-e2e.yml index bc996f08f7..8ccd621401 100644 --- a/.github/workflows/live-e2e.yml +++ b/.github/workflows/live-e2e.yml @@ -99,11 +99,7 @@ jobs: # Non-secret config is job-level; the staging token is scoped to only the two # steps that need it (run + cleanup) so build/checkout/docker never see it. env: - SUPABASE_LIVE_MODE: managed - SUPABASE_LIVE_LOCAL_STACK: "0" - SUPABASE_PROFILE: supabase-staging SUPABASE_LIVE_API_URL: https://api.supabase.green - SUPABASE_LIVE_PROJECT_HOST: supabase.red SUPABASE_LIVE_PROJECT_NAME: supabase-cli-live-${{ matrix.target }} CLI_HARNESS_TARGET: ${{ matrix.target }} steps: diff --git a/apps/cli/AGENTS.md b/apps/cli/AGENTS.md index 9dc8611f42..bba3413c46 100644 --- a/apps/cli/AGENTS.md +++ b/apps/cli/AGENTS.md @@ -471,7 +471,7 @@ Read https://www.effect.solutions/testing for Effect testing patterns. Note that - `*.unit.test.ts` belongs to the `unit` Vitest project and is the default for unit-style and other fast in-process tests. - `*.integration.test.ts` belongs to the `integration` project and is for in-process integration tests that exercise real handler or service behavior with layered dependency replacement. - `*.e2e.test.ts` belongs to the `e2e` Vitest project and is for black-box CLI subprocess tests. -- `*.live.test.ts` belongs to the `live` Vitest project and is for black-box CLI subprocess tests that run against a **real, running Supabase platform or local Docker stack** — see "Live tests" below. +- `*.live.test.ts` belongs to the `live` Vitest project and is for black-box CLI subprocess tests whose asserted command reaches a real Supabase platform or project data plane — see "Live tests" below. ### Testing policy @@ -489,23 +489,41 @@ Read https://www.effect.solutions/testing for Effect testing patterns. Note that ### Live tests (`*.live.test.ts`) -Live tests are black-box CLI subprocess tests — like `*.e2e.test.ts`, but run against a **real backend** instead of local fakes/mocks: either the real Management API (a full [supabox](https://github.com/supabase/supabox) platform stack) or a real local Docker dev stack (`supabase start`'s actual containers). They are the highest-fidelity, most expensive tier — reserved for the small set of behaviors that only a genuinely running backend can prove (auth round-trips, real Docker label filtering, real container lifecycle), not for anything an integration test can already cover with mocks. - -- **Where they run:** authored and executed from this workspace's `live` Vitest project (`pnpm test:live`). CI may run the same files from the [`supabase/cli-e2e-ci`](https://github.com/supabase/cli-e2e-ci) Supabox harness or the managed-staging workflow. They never run as part of the default unit/integration/e2e loop, and locally they no-op unless the live environment is configured (see below). -- **Add one whenever you add or change a command whose correctness genuinely depends on a real backend** — a new Management API command, or a change to `start`/`stop`/`status`'s real Docker interaction. Colocate it with the command, same as `*.e2e.test.ts`: `src/legacy/commands//[/].live.test.ts`. -- **Gating:** every live suite must be wrapped in one of `tests/helpers/live.ts`'s `describe.skipIf` gates so the file is inert (skipped, not failed) outside a configured live runner: - - `describeLive` — runs whenever `SUPABASE_ACCESS_TOKEN` is set (the live environment is configured at all). Use it for account-level and platform/control-plane scenarios that do not require a provisioned project or a local Docker stack. Commands that manage the local dev stack must use `describeLocalStackLive` instead. - - `describeDockerLive` — the configured-live gate composed with a `docker info` probe; use for Docker-dependent remote scenarios such as `functions deploy` that still run under managed staging. It never runs on a machine that merely exposes Docker — `SUPABASE_ACCESS_TOKEN` must still be set, so the file stays inert outside a configured live runner like every other live suite. - - `describeLocalStackLive` — additionally requires `SUPABASE_LIVE_LOCAL_STACK` to be enabled and a reachable Docker daemon. Attached Supabox/local runs default to enabled; managed staging defaults to disabled so it does not launch unrelated local stacks. Set `SUPABASE_LIVE_LOCAL_STACK=1` to opt into those suites, or `0` to disable them explicitly. Invalid values fail loudly. - - `describeLiveProject` — additionally requires a provisioned project (`SUPABASE_LIVE_PROJECT_REF`); use for project-scoped Management API commands (branches, functions, project-scoped db). - - `testLiveDataPlane` / `testLiveDestructiveDataPlane` — fixture gates for data-plane commands; the latter additionally requires `SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1` in attached mode because its setup/cleanup resets remote state. -- **Invocation:** use `runSupabaseLive(args, options?)` (wraps `runSupabase` with the `legacy` entrypoint and the live profile/timeout defaults) rather than calling `runSupabase` directly, so every live test picks up the same environment plumbing. - - **Environment:** attached mode is the default and preserves the local Supabox contract (`SUPABASE_PROFILE=supabase-local`, `SUPABASE_LIVE_API_URL=http://localhost:8080`, `SUPABASE_LIVE_PROJECT_REF`, and optional data-plane keys). It only reads the existing project and never deletes it. Destructive remote DB tests are opt-in with `SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1`. Set `SUPABASE_LIVE_MODE=managed` for an explicit staging run; global setup then provisions one uniquely named project, shares it across files, and deletes exactly that project at teardown. See `live.env.example` for the full contract. -- **Assertion scope:** each file should assert one command's golden path. Setup and teardown may call other commands to prepare or clean state, but those calls are not assertions. Keep the suite focused on one common success workflow per command; use integration tests for exhaustive branches, validation, formatting, errors, and matrices. -- **Local-dev-stack live tests** (`start`/`stop`/`status`, and anything else that manages real Docker containers rather than calling the Management API) use `describeLocalStackLive` and don't need `SUPABASE_PROFILE`/project-ref machinery. Pattern: `mkdtemp` a project dir, run `init` and any bootstrap `start`/`status` commands with diagnostic failures but no Vitest assertions, exercise the command under test, then clean up in `afterEach` (best-effort `stop --no-backup` + `rm` the temp dir) so a failed assertion never leaks containers onto the CI runner. See `commands/stop/stop.live.test.ts` and `commands/status/status.live.test.ts` for the canonical example. -- **Managed CI execution** provisions one shared project per run and invokes `pnpm --filter supabase test:live` once with a 20-minute step timeout. It does not retry the whole suite; deterministic failures should be diagnosed from the first run, while the separate cleanup step removes any leftover managed project. -- **Keep the suite small and golden-path only** — same philosophy as `*.e2e.test.ts`, but even more so given the cost of a real backend. One or two scenarios per command is normal; branch-by-branch coverage belongs in `*.integration.test.ts`. -- Timeouts are generous by default (`testTimeout`/`hookTimeout: 300_000` for the whole `live` project) because real platform/Docker operations are slow — pass an explicit per-`test()` timeout when a scenario needs less (or, for a real local-stack `start`, close to the full budget). +Live tests are black-box CLI subprocess tests whose asserted command reaches a +real Management API, its suite-owned project, or that project's data plane. +They are serial, explicit, and expensive; keep them to one golden path per +command. The file name selects the live Vitest project and the file imports one +extended fixture as `test` from `tests/helpers/live.ts`: + +```ts +import { expect } from "vitest"; +import { test } from "../../../../../tests/helpers/live.ts"; + +test("lists projects", async ({ cli, project }) => { + const result = await cli(["projects", "list", "--output-format", "json"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain(project.ref); +}); +``` + +Global setup requires `SUPABASE_LIVE_API_URL` and `SUPABASE_ACCESS_TOKEN`, +provisions one disposable project through the typed Effect `@supabase/api` +client, waits for `ACTIVE_HEALTHY`, resolves project wiring, writes a temporary +YAML profile, and shares it across the serial suite. Teardown deletes exactly +that project and the temporary profile. Supabox, a Docker-hosted API platform, +and staging are interchangeable; changing the URL and token retargets the +run. `SUPABASE_LIVE_KEEP_PROJECT=1` keeps the project for debugging. + +Local Docker-stack lifecycle tests (`start`, `stop`, `status`, `db start`, +`db diff`, declarative sync, and `functions dev`) are `*.e2e.test.ts`, use +`runSupabase` plus the existing e2e stack cleanup, and require no platform +credentials. `functions deploy` remains live because its assertion is remote +deployment and invocation, even though Docker is a runner prerequisite. + +Setup/teardown may invoke other commands, but assertions stay focused on the +one command named by the test. The live workflow runs one serial attempt with a +20-minute bound, retains Docker preflight, and sweeps only projects owned by +that run after crashes. --- diff --git a/apps/cli/live.env.example b/apps/cli/live.env.example index c0ea5c4988..b859ed76f0 100644 --- a/apps/cli/live.env.example +++ b/apps/cli/live.env.example @@ -1,45 +1,15 @@ -# Live CLI e2e environment. The suite is in the `live` Vitest project and is -# intentionally separate from the replay/record harness in apps/cli-e2e. - -# Required in both modes. The default attached mode never creates or deletes a -# project; it uses the project supplied by the local Supabox/stack harness. -SUPABASE_ACCESS_TOKEN=sbp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -SUPABASE_LIVE_PROJECT_REF=abcdefghijklmnopqrst -# Local Docker-stack lifecycle suites default on in attached mode and off in -# managed staging. Set explicitly to override the mode default (only 0 or 1). -# SUPABASE_LIVE_LOCAL_STACK=1 - -# Attached mode (default): point these at a running Supabox/local platform. -SUPABASE_LIVE_MODE=attached -SUPABASE_PROFILE=supabase-local +# Live CLI e2e environment. The suite provisions one disposable project +# against the configured Management API URL. Supabox, Docker-hosted API, and +# staging use the same contract; only this URL and token change. SUPABASE_LIVE_API_URL=http://localhost:8080 -SUPABASE_LIVE_PROJECT_HOST=supabase.red -NODE_EXTRA_CA_CERTS=/path/to/supabox/ca.pem - -# Optional attached data-plane values. Read-only database tests fall back to -# `--linked --project-ref` when DB_URL is omitted. Destructive remote database -# tests additionally require SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1. -# SUPABASE_LIVE_ANON_KEY=eyJ... -# SUPABASE_LIVE_FUNCTIONS_URL=https://abcdefghijklmnopqrst.supabase.red/functions/v1 # optional; derived from project ref when omitted -# SUPABASE_LIVE_DB_URL=postgres://... -# SUPABASE_LIVE_DB_PASSWORD=... -# SUPABASE_LIVE_STORAGE_BUCKET=supabase-cli-live-bucket -# Destructive remote DB tests (db push/pull/reset) are opt-in when attached. -# SUPABASE_LIVE_ALLOW_DESTRUCTIVE=1 +SUPABASE_ACCESS_TOKEN=sbp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -# Managed mode (explicit opt-in): provision one staging project for the whole -# Vitest run, then delete exactly that project during global teardown. -# SUPABASE_LIVE_MODE=managed -# SUPABASE_PROFILE=supabase-staging -# SUPABASE_LIVE_API_URL=https://api.supabase.green -# SUPABASE_LIVE_PROJECT_HOST=supabase.red +# Optional provisioning/debug values. # SUPABASE_LIVE_ORG_ID=... # SUPABASE_LIVE_REGION=us-east-1 # SUPABASE_LIVE_PROJECT_NAME=supabase-cli-live -# SUPABASE_LIVE_DB_PASSWORD= # optional; generated randomly when unset -# SUPABASE_LIVE_KEEP_PROJECT=1 # debugging only; skips managed teardown -# Managed runs do not retry the entire live suite; failures stop the job and -# leave the separate scoped cleanup step to remove any provisioned project. +# SUPABASE_LIVE_KEEP_PROJECT=1 +# NODE_EXTRA_CA_CERTS=/path/to/supabox/ca.pem -# Run with no external harness: +# Run explicitly (Docker is required by the runner): # pnpm test:live diff --git a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts index 18a6c602ba..fbb5152748 100644 --- a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { test } from "../../../../../tests/helpers/live.ts"; async function deleteBranch( run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, @@ -16,7 +16,7 @@ async function deleteBranch( } } -testLiveProject("creates a preview branch", async ({ run, projectRef, skip }) => { +test("creates a preview branch", async ({ run, projectRef, skip }) => { const name = `cli-e2e-create-${randomUUID().slice(0, 8)}`; const result = await run(["branches", "create", name, "--project-ref", projectRef]); diff --git a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts index 182fd7fc35..c8b72e6bdf 100644 --- a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts @@ -1,9 +1,9 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -testLiveProject("deletes a preview branch", async ({ run, projectRef, skip }) => { +test("deletes a preview branch", async ({ run, projectRef, skip }) => { const name = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; const created = await run(["branches", "create", name, "--project-ref", projectRef]); if ( diff --git a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts index 85e647635a..2835d39535 100644 --- a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts @@ -1,9 +1,9 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -testLiveProject("lists a preview branch for the project", async ({ run, projectRef, skip }) => { +test("lists a preview branch for the project", async ({ run, projectRef, skip }) => { const name = `cli-e2e-list-${randomUUID().slice(0, 8)}`; const created = await run(["branches", "create", name, "--project-ref", projectRef]); if ( diff --git a/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts similarity index 83% rename from apps/cli/src/legacy/commands/db/diff/diff.live.test.ts rename to apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts index 0f9ea564ab..03cc95b35c 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.live.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts @@ -4,8 +4,8 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { afterEach, expect, test } from "vitest"; -import { describeLocalStackLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; -import { requireLiveSuccess } from "../../../../../tests/helpers/live-context.ts"; +import { describe } from "vitest"; +import { requireCliSuccess, runSupabase } from "../../../../../tests/helpers/cli.ts"; const START_TIMEOUT_MS = 280_000; @@ -19,14 +19,14 @@ const START_TIMEOUT_MS = 280_000; // FUNCTION statement; the REVOKE is silently absent. This suite uses the local // Docker-stack gate and never calls the Management API. See AGENTS.md's "Live // tests" section. -describeLocalStackLive("supabase db diff (live, pg-delta declarative privileges)", () => { +describe("supabase db diff (live, pg-delta declarative privileges)", () => { let projectDir: string | undefined; afterEach(async () => { if (projectDir === undefined) return; // Best-effort cleanup even if an assertion above failed mid-lifecycle — a // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabaseLive(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); projectDir = undefined; }); @@ -37,21 +37,21 @@ describeLocalStackLive("supabase db diff (live, pg-delta declarative privileges) async () => { projectDir = await mkdtemp(path.join(tmpdir(), "sb-db-diff-live-")); - const init = await runSupabaseLive(["init"], { cwd: projectDir }); - requireLiveSuccess(init, "init setup"); + const init = await runSupabase(["init"], { cwd: projectDir }); + requireCliSuccess(init, "init setup"); // Exclude the heaviest, least relevant services — `db diff` only needs the // local Postgres container reachable, same rationale as stop/status. - const start = await runSupabaseLive( + const start = await runSupabase( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - requireLiveSuccess(start, "start setup"); + requireCliSuccess(start, "start setup"); // Minimal, deterministic repro: execute a fresh function's implicit PUBLIC // EXECUTE grant, explicitly revoked, directly against the local database. // `db query` is setup only; the command under test remains `db diff`. - const query = await runSupabaseLive( + const query = await runSupabase( [ "db", "query", @@ -65,9 +65,9 @@ revoke execute on function public.probe_fn() from public;`, ], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - requireLiveSuccess(query, "db query setup"); + requireCliSuccess(query, "db query setup"); - const diff = await runSupabaseLive( + const diff = await runSupabase( ["db", "diff", "--local", "--use-pg-delta", "-f", "revoke_public_execute"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); diff --git a/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts b/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts index 121f0d5978..eb5afb387c 100644 --- a/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts +++ b/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts @@ -2,23 +2,17 @@ import { existsSync } from "node:fs"; import { join } from "node:path"; import { expect } from "vitest"; -import { - liveDatabaseTargetArgs, - testLiveDataPlane, -} from "../../../../../tests/helpers/live-context.ts"; +import { liveDatabaseTargetArgs, test } from "../../../../../tests/helpers/live.ts"; -testLiveDataPlane( - "dumps the remote schema to a file", - async ({ run, dbUrl, projectRef, workspace }) => { - const outFile = join(workspace.path, "schema.sql"); - const result = await run([ - "db", - "dump", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "-f", - outFile, - ]); - expect(result.exitCode, result.stderr).toBe(0); - expect(existsSync(outFile)).toBe(true); - }, -); +test("dumps the remote schema to a file", async ({ run, dbUrl, projectRef, workspace }) => { + const outFile = join(workspace.path, "schema.sql"); + const result = await run([ + "db", + "dump", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "-f", + outFile, + ]); + expect(result.exitCode, result.stderr).toBe(0); + expect(existsSync(outFile)).toBe(true); +}); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts index ce40e7c12b..3bbf18d179 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts @@ -5,82 +5,69 @@ import { expect } from "vitest"; import { liveDatabaseTargetArgs, requireLiveSuccess, - testLiveDestructiveDataPlane, -} from "../../../../../tests/helpers/live-context.ts"; + test, +} from "../../../../../tests/helpers/live.ts"; -testLiveDestructiveDataPlane( - "pulls the remote schema after a local migration is applied", - async ({ run, dbUrl, projectRef, workspace }) => { - const version = `${Date.now()}${Math.floor(Math.random() * 10_000) - .toString() - .padStart(4, "0")}`; - const migrations = join(workspace.path, "supabase", "migrations"); - await mkdir(migrations, { recursive: true }); - const existingMigrations = new Set(await readdir(migrations)); - const migrationFile = join(migrations, `${version}_e2e_pull.sql`); - await writeFile(migrationFile, `create table if not exists e2e_pull_${version} (id int);\n`); +test("pulls the remote schema after a local migration is applied", async ({ + run, + dbUrl, + projectRef, + workspace, +}) => { + const version = `${Date.now()}${Math.floor(Math.random() * 10_000) + .toString() + .padStart(4, "0")}`; + const migrations = join(workspace.path, "supabase", "migrations"); + await mkdir(migrations, { recursive: true }); + const existingMigrations = new Set(await readdir(migrations)); + const migrationFile = join(migrations, `${version}_e2e_pull.sql`); + await writeFile(migrationFile, `create table if not exists e2e_pull_${version} (id int);\n`); - let targetError: unknown; - try { - const pushed = await run([ - "db", - "push", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--yes", - ]); - requireLiveSuccess(pushed, "db push setup"); + let targetError: unknown; + try { + const pushed = await run(["db", "push", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + requireLiveSuccess(pushed, "db push setup"); - const result = await run([ - "db", - "pull", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--yes", - ]); - expect(result.exitCode, result.stderr).toBe(0); - expect(`${result.stdout}${result.stderr}`).not.toMatch( - /dial|no route|connection refused|could not connect|server closed the connection|i\/o timeout/i, - ); - } catch (error) { - targetError = error; - } + const result = await run(["db", "pull", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(`${result.stdout}${result.stderr}`).not.toMatch( + /dial|no route|connection refused|could not connect|server closed the connection|i\/o timeout/i, + ); + } catch (error) { + targetError = error; + } - let cleanupError: unknown; - try { - // Remove all migrations created by this test before resetting. This - // includes both the seed migration and the migration generated by - // `db pull`; resetting with only the generated grant statements left - // behind can reference a table that no longer exists. - const currentMigrations = await readdir(migrations).catch(() => []); - for (const file of currentMigrations.filter( - (candidate) => !existingMigrations.has(candidate), - )) { - try { - await unlink(join(migrations, file)); - } catch (error) { - throw new Error( - `db pull cleanup could not remove test migration ${join(migrations, file)}: ${ - error instanceof Error ? error.message : String(error) - }`, - ); - } + let cleanupError: unknown; + try { + // Remove all migrations created by this test before resetting. This + // includes both the seed migration and the migration generated by + // `db pull`; resetting with only the generated grant statements left + // behind can reference a table that no longer exists. + const currentMigrations = await readdir(migrations).catch(() => []); + for (const file of currentMigrations.filter( + (candidate) => !existingMigrations.has(candidate), + )) { + try { + await unlink(join(migrations, file)); + } catch (error) { + throw new Error( + `db pull cleanup could not remove test migration ${join(migrations, file)}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); } - - const reset = await run([ - "db", - "reset", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--yes", - ]); - requireLiveSuccess(reset, "db reset cleanup after db pull"); - } catch (error) { - cleanupError = error; } - if (targetError !== undefined) { - throw targetError; - } - if (cleanupError !== undefined) { - throw cleanupError; - } - }, -); + const reset = await run(["db", "reset", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + requireLiveSuccess(reset, "db reset cleanup after db pull"); + } catch (error) { + cleanupError = error; + } + + if (targetError !== undefined) { + throw targetError; + } + if (cleanupError !== undefined) { + throw cleanupError; + } +}); diff --git a/apps/cli/src/legacy/commands/db/push/push.live.test.ts b/apps/cli/src/legacy/commands/db/push/push.live.test.ts index d6f846deda..6211a3d987 100644 --- a/apps/cli/src/legacy/commands/db/push/push.live.test.ts +++ b/apps/cli/src/legacy/commands/db/push/push.live.test.ts @@ -5,38 +5,30 @@ import { expect } from "vitest"; import { liveDatabaseTargetArgs, requireLiveSuccess, - testLiveDestructiveDataPlane, -} from "../../../../../tests/helpers/live-context.ts"; + test, +} from "../../../../../tests/helpers/live.ts"; -testLiveDestructiveDataPlane( - "pushes a local migration to the remote database", - async ({ run, dbUrl, projectRef, workspace }) => { - const version = `${Date.now()}${Math.floor(Math.random() * 10_000) - .toString() - .padStart(4, "0")}`; - const migrations = join(workspace.path, "supabase", "migrations"); - await mkdir(migrations, { recursive: true }); - const migrationFile = join(migrations, `${version}_e2e_push.sql`); - await writeFile(migrationFile, `create table if not exists e2e_push_${version} (id int);\n`); +test("pushes a local migration to the remote database", async ({ + run, + dbUrl, + projectRef, + workspace, +}) => { + const version = `${Date.now()}${Math.floor(Math.random() * 10_000) + .toString() + .padStart(4, "0")}`; + const migrations = join(workspace.path, "supabase", "migrations"); + await mkdir(migrations, { recursive: true }); + const migrationFile = join(migrations, `${version}_e2e_push.sql`); + await writeFile(migrationFile, `create table if not exists e2e_push_${version} (id int);\n`); - try { - const result = await run([ - "db", - "push", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--yes", - ]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).toContain("Finished supabase db push"); - } finally { - await unlink(migrationFile).catch(() => undefined); - const reset = await run([ - "db", - "reset", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--yes", - ]); - requireLiveSuccess(reset, "db reset cleanup after db push"); - } - }, -); + try { + const result = await run(["db", "push", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Finished supabase db push"); + } finally { + await unlink(migrationFile).catch(() => undefined); + const reset = await run(["db", "reset", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + requireLiveSuccess(reset, "db reset cleanup after db push"); + } +}); diff --git a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts index 34beb90397..8d49ce8c2a 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts @@ -5,38 +5,35 @@ import { expect } from "vitest"; import { liveDatabaseTargetArgs, requireLiveSuccess, - testLiveDestructiveDataPlane, -} from "../../../../../tests/helpers/live-context.ts"; + test, +} from "../../../../../tests/helpers/live.ts"; -testLiveDestructiveDataPlane( - "resets the remote database with local migrations", - async ({ run, dbUrl, projectRef, workspace }) => { - const version = `${Date.now()}${Math.floor(Math.random() * 10_000) - .toString() - .padStart(4, "0")}`; - const migrations = join(workspace.path, "supabase", "migrations"); - await mkdir(migrations, { recursive: true }); - const migrationFile = join(migrations, `${version}_e2e_reset.sql`); - await writeFile(migrationFile, `create table if not exists e2e_reset_${version} (id int);\n`); +test("resets the remote database with local migrations", async ({ + run, + dbUrl, + projectRef, + workspace, +}) => { + const version = `${Date.now()}${Math.floor(Math.random() * 10_000) + .toString() + .padStart(4, "0")}`; + const migrations = join(workspace.path, "supabase", "migrations"); + await mkdir(migrations, { recursive: true }); + const migrationFile = join(migrations, `${version}_e2e_reset.sql`); + await writeFile(migrationFile, `create table if not exists e2e_reset_${version} (id int);\n`); - try { - const result = await run([ - "db", - "reset", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--yes", - ]); - expect(result.exitCode, result.stderr).toBe(0); - expect(`${result.stdout}${result.stderr}`).toContain("Resetting remote database"); - } finally { - await unlink(migrationFile).catch(() => undefined); - const reset = await run([ - "db", - "reset", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--yes", - ]); - requireLiveSuccess(reset, "db reset cleanup"); - } - }, -); + try { + const result = await run([ + "db", + "reset", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + "--yes", + ]); + expect(result.exitCode, result.stderr).toBe(0); + expect(`${result.stdout}${result.stderr}`).toContain("Resetting remote database"); + } finally { + await unlink(migrationFile).catch(() => undefined); + const reset = await run(["db", "reset", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + requireLiveSuccess(reset, "db reset cleanup"); + } +}); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.live.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts similarity index 85% rename from apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.live.test.ts rename to apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts index 153690cdd0..bbdb5391a9 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.live.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts @@ -4,11 +4,8 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { afterAll, beforeAll, expect, test } from "vitest"; -import { - describeLocalStackLive, - runSupabaseLive, -} from "../../../../../../../tests/helpers/live.ts"; -import { requireLiveSuccess } from "../../../../../../../tests/helpers/live-context.ts"; +import { describe } from "vitest"; +import { requireCliSuccess, runSupabase } from "../../../../../../../tests/helpers/cli.ts"; const COMMAND_TIMEOUT_MS = 280_000; const SCENARIO_TIMEOUT_MS = 900_000; @@ -39,17 +36,17 @@ function migrationFiles(projectDir: string): ReadonlyArray { : []; } -describeLocalStackLive("db schema declarative sync (live)", () => { +describe("db schema declarative sync (e2e)", () => { let projectDir = ""; beforeAll(async () => { projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-live-")); - const init = await runSupabaseLive(["init"], { + const init = await runSupabase(["init"], { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS, }); - requireLiveSuccess(init, "init setup"); + requireCliSuccess(init, "init setup"); const configPath = path.join(projectDir, "supabase", "config.toml"); const config = readFileSync(configPath, "utf8"); @@ -68,7 +65,7 @@ describeLocalStackLive("db schema declarative sync (live)", () => { mkdirSync(schemasDir, { recursive: true }); writeFileSync(path.join(schemasDir, "public.sql"), initialDesiredSchema); - const start = await runSupabaseLive( + const start = await runSupabase( [ "start", "--exclude", @@ -86,12 +83,12 @@ describeLocalStackLive("db schema declarative sync (live)", () => { ], { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, ); - requireLiveSuccess(start, "start setup"); + requireCliSuccess(start, "start setup"); }, COMMAND_TIMEOUT_MS); afterAll(async () => { if (projectDir.length === 0) return; - await runSupabaseLive(["stop", "--no-backup"], { + await runSupabase(["stop", "--no-backup"], { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS, }).catch(() => undefined); @@ -102,7 +99,7 @@ describeLocalStackLive("db schema declarative sync (live)", () => { "applies a representative declarative schema and converges", { timeout: SCENARIO_TIMEOUT_MS }, async () => { - const sync = await runSupabaseLive( + const sync = await runSupabase( [ "db", "schema", @@ -130,13 +127,13 @@ describeLocalStackLive("db schema declarative sync (live)", () => { /CREATE\s+(?:SCHEMA|TABLE)\s+(?:IF\s+NOT\s+EXISTS\s+)?["']?(?:auth|storage|realtime)["']?/iu, ); - const reset = await runSupabaseLive(["db", "reset", "--local", "--no-seed"], { + const reset = await runSupabase(["db", "reset", "--local", "--no-seed"], { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS, }); - requireLiveSuccess(reset, "db reset setup"); + requireCliSuccess(reset, "db reset setup"); - const converged = await runSupabaseLive( + const converged = await runSupabase( ["db", "schema", "declarative", "sync", "--no-apply", "--experimental"], { cwd: projectDir, diff --git a/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts new file mode 100644 index 0000000000..2798bdfad4 --- /dev/null +++ b/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts @@ -0,0 +1,17 @@ +import { describe, expect, test } from "vitest"; + +import { runSupabase } from "../../../../../tests/helpers/cli.ts"; + +describe("supabase db start (e2e)", () => { + test("boots the local database", async () => { + try { + const started = await runSupabase(["db", "start"]); + expect(started.exitCode, started.stderr).toBe(0); + expect(`${started.stdout}${started.stderr}`).toMatch( + /Starting database|Initialising schema/i, + ); + } finally { + await runSupabase(["stop", "--no-backup"]).catch(() => undefined); + } + }, 600_000); +}); diff --git a/apps/cli/src/legacy/commands/db/start/start.live.test.ts b/apps/cli/src/legacy/commands/db/start/start.live.test.ts deleted file mode 100644 index 00d2fa6ab3..0000000000 --- a/apps/cli/src/legacy/commands/db/start/start.live.test.ts +++ /dev/null @@ -1,22 +0,0 @@ -import { expect } from "vitest"; - -import { describeLocalStackLive } from "../../../../../tests/helpers/live.ts"; -import { testLive } from "../../../../../tests/helpers/live-context.ts"; - -describeLocalStackLive("supabase db start (live)", () => { - testLive( - "boots the local database", - async ({ run }) => { - try { - const started = await run(["db", "start"]); - expect(started.exitCode, started.stderr).toBe(0); - expect(`${started.stdout}${started.stderr}`).toMatch( - /Starting database|Initialising schema/i, - ); - } finally { - await run(["stop", "--no-backup"]).catch(() => undefined); - } - }, - 600_000, - ); -}); diff --git a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts index b4eb173da2..774e2e8e10 100644 --- a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts @@ -2,9 +2,9 @@ import { randomUUID } from "node:crypto"; import { mkdir, writeFile } from "node:fs/promises"; import { expect } from "vitest"; -import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -testLiveProject("deletes a deployed function", async ({ run, projectRef, workspace }) => { +test("deletes a deployed function", async ({ run, projectRef, workspace }) => { const slug = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; const directory = `${workspace.path}/supabase/functions/${slug}`; await mkdir(directory, { recursive: true }); diff --git a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts index 6b2079ceb4..4c447a0b69 100644 --- a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts @@ -2,41 +2,39 @@ import { randomUUID } from "node:crypto"; import { mkdir, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; +import { describe } from "vitest"; -import { describeDockerLive } from "../../../../../tests/helpers/live.ts"; -import { - expectFunctionOk, - requireLiveSuccess, - testLiveFunctions, -} from "../../../../../tests/helpers/live-context.ts"; +import { expectFunctionOk, requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -describeDockerLive("functions deploy (live)", () => { - testLiveFunctions( - "deploys a function that responds over HTTP", - async ({ run, invoke, projectRef, workspace }) => { - const slug = `cli-e2e-deploy-${randomUUID().slice(0, 8)}`; - const directory = join(workspace.path, "supabase", "functions", slug); - await mkdir(directory, { recursive: true }); - await writeFile( - join(directory, "index.ts"), - `Deno.serve(() => Response.json({ case: ${JSON.stringify(slug)}, ok: true }));\n`, - ); - await writeFile(join(directory, "deno.json"), '{\n "imports": {}\n}\n'); +describe("functions deploy (live)", () => { + test("deploys a function that responds over HTTP", async ({ + cli, + invoke, + project, + workspace, + }) => { + const slug = `cli-e2e-deploy-${randomUUID().slice(0, 8)}`; + const directory = join(workspace.path, "supabase", "functions", slug); + await mkdir(directory, { recursive: true }); + await writeFile( + join(directory, "index.ts"), + `Deno.serve(() => Response.json({ case: ${JSON.stringify(slug)}, ok: true }));\n`, + ); + await writeFile(join(directory, "deno.json"), '{\n "imports": {}\n}\n'); - let deployed = false; - try { - const result = await run(["functions", "deploy", "--project-ref", projectRef]); - expect(result.exitCode, result.stderr).toBe(0); - deployed = true; - expect(result.stdout).toMatch(/Deployed Function/i); + let deployed = false; + try { + const result = await cli(["functions", "deploy", "--project-ref", project.ref]); + expect(result.exitCode, result.stderr).toBe(0); + deployed = true; + expect(result.stdout).toMatch(/Deployed Function/i); - expectFunctionOk(await invoke(slug), slug); - } finally { - if (deployed) { - const deleted = await run(["functions", "delete", slug, "--project-ref", projectRef]); - requireLiveSuccess(deleted, "functions delete cleanup"); - } + expectFunctionOk(await invoke(slug), slug); + } finally { + if (deployed) { + const deleted = await cli(["functions", "delete", slug, "--project-ref", project.ref]); + requireLiveSuccess(deleted, "functions delete cleanup"); } - }, - ); + } + }); }); diff --git a/apps/cli/src/legacy/commands/functions/list/list.live.test.ts b/apps/cli/src/legacy/commands/functions/list/list.live.test.ts index 2bfa93b86f..daa1f8ef2d 100644 --- a/apps/cli/src/legacy/commands/functions/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/list/list.live.test.ts @@ -1,52 +1,50 @@ -import { expect, test } from "vitest"; +import { describe, expect } from "vitest"; -import { - describeLive, - describeLiveProject, - requireLiveProjectRef, - runSupabaseLive, -} from "../../../../../tests/helpers/live.ts"; +import { test } from "../../../../../tests/helpers/live.ts"; const LIVE_TIMEOUT_MS = 120_000; -// Project-scoped read-only scenario. Skipped unless SUPABASE_LIVE_PROJECT_REF is -// set — i.e. a project has been provisioned on the stack (the cli-e2e-ci runner -// does this; a control-plane-only stack, like local macOS, skips it). -// // This is the entry point for the broader edge-functions coverage tracked in // CLI-1834 (deploy + invoke over :443 / {ref}.supabase.red), which needs the // project's gateway reachable from the host — author those here as they become // runnable on the full stack. -describeLiveProject("supabase functions list (live)", () => { - test("lists edge functions for the project", { timeout: LIVE_TIMEOUT_MS }, async () => { - const ref = requireLiveProjectRef(); - const { exitCode, stdout, stderr } = await runSupabaseLive([ - "functions", - "list", - "--project-ref", - ref, - ]); - expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); - expect(exitCode).toBe(0); - }); +describe("supabase functions list (live)", () => { + test( + "lists edge functions for the project", + { timeout: LIVE_TIMEOUT_MS }, + async ({ cli, project }) => { + const { exitCode, stdout, stderr } = await cli([ + "functions", + "list", + "--project-ref", + project.ref, + ]); + expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); + expect(exitCode).toBe(0); + }, + ); }); // Project-scoped error path that needs NO provisioned project: a valid token // with an unknown `--project-ref` must reach the live Management API, come back // 404, and surface as a non-zero exit (not a crash, not "Unauthorized"). This // exercises the `--project-ref` request path + error mapping on a control-plane- -// only stack, so it runs under `describeLive`, not `describeLiveProject`. -describeLive("supabase functions list — unknown project (live)", () => { - test("fails with a 404 for an unknown project ref", { timeout: LIVE_TIMEOUT_MS }, async () => { - const { exitCode, stdout, stderr } = await runSupabaseLive([ - "functions", - "list", - "--project-ref", - "a".repeat(20), // well-formed (20 lowercase chars) but nonexistent ref - ]); - const out = `${stdout}${stderr}`; - expect(exitCode).not.toBe(0); - expect(out).not.toContain("Unauthorized"); - expect(out).toContain("404"); - }); +// only stack, so it runs under the same shared live fixture. +describe("supabase functions list — unknown project (live)", () => { + test( + "fails with a 404 for an unknown project ref", + { timeout: LIVE_TIMEOUT_MS }, + async ({ cli }) => { + const { exitCode, stdout, stderr } = await cli([ + "functions", + "list", + "--project-ref", + "a".repeat(20), // well-formed (20 lowercase chars) but nonexistent ref + ]); + const out = `${stdout}${stderr}`; + expect(exitCode).not.toBe(0); + expect(out).not.toContain("Unauthorized"); + expect(out).toContain("404"); + }, + ); }); diff --git a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts index cb7cd2fd2a..b5b8dd79b7 100644 --- a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts +++ b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts @@ -1,13 +1,10 @@ import { expect } from "vitest"; -import { testLiveDataPlane } from "../../../../../tests/helpers/live-context.ts"; +import { test } from "../../../../../tests/helpers/live.ts"; -testLiveDataPlane( - "generates TypeScript types from the remote schema", - async ({ run, dbUrl, projectRef }) => { - const targetArgs = dbUrl.length > 0 ? ["--db-url", dbUrl] : ["--project-id", projectRef]; - const result = await run(["gen", "types", ...targetArgs, "--lang", "typescript"]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).toMatch(/export type (Database|Json)/); - }, -); +test("generates TypeScript types from the remote schema", async ({ run, dbUrl, projectRef }) => { + const targetArgs = dbUrl.length > 0 ? ["--db-url", dbUrl] : ["--project-id", projectRef]; + const result = await run(["gen", "types", ...targetArgs, "--lang", "typescript"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toMatch(/export type (Database|Json)/); +}); diff --git a/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts b/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts index 3a27a75b65..b143ca9d8a 100644 --- a/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts +++ b/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts @@ -1,20 +1,14 @@ import { expect } from "vitest"; -import { - liveDatabaseTargetArgs, - testLiveDataPlane, -} from "../../../../../../tests/helpers/live-context.ts"; +import { liveDatabaseTargetArgs, test } from "../../../../../../tests/helpers/live.ts"; -testLiveDataPlane( - "reports statistics from the remote database", - async ({ run, dbUrl, projectRef }) => { - const result = await run([ - "inspect", - "db", - "db-stats", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - ]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).toContain("Database Size"); - }, -); +test("reports statistics from the remote database", async ({ run, dbUrl, projectRef }) => { + const result = await run([ + "inspect", + "db", + "db-stats", + ...liveDatabaseTargetArgs(dbUrl, projectRef), + ]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Database Size"); +}); diff --git a/apps/cli/src/legacy/commands/link/link.live.test.ts b/apps/cli/src/legacy/commands/link/link.live.test.ts index b5b790a760..293e79788a 100644 --- a/apps/cli/src/legacy/commands/link/link.live.test.ts +++ b/apps/cli/src/legacy/commands/link/link.live.test.ts @@ -2,14 +2,11 @@ import { existsSync } from "node:fs"; import { join } from "node:path"; import { expect } from "vitest"; -import { testLiveProject } from "../../../../tests/helpers/live-context.ts"; +import { test } from "../../../../tests/helpers/live.ts"; -testLiveProject( - "links a project and writes its workspace cache", - async ({ run, projectRef, workspace }) => { - const result = await run(["link", "--project-ref", projectRef, "--skip-pooler"]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).toContain("Finished supabase link"); - expect(existsSync(join(workspace.path, "supabase", ".temp", "linked-project.json"))).toBe(true); - }, -); +test("links a project and writes its workspace cache", async ({ run, projectRef, workspace }) => { + const result = await run(["link", "--project-ref", projectRef, "--skip-pooler"]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain("Finished supabase link"); + expect(existsSync(join(workspace.path, "supabase", ".temp", "linked-project.json"))).toBe(true); +}); diff --git a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts index 0845367046..f8b61db945 100644 --- a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts @@ -6,8 +6,8 @@ import { expect } from "vitest"; import { liveDatabaseTargetArgs, requireLiveSuccess, - testLiveDestructiveDataPlane, -} from "../../../../../tests/helpers/live-context.ts"; + test, +} from "../../../../../tests/helpers/live.ts"; const LIVE_TIMEOUT_MS = 120_000; @@ -34,7 +34,7 @@ function liveMigrationVersion(): string { // table then upserts the version from the local file), establishing // the table + a row for `fetch` to read back. The shared fixture's pooler URL is // passed explicitly so the test does not fall back to a direct IPv6 host. -testLiveDestructiveDataPlane( +test( "fetches a seeded remote migration into the local migrations directory", { timeout: LIVE_TIMEOUT_MS }, async ({ run, dbUrl, projectRef }) => { diff --git a/apps/cli/src/legacy/commands/migration/list/list.live.test.ts b/apps/cli/src/legacy/commands/migration/list/list.live.test.ts index 2a29c2ee53..e8ef58dd11 100644 --- a/apps/cli/src/legacy/commands/migration/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/list/list.live.test.ts @@ -1,20 +1,14 @@ import { expect } from "vitest"; -import { - liveDatabaseTargetArgs, - testLiveDataPlane, -} from "../../../../../tests/helpers/live-context.ts"; +import { liveDatabaseTargetArgs, test } from "../../../../../tests/helpers/live.ts"; -testLiveDataPlane( - "lists migrations from the remote database", - async ({ run, dbUrl, projectRef }) => { - const result = await run(["migration", "list", ...liveDatabaseTargetArgs(dbUrl, projectRef)]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).not.toContain("Unauthorized"); - }, -); +test("lists migrations from the remote database", async ({ run, dbUrl, projectRef }) => { + const result = await run(["migration", "list", ...liveDatabaseTargetArgs(dbUrl, projectRef)]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).not.toContain("Unauthorized"); +}); -testLiveDataPlane("emits migration list as JSON", async ({ run, dbUrl, projectRef }) => { +test("emits migration list as JSON", async ({ run, dbUrl, projectRef }) => { const result = await run([ "migration", "list", diff --git a/apps/cli/src/legacy/commands/orgs/list/list.live.test.ts b/apps/cli/src/legacy/commands/orgs/list/list.live.test.ts index 515e8a855d..2e1dcf7e88 100644 --- a/apps/cli/src/legacy/commands/orgs/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/orgs/list/list.live.test.ts @@ -1,5 +1,5 @@ -import { expect, test } from "vitest"; -import { describeLive, runSupabaseLive } from "../../../../../tests/helpers/live.ts"; +import { describe, expect } from "vitest"; +import { test } from "../../../../../tests/helpers/live.ts"; const LIVE_TIMEOUT_MS = 60_000; @@ -8,15 +8,15 @@ const LIVE_TIMEOUT_MS = 60_000; // → authenticated Management API request against the running platform — with a // read-only call, so it is safe to run repeatedly and creates no resources. // -// Gated by `describeLive`: skipped unless SUPABASE_ACCESS_TOKEN is set (the +// Gated by `describe`: skipped unless SUPABASE_ACCESS_TOKEN is set (the // cli-e2e-ci runner provides supabox's seeded PAT). Broader lifecycle scenarios // (projects, functions, branching, db, storage) build on this same harness. -describeLive("supabase orgs list (live)", () => { +describe("supabase orgs list (live)", () => { test( "lists organizations for the authenticated token", { timeout: LIVE_TIMEOUT_MS }, - async () => { - const { exitCode, stdout, stderr } = await runSupabaseLive(["orgs", "list"]); + async ({ cli }) => { + const { exitCode, stdout, stderr } = await cli(["orgs", "list"]); expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); expect(exitCode).toBe(0); }, @@ -25,13 +25,8 @@ describeLive("supabase orgs list (live)", () => { test( "emits machine-readable JSON with --output-format json", { timeout: LIVE_TIMEOUT_MS }, - async () => { - const { exitCode, stdout } = await runSupabaseLive([ - "orgs", - "list", - "--output-format", - "json", - ]); + async ({ cli }) => { + const { exitCode, stdout } = await cli(["orgs", "list", "--output-format", "json"]); expect(exitCode).toBe(0); // stdout must be payload-only valid JSON in json mode (no spinner/log noise). expect(() => JSON.parse(stdout)).not.toThrow(); @@ -42,11 +37,15 @@ describeLive("supabase orgs list (live)", () => { // back 401, and surface as a non-zero exit with the upstream "Unauthorized" // message — i.e. the cli's auth + error mapping work against the live stack, // not just the golden path. Overrides only the token (profile stays set). - test("fails with Unauthorized for an invalid token", { timeout: LIVE_TIMEOUT_MS }, async () => { - const { exitCode, stdout, stderr } = await runSupabaseLive(["orgs", "list"], { - env: { SUPABASE_ACCESS_TOKEN: `sbp_${"0".repeat(40)}` }, - }); - expect(exitCode).not.toBe(0); - expect(`${stdout}${stderr}`).toContain("Unauthorized"); - }); + test( + "fails with Unauthorized for an invalid token", + { timeout: LIVE_TIMEOUT_MS }, + async ({ cli }) => { + const { exitCode, stdout, stderr } = await cli(["orgs", "list"], { + env: { SUPABASE_ACCESS_TOKEN: `sbp_${"0".repeat(40)}` }, + }); + expect(exitCode).not.toBe(0); + expect(`${stdout}${stderr}`).toContain("Unauthorized"); + }, + ); }); diff --git a/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts b/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts index 08cdfbb250..440847789a 100644 --- a/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts +++ b/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts @@ -1,8 +1,8 @@ import { expect } from "vitest"; -import { testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { test } from "../../../../../tests/helpers/live.ts"; -testLiveProject("lists API keys for a project", async ({ run, projectRef }) => { +test("lists API keys for a project", async ({ run, projectRef }) => { const result = await run([ "projects", "api-keys", diff --git a/apps/cli/src/legacy/commands/projects/list/list.live.test.ts b/apps/cli/src/legacy/commands/projects/list/list.live.test.ts index 8737bc5315..54aba6edda 100644 --- a/apps/cli/src/legacy/commands/projects/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/projects/list/list.live.test.ts @@ -1,33 +1,30 @@ import { expect } from "vitest"; -import { testLive, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { test } from "../../../../../tests/helpers/live.ts"; -testLiveProject( - "lists the live project for the authenticated token", - async ({ run, projectRef }) => { - const result = await run(["projects", "list", "--output-format", "json"]); - expect(result.exitCode, result.stderr).toBe(0); - const parsed: unknown = JSON.parse(result.stdout); - expect(parsed).toEqual(expect.objectContaining({ projects: expect.any(Array) })); - if ( - parsed === null || - typeof parsed !== "object" || - !("projects" in parsed) || - !Array.isArray(parsed.projects) - ) { - throw new Error("projects list JSON response did not contain a projects array"); - } - const refs = parsed.projects.flatMap((project) => { - if (project === null || typeof project !== "object") return []; - if ("ref" in project && typeof project.ref === "string") return [project.ref]; - if ("id" in project && typeof project.id === "string") return [project.id]; - return []; - }); - expect(refs).toContain(projectRef); - }, -); +test("lists the live project for the authenticated token", async ({ run, projectRef }) => { + const result = await run(["projects", "list", "--output-format", "json"]); + expect(result.exitCode, result.stderr).toBe(0); + const parsed: unknown = JSON.parse(result.stdout); + expect(parsed).toEqual(expect.objectContaining({ projects: expect.any(Array) })); + if ( + parsed === null || + typeof parsed !== "object" || + !("projects" in parsed) || + !Array.isArray(parsed.projects) + ) { + throw new Error("projects list JSON response did not contain a projects array"); + } + const refs = parsed.projects.flatMap((project) => { + if (project === null || typeof project !== "object") return []; + if ("ref" in project && typeof project.ref === "string") return [project.ref]; + if ("id" in project && typeof project.id === "string") return [project.id]; + return []; + }); + expect(refs).toContain(projectRef); +}); -testLive("emits projects as JSON for an account-level read", async ({ run }) => { +test("emits projects as JSON for an account-level read", async ({ run }) => { const result = await run(["projects", "list", "--output-format", "json"]); expect(result.exitCode, result.stderr).toBe(0); expect(() => JSON.parse(result.stdout)).not.toThrow(); diff --git a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts index ebca088bf3..1385301f19 100644 --- a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, @@ -14,7 +14,7 @@ async function unsetSecret( } } -testLiveProject("lists a secret created on the remote project", async ({ run, projectRef }) => { +test("lists a secret created on the remote project", async ({ run, projectRef }) => { const name = `CLI_E2E_LIST_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; const created = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); requireLiveSuccess(created, "secrets set setup"); diff --git a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts index fc24a48e53..82cb1294aa 100644 --- a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { test } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, @@ -14,7 +14,7 @@ async function unsetSecret( } } -testLiveProject("sets a secret on the remote project", async ({ run, projectRef }) => { +test("sets a secret on the remote project", async ({ run, projectRef }) => { const name = `CLI_E2E_SET_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; const result = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); try { diff --git a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts index 314e265a15..7dc3771590 100644 --- a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { requireLiveSuccess, testLiveProject } from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, @@ -14,7 +14,7 @@ async function unsetSecret( } } -testLiveProject("unsets a secret from the remote project", async ({ run, projectRef }) => { +test("unsets a secret from the remote project", async ({ run, projectRef }) => { const name = `CLI_E2E_UNSET_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; const created = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); requireLiveSuccess(created, "secrets set setup"); diff --git a/apps/cli/src/legacy/commands/start/start.live.test.ts b/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts similarity index 88% rename from apps/cli/src/legacy/commands/start/start.live.test.ts rename to apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts index fc664f4487..b2ece5fc36 100644 --- a/apps/cli/src/legacy/commands/start/start.live.test.ts +++ b/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts @@ -5,10 +5,9 @@ import { createServer } from "node:net"; import { tmpdir } from "node:os"; import path from "node:path"; import { promisify } from "node:util"; -import { afterEach, expect, test } from "vitest"; +import { afterEach, describe, expect, test } from "vitest"; -import { describeLocalStackLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; -import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; +import { requireCliSuccess, runSupabase } from "../../../../tests/helpers/cli.ts"; import { legacySanitizeProjectId, legacyServiceContainerName, @@ -26,7 +25,7 @@ const LIFECYCLE_OVERHEAD_MS = 90_000; /** * `--exclude` values for the 3 heaviest/least-relevant services — same intent - * `stop.live.test.ts`/`status.live.test.ts` already documented for their own + * `stop.e2e.test.ts`/`status.e2e.test.ts` already documented for their own * reduced-stack `start` call (Studio's Next.js build, the Logflare/Vector * logging pipeline), but "logflare" here, NOT "analytics" like those two * siblings. `LEGACY_SERVICE_CATALOG`'s `excludeKey` for the logflare service @@ -43,10 +42,10 @@ const EXCLUDED_SERVICE_KEYS: ReadonlySet = new Set(["studio", "logflare" * Services the running-container assertion below must NOT expect to be running, even though * they are neither in `EXCLUDED_SERVICE_KEYS` nor `--exclude`d on the `start` call itself: * - `supavisor` — `db.pooler.enabled` defaults to `false` (`packages/config/src/db.ts`, - * `defaultPoolerEnabled`), and `runSupabaseLive(["init"], ...)` above writes a config.toml + * `defaultPoolerEnabled`), and `runSupabase(["init"], ...)` above writes a config.toml * with no override, so it's genuinely disabled on this test's stack, not merely unasserted. * - `imgproxy` — gated on `storage.image_transformation.enabled` (`start.gates.ts:169`), - * which defaults to `false`/absent; `runSupabaseLive(["init"], ...)` writes a config.toml + * which defaults to `false`/absent; `runSupabase(["init"], ...)` writes a config.toml * with `[storage.image_transformation]` still commented out * (`project-init.templates.ts:132-133`), so imgproxy is genuinely disabled on this test's stack. */ @@ -61,18 +60,18 @@ function splitNonEmptyLines(text: string): ReadonlyArray { // `start` is the one local-dev-stack command whose correctness genuinely // depends on a real Docker daemon — real label filtering and real container -// lifecycle, not just CLI exit codes. `describeLocalStackLive` gates the -// "we're in a configured live runner" signal (see stop.live.test.ts's own +// lifecycle, not just CLI exit codes. `describe` gates the +// "we're in a configured e2e runner" signal (see stop.e2e.test.ts's own // comment for why this, not a Management-API gate, is correct here). See -// AGENTS.md's "Live tests" section for the full convention. -describeLocalStackLive("supabase start (live)", () => { +// AGENTS.md's "e2e tests" section for the full convention. +describe("supabase start (e2e)", () => { let projectDir: string | undefined; afterEach(async () => { if (projectDir === undefined) return; // Best-effort cleanup even if an assertion above failed mid-lifecycle — a // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabaseLive(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); projectDir = undefined; }); @@ -99,13 +98,13 @@ describeLocalStackLive("supabase start (live)", () => { "vector", ]; - const init = await runSupabaseLive(["init"], { + const init = await runSupabase(["init"], { cwd: projectDir, exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, }); - requireLiveSuccess(init, "init setup"); + requireCliSuccess(init, "init setup"); - const start = await runSupabaseLive(startArgs, { + const start = await runSupabase(startArgs, { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS, }); @@ -151,7 +150,7 @@ describeLocalStackLive("supabase start (live)", () => { Status: "exited", }); - const restart = await runSupabaseLive(startArgs, { + const restart = await runSupabase(startArgs, { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS, }); @@ -196,11 +195,11 @@ describeLocalStackLive("supabase start (live)", () => { ).toBe(!isExcluded); } - const status = await runSupabaseLive(["status"], { + const status = await runSupabase(["status"], { cwd: projectDir, exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, }); - requireLiveSuccess(status, "status setup"); + requireCliSuccess(status, "status setup"); }, ); @@ -210,11 +209,11 @@ describeLocalStackLive("supabase start (live)", () => { async () => { projectDir = await mkdtemp(path.join(tmpdir(), "sb-start-live-proxy-")); - const init = await runSupabaseLive(["init"], { + const init = await runSupabase(["init"], { cwd: projectDir, exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, }); - requireLiveSuccess(init, "init setup"); + requireCliSuccess(init, "init setup"); let proxyConnections = 0; const proxy = createServer((socket) => { @@ -238,7 +237,7 @@ describeLocalStackLive("supabase start (live)", () => { : ["--exclude", entry.excludeKey], ); const proxyUrl = `http://127.0.0.1:${address.port}`; - const start = await runSupabaseLive(["start", ...excludeArgs], { + const start = await runSupabase(["start", ...excludeArgs], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS, env: { @@ -281,11 +280,11 @@ describeLocalStackLive("supabase start (live)", () => { // finds this deliberately broken build and never reaches a registry. const mailpitImage = legacyGetRegistryImageUrl(dockerfileServiceImage("mailpit")); - const init = await runSupabaseLive(["init"], { + const init = await runSupabase(["init"], { cwd: projectDir, exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, }); - requireLiveSuccess(init, "init setup"); + requireCliSuccess(init, "init setup"); // A `scratch` image whose entrypoint is not an executable binary — the // kernel refuses it with exactly the "exec format error" this diagnoses. @@ -306,7 +305,7 @@ describeLocalStackLive("supabase start (live)", () => { ? [] : ["--exclude", entry.excludeKey], ); - const start = await runSupabaseLive(["start", ...excludeArgs], { + const start = await runSupabase(["start", ...excludeArgs], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS, }); diff --git a/apps/cli/src/legacy/commands/status/status.live.test.ts b/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts similarity index 66% rename from apps/cli/src/legacy/commands/status/status.live.test.ts rename to apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts index c97c2827e7..14425148ca 100644 --- a/apps/cli/src/legacy/commands/status/status.live.test.ts +++ b/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts @@ -3,21 +3,21 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { afterEach, expect, test } from "vitest"; -import { describeLocalStackLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; -import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; +import { describe } from "vitest"; +import { requireCliSuccess, runSupabase } from "../../../../tests/helpers/cli.ts"; const START_TIMEOUT_MS = 280_000; -// See stop.live.test.ts for why `describeLocalStackLive` (not a Management-API gate) is +// See stop.e2e.test.ts for why `describe` (not a Management-API gate) is // the right reuse here: `status` never calls the Management API, only the real -// Docker daemon the cli-e2e-ci runner provides. See AGENTS.md's "Live tests" +// Docker daemon the cli-e2e-ci runner provides. See AGENTS.md's "e2e tests" // section for the full convention. -describeLocalStackLive("supabase status (live)", () => { +describe("supabase status (e2e)", () => { let projectDir: string | undefined; afterEach(async () => { if (projectDir === undefined) return; - await runSupabaseLive(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); projectDir = undefined; }); @@ -28,22 +28,22 @@ describeLocalStackLive("supabase status (live)", () => { async () => { projectDir = await mkdtemp(path.join(tmpdir(), "sb-status-live-")); - const init = await runSupabaseLive(["init"], { cwd: projectDir }); - requireLiveSuccess(init, "init setup"); + const init = await runSupabase(["init"], { cwd: projectDir }); + requireCliSuccess(init, "init setup"); - const start = await runSupabaseLive( + const start = await runSupabase( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - requireLiveSuccess(start, "start setup"); + requireCliSuccess(start, "start setup"); - const pretty = await runSupabaseLive(["status"], { cwd: projectDir }); + const pretty = await runSupabase(["status"], { cwd: projectDir }); expect(pretty.exitCode, `stdout:\n${pretty.stdout}\nstderr:\n${pretty.stderr}`).toBe(0); expect(`${pretty.stdout}${pretty.stderr}`).toContain("is running"); expect(pretty.stdout).toContain("Project URL"); expect(pretty.stdout).toContain("Database"); - const json = await runSupabaseLive(["status", "-o", "json"], { cwd: projectDir }); + const json = await runSupabase(["status", "-o", "json"], { cwd: projectDir }); expect(json.exitCode, `stdout:\n${json.stdout}\nstderr:\n${json.stderr}`).toBe(0); const parsed: unknown = JSON.parse(json.stdout); expect(parsed).toMatchObject({ diff --git a/apps/cli/src/legacy/commands/stop/stop.live.test.ts b/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts similarity index 79% rename from apps/cli/src/legacy/commands/stop/stop.live.test.ts rename to apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts index 0a1d2a1c4c..dcd252bde8 100644 --- a/apps/cli/src/legacy/commands/stop/stop.live.test.ts +++ b/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts @@ -3,10 +3,9 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { promisify } from "node:util"; -import { afterEach, expect, test } from "vitest"; +import { afterEach, describe, expect, test } from "vitest"; -import { describeLocalStackLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; -import { requireLiveSuccess } from "../../../../tests/helpers/live-context.ts"; +import { requireCliSuccess, runSupabase } from "../../../../tests/helpers/cli.ts"; import { legacySanitizeProjectId } from "../../shared/legacy-docker-ids.ts"; const execFileAsync = promisify(execFile); @@ -14,12 +13,12 @@ const execFileAsync = promisify(execFile); const START_TIMEOUT_MS = 280_000; // `stop` never calls the Management API — it talks directly to the real local -// Docker stack `start` creates. `describeLocalStackLive` gates +// Docker stack `start` creates. `describe` gates // purely as the "we're in the full cli-e2e-ci runner" signal (it also has a // real Docker daemon, since that's how supabox itself runs); the // SUPABASE_ACCESS_TOKEN it gates on is otherwise irrelevant here. See -// AGENTS.md's "Live tests" section for the full convention. -describeLocalStackLive("supabase stop (live)", () => { +// AGENTS.md's "e2e tests" section for the full convention. +describe("supabase stop (e2e)", () => { let projectDir: string | undefined; let projectId: string | undefined; @@ -27,7 +26,7 @@ describeLocalStackLive("supabase stop (live)", () => { if (projectDir === undefined) return; // Best-effort cleanup even if an assertion above failed mid-lifecycle — a // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabaseLive(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); projectDir = undefined; projectId = undefined; @@ -42,24 +41,24 @@ describeLocalStackLive("supabase stop (live)", () => { // basename (see legacy-docker-ids.ts). projectId = path.basename(projectDir); - const init = await runSupabaseLive(["init"], { cwd: projectDir }); - requireLiveSuccess(init, "init setup"); + const init = await runSupabase(["init"], { cwd: projectDir }); + requireCliSuccess(init, "init setup"); // Exclude the heaviest, least relevant services (Next.js Studio build, the // logging pipeline) — `stop`'s Docker label-filtering logic doesn't care // which services are running, only that at least one real container // exists to stop. - const start = await runSupabaseLive( + const start = await runSupabase( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - requireLiveSuccess(start, "start setup"); + requireCliSuccess(start, "start setup"); // Sanity: confirm the stack is actually up before testing `stop` against it. - const before = await runSupabaseLive(["status"], { cwd: projectDir }); - requireLiveSuccess(before, "status setup"); + const before = await runSupabase(["status"], { cwd: projectDir }); + requireCliSuccess(before, "status setup"); - const stop = await runSupabaseLive(["stop"], { cwd: projectDir }); + const stop = await runSupabase(["stop"], { cwd: projectDir }); expect(stop.exitCode, `stdout:\n${stop.stdout}\nstderr:\n${stop.stderr}`).toBe(0); expect(stop.stdout).toContain("Stopped"); @@ -85,24 +84,24 @@ describeLocalStackLive("supabase stop (live)", () => { projectDir = await mkdtemp(path.join(tmpdir(), "sb-stop-live-")); // Sanitizing is a no-op for a `mkdtemp`-generated basename (already // alphanumeric/`-`), but mirrors the port's actual resolution rather - // than assuming that stays true (same note as `start.live.test.ts`). + // than assuming that stays true (same note as `start.e2e.test.ts`). projectId = legacySanitizeProjectId(path.basename(projectDir)); - const init = await runSupabaseLive(["init"], { cwd: projectDir }); - requireLiveSuccess(init, "init setup"); + const init = await runSupabase(["init"], { cwd: projectDir }); + requireCliSuccess(init, "init setup"); - const start = await runSupabaseLive( + const start = await runSupabase( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - requireLiveSuccess(start, "start setup"); + requireCliSuccess(start, "start setup"); // `--no-backup` exercises the volume-prune branch; `--debug` turns on // the `Pruned …:` stderr reports, which are // backed by parsing REAL `docker`/`podman` prune stdout — the format // assumption (`Deleted …:` headers, `Total reclaimed space:` trailer) // that mocked integration fixtures cannot validate by construction. - const stop = await runSupabaseLive(["stop", "--no-backup", "--debug"], { cwd: projectDir }); + const stop = await runSupabase(["stop", "--no-backup", "--debug"], { cwd: projectDir }); expect(stop.exitCode, `stdout:\n${stop.stdout}\nstderr:\n${stop.stderr}`).toBe(0); expect(stop.stdout).toContain("Stopped"); diff --git a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts index 9350b6050e..a45816bed2 100644 --- a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts @@ -3,7 +3,7 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, testLiveStorage } from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; @@ -17,26 +17,29 @@ async function removeObject( } } -testLiveStorage( - "copies a local file to the remote bucket", - async ({ run, projectRef, dbPassword, storageBucket, workspace }) => { - const suffix = randomUUID().slice(0, 8); - const local = join(workspace.path, `upload-${suffix}.txt`); - const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; - await writeFile(local, "live-e2e storage payload\n"); +test("copies a local file to the remote bucket", async ({ + run, + projectRef, + dbPassword, + storageBucket, + workspace, +}) => { + const suffix = randomUUID().slice(0, 8); + const local = join(workspace.path, `upload-${suffix}.txt`); + const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + await writeFile(local, "live-e2e storage payload\n"); - const linked = await run(["link", "--project-ref", projectRef], { - env: { SUPABASE_DB_PASSWORD: dbPassword }, - }); - requireLiveSuccess(linked, "link setup for storage cp"); + const linked = await run(["link", "--project-ref", projectRef], { + env: { SUPABASE_DB_PASSWORD: dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage cp"); - let uploaded = false; - try { - const result = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); - expect(result.exitCode, result.stderr).toBe(0); - uploaded = true; - } finally { - if (uploaded) await removeObject(run, remote); - } - }, -); + let uploaded = false; + try { + const result = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + expect(result.exitCode, result.stderr).toBe(0); + uploaded = true; + } finally { + if (uploaded) await removeObject(run, remote); + } +}); diff --git a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts index d0147d611e..e963b1418b 100644 --- a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts @@ -3,7 +3,7 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, testLiveStorage } from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; @@ -17,27 +17,30 @@ async function removeObject( } } -testLiveStorage( - "lists an uploaded object", - async ({ run, projectRef, dbPassword, storageBucket, workspace }) => { - const suffix = randomUUID().slice(0, 8); - const local = join(workspace.path, `upload-${suffix}.txt`); - const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; - await writeFile(local, "live-e2e storage payload\n"); +test("lists an uploaded object", async ({ + run, + projectRef, + dbPassword, + storageBucket, + workspace, +}) => { + const suffix = randomUUID().slice(0, 8); + const local = join(workspace.path, `upload-${suffix}.txt`); + const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + await writeFile(local, "live-e2e storage payload\n"); - const linked = await run(["link", "--project-ref", projectRef], { - env: { SUPABASE_DB_PASSWORD: dbPassword }, - }); - requireLiveSuccess(linked, "link setup for storage ls"); - const uploaded = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); - requireLiveSuccess(uploaded, "storage cp setup for storage ls"); + const linked = await run(["link", "--project-ref", projectRef], { + env: { SUPABASE_DB_PASSWORD: dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage ls"); + const uploaded = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + requireLiveSuccess(uploaded, "storage cp setup for storage ls"); - try { - const result = await run(["storage", "ls", `ss:///${storageBucket}/`, ...STORAGE_FLAGS]); - expect(result.exitCode, result.stderr).toBe(0); - expect(result.stdout).toContain(`upload-${suffix}.txt`); - } finally { - await removeObject(run, remote); - } - }, -); + try { + const result = await run(["storage", "ls", `ss:///${storageBucket}/`, ...STORAGE_FLAGS]); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.stdout).toContain(`upload-${suffix}.txt`); + } finally { + await removeObject(run, remote); + } +}); diff --git a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts index b287750c4c..0e4a0061e2 100644 --- a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts @@ -3,7 +3,7 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, testLiveStorage } from "../../../../../tests/helpers/live-context.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; @@ -17,28 +17,31 @@ async function removeObject( } } -testLiveStorage( - "removes an uploaded object", - async ({ run, projectRef, dbPassword, storageBucket, workspace }) => { - const suffix = randomUUID().slice(0, 8); - const local = join(workspace.path, `upload-${suffix}.txt`); - const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; - await writeFile(local, "live-e2e storage payload\n"); +test("removes an uploaded object", async ({ + run, + projectRef, + dbPassword, + storageBucket, + workspace, +}) => { + const suffix = randomUUID().slice(0, 8); + const local = join(workspace.path, `upload-${suffix}.txt`); + const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + await writeFile(local, "live-e2e storage payload\n"); - const linked = await run(["link", "--project-ref", projectRef], { - env: { SUPABASE_DB_PASSWORD: dbPassword }, - }); - requireLiveSuccess(linked, "link setup for storage rm"); - const uploaded = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); - requireLiveSuccess(uploaded, "storage cp setup for storage rm"); + const linked = await run(["link", "--project-ref", projectRef], { + env: { SUPABASE_DB_PASSWORD: dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage rm"); + const uploaded = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + requireLiveSuccess(uploaded, "storage cp setup for storage rm"); - let removed = false; - try { - const result = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); - expect(result.exitCode, result.stderr).toBe(0); - removed = true; - } finally { - if (!removed) await removeObject(run, remote); - } - }, -); + let removed = false; + try { + const result = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + expect(result.exitCode, result.stderr).toBe(0); + removed = true; + } finally { + if (!removed) await removeObject(run, remote); + } +}); diff --git a/apps/cli/src/next/commands/functions/dev/dev.live.test.ts b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts similarity index 95% rename from apps/cli/src/next/commands/functions/dev/dev.live.test.ts rename to apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts index 0277ad9ebc..3b38671458 100644 --- a/apps/cli/src/next/commands/functions/dev/dev.live.test.ts +++ b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts @@ -1,4 +1,4 @@ -import { afterEach, expect, test } from "vitest"; +import { afterEach, describe, expect, test } from "vitest"; import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { @@ -7,7 +7,6 @@ import { runSupabase, spawnSupabase, } from "../../../../../tests/helpers/cli.ts"; -import { describeLocalStackLive } from "../../../../../tests/helpers/live.ts"; import { cleanupRegisteredStackProjects } from "../../../../../tests/helpers/stack-e2e-cleanup.ts"; const FUNCTIONS_DEV_STARTUP_TIMEOUT_MS = 60_000; @@ -73,8 +72,8 @@ async function waitForFunctionResponse( // This crosses the compiled CLI, detached supervisor, full local stack, file // watcher, and HTTP runtime boundaries. Keep the one golden path in the -// opt-in live suite instead of slowing and destabilizing ordinary e2e shards. -describeLocalStackLive("supabase functions dev (live)", () => { +// opt-in e2e suite instead of slowing and destabilizing ordinary e2e shards. +describe("supabase functions dev (e2e)", () => { afterEach(cleanupRegisteredStackProjects); test( diff --git a/apps/cli/src/next/commands/start/start.live.test.ts b/apps/cli/src/next/commands/start/start.docker.e2e.test.ts similarity index 81% rename from apps/cli/src/next/commands/start/start.live.test.ts rename to apps/cli/src/next/commands/start/start.docker.e2e.test.ts index bf0283023a..724248c0c5 100644 --- a/apps/cli/src/next/commands/start/start.live.test.ts +++ b/apps/cli/src/next/commands/start/start.docker.e2e.test.ts @@ -1,6 +1,5 @@ -import { afterEach, expect, test } from "vitest"; -import { makeTempHome, makeTempStackProject } from "../../../../tests/helpers/cli.ts"; -import { describeLocalStackLive, runSupabaseLive } from "../../../../tests/helpers/live.ts"; +import { afterEach, describe, expect, test } from "vitest"; +import { makeTempHome, makeTempStackProject, runSupabase } from "../../../../tests/helpers/cli.ts"; import { cleanupRegisteredStackProjects } from "../../../../tests/helpers/stack-e2e-cleanup.ts"; const START_TIMEOUT_MS = 180_000; @@ -32,7 +31,7 @@ const LIGHTWEIGHT_DOCKER_ARGS = [ // Lazy service activation crosses the real proxy, daemon, Docker network, and // container lifecycle boundaries, so keep one gated golden-path live test. -describeLocalStackLive("supabase start lazy lifecycle (live)", () => { +describe("supabase start lazy lifecycle (e2e)", () => { let project: Awaited> | undefined; let home: ReturnType | undefined; @@ -49,7 +48,7 @@ describeLocalStackLive("supabase start lazy lifecycle (live)", () => { project = await makeTempStackProject("supabase-lazy-start-live-"); home = makeTempHome(); - const started = await runSupabaseLive([...LIGHTWEIGHT_DOCKER_ARGS], { + const started = await runSupabase([...LIGHTWEIGHT_DOCKER_ARGS], { ...COMMAND_OPTIONS, cwd: project.dir, home: home.dir, @@ -57,7 +56,7 @@ describeLocalStackLive("supabase start lazy lifecycle (live)", () => { }); expect(started.exitCode, `stdout:\n${started.stdout}\nstderr:\n${started.stderr}`).toBe(0); - const before = await runSupabaseLive(["status"], { + const before = await runSupabase(["status"], { ...COMMAND_OPTIONS, cwd: project.dir, home: home.dir, @@ -70,7 +69,7 @@ describeLocalStackLive("supabase start lazy lifecycle (live)", () => { }); expect(response.ok).toBe(true); - const after = await runSupabaseLive(["status"], { + const after = await runSupabase(["status"], { ...COMMAND_OPTIONS, cwd: project.dir, home: home.dir, diff --git a/apps/cli/tests/helpers/cli.ts b/apps/cli/tests/helpers/cli.ts index 5e043a3964..fed5b88b23 100644 --- a/apps/cli/tests/helpers/cli.ts +++ b/apps/cli/tests/helpers/cli.ts @@ -473,3 +473,14 @@ export async function runSupabase( const result = await spawned.waitForExit(); return { ...result, exitCode: killedByUntil ? 0 : result.exitCode }; } + +export function requireCliSuccess( + result: { readonly exitCode: number; readonly stdout: string; readonly stderr: string }, + command: string, +): void { + if (result.exitCode !== 0) { + throw new Error( + `${command} failed (exit ${result.exitCode})\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`, + ); + } +} diff --git a/apps/cli/tests/helpers/live-context.ts b/apps/cli/tests/helpers/live-context.ts deleted file mode 100644 index 6c861839f3..0000000000 --- a/apps/cli/tests/helpers/live-context.ts +++ /dev/null @@ -1,215 +0,0 @@ -import { mkdtempSync, rmSync } from "node:fs"; -import { tmpdir } from "node:os"; -import path from "node:path"; - -import { inject, test } from "vitest"; - -import { makeTempHome } from "./cli.ts"; -import { runSupabaseLive } from "./live.ts"; -import { - isLiveConfigured, - isManagedLive, - liveProjectDataPlaneReady, - liveProjectRef, -} from "./live-env.ts"; - -type RunOptions = NonNullable[1]>; -type RunResult = Awaited>; -type TempHome = ReturnType; - -export interface LiveWorkspace { - readonly path: string; -} - -export interface InvokeResult { - readonly status: number; - readonly body: unknown; - readonly text: string; -} - -export interface LiveFixtures { - readonly projectRef: string; - readonly anonKey: string; - readonly functionsUrl: string; - readonly dbUrl: string; - readonly dbPassword: string; - readonly storageBucket: string; - readonly home: TempHome; - readonly workspace: LiveWorkspace; - readonly run: (args: string[], options?: RunOptions) => Promise; - readonly invoke: ( - slug: string, - options?: { anonKey?: string; payload?: unknown }, - ) => Promise; -} - -const base = test.extend({ - // eslint-disable-next-line no-empty-pattern - projectRef: async ({}, use) => { - await use(inject("projectRef")); - }, - - // eslint-disable-next-line no-empty-pattern - anonKey: async ({}, use) => { - await use(inject("anonKey")); - }, - - // eslint-disable-next-line no-empty-pattern - functionsUrl: async ({}, use) => { - await use(inject("functionsUrl")); - }, - - // eslint-disable-next-line no-empty-pattern - dbUrl: async ({}, use) => { - await use(inject("dbUrl")); - }, - - // eslint-disable-next-line no-empty-pattern - dbPassword: async ({}, use) => { - await use(inject("dbPassword")); - }, - - // eslint-disable-next-line no-empty-pattern - storageBucket: async ({}, use) => { - await use(inject("storageBucket")); - }, - - home: async ({ task: _task }, use) => { - const home = makeTempHome(); - try { - await use(home); - } finally { - home[Symbol.dispose](); - } - }, - - workspace: async ({ task, home }, use) => { - const suffix = task.name.replace(/[^a-z0-9-]+/giu, "-").slice(0, 40); - const directory = mkdtempSync(path.join(tmpdir(), `supabase-live-${suffix || "test"}-`)); - try { - const initialized = await runSupabaseLive(["init"], { - cwd: directory, - home: home.dir, - }); - if (initialized.exitCode !== 0) { - throw new Error( - `supabase init failed (exit ${initialized.exitCode})\n${initialized.stderr || initialized.stdout}`, - ); - } - await use({ path: directory }); - } finally { - rmSync(directory, { recursive: true, force: true }); - } - }, - - run: async ({ workspace, home }, use) => { - await use((args, options) => - runSupabaseLive(args, { - ...options, - cwd: options?.cwd ?? workspace.path, - // A test may use a subdirectory as cwd, but must share this HOME so - // setup/command/teardown observe the same link and config state. - home: home.dir, - }), - ); - }, - - invoke: async ({ functionsUrl, anonKey }, use) => { - await use(async (slug, options) => { - const key = options?.anonKey ?? anonKey; - const headers: Record = { "Content-Type": "application/json" }; - if (key.length > 0) { - headers["Authorization"] = `Bearer ${key}`; - headers["apikey"] = key; - } - const response = await fetch(`${functionsUrl}/${slug}`, { - method: "POST", - headers, - body: JSON.stringify(options?.payload ?? {}), - }); - const text = await response.text(); - let body: unknown; - try { - body = JSON.parse(text); - } catch { - body = text; - } - return { status: response.status, body, text }; - }); - }, -}); - -/** Live subprocess fixture. The global setup owns the shared platform project; - * this fixture owns only one isolated workspace and HOME per test. */ -export const testLive = base.skipIf(!isLiveConfigured()); - -/** Fixture for scenarios that require a project ref from managed or attached setup. */ -export const testLiveProject = base.skipIf(!isLiveConfigured() || !liveProjectRef()); - -/** Fixture for Edge Function deploy/invoke scenarios. */ -export const testLiveFunctions = base.skipIf( - !isLiveConfigured() || - !liveProjectRef() || - (!isManagedLive() && (process.env["SUPABASE_LIVE_ANON_KEY"] ?? "").length === 0), -); - -/** Fixture for scenarios that require the project's Postgres data plane. */ -export const testLiveDataPlane = base.skipIf(!(await liveProjectDataPlaneReady())); - -/** Fixture for remote database tests that create/drop transient schema state. */ -export const testLiveDestructiveDataPlane = base.skipIf( - !(await liveProjectDataPlaneReady()) || - (!isManagedLive() && process.env["SUPABASE_LIVE_ALLOW_DESTRUCTIVE"] !== "1"), -); - -/** Fixture for Storage scenarios requiring a linked database password. */ -export const testLiveStorage = base.skipIf( - !(await liveProjectDataPlaneReady()) || - !liveProjectRef() || - (!isManagedLive() && - ((process.env["SUPABASE_LIVE_DB_PASSWORD"] ?? "").length === 0 || - (process.env["SUPABASE_LIVE_STORAGE_BUCKET"] ?? "").length === 0)), -); - -/** Throw with command diagnostics when a setup/teardown command fails. */ -export function requireLiveSuccess( - result: { readonly exitCode: number; readonly stdout: string; readonly stderr: string }, - command: string, -): void { - if (result.exitCode !== 0) { - throw new Error( - `${command} failed (exit ${result.exitCode})\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`, - ); - } -} - -/** Resolve a remote database target without requiring staging-only pooler data. */ -export function liveDatabaseTargetArgs(dbUrl: string, projectRef: string): string[] { - if (dbUrl.length > 0) return ["--db-url", dbUrl]; - if (projectRef.length === 0) { - throw new Error("A project ref is required when SUPABASE_LIVE_DB_URL is unavailable"); - } - return ["--linked", "--project-ref", projectRef]; -} - -export function expectFunctionOk( - result: InvokeResult, - slug: string, - extra?: Record, -): void { - if (result.status !== 200) { - throw new Error( - `Expected function ${slug} to return 200, got ${result.status}: ${result.text}`, - ); - } - if (typeof result.body !== "object" || result.body === null) { - throw new Error(`Expected function ${slug} to return JSON: ${result.text}`); - } - const body = result.body as Record; - if (body.case !== slug || body.ok !== true) { - throw new Error(`Unexpected response from ${slug}: ${result.text}`); - } - for (const [key, value] of Object.entries(extra ?? {})) { - if (body[key] !== value) throw new Error(`Unexpected ${key} from ${slug}: ${result.text}`); - } -} diff --git a/apps/cli/tests/helpers/live-env.ts b/apps/cli/tests/helpers/live-env.ts index 76ed269359..3730741d42 100644 --- a/apps/cli/tests/helpers/live-env.ts +++ b/apps/cli/tests/helpers/live-env.ts @@ -1,209 +1,71 @@ -/** - * Environment-only helpers for the `live` Vitest project, with **no Vitest test - * APIs imported**. Vitest evaluates `globalSetup` (live-global-setup.ts) in a - * separate context before the test workers, where importing `describe`/`test` - * is not valid — so the global setup imports the env helpers from here, while - * the test-facing pieces (`describeLive`, `runSupabaseLive`, …) live in - * `live.ts` and re-export these. - * - * Environment contract (provided by the managed-staging workflow or an - * attached Supabox/local runner): - * - `SUPABASE_ACCESS_TOKEN` — required; the platform PAT (supabox seeds a - * deterministic `sbp_…` token into its mgmt-api database). - * - `SUPABASE_PROFILE` — selects the API base URL; defaults to `supabase-local` - * (→ `http://localhost:8080`, `project_host: supabase.red`). Note the cli does - * NOT honor `SUPABASE_API_URL` (Go parity) — the profile is the override. - * - `SUPABASE_LIVE_API_URL` — base URL the readiness check probes; attached - * mode defaults to `http://localhost:8080`, managed staging defaults to - * `https://api.supabase.green`. - * - `SUPABASE_LIVE_PROJECT_REF` — the shared project; gates project-scoped - * suites (functions, branches, db, storage). Managed setup populates it. - * - `SUPABASE_LIVE_LOCAL_STACK` — whether local Docker-stack lifecycle suites - * run. Attached runs default to `1`; managed staging defaults to `0`. - * Set explicitly to `0` or `1` to override the mode default. - * - `NODE_EXTRA_CA_CERTS` — trusts the supabox CA for `*.supabase.red` TLS; - * inherited by the subprocess via the parent environment. - */ +/** Environment-only live-suite configuration. */ -/** Default profile for the host runner: api_url → localhost:8080, project_host → supabase.red. */ -export const LIVE_DEFAULT_PROFILE = "supabase-local"; - -export type LiveMode = "attached" | "managed"; - -/** - * Selects how the run-scoped live environment is obtained. - * - * The default is deliberately attached: a token in a developer's environment - * must never cause the test suite to create or delete a project. Managed mode - * is an explicit opt-in for staging runs and is selected with - * `SUPABASE_LIVE_MODE=managed`. - */ -export function liveMode(): LiveMode { - const value = process.env["SUPABASE_LIVE_MODE"]; - if (value === undefined || value === "" || value === "attached") { - return "attached"; - } - if (value === "managed") { - return "managed"; - } - throw new Error( - `Unsupported SUPABASE_LIVE_MODE ${JSON.stringify(value)}; expected "attached" or "managed"`, - ); -} - -export function isManagedLive(): boolean { - return liveMode() === "managed"; -} +export const LIVE_EXIT_TIMEOUT_MS = 240_000; -/** - * Whether suites that own a local Docker development stack should run. - * Attached runners preserve the historical default; managed project runners - * only exercise the remote project unless explicitly opted into local-stack - * coverage. - */ -export function localStackLiveEnabled(): boolean { - const value = process.env["SUPABASE_LIVE_LOCAL_STACK"]; - if (value === undefined) { - return !isManagedLive(); +export function liveApiUrl(): string { + const value = process.env["SUPABASE_LIVE_API_URL"]?.trim(); + if (value === undefined || value.length === 0) { + throw new Error("SUPABASE_LIVE_API_URL is required to run the live suite"); } - if (value === "0") { - return false; + let url: URL; + try { + url = new URL(value); + } catch { + throw new Error(`SUPABASE_LIVE_API_URL must be an absolute HTTP(S) URL: ${value}`); } - if (value === "1") { - return true; + if (url.protocol !== "http:" && url.protocol !== "https:") { + throw new Error(`SUPABASE_LIVE_API_URL must use http:// or https://: ${value}`); } - throw new Error( - `Unsupported SUPABASE_LIVE_LOCAL_STACK ${JSON.stringify(value)}; expected "0" or "1"`, - ); + return url.toString().replace(/\/+$/u, ""); } -/** - * Default subprocess exit timeout for live runs. `runSupabase` otherwise caps at - * 60s, which would kill a slow-but-valid supabox call before the live tests' - * own (60–120s+) timeouts fire. Generous, but under the `live` project's 300s - * cap so the per-test timeout stays the real gate. Callers may override. - */ -export const LIVE_EXIT_TIMEOUT_MS = 240_000; - -/** Management API base URL probed by the live readiness check. */ -export function liveApiBaseUrl(): string { - return ( - process.env["SUPABASE_LIVE_API_URL"] ?? - (isManagedLive() ? "https://api.supabase.green" : "http://localhost:8080") - ); -} - -/** Profile used by the CLI subprocess. Attached runs preserve the historical - * local Supabox default; managed staging runs use the built-in staging profile - * unless the caller explicitly selected another profile. */ -export function liveProfile(): string { - return ( - process.env["SUPABASE_PROFILE"] ?? (isManagedLive() ? "supabase-staging" : LIVE_DEFAULT_PROFILE) - ); -} - -/** Host used for project-scoped endpoints such as Edge Functions and Storage. */ -export function liveProjectHost(): string { - if (process.env["SUPABASE_LIVE_PROJECT_HOST"] !== undefined) { - return process.env["SUPABASE_LIVE_PROJECT_HOST"]!; +export function liveAccessToken(): string { + const token = process.env["SUPABASE_ACCESS_TOKEN"]?.trim(); + if (token === undefined || token.length === 0) { + throw new Error("SUPABASE_ACCESS_TOKEN is required to run the live suite"); } + return token; +} - switch (liveProfile()) { - case "supabase": - return "supabase.co"; - case "supabase-staging": - case "supabase-local": - return "supabase.red"; - default: - return "supabase.red"; - } +export function validateLiveConfig(): { readonly apiUrl: string; readonly accessToken: string } { + return { apiUrl: liveApiUrl(), accessToken: liveAccessToken() }; } -/** Keep an explicitly managed project alive for debugging instead of deleting it. */ export function keepLiveProject(): boolean { return process.env["SUPABASE_LIVE_KEEP_PROJECT"] === "1"; } -/** - * True when the environment carries a platform access token, i.e. the live - * suite is expected to run. Used to gate `describeLive` so live tests are inert - * in the default test loop. - */ -export function isLiveConfigured(): boolean { - return Boolean(process.env["SUPABASE_ACCESS_TOKEN"]); +export function liveProjectName(): string { + return process.env["SUPABASE_LIVE_PROJECT_NAME"]?.trim() || "supabase-cli-live"; +} + +export function liveRegion(): string { + return process.env["SUPABASE_LIVE_REGION"]?.trim() || "us-east-1"; } -/** - * Project ref for project-scoped live scenarios (functions, branches, db, - * storage, …). The cli-e2e-ci runner sets this once a project has been - * provisioned on the stack; absent → those suites skip. Returns `undefined` - * when unset so callers can branch; use `requireLiveProjectRef` inside a - * `describeLiveProject` block where presence is already guaranteed. - */ -export function liveProjectRef(): string | undefined { - return process.env["SUPABASE_LIVE_PROJECT_REF"]; +export function liveOrgId(): string | undefined { + const value = process.env["SUPABASE_LIVE_ORG_ID"]?.trim(); + return value === undefined || value.length === 0 ? undefined : value; } -/** - * The live project ref, or a thrown error if unset. Safe to call inside a - * `describeLiveProject` block (the gate guarantees it is present) and gives a - * typed `string` without a non-null assertion. - */ -export function requireLiveProjectRef(): string { - const ref = liveProjectRef(); - if (!ref) { +/** Resolve `.` from a database host such as `db..supabase.co`. */ +export function deriveLiveProjectHost(databaseHost: string, projectRef: string): string { + const prefix = `db.${projectRef}.`; + if (!databaseHost.startsWith(prefix)) { throw new Error( - "SUPABASE_LIVE_PROJECT_REF must be set for project-scoped live tests " + - "(managed global setup sets it after provisioning a project).", + `Cannot derive project host for ${projectRef} from database host ${databaseHost}; expected a ${prefix} name`, ); } - return ref; -} - -/** - * Whether the live project's *data-plane* — its own Postgres instance — is up - * and healthy. A control-plane-only platform may expose a project record without - * a reachable database; commands that talk to project Postgres (migration, db, - * storage) gate on this and SKIP until the data plane is ready. - * - * Probes `GET /v1/projects` (already proven reachable by `projects list`) and - * matches the live ref. Any failure or missing prerequisite returns `false` — - * "not ready" is the safe default, so a probe error skips rather than fails the - * suite. - */ -export async function liveProjectDataPlaneReady(): Promise { - const token = process.env["SUPABASE_ACCESS_TOKEN"]; - const ref = liveProjectRef(); - if (token === undefined || token.length === 0 || ref === undefined) { - return false; + const host = databaseHost.slice(prefix.length); + if (host.length === 0 || host.includes("/")) { + throw new Error(`Cannot derive a valid project host from database host ${databaseHost}`); } + return host; +} - const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), 15_000); - try { - const response = await fetch(`${liveApiBaseUrl()}/v1/projects`, { - headers: { Authorization: `Bearer ${token}` }, - signal: controller.signal, - }); - if (!response.ok) { - return false; - } - const projects: unknown = await response.json(); - if (!Array.isArray(projects)) { - return false; - } - return projects.some( - (candidate) => - candidate !== null && - typeof candidate === "object" && - "ref" in candidate && - candidate.ref === ref && - "status" in candidate && - candidate.status === "ACTIVE_HEALTHY", - ); - } catch { - return false; - } finally { - clearTimeout(timeout); - } +export function isLiveConfigured(): boolean { + return ( + (process.env["SUPABASE_LIVE_API_URL"]?.trim().length ?? 0) > 0 && + (process.env["SUPABASE_ACCESS_TOKEN"]?.trim().length ?? 0) > 0 + ); } diff --git a/apps/cli/tests/helpers/live-env.unit.test.ts b/apps/cli/tests/helpers/live-env.unit.test.ts index 96b9249062..b5704a6a9c 100644 --- a/apps/cli/tests/helpers/live-env.unit.test.ts +++ b/apps/cli/tests/helpers/live-env.unit.test.ts @@ -1,52 +1,44 @@ import { afterEach, describe, expect, it } from "vitest"; -import { localStackLiveEnabled } from "./live-env.ts"; +import { deriveLiveProjectHost, liveApiUrl, validateLiveConfig } from "./live-env.ts"; -const originalMode = process.env["SUPABASE_LIVE_MODE"]; -const originalLocalStack = process.env["SUPABASE_LIVE_LOCAL_STACK"]; +const originalApiUrl = process.env["SUPABASE_LIVE_API_URL"]; +const originalToken = process.env["SUPABASE_ACCESS_TOKEN"]; afterEach(() => { - if (originalMode === undefined) { - delete process.env["SUPABASE_LIVE_MODE"]; - } else { - process.env["SUPABASE_LIVE_MODE"] = originalMode; - } - if (originalLocalStack === undefined) { - delete process.env["SUPABASE_LIVE_LOCAL_STACK"]; - } else { - process.env["SUPABASE_LIVE_LOCAL_STACK"] = originalLocalStack; - } + if (originalApiUrl === undefined) delete process.env["SUPABASE_LIVE_API_URL"]; + else process.env["SUPABASE_LIVE_API_URL"] = originalApiUrl; + if (originalToken === undefined) delete process.env["SUPABASE_ACCESS_TOKEN"]; + else process.env["SUPABASE_ACCESS_TOKEN"] = originalToken; }); -describe("localStackLiveEnabled", () => { - it("defaults to enabled for attached live runs", () => { - delete process.env["SUPABASE_LIVE_MODE"]; - delete process.env["SUPABASE_LIVE_LOCAL_STACK"]; +describe("live environment", () => { + it("requires both the API URL and access token", () => { + delete process.env["SUPABASE_LIVE_API_URL"]; + delete process.env["SUPABASE_ACCESS_TOKEN"]; + expect(() => validateLiveConfig()).toThrow("SUPABASE_LIVE_API_URL is required"); - expect(localStackLiveEnabled()).toBe(true); + process.env["SUPABASE_LIVE_API_URL"] = "http://localhost:8080"; + expect(() => validateLiveConfig()).toThrow("SUPABASE_ACCESS_TOKEN is required"); }); - it("defaults to disabled for managed live runs", () => { - process.env["SUPABASE_LIVE_MODE"] = "managed"; - delete process.env["SUPABASE_LIVE_LOCAL_STACK"]; - - expect(localStackLiveEnabled()).toBe(false); + it("normalizes and validates HTTP API URLs", () => { + process.env["SUPABASE_LIVE_API_URL"] = "http://localhost:8080///"; + process.env["SUPABASE_ACCESS_TOKEN"] = " token "; + expect(validateLiveConfig()).toEqual({ + apiUrl: "http://localhost:8080", + accessToken: "token", + }); + process.env["SUPABASE_LIVE_API_URL"] = "not-a-url"; + expect(() => liveApiUrl()).toThrow("absolute HTTP(S) URL"); }); - it("honors an explicit enabled or disabled value", () => { - process.env["SUPABASE_LIVE_MODE"] = "managed"; - process.env["SUPABASE_LIVE_LOCAL_STACK"] = "1"; - expect(localStackLiveEnabled()).toBe(true); - - process.env["SUPABASE_LIVE_LOCAL_STACK"] = "0"; - expect(localStackLiveEnabled()).toBe(false); - }); - - it("rejects unsupported values", () => { - process.env["SUPABASE_LIVE_LOCAL_STACK"] = "yes"; - - expect(() => localStackLiveEnabled()).toThrow( - 'Unsupported SUPABASE_LIVE_LOCAL_STACK "yes"; expected "0" or "1"', + it("derives the project host from the typed database host", () => { + expect( + deriveLiveProjectHost("db.abcdefghijklmnopqrst.supabase.co", "abcdefghijklmnopqrst"), + ).toBe("supabase.co"); + expect(() => deriveLiveProjectHost("postgres.supabase.co", "abcdefghijklmnopqrst")).toThrow( + "Cannot derive project host", ); }); }); diff --git a/apps/cli/tests/helpers/live-project.ts b/apps/cli/tests/helpers/live-project.ts index dbd6720e67..e6e18732cb 100644 --- a/apps/cli/tests/helpers/live-project.ts +++ b/apps/cli/tests/helpers/live-project.ts @@ -1,381 +1,284 @@ import { randomBytes, randomUUID } from "node:crypto"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; -import { runSupabase } from "./cli.ts"; -import { keepLiveProject, liveApiBaseUrl, liveProfile, liveProjectHost } from "./live-env.ts"; +import { makeApiClient, type OperationOutput } from "@supabase/api/effect"; +import { Effect } from "effect"; +import { FetchHttpClient } from "effect/unstable/http"; + +import { + deriveLiveProjectHost, + keepLiveProject, + liveAccessToken, + liveApiUrl, + liveOrgId, + liveProjectName, + liveRegion, +} from "./live-env.ts"; const PROJECT_REF_RE = /^[a-z]{20}$/u; const TERMINAL_BAD_STATUSES = new Set(["INIT_FAILED", "RESTORE_FAILED", "REMOVED"]); -const DEFAULT_STORAGE_BUCKET = "supabase-cli-live-bucket"; +const PROFILE_NAME = "supabase-cli-live"; -export interface LiveProjectEnvironment { - readonly projectRef: string; - readonly anonKey: string; - readonly functionsUrl: string; - readonly dbUrl: string; - readonly dbPassword: string; - readonly storageBucket: string; - readonly projectName: string; - readonly owned: true; -} +type Project = OperationOutput<"v1GetProject">; +type Organization = OperationOutput<"v1ListAllOrganizations">[number]; +type ApiKey = OperationOutput<"v1GetProjectApiKeys">[number]; +type Region = + | "us-east-1" + | "us-east-2" + | "us-west-1" + | "us-west-2" + | "ap-east-1" + | "ap-southeast-1" + | "ap-northeast-1" + | "ap-northeast-2" + | "ap-southeast-2" + | "eu-west-1" + | "eu-west-2" + | "eu-west-3" + | "eu-north-1" + | "eu-central-1" + | "eu-central-2" + | "ca-central-1" + | "ap-south-1" + | "sa-east-1"; -function accessToken(): string { - const token = process.env["SUPABASE_ACCESS_TOKEN"]; - if (token === undefined || token.length === 0) { - throw new Error( - "Managed live mode requires SUPABASE_ACCESS_TOKEN; refusing to provision with an empty token.", - ); - } - return token; -} +const REGIONS: ReadonlyArray = [ + "us-east-1", + "us-east-2", + "us-west-1", + "us-west-2", + "ap-east-1", + "ap-southeast-1", + "ap-northeast-1", + "ap-northeast-2", + "ap-southeast-2", + "eu-west-1", + "eu-west-2", + "eu-west-3", + "eu-north-1", + "eu-central-1", + "eu-central-2", + "ca-central-1", + "ap-south-1", + "sa-east-1", +]; -function apiBaseUrl(): string { - return liveApiBaseUrl().replace(/\/+$/u, ""); +function supportedRegion(value: string): Region { + const region = REGIONS.find((candidate) => candidate === value); + if (region !== undefined) return region; + throw new Error(`Unsupported SUPABASE_LIVE_REGION ${JSON.stringify(value)}`); } -/** - * Keep both connection establishment and response-body consumption inside the - * caller's wall-clock deadline. A fetch can resolve its headers while a body - * read remains hung, so the timer intentionally surrounds `consume` too. - */ -async function fetchWithinDeadline( - operation: string, - url: string, - init: RequestInit, - deadline: number, - consume: (response: Response) => Promise, -): Promise { - const remaining = deadline - Date.now(); - if (remaining <= 0) { - throw new Error(`${operation} exceeded its provisioning deadline before starting: ${url}`); - } - - const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), remaining); - try { - const response = await fetch(url, { ...init, signal: controller.signal }); - return await consume(response); - } catch (error) { - if (controller.signal.aborted) { - throw new Error(`${operation} exceeded its provisioning deadline: ${url}`, { cause: error }); - } - throw error; - } finally { - clearTimeout(timeout); - } +function apiError(error: unknown): Error { + return error instanceof Error ? error : new Error(String(error)); } -function cliEnv(): Record { - return { SUPABASE_PROFILE: liveProfile() }; +function makeLiveApi() { + return makeApiClient({ baseUrl: liveApiUrl(), accessToken: liveAccessToken() }).pipe( + Effect.provide(FetchHttpClient.layer), + ); } -async function managementCommand(args: string[]) { - return runSupabase(args, { - entrypoint: "legacy", - env: cliEnv(), - exitTimeoutMs: 240_000, - }); +async function runLiveEffect(effect: Effect.Effect): Promise { + return Effect.runPromise(effect); } -function jsonError( - result: { exitCode: number; stderr: string; stdout: string }, - command: string, -): Error { - return new Error( - `${command} failed (exit ${result.exitCode}): ${result.stderr || result.stdout}`, - ); +function uniqueProjectName(): string { + const runId = process.env["GITHUB_RUN_ID"] ?? process.env["CI_JOB_ID"] ?? String(Date.now()); + return `${liveProjectName()}-${runId}-${randomUUID().slice(0, 8)}`; } -async function resolveOrgId(): Promise { - const override = process.env["SUPABASE_LIVE_ORG_ID"]; - if (override !== undefined && override.length > 0) return override; +function databasePassword(): string { + return `supabase-cli-live-${randomBytes(12).toString("hex")}`; +} - const result = await managementCommand(["orgs", "list", "--output", "json"]); - if (result.exitCode !== 0) throw jsonError(result, "orgs list"); - const rows = JSON.parse(result.stdout) as Array<{ id?: string }>; - const id = rows[0]?.id; - if (id === undefined || id.length === 0) { - throw new Error("No organizations found; cannot create the managed live project"); +async function resolveOrganization(): Promise { + const api = await runLiveEffect(makeLiveApi()); + const organizations = await runLiveEffect(api.v1.listAllOrganizations()); + const requested = liveOrgId(); + const organization = + (requested === undefined + ? organizations[0] + : organizations.find( + (candidate) => candidate.id === requested || candidate.slug === requested, + )) ?? undefined; + if (organization === undefined) { + throw new Error( + requested === undefined + ? "No organizations found; cannot create the live project" + : `Organization ${requested} was not found; cannot create the live project`, + ); } - return id; + return organization; } -function generateDbPassword(): string { - return ( - process.env["SUPABASE_LIVE_DB_PASSWORD"] ?? - `supabase-cli-live-${randomBytes(12).toString("hex")}` +async function createProject(name: string, password: string): Promise { + const organization = await resolveOrganization(); + const api = await runLiveEffect(makeLiveApi()); + const project = await runLiveEffect( + api.v1.createAProject({ + name, + db_pass: password, + organization_slug: organization.slug, + region: supportedRegion(liveRegion()), + }), ); -} - -async function createProject(name: string, orgId: string, dbPassword: string): Promise { - const region = process.env["SUPABASE_LIVE_REGION"] ?? "us-east-1"; - const result = await managementCommand([ - "projects", - "create", - name, - "--org-id", - orgId, - "--db-password", - dbPassword, - "--region", - region, - "--output", - "json", - ]); - if (result.exitCode !== 0) throw jsonError(result, "projects create"); - - const project = JSON.parse(result.stdout) as { id?: string; ref?: string }; - const ref = project.ref ?? project.id; - if (ref === undefined || !PROJECT_REF_RE.test(ref)) { - throw new Error(`Unexpected project ref from projects create: ${result.stdout}`); + if (!PROJECT_REF_RE.test(project.ref)) { + throw new Error(`Unexpected project ref from project creation: ${project.ref}`); } - return ref; + return project.ref; } -async function deleteProject(projectRef: string): Promise { - const result = await managementCommand(["projects", "delete", projectRef, "--yes"]); - if (result.exitCode !== 0) throw jsonError(result, `projects delete ${projectRef}`); +async function deleteProject(ref: string): Promise { + const api = await runLiveEffect(makeLiveApi()); + await runLiveEffect(api.v1.deleteAProject({ ref })); } -async function waitForProjectReady(projectRef: string, timeoutMs = 300_000): Promise { - const deadline = Date.now() + timeoutMs; +async function waitForProject(ref: string): Promise { + const deadline = Date.now() + 300_000; while (Date.now() < deadline) { - const result = await fetchWithinDeadline( - `Waiting for project ${projectRef} readiness`, - `${apiBaseUrl()}/v1/projects/${projectRef}`, - { headers: { Authorization: `Bearer ${accessToken()}` } }, - deadline, - async (response) => { - if (response.ok) { - return { - ok: true, - project: (await response.json()) as { status?: string }, - }; - } - await response.body?.cancel(); - return { ok: false, project: undefined }; - }, - ); - if (result.ok) { - if (result.project?.status === "ACTIVE_HEALTHY") return; - if ( - result.project?.status !== undefined && - TERMINAL_BAD_STATUSES.has(result.project.status) - ) { - throw new Error( - `Project ${projectRef} entered terminal status ${result.project.status} during provisioning`, - ); - } + const api = await runLiveEffect(makeLiveApi()); + const project = await runLiveEffect(api.v1.getProject({ ref })); + if (project.status === "ACTIVE_HEALTHY") return project; + if (TERMINAL_BAD_STATUSES.has(project.status)) { + throw new Error(`Project ${ref} entered terminal status ${project.status}`); } - const remaining = deadline - Date.now(); - if (remaining <= 0) break; - await new Promise((resolve) => setTimeout(resolve, Math.min(5_000, remaining))); - } - throw new Error(`Project ${projectRef} did not become ACTIVE_HEALTHY within ${timeoutMs}ms`); -} - -interface ApiKey { - readonly name?: string; - readonly api_key?: string; -} - -/** - * API keys are eventually consistent after project readiness. Polling is an - * intrinsic part of provisioning because the Management API exposes no - * readiness event; bound it by wall-clock time rather than attempt count. - */ -async function getProjectKeys(projectRef: string, timeoutMs = 180_000): Promise { - const deadline = Date.now() + timeoutMs; - let lastStatus = "unknown"; - while (Date.now() < deadline) { - const result = await fetchWithinDeadline( - `Resolving API keys for project ${projectRef}`, - `${apiBaseUrl()}/v1/projects/${projectRef}/api-keys`, - { headers: { Authorization: `Bearer ${accessToken()}` } }, - deadline, - async (response) => { - if (response.ok) { - return { status: response.status, keys: (await response.json()) as ApiKey[] }; - } - await response.body?.cancel(); - return { status: response.status, keys: undefined }; - }, - ); - lastStatus = String(result.status); - if (result.keys !== undefined && result.keys.length > 0) return result.keys; - const remaining = deadline - Date.now(); - if (remaining <= 0) break; - await new Promise((resolve) => setTimeout(resolve, Math.min(10_000, remaining))); + await Effect.runPromise(Effect.sleep("5 seconds")); } - throw new Error( - `Failed to resolve API keys for ${projectRef} within ${timeoutMs}ms (${lastStatus})`, - ); -} - -async function getAnonKey(projectRef: string): Promise { - const keys = await getProjectKeys(projectRef); - const anon = keys.find((key) => key.name === "anon" && key.api_key)?.api_key; - if (anon !== undefined) return anon; - throw new Error(`Project ${projectRef} returned no legacy anon JWT for function invokes`); + throw new Error(`Project ${ref} did not become ACTIVE_HEALTHY within 300000ms`); } -async function getServiceRoleKey(projectRef: string): Promise { - const keys = await getProjectKeys(projectRef); +async function resolveKeys(ref: string): Promise<{ anonKey: string; serviceRoleKey: string }> { + const api = await runLiveEffect(makeLiveApi()); + const keys = await runLiveEffect(api.v1.getProjectApiKeys({ ref, reveal: true })); + const keyValue = (key: ApiKey): string | undefined => + key.api_key === null || key.api_key === undefined ? undefined : key.api_key; + const anon = keys.find((key) => key.name === "anon"); const serviceRole = - keys.find((key) => key.name === "service_role" && key.api_key)?.api_key ?? - keys.find((key) => key.api_key?.startsWith("sb_secret_"))?.api_key; - if (serviceRole !== undefined) return serviceRole; - throw new Error(`Project ${projectRef} returned no service-role key`); -} - -interface PoolerConfig { - readonly database_type?: string; - readonly connection_string?: string; + keys.find((key) => key.name === "service_role") ?? + keys.find((key) => key.api_key?.startsWith("sb_secret_")); + const anonKey = anon === undefined ? undefined : keyValue(anon); + const serviceRoleKey = serviceRole === undefined ? undefined : keyValue(serviceRole); + if (anonKey === undefined || serviceRoleKey === undefined) { + throw new Error(`Project ${ref} returned no anon and service-role API keys`); + } + return { anonKey, serviceRoleKey }; } -async function getPoolerSessionUrl( - projectRef: string, - password: string, - timeoutMs = 180_000, -): Promise { - const deadline = Date.now() + timeoutMs; - let lastStatus = "unknown"; - while (Date.now() < deadline) { - const result = await fetchWithinDeadline( - `Resolving pooler config for project ${projectRef}`, - `${apiBaseUrl()}/v1/projects/${projectRef}/config/database/pooler`, - { headers: { Authorization: `Bearer ${accessToken()}` } }, - deadline, - async (response) => { - if (!response.ok) { - await response.body?.cancel(); - return { status: response.status, connectionString: undefined }; - } - const raw = (await response.json()) as PoolerConfig | PoolerConfig[]; - const configs = Array.isArray(raw) ? raw : [raw]; - const primary = configs.find((config) => config.database_type === "PRIMARY") ?? configs[0]; - return { status: response.status, connectionString: primary?.connection_string }; - }, - ); - lastStatus = String(result.status); - if (result.connectionString !== undefined) { - const url = new URL(result.connectionString); - url.password = password; - url.port = "5432"; - if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); - return url.toString(); - } - const remaining = deadline - Date.now(); - if (remaining <= 0) break; - await new Promise((resolve) => setTimeout(resolve, Math.min(10_000, remaining))); +async function resolveDbUrl(ref: string, password: string): Promise { + const api = await runLiveEffect(makeLiveApi()); + const config = await runLiveEffect(api.v1.getProjectPgbouncerConfig({ ref })); + if (config.connection_string === undefined) { + throw new Error(`Project ${ref} returned no pooler connection string`); } - throw new Error( - `Failed to resolve pooler config for ${projectRef} within ${timeoutMs}ms (${lastStatus})`, - ); + const url = new URL(config.connection_string); + url.password = password; + url.port = "5432"; + if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); + return url.toString(); } async function createStorageBucket( - projectRef: string, + ref: string, + host: string, serviceRoleKey: string, bucket: string, ): Promise { - const url = `https://${projectRef}.${liveProjectHost()}/storage/v1/bucket`; - await fetchWithinDeadline( - `Creating storage bucket ${bucket}`, - url, - { - method: "POST", - headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, - body: JSON.stringify({ id: bucket, name: bucket, public: false }), - }, - Date.now() + 60_000, - async (response) => { - if (!response.ok && response.status !== 409) { - throw new Error( - `Failed to create storage bucket ${bucket}: ${response.status} ${await response.text()}`, - ); - } - }, - ); + const response = await fetch(`https://${ref}.${host}/storage/v1/bucket`, { + method: "POST", + headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, + body: JSON.stringify({ id: bucket, name: bucket, public: false }), + }); + if (!response.ok && response.status !== 409) { + throw new Error( + `Failed to create storage bucket ${bucket}: ${response.status} ${await response.text()}`, + ); + } } -/** Provision one unique staging project and all values needed by project live tests. */ -export async function provisionManagedLiveProject(): Promise { - const orgId = await resolveOrgId(); - const baseName = process.env["SUPABASE_LIVE_PROJECT_NAME"] ?? "supabase-cli-live"; - const runId = process.env["GITHUB_RUN_ID"] ?? process.env["CI_JOB_ID"] ?? String(Date.now()); - const projectName = `${baseName}-${runId}-${randomUUID().slice(0, 8)}`; - const dbPassword = generateDbPassword(); - const projectRef = await createProject(projectName, orgId, dbPassword); - +async function writeProfile( + projectRef: string, + projectHost: string, + dbUrl: string, +): Promise { + const directory = await mkdtemp(path.join(tmpdir(), "supabase-live-profile-")); + const profilePath = path.join(directory, "profile.yaml"); + const poolerHost = new URL(dbUrl).hostname; try { - await waitForProjectReady(projectRef); - const anonKey = await getAnonKey(projectRef); - const serviceRoleKey = await getServiceRoleKey(projectRef); - const dbUrl = await getPoolerSessionUrl(projectRef, dbPassword); - const storageBucket = process.env["SUPABASE_LIVE_STORAGE_BUCKET"] ?? DEFAULT_STORAGE_BUCKET; - await createStorageBucket(projectRef, serviceRoleKey, storageBucket); - - return { - projectRef, - anonKey, - functionsUrl: `https://${projectRef}.${liveProjectHost()}/functions/v1`, - dbUrl, - dbPassword, - storageBucket, - projectName, - owned: true, - }; + await writeFile( + profilePath, + [ + `name: ${PROFILE_NAME}`, + `api_url: ${JSON.stringify(liveApiUrl())}`, + `dashboard_url: ${JSON.stringify(liveApiUrl())}`, + `project_host: ${projectHost}`, + `pooler_host: ${poolerHost}`, + `# provisioned project: ${projectRef}`, + "", + ].join("\n"), + ); } catch (error) { - if (!keepLiveProject()) { - await deleteProject(projectRef).catch((cleanupError) => { - console.error(`Failed to delete managed live project ${projectRef}:`, cleanupError); - }); - } + await rm(directory, { recursive: true, force: true }); throw error; } + return profilePath; } -export async function deleteManagedLiveProject(projectRef: string): Promise { - if (keepLiveProject()) { - console.log(`SUPABASE_LIVE_KEEP_PROJECT=1 — leaving managed live project ${projectRef} alive`); - return; - } - await deleteProject(projectRef); +export interface LiveProjectEnvironment { + readonly project: { + readonly ref: string; + readonly dbUrl: string; + readonly dbPassword: string; + readonly anonKey: string; + readonly serviceRoleKey: string; + readonly functionsUrl: string; + readonly storageBucket: string; + }; + readonly profilePath: string; } -/** Read-only API values supplied by an attached Supabox/local harness. */ -export function attachedLiveValues( - projectRef: string | undefined, -): Omit { - const anonKey = process.env["SUPABASE_LIVE_ANON_KEY"] ?? ""; - const dbUrl = process.env["SUPABASE_LIVE_DB_URL"] ?? ""; - const dbPassword = process.env["SUPABASE_LIVE_DB_PASSWORD"] ?? ""; - const storageBucket = process.env["SUPABASE_LIVE_STORAGE_BUCKET"] ?? DEFAULT_STORAGE_BUCKET; - return { - projectRef: projectRef ?? "", - anonKey, - functionsUrl: - process.env["SUPABASE_LIVE_FUNCTIONS_URL"] ?? - (projectRef === undefined ? "" : `https://${projectRef}.${liveProjectHost()}/functions/v1`), - dbUrl, - dbPassword, - storageBucket, - }; +export async function provisionLiveEnvironment(): Promise { + const password = databasePassword(); + const ref = await createProject(uniqueProjectName(), password); + let profilePath: string | undefined; + try { + const project = await waitForProject(ref); + const projectHost = deriveLiveProjectHost(project.database.host, ref); + const keys = await resolveKeys(ref); + const dbUrl = await resolveDbUrl(ref, password); + const storageBucket = "supabase-cli-live-bucket"; + await createStorageBucket(ref, projectHost, keys.serviceRoleKey, storageBucket); + profilePath = await writeProfile(ref, projectHost, dbUrl); + return { + project: { + ref, + dbUrl, + dbPassword: password, + anonKey: keys.anonKey, + serviceRoleKey: keys.serviceRoleKey, + functionsUrl: `https://${ref}.${projectHost}/functions/v1`, + storageBucket, + }, + profilePath, + }; + } catch (error) { + if (profilePath !== undefined) + await rm(path.dirname(profilePath), { recursive: true, force: true }); + if (!keepLiveProject()) await deleteProject(ref).catch(() => undefined); + throw apiError(error); + } } -/** Resolve the attached environment without ever creating or deleting a project. */ -export async function assertAttachedLiveReachable(): Promise { - const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), 30_000); +export async function cleanupLiveEnvironment(environment: LiveProjectEnvironment): Promise { + let profileError: unknown; try { - await fetch(`${apiBaseUrl()}/v1/organizations`, { signal: controller.signal }); + await rm(path.dirname(environment.profilePath), { recursive: true, force: true }); } catch (error) { - const reason = error instanceof Error ? error.message : String(error); - throw new Error( - `Live platform is not reachable at ${apiBaseUrl()}/v1/organizations: ${reason}.\n` + - "Ensure the Supabox/local API platform is running and reachable.", - ); - } finally { - clearTimeout(timeout); + profileError = error; } + if (!keepLiveProject()) await deleteProject(environment.project.ref); + else console.log(`SUPABASE_LIVE_KEEP_PROJECT=1 — leaving ${environment.project.ref} alive`); + if (profileError !== undefined) throw apiError(profileError); } diff --git a/apps/cli/tests/helpers/live-provided-context.ts b/apps/cli/tests/helpers/live-provided-context.ts index 9a177d8d48..deb9a63adf 100644 --- a/apps/cli/tests/helpers/live-provided-context.ts +++ b/apps/cli/tests/helpers/live-provided-context.ts @@ -1,20 +1,18 @@ -// Vitest evaluates global setup separately from test modules. Keep the -// `ProvidedContext` augmentation in a side-effect-free module so global setup -// can import it without loading Vitest's test APIs. +// Vitest evaluates global setup separately from test modules. Keep this module +// side-effect-free so global setup can provide the shared live environment. export {}; declare module "vitest" { export interface ProvidedContext { - /** Environment selected by the live global setup. */ - liveMode: "attached" | "managed"; - /** Shared project wiring. Empty strings mean the attached harness did not - * provide a project-scoped value and the corresponding suite should use a - * project/data-plane gate. */ - projectRef: string; - anonKey: string; - functionsUrl: string; - dbUrl: string; - dbPassword: string; - storageBucket: string; + liveProject: { + readonly ref: string; + readonly dbUrl: string; + readonly dbPassword: string; + readonly anonKey: string; + readonly serviceRoleKey: string; + readonly functionsUrl: string; + readonly storageBucket: string; + }; + liveProfilePath: string; } } diff --git a/apps/cli/tests/helpers/live.ts b/apps/cli/tests/helpers/live.ts index 9ad2fd96d8..0b4a1c9948 100644 --- a/apps/cli/tests/helpers/live.ts +++ b/apps/cli/tests/helpers/live.ts @@ -1,115 +1,166 @@ -import { execSync } from "node:child_process"; -import { describe } from "vitest"; - -import { runSupabase } from "./cli.ts"; -import { - isLiveConfigured, - LIVE_EXIT_TIMEOUT_MS, - localStackLiveEnabled, - liveProjectDataPlaneReady, - liveProfile, - liveProjectRef, -} from "./live-env.ts"; - -/** - * Test-facing helpers for the `live` Vitest project (`*.live.test.ts`): - * black-box CLI subprocess tests that run against a *real* Supabase platform — - * in CI a local Supabox stack or a managed staging project. - * - * This module imports Vitest test APIs (`describe`), so it must NOT be imported - * from `globalSetup` (Vitest evaluates that in a different context). The - * env-only helpers live in `./live-env.ts`; `globalSetup` imports from there. - * They are re-exported below so test files have a single import site. - */ - -// Re-export the env-only helpers so `*.live.test.ts` files import everything -// from `helpers/live.ts`. -export { - isLiveConfigured, - LIVE_DEFAULT_PROFILE, - LIVE_EXIT_TIMEOUT_MS, - liveApiBaseUrl, - isManagedLive, - localStackLiveEnabled, - keepLiveProject, - liveProjectDataPlaneReady, - liveMode, - liveProfile, - liveProjectHost, - liveProjectRef, - type LiveMode, - requireLiveProjectRef, -} from "./live-env.ts"; - -/** - * `describe` that runs only when the live environment is configured. Use this - * for every live suite so the file is inert (skipped, not failed) outside a - * configured live environment. - */ -export const describeLive = describe.skipIf(!isLiveConfigured()); - -function hasDockerDaemon(): boolean { - try { - execSync("docker info", { stdio: "ignore" }); - return true; - } catch { - return false; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import { inject, test as vitestTest } from "vitest"; + +import { makeTempHome, runSupabase } from "./cli.ts"; +import { LIVE_EXIT_TIMEOUT_MS } from "./live-env.ts"; +import type { LiveProjectEnvironment } from "./live-project.ts"; + +export type LiveProject = LiveProjectEnvironment["project"]; +type RunOptions = NonNullable[1]>; +type RunResult = Awaited>; + +export interface LiveWorkspace { + readonly path: string; +} + +export interface InvokeResult { + readonly status: number; + readonly body: unknown; + readonly text: string; +} + +export interface LiveFixtures { + readonly project: LiveProject; + /** Compatibility aliases while command suites migrate to `project`/`cli`. */ + readonly projectRef: string; + readonly dbUrl: string; + readonly dbPassword: string; + readonly anonKey: string; + readonly functionsUrl: string; + readonly storageBucket: string; + readonly workspace: LiveWorkspace; + readonly home: ReturnType; + readonly cli: (args: string[], options?: RunOptions) => Promise; + readonly run: (args: string[], options?: RunOptions) => Promise; + readonly invoke: ( + slug: string, + options?: { readonly anonKey?: string; readonly payload?: unknown }, + ) => Promise; +} + +const base = vitestTest.extend({ + // eslint-disable-next-line no-empty-pattern + project: async ({}, use) => use(inject("liveProject")), + + projectRef: async ({ project }, use) => use(project.ref), + dbUrl: async ({ project }, use) => use(project.dbUrl), + dbPassword: async ({ project }, use) => use(project.dbPassword), + anonKey: async ({ project }, use) => use(project.anonKey), + functionsUrl: async ({ project }, use) => use(project.functionsUrl), + storageBucket: async ({ project }, use) => use(project.storageBucket), + + home: async ({ task: _task }, use) => { + const home = makeTempHome(); + try { + await use(home); + } finally { + home[Symbol.dispose](); + } + }, + + workspace: async ({ task, home }, use) => { + const suffix = task.name.replace(/[^a-z0-9-]+/giu, "-").slice(0, 40); + const directory = mkdtempSync(path.join(tmpdir(), `supabase-live-${suffix || "test"}-`)); + try { + const initialized = await runSupabase(["init"], { + entrypoint: "legacy", + cwd: directory, + home: home.dir, + env: { SUPABASE_PROFILE: inject("liveProfilePath") }, + }); + if (initialized.exitCode !== 0) { + throw new Error( + `supabase init failed (exit ${initialized.exitCode})\n${initialized.stderr || initialized.stdout}`, + ); + } + await use({ path: directory }); + } finally { + rmSync(directory, { recursive: true, force: true }); + } + }, + + cli: async ({ workspace, home }, use) => { + await use((args, options) => + runSupabase(args, { + entrypoint: "legacy", + ...options, + cwd: options?.cwd ?? workspace.path, + home: home.dir, + exitTimeoutMs: options?.exitTimeoutMs ?? LIVE_EXIT_TIMEOUT_MS, + env: { + SUPABASE_PROFILE: inject("liveProfilePath"), + ...options?.env, + }, + }), + ); + }, + + run: async ({ cli }, use) => use(cli), + + invoke: async ({ project }, use) => { + await use(async (slug, options) => { + const key = options?.anonKey ?? project.anonKey; + const headers: Record = { "Content-Type": "application/json" }; + if (key.length > 0) { + headers["Authorization"] = `Bearer ${key}`; + headers["apikey"] = key; + } + const response = await fetch(`${project.functionsUrl}/${slug}`, { + method: "POST", + headers, + body: JSON.stringify(options?.payload ?? {}), + }); + const text = await response.text(); + let body: unknown; + try { + body = JSON.parse(text); + } catch { + body = text; + } + return { status: response.status, body, text }; + }); + }, +}); + +/** The sole live fixture. The live global setup owns the shared project. */ +export const test = base; + +export function requireLiveSuccess( + result: { readonly exitCode: number; readonly stdout: string; readonly stderr: string }, + command: string, +): void { + if (result.exitCode !== 0) { + throw new Error( + `${command} failed (exit ${result.exitCode})\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`, + ); } } -/** - * `describe` for local-stack live tests that additionally require a reachable - * Docker daemon. Composes the configured-live gate (`isLiveConfigured`) with a - * `docker info` probe so these suites stay inert (skipped, not failed) outside - * a configured live runner — a machine that merely exposes Docker must never - * launch a real stack just by collecting the live Vitest project. The - * synchronous read-only probe runs once when this helper module is collected, - * and only when the live environment is configured. - */ -export const describeDockerLive = describe.skipIf(!isLiveConfigured() || !hasDockerDaemon()); - -/** - * `describe` for suites that own a local Docker development stack. Attached - * Supabox/local runners enable these by default; managed staging only runs - * them when `SUPABASE_LIVE_LOCAL_STACK=1` is explicitly set. - */ -export const describeLocalStackLive = describe.skipIf( - !isLiveConfigured() || !localStackLiveEnabled() || !hasDockerDaemon(), -); - -/** - * `describe` for project-scoped live suites: runs only when the live env is - * configured AND a project ref is available. On a control-plane-only stack - * these skip rather than fail. See `requireLiveProjectRef`. - */ -export const describeLiveProject = describe.skipIf(!isLiveConfigured() || !liveProjectRef()); - -/** - * `describe` for data-plane live suites (migration / db / storage): runs only - * when the live env is configured AND the project's own Postgres instance is - * `ACTIVE_HEALTHY`. On a control-plane-only stack where the project DB - * is unreachable, so these SKIP rather than fail. They activate automatically - * once the full data-plane is provisioned. The readiness probe runs once at - * collection time (top-level await); see `liveProjectDataPlaneReady`. - */ -export const describeLiveDataPlane = describe.skipIf(!(await liveProjectDataPlaneReady())); - -/** - * Spawn the built CLI against the live platform, injecting the profile so the - * Management API base resolves to the stack. Defaults to the `legacy` shell, - * which hosts the platform commands (orgs, projects, branches, functions, …). - */ -export function runSupabaseLive( - args: string[], - options?: Parameters[1], -): ReturnType { - return runSupabase(args, { - entrypoint: "legacy", - ...options, - exitTimeoutMs: options?.exitTimeoutMs ?? LIVE_EXIT_TIMEOUT_MS, - env: { - SUPABASE_PROFILE: liveProfile(), - ...options?.env, - }, - }); +export function liveDatabaseTargetArgs(dbUrl: string, _projectRef?: string): string[] { + return ["--db-url", dbUrl]; +} + +export function expectFunctionOk( + result: InvokeResult, + slug: string, + extra?: Record, +): void { + if (result.status !== 200) { + throw new Error( + `Expected function ${slug} to return 200, got ${result.status}: ${result.text}`, + ); + } + if (typeof result.body !== "object" || result.body === null) { + throw new Error(`Expected function ${slug} to return JSON: ${result.text}`); + } + const body = result.body as Record; + if (body.case !== slug || body.ok !== true) { + throw new Error(`Unexpected response from ${slug}: ${result.text}`); + } + for (const [key, value] of Object.entries(extra ?? {})) { + if (body[key] !== value) throw new Error(`Unexpected ${key} from ${slug}: ${result.text}`); + } } diff --git a/apps/cli/tests/live-global-setup.ts b/apps/cli/tests/live-global-setup.ts index 97d4b5ffbb..71279a6a07 100644 --- a/apps/cli/tests/live-global-setup.ts +++ b/apps/cli/tests/live-global-setup.ts @@ -1,80 +1,20 @@ import type { ProvidedContext } from "vitest"; import "./helpers/live-provided-context.ts"; -import { - assertAttachedLiveReachable, - attachedLiveValues, - deleteManagedLiveProject, - provisionManagedLiveProject, -} from "./helpers/live-project.ts"; -import { isLiveConfigured, liveMode, liveProfile, liveProjectRef } from "./helpers/live-env.ts"; +import { cleanupLiveEnvironment, provisionLiveEnvironment } from "./helpers/live-project.ts"; +import { validateLiveConfig } from "./helpers/live-env.ts"; type LiveSetupContext = { provide: (key: K, value: ProvidedContext[K]) => void; }; -function provideEnvironment( - provide: LiveSetupContext["provide"], - values: { - projectRef: string; - anonKey: string; - functionsUrl: string; - dbUrl: string; - dbPassword: string; - storageBucket: string; - }, - mode: "attached" | "managed", -): void { - provide("liveMode", mode); - provide("projectRef", values.projectRef); - provide("anonKey", values.anonKey); - provide("functionsUrl", values.functionsUrl); - provide("dbUrl", values.dbUrl); - provide("dbPassword", values.dbPassword); - provide("storageBucket", values.storageBucket); -} - -/** - * Own one live environment for the entire Vitest run. - * - * Attached mode is the historical Supabox/local contract: it only probes the - * configured platform and never mutates or deletes a project. Managed mode is - * an explicit staging opt-in (`SUPABASE_LIVE_MODE=managed`) and creates one - * uniquely named project, provides its wiring to workers, then deletes exactly - * that project during global teardown unless `SUPABASE_LIVE_KEEP_PROJECT=1`. - */ +/** Provision one disposable project for the entire serial live Vitest run. */ export async function setup({ provide }: LiveSetupContext): Promise<() => Promise> { - const mode = liveMode(); - - if (mode === "managed") { - if (!isLiveConfigured()) { - throw new Error( - "SUPABASE_LIVE_MODE=managed requires SUPABASE_ACCESS_TOKEN; refusing to provision with an ambient empty token.", - ); - } - - // Make the managed profile/ref visible to existing collection-time gates in - // addition to providing the values to fixtures through Vitest's context. - if (process.env["SUPABASE_PROFILE"] === undefined) { - process.env["SUPABASE_PROFILE"] = liveProfile(); - } - const environment = await provisionManagedLiveProject(); - process.env["SUPABASE_LIVE_PROJECT_REF"] = environment.projectRef; - provideEnvironment(provide, environment, mode); - - return async () => { - await deleteManagedLiveProject(environment.projectRef); - }; - } - - if (!isLiveConfigured()) { - return async () => {}; - } - - await assertAttachedLiveReachable(); - const values = attachedLiveValues(liveProjectRef()); - provideEnvironment(provide, values, mode); - return async () => {}; + validateLiveConfig(); + const environment = await provisionLiveEnvironment(); + provide("liveProject", environment.project); + provide("liveProfilePath", environment.profilePath); + return async () => cleanupLiveEnvironment(environment); } export default setup; diff --git a/apps/cli/vitest.config.ts b/apps/cli/vitest.config.ts index a8c53dd4e5..fcfa681541 100644 --- a/apps/cli/vitest.config.ts +++ b/apps/cli/vitest.config.ts @@ -70,9 +70,9 @@ export default defineConfig({ { plugins: [dockerfileTextPlugin()], test: { - // Live tests run against a real platform (a supabox stack in CI) and - // are gated by `describeLive`, so they are inert unless the live env - // is configured. Never part of the default unit/integration/e2e loop. + // Live tests run against one provisioned project on the configured + // platform. They are never part of the default unit/integration/e2e + // loop; an explicit run fails fast when required configuration is absent. name: "live", include: ["**/*.live.test.ts"], fileParallelism: false, diff --git a/docs/adr/0013-live-e2e-bypasses-replay-server.md b/docs/adr/0013-live-e2e-bypasses-replay-server.md index cf8ce50f08..b4730453e0 100644 --- a/docs/adr/0013-live-e2e-bypasses-replay-server.md +++ b/docs/adr/0013-live-e2e-bypasses-replay-server.md @@ -3,146 +3,54 @@ **Status**: accepted **Date**: 2026-06-16 -## Problem Statement +## Problem -The CLI has no true end-to-end tests. `apps/cli-e2e` is a replay/record harness: -in **replay** mode it serves recorded HTTP fixtures (fast, deterministic, no -network); in **record** mode it proxies the CLI's Management API and Docker -traffic to staging only to *capture* those fixtures. Tests always assert against -replayed fixtures, never live responses. Behaviour that cannot be mocked — real -Management API calls and the real Docker bundler (e.g. `functions deploy`) — is -therefore untested. - -[CLI-1630](https://linear.app/supabase/issue/CLI-1630/set-up-proper-live-e2e-tests-for-the-cli) -adds a structured Vitest **live** suite under `apps/cli` that runs the real CLI -against a real backend (managed staging or an attached Supabox/local stack) as -a non-blocking smoke test before a stable deploy. - -The open architectural question was *how* live mode should reach the backend. -The first instinct was to add a third runtime mode inside `replay-server.ts` -alongside `replay` and `record` — taking record mode's passthrough path -(CLI → replay server → real API) but skipping fixture I/O. That keeps the -existing Docker and storage proxies "for free." +`apps/cli-e2e` is a replay/record harness. Replay tests use recorded HTTP and +Docker fixtures, so they do not exercise a real Management API, project data +plane, or Docker bundler. The CLI needs a small, non-blocking golden-path suite +that crosses those boundaries for real. ## Decision -Live mode **does not route through the replay server**. It is a harness-wiring -mode, not a `replay-server.ts` branch. - -- Live tests are collocated with their command under `apps/cli/src/**` as - `*.live.test.ts` files and use `runSupabaseLive`, which wires the CLI directly - to the selected `SUPABASE_PROFILE` and real Docker socket. -- Global setup provisions one shared environment per Vitest run. Attached mode - (the default) probes the existing Supabox/local platform and never mutates or - deletes its project. Explicit managed mode (`SUPABASE_LIVE_MODE=managed`) - provisions one uniquely named staging project, shares it across tests, and - deletes exactly that project during teardown unless explicitly kept. -- `replay-server.ts` is untouched — no `live` branch, no live Docker or storage - proxy. -- Local Docker-stack lifecycle suites are gated separately by - `SUPABASE_LIVE_LOCAL_STACK`: attached Supabox/local runs default to enabled, - while managed staging defaults to disabled. Docker-dependent remote scenarios - such as `functions deploy` retain the broader Docker gate. -- Assertions are **outcome-based**, modeled on the manual deploy playbook: - 1. run the real CLI (`run([...])`) and assert `exitCode` / `stdout`; - 2. **invoke the deployed function over HTTP directly** and assert HTTP status + - the JSON body the function itself returns (e.g. `{case, ok:true}`). - The invoke is a direct HTTP call to `https://{ref}.{SUPABASE_LIVE_PROJECT_HOST}/functions/v1`, - not a proxied call — the replay server is nowhere in the assertion path. -- Because the assertion target is the function's own deterministic response (plus - exit codes / stdout substrings), the suite is **ID-agnostic** — no response - normalization or snapshot machinery by default. The function invoke URL and - anon key are resolved at setup from the freshly created project (anon key via - `GET /v1/projects/{ref}/api-keys`). - -The live CI currently targets the shipped `ts-legacy` shell. A live file asserts -one command's golden path; setup and teardown may invoke other commands without -asserting them. This keeps cross-command scenarios for a later phase while the -suite grows toward one representative live test per command. - -## Rationale - -For the assertions live mode actually makes, intercepting the Management API buys -nothing — nothing inspects a proxied API body. The only thing the replay server -would do in live mode for `functions deploy` is relay Docker traffic -(CLI → relay → real socket) through its streaming/idle-timeout proxy. That -streaming relay is the most complex, most failure-prone code path in the harness, -and it would sit in front of the slowest, flakiest real operation (image pull + -bundle) for zero assertion benefit. Pointing `DOCKER_HOST` at the real socket -removes that failure surface entirely. - -Keeping `replay-server.ts` out of the live path also means live and record modes -stay decoupled: record mode's destructive fixture-tree rewrite, scenario logging, -and placeholder normalization never have to grow `isLive` guards, and a future -reader is not left wondering why a "transparent proxy" mode exists that records -nothing. - -The storage proxy (the other "free" proxy) is not exercised by the -`functions deploy` pilot, so it is not a reason to keep the server in front. If a -later live command genuinely needs host rewriting (e.g. storage on a different -host than the Management API), a scoped passthrough can be introduced *then* for -that command — YAGNI until a concrete need exists. - -The attached/managed environment contract keeps local Supabox and staging -runs interchangeable: switch `SUPABASE_PROFILE` and the `SUPABASE_LIVE_*` -values without changing test code. +Live tests are collocated under `apps/cli` as `*.live.test.ts` and run directly +against a configured platform URL. They never route through the replay server. + +Local Docker-stack lifecycle tests are ordinary `*.e2e.test.ts` tests. They use +the existing e2e global setup and registered-stack cleanup and do not require a +platform token. A live test means the command under assertion reaches the +Management API, its provisioned project, or that project's data plane. Docker +is a runner prerequisite, including for live `functions deploy`; there is no +Docker-specific live fixture. + +The live global setup requires `SUPABASE_LIVE_API_URL` and +`SUPABASE_ACCESS_TOKEN`, then: + +1. Creates one uniquely named disposable project through the typed Effect + `@supabase/api` client pointed at the configured URL. +2. Waits for `ACTIVE_HEALTHY`, resolves API keys and pooler connection details, + creates a storage bucket, and derives the project host from the returned + database host. +3. Writes a temporary YAML profile containing the same API URL and project + host, and injects it into every CLI subprocess in the serial suite. +4. Deletes exactly that project and the temporary profile during teardown. + +`SUPABASE_LIVE_KEEP_PROJECT=1` skips project deletion for debugging but never +skips temporary profile cleanup. Provisioning failure attempts cleanup of the +exact project it created. + +All three supported targets—Supabox, a Docker-hosted API platform, and staging— +implement the same HTTP API contract. Retargeting a run only changes +`SUPABASE_LIVE_API_URL` and its access token. The live workflow keeps a Docker +preflight, one serial attempt, a 20-minute bound, and a scoped leftover-project +sweeper. ## Consequences -### Positive - -- The live path has fewer moving parts: no proxy, no streaming relay, no fixture - guards. The Docker bundler talks to the real daemon as users' machines do. -- `replay-server.ts` and the replay/record contract are unchanged, so the - PR-blocking `e2e` suite is unaffected. -- Tests are trivial to add: colocate one `*.live.test.ts` next to the command, - prepare any state with unasserted setup commands, then assert the target - command's real outcome. -- Retargeting from staging to Supabox/local is an environment swap - (`SUPABASE_PROFILE` + `SUPABASE_LIVE_API_URL` + `SUPABASE_LIVE_PROJECT_HOST` - + token), because assertions key off behavior, not hostnames. - -### Negative - -- Live mode requires a working Docker daemon on the runner (enforced by a - `docker info` preflight) — unlike the replay suite, which served Docker - fixtures and needed no daemon. -- Explicit managed runs provision and tear down one real staging project, so those - runs are inherently slower and subject to provisioning flake. The default attached - mode uses the caller-provided Supabox/local project; global setup never deletes it. - Managed CI runs the live suite once with a 20-minute step timeout and always runs - the separate scoped project-cleanup step; it does not amplify deterministic failures - by retrying the entire suite. -- A second wiring path now exists for the same harness (replay-via-server vs - live-direct); contributors must know which mode wires the CLI how. - -## Alternatives Considered - -1. **Third `live` branch inside `replay-server.ts`** (the initial plan): rejected. - It adds `isLive` guards throughout record-mode code, keeps the fragile Docker - stream relay in the hot path for no assertion benefit, and couples live mode to - machinery it does not use. -2. **Snapshot/normalization-first assertions**: rejected as the default. Outcome - assertions on function bodies are naturally ID-agnostic; a scoped normalizer is - added only if a future case makes CLI diagnostic output itself the assertion - target. -3. **Keep live tests in `apps/cli-e2e`**: rejected. The replay/record harness and - its fixtures stay isolated there, while command live tests belong with the - command implementation and share the CLI package's global setup. -4. **One shared long-lived staging project**: rejected. State would leak between - runs and overlapping runs would collide; managed mode owns one project per - run with scoped teardown, while attached mode leaves the caller's project - untouched. - -## Related Decisions - -- [Compiled Bun self-dispatch](../../packages/process-compose/docs/architecture.md#compiled-bun-self-dispatch): - the next CLI e2e harness runs against the compiled binary and therefore exercises its process - re-entry contract -- [ADR 0011](0011-cli-release-and-distribution-strategy.md): CLI Release & Distribution Strategy - -## See Also +The live path has no fixture proxy, host-rewrite layer, attached/managed mode, +ambient profile, project-ref gate, or capability-specific skip wrapper. The +single extended Vitest fixture is imported as `test` from +`apps/cli/tests/helpers/live.ts`; its context exposes `cli`, `project`, and an +isolated workspace. Setup and teardown may invoke other commands, but each +assertion stays focused on one command. -- [CLI live-test guidance](../../apps/cli/AGENTS.md) -- [Live environment example](../../apps/cli/live.env.example) -- [Replay/record harness](../../apps/cli-e2e/AGENTS.md) +Replay/record behavior and fixtures in `apps/cli-e2e` remain unchanged. From faabe4d9c6dc74aae28ec8d08b81eb5fd7ae7afa Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 22:33:16 +0200 Subject: [PATCH 11/23] test(cli): tighten live e2e lifecycle --- .../branches/create/create.live.test.ts | 22 +- .../branches/delete/delete.live.test.ts | 16 +- .../commands/branches/list/list.live.test.ts | 21 +- .../legacy/commands/db/dump/dump.live.test.ts | 12 +- .../legacy/commands/db/pull/pull.live.test.ts | 17 +- .../legacy/commands/db/push/push.live.test.ts | 17 +- .../commands/db/reset/reset.live.test.ts | 22 +- .../db/start/start.docker.e2e.test.ts | 18 +- .../functions/delete/delete.live.test.ts | 10 +- .../commands/functions/list/list.live.test.ts | 24 - .../commands/gen/types/types.live.test.ts | 5 +- .../inspect/db/db-stats/db-stats.live.test.ts | 11 +- .../legacy/commands/link/link.live.test.ts | 4 +- .../migration/fetch/fetch.live.test.ts | 18 +- .../commands/migration/list/list.live.test.ts | 18 +- .../commands/orgs/list/list.live.test.ts | 56 +- .../projects/api-keys/api-keys.live.test.ts | 6 +- .../commands/projects/list/list.live.test.ts | 12 +- .../commands/secrets/list/list.live.test.ts | 14 +- .../commands/secrets/set/set.live.test.ts | 12 +- .../commands/secrets/unset/unset.live.test.ts | 14 +- .../commands/storage/cp/cp.live.test.ts | 22 +- .../commands/storage/ls/ls.live.test.ts | 29 +- .../commands/storage/rm/rm.live.test.ts | 24 +- .../functions/dev/dev.docker.e2e.test.ts | 112 ++-- apps/cli/tests/helpers/cli.ts | 17 +- apps/cli/tests/helpers/live-project.ts | 530 ++++++++++++------ .../tests/helpers/live-project.unit.test.ts | 53 ++ apps/cli/tests/helpers/live.ts | 21 - apps/cli/tests/live-global-setup.ts | 18 +- 30 files changed, 642 insertions(+), 533 deletions(-) create mode 100644 apps/cli/tests/helpers/live-project.unit.test.ts diff --git a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts index fbb5152748..a82659e30b 100644 --- a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts @@ -4,11 +4,11 @@ import { expect } from "vitest"; import { test } from "../../../../../tests/helpers/live.ts"; async function deleteBranch( - run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, name: string, - projectRef: string, + ref: string, ): Promise { - const deleted = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); + const deleted = await cli(["branches", "delete", name, "--project-ref", ref, "--yes"]); if (deleted.exitCode !== 0) { throw new Error( `branches delete cleanup failed (exit ${deleted.exitCode})\n${deleted.stdout}\n${deleted.stderr}`, @@ -16,22 +16,14 @@ async function deleteBranch( } } -test("creates a preview branch", async ({ run, projectRef, skip }) => { +test("creates a preview branch", async ({ cli, project }) => { const name = `cli-e2e-create-${randomUUID().slice(0, 8)}`; - const result = await run(["branches", "create", name, "--project-ref", projectRef]); - - // Branching is plan-gated. A free Supabox org lacks the optional fixture for - // this golden path, so skip only the recognized plan-gate response. - if (result.exitCode !== 0) { - if (/paid plan|upgrade|not.*support/i.test(`${result.stdout}\n${result.stderr}`)) { - skip("Preview branches require a paid plan"); - } - expect(result.exitCode, result.stderr).toBe(0); - } + const result = await cli(["branches", "create", name, "--project-ref", project.ref]); + expect(result.exitCode, result.stderr).toBe(0); try { expect(result.stdout).toContain("Created preview branch"); } finally { - await deleteBranch(run, name, projectRef); + await deleteBranch(cli, name, project.ref); } }); diff --git a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts index c8b72e6bdf..235e8c1473 100644 --- a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts @@ -3,22 +3,16 @@ import { expect } from "vitest"; import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -test("deletes a preview branch", async ({ run, projectRef, skip }) => { +test("deletes a preview branch", async ({ cli, project }) => { const name = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; - const created = await run(["branches", "create", name, "--project-ref", projectRef]); - if ( - created.exitCode !== 0 && - /paid plan|upgrade|not.*support/i.test(`${created.stdout}\n${created.stderr}`) - ) { - skip("Preview branches require a paid plan"); - } + const created = await cli(["branches", "create", name, "--project-ref", project.ref]); requireLiveSuccess(created, "branches create"); let deleted = false; let targetError: unknown; let cleanupError: unknown; try { - const removed = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); + const removed = await cli(["branches", "delete", name, "--project-ref", project.ref, "--yes"]); expect(removed.exitCode, removed.stderr).toBe(0); deleted = true; expect(removed.stderr).toContain("Deleted preview branch"); @@ -27,12 +21,12 @@ test("deletes a preview branch", async ({ run, projectRef, skip }) => { } finally { if (!deleted) { try { - const cleanup = await run([ + const cleanup = await cli([ "branches", "delete", name, "--project-ref", - projectRef, + project.ref, "--yes", ]); if (cleanup.exitCode !== 0 && !/not found/i.test(`${cleanup.stdout}\n${cleanup.stderr}`)) { diff --git a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts index 2835d39535..27561ae802 100644 --- a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts @@ -3,24 +3,25 @@ import { expect } from "vitest"; import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -test("lists a preview branch for the project", async ({ run, projectRef, skip }) => { +test("lists a preview branch for the project", async ({ cli, project }) => { const name = `cli-e2e-list-${randomUUID().slice(0, 8)}`; - const created = await run(["branches", "create", name, "--project-ref", projectRef]); - if ( - created.exitCode !== 0 && - /paid plan|upgrade|not.*support/i.test(`${created.stdout}\n${created.stderr}`) - ) { - skip("Preview branches require a paid plan"); - } + const created = await cli(["branches", "create", name, "--project-ref", project.ref]); requireLiveSuccess(created, "branches create setup"); try { - const result = await run(["branches", "list", "--output", "json", "--project-ref", projectRef]); + const result = await cli([ + "branches", + "list", + "--output", + "json", + "--project-ref", + project.ref, + ]); expect(result.exitCode, result.stderr).toBe(0); const branches = JSON.parse(result.stdout) as Array<{ name?: string }>; expect(branches.map((branch) => branch.name)).toContain(name); } finally { - const deleted = await run(["branches", "delete", name, "--project-ref", projectRef, "--yes"]); + const deleted = await cli(["branches", "delete", name, "--project-ref", project.ref, "--yes"]); requireLiveSuccess(deleted, "branches delete cleanup"); } }); diff --git a/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts b/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts index eb5afb387c..8bf824a885 100644 --- a/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts +++ b/apps/cli/src/legacy/commands/db/dump/dump.live.test.ts @@ -2,17 +2,11 @@ import { existsSync } from "node:fs"; import { join } from "node:path"; import { expect } from "vitest"; -import { liveDatabaseTargetArgs, test } from "../../../../../tests/helpers/live.ts"; +import { test } from "../../../../../tests/helpers/live.ts"; -test("dumps the remote schema to a file", async ({ run, dbUrl, projectRef, workspace }) => { +test("dumps the remote schema to a file", async ({ cli, project, workspace }) => { const outFile = join(workspace.path, "schema.sql"); - const result = await run([ - "db", - "dump", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "-f", - outFile, - ]); + const result = await cli(["db", "dump", "--db-url", project.dbUrl, "-f", outFile]); expect(result.exitCode, result.stderr).toBe(0); expect(existsSync(outFile)).toBe(true); }); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts index 3bbf18d179..360a286c62 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts @@ -2,16 +2,11 @@ import { mkdir, readdir, unlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { - liveDatabaseTargetArgs, - requireLiveSuccess, - test, -} from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; test("pulls the remote schema after a local migration is applied", async ({ - run, - dbUrl, - projectRef, + cli, + project, workspace, }) => { const version = `${Date.now()}${Math.floor(Math.random() * 10_000) @@ -25,10 +20,10 @@ test("pulls the remote schema after a local migration is applied", async ({ let targetError: unknown; try { - const pushed = await run(["db", "push", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + const pushed = await cli(["db", "push", "--db-url", project.dbUrl, "--yes"]); requireLiveSuccess(pushed, "db push setup"); - const result = await run(["db", "pull", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + const result = await cli(["db", "pull", "--db-url", project.dbUrl, "--yes"]); expect(result.exitCode, result.stderr).toBe(0); expect(`${result.stdout}${result.stderr}`).not.toMatch( /dial|no route|connection refused|could not connect|server closed the connection|i\/o timeout/i, @@ -58,7 +53,7 @@ test("pulls the remote schema after a local migration is applied", async ({ } } - const reset = await run(["db", "reset", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + const reset = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); requireLiveSuccess(reset, "db reset cleanup after db pull"); } catch (error) { cleanupError = error; diff --git a/apps/cli/src/legacy/commands/db/push/push.live.test.ts b/apps/cli/src/legacy/commands/db/push/push.live.test.ts index 6211a3d987..31dce484d2 100644 --- a/apps/cli/src/legacy/commands/db/push/push.live.test.ts +++ b/apps/cli/src/legacy/commands/db/push/push.live.test.ts @@ -2,18 +2,9 @@ import { mkdir, unlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { - liveDatabaseTargetArgs, - requireLiveSuccess, - test, -} from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -test("pushes a local migration to the remote database", async ({ - run, - dbUrl, - projectRef, - workspace, -}) => { +test("pushes a local migration to the remote database", async ({ cli, project, workspace }) => { const version = `${Date.now()}${Math.floor(Math.random() * 10_000) .toString() .padStart(4, "0")}`; @@ -23,12 +14,12 @@ test("pushes a local migration to the remote database", async ({ await writeFile(migrationFile, `create table if not exists e2e_push_${version} (id int);\n`); try { - const result = await run(["db", "push", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + const result = await cli(["db", "push", "--db-url", project.dbUrl, "--yes"]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Finished supabase db push"); } finally { await unlink(migrationFile).catch(() => undefined); - const reset = await run(["db", "reset", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + const reset = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); requireLiveSuccess(reset, "db reset cleanup after db push"); } }); diff --git a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts index 8d49ce8c2a..3a03886206 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts @@ -2,18 +2,9 @@ import { mkdir, unlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { - liveDatabaseTargetArgs, - requireLiveSuccess, - test, -} from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -test("resets the remote database with local migrations", async ({ - run, - dbUrl, - projectRef, - workspace, -}) => { +test("resets the remote database with local migrations", async ({ cli, project, workspace }) => { const version = `${Date.now()}${Math.floor(Math.random() * 10_000) .toString() .padStart(4, "0")}`; @@ -23,17 +14,12 @@ test("resets the remote database with local migrations", async ({ await writeFile(migrationFile, `create table if not exists e2e_reset_${version} (id int);\n`); try { - const result = await run([ - "db", - "reset", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--yes", - ]); + const result = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); expect(result.exitCode, result.stderr).toBe(0); expect(`${result.stdout}${result.stderr}`).toContain("Resetting remote database"); } finally { await unlink(migrationFile).catch(() => undefined); - const reset = await run(["db", "reset", ...liveDatabaseTargetArgs(dbUrl, projectRef), "--yes"]); + const reset = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); requireLiveSuccess(reset, "db reset cleanup"); } }); diff --git a/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts index 2798bdfad4..53c916d275 100644 --- a/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts @@ -1,17 +1,29 @@ import { describe, expect, test } from "vitest"; -import { runSupabase } from "../../../../../tests/helpers/cli.ts"; +import { + makeTempHome, + makeTempStackProject, + runSupabase, +} from "../../../../../tests/helpers/cli.ts"; describe("supabase db start (e2e)", () => { test("boots the local database", async () => { + const home = makeTempHome(); + const project = await makeTempStackProject("supabase-db-start-e2e-"); try { - const started = await runSupabase(["db", "start"]); + const started = await runSupabase(["db", "start"], { + cwd: project.dir, + home: home.dir, + }); expect(started.exitCode, started.stderr).toBe(0); expect(`${started.stdout}${started.stderr}`).toMatch( /Starting database|Initialising schema/i, ); } finally { - await runSupabase(["stop", "--no-backup"]).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { + cwd: project.dir, + home: home.dir, + }).catch(() => undefined); } }, 600_000); }); diff --git a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts index 774e2e8e10..b0ee8a692a 100644 --- a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts @@ -4,32 +4,32 @@ import { expect } from "vitest"; import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; -test("deletes a deployed function", async ({ run, projectRef, workspace }) => { +test("deletes a deployed function", async ({ cli, project, workspace }) => { const slug = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; const directory = `${workspace.path}/supabase/functions/${slug}`; await mkdir(directory, { recursive: true }); await writeFile(`${directory}/index.ts`, "Deno.serve(() => Response.json({ ok: true }));\n"); await writeFile(`${directory}/deno.json`, '{\n "imports": {}\n}\n'); - const deployed = await run([ + const deployed = await cli([ "functions", "deploy", slug, "--project-ref", - projectRef, + project.ref, "--use-api", ]); requireLiveSuccess(deployed, "functions deploy setup"); let deleted = false; try { - const result = await run(["functions", "delete", slug, "--project-ref", projectRef]); + const result = await cli(["functions", "delete", slug, "--project-ref", project.ref]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Deleted Function"); deleted = true; } finally { if (!deleted) { - const cleanup = await run(["functions", "delete", slug, "--project-ref", projectRef]); + const cleanup = await cli(["functions", "delete", slug, "--project-ref", project.ref]); requireLiveSuccess(cleanup, "functions delete cleanup"); } } diff --git a/apps/cli/src/legacy/commands/functions/list/list.live.test.ts b/apps/cli/src/legacy/commands/functions/list/list.live.test.ts index daa1f8ef2d..801dbd4906 100644 --- a/apps/cli/src/legacy/commands/functions/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/list/list.live.test.ts @@ -24,27 +24,3 @@ describe("supabase functions list (live)", () => { }, ); }); - -// Project-scoped error path that needs NO provisioned project: a valid token -// with an unknown `--project-ref` must reach the live Management API, come back -// 404, and surface as a non-zero exit (not a crash, not "Unauthorized"). This -// exercises the `--project-ref` request path + error mapping on a control-plane- -// only stack, so it runs under the same shared live fixture. -describe("supabase functions list — unknown project (live)", () => { - test( - "fails with a 404 for an unknown project ref", - { timeout: LIVE_TIMEOUT_MS }, - async ({ cli }) => { - const { exitCode, stdout, stderr } = await cli([ - "functions", - "list", - "--project-ref", - "a".repeat(20), // well-formed (20 lowercase chars) but nonexistent ref - ]); - const out = `${stdout}${stderr}`; - expect(exitCode).not.toBe(0); - expect(out).not.toContain("Unauthorized"); - expect(out).toContain("404"); - }, - ); -}); diff --git a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts index b5b8dd79b7..ac1acc30fe 100644 --- a/apps/cli/src/legacy/commands/gen/types/types.live.test.ts +++ b/apps/cli/src/legacy/commands/gen/types/types.live.test.ts @@ -2,9 +2,8 @@ import { expect } from "vitest"; import { test } from "../../../../../tests/helpers/live.ts"; -test("generates TypeScript types from the remote schema", async ({ run, dbUrl, projectRef }) => { - const targetArgs = dbUrl.length > 0 ? ["--db-url", dbUrl] : ["--project-id", projectRef]; - const result = await run(["gen", "types", ...targetArgs, "--lang", "typescript"]); +test("generates TypeScript types from the remote schema", async ({ cli, project }) => { + const result = await cli(["gen", "types", "--db-url", project.dbUrl, "--lang", "typescript"]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toMatch(/export type (Database|Json)/); }); diff --git a/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts b/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts index b143ca9d8a..c747836609 100644 --- a/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts +++ b/apps/cli/src/legacy/commands/inspect/db/db-stats/db-stats.live.test.ts @@ -1,14 +1,9 @@ import { expect } from "vitest"; -import { liveDatabaseTargetArgs, test } from "../../../../../../tests/helpers/live.ts"; +import { test } from "../../../../../../tests/helpers/live.ts"; -test("reports statistics from the remote database", async ({ run, dbUrl, projectRef }) => { - const result = await run([ - "inspect", - "db", - "db-stats", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - ]); +test("reports statistics from the remote database", async ({ cli, project }) => { + const result = await cli(["inspect", "db", "db-stats", "--db-url", project.dbUrl]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Database Size"); }); diff --git a/apps/cli/src/legacy/commands/link/link.live.test.ts b/apps/cli/src/legacy/commands/link/link.live.test.ts index 293e79788a..47bf065f1d 100644 --- a/apps/cli/src/legacy/commands/link/link.live.test.ts +++ b/apps/cli/src/legacy/commands/link/link.live.test.ts @@ -4,8 +4,8 @@ import { expect } from "vitest"; import { test } from "../../../../tests/helpers/live.ts"; -test("links a project and writes its workspace cache", async ({ run, projectRef, workspace }) => { - const result = await run(["link", "--project-ref", projectRef, "--skip-pooler"]); +test("links a project and writes its workspace cache", async ({ cli, project, workspace }) => { + const result = await cli(["link", "--project-ref", project.ref, "--skip-pooler"]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Finished supabase link"); expect(existsSync(join(workspace.path, "supabase", ".temp", "linked-project.json"))).toBe(true); diff --git a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts index f8b61db945..09b530c29d 100644 --- a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts @@ -3,11 +3,7 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { expect } from "vitest"; -import { - liveDatabaseTargetArgs, - requireLiveSuccess, - test, -} from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; const LIVE_TIMEOUT_MS = 120_000; @@ -20,7 +16,7 @@ function liveMigrationVersion(): string { // Destructive data-plane scenario (Postgres over the pooler) — the setup repairs // remote migration history and the teardown reverts that exact row. The fixture -// keeps this opt-in for attached runs and waits for an ACTIVE_HEALTHY project. +// provisions one ACTIVE_HEALTHY project for the serial live suite. // // Golden path: `migration fetch` reads the remote `schema_migrations` history and // writes each row to `supabase/migrations/_.sql`. @@ -37,8 +33,8 @@ function liveMigrationVersion(): string { test( "fetches a seeded remote migration into the local migrations directory", { timeout: LIVE_TIMEOUT_MS }, - async ({ run, dbUrl, projectRef }) => { - const targetArgs = liveDatabaseTargetArgs(dbUrl, projectRef); + async ({ cli, project }) => { + const targetArgs = ["--db-url", project.dbUrl]; const version = liveMigrationVersion(); const migrationFile = `${version}_${NAME}.sql`; const seedDir = await mkdtemp(path.join(tmpdir(), "sb-migration-seed-live-")); @@ -54,7 +50,7 @@ test( path.join(seedDir, "supabase", "migrations", migrationFile), "create table if not exists public.cli_live_roundtrip (id int);\n", ); - const repairResult = await run( + const repairResult = await cli( ["migration", "repair", version, "--status", "applied", ...targetArgs], { cwd: seedDir }, ); @@ -63,7 +59,7 @@ test( // Fetch into a fresh (empty) dir so no overwrite prompt fires; it reads the // remote history and writes _.sql. - const fetched = await run(["migration", "fetch", ...targetArgs], { cwd: fetchDir }); + const fetched = await cli(["migration", "fetch", ...targetArgs], { cwd: fetchDir }); expect(fetched.exitCode, `stdout:\n${fetched.stdout}\nstderr:\n${fetched.stderr}`).toBe(0); // fetch wrote the seeded migration back, under its established filename format. @@ -74,7 +70,7 @@ test( } finally { if (repaired) { try { - const reverted = await run( + const reverted = await cli( ["migration", "repair", version, "--status", "reverted", ...targetArgs], { cwd: seedDir }, ); diff --git a/apps/cli/src/legacy/commands/migration/list/list.live.test.ts b/apps/cli/src/legacy/commands/migration/list/list.live.test.ts index e8ef58dd11..d2cc786163 100644 --- a/apps/cli/src/legacy/commands/migration/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/list/list.live.test.ts @@ -1,21 +1,9 @@ import { expect } from "vitest"; -import { liveDatabaseTargetArgs, test } from "../../../../../tests/helpers/live.ts"; +import { test } from "../../../../../tests/helpers/live.ts"; -test("lists migrations from the remote database", async ({ run, dbUrl, projectRef }) => { - const result = await run(["migration", "list", ...liveDatabaseTargetArgs(dbUrl, projectRef)]); +test("lists migrations from the remote database", async ({ cli, project }) => { + const result = await cli(["migration", "list", "--db-url", project.dbUrl]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).not.toContain("Unauthorized"); }); - -test("emits migration list as JSON", async ({ run, dbUrl, projectRef }) => { - const result = await run([ - "migration", - "list", - ...liveDatabaseTargetArgs(dbUrl, projectRef), - "--output-format", - "json", - ]); - expect(result.exitCode, result.stderr).toBe(0); - expect(() => JSON.parse(result.stdout)).not.toThrow(); -}); diff --git a/apps/cli/src/legacy/commands/orgs/list/list.live.test.ts b/apps/cli/src/legacy/commands/orgs/list/list.live.test.ts index 2e1dcf7e88..2b4217489a 100644 --- a/apps/cli/src/legacy/commands/orgs/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/orgs/list/list.live.test.ts @@ -1,51 +1,19 @@ -import { describe, expect } from "vitest"; +import { expect } from "vitest"; import { test } from "../../../../../tests/helpers/live.ts"; const LIVE_TIMEOUT_MS = 60_000; -// Harness smoke for the `live` Vitest project: the canonical example of a live -// test. It exercises the full path — built binary → SUPABASE_PROFILE resolution +// Harness smoke for the live Vitest project: the canonical example of a live +// test. It exercises the full path — built binary → temporary profile resolution // → authenticated Management API request against the running platform — with a // read-only call, so it is safe to run repeatedly and creates no resources. // -// Gated by `describe`: skipped unless SUPABASE_ACCESS_TOKEN is set (the -// cli-e2e-ci runner provides supabox's seeded PAT). Broader lifecycle scenarios -// (projects, functions, branching, db, storage) build on this same harness. -describe("supabase orgs list (live)", () => { - test( - "lists organizations for the authenticated token", - { timeout: LIVE_TIMEOUT_MS }, - async ({ cli }) => { - const { exitCode, stdout, stderr } = await cli(["orgs", "list"]); - expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); - expect(exitCode).toBe(0); - }, - ); - - test( - "emits machine-readable JSON with --output-format json", - { timeout: LIVE_TIMEOUT_MS }, - async ({ cli }) => { - const { exitCode, stdout } = await cli(["orgs", "list", "--output-format", "json"]); - expect(exitCode).toBe(0); - // stdout must be payload-only valid JSON in json mode (no spinner/log noise). - expect(() => JSON.parse(stdout)).not.toThrow(); - }, - ); - - // Negative path: a bad token must round-trip to the real Management API, come - // back 401, and surface as a non-zero exit with the upstream "Unauthorized" - // message — i.e. the cli's auth + error mapping work against the live stack, - // not just the golden path. Overrides only the token (profile stays set). - test( - "fails with Unauthorized for an invalid token", - { timeout: LIVE_TIMEOUT_MS }, - async ({ cli }) => { - const { exitCode, stdout, stderr } = await cli(["orgs", "list"], { - env: { SUPABASE_ACCESS_TOKEN: `sbp_${"0".repeat(40)}` }, - }); - expect(exitCode).not.toBe(0); - expect(`${stdout}${stderr}`).toContain("Unauthorized"); - }, - ); -}); +test( + "lists organizations for the authenticated token", + { timeout: LIVE_TIMEOUT_MS }, + async ({ cli }) => { + const { exitCode, stdout, stderr } = await cli(["orgs", "list"]); + expect(`${stdout}${stderr}`).not.toContain("Unauthorized"); + expect(exitCode).toBe(0); + }, +); diff --git a/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts b/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts index 440847789a..b7b7ba7097 100644 --- a/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts +++ b/apps/cli/src/legacy/commands/projects/api-keys/api-keys.live.test.ts @@ -2,12 +2,12 @@ import { expect } from "vitest"; import { test } from "../../../../../tests/helpers/live.ts"; -test("lists API keys for a project", async ({ run, projectRef }) => { - const result = await run([ +test("lists API keys for a project", async ({ cli, project }) => { + const result = await cli([ "projects", "api-keys", "--project-ref", - projectRef, + project.ref, "--output", "json", ]); diff --git a/apps/cli/src/legacy/commands/projects/list/list.live.test.ts b/apps/cli/src/legacy/commands/projects/list/list.live.test.ts index 54aba6edda..12db8ce9b7 100644 --- a/apps/cli/src/legacy/commands/projects/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/projects/list/list.live.test.ts @@ -2,8 +2,8 @@ import { expect } from "vitest"; import { test } from "../../../../../tests/helpers/live.ts"; -test("lists the live project for the authenticated token", async ({ run, projectRef }) => { - const result = await run(["projects", "list", "--output-format", "json"]); +test("lists the live project for the authenticated token", async ({ cli, project }) => { + const result = await cli(["projects", "list", "--output-format", "json"]); expect(result.exitCode, result.stderr).toBe(0); const parsed: unknown = JSON.parse(result.stdout); expect(parsed).toEqual(expect.objectContaining({ projects: expect.any(Array) })); @@ -21,11 +21,5 @@ test("lists the live project for the authenticated token", async ({ run, project if ("id" in project && typeof project.id === "string") return [project.id]; return []; }); - expect(refs).toContain(projectRef); -}); - -test("emits projects as JSON for an account-level read", async ({ run }) => { - const result = await run(["projects", "list", "--output-format", "json"]); - expect(result.exitCode, result.stderr).toBe(0); - expect(() => JSON.parse(result.stdout)).not.toThrow(); + expect(refs).toContain(project.ref); }); diff --git a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts index 1385301f19..391c472d13 100644 --- a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts @@ -4,29 +4,29 @@ import { expect } from "vitest"; import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( - run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, name: string, - projectRef: string, + ref: string, ): Promise { - const cleanup = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); + const cleanup = await cli(["secrets", "unset", name, "--project-ref", ref, "--yes"]); if (cleanup.exitCode !== 0) { throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); } } -test("lists a secret created on the remote project", async ({ run, projectRef }) => { +test("lists a secret created on the remote project", async ({ cli, project }) => { const name = `CLI_E2E_LIST_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; - const created = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); + const created = await cli(["secrets", "set", `${name}=live-value`, "--project-ref", project.ref]); requireLiveSuccess(created, "secrets set setup"); try { - const result = await run(["secrets", "list", "--output", "json", "--project-ref", projectRef]); + const result = await cli(["secrets", "list", "--output", "json", "--project-ref", project.ref]); expect(result.exitCode, result.stderr).toBe(0); const names = (JSON.parse(result.stdout) as Array<{ name: string }>).map( (secret) => secret.name, ); expect(names).toContain(name); } finally { - await unsetSecret(run, name, projectRef); + await unsetSecret(cli, name, project.ref); } }); diff --git a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts index 82cb1294aa..8bffcf29f8 100644 --- a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts @@ -4,23 +4,23 @@ import { expect } from "vitest"; import { test } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( - run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, name: string, - projectRef: string, + ref: string, ): Promise { - const cleanup = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); + const cleanup = await cli(["secrets", "unset", name, "--project-ref", ref, "--yes"]); if (cleanup.exitCode !== 0) { throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); } } -test("sets a secret on the remote project", async ({ run, projectRef }) => { +test("sets a secret on the remote project", async ({ cli, project }) => { const name = `CLI_E2E_SET_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; - const result = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); + const result = await cli(["secrets", "set", `${name}=live-value`, "--project-ref", project.ref]); try { expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Finished"); } finally { - await unsetSecret(run, name, projectRef); + await unsetSecret(cli, name, project.ref); } }); diff --git a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts index 7dc3771590..1bc427d11d 100644 --- a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts @@ -4,26 +4,26 @@ import { expect } from "vitest"; import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( - run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, name: string, - projectRef: string, + ref: string, ): Promise { - const cleanup = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); + const cleanup = await cli(["secrets", "unset", name, "--project-ref", ref, "--yes"]); if (cleanup.exitCode !== 0 && !/not found/i.test(`${cleanup.stdout}\n${cleanup.stderr}`)) { throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); } } -test("unsets a secret from the remote project", async ({ run, projectRef }) => { +test("unsets a secret from the remote project", async ({ cli, project }) => { const name = `CLI_E2E_UNSET_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; - const created = await run(["secrets", "set", `${name}=live-value`, "--project-ref", projectRef]); + const created = await cli(["secrets", "set", `${name}=live-value`, "--project-ref", project.ref]); requireLiveSuccess(created, "secrets set setup"); let deleted = false; let targetError: unknown; let cleanupError: unknown; try { - const result = await run(["secrets", "unset", name, "--project-ref", projectRef, "--yes"]); + const result = await cli(["secrets", "unset", name, "--project-ref", project.ref, "--yes"]); expect(result.exitCode, result.stderr).toBe(0); deleted = true; expect(result.stdout).toContain("Finished"); @@ -32,7 +32,7 @@ test("unsets a secret from the remote project", async ({ run, projectRef }) => { } finally { if (!deleted) { try { - await unsetSecret(run, name, projectRef); + await unsetSecret(cli, name, project.ref); } catch (error) { cleanupError = error; } diff --git a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts index a45816bed2..98dc95bdc8 100644 --- a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts @@ -8,38 +8,32 @@ import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; async function removeObject( - run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, remote: string, ): Promise { - const removed = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + const removed = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); if (removed.exitCode !== 0) { throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); } } -test("copies a local file to the remote bucket", async ({ - run, - projectRef, - dbPassword, - storageBucket, - workspace, -}) => { +test("copies a local file to the remote bucket", async ({ cli, project, workspace }) => { const suffix = randomUUID().slice(0, 8); const local = join(workspace.path, `upload-${suffix}.txt`); - const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + const remote = `ss:///${project.storageBucket}/upload-${suffix}.txt`; await writeFile(local, "live-e2e storage payload\n"); - const linked = await run(["link", "--project-ref", projectRef], { - env: { SUPABASE_DB_PASSWORD: dbPassword }, + const linked = await cli(["link", "--project-ref", project.ref], { + env: { SUPABASE_DB_PASSWORD: project.dbPassword }, }); requireLiveSuccess(linked, "link setup for storage cp"); let uploaded = false; try { - const result = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + const result = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); expect(result.exitCode, result.stderr).toBe(0); uploaded = true; } finally { - if (uploaded) await removeObject(run, remote); + if (uploaded) await removeObject(cli, remote); } }); diff --git a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts index e963b1418b..a73463825e 100644 --- a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts @@ -8,39 +8,38 @@ import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; async function removeObject( - run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, remote: string, ): Promise { - const removed = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + const removed = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); if (removed.exitCode !== 0) { throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); } } -test("lists an uploaded object", async ({ - run, - projectRef, - dbPassword, - storageBucket, - workspace, -}) => { +test("lists an uploaded object", async ({ cli, project, workspace }) => { const suffix = randomUUID().slice(0, 8); const local = join(workspace.path, `upload-${suffix}.txt`); - const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + const remote = `ss:///${project.storageBucket}/upload-${suffix}.txt`; await writeFile(local, "live-e2e storage payload\n"); - const linked = await run(["link", "--project-ref", projectRef], { - env: { SUPABASE_DB_PASSWORD: dbPassword }, + const linked = await cli(["link", "--project-ref", project.ref], { + env: { SUPABASE_DB_PASSWORD: project.dbPassword }, }); requireLiveSuccess(linked, "link setup for storage ls"); - const uploaded = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + const uploaded = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); requireLiveSuccess(uploaded, "storage cp setup for storage ls"); try { - const result = await run(["storage", "ls", `ss:///${storageBucket}/`, ...STORAGE_FLAGS]); + const result = await cli([ + "storage", + "ls", + `ss:///${project.storageBucket}/`, + ...STORAGE_FLAGS, + ]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain(`upload-${suffix}.txt`); } finally { - await removeObject(run, remote); + await removeObject(cli, remote); } }); diff --git a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts index 0e4a0061e2..0cfb89ddaf 100644 --- a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts @@ -8,40 +8,34 @@ import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; async function removeObject( - run: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, remote: string, ): Promise { - const removed = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + const removed = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); if (removed.exitCode !== 0) { throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); } } -test("removes an uploaded object", async ({ - run, - projectRef, - dbPassword, - storageBucket, - workspace, -}) => { +test("removes an uploaded object", async ({ cli, project, workspace }) => { const suffix = randomUUID().slice(0, 8); const local = join(workspace.path, `upload-${suffix}.txt`); - const remote = `ss:///${storageBucket}/upload-${suffix}.txt`; + const remote = `ss:///${project.storageBucket}/upload-${suffix}.txt`; await writeFile(local, "live-e2e storage payload\n"); - const linked = await run(["link", "--project-ref", projectRef], { - env: { SUPABASE_DB_PASSWORD: dbPassword }, + const linked = await cli(["link", "--project-ref", project.ref], { + env: { SUPABASE_DB_PASSWORD: project.dbPassword }, }); requireLiveSuccess(linked, "link setup for storage rm"); - const uploaded = await run(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + const uploaded = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); requireLiveSuccess(uploaded, "storage cp setup for storage rm"); let removed = false; try { - const result = await run(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); + const result = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); expect(result.exitCode, result.stderr).toBe(0); removed = true; } finally { - if (!removed) await removeObject(run, remote); + if (!removed) await removeObject(cli, remote); } }); diff --git a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts index 3b38671458..3bf4f4d774 100644 --- a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts +++ b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts @@ -1,9 +1,10 @@ -import { afterEach, describe, expect, test } from "vitest"; -import { writeFile } from "node:fs/promises"; +import { mkdir, writeFile } from "node:fs/promises"; import { join } from "node:path"; +import { afterEach, describe, expect, test } from "vitest"; + import { + makeTempCliProject, makeTempHome, - makeTempStackProject, runSupabase, spawnSupabase, } from "../../../../../tests/helpers/cli.ts"; @@ -13,66 +14,25 @@ const FUNCTIONS_DEV_STARTUP_TIMEOUT_MS = 60_000; const FUNCTIONS_DEV_STEP_TIMEOUT_MS = 30_000; const FUNCTIONS_DEV_TEST_TIMEOUT_MS = 90_000; const FUNCTION_FILES_RESTART_PATTERN = /Function files changed\. Restarting edge-runtime\./; -const FUNCTION_FILES_RESTART_PATTERN_GLOBAL = /Function files changed\. Restarting edge-runtime\./g; type SpawnedSupabase = ReturnType; -function countOutputMatches(proc: SpawnedSupabase, pattern: RegExp): number { - return [...`${proc.stdout()}\n${proc.stderr()}`.matchAll(pattern)].length; -} - -async function waitForOutputMatchCount( - proc: SpawnedSupabase, - pattern: RegExp, - expectedCount: number, -) { - const deadline = Date.now() + FUNCTIONS_DEV_STEP_TIMEOUT_MS; - - while (Date.now() < deadline) { - if (countOutputMatches(proc, pattern) >= expectedCount) { - return; - } - await new Promise((resolve) => setTimeout(resolve, 250)); - } - - throw new Error( - `Timed out waiting for ${expectedCount.toString()} occurrences of ${pattern.toString()}`, - ); -} - -async function waitForFunctionResponse( +async function assertFunctionResponse( url: string, init: RequestInit, assertResponse: (response: Response, body: string) => void, -) { - const deadline = Date.now() + FUNCTIONS_DEV_STEP_TIMEOUT_MS; - let lastError: unknown; - - while (Date.now() < deadline) { - try { - const response = await fetch(url, init); - const body = await response.text(); - try { - assertResponse(response, body); - return; - } catch (error) { - lastError = error; - } - } catch (error) { - lastError = error; - } - - await new Promise((resolve) => setTimeout(resolve, 250)); +): Promise { + try { + const response = await fetch(url, init); + const body = await response.text(); + assertResponse(response, body); + } catch (error) { + throw new Error( + `Function request ${url} failed: ${error instanceof Error ? error.message : String(error)}`, + ); } - - throw lastError instanceof Error - ? lastError - : new Error(`Timed out waiting for function response: ${String(lastError)}`); } -// This crosses the compiled CLI, detached supervisor, full local stack, file -// watcher, and HTTP runtime boundaries. Keep the one golden path in the -// opt-in e2e suite instead of slowing and destabilizing ordinary e2e shards. describe("supabase functions dev (e2e)", () => { afterEach(cleanupRegisteredStackProjects); @@ -81,10 +41,16 @@ describe("supabase functions dev (e2e)", () => { { timeout: FUNCTIONS_DEV_TEST_TIMEOUT_MS }, async () => { const home = makeTempHome(); - const project = await makeTempStackProject("supabase-functions-dev-e2e-"); + // The next functions runtime owns managed port allocation. This project + // intentionally contains no released-port reservations from the test. + const project = await makeTempCliProject("supabase-functions-dev-e2e-"); + await mkdir(join(project.dir, "supabase"), { recursive: true }); + await writeFile( + join(project.dir, "supabase", "config.toml"), + 'project_id = "functions-dev-e2e"\n', + ); const functionPath = join(project.dir, "supabase", "functions", "hello-world", "index.ts"); - const functionUrl = `http://127.0.0.1:${project.ports.apiPort}/functions/v1/hello-world`; - let devProc: ReturnType | undefined; + let devProc: SpawnedSupabase | undefined; try { devProc = spawnSupabase(["functions", "dev"], { @@ -98,7 +64,15 @@ describe("supabase functions dev (e2e)", () => { /Edge Functions dev server is running\./, FUNCTIONS_DEV_STARTUP_TIMEOUT_MS, ); - await new Promise((resolve) => setTimeout(resolve, 500)); + const functionUrlMatch = `${devProc.stdout()}\n${devProc.stderr()}`.match( + /Functions URL:\s+(https?:\/\/[^\s/]+\/functions\/v1)/, + ); + if (functionUrlMatch?.[1] === undefined) { + throw new Error( + `Functions dev output did not include a URL.\nstdout:\n${devProc.stdout()}\nstderr:\n${devProc.stderr()}`, + ); + } + const functionUrl = `${functionUrlMatch[1]}/hello-world`; const newResult = await runSupabase(["functions", "new", "hello-world"], { cwd: project.dir, @@ -106,14 +80,14 @@ describe("supabase functions dev (e2e)", () => { exitTimeoutMs: FUNCTIONS_DEV_STEP_TIMEOUT_MS, }); expect(newResult.exitCode).toBe(0); - await devProc.waitForOutput(FUNCTION_FILES_RESTART_PATTERN, FUNCTIONS_DEV_STEP_TIMEOUT_MS); - await waitForFunctionResponse(functionUrl, {}, (response, body) => { + await assertFunctionResponse(functionUrl, {}, (response, body) => { expect(response.status).toBe(401); expect(body).toContain("Missing authorization header"); }); + const configOffset = devProc.stdout().length; await writeFile( join(project.dir, "supabase", "config.toml"), `project_id = "functions-dev-e2e" @@ -122,13 +96,13 @@ describe("supabase functions dev (e2e)", () => { verify_jwt = false `, ); - await devProc.waitForOutput( /Edge runtime config changed\. Restarting edge-runtime\./, FUNCTIONS_DEV_STEP_TIMEOUT_MS, + configOffset, ); - await waitForFunctionResponse( + await assertFunctionResponse( functionUrl, { method: "POST", @@ -141,7 +115,7 @@ verify_jwt = false }, ); - const restartCount = countOutputMatches(devProc, FUNCTION_FILES_RESTART_PATTERN_GLOBAL); + const sourceOffset = devProc.stdout().length; await writeFile( functionPath, `Deno.serve(() => { @@ -151,13 +125,13 @@ verify_jwt = false }); `, ); - await waitForOutputMatchCount( - devProc, - FUNCTION_FILES_RESTART_PATTERN_GLOBAL, - restartCount + 1, + await devProc.waitForOutput( + FUNCTION_FILES_RESTART_PATTERN, + FUNCTIONS_DEV_STEP_TIMEOUT_MS, + sourceOffset, ); - await waitForFunctionResponse( + await assertFunctionResponse( functionUrl, { method: "POST", @@ -171,7 +145,7 @@ verify_jwt = false ); } finally { devProc?.kill("SIGTERM"); - await devProc?.waitForExit().catch(() => {}); + await devProc?.waitForExit().catch(() => undefined); } }, ); diff --git a/apps/cli/tests/helpers/cli.ts b/apps/cli/tests/helpers/cli.ts index fed5b88b23..3110d7cdc0 100644 --- a/apps/cli/tests/helpers/cli.ts +++ b/apps/cli/tests/helpers/cli.ts @@ -84,7 +84,7 @@ interface SpawnedSupabase { readonly stdout: () => string; readonly stderr: () => string; readonly kill: (signal?: NodeJS.Signals) => void; - readonly waitForOutput: (pattern: RegExp, timeoutMs?: number) => Promise; + readonly waitForOutput: (pattern: RegExp, timeoutMs?: number, startAt?: number) => Promise; readonly waitForExit: (timeoutMs?: number) => Promise; } @@ -141,6 +141,13 @@ async function makeTempProject(prefix = "supabase-project-e2e-") { }; } +/** Create an isolated CLI project without pre-allocating released ports. */ +export async function makeTempCliProject(prefix = "supabase-cli-e2e-") { + const project = await makeTempProject(prefix); + registerTempStackProject(project); + return project; +} + export async function makeTempStackProject(prefix = "supabase-stack-e2e-") { const project = await makeTempProject(prefix); const ports = { @@ -369,8 +376,9 @@ export function spawnSupabase( proc.kill(signal); } catch {} }, - waitForOutput: async (pattern: RegExp, timeoutMs = 60_000) => { - if (pattern.test(stdout)) { + waitForOutput: async (pattern: RegExp, timeoutMs = 60_000, startAt = 0) => { + pattern.lastIndex = 0; + if (pattern.test(stdout.slice(startAt))) { return; } if (closeResult) { @@ -402,7 +410,8 @@ export function spawnSupabase( }, timeoutMs); const onStdout = (_data: Buffer) => { - if (pattern.test(stdout)) { + pattern.lastIndex = 0; + if (pattern.test(stdout.slice(startAt))) { cleanup(); resolve(); } diff --git a/apps/cli/tests/helpers/live-project.ts b/apps/cli/tests/helpers/live-project.ts index e6e18732cb..1686dd63e3 100644 --- a/apps/cli/tests/helpers/live-project.ts +++ b/apps/cli/tests/helpers/live-project.ts @@ -4,13 +4,11 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { makeApiClient, type OperationOutput } from "@supabase/api/effect"; -import { Effect } from "effect"; -import { FetchHttpClient } from "effect/unstable/http"; +import { Cause, Data, Effect, Exit, Schedule } from "effect"; import { deriveLiveProjectHost, keepLiveProject, - liveAccessToken, liveApiUrl, liveOrgId, liveProjectName, @@ -20,10 +18,13 @@ import { const PROJECT_REF_RE = /^[a-z]{20}$/u; const TERMINAL_BAD_STATUSES = new Set(["INIT_FAILED", "RESTORE_FAILED", "REMOVED"]); const PROFILE_NAME = "supabase-cli-live"; +const POLL_INTERVAL = "5 seconds"; +const POLL_TIMEOUT = "5 minutes"; type Project = OperationOutput<"v1GetProject">; type Organization = OperationOutput<"v1ListAllOrganizations">[number]; type ApiKey = OperationOutput<"v1GetProjectApiKeys">[number]; +type LiveApi = Effect.Success>; type Region = | "us-east-1" | "us-east-2" @@ -65,24 +66,72 @@ const REGIONS: ReadonlyArray = [ "sa-east-1", ]; +class LiveTransientPoll extends Data.TaggedError("LiveTransientPoll")<{ + readonly phase: string; + readonly cause?: unknown; +}> {} + +class LiveTerminalPoll extends Data.TaggedError("LiveTerminalPoll")<{ + readonly phase: string; + readonly message: string; +}> {} + +class LivePollTimeout extends Data.TaggedError("LivePollTimeout")<{ + readonly phase: string; +}> { + override get message(): string { + return `${this.phase} timed out`; + } +} + +class LiveStorageError extends Data.TaggedError("LiveStorageError")<{ + readonly message: string; +}> {} + +function apiError(error: unknown): Error { + return error instanceof Error ? error : new Error(String(error)); +} + function supportedRegion(value: string): Region { const region = REGIONS.find((candidate) => candidate === value); if (region !== undefined) return region; throw new Error(`Unsupported SUPABASE_LIVE_REGION ${JSON.stringify(value)}`); } -function apiError(error: unknown): Error { - return error instanceof Error ? error : new Error(String(error)); +/** Retry a transient management operation using Effect's schedule and deadline semantics. */ +export function retryLiveEffect( + phase: string, + effect: Effect.Effect, + options: { + readonly interval?: import("effect").Duration.Input; + readonly timeout?: import("effect").Duration.Input; + readonly shouldRetry?: (error: E) => boolean; + } = {}, +): Effect.Effect { + const retrying = Effect.retry(effect, { + schedule: Schedule.spaced(options.interval ?? POLL_INTERVAL), + ...(options.shouldRetry === undefined ? {} : { while: options.shouldRetry }), + }); + return Effect.timeoutOrElse(retrying, { + duration: options.timeout ?? POLL_TIMEOUT, + orElse: () => Effect.fail(new LivePollTimeout({ phase })), + }); } -function makeLiveApi() { - return makeApiClient({ baseUrl: liveApiUrl(), accessToken: liveAccessToken() }).pipe( - Effect.provide(FetchHttpClient.layer), - ); +/** Build one diagnostic while retaining every target and cleanup failure. */ +export function cleanupErrors(primary: unknown, cleanup: ReadonlyArray): AggregateError { + const errors = [primary, ...cleanup].map(apiError); + return new AggregateError(errors, "Live e2e lifecycle failed"); } -async function runLiveEffect(effect: Effect.Effect): Promise { - return Effect.runPromise(effect); +function timeoutLiveRequest( + phase: string, + effect: Effect.Effect, +): Effect.Effect { + return Effect.timeoutOrElse(effect, { + duration: POLL_TIMEOUT, + orElse: () => Effect.fail(new Error(`${phase} timed out`)), + }); } function uniqueProjectName(): string { @@ -94,136 +143,289 @@ function databasePassword(): string { return `supabase-cli-live-${randomBytes(12).toString("hex")}`; } -async function resolveOrganization(): Promise { - const api = await runLiveEffect(makeLiveApi()); - const organizations = await runLiveEffect(api.v1.listAllOrganizations()); - const requested = liveOrgId(); - const organization = - (requested === undefined - ? organizations[0] - : organizations.find( - (candidate) => candidate.id === requested || candidate.slug === requested, - )) ?? undefined; - if (organization === undefined) { - throw new Error( - requested === undefined - ? "No organizations found; cannot create the live project" - : `Organization ${requested} was not found; cannot create the live project`, - ); - } - return organization; +function resolveOrganization(api: LiveApi): Effect.Effect { + return timeoutLiveRequest("organization lookup", api.v1.listAllOrganizations()).pipe( + Effect.mapError(apiError), + Effect.flatMap((organizations) => { + const requested = liveOrgId(); + const organization = + requested === undefined + ? organizations[0] + : organizations.find( + (candidate) => candidate.id === requested || candidate.slug === requested, + ); + return organization === undefined + ? Effect.fail( + new Error( + requested === undefined + ? "No organizations found; cannot create the live project" + : `Organization ${requested} was not found; cannot create the live project`, + ), + ) + : Effect.succeed(organization); + }), + ); +} + +function createProject( + api: LiveApi, + name: string, + password: string, +): Effect.Effect { + return resolveOrganization(api).pipe( + Effect.flatMap((organization) => + timeoutLiveRequest( + "project creation", + api.v1.createAProject({ + name, + db_pass: password, + organization_slug: organization.slug, + region: supportedRegion(liveRegion()), + }), + ).pipe(Effect.mapError(apiError)), + ), + Effect.flatMap((project) => + PROJECT_REF_RE.test(project.ref) + ? Effect.succeed(project.ref) + : Effect.fail(new Error(`Unexpected project ref from project creation: ${project.ref}`)), + ), + ); +} + +function deleteProject(api: LiveApi, ref: string): Effect.Effect { + return timeoutLiveRequest("project deletion", api.v1.deleteAProject({ ref })).pipe( + Effect.mapError(apiError), + Effect.asVoid, + ); +} + +function projectReadiness( + api: LiveApi, + ref: string, +): Effect.Effect { + return api.v1.getProject({ ref }).pipe( + Effect.mapError( + (cause): LiveTransientPoll => new LiveTransientPoll({ phase: "project readiness", cause }), + ), + Effect.flatMap( + (project): Effect.Effect => { + if (project.status === "ACTIVE_HEALTHY") return Effect.succeed(project); + if (TERMINAL_BAD_STATUSES.has(project.status)) { + return Effect.fail( + new LiveTerminalPoll({ + phase: "project readiness", + message: `Project ${ref} entered terminal status ${project.status}`, + }), + ); + } + return Effect.fail( + new LiveTransientPoll({ + phase: "project readiness", + cause: `status=${project.status}`, + }), + ); + }, + ), + ); } -async function createProject(name: string, password: string): Promise { - const organization = await resolveOrganization(); - const api = await runLiveEffect(makeLiveApi()); - const project = await runLiveEffect( - api.v1.createAProject({ - name, - db_pass: password, - organization_slug: organization.slug, - region: supportedRegion(liveRegion()), +function waitForProject(api: LiveApi, ref: string): Effect.Effect { + return retryLiveEffect("project readiness", projectReadiness(api, ref), { + shouldRetry: (error) => error instanceof LiveTransientPoll, + }).pipe( + Effect.mapError((error) => { + if (error instanceof LiveTerminalPoll) return new Error(error.message); + if (error instanceof LivePollTimeout) return new Error(error.message); + return apiError(error); }), ); - if (!PROJECT_REF_RE.test(project.ref)) { - throw new Error(`Unexpected project ref from project creation: ${project.ref}`); - } - return project.ref; } -async function deleteProject(ref: string): Promise { - const api = await runLiveEffect(makeLiveApi()); - await runLiveEffect(api.v1.deleteAProject({ ref })); +function keysReadiness( + api: LiveApi, + ref: string, +): Effect.Effect<{ anonKey: string; serviceRoleKey: string }, LiveTransientPoll, never> { + return api.v1.getProjectApiKeys({ ref, reveal: true }).pipe( + Effect.mapError((cause) => new LiveTransientPoll({ phase: "project API keys", cause })), + Effect.flatMap((keys) => { + const keyValue = (key: ApiKey): string | undefined => key.api_key ?? undefined; + const anonKey = keys.find((key) => key.name === "anon"); + const serviceRoleKey = + keys.find((key) => key.name === "service_role") ?? + keys.find((key) => key.api_key?.startsWith("sb_secret_")); + const anon = anonKey === undefined ? undefined : keyValue(anonKey); + const service = serviceRoleKey === undefined ? undefined : keyValue(serviceRoleKey); + return anon === undefined || service === undefined + ? Effect.fail( + new LiveTransientPoll({ phase: "project API keys", cause: "keys incomplete" }), + ) + : Effect.succeed({ anonKey: anon, serviceRoleKey: service }); + }), + ); } -async function waitForProject(ref: string): Promise { - const deadline = Date.now() + 300_000; - while (Date.now() < deadline) { - const api = await runLiveEffect(makeLiveApi()); - const project = await runLiveEffect(api.v1.getProject({ ref })); - if (project.status === "ACTIVE_HEALTHY") return project; - if (TERMINAL_BAD_STATUSES.has(project.status)) { - throw new Error(`Project ${ref} entered terminal status ${project.status}`); - } - await Effect.runPromise(Effect.sleep("5 seconds")); - } - throw new Error(`Project ${ref} did not become ACTIVE_HEALTHY within 300000ms`); +function resolveKeys( + api: LiveApi, + ref: string, +): Effect.Effect<{ anonKey: string; serviceRoleKey: string }, Error, never> { + return retryLiveEffect("project API keys", keysReadiness(api, ref), { + shouldRetry: (error) => error instanceof LiveTransientPoll, + }).pipe( + Effect.mapError((error) => + error instanceof LivePollTimeout + ? new Error(`Project ${ref} did not return API keys within ${POLL_TIMEOUT}`) + : apiError(error), + ), + ); } -async function resolveKeys(ref: string): Promise<{ anonKey: string; serviceRoleKey: string }> { - const api = await runLiveEffect(makeLiveApi()); - const keys = await runLiveEffect(api.v1.getProjectApiKeys({ ref, reveal: true })); - const keyValue = (key: ApiKey): string | undefined => - key.api_key === null || key.api_key === undefined ? undefined : key.api_key; - const anon = keys.find((key) => key.name === "anon"); - const serviceRole = - keys.find((key) => key.name === "service_role") ?? - keys.find((key) => key.api_key?.startsWith("sb_secret_")); - const anonKey = anon === undefined ? undefined : keyValue(anon); - const serviceRoleKey = serviceRole === undefined ? undefined : keyValue(serviceRole); - if (anonKey === undefined || serviceRoleKey === undefined) { - throw new Error(`Project ${ref} returned no anon and service-role API keys`); - } - return { anonKey, serviceRoleKey }; +function dbReadiness( + api: LiveApi, + ref: string, + password: string, +): Effect.Effect { + return api.v1.getProjectPgbouncerConfig({ ref }).pipe( + Effect.mapError((cause) => new LiveTransientPoll({ phase: "pooler configuration", cause })), + Effect.flatMap((config) => { + if (config.connection_string === undefined) { + return Effect.fail( + new LiveTransientPoll({ + phase: "pooler configuration", + cause: "connection string missing", + }), + ); + } + try { + const url = new URL(config.connection_string); + url.password = password; + url.port = "5432"; + if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); + return Effect.succeed(url.toString()); + } catch (cause) { + return Effect.fail(new LiveTransientPoll({ phase: "pooler configuration", cause })); + } + }), + ); } -async function resolveDbUrl(ref: string, password: string): Promise { - const api = await runLiveEffect(makeLiveApi()); - const config = await runLiveEffect(api.v1.getProjectPgbouncerConfig({ ref })); - if (config.connection_string === undefined) { - throw new Error(`Project ${ref} returned no pooler connection string`); - } - const url = new URL(config.connection_string); - url.password = password; - url.port = "5432"; - if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); - return url.toString(); +function resolveDbUrl( + api: LiveApi, + ref: string, + password: string, +): Effect.Effect { + return retryLiveEffect("pooler configuration", dbReadiness(api, ref, password), { + shouldRetry: (error) => error instanceof LiveTransientPoll, + }).pipe( + Effect.mapError((error) => + error instanceof LivePollTimeout + ? new Error( + `Project ${ref} did not return a pooler connection string within ${POLL_TIMEOUT}`, + ) + : apiError(error), + ), + ); } -async function createStorageBucket( +function createStorageBucket( ref: string, host: string, serviceRoleKey: string, bucket: string, -): Promise { - const response = await fetch(`https://${ref}.${host}/storage/v1/bucket`, { - method: "POST", - headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, - body: JSON.stringify({ id: bucket, name: bucket, public: false }), - }); - if (!response.ok && response.status !== 409) { - throw new Error( - `Failed to create storage bucket ${bucket}: ${response.status} ${await response.text()}`, - ); - } +): Effect.Effect { + return Effect.tryPromise({ + try: async () => { + const response = await fetch(`https://${ref}.${host}/storage/v1/bucket`, { + method: "POST", + headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, + body: JSON.stringify({ id: bucket, name: bucket, public: false }), + }); + if (!response.ok && response.status !== 409) { + throw new LiveStorageError({ + message: `Failed to create storage bucket ${bucket}: ${response.status} ${await response.text()}`, + }); + } + }, + catch: (cause) => (cause instanceof LiveStorageError ? cause : new Error(String(cause))), + }).pipe( + Effect.timeoutOrElse({ + duration: POLL_TIMEOUT, + orElse: () => Effect.fail(new Error(`storage bucket ${bucket} creation timed out`)), + }), + ); } -async function writeProfile( +function writeProfile( projectRef: string, projectHost: string, dbUrl: string, -): Promise { - const directory = await mkdtemp(path.join(tmpdir(), "supabase-live-profile-")); - const profilePath = path.join(directory, "profile.yaml"); - const poolerHost = new URL(dbUrl).hostname; - try { - await writeFile( - profilePath, - [ - `name: ${PROFILE_NAME}`, - `api_url: ${JSON.stringify(liveApiUrl())}`, - `dashboard_url: ${JSON.stringify(liveApiUrl())}`, - `project_host: ${projectHost}`, - `pooler_host: ${poolerHost}`, - `# provisioned project: ${projectRef}`, - "", - ].join("\n"), - ); - } catch (error) { - await rm(directory, { recursive: true, force: true }); - throw error; - } - return profilePath; +): Effect.Effect { + return Effect.tryPromise({ + try: async () => { + const directory = await mkdtemp(path.join(tmpdir(), "supabase-live-profile-")); + const profilePath = path.join(directory, "profile.yaml"); + try { + const poolerHost = new URL(dbUrl).hostname; + await writeFile( + profilePath, + [ + `name: ${PROFILE_NAME}`, + `api_url: ${JSON.stringify(liveApiUrl())}`, + `dashboard_url: ${JSON.stringify(liveApiUrl())}`, + `project_host: ${projectHost}`, + `pooler_host: ${poolerHost}`, + `# provisioned project: ${projectRef}`, + "", + ].join("\n"), + ); + return profilePath; + } catch (cause) { + try { + await rm(directory, { recursive: true, force: true }); + } catch (cleanup) { + throw cleanupErrors(cause, [cleanup]); + } + throw cause; + } + }, + catch: apiError, + }); +} + +function cleanupDirectory(profilePath: string): Effect.Effect { + return Effect.tryPromise({ + try: () => rm(path.dirname(profilePath), { recursive: true, force: true }), + catch: apiError, + }); +} + +function cleanupRemote( + api: LiveApi, + environment: LiveProjectEnvironment, +): Effect.Effect { + return keepLiveProject() + ? Effect.sync(() => { + console.log(`SUPABASE_LIVE_KEEP_PROJECT=1 — leaving ${environment.project.ref} alive`); + }) + : deleteProject(api, environment.project.ref); +} + +function cleanupCreatedProject(api: LiveApi, ref: string): Effect.Effect { + return keepLiveProject() + ? Effect.sync(() => { + console.log( + `SUPABASE_LIVE_KEEP_PROJECT=1 — leaving ${ref} alive after provisioning failure`, + ); + }) + : deleteProject(api, ref); +} + +function combineCleanupExits( + exits: ReadonlyArray>, +): Effect.Effect { + const errors = exits.flatMap((exit) => (Exit.isFailure(exit) ? [Cause.squash(exit.cause)] : [])); + return errors.length === 0 + ? Effect.void + : Effect.fail(new AggregateError(errors, "Live cleanup failed")); } export interface LiveProjectEnvironment { @@ -239,46 +441,58 @@ export interface LiveProjectEnvironment { readonly profilePath: string; } -export async function provisionLiveEnvironment(): Promise { - const password = databasePassword(); - const ref = await createProject(uniqueProjectName(), password); - let profilePath: string | undefined; - try { - const project = await waitForProject(ref); - const projectHost = deriveLiveProjectHost(project.database.host, ref); - const keys = await resolveKeys(ref); - const dbUrl = await resolveDbUrl(ref, password); - const storageBucket = "supabase-cli-live-bucket"; - await createStorageBucket(ref, projectHost, keys.serviceRoleKey, storageBucket); - profilePath = await writeProfile(ref, projectHost, dbUrl); - return { - project: { - ref, - dbUrl, - dbPassword: password, - anonKey: keys.anonKey, - serviceRoleKey: keys.serviceRoleKey, - functionsUrl: `https://${ref}.${projectHost}/functions/v1`, - storageBucket, - }, - profilePath, - }; - } catch (error) { - if (profilePath !== undefined) - await rm(path.dirname(profilePath), { recursive: true, force: true }); - if (!keepLiveProject()) await deleteProject(ref).catch(() => undefined); - throw apiError(error); - } +/** Provision one project; the caller owns the outer Effect runtime boundary. */ +export function provisionLiveEnvironment( + api: LiveApi, +): Effect.Effect { + return Effect.gen(function* () { + const password = databasePassword(); + const ref = yield* createProject(api, uniqueProjectName(), password); + const setup = Effect.gen(function* () { + const project = yield* waitForProject(api, ref); + const projectHost = deriveLiveProjectHost(project.database.host, ref); + const keys = yield* resolveKeys(api, ref); + const dbUrl = yield* resolveDbUrl(api, ref, password); + const storageBucket = "supabase-cli-live-bucket"; + yield* createStorageBucket(ref, projectHost, keys.serviceRoleKey, storageBucket); + const profilePath = yield* writeProfile(ref, projectHost, dbUrl); + return { + project: { + ref, + dbUrl, + dbPassword: password, + anonKey: keys.anonKey, + serviceRoleKey: keys.serviceRoleKey, + functionsUrl: `https://${ref}.${projectHost}/functions/v1`, + storageBucket, + }, + profilePath, + } satisfies LiveProjectEnvironment; + }); + const setupExit = yield* Effect.exit(setup); + if (Exit.isSuccess(setupExit)) return setupExit.value; + + const cleanupExit = yield* Effect.exit(cleanupCreatedProject(api, ref)); + if (Exit.isSuccess(cleanupExit)) return yield* Effect.failCause(setupExit.cause); + return yield* Effect.fail( + cleanupErrors(Cause.squash(setupExit.cause), [Cause.squash(cleanupExit.cause)]), + ); + }); } -export async function cleanupLiveEnvironment(environment: LiveProjectEnvironment): Promise { - let profileError: unknown; - try { - await rm(path.dirname(environment.profilePath), { recursive: true, force: true }); - } catch (error) { - profileError = error; - } - if (!keepLiveProject()) await deleteProject(environment.project.ref); - else console.log(`SUPABASE_LIVE_KEEP_PROJECT=1 — leaving ${environment.project.ref} alive`); - if (profileError !== undefined) throw apiError(profileError); +/** Delete the exact owned project and always remove its temporary profile. */ +export function cleanupLiveEnvironment( + api: LiveApi, + environment: LiveProjectEnvironment, +): Effect.Effect { + return Effect.gen(function* () { + const [profileExit, projectExit] = yield* Effect.all( + [ + Effect.exit(cleanupDirectory(environment.profilePath)), + Effect.exit(cleanupRemote(api, environment)), + ], + { concurrency: "unbounded" }, + ); + yield* combineCleanupExits([profileExit, projectExit]); + }); } diff --git a/apps/cli/tests/helpers/live-project.unit.test.ts b/apps/cli/tests/helpers/live-project.unit.test.ts new file mode 100644 index 0000000000..e91032220e --- /dev/null +++ b/apps/cli/tests/helpers/live-project.unit.test.ts @@ -0,0 +1,53 @@ +import { Effect } from "effect"; +import { describe, expect, it } from "vitest"; + +import { cleanupErrors, retryLiveEffect } from "./live-project.ts"; + +describe("live project lifecycle", () => { + it("retries transient failures until the management operation succeeds", async () => { + let attempts = 0; + const result = await Effect.runPromise( + retryLiveEffect( + "project readiness", + Effect.suspend(() => + Effect.sync(() => { + attempts += 1; + return attempts < 3 + ? Effect.fail(new Error("temporarily unavailable")) + : Effect.succeed("ACTIVE_HEALTHY"); + }).pipe(Effect.flatten), + ), + { interval: "1 millis", timeout: "100 millis" }, + ), + ); + + expect(result).toBe("ACTIVE_HEALTHY"); + expect(attempts).toBe(3); + }); + + it("fails a poll when its wall-clock deadline expires", async () => { + const result = Effect.runPromise( + retryLiveEffect("project keys", Effect.never, { + interval: "1 millis", + timeout: "10 millis", + }), + ); + + await expect(result).rejects.toThrow("project keys timed out"); + }); + + it("preserves both target and cleanup failures", () => { + const error = cleanupErrors(new Error("provision failed"), [ + new Error("profile cleanup failed"), + new Error("project deletion failed"), + ]); + + expect(error).toBeInstanceOf(AggregateError); + expect(error.errors).toHaveLength(3); + expect(error.errors.map((entry) => String(entry))).toEqual([ + "Error: provision failed", + "Error: profile cleanup failed", + "Error: project deletion failed", + ]); + }); +}); diff --git a/apps/cli/tests/helpers/live.ts b/apps/cli/tests/helpers/live.ts index 0b4a1c9948..b365a40365 100644 --- a/apps/cli/tests/helpers/live.ts +++ b/apps/cli/tests/helpers/live.ts @@ -24,17 +24,9 @@ export interface InvokeResult { export interface LiveFixtures { readonly project: LiveProject; - /** Compatibility aliases while command suites migrate to `project`/`cli`. */ - readonly projectRef: string; - readonly dbUrl: string; - readonly dbPassword: string; - readonly anonKey: string; - readonly functionsUrl: string; - readonly storageBucket: string; readonly workspace: LiveWorkspace; readonly home: ReturnType; readonly cli: (args: string[], options?: RunOptions) => Promise; - readonly run: (args: string[], options?: RunOptions) => Promise; readonly invoke: ( slug: string, options?: { readonly anonKey?: string; readonly payload?: unknown }, @@ -45,13 +37,6 @@ const base = vitestTest.extend({ // eslint-disable-next-line no-empty-pattern project: async ({}, use) => use(inject("liveProject")), - projectRef: async ({ project }, use) => use(project.ref), - dbUrl: async ({ project }, use) => use(project.dbUrl), - dbPassword: async ({ project }, use) => use(project.dbPassword), - anonKey: async ({ project }, use) => use(project.anonKey), - functionsUrl: async ({ project }, use) => use(project.functionsUrl), - storageBucket: async ({ project }, use) => use(project.storageBucket), - home: async ({ task: _task }, use) => { const home = makeTempHome(); try { @@ -98,8 +83,6 @@ const base = vitestTest.extend({ ); }, - run: async ({ cli }, use) => use(cli), - invoke: async ({ project }, use) => { await use(async (slug, options) => { const key = options?.anonKey ?? project.anonKey; @@ -139,10 +122,6 @@ export function requireLiveSuccess( } } -export function liveDatabaseTargetArgs(dbUrl: string, _projectRef?: string): string[] { - return ["--db-url", dbUrl]; -} - export function expectFunctionOk( result: InvokeResult, slug: string, diff --git a/apps/cli/tests/live-global-setup.ts b/apps/cli/tests/live-global-setup.ts index 71279a6a07..40b4d31c57 100644 --- a/apps/cli/tests/live-global-setup.ts +++ b/apps/cli/tests/live-global-setup.ts @@ -1,8 +1,12 @@ import type { ProvidedContext } from "vitest"; +import { makeApiClient } from "@supabase/api/effect"; +import { Effect } from "effect"; +import { FetchHttpClient } from "effect/unstable/http"; + import "./helpers/live-provided-context.ts"; import { cleanupLiveEnvironment, provisionLiveEnvironment } from "./helpers/live-project.ts"; -import { validateLiveConfig } from "./helpers/live-env.ts"; +import { liveAccessToken, liveApiUrl, validateLiveConfig } from "./helpers/live-env.ts"; type LiveSetupContext = { provide: (key: K, value: ProvidedContext[K]) => void; @@ -11,10 +15,18 @@ type LiveSetupContext = { /** Provision one disposable project for the entire serial live Vitest run. */ export async function setup({ provide }: LiveSetupContext): Promise<() => Promise> { validateLiveConfig(); - const environment = await provisionLiveEnvironment(); + const { api, environment } = await Effect.runPromise( + Effect.gen(function* () { + const api = yield* makeApiClient({ baseUrl: liveApiUrl(), accessToken: liveAccessToken() }); + const environment = yield* provisionLiveEnvironment(api); + return { api, environment }; + }).pipe(Effect.provide(FetchHttpClient.layer)), + ); provide("liveProject", environment.project); provide("liveProfilePath", environment.profilePath); - return async () => cleanupLiveEnvironment(environment); + return async () => { + await Effect.runPromise(cleanupLiveEnvironment(api, environment)); + }; } export default setup; From b2a3946d6693b2db52f79b102f6665c4bb37dc8a Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 22:34:20 +0200 Subject: [PATCH 12/23] docs(cli): refresh live e2e contributor guidance --- CONTRIBUTING.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c63d7d1671..c02e794bd3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -191,21 +191,21 @@ The replay/record harness has two modes: ### Live remote-project coverage -The live suite lives in `apps/cli/src/**` as collocated `*.live.test.ts` files and runs in the CLI package's `live` Vitest project. Global setup provisions one shared environment per run. Attached mode (the default) preserves the local Supabox contract: `SUPABASE_PROFILE` selects the platform, `SUPABASE_LIVE_API_URL` is used for readiness, and `SUPABASE_LIVE_PROJECT_REF` identifies the existing project; attached teardown never deletes it. Managed mode is explicit (`SUPABASE_LIVE_MODE=managed`) and provisions one uniquely named staging project, shares it across all live tests, then deletes exactly that project during teardown unless `SUPABASE_LIVE_KEEP_PROJECT=1` is set. +The live suite lives in `apps/cli/src/**` as collocated `*.live.test.ts` files and runs in the CLI package's separate, serial `live` Vitest project. Global setup requires `SUPABASE_LIVE_API_URL` and `SUPABASE_ACCESS_TOKEN`, then provisions one uniquely named project through the typed Management API client, waits for it to become healthy, creates the shared storage fixture, and writes a temporary YAML profile. Every live subprocess receives that profile, so the same contract works with Supabox, a Docker-hosted API platform, or staging by changing only the URL and token. Teardown always removes the temporary profile and deletes the exact owned project unless `SUPABASE_LIVE_KEEP_PROJECT=1` is set. Live coverage is smoke coverage, not an exhaustive command matrix. Add one representative golden-path test for each user-facing command, colocated beside that command. A live test should assert one target command; setup and teardown may invoke other commands when they prepare or clean up state, but those commands are not asserted in that test. Keep validation, formatting, fallback, error, and matrix details in integration tests unless the remote/runtime boundary itself is the behavior under test. See [ADR 0013](docs/adr/0013-live-e2e-bypasses-replay-server.md) and [`apps/cli/live.env.example`](apps/cli/live.env.example). -To run the live suite locally against an attached Supabox/local platform, copy [`apps/cli/live.env.example`](apps/cli/live.env.example), fill in the platform values, and run: +To run the live suite locally, copy [`apps/cli/live.env.example`](apps/cli/live.env.example), set the API URL and access token for the target platform, and run: ```sh cd apps/cli pnpm test:live ``` -For an explicit managed staging run, set `SUPABASE_LIVE_MODE=managed`, -`SUPABASE_PROFILE=supabase-staging`, `SUPABASE_LIVE_API_URL=https://api.supabase.green`, -and `SUPABASE_ACCESS_TOKEN` before invoking the same command. The suite's global setup -handles project provisioning and teardown. +Optional `SUPABASE_LIVE_ORG_ID`, `SUPABASE_LIVE_REGION`, and +`SUPABASE_LIVE_PROJECT_NAME` values select provisioning details. Set +`SUPABASE_LIVE_KEEP_PROJECT=1` only when debugging a failed run; the temporary +profile is still cleaned up. Live CI is manual or daily scheduled and is not PR-blocking; run it manually on a PR branch when you need pre-merge remote coverage. From eac71776089ee2ae1253c26f363f307452b56d2b Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 22:38:54 +0200 Subject: [PATCH 13/23] test(cli): honor live data plane protocol --- apps/cli/tests/helpers/live-project.ts | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/apps/cli/tests/helpers/live-project.ts b/apps/cli/tests/helpers/live-project.ts index 1686dd63e3..f8477ae23b 100644 --- a/apps/cli/tests/helpers/live-project.ts +++ b/apps/cli/tests/helpers/live-project.ts @@ -143,6 +143,11 @@ function databasePassword(): string { return `supabase-cli-live-${randomBytes(12).toString("hex")}`; } +function liveProjectBaseUrl(ref: string, host: string): string { + const protocol = new URL(liveApiUrl()).protocol; + return `${protocol}//${ref}.${host}`; +} + function resolveOrganization(api: LiveApi): Effect.Effect { return timeoutLiveRequest("organization lookup", api.v1.listAllOrganizations()).pipe( Effect.mapError(apiError), @@ -334,7 +339,7 @@ function createStorageBucket( ): Effect.Effect { return Effect.tryPromise({ try: async () => { - const response = await fetch(`https://${ref}.${host}/storage/v1/bucket`, { + const response = await fetch(`${liveProjectBaseUrl(ref, host)}/storage/v1/bucket`, { method: "POST", headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, body: JSON.stringify({ id: bucket, name: bucket, public: false }), @@ -463,7 +468,7 @@ export function provisionLiveEnvironment( dbPassword: password, anonKey: keys.anonKey, serviceRoleKey: keys.serviceRoleKey, - functionsUrl: `https://${ref}.${projectHost}/functions/v1`, + functionsUrl: `${liveProjectBaseUrl(ref, projectHost)}/functions/v1`, storageBucket, }, profilePath, From eab3f7910e75dff7d800a36d2f81fa35ed0694eb Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 23:01:31 +0200 Subject: [PATCH 14/23] test(cli): harden live e2e lifecycle --- CONTRIBUTING.md | 4 + apps/cli/AGENTS.md | 3 + apps/cli/live.env.example | 4 +- .../branches/create/create.live.test.ts | 37 +++++-- .../branches/delete/delete.live.test.ts | 5 +- .../commands/branches/list/list.live.test.ts | 22 +++- .../legacy/commands/db/pull/pull.live.test.ts | 9 +- .../legacy/commands/db/push/push.live.test.ts | 17 ++- .../commands/db/reset/reset.live.test.ts | 17 ++- .../functions/delete/delete.live.test.ts | 15 ++- .../functions/deploy/deploy.live.test.ts | 20 +++- .../migration/fetch/fetch.live.test.ts | 21 ++-- .../commands/secrets/list/list.live.test.ts | 13 ++- .../commands/secrets/set/set.live.test.ts | 21 +++- .../commands/secrets/unset/unset.live.test.ts | 5 +- .../commands/storage/cp/cp.live.test.ts | 15 ++- .../commands/storage/ls/ls.live.test.ts | 13 ++- .../commands/storage/rm/rm.live.test.ts | 15 ++- .../functions/dev/dev.docker.e2e.test.ts | 47 ++++++-- .../functions/dev/functions-dev-runtime.ts | 5 + apps/cli/tests/helpers/live-env.ts | 7 -- apps/cli/tests/helpers/live-project.ts | 100 +++++++++++++----- .../tests/helpers/live-project.unit.test.ts | 68 +++++++++++- apps/cli/tests/helpers/live.ts | 15 +++ .../0013-live-e2e-bypasses-replay-server.md | 6 ++ 25 files changed, 401 insertions(+), 103 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c02e794bd3..17d6bed3b1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -193,6 +193,10 @@ The replay/record harness has two modes: The live suite lives in `apps/cli/src/**` as collocated `*.live.test.ts` files and runs in the CLI package's separate, serial `live` Vitest project. Global setup requires `SUPABASE_LIVE_API_URL` and `SUPABASE_ACCESS_TOKEN`, then provisions one uniquely named project through the typed Management API client, waits for it to become healthy, creates the shared storage fixture, and writes a temporary YAML profile. Every live subprocess receives that profile, so the same contract works with Supabox, a Docker-hosted API platform, or staging by changing only the URL and token. Teardown always removes the temporary profile and deletes the exact owned project unless `SUPABASE_LIVE_KEEP_PROJECT=1` is set. +The configured URL is the Management API endpoint. Tenant data-plane URLs keep +the CLI profile contract (`https://.`) using the host derived +from the provisioned project's database metadata. + Live coverage is smoke coverage, not an exhaustive command matrix. Add one representative golden-path test for each user-facing command, colocated beside that command. A live test should assert one target command; setup and teardown may invoke other commands when they prepare or clean up state, but those commands are not asserted in that test. Keep validation, formatting, fallback, error, and matrix details in integration tests unless the remote/runtime boundary itself is the behavior under test. See [ADR 0013](docs/adr/0013-live-e2e-bypasses-replay-server.md) and [`apps/cli/live.env.example`](apps/cli/live.env.example). To run the live suite locally, copy [`apps/cli/live.env.example`](apps/cli/live.env.example), set the API URL and access token for the target platform, and run: diff --git a/apps/cli/AGENTS.md b/apps/cli/AGENTS.md index bba3413c46..a48cd0a384 100644 --- a/apps/cli/AGENTS.md +++ b/apps/cli/AGENTS.md @@ -513,6 +513,9 @@ YAML profile, and shares it across the serial suite. Teardown deletes exactly that project and the temporary profile. Supabox, a Docker-hosted API platform, and staging are interchangeable; changing the URL and token retargets the run. `SUPABASE_LIVE_KEEP_PROJECT=1` keeps the project for debugging. +`SUPABASE_LIVE_API_URL` configures the Management API only; tenant data-plane +URLs retain the profile contract `https://.`, with +`project_host` derived from the provisioned project's database host. Local Docker-stack lifecycle tests (`start`, `stop`, `status`, `db start`, `db diff`, declarative sync, and `functions dev`) are `*.e2e.test.ts`, use diff --git a/apps/cli/live.env.example b/apps/cli/live.env.example index b859ed76f0..a0ccbe76c9 100644 --- a/apps/cli/live.env.example +++ b/apps/cli/live.env.example @@ -1,6 +1,8 @@ # Live CLI e2e environment. The suite provisions one disposable project # against the configured Management API URL. Supabox, Docker-hosted API, and -# staging use the same contract; only this URL and token change. +# staging use the same contract; only this URL and token change. Tenant data +# plane URLs remain https://., derived from the project DB +# host returned by the Management API. SUPABASE_LIVE_API_URL=http://localhost:8080 SUPABASE_ACCESS_TOKEN=sbp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx diff --git a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts index a82659e30b..2b9c4baca8 100644 --- a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts @@ -1,15 +1,30 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { test } from "../../../../../tests/helpers/live.ts"; +import { test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; -async function deleteBranch( +async function cleanupBranch( cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, name: string, ref: string, ): Promise { + const listed = await cli(["branches", "list", "--output", "json", "--project-ref", ref]); + if (listed.exitCode !== 0) { + throw new Error( + `branches list cleanup failed (exit ${listed.exitCode})\n${listed.stdout}\n${listed.stderr}`, + ); + } + const branches: unknown = JSON.parse(listed.stdout); + const exists = + Array.isArray(branches) && + branches.some( + (branch) => + typeof branch === "object" && branch !== null && "name" in branch && branch.name === name, + ); + if (!exists) return; + const deleted = await cli(["branches", "delete", name, "--project-ref", ref, "--yes"]); - if (deleted.exitCode !== 0) { + if (deleted.exitCode !== 0 && !/not found/i.test(`${deleted.stdout}\n${deleted.stderr}`)) { throw new Error( `branches delete cleanup failed (exit ${deleted.exitCode})\n${deleted.stdout}\n${deleted.stderr}`, ); @@ -18,12 +33,20 @@ async function deleteBranch( test("creates a preview branch", async ({ cli, project }) => { const name = `cli-e2e-create-${randomUUID().slice(0, 8)}`; - const result = await cli(["branches", "create", name, "--project-ref", project.ref]); - expect(result.exitCode, result.stderr).toBe(0); - + let targetError: unknown; + let cleanupError: unknown; try { + const result = await cli(["branches", "create", name, "--project-ref", project.ref]); + expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Created preview branch"); + } catch (error) { + targetError = error; } finally { - await deleteBranch(cli, name, project.ref); + try { + await cleanupBranch(cli, name, project.ref); + } catch (error) { + cleanupError = error; + } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts index 235e8c1473..2de79561cb 100644 --- a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; test("deletes a preview branch", async ({ cli, project }) => { const name = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; @@ -39,6 +39,5 @@ test("deletes a preview branch", async ({ cli, project }) => { } } } - if (targetError !== undefined) throw targetError; - if (cleanupError !== undefined) throw cleanupError; + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts index 27561ae802..555507c10d 100644 --- a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts @@ -1,13 +1,15 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; test("lists a preview branch for the project", async ({ cli, project }) => { const name = `cli-e2e-list-${randomUUID().slice(0, 8)}`; const created = await cli(["branches", "create", name, "--project-ref", project.ref]); requireLiveSuccess(created, "branches create setup"); + let targetError: unknown; + let cleanupError: unknown; try { const result = await cli([ "branches", @@ -20,8 +22,22 @@ test("lists a preview branch for the project", async ({ cli, project }) => { expect(result.exitCode, result.stderr).toBe(0); const branches = JSON.parse(result.stdout) as Array<{ name?: string }>; expect(branches.map((branch) => branch.name)).toContain(name); + } catch (error) { + targetError = error; } finally { - const deleted = await cli(["branches", "delete", name, "--project-ref", project.ref, "--yes"]); - requireLiveSuccess(deleted, "branches delete cleanup"); + try { + const deleted = await cli([ + "branches", + "delete", + name, + "--project-ref", + project.ref, + "--yes", + ]); + requireLiveSuccess(deleted, "branches delete cleanup"); + } catch (error) { + cleanupError = error; + } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts index 360a286c62..526c543c35 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts @@ -2,7 +2,7 @@ import { mkdir, readdir, unlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; test("pulls the remote schema after a local migration is applied", async ({ cli, @@ -59,10 +59,5 @@ test("pulls the remote schema after a local migration is applied", async ({ cleanupError = error; } - if (targetError !== undefined) { - throw targetError; - } - if (cleanupError !== undefined) { - throw cleanupError; - } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/db/push/push.live.test.ts b/apps/cli/src/legacy/commands/db/push/push.live.test.ts index 31dce484d2..d4d4ed890f 100644 --- a/apps/cli/src/legacy/commands/db/push/push.live.test.ts +++ b/apps/cli/src/legacy/commands/db/push/push.live.test.ts @@ -2,7 +2,7 @@ import { mkdir, unlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; test("pushes a local migration to the remote database", async ({ cli, project, workspace }) => { const version = `${Date.now()}${Math.floor(Math.random() * 10_000) @@ -13,13 +13,22 @@ test("pushes a local migration to the remote database", async ({ cli, project, w const migrationFile = join(migrations, `${version}_e2e_push.sql`); await writeFile(migrationFile, `create table if not exists e2e_push_${version} (id int);\n`); + let targetError: unknown; + const cleanupErrors: Array = []; try { const result = await cli(["db", "push", "--db-url", project.dbUrl, "--yes"]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Finished supabase db push"); + } catch (error) { + targetError = error; } finally { - await unlink(migrationFile).catch(() => undefined); - const reset = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); - requireLiveSuccess(reset, "db reset cleanup after db push"); + await unlink(migrationFile).catch((error) => cleanupErrors.push(error)); + try { + const reset = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); + requireLiveSuccess(reset, "db reset cleanup after db push"); + } catch (error) { + cleanupErrors.push(error); + } } + throwWithCleanup(targetError, cleanupErrors); }); diff --git a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts index 3a03886206..738446318c 100644 --- a/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts +++ b/apps/cli/src/legacy/commands/db/reset/reset.live.test.ts @@ -2,7 +2,7 @@ import { mkdir, unlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; test("resets the remote database with local migrations", async ({ cli, project, workspace }) => { const version = `${Date.now()}${Math.floor(Math.random() * 10_000) @@ -13,13 +13,22 @@ test("resets the remote database with local migrations", async ({ cli, project, const migrationFile = join(migrations, `${version}_e2e_reset.sql`); await writeFile(migrationFile, `create table if not exists e2e_reset_${version} (id int);\n`); + let targetError: unknown; + const cleanupErrors: Array = []; try { const result = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); expect(result.exitCode, result.stderr).toBe(0); expect(`${result.stdout}${result.stderr}`).toContain("Resetting remote database"); + } catch (error) { + targetError = error; } finally { - await unlink(migrationFile).catch(() => undefined); - const reset = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); - requireLiveSuccess(reset, "db reset cleanup"); + await unlink(migrationFile).catch((error) => cleanupErrors.push(error)); + try { + const reset = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); + requireLiveSuccess(reset, "db reset cleanup"); + } catch (error) { + cleanupErrors.push(error); + } } + throwWithCleanup(targetError, cleanupErrors); }); diff --git a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts index b0ee8a692a..44e659cc3d 100644 --- a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts @@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto"; import { mkdir, writeFile } from "node:fs/promises"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; test("deletes a deployed function", async ({ cli, project, workspace }) => { const slug = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; @@ -22,15 +22,24 @@ test("deletes a deployed function", async ({ cli, project, workspace }) => { requireLiveSuccess(deployed, "functions deploy setup"); let deleted = false; + let targetError: unknown; + let cleanupError: unknown; try { const result = await cli(["functions", "delete", slug, "--project-ref", project.ref]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Deleted Function"); deleted = true; + } catch (error) { + targetError = error; } finally { if (!deleted) { - const cleanup = await cli(["functions", "delete", slug, "--project-ref", project.ref]); - requireLiveSuccess(cleanup, "functions delete cleanup"); + try { + const cleanup = await cli(["functions", "delete", slug, "--project-ref", project.ref]); + requireLiveSuccess(cleanup, "functions delete cleanup"); + } catch (error) { + cleanupError = error; + } } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts index 4c447a0b69..cdcf4f7853 100644 --- a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts @@ -4,7 +4,12 @@ import { join } from "node:path"; import { expect } from "vitest"; import { describe } from "vitest"; -import { expectFunctionOk, requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { + expectFunctionOk, + requireLiveSuccess, + test, + throwWithCleanup, +} from "../../../../../tests/helpers/live.ts"; describe("functions deploy (live)", () => { test("deploys a function that responds over HTTP", async ({ @@ -23,6 +28,8 @@ describe("functions deploy (live)", () => { await writeFile(join(directory, "deno.json"), '{\n "imports": {}\n}\n'); let deployed = false; + let targetError: unknown; + let cleanupError: unknown; try { const result = await cli(["functions", "deploy", "--project-ref", project.ref]); expect(result.exitCode, result.stderr).toBe(0); @@ -30,11 +37,18 @@ describe("functions deploy (live)", () => { expect(result.stdout).toMatch(/Deployed Function/i); expectFunctionOk(await invoke(slug), slug); + } catch (error) { + targetError = error; } finally { if (deployed) { - const deleted = await cli(["functions", "delete", slug, "--project-ref", project.ref]); - requireLiveSuccess(deleted, "functions delete cleanup"); + try { + const deleted = await cli(["functions", "delete", slug, "--project-ref", project.ref]); + requireLiveSuccess(deleted, "functions delete cleanup"); + } catch (error) { + cleanupError = error; + } } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); }); diff --git a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts index 09b530c29d..822bacbc14 100644 --- a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts @@ -3,7 +3,7 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; const LIVE_TIMEOUT_MS = 120_000; @@ -41,7 +41,7 @@ test( const fetchDir = await mkdtemp(path.join(tmpdir(), "sb-migration-fetch-live-")); let repaired = false; let targetError: unknown; - let cleanupError: unknown; + const cleanupErrors: Array = []; try { // Seed: record one migration in the remote history. `repair --status applied` // reads the local file for the version's name/statements, so write it first. @@ -76,17 +76,16 @@ test( ); requireLiveSuccess(reverted, "migration repair cleanup"); } catch (error) { - cleanupError = error; + cleanupErrors.push(error); } } - try { - await rm(seedDir, { recursive: true, force: true }); - await rm(fetchDir, { recursive: true, force: true }); - } catch (error) { - cleanupError ??= error; - } + await rm(seedDir, { recursive: true, force: true }).catch((error) => + cleanupErrors.push(error), + ); + await rm(fetchDir, { recursive: true, force: true }).catch((error) => + cleanupErrors.push(error), + ); } - if (targetError !== undefined) throw targetError; - if (cleanupError !== undefined) throw cleanupError; + throwWithCleanup(targetError, cleanupErrors); }, ); diff --git a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts index 391c472d13..a9c5930f52 100644 --- a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, @@ -19,6 +19,8 @@ test("lists a secret created on the remote project", async ({ cli, project }) => const created = await cli(["secrets", "set", `${name}=live-value`, "--project-ref", project.ref]); requireLiveSuccess(created, "secrets set setup"); + let targetError: unknown; + let cleanupError: unknown; try { const result = await cli(["secrets", "list", "--output", "json", "--project-ref", project.ref]); expect(result.exitCode, result.stderr).toBe(0); @@ -26,7 +28,14 @@ test("lists a secret created on the remote project", async ({ cli, project }) => (secret) => secret.name, ); expect(names).toContain(name); + } catch (error) { + targetError = error; } finally { - await unsetSecret(cli, name, project.ref); + try { + await unsetSecret(cli, name, project.ref); + } catch (error) { + cleanupError = error; + } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts index 8bffcf29f8..c7862bb93c 100644 --- a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { test } from "../../../../../tests/helpers/live.ts"; +import { test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, @@ -16,11 +16,26 @@ async function unsetSecret( test("sets a secret on the remote project", async ({ cli, project }) => { const name = `CLI_E2E_SET_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; - const result = await cli(["secrets", "set", `${name}=live-value`, "--project-ref", project.ref]); + let targetError: unknown; + let cleanupError: unknown; try { + const result = await cli([ + "secrets", + "set", + `${name}=live-value`, + "--project-ref", + project.ref, + ]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Finished"); + } catch (error) { + targetError = error; } finally { - await unsetSecret(cli, name, project.ref); + try { + await unsetSecret(cli, name, project.ref); + } catch (error) { + cleanupError = error; + } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts index 1bc427d11d..3818a9f91e 100644 --- a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; async function unsetSecret( cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, @@ -38,6 +38,5 @@ test("unsets a secret from the remote project", async ({ cli, project }) => { } } } - if (targetError !== undefined) throw targetError; - if (cleanupError !== undefined) throw cleanupError; + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts index 98dc95bdc8..2be147f616 100644 --- a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts @@ -3,7 +3,7 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; @@ -29,11 +29,22 @@ test("copies a local file to the remote bucket", async ({ cli, project, workspac requireLiveSuccess(linked, "link setup for storage cp"); let uploaded = false; + let targetError: unknown; + let cleanupError: unknown; try { const result = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); expect(result.exitCode, result.stderr).toBe(0); uploaded = true; + } catch (error) { + targetError = error; } finally { - if (uploaded) await removeObject(cli, remote); + if (uploaded) { + try { + await removeObject(cli, remote); + } catch (error) { + cleanupError = error; + } + } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts index a73463825e..d8b5f8504b 100644 --- a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts @@ -3,7 +3,7 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; @@ -30,6 +30,8 @@ test("lists an uploaded object", async ({ cli, project, workspace }) => { const uploaded = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); requireLiveSuccess(uploaded, "storage cp setup for storage ls"); + let targetError: unknown; + let cleanupError: unknown; try { const result = await cli([ "storage", @@ -39,7 +41,14 @@ test("lists an uploaded object", async ({ cli, project, workspace }) => { ]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain(`upload-${suffix}.txt`); + } catch (error) { + targetError = error; } finally { - await removeObject(cli, remote); + try { + await removeObject(cli, remote); + } catch (error) { + cleanupError = error; + } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts index 0cfb89ddaf..c2b550e58a 100644 --- a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts @@ -3,7 +3,7 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; -import { requireLiveSuccess, test } from "../../../../../tests/helpers/live.ts"; +import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; const STORAGE_FLAGS = ["--linked", "--experimental"]; @@ -31,11 +31,22 @@ test("removes an uploaded object", async ({ cli, project, workspace }) => { requireLiveSuccess(uploaded, "storage cp setup for storage rm"); let removed = false; + let targetError: unknown; + let cleanupError: unknown; try { const result = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); expect(result.exitCode, result.stderr).toBe(0); removed = true; + } catch (error) { + targetError = error; } finally { - if (!removed) await removeObject(cli, remote); + if (!removed) { + try { + await removeObject(cli, remote); + } catch (error) { + cleanupError = error; + } + } } + throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); }); diff --git a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts index 3bf4f4d774..7c69065346 100644 --- a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts +++ b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts @@ -14,6 +14,8 @@ const FUNCTIONS_DEV_STARTUP_TIMEOUT_MS = 60_000; const FUNCTIONS_DEV_STEP_TIMEOUT_MS = 30_000; const FUNCTIONS_DEV_TEST_TIMEOUT_MS = 90_000; const FUNCTION_FILES_RESTART_PATTERN = /Function files changed\. Restarting edge-runtime\./; +const FUNCTION_RELOAD_COMPLETE_PATTERN = /Function reload complete\./; +const EDGE_RUNTIME_RELOAD_COMPLETE_PATTERN = /Edge runtime reload complete\./; type SpawnedSupabase = ReturnType; @@ -74,13 +76,24 @@ describe("supabase functions dev (e2e)", () => { } const functionUrl = `${functionUrlMatch[1]}/hello-world`; + const functionOffset = devProc.stdout().length; + const functionRestart = devProc.waitForOutput( + FUNCTION_FILES_RESTART_PATTERN, + FUNCTIONS_DEV_STEP_TIMEOUT_MS, + functionOffset, + ); + const functionReload = devProc.waitForOutput( + FUNCTION_RELOAD_COMPLETE_PATTERN, + FUNCTIONS_DEV_STEP_TIMEOUT_MS, + functionOffset, + ); const newResult = await runSupabase(["functions", "new", "hello-world"], { cwd: project.dir, home: home.dir, exitTimeoutMs: FUNCTIONS_DEV_STEP_TIMEOUT_MS, }); expect(newResult.exitCode).toBe(0); - await devProc.waitForOutput(FUNCTION_FILES_RESTART_PATTERN, FUNCTIONS_DEV_STEP_TIMEOUT_MS); + await Promise.all([functionRestart, functionReload]); await assertFunctionResponse(functionUrl, {}, (response, body) => { expect(response.status).toBe(401); @@ -88,6 +101,16 @@ describe("supabase functions dev (e2e)", () => { }); const configOffset = devProc.stdout().length; + const configRestart = devProc.waitForOutput( + /Edge runtime config changed\. Restarting edge-runtime\./, + FUNCTIONS_DEV_STEP_TIMEOUT_MS, + configOffset, + ); + const configReload = devProc.waitForOutput( + EDGE_RUNTIME_RELOAD_COMPLETE_PATTERN, + FUNCTIONS_DEV_STEP_TIMEOUT_MS, + configOffset, + ); await writeFile( join(project.dir, "supabase", "config.toml"), `project_id = "functions-dev-e2e" @@ -96,11 +119,7 @@ describe("supabase functions dev (e2e)", () => { verify_jwt = false `, ); - await devProc.waitForOutput( - /Edge runtime config changed\. Restarting edge-runtime\./, - FUNCTIONS_DEV_STEP_TIMEOUT_MS, - configOffset, - ); + await Promise.all([configRestart, configReload]); await assertFunctionResponse( functionUrl, @@ -116,6 +135,16 @@ verify_jwt = false ); const sourceOffset = devProc.stdout().length; + const sourceRestart = devProc.waitForOutput( + FUNCTION_FILES_RESTART_PATTERN, + FUNCTIONS_DEV_STEP_TIMEOUT_MS, + sourceOffset, + ); + const sourceReload = devProc.waitForOutput( + FUNCTION_RELOAD_COMPLETE_PATTERN, + FUNCTIONS_DEV_STEP_TIMEOUT_MS, + sourceOffset, + ); await writeFile( functionPath, `Deno.serve(() => { @@ -125,11 +154,7 @@ verify_jwt = false }); `, ); - await devProc.waitForOutput( - FUNCTION_FILES_RESTART_PATTERN, - FUNCTIONS_DEV_STEP_TIMEOUT_MS, - sourceOffset, - ); + await Promise.all([sourceRestart, sourceReload]); await assertFunctionResponse( functionUrl, diff --git a/apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts b/apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts index 680c67417e..90ca964dc6 100644 --- a/apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts +++ b/apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts @@ -122,6 +122,9 @@ const ensureFunctionsDirectory = Effect.fnUntraced(function* () { yield* fs.makeDirectory(join(projectHome.supabaseDir, "functions"), { recursive: true }); }); +const EDGE_RUNTIME_RELOAD_COMPLETE = "Edge runtime reload complete."; +const FUNCTIONS_RELOAD_COMPLETE = "Function reload complete."; + function watchEventMatches(spec: FunctionsDevWatchPath, event: FileWatchEvent): boolean { if (spec.names === undefined) { return true; @@ -228,11 +231,13 @@ export const runFunctionsDevRuntime = Effect.fnUntraced(function* ( yield* output.info("Edge runtime config changed. Restarting edge-runtime..."); yield* reloadEdgeRuntime(stack, opts, result.state.config); edgeRuntimeState = result.state; + yield* output.info(EDGE_RUNTIME_RELOAD_COMPLETE); return; } edgeRuntimeState = result.state; yield* output.info("Function files changed. Restarting edge-runtime..."); yield* stack.reloadFunctions({ functions: yield* resolveFunctionsBundle(opts) }); + yield* output.info(FUNCTIONS_RELOAD_COMPLETE); }).pipe( Effect.catch((error) => output.error(error instanceof Error ? error.message : String(error)), diff --git a/apps/cli/tests/helpers/live-env.ts b/apps/cli/tests/helpers/live-env.ts index 3730741d42..92c2519b79 100644 --- a/apps/cli/tests/helpers/live-env.ts +++ b/apps/cli/tests/helpers/live-env.ts @@ -62,10 +62,3 @@ export function deriveLiveProjectHost(databaseHost: string, projectRef: string): } return host; } - -export function isLiveConfigured(): boolean { - return ( - (process.env["SUPABASE_LIVE_API_URL"]?.trim().length ?? 0) > 0 && - (process.env["SUPABASE_ACCESS_TOKEN"]?.trim().length ?? 0) > 0 - ); -} diff --git a/apps/cli/tests/helpers/live-project.ts b/apps/cli/tests/helpers/live-project.ts index f8477ae23b..eea6f49064 100644 --- a/apps/cli/tests/helpers/live-project.ts +++ b/apps/cli/tests/helpers/live-project.ts @@ -5,6 +5,7 @@ import path from "node:path"; import { makeApiClient, type OperationOutput } from "@supabase/api/effect"; import { Cause, Data, Effect, Exit, Schedule } from "effect"; +import * as HttpClientError from "effect/unstable/http/HttpClientError"; import { deriveLiveProjectHost, @@ -74,6 +75,7 @@ class LiveTransientPoll extends Data.TaggedError("LiveTransientPoll")<{ class LiveTerminalPoll extends Data.TaggedError("LiveTerminalPoll")<{ readonly phase: string; readonly message: string; + readonly cause?: unknown; }> {} class LivePollTimeout extends Data.TaggedError("LivePollTimeout")<{ @@ -86,6 +88,8 @@ class LivePollTimeout extends Data.TaggedError("LivePollTimeout")<{ class LiveStorageError extends Data.TaggedError("LiveStorageError")<{ readonly message: string; + readonly retryable: boolean; + readonly cause?: unknown; }> {} function apiError(error: unknown): Error { @@ -98,6 +102,31 @@ function supportedRegion(value: string): Region { throw new Error(`Unsupported SUPABASE_LIVE_REGION ${JSON.stringify(value)}`); } +/** HTTP statuses that can occur while a newly-created project propagates. */ +export function isTransientStorageStatus(status: number): boolean { + return status === 404 || status === 408 || status === 425 || status === 429 || status >= 500; +} + +/** Retry only transport failures and statuses plausibly caused by propagation. */ +export function isTransientLiveError(error: unknown): boolean { + if (!HttpClientError.isHttpClientError(error)) return false; + if (error.reason._tag === "TransportError") return true; + return ( + error.reason._tag === "StatusCodeError" && + isTransientStorageStatus(error.reason.response.status) + ); +} + +function classifyPollError(phase: string, cause: unknown): LiveTransientPoll | LiveTerminalPoll { + return isTransientLiveError(cause) + ? new LiveTransientPoll({ phase, cause }) + : new LiveTerminalPoll({ + phase, + message: `${phase} failed: ${apiError(cause).message}`, + cause, + }); +} + /** Retry a transient management operation using Effect's schedule and deadline semantics. */ export function retryLiveEffect( phase: string, @@ -143,11 +172,6 @@ function databasePassword(): string { return `supabase-cli-live-${randomBytes(12).toString("hex")}`; } -function liveProjectBaseUrl(ref: string, host: string): string { - const protocol = new URL(liveApiUrl()).protocol; - return `${protocol}//${ref}.${host}`; -} - function resolveOrganization(api: LiveApi): Effect.Effect { return timeoutLiveRequest("organization lookup", api.v1.listAllOrganizations()).pipe( Effect.mapError(apiError), @@ -209,9 +233,7 @@ function projectReadiness( ref: string, ): Effect.Effect { return api.v1.getProject({ ref }).pipe( - Effect.mapError( - (cause): LiveTransientPoll => new LiveTransientPoll({ phase: "project readiness", cause }), - ), + Effect.mapError((cause) => classifyPollError("project readiness", cause)), Effect.flatMap( (project): Effect.Effect => { if (project.status === "ACTIVE_HEALTHY") return Effect.succeed(project); @@ -249,9 +271,13 @@ function waitForProject(api: LiveApi, ref: string): Effect.Effect { +): Effect.Effect< + { anonKey: string; serviceRoleKey: string }, + LiveTransientPoll | LiveTerminalPoll, + never +> { return api.v1.getProjectApiKeys({ ref, reveal: true }).pipe( - Effect.mapError((cause) => new LiveTransientPoll({ phase: "project API keys", cause })), + Effect.mapError((cause) => classifyPollError("project API keys", cause)), Effect.flatMap((keys) => { const keyValue = (key: ApiKey): string | undefined => key.api_key ?? undefined; const anonKey = keys.find((key) => key.name === "anon"); @@ -279,7 +305,9 @@ function resolveKeys( Effect.mapError((error) => error instanceof LivePollTimeout ? new Error(`Project ${ref} did not return API keys within ${POLL_TIMEOUT}`) - : apiError(error), + : error instanceof LiveTerminalPoll + ? new Error(error.message) + : apiError(error), ), ); } @@ -288,10 +316,12 @@ function dbReadiness( api: LiveApi, ref: string, password: string, -): Effect.Effect { +): Effect.Effect { return api.v1.getProjectPgbouncerConfig({ ref }).pipe( - Effect.mapError((cause) => new LiveTransientPoll({ phase: "pooler configuration", cause })), - Effect.flatMap((config) => { + Effect.mapError((cause): LiveTransientPoll | LiveTerminalPoll => + classifyPollError("pooler configuration", cause), + ), + Effect.flatMap((config): Effect.Effect => { if (config.connection_string === undefined) { return Effect.fail( new LiveTransientPoll({ @@ -307,7 +337,13 @@ function dbReadiness( if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); return Effect.succeed(url.toString()); } catch (cause) { - return Effect.fail(new LiveTransientPoll({ phase: "pooler configuration", cause })); + return Effect.fail( + new LiveTerminalPoll({ + phase: "pooler configuration", + message: `pooler configuration returned an invalid connection string: ${apiError(cause).message}`, + cause, + }), + ); } }), ); @@ -326,7 +362,9 @@ function resolveDbUrl( ? new Error( `Project ${ref} did not return a pooler connection string within ${POLL_TIMEOUT}`, ) - : apiError(error), + : error instanceof LiveTerminalPoll + ? new Error(error.message) + : apiError(error), ), ); } @@ -337,9 +375,9 @@ function createStorageBucket( serviceRoleKey: string, bucket: string, ): Effect.Effect { - return Effect.tryPromise({ + const attempt = Effect.tryPromise({ try: async () => { - const response = await fetch(`${liveProjectBaseUrl(ref, host)}/storage/v1/bucket`, { + const response = await fetch(`https://${ref}.${host}/storage/v1/bucket`, { method: "POST", headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, body: JSON.stringify({ id: bucket, name: bucket, public: false }), @@ -347,15 +385,29 @@ function createStorageBucket( if (!response.ok && response.status !== 409) { throw new LiveStorageError({ message: `Failed to create storage bucket ${bucket}: ${response.status} ${await response.text()}`, + retryable: isTransientStorageStatus(response.status), }); } }, - catch: (cause) => (cause instanceof LiveStorageError ? cause : new Error(String(cause))), + catch: (cause) => + cause instanceof LiveStorageError + ? cause + : new LiveStorageError({ + message: `Failed to create storage bucket ${bucket}: ${apiError(cause).message}`, + retryable: true, + cause, + }), + }); + return retryLiveEffect("storage bucket", attempt, { + shouldRetry: (error) => error instanceof LiveStorageError && error.retryable, }).pipe( - Effect.timeoutOrElse({ - duration: POLL_TIMEOUT, - orElse: () => Effect.fail(new Error(`storage bucket ${bucket} creation timed out`)), - }), + Effect.mapError((error) => + error instanceof LivePollTimeout + ? new Error(`storage bucket ${bucket} creation timed out`) + : error instanceof LiveStorageError + ? new Error(error.message) + : apiError(error), + ), ); } @@ -468,7 +520,7 @@ export function provisionLiveEnvironment( dbPassword: password, anonKey: keys.anonKey, serviceRoleKey: keys.serviceRoleKey, - functionsUrl: `${liveProjectBaseUrl(ref, projectHost)}/functions/v1`, + functionsUrl: `https://${ref}.${projectHost}/functions/v1`, storageBucket, }, profilePath, diff --git a/apps/cli/tests/helpers/live-project.unit.test.ts b/apps/cli/tests/helpers/live-project.unit.test.ts index e91032220e..fd48f59d34 100644 --- a/apps/cli/tests/helpers/live-project.unit.test.ts +++ b/apps/cli/tests/helpers/live-project.unit.test.ts @@ -1,7 +1,23 @@ import { Effect } from "effect"; +import * as HttpClientError from "effect/unstable/http/HttpClientError"; +import * as HttpClientRequest from "effect/unstable/http/HttpClientRequest"; +import * as HttpClientResponse from "effect/unstable/http/HttpClientResponse"; import { describe, expect, it } from "vitest"; -import { cleanupErrors, retryLiveEffect } from "./live-project.ts"; +import { + cleanupErrors, + isTransientLiveError, + isTransientStorageStatus, + retryLiveEffect, +} from "./live-project.ts"; + +function statusError(status: number): HttpClientError.HttpClientError { + const request = HttpClientRequest.get("https://api.supabase.com/v1/projects/test"); + const response = HttpClientResponse.fromWeb(request, new Response(null, { status })); + return new HttpClientError.HttpClientError({ + reason: new HttpClientError.StatusCodeError({ request, response }), + }); +} describe("live project lifecycle", () => { it("retries transient failures until the management operation succeeds", async () => { @@ -50,4 +66,54 @@ describe("live project lifecycle", () => { "Error: project deletion failed", ]); }); + + it("retries transient API statuses but fails authorization errors immediately", async () => { + expect(isTransientLiveError(statusError(404))).toBe(true); + expect(isTransientLiveError(statusError(503))).toBe(true); + expect(isTransientLiveError(statusError(401))).toBe(false); + expect(isTransientLiveError(statusError(403))).toBe(false); + + let attempts = 0; + const transient = statusError(503); + const result = await Effect.runPromise( + retryLiveEffect( + "storage bucket", + Effect.suspend(() => { + attempts += 1; + return attempts < 3 ? Effect.fail(transient) : Effect.succeed("created"); + }), + { interval: "1 millis", timeout: "100 millis", shouldRetry: isTransientLiveError }, + ), + ); + expect(result).toBe("created"); + expect(attempts).toBe(3); + + attempts = 0; + await expect( + Effect.runPromise( + retryLiveEffect( + "project readiness", + Effect.suspend(() => { + attempts += 1; + return Effect.fail(statusError(403)); + }), + { + interval: "1 millis", + timeout: "100 millis", + shouldRetry: isTransientLiveError, + }, + ), + ), + ).rejects.toThrow(); + expect(attempts).toBe(1); + }); + + it("classifies storage responses for retry without retrying terminal client errors", () => { + expect(isTransientStorageStatus(408)).toBe(true); + expect(isTransientStorageStatus(429)).toBe(true); + expect(isTransientStorageStatus(500)).toBe(true); + expect(isTransientStorageStatus(401)).toBe(false); + expect(isTransientStorageStatus(403)).toBe(false); + expect(isTransientStorageStatus(422)).toBe(false); + }); }); diff --git a/apps/cli/tests/helpers/live.ts b/apps/cli/tests/helpers/live.ts index b365a40365..a5d3e5e848 100644 --- a/apps/cli/tests/helpers/live.ts +++ b/apps/cli/tests/helpers/live.ts @@ -122,6 +122,21 @@ export function requireLiveSuccess( } } +/** Rethrow a target failure without discarding failures from exact cleanup. */ +export function throwWithCleanup( + primary: unknown | undefined, + cleanup: ReadonlyArray, +): void { + if (primary !== undefined) { + if (cleanup.length > 0) { + throw new AggregateError([primary, ...cleanup], "Live e2e target and cleanup failed"); + } + throw primary; + } + if (cleanup.length === 1) throw cleanup[0]; + if (cleanup.length > 1) throw new AggregateError(cleanup, "Live e2e cleanup failed"); +} + export function expectFunctionOk( result: InvokeResult, slug: string, diff --git a/docs/adr/0013-live-e2e-bypasses-replay-server.md b/docs/adr/0013-live-e2e-bypasses-replay-server.md index b4730453e0..570a35ce8d 100644 --- a/docs/adr/0013-live-e2e-bypasses-replay-server.md +++ b/docs/adr/0013-live-e2e-bypasses-replay-server.md @@ -44,6 +44,12 @@ implement the same HTTP API contract. Retargeting a run only changes preflight, one serial attempt, a 20-minute bound, and a scoped leftover-project sweeper. +`SUPABASE_LIVE_API_URL` configures the Management API only. Tenant data-plane +URLs continue to use the CLI profile contract, `https://.`; +`project_host` is derived from the provisioned project's typed database host. +This keeps tenant routing correct even when a local platform exposes its +Management API over plain HTTP. + ## Consequences The live path has no fixture proxy, host-rewrite layer, attached/managed mode, From 1c5aeacc7d54b6eee0fcc0778487396e3c4d5a3f Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 23:13:28 +0200 Subject: [PATCH 15/23] test(cli): preserve live e2e cleanup --- .../branches/create/create.live.test.ts | 5 +- .../branches/delete/delete.live.test.ts | 43 ++++++++------- .../commands/branches/list/list.live.test.ts | 15 ++++-- .../legacy/commands/db/pull/pull.live.test.ts | 39 ++++++++------ .../functions/delete/delete.live.test.ts | 53 ++++++++++++------- .../functions/deploy/deploy.live.test.ts | 34 ++++++------ .../migration/fetch/fetch.live.test.ts | 23 ++++---- .../commands/secrets/list/list.live.test.ts | 17 ++++-- .../commands/secrets/set/set.live.test.ts | 5 +- .../commands/secrets/unset/unset.live.test.ts | 29 +++++----- .../commands/storage/cp/cp.live.test.ts | 27 +++++----- .../commands/storage/ls/ls.live.test.ts | 19 ++++--- .../commands/storage/rm/rm.live.test.ts | 31 ++++++----- apps/cli/tests/helpers/live.unit.test.ts | 33 ++++++++++++ 14 files changed, 229 insertions(+), 144 deletions(-) create mode 100644 apps/cli/tests/helpers/live.unit.test.ts diff --git a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts index 2b9c4baca8..880f0a7c9c 100644 --- a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts @@ -24,7 +24,10 @@ async function cleanupBranch( if (!exists) return; const deleted = await cli(["branches", "delete", name, "--project-ref", ref, "--yes"]); - if (deleted.exitCode !== 0 && !/not found/i.test(`${deleted.stdout}\n${deleted.stderr}`)) { + if ( + deleted.exitCode !== 0 && + !/not found|does not exist/i.test(`${deleted.stdout}\n${deleted.stderr}`) + ) { throw new Error( `branches delete cleanup failed (exit ${deleted.exitCode})\n${deleted.stdout}\n${deleted.stderr}`, ); diff --git a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts index 2de79561cb..cd3eab112f 100644 --- a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts @@ -5,38 +5,37 @@ import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests test("deletes a preview branch", async ({ cli, project }) => { const name = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; - const created = await cli(["branches", "create", name, "--project-ref", project.ref]); - requireLiveSuccess(created, "branches create"); - - let deleted = false; let targetError: unknown; let cleanupError: unknown; try { + const created = await cli(["branches", "create", name, "--project-ref", project.ref]); + requireLiveSuccess(created, "branches create"); + const removed = await cli(["branches", "delete", name, "--project-ref", project.ref, "--yes"]); expect(removed.exitCode, removed.stderr).toBe(0); - deleted = true; expect(removed.stderr).toContain("Deleted preview branch"); } catch (error) { targetError = error; } finally { - if (!deleted) { - try { - const cleanup = await cli([ - "branches", - "delete", - name, - "--project-ref", - project.ref, - "--yes", - ]); - if (cleanup.exitCode !== 0 && !/not found/i.test(`${cleanup.stdout}\n${cleanup.stderr}`)) { - cleanupError = new Error( - `branches delete cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`, - ); - } - } catch (error) { - cleanupError = error; + try { + const cleanup = await cli([ + "branches", + "delete", + name, + "--project-ref", + project.ref, + "--yes", + ]); + if ( + cleanup.exitCode !== 0 && + !/not found|does not exist/i.test(`${cleanup.stdout}\n${cleanup.stderr}`) + ) { + cleanupError = new Error( + `branches delete cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`, + ); } + } catch (error) { + cleanupError = error; } } throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); diff --git a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts index 555507c10d..8ac0529fac 100644 --- a/apps/cli/src/legacy/commands/branches/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/list/list.live.test.ts @@ -5,12 +5,12 @@ import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests test("lists a preview branch for the project", async ({ cli, project }) => { const name = `cli-e2e-list-${randomUUID().slice(0, 8)}`; - const created = await cli(["branches", "create", name, "--project-ref", project.ref]); - requireLiveSuccess(created, "branches create setup"); - let targetError: unknown; let cleanupError: unknown; try { + const created = await cli(["branches", "create", name, "--project-ref", project.ref]); + requireLiveSuccess(created, "branches create setup"); + const result = await cli([ "branches", "list", @@ -34,7 +34,14 @@ test("lists a preview branch for the project", async ({ cli, project }) => { project.ref, "--yes", ]); - requireLiveSuccess(deleted, "branches delete cleanup"); + if ( + deleted.exitCode !== 0 && + !/not found|does not exist/i.test(`${deleted.stdout}\n${deleted.stderr}`) + ) { + cleanupError = new Error( + `branches delete cleanup failed:\n${deleted.stdout}\n${deleted.stderr}`, + ); + } } catch (error) { cleanupError = error; } diff --git a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts index 526c543c35..bdb4119e38 100644 --- a/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts +++ b/apps/cli/src/legacy/commands/db/pull/pull.live.test.ts @@ -32,32 +32,37 @@ test("pulls the remote schema after a local migration is applied", async ({ targetError = error; } - let cleanupError: unknown; + const cleanupErrors: Array = []; + // Remove all migrations created by this test before resetting. This + // includes both the seed migration and the migration generated by + // `db pull`; resetting with only the generated grant statements left + // behind can reference a table that no longer exists. + let currentMigrations: ReadonlyArray = []; try { - // Remove all migrations created by this test before resetting. This - // includes both the seed migration and the migration generated by - // `db pull`; resetting with only the generated grant statements left - // behind can reference a table that no longer exists. - const currentMigrations = await readdir(migrations).catch(() => []); - for (const file of currentMigrations.filter( - (candidate) => !existingMigrations.has(candidate), - )) { - try { - await unlink(join(migrations, file)); - } catch (error) { - throw new Error( + currentMigrations = await readdir(migrations); + } catch (error) { + cleanupErrors.push(error); + } + for (const file of currentMigrations.filter((candidate) => !existingMigrations.has(candidate))) { + try { + await unlink(join(migrations, file)); + } catch (error) { + cleanupErrors.push( + new Error( `db pull cleanup could not remove test migration ${join(migrations, file)}: ${ error instanceof Error ? error.message : String(error) }`, - ); - } + ), + ); } + } + try { const reset = await cli(["db", "reset", "--db-url", project.dbUrl, "--yes"]); requireLiveSuccess(reset, "db reset cleanup after db pull"); } catch (error) { - cleanupError = error; + cleanupErrors.push(error); } - throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); + throwWithCleanup(targetError, cleanupErrors); }); diff --git a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts index 44e659cc3d..9b4b06f88c 100644 --- a/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/delete/delete.live.test.ts @@ -2,7 +2,21 @@ import { randomUUID } from "node:crypto"; import { mkdir, writeFile } from "node:fs/promises"; import { expect } from "vitest"; -import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; +import { test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; + +async function cleanupFunction( + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + slug: string, + ref: string, +): Promise { + const deleted = await cli(["functions", "delete", slug, "--project-ref", ref]); + if ( + deleted.exitCode !== 0 && + !/not found|does not exist/i.test(`${deleted.stdout}\n${deleted.stderr}`) + ) { + throw new Error(`functions delete cleanup failed:\n${deleted.stdout}\n${deleted.stderr}`); + } +} test("deletes a deployed function", async ({ cli, project, workspace }) => { const slug = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; @@ -11,34 +25,33 @@ test("deletes a deployed function", async ({ cli, project, workspace }) => { await writeFile(`${directory}/index.ts`, "Deno.serve(() => Response.json({ ok: true }));\n"); await writeFile(`${directory}/deno.json`, '{\n "imports": {}\n}\n'); - const deployed = await cli([ - "functions", - "deploy", - slug, - "--project-ref", - project.ref, - "--use-api", - ]); - requireLiveSuccess(deployed, "functions deploy setup"); - - let deleted = false; let targetError: unknown; let cleanupError: unknown; try { + const deployed = await cli([ + "functions", + "deploy", + slug, + "--project-ref", + project.ref, + "--use-api", + ]); + if (deployed.exitCode !== 0) { + throw new Error( + `functions deploy setup failed (exit ${deployed.exitCode})\nstdout:\n${deployed.stdout}\nstderr:\n${deployed.stderr}`, + ); + } + const result = await cli(["functions", "delete", slug, "--project-ref", project.ref]); expect(result.exitCode, result.stderr).toBe(0); expect(result.stdout).toContain("Deleted Function"); - deleted = true; } catch (error) { targetError = error; } finally { - if (!deleted) { - try { - const cleanup = await cli(["functions", "delete", slug, "--project-ref", project.ref]); - requireLiveSuccess(cleanup, "functions delete cleanup"); - } catch (error) { - cleanupError = error; - } + try { + await cleanupFunction(cli, slug, project.ref); + } catch (error) { + cleanupError = error; } } throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); diff --git a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts index cdcf4f7853..76e83ced69 100644 --- a/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts +++ b/apps/cli/src/legacy/commands/functions/deploy/deploy.live.test.ts @@ -4,12 +4,21 @@ import { join } from "node:path"; import { expect } from "vitest"; import { describe } from "vitest"; -import { - expectFunctionOk, - requireLiveSuccess, - test, - throwWithCleanup, -} from "../../../../../tests/helpers/live.ts"; +import { expectFunctionOk, test, throwWithCleanup } from "../../../../../tests/helpers/live.ts"; + +async function cleanupFunction( + cli: (args: string[]) => Promise<{ exitCode: number; stdout: string; stderr: string }>, + slug: string, + ref: string, +): Promise { + const deleted = await cli(["functions", "delete", slug, "--project-ref", ref]); + if ( + deleted.exitCode !== 0 && + !/not found|does not exist/i.test(`${deleted.stdout}\n${deleted.stderr}`) + ) { + throw new Error(`functions delete cleanup failed:\n${deleted.stdout}\n${deleted.stderr}`); + } +} describe("functions deploy (live)", () => { test("deploys a function that responds over HTTP", async ({ @@ -27,26 +36,21 @@ describe("functions deploy (live)", () => { ); await writeFile(join(directory, "deno.json"), '{\n "imports": {}\n}\n'); - let deployed = false; let targetError: unknown; let cleanupError: unknown; try { const result = await cli(["functions", "deploy", "--project-ref", project.ref]); expect(result.exitCode, result.stderr).toBe(0); - deployed = true; expect(result.stdout).toMatch(/Deployed Function/i); expectFunctionOk(await invoke(slug), slug); } catch (error) { targetError = error; } finally { - if (deployed) { - try { - const deleted = await cli(["functions", "delete", slug, "--project-ref", project.ref]); - requireLiveSuccess(deleted, "functions delete cleanup"); - } catch (error) { - cleanupError = error; - } + try { + await cleanupFunction(cli, slug, project.ref); + } catch (error) { + cleanupError = error; } } throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); diff --git a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts index 822bacbc14..48fa94e903 100644 --- a/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts +++ b/apps/cli/src/legacy/commands/migration/fetch/fetch.live.test.ts @@ -39,7 +39,6 @@ test( const migrationFile = `${version}_${NAME}.sql`; const seedDir = await mkdtemp(path.join(tmpdir(), "sb-migration-seed-live-")); const fetchDir = await mkdtemp(path.join(tmpdir(), "sb-migration-fetch-live-")); - let repaired = false; let targetError: unknown; const cleanupErrors: Array = []; try { @@ -55,7 +54,6 @@ test( { cwd: seedDir }, ); requireLiveSuccess(repairResult, "migration repair setup"); - repaired = true; // Fetch into a fresh (empty) dir so no overwrite prompt fires; it reads the // remote history and writes _.sql. @@ -68,16 +66,21 @@ test( } catch (error) { targetError = error; } finally { - if (repaired) { - try { - const reverted = await cli( - ["migration", "repair", version, "--status", "reverted", ...targetArgs], - { cwd: seedDir }, + try { + const reverted = await cli( + ["migration", "repair", version, "--status", "reverted", ...targetArgs], + { cwd: seedDir }, + ); + if ( + reverted.exitCode !== 0 && + !/not found|does not exist/i.test(`${reverted.stdout}\n${reverted.stderr}`) + ) { + cleanupErrors.push( + new Error(`migration repair cleanup failed:\n${reverted.stdout}\n${reverted.stderr}`), ); - requireLiveSuccess(reverted, "migration repair cleanup"); - } catch (error) { - cleanupErrors.push(error); } + } catch (error) { + cleanupErrors.push(error); } await rm(seedDir, { recursive: true, force: true }).catch((error) => cleanupErrors.push(error), diff --git a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts index a9c5930f52..1b5e062d9e 100644 --- a/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/list/list.live.test.ts @@ -9,19 +9,28 @@ async function unsetSecret( ref: string, ): Promise { const cleanup = await cli(["secrets", "unset", name, "--project-ref", ref, "--yes"]); - if (cleanup.exitCode !== 0) { + if ( + cleanup.exitCode !== 0 && + !/not found|does not exist/i.test(`${cleanup.stdout}\n${cleanup.stderr}`) + ) { throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); } } test("lists a secret created on the remote project", async ({ cli, project }) => { const name = `CLI_E2E_LIST_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; - const created = await cli(["secrets", "set", `${name}=live-value`, "--project-ref", project.ref]); - requireLiveSuccess(created, "secrets set setup"); - let targetError: unknown; let cleanupError: unknown; try { + const created = await cli([ + "secrets", + "set", + `${name}=live-value`, + "--project-ref", + project.ref, + ]); + requireLiveSuccess(created, "secrets set setup"); + const result = await cli(["secrets", "list", "--output", "json", "--project-ref", project.ref]); expect(result.exitCode, result.stderr).toBe(0); const names = (JSON.parse(result.stdout) as Array<{ name: string }>).map( diff --git a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts index c7862bb93c..617ad85c8c 100644 --- a/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/set/set.live.test.ts @@ -9,7 +9,10 @@ async function unsetSecret( ref: string, ): Promise { const cleanup = await cli(["secrets", "unset", name, "--project-ref", ref, "--yes"]); - if (cleanup.exitCode !== 0) { + if ( + cleanup.exitCode !== 0 && + !/not found|does not exist/i.test(`${cleanup.stdout}\n${cleanup.stderr}`) + ) { throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); } } diff --git a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts index 3818a9f91e..ede0a6dc6b 100644 --- a/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts +++ b/apps/cli/src/legacy/commands/secrets/unset/unset.live.test.ts @@ -9,33 +9,38 @@ async function unsetSecret( ref: string, ): Promise { const cleanup = await cli(["secrets", "unset", name, "--project-ref", ref, "--yes"]); - if (cleanup.exitCode !== 0 && !/not found/i.test(`${cleanup.stdout}\n${cleanup.stderr}`)) { + if ( + cleanup.exitCode !== 0 && + !/not found|does not exist/i.test(`${cleanup.stdout}\n${cleanup.stderr}`) + ) { throw new Error(`secrets unset cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`); } } test("unsets a secret from the remote project", async ({ cli, project }) => { const name = `CLI_E2E_UNSET_${randomUUID().replaceAll("-", "").slice(0, 12).toUpperCase()}`; - const created = await cli(["secrets", "set", `${name}=live-value`, "--project-ref", project.ref]); - requireLiveSuccess(created, "secrets set setup"); - - let deleted = false; let targetError: unknown; let cleanupError: unknown; try { + const created = await cli([ + "secrets", + "set", + `${name}=live-value`, + "--project-ref", + project.ref, + ]); + requireLiveSuccess(created, "secrets set setup"); + const result = await cli(["secrets", "unset", name, "--project-ref", project.ref, "--yes"]); expect(result.exitCode, result.stderr).toBe(0); - deleted = true; expect(result.stdout).toContain("Finished"); } catch (error) { targetError = error; } finally { - if (!deleted) { - try { - await unsetSecret(cli, name, project.ref); - } catch (error) { - cleanupError = error; - } + try { + await unsetSecret(cli, name, project.ref); + } catch (error) { + cleanupError = error; } } throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); diff --git a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts index 2be147f616..c66f82c981 100644 --- a/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/cp/cp.live.test.ts @@ -12,7 +12,10 @@ async function removeObject( remote: string, ): Promise { const removed = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); - if (removed.exitCode !== 0) { + if ( + removed.exitCode !== 0 && + !/not found|does not exist/i.test(`${removed.stdout}\n${removed.stderr}`) + ) { throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); } } @@ -23,27 +26,23 @@ test("copies a local file to the remote bucket", async ({ cli, project, workspac const remote = `ss:///${project.storageBucket}/upload-${suffix}.txt`; await writeFile(local, "live-e2e storage payload\n"); - const linked = await cli(["link", "--project-ref", project.ref], { - env: { SUPABASE_DB_PASSWORD: project.dbPassword }, - }); - requireLiveSuccess(linked, "link setup for storage cp"); - - let uploaded = false; let targetError: unknown; let cleanupError: unknown; try { + const linked = await cli(["link", "--project-ref", project.ref], { + env: { SUPABASE_DB_PASSWORD: project.dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage cp"); + const result = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); expect(result.exitCode, result.stderr).toBe(0); - uploaded = true; } catch (error) { targetError = error; } finally { - if (uploaded) { - try { - await removeObject(cli, remote); - } catch (error) { - cleanupError = error; - } + try { + await removeObject(cli, remote); + } catch (error) { + cleanupError = error; } } throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); diff --git a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts index d8b5f8504b..b678ceb7df 100644 --- a/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/ls/ls.live.test.ts @@ -12,7 +12,10 @@ async function removeObject( remote: string, ): Promise { const removed = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); - if (removed.exitCode !== 0) { + if ( + removed.exitCode !== 0 && + !/not found|does not exist/i.test(`${removed.stdout}\n${removed.stderr}`) + ) { throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); } } @@ -23,16 +26,16 @@ test("lists an uploaded object", async ({ cli, project, workspace }) => { const remote = `ss:///${project.storageBucket}/upload-${suffix}.txt`; await writeFile(local, "live-e2e storage payload\n"); - const linked = await cli(["link", "--project-ref", project.ref], { - env: { SUPABASE_DB_PASSWORD: project.dbPassword }, - }); - requireLiveSuccess(linked, "link setup for storage ls"); - const uploaded = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); - requireLiveSuccess(uploaded, "storage cp setup for storage ls"); - let targetError: unknown; let cleanupError: unknown; try { + const linked = await cli(["link", "--project-ref", project.ref], { + env: { SUPABASE_DB_PASSWORD: project.dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage ls"); + const uploaded = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + requireLiveSuccess(uploaded, "storage cp setup for storage ls"); + const result = await cli([ "storage", "ls", diff --git a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts index c2b550e58a..674268a2cc 100644 --- a/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts +++ b/apps/cli/src/legacy/commands/storage/rm/rm.live.test.ts @@ -12,7 +12,10 @@ async function removeObject( remote: string, ): Promise { const removed = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); - if (removed.exitCode !== 0) { + if ( + removed.exitCode !== 0 && + !/not found|does not exist/i.test(`${removed.stdout}\n${removed.stderr}`) + ) { throw new Error(`storage rm cleanup failed:\n${removed.stdout}\n${removed.stderr}`); } } @@ -23,29 +26,25 @@ test("removes an uploaded object", async ({ cli, project, workspace }) => { const remote = `ss:///${project.storageBucket}/upload-${suffix}.txt`; await writeFile(local, "live-e2e storage payload\n"); - const linked = await cli(["link", "--project-ref", project.ref], { - env: { SUPABASE_DB_PASSWORD: project.dbPassword }, - }); - requireLiveSuccess(linked, "link setup for storage rm"); - const uploaded = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); - requireLiveSuccess(uploaded, "storage cp setup for storage rm"); - - let removed = false; let targetError: unknown; let cleanupError: unknown; try { + const linked = await cli(["link", "--project-ref", project.ref], { + env: { SUPABASE_DB_PASSWORD: project.dbPassword }, + }); + requireLiveSuccess(linked, "link setup for storage rm"); + const uploaded = await cli(["storage", "cp", local, remote, ...STORAGE_FLAGS]); + requireLiveSuccess(uploaded, "storage cp setup for storage rm"); + const result = await cli(["storage", "rm", remote, "--yes", ...STORAGE_FLAGS]); expect(result.exitCode, result.stderr).toBe(0); - removed = true; } catch (error) { targetError = error; } finally { - if (!removed) { - try { - await removeObject(cli, remote); - } catch (error) { - cleanupError = error; - } + try { + await removeObject(cli, remote); + } catch (error) { + cleanupError = error; } } throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); diff --git a/apps/cli/tests/helpers/live.unit.test.ts b/apps/cli/tests/helpers/live.unit.test.ts new file mode 100644 index 0000000000..99ac0b4d6b --- /dev/null +++ b/apps/cli/tests/helpers/live.unit.test.ts @@ -0,0 +1,33 @@ +import { describe, expect, it } from "vitest"; + +import { throwWithCleanup } from "./live.ts"; + +describe("throwWithCleanup", () => { + it("rethrows the primary failure when cleanup succeeds", () => { + const primary = new Error("target failed"); + + expect(() => throwWithCleanup(primary, [])).toThrow(primary); + }); + + it("throws the cleanup failure when the target succeeds", () => { + const cleanup = new Error("cleanup failed"); + + expect(() => throwWithCleanup(undefined, [cleanup])).toThrow(cleanup); + }); + + it("preserves the primary and every cleanup failure", () => { + const primary = new Error("target failed"); + const cleanup = [new Error("first cleanup failed"), new Error("second cleanup failed")]; + let thrown: unknown; + + try { + throwWithCleanup(primary, cleanup); + } catch (error) { + thrown = error; + } + + expect(thrown).toBeInstanceOf(AggregateError); + if (!(thrown instanceof AggregateError)) return; + expect(thrown.errors).toEqual([primary, ...cleanup]); + }); +}); From 6f64b9a8aa6bd11fc84cbbcf042f2ec93a53823e Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 23:18:27 +0200 Subject: [PATCH 16/23] test(cli): simplify branch cleanup --- .../commands/branches/create/create.live.test.ts | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts index 880f0a7c9c..3a4b1fd4fd 100644 --- a/apps/cli/src/legacy/commands/branches/create/create.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/create/create.live.test.ts @@ -8,21 +8,6 @@ async function cleanupBranch( name: string, ref: string, ): Promise { - const listed = await cli(["branches", "list", "--output", "json", "--project-ref", ref]); - if (listed.exitCode !== 0) { - throw new Error( - `branches list cleanup failed (exit ${listed.exitCode})\n${listed.stdout}\n${listed.stderr}`, - ); - } - const branches: unknown = JSON.parse(listed.stdout); - const exists = - Array.isArray(branches) && - branches.some( - (branch) => - typeof branch === "object" && branch !== null && "name" in branch && branch.name === name, - ); - if (!exists) return; - const deleted = await cli(["branches", "delete", name, "--project-ref", ref, "--yes"]); if ( deleted.exitCode !== 0 && From 44ffc5a78b6b7e6d71303887f4b49b42f44d1784 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 23:36:25 +0200 Subject: [PATCH 17/23] test(cli): use primary pooler for live database --- apps/cli/tests/helpers/live-project.ts | 75 ++++++++++++------- .../tests/helpers/live-project.unit.test.ts | 56 ++++++++++++++ 2 files changed, 105 insertions(+), 26 deletions(-) diff --git a/apps/cli/tests/helpers/live-project.ts b/apps/cli/tests/helpers/live-project.ts index eea6f49064..8246a45398 100644 --- a/apps/cli/tests/helpers/live-project.ts +++ b/apps/cli/tests/helpers/live-project.ts @@ -25,6 +25,7 @@ const POLL_TIMEOUT = "5 minutes"; type Project = OperationOutput<"v1GetProject">; type Organization = OperationOutput<"v1ListAllOrganizations">[number]; type ApiKey = OperationOutput<"v1GetProjectApiKeys">[number]; +export type PoolerConfig = OperationOutput<"v1GetPoolerConfig">[number]; type LiveApi = Effect.Success>; type Region = | "us-east-1" @@ -117,6 +118,24 @@ export function isTransientLiveError(error: unknown): boolean { ); } +export function selectPrimaryPoolerConfig( + configs: ReadonlyArray, +): PoolerConfig | undefined { + return configs.find((config) => config.database_type === "PRIMARY"); +} + +export function resolvePoolerDatabaseUrl( + connectionString: string, + poolMode: PoolerConfig["pool_mode"], + password: string, +): string { + const url = new URL(connectionString); + url.password = password; + if (poolMode !== "session" && url.port === "6543") url.port = "5432"; + if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); + return url.toString(); +} + function classifyPollError(phase: string, cause: unknown): LiveTransientPoll | LiveTerminalPoll { return isTransientLiveError(cause) ? new LiveTransientPoll({ phase, cause }) @@ -317,35 +336,39 @@ function dbReadiness( ref: string, password: string, ): Effect.Effect { - return api.v1.getProjectPgbouncerConfig({ ref }).pipe( + return api.v1.getPoolerConfig({ ref }).pipe( Effect.mapError((cause): LiveTransientPoll | LiveTerminalPoll => classifyPollError("pooler configuration", cause), ), - Effect.flatMap((config): Effect.Effect => { - if (config.connection_string === undefined) { - return Effect.fail( - new LiveTransientPoll({ - phase: "pooler configuration", - cause: "connection string missing", - }), - ); - } - try { - const url = new URL(config.connection_string); - url.password = password; - url.port = "5432"; - if (!url.searchParams.has("connect_timeout")) url.searchParams.set("connect_timeout", "30"); - return Effect.succeed(url.toString()); - } catch (cause) { - return Effect.fail( - new LiveTerminalPoll({ - phase: "pooler configuration", - message: `pooler configuration returned an invalid connection string: ${apiError(cause).message}`, - cause, - }), - ); - } - }), + Effect.flatMap( + (configs): Effect.Effect => { + const primary = selectPrimaryPoolerConfig(configs); + if (primary === undefined || primary.connection_string.trim().length === 0) { + return Effect.fail( + new LiveTransientPoll({ + phase: "pooler configuration", + cause: + primary === undefined + ? "primary pooler config missing" + : "connection string missing", + }), + ); + } + try { + return Effect.succeed( + resolvePoolerDatabaseUrl(primary.connection_string, primary.pool_mode, password), + ); + } catch (cause) { + return Effect.fail( + new LiveTerminalPoll({ + phase: "pooler configuration", + message: `pooler configuration returned an invalid connection string: ${apiError(cause).message}`, + cause, + }), + ); + } + }, + ), ); } diff --git a/apps/cli/tests/helpers/live-project.unit.test.ts b/apps/cli/tests/helpers/live-project.unit.test.ts index fd48f59d34..9ff3cea4ed 100644 --- a/apps/cli/tests/helpers/live-project.unit.test.ts +++ b/apps/cli/tests/helpers/live-project.unit.test.ts @@ -8,7 +8,10 @@ import { cleanupErrors, isTransientLiveError, isTransientStorageStatus, + resolvePoolerDatabaseUrl, retryLiveEffect, + selectPrimaryPoolerConfig, + type PoolerConfig, } from "./live-project.ts"; function statusError(status: number): HttpClientError.HttpClientError { @@ -19,6 +22,24 @@ function statusError(status: number): HttpClientError.HttpClientError { }); } +function poolerConfig(overrides: Partial = {}): PoolerConfig { + return { + identifier: "primary", + database_type: "PRIMARY", + is_using_scram_auth: false, + db_user: "postgres", + db_host: "pooler.example.com", + db_port: 6543, + db_name: "postgres", + connection_string: "postgresql://postgres.ref:[YOUR-PASSWORD]@pooler.example.com:6543/postgres", + connectionString: "", + default_pool_size: null, + max_client_conn: null, + pool_mode: "transaction", + ...overrides, + }; +} + describe("live project lifecycle", () => { it("retries transient failures until the management operation succeeds", async () => { let attempts = 0; @@ -116,4 +137,39 @@ describe("live project lifecycle", () => { expect(isTransientStorageStatus(403)).toBe(false); expect(isTransientStorageStatus(422)).toBe(false); }); + + it("selects the primary pooler config", () => { + const replica = poolerConfig({ identifier: "replica", database_type: "READ_REPLICA" }); + const primary = poolerConfig(); + + expect(selectPrimaryPoolerConfig([replica, primary])).toBe(primary); + }); + + it("translates transaction pooler port and encodes the password", () => { + const resolved = new URL( + resolvePoolerDatabaseUrl( + "postgresql://postgres.ref:[YOUR-PASSWORD]@pooler.example.com:6543/postgres", + "transaction", + "p@ss word", + ), + ); + + expect(resolved.hostname).toBe("pooler.example.com"); + expect(resolved.port).toBe("5432"); + expect(decodeURIComponent(resolved.password)).toBe("p@ss word"); + expect(resolved.searchParams.get("connect_timeout")).toBe("30"); + }); + + it("preserves the API port and timeout for session pooler mode", () => { + const resolved = new URL( + resolvePoolerDatabaseUrl( + "postgresql://postgres.ref:[YOUR-PASSWORD]@pooler.example.com:6543/postgres?connect_timeout=7", + "session", + "secret", + ), + ); + + expect(resolved.port).toBe("6543"); + expect(resolved.searchParams.get("connect_timeout")).toBe("7"); + }); }); From abcbb474a64068e93f8d4b0c2af3aac354195834 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Fri, 21 Aug 2026 23:48:06 +0200 Subject: [PATCH 18/23] test(cli): avoid duplicate branch cleanup --- .../branches/delete/delete.live.test.ts | 41 +++++++++++-------- 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts index cd3eab112f..3bd193d878 100644 --- a/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts +++ b/apps/cli/src/legacy/commands/branches/delete/delete.live.test.ts @@ -5,37 +5,42 @@ import { requireLiveSuccess, test, throwWithCleanup } from "../../../../../tests test("deletes a preview branch", async ({ cli, project }) => { const name = `cli-e2e-delete-${randomUUID().slice(0, 8)}`; + let mayExist = false; let targetError: unknown; let cleanupError: unknown; try { + mayExist = true; const created = await cli(["branches", "create", name, "--project-ref", project.ref]); requireLiveSuccess(created, "branches create"); const removed = await cli(["branches", "delete", name, "--project-ref", project.ref, "--yes"]); + if (removed.exitCode === 0) mayExist = false; expect(removed.exitCode, removed.stderr).toBe(0); expect(removed.stderr).toContain("Deleted preview branch"); } catch (error) { targetError = error; } finally { - try { - const cleanup = await cli([ - "branches", - "delete", - name, - "--project-ref", - project.ref, - "--yes", - ]); - if ( - cleanup.exitCode !== 0 && - !/not found|does not exist/i.test(`${cleanup.stdout}\n${cleanup.stderr}`) - ) { - cleanupError = new Error( - `branches delete cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`, - ); + if (mayExist) { + try { + const cleanup = await cli([ + "branches", + "delete", + name, + "--project-ref", + project.ref, + "--yes", + ]); + if ( + cleanup.exitCode !== 0 && + !/not found|does not exist/i.test(`${cleanup.stdout}\n${cleanup.stderr}`) + ) { + cleanupError = new Error( + `branches delete cleanup failed:\n${cleanup.stdout}\n${cleanup.stderr}`, + ); + } + } catch (error) { + cleanupError = error; } - } catch (error) { - cleanupError = error; } } throwWithCleanup(targetError, cleanupError === undefined ? [] : [cleanupError]); From 980bb111ea6fe45606625e24cb0065068f7acdaf Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Sat, 22 Aug 2026 00:30:44 +0200 Subject: [PATCH 19/23] test(cli): fix Docker e2e execution --- .../commands/db/diff/diff.docker.e2e.test.ts | 21 +-- .../declarative/sync/sync.docker.e2e.test.ts | 15 +- .../db/start/start.docker.e2e.test.ts | 2 + .../commands/start/start.docker.e2e.test.ts | 31 ++-- .../commands/status/status.docker.e2e.test.ts | 18 ++- .../commands/stop/stop.docker.e2e.test.ts | 26 ++-- .../functions/dev/dev.docker.e2e.test.ts | 66 +++++---- .../functions/dev/functions-dev-runtime.ts | 5 - .../commands/start/start.docker.e2e.test.ts | 4 +- .../runtime/stack-e2e-cleanup.unit.test.ts | 134 ++++++++++++++++++ apps/cli/tests/helpers/stack-e2e-cleanup.ts | 69 ++++++--- 11 files changed, 297 insertions(+), 94 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts index 03cc95b35c..3662c5e293 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts @@ -17,16 +17,19 @@ const START_TIMEOUT_MS = 280_000; // (supabase/pg-toolbelt#357). Verified directly against this repo's build: with // the pre-fix pin (1.0.0-alpha.32) the migration below contains only the CREATE // FUNCTION statement; the REVOKE is silently absent. This suite uses the local -// Docker-stack gate and never calls the Management API. See AGENTS.md's "Live -// tests" section. -describe("supabase db diff (live, pg-delta declarative privileges)", () => { +// Docker-stack e2e coverage and never calls the Management API. See AGENTS.md's +// "E2e tests" section. +describe("supabase db diff (e2e, pg-delta declarative privileges)", () => { let projectDir: string | undefined; afterEach(async () => { if (projectDir === undefined) return; // Best-effort cleanup even if an assertion above failed mid-lifecycle — a // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabase(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { + entrypoint: "legacy", + cwd: projectDir, + }).catch(() => undefined); await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); projectDir = undefined; }); @@ -35,16 +38,16 @@ describe("supabase db diff (live, pg-delta declarative privileges)", () => { "keeps REVOKE ... FROM PUBLIC on a function when diffing a declarative schema against local", { timeout: START_TIMEOUT_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-db-diff-live-")); + projectDir = await mkdtemp(path.join(tmpdir(), "sb-db-diff-e2e-")); - const init = await runSupabase(["init"], { cwd: projectDir }); + const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir }); requireCliSuccess(init, "init setup"); // Exclude the heaviest, least relevant services — `db diff` only needs the // local Postgres container reachable, same rationale as stop/status. const start = await runSupabase( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); @@ -63,13 +66,13 @@ as $$ select 1; $$; revoke execute on function public.probe_fn() from public;`, "--local", ], - { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); requireCliSuccess(query, "db query setup"); const diff = await runSupabase( ["db", "diff", "--local", "--use-pg-delta", "-f", "revoke_public_execute"], - { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); expect(diff.exitCode, `stdout:\n${diff.stdout}\nstderr:\n${diff.stderr}`).toBe(0); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts index bbdb5391a9..06587a8fd6 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts @@ -40,9 +40,10 @@ describe("db schema declarative sync (e2e)", () => { let projectDir = ""; beforeAll(async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-live-")); + projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-e2e-")); const init = await runSupabase(["init"], { + entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS, }); @@ -81,7 +82,7 @@ describe("db schema declarative sync (e2e)", () => { "--exclude", "storage-api", ], - { cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); }, COMMAND_TIMEOUT_MS); @@ -89,6 +90,7 @@ describe("db schema declarative sync (e2e)", () => { afterAll(async () => { if (projectDir.length === 0) return; await runSupabase(["stop", "--no-backup"], { + entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS, }).catch(() => undefined); @@ -110,7 +112,12 @@ describe("db schema declarative sync (e2e)", () => { "initial_declarative", "--experimental", ], - { cwd: projectDir, env: NEXT_ENV, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + { + entrypoint: "legacy", + cwd: projectDir, + env: NEXT_ENV, + exitTimeoutMs: COMMAND_TIMEOUT_MS, + }, ); expect(sync.exitCode, commandFailure(sync)).toBe(0); @@ -128,6 +135,7 @@ describe("db schema declarative sync (e2e)", () => { ); const reset = await runSupabase(["db", "reset", "--local", "--no-seed"], { + entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS, }); @@ -136,6 +144,7 @@ describe("db schema declarative sync (e2e)", () => { const converged = await runSupabase( ["db", "schema", "declarative", "sync", "--no-apply", "--experimental"], { + entrypoint: "legacy", cwd: projectDir, env: NEXT_ENV, exitTimeoutMs: COMMAND_TIMEOUT_MS, diff --git a/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts index 53c916d275..48681124f4 100644 --- a/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts @@ -12,6 +12,7 @@ describe("supabase db start (e2e)", () => { const project = await makeTempStackProject("supabase-db-start-e2e-"); try { const started = await runSupabase(["db", "start"], { + entrypoint: "legacy", cwd: project.dir, home: home.dir, }); @@ -21,6 +22,7 @@ describe("supabase db start (e2e)", () => { ); } finally { await runSupabase(["stop", "--no-backup"], { + entrypoint: "legacy", cwd: project.dir, home: home.dir, }).catch(() => undefined); diff --git a/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts b/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts index b2ece5fc36..bbb658f5e0 100644 --- a/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts @@ -20,7 +20,7 @@ import { dockerfileServiceImage } from "../../../shared/services/dockerfile-imag const execFileAsync = promisify(execFile); const START_TIMEOUT_MS = 280_000; -const SHORT_LIVE_TIMEOUT_MS = 30_000; +const SHORT_E2E_TIMEOUT_MS = 30_000; const LIFECYCLE_OVERHEAD_MS = 90_000; /** @@ -71,7 +71,10 @@ describe("supabase start (e2e)", () => { if (projectDir === undefined) return; // Best-effort cleanup even if an assertion above failed mid-lifecycle — a // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabase(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { + entrypoint: "legacy", + cwd: projectDir, + }).catch(() => undefined); await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); projectDir = undefined; }); @@ -80,7 +83,7 @@ describe("supabase start (e2e)", () => { "recreates a stopped real stack and preserves database data", { timeout: START_TIMEOUT_MS * 2 + LIFECYCLE_OVERHEAD_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-start-live-")); + projectDir = await mkdtemp(path.join(tmpdir(), "sb-start-e2e-")); // No `project_id` override, so the cli resolves it from the workdir // basename (see legacy-docker-ids.ts). Sanitizing is a no-op for a // `mkdtemp`-generated basename (already alphanumeric/`-`), but mirrors @@ -99,12 +102,14 @@ describe("supabase start (e2e)", () => { ]; const init = await runSupabase(["init"], { + entrypoint: "legacy", cwd: projectDir, - exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, + exitTimeoutMs: SHORT_E2E_TIMEOUT_MS, }); requireCliSuccess(init, "init setup"); const start = await runSupabase(startArgs, { + entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS, }); @@ -135,7 +140,7 @@ describe("supabase start (e2e)", () => { const containerIds = splitNonEmptyLines(containerIdOutput); expect(containerIds.length).toBeGreaterThan(0); await execFileAsync("docker", ["stop", "--time", "0", ...containerIds], { - timeout: SHORT_LIVE_TIMEOUT_MS, + timeout: SHORT_E2E_TIMEOUT_MS, }); const { stdout: stoppedState } = await execFileAsync("docker", [ @@ -151,6 +156,7 @@ describe("supabase start (e2e)", () => { }); const restart = await runSupabase(startArgs, { + entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS, }); @@ -196,8 +202,9 @@ describe("supabase start (e2e)", () => { } const status = await runSupabase(["status"], { + entrypoint: "legacy", cwd: projectDir, - exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, + exitTimeoutMs: SHORT_E2E_TIMEOUT_MS, }); requireCliSuccess(status, "status setup"); }, @@ -207,11 +214,12 @@ describe("supabase start (e2e)", () => { "bypasses an HTTPS proxy for loopback gateway health checks", { timeout: START_TIMEOUT_MS + LIFECYCLE_OVERHEAD_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-start-live-proxy-")); + projectDir = await mkdtemp(path.join(tmpdir(), "sb-start-e2e-proxy-")); const init = await runSupabase(["init"], { + entrypoint: "legacy", cwd: projectDir, - exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, + exitTimeoutMs: SHORT_E2E_TIMEOUT_MS, }); requireCliSuccess(init, "init setup"); @@ -238,6 +246,7 @@ describe("supabase start (e2e)", () => { ); const proxyUrl = `http://127.0.0.1:${address.port}`; const start = await runSupabase(["start", ...excludeArgs], { + entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS, env: { @@ -273,7 +282,7 @@ describe("supabase start (e2e)", () => { "names the container and its image when a cached image cannot be executed", { timeout: START_TIMEOUT_MS + LIFECYCLE_OVERHEAD_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-start-live-exec-")); + projectDir = await mkdtemp(path.join(tmpdir(), "sb-start-e2e-exec-")); const projectId = legacySanitizeProjectId(path.basename(projectDir)); const mailpitContainer = legacyServiceContainerName("inbucket", projectId); // The exact tag `start` resolves for Mailpit, so its already-cached check @@ -281,8 +290,9 @@ describe("supabase start (e2e)", () => { const mailpitImage = legacyGetRegistryImageUrl(dockerfileServiceImage("mailpit")); const init = await runSupabase(["init"], { + entrypoint: "legacy", cwd: projectDir, - exitTimeoutMs: SHORT_LIVE_TIMEOUT_MS, + exitTimeoutMs: SHORT_E2E_TIMEOUT_MS, }); requireCliSuccess(init, "init setup"); @@ -306,6 +316,7 @@ describe("supabase start (e2e)", () => { : ["--exclude", entry.excludeKey], ); const start = await runSupabase(["start", ...excludeArgs], { + entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS, }); diff --git a/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts b/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts index 14425148ca..9d52635a37 100644 --- a/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts @@ -17,7 +17,10 @@ describe("supabase status (e2e)", () => { afterEach(async () => { if (projectDir === undefined) return; - await runSupabase(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { + entrypoint: "legacy", + cwd: projectDir, + }).catch(() => undefined); await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); projectDir = undefined; }); @@ -26,24 +29,27 @@ describe("supabase status (e2e)", () => { "reports a running local stack in pretty and json modes", { timeout: START_TIMEOUT_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-status-live-")); + projectDir = await mkdtemp(path.join(tmpdir(), "sb-status-e2e-")); - const init = await runSupabase(["init"], { cwd: projectDir }); + const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir }); requireCliSuccess(init, "init setup"); const start = await runSupabase( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); - const pretty = await runSupabase(["status"], { cwd: projectDir }); + const pretty = await runSupabase(["status"], { entrypoint: "legacy", cwd: projectDir }); expect(pretty.exitCode, `stdout:\n${pretty.stdout}\nstderr:\n${pretty.stderr}`).toBe(0); expect(`${pretty.stdout}${pretty.stderr}`).toContain("is running"); expect(pretty.stdout).toContain("Project URL"); expect(pretty.stdout).toContain("Database"); - const json = await runSupabase(["status", "-o", "json"], { cwd: projectDir }); + const json = await runSupabase(["status", "-o", "json"], { + entrypoint: "legacy", + cwd: projectDir, + }); expect(json.exitCode, `stdout:\n${json.stdout}\nstderr:\n${json.stderr}`).toBe(0); const parsed: unknown = JSON.parse(json.stdout); expect(parsed).toMatchObject({ diff --git a/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts b/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts index dcd252bde8..a1c7f44dab 100644 --- a/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts @@ -26,7 +26,10 @@ describe("supabase stop (e2e)", () => { if (projectDir === undefined) return; // Best-effort cleanup even if an assertion above failed mid-lifecycle — a // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabase(["stop", "--no-backup"], { cwd: projectDir }).catch(() => undefined); + await runSupabase(["stop", "--no-backup"], { + entrypoint: "legacy", + cwd: projectDir, + }).catch(() => undefined); await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); projectDir = undefined; projectId = undefined; @@ -36,12 +39,12 @@ describe("supabase stop (e2e)", () => { "starts a real local stack, then stops it and removes its containers", { timeout: START_TIMEOUT_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-stop-live-")); + projectDir = await mkdtemp(path.join(tmpdir(), "sb-stop-e2e-")); // No `project_id` override, so the cli resolves it from the workdir // basename (see legacy-docker-ids.ts). projectId = path.basename(projectDir); - const init = await runSupabase(["init"], { cwd: projectDir }); + const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir }); requireCliSuccess(init, "init setup"); // Exclude the heaviest, least relevant services (Next.js Studio build, the @@ -50,15 +53,15 @@ describe("supabase stop (e2e)", () => { // exists to stop. const start = await runSupabase( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); // Sanity: confirm the stack is actually up before testing `stop` against it. - const before = await runSupabase(["status"], { cwd: projectDir }); + const before = await runSupabase(["status"], { entrypoint: "legacy", cwd: projectDir }); requireCliSuccess(before, "status setup"); - const stop = await runSupabase(["stop"], { cwd: projectDir }); + const stop = await runSupabase(["stop"], { entrypoint: "legacy", cwd: projectDir }); expect(stop.exitCode, `stdout:\n${stop.stdout}\nstderr:\n${stop.stderr}`).toBe(0); expect(stop.stdout).toContain("Stopped"); @@ -81,18 +84,18 @@ describe("supabase stop (e2e)", () => { "stop --no-backup --debug reports real pruned containers, volumes, and network", { timeout: START_TIMEOUT_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-stop-live-")); + projectDir = await mkdtemp(path.join(tmpdir(), "sb-stop-e2e-")); // Sanitizing is a no-op for a `mkdtemp`-generated basename (already // alphanumeric/`-`), but mirrors the port's actual resolution rather // than assuming that stays true (same note as `start.e2e.test.ts`). projectId = legacySanitizeProjectId(path.basename(projectDir)); - const init = await runSupabase(["init"], { cwd: projectDir }); + const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir }); requireCliSuccess(init, "init setup"); const start = await runSupabase( ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); @@ -101,7 +104,10 @@ describe("supabase stop (e2e)", () => { // backed by parsing REAL `docker`/`podman` prune stdout — the format // assumption (`Deleted …:` headers, `Total reclaimed space:` trailer) // that mocked integration fixtures cannot validate by construction. - const stop = await runSupabase(["stop", "--no-backup", "--debug"], { cwd: projectDir }); + const stop = await runSupabase(["stop", "--no-backup", "--debug"], { + entrypoint: "legacy", + cwd: projectDir, + }); expect(stop.exitCode, `stdout:\n${stop.stdout}\nstderr:\n${stop.stderr}`).toBe(0); expect(stop.stdout).toContain("Stopped"); diff --git a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts index 7c69065346..2a5f6461f5 100644 --- a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts +++ b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts @@ -13,9 +13,9 @@ import { cleanupRegisteredStackProjects } from "../../../../../tests/helpers/sta const FUNCTIONS_DEV_STARTUP_TIMEOUT_MS = 60_000; const FUNCTIONS_DEV_STEP_TIMEOUT_MS = 30_000; const FUNCTIONS_DEV_TEST_TIMEOUT_MS = 90_000; +const FUNCTION_RESPONSE_ATTEMPT_TIMEOUT_MS = 5_000; +const FUNCTION_RESPONSE_RETRY_BACKOFF_MS = 250; const FUNCTION_FILES_RESTART_PATTERN = /Function files changed\. Restarting edge-runtime\./; -const FUNCTION_RELOAD_COMPLETE_PATTERN = /Function reload complete\./; -const EDGE_RUNTIME_RELOAD_COMPLETE_PATTERN = /Edge runtime reload complete\./; type SpawnedSupabase = ReturnType; @@ -23,23 +23,46 @@ async function assertFunctionResponse( url: string, init: RequestInit, assertResponse: (response: Response, body: string) => void, + timeoutMs = FUNCTIONS_DEV_STEP_TIMEOUT_MS, ): Promise { - try { - const response = await fetch(url, init); - const body = await response.text(); - assertResponse(response, body); - } catch (error) { - throw new Error( - `Function request ${url} failed: ${error instanceof Error ? error.message : String(error)}`, - ); + const deadline = Date.now() + timeoutMs; + let lastFailure: unknown = new Error("No response received"); + + while (Date.now() < deadline) { + const remainingMs = deadline - Date.now(); + try { + const response = await fetch(url, { + ...init, + signal: AbortSignal.timeout(Math.min(remainingMs, FUNCTION_RESPONSE_ATTEMPT_TIMEOUT_MS)), + }); + const body = await response.text(); + assertResponse(response, body); + return; + } catch (error) { + lastFailure = error; + // Bound request frequency while the worker catches up after a reload; + // the wall-clock deadline, rather than an attempt count, remains the guard. + const retryDelayMs = Math.min( + FUNCTION_RESPONSE_RETRY_BACKOFF_MS, + Math.max(0, deadline - Date.now()), + ); + if (retryDelayMs > 0) { + await new Promise((resolve) => setTimeout(resolve, retryDelayMs)); + } + } } + + throw new Error( + `Function request ${url} did not reach the expected response within ${timeoutMs}ms. ` + + `Last failure: ${lastFailure instanceof Error ? lastFailure.message : String(lastFailure)}`, + ); } describe("supabase functions dev (e2e)", () => { afterEach(cleanupRegisteredStackProjects); test( - "serves a function created while running and applies live config and source changes", + "serves a function created while running and applies config and source changes", { timeout: FUNCTIONS_DEV_TEST_TIMEOUT_MS }, async () => { const home = makeTempHome(); @@ -82,18 +105,13 @@ describe("supabase functions dev (e2e)", () => { FUNCTIONS_DEV_STEP_TIMEOUT_MS, functionOffset, ); - const functionReload = devProc.waitForOutput( - FUNCTION_RELOAD_COMPLETE_PATTERN, - FUNCTIONS_DEV_STEP_TIMEOUT_MS, - functionOffset, - ); const newResult = await runSupabase(["functions", "new", "hello-world"], { cwd: project.dir, home: home.dir, exitTimeoutMs: FUNCTIONS_DEV_STEP_TIMEOUT_MS, }); expect(newResult.exitCode).toBe(0); - await Promise.all([functionRestart, functionReload]); + await functionRestart; await assertFunctionResponse(functionUrl, {}, (response, body) => { expect(response.status).toBe(401); @@ -106,11 +124,6 @@ describe("supabase functions dev (e2e)", () => { FUNCTIONS_DEV_STEP_TIMEOUT_MS, configOffset, ); - const configReload = devProc.waitForOutput( - EDGE_RUNTIME_RELOAD_COMPLETE_PATTERN, - FUNCTIONS_DEV_STEP_TIMEOUT_MS, - configOffset, - ); await writeFile( join(project.dir, "supabase", "config.toml"), `project_id = "functions-dev-e2e" @@ -119,7 +132,7 @@ describe("supabase functions dev (e2e)", () => { verify_jwt = false `, ); - await Promise.all([configRestart, configReload]); + await configRestart; await assertFunctionResponse( functionUrl, @@ -140,11 +153,6 @@ verify_jwt = false FUNCTIONS_DEV_STEP_TIMEOUT_MS, sourceOffset, ); - const sourceReload = devProc.waitForOutput( - FUNCTION_RELOAD_COMPLETE_PATTERN, - FUNCTIONS_DEV_STEP_TIMEOUT_MS, - sourceOffset, - ); await writeFile( functionPath, `Deno.serve(() => { @@ -154,7 +162,7 @@ verify_jwt = false }); `, ); - await Promise.all([sourceRestart, sourceReload]); + await sourceRestart; await assertFunctionResponse( functionUrl, diff --git a/apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts b/apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts index 90ca964dc6..680c67417e 100644 --- a/apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts +++ b/apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts @@ -122,9 +122,6 @@ const ensureFunctionsDirectory = Effect.fnUntraced(function* () { yield* fs.makeDirectory(join(projectHome.supabaseDir, "functions"), { recursive: true }); }); -const EDGE_RUNTIME_RELOAD_COMPLETE = "Edge runtime reload complete."; -const FUNCTIONS_RELOAD_COMPLETE = "Function reload complete."; - function watchEventMatches(spec: FunctionsDevWatchPath, event: FileWatchEvent): boolean { if (spec.names === undefined) { return true; @@ -231,13 +228,11 @@ export const runFunctionsDevRuntime = Effect.fnUntraced(function* ( yield* output.info("Edge runtime config changed. Restarting edge-runtime..."); yield* reloadEdgeRuntime(stack, opts, result.state.config); edgeRuntimeState = result.state; - yield* output.info(EDGE_RUNTIME_RELOAD_COMPLETE); return; } edgeRuntimeState = result.state; yield* output.info("Function files changed. Restarting edge-runtime..."); yield* stack.reloadFunctions({ functions: yield* resolveFunctionsBundle(opts) }); - yield* output.info(FUNCTIONS_RELOAD_COMPLETE); }).pipe( Effect.catch((error) => output.error(error instanceof Error ? error.message : String(error)), diff --git a/apps/cli/src/next/commands/start/start.docker.e2e.test.ts b/apps/cli/src/next/commands/start/start.docker.e2e.test.ts index 724248c0c5..4771849e5b 100644 --- a/apps/cli/src/next/commands/start/start.docker.e2e.test.ts +++ b/apps/cli/src/next/commands/start/start.docker.e2e.test.ts @@ -30,7 +30,7 @@ const LIGHTWEIGHT_DOCKER_ARGS = [ ] as const; // Lazy service activation crosses the real proxy, daemon, Docker network, and -// container lifecycle boundaries, so keep one gated golden-path live test. +// container lifecycle boundaries, so keep one golden-path Docker e2e test. describe("supabase start lazy lifecycle (e2e)", () => { let project: Awaited> | undefined; let home: ReturnType | undefined; @@ -45,7 +45,7 @@ describe("supabase start lazy lifecycle (e2e)", () => { "keeps an HTTP service dormant until its first proxied request", { timeout: START_TIMEOUT_MS + 120_000 }, async () => { - project = await makeTempStackProject("supabase-lazy-start-live-"); + project = await makeTempStackProject("supabase-lazy-start-e2e-"); home = makeTempHome(); const started = await runSupabase([...LIGHTWEIGHT_DOCKER_ARGS], { diff --git a/apps/cli/src/shared/runtime/stack-e2e-cleanup.unit.test.ts b/apps/cli/src/shared/runtime/stack-e2e-cleanup.unit.test.ts index 4d3a374a95..a25d1f13bf 100644 --- a/apps/cli/src/shared/runtime/stack-e2e-cleanup.unit.test.ts +++ b/apps/cli/src/shared/runtime/stack-e2e-cleanup.unit.test.ts @@ -112,6 +112,44 @@ describe("stack e2e cleanup manager", () => { } }); + it("preserves project and home cleanup receivers", async () => { + class ReceiverHome { + readonly dir = "/tmp/home"; + disposed = false; + + dispose() { + this.disposed = true; + } + } + + class ReceiverProject { + readonly dir = "/tmp/project"; + cleaned = false; + + async cleanup() { + this.cleaned = true; + } + } + + const home = new ReceiverHome(); + const project = new ReceiverProject(); + const manager = createStackE2eCleanupManager(cleanupEnvironment([])); + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + try { + manager.registerHome(home); + manager.registerStackProject(project); + manager.associateHome(project.dir, home.dir); + + await manager.drain(); + + expect(project.cleaned).toBe(true); + expect(home.disposed).toBe(true); + expect(warn).not.toHaveBeenCalled(); + } finally { + warn.mockRestore(); + } + }); + it("ignores non-stack homes", async () => { const calls: Array = []; const manager = createStackE2eCleanupManager(cleanupEnvironment(calls)); @@ -226,6 +264,102 @@ describe("stack e2e cleanup manager", () => { expect(calls).toEqual(["cleanup-project", "docker-remove"]); }); + it("removes permission-blocked associated homes with the Docker root fallback", async () => { + const calls: Array = []; + const manager = createStackE2eCleanupManager( + cleanupEnvironment(calls, { + removeProjectWithDocker: async () => { + calls.push("docker-remove"); + return true; + }, + }), + ); + + manager.registerHome({ + dir: "/tmp/home", + dispose: () => { + calls.push("dispose-home"); + throw permissionError(); + }, + }); + manager.registerStackProject({ + dir: "/tmp/project", + cleanup: async () => { + calls.push("cleanup-project"); + }, + }); + manager.associateHome("/tmp/project", "/tmp/home"); + + await expect(manager.drain()).resolves.toBeUndefined(); + + expect(calls).toEqual(["cleanup-project", "dispose-home", "docker-remove"]); + }); + + it("warns when an associated home remains after permission fallback", async () => { + const calls: Array = []; + const manager = createStackE2eCleanupManager(cleanupEnvironment(calls)); + + manager.registerHome({ + dir: "/tmp/home", + dispose: () => { + calls.push("dispose-home"); + throw permissionError(); + }, + }); + manager.registerStackProject({ + dir: "/tmp/project", + cleanup: async () => { + calls.push("cleanup-project"); + }, + }); + manager.associateHome("/tmp/project", "/tmp/home"); + + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + try { + await expect(manager.drain()).resolves.toBeUndefined(); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("Failed to remove temp home")); + } finally { + warn.mockRestore(); + } + expect(calls).toEqual([ + "cleanup-project", + "dispose-home", + "docker-remove", + "chmod", + "dispose-home", + ]); + }); + + it("disposes an associated home once after all projects sharing it are cleaned", async () => { + const calls: Array = []; + const manager = createStackE2eCleanupManager(cleanupEnvironment(calls)); + + manager.registerHome({ + dir: "/tmp/home", + dispose: () => { + calls.push("dispose-home"); + }, + }); + manager.registerStackProject({ + dir: "/tmp/project-one", + cleanup: async () => { + calls.push("cleanup-project-one"); + }, + }); + manager.registerStackProject({ + dir: "/tmp/project-two", + cleanup: async () => { + calls.push("cleanup-project-two"); + }, + }); + manager.associateHome("/tmp/project-one", "/tmp/home"); + manager.associateHome("/tmp/project-two", "/tmp/home"); + + await manager.drain(); + + expect(calls).toEqual(["cleanup-project-one", "cleanup-project-two", "dispose-home"]); + }); + it("falls back to chmod and retries cleanup when Docker cannot remove the project", async () => { const calls: Array = []; let attempts = 0; diff --git a/apps/cli/tests/helpers/stack-e2e-cleanup.ts b/apps/cli/tests/helpers/stack-e2e-cleanup.ts index b6815fecc8..8d571f89f7 100644 --- a/apps/cli/tests/helpers/stack-e2e-cleanup.ts +++ b/apps/cli/tests/helpers/stack-e2e-cleanup.ts @@ -139,8 +139,12 @@ function readDocumentPid(documentFile: string): number | undefined { } } -function cleanupErrorDetail(projectDir: string, error: unknown): string { - return `Failed to remove temp stack project ${projectDir}: ${ +function cleanupErrorDetail( + pathname: string, + error: unknown, + resource = "temp stack project", +): string { + return `Failed to remove ${resource} ${pathname}: ${ error instanceof Error ? error.message : String(error) }`; } @@ -222,38 +226,49 @@ async function removeProjectWithDocker(projectDir: string): Promise { return removed; } -async function cleanupProject( - project: StackProject, +async function cleanupOwnedPath( + pathname: string, + cleanup: () => void | Promise, environment: Pick< CleanupEnvironment, "removeProjectWithDocker" | "repairProjectPermissions" | "describeProjectPermissions" >, ): Promise { try { - await project.cleanup(); + await cleanup(); } catch (error) { if (!isPermissionError(error)) { throw error; } - const removedByDocker = await environment.removeProjectWithDocker(project.dir); + const removedByDocker = await environment.removeProjectWithDocker(pathname); if (removedByDocker) { return; } - environment.repairProjectPermissions(project.dir); + environment.repairProjectPermissions(pathname); try { - await project.cleanup(); + await cleanup(); } catch (retryError) { throw new Error( `${retryError instanceof Error ? retryError.message : String(retryError)}\n${environment.describeProjectPermissions( - project.dir, + pathname, )}`, ); } } } +async function cleanupProject( + project: StackProject, + environment: Pick< + CleanupEnvironment, + "removeProjectWithDocker" | "repairProjectPermissions" | "describeProjectPermissions" + >, +): Promise { + await cleanupOwnedPath(project.dir, project.cleanup, environment); +} + function captureSnapshot(projectDir: string, homeDir?: string): StackRuntimeSnapshot { const normalized = normalizeDir(projectDir); const managedStacksRoot = @@ -382,12 +397,17 @@ export function createStackE2eCleanupManager( return { registerHome(home) { - homes.set(normalizeDir(home.dir), home); + const dir = normalizeDir(home.dir); + homes.set(dir, { + dir, + dispose: () => home.dispose(), + }); }, registerStackProject(project) { - projects.set(normalizeDir(project.dir), { - dir: normalizeDir(project.dir), - cleanup: project.cleanup, + const dir = normalizeDir(project.dir); + projects.set(dir, { + dir, + cleanup: () => project.cleanup(), }); }, associateHome(projectDir, homeDir) { @@ -403,9 +423,14 @@ export function createStackE2eCleanupManager( homes.clear(); const failures: Array = []; + const associatedHomes = new Map(); for (const project of pendingProjects) { - const home = project.homeDir ? pendingHomes.get(project.homeDir) : undefined; + const homeDir = project.homeDir; + const home = homeDir === undefined ? undefined : pendingHomes.get(homeDir); + if (home !== undefined && homeDir !== undefined) { + associatedHomes.set(homeDir, home); + } const snapshot = environment.captureSnapshot(project.dir, project.homeDir); const hasRuntimeArtifacts = snapshot.documentFiles.length > 0 || @@ -439,14 +464,18 @@ export function createStackE2eCleanupManager( await cleanupProject(project, environment); } catch (error) { failures.push(cleanupErrorDetail(project.dir, error)); - } finally { - if (home !== undefined) { - home.dispose(); - } } } - // Cleanup of leaked stack projects is best-effort: assertions in the + for (const home of associatedHomes.values()) { + try { + await cleanupOwnedPath(home.dir, home.dispose, environment); + } catch (error) { + failures.push(cleanupErrorDetail(home.dir, error, "temp home")); + } + } + + // Cleanup of leaked stack resources is best-effort: assertions in the // test itself have already passed by the time `drain()` runs, and CI // runners are ephemeral so a leaked temp dir doesn't affect // correctness. Surface the details so developers can still see them @@ -457,7 +486,7 @@ export function createStackE2eCleanupManager( // sandbox). if (failures.length > 0) { console.warn( - `[stack-e2e-cleanup] ${failures.length} project(s) could not be cleaned up:\n${failures.join("\n")}`, + `[stack-e2e-cleanup] ${failures.length} resource(s) could not be cleaned up:\n${failures.join("\n")}`, ); } }, From 375c55aac0b18bf83465f4888805ba691e3f2d09 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Sat, 22 Aug 2026 01:06:21 +0200 Subject: [PATCH 20/23] test(cli): align Docker e2e fixtures --- .../commands/db/diff/diff.docker.e2e.test.ts | 23 ++++++++++++------- .../declarative/sync/sync.docker.e2e.test.ts | 8 +++++++ .../functions/dev/dev.docker.e2e.test.ts | 2 +- 3 files changed, 24 insertions(+), 9 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts index 3662c5e293..cf84a20f7a 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts @@ -53,22 +53,27 @@ describe("supabase db diff (e2e, pg-delta declarative privileges)", () => { // Minimal, deterministic repro: execute a fresh function's implicit PUBLIC // EXECUTE grant, explicitly revoked, directly against the local database. - // `db query` is setup only; the command under test remains `db diff`. - const query = await runSupabase( + // `db query` is setup only; keep each statement in its own invocation + // because the legacy query command sends one prepared statement at a time. + const createFunction = await runSupabase( [ "db", "query", `create function public.probe_fn() returns void language sql -as $$ select 1; $$; - -revoke execute on function public.probe_fn() from public;`, +as $$ select 1; $$;`, "--local", ], { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, ); - requireCliSuccess(query, "db query setup"); + requireCliSuccess(createFunction, "db query create-function setup"); + + const revoke = await runSupabase( + ["db", "query", "revoke execute on function public.probe_fn() from public;", "--local"], + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + ); + requireCliSuccess(revoke, "db query revoke setup"); const diff = await runSupabase( ["db", "diff", "--local", "--use-pg-delta", "-f", "revoke_public_execute"], @@ -88,9 +93,11 @@ revoke execute on function public.probe_fn() from public;`, // survives). Anchor the match to the function's own REVOKE statement — up to // its terminating `;` — so this cannot pass on an unrelated PUBLIC mention // elsewhere in the file. - expect(sql).toContain("CREATE FUNCTION public.probe_fn()"); expect(sql).toMatch( - /REVOKE\s+(?:ALL|EXECUTE)\s+ON\s+FUNCTION\s+public\.probe_fn\(\)\s+FROM\s+[^;]*PUBLIC[^;]*;/i, + /CREATE(?:\s+OR\s+REPLACE)?\s+FUNCTION\s+"?public"?\s*\.\s*"?probe_fn"?\s*\(\)/i, + ); + expect(sql).toMatch( + /REVOKE\s+(?:ALL|EXECUTE)\s+ON\s+FUNCTION\s+"?public"?\s*\.\s*"?probe_fn"?\s*\(\)\s+FROM\s+[^;]*PUBLIC[^;]*;/i, ); }, ); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts index 06587a8fd6..ceb3157049 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts @@ -65,6 +65,14 @@ describe("db schema declarative sync (e2e)", () => { const schemasDir = path.join(projectDir, "supabase", "schemas"); mkdirSync(schemasDir, { recursive: true }); writeFileSync(path.join(schemasDir, "public.sql"), initialDesiredSchema); + const extensionsDir = path.join(schemasDir, "cluster", "extensions"); + mkdirSync(extensionsDir, { recursive: true }); + for (const extension of ["pg_net", "pgcrypto", "uuid-ossp"]) { + writeFileSync( + path.join(extensionsDir, `${extension}.sql`), + `CREATE EXTENSION IF NOT EXISTS "${extension}" WITH SCHEMA "extensions";\n`, + ); + } const start = await runSupabase( [ diff --git a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts index 2a5f6461f5..a4440d2dbf 100644 --- a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts +++ b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts @@ -120,7 +120,7 @@ describe("supabase functions dev (e2e)", () => { const configOffset = devProc.stdout().length; const configRestart = devProc.waitForOutput( - /Edge runtime config changed\. Restarting edge-runtime\./, + FUNCTION_FILES_RESTART_PATTERN, FUNCTIONS_DEV_STEP_TIMEOUT_MS, configOffset, ); From 7e08e3ec853bc2ff0785bfb718373fbd655753d6 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Sat, 22 Aug 2026 01:30:15 +0200 Subject: [PATCH 21/23] test(cli): harden e2e lifecycle checks --- CONTRIBUTING.md | 5 +- .../db/start/start.docker.e2e.test.ts | 52 +++++++++++-------- .../functions/dev/dev.docker.e2e.test.ts | 6 ++- .../commands/start/start.docker.e2e.test.ts | 28 ++++++++-- apps/cli/tests/helpers/live-project.ts | 36 +++++++------ .../tests/helpers/live-project.unit.test.ts | 10 ++++ 6 files changed, 93 insertions(+), 44 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 17d6bed3b1..fe01bcedb5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -199,11 +199,10 @@ from the provisioned project's database metadata. Live coverage is smoke coverage, not an exhaustive command matrix. Add one representative golden-path test for each user-facing command, colocated beside that command. A live test should assert one target command; setup and teardown may invoke other commands when they prepare or clean up state, but those commands are not asserted in that test. Keep validation, formatting, fallback, error, and matrix details in integration tests unless the remote/runtime boundary itself is the behavior under test. See [ADR 0013](docs/adr/0013-live-e2e-bypasses-replay-server.md) and [`apps/cli/live.env.example`](apps/cli/live.env.example). -To run the live suite locally, copy [`apps/cli/live.env.example`](apps/cli/live.env.example), set the API URL and access token for the target platform, and run: +To run the live suite locally, copy [`apps/cli/live.env.example`](apps/cli/live.env.example), set the API URL and access token for the target platform, and run the Nx target from the repository root. The target's build dependency prepares the CLI artifacts before Vitest starts: ```sh -cd apps/cli -pnpm test:live +pnpm exec nx run supabase:test:live ``` Optional `SUPABASE_LIVE_ORG_ID`, `SUPABASE_LIVE_REGION`, and diff --git a/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts index 48681124f4..46ffcec0b3 100644 --- a/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts @@ -6,26 +6,36 @@ import { runSupabase, } from "../../../../../tests/helpers/cli.ts"; +const DB_START_COMMAND_TIMEOUT_MS = 480_000; +const DB_START_CLEANUP_TIMEOUT_MS = 120_000; +const DB_START_TEST_TIMEOUT_MS = DB_START_COMMAND_TIMEOUT_MS + DB_START_CLEANUP_TIMEOUT_MS; + describe("supabase db start (e2e)", () => { - test("boots the local database", async () => { - const home = makeTempHome(); - const project = await makeTempStackProject("supabase-db-start-e2e-"); - try { - const started = await runSupabase(["db", "start"], { - entrypoint: "legacy", - cwd: project.dir, - home: home.dir, - }); - expect(started.exitCode, started.stderr).toBe(0); - expect(`${started.stdout}${started.stderr}`).toMatch( - /Starting database|Initialising schema/i, - ); - } finally { - await runSupabase(["stop", "--no-backup"], { - entrypoint: "legacy", - cwd: project.dir, - home: home.dir, - }).catch(() => undefined); - } - }, 600_000); + test( + "boots the local database", + async () => { + const home = makeTempHome(); + const project = await makeTempStackProject("supabase-db-start-e2e-"); + try { + const started = await runSupabase(["db", "start"], { + entrypoint: "legacy", + cwd: project.dir, + home: home.dir, + exitTimeoutMs: DB_START_COMMAND_TIMEOUT_MS, + }); + expect(started.exitCode, started.stderr).toBe(0); + expect(`${started.stdout}${started.stderr}`).toMatch( + /Starting database|Initialising schema/i, + ); + } finally { + await runSupabase(["stop", "--no-backup"], { + entrypoint: "legacy", + cwd: project.dir, + home: home.dir, + exitTimeoutMs: DB_START_CLEANUP_TIMEOUT_MS, + }).catch(() => undefined); + } + }, + DB_START_TEST_TIMEOUT_MS, + ); }); diff --git a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts index a4440d2dbf..be5905217b 100644 --- a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts +++ b/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts @@ -12,7 +12,11 @@ import { cleanupRegisteredStackProjects } from "../../../../../tests/helpers/sta const FUNCTIONS_DEV_STARTUP_TIMEOUT_MS = 60_000; const FUNCTIONS_DEV_STEP_TIMEOUT_MS = 30_000; -const FUNCTIONS_DEV_TEST_TIMEOUT_MS = 90_000; +const FUNCTIONS_DEV_CLEANUP_TIMEOUT_MS = 30_000; +const FUNCTIONS_DEV_TEST_TIMEOUT_MS = + FUNCTIONS_DEV_STARTUP_TIMEOUT_MS + + FUNCTIONS_DEV_STEP_TIMEOUT_MS * 7 + + FUNCTIONS_DEV_CLEANUP_TIMEOUT_MS; const FUNCTION_RESPONSE_ATTEMPT_TIMEOUT_MS = 5_000; const FUNCTION_RESPONSE_RETRY_BACKOFF_MS = 250; const FUNCTION_FILES_RESTART_PATTERN = /Function files changed\. Restarting edge-runtime\./; diff --git a/apps/cli/src/next/commands/start/start.docker.e2e.test.ts b/apps/cli/src/next/commands/start/start.docker.e2e.test.ts index 4771849e5b..898331795e 100644 --- a/apps/cli/src/next/commands/start/start.docker.e2e.test.ts +++ b/apps/cli/src/next/commands/start/start.docker.e2e.test.ts @@ -1,5 +1,7 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { basename, join } from "node:path"; import { afterEach, describe, expect, test } from "vitest"; -import { makeTempHome, makeTempStackProject, runSupabase } from "../../../../tests/helpers/cli.ts"; +import { makeTempCliProject, makeTempHome, runSupabase } from "../../../../tests/helpers/cli.ts"; import { cleanupRegisteredStackProjects } from "../../../../tests/helpers/stack-e2e-cleanup.ts"; const START_TIMEOUT_MS = 180_000; @@ -32,7 +34,7 @@ const LIGHTWEIGHT_DOCKER_ARGS = [ // Lazy service activation crosses the real proxy, daemon, Docker network, and // container lifecycle boundaries, so keep one golden-path Docker e2e test. describe("supabase start lazy lifecycle (e2e)", () => { - let project: Awaited> | undefined; + let project: Awaited> | undefined; let home: ReturnType | undefined; afterEach(async () => { @@ -45,8 +47,16 @@ describe("supabase start lazy lifecycle (e2e)", () => { "keeps an HTTP service dormant until its first proxied request", { timeout: START_TIMEOUT_MS + 120_000 }, async () => { - project = await makeTempStackProject("supabase-lazy-start-e2e-"); + project = await makeTempCliProject("supabase-lazy-start-e2e-"); home = makeTempHome(); + await mkdir(join(project.dir, "supabase"), { recursive: true }); + const projectId = basename(project.dir) + .replace(/[^a-z0-9]/giu, "") + .toLowerCase(); + await writeFile( + join(project.dir, "supabase", "config.toml"), + `project_id = "${projectId}"\n`, + ); const started = await runSupabase([...LIGHTWEIGHT_DOCKER_ARGS], { ...COMMAND_OPTIONS, @@ -64,7 +74,17 @@ describe("supabase start lazy lifecycle (e2e)", () => { expect(before.exitCode, `stdout:\n${before.stdout}\nstderr:\n${before.stderr}`).toBe(0); expect(before.stdout).toContain("auth: Dormant"); - const response = await fetch(`http://127.0.0.1:${project.ports.apiPort}/auth/v1/health`, { + const apiUrlMatch = + `${started.stdout}\n${started.stderr}\n${before.stdout}\n${before.stderr}`.match( + /API URL:\s+(https?:\/\/[^\s]+)/, + ); + if (apiUrlMatch?.[1] === undefined) { + throw new Error( + `Start/status output did not include an API URL.\nstdout:\n${before.stdout}\nstderr:\n${before.stderr}`, + ); + } + + const response = await fetch(`${apiUrlMatch[1]}/auth/v1/health`, { signal: AbortSignal.timeout(60_000), }); expect(response.ok).toBe(true); diff --git a/apps/cli/tests/helpers/live-project.ts b/apps/cli/tests/helpers/live-project.ts index 8246a45398..7beededf64 100644 --- a/apps/cli/tests/helpers/live-project.ts +++ b/apps/cli/tests/helpers/live-project.ts @@ -97,10 +97,11 @@ function apiError(error: unknown): Error { return error instanceof Error ? error : new Error(String(error)); } -function supportedRegion(value: string): Region { +export function supportedRegion(value: string): Effect.Effect { const region = REGIONS.find((candidate) => candidate === value); - if (region !== undefined) return region; - throw new Error(`Unsupported SUPABASE_LIVE_REGION ${JSON.stringify(value)}`); + return region === undefined + ? Effect.fail(new Error(`Unsupported SUPABASE_LIVE_REGION ${JSON.stringify(value)}`)) + : Effect.succeed(region); } /** HTTP statuses that can occur while a newly-created project propagates. */ @@ -220,17 +221,21 @@ function createProject( name: string, password: string, ): Effect.Effect { - return resolveOrganization(api).pipe( - Effect.flatMap((organization) => - timeoutLiveRequest( - "project creation", - api.v1.createAProject({ - name, - db_pass: password, - organization_slug: organization.slug, - region: supportedRegion(liveRegion()), - }), - ).pipe(Effect.mapError(apiError)), + return supportedRegion(liveRegion()).pipe( + Effect.flatMap((region) => + resolveOrganization(api).pipe( + Effect.flatMap((organization) => + timeoutLiveRequest( + "project creation", + api.v1.createAProject({ + name, + db_pass: password, + organization_slug: organization.slug, + region, + }), + ).pipe(Effect.mapError(apiError)), + ), + ), ), Effect.flatMap((project) => PROJECT_REF_RE.test(project.ref) @@ -399,11 +404,12 @@ function createStorageBucket( bucket: string, ): Effect.Effect { const attempt = Effect.tryPromise({ - try: async () => { + try: async (signal) => { const response = await fetch(`https://${ref}.${host}/storage/v1/bucket`, { method: "POST", headers: { Authorization: `Bearer ${serviceRoleKey}`, "Content-Type": "application/json" }, body: JSON.stringify({ id: bucket, name: bucket, public: false }), + signal, }); if (!response.ok && response.status !== 409) { throw new LiveStorageError({ diff --git a/apps/cli/tests/helpers/live-project.unit.test.ts b/apps/cli/tests/helpers/live-project.unit.test.ts index 9ff3cea4ed..4da9bd2bc0 100644 --- a/apps/cli/tests/helpers/live-project.unit.test.ts +++ b/apps/cli/tests/helpers/live-project.unit.test.ts @@ -11,6 +11,7 @@ import { resolvePoolerDatabaseUrl, retryLiveEffect, selectPrimaryPoolerConfig, + supportedRegion, type PoolerConfig, } from "./live-project.ts"; @@ -41,6 +42,15 @@ function poolerConfig(overrides: Partial = {}): PoolerConfig { } describe("live project lifecycle", () => { + it("fails invalid regions before provisioning", async () => { + await expect( + Effect.runPromise(Effect.flip(supportedRegion("not-a-region"))), + ).resolves.toMatchObject({ + message: expect.stringContaining("Unsupported SUPABASE_LIVE_REGION"), + }); + await expect(Effect.runPromise(supportedRegion("us-east-1"))).resolves.toBe("us-east-1"); + }); + it("retries transient failures until the management operation succeeds", async () => { let attempts = 0; const result = await Effect.runPromise( From 08aadb12637841861ea155f133e66ec7d623bf4a Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Sat, 22 Aug 2026 02:05:57 +0200 Subject: [PATCH 22/23] test(cli): own legacy e2e lifecycle cleanup --- .../commands/db/diff/diff.docker.e2e.test.ts | 57 +++++---- .../declarative/sync/sync.docker.e2e.test.ts | 49 ++++--- .../commands/start/start.docker.e2e.test.ts | 13 +- .../commands/status/status.docker.e2e.test.ts | 56 +++++--- .../commands/stop/stop.docker.e2e.test.ts | 121 +++++++++++------- apps/cli/tests/helpers/cli.ts | 39 ++++++ 6 files changed, 218 insertions(+), 117 deletions(-) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts index cf84a20f7a..2d8a044702 100644 --- a/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts @@ -1,13 +1,26 @@ import { existsSync, readdirSync, readFileSync } from "node:fs"; -import { mkdtemp, rm } from "node:fs/promises"; -import { tmpdir } from "node:os"; import path from "node:path"; import { afterEach, expect, test } from "vitest"; import { describe } from "vitest"; -import { requireCliSuccess, runSupabase } from "../../../../../tests/helpers/cli.ts"; +import { + makeTempLegacyStackProject, + requireCliSuccess, + runSupabase, +} from "../../../../../tests/helpers/cli.ts"; -const START_TIMEOUT_MS = 280_000; +const CLI_COMMAND_TIMEOUT_MS = 60_000; +const STACK_START_TIMEOUT_MS = 280_000; +const DIFF_COMMAND_TIMEOUT_MS = 280_000; +const CLEANUP_TIMEOUT_MS = 120_000; +const LIFECYCLE_MARGIN_MS = 30_000; +const CLEANUP_HOOK_TIMEOUT_MS = CLEANUP_TIMEOUT_MS + LIFECYCLE_MARGIN_MS; +const DIFF_TEST_TIMEOUT_MS = + CLI_COMMAND_TIMEOUT_MS + + STACK_START_TIMEOUT_MS + + CLI_COMMAND_TIMEOUT_MS * 2 + + DIFF_COMMAND_TIMEOUT_MS + + LIFECYCLE_MARGIN_MS; // CLI-1947 regression: pg-delta's `filterPublicBuiltInDefaults()` unconditionally // treated PUBLIC's implicit built-in privilege as a no-op on both sides of a diff, @@ -20,34 +33,32 @@ const START_TIMEOUT_MS = 280_000; // Docker-stack e2e coverage and never calls the Management API. See AGENTS.md's // "E2e tests" section. describe("supabase db diff (e2e, pg-delta declarative privileges)", () => { - let projectDir: string | undefined; + let project: Awaited> | undefined; afterEach(async () => { - if (projectDir === undefined) return; - // Best-effort cleanup even if an assertion above failed mid-lifecycle — a - // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabase(["stop", "--no-backup"], { - entrypoint: "legacy", - cwd: projectDir, - }).catch(() => undefined); - await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); - projectDir = undefined; - }); + await project?.cleanup().catch(() => undefined); + project = undefined; + }, CLEANUP_HOOK_TIMEOUT_MS); test( "keeps REVOKE ... FROM PUBLIC on a function when diffing a declarative schema against local", - { timeout: START_TIMEOUT_MS }, + { timeout: DIFF_TEST_TIMEOUT_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-db-diff-e2e-")); + project = await makeTempLegacyStackProject("sb-db-diff-e2e-"); + const projectDir = project.dir; - const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir }); + const init = await runSupabase(["init"], { + entrypoint: "legacy", + cwd: projectDir, + exitTimeoutMs: CLI_COMMAND_TIMEOUT_MS, + }); requireCliSuccess(init, "init setup"); // Exclude the heaviest, least relevant services — `db diff` only needs the // local Postgres container reachable, same rationale as stop/status. const start = await runSupabase( - ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + ["start", "--exclude", "studio", "--exclude", "logflare", "--exclude", "vector"], + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: STACK_START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); @@ -65,19 +76,19 @@ language sql as $$ select 1; $$;`, "--local", ], - { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: CLI_COMMAND_TIMEOUT_MS }, ); requireCliSuccess(createFunction, "db query create-function setup"); const revoke = await runSupabase( ["db", "query", "revoke execute on function public.probe_fn() from public;", "--local"], - { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: CLI_COMMAND_TIMEOUT_MS }, ); requireCliSuccess(revoke, "db query revoke setup"); const diff = await runSupabase( ["db", "diff", "--local", "--use-pg-delta", "-f", "revoke_public_execute"], - { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: DIFF_COMMAND_TIMEOUT_MS }, ); expect(diff.exitCode, `stdout:\n${diff.stdout}\nstderr:\n${diff.stderr}`).toBe(0); diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts index ceb3157049..f39d724f50 100644 --- a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts @@ -1,13 +1,21 @@ import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; -import { mkdtemp, rm } from "node:fs/promises"; -import { tmpdir } from "node:os"; import path from "node:path"; import { afterAll, beforeAll, expect, test } from "vitest"; import { describe } from "vitest"; -import { requireCliSuccess, runSupabase } from "../../../../../../../tests/helpers/cli.ts"; - -const COMMAND_TIMEOUT_MS = 280_000; +import { + makeTempLegacyStackProject, + requireCliSuccess, + runSupabase, +} from "../../../../../../../tests/helpers/cli.ts"; + +const CLI_COMMAND_TIMEOUT_MS = 60_000; +const STACK_START_TIMEOUT_MS = 280_000; +const CLEANUP_TIMEOUT_MS = 120_000; +const LIFECYCLE_MARGIN_MS = 30_000; +const CLEANUP_HOOK_TIMEOUT_MS = CLEANUP_TIMEOUT_MS + LIFECYCLE_MARGIN_MS; +const SCENARIO_COMMAND_TIMEOUT_MS = 280_000; +const BEFORE_ALL_TIMEOUT_MS = CLI_COMMAND_TIMEOUT_MS + STACK_START_TIMEOUT_MS + LIFECYCLE_MARGIN_MS; const SCENARIO_TIMEOUT_MS = 900_000; const NEXT_ENV = { SUPABASE_USE_PG_DELTA_NEXT: "true" }; @@ -37,15 +45,16 @@ function migrationFiles(projectDir: string): ReadonlyArray { } describe("db schema declarative sync (e2e)", () => { - let projectDir = ""; + let project: Awaited> | undefined; beforeAll(async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-pgdelta-next-e2e-")); + project = await makeTempLegacyStackProject("sb-pgdelta-next-e2e-"); + const projectDir = project.dir; const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, + exitTimeoutMs: CLI_COMMAND_TIMEOUT_MS, }); requireCliSuccess(init, "init setup"); @@ -90,25 +99,23 @@ describe("db schema declarative sync (e2e)", () => { "--exclude", "storage-api", ], - { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: COMMAND_TIMEOUT_MS }, + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: STACK_START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); - }, COMMAND_TIMEOUT_MS); + }, BEFORE_ALL_TIMEOUT_MS); afterAll(async () => { - if (projectDir.length === 0) return; - await runSupabase(["stop", "--no-backup"], { - entrypoint: "legacy", - cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, - }).catch(() => undefined); - await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); - }, COMMAND_TIMEOUT_MS); + await project?.cleanup().catch(() => undefined); + project = undefined; + }, CLEANUP_HOOK_TIMEOUT_MS); test( "applies a representative declarative schema and converges", { timeout: SCENARIO_TIMEOUT_MS }, async () => { + const projectDir = project?.dir; + if (projectDir === undefined) throw new Error("declarative sync project was not initialized"); + const sync = await runSupabase( [ "db", @@ -124,7 +131,7 @@ describe("db schema declarative sync (e2e)", () => { entrypoint: "legacy", cwd: projectDir, env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, + exitTimeoutMs: SCENARIO_COMMAND_TIMEOUT_MS, }, ); expect(sync.exitCode, commandFailure(sync)).toBe(0); @@ -145,7 +152,7 @@ describe("db schema declarative sync (e2e)", () => { const reset = await runSupabase(["db", "reset", "--local", "--no-seed"], { entrypoint: "legacy", cwd: projectDir, - exitTimeoutMs: COMMAND_TIMEOUT_MS, + exitTimeoutMs: SCENARIO_COMMAND_TIMEOUT_MS, }); requireCliSuccess(reset, "db reset setup"); @@ -155,7 +162,7 @@ describe("db schema declarative sync (e2e)", () => { entrypoint: "legacy", cwd: projectDir, env: NEXT_ENV, - exitTimeoutMs: COMMAND_TIMEOUT_MS, + exitTimeoutMs: SCENARIO_COMMAND_TIMEOUT_MS, }, ); expect(converged.exitCode, commandFailure(converged)).toBe(0); diff --git a/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts b/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts index bbb658f5e0..2bc0483e9c 100644 --- a/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts @@ -25,16 +25,9 @@ const LIFECYCLE_OVERHEAD_MS = 90_000; /** * `--exclude` values for the 3 heaviest/least-relevant services — same intent - * `stop.e2e.test.ts`/`status.e2e.test.ts` already documented for their own - * reduced-stack `start` call (Studio's Next.js build, the Logflare/Vector - * logging pipeline), but "logflare" here, NOT "analytics" like those two - * siblings. `LEGACY_SERVICE_CATALOG`'s `excludeKey` for the logflare service - * is "logflare" — "analytics" is only that service's *container suffix* - * (`legacy-service-catalog.ts`), never a valid `--exclude` value. The - * siblings' `--exclude analytics` is a silent no-op — harmless for their own - * coarse "is the stack up/down" assertions, but this suite's exact- - * container-set assertions need the genuinely valid key so logflare is - * actually excluded. + * as the reduced-stack `start` calls in the sibling Docker e2e suites (Studio's + * Next.js build and the Logflare/Vector logging pipeline). The legacy service + * catalog uses `logflare` as the exclusion key for that logging service. */ const EXCLUDED_SERVICE_KEYS: ReadonlySet = new Set(["studio", "logflare", "vector"]); diff --git a/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts b/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts index 9d52635a37..5b4e47413d 100644 --- a/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts @@ -1,46 +1,61 @@ -import { mkdtemp, rm } from "node:fs/promises"; -import { tmpdir } from "node:os"; -import path from "node:path"; import { afterEach, expect, test } from "vitest"; import { describe } from "vitest"; -import { requireCliSuccess, runSupabase } from "../../../../tests/helpers/cli.ts"; +import { + makeTempLegacyStackProject, + requireCliSuccess, + runSupabase, +} from "../../../../tests/helpers/cli.ts"; -const START_TIMEOUT_MS = 280_000; +const CLI_COMMAND_TIMEOUT_MS = 60_000; +const STACK_START_TIMEOUT_MS = 280_000; +const STATUS_COMMAND_TIMEOUT_MS = 60_000; +const CLEANUP_TIMEOUT_MS = 120_000; +const LIFECYCLE_MARGIN_MS = 30_000; +const CLEANUP_HOOK_TIMEOUT_MS = CLEANUP_TIMEOUT_MS + LIFECYCLE_MARGIN_MS; +const STATUS_TEST_TIMEOUT_MS = + CLI_COMMAND_TIMEOUT_MS + + STACK_START_TIMEOUT_MS + + STATUS_COMMAND_TIMEOUT_MS * 2 + + LIFECYCLE_MARGIN_MS; // See stop.e2e.test.ts for why `describe` (not a Management-API gate) is // the right reuse here: `status` never calls the Management API, only the real // Docker daemon the cli-e2e-ci runner provides. See AGENTS.md's "e2e tests" // section for the full convention. describe("supabase status (e2e)", () => { - let projectDir: string | undefined; + let project: Awaited> | undefined; afterEach(async () => { - if (projectDir === undefined) return; - await runSupabase(["stop", "--no-backup"], { - entrypoint: "legacy", - cwd: projectDir, - }).catch(() => undefined); - await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); - projectDir = undefined; - }); + await project?.cleanup().catch(() => undefined); + project = undefined; + }, CLEANUP_HOOK_TIMEOUT_MS); test( "reports a running local stack in pretty and json modes", - { timeout: START_TIMEOUT_MS }, + { timeout: STATUS_TEST_TIMEOUT_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-status-e2e-")); + project = await makeTempLegacyStackProject("sb-status-e2e-"); + const projectDir = project.dir; - const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir }); + const init = await runSupabase(["init"], { + entrypoint: "legacy", + cwd: projectDir, + exitTimeoutMs: CLI_COMMAND_TIMEOUT_MS, + }); requireCliSuccess(init, "init setup"); const start = await runSupabase( - ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + ["start", "--exclude", "studio", "--exclude", "logflare", "--exclude", "vector"], + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: STACK_START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); - const pretty = await runSupabase(["status"], { entrypoint: "legacy", cwd: projectDir }); + const pretty = await runSupabase(["status"], { + entrypoint: "legacy", + cwd: projectDir, + exitTimeoutMs: STATUS_COMMAND_TIMEOUT_MS, + }); expect(pretty.exitCode, `stdout:\n${pretty.stdout}\nstderr:\n${pretty.stderr}`).toBe(0); expect(`${pretty.stdout}${pretty.stderr}`).toContain("is running"); expect(pretty.stdout).toContain("Project URL"); @@ -49,6 +64,7 @@ describe("supabase status (e2e)", () => { const json = await runSupabase(["status", "-o", "json"], { entrypoint: "legacy", cwd: projectDir, + exitTimeoutMs: STATUS_COMMAND_TIMEOUT_MS, }); expect(json.exitCode, `stdout:\n${json.stdout}\nstderr:\n${json.stderr}`).toBe(0); const parsed: unknown = JSON.parse(json.stdout); diff --git a/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts b/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts index a1c7f44dab..6e18d79786 100644 --- a/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts +++ b/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts @@ -1,16 +1,31 @@ import { execFile } from "node:child_process"; -import { mkdtemp, rm } from "node:fs/promises"; -import { tmpdir } from "node:os"; import path from "node:path"; import { promisify } from "node:util"; import { afterEach, describe, expect, test } from "vitest"; -import { requireCliSuccess, runSupabase } from "../../../../tests/helpers/cli.ts"; +import { + makeTempLegacyStackProject, + requireCliSuccess, + runSupabase, +} from "../../../../tests/helpers/cli.ts"; import { legacySanitizeProjectId } from "../../shared/legacy-docker-ids.ts"; const execFileAsync = promisify(execFile); -const START_TIMEOUT_MS = 280_000; +const CLI_COMMAND_TIMEOUT_MS = 60_000; +const STACK_START_TIMEOUT_MS = 280_000; +const STOP_COMMAND_TIMEOUT_MS = 120_000; +const DOCKER_INSPECT_TIMEOUT_MS = 30_000; +const CLEANUP_TIMEOUT_MS = 120_000; +const LIFECYCLE_MARGIN_MS = 30_000; +const CLEANUP_HOOK_TIMEOUT_MS = CLEANUP_TIMEOUT_MS + LIFECYCLE_MARGIN_MS; +const STOP_TEST_TIMEOUT_MS = + CLI_COMMAND_TIMEOUT_MS + + STACK_START_TIMEOUT_MS + + CLI_COMMAND_TIMEOUT_MS + + STOP_COMMAND_TIMEOUT_MS + + DOCKER_INSPECT_TIMEOUT_MS + + LIFECYCLE_MARGIN_MS; // `stop` never calls the Management API — it talks directly to the real local // Docker stack `start` creates. `describe` gates @@ -19,32 +34,30 @@ const START_TIMEOUT_MS = 280_000; // SUPABASE_ACCESS_TOKEN it gates on is otherwise irrelevant here. See // AGENTS.md's "e2e tests" section for the full convention. describe("supabase stop (e2e)", () => { - let projectDir: string | undefined; + let project: Awaited> | undefined; let projectId: string | undefined; afterEach(async () => { - if (projectDir === undefined) return; - // Best-effort cleanup even if an assertion above failed mid-lifecycle — a - // leaked local stack would otherwise pollute the CI runner for later jobs. - await runSupabase(["stop", "--no-backup"], { - entrypoint: "legacy", - cwd: projectDir, - }).catch(() => undefined); - await rm(projectDir, { recursive: true, force: true }).catch(() => undefined); - projectDir = undefined; + await project?.cleanup().catch(() => undefined); + project = undefined; projectId = undefined; - }); + }, CLEANUP_HOOK_TIMEOUT_MS); test( "starts a real local stack, then stops it and removes its containers", - { timeout: START_TIMEOUT_MS }, + { timeout: STOP_TEST_TIMEOUT_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-stop-e2e-")); + project = await makeTempLegacyStackProject("sb-stop-e2e-"); + const projectDir = project.dir; // No `project_id` override, so the cli resolves it from the workdir // basename (see legacy-docker-ids.ts). projectId = path.basename(projectDir); - const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir }); + const init = await runSupabase(["init"], { + entrypoint: "legacy", + cwd: projectDir, + exitTimeoutMs: CLI_COMMAND_TIMEOUT_MS, + }); requireCliSuccess(init, "init setup"); // Exclude the heaviest, least relevant services (Next.js Studio build, the @@ -52,50 +65,67 @@ describe("supabase stop (e2e)", () => { // which services are running, only that at least one real container // exists to stop. const start = await runSupabase( - ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + ["start", "--exclude", "studio", "--exclude", "logflare", "--exclude", "vector"], + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: STACK_START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); // Sanity: confirm the stack is actually up before testing `stop` against it. - const before = await runSupabase(["status"], { entrypoint: "legacy", cwd: projectDir }); + const before = await runSupabase(["status"], { + entrypoint: "legacy", + cwd: projectDir, + exitTimeoutMs: CLI_COMMAND_TIMEOUT_MS, + }); requireCliSuccess(before, "status setup"); - const stop = await runSupabase(["stop"], { entrypoint: "legacy", cwd: projectDir }); + const stop = await runSupabase(["stop"], { + entrypoint: "legacy", + cwd: projectDir, + exitTimeoutMs: STOP_COMMAND_TIMEOUT_MS, + }); expect(stop.exitCode, `stdout:\n${stop.stdout}\nstderr:\n${stop.stderr}`).toBe(0); expect(stop.stdout).toContain("Stopped"); // The real Docker daemon must agree: no container carrying this project's // label survives `stop` — the actual behavior under test, not just the // cli's own exit code. - const { stdout: remaining } = await execFileAsync("docker", [ - "ps", - "-a", - "--filter", - `label=com.supabase.cli.project=${projectId}`, - "--format", - "{{.ID}}", - ]); + const { stdout: remaining } = await execFileAsync( + "docker", + [ + "ps", + "-a", + "--filter", + `label=com.supabase.cli.project=${projectId}`, + "--format", + "{{.ID}}", + ], + { timeout: DOCKER_INSPECT_TIMEOUT_MS }, + ); expect(remaining.trim()).toBe(""); }, ); test( "stop --no-backup --debug reports real pruned containers, volumes, and network", - { timeout: START_TIMEOUT_MS }, + { timeout: STOP_TEST_TIMEOUT_MS }, async () => { - projectDir = await mkdtemp(path.join(tmpdir(), "sb-stop-e2e-")); + project = await makeTempLegacyStackProject("sb-stop-e2e-"); + const projectDir = project.dir; // Sanitizing is a no-op for a `mkdtemp`-generated basename (already // alphanumeric/`-`), but mirrors the port's actual resolution rather // than assuming that stays true (same note as `start.e2e.test.ts`). projectId = legacySanitizeProjectId(path.basename(projectDir)); - const init = await runSupabase(["init"], { entrypoint: "legacy", cwd: projectDir }); + const init = await runSupabase(["init"], { + entrypoint: "legacy", + cwd: projectDir, + exitTimeoutMs: CLI_COMMAND_TIMEOUT_MS, + }); requireCliSuccess(init, "init setup"); const start = await runSupabase( - ["start", "--exclude", "studio", "--exclude", "analytics", "--exclude", "vector"], - { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: START_TIMEOUT_MS }, + ["start", "--exclude", "studio", "--exclude", "logflare", "--exclude", "vector"], + { entrypoint: "legacy", cwd: projectDir, exitTimeoutMs: STACK_START_TIMEOUT_MS }, ); requireCliSuccess(start, "start setup"); @@ -107,6 +137,7 @@ describe("supabase stop (e2e)", () => { const stop = await runSupabase(["stop", "--no-backup", "--debug"], { entrypoint: "legacy", cwd: projectDir, + exitTimeoutMs: STOP_COMMAND_TIMEOUT_MS, }); expect(stop.exitCode, `stdout:\n${stop.stdout}\nstderr:\n${stop.stderr}`).toBe(0); expect(stop.stdout).toContain("Stopped"); @@ -126,14 +157,18 @@ describe("supabase stop (e2e)", () => { // The real Docker daemon must agree with the report: nothing carrying // this project's label survives. - const { stdout: remaining } = await execFileAsync("docker", [ - "ps", - "-a", - "--filter", - `label=com.supabase.cli.project=${projectId}`, - "--format", - "{{.ID}}", - ]); + const { stdout: remaining } = await execFileAsync( + "docker", + [ + "ps", + "-a", + "--filter", + `label=com.supabase.cli.project=${projectId}`, + "--format", + "{{.ID}}", + ], + { timeout: DOCKER_INSPECT_TIMEOUT_MS }, + ); expect(remaining.trim()).toBe(""); }, ); diff --git a/apps/cli/tests/helpers/cli.ts b/apps/cli/tests/helpers/cli.ts index 3110d7cdc0..51a76fbaaf 100644 --- a/apps/cli/tests/helpers/cli.ts +++ b/apps/cli/tests/helpers/cli.ts @@ -76,6 +76,7 @@ type RunResult = { }; const DEFAULT_EXIT_TIMEOUT_MS = 60_000; +const DEFAULT_LEGACY_STACK_CLEANUP_TIMEOUT_MS = 120_000; const OUTPUT_TAIL_LENGTH = 4_000; interface SpawnedSupabase { @@ -148,6 +149,44 @@ export async function makeTempCliProject(prefix = "supabase-cli-e2e-") { return project; } +export async function makeTempLegacyStackProject( + prefix = "supabase-legacy-stack-e2e-", + cleanupTimeoutMs = DEFAULT_LEGACY_STACK_CLEANUP_TIMEOUT_MS, +) { + const project = await makeTempProject(prefix); + const cleanup = async () => { + if (!existsSync(project.dir)) return; + + // `init` can fail before creating a project config. There is no stack to + // stop in that case, so remove the exact owned directory directly. + if (!existsSync(path.join(project.dir, "supabase", "config.toml"))) { + await rm(project.dir, { recursive: true, force: true }); + return; + } + + const stopped = await runSupabase(["stop", "--no-backup"], { + entrypoint: "legacy", + cwd: project.dir, + exitTimeoutMs: cleanupTimeoutMs, + }); + if (stopped.exitCode !== 0) { + throw new Error( + [ + `Failed to stop legacy stack in ${project.dir} (exit code ${stopped.exitCode}).`, + `stdout:\n${stopped.stdout}`, + `stderr:\n${stopped.stderr}`, + ].join("\n"), + ); + } + + await rm(project.dir, { recursive: true, force: true }); + }; + + const stackProject = { dir: project.dir, cleanup }; + registerTempStackProject(stackProject); + return stackProject; +} + export async function makeTempStackProject(prefix = "supabase-stack-e2e-") { const project = await makeTempProject(prefix); const ports = { From 85555d3dbdf7bb62d9067660bc94b99855e45a08 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Sat, 22 Aug 2026 08:04:33 +0200 Subject: [PATCH 23/23] test(cli): use standard e2e filenames --- .../{diff.docker.e2e.test.ts => diff.declarative.e2e.test.ts} | 0 .../sync/{sync.docker.e2e.test.ts => sync.e2e.test.ts} | 0 .../db/start/{start.docker.e2e.test.ts => start.e2e.test.ts} | 0 .../{start.docker.e2e.test.ts => start.lifecycle.e2e.test.ts} | 0 .../status/{status.docker.e2e.test.ts => status.e2e.test.ts} | 0 .../commands/stop/{stop.docker.e2e.test.ts => stop.e2e.test.ts} | 0 .../functions/dev/{dev.docker.e2e.test.ts => dev.e2e.test.ts} | 0 .../start/{start.docker.e2e.test.ts => start.e2e.test.ts} | 0 8 files changed, 0 insertions(+), 0 deletions(-) rename apps/cli/src/legacy/commands/db/diff/{diff.docker.e2e.test.ts => diff.declarative.e2e.test.ts} (100%) rename apps/cli/src/legacy/commands/db/schema/declarative/sync/{sync.docker.e2e.test.ts => sync.e2e.test.ts} (100%) rename apps/cli/src/legacy/commands/db/start/{start.docker.e2e.test.ts => start.e2e.test.ts} (100%) rename apps/cli/src/legacy/commands/start/{start.docker.e2e.test.ts => start.lifecycle.e2e.test.ts} (100%) rename apps/cli/src/legacy/commands/status/{status.docker.e2e.test.ts => status.e2e.test.ts} (100%) rename apps/cli/src/legacy/commands/stop/{stop.docker.e2e.test.ts => stop.e2e.test.ts} (100%) rename apps/cli/src/next/commands/functions/dev/{dev.docker.e2e.test.ts => dev.e2e.test.ts} (100%) rename apps/cli/src/next/commands/start/{start.docker.e2e.test.ts => start.e2e.test.ts} (100%) diff --git a/apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/diff/diff.declarative.e2e.test.ts similarity index 100% rename from apps/cli/src/legacy/commands/db/diff/diff.docker.e2e.test.ts rename to apps/cli/src/legacy/commands/db/diff/diff.declarative.e2e.test.ts diff --git a/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.e2e.test.ts similarity index 100% rename from apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.docker.e2e.test.ts rename to apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.e2e.test.ts diff --git a/apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts b/apps/cli/src/legacy/commands/db/start/start.e2e.test.ts similarity index 100% rename from apps/cli/src/legacy/commands/db/start/start.docker.e2e.test.ts rename to apps/cli/src/legacy/commands/db/start/start.e2e.test.ts diff --git a/apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts b/apps/cli/src/legacy/commands/start/start.lifecycle.e2e.test.ts similarity index 100% rename from apps/cli/src/legacy/commands/start/start.docker.e2e.test.ts rename to apps/cli/src/legacy/commands/start/start.lifecycle.e2e.test.ts diff --git a/apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts b/apps/cli/src/legacy/commands/status/status.e2e.test.ts similarity index 100% rename from apps/cli/src/legacy/commands/status/status.docker.e2e.test.ts rename to apps/cli/src/legacy/commands/status/status.e2e.test.ts diff --git a/apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts b/apps/cli/src/legacy/commands/stop/stop.e2e.test.ts similarity index 100% rename from apps/cli/src/legacy/commands/stop/stop.docker.e2e.test.ts rename to apps/cli/src/legacy/commands/stop/stop.e2e.test.ts diff --git a/apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts b/apps/cli/src/next/commands/functions/dev/dev.e2e.test.ts similarity index 100% rename from apps/cli/src/next/commands/functions/dev/dev.docker.e2e.test.ts rename to apps/cli/src/next/commands/functions/dev/dev.e2e.test.ts diff --git a/apps/cli/src/next/commands/start/start.docker.e2e.test.ts b/apps/cli/src/next/commands/start/start.e2e.test.ts similarity index 100% rename from apps/cli/src/next/commands/start/start.docker.e2e.test.ts rename to apps/cli/src/next/commands/start/start.e2e.test.ts