Migrate app database: Cloudflare D1 → PlanetScale Postgres

[Makisuo]

Moves Maple's relational app DB off Cloudflare D1 (SQLite) onto PlanetScale Postgres, reached from Workers via Cloudflare Hyperdrive. Drizzle stays the ORM; schema is modernized to native Postgres types.

Why

D1 forced ms-integer timestamps, 0/1 booleans, JSON-as-text, a 100 bound-param insert cap, and a drift-prone dual migration-runner setup. Postgres gives real timestamptz/jsonb/boolean, no param cap, and one migration history.

What changed

Schema & migrations (packages/db)

• All 20 schema files sqliteTable → pgTable: 114 ms columns → timestamptz (Date), 23 text-JSON → jsonb, native boolean, real → doublePrecision, unixepoch defaults → defaultNow().
• Re-baselined to a single Postgres migration (drizzle/0000_*.sql); deleted the 25 SQLite migrations + the 0012/0013 data migrations.
• New pg/PGlite client factories; deleted d1-limits.ts.

Effect layers (apps/api)

• DatabasePgLive (Workers → Hyperdrive; short-lived per-execute postgres.js client, since Workers TCP sockets are request-bound and Hyperdrive owns the pool) and DatabasePgliteLive (local/tests). New hyperdrive-connection.ts; deleted DatabaseD1Live/DatabaseLibsqlLive/d1-connection.ts. worker/agent/alerting/chat-agent rewired; dynamic-import startup-CPU pattern preserved.

Call-site sweep — new Date() at write/where boundaries, .getTime()/.toISOString() on reads (via apps/api/src/lib/time.ts), ~44 Schema.fromJsonString → direct jsonb decode, boolean comparisons, 2 db.batch → db.transaction, chunked-insert removal. Domain/HTTP contracts unchanged (still ms numbers / ISO strings) — conversion lives only at the drizzle boundary.

Tests — test-pglite.ts (in-memory PGlite per file); 20 test files converted. Full suite green.

Infra

• alchemy.run.ts: Hyperdrive binding parsed from a single MAPLE_PG_URL per stage (the Cloudflare origin API is structured-only, so the URL is parsed into host/user/…); legacy D1 kept with delete:false as a rollback snapshot.
• stage.ts: resolveHyperdriveName / resolvePlanetScaleBranch.
• 3× wrangler.jsonc: hyperdrive block + localConnectionString; docker-compose Postgres (bun db:up / db:migrate:local).
• CI: drizzle-kit migrate against the branch's direct 5432 before alchemy deploy, in all three deploy workflows.

Migration tooling (packages/db/scripts/)

• planetscale-connection.ts — brokers an ephemeral pscale role (Postgres uses roles, not Vitess passwords) with a TTL safety net, hands a connection URL to a callback, revokes after.
• ps:apply-schema <branch> — drizzle migrate against a branch.
• ps:migrate-data <branch> [dump] — auto-exports the D1 dump via wrangler d1 export, then imports. d1-to-postgres-migrate.ts streams the dump, skips/prunes disposable error-history churn (IMPORT_SKIP_ERROR_HISTORY), rounds float-in-int columns (SQLite loose typing), realigns identity sequences, and verifies per-table counts.
• planetscale-pr-branch.ts — per-PR ephemeral branch create/delete, exports MAPLE_PG_URL.

Ingest gateway (apps/ingest, Rust) — new PostgresKeyStore behind INGEST_KEY_STORE_BACKEND=postgres (PSBouncer 6432, no Hyperdrive); D1 keystore kept for the rollback window. cargo build + tests pass.

Verified

• Repo-wide bun typecheck (22/22) · vitest full suite · ingest cargo build/test.
• Live e2e: wrangler dev → Hyperdrive → docker Postgres round-tripped a row (jsonb came back as an object, boolean filter worked, lazy ingest-key write succeeded).
• Real PlanetScale: schema applied + data imported & count-verified on both main and stg (config/operational tables; error-history churn skipped by choice).

Reviewer notes

• Single MAPLE_PG_URL secret per stage (Doppler); MAPLE_PG_DATABASE connect-name is postgres (cluster default), not the PS resource name maple.
• Error-history tables (error_incidents/error_issue_events/error_issue_states, and optionally error_issues) are skipped in the data import — they rebuild from live telemetry. Flip IMPORT_KEEP_INCIDENTS / drop IMPORT_SKIP_ERROR_HISTORY to keep them.
• Cutover is not in this PR. Deploying still needs the per-stage MAPLE_PG_URL in Doppler + a stable app role; do it on stg first. The MAPLE_MAINTENANCE freeze knob and the final D1 decommission are deferred to the cutover-execution PR.

🤖 Generated with Claude Code

[github-actions]

Ingest Rust Test + Benchmark Results

Commit: 3b72218b66f66e8216bff34e5f976c0837c9caa9

Load Benchmark — tinybird mode, median of 3 run(s) vs main

Metric	main (median)	PR (median)	Delta
Requests/sec	2728.54	3033.85	+11.2% better
Rows/sec	27285.38	30338.55	+11.2% better
p50 latency	21.64 ms	20.27 ms	-6.3% better
p95 latency	39.21 ms	39.69 ms	+1.2% worse
p99 latency	50.38 ms	58.04 ms	+15.2% worse
Export catch-up	0.026 s	0.026 s	-1.6% better
Max RSS	99.91 MiB	103.49 MiB	+3.6% worse
Failures	0	0	same

Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.

PR load benchmark JSON (per-iteration)

[
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 21,
    "duration_seconds": 0.707105125,
    "export_catchup_seconds": 0.025982827,
    "request_rps": 2828.4337495078967,
    "row_rps": 28284.337495078966,
    "p50_ms": 21.858,
    "p95_ms": 28.3,
    "p99_ms": 43.098,
    "max_rss_mb": 102.77734375,
    "max_cpu_percent": 91.0,
    "avg_cpu_percent": 53.8
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 20,
    "duration_seconds": 0.65922738,
    "export_catchup_seconds": 0.026286323,
    "request_rps": 3033.854570785576,
    "row_rps": 30338.545707855763,
    "p50_ms": 20.27,
    "p95_ms": 40.153,
    "p99_ms": 58.897,
    "max_rss_mb": 103.81640625,
    "max_cpu_percent": 94.6,
    "avg_cpu_percent": 63.949999999999996
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 23,
    "duration_seconds": 0.640499373,
    "export_catchup_seconds": 0.025699025,
    "request_rps": 3122.563556358079,
    "row_rps": 31225.635563580792,
    "p50_ms": 19.973,
    "p95_ms": 39.688,
    "p99_ms": 58.044,
    "max_rss_mb": 103.4921875,
    "max_cpu_percent": 96.4,
    "avg_cpu_percent": 56.5
  }
]

main load benchmark JSON (per-iteration)

[
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 25,
    "duration_seconds": 0.77716588,
    "export_catchup_seconds": 0.026153257,
    "request_rps": 2573.4531732144496,
    "row_rps": 25734.531732144496,
    "p50_ms": 23.306,
    "p95_ms": 34.21,
    "p99_ms": 50.379,
    "max_rss_mb": 99.9140625,
    "max_cpu_percent": 80.3,
    "avg_cpu_percent": 48.45
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 22,
    "duration_seconds": 0.732993263,
    "export_catchup_seconds": 0.026409083,
    "request_rps": 2728.538038418506,
    "row_rps": 27285.38038418506,
    "p50_ms": 21.644,
    "p95_ms": 44.013,
    "p99_ms": 68.738,
    "max_rss_mb": 98.60546875,
    "max_cpu_percent": 83.9,
    "avg_cpu_percent": 58.6
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 23,
    "duration_seconds": 0.665687999,
    "export_catchup_seconds": 0.027037966,
    "request_rps": 3004.41047908992,
    "row_rps": 30044.104790899197,
    "p50_ms": 20.73,
    "p95_ms": 39.21,
    "p99_ms": 42.333,
    "max_rss_mb": 105.7265625,
    "max_cpu_percent": 92.8,
    "avg_cpu_percent": 54.7
  }
]

WAL-acked microbench (cargo bench --bench ingest_bench)

   Compiling maple-ingest v0.1.0 (/home/runner/work/maple/maple/apps/ingest)
warning: unused import: `AtomicBool`
 --> src/main.rs:8:25
  |
8 | use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
  |                         ^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default

warning: `maple-ingest` (bin "maple-ingest") generated 1 warning (run `cargo fix --bin "maple-ingest" -p maple-ingest` to apply 1 suggestion)
    Finished `bench` profile [optimized] target(s) in 41.75s
     Running benches/ingest_bench.rs (target/release/deps/ingest_bench-b2de825453a1d881)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench:      334192 ns/iter (+/- 27140)
test ingest_accept/traces_10_spans_wal_ack ... bench:      396896 ns/iter (+/- 13414)

cargo test

test telemetry::tests::hex_empty_for_zero_ids ... ok
test telemetry::tests::logs_use_observed_time_when_time_unix_nano_is_zero ... ok
test telemetry::tests::metric_encoder_matches_all_tinybird_datasource_shapes ... ok
test telemetry::tests::metrics_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::metrics_summary_data_points_are_dropped ... ok
test telemetry::tests::pipeline_can_start_for_clickhouse_only_without_tinybird_credentials ... ok
test telemetry::tests::clickhouse_export_drops_passworded_non_https_endpoint_without_sending ... ok
test telemetry::tests::pipeline_e2e_exports_metrics_to_fake_tinybird ... ok
test telemetry::tests::sampling_keeps_errors_even_when_ratio_low ... ok
test telemetry::tests::scraper_contract::scraper_otlp_json_decodes_with_gateway_serde_and_encodes_to_rows ... ok
test telemetry::tests::timestamp_has_nano_precision ... ok
test telemetry::tests::timestamps_match_clickhouse_datetime64_nine_format ... ok
test telemetry::tests::trace_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::traces_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::pipeline_e2e_exports_gzip_ndjson_to_fake_tinybird ... ok
test telemetry::tests::wal_round_trips_frame ... ok
test telemetry::tests::wal_truncates_after_full_drain_allowing_further_appends ... ok
test telemetry::tests::wal_partial_drain_advances_cursor_without_truncating ... ok
test telemetry::tests::pipeline_e2e_exports_traces_to_fake_tinybird ... ok
test telemetry::tests::pipeline_exports_ready_org_to_clickhouse_without_tinybird_calls ... ok

test result: ok. 28 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s

     Running unittests src/bin/load_test.rs (target/debug/deps/load_test-1467db0053340276)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/maple_ingest-0081139dbd22711e)

running 40 tests
test autumn::tests::allowed_only_no_balance_field ... ok
test autumn::tests::flat_hardcap_with_remaining_allows ... ok
test autumn::tests::flat_hardcap_depleted_blocks ... ok
test autumn::tests::flat_overage_allows ... ok
test autumn::tests::flat_sub_one_gb_remaining_still_allows ... ok
test autumn::tests::flat_unlimited_allows ... ok
test autumn::tests::nested_balance_object_depleted_blocks ... ok
test autumn::tests::nested_balance_object_with_remaining_allows ... ok
test autumn::tests::null_balance_no_subscription_blocks ... ok
test autumn::tests::nested_overage_allows ... ok
test autumn::tests::unrecognized_shape_returns_none ... ok
test tests::clickhouse_destination_is_terminal_in_dual_mode ... ok
test tests::clickhouse_destination_uses_native_pipeline_even_in_forward_mode ... ok
test tests::clickhouse_target_resolver_requires_current_schema ... ok
test tests::clickhouse_target_resolver_decrypts_current_schema_password ... ok
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::clickhouse_target_resolver_rejects_password_over_http ... ok
test tests::d1_response_parses_empty_results_as_no_match ... ok
test tests::d1_response_parses_failure_with_errors ... ok
test tests::d1_response_parses_success_with_rows ... ok
test tests::decrypt_aes256_gcm_matches_node_crypto_fixture ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::d1_truthy_accepts_int_and_bool_self_managed ... ok
test tests::rejection_span_status_is_error_only_for_5xx ... ok
test tests::hash_is_deterministic ... ok
test tests::resolve_ingest_key_keeps_stale_schema_on_managed_native_path ... ok
test tests::resolve_connector_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::tinybird_destination_keeps_forward_mode_on_forward_path ... ok
test tests::resolve_ingest_key_serves_last_known_routing_when_refresh_fails ... ok
test autumn::tests::fails_open_on_transport_error ... ok
test tests::forward_mode_switches_ready_org_to_clickhouse_without_forwarding_again ... ok

test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s

   Doc-tests maple_ingest

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

[pullfrog]

ℹ️ No critical issues — one error-handling rough edge inline, plus two cutover/teardown sequencing notes for awareness.

Reviewed changes — initial review of the full D1 (SQLite) → PlanetScale Postgres migration: schema retype, Effect DB layers, the ~44-file call-site sweep, infra wiring, migration tooling, and the Rust ingest keystore. The conversion is careful and internally consistent — boolean/jsonb/timestamptz boundary handling is uniform and the domain/HTTP contracts are genuinely untouched.

• Schema retype + re-baseline — 20 sqliteTable→pgTable files (ms→timestamptz, text-JSON→jsonb, 0/1→boolean, real→doublePrecision) collapsed into a single drizzle/0000_windy_raza.sql; 25 legacy SQLite migrations + the 0012/0013 data migrations deleted.
• New DB layers — DatabasePgLive (Workers→Hyperdrive, fresh postgres.js client per execute) and DatabasePgliteLive (local/tests); createMaplePgClient in packages/db/src/client.ts; hyperdrive-connection.ts replacing d1-connection.ts.
• Boundary sweep — msToDate/dateToMs in apps/api/src/lib/time.ts, direct jsonb decode (removed Schema.fromJsonString), native boolean comparisons, 2× db.batch→db.transaction, and removal of the chunked-insert / d1-limits.ts machinery.
• Infra — alchemy.run.ts parses MAPLE_PG_URL into a structured Hyperdrive origin (legacy D1 kept delete:false as a rollback snapshot, caching:{disabled:true}); 3 deploy workflows run drizzle-kit migrate against the direct 5432 port before alchemy deploy.
• Migration tooling — planetscale-connection.ts (ephemeral pscale role + TTL + revoke), d1-to-postgres-migrate.ts (float rounding, sequence realignment, count verification, error-history skip), scripts/planetscale-pr-branch.ts.
• Ingest — Rust PostgresKeyStore behind INGEST_KEY_STORE_BACKEND=postgres (deadpool over PSBouncer 6432, rustls + webpki, startup probe), with the D1 keystore retained for the rollback window.

ℹ️ Ingest/API key-store backends flip independently during cutover

The ingest gateway selects its key store from INGEST_KEY_STORE_BACKEND at startup, while the API selects its DB layer at deploy time. These are separate services with separate restarts, so a cutover deploy has a window where one side is on Postgres and the other still on D1.

• API on PG + ingest still on D1 → ingest keys written after the API deploy resolve against the old D1 keystore and bounce as invalid ingest key until ingest restarts.
• The reverse ordering strands keys written to D1 during the gap.

The PR explicitly defers cutover to a later PR, so this is not a blocker — flagging it so the cutover runbook pins a deterministic order (or a brief dual-read) rather than leaving the two backends to drift during the rollout.

Technical details

# Ingest/API key-store backends flip independently during cutover

## Affected sites
- apps/ingest/src/main.rs (`resolve_key_store_backend`) — backend chosen from `INGEST_KEY_STORE_BACKEND` at process start.
- apps/api DB layer selection (`DatabasePgLive` vs prior D1) — flips at deploy.

## Required outcome
- The cutover sequence guarantees ingest-key reads and writes hit the same backend at all times (no read-from-D1 / write-to-PG gap), OR an explicit short-lived dual-read window covers the transition.

## Suggested approach (optional)
- Document the flip order in the cutover-execution PR's runbook: e.g. backfill PG, point both API + ingest at PG in the same release train, keep D1 readable for rollback only.

## Open questions for the human
- Is there an intended single moment where both services are restarted onto PG, or is dual-read expected during the window?

ℹ️ PR-preview branch teardown is best-effort under cancel-in-progress

PR-preview deploys create an ephemeral PlanetScale pr-<n> branch and tear it down on closed. The teardown is correct and idempotent (down tolerates a missing branch), but the workflow's cancel-in-progress: true plus the un-guarded chaining of alchemy:destroy:pr → planetscale-pr-branch.ts down means an unlucky event interleaving or a failed destroy step can leave the branch undeleted.

• The minted role carries a 24h TTL, so a stranded credential self-expires.
• The branch itself has no TTL — a stranded one keeps billing (PS-DEV) and consumes a Hyperdrive config slot until manually removed.

Not blocking and not unique to this PR (the Tinybird branch script has the same shape), but worth a periodic orphan-sweep if PS-DEV quota/billing becomes a concern.

ℹ️ Nitpicks

• ClickHouseSchemaApplyWorkflow.run.ts already guards its client teardown with await connection.end().catch(() => undefined) — the request-path DatabasePgLive should adopt the same pattern (see inline comment) so the two converge.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes — incremental review of the two commits pushed since the prior review (5567def): a knip-config fix and a CI gating change to the PR-preview workflow. Both are clean tooling/CI adjustments with no new behavioral risk.

• Gate PR-preview PlanetScale/Alchemy steps on PLANETSCALE_SERVICE_TOKEN — every PlanetScale + Alchemy step in deploy-pr-preview.yml now carries && env.PLANETSCALE_SERVICE_TOKEN != '', so PR previews skip cleanly (check stays green) until the service token lands in the Doppler pr config rather than failing every PR. The Doppler inject-env-vars: true write to $GITHUB_ENV is visible in the env context of the later steps' if: conditions, so the gate resolves correctly.
• knip: register packages/db scripts as entry + drop unused effect dep — knip.json adds packages/db scripts/**/*.ts as entry points (the migration CLIs) and removes the now-unused effect dev dependency from lib/effect-cloudflare.

The prior review's inline note on the unguarded await end() in DatabasePgLive.ts is unchanged by these commits and still stands.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes — incremental review of the single commit pushed since the prior review (759bbca): the fix for the DatabasePgLive error-masking finding from the initial review.

• Guard postgres.js end() in DatabasePgLive finally — the per-execute teardown is now await end().catch(() => undefined), so a socket-teardown error can no longer shadow the real DB error from fn(db) before it reaches toDatabaseError. Matches the precedent in ClickHouseSchemaApplyWorkflow.run.ts. This fully resolves the inline finding from the prior review (thread now resolved).

The two informational notes from the initial review (independent cutover backend flip; best-effort PR-preview branch teardown) remain open as awareness items for the deferred cutover PR — neither is a blocker for this PR.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

ℹ️ No new issues — the test-setup change is sound. One orphaned-JSDoc nit inline.

Reviewed changes — incremental review of the single commit since the prior review (bc7a346): making the PGlite test setup cheaper and resilient under CI contention. Deployed Postgres is untouched (still drizzle-kit migrate); this is purely the local/test path.

• Apply test schema via a single pglite.exec() instead of the drizzle migrator — test-pglite.ts now reads the concatenated migration SQL once at module load and applies it per instance, memoized via migrated ??=, avoiding the migrator's per-test filesystem reads and __drizzle_migrations bookkeeping that starve under parallel turbo test.
• Add readBundledMigrationsSql() to @maple/db/migrate — reads + concatenates drizzle/*.sql in .sort() (filename) order, cached at module scope. Today only 0000_windy_raza.sql exists, so ordering is moot in practice; the --> statement-breakpoint markers are SQL line comments that pglite.exec()'s multi-statement script handles fine.
• Extract databaseFromInstance in DatabasePgliteLive.ts — a no-migration wrapper around an already-migrated PGlite instance, replacing the removed DatabasePgliteInstanceLive.
• Raise vitest testTimeout/hookTimeout to 60s — headroom for WASM-PGlite boot + real-backoff retry tests under CI CPU starvation.

Note: I could not execute the DB-backed tests in the review sandbox — @electric-sql/pglite is declared in both apps/api and packages/db package.json but isn't installed in this environment. That's a sandbox artifact, not a code issue; the change is sound on inspection.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes — incremental review of the single commit since the prior review (1342ca0): the merge of origin/main that resolves apps/ingest conflicts introduced by #91 (direct ClickHouse ingest routing). The conflict resolution faithfully extends the new PostgresKeyStore to #91's expanded KeyStore interface — the Postgres backend now behaves identically to the D1 backend that #91 added.

• Extend PostgresKeyStore for the clickhouse_ready signal — KeyRow/ConnectorRow gain clickhouse_ready, computed via COALESCE(s.sync_status = 'connected' AND s.schema_version = $1, false). Param ordering is correct ($1 = CLICKHOUSE_PROJECT_REVISION, $2/$3 = hash/ids), the COALESCE correctly maps the LEFT-JOIN NULL to false, and the projection is semantically equivalent to D1's CASE WHEN ... schema_version = ? THEN 1 ELSE 0 END.
• Add PostgresKeyStore::fetch_clickhouse_target — reads org_clickhouse_settings filtered on sync_status = 'connected' AND schema_version = $2. Nullable columns (ch_password_*) map to Option<String> per the schema; the non-null schema_version read is safe because the WHERE filter guarantees a matched row has it set.
• Fold the Postgres arm into the clickhouse_encryption_key match — both DB backends now require MAPLE_INGEST_KEY_ENCRYPTION_KEY since both decrypt BYO-ClickHouse credentials.
• Regenerate Cargo.lock — merges this PR's postgres crates with #91's aes-gcm dependency tree; cargo build is green.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes — incremental review of the single commit since the prior review (e4142f4): a CI idempotency fix to the PR-preview PlanetScale branch script. Scoped to scripts/planetscale-pr-branch.ts; correct and self-contained.

• Tolerate PlanetScale's real "already taken" wording — isAlreadyExists now also matches already been taken (the message branch create actually emits), so re-running up on a PR synchronize treats the persistent pr-<n> branch as a benign no-op instead of failing the step.
• Mint a unique CI role per run — createCredential now uses ci-<branch>-<pid>-<Date.now()> rather than a fixed ci-<branch>, which previously collided on the second synchronize because the branch (and its leftover role) survives across pushes. Cleanup is unaffected: down deletes the branch, which revokes all of its roles wholesale, and the 24h TTL remains as a safety net.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Caution

This merge does not compile. cargo build for apps/ingest fails with 4 errors — the PostgresKeyStore was not reconciled with the changes this merge pulled in from origin/main (fb8b043, which re-keyed BYO-ClickHouse ingest readiness onto the migration version and expanded the KeyStore trait).

Reviewed changes — incremental review of the single merge commit since the prior review (531a334): 589e4ba merges origin/main into migration-pg, pulling in fb8b043 ("key BYO-ClickHouse ingest readiness on migration version").

• Re-baseline correctly subsumes main's new SQLite data migration — origin/main added 0025-clickhouse-schema-version-rekey.ts (+ test); the merge drops both, which is right because this PR collapsed the whole migration history into 0000_windy_raza.sql. migrate.ts carries no data-migration calls.
• Schema + workflow adopt the new keying semantics — org-clickhouse-settings.ts now documents schema_version as holding clickHouseSchemaVersion (migration version), and ClickHouseSchemaApplyWorkflow.run.ts:340 stamps clickHouseSchemaVersion instead of clickHouseProjectRevision.
• Ingest D1 keystore tracks main — the D1 KeyStore impl binds CLICKHOUSE_SCHEMA_VERSION and implements the new fetch_org_routing trait method.
• PostgresKeyStore was left behind — it still references the removed CLICKHOUSE_PROJECT_REVISION constant and never implements fetch_org_routing, so the crate fails to build (see inline).

🚨 Ingest gateway does not compile after the merge

The merge pulled main's fb8b043, which renamed the ingest readiness key from the Tinybird-coupled PROJECT_REVISION to the migration SCHEMA_VERSION and added a fetch_org_routing method to the KeyStore trait. The D1 keystore was updated for both, but the PostgresKeyStore (authored in this PR) was not — leaving it referencing a constant that no longer exists and missing a required trait method.

cargo build in apps/ingest fails with four errors:

error[E0425]: cannot find value `CLICKHOUSE_PROJECT_REVISION` in this scope   (×3)
error[E0046]: not all trait items implemented, missing: `fetch_org_routing`

Beyond the build break, the constant swap is also a behavior bug: even if CLICKHOUSE_PROJECT_REVISION still resolved, the Postgres queries would compare the 64-char project-revision hash against an org_clickhouse_settings.schema_version column that now stores "4", so every BYO-ClickHouse org would read back clickhouse_ready = false and fetch_clickhouse_target would return None — silently disabling direct ClickHouse routing once the ingest gateway moves to the Postgres backend. Both fixes live inline.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

🚨 CLICKHOUSE_PROJECT_REVISION is no longer defined — main's fb8b043 renamed the readiness key, and only SCHEMA_VERSION as CLICKHOUSE_SCHEMA_VERSION is imported (line 33). All three PostgresKeyStore sites (4013, 4042, 4121) must use CLICKHOUSE_SCHEMA_VERSION, matching the D1 keystore. This is both a compile error (E0425 ×3) and a behavior bug: binding the project-revision hash against a schema_version column that now stores "4" makes every BYO-ClickHouse org read clickhouse_ready = false.

Technical details

# PostgresKeyStore binds the removed CLICKHOUSE_PROJECT_REVISION

## Affected sites
- apps/ingest/src/main.rs:4013 — `fetch_ingest_key`, `let revision = CLICKHOUSE_PROJECT_REVISION;`
- apps/ingest/src/main.rs:4042 — `fetch_connector`, same
- apps/ingest/src/main.rs:4121 — `fetch_clickhouse_target`, same (also bound as `$2` in the WHERE)
- apps/ingest/src/main.rs:33 — only `SCHEMA_VERSION as CLICKHOUSE_SCHEMA_VERSION` is imported; `PROJECT_REVISION` is not

## Required outcome
- The crate compiles (no `E0425`).
- The Postgres `clickhouse_ready` projection and `fetch_clickhouse_target` filter compare `schema_version` against the migration version `"4"` (`CLICKHOUSE_SCHEMA_VERSION`), identical to the D1 keystore at main.rs:3677/3888 and the `WHERE ... schema_version = ?` filter at 3854.

## Suggested approach
- Replace `CLICKHOUSE_PROJECT_REVISION` with `CLICKHOUSE_SCHEMA_VERSION` at all three sites. No other change needed — the SQL shape already matches the D1 path.

[pullfrog]

🚨 PostgresKeyStore is missing the fetch_org_routing trait method that main's fb8b043 added to KeyStore (declared at main.rs:562, consumed live in OrgRoutingResolver::resolve_org_routing at 3309). The D1 keystore implements it at 3879; this impl block ends after fetch_clickhouse_target without it, so the build fails with E0046.

Technical details

# PostgresKeyStore does not implement fetch_org_routing

## Affected sites
- apps/ingest/src/main.rs:562 — `KeyStore` trait now requires `async fn fetch_org_routing(&self, org_id: &str) -> Result<Option<OrgRouting>, String>`
- apps/ingest/src/main.rs:~4005-4145 — `impl KeyStore for PostgresKeyStore` has no `fetch_org_routing`
- apps/ingest/src/main.rs:3879-3900 — reference D1 implementation
- apps/ingest/src/main.rs:3309 — live caller via `OrgRoutingResolver`

## Required outcome
- `PostgresKeyStore` implements `fetch_org_routing`, returning the same `OrgRouting { self_managed, clickhouse_ready }` shape the D1 path returns.

## Suggested approach
- Port the D1 impl (3879-3900) to Postgres: `SELECT (sync_status = 'connected') AS self_managed, COALESCE(sync_status = 'connected' AND schema_version = $1, false) AS clickhouse_ready FROM org_clickhouse_settings WHERE org_id = $2 LIMIT 1`, bind `&[&CLICKHOUSE_SCHEMA_VERSION, &org_id]`, and read native bools via `row.get(...)` (no `d1_truthy`).