feat(analytics): restore from an S3-compatible bucket's dumps

[bryangingechen]

Motivation

The analytics workflow (upload_backup.yaml) can only read Heroku PGBackups. This lets it read the database's pg_dump archives from an S3-compatible bucket instead. Switching sources is a single repository variable, and with the variable unset the workflow does what it does today.

What changes

• BACKUP_SOURCE repository variable. Unset or heroku: the Heroku path, unchanged. s3: the bucket. Any other value fails the run's first step.
• scripts/download_s3_dump.py lists keys named <prefix>-YYYYMMDDTHHMMSSZ.dump, which sort by time, and ignores any other key.
  • It takes the newest dump whose pg_restore --list shows TABLE DATA entries, matched on the TOC line's shape. A dump of a database with no tables yet is skipped for an older one.
  • It gives up after 48 such dumps, and stops at once on a dump pg_restore can't read.
  • It prints which pg_restore it runs.
• Guard against empty snapshots. An empty table still gets a TABLE DATA entry, so a dump of a freshly migrated database would be taken. The restore step therefore fails when the restored database holds no pull requests, whatever the source, so an empty snapshot is never published.
• Manual runs: the new dump_key input names a dump for the S3 source; backup_id stays the Heroku source's. Each source refuses the other's input.
• Configuration:
  • Secrets: BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY_ID, BACKUP_S3_SECRET_ACCESS_KEY. Read-only credentials suffice. With the S3 source, a missing one fails the first step.
  • Optional variables: BACKUP_S3_KEY_PREFIX (default queueboard) and BACKUP_S3_REGION (default us-east-1).
  • The script reads all of these from the environment. Its errors never repeat the endpoint.
  • The backup-metadata artifact names the dump, its size, its upload time and the archive's format and versions. It never names the endpoint or the bucket, nor when or from which database the archive was made.
• Postgres 18 for the service and the client. pg_restore reads its own major version and older ones, so it still takes Heroku's 17 dumps. The published sanitized-pg-dump is now made by pg_dump 18; restore it with pg_restore 18. scripts/local_sanitize_backup.sh defaults to postgres:18-alpine.
• scripts/install_postgres_client.sh installs the client and puts its bin directory first on PATH. The runner image has its own Postgres 16 cluster, and Debian's pg_wrapper picks that version for a command that names no host. So without the pin, pg_restore --list runs version 16, which refuses newer dumps (unsupported version (1.16) in file header). The workflow and the CI job both use the script.
• The workflow's Python pins move to scripts/requirements-analytics.txt (adding boto3). The Heroku CLI is installed only for the Heroku source. Design doc 016 gets the operational notes.

Safe to merge before the bucket exists

With BACKUP_SOURCE unset, nothing reads the new secrets (beyond the first step testing them for emptiness, which it does only for the S3 source). The S3 step is skipped. The other differences on the Heroku path are Postgres 18, the client install script, the extra boto3 install, and the empty-database guard.

Switching

1. Add the four secrets.
2. Optionally, run the workflow by hand once the bucket holds a dump with data.
3. Set the repository variable BACKUP_SOURCE=s3. Unsetting it switches back.

Testing

• scripts/test_download_s3_dump.sh, CI job backup-download. It runs versitygw as a local S3 server, with Postgres 18 and 17 containers, and needs no credentials.
  • In CI it runs against the runner's pg_restore as the workflow sets it up (REQUIRE_HOST_PG_RESTORE=1). Locally, without Postgres 18, it falls back to the container's.
  • The dumps come from Postgres 18, of a database owned by another role. One is over boto3's 8 MiB multipart threshold. There is also a dump of a database with no tables, which holds a sequence named like a TABLE DATA entry, plus garbage and keys that don't match the pattern. Unset variables are passed empty, as the workflow passes them.
  • Eleven checks:
    • takes the newest dump with table data (a multipart download), skipping two newer empty ones; records what it skipped; restores with the workflow's flags to the right data;
    • the output names no endpoint, bucket or secret, and the metadata says nothing of when or from which database;
    • takes a named dump;
    • refuses a named dump with no table data, and fails on a missing key;
    • stops after --max-candidates, and at an unreadable dump;
    • fails on an empty prefix, wrong credentials, a missing endpoint, and an endpoint without a scheme (without repeating it);
    • restores a Postgres 17 dump (with pg_stat_statements, as Heroku's has) using pg_restore 18.
  • Restores fail their checks if pg_restore reports errors.
• Mutation checks:
  • accepting dumps without table data, or keys outside the pattern, fails the test;
  • an unanchored TABLE DATA match fails it (the decoy sequence);
  • a Postgres 17 source that restores with errors fails it.
• On a real runner (CI): the install script logs pg_restore 16.15 through pg_wrapper with no host, and 18.6 installed. The test then ran against 18.6, and all eleven checks passed.
• The Heroku path on Postgres 18, by manual runs of this branch's workflow with the variable unset. Both passed, and the S3 step was skipped.
  • Run 38176354840 (ba809eab): postgresql-client-18 and the postgres:18-alpine service restored the day's Heroku backup (a Postgres 17 database, 172.9 MB), with data for all 54 tables and no errors.
  • Run 38178884130 (24bc7d6b): the same, with the install script pinning 18.6 over the runner's 16.15, and the guard passing (44,079 pull requests restored). Sanitize, export and uploads passed in both.
• actionlint on both workflows reports only the existing for i loop in the restore step. shellcheck is clean on the new scripts. ruff check and ruff format --check pass.
• Review (a separate agent) found the pg_wrapper problem, the unchecked restore status, the empty-snapshot gap and the rest; all of these are fixed in 24bc7d6b. Not changed here: the existing pandas/pyarrow pins and readiness loop in local_sanitize_backup.sh.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QoXLf4KSfqzMFrdZGQG9Up