Skip to content

Add utopia CLI with backup subcommand (restore stubbed) - #712

Open
rollroyces wants to merge 1 commit into
deeplethe:devfrom
rollroyces:feat/backup-restore
Open

rollroyces wants to merge 1 commit into
deeplethe:devfrom
rollroyces:feat/backup-restore

Conversation

@rollroyces

Copy link
Copy Markdown
Contributor

Summary

Roadmap item: Enterprise — backup and restore commands (README §Roadmap). First cut: a separate utopia operator CLI with backup implemented and restore stubbed.

What's in this PR

  • crates/utopia-cli/ — new crate, depends only on utopia-core for AppConfig::load. So utopia backup doesn't pull in the server, extractor, chat loop, or any of the ~50 server modules.
  • crates/utopia-cli/src/main.rs — 589 lines, hand-rolled arg parser, manifest schema with sha256 checksums, backup shells out to pg_dump -Fc and tar, restore parses flags and bails with a design-doc pointer.
  • Cargo.toml — workspace members += ["crates/utopia-cli"]
  • .roadmap-proposals/backup-restore.md — design with 5 open questions

Why a separate crate (not a subcommand of utopia-server)

  1. The CLI doesn't need the HTTP server, the extractor, the chat loop, the mapping engine, or any of the 50+ modules pulled in by utopia-server. utopia-server is the largest crate in the workspace; making the operator compile it just to dump the database is a real CI / image-size cost.
  2. The CLI is its own surface: containers, Kubernetes Jobs, cron, operators' laptops. A separate boundary makes the attack surface and the image artifact explicit.
  3. The CLI reuses AppConfig::load() so connection info (UTOPIA_DATABASE_URL, UTOPIA_DATA_DIR, .env) is shared with the server. No duplicate config wiring.

utopia backup

$ utopia backup [flags]
  --output <PATH>            archive path; default utopia-<timestamp>.tar.gz
  --include-data-dir         also tar the data/ directory
  --dry-run                  print the plan, don't write anything
  --pg-dump <PATH>           path to pg_dump binary
  --tar <PATH>               path to tar binary
  --migration-url <URL>      override connection string for pg_dump

What it does:

  1. Resolve connection string and data_dir from AppConfig.
  2. Run pg_dump -Fc to a temporary file under the current directory. -Fc (custom format) is the only format pg_restore consumes.
  3. Optionally tar data/ into a second temporary file.
  4. Build manifest.json with schema_version, utopia_version, created_at, component paths/bytes, sha256 checksums.
  5. Write a single *.tar.gz containing manifest.json + pg_dump.custom + (optional) data/. Refuses to overwrite an existing archive.

utopia restore

Stubbed in this cut — parses flags, prints the plan, then bail!s with a pointer to the design doc. The backup side is independently useful (operators want to take backups even before restore works) and the restore side has design questions that benefit from running backup in production first.

Tests

8 unit tests, all pure (no live DB, no docker):

  1. parses_backup_minimalutopia backup --dry-run produces a BackupArgs with all defaults.
  2. parses_backup_full — every flag set, every field populated.
  3. parses_restore_requires_from--from is mandatory on restore.
  4. parses_restore_full — every flag set, every field populated.
  5. rejects_unknown_subcommandutopia frobnicate fails clearly.
  6. redact_url_host_keeps_userinfo_at_hostpostgres://u:***@h/db stays redacted on round-trip.
  7. redact_url_host_handles_no_atpostgres://h/db redacts nothing (no userinfo).
  8. hex_encode_known_valuehex_encode(&[0xde, 0xad]) == "dead".

Full workspace test: 799 passed, 0 failed.

Open questions for the maintainer

Full text in .roadmap-proposals/backup-restore.md. In priority order:

  1. Where does the CLI live? crates/utopia-cli/ (this proposal) vs. a bin/utopia.rs inside utopia-server. Recommendation: separate crate.
  2. Restore strategy. Drop-and-recreate (needs CREATEDB) vs. in-place pg_restore --clean. Recommendation: in-place --clean.
  3. Should the app docker image install postgresql-client? Recommendation: no — keep the runtime image minimal, ship a separate utopia-cli image with pg_dump and pg_restore. (Out of scope for this PR — lands the binary first.)
  4. Manifest version policy. Refuse forward-incompatible manifests on read; warn on older. Recommendation: refuse forward, warn on older.
  5. Should UTOPIA_BACKUP_DIR become a new config knob for the default output location? Recommendation: yes, trivially.

Out of scope (intentionally)

  • No restore implementation (stubbed, follow-up PR once Q2 is resolved).
  • No docker image change (Q3).
  • No UTOPIA_BACKUP_DIR config (Q5).
  • No automatic migration of older manifests — restore will only read schema_version == current.
  • No encryption-at-rest. Out of scope; the operator's filesystem encryption is the right layer.
  • No streaming upload to S3. Out of scope; the operator can pipe utopia backup --output - to aws s3 cp - s3://… today.

Checklist

  • DCO sign-off (Signed-off-by: in commit)
  • cargo fmt --all clean (verified locally)
  • cargo test -p utopia-cli 8 passed, 0 failed
  • cargo test --workspace 799 passed, 0 failed
  • cargo clippy -p utopia-cli clean (no warnings on the new crate)
  • Back-compat: no changes to existing server / extractor / store code
  • Existing PRs (PR GitHub and Jira write their timestamps to the second #691 on feat/instant-precision) unaffected — different branch, different files

Roadmap item #2 (Enterprise — backup and restore commands). First
cut: a separate `utopia-cli` crate that depends only on `utopia-core`
for config loading, so backup/restore don't pull in the server,
extractor, or chat loop.

What lands:
- crates/utopia-cli/src/main.rs (589 lines): hand-rolled arg parser,
  Manifest schema (schema_version, utopia_version, created_at,
  component paths + sha256 checksums), backup() that shells out to
  pg_dump -Fc + tar, restore() stubbed with a clear bail.
- crates/utopia-cli/Cargo.toml
- Cargo.toml workspace: add crates/utopia-cli to members
- .roadmap-proposals/backup-restore.md: design with 5 open questions

Flags on backup:
  --output, --include-data-dir, --dry-run, --pg-dump, --tar, --migration-url

Flags on restore (stubbed):
  --from, --target-data-dir, --pg-restore, --dry-run, --force, --yes

Tests: 8 unit tests, all pure (no live DB):
  arg parsing for both subcommands (minimal + full),
  --from is required on restore,
  unknown subcommand rejected,
  url-host redaction round-trips on urls with/without userinfo,
  hex_encode known value.

Restore side is intentionally stubbed — design doc explains why
(independent usefulness of backup, restore design questions benefit
from running backup in production first). Restore cut follows when
the maintainer answers the 5 open questions in the design doc.

The CLI reuses AppConfig::load() for connection info, so it picks
up the same .env / UTOPIA_DATABASE_URL / UTOPIA_DATA_DIR the server
uses. No duplicate config wiring.

Signed-off-by: Royce Lam <roycelam@umich.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant