docs(backup): design revision for off-host backup and restore [backup/restore 1/4]

[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged first) · #393 store · #394 format · #395 SnapshotEngine · next: PITR surface (#335 rebased), catalog index, backup job, restore job.

What

Revision of docs/design/15-backup-restore.md from #348, written against main 9f43d03, #384, and the companion PRs #393, #394, #395 as they stand after two adversarial review rounds. One commit, squashed with #348's work (Lee is co-author); supersedes #348. The document carries a revision log (items 1–14); the entries that change behavior:

Snapshot semantics. SnapshotEngine returns the table definition together with the items from one operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or a DeleteTable to commit between the definition read and the data snapshot, which #384 had already closed on PostgreSQL. The snapshot is opened in the CreateBackup request, so the backup represents the request time, as the service documents; backup_creation_date_time is the instant the snapshot pinned. Indexes are reported in whatever build state they had and only an index being deleted is dropped; a restore builds them from the items, and an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good.

Coordination. Per-job claims with fencing tokens replace the single-instance lease the draft assumed (PostgreSQL and MongoDB have none; readiness P0-7 is open). The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB (born owned), an in-process slot on SQLite; no row ever exists unowned. Heartbeats are written on a timer, not per uploaded part (the store trait exposes no part boundary); staleness is measured on the database's clock on every backend; a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation, conditioned on status = CREATING, and deletes its prefix only after the stale window.

DeleteBackup. The row is tombstoned DELETING, the manifest is deleted first, and the row goes to DELETED when the prefix lists empty. backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention.

Restore. The request creates the target table and a restore_jobs row atomically and returns CREATING (ResourceInUseException on a name race, LimitExceededException over the job limit); the job spools and verifies every data file (with .verified markers in a per-host spool) before writing any item; progress is recorded in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set, so restore memory does not grow with the table. N values are normalized before key text or key order is computed. The ACTIVE flip is conditional on the table still being a CREATING restore target; DeleteTable on a restore target fails the job. Index population stays per backend behind bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB.

Format. The manifest self-checksum is replaced by a sidecar over stored bytes. backup_arn is the backup's identity and source_table must agree with it. item_order with a per-file key_range (normalized N endpoints; none on an unordered manifest); the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL; md5 and uncompressed_bytes per file; explicit nulls; three-state vector_indexes of creation specifications; optional etag; dataFileS3Key as the full object key; one partition-aware BackupArn.

Store. Owner-only root re-checked by every operation, the owner-only rule on every descended directory including list and delete_prefix, an ancestor ownership rule, 0600/0700 at creation; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto.

Also specified: ListBackups paging and filtering; the restore override contract; sqlite-memory refusal; extenddb destroy leaves the store intact and backup purge is scoped to this instance's accounts; a restore spool outside the store root; max_concurrent_jobs as a semaphore shared by backups and restores; heartbeat_interval_secs; a concrete pool-size rule (pool_size >= max_concurrent_jobs + 10); SnapshotExpired codes (PostgreSQL 40001 and 72000, MongoDB 239 and 246); SnapshotInfo as a re-export once #394 merges.

Phasing is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers (catalog 004/005, data 005) are reserved up front so the PITR PR is off the critical path.

Why

The draft was written on 2026-09-10 against 4fd2c5f8. Since then #384 fixed two of its seven motivating defects, and a review of the draft against the code found three design-level problems (snapshot/definition atomicity, delete ordering, non-idempotent resume) and several assumptions the code does not satisfy (the lease, the backfill workers, a job queue). Two further review rounds against this revision and the companion PRs found internal contradictions (the checklist still described the pre-revision restore order; "heartbeat per part" could not be built against the store trait; "claim before row" is impossible on MongoDB; step 6 resolved rows "immediately" while step 4 forbade deleting for a stale window; an unordered-backup key set contradicted the memory formula) and undefined pieces (the restore job's row, claim and retention; where "verified" is recorded; the ACTIVE flip's condition), all closed in this revision. The implementation PRs in this series follow the revised text, so it should land with or before them.

Testing done

Documentation only. cargo test -p extenddb-storage-postgres --test doc_version_literals passes (the two forward-looking catalog version literals carry version-literal-ok markers). Measurements cited in the text (read, compression, bulk-load throughput; vacuum and WAL growth under a held snapshot) were taken on a 16-core host with PostgreSQL 16 and SQLite against a 1M-item fixture; each number carries its host, item size, transaction granularity, and tool caveats in the document.

Not changed here

• Sections 6 (PITR) and 9 (observability) beyond the items above.
• The PITR exception type already matched fix(backup): report point-in-time recovery as unsupported #335 in docs(design): add the backup and restore design #348 and is unchanged.
• The deployment guide and the backup manual (section 10) are PR 12.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: this document is the design record for the series.

Breaking changes

None (documentation).

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.