Repository navigation
fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores - #384
fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores#384robinnsc wants to merge 1 commit into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
robinnsc
changed the title
Nothing stopped two `extenddb serve` processes from opening the same SQLite file. The backend serializes writers with a lock inside the server process, so two servers write concurrently and can hit SQLITE_BUSY mid-transaction; and startup recovery (control-plane transitions, GSI and vector index rebuilds, and, with #384, abandoned restores) assumes no other server is running, so a second server starting up would undo the first one's work in progress. Nothing stopped `destroy` from unlinking the file under a running server either, which left that server serving an unlinked inode while a later `init` created a fresh database at the same path. `extenddb serve` on a file database now takes an exclusive flock(2) on `<database>.lock` before opening the database and holds it for the life of the process (the lock lives in the engine, so it is held while any clone is). `init` and `migrate` take the same lock for their duration through the bootstrapper's migration-lock hook, and `destroy` takes it before it removes the file; each refuses, rather than waits, when a server holds it. Read-only commands (`settings`, `manage`, `verify`, `status`) take no lock. A second holder fails with: another extenddb process is already using <db> (lock held on <db>.lock); stop it first, or point this command at a different database The lock file is named after the file SQLite actually opens, not the configured string: the location is parsed with sqlx's own SqliteConnectOptions (so `sqlite:` URL forms, percent-encoding such as `a%20b.sqlite`, and `file:` URIs resolve as the engine resolves them), and the path is canonicalized, so two spellings of one file, a `..` path, or a symlink and its target all share one lock. A disk file whose name happens to contain `mode=memory` is a file. In-memory databases take no lock. The kernel releases the lock when the process exits, including on kill -9, so there is no stale-lock cleanup. On non-Unix platforms no lock is taken and a warning is logged. flock is advisory and its behaviour on network filesystems depends on the server and mount; the docs say to keep SQLite databases on local disk. The directory holding the database must be writable so the lock file can be created. flock through libc rather than std's File::try_lock, which needs Rust 1.89; the workspace MSRV is 1.88. libc is already a dependency of the workspace and listed in every license notices file. Docs: troubleshooting entry for the error with its scope; README and the deployment and design guides now state that one instance per catalog is the supported deployment, that SQLite enforces it, and that PostgreSQL and MongoDB do not (per-instance caches without cross-instance invalidation, every worker on every instance, a liveness-only /health). The deployment guide previously said multiple PostgreSQL instances were consistent because there was no in-process cache, which was not true. Tests: database_file resolution for plain paths, sqlite: URLs, percent-encoding, file: URIs, in-memory forms, and a disk file named like a memory parameter; ServeLock refused on the same file, independent across files, shared across `..` spellings and across file and directory symlinks; the server factory refusing a second server on one file; destroy and migrate refused while a server holds the file and a server refused while migrate holds it. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
13c7f24 to
dc980eb
Compare
There was a problem hiding this comment.
Blocking on two things, both inline:
- the Postgres CreateBackup barrier doesn't pin the snapshot the way the code and docs say it does, and
- SQLite CreateBackup now holds the write lock for the whole read, which is a server-wide write stall on a big table. The rest are non-blocking.
| .execute(&mut *snapshot) | ||
| .await | ||
| .map_err(db_err)?; | ||
| meta.commit().await.map_err(db_err)?; |
There was a problem hiding this comment.
Blocking. This barrier doesn't hold. LOCK TABLE is a utility statement and doesn't take a snapshot in Postgres. The REPEATABLE READ snapshot is taken by the first SELECT, which here runs after meta.commit() has already released the FOR SHARE row. So the definition is captured at one instant and the items at a later one, with UpdateTable free to commit in between. I checked this on PG 15: BEGIN ... REPEATABLE READ; LOCK TABLE t ...; then an insert from another session, then SELECT count(*) sees the new row.
There was a problem hiding this comment.
Yeah good catch, swapped that to BEGIN RR RO; LOCK TABLE … ACCESS SHARE; SELECT 1 FROM <data> LIMIT 1; and only then releases the FOR SHARE row
| // instant, including against a concurrent UpdateTable. Items are | ||
| // read in rowid batches on the same connection, so memory is | ||
| // bounded by the batch size. | ||
| let _writer = self.write_lock.lock().await; |
There was a problem hiding this comment.
Blocking. This now holds the engine write lock for the whole read and write of the table. On main the item read ran before the lock, only the inserts were under it. The pool is WAL mode, so a deferred read transaction on a second connection already gives a stable snapshot without blocking writers.
There was a problem hiding this comment.
FIxed, the lock is now held only to read the definition, insert the backup row as CREATING, and open the deferred read transaction (snapshot fixed under the lock, so definition and items agree), items stream unlocked and are written in batches under short holds
| "SSESpecificationOverride", | ||
| ]; | ||
|
|
||
| fn validate_restore_table_overrides(body: &Value) -> Result<(), DynamoDbError> { |
There was a problem hiding this comment.
Now that the definition is in hand, BillingModeOverride and ProvisionedThroughputOverride are a few lines on CreateTableInput. Refusing them is a wire regression for clients that pass them (the console does). I'd implement at least those two and refuse only the index and SSE overrides.
| | `RestoreTableFromBackup` overrides | `BillingModeOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `ProvisionedThroughputOverride`, `SSESpecificationOverride` apply to the restored table | Not supported; a request carrying any of them returns `ValidationException` rather than restoring a table that differs from what was asked for | | ||
| | Multi-part base table keys (preview) | Not supported | A backup of a table created with `enable_multipart_keys` is refused at restore with `ValidationException`; the item paths address such tables by their first HASH and RANGE attribute only | | ||
| | `DeleteBackup` during a restore from that backup | `BackupInUseException` | Matches: `BackupInUseException` (HTTP 400) until the restore completes or fails | | ||
| | `DeleteTable` on a table being restored | `ResourceInUseException` | Matches on a restore target. An ordinary table in `CREATING` can still be deleted during its control-plane delay, where the service refuses | |
There was a problem hiding this comment.
This says "Matches on a restore target" without a backend qualifier, but the MongoDB diff only adds RestoreSummary and the DeleteBackup refusal. DeleteTable during a Mongo restore is not refused, and Mongo has no abandoned-restore sweep. Either qualify these rows or add the behaviour.
There was a problem hiding this comment.
Qualified every row by backend, and implemented the Mongo DeleteTable, and clarified that Mongo still has no abandoned restore sweep
| ## Version History | ||
|
|
||
| ### Catalog 0.0.3 (Current) | ||
| ### Catalog 0.0.4 (Current) |
There was a problem hiding this comment.
Worth a sentence here: any Postgres target the old bug left in CREATING matches the sweep predicate and gets removed on the first control-plane pass after upgrade. Right outcome, but operators will see tables disappear from ListTables and should know why.
| } | ||
| let rows_sql = format!( | ||
| "SELECT rowid, item_data FROM {table} \ | ||
| WHERE {cond} AND rowid > ? AND rowid <= ? ORDER BY rowid" |
There was a problem hiding this comment.
The restore cursor is the implicit rowid of backup_items, read across separately committed batches. backup_items has no INTEGER PRIMARY KEY, so a VACUUM between batches can renumber rowids and the restore skips or duplicates rows. Nothing in-tree runs VACUUM and #391 keeps a second extenddb out, but not a sqlite3 CLI. Give backup_items an INTEGER PRIMARY KEY or cursor on something stable.
There was a problem hiding this comment.
Done, backup_items has id INTEGER PRIMARY KEY and the restore cursors on it. An existing table is rebuilt in place by migrate, and the fixture test now checks id is present with rows in order.
| // This session sits idle for the whole copy. A server-side | ||
| // idle_session_timeout would end it and release the lock while the | ||
| // restore is still running, so it is disabled for this session only. | ||
| sqlx::query("SET idle_session_timeout = 0") |
There was a problem hiding this comment.
If this SET fails (managed instance that rejects it), every restore becomes a 500. Make it best-effort with a warning.
There was a problem hiding this comment.
Done, now best-effort with a warning
| // mid-copy leaves its target CREATING with no scheduled transition, | ||
| // which nothing above would ever move on. Logged and skipped on | ||
| // failure, so it cannot hold up the transitions above. | ||
| match self.sweep_abandoned_restores().await { |
There was a problem hiding this comment.
This pass is notify-driven with a 60 s idle timeout, so an abandoned target is removed between 60 and ~120 s, not "after 60 s". Each pass also opens a fresh out-of-pool connection per still-locked candidate. Fine, but say so in the comment.
There was a problem hiding this comment.
Corrected the comment
| .map_err(|e| StorageError::Internal(format!("Database error: {e}")))?; | ||
| .map_err(db_err)?; | ||
| sqlx::query(&format!( | ||
| "LOCK TABLE {} IN ACCESS SHARE MODE", |
There was a problem hiding this comment.
This lock is now held across the read and the interleaved catalog writes. A DeleteTable issued during the backup reaches the control-plane worker, whose DROP TABLE blocks behind it and stalls the whole worker pass until the backup finishes. Main had the same shape for the read only. Either SET LOCAL lock_timeout in the worker's drop or document it.
There was a problem hiding this comment.
Went with using SET LOCAL lock_timeout
| .unwrap_or(i64::MAX); | ||
| sqlx::query( | ||
| "INSERT INTO backup_items (backup_arn, pk, sk, item_data) \ | ||
| VALUES (?, '', NULL, ?)", |
There was a problem hiding this comment.
SQLite stores pk = '' while Postgres stores the real pk. Harmless today but the #348 migration-from-rows will want it. Also COPY_BATCH_ITEMS is imported but this file uses its own COPY_BATCH.
There was a problem hiding this comment.
Swapped to storing the Real pk
Nothing stopped two `extenddb serve` processes from opening the same SQLite file. The backend serializes writers with a lock inside the server process, so two servers write concurrently and can hit SQLITE_BUSY mid-transaction; and startup recovery (control-plane transitions, GSI and vector index rebuilds, and, with #384, abandoned restores) assumes no other server is running, so a second server starting up would undo the first one's work in progress. Nothing stopped `destroy` from unlinking the file under a running server either, which left that server serving an unlinked inode while a later `init` created a fresh database at the same path. `extenddb serve` on a file database now takes an exclusive flock(2) on `<database>.lock` before opening the database and holds it for the life of the process (the lock lives in the engine, so it is held while any clone is). `init` and `migrate` take the same lock for their duration through the bootstrapper's migration-lock hook, and `destroy` takes it before it removes the file; each refuses, rather than waits, when a server holds it. `init` holds it through the encryption key, default account, and admin user, not only the schema, so a server cannot start in the window where the schema exists but the key does not. Read-only commands (`settings`, `manage`, `verify`, `status`) take no lock. A second holder fails with: another extenddb process is already using <db> (lock held on <db>.lock); stop it first, or point this command at a different database The lock file is named after the file SQLite actually opens, not the configured string: the location is parsed with sqlx's own SqliteConnectOptions (so `sqlite:` URL forms, percent-encoding such as `a%20b.sqlite`, and `file:` URIs resolve as the engine resolves them), and the path is canonicalized, so two spellings of one file, a `..` path, or a symlink and its target all share one lock. A dangling symlink (the target not created yet) is followed by hand, since canonicalize refuses it. A disk file whose name happens to contain `mode=memory` is a file. In-memory databases take no lock. The lock file is created 0600 and an existing one is tightened to 0600 on open, matching the database and its sidecars: flock needs only a read handle, so a world-readable lock file would let any local user hold the lock and keep the server from starting. The kernel releases the lock when the process exits, including on kill -9, so there is no stale-lock cleanup. On non-Unix platforms no lock is taken and a warning is logged. flock is advisory and its behaviour on network filesystems depends on the server and mount; the docs say to keep SQLite databases on local disk. The directory holding the database must be writable so the lock file can be created. flock through libc rather than std's File::try_lock, which needs Rust 1.89; the workspace MSRV is 1.88. libc is already a dependency of the workspace and listed in every license notices file. Docs: troubleshooting entry for the error with its scope; README and the deployment and design guides now state that one instance per catalog is the supported deployment, that SQLite enforces it, and that PostgreSQL and MongoDB do not (per-instance caches without cross-instance invalidation, every worker on every instance, a liveness-only /health). The deployment guide previously said multiple PostgreSQL instances were consistent because there was no in-process cache, which was not true. Tests: database_file resolution for plain paths, sqlite: URLs, percent-encoding, file: URIs, in-memory forms, and a disk file named like a memory parameter; ServeLock refused on the same file, independent across files, shared across `..` spellings, across file and directory symlinks, and across a dangling symlink and its future target (one and two links); the lock file created 0600 and an existing 0644 one tightened; the server factory refusing a second server on one file; destroy and migrate refused while a server holds the file and a server refused while migrate holds it. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
Nothing stopped two `extenddb serve` processes from opening the same SQLite file. The backend serializes writers with a lock inside the server process, so two servers write concurrently and can hit SQLITE_BUSY mid-transaction; and startup recovery (control-plane transitions, GSI and vector index rebuilds, and, with ExtendDB#384, abandoned restores) assumes no other server is running, so a second server starting up would undo the first one's work in progress. Nothing stopped `destroy` from unlinking the file under a running server either, which left that server serving an unlinked inode while a later `init` created a fresh database at the same path. `extenddb serve` on a file database now takes an exclusive flock(2) on `<database>.lock` before opening the database and holds it for the life of the process (the lock lives in the engine, so it is held while any clone is). `init` and `migrate` take the same lock for their duration through the bootstrapper's migration-lock hook, and `destroy` takes it before it removes the file; each refuses, rather than waits, when a server holds it. `init` holds it through the encryption key, default account, and admin user, not only the schema, so a server cannot start in the window where the schema exists but the key does not. Read-only commands (`settings`, `manage`, `verify`, `status`) take no lock. A second holder fails with: another extenddb process is already using <db> (lock held on <db>.lock); stop it first, or point this command at a different database The lock file is named after the file SQLite actually opens, not the configured string: the location is parsed with sqlx's own SqliteConnectOptions (so `sqlite:` URL forms, percent-encoding such as `a%20b.sqlite`, and `file:` URIs resolve as the engine resolves them), and the path is canonicalized, so two spellings of one file, a `..` path, or a symlink and its target all share one lock. A dangling symlink (the target not created yet) is followed by hand, since canonicalize refuses it. A disk file whose name happens to contain `mode=memory` is a file. In-memory databases take no lock. The lock file is created 0600 and an existing one is tightened to 0600 on open, matching the database and its sidecars: flock needs only a read handle, so a world-readable lock file would let any local user hold the lock and keep the server from starting. The kernel releases the lock when the process exits, including on kill -9, so there is no stale-lock cleanup. On non-Unix platforms no lock is taken and a warning is logged. flock is advisory and its behaviour on network filesystems depends on the server and mount; the docs say to keep SQLite databases on local disk. The directory holding the database must be writable so the lock file can be created. flock through libc rather than std's File::try_lock, which needs Rust 1.89; the workspace MSRV is 1.88. libc is already a dependency of the workspace and listed in every license notices file. Docs: troubleshooting entry for the error with its scope; README and the deployment and design guides now state that one instance per catalog is the supported deployment, that SQLite enforces it, and that PostgreSQL and MongoDB do not (per-instance caches without cross-instance invalidation, every worker on every instance, a liveness-only /health). The deployment guide previously said multiple PostgreSQL instances were consistent because there was no in-process cache, which was not true. Tests: database_file resolution for plain paths, sqlite: URLs, percent-encoding, file: URIs, in-memory forms, and a disk file named like a memory parameter; ServeLock refused on the same file, independent across files, shared across `..` spellings, across file and directory symlinks, and across a dangling symlink and its future target (one and two links); the lock file created 0600 and an existing 0644 one tightened; the server factory refusing a second server on one file; destroy and migrate refused while a server holds the file and a server refused while migrate holds it. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
dc980eb to
8f83169
Compare
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. create_backup and restore_table_from_backup on PostgreSQL checked for a column named `sk`, which data tables do not have (sk_s, sk_n, sk_b). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store pk and item_data only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in item_data and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. RestoreTableFromBackup overrides. BillingModeOverride and ProvisionedThroughputOverride are implemented on all three backends and passed to storage as an explicit `overrides: RestoreTableOverrides` parameter on BackupEngine::restore_table_from_backup (a trait change). BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST normalization and refuses, before any target is created, a PROVISIONED restore of a backup whose GSIs carry no throughput (the service needs GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, SSESpecificationOverride, OnDemandThroughputOverride, and VectorIndexOverride are refused with ValidationException before any storage read; they were silently ignored. Throughput values reuse CreateTable's validator so the error text matches. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add backup_definitions (one row per backup, the definition as JSON behind a version marker; extenddb_storage::backup_definition) and table_restores (one row per restored table: source backup ARN and restore time, not a foreign key to backups so the summary outlives the backup, as on the service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore cursors on it rather than the implicit rowid, which a VACUUM between batches could renumber. An existing backup_items is rebuilt in place (copy in rowid order, idempotent). backup_items.pk now holds the real partition key on SQLite as it does on PostgreSQL. Backups and restores from before the upgrade have no definition rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and the snapshot is pinned by a real read of the table (LOCK TABLE is a utility statement and does not establish a REPEATABLE READ snapshot; the first revision relied on it and the definition and the items could come from different instants). UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. SQLite CreateBackup takes the engine write lock only to read the definition, insert the backup row as CREATING, and open the item reader's WAL snapshot; the items are then streamed without the lock and written in batches under short lock holds, so other writers wait for a batch rather than the whole backup (a 100,000-item backup previously stalled every write for about twelve seconds). The copy runs detached from the request, so a client that disconnects does not leave a half-written CREATING backup; one left by a crash is removed at startup. A CREATING backup is visible to ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses it. The long reader pins WAL checkpointing for the backup's duration. A file-backed SQLite pool is clamped to at least two connections so the reader and the batch writer can coexist. A restore registers itself in the same transaction that creates its target on PostgreSQL (backup row FOR SHARE, table_restores inserted with the tables row) and SQLite, so DeleteBackup orders strictly before (the restore then sees no backup) or after (it sees the restore, BackupInUseException). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims before reading and releases once its CREATING target exists; DeleteBackup claims the same way before dropping data; stale claims from a crashed process are cleared at startup. DeleteTable on a table being restored returns ResourceInUseException on all three backends. DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection (its idle_session_timeout SET is best-effort, for managed instances that refuse it); the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free, which with the pass's 60 s idle timeout means between 60 and about 120 s. SQLite removes the same targets at startup, which is safe because one server at a time holds the file (#391). The control-plane worker's DeleteTable drop runs under a 3 s lock_timeout and drops data tables before deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE lock fails fast and the row stays DELETING for the next pass (the previous order deleted the row first, so a failed drop was never retried). The stale-vector-index recovery skips indexes whose parent table is DELETING and cleans up if the parent vanishes under it, so the reordered drop cannot leave an orphan vector table. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and the unsupported overrides above. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported and its summary row is recounted; docs/differences-from- dynamodb.md gains a Backup and Restore section with every row qualified by backend (what is restored, the seven overrides, the refusals, CREATING visibility, the delete race, crash recovery, where backups live, the consistency model and WAL cost); the upgrade manual has a 0.0.4 section that warns PostgreSQL operators that targets the old bug left in CREATING are removed on the first control-plane pass after the upgrade, describes the SQLite lock, and notes WAL headroom; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash and sort keys compared item by item; a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads; RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput; BillingModeOverride in both directions; a refused PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no target). crates/storage-postgres/tests/backup_restore.rs (21, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier including a production-path test that fails if the snapshot-pinning read is removed and the two transaction shapes that show why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both orders, the abandoned-restore sweep, the drop lock_timeout leaving a row DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a writer that keeps making progress during a backup, items written after the snapshot excluded, a cancelled request finishing with every item, the startup sweeps, a 0.0.3 database in the old row format, the backup_items rebuild). Unit tests for the definition normalization and override preflight (12), the engine's override parsing and refusals (15), Mongo's claim predicates and GSI throughput handling, CatalogVersion ordering, the newer-catalog refusals, and the vector-recovery predicate. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
8f83169 to
ba9f869
Compare
|
#395 (SnapshotEngine) is branched from this PR and depends on it; its diff shows this PR's commit until this merges. |
…t code Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture, and two review rounds against the companion PRs #393, #394 and #395. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time, as DynamoDB documents; backup_creation_date_time is the instant the snapshot pinned. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite; no row ever exists unowned. Heartbeats are a timer, staleness is measured on the database's clock, and a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING and deletes its prefix only after the stale window. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically; ResourceInUseException on a name race, LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional; DeleteTable on a restore target fails the job. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto (MinIO no longer publishes images). - ListBackups paging and filtering, the restore override contract, sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs and a concrete pool-size rule are specified. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path. Co-authored-by: Lee Hannigan <lhnng@amazon.com>
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
Requesting changes. The storage engineering under this PR held under reproduction: both of yesyayen's blocking items are fixed and each fix fails its test when reverted; restore atomicity against DeleteBackup held on PostgreSQL in both orders and across 100 race iterations on MongoDB; the migration, the sweep predicate, the backup_items cursor, the downgrade gate, and crash cleanup all behaved as described; cross-account access to backups is refused on every backend under every ARN variation tried; the restored data matched DynamoDB item for item on every index and projection. The problems are in the surface customers and operators touch, and four of them block the merge.
Blocking
1. The merged tree does not compile. #338 landed the Cassandra backend on main this morning with the pre-PR four-argument restore_table_from_backup. cargo check --workspace --all-targets on the merge of origin/main and ba9f869 fails in one crate, extenddb-storage-cassandra, E0050 (4 parameters, trait has 5). GitHub reports MERGEABLE because no file overlaps. Adding the parameter alone leaves three defects: the Cassandra restore hardcodes global_secondary_indexes: None and local_secondary_indexes: None, ignores both overrides, and never calls ensure_single_part_base_key or ensure_restorable, because those refusals live in each SQL storage backend rather than in the engine. Either port the definition, the overrides, and both refusals into Cassandra, or add a Cassandra row to docs/differences-from-dynamodb.md saying what its restore does not honour. In both cases move the two refusals into crates/engine/src/backup.rs so a backend cannot skip them.
2. RestoreTableFromBackup copies every item inside the request on PostgreSQL and SQLite, and a client timeout cancels it. Both backends await copy_backup_items and only then return CREATING for a table that is already ACTIVE (the comment at backup_engine.rs:1282 says so). Reproduced with 40,000 items and two GSIs, client read timeout 3 s, no retries. PostgreSQL: the target sat CREATING with zero items, then the sweep removed it 64 s after the disconnect; the restore was lost. SQLite: the target is still CREATING four minutes later, a retry into the same name returns ResourceInUseException, DeleteTable on it returns ResourceInUseException ("Table is being restored"), and DeleteBackup on the source returns BackupInUseException. The customer is locked out of the name and the backup until the process restarts. boto3 and the CLI default to a 60 s read timeout, and with one INSERT per item per index over a network a few hundred thousand items cross it, so large restores cannot succeed with stock clients. SQLite CreateBackup in this PR already detaches its copy with tokio::spawn for this reason (backup.rs:799). Do the same here on both backends: commit the CREATING target and its table_restores row, spawn the copy, return at once. Add a live test that drops the client mid-restore and asserts ACTIVE with the full count. This also makes the DeleteBackup and DeleteTable refusals observable the way the docs describe them.
3. DescribeBackup still returns the pre-PR subset. Against DynamoDB, a backup of a table with a GSI, an LSI, and TTL enabled returns SourceTableFeatureDetails with GlobalSecondaryIndexes, LocalSecondaryIndexes, and TimeToLiveDescription, and SourceTableDetails.ProvisionedThroughput ({0, 0} for on-demand; the member is required in the shape). ExtendDB omits all of them on all three backends, although backup_definitions now holds the data. A customer auditing a backup is told it has no indexes; a strongly typed SDK can fail to deserialize the response. Populate both from the stored definition.
4. The PR body and the docs state behaviour the build does not have. Checked on servers built from ba9f869: "DeleteTable on a table that is being restored returns ResourceInUseException on all three backends" (only MongoDB; see item 2); "SQLite similarly exposes CREATING while it copies" (the caller blocks until AVAILABLE; only a concurrent observer sees CREATING); "a backup is never stuck undeletable" (false after a MongoDB crash during $out, and after item 2 on SQLite); "backups of tables with vector indexes are refused" (only the restore is refused; CreateBackup returns an AVAILABLE backup). Every claim in the body and in differences-from-dynamodb.md needs a live check per backend before the next revision.
Wire behaviour, measured against DynamoDB
Each of these was run on the service and on all three ExtendDB backends.
RestoreSummary.RestoreDateTimeis the restore wall clock. The service reports the source backup's creation time (measured:RestoreDateTime1791976431.438 equal toBackupCreationDateTime, restore issued 74 s later). A client reading it to learn which point in time was restored gets the wrong instant.- The service drops
RestoreSummaryfrom DescribeTable once the table is ACTIVE (absent at ACTIVE and 30 s later). ExtendDB keeps it withRestoreInProgress: false, andtests/test_backup_restore_fidelity.py:252-254pins that. RestoreSummary.SourceTableArnis never emitted; the service sets it.- The CREATING restore response carries GSI
IndexStatus: ACTIVE; the service omits the member. - A malformed
BackupArnreturns AccessDeniedException; the service returns ValidationException. - Restoring into an existing table name returns ResourceInUseException; the service returns TableAlreadyExistsException.
- An empty
GlobalSecondaryIndexOverrideorLocalSecondaryIndexOverridelist and anSSESpecificationOverrideare refused; the service accepts them as no-ops. The blanket refusal is documented, the no-op case is not. - The PROVISIONED-with-GSIs refusal tells the caller to pass
GlobalSecondaryIndexOverride, which ExtendDB then refuses. The fidelity test pins that text at line 649. ListBackupsignoresLimitand never returnsLastEvaluatedBackupArn; every row comes back in one response. The service caps at 100 with a token.
PostgreSQL
- The restore's ownership is a session advisory lock on a dedicated connection that issues no statement for the whole copy.
SET idle_session_timeout = 0is best-effort and the comment atbackup_engine.rs:118-123says managed services may reject it. On RDS, Aurora, or behind a transaction-mode pooler the idle session can end while the copy continues; after 60 s the sweep flips the target to DELETING and the restore fails with "deleted while it was being restored". Heartbeat the lock session, or gate the sweep on a liveness signal that does not depend on an idle connection. Item 2 does not remove this; a long detached copy has the same exposure. - "Bounded memory" bounds the Rust buffer. The copy is one PostgreSQL transaction (
beginat line 325, onecommitafter the loop), and secondary-index rows are written one INSERT per item per index in the nested loop at line 464 while base rows are batched. A million-item table with four GSIs is one transaction with four million round trips, WAL held throughout, target pinned CREATING throughout. - A failure to open the lock connection (
max_connectionsreached) maps to an Internal error. Map it to LimitExceeded so clients back off. restore_table_to_point_in_time(lines 1393-1420) is unreachable, uses a fixed backup name, and discards its cleanup error. Delete it.
SQLite and MongoDB
- CreateBackup on a table with a vector index returns an AVAILABLE backup that RestoreTableFromBackup always refuses. The customer finds out at restore time. Refuse at CreateBackup with the same message.
- MongoDB: a crash during the base
$outcopy, beforerestore_backfill_pendingis set, leaves a CREATING target that nothing sweeps or resumes. DeleteTable refuses it and DeleteBackup of the source returns BackupInUse until the catalog is edited by hand. Crashes after$outself-heal. Add a sweep for CREATING targets carryingrestore_source_backup_arnwith no pending flag and no live operation, or let DeleteTable remove them. - MongoDB ignores
--data-dband--catalog-dband has no serve lock, so two processes on one replica set share one catalog, andsweep_stale_backup_claimsat startup clears the other process's in-flight markers. Pre-existing, but the sweep makes it matter. pool_size = 1is raised to 2 for a file-backed database with no log line and no mention in the config docs. A file-backed backup also pins one pooled connection for its duration, sopool_size >= 3is wanted for undisturbed concurrency; say so.BackupMetadataClaim's Drop release only runs when a runtime handle is current; a comment should say the startup sweep is the backstop.- The MongoDB backup is not a point-in-time snapshot (
$outreads the live collection). Line 93 says so in the general consistency row; the Backup and Restore section should state the per-item mixed-state consequence where an operator choosing MongoDB will read it.
Tenancy and validation
BackupTableDefinition::apply_toapplies the stored definition withoutvalidate_create_table. A definition with 30 GSIs, a GSI on an undefined attribute, 101 projected attributes, or negative throughput restores to an ACTIVE table that CreateTable would reject. Triggering it needs catalog write access; it is still wrong data stored, and the service re-validates.RESTORE_SLOTSis a process-global semaphore of four. One account can hold all four and every other account's restores wait 30 s and get LimitExceededException. Make the limit per account.converge_catalog_versionandrefuse_newer_catalogtreat a storedcatalog_versionthe binary cannot parse as "proceed" and stamp the compiled version over it (older_or_equal_or_absent_is_allowedassertsSome("garbage")is allowed). That defeats the downgrade guard in the one case it exists for. Fail closed.- Pre-existing, outside this diff:
RestoreTableFromBackup,DescribeBackup, andDeleteBackupare authorized againstarn:...:table/*becauseextract_table_namereads onlyTableName(crates/server/src/request_helpers.rs:42-46, 289-294; the server log showsresource="...:table/*"on a denial). A policy scoped to a target table or a backup ARN can never match. The service authorizes on the backup ARN and the target table. This PR makes restore a first-class feature on top of that gap; fix it here or file an issue; either way it needs recording.
Upgrade path and docs
Walked literally from docs/manuals/07-upgrade-manual.md on a 0.0.3 deployment created by the 0.1.13 binary.
extenddb migrate --config extenddb.tomlas written fails: it needs PostgreSQL admin credentials the manual never mentions, and the error isError: Internal("Internal(\"Cannot connect as admin: error returned from database: password authentication failed for user \"lhnng\"\")"). Document--pg-userandEXTENDDB_PG_PASSWORDin the command, flatten the nested error, and say what to pass.- The manual says to run
migratewithout--yesfirst to see what is pending; the tool prints the plan and thenError: --yes is required to apply migrationswith exit status 1, so a scripted upgrade underset -estops at the preview. Exit 0 for the preview, or a documented distinct code. - On SQLite the upgrade rebuilds
backup_itemsby copying every row into a new table in one write transaction (schema.rs:494-553). No manual mentions it, the dry run does not name it, and the function logs nothing. Document it, name it in the dry run when the old shape is present, and log start and finish with the row count. - The sweep that removes targets the old bug left in CREATING logs nothing. Operators will see tables vanish from ListTables with no server record of why. Log each removal by name.
- The manual's backup and rollback commands hardcode
extenddb_catalog,extenddb, andOWNER extenddb;initlets the operator choose all three. There is no SQLite subsection (stop, copy the file, migrate, roll back by restoring the copy). - 0.0.4 and migration 003 are also claimed by #368 and #335. The migration ledger keys on filename and the gate compares version numbers, so if #368 merges first and stamps 0.0.4, this binary applies its own 003 and converges with no refusal: two schemas named 0.0.4. Whoever merges second renumbers to 004 and 0.0.5 (and the two assertions in
migrations.rstests, and the manual's 0.0.4 section). The order should be agreed now. differences-from-dynamodb.mdnames three backends; the merged tree ships four.- The
Storagetrait and the on-disk format changed.CONTRIBUTING.mdanddocs/rfcs/README.mdrequire an accepted RFC for both; the PR links #348, a design PR, not a numbered accepted RFC. - The PR template asks for a
## Breaking changessection; the changes are spread through the prose, and the one customers will hit first (overrides that were silently ignored are now hard errors) deserves its own line.
Minor
Backup ARN timestamp is 13 digits where the service zero-pads to 14; BackupCreationDateTime is whole seconds where the service is sub-second; the restore response adds LastUpdateToPayPerRequestDateTime, which the service omits; ListBackups summaries omit TableId; error __type namespace differs for ValidationException and AccessDeniedException; with two invalid inputs the service checks the ARN account before the target name.
| account_id: &str, | ||
| target_table_name: &str, | ||
| backup_arn: &str, | ||
| overrides: RestoreTableOverrides, |
There was a problem hiding this comment.
main now has crates/storage-cassandra/src/backup_engine.rs (#338, merged this morning) with the old four-argument signature, so the merge fails to compile: cargo check --workspace --all-targets on origin/main plus this branch gives E0050 in extenddb-storage-cassandra and nothing else. Adding the parameter there leaves three defects: the Cassandra restore hardcodes global_secondary_indexes: None and local_secondary_indexes: None, ignores the overrides, and never calls ensure_single_part_base_key or ensure_restorable, which this PR enforces inside the PostgreSQL and SQLite backends rather than in the engine. Port the definition and both refusals into Cassandra, or document in the differences matrix what its restore does not honour, and move the two refusals into the engine so a backend cannot skip them.
| // CREATING with no scheduled transition, so nothing else would | ||
| // move it on, and the name would stay taken. Deleting the target | ||
| // cascades its table_restores row. | ||
| let copied = self.copy_backup_items(&desc, &backup_arn).await; |
There was a problem hiding this comment.
The copy is awaited inside the request future, so the response at line 1282 says CREATING for a table that is already ACTIVE, and a client that times out drops the future mid-copy. Reproduced with 40,000 items and two GSIs, read timeout 3 s, no retries: the target sat CREATING with zero items and the sweep removed it 64 s after the disconnect. The restore was lost. boto3 and the CLI default to 60 s, so any restore whose copy takes longer cannot succeed with a stock client. Commit the CREATING target and its table_restores row, spawn the copy, and return at once, the way SQLite CreateBackup does at backup.rs:799. Add a live test that drops the client and asserts the target reaches ACTIVE with the full count.
| // last one; until then the target is CREATING and refuses every | ||
| // data-plane request. A failure part-way is cleaned up here, a | ||
| // crash part-way by the startup sweep. | ||
| let copied = self.copy_backup_items(&desc, &backup_arn).await; |
There was a problem hiding this comment.
The SQLite path has the same shape. With the client gone after 2 s on a 40,000-item restore, the target is still CREATING with RestoreInProgress: true four minutes later; a retry into the same name returns ResourceInUseException, DeleteTable on the target returns ResourceInUseException ("Table is being restored"), and DeleteBackup on the source returns BackupInUseException. SQLite sweeps abandoned restores only at startup, so the name and the backup are locked until the process restarts. create_backup at line 799 already detaches its copy with tokio::spawn and the comment there names this hazard; the restore path needs the same treatment.
| // restore is still running, so disable it for this session when the | ||
| // server permits. Some managed PostgreSQL services reject this SET; | ||
| // ownership still works there, but remains subject to their idle timeout. | ||
| if let Err(e) = sqlx::query("SET idle_session_timeout = 0") |
There was a problem hiding this comment.
This connection issues no statement after pg_try_advisory_lock for the whole copy. When the SET is refused (the comment says managed services refuse it) or the connection sits behind a transaction-mode pooler, the idle session can end while the copy continues, the lock frees, and after 60 s the sweep flips the target to DELETING; the restore then fails with "deleted while it was being restored". That covers RDS, Aurora, and PgBouncer. Heartbeat the session while the copy runs, or gate the sweep on a liveness signal that does not depend on an idle connection.
| "Backup not found: {backup_arn}" | ||
| ))); | ||
| } | ||
| let mut tx = self.data_pool.begin().await.map_err(db_err)?; |
There was a problem hiding this comment.
The whole copy is one transaction: begin here, one commit after the loop. The 500-item / 4 MiB bound limits the Rust buffer only. In write_restore_batch the base rows go in one multi-row INSERT, but the loop at line 464 writes secondary-index rows one INSERT per item per index. A million-item table with four GSIs is one transaction with four million round trips, WAL held for the duration, and the target pinned CREATING throughout, which widens the sweep exposure above. Batch the index rows per index per batch, and either commit per batch with cleanup on failure or document the single-transaction cost for large restores.
| # DescribeTable keeps reporting where the table came from, finished. | ||
| summary = table["RestoreSummary"] | ||
| assert summary["SourceBackupArn"] == backup_arn | ||
| assert summary["RestoreInProgress"] is False |
There was a problem hiding this comment.
DynamoDB does not return RestoreSummary on DescribeTable once the table is ACTIVE (checked at ACTIVE and 30 s later). This assertion pins the ExtendDB behaviour as correct. The test should assert the member is absent after ACTIVE, and the engine should stop emitting it.
| | What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, SSE settings. Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand | The same set on PostgreSQL, SQLite, and MongoDB, plus table class and on-demand throughput. Streams, TTL, tags, and deletion protection are not restored. A backup written by a version before catalog 0.0.4 carries no table definition and restores as keys and items only, with 5/5 provisioned throughput | | ||
| | `RestoreTableFromBackup` overrides | Current request members are `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, and `VectorIndexOverride`; DynamoDB applies them to the restored table | PostgreSQL, SQLite, and MongoDB support `BillingModeOverride` and `ProvisionedThroughputOverride`. Switching to `PAY_PER_REQUEST` drops table and GSI throughput. `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, and `VectorIndexOverride` are refused with `ValidationException` before the target table is created. Switching an on-demand backup with GSIs to `PROVISIONED` is likewise refused because it requires `GlobalSecondaryIndexOverride` | | ||
| | Multi-part base table keys (preview) | Not supported | A backup of a table created with `enable_multipart_keys` is refused at restore with `ValidationException`; the item paths address such tables by their first HASH and RANGE attribute only | | ||
| | `DeleteBackup` during a restore from that backup | `BackupInUseException` | `BackupInUseException` (HTTP 400) until the restore completes or fails, on all three backends. PostgreSQL and SQLite register the restore in the same transaction that creates its target, with the backup row held so a concurrent `DeleteBackup` orders strictly before (the restore then sees no backup) or after (it sees the restore). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims the backup before reading it and releases the claim once its `CREATING` target exists; `DeleteBackup` claims it the same way before dropping data. A claim left by a crashed process is cleared at the next startup, so a backup is never stuck undeletable; until then a restore or delete of that backup gets `BackupInUseException` | |
There was a problem hiding this comment.
"A backup is never stuck undeletable" is false in two cases on this branch: a MongoDB crash during the base $out copy leaves a CREATING target that blocks DeleteBackup with BackupInUse until the catalog is hand-edited, and on SQLite a restore whose client disconnected leaves the same state until restart. The next row's "PostgreSQL, SQLite, and MongoDB match" is only observable on MongoDB: on the SQL backends the restore completes inside the request, so a sequential client's DeleteTable or DeleteBackup after the response succeeds every time.
| | Restore interrupted by a server crash | Managed by the service | PostgreSQL removes an abandoned `CREATING` restore target on a control-plane pass; SQLite removes one when the next server starts. MongoDB has no abandoned-restore sweep: the target remains `CREATING`, the API continues to refuse `DeleteTable`, and an operator must remove its catalog and data collections by hand | | ||
| | Point-in-time recovery | 35-day window | Not supported; `UpdateContinuousBackups` and `RestoreTableToPointInTime` are refused | | ||
| | Where backups live | Managed storage, independent of the table | Inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database. Take database-level backups for off-host copies until the backup store program ships | | ||
| | Backup creation status | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE` | SQLite similarly exposes `CREATING` while it copies: size and item count remain zero, `DeleteBackup` is refused, and `RestoreTableFromBackup` reports the backup missing. PostgreSQL and MongoDB complete their copy before returning and expose only `AVAILABLE` | |
There was a problem hiding this comment.
"SQLite similarly exposes CREATING while it copies" reads as if the caller gets a CREATING response. The caller blocks until AVAILABLE on all three backends (1.4 s for 50,000 items; minutes for a large table), and only a concurrent observer sees the CREATING row. Say that, and note that CreateBackup blocks for the copy duration, or return CREATING and let the detached task finish, which the SQLite code already supports.
| Upgrade sequence, on PostgreSQL and SQLite alike: | ||
|
|
||
| ```bash | ||
| extenddb stop --config extenddb.toml |
There was a problem hiding this comment.
Followed literally on a 0.0.3 deployment this fails: migrate connects as a PostgreSQL admin and the manual gives no credentials, so it falls back to the OS user and prints Error: Internal("Internal(\"Cannot connect as admin: ... password authentication failed for user \"lhnng\"\")"). Show --pg-user and EXTENDDB_PG_PASSWORD here, and flatten that nested error in the tool. The preview step on line 203 also exits 1 with Error: --yes is required, so a scripted upgrade under set -e stops at the preview; exit 0 for the preview or say the non-zero exit is intended. Nothing here tells a SQLite operator about the backup_items rebuild, and the backup and rollback steps above hardcode extenddb_catalog, extenddb, and OWNER extenddb.
|
|
||
| CREATE INDEX IF NOT EXISTS idx_table_restores_backup ON table_restores (source_backup_arn); | ||
|
|
||
| UPDATE settings SET value = '0.0.4' WHERE key = 'catalog_version'; |
There was a problem hiding this comment.
#368 and #335 also ship a 003_*.sql that writes 0.0.4. The ledger keys on filename and converge_catalog_version compares numbers, so if #368 merges first and stamps 0.0.4, this binary applies its own 003 and converges with no refusal: two different schemas both called 0.0.4, and the startup gate cannot tell them apart. Whoever merges second renumbers to 004 and 0.0.5 (plus the two assertions in the migrations.rs tests and the manual's 0.0.4 section). The order should be agreed before any of the three lands.
This branch has not been deployed
4 participants
What
PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. One commit, rebased on current main (which includes #391).
Scope against the readiness review's P0-2. This closes the two restore defects that review said needed fixes of their own (PostgreSQL loses sort keys and hangs the target in CREATING; both backends drop indexes) and adds the restore-fidelity test it asked for. It does not add point-in-time recovery (
UpdateContinuousBackupsandRestoreTableToPointInTimeare still refused; #335 reports PITR honestly) or off-host backup storage (backups still live in the database; #336 and #337). Those remain open.Sort keys.
create_backupandrestore_table_from_backupchecked for a column namedsk, which data tables do not have (sk_s,sk_n,sk_b). Backups of every table with a sort key lost their sort keys, and restoring one left the target in CREATING for good. Backups now storepkanditem_dataonly; restore derives every key column from the item. Backups written by earlier binaries carry the whole item initem_dataand restore correctly.Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (it was always 5/5); table class, SSE specification, on-demand throughput. Streams, TTL, tags, and deletion protection are not restored, as on the service. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED.
RestoreTableFromBackupoverrides.BillingModeOverrideandProvisionedThroughputOverrideare implemented on all three backends. They reach storage as an explicitoverrides: RestoreTableOverridesparameter onBackupEngine::restore_table_from_backup(aStoragetrait change, see below).BackupTableDefinition::apply_toapplies them before the PAY_PER_REQUEST normalization and refuses, before any target is created, a PROVISIONED restore of a backup whose GSIs carry no throughput (the service needsGlobalSecondaryIndexOverridethere).GlobalSecondaryIndexOverride,LocalSecondaryIndexOverride,SSESpecificationOverride,OnDemandThroughputOverride, andVectorIndexOverrideare refused withValidationExceptionbefore any storage read; previously every override was silently ignored. Throughput values reuse CreateTable's validator, so the error text matches CreateTable's.Catalog 0.0.4:
crates/storage-postgres/migrations/003_backup_definitions.sqland the SQLite schema addbackup_definitions(one row per backup; the definition as JSON behind a version marker,crates/storage/src/backup_definition.rs) andtable_restores(one row per restored table; not a foreign key tobackups, so the summary outlives the backup as on the service).backup_itemsgainsid INTEGER PRIMARY KEY; the restore cursors on it rather than the implicit rowid, which aVACUUMbetween separately committed batches could renumber. An existingbackup_itemsis rebuilt in place byextenddb migrate(copy in rowid order, idempotent).backup_items.pknow holds the real partition key on SQLite, as on PostgreSQL.RestoreSummary.SET_CATALOG_VERSION_SQL), the same change fix(backup): report point-in-time recovery as unsupported #335 makes. They refuse to write it when the stored version is newer, andextenddb migraterefuses a catalog newer than the binary outright, so an older binary can no longer "migrate" a newer catalog, stamp its version onto it, and then pass its own startup gate.Consistency:
FOR SHAREfrom its first catalog read until its data snapshot is taken, and theREPEATABLE READsnapshot is pinned by a real read of the table before the row is released. The first revision relied onLOCK TABLEfor this; it is a utility statement and does not establish the snapshot, so the definition and the items could come from different instants (reviewer finding, reproduced). UpdateTable and DeleteTable take the rowFOR UPDATE, so the definition recorded is the one in force when the items were read.CREATING, and open the item reader's WAL snapshot; items are then streamed without the lock and written in batches under short lock holds, so other writers wait for a batch, not the whole backup (the first revision stalled every write for the duration: about twelve seconds for 100,000 items; reviewer finding). The copy runs detached from the request, so a client that disconnects does not leave a half-written backup; one left by a crash is removed at startup. ACREATINGbackup is visible toListBackups/DescribeBackup, as on the service;DeleteBackuprefuses it andRestoreTableFromBackupreports it missing. The long reader pins WAL checkpointing for the backup's duration; documented. A file-backed SQLite pool is clamped to at least two connections so the reader and the batch writer can coexist.FOR SHAREand thetable_restoresrow inserted with thetablesrow) and on SQLite, so a concurrentDeleteBackuporders strictly before (the restore sees no backup, no target is created) or after (it sees the restore and returnsBackupInUseException). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata: a restore claims before reading and releases once itsCREATINGtarget exists;DeleteBackupclaims the same way before dropping data; a claim left by a crashed process is cleared at the next startup.ResourceInUseExceptionon all three backends.RestoreSummaryfor a restored table;RestoreInProgressis true while CREATING. The restore response and later DescribeTable calls carry the sameRestoreDateTime.Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with
LimitExceededException.Failure and crash:
SET idle_session_timeoutis best-effort (managed instances may refuse it). The control-plane pass removes targets that are CREATING with no scheduled transition, older than 60 s, whose lock is free; with the pass's 60 s idle timeout that is between 60 and about 120 s after abandonment.SET LOCAL lock_timeout = '3s'and drops data tables before deleting the catalog row, so a drop blocked behind a backup'sACCESS SHARElock fails fast and the row stays DELETING for the next pass. The previous order deleted the catalog row first, so a failed drop was never retried. The stale-vector-index recovery now skips indexes whose parent table is DELETING and cleans up if the parent vanishes under it, so the reordered drop cannot leave an orphan vector table.Refused rather than restored wrongly: backups of tables with vector indexes (both backends; SQLite previously restored the table silently without them); backups of multi-part-key tables; table definitions with an unknown version; and the unsupported overrides above. RestoreTableFromBackup validates
TargetTableName. NewDynamoDbError::BackupInUseExceptionandStorageError::BackupInUse.Docs:
docs/dynamodb-limits.mdno longer says backup and restore are unsupported, and its summary row is recounted;docs/differences-from-dynamodb.mdgains a Backup and Restore section with every row qualified by backend; the upgrade manual has a 0.0.4 section that warns PostgreSQL operators that targets the old bug left in CREATING are removed on the first control-plane pass after the upgrade, describes the SQLite lock, and notes WAL headroom; the admin guide's version literal and migrate guidance are current.Why
No issue filed. The 1.0 readiness review's P0-2 found:
The backup-store design in #348 replaces this path. Until it lands, this is the path users have.
Testing done
tests/test_backup_restore_fidelity.py(11, any backend): S/N/B hash and sort keys compared item by item by Scan and GetItem; a provisioned table with four GSIs and an LSI (projections ALL/INCLUDE/KEYS_ONLY, S/N hash and N/B range keys, sparse items, table and GSI throughput,IndexStatus, every index's contents, projected attribute sets, ordered ranged reads both directions);RestoreSummaryon the response and on DescribeTable; a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI;BillingModeOverridein both directions withProvisionedThroughputOverride; a refused PROVISIONED override on a PAY_PER_REQUEST backup with a GSI, with DescribeTable then returningResourceNotFoundException(no target left). 11/11 on PostgreSQL and on SQLite against servers built from this branch.crates/storage-postgres/tests/backup_restore.rs(21, in the PostgreSQL storage-level CI step): old backup formats; failure cleanup; duplicate rows; the multi-part refusal; restored indexes, throughput, table class, SSE, on-demand throughput; more than 1,000 rows including 300 KB items; the abandoned-restore sweep (inside the grace period, lock held, unowned, lock held by another); the backup barrier against an uncommitted UpdateTable; a production-path snapshot test that blockscreate_backupafter its snapshot is pinned, commits an item, and asserts the backup excludes it (verified to fail when the pinning read is removed), plus the two transaction shapes that show whyLOCK TABLEalone was not enough; a half-built GSI; DeleteTable during a restore; DeleteBackup racing a restore in both orders; the droplock_timeoutleaving a row DELETING and the next pass finishing it; RestoreSummary and DeleteBackup (absent, in progress, done, refused then allowed, kept after the backup is deleted). The fixture appends/postgresitself, soEXTENDDB_TEST_PG_CONNECTION_STRINGis the server, not a database.SQLite
restore_tests(16): indexes and throughput; the startup sweeps for restores and backups; failed-copy cleanup; the multi-part refusal; table class, SSE, on-demand throughput; 730 rows across batch boundaries compared to what the write path produces; a database with the exact 0.0.3 schema holding a backup in the 0.0.3 row format, migrated and restored withidpresent and rows in order; byte-bounded batch cuts; DeleteTable during a restore; the restore intent committed with its target; an unrelated writer keeping progress during a 5,000-item backup (fails on the whole-lock shape); items written after the snapshot excluded; a cancelled CreateBackup finishing with every item; thebackup_itemsrebuild being idempotent and order-preserving.Unit tests: definition normalization and the override preflight (12); the engine's override parsing and refusals including every unsupported member and the CreateTable throughput text (15); MongoDB's claim predicates, GSI throughput handling, and the two-phase claim filters (7);
CatalogVersionordering; the newer-catalog refusal inextenddb migrateand in both schema runners; the vector-recovery selection predicate.By hand, against release builds from this branch: the override matrix on PostgreSQL (throughput-only override on a provisioned backup;
BillingModeOverride=PROVISIONEDwithout throughput refused; invalid enum text;OnDemandThroughputOverride/VectorIndexOverriderefused; PROVISIONED+GSI → PAY_PER_REQUEST reports 0/0); a cancelled CreateBackup on SQLite observed CREATING then AVAILABLE with the full count; a 20,000-item SQLite backup with an unrelated writer at a 126 ms maximum gap (previously the whole 0.9 s); a 0.0.3 SQLite file migrated by the binary withbackup_items.idpresent and a secondmigratea no-op. Fulltests/on PostgreSQL against this branch and main: identical 8 environment-dependent failures on both (this host's botocore), no regression; the branch adds 18 passing tests.Review: two rounds of independent adversarial review (four and two reviewers) on this revision after the human review; every blocking and major finding from both is fixed above. MongoDB's claim protocol was checked by an exhaustive interleaving model (20 orderings, no dual claim) but not against a live replica set on this host.
Not changed here
deny_unknown_fieldsis a follow-up before a second definition version exists.ResourceNotFoundException, and the claim and the activation cannot both succeed.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).ADR / RFC: #348 (backup and restore design) describes the target this moves toward.
Storagetrait:BackupEngine::restore_table_from_backupgains anoverrides: RestoreTableOverridesparameter (Defaultmeans none); the overrides are request input the service defines, and an explicit parameter was preferred over a task-local so every backend author sees it. On-disk: two new catalog tables and one new SQLite column. No wire, auth, or CLI changes beyond honouring request members the API already defines.Breaking changes
extenddb migrate; the server refuses to start against 0.0.3.extenddb migraterefuses a catalog newer than the binary.BackupEngine::restore_table_from_backuptakes an additionaloverridesparameter; out-of-tree backends must add it.GlobalSecondaryIndexOverride,LocalSecondaryIndexOverride,SSESpecificationOverride,OnDemandThroughputOverride, orVectorIndexOverrideare refused (previously ignored).ResourceInUseException.CREATING, returnsBackupInUseException.pool_size.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.