Repository navigation
feat(backup): add backup store crate with filesystem and S3 implementations - #336
Open
LeeroyHannigan wants to merge 4 commits into
Open
LeeroyHannigan wants to merge 4 commits into
LeeroyHannigan wants to merge 4 commits into
Conversation
…ations New workspace crate extenddb-backup-store: the BackupStore trait (put/get/head/list/delete_prefix/validate) behind which the engine will write portable backups, per docs/design/15-backup-restore.md section 2. Key space is shared by both stores: /-separated components, each non-empty, no dot or dot-dot, no backslashes, no control characters, 1024 bytes total. Prefix matching is component-aligned so a/b never matches a/bc, on S3 included via client-side filtering. FilesystemStore canonicalizes its root at construction and refuses any operation whose resolved target escapes the root. Because create_dir_all follows symlinks in existing path components, put resolves the deepest existing ancestor of its destination and checks it against the root BEFORE creating any directory, then creates the remaining components one at a time with create_dir, re-verifying each is a real directory and not a symlink. Writes go through a temporary sibling that is synced before the rename, and the parent directory is synced after it, so an Ok return means the object and its directory entry are on disk. Prefix deletion removes files and the directories the deletion emptied; temporary names are invisible to listings. Escape refusals name the offending path. S3Store builds on aws-sdk-s3 with credentials from the SDK default chain; supports bucket, prefix, region, endpoint override and path-style addressing for S3-compatible stores. Bodies above the configured part size (default 8 MiB, enforced floor 5 MiB) upload as multipart with bounded concurrency. Every failure after CreateMultipartUpload reaches the abort, a failed complete included, so no orphaned parts accrue storage. Listing paginates; prefix deletion batches DeleteObjects requests of 1000. BackupStoreConfig models the store subset of the [backup] section with deny_unknown_fields; open(config) constructs Arc<dyn BackupStore>. Server wiring is deliberately left to the configuration task. A shared test module exercises both stores: round trips, listing with and without prefix, prefix deletion counts, missing-object behavior, every rejected key shape, bodies larger than the part size, abort on a mid-body failure and on a complete-time failure injected through a client interceptor, and traversal and symlink escapes for the filesystem store, including multi-level keys under planted directory and file symlinks asserting nothing is created outside the root. S3 tests run against a local endpoint when EXTENDDB_TEST_S3_ENDPOINT is set and announce the skip on raw stderr otherwise; the crate README documents how to run them. aws-config and aws-sdk-s3 are pinned exactly to the newest versions whose minimum supported rustc matches the workspace rust-version of 1.88; later releases require a newer compiler and fail the 1.88.0 CI job. Their transitive aws-smithy-* crates are held in Cargo.lock for the same reason, and crc moves to 3.3.0 so sqlx and the checksum path share one crc major. Both license notices are regenerated for the crc version change and both checks pass. The workspace builds and the crate tests pass on the 1.88.0 toolchain and on stable, with the S3 half of the suite additionally run against a local MinIO on both toolchains. Signed-off-by: Lee Hannigan <lhnng@amazon.com>
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
September 10, 2026 19:03
…mlink claim
An in-root directory symlink (root/a pointing at root/z) aliased one key
onto another: put("a/b/file") landed at z/b/file, list("a") was empty,
delete_prefix("a") returned 0 while get("a/b/file") kept serving bytes,
and a planted file symlink redirected get to another key's content. The
canonicalize check only refused targets outside the root, so redirection
inside the root passed, and keys are account-id-prefixed, so an alias
maps one account's writes into another account's prefix.
The filesystem store now resolves every key one component at a time with
symlink_metadata and refuses any component that is a symlink, whether it
points inside or outside the root: get and head walk the path without
following links, put refuses symlinked ancestors in prepare_parent and a
symlink at the destination, and list and delete_prefix already skip
symlinks, so all five operations agree on what a key names.
The module doc and README claimed a symlink planted inside the root
cannot redirect reads or writes. That claim is now bounded to what the
checks deliver: they defeat symlinks present when an operation starts
and do not defend against another local process mutating the tree while
an operation runs, because the kernel re-resolves the textual path
between a component check and the next path step. Closing that window
needs directory-handle resolution (openat2 with RESOLVE_BENEATH or an
openat walk with O_NOFOLLOW at every step), which the standard library
does not expose; it is documented in the module doc as a follow-up, and
the backup root must not be writable by less-trusted users.
Also: open_with_interceptor is gated behind a new test-util feature so
it is no longer part of the crate's default API surface, and the module
doc notes the two filesystem-specific key space limits (NAME_MAX on a
component, and the temporary-file name pattern hidden from listings).
Signed-off-by: Lee Hannigan <lhnng@amazon.com>
The MongoDB notices file predated the merge of main into this branch and still recorded crc 3.4.0; the branch's lock holds crc 3.3.0 so sqlx and the checksum path share one crc major. Regenerated with the pinned cargo-about 0.9.0 for all three variants the Licenses workflow checks; only the MongoDB file changed and all three checks pass. Signed-off-by: Lee Hannigan <lhnng@amazon.com>
robinnsc
reviewed
Sep 19, 2026
Collaborator
There was a problem hiding this comment.
Three comments inline, none blocking, plus one Cargo.lock question here since the lock diff is unwieldy to anchor: the diff changes winapi-util from windows-sys 0.61.2 to windows-sys 0.48.0 — a downgrade unrelated to the AWS additions, which looks like accidental churn from the --precise pinning of the smithy crates. Intentional? If not, a targeted cargo update should put it back.
| Ok(deleted) | ||
| } | ||
|
|
||
| async fn validate_impl(&self) -> Result<(), StoreError> { |
Collaborator
There was a problem hiding this comment.
The filesystem store's validate() writes and removes a probe file, but this only calls HeadBucket, which succeeds with list-only permission. The trait doc says validate exists so a misconfigured store fails at startup rather than at the first CreateBackup — a bucket the server can see but not write to defeats exactly that. A probe put/delete under the configured prefix would make the two stores equivalent.
Related smaller thing that could fold in: head/get map a missing bucket (NoSuchBucket → 404) to NotFound/None, so after a bucket deletion reads look like "no such object" rather than "store is broken". Acceptable if validate is strengthened; worth a doc sentence either way.
| })) | ||
| } | ||
|
|
||
| async fn list_impl(&self, prefix: &str) -> Result<Vec<ObjectMeta>, StoreError> { |
Collaborator
There was a problem hiding this comment.
collect_objects walks the whole prefix into a Vec inside spawn_blocking before wrapping it in a stream, while the S3 store streams page by page. For the backup layout (bounded per-backup prefixes) this is fine in practice, but the trait presents a stream, and a caller listing the root of a large store gets surprise memory behavior. Fine to keep as is — just worth a sentence in the module doc, or a follow-up note.
|
|
||
| /// Upload every part with bounded concurrency; returns the completed part | ||
| /// list sorted by part number. | ||
| async fn upload_parts( |
Collaborator
There was a problem hiding this comment.
With the default 8 MiB part size, S3's 10,000-part limit caps an object at about 78 GiB; past that the upload fails mid-flight with a service error rather than anything actionable. A cheap client-side check when the part number passes 10,000 (or a doc note telling callers to raise upload_part_size_bytes for huge objects) would turn that into a clear error.
robinnsc
approved these changes
Sep 19, 2026
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
8 tasks done
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 9, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. create_backup and restore_table_from_backup on PostgreSQL checked for a column named `sk`, which data tables do not have (sk_s, sk_n, sk_b). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store pk and item_data only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in item_data and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. RestoreTableFromBackup overrides. BillingModeOverride and ProvisionedThroughputOverride are implemented on all three backends and passed to storage as an explicit `overrides: RestoreTableOverrides` parameter on BackupEngine::restore_table_from_backup (a trait change). BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST normalization and refuses, before any target is created, a PROVISIONED restore of a backup whose GSIs carry no throughput (the service needs GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, SSESpecificationOverride, OnDemandThroughputOverride, and VectorIndexOverride are refused with ValidationException before any storage read; they were silently ignored. Throughput values reuse CreateTable's validator so the error text matches. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add backup_definitions (one row per backup, the definition as JSON behind a version marker; extenddb_storage::backup_definition) and table_restores (one row per restored table: source backup ARN and restore time, not a foreign key to backups so the summary outlives the backup, as on the service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore cursors on it rather than the implicit rowid, which a VACUUM between batches could renumber. An existing backup_items is rebuilt in place (copy in rowid order, idempotent). backup_items.pk now holds the real partition key on SQLite as it does on PostgreSQL. Backups and restores from before the upgrade have no definition rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and the snapshot is pinned by a real read of the table (LOCK TABLE is a utility statement and does not establish a REPEATABLE READ snapshot; the first revision relied on it and the definition and the items could come from different instants). UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. SQLite CreateBackup takes the engine write lock only to read the definition, insert the backup row as CREATING, and open the item reader's WAL snapshot; the items are then streamed without the lock and written in batches under short lock holds, so other writers wait for a batch rather than the whole backup (a 100,000-item backup previously stalled every write for about twelve seconds). The copy runs detached from the request, so a client that disconnects does not leave a half-written CREATING backup; one left by a crash is removed at startup. A CREATING backup is visible to ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses it. The long reader pins WAL checkpointing for the backup's duration. A file-backed SQLite pool is clamped to at least two connections so the reader and the batch writer can coexist. A restore registers itself in the same transaction that creates its target on PostgreSQL (backup row FOR SHARE, table_restores inserted with the tables row) and SQLite, so DeleteBackup orders strictly before (the restore then sees no backup) or after (it sees the restore, BackupInUseException). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims before reading and releases once its CREATING target exists; DeleteBackup claims the same way before dropping data; stale claims from a crashed process are cleared at startup. DeleteTable on a table being restored returns ResourceInUseException on all three backends. DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection (its idle_session_timeout SET is best-effort, for managed instances that refuse it); the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free, which with the pass's 60 s idle timeout means between 60 and about 120 s. SQLite removes the same targets at startup, which is safe because one server at a time holds the file (#391). The control-plane worker's DeleteTable drop runs under a 3 s lock_timeout and drops data tables before deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE lock fails fast and the row stays DELETING for the next pass (the previous order deleted the row first, so a failed drop was never retried). The stale-vector-index recovery skips indexes whose parent table is DELETING and cleans up if the parent vanishes under it, so the reordered drop cannot leave an orphan vector table. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and the unsupported overrides above. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported and its summary row is recounted; docs/differences-from- dynamodb.md gains a Backup and Restore section with every row qualified by backend (what is restored, the seven overrides, the refusals, CREATING visibility, the delete race, crash recovery, where backups live, the consistency model and WAL cost); the upgrade manual has a 0.0.4 section that warns PostgreSQL operators that targets the old bug left in CREATING are removed on the first control-plane pass after the upgrade, describes the SQLite lock, and notes WAL headroom; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash and sort keys compared item by item; a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads; RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput; BillingModeOverride in both directions; a refused PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no target). crates/storage-postgres/tests/backup_restore.rs (21, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier including a production-path test that fails if the snapshot-pinning read is removed and the two transaction shapes that show why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both orders, the abandoned-restore sweep, the drop lock_timeout leaving a row DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a writer that keeps making progress during a backup, items written after the snapshot excluded, a cancelled request finishing with every item, the startup sweeps, a 0.0.3 database in the old row format, the backup_items rebuild). Unit tests for the definition normalization and override preflight (12), the engine's override parsing and refusals (15), Mongo's claim predicates and GSI throughput handling, CatalogVersion ordering, the newer-catalog refusals, and the vector-recovery predicate. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 12, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. create_backup and restore_table_from_backup on PostgreSQL checked for a column named `sk`, which data tables do not have (sk_s, sk_n, sk_b). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store pk and item_data only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in item_data and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. RestoreTableFromBackup overrides. BillingModeOverride and ProvisionedThroughputOverride are implemented on all three backends and passed to storage as an explicit `overrides: RestoreTableOverrides` parameter on BackupEngine::restore_table_from_backup (a trait change). BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST normalization and refuses, before any target is created, a PROVISIONED restore of a backup whose GSIs carry no throughput (the service needs GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, SSESpecificationOverride, OnDemandThroughputOverride, and VectorIndexOverride are refused with ValidationException before any storage read; they were silently ignored. Throughput values reuse CreateTable's validator so the error text matches. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add backup_definitions (one row per backup, the definition as JSON behind a version marker; extenddb_storage::backup_definition) and table_restores (one row per restored table: source backup ARN and restore time, not a foreign key to backups so the summary outlives the backup, as on the service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore cursors on it rather than the implicit rowid, which a VACUUM between batches could renumber. An existing backup_items is rebuilt in place (copy in rowid order, idempotent). backup_items.pk now holds the real partition key on SQLite as it does on PostgreSQL. Backups and restores from before the upgrade have no definition rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and the snapshot is pinned by a real read of the table (LOCK TABLE is a utility statement and does not establish a REPEATABLE READ snapshot; the first revision relied on it and the definition and the items could come from different instants). UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. SQLite CreateBackup takes the engine write lock only to read the definition, insert the backup row as CREATING, and open the item reader's WAL snapshot; the items are then streamed without the lock and written in batches under short lock holds, so other writers wait for a batch rather than the whole backup (a 100,000-item backup previously stalled every write for about twelve seconds). The copy runs detached from the request, so a client that disconnects does not leave a half-written CREATING backup; one left by a crash is removed at startup. A CREATING backup is visible to ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses it. The long reader pins WAL checkpointing for the backup's duration. A file-backed SQLite pool is clamped to at least two connections so the reader and the batch writer can coexist. A restore registers itself in the same transaction that creates its target on PostgreSQL (backup row FOR SHARE, table_restores inserted with the tables row) and SQLite, so DeleteBackup orders strictly before (the restore then sees no backup) or after (it sees the restore, BackupInUseException). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims before reading and releases once its CREATING target exists; DeleteBackup claims the same way before dropping data; stale claims from a crashed process are cleared at startup. DeleteTable on a table being restored returns ResourceInUseException on all three backends. DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection (its idle_session_timeout SET is best-effort, for managed instances that refuse it); the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free, which with the pass's 60 s idle timeout means between 60 and about 120 s. SQLite removes the same targets at startup, which is safe because one server at a time holds the file (#391). The control-plane worker's DeleteTable drop runs under a 3 s lock_timeout and drops data tables before deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE lock fails fast and the row stays DELETING for the next pass (the previous order deleted the row first, so a failed drop was never retried). The stale-vector-index recovery skips indexes whose parent table is DELETING and cleans up if the parent vanishes under it, so the reordered drop cannot leave an orphan vector table. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and the unsupported overrides above. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported and its summary row is recounted; docs/differences-from- dynamodb.md gains a Backup and Restore section with every row qualified by backend (what is restored, the seven overrides, the refusals, CREATING visibility, the delete race, crash recovery, where backups live, the consistency model and WAL cost); the upgrade manual has a 0.0.4 section that warns PostgreSQL operators that targets the old bug left in CREATING are removed on the first control-plane pass after the upgrade, describes the SQLite lock, and notes WAL headroom; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash and sort keys compared item by item; a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads; RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput; BillingModeOverride in both directions; a refused PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no target). crates/storage-postgres/tests/backup_restore.rs (21, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier including a production-path test that fails if the snapshot-pinning read is removed and the two transaction shapes that show why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both orders, the abandoned-restore sweep, the drop lock_timeout leaving a row DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a writer that keeps making progress during a backup, items written after the snapshot excluded, a cancelled request finishing with every item, the startup sweeps, a 0.0.3 database in the old row format, the backup_items rebuild). Unit tests for the definition normalization and override preflight (12), the engine's override parsing and refusals (15), Mongo's claim predicates and GSI throughput handling, CatalogVersion ordering, the newer-catalog refusals, and the vector-recovery predicate. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
Collaborator
|
Superseded by #393, which includes this branch's commits unchanged and adds one on top with the changes from the design review (see that PR's description). Leaving this open for the author to close or to pick the changes back here, whichever is preferred. |
robinnsc
added a commit
that referenced
this pull request
Oct 13, 2026
…t code Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture, and two review rounds against the companion PRs #393, #394 and #395. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time, as DynamoDB documents; backup_creation_date_time is the instant the snapshot pinned. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite; no row ever exists unowned. Heartbeats are a timer, staleness is measured on the database's clock, and a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING and deletes its prefix only after the stale window. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically; ResourceInUseException on a name race, LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional; DeleteTable on a restore target fails the job. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto (MinIO no longer publishes images). - ListBackups paging and filtering, the restore override contract, sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs and a concrete pool-size rule are specified. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path. Co-authored-by: Lee Hannigan <lhnng@amazon.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the backup store crate,
crates/backup-store(extenddb-backup-store): theBackupStoretrait the engine will write portable backups through, with a filesystem implementation and an S3 implementation. This is the store layer only; nothing is wired into the server orcrates/configyet, and no API behavior changes.BackupStoretrait:put(streaming body),get(streaming body),headreturningOption<ObjectMeta { size, last_modified }>,list(prefix)streamingObjectMetain key order,delete_prefix(prefix)returning the deleted count, andvalidate()for startup checks. Every operation returns aStoreError(NotFound,PermissionDenied,InvalidKey,Io/Transportwith sources,Other) whose display output never carries credentials./-separated components, each non-empty, no.or.., no backslashes, no control characters, at most 1024 bytes total. Prefix matching is component-aligned, soa/bnever matchesa/bc; the S3 store filters listings client-side to the same rule.FilesystemStore: canonicalizes its root at construction and resolves every key one component at a time without following symlinks: a symlink anywhere in a key's path, pointing inside or outside the root, is refused, so a key never resolves through a link and put, get, head, list, and delete_prefix agree on what a key names. These checks defeat symlinks present when an operation starts; they do not defend against another local process mutating the tree during an operation, so the backup root must not be writable by less-trusted users. Becausecreate_dir_allfollows symlinks in existing components,putwalks to the deepest existing ancestor of its destination, refusing any symlinked component before creating a directory, then creates the remaining components one at a time withcreate_dir, re-verifying each is a real directory and not a symlink. Writes go to a temporary sibling that is synced before the rename, and the parent directory is synced after it, so an Ok put means the object and its directory entry are on disk; a crash never leaves a truncated object under its final name, and temporary names are invisible to listings. Prefix deletion removes files and the directories the deletion emptied. Refusals name the offending path, never a link's target.S3Storeon the officialaws-sdk-s3crate: credentials from the SDK default chain, withbucket,prefix,region, optionalendpoint,force_path_style,upload_part_size_bytes(default 8 MiB, enforced 5 MiB floor), andmax_concurrent_uploads(default 4). Bodies above the part size upload as multipart with bounded concurrency; every failure after CreateMultipartUpload reaches the abort, a failed CompleteMultipartUpload included, so no orphaned parts accrue storage. Listing paginates; prefix deletion batches DeleteObjects requests of 1000.validate()calls HeadBucket and names the bucket in the failure message. The failure-injection interceptor seam the test suite uses is gated behind atest-utilcargo feature and absent from the default API surface.BackupStoreConfigwith serdedeny_unknown_fields, modeling the store subset of the[backup]section (store = "filesystem" | "s3",[backup.filesystem] path,[backup.s3]fields), plusopen(config) -> Result<Arc<dyn BackupStore>>.Dependencies:
aws-config = "=1.8.13"andaws-sdk-s3 = "=1.122.0", pinned exactly to the newest versions whose minimum supported rustc matches the workspacerust-versionof 1.88 (later releases require a newer compiler and fail the 1.88.0 CI job); transitiveaws-smithy-*crates are held inCargo.lockto 1.88-compatible versions for the same reason, andcrcmoves from 3.4.0 to 3.3.0 so sqlx and the checksum path share one crc major. The license notices files are regenerated for the crc version change and all three checks pass.Why
Every backup today shares the failure domain of its data, backups cannot cross backends, and a lost catalog destroys the ability to restore. The backup and restore design (
docs/design/15-backup-restore.md) closes that with one engine-owned path writing a portable format to a configured store. This PR is the store abstraction that path writes through (task T1.1); the manifest types, snapshot engine, and job workers build on it in the following PRs.Testing done
cargo test -p extenddb-backup-store: 52 tests, 0 filtered. A shared test module runs identical checks against both stores: put/get/head round trip, overwrite, empty body, list with and without prefix (including trailing slash and thea/bvsa/bcboundary), delete_prefix counts and emptiness after, NotFound on missing get and None on missing head, every rejected key shape on every operation, a body larger than the part size verified byte-for-byte, and a put whose body fails midway leaving nothing under the key. Filesystem-only: traversal and symlink escapes refused, including multi-level keys under planted directory and file symlinks with the outside filesystem asserted untouched (no directory may be created outside the root); an in-root directory symlink refused on put, get, and head, with list of the aliased prefix empty, delete_prefix on it returning 0 and removing nothing, and the object under its real key surviving; an in-root file symlink refused on get, head, and put with the link target untouched; an ancestor collision with an existing object; no stray temporary file after a failed put; and the validate() probe on an unwritable root.EXTENDDB_TEST_S3_ENDPOINTthe S3 tests announce the skip on raw stderr, visible in default output; the crate README documents the docker and plain-binary MinIO invocations. CI wiring for a MinIO service is task T1.9.cargo test --workspace: 1233 passed, 0 failed.cargo +1.88.0 check --workspace --lockedandcargo +1.88.0 test -p extenddb-backup-storepass on the 1.88.0 toolchain (the CI minimum), and all gates below pass on stable.cargo fmt --all -- --checkandcargo clippy --all-targets -- -D warnings: clean.devtools/generate-software-license-notices --check,devtools/generate-software-license-notices --check --backend mongodb, anddevtools/generate-dev-license-notices --check: all three current after regeneration.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: docs/design/15-backup-restore.md (landing in PR 1)
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.