Expose tiered storage over FFI and add integration tests#871
Open
enigbe wants to merge 7 commits into
Open
Conversation
|
👋 I see @tnull was un-assigned. |
enigbe
force-pushed
the
2025-10-tiered-data-storage-ffi-and-tests
branch
from
5badddd to
d63381a
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
Collaborator
|
Please re-request review after #692 landed. |
enigbe
force-pushed
the
2025-10-tiered-data-storage-ffi-and-tests
branch
2 times, most recently
from
38b37be to
32fcfbf
Compare
This commit: Adds `TierStore`, a tiered `KVStore`/`KVStoreSync` implementation that routes node persistence across three storage roles: - a primary store for durable, authoritative data - an optional backup store for a second durable copy of primary-backed data - an optional ephemeral store for rebuildable cached data such as the network graph and scorer TierStore routes ephemeral cache data to the ephemeral store when configured, while durable data remains primary+backup. Reads and lists do not consult the backup store during normal operation. For primary+backup writes and removals, this implementation treats the backup store as part of the persistence success path rather than as a best-effort background mirror. Earlier designs used asynchronous backup queueing to avoid blocking the primary path, but that weakens the durability contract by allowing primary success to be reported before backup persistence has completed. TierStore now issues primary and backup operations together and only returns success once both complete. This gives callers a clearer persistence guarantee when a backup store is configured: acknowledged primary+backup mutations have been attempted against both durable stores. The tradeoff is that dual-store operations are not atomic across stores, so an error may still be returned after one store has already been updated. TierStore also implements `KVStoreSync` in terms of dedicated synchronous helpers that call the wrapped stores' sync interfaces directly. This preserves the inner stores' synchronous semantics instead of routing sync operations through a previously held async runtime. Additionally, adds unit coverage for the current contract, including: - basic read/write/remove/list persistence - routing of ephemeral data away from the primary store - backup participation in the foreground success path for writes and removals
Add native builder support for tiered storage by introducing `TierStoreConfig` and builder methods for configuring backup and ephemeral stores. During node construction, wrap the configured primary store in `TierStore` and attach any configured secondary tiers: ephemeral storage for cache-like data and backup storage for mirrored durable writes.
Refactor backup storage to local SQLite Replaces the builder's BYO backup-store configuration with a path-based local SQLite backup mirror. The builder now constructs the backup store internally using a dedicated backup database file name and rejects configurations where the backup path conflicts with the primary storage path. Also adds test coverage for full-cycle backup mirroring and same-path rejection, as well as a `setup_node_with_builder` test helper to allow builder customization in integration tests.
- Make setup_builder! use a mutable binding for Builder under uniffi to preserve test helper compatibility for the FFI-backed builder - Add ArcedNodeBuilder forwarding methods set_backup_storage_dir_path and set_ephemeral_store Co-authored-by: Copilot <copilot@github.com>
Add UniFFI-facing store abstractions and builder APIs so foreign-language callers can configure a custom primary store, an ephemeral store, and a local SQLite backup storage path when constructing nodes. This introduces `FfiDynStoreTrait` as an FFI-safe equivalent of `DynStoreTrait`, along with a Rust-side adapter that bridges foreign store implementations into the internal dynamic store abstraction used by the builder. As part of this change, we: - add UniFFI bindings for custom primary and ephemeral stores, plus the backup storage directory - expose `Builder::set_backup_storage_dir_path`, `Builder::set_ephemeral_store`, and `Builder::build_with_store` on the FFI surface - route FFI-backed builder construction through the native dyn-store path - move FFI I/O-related types into a dedicated module - preserve per-key write ordering across the FFI boundary - route Rust-side synchronous access through the async mutation path so sync and async callers share the same ordering behavior
Extend the Rust test harness to support tiered store configurations and add integration coverage for routing data across primary, backup, and ephemeral stores. This introduces tier-store-aware test helpers, including support for configuring separate stores per node, opening the internally-created SQLite backup store for inspection, and reading test stores in both native and UniFFI-enabled builds. Add an integration test covering the tiered-storage channel lifecycle and verifying that: - durable node data is persisted to both the primary and backup stores - ephemeral-routed data is stored in the ephemeral tier - ephemeral data is not mirrored back into the durable tiers
Add a Python in-memory KV store implementation that satisfies the UniFFI store callback interface, including both sync and async accessors. Refactor the Python integration test setup so channel lifecycle tests can share node construction, cleanup, funding, channel open, payment, and close logic. Add a tier-store test that exercises the FFI builder path with Python primary and ephemeral stores while configuring the backup through the builder's SQLite backup path. The test verifies that Python-backed stores remain usable throughout a full channel lifecycle and that ephemeral data is routed to the ephemeral store.
enigbe
force-pushed
the
2025-10-tiered-data-storage-ffi-and-tests
branch
from
32fcfbf to
856d30b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR is the second half of the tiered-storage work and is intended to be reviewed on top of #692. This split is intended to keep review manageable:
While Support tiered data storage #692 is limited to the Rust-side
TierStoreimplementation and nativeNodeBuilderintegration, this PR adds the FFI surface and end-to-end test coverage on top of that foundation.The changes introduced include:
Expose tiered storage over FFI
This introduces
FfiDynStoreTraitas an FFI-safe analogue toDynStoreTraitand wires theArcedNodeBuilderAPIs so foreign-language callers can configure:It also routes FFI-backed builder construction through the same internal
dyn-storepath used by the native builder.To support FFI-backed custom stores safely, this PR also:
Add Rust integration coverage
This extends the Rust test harness with tier-store-aware setup helpers and adds integration coverage verifying that:
Add Python FFI tests
This adds a Python in-memory key-value store implementation and a Python FFI test that exercises a full channel lifecycle using custom primary, backup, and ephemeral stores configured through the exposed builder APIs.
The Python test offer the same verification as the native Rust test.