Repository navigation
Conversation
- WatcherCheckpoint and WatcherRollingState in syft/sync/checkpoints/watcher_state.py hold the files received from one DO, the changes since then and the cursor in its outbox. They refuse a state that a later client wrote. - parse_versioned_json() checks the version of JSON text or bytes before the fields. read_versioned_json() now calls it for a file. - DriveWatcherStoreConnection keeps one folder per DO in /SyftBox/<me>-<version>-watcher-state, which is shared with nobody. A save replaces the content of the checkpoint file or the rolling-state file in place. - put_checkpoint() and put_rolling_state() take no file name, because each DO has one file of each kind.
…crypted with the DS key - watcher_save_checkpoint() and watcher_save_rolling_state() encrypt the state for the DS when encryption is required, then store it. - watcher_load_checkpoint() and watcher_load_rolling_state() return None only when nothing is saved. They raise when the state cannot be decrypted or a later client saved it. - When the DS key cannot decrypt the state, the error tells the DS to log in with the key_file of the key that saved it.
…at startup - At startup, a DS in on-disk mode restores every DO that has saved state. It writes the files of the checkpoint and the rolling state, and records their hashes and the received message ids. - After a pull that receives messages, the DS saves the rolling state of that DO, then marks the messages processed. At 50 changed paths, it folds them into the checkpoint. - The DS keeps the received message ids per DO. An event that pre_write_filter refuses is not saved, so a restore does not write it. - WatcherRollingState has no last_msg_id, because the cursor is the newest received id. - peers_with_saved_state() lists the DOs that have saved state.
… holds its state - In on-disk mode, the DS keeps its record of received messages in memory, as in-memory mode does. A restart restores only from the saved state on Drive. - The code that rebuilt the cursor and the file hashes from the local folder is gone, and so is the unused events_base_path. - The local cleanup no longer deletes a -event-messages folder. - The three tests that restarted from the local folder now check the state restored from Drive.
…checkpoint models - Watcher state is not encrypted until key recovery exists, as the ticket says. With the DS key, an enclave could not boot twice: its new key could not read the state of the last boot. - The DS saves a Checkpoint and a RollingState from syft/sync/checkpoints/, as the ticket says. watcher_state.py and parse_versioned_json are gone. - CheckpointFile.content holds str or bytes, so a binary file survives a checkpoint. RollingState.received_msg_ids holds the ids that a DS received. - The DS saves and loads the state of a DO under its email without case, so two spellings share one Drive folder. A restore writes the checkpoint files under the email as the DO's events spell it.
- Such a change blocked every later sync of the DS. A change with no hash made each fold raise, and a change with no content raised when it was applied. - The DS now drops it with a warning, as it does for an event aimed at another datasite.
…e DO and the DS - syft/sync/utils/file_content.py holds the text-or-bytes content code that ProposedFileChange, FileChangeEvent and CheckpointFile used to repeat. The JSON shape does not change. - Checkpoint.with_events() applies events to a checkpoint. The DO compaction and the DS fold both use it, and compact_incremental_checkpoints is gone. - The DO compaction now raises on a change without content or hash. Before, it dropped the file and could write "None" as a hash.
- decode_content() dropped characters outside the base64 alphabet and returned other bytes, with no error. It now raises, so the model refuses the content.
- DEFAULT_CHECKPOINT_EVENT_THRESHOLD moved to syft/sync/checkpoints/checkpoint.py. The DO syncer, the DS cache and SyftboxManager all read it. - WATCHER_CHECKPOINT_THRESHOLD and three literal defaults of 50 in SyftboxManager are gone.
- DriveWatcherStoreConnection.clone_for_thread() returned the same object, so another thread would share one Drive client. It now returns a store over a new client, as the other Drive connections do.
…ndle - WatcherStoreConnection.latest_checkpoint() is now get_checkpoint(), because each DO has one checkpoint. put_checkpoint() and put_rolling_state() return nothing, because no caller read the file id. - DriveWatcherStoreConnection names its helpers after the watcher folder and the files: _watcher_folder_id(), _save_file(), _read_file() and _find_id(). - DataSiteWatcherCache has _save_watcher_state() and _restore_watcher_state(). The DO compaction names its sorted checkpoints incrementals_in_order.
…test - test_ds_cache_handles_deletions_correctly reloaded over an empty mock Drive, so it passed even with no restore. It now reloads over the DS's own router. - The test also checks that a kept file comes back, so a restore that does nothing fails it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The DS saves a checkpoint and a rolling state for each DO on its own Drive, and restores them at startup. A lost local disk, such as a Colab runtime reset, no longer loses the DO files or the outbox cursor. PR 2 of "connsplit v1.3: Scientist durability", stacked on #9611.
Changes
DriveWatcherStoreConnection(drive_watcher_store.py):/SyftBox/<me>-<version>-watcher-state, shared with nobody, with one folder per DO. Each save replaces the checkpoint or rolling-state file in place.CheckpointandRollingStatefromsyft/sync/checkpoints/.CheckpointFile.contentnow holds text or bytes, andRollingState.received_msg_idsholds the ids the DS received.<syftbox>-event-messagescache is gone.file_content.pyfor text-or-bytes content,Checkpoint.with_events()for the DO compaction and the DS fold, and oneDEFAULT_CHECKPOINT_EVENT_THRESHOLD.WatcherStoreConnection: the put methods take no name and return nothing,latest_checkpoint()is nowget_checkpoint(), andpeers_with_saved_state()is new.Testing
tests/unit/test_watcher_state.py: restart on an empty folder, restart with a new key, delete after a fold, refused write, binary file, email case, broken changes.CheckpointandRollingState, and a corrupt base64 test.Asana task