Implement async state store methods for object storage backend - #72131
Open
jason810496 wants to merge 1 commit into
Open
Implement async state store methods for object storage backend#72131jason810496 wants to merge 1 commit into
jason810496 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
StateStoreObjectStorageBackendraisedNotImplementedErrorfrom all four of its async methods, so the async state store accessors added in #68232 fail against the shipped object storage backend.adelete/aclearclear the DB reference before calling the backend, so the call raised only after the reference was gone, orphaning the stored object.What
aget,aset,adelete,aclearby offloading the existing sync methods withasyncio.to_thread.get().Why
asyncio.to_threadand not fsspec's async APIfsspec does have one, but it is not reachable from where this backend sits:
AsyncFileSystemand are all private (_cat_file,_pipe_file,_rm,_glob).open_asyncis the only public one.LocalFileSystemsetsasync_impl = False, so a native path still needs anisinstancebranch and a sync fallback.ObjectStoragePathexposes no coroutines at all, so going native means bypassing it and re-implementing the path building and compression that_write_to_object_storage/_read_from_object_storagealready handle.run_coroutine_threadsafeand block the caller.asyncio.to_threadonly moves that block off Airflow's loop, so this follows fsspec's own concurrency model rather than working around it.Was generative AI tooling used to co-author this PR?