Conversation
Report upload granularity at session creation and in progress responses. Reject undersized non-final chunks in GCS and tiered storage, and let the Rust client preflight known chunk lengths.
…e-upload-granularity
This comment has been minimized.
This comment has been minimized.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## feat/tiered-resumable-uploads #652 +/- ##
=================================================================
- Coverage 91.32% 91.28% -0.04%
=================================================================
Files 116 116
Lines 23625 23725 +100
=================================================================
+ Hits 21575 21657 +82
- Misses 2050 2068 +18
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 5b08a22. Configure here.
Keep creation-time granularity on new client handles and leave reconstructed handles without it. Remove progress-response granularity metadata and inline the client chunk-length check.
| #[error("resumable upload session is not available")] | ||
| ResumableUploadUnavailable, | ||
| /// A non-final chunk is shorter than the upload granularity. | ||
| #[cfg(feature = "resumable-upload-api")] |
There was a problem hiding this comment.
I'm noticing that our error type isn't marked as non_exhaustive, yet we add or remove variants with feature flags, which seems disruptive. By adding a feature flag, existing code may stop to compile.
When this stabilizes, I think we should keep those un-flagged and revisit our idea of hiding the function behind a feature flag permanently. It seems that it's better to just have the code in always, and instead:
- either use an extension trait on session to make opt-in to the low-level methods explicit
- or simply keep calling them out as low-level in the docs and that's it.
There was a problem hiding this comment.
Good point. The ultimate cause of this though is that this enum is not marked non_exhaustive so whenever we introduce an error kind it will be a breaking change.
Should we consider marking this as non_exhaustive or would you rather do so when we know all the APIs are relatively stable?
This introduces the concept of upload granularity for resumable uploads.
The only backend that has a granularity is effectively GCS with its documented limit of 256 KiB. Tiered inherits from long-term. The other backends get the default value of 0.
The Rust client learns the granularity from session creation responses, and rejects
put_chunk/put_stream/put_readcalls upfront whenlength < granularityand the chunk is not the last one, avoiding a round-trip to the server.Note that a resumable upload handle reconstructed on the client from a persisted token keeps its granularity unknown and leaves chunk-size validation to the server which will return a generic 400.
I aim to address this in a follow-up where we will introduce an error code on our errors and parse it back on the client to give a more useful error.
Refs FS-514