feat(storage): enable default CRC32C checksum validation for object downloads#9210
feat(storage): enable default CRC32C checksum validation for object downloads#9210salilg-eng wants to merge 1 commit into
Conversation
| // the partial stream, we can just return the stream we fetched. | ||
| if ($transcodedObj) { | ||
| return $fetchedStream; | ||
| return $this->maybeWrapWithHashValidatingStream($fetchedStream, $args, $response); |
There was a problem hiding this comment.
IIUC, wrapping the transcoded stream in maybeWrapWithHashValidatingStream here will cause a 100% failure rate for compressed files because GCS serves uncompressed bytes on transcoding, whereas X-Goog-Hash contains the hash of the stored compressed bytes. Can we implement on-the-fly decompression and verification?
cc: @v-pratap what do you recommend?
There was a problem hiding this comment.
I have updated the logic to check GCS's X-Goog-Stored-Content-Encoding header. If this header is present (indicating transcoding occurred), the validator now safely and automatically bypasses validation, which aligns with how GCS client libraries in other languages (like Java, Go, and Python) handle transcoding. I also added a dedicated unit test testDownloadObjectWithTranscodedObjectValidationBypassed to verify this behavior.
There was a problem hiding this comment.
Thanks. I'll let Vaibhav confirm if bypassing this is okay. Let's keep this unresolved for now.
d52e22f to
1ed7da9
Compare
1ed7da9 to
1e43dbb
Compare
Overview
Enables full object checksum validation (CRC32C by default) across GCS JSON read paths, covering all four download methods:
downloadAsString,downloadToFile,downloadAsStream, anddownloadAsStreamAsync.It integrates a
HashValidatingStreamdecorator that computes hashes on-the-fly and validates them upon reaching the end of the stream.Key Features
'validate'parameter (crc32,md5, orfalseto disable).206 Partial Content) where full object validation is incompatible.Buganizer Tasks Resolved
[PHP] SDK: Full object checksum for all JSON and XML Reads as well)Testing
HashValidatingStreamTest.php.RestTest.php.