Feat(Storage): Enable full object checksum validation on JSON path by thiyaguk09 · Pull Request #8825 · googleapis/google-cloud-php

thiyaguk09 · 2025-12-26T05:23:12Z

Enhanced Checksum Validation & Header Logic

This PR implements comprehensive MD5 and CRC32c checksum validation for object uploads, ensuring data integrity via the X-Goog-Hash header, improving data integrity verification. It refactors the upload architecture to handle hashes dynamically across different upload strategies.

Key Technical Changes

1. Core Library Enhancements (google-cloud-core)

ResumableUploader & StreamableUploader: Added type-safe logic (int)($rangeEnd + 1) === (int)$size to ensure X-Goog-Hash is transmitted only on the final chunk/request, preventing intermediate validation errors.
MultipartUploader: Standardized header merging to ensure hashes calculated by the connection layer are always included in single-shot uploads.
Header Integrity: Refactored restOptions merging to ensure custom metadata and encryption headers are preserved alongside checksums.

2. Storage Package Improvements (google-cloud-storage)

Automatic Hashing: Implemented logic to calculate missing MD5 or CRC32c hashes when the validate option is enabled.
Validation Logic: Updated Bucket::upload() to honor user-provided checksums and prevent redundant re-calculation.
Test Coverage: Added unit tests in BucketTest and RestTest to verify hash behavior in resumable, streamable, and multipart scenarios.

Note

CI "Lowest Dependencies" Failure: This failure occurs because the CI environment pulls the tagged version of google-cloud-core from Packagist instead of using the local changes in this PR. This will resolve once the Core changes are merged.

Refactor Resumable, Streamable, and Multipart uploaders to ensure integrity headers (X-Goog-Hash) are only attached to the request when an upload is being finalized. - In StreamableUploader, introduced `$isFinalRequest` to track intent before writeSize recalculations. - In ResumableUploader, added a boundary check to only attach the hash when the current range matches the total file size. - Aligns with GCS best practices for resumable upload integrity.

bshaffer

Just one question and a few code-cleanup nits! Otherwise this looks great

bshaffer · 2026-03-31T18:47:25Z

Storage/src/Connection/Rest.php

+        $md5Hash = null;
+        $crc32c = null;
+
+        if ($validate !== false) {
+            $md5Hash = base64_encode(Utils::hash($args['data'], 'md5', true));
+            $crc32c = $this->crcFromStream($args['data']);
+
+            if ($validate === 'md5') {
+                $args['metadata']['md5Hash'] = $md5Hash;
+            } elseif ($validate === 'crc32') {
+                $args['metadata']['crc32c'] = $crc32c;
+            }
        }

+        // Prepare the X-Goog-Hash header string
+        $xGoogHash = [];
+        if ($crc32c) {
+            $xGoogHash[] = 'crc32c=' . $crc32c;
+        }
+        if ($md5Hash) {
+            $xGoogHash[] = 'md5=' . $md5Hash;
+        }
+        $xGoogHashHeader = implode(',', $xGoogHash);


This logic is a bit hard to follow. I think it could be improved by 1) placing everything inside the if($validate) block, and 2) forming the $xGoogHash in a single statement

$xGoogHashHeader = ''; if ($validate !== false) { $md5Hash = base64_encode(Utils::hash($args['data'], 'md5', true)); $crc32c = $this->crcFromStream($args['data']); // Add validation metadata if ($validate === 'md5') { $args['metadata']['md5Hash'] = $md5Hash; } elseif ($validate === 'crc32') { $args['metadata']['crc32c'] = $crc32c; } // Prepare the X-Goog-Hash header string $xGoogHashHeader = implode(',', array_filter([ $md5Hash ? 'md5=' . $md5Hash : null, $crc32c ? 'crc32c=' . $crc32c : null, ])); }

I've refactored the hash logic to stay within the $validate block and used the array_filter approach for a cleaner implode.

bshaffer · 2026-03-31T19:03:41Z

Storage/src/Connection/Rest.php

+        $args['uploaderOptions']['restOptions'] ??= [];
+        $args['uploaderOptions']['restOptions']['headers'] ??= [];


fun fact - thanks to "autovivification", these two lines aren't necessary as long as you modify your line below (see my other comment)

Suggested change

$args['uploaderOptions']['restOptions'] ??= [];

$args['uploaderOptions']['restOptions']['headers'] ??= [];

Good point on autovivification—I've removed the explicit array initializations and updated the array_merge to use the null coalescing operator instead. This definitely cleans up the Rest.php logic.

bshaffer · 2026-03-31T19:06:12Z

Storage/src/Connection/Rest.php

+
+        if (!empty($args['headers'])) {
+            $args['uploaderOptions']['restOptions']['headers'] = array_merge(
+                $args['uploaderOptions']['restOptions']['headers'],


This will work as expected thanks to autovivification:

Suggested change

$args['uploaderOptions']['restOptions']['headers'],

$args['uploaderOptions']['restOptions']['headers'] ?? [],

Updated—thanks for the tip! I've updated the logic to leverage autovivification and simplified the initialization.

bshaffer · 2026-03-31T19:06:46Z

Core/src/Upload/ResumableUploader.php

+            if ($size !== '*' && ($rangeEnd + 1) == (int) $size) {
+                $customHeaders = $this->requestOptions['restOptions']['headers'] ?? [];
+                if (isset($customHeaders['X-Goog-Hash'])) {
+                    $headers['X-Goog-Hash'] = $customHeaders['X-Goog-Hash'];


Is there a reason we are only setting this one header instead of supporting all custom headers?

You're right. The intent was to isolate the X-Goog-Hash to the final chunk to avoid intermediate validation errors, but this implementation accidentally drops other custom headers. I will refactor this to merge all $customHeaders while specifically unsetting X-Goog-Hash only on non-final chunks.

bshaffer · 2026-03-31T19:07:16Z

Core/src/Upload/StreamableUploader.php

+        if ($isFinalRequest) {
+            $customHeaders = $this->requestOptions['restOptions']['headers'] ?? [];
+            if (isset($customHeaders['X-Goog-Hash'])) {
+                $headers['X-Goog-Hash'] = $customHeaders['X-Goog-Hash'];


same question here - is there a reason we're only supporting the one custom header instead of supporting all custom headers?

The intent was to isolate the X-Goog-Hash to the final chunk to avoid intermediate validation errors, but this implementation accidentally drops other custom headers. I will refactor this to merge all $customHeaders while specifically unsetting X-Goog-Hash only on non-final chunks.

- Refactor Rest.php hash calculation to be more concise using array_filter. - Remove redundant array initializations in Rest.php by utilizing PHP autovivification. - Improve readability of X-Goog-Hash header generation.

product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Dec 26, 2025

thiyaguk09 force-pushed the feat/enable-full-checksum-validation branch 2 times, most recently from d61822d to 4091fcc Compare March 23, 2026 11:33

thiyaguk09 added 5 commits March 23, 2026 11:40

Feat(Storage): Enable full object checksum validation on JSON path

053370b

bug fix

69be82e

bug fix

db306f4

lint fix

2ac9aad

thiyaguk09 force-pushed the feat/enable-full-checksum-validation branch from 4091fcc to 2ac9aad Compare March 23, 2026 11:40

chore: code refactor

1e627f4

thiyaguk09 marked this pull request as ready for review March 23, 2026 12:54

thiyaguk09 requested review from a team as code owners March 23, 2026 12:54

Dhriti07 requested a review from nidhiii-27 March 24, 2026 08:37

bshaffer reviewed Mar 31, 2026

View reviewed changes

thiyaguk09 added 4 commits April 1, 2026 08:16

Merge branch 'main' into feat/enable-full-checksum-validation

34f2f0b

feat(core): support all custom headers and fix final chunk hash logic

4895c98

Lint fix

e24a1e3

thiyaguk09 requested a review from bshaffer April 1, 2026 12:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat(Storage): Enable full object checksum validation on JSON path#8825

Feat(Storage): Enable full object checksum validation on JSON path#8825
thiyaguk09 wants to merge 10 commits intogoogleapis:mainfrom
thiyaguk09:feat/enable-full-checksum-validation

thiyaguk09 commented Dec 26, 2025 •

edited

Loading

Uh oh!

bshaffer left a comment

Uh oh!

bshaffer Mar 31, 2026 •

edited

Loading

Uh oh!

thiyaguk09 Apr 1, 2026

Uh oh!

bshaffer Mar 31, 2026

Uh oh!

thiyaguk09 Apr 1, 2026

Uh oh!

bshaffer Mar 31, 2026

Uh oh!

thiyaguk09 Apr 1, 2026

Uh oh!

bshaffer Mar 31, 2026

Uh oh!

thiyaguk09 Apr 1, 2026

Uh oh!

bshaffer Mar 31, 2026

Uh oh!

thiyaguk09 Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		$args['uploaderOptions']['restOptions'] ??= [];
		$args['uploaderOptions']['restOptions']['headers'] ??= [];

Conversation

thiyaguk09 commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Enhanced Checksum Validation & Header Logic

Key Technical Changes

Uh oh!

bshaffer left a comment

Choose a reason for hiding this comment

Uh oh!

bshaffer Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

thiyaguk09 commented Dec 26, 2025 •

edited

Loading

bshaffer Mar 31, 2026 •

edited

Loading