Conversation
| expectedLength = expectedLengthBuffer.getLong(); | ||
| if (expectedLength < 0) { | ||
| throw new IllegalStateException("Invalid expected ciphertext length."); | ||
| if (expectedLength < LENGTH_HEADER_BYTES + (long) headerLength) { |
There was a problem hiding this comment.
nit. (long) looks unnecessary here because both operands are small int values.
| if (expectedRemaining <= 0) { | ||
| throw new IllegalStateException( | ||
| "Invalid ciphertext state: expectedLength=" + expectedLength | ||
| + ", ciphertextRead=" + ciphertextRead); | ||
| } |
There was a problem hiding this comment.
Is this reachable? With the new lower-bound check in initializeExpectedLength, ciphertextRead <= expectedLength always holds after the header is read, and completed becomes true when they are equal, so this loop is not entered with expectedRemaining <= 0. Shall we remove this?
| "Invalid ciphertext state: expectedLength=" + expectedLength | ||
| + ", ciphertextRead=" + ciphertextRead); | ||
| } | ||
| int bytesToRead = (int) Math.min((long) readableBytes, expectedRemaining); |
There was a problem hiding this comment.
nit. (long) is unnecessary because Math.min(long, long) promotes readableBytes automatically.
| long expectedLength = (long) GcmTransportCipher.LENGTH_HEADER_BYTES + | ||
| streaming.getHeaderLength() + Integer.MAX_VALUE + 1L; |
There was a problem hiding this comment.
nit. Could you add a comment explaining that the remaining ciphertext length is Integer.MAX_VALUE + 1, which becomes Integer.MIN_VALUE when cast to int?
2 participants
What changes were proposed in this pull request?
Fix integer overflow in GcmTransportCipher
Why are the changes needed?
To fix shuffle error in large shuffle block (>2GB) when AES-GCM is enabled for RPC encryption:
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added regression tests. Also, applied the patch and ran Spark jobs with 3GB shuffle blocks successfully in Spark on YARN cluster.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenCode (GPT-5.6 Sol)