Skip to content

HADOOP-19900. EC write can fail with ArrayIndexOutOfBoundsException due to CoderUtil emptyChunk resize race#8518

Merged
smengcl merged 2 commits into
apache:trunkfrom
smengcl:HADOOP-19900-ec
May 28, 2026
Merged

HADOOP-19900. EC write can fail with ArrayIndexOutOfBoundsException due to CoderUtil emptyChunk resize race#8518
smengcl merged 2 commits into
apache:trunkfrom
smengcl:HADOOP-19900-ec

Conversation

@smengcl
Copy link
Copy Markdown
Contributor

@smengcl smengcl commented May 27, 2026

https://issues.apache.org/jira/browse/HADOOP-19900

Contains content generated by Claude Code (Opus 4.7)

Description of PR

Fix a race in CoderUtil.getEmptyChunk() that can cause EC writes to fail with ArrayIndexOutOfBoundsException during parity encoding.

This can be hit when multiple EC key output streams in the same client JVM use the Java raw EC encoder concurrently with different encode/reset lengths. Each ECKeyOutputStream has its own encoder, but all Java encoders share the static CoderUtil.emptyChunk cache. If native ISA-L is unavailable or not selected, the Java RSRawEncoder clears parity output buffers through CoderUtil.resetOutputBuffers(). Under concurrent close/flush paths, especially with partial final stripes of different sizes, one stream can grow the shared zero buffer for a larger encode while another smaller encode races and shrinks it, causing the larger encode’s later System.arraycopy() to throw ArrayIndexOutOfBoundsException.

For more details, see apache/ozone#10324

How was this patch tested?

  • Added Claude-generated repro test.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

smengcl added 2 commits May 26, 2026 20:19
Generated-by: Claude Code (Opus 4.7)
Copy link
Copy Markdown
Contributor

@ZanderXu ZanderXu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch.

LGTM +1

@hadoop-yetus

This comment was marked as resolved.

@smengcl smengcl marked this pull request as ready for review May 28, 2026 00:30
Copilot AI review requested due to automatic review settings May 28, 2026 00:30
@smengcl
Copy link
Copy Markdown
Contributor Author

smengcl commented May 28, 2026

Thanks @ZanderXu for reviewing this!

Merging this as yetus looks good.

@smengcl smengcl merged commit e33eb71 into apache:trunk May 28, 2026
6 of 7 checks passed
@smengcl smengcl deleted the HADOOP-19900-ec branch May 28, 2026 00:33
@smengcl smengcl review requested due to automatic review settings May 28, 2026 00:52
smengcl added a commit that referenced this pull request May 28, 2026
…ue to CoderUtil emptyChunk resize race (#8518)

Generated-by: Claude Code (Opus 4.7)
(cherry picked from commit e33eb71)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants