HDDS-15734. Implementing position read in BlockInputStream by chungen0126 · Pull Request #10765 · apache/ozone

chungen0126 · 2026-07-14T15:17:04Z

What changes were proposed in this pull request?

Summary

To resolve the significant lock contention experienced during concurrent readVectored operations, this PR implements the ByteBufferPositionedReadable interface for both BlockInputStream and ChunkInputStream. By supporting thread-safe, position-based reads directly into ByteBuffer, we bypass the synchronization overhead of traditional stream locks and greatly improve parallel read performance.

Detailed Changes

Improve Visibility of Initialization State

Declared the initialized flag as volatile. This ensures that once the stream is initialized, the state change is instantly visible to all other concurrent threads, preventing redundant initialization.

Locking for Stream Refresh/Re-acquisition

Introduced a ReentrantLock (passed from BlockInputStream to ChunkInputStream) to guard critical operations such as refreshBlockInfo, acquireClient, and handling read errors. This ensures thread safety during network/client state transitions without locking the entire read path.

Avoid Redundant Pipeline Refreshing

Added a failedPipeline variable to track the pipeline that encountered an issue.
During refreshBlockInfo, the stream now checks if the current pipeline matches failedPipeline. If it does, the redundant refresh operation is skipped, preventing multiple threads from hammering the refresh logic concurrently for the same failure.

Adopt BlockInputStream Positional Read in OzoneFSInputStream

Updated OzoneFSInputStream to leverage the positional read capabilities of BlockInputStream. This allows OzoneFSInputStream to delegate position-based reads directly to the underlying block stream, bypassing global stream-level locks and enabling fully concurrent read operations.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15734

How was this patch tested?

Add test in TestChunkInputStream and TestBlockInputStream. Also test by TestOzoneFSInputStream.
CI: https://github.com/chungen0126/ozone/actions/runs/29289303279

chungen0126 · 2026-07-15T04:01:59Z

cc @yandrey321

jojochuang · 2026-07-15T05:45:06Z

Bugbot review

readFully succeeds at EOF — hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java:328-338 · Severity: high

readFully returns true when the target offset is at or past the end of the block, or when no bytes can be read from the current chunk, instead of throwing EOFException or otherwise signaling failure. Callers such as OzoneFSInputStream treat a true return as a completed positioned read and compute bytes transferred from buffer remaining; an unfilled buffer yields zero bytes read while hasRemaining() stays true, which can spin forever or return silently short reads.

Positioned read leaks clients — hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java:833-851 · Severity: high

getClientAndUpdateBlock calls acquireClientForReadData on every positioned read(long, ByteBuffer) but never calls releaseClientForReadData. BlockInputStream.readFully uses this path for each chunk segment, so each positioned read leaks an xceiver client and its connections.

readFully checks buffer capacity — hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java:824-830 · Severity: medium

readFully compares the result of read against byteBuffer.capacity() instead of the number of bytes the caller asked to fill (byteBuffer.remaining() at entry). For a buffer with a non-zero position or a limit below capacity, a successful read can incorrectly throw EOFException, or accept a short read as complete.

Refresh skipped after first failure — hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java:217-223 · Severity: medium

After the first refresh attempt for a pipeline, failedPipeline is set to the current pipelineRef even when refresh did not obtain new location info. Later connectivity failures on the same pipeline skip refreshBlockInfo entirely, so stale pipeline/token metadata may never be refreshed again during retries.

Generated-by: Cursor Bugbot

yandrey321 · 2026-07-16T13:17:26Z


  private BlockData blockData;

+  private Pipeline failedPipeline;


why do we need to track previously failed pipelines?

The reason I track previously failed pipelines is that, in my design, I assume a scenario where multiple position read in BlockInputStream might fail concurrently and attempt to refresh the block info at the same time. Once the first BlockInputStream completes the refresh, subsequent streams will obtain the updated information. If the position read detects that its failed pipeline belongs to the old one and a new pipeline is now available, it should not need to trigger another redundant refresh.

yandrey321 · 2026-07-16T13:18:47Z

+    int len = buffer.remaining();
+    int innerRetries = 0;
+    int chunkIdx = Arrays.binarySearch(chunkOffsets, pos);
+    if (chunkIdx < 0) {


when index can be negative?

If the binary search doesn't find an exact match in the array, it returns a negative number. For details, please see: https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#binarySearch-int:A-int-

yandrey321 · 2026-07-16T13:38:58Z

+      final List<ChunkInputStream> inputStreams = this.chunkStreams;
+      if (inputStreams != null) {
+        for (ChunkInputStream is : inputStreams) {
+          is.releaseClient();


should it be wrapped in try catch?

I don't think it's necessary. They didn't throw any exceptions.

yandrey321 · 2026-07-16T14:05:24Z

  }
+
+  @Test
+  public void testPositionedRead() throws Exception {


these should cover moving forward and back in the file.

I have updated the current tests. If you have any suggestions for improvement, please let me know and I'll update them right away.

yandrey321 · 2026-07-16T14:06:38Z

+    ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
+    int bytesRead = chunkStream.read(30, byteBuffer);
+
+    assertEquals(50, bytesRead);


what if chunk stream has less than 50 bytes?

The buffer is filled until the stream reaches its end. For example, if the buffer capacity is 50 but only 10 bytes are left in the stream, only the first 10 bytes of the buffer will be filled.

yandrey321 · 2026-07-16T14:08:09Z

+  }
+
+  @Test
+  public void testPositionedReadFully() throws Exception {


how do we assert that stream was read till the end as text name suggest?

As it stands, this method will read as much as possible until either the buffer is full or the stream reaches EOF (End of File). The only way we can verify if we've reached the end of the stream is by checking the final remaining length of the buffer.

However, looking at how the upper-level BlockInputStream actually uses it, we might not even need this method. I think we should consider removing it entirely. What do you think?

Here is how the overall architecture/flow would look:

If the stream reaches EOF but the buffer is not yet full:

ChunkInputStream returns normally to BlockInputStream.

BlockInputStream has two choices:

a. If there is another ChunkInputStream available, switch to it and continue reading (loop back).
b. If there are no more chunks, return to MultiPartInputStream.

MultiPartInputStream also has two choices:

a. If there is another BlockInputStream available, switch to it (loop back).
b. If there are no more blocks AND the buffer is still not full, throw an EOFException.

chungen0126 added 3 commits July 14, 2026 04:16

HDDS-15734. Implementing position read in BlockInputStream

9f551e8

fix checkstyle

a882c23

Merge branch 'master' into HDDS-15734

5e07fcd

chungen0126 mentioned this pull request Jul 15, 2026

HDDS-13660. Ozone client support readVectored. #9402

Closed

chungen0126 requested review from adoroszlai, ashishkumar50 and jojochuang July 15, 2026 03:52

address comments

a17f876

peterxcli self-requested a review July 15, 2026 16:41

chungen0126 added 3 commits July 16, 2026 00:53

address comments

4ef5a92

make OzoneFSInputStream adopt BlockInputStream position read

c5ba62d

fix checkstyle

ec3ee4c

yandrey321 reviewed Jul 16, 2026

View reviewed changes

update tests

0c294de


		private BlockData blockData;

		private Pipeline failedPipeline;

Uh oh!

Conversation

chungen0126 commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Summary

Detailed Changes

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

chungen0126 commented Jul 15, 2026

Uh oh!

jojochuang commented Jul 15, 2026

Bugbot review

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chungen0126 commented Jul 14, 2026 •

edited

Loading