Skip to content

[filesystem] avoid zero-skip loop in HadoopDataInputStream#3738

Draft
wolfkill wants to merge 1 commit into
apache:mainfrom
wolfkill:fix/hadoop-seek-past-eof
Draft

[filesystem] avoid zero-skip loop in HadoopDataInputStream#3738
wolfkill wants to merge 1 commit into
apache:mainfrom
wolfkill:fix/hadoop-seek-past-eof

Conversation

@wolfkill

@wolfkill wolfkill commented Jul 22, 2026

Copy link
Copy Markdown

Fixes #3718.

Root Cause

HadoopDataInputStream.seek() optimizes small forward seeks by skipping instead of issuing a Hadoop seek().

The old local skipFully() loop assumed every FSDataInputStream.skip(bytes) call makes forward progress. Some streams return 0 at EOF or when no bytes can currently be skipped. In that case the remaining byte count never decreases and the loop can spin forever.

Changes

  • Delegate small forward skips to Hadoop's org.apache.hadoop.io.IOUtils.skipFully(InputStream, long).
  • Remove the local skipFully implementation.
  • Add a regression test using a seekable stream that returns one zero-byte skip and then EOF on read, verifying EOFException and that zero-progress skip is not retried.

Validation

  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -Dtest=HadoopDataInputStreamTest#testSeekPastEndOfStreamThrowsEOFException -DfailIfNoTests=false test
  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -Dtest=HadoopDataInputStreamTest -DfailIfNoTests=false test
  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -DfailIfNoTests=false test
  • git diff --check upstream/main...HEAD

@litiliu

litiliu commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

would it be better to use the org.apache.hadoop.io.IOUtils.skipFully?

@wolfkill

Copy link
Copy Markdown
Author

Updated this to use org.apache.hadoop.io.IOUtils.skipFully(fsDataInputStream, delta).

I checked the Hadoop 3.4.0 implementation before changing it: when skip() returns 0, IOUtils.skipFully attempts a single-byte read() and throws EOFException if that returns -1, so it avoids the zero-skip loop and gives the expected prompt EOF signal for seek-past-end.

The regression test now verifies that path by asserting EOFException, one skip() call, and one EOF read() call. I also rebased the branch onto latest apache/main; current branch head is e324d87e, and compare reports behind_by=0, ahead_by=1.

Local verification:

  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -Dtest=HadoopDataInputStreamTest#testSeekPastEndOfStreamThrowsEOFException -DfailIfNoTests=false test
  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -Dtest=HadoopDataInputStreamTest -DfailIfNoTests=false test
  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -DfailIfNoTests=false test
  • git diff --check upstream/main...HEAD

@wolfkill
wolfkill force-pushed the fix/hadoop-seek-past-eof branch from 9c887a3 to e324d87 Compare July 23, 2026 08:39
@litiliu

litiliu commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

looks good to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[filesystem] HadoopDataInputStream.seek loops forever past EOF

2 participants