io: Fix checksum seek at end

[Lloyd-Pottiger]

What problem does this PR solve?

Issue Number: close #10340

Problem Summary:

When there are multiple NULL rows of Strings-like column or vector-like column, TiFlash may save some "empty blocks" into the DMFile. And it happen to be the "empty blocks" are stored at the end of the DMFile.

When TiFlash try to read data from disk, DMFileReader::readFromDisk will first call FramedChecksumReadBuffer::doSeek and seek to the "empty block" at the file end. The working_buffer will be release because we read to the end of the file. When TiFlash try to seek to the next "empty block", pos move forward the released working_buffer, leading to reading random data and cause random failure.

tiflash/dbms/src/Storages/DeltaMerge/File/DMFileReader.cpp

Lines 542 to 555 in 76a4ec4

	type_on_disk->deserializeBinaryBulkWithMultipleStreams( //
	*mutable_col,
	[&](const IDataType::SubstreamPath & substream_path) {
	const auto substream_name = DMFile::getFileNameBase(cd.id, substream_path);
	auto & sub_stream = column_streams.at(substream_name);
	sub_stream->buf->seek(
	sub_stream->getOffsetInFile(start_pack_id),
	sub_stream->getOffsetInDecompressedBlock(start_pack_id));
	return sub_stream->buf.get();
	},
	read_rows,
	top_stream->avg_size_hint,
	true,
	{});

tiflash/dbms/src/IO/Checksum/ChecksumBuffer.h

Lines 436 to 440 in dce9588

	if (target_frame == current_frame)
	{
	pos = working_buffer.begin() + target_offset;
	return offset;
	}

What is changed and how it works?

1. In `FramedChecksumReadBuffer::doSeek`, add guard check for `working_buffer` size and `target_offset`. If the working_buffer is cleared, then re-seek the working buffer according to the offset and load data from the underlying file again.
2. In order to support the behavior above, `S3RandomAccessFile::seekImpl` support seek backward, which is implemented by reopening the file again
3. Add retry when reading from S3 meet errno=115, EINPROGRESS

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Fix possible read fail when column data contains lots of "NULL" values.

[Lloyd-Pottiger]

/check-issue-triage-complete

[Lloyd-Pottiger]

/hold

[JaySon-Huang]

why do we need to seek backward?

[breezewish]

// If we have already seek to EOF, then working_buffer was cleared
    if (target_frame == current_frame && working_buffer.size() > 0)
    {
        if (unlikely(target_offset > working_buffer.size()))
            pos = working_buffer.end();
        else
            pos = working_buffer.begin() + target_offset;
        return offset;
    }

If the same target frame is seeked again it is seeked in backward.

BTW it would be great if we could figure out some ways to avoid seeking backward.

[Lloyd-Pottiger]

/unhold

[JaySon-Huang]

highlight what is changed

[JaySon-Huang]

The main change in here is to change

        pos = working_buffer.begin() + target_offset;

to

        if (unlikely(target_offset > working_buffer.size()))
            pos = working_buffer.end();
        else
            pos = working_buffer.begin() + target_offset;

[JaySon-Huang]

The change in here is change

    if (target_frame == current_frame)
    {
        pos = working_buffer.begin() + target_offset;
        return offset;
    }

to

    // If we have already seek to EOF, then working_buffer was cleared
    if (target_frame == current_frame && working_buffer.size() > 0)
    {
        if (unlikely(target_offset > working_buffer.size()))
            pos = working_buffer.end();
        else
            pos = working_buffer.begin() + target_offset;
        return offset;
    }

[JaySon-Huang]

LGTM

[Lloyd-Pottiger]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, JinheLin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,JinheLin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-08-19 16:05:39.472356602 +0000 UTC m=+369547.415532117: ☑️ agreed by JaySon-Huang.
• 2025-08-20 03:54:43.773329433 +0000 UTC m=+412091.716504949: ☑️ agreed by JinheLin.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.5: #10354.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.1: #10355.
But this PR has conflicts, please resolve them!