[fix](be) Fix row binlog memtable flush backpressure - #66847
Draft
sollhui wants to merge 3 commits into
Draft
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#62007 Problem Summary: A row-binlog group memtable produces two physical flush tasks, but write backpressure used the same physical task limit as a normal data memtable. This effectively halved logical flush concurrency and serialized writes behind flushes. Use twice the configured task limit for group writes in both local and cloud modes. In cloud mode, also apply backpressure when the S3 upload queue is busy, while retaining the flush-running-count memory bound. ### Release note None ### Check List (For Author) - Test: Not run (per request) - Behavior changed: Yes, row-binlog writes use the intended logical flush concurrency and cloud writes also honor S3 upload queue pressure - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Keep this change focused on correcting the physical flush task limit for row-binlog group writes. Remove the additional cloud S3 upload queue condition and its supporting initialization and test changes.
Use the doubled physical flush task limit only for local row-binlog group writers. In cloud mode, replace flush-count backpressure with the S3 upload queue busy signal and initialize that signal independently of adaptive flush-thread adjustment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #62007
Problem summary
A row-binlog group memtable flush creates two physical flush tasks: one for data and one for row binlog. Both tasks increment
flush_running_count, but write backpressure previously compared that physical count with the same configured limit used by a normal data-only memtable.As a result, enabling row binlog effectively halved the number of logical memtables allowed to flush concurrently in storage-compute integrated mode. With a limit of 2, one group memtable occupied the whole allowance, so subsequent writes waited for the previous group flush and the write/flush pipeline became nearly serial.
In storage-compute separated mode, physical flush task completion does not represent completion of the asynchronous S3 upload. Backpressure should follow the S3 upload queue instead of the number of running memtable flush tasks.
Changes
configured_limit * 2for a row-binlogGROUPrequest and keep the configured limit unchanged for normal requests.AdaptiveThreadPoolController::is_io_busy(), which checks the S3 upload queue.Impact
Row-binlog imports can keep memtable writing and flushing overlapped instead of being prematurely serialized. Cloud writes are throttled when asynchronous S3 uploads accumulate.
Release note
None
Check List (For Author)