Skip to content

[fix](be) Fix row binlog memtable flush backpressure - #66847

Draft
sollhui wants to merge 3 commits into
apache:masterfrom
sollhui:agent/fix-row-binlog-flush-backpressure
Draft

[fix](be) Fix row binlog memtable flush backpressure#66847
sollhui wants to merge 3 commits into
apache:masterfrom
sollhui:agent/fix-row-binlog-flush-backpressure

Conversation

@sollhui

@sollhui sollhui commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #62007

Problem summary

A row-binlog group memtable flush creates two physical flush tasks: one for data and one for row binlog. Both tasks increment flush_running_count, but write backpressure previously compared that physical count with the same configured limit used by a normal data-only memtable.

As a result, enabling row binlog effectively halved the number of logical memtables allowed to flush concurrently in storage-compute integrated mode. With a limit of 2, one group memtable occupied the whole allowance, so subsequent writes waited for the previous group flush and the write/flush pipeline became nearly serial.

In storage-compute separated mode, physical flush task completion does not represent completion of the asynchronous S3 upload. Backpressure should follow the S3 upload queue instead of the number of running memtable flush tasks.

Changes

  • In storage-compute integrated mode, use configured_limit * 2 for a row-binlog GROUP request and keep the configured limit unchanged for normal requests.
  • In storage-compute separated mode, replace flush-running-count backpressure with AdaptiveThreadPoolController::is_io_busy(), which checks the S3 upload queue.
  • Initialize the controller's I/O signal even when adaptive flush-thread adjustment is disabled.
  • Add unit coverage for the S3 upload queue busy threshold.

Impact

Row-binlog imports can keep memtable writing and flushing overlapped instead of being prematurely serialized. Cloud writes are throttled when asynchronous S3 uploads accumulate.

Release note

None

Check List (For Author)

  • Test: Not run (per request)
  • Behavior changed: Yes. Local row-binlog writes use the intended logical flush concurrency, while cloud writes use S3 upload queue pressure.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: apache#62007

Problem Summary: A row-binlog group memtable produces two physical flush tasks, but write backpressure used the same physical task limit as a normal data memtable. This effectively halved logical flush concurrency and serialized writes behind flushes. Use twice the configured task limit for group writes in both local and cloud modes. In cloud mode, also apply backpressure when the S3 upload queue is busy, while retaining the flush-running-count memory bound.

### Release note

None

### Check List (For Author)

- Test: Not run (per request)
- Behavior changed: Yes, row-binlog writes use the intended logical flush concurrency and cloud writes also honor S3 upload queue pressure
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Keep this change focused on correcting the physical flush task limit for row-binlog group writes. Remove the additional cloud S3 upload queue condition and its supporting initialization and test changes.
Use the doubled physical flush task limit only for local row-binlog group writers. In cloud mode, replace flush-count backpressure with the S3 upload queue busy signal and initialize that signal independently of adaptive flush-thread adjustment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants