[fix](be) Account for TaskExecutor admission in scanner scheduling

[mrhhsg]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

ScannerContext counted every scanner submitted to TaskExecutor as progress-capable in-flight work. However, TaskExecutor admits only the TaskHandle target concurrency and keeps excess leaf splits in a per-handle admission queue. When an admitted non-EOS scanner yielded, the queued splits could keep the ScannerContext scheduling margin closed, preventing the yielded scanner from being re-enqueued even though it still owned an admission slot. If every admitted scanner reached that state, neither scheduler could make progress.

This change exposes the number of accepted but unadmitted leaf splits through TaskHandle and separates two scheduling budgets:

1. First-time scanner submission keeps the original margin based on total in-flight tasks, including TaskExecutor's private admission queue.
2. Admission-aware margin can only select yielded tasks whose split already exists in TaskExecutor (is_first_schedule=false) and therefore calls re_enqueue_split. It cannot create a new split or enlarge the TaskExecutor admission backlog.

The ScannerContext limit is therefore no longer reduced to the TaskExecutor admission limit, while TaskExecutor can continue to adapt its own admission target without blocking the continuation of already-admitted scanners.

A deterministic integration unit test uses six yielding scanners, a ScannerContext limit of four, and a TaskHandle admission limit of two. It verifies that the initial excess splits enter the admission queue, yielded scanners remain schedulable, all six scanners reach EOS, and total submitted work never exceeds the ScannerContext limit. A focused selection assertion also verifies that admission-aware scheduling refuses a first-time scanner and only selects an existing split.

Release note

Fix a possible file scan scheduling stall when ScannerContext concurrency exceeds TaskExecutor admission.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
    • ENABLE_UNITY_BUILD=OFF ./run-be-ut.sh --run --filter='ScannerContextTest.*:TimeSharingTaskExecutorTest.*' -j 32 (43 passed, ASAN UT)
    • ENABLE_UNITY_BUILD=OFF ./run-be-ut.sh --run --filter='ScannerContextTest.task_executor_admission_queue_keeps_scanners_live' -j 32
    • build-support/clang-format.sh
    • build-support/check-format.sh
    • git diff --check
    • Clang-tidy was attempted. Analysis of ScannerContext files is blocked by the existing unmatched NOLINTEND in be/src/core/types.h:576; changed TaskHandle files reported no warnings.
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes. Yielded admitted scanners can resume without granting additional first-time TaskExecutor submissions.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[mrhhsg]

/review

[github-actions]

Request changes: the fixed-cap case now avoids the private TaskHandle backlog, but one performance/configuration defect remains because the initial admission target becomes the ScannerContext lifetime maximum.

Critical checkpoints:

• Goal and proof: the real-executor test deterministically covers the original four-scanner/two-slot liveness stall, and the fixed initial=max path now reaches EOS.
• Scope and reuse: the two-file change is focused. The duplicated effective-capacity formula matches current scheduler construction, though an executor/handle capacity API would be safer when addressing the inline issue.
• Concurrency: executor workers, the pipeline consumer, _transfer_lock, scheduler lock, and handle/executor locks were traced end to end. No new lock-order, atomic-order, or wake-up defect was found at a fixed cap.
• Lifecycle and errors: init failure, non-EOS park/re-enqueue, both EOS replacement orders, cancellation, stop, destruction, and repeated handle removal retain cleanup/progress paths.
• Configuration, parallel paths, and compatibility: existing startup-immutable initial/max settings match for internal and external task-executor schedulers; thread-pool and one-split paths are appropriately unaffected. There are no protocol, storage-format, symbol, or rolling-upgrade changes.
• Tests: test state restoration, scheduler lifetime, dependency polling, and expected row/EOS checks are sound. The missing initial < max case is the gap identified inline. No build or test was run in this review because the review runner forbids builds; git diff --check passed.
• Observability, persistence, and writes: the profile records the resulting max concurrency; no additional logging/metric need was found. Transactions, persistence, data writes, and FE-BE variable propagation are not involved.
• Performance: the accepted inline issue can pin a file scan at the initial target even after the handle raises its live target. No other CPU, memory, or redundant-work issue survived review.
• User focus: no additional focus was provided.

Review status: converged after two rounds of normal full review plus separate risk-focused scans. One inline issue is submitted; every other candidate was deduplicated or dismissed with code evidence.

[github-actions]

The assignment immediately below turns task_executor_initial_max_concurrency_per_task into a lifetime cap for the context. That setting is documented as an initial target, and SplitConcurrencyController plus test_min_concurrency_per_task_when_target_concurrency_increases explicitly support growing it. With initial=2, max=16, and a file scan eligible for 16 scanners, the handle target can grow but _pull_next_scan_task() can never submit a third occupied slot after this assignment, so the scan stays pinned at two scanners. The new test uses initial=max=2, which hides this regression. Please preserve the eventual local/configured maximum while coordinating submissions with the handle's current admission capacity (or otherwise fixing the hidden-slot accounting), and add an initial < max test that proves both liveness and scale-up.