Refactor scheduler task concurrency evaluation into a dedicated helper.#67589
Draft
SameerMesiah97 wants to merge 1 commit into
Draft
Refactor scheduler task concurrency evaluation into a dedicated helper.#67589SameerMesiah97 wants to merge 1 commit into
SameerMesiah97 wants to merge 1 commit into
Conversation
Move serialized DAG retrieval, concurrency checks, and starvation bookkeeping out of _executable_task_instances_to_queued.
boring-cyborg
Bot
added
the
area:Scheduler
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change refactors task concurrency evaluation logic out of
SchedulerJobRunner._executable_task_instances_to_queuedinto a dedicated helper method.Task concurrency checks for
max_active_tis_per_dagandmax_active_tis_per_dagrun, along with their associated serialized DAG loading, starvation bookkeeping, and missing DAG handling, were previously implemented inline inside the main scheduling loop. This logic has now been extracted into_task_concurrency_allows_execution.The new helper encapsulates serialized DAG retrieval and missing DAG handling, task-level and task-per-DAG-run concurrency checks, and concurrency starvation bookkeeping and logging.
Rationale
_executable_task_instances_to_queuedcurrently performs a large number of responsibilities, including pool management, DAG-level concurrency enforcement, task-level concurrency evaluation, executor routing, starvation tracking, and queue state transitions.The task concurrency section in particular introduced deeply nested control flow and intermixed concurrency-specific behavior directly inside the main scheduling loop, making the scheduling path harder to visually scan and maintain.
Tests
No test adjustments were required because this change is a refactor only and does not introduce any behavioral changes.
Backwards Compatibility
This change is a refactor only and does not modify scheduler behavior, public APIs, task concurrency semantics, or executor interactions.
Was generative AI tooling used to co-author this PR?
Generated-by: [GPT 5.5] following the guidelines