Unify executor workload queues

[anishgirianish]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Summary

Refactors executor workload queue management for extensibility. No behavioral change , scheduling order, slot accounting, and all provider executors work identically to before.

Follows the direction proposed by @ferruzzi #62343 (comment).

Problem

Adding a new workload type (like ExecuteCallback or TestConnection) required touching ~6 places in BaseExecutor: a new queue dict, a new supports_* flag,slots calculation, an isinstance branch in queue_workload, a dedicated scheduling method, and isinstance branches in dequeue/trigger logic. Each provider executor that overrode queue_workload also needed updating. This made extending the executor interface unnecessarily painful.

What this does

Replaces the per-type queue dicts and boolean capability flags with three simple primitives:

• executor_queues: a single defaultdict(dict) keyed by workload type string (e.g. "ExecuteTask","ExecuteCallback") instead of separatequeued_tasks / queued_callbacks dicts
• supported_workload_types: a frozenset of type strings instead of individual supports_callbacks booleans
• WORKLOAD_TYPE_PRIORITY + sort_key / queue_key : properties on each workload schema that control scheduling priority and queue indexing

The base class queue_workload is now generic: validate the type, store by key. Four provider executors (K8s, ECS, Batch, Lambda) no longer need their own queue_workload overrides. trigger_tasks becomes trigger_workloads since it handles all workload types now.

Adding a new workload type after this refactor

1. Define queue_key and sort_key on the workload schema
2. Add the type string to supported_workload_types on supporting executors
3. Handle the type in _process_workloads, done

No changes needed in BaseExecutor itself.

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[ferruzzi]

Made a real quick pass and left some comments and questions, I'll try to get a more thorough one tomorrow.

[kaxil]

Left 8 inline comments on this PR.

[Copilot]

Pull request overview

Refactors Airflow executor workload queue management to use a unified, type-driven queue structure (instead of per-workload dicts and per-feature flags), aiming to make new workload types easier to add while keeping scheduling/slot behavior consistent.

Changes:

• Introduces WorkloadType, WORKLOAD_TYPE_TIER, and per-workload queue_key/sort_key to drive queueing and scheduling.
• Replaces queued_tasks/queued_callbacks internal storage with BaseExecutor.executor_queues and replaces supports_callbacks with supported_workload_types (with deprecation shims).
• Renames scheduling entrypoint from trigger_tasks to trigger_workloads and updates core/provider tests accordingly.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
airflow-core/src/airflow/executors/base_executor.py	Core refactor: unified executor_queues, supported_workload_types, trigger_workloads, and backward-compat properties.
airflow-core/src/airflow/executors/local_executor.py	Updates LocalExecutor to use unified queues and generic dequeue by queue_key.
airflow-core/src/airflow/executors/workloads/base.py	Adds WorkloadType registry and WORKLOAD_TYPE_TIER plus base queue_key/sort_key API.
airflow-core/src/airflow/executors/workloads/task.py	Makes ExecuteTask.type use WorkloadType and adds queue_key/sort_key.
airflow-core/src/airflow/executors/workloads/callback.py	Makes ExecuteCallback.type use WorkloadType and adds queue_key.
airflow-core/src/airflow/executors/workloads/types.py	Adds QueueableWorkload type alias for workloads that flow through executor queues.
airflow-core/src/airflow/executors/workloads/init.py	Exports WorkloadType and WORKLOAD_TYPE_TIER from the workloads package.
airflow-core/tests/unit/executors/test_base_executor.py	Updates tests to new APIs; adds coverage for backward-compat properties.
airflow-core/tests/unit/executors/test_local_executor.py	Updates tests to new unified queue access patterns.
devel-common/src/tests_common/test_utils/mock_executor.py	Updates mock executor to use executor_queues keyed by WorkloadType.
devel-common/src/tests_common/test_utils/version_compat.py	Adds AIRFLOW_V_3_3_PLUS test constant for compatibility branching.
providers/celery/src/airflow/providers/celery/executors/celery_executor.py	Switches CeleryExecutor to supported_workload_types instead of supports_callbacks.
providers/celery/src/airflow/providers/celery/executors/celery_kubernetes_executor.py	Updates queued task merging implementation (copy/update instead of dict union).
providers/celery/tests/unit/celery/executors/test_celery_executor.py	Updates test patching for renamed trigger method (currently version-gated).
providers/celery/tests/integration/celery/test_celery_executor.py	Updates integration test to call trigger_workloads.
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py	Removes legacy TYPE_CHECKING queued_tasks typing branch.
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/local_kubernetes_executor.py	Adjusts queued task merging implementation (copy/update) and adds type-ignore/TODO.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py	Updates test patching for renamed trigger method (currently version-gated).
providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py	Removes legacy TYPE_CHECKING queued_tasks typing branch.
providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py	Removes legacy TYPE_CHECKING queued_tasks typing branch and unused version compat import.
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py	Removes legacy TYPE_CHECKING queued_tasks typing branch and unused version compat import.
providers/edge3/tests/unit/edge3/executors/test_edge_executor.py	Adjusts test setup to mutate queued tasks mapping in-place.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py	Imports new AIRFLOW_V_3_3_PLUS and gates patch target between old/new method name.

[potiuk]

@anishgirianish This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Merge conflicts: This PR has merge conflicts with the main branch. Your branch is 113 commits behind main. Please rebase your branch (git fetch origin && git rebase origin/main), resolve the conflicts, and push again. See contributing quick start.
• ❌ Provider tests: Failing: provider distributions tests / Compat 2.11.1:P3.10:, provider distributions tests / Compat 3.0.6:P3.10:, provider distributions tests / Compat 3.1.8:P3.10:, Non-DB tests: providers / Non-DB-prov::3.10:-amazon,celer...standard, Special tests / Latest Boto test: providers / All-prov:LatestBoto-Postgres:14:3.10:-amazon,celer...standard (+1 more). Run provider tests with breeze run pytest <provider-test-path> -xvs. See Provider tests docs.
• ❌ Unit tests: Failing: Helm tests / Unit tests Helm: apiserver (K8S 1.30.13). Run failing tests with breeze run pytest <path> -xvs. See Unit tests docs.

Note: Your branch is 113 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

[potiuk]

@anishgirianish This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Merge conflicts: This PR has merge conflicts with the main branch. Your branch is 68 commits behind main. Please rebase your branch (git fetch origin && git rebase origin/main), resolve the conflicts, and push again. See contributing quick start.
• ❌ Pre-commit / static checks: Failing: CI image checks / Static checks. Run prek run --from-ref main locally to find and fix issues. See Pre-commit / static checks docs.
• ❌ Provider tests: Failing: provider distributions tests / Compat 2.11.1:P3.10:, provider distributions tests / Compat 3.0.6:P3.10:, provider distributions tests / Compat 3.1.8:P3.10:, Non-DB tests: providers / Non-DB-prov::3.10:-amazon,celer...standard. Run provider tests with breeze run pytest <provider-test-path> -xvs. See Provider tests docs.
• ❌ Other failing CI checks: Failing: Build info, Platform: ARM, Platform: AMD, Verify release calendar, Go SDK tests. Run prek run --from-ref main locally to reproduce. See static checks docs.

Note: Your branch is 68 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

[potiuk]

@anishgirianish This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Merge conflicts: This PR has merge conflicts with the main branch. Your branch is 78 commits behind main. Please rebase your branch (git fetch origin && git rebase origin/main), resolve the conflicts, and push again. See contributing quick start.
• ❌ Pre-commit / static checks: Failing: CI image checks / Static checks. Run prek run --from-ref main locally to find and fix issues. See Pre-commit / static checks docs.
• ❌ Provider tests: Failing: provider distributions tests / Compat 2.11.1:P3.10:, provider distributions tests / Compat 3.0.6:P3.10:, provider distributions tests / Compat 3.1.8:P3.10:, Non-DB tests: providers / Non-DB-prov::3.10:-amazon,celer...standard. Run provider tests with breeze run pytest <provider-test-path> -xvs. See Provider tests docs.
• ❌ Other failing CI checks: Failing: Build info, Platform: ARM, Platform: AMD, Verify release calendar, Go SDK tests. Run prek run --from-ref main locally to reproduce. See static checks docs.

Note: Your branch is 78 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

[potiuk]

You got asked to fix the issues - before marking it ready . Don't do it next time.

[anishgirianish]

You got asked to fix the issues - before marking it ready . Don't do it next time.

Hi @potiuk,

Thank you so much for taking the time to review and for all the work you do maintaining the triage process. I truly appreciate it.

I sincerely apologize for marking the PR as ready without resolving the merge conflicts first. That was my mistake, and I completely understand why it was flagged. I'll be more careful going forward and will make sure everything is in order before marking ready again. Thank you for pointing that out it helps me learn the process better.

I had a question about the provider test failures and would really appreciate your guidance. The few failing tests regarding providers are related to the trigger_tasks AttributeError, No module named'airflow.executors.workloads', 'KubernetesExecutor' has no attribute 'executor_queues' that this PR introduces for 3.3. The tests already have a version guard using AIRFLOW_V_3_3_PLUS to handle backward compatibility, but since main is currently on 3.2, the flag evaluates to False and the tests try to mock the old method name, which no longer exists on this branch.

Would the correct approach be to keep this PR in draft state until main moves to 3.3, and then rebase and mark it ready at that point? I'd love to know what you'd recommend so I can follow the right process.

Thank you so much for your patience and guidance!

[potiuk]

Would the correct approach be to keep this PR in draft state until main moves to 3.3, and then rebase and mark it ready at that point? I'd love to know what you'd recommend so I can follow the right process.

I think we should change the version to 3.3 about now (@rahulvats) ?

Though the thing is that compatibility tests should be adapted to compat tests - this comment when they fail explain it and links to the docs about it:

Read more on how to run the test locally and how to deal with Provider's compatibility with older Airflow versions at:
https://github.com/apache/airflow/blob/main/contributing-docs/testing/unit_tests.rst#compatibility-provider-unit-tests-against-older-airflow-releases

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[ferruzzi]

@seanghaeli - With #62645 merged, this is the next big piece of the puzzle.