Bump apache-airflow from 2.10.3 to 3.3.0

[dependabot]

Bumps apache-airflow from 2.10.3 to 3.3.0.

Release notes

Sourced from apache-airflow's releases.

Apache Airflow 3.3.0

📦 PyPI: https://pypi.org/project/apache-airflow/3.3.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.3.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.3.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:3.3.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.3.0

Significant Changes

Asset Partitioning (#64571, #65447, #66030, #66848, #67184, #67475, #67716, #68978)

Building on the asset partitioning introduced in 3.2.0, Airflow 3.3.0 substantially expands how a single upstream asset event fans out to partitioned downstream Dag runs. New partition mappers — RollupMapper (many-to-one), FanOutMapper (one-to-many), and FixedKeyMapper + SegmentWindow (categorical rollup) — compose with time windows (day/week/month/quarter/year) and a wait_policy (WaitForAll or MinimumCount(n)) to control when partitioned runs fire. Windows can fan out forward or backward in time, and total fan-out per upstream event is bounded by the new [scheduler] partition_mapper_max_downstream_keys config (configurable per mapper). Airflow 3.3.0 also adds the PartitionedAtRuntime timetable, which lets a Dag declare that its partition key(s) are assigned when the run starts rather than mapped from an upstream event.

For detailed usage instructions, see :doc:/authoring-and-scheduling/assets.

Task and Asset State Store (#65759, #66073, #66160, #66463, #66586, #66859, #67041, #67292, #67319)

Airflow 3.3.0 introduces a first-class state store for tasks and assets (AIP-103). Tasks can persist arbitrary key-value state that survives across retries and runs via a new task_state_store accessor, and assets can carry their own state via asset_state_store — both available from the Task SDK. State is kept in the metadata database by default, or in a custom worker-side backend ([workers] state_store_backend), supports per-key retention with periodic garbage collection and an optional clear_on_success, and is fully manageable through the Core API and Execution API.

For detailed usage instructions, see :doc:/core-concepts/task-and-asset-state-store.

Pluggable Retry Policies (#65474)

Task retry behaviour is now pluggable (AIP-105). In addition to a fixed retries count, you can attach a custom retry policy that decides whether and when a task is retried, enabling strategies such as retrying only on specific exceptions or backing off based on custom logic.

For detailed usage instructions, see :ref:concepts:retry-policies.

Language Task SDK (Java and Go) (#65958, #67161, #67635, #67699)

Airflow 3.3.0 adds a Coordinator layer (AIP-108) that lets individual task implementations be written in non-Python languages while the Dag and its scheduling stay in Python. A task is declared in the Dag

... (truncated)

Changelog

Sourced from apache-airflow's changelog.

.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Dockerfile Changelog

The Dockerfile does not strictly follow the SemVer <https://semver.org/>_ approach of Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly follows it, the Dockerfile is really a way to give users a conveniently packaged Airflow using standard container approach, so occasionally there are some changes in the building process or in the entrypoint of the image that require slight adaptation of how it is used or built.

The Changelog below describes the changes introduced in each version of the docker images released by the Airflow team.

:note: The Changelog below concerns only the convenience production images released at Airflow DockerHub <https://hub.docker.com/r/apache/airflow>_ . The images that are released there are usually built using the Dockerfile released together with Airflow. However, you are free to take latest released Dockerfile from Airflow and use it to build an image for any Airflow version from the Airflow 2 line. There is no guarantee that it will work, but if it does, then you can use latest features from that image to build images for previous Airflow versions.

Airflow 3.4.0

  * The ``tdsodbc`` package was added to the image so that the FreeTDS ODBC driver
    (``libtdsodbc.so``) is available for connecting to Sybase/TDS databases via ODBC.
Airflow 3.1.4

In Airflow 3.1.4, the images are build without removing of .pyc and .pyo files when Python is built. This increases the size of the image slightly (<0.5%), but improves performance of Python in the container because Python does not need to recompile the files on the first run but more importantly, if you use exec to run Health Checks, removed .pyc files caused a small but ever growing memory leak in the Unix kernel connected to negative dentries created when .pyc files were attempted to be compiled and failed.

... (truncated)

Commits

• 1438ea3 update release notes for 3.3.0rc2
• fa08188 [v3-3-test] Fix asset event ingestion crash for Dags using FixedKeyMapper (#6...
• 43562ca [v3-3-test] Fix details panel header overlapping the tabs (#69255) (#69318)
• 428b89b [v3-3-test] Rename state-store cleanup-task-state-store command to `state-s...
• a1579dc [v3-3-test] Fix new DAG versions from serializing a retry_policy object (#692...
• e494673 [v3-3-test] Clarify when last_automated_data_interval is None in timetable do...
• def55d0 [v3-3-test] Fix deadline callback data persistence (#69073) (#69259)
• 9e0fb00 [v3-3-test] Fix retry policy overrides not persisted to task instance history...
• bf56f0f [v3-3-test] Add new Committers to CODEOWNERS (#69215) (#69219)
• 95128d0 Update SQLAlchemy documentation links from 1.4 to 2.0 (#69200) (#69203)
• Additional commits viewable in compare view

[dmiltr3]

The upgrade of apache-airflow from 2.10.3 to 3.3.0 introduces breaking changes across the major version boundary, causing CI e2e test failures (tests (3.10, e2e, DEFAULT) and tests (3.11, e2e, DEFAULT)).

TFX Airflow orchestration integration and provider packages currently depend on Airflow 2.x APIs. Please update the test constraints to a compatible 2.x release (e.g. latest 2.10.x patch release) or ensure full compatibility with Airflow 3.x before bumping the major version constraint.

[dmiltr3]

CI unit test checks are failing on Python 3.10 and 3.11 (tests (3.10, e2e, DEFAULT) and tests (3.11, e2e, DEFAULT)). Please rebase this pull request onto the latest master branch (e.g. by commenting @dependabot rebase) and ensure all CI test suites pass.

[nithyak0204]

@dependabot rebase