Skip to content

Comments

fix : Improve historical_metrics_data endpoint #62158

Open
iharsh02 wants to merge 2 commits intoapache:mainfrom
iharsh02:api-enhance
Open

fix : Improve historical_metrics_data endpoint #62158
iharsh02 wants to merge 2 commits intoapache:mainfrom
iharsh02:api-enhance

Conversation

@iharsh02
Copy link
Contributor

@iharsh02 iharsh02 commented Feb 19, 2026

Improve historical_metrics_data endpoint

  1. Refactor queries to be "Sargable" (index-friendly)
  2. Combine dag_run_types and dag_run_states queries

Key findings

  • Wrapping columns in functions (like coalesce) effectively prevents the database from using standard B-Tree indexes on those columns (unless a specific function-based index exists). This forces a full table scan or index scan, which is slow on large tables.
  • The DagRun table has no index on start_date or end_date , the DB must linearly scan every remaining row to check the date range.

closes : #62021
related: ##62020


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Copy link
Contributor

@viiccwen viiccwen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall lg.
but...pls fix CI first, and ensure testing all good. : )
also give a benchmark result to make sure your optimized solution is better.

@henry3260
Copy link
Contributor

I think this is duplicated with #62152

@viiccwen
Copy link
Contributor

right.

@iharsh02 iharsh02 marked this pull request as draft February 19, 2026 10:21
@iharsh02
Copy link
Contributor Author

  • The DagRun table has no index on start_date or end_date , the DB must linearly scan every remaining row to check the date range.

@iharsh02 iharsh02 marked this pull request as ready for review February 19, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API - Improve historical_metrics_data endpoint

3 participants