From 801df6b0ef5c1b4ca4b1c25412c7200c64b547d1 Mon Sep 17 00:00:00 2001 From: Alejandro Morera Date: Tue, 14 Jul 2026 14:51:22 -0600 Subject: [PATCH 1/5] [DCV-3797] Add Airflow metadata database cleanup how-to Co-Authored-By: Claude Opus 4.8 (1M context) --- .../airflow/cleanup-metadata-database.mdx | 85 +++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 docs/how-tos/airflow/cleanup-metadata-database.mdx diff --git a/docs/how-tos/airflow/cleanup-metadata-database.mdx b/docs/how-tos/airflow/cleanup-metadata-database.mdx new file mode 100644 index 0000000..5ccca27 --- /dev/null +++ b/docs/how-tos/airflow/cleanup-metadata-database.mdx @@ -0,0 +1,85 @@ +--- +title: Clean up the Airflow metadata database in Datacoves +sidebar_label: Cleanup Metadata Database +description: "How to schedule cleanup of the Airflow metadata database in Datacoves with the datacoves_airflow_db_cleanup decorator to keep task_instance, log and other tables from growing unbounded." +sidebar_position: 4 +--- + +# Clean up the Airflow metadata database + +Over time the Airflow metadata database accumulates rows in tables such as `task_instance`, `log`, `job`, `xcom`, `dag_run` and `rendered_task_instance_fields`. Left unchecked it grows to several GB and queries on these tables slow down, which can eventually affect scheduler performance. Datacoves monitors the size of your metadata database and notifies account admins when it crosses recommended thresholds, but the cleanup itself is scheduled by you, in your own DAG, so you stay in control of the retention window. + +:::note +This is available for Airflow 3 environments only. +::: + +## Which tables are cleaned + +By default the cleanup targets the metadata tables that grow with every DAG run: + +- `task_instance` +- `log` +- `rendered_task_instance_fields` +- `job` +- `xcom` +- `dag_run` + +These are the same tables Datacoves monitors for size. `dag` and `dag_version` are never touched, so your DAG history relationships are preserved. You only need to pass `table_names` if you want to restrict a run to a subset of these tables; leaving it unset cleans all of them, which is the recommended setup. + +Not sure how much each table holds or how far back your data goes? Run the decorator with `dry_run=True` first (see [Recommended first run](#recommended-first-run)): the task logs report, per table, how many rows would be removed, without deleting anything. + +## Cleanup decorator + +To clean up the metadata database, use an Airflow DAG with the Datacoves cleanup decorator: + +```python +@task.datacoves_airflow_db_cleanup +``` + +By default it removes records older than 90 days from the tables listed above. + +The decorator can receive: + +- `retention_days`: keep records newer than this many days; older rows are deleted. Defaults to `90`. +- `table_names`: restrict the cleanup to specific tables (any subset of the list above). Defaults to all of them. +- `dry_run`: when `True`, report what would be deleted without deleting anything. Use this first. +- `skip_archive`: when `True` (the default) records are deleted directly. When `False`, deleted rows are copied into `_airflow_deleted__*` archive tables first, giving you a recovery window at the cost of extra space. +- `batch_size`: delete in batches of this many rows instead of all at once. Useful the first time you clean a very large table. +- `connection_id`: the Airflow connection pointing at the api-server. Defaults to `datacoves_airflow_api`, which Datacoves creates and maintains for you. + +## Example DAG + +```python +from pendulum import datetime + +from airflow.sdk import dag, task + + +@dag( + default_args={ + "start_date": datetime(2026, 1, 1), + "owner": "Noel", + "retries": 1, + }, + description="Clean up the Airflow metadata database", + schedule="@weekly", + tags=["maintenance"], + catchup=False, +) +def airflow_db_cleanup(): + @task.datacoves_airflow_db_cleanup( + retention_days=90, + # dry_run=True, # uncomment for a first, non-destructive run + ) + def cleanup(): + pass + + cleanup() + + +airflow_db_cleanup() +``` + +## Recommended first run + +Cleanup is destructive. Run once with `dry_run=True` and check the task logs to confirm the volume of rows that would be removed, then switch to a real run. Start with a conservative `retention_days` (for example 90) and lower it gradually if the database is still large. From c1a480f0b6f933cf77b06496d3e6969312314955 Mon Sep 17 00:00:00 2001 From: Alejandro Morera Date: Thu, 23 Jul 2026 09:51:02 -0600 Subject: [PATCH 2/5] [DCV-3797] Add Airflow 2 cleanup section with airflow db clean example DAG --- .../airflow/cleanup-metadata-database.mdx | 80 ++++++++++++++++--- 1 file changed, 71 insertions(+), 9 deletions(-) diff --git a/docs/how-tos/airflow/cleanup-metadata-database.mdx b/docs/how-tos/airflow/cleanup-metadata-database.mdx index 5ccca27..9dba175 100644 --- a/docs/how-tos/airflow/cleanup-metadata-database.mdx +++ b/docs/how-tos/airflow/cleanup-metadata-database.mdx @@ -1,7 +1,7 @@ --- title: Clean up the Airflow metadata database in Datacoves sidebar_label: Cleanup Metadata Database -description: "How to schedule cleanup of the Airflow metadata database in Datacoves with the datacoves_airflow_db_cleanup decorator to keep task_instance, log and other tables from growing unbounded." +description: "How to schedule cleanup of the Airflow metadata database in Datacoves to keep task_instance, log and other tables from growing unbounded, using the datacoves_airflow_db_cleanup decorator on Airflow 3 or an airflow db clean DAG on Airflow 2." sidebar_position: 4 --- @@ -9,9 +9,10 @@ sidebar_position: 4 Over time the Airflow metadata database accumulates rows in tables such as `task_instance`, `log`, `job`, `xcom`, `dag_run` and `rendered_task_instance_fields`. Left unchecked it grows to several GB and queries on these tables slow down, which can eventually affect scheduler performance. Datacoves monitors the size of your metadata database and notifies account admins when it crosses recommended thresholds, but the cleanup itself is scheduled by you, in your own DAG, so you stay in control of the retention window. -:::note -This is available for Airflow 3 environments only. -::: +The way you run the cleanup depends on your Airflow version: + +- **[Airflow 3](#airflow-3-cleanup-decorator)**: use the `@task.datacoves_airflow_db_cleanup` decorator. Airflow 3 tasks no longer have direct access to the metadata database, so the decorator performs the cleanup through the Datacoves api-server for you. +- **[Airflow 2](#airflow-2-cleanup-dag-with-airflow-db-clean)**: use a DAG that runs the built-in `airflow db clean` command. Airflow 2 workers connect directly to the metadata database, so no extra setup is needed. ## Which tables are cleaned @@ -24,11 +25,11 @@ By default the cleanup targets the metadata tables that grow with every DAG run: - `xcom` - `dag_run` -These are the same tables Datacoves monitors for size. `dag` and `dag_version` are never touched, so your DAG history relationships are preserved. You only need to pass `table_names` if you want to restrict a run to a subset of these tables; leaving it unset cleans all of them, which is the recommended setup. +These are the same tables Datacoves monitors for size. On Airflow 3, `dag` and `dag_version` are never touched, so your DAG history relationships are preserved. You only need to restrict the tables if you want a run to touch a subset of these; leaving the table selection unset cleans all of them, which is the recommended setup. -Not sure how much each table holds or how far back your data goes? Run the decorator with `dry_run=True` first (see [Recommended first run](#recommended-first-run)): the task logs report, per table, how many rows would be removed, without deleting anything. +Not sure how much each table holds or how far back your data goes? Do a dry run first (see [Recommended first run](#recommended-first-run)): the task logs report what would be removed, without deleting anything. -## Cleanup decorator +## Airflow 3: cleanup decorator To clean up the metadata database, use an Airflow DAG with the Datacoves cleanup decorator: @@ -47,7 +48,7 @@ The decorator can receive: - `batch_size`: delete in batches of this many rows instead of all at once. Useful the first time you clean a very large table. - `connection_id`: the Airflow connection pointing at the api-server. Defaults to `datacoves_airflow_api`, which Datacoves creates and maintains for you. -## Example DAG +### Example DAG (Airflow 3) ```python from pendulum import datetime @@ -80,6 +81,67 @@ def airflow_db_cleanup(): airflow_db_cleanup() ``` +## Airflow 2: cleanup DAG with airflow db clean + +On Airflow 2 the cleanup runs the built-in [`airflow db clean`](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#clean) command from a regular task. Datacoves Airflow 2 workers already have the metadata database connection configured, so the DAG below works as is. + +The command accepts: + +- `--clean-before-timestamp`: delete records older than this timestamp. The example computes it from the run date minus 90 days. +- `--tables`: comma-separated list to restrict the cleanup to specific tables (any subset of the list above). Defaults to all cleanable tables. +- `--dry-run`: report what would be deleted without deleting anything. Use this first. +- `--skip-archive`: delete records directly instead of copying them into `_airflow_deleted__*` archive tables. Without it, archived copies keep using disk space, so the example also runs `airflow db drop-archived` afterwards to remove any archive tables left behind by previous runs. + +### Example DAG (Airflow 2) + +```python +from datetime import datetime + +from airflow.decorators import dag +from airflow.operators.bash import BashOperator + + +@dag( + default_args={ + "start_date": datetime(2026, 1, 1), + "owner": "Noel", + "retries": 1, + }, + description="Clean up the Airflow metadata database", + schedule="@weekly", + tags=["maintenance"], + catchup=False, + max_active_runs=1, +) +def airflow_db_cleanup(): + db_cleanup = BashOperator( + task_id="db_cleanup", + bash_command=( + "airflow db clean " + "--clean-before-timestamp '{{ macros.ds_add(ds, -90) }}' " + # "--dry-run " # uncomment for a first, non-destructive run + "--skip-archive " + "--verbose " + "--yes" + ), + ) + + drop_archived = BashOperator( + task_id="drop_archived_tables", + bash_command="airflow db drop-archived --yes", + trigger_rule="all_done", + ) + + db_cleanup >> drop_archived + + +airflow_db_cleanup() +``` + +:::tip +If your database has years of history, deleting everything in one pass can produce a very long transaction. Astronomer maintains a [more elaborate version of this DAG](https://www.astronomer.io/docs/learn/2.x/cleanup-dag-tutorial) that splits the cleanup into date-range batches; consider it for the first cleanup of a very large database, then switch to the simpler DAG above for the recurring run. +::: + ## Recommended first run -Cleanup is destructive. Run once with `dry_run=True` and check the task logs to confirm the volume of rows that would be removed, then switch to a real run. Start with a conservative `retention_days` (for example 90) and lower it gradually if the database is still large. +Cleanup is destructive. Do a dry run first (`dry_run=True` on the Airflow 3 decorator, `--dry-run` on `airflow db clean` in Airflow 2) and check the task logs to confirm the volume of rows that would be removed, then switch to a real run. Start with a conservative retention window (for example 90 days) and lower it gradually if the database is still large. From 07c61a2bbe9ac2868e995958e7c3c17293d8ce60 Mon Sep 17 00:00:00 2001 From: Alejandro Morera Date: Fri, 24 Jul 2026 16:40:55 -0600 Subject: [PATCH 3/5] Document datacoves my parse-logs command --- docs/reference/airflow/datacoves-commands.md | 37 +++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/docs/reference/airflow/datacoves-commands.md b/docs/reference/airflow/datacoves-commands.md index 6aabd68..9250bc7 100644 --- a/docs/reference/airflow/datacoves-commands.md +++ b/docs/reference/airflow/datacoves-commands.md @@ -1,7 +1,7 @@ --- title: "Datacoves CLI Commands Reference: datacoves my" sidebar_label: CLI Commands -description: "Reference for Datacoves CLI commands: datacoves my import, my pytest, and my api-key for managing My Airflow instances from the VS Code terminal." +description: "Reference for Datacoves CLI commands: datacoves my import, my pytest, my api-key, and my parse-logs for managing My Airflow instances from the VS Code terminal." sidebar_position: 126 --- # Datacoves CLI Commands @@ -17,6 +17,7 @@ Currently, the `datacoves my` subcommand has the following subcommands: - `my import` - `my pytest` - `my api-key` +- `my parse-logs` ### datacoves my import @@ -79,3 +80,37 @@ Use the first 8 characters of the token (shown in the list command) to identify ```bash datacoves my api-key delete abc12345 ``` + +### datacoves my parse-logs + +:::note +My Airflow [must be instantiated](/docs/how-tos/my_airflow/start-my-airflow) for this command to work. +::: + +This command helps you inspect and optimize how your DAGs are parsed by My Airflow's dag-processor. With no arguments it shows a report of how long each DAG file takes to parse, sorted slowest first, so you can quickly see which files are worth optimizing. + +```bash +datacoves my parse-logs +``` + +#### Show the native Airflow report + +Add `--raw` to print the report exactly as `airflow dags report` produces it: + +```bash +datacoves my parse-logs --raw +``` + +#### Inspect a single DAG's parse log + +Pass `--file` with a DAG file name (a partial name is fine) to see the detailed parse log for that file. This is useful for diagnosing import errors or slow imports on a specific DAG. + +```bash +datacoves my parse-logs --file my_dag.py +``` + +Add `--follow` (`-f`) to stream the log live as you edit and save your DAG, and `--lines` (`-n`) to control how many lines are shown: + +```bash +datacoves my parse-logs --file my_dag.py --follow +``` From bc5a469a61c7e8abf4ffa7ce95998aa589f5d5ea Mon Sep 17 00:00:00 2001 From: Alejandro Morera Date: Fri, 24 Jul 2026 16:43:35 -0600 Subject: [PATCH 4/5] Add how-to for optimizing DAG parsing with parse-logs --- .../my_airflow/optimize-dag-parsing.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 docs/how-tos/my_airflow/optimize-dag-parsing.md diff --git a/docs/how-tos/my_airflow/optimize-dag-parsing.md b/docs/how-tos/my_airflow/optimize-dag-parsing.md new file mode 100644 index 0000000..5984d0e --- /dev/null +++ b/docs/how-tos/my_airflow/optimize-dag-parsing.md @@ -0,0 +1,61 @@ +--- +title: "Optimize and Debug DAG Parsing in My Airflow" +sidebar_label: "Optimize DAG Parsing" +description: "Use the datacoves my parse-logs command to find slow-parsing DAGs and diagnose import errors in your personal My Airflow instance in Datacoves." +sidebar_position: 79 +--- +# Optimize and debug DAG parsing + +Every DAG file is repeatedly read and executed by Airflow's dag-processor so it can build the DAGs it schedules. Slow or failing parsing shows up as DAGs that take a long time to appear, DAGs that never appear, or import errors in the UI. The `datacoves my parse-logs` command surfaces this information directly in your terminal so you can act on it while you develop. + +:::note +My Airflow [must be instantiated](/docs/how-tos/my_airflow/start-my-airflow) for this command to work. +::: + +## Find slow-parsing DAGs + +Run the command with no arguments to get a report of how long each DAG file takes to parse, sorted slowest first: + +```bash +datacoves my parse-logs +``` + +```text + My Airflow DAG parsing times (slowest first) + File Parse time DAGs Tasks + dbt_dag.py 0.794s 1 1 + daily_loan_run.py 0.400s 1 7 + other_examples/variable_via_secrets_... 0.250s 0 0 + notifications_examples/yaml_slack_dag.py 0.029s 1 1 + ... +``` + +The files at the top of the list are the ones worth optimizing. Common causes of slow parsing are heavy top-level imports, network or database calls made at parse time (rather than inside a task), and expensive module-level code. Move that work inside your tasks so it runs only when the task executes, not every time the file is parsed. + +If you prefer the output exactly as Airflow produces it, add `--raw`: + +```bash +datacoves my parse-logs --raw +``` + +## Debug a DAG that fails to parse or does not appear + +When a DAG is missing from the UI or shows an import error, look at the parse log for that specific file. Pass `--file` with the DAG file name (a partial name is fine): + +```bash +datacoves my parse-logs --file my_dag.py +``` + +The output shows the import errors, deprecation warnings, and other messages the dag-processor emitted while parsing that file, which usually points straight at the problem. + +To watch the log update live while you edit and save the file, add `--follow` (`-f`): + +```bash +datacoves my parse-logs --file my_dag.py --follow +``` + +Use `--lines` (`-n`) to control how many lines are shown (the default is 100). + +## Reference + +See [Datacoves CLI Commands](/docs/reference/airflow/datacoves-commands#datacoves-my-parse-logs) for the full list of options. From 00cc41ba80c27f5ad155ec0c4c8547799b8257d8 Mon Sep 17 00:00:00 2001 From: Alejandro Morera Date: Fri, 24 Jul 2026 16:44:37 -0600 Subject: [PATCH 5/5] Document datacoves my db-clean command --- docs/reference/airflow/datacoves-commands.md | 21 +++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/docs/reference/airflow/datacoves-commands.md b/docs/reference/airflow/datacoves-commands.md index 9250bc7..f3e6620 100644 --- a/docs/reference/airflow/datacoves-commands.md +++ b/docs/reference/airflow/datacoves-commands.md @@ -1,7 +1,7 @@ --- title: "Datacoves CLI Commands Reference: datacoves my" sidebar_label: CLI Commands -description: "Reference for Datacoves CLI commands: datacoves my import, my pytest, my api-key, and my parse-logs for managing My Airflow instances from the VS Code terminal." +description: "Reference for Datacoves CLI commands: datacoves my import, my pytest, my db-clean, my api-key, and my parse-logs for managing My Airflow instances from the VS Code terminal." sidebar_position: 126 --- # Datacoves CLI Commands @@ -16,6 +16,7 @@ The `my` subcommand executes commands for My Airflow. Currently, the `datacoves my` subcommand has the following subcommands: - `my import` - `my pytest` +- `my db-clean` - `my api-key` - `my parse-logs` @@ -43,6 +44,24 @@ This command allows you to run pytest validations straight from the command line datacoves my pytest -- orchestrate/test_dags/validate_dags.py ``` +### datacoves my db-clean + +:::note +My Airflow [must be instantiated](/docs/how-tos/my_airflow/start-my-airflow) for this command to work. +::: + +My Airflow runs on a SQLite database that grows as DAG runs, task instances, and logs accumulate. This command prunes entries older than a given number of days (5 by default) to keep it small. + +```bash +datacoves my db-clean +``` + +Use `--days` to change how many days of history to keep: + +```bash +datacoves my db-clean --days 10 +``` + ### datacoves my api-key Manage My Airflow API keys from the command line. These keys allow you to access the My Airflow API programmatically.