Commit 67c8d78
fix: Avoid heavy top-level imports in operators
For complex DAGs, the import time added by this library could generate
`DagBag` import timeouts if the configured value is low enough. An
Airflow documented best practice is to
[avoid heavy Python code](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code)
that runs on DAG and Operator creation, and `dbt` imports are slow,
based on profiling.
Profiling can be easily run locally, with the following command:
```shell
python -X importtime -c "from airflow_dbt_python.operators.dbt import DbtRunOperator" 2>import-times.log
```
And then parsed using a tool like
[tuna](https://github.com/nschloe/tuna).
Before this change, the operator import takes ~1.37s, which is reduced
to ~0.25s with this fix.
It's important to note that, from those 0.25s, more than 80% of the time
is spent importing `airflow` components, which will be commonly already
loaded in DAGs, so this library's import time for operators becomes
insignificant.1 parent 3844351 commit 67c8d78
1 file changed
+15
-4
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | 17 | | |
21 | 18 | | |
22 | 19 | | |
| |||
30 | 27 | | |
31 | 28 | | |
32 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
| |||
107 | 110 | | |
108 | 111 | | |
109 | 112 | | |
| 113 | + | |
| 114 | + | |
110 | 115 | | |
111 | 116 | | |
112 | 117 | | |
| |||
314 | 319 | | |
315 | 320 | | |
316 | 321 | | |
| 322 | + | |
| 323 | + | |
317 | 324 | | |
318 | 325 | | |
319 | 326 | | |
| |||
611 | 618 | | |
612 | 619 | | |
613 | 620 | | |
| 621 | + | |
| 622 | + | |
614 | 623 | | |
615 | 624 | | |
616 | 625 | | |
| |||
757 | 766 | | |
758 | 767 | | |
759 | 768 | | |
| 769 | + | |
| 770 | + | |
760 | 771 | | |
761 | 772 | | |
762 | 773 | | |
| |||
0 commit comments