How to manage utility code

Airflow
1

Hi,
I have some some python functions that are mainly there to build tasks in DAG. Is it possible to have these functions in a separate module and import them per dag file?

For example,

dag/
   dag1/
      dag1.py
      util/
         utilA.py
         utilB.py

Then I want to be able to execute these simple python methods from dag1.py by using regular python syntax

#dag1.py
import util.utilA

utilA.function1()

I’ve considered few options:

  1. operators/hooks: I don’t feel like these util functions should be built as operators/hooks since these are simple python functions.
  2. adding python module/package: if i understood it correctly, it affects all the dags. i want util module to be available to dag1.py only
2

Hi @danielyahn!

Absolutely you can store functions/modules outside of DAG files and honestly it’s very much encouraged! Ideally DAG files closely resemble configuration files and only contain what’s needed to define the DAG itself, the tasks within the DAG, and the task dependencies. All other relevant code should live outside of the DAG file itself. Just make sure you have __init__.py files in your util directories so they can be imported into the DAG as necessary. With astro dev init you can take advantage of the provided the include folder for this very use case too.