I would like to use some of the existing Astronomer airflow plugins. What’s the best practice for adding these plugins into an astronomer project so that they’re included in the deployment?
Has anyone used third-party airflow plugins such as [https://github.com/airflow-plugins](https://github.com/airflow-plugins? If so how are you adding it to the airflow plugins directory and how are you pulling in changes if there are updates for the plugin?
The only solution that I can think of is creating a separate repo to manage all plugins then use git submodule to add third-party plugins.
Hi @minhvu! Thanks for reaching out here. The best way to add and import your plugins is by adding them to the plugins file within your Astronomer project, which was automatically generated when you initialized it (on astro dev init).
.
├── dags # Where your DAGs go
│ ├── example-dag.py # An example dag that comes with the initialized project
├── Dockerfile # For Astronomer's Docker image and runtime overrides
├── include # For any other files you'd like to include
├── plugins # For any custom or community Airflow plugins
├── operators
└── my_operator.py
├──airflow_settings.yaml #For your Airflow Connections, Variables and Pools (local only)
├──packages.txt # For OS-level packages
└── requirements.txt # For any Python packages
From here, any Python files in those folders should be accessible to import (e.g. from operators.my_operator import MyOperator).
A few pro-tips:
- Make sure you’re importing plugins from the GitHub repo branch that corresponds to the version of Airflow you’re running
- When the content of any plugins change in core Airflow, those changes go out in a stable Airflow branch (though the import path typically doesn’t change)
Here’s a guide that walks you through it: https://www.astronomer.io/guides/airflow-importing-custom-hooks-operators/
Does that answer your question, @minhvu?