Hi! I have the following setup:
astronomer-airflow/
|-- dags/
|-- my-python-library
|-- Dockerfile
...
After I run astro airflow start, any changes I make to my-python-library are not reflected live, but rather requires me to restart everything for them to be updated. How can I make changes to the codebase update live?
Hi @miroli! Happy to help here.
So, once you’ve initialized a project via the Astronomer CLI, you’ll effectively have to run astro airflow start to start up a local instance of Airflow on your machine.
To make changes once you have Airflow up and running:
If you’re making changes to any of the following, you have to run astro airflow stop and then astro airflow start to re-build your image:
If you make changes to your CODE (dags, include, plugins), you can:
Does that answer your question?
Thank you for the quick reply Paola.
This gives some clarification, but my case does not really fall into either of these categories. The code I want to be able to “update live” rather lives in a directory/package outside of the dags directory. This code is then imported by the dags.
But perhaps that directory structured is discouraged?
Hi @miroli. So, only the 3 directories listed above are “watched” by astro airflow start. You can check out the source code here: https://github.com/astronomer/astro-cli/blob/master/airflow/include/composeyml.go#L54
What’s in your my-python-library folder? I’d recommend you throw any python packages in requirements.txt, create a sub-folder within your dags directory if you want, throw plugins into plugins and anything else you want to call from Airflow into include.
On Astronomer v0.9+, you’ll be able to overwrite files + specify their own volume mounts. If you’re on Astronomer Cloud at the moment (v0.7.5) you won’t be able to do so just yet - but soon!
Any update on how this can be achieved? I think the cli is on v0.10+ so this should be available. I couldnt find docs on how to do this.