In my use case I need to setup configuration files on the worker nodes for my DAGs to function properly. I am using Astronomer Cloud, and as far as I understand the platform, I have currently two options to setup those
- Generate the files before or during the image build process in my CI/CD pipeline and COPY them into the image
- Create a specific airflow DAG that I can run on demand and that fetches the config and sets up the files in the container.
Putting the config files, with potential secrets on them, in the image does not feel very safe. On the other hand having a full DAG to do the config does not work well neither, as I will have to remember to run it every time I re-deploy. Some other, more convenient approaches that I do not think are possible are
- Bootstraping the entrypoint to fetch and setup the configuration on start-up
- SSH to the container and put the files there after deployment
What are the alternatives here?