What is the new airflow_settings.yaml file for?

I see this Skipping Settings Creation: airflow_settings.yaml not found... message on astro airflow start and I now I’m curious about this file.

It’s something for overriding the airflow.cfg?

Thanks

@edbizarro The airflow_settings.yaml feature is an experimental feature available in Astronomer Cloud v0.8 and CLI v0.7.5-2.

Check out this page on our CLI repo for a breakdown of that file and how you’ll soon be able to use it to generate Connections, Pools, and Variables via astro airflow start directly from our CLI.

Note that it’s unrelated to airflow.cfg - it’ll get injected as connection/pools/variables that are stored in the database (e.g. the Variables you have access to in the Airflow UI).

2 Likes

Thanks @paola for the explanation, that’s very useful!

1 Like

I am trying to use this file (locally on the latest Alpha) and if the file isn’t there, I see the skipping message, but when the file IS there, it doesn’t load my variables…

Hi @btallman! We just uncovered a bug that might have something to do with what you’re seeing. Can you make sure the name of your config.yaml project does not have any spaces or symbols (dashes, underscores, etc)? E.g. if your project name was airflow-home, changing it to airflowhome should do the trick.

Note: Starting a new project by renaming it will create new docker volumes, so your previous connections, env vars, pools, and DAG/task history will be gone as soon as you run astro airflow start. Hopefully not a huge inconvenience if you’re just testing locally.

Let us know if you see this issue persist and we’ll dig in deeper, but do keep in mind that you should still be on v0.7.5 of our CLI for optimal performance as a current Cloud customer.

Update: We recently released a new version of our CLI for current Cloud customers on v0.7 that backports the airflow_settings.yaml feature :smiley:

If you’d like access to this feature (and the .env feature) on v0.7, run the following:

curl -sSL https://install.astronomer.io | sudo bash -s -- v0.7.5-2

Hi @paola Will this feature work with astro airflow deploy as well? Such as if we wanted to create a new Astronomer deployment, and wanted to use this to seed it with all the Connections & Variables?

Hi @joshuamoore-procore - the airflow_settings.yaml feature is currently only for local development, not for when you’re pushing up to an Astro deployment with astro airflow deploy. Certainly something we’re looking to incorporate into our platform in the future.

In the meantime, you could always incorporate a CI process and use the API to programmatically update Airflow Variables. We don’t have a step-by-step guide for that just yet, but it should be out soon (GitHub issue here).

1 Like