What is the new airflow_settings.yaml file for?

I see this Skipping Settings Creation: airflow_settings.yaml not found... message on astro airflow start and I now I’m curious about this file.

It’s something for overriding the airflow.cfg?

Thanks

@edbizarro The airflow_settings.yaml feature is an experimental feature available in Astronomer Cloud v0.8 and CLI v0.7.5-2.

Check out this page on our CLI repo for a breakdown of that file and how you’ll soon be able to use it to generate Connections, Pools, and Variables via astro airflow start directly from our CLI.

Note that it’s unrelated to airflow.cfg - it’ll get injected as connection/pools/variables that are stored in the database (e.g. the Variables you have access to in the Airflow UI).

2 Likes

Thanks @paola for the explanation, that’s very useful!

1 Like

I am trying to use this file (locally on the latest Alpha) and if the file isn’t there, I see the skipping message, but when the file IS there, it doesn’t load my variables…

Hi @btallman! We just uncovered a bug that might have something to do with what you’re seeing. Can you make sure the name of your config.yaml project does not have any spaces or symbols (dashes, underscores, etc)? E.g. if your project name was airflow-home, changing it to airflowhome should do the trick.

Note: Starting a new project by renaming it will create new docker volumes, so your previous connections, env vars, pools, and DAG/task history will be gone as soon as you run astro airflow start. Hopefully not a huge inconvenience if you’re just testing locally.

Let us know if you see this issue persist and we’ll dig in deeper, but do keep in mind that you should still be on v0.7.5 of our CLI for optimal performance as a current Cloud customer.

Update: We recently released a new version of our CLI for current Cloud customers on v0.7 that backports the airflow_settings.yaml feature :smiley:

If you’d like access to this feature (and the .env feature) on v0.7, run the following:

curl -sSL https://install.astronomer.io | sudo bash -s -- v0.7.5-2

Hi @paola Will this feature work with astro airflow deploy as well? Such as if we wanted to create a new Astronomer deployment, and wanted to use this to seed it with all the Connections & Variables?

Hi @joshuamoore-procore - the airflow_settings.yaml feature is currently only for local development, not for when you’re pushing up to an Astro deployment with astro airflow deploy. Certainly something we’re looking to incorporate into our platform in the future.

In the meantime, you could always incorporate a CI process and use the API to programmatically update Airflow Variables. We don’t have a step-by-step guide for that just yet, but it should be out soon (GitHub issue here).

1 Like

Update -

In an effort to make it easier for Astronomer users to use Airflow, we’re working to address a few pain points associated with creating new Connections/Variables/Pools in Airflow, both locally and on Astronomer.

1. Copying Connections/Variables/Pools between Deployments

To address the pain point of having to manually copy over Connections, Variables, and Pools on the Airflow UI from one deployment to another for those migrating between deployments, we have a feature on our roadmap that will allow users to copy those components between Airflow deployments on Astronomer Cloud + Enterprise.

GitHub issue here for reference: https://github.com/astronomer/astronomer/issues/174

2. Injecting Connections/Variables/Pools locally up to a deployment on Astronomer Cloud

The functionality that airflow_settings.yaml gives you is currently only available when you’re developing locally.

While it sounds ideal to be able to do that on an Astronomer deployment, we’re looking to build a more secure and robust way to sync those values in a centralized way among users. Once we scope out a solution, we’ll link an actionable GitHub issue in here for everyone to follow.

Just chiming in on why this doesn’t work for deployments. Your airflow_settings.yaml file usually contains sensitive credentials you don’t want pushed to github, so it should be in your .gitignore file. This means when your colleague does a git pull they won’t have all those settings defined. Then when they do a astro airflow deploy, what happens? Do your settings get changed to what they put in their airflow_settings.yaml file? How do you keep these files in sync across your whole team? So this is why it is designed only for local development. We are brainstorming on some ideas to make adding these settings to your deployments easier. Would love to hear thoughts from the community. I for one would love a integration with Vault or AWS Secrets Manager.

1 Like