What is the new airflow_settings.yaml file for?

I see this Skipping Settings Creation: airflow_settings.yaml not found... message on astro airflow start and I now I’m curious about this file.

It’s something for overriding the airflow.cfg?

Thanks

@edbizarro The airflow_settings.yaml feature is an experimental feature available in Astronomer Cloud v0.8 and CLI v0.7.5-2.

Check out this page on our CLI repo for a breakdown of that file and how you’ll soon be able to use it to generate Connections, Pools, and Variables via astro airflow start directly from our CLI.

Note that it’s unrelated to airflow.cfg - it’ll get injected as connection/pools/variables that are stored in the database (e.g. the Variables you have access to in the Airflow UI).

2 Likes

Thanks @paola for the explanation, that’s very useful!

1 Like

I am trying to use this file (locally on the latest Alpha) and if the file isn’t there, I see the skipping message, but when the file IS there, it doesn’t load my variables…

Hi @btallman! We just uncovered a bug that might have something to do with what you’re seeing. Can you make sure the name of your config.yaml project does not have any spaces or symbols (dashes, underscores, etc)? E.g. if your project name was airflow-home, changing it to airflowhome should do the trick.

Note: Starting a new project by renaming it will create new docker volumes, so your previous connections, env vars, pools, and DAG/task history will be gone as soon as you run astro airflow start. Hopefully not a huge inconvenience if you’re just testing locally.

Let us know if you see this issue persist and we’ll dig in deeper, but do keep in mind that you should still be on v0.7.5 of our CLI for optimal performance as a current Cloud customer.

Update: We recently released a new version of our CLI for current Cloud customers on v0.7 that backports the airflow_settings.yaml feature :smiley:

If you’d like access to this feature (and the .env feature) on v0.7, run the following:

curl -sSL https://install.astronomer.io | sudo bash -s -- v0.7.5-2

Hi @paola Will this feature work with astro airflow deploy as well? Such as if we wanted to create a new Astronomer deployment, and wanted to use this to seed it with all the Connections & Variables?

Hi @joshuamoore-procore - the airflow_settings.yaml feature is currently only for local development, not for when you’re pushing up to an Astro deployment with astro airflow deploy. Certainly something we’re looking to incorporate into our platform in the future.

In the meantime, you could always incorporate a CI process and use the API to programmatically update Airflow Variables. We don’t have a step-by-step guide for that just yet, but it should be out soon (GitHub issue here).

1 Like

Update -

In an effort to make it easier for Astronomer users to use Airflow, we’re working to address a few pain points associated with creating new Connections/Variables/Pools in Airflow, both locally and on Astronomer.

1. Copying Connections/Variables/Pools between Deployments

To address the pain point of having to manually copy over Connections, Variables, and Pools on the Airflow UI from one deployment to another for those migrating between deployments, we have a feature on our roadmap that will allow users to copy those components between Airflow deployments on Astronomer Cloud + Enterprise.

GitHub issue here for reference: https://github.com/astronomer/astronomer/issues/174

2. Injecting Connections/Variables/Pools locally up to a deployment on Astronomer Cloud

The functionality that airflow_settings.yaml gives you is currently only available when you’re developing locally.

While it sounds ideal to be able to do that on an Astronomer deployment, we’re looking to build a more secure and robust way to sync those values in a centralized way among users. Once we scope out a solution, we’ll link an actionable GitHub issue in here for everyone to follow.

Just chiming in on why this doesn’t work for deployments. Your airflow_settings.yaml file usually contains sensitive credentials you don’t want pushed to github, so it should be in your .gitignore file. This means when your colleague does a git pull they won’t have all those settings defined. Then when they do a astro airflow deploy, what happens? Do your settings get changed to what they put in their airflow_settings.yaml file? How do you keep these files in sync across your whole team? So this is why it is designed only for local development. We are brainstorming on some ideas to make adding these settings to your deployments easier. Would love to hear thoughts from the community. I for one would love a integration with Vault or AWS Secrets Manager.

1 Like

Just one thought, would it work to just point to environment variables in the Astronomer cloud?

@rcleary The connections and variables you’d want to configure here are ones within Airflow that you can reference in your tasks, not more global deployment-level env vars that you’d set on Astronomer.

Can you describe what you mean in a bit more detail? FWIW, the .env file in your directory should allow for parallel functionality locally with environment variables!

Quick question on this topic: If my astro project predates airflow_settings.yaml creation, am I safe just creating my own blank one? Or am I safe just doing astro airflow init ? I don’t want to clobber my current environment.

Both are safe. airflow init will not overwrite any dags, plugins, dockerfiles or anything like that. But it will create missing files, like the example dag. Creating an empty airflow_settings.yaml would work with the least amount of changes to your directory

1 Like

How do i go about adding connections that have special fields?

For example connections of typ “google_cloud_platform” have a field named “Keyfile JSON”

Hi @paola I have added some credentials to airflow_settings.yaml then performed astro airflow kill and astro airflow start but the new credentials are not populated in Airflow. Are there additional steps?

A second related question, are there example connections? I cannot find these documented other than your example above.
Many thanks

Hi @robmarkcole! What version of the Astronomer CLI are you running? Can you try $ astro dev stop and then $ astro dev start? We’ve deprecated $ astro airflow start and $ astro airflow kill - I’m not sure if that’d be the problem, but let’s give that a try.

In terms of an example connection, the only one we have is defined in this doc: https://www.astronomer.io/docs/cloud/stable/develop/customize-image/. I can see if I can gather another from our team! Anything in particular you’re having trouble with or would like to see?

Hi @paola
I have Astro CLI Version: 0.10.3

I inherited a bunch of bash scripts which include one to run:

set -e

astro airflow stop
line_count=$(lsof -i :5432 |  grep postgresql | wc -l )
if [ $line_count -gt 0 ]; then echo "Postgres is running, try\n\nbrew services stop postgresql"; exit; fi

astro airflow start

If you can advise where deprecation is publicised I can begin on an issue for our team?

Re example connections I guessed snowflake (below). For aws would it be aws or something else?

  connections:
    - conn_id: snowflake
      conn_type: snowflake
      conn_login: you@bla.com
      conn_password: yours
      conn_extra: {"account":"you",  "database":"bla",,,} 

Appears I am unable to edit the previous post, but for the conn_extra it is necessary to add ’ around the json, so conn_extra: '{"account":"you", "database":"bla",,,}'

Sorry to be jumping on this old thread, but recently our organization was migrating to astronomer in GCP and am facing similar issue with loading of airflow_settings.yaml file.

I am unable to see this file being loaded automatically in my docker images anywhere in the logs. Also, which i check inside the docker containers, this file has not been loaded

local folder :

Dockerfile airflow_settings.yaml dags include packages.txt plugins requirements.txt

scheduler container :

Dockerfile airflow.cfg dags include logs packages.txt plugins requirements.txt unittests.cfg

Do i need to explicitly load this file at the time of astro start ?

Astro CLI Version: 0.23.3