What is the new airflow_settings.yaml file for?

edbizarro · March 8, 2019, 10:12pm

I see this Skipping Settings Creation: airflow_settings.yaml not found... message on astro airflow start and I now I’m curious about this file.

It’s something for overriding the airflow.cfg?

Thanks

paola · March 12, 2019, 3:01pm

@edbizarro The airflow_settings.yaml feature is an experimental feature available in Astronomer Cloud v0.8 and CLI v0.7.5-2.

Check out this page on our CLI repo for a breakdown of that file and how you’ll soon be able to use it to generate Connections, Pools, and Variables via astro airflow start directly from our CLI.

Note that it’s unrelated to airflow.cfg - it’ll get injected as connection/pools/variables that are stored in the database (e.g. the Variables you have access to in the Airflow UI).

github.com

astronomer/docs/blob/master/v0.9/cli-airflow-configuration.md

---
title: "Adding Airflow Configuration with Astro CLI"
description: "Set up connections, variables, and pools automatically with the Astronomer CLI."
link: "Adding Airflow Configuration"
date: 2019-01-31T00:00:00.000Z
slug: "cli-airflow-configuration"
---

One of the most useful features of the Astro CLI during development is the ability to have connections, variables, and pools automatically generated on `astro airflow start`. Using this feature, you can kill your environment and automatically start back up with all of your necessary configuration to start again.

**NOTE**: Connections, variables, and pools defined through this process will only be available locally. To ensure they are available in your remote deployments, please add them via the Airflow UI.

## airflow_settings.yaml
When you first `astro airflow init` to create a new project, a file titled `airflow_settings.yaml` will be created to add connections, pools, and variables. By default, the structure within this file will look like the following:

```
airflow:
  connections:
    - conn_id: my_new_connection
      conn_type: postgres

This file has been truncated. show original

edbizarro · March 12, 2019, 3:42pm

Thanks @paola for the explanation, that’s very useful!

btallman · April 18, 2019, 1:08pm

I am trying to use this file (locally on the latest Alpha) and if the file isn’t there, I see the skipping message, but when the file IS there, it doesn’t load my variables…

paola · April 18, 2019, 3:00pm

Hi @btallman! We just uncovered a bug that might have something to do with what you’re seeing. Can you make sure the name of your config.yaml project does not have any spaces or symbols (dashes, underscores, etc)? E.g. if your project name was airflow-home, changing it to airflowhome should do the trick.

Note: Starting a new project by renaming it will create new docker volumes, so your previous connections, env vars, pools, and DAG/task history will be gone as soon as you run astro airflow start. Hopefully not a huge inconvenience if you’re just testing locally.

Let us know if you see this issue persist and we’ll dig in deeper, but do keep in mind that you should still be on v0.7.5 of our CLI for optimal performance as a current Cloud customer.

paola · May 7, 2019, 3:21pm

Update: We recently released a new version of our CLI for current Cloud customers on v0.7 that backports the airflow_settings.yaml feature

If you’d like access to this feature (and the .env feature) on v0.7, run the following:

curl -sSL https://install.astronomer.io | sudo bash -s -- v0.7.5-2

joshuamoore-procore · May 13, 2019, 11:27pm

Hi @paola Will this feature work with astro airflow deploy as well? Such as if we wanted to create a new Astronomer deployment, and wanted to use this to seed it with all the Connections & Variables?

paola · May 14, 2019, 3:17pm

Hi @joshuamoore-procore - the airflow_settings.yaml feature is currently only for local development, not for when you’re pushing up to an Astro deployment with astro airflow deploy. Certainly something we’re looking to incorporate into our platform in the future.

In the meantime, you could always incorporate a CI process and use the API to programmatically update Airflow Variables. We don’t have a step-by-step guide for that just yet, but it should be out soon (GitHub issue here).

paola · August 22, 2019, 7:59pm

Update -

In an effort to make it easier for Astronomer users to use Airflow, we’re working to address a few pain points associated with creating new Connections/Variables/Pools in Airflow, both locally and on Astronomer.

1. Copying Connections/Variables/Pools between Deployments

To address the pain point of having to manually copy over Connections, Variables, and Pools on the Airflow UI from one deployment to another for those migrating between deployments, we have a feature on our roadmap that will allow users to copy those components between Airflow deployments on Astronomer Cloud + Enterprise.

GitHub issue here for reference: https://github.com/astronomer/astronomer/issues/174

2. Injecting Connections/Variables/Pools locally up to a deployment on Astronomer Cloud

The functionality that airflow_settings.yaml gives you is currently only available when you’re developing locally.

While it sounds ideal to be able to do that on an Astronomer deployment, we’re looking to build a more secure and robust way to sync those values in a centralized way among users. Once we scope out a solution, we’ll link an actionable GitHub issue in here for everyone to follow.

AndrewHarmon · August 22, 2019, 9:27pm

Just chiming in on why this doesn’t work for deployments. Your airflow_settings.yaml file usually contains sensitive credentials you don’t want pushed to github, so it should be in your .gitignore file. This means when your colleague does a git pull they won’t have all those settings defined. Then when they do a astro airflow deploy, what happens? Do your settings get changed to what they put in their airflow_settings.yaml file? How do you keep these files in sync across your whole team? So this is why it is designed only for local development. We are brainstorming on some ideas to make adding these settings to your deployments easier. Would love to hear thoughts from the community. I for one would love a integration with Vault or AWS Secrets Manager.

rcleary · October 1, 2019, 7:13pm

Just one thought, would it work to just point to environment variables in the Astronomer cloud?

paola · October 4, 2019, 1:21am

@rcleary The connections and variables you’d want to configure here are ones within Airflow that you can reference in your tasks, not more global deployment-level env vars that you’d set on Astronomer.

Can you describe what you mean in a bit more detail? FWIW, the .env file in your directory should allow for parallel functionality locally with environment variables!

andrewcstewart · October 7, 2019, 8:38pm

Quick question on this topic: If my astro project predates airflow_settings.yaml creation, am I safe just creating my own blank one? Or am I safe just doing astro airflow init ? I don’t want to clobber my current environment.

AndrewHarmon · October 8, 2019, 5:22pm

Both are safe. airflow init will not overwrite any dags, plugins, dockerfiles or anything like that. But it will create missing files, like the example dag. Creating an empty airflow_settings.yaml would work with the least amount of changes to your directory

MaxKrog · April 15, 2020, 12:45pm

How do i go about adding connections that have special fields?

For example connections of typ “google_cloud_platform” have a field named “Keyfile JSON”

robmarkcole · September 14, 2020, 2:25pm

Hi @paola I have added some credentials to airflow_settings.yaml then performed astro airflow kill and astro airflow start but the new credentials are not populated in Airflow. Are there additional steps?

A second related question, are there example connections? I cannot find these documented other than your example above.
Many thanks

paola · September 15, 2020, 7:42pm

Hi @robmarkcole! What version of the Astronomer CLI are you running? Can you try $ astro dev stop and then $ astro dev start? We’ve deprecated $ astro airflow start and $ astro airflow kill - I’m not sure if that’d be the problem, but let’s give that a try.

In terms of an example connection, the only one we have is defined in this doc: https://www.astronomer.io/docs/cloud/stable/develop/customize-image/. I can see if I can gather another from our team! Anything in particular you’re having trouble with or would like to see?

robmarkcole · September 16, 2020, 8:43am

Hi @paola
I have Astro CLI Version: 0.10.3

I inherited a bunch of bash scripts which include one to run:

set -e

astro airflow stop
line_count=$(lsof -i :5432 |  grep postgresql | wc -l )
if [ $line_count -gt 0 ]; then echo "Postgres is running, try\n\nbrew services stop postgresql"; exit; fi

astro airflow start

If you can advise where deprecation is publicised I can begin on an issue for our team?

Re example connections I guessed snowflake (below). For aws would it be aws or something else?

  connections:
    - conn_id: snowflake
      conn_type: snowflake
      conn_login: you@bla.com
      conn_password: yours
      conn_extra: {"account":"you",  "database":"bla",,,}

robmarkcole · September 17, 2020, 9:24am

Appears I am unable to edit the previous post, but for the conn_extra it is necessary to add ’ around the json, so conn_extra: '{"account":"you", "database":"bla",,,}'

saurabhmishra · April 12, 2021, 2:39am

Sorry to be jumping on this old thread, but recently our organization was migrating to astronomer in GCP and am facing similar issue with loading of airflow_settings.yaml file.

have referred the variable definition syntax from your official guide - Astro Documentation | Astronomer Documentation
Also, am trying to use the configuration in local mode only
Commands executed - astro dev start

I am unable to see this file being loaded automatically in my docker images anywhere in the logs. Also, which i check inside the docker containers, this file has not been loaded

local folder :

Dockerfile airflow_settings.yaml dags include packages.txt plugins requirements.txt

scheduler container :

Dockerfile airflow.cfg dags include logs packages.txt plugins requirements.txt unittests.cfg

Do i need to explicitly load this file at the time of astro start ?

Astro CLI Version: 0.23.3

Topic		Replies	Views
How do I set astronomer config file options (Env Vars)? Airflow	2	7031	January 7, 2020
Can I programmatically add/manage Airflow Connections in code? Airflow	0	3625	August 21, 2020
Updating variables on remote deployments Astronomer Nebula	6	5911	May 11, 2020
Can I use my own airflow.cfg instead of Env Vars in the Astro UI? Astro CLI	10	7630	December 18, 2023
Unable to set-up an external database backend for Airflow Astro CLI	2	2440	October 31, 2022

What is the new airflow_settings.yaml file for?

Update -

1. Copying Connections/Variables/Pools between Deployments

2. Injecting Connections/Variables/Pools locally up to a deployment on Astronomer Cloud

Related Topics