Generally speaking, not wanting to rely on manually creating Airflow Connections from the Airflow UI every time you spin up a new Airflow Deployment is a very fair and common question/pain point.
If you’re running both Airflow and Astronomer, there are a few answers to this question:
Locally
Locally, we recommend leveraging Astronomer’s airflow_settings.yaml. It’s a yaml file that’s automatically generated when you initialize your Airflow project on Astronomer and a place where you can specify Airflow Connections/Variables/Pools. They’ll get applied every time you run $ astro dev start without requiring you to touch the Airflow UI and re-enter them every time you spin up a local Airflow instance on your machine. We’d recommend adding this to your .gitignore, assuming there are sensitive creds in there.
More information on this here.
On Astronomer
On Astronomer, you won’t be able to use the airflow_settings.yaml file for security reasons but you have two options:
-
Incorporate a secret backend (e.g. AWS SSM Parameter Store, Hashicorp Vault) to manage connections (and other secrets) programmatically across the board
-
Add Airflow Connections as Env Vars.
By adding Connections and Variables as Environment Variables, you can refer to them more easily in your code and lower the amount of open connections to Airflow’s Metadata Database, preventing a strain on your Database and resources. You can add Environment Variables via the Astro UI or via your Dockerfile, but we recommend doing so via the Astro UI to prevent exposing sensitive credentials in plaintext. Adding Connections as Airflow Environment Variables still means you have to “manage” connections and manually create them for every Airflow Deployment, but it’s easier to do it this way relative to working exclusively within the the Airflow UI (and generally better practice).
Here are some additional docs on the above that can help:
If you have additional questions or feedback, the Astronomer Support Portal (support.astronomer.io) is a great resource.