Can I use my own airflow.cfg instead of Env Vars in the Astro UI?

Astro CLI
1

As of Astronomer v0.9, users cannot leverage their own airflow.cfg. To add Environment Variables to a deployment on Astronomer, you can either:

  1. Add them via the Astronomer UI (Deployment > Configure > Env Vars)
  2. Add them to your Dockerfile (a file programmatically generated when you initialize a project on Astronomer via the CLI)

For more details on how to do the above (or leverage the -env flag locally), check out this forum post: How do I set astronomer config file options (Env Vars)?

Note: Prior to Astronomer v0.9, the answer would have been yes, you can. Historically, users had been able to add airflow.cfg to the main directory of their project and it would be included when the image built via astro airflow start or astro airflow deploy.

1 Like
2

If I use my own airflow.cfg with astro airflow deploy - does it over ride the env set in the docker image? Same for the env defined in the UI?

3

if you set airflow settings as environment variables in either the dockerfile or the orbit ui, those will take precedent over setting ins airflow.cfg

1 Like
4

@paola Its not working for me :frowning:

$ tree -a
.
├── airflow.cfg          <======== here is my airflow.cfg
├── airflow_settings.yaml
├── .astro
│   └── config.yaml
├── dags
│   └── example-dag.py
├── Dockerfile
├── .dockerignore
├── .env
├── .gitignore
├── include
├── packages.txt
├── plugins
│   └── example-plugin.py
└── requirements.txt

Here are the contents of my airflow.cfg. My smtp relay doesn’t require credentials:

$ cat airflow.cfg 
[email]
email_backend = airflow.utils.email.send_email_smtp

[smtp]
smtp_host = smtp-relay.example.com
smtp_starttls = False
smtp_ssl = False
smtp_port = 25
smtp_mail_from = noreply@example.com

I deployed to my workspace via astro deploy -f and my custom airflow.cfg settings don’t show up on the configuration section of Airflow:

Screen Shot 2019-12-11 at 18.08.26

What’s the trick to make this work?

5

I ended up deleting the airflow.cfg file and setting the ENV vars on the Dockerfile like so:

FROM    astronomerinc/ap-airflow:0.10.3-1.10.5-onbuild AS base
  
ENV     AIRFLOW__SMTP__SMTP_HOST="smtp-relay.example.com" \
        AIRFLOW__SMTP__SMTP_PORT="25" \
        AIRFLOW__SMTP__SMTP_SSL="false" \
        AIRFLOW__SMTP__SMTP_STARTTLS="false" \
        AIRFLOW__SMTP__SMTP_MAIL_FROM="noreply@example.com"

And now they show up properly on the Airflow config section:

Screen Shot 2019-12-12 at 11.40.27

My SMTP relay for the PoC doesn’t require SSL/TLS or authentication though.

1 Like
6

Hi @tonejito! I just updated the original answer here, but I chatted with my team and discovered that if you’re running Astronomer v0.9 or later, you actually won’t be able to leverage your own airflow.cfg.

Happy to hear you were able to add them via your Dockerfile! You’ll always be able to either do that or inject them directly via the Astronomer UI (on your Deployment’s “Configure” page). Let us know if there’s anything else we can do to help!

7

Hi @paola:

I tried this tactic of using Dockerfile env to enable DebugExecutor mode (I would like to be able to run a dag in my IDE’s debugger as is detailed here: https://airflow.apache.org/docs/stable/executor/debug.html).

Using this in my Dockerfile doesn’t seem to override the LocalExecutor though :frowning: :

# For debugging in one's IDE
ENV AIRFLOW__CORE__EXECUTOR="DebugExecutor"

After inspecting the environment of running container (via astro dev start):

$ docker exec -it dataairflowdags8cfa94_scheduler_1 bash
astro@879253d9d962:/usr/local/airflow$ env | grep AIRFLOW__CORE__EXECUTOR
AIRFLOW__CORE__EXECUTOR=LocalExecutor
astro@879253d9d962:/usr/local/airflow$

I’m guessing Astronomer doesn’t support ability to do this and forces LocalExecutor mode for local runs of astro dev start? Is it possible to also allow for DebugExecutor as it would greatly help those who’d like to debug dags from their IDE (e.g., PyCharm)?

JD

8

Isn’t the .env file exactly for that? I added my env’s in this file, hit astro dev restart and it worked

# inside .env in the root directory
AIRFLOW__CORE__TEST_CONNECTION=Enabled

I see problems with the other approaches:

  1. Add them via the Astronomer UI (Deployment > Configure > Env Vars)
    → Does not work in local def mode
  2. Add them to your Dockerfile (a file programmatically generated when you initialize a project on Astronomer via the CLI)
    → The container needs to be rebuild every time you change a env, which takes time

They also do it like this in the docu, e.g. in xcom-backend-tutorial

However, the possibility to use airflow.cfg would be great!

9

Hey @turbotimon

This is a pretty old post and Astronomer has come a long way in how we manage Env variables. I would suggest reading Manage Environment Variables in Astro.

Please feel free to ask a new question on this topic. I will close this topic as it is stale.

Thanks
Manmeet