How do I run Airflow 1.10 on Astronomer v0.7?


#1

I’m an Astronomer customer, and would love access to Airflow 1.10 features on Astronomer v0.7.

What do I need to do?


When will Astronomer have Airflow 1.10?
Issue installing the Databricks Operator
Delete old dags
Delete old dags
#2

Yes, using airflow 1.10.1 is a simple change.

  1. Upgrade the CLI to the latest version (astro upgrade).

  2. Change the FROM link in your Dockerfile to:

FROM astronomerinc/ap-airflow:0.7.5-1.10.1-onbuild

This will pull down our 1.10.1 image instead of the 1.9 image.
3) Confirm this works by running astro airflow start and you should see the 1.10.1 UI at localhost:8080/admin.

Notice that there’s the little Red X to delete DAGs from the UI, confirming that you are running Airflow 1.10.1 (or go to About -> Version

Note: Astronomer’s 1.10.1 image does not support the Airflow RBAC on v0.7 - this will be a part of our 0.9 release.

Note: The 1.10.1 webserver is more CPU intensive than the 1.9 version - you might need to scale up the webserver a bit.


#4

I like the ability to clear out old DAGs from the UI, that’s handy. Are there other improvements I should upgrade for? Anything that 1.9 does badly that 1.10 will improve?

The changelog lists new support for SageMaker and IMAP, both of which are interesting and I can see being useful for me in the future, but not immediately necessary.

Thanks for this topic!


#5

Hey @mplovepop!

This is a really good writeup of some of the changes:

The biggest takeaways are additions of AWS hooks/operators and adding a few docs.

Please note that we don’t support the K8s executor or the RBAC UI - we’ll be adding those shortly.

Also, we’ve found that the new webserver is a little more CPU greedy than the 1.9 webserver - if you are seeing the UI lag a bit, I’d recommend giving it some more AUs (Astronomer units) on that sweet new slider in the Configure tab :slight_smile: