Can we run a custom Airflow image on Astronomer?

Airflow
#1

Hi,

I wanted to know if we can run a custom Airflow image that is build by forking the following repo - GitHub - apache/airflow: Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Our use is that we want to use a modified version of Airflow to suit are custom needs. Let me know if this is possible.

Thanks

#2

Hi @tanuj-ahuja ! Thanks for reaching out. Whether you’re running Astronomer or running Airflow on your own, we strongly recommend Astronomer Certified, which is available as a Docker Image and Python package for Airflow versions 1.10.7 - 2.0.0. These images have been battle-tested by our QA team, are used by our customer base, and incorporate upstream bug and security fixes in subsequent patch versions that may not yet be released in OSS.

If you want to give it a try, install our open source CLI and run our latest 2.0 image by following this guide: https://www.astronomer.io/guides/get-started-airflow-2

If you’re curious on what’s built into the image, reference Astronomer’s ap-airflow repo. For AC 2.0, Dockerfile here.

Does this help at all? Can I ask what you’re looking to customize?

#3

Hi @paola ,

Yes, correct we shouldn’t run a custom image given the points to mentioned above. Thank you for the references.

Our use here is to support a custom attribute for task instance that is not present in task_intance table in postgres database of Airflow. In particular we want to expose -

debug_log_path: Path where debug logs are stored for a task_instance. These logs are provided by a third party tool (data-build-tool) that we are orchestrating using Airflow

Let me know if there is some kind of a plugin available to accomplish this.