Can I run both Python2 and Python3 DAGs on Astronomer?

Astronomer
1

We have some legacy Python2 DAGs that we’d like to run on Astronomer, in addition to our newer Python3 DAGS (yes I know we can convert them, we’ll get to it soon!)

Given that Astronomer supports Python3 by default, is it possible to run Python2 instead for our legacy DAGs for now?

2

Doing this with arbitrary python code is easy:

PythonVirtualEnvOperator
A solution that involves fewer moving parts - the PythonVirtualEnvOperator allows to run an arbitary python function (with some caveats).

Define a python callable:

def test_func_two():
    import sys
    print(sys.version)

And then call the PythonVirtualEnvOperator:

t2 = PythonVirtualenvOperator(
    task_id='test_two',
    python_version='2',
    python_callable=test_func_two,
    dag=dag
)

The output confirms that the function is ran in python2:

b’2.7.14 (default, Dec 14 2017, 15:51:29) \n[GCC 6.4.0]\nhi\n
This same pattern can be used to call run any code that interacts with Snakebite or ApacheBeam.

Note that to get this to run, python2 needs to exist in the box/container the task is executing in. If you are using Astronomer Enterprise, you can add the following to your project’s Dockerfile:

RUN apk add py-virtualenv

(or add py-virtualenv to packages.txt)

It gets a bit tougher when there are a bunch of legacy util packages and such that need to run in python2, since that involves changing packages, config, etc.