Doing this with arbitrary python code is easy:
A solution that involves fewer moving parts - the PythonVirtualEnvOperator allows to run an arbitary python function (with some caveats).
Define a python callable:
And then call the PythonVirtualEnvOperator:
t2 = PythonVirtualenvOperator(
The output confirms that the function is ran in python2:
b’2.7.14 (default, Dec 14 2017, 15:51:29) \n[GCC 6.4.0]\nhi\n
This same pattern can be used to call run any code that interacts with Snakebite or ApacheBeam.
Note that to get this to run, python2 needs to exist in the box/container the task is executing in. If you are using Astronomer Enterprise, you can add the following to your project’s Dockerfile:
RUN apk add py-virtualenv
(or add py-virtualenv to packages.txt)
It gets a bit tougher when there are a bunch of legacy util packages and such that need to run in python2, since that involves changing packages, config, etc.