Astronomer Conda Packages

Astronomer
1

I’m trying to run a DAG which uses geopandas as one of its imports. I was able to install geopandas via Conda on the astronomerinc/ap-airflow:1.10.7-alpine3.10-onbuild Docker image after getting Conda installed. However, when I try to import it in the DAG, it cannot find the module.

Doing a “docker exec -it CONTAINER_ID python” then “import geopandas” does successfully run so it is installed.

When I run ps inside the webserver container I noticed, it’s using the default install of python
{/usr/bin/python} /usr/bin/python3.7 /usr/bin/airflow webserver
and not /opt/conda/bin/python which I’ve installed and has the geopandas library.

Is there any way to have airflow use conda packages? Or have conda install packages to the base python install(I’m not sure if this is even possible).

Note: Installing geopandas with pip on alpine Linux does not appear to be possible. It’s strongly recommended to use conda if possible.

Thanks,
Chris

2

Hey @Chrisreed278! Interesting that geopandas can’t be installed on Alpine via pip. Have you tried our debian base image? You might have more luck getting that package installed that way- you can use astronomerinc/ap-airflow:1.10.7-buster-onbuild if you’d like to give that a shot.

Let me know how it goes!

1 Like
3

Hi @pete I have the same question as @Chrisreed278 : I cannot use pip to install my packages because my company provides conda repos in artifactory, but it doesn’t have a PyPI repo available there. I am using the astro CLI with the quay.io/astronomer/astro-runtime:9.1.0-python-3.10-base image and trying to follow the strategy outlined here, but I have a private conda repo instead. Is it possible to somehow copy conda packages over in a similar way to stage 3 of the example there? Any suggestions are welcome! Thanks.