Cannot install some Python packages

#1

I’ve been trying to install scikit-learn and spacy without success, by adding them to requirements.txt. When running astro airflow start I get the following output and then the process seems to just hang.

Sending build context to Docker daemon  236.5kB
Step 1/13 : FROM astronomerinc/ap-airflow:0.9.0-1.10.3-onbuild
# Executing 5 build triggers
 ---> Using cache
 ---> Using cache
 ---> Running in a27916d656c4

Here’s the contents of packages.txt:

libc-dev
musl
libc6-compat
gcc
python3-dev
build-base
gfortran
freetype-dev
libpng-dev
openblas-dev
gfortran
build-base
g++
make
musl-dev
libxslt-dev
libxslt
subversion
nodejs
git
npm

#2

Ok so it turns out that the process didn’t actually freeze but rather was compiling the source distributions of the python packages, which takes a long time. Am I correct to assume that there are no pre-compiled binaries of scikit-learn and spacy available for Linux Alpine, and that this is what causes the long installations?

#3

I think that’s correct. And yes, it takes quite a long time. We are testing out a Beta image on Debian instead of Alpine if that’s something you’re interested in trying.

#4

I would love to give that a try. How do I proceed?

#5

@miroli Hi there! If you’re interested in testing out our Debian image, you can go ahead and throw the following in your Dockerfile:

FROM astronomerio/ap-airflow:debian-alpha.1-onbuild

If you have any issues, feel free to reply back in here or email us directly at support@astronomer.io

#6

Hi Paola,

I also am interested in using the Debian image. I was able to build the dockerfile but when I clicked run on any of the DAGs I got this error: “AttributeError: ‘NoneType’ object has no attribute ‘create_dagrun’”

Thanks for your help,
Ryan

#7

Hi there!
This is usually due to an airfow version mismatch. Try astro airflow kill and astro airflow start and you should be good to go!

1 Like