Cannot install some Python packages

#1

I’ve been trying to install scikit-learn and spacy without success, by adding them to requirements.txt. When running astro airflow start I get the following output and then the process seems to just hang.

Sending build context to Docker daemon  236.5kB
Step 1/13 : FROM astronomerinc/ap-airflow:0.9.0-1.10.3-onbuild
# Executing 5 build triggers
 ---> Using cache
 ---> Using cache
 ---> Running in a27916d656c4

Here’s the contents of packages.txt:

libc-dev
musl
libc6-compat
gcc
python3-dev
build-base
gfortran
freetype-dev
libpng-dev
openblas-dev
gfortran
build-base
g++
make
musl-dev
libxslt-dev
libxslt
subversion
nodejs
git
npm

#2

Ok so it turns out that the process didn’t actually freeze but rather was compiling the source distributions of the python packages, which takes a long time. Am I correct to assume that there are no pre-compiled binaries of scikit-learn and spacy available for Linux Alpine, and that this is what causes the long installations?

#3

I think that’s correct. And yes, it takes quite a long time. We are testing out a Beta image on Debian instead of Alpine if that’s something you’re interested in trying.

#4

I would love to give that a try. How do I proceed?

#5

@miroli Hi there! If you’re interested in testing out our Debian image, you can go ahead and throw the following in your Dockerfile:

FROM astronomerio/ap-airflow:debian-alpha.1-onbuild

If you have any issues, feel free to reply back in here or email us directly at support@astronomer.io

#6

Hi Paola,

I also am interested in using the Debian image. I was able to build the dockerfile but when I clicked run on any of the DAGs I got this error: “AttributeError: ‘NoneType’ object has no attribute ‘create_dagrun’”

Thanks for your help,
Ryan

#7

Hi there!
This is usually due to an airfow version mismatch. Try astro airflow kill and astro airflow start and you should be good to go!

1 Like
#8

Hi @paola. Is there any official version of the Debian Image?

#9

Hi @gonzalodiazrappi! We should have a Debian image available for testing in the next few days, we’re very excited to get this released and to get feedback on it as it should address many of the common OS-level package dependency issues encountered when installing python libraries in Alpine.

1 Like
#10

Hi all! Quick update here that the Astronomer Platform (v0.11+) now officially supports a Debian Image, alongside Alpine, as a part of our Airflow build process.

A few notes:

  • At the time of this post, we support a Debian image for Airflow 1.10.5, 1.10.6 and 1.10.7 (1.10.10 coming soon)
  • Alpine continues to be our default for local development + new projects created on astro dev init, but we envisioning transitioning to Debian by default in the future
  • For our list of images and instructions on how to implement, check out this doc: https://www.astronomer.io/docs/airflow-versioning/
  • In addition to our docs, you can always find all available Docker images in Astronomer’s DockerHub repo