Pip install failing image build and determining dependencies

Locally, I am using conda to manage my python env. Pip freezing gives me the following requirements (which could be probably be trimmed down). Also in my env => python --version
Python 3.7.4.

#requirements.txt
alembic==1.0.11
apache-airflow==1.10.4
asn1crypto==0.24.0
attrs==19.1.0
Babel==2.7.0
backcall==0.1.0
bleach==3.1.0
boto3==1.9.199
botocore==1.12.199
cached-property==1.5.1
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
Click==7.0
colorama==0.4.1
colorlog==4.0.2
configparser==3.5.0
croniter==0.3.30
cryptography==2.7
decorator==4.4.0
defusedxml==0.5.0
dill==0.2.9
dnspython==1.16.0
docutils==0.14
entrypoints==0.3
Flask==1.1.1
Flask-Admin==1.5.3
Flask-AppBuilder==1.12.3
Flask-Babel==0.12.2
Flask-Caching==1.3.3
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.0
flask-swagger==0.2.13
Flask-WTF==0.14.2
future==0.16.0
gitdb2==2.0.5
GitPython==3.0.1
gunicorn==19.9.0
idna==2.8
ipykernel==5.1.2
ipython==7.7.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
iso8601==0.1.12
itsdangerous==1.1.0
jedi==0.15.1
Jinja2==2.10.1
jmespath==0.9.4
json-merge-patch==0.2
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-core==4.5.0
lazy-object-proxy==1.3.1
lockfile==0.12.2
lxml==4.4.1
Mako==1.1.0
Markdown==2.6.11
MarkupSafe==1.1.1
mistune==0.8.4
nbconvert==5.5.0
nbformat==4.4.0
notebook==6.0.0
numpy==1.17.0
pandas==0.25.0
pandocfilters==1.4.2
parso==0.5.1
pendulum==1.4.4
pexpect==4.7.0
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==2.0.9
psutil==5.6.3
psycopg2==2.7.6.1
ptyprocess==0.6.0
pycparser==2.19
Pygments==2.4.2
pymongo==3.8.0
pyOpenSSL==19.0.0
pyrsistent==0.14.11
PySocks==1.7.0
python-daemon==2.1.2
python-dateutil==2.8.0
python-editor==1.0.4
python-nvd3==0.15.0
python-slugify==1.2.6
python3-openid==3.1.0
pytz==2019.2
pytzdata==2019.2
PyYAML==5.1.2
pyzmq==18.1.0
qtconsole==4.5.4
requests==2.22.0
s3transfer==0.2.1
Send2Trash==1.5.0
setproctitle==1.1.10
six==1.12.0
smmap2==2.0.5
SQLAlchemy==1.3.7
tabulate==0.8.3
tenacity==4.12.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
text-unidecode==1.2
thrift==0.11.0
tornado==6.0.3
traitlets==4.3.2
tzlocal==1.5.1
unicodecsv==0.14.1
Unidecode==1.1.1
urllib3==1.25.3
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.5
widgetsnbextension==3.5.1
WTForms==2.2.1
zope.deprecation==4.4.0

astro airflow start using the requirements above fails with:

Command “/usr/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c9pb7l_7 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple – setuptools>=40.6.0 wheel “cffi>=1.8,!=1.11.3; python_implementation != ‘PyPy’”” failed with error code 1 in None
You are using pip version 18.1, however version 19.2.3 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
The command ‘/bin/sh -c pip install --no-cache-dir -q -r requirements.txt’ returned a non-zero code: 1
Error: command 'docker build -t orders-lifecycle-etl/airflow:latest failed: failed to execute cmd: exit status 1

I tried adding the suggested list of packages to packages.txt:

libc-dev
musl
libc6-compat
gcc
python3-dev
build-base
gfortran
freetype-dev
libpng-dev
openblas-dev
gfortran
build-base
g++
make
musl-dev

But I still get the error. I am not sure which dependencies I actually need or if there is an easy way to determine them.

Hi @ced! We ended up getting this fixed via a private support ticket, but some best practices for anyone following:

  1. Make sure that you need all the packages in here

    • Astronomer docker images already have the python libraries that users typically need built-in, so we’d recommend only adding additional libraries that are needed for you DAGs as you need them
  2. If you’re developing locally, make sure that the Airflow version you’re running locally versus Astronomer Cloud are consistent

  3. If you DO need all of these packages, if might be better to start with an empty requirements.txt, bash into one of the running containers (by running docker exec -it <container ID> bash), run pip and try and install each package until you get an error message that’ll diagnose with OS system package you need.

    • These containers are running on Alpine linux, so you can use apk to add those missing os packages