Locally, I am using conda to manage my python env. Pip freezing gives me the following requirements (which could be probably be trimmed down). Also in my env => python --version
Python 3.7.4.
#requirements.txt
alembic==1.0.11
apache-airflow==1.10.4
asn1crypto==0.24.0
attrs==19.1.0
Babel==2.7.0
backcall==0.1.0
bleach==3.1.0
boto3==1.9.199
botocore==1.12.199
cached-property==1.5.1
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
Click==7.0
colorama==0.4.1
colorlog==4.0.2
configparser==3.5.0
croniter==0.3.30
cryptography==2.7
decorator==4.4.0
defusedxml==0.5.0
dill==0.2.9
dnspython==1.16.0
docutils==0.14
entrypoints==0.3
Flask==1.1.1
Flask-Admin==1.5.3
Flask-AppBuilder==1.12.3
Flask-Babel==0.12.2
Flask-Caching==1.3.3
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.0
flask-swagger==0.2.13
Flask-WTF==0.14.2
future==0.16.0
gitdb2==2.0.5
GitPython==3.0.1
gunicorn==19.9.0
idna==2.8
ipykernel==5.1.2
ipython==7.7.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
iso8601==0.1.12
itsdangerous==1.1.0
jedi==0.15.1
Jinja2==2.10.1
jmespath==0.9.4
json-merge-patch==0.2
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-core==4.5.0
lazy-object-proxy==1.3.1
lockfile==0.12.2
lxml==4.4.1
Mako==1.1.0
Markdown==2.6.11
MarkupSafe==1.1.1
mistune==0.8.4
nbconvert==5.5.0
nbformat==4.4.0
notebook==6.0.0
numpy==1.17.0
pandas==0.25.0
pandocfilters==1.4.2
parso==0.5.1
pendulum==1.4.4
pexpect==4.7.0
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==2.0.9
psutil==5.6.3
psycopg2==2.7.6.1
ptyprocess==0.6.0
pycparser==2.19
Pygments==2.4.2
pymongo==3.8.0
pyOpenSSL==19.0.0
pyrsistent==0.14.11
PySocks==1.7.0
python-daemon==2.1.2
python-dateutil==2.8.0
python-editor==1.0.4
python-nvd3==0.15.0
python-slugify==1.2.6
python3-openid==3.1.0
pytz==2019.2
pytzdata==2019.2
PyYAML==5.1.2
pyzmq==18.1.0
qtconsole==4.5.4
requests==2.22.0
s3transfer==0.2.1
Send2Trash==1.5.0
setproctitle==1.1.10
six==1.12.0
smmap2==2.0.5
SQLAlchemy==1.3.7
tabulate==0.8.3
tenacity==4.12.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
text-unidecode==1.2
thrift==0.11.0
tornado==6.0.3
traitlets==4.3.2
tzlocal==1.5.1
unicodecsv==0.14.1
Unidecode==1.1.1
urllib3==1.25.3
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.5
widgetsnbextension==3.5.1
WTForms==2.2.1
zope.deprecation==4.4.0
astro airflow start using the requirements above fails with:
Command “/usr/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c9pb7l_7 --no-warn-script-location --no-binary :none: --only-binary :none: -i Simple index – setuptools>=40.6.0 wheel “cffi>=1.8,!=1.11.3; python_implementation != ‘PyPy’”” failed with error code 1 in None
You are using pip version 18.1, however version 19.2.3 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
The command ‘/bin/sh -c pip install --no-cache-dir -q -r requirements.txt’ returned a non-zero code: 1
Error: command 'docker build -t orders-lifecycle-etl/airflow:latest failed: failed to execute cmd: exit status 1
I tried adding the suggested list of packages to packages.txt:
libc-dev
musl
libc6-compat
gcc
python3-dev
build-base
gfortran
freetype-dev
libpng-dev
openblas-dev
gfortran
build-base
g++
make
musl-dev
But I still get the error. I am not sure which dependencies I actually need or if there is an easy way to determine them.