Building the new astronomer image fails during migration on macos

Astro CLI
1

I’m currently migrating to the new Astronomer Cloud. Datastory: newtonian-inclination-8019
MacOS 10.15.2 (19C57)

I’m having an issue while building the new image (astronomerinc/ap-airflow:0.10.3-1.10.5-onbuild). Any thoughts on how I can approach this?

> Astro dev start

gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/

unable to execute 'gcc': No such file or directory error: command 'gcc' failed with exit status 1 ----------------------------------------

ERROR: Command errored out with exit status 1: command: /usr/bin/python3.7 **/usr/lib/python3.7/site-packages/pip** install --ignore-installed --no-user --prefix /tmp/pip-build-env-x6ylw9es/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"''

I guess, the python installation folders are different in Mac and Ubuntu. In Mac they are installed in /usr/local/lib/

~$ find /usr -name 'site-packages'

/usr/local/lib/python3.7/site-packages

/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages

~$ pip --version

pip 20.0.2 from /Users/papanash/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pip (python 3.7)
2

@papanash can you please share you Dockerfile?

cat Dockerfile
3

This is my Dockerfile and it fails on the first line.

FROM astronomerio/ap-airflow:debian-alpha.1-onbuild
ENV AIRFLOW__KUBERNETES__IN_CLUSTER True
…

Step 1/9 : FROM astronomerinc/ap-airflow:0.10.3-1.10.5-onbuild
# Executing 5 build triggers
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> Running in 171962e345c3
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3.7 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-x6ylw9es/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"''
4

when i run docker image history on the new astronomer image i noticed that the path of pip is hardcoded to /usr/bin.

docker image history
<missing> 5 months ago |4 BUILD_NUMBER=1503 ORG=astronomer SUBMODULES=all, statsd, elasticsearch VERSION=1.10.5-2 /bin/sh -c apk update && apk add --no-cache --virtual .build-deps build-base cyrus-sasl-dev freetds-dev freetype-dev git krb5-dev libffi-dev libxml2-dev libxslt-dev linux-headers nodejs nodejs-npm python3-dev tzdata && apk add --no-cache bash ca-certificates cyrus-sasl cython3@edge freetds krb5-libs py3-bcrypt@edge py3-cassandra-driver@edge-testing'>='3 py3-cryptography@edge py3-fastavro@edge-testing py3-grpcio@edge-testing py3-mysqlclient py3-numpy-dev@edge-community py3-numpy@edge-community py3-pandas@edge-testing py3-psycopg2 py3-pycryptodome@edge-community py3-pynacl@edge py3-typed-ast@edge-testing python3 tini && update-ca-certificates && cp /usr/share/zoneinfo/UTC /etc/localtime && pip3 install --no-cache-dir --upgrade pip==19.2.3 && pip3 install --no-cache-dir --upgrade setuptools==41.0.1 && pip3 install --no-cache-dir --upgrade snowflake-connector-python==1.9.1 && pip3 install --no-cache-dir "${AIRFLOW_MODULE}" && pip3 install --no-cache-dir "https://github.com/astronomer/astronomer-fab-securitymanager/releases/download/v1.2.1/astronomer_fab_security_manager-1.2.1-py3-none-any.whl" && cd /usr/lib/python3.7/site-packages/airflow/www_rbac && npm install && npm run build && rm -rf node_modules && apk del .build-deps py3-numpy-dev && ln -sf /usr/bin/python3 /usr/bin/python && ln -sf /usr/bin/pip3 /usr/bin/pip 586MB

5

@papanash can you please try:

cat Dockerfile
FROM astronomerinc/ap-airflow:1.10.5-buster-onbuild
7

This image worked. Thank you!

I’m having trouble in the last step which is to install mongodb.

* sudo apt-get install -y mongodb-org
Reading package lists…
Building dependency tree…
Reading state information…
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
mongodb-org : Depends: mongodb-org-shell but it is not going to be installed
Depends: mongodb-org-server but it is not going to be installed
Depends: mongodb-org-mongos but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command ‘/bin/bash -o pipefail -e -u -x -c sudo apt-get update && sudo apt-get install -y mongodb-org’ returned a non-zero code: 100

I have tried to install the dependencies before.
RUN sudo apt-get install libcurl4 openssl

and i have to tried to check if it there are failed installation which seems as it should be.
RUN sudo dpkg -l | grep mongo

Are there any more ways to troubleshoot this?

8

Hi @papanash - happy to hear you’re working on migrating to “New” Astronomer Cloud.

Regarding the installation of openssl, libcurl4 etc, do you need those to be installed at run-time? I’d try throwing those packages directly into your project directory under the packages.txt (OS-level packages) and requirements.txt (Python packages) files that are automatically generated when you initialized your Astronomer project.

To install them, just add the name of the package to the respective file and re-build your image locally with $ astro dev stop and $ astro dev start to see if it was successful.

If you’re still having issues, can you reach out to support.astronomer.io? We’ll make sure to help you out there and dig into your image/issues more specifically.