Add apache-airflow-backport-providers-amazon requirement to astronmer-certified docker

I’m following https://www.astronomer.io/docs/ac-local/ and running the docker image successfully

I’d like to add the a dependency for https://pypi.org/project/apache-airflow-backport-providers-amazon/

I’ve added apache-airflow-backport-providers-amazon==2020.6.24 to ‘requirements.txt’, stopping the process and running docker down/up. but still getting error.

Broken DAG: [/usr/local/airflow/dags/gluejob.py] No module named ‘airflow.providers’

Is there a step I’m missing?

Looks like I need something similar to https://github.com/docker/docker-py/blob/master/Dockerfile

Using

$ docker-compose up --build
Building scheduler
Step 1/2 : FROM astronomerinc/ap-airflow:1.10.10-alpine3.10-onbuild

Executing 5 build triggers

—> Using cache
—> Using cache
—> Running in 8df2a795b077
Do not upgrade by specifying ‘apache-airflow’ in your requirements.txt, change the base image instead!
ERROR: Service ‘scheduler’ failed to build: The command ‘/bin/sh -c if grep -q ‘apache-airflow’ requirements.txt; then echo >&2 “Do not upgrade by specifying ‘apache-airflow’ in your requirements.txt, change the base image instead!”; exit 1; fi; pip install --no-cache-dir -q -r requirements.txt’ returned a non-zero code: 1

Hi @ctopinka! Great to see you in here - thanks for reporting this. I just spoke to our Airflow team and it looks like the check we have in place for users that include apache-airflow in their requirements.txt file is too widely encompassing.

Users should get the error you see when including apache-airflow (not needed on Astronomer), but naturally NOT if that’s the prefix of an entirely different package such as apache-airflow-backport-providers-amazon.

For now, can you try adding the following to the Dockerfile in your project?

FROM astronomerinc/ap-airflow:1.10.10-alpine3.10-onbuild
RUN pip install 'apache-airflow-backport-providers-amazon==2020.6.24'

That’ll install the package you’re looking for at runtime and get around the check in requirements.txt. For reference, I’ve added this bug to be fixed in our upcoming AC release, which will be coming out in the next week or two. To stay updated on that release, subscribe to our AC Mailing list.

Let me know if that works, @ctopinka.

Hello @paola! You too! Hope everything is great. I’m seeing a different errors and behavior now after adding recommended and running ‘docker-compse up --build’

In console with context I have

Requirement already satisfied: defusedxml in /usr/lib/python3.7/site-packages (from python3-openid>=2.0->Flask-OpenID<2,>=1.2.5->flask-appbuilder<2.3.3->-c /usr/local/share/astronomer-pip-constraints.txt (line 14)) (0.6.0)
ERROR: snowflake-connector-python 1.9.1 has requirement boto3<1.10.0,>=1.4.4, but you'll have boto3 1.14.5 which is incompatible.
ERROR: snowflake-connector-python 1.9.1 has requirement botocore<1.13.0,>=1.5.0, but you'll have botocore 1.17.24 which is incompatible.
ERROR: astronomer-certified 1.10.10.post3 has requirement apache-airflow==1!1.10.10+astro.3, but you'll have apache-airflow 1.10.11 which is incompatible.
Installing collected packages: watchtower, email-validator, text-unidecode, python-slugify, python-nvd3, cattrs, apache-airflow, apache-airflow-backport-providers-amazon, botocore

In UI the glue-job dag isn’t loaded and have error:

Broken plugin: [astronomer.airflow.version_check.plugin] (apache-airflow 1.10.11 (/usr/lib/python3.7/site-packages), Requirement.parse('apache-airflow==1!1.10.10+astro.3'), {'astronomer-certified'})

Thanks!

@ctopinka

Can you try with the following please:

FROM astronomerinc/ap-airflow:1.10.10-alpine3.10-onbuild
RUN pip install 'apache-airflow-backport-providers-amazon'

This will install the latest backport packages that are compatible with Astronomer Certified