I’m following https://www.astronomer.io/docs/ac-local/ and running the docker image successfully
I’d like to add the a dependency for https://pypi.org/project/apache-airflow-backport-providers-amazon/
I’ve added apache-airflow-backport-providers-amazon==2020.6.24 to ‘requirements.txt’, stopping the process and running docker down/up. but still getting error.
Broken DAG: [/usr/local/airflow/dags/gluejob.py] No module named ‘airflow.providers’
Is there a step I’m missing?
Looks like I need something similar to https://github.com/docker/docker-py/blob/master/Dockerfile
Using
$ docker-compose up --build
Building scheduler
Step 1/2 : FROM astronomerinc/ap-airflow:1.10.10-alpine3.10-onbuild
—> Using cache
—> Using cache
—> Running in 8df2a795b077
Do not upgrade by specifying ‘apache-airflow’ in your requirements.txt, change the base image instead!
ERROR: Service ‘scheduler’ failed to build: The command ‘/bin/sh -c if grep -q ‘apache-airflow’ requirements.txt; then echo >&2 “Do not upgrade by specifying ‘apache-airflow’ in your requirements.txt, change the base image instead!”; exit 1; fi; pip install --no-cache-dir -q -r requirements.txt’ returned a non-zero code: 1
Hi @ctopinka! Great to see you in here - thanks for reporting this. I just spoke to our Airflow team and it looks like the check we have in place for users that include apache-airflow in their requirements.txt file is too widely encompassing.
Users should get the error you see when including apache-airflow (not needed on Astronomer), but naturally NOT if that’s the prefix of an entirely different package such as apache-airflow-backport-providers-amazon.
For now, can you try adding the following to the Dockerfile in your project?
FROM astronomerinc/ap-airflow:1.10.10-alpine3.10-onbuild
RUN pip install 'apache-airflow-backport-providers-amazon==2020.6.24'
That’ll install the package you’re looking for at runtime and get around the check in requirements.txt. For reference, I’ve added this bug to be fixed in our upcoming AC release, which will be coming out in the next week or two. To stay updated on that release, subscribe to our AC Mailing list.
Let me know if that works, @ctopinka.
Hello @paola! You too! Hope everything is great. I’m seeing a different errors and behavior now after adding recommended and running ‘docker-compse up --build’
In console with context I have
Requirement already satisfied: defusedxml in /usr/lib/python3.7/site-packages (from python3-openid>=2.0->Flask-OpenID<2,>=1.2.5->flask-appbuilder<2.3.3->-c /usr/local/share/astronomer-pip-constraints.txt (line 14)) (0.6.0)
ERROR: snowflake-connector-python 1.9.1 has requirement boto3<1.10.0,>=1.4.4, but you'll have boto3 1.14.5 which is incompatible.
ERROR: snowflake-connector-python 1.9.1 has requirement botocore<1.13.0,>=1.5.0, but you'll have botocore 1.17.24 which is incompatible.
ERROR: astronomer-certified 1.10.10.post3 has requirement apache-airflow==1!1.10.10+astro.3, but you'll have apache-airflow 1.10.11 which is incompatible.
Installing collected packages: watchtower, email-validator, text-unidecode, python-slugify, python-nvd3, cattrs, apache-airflow, apache-airflow-backport-providers-amazon, botocore
In UI the glue-job dag isn’t loaded and have error:
Broken plugin: [astronomer.airflow.version_check.plugin] (apache-airflow 1.10.11 (/usr/lib/python3.7/site-packages), Requirement.parse('apache-airflow==1!1.10.10+astro.3'), {'astronomer-certified'})
Thanks!
Can you try with the following please:
FROM astronomerinc/ap-airflow:1.10.10-alpine3.10-onbuild
RUN pip install 'apache-airflow-backport-providers-amazon'
This will install the latest backport packages that are compatible with Astronomer Certified
Hi @ctopinka - i followed your suggestion for google backport package. we are using airflow 10.7 currently .
but i get this error when starting the docker image
Broken DAG: [/usr/local/airflow/dags/gcp_smoke_bq.py] cannot import name 'aio' from 'grpc.experimental' (/usr/lib/python3.7/site-packages/grpc/experimental/__init__.py)
what am i missing ?
I’m also getting this same error as need to depend on backport providers packages too. Seems like the astro cli is trying to be too smart. Current astro cli version I’m using is from a homebrew install: Astro CLI Version: 0.20.0, Git Commit: ab454f34716757666971e152b42b66d9ad0c3a04.
Has there been an update yet to the cli so I can simply added the valid requirement of apache-airflow-backport-providers-amazon and apache-airflow-backport-providers-microsoft-azure to requirements.txt to name a couple?
Or should we still be using the workaround of modifying the Dockerfile directly via RUN pip install?
Hi @jfagan! Happy to take a look at this. Can you confirm a few things?
The bug I mentioned above has been fixed. You should now be able to include apache-airflow-backport-providers-amazon etc in your requirements.txt file as you otherwise would. The fix for this is included in the following versions of AC:
Hi @shlomiken! Did you include that package in your requirements.txt file? Can you try updating your grpc version?
If you’re still having trouble, go ahead and reach out to Astronomer Support.
Hi Paula
i decided to go with ubuntu image to avoid these kind of problems .
thanks
I’m using 1.10.12. I’ve worked around issue via my own Dockerfile for now, like so:
FROM astronomerinc/ap-airflow:1.10.12-1-buster-onbuild
# Work around bug in Astronomer cli - see https://forum.astronomer.io/t/add-apache-airflow-backport-providers-amazon-requirement-to-astronmer-certified-docker/718/8?u=jfagan
ENV PATH "/home/astro/.local/bin:${PATH}"
USER root
RUN pip install --no-cache-dir --upgrade pip
RUN pip install --no-cache-dir --upgrade setuptools wheel --use-feature=2020-resolver
USER astro
RUN pip install --no-cache-dir -r requirements.prod.txt --use-feature=2020-resolver
# Fix broken Airflow backport-providers (backport providers only work when they are installed in the same namespace as
# the ‘apache-airflow’ 1.10 package, and since airflow package is inherited from base image, the namespaces are
# different from our pip installs we've added in this child Docker image.)
# See https://airflow.readthedocs.io/en/latest/backport-providers.html#troubleshooting-installing-backport-packages
USER root
RUN cd /usr/local/lib/python3.7/site-packages/airflow \
&& ln -s /home/astro/.local/lib/python3.7/site-packages/airflow/providers
USER astro@jfagan Glad to hear you got around this, then! Let us know if you have any other trouble with this.