Issue with Astronomer Certified 2.0.2 and Airflow Helm Chart 0.19.0

mgoeke · May 12, 2021, 8:51pm

Hi all!

First I want to commend y’all on some fantastic repos under your org. I’m currently leveraging the 0.19.0 helm chart and trying to find the most stable ^2.0.0 image to use for both local (docker-compose) and remote (k8s) development.

Recently I’ve run into some issues regarding 2.0.0 (post 1 through 5) where the docker compose is failing during start-up due to an issue with the utils/db.py run and I’m trying to avoid going too deep down the rabbit hole triage wise. It looks like the migration is failing part of the way through (stack trace at the bottom of the post) and I’m curious if this is a known issue or if the docker-compose.yml generated from the 0.23.4 CLI cut is out of date now for these images?

Secondly it looks like 2.0.2-* is not compatible with the 0.19.0 chart and I’m curious if there is a branch cut I can be leveraging that fixes that? Scouring the repo last night I didn’t see anything promising and was this close to forking and testing the chart pointing the default version to 2.0.2-* just in case this was a migrations incompatibility between 2.0.0 and 2.0.2.

Lastly, I’m trying to leverage poetry for our artifact build and I’m running into an interesting solver conflict with putting astronomer_certified in:

Summary

Because astronomer-certified (2.0.0.post5) depends on apache-airflow (1)
   and apache-airflow-providers-databricks (1.0.1) depends on apache-airflow (>=2.0.0), astronomer-certified (2.0.0.post5) is incompatible with apache-airflow-providers-databricks (1.0.1).
  So, because airflow-libs depends on both apache-airflow-providers-databricks (1.0.1) and astronomer_certified (2.0.0-5), version solving failed.

I’m curious if the transitive bind for ^2.0.0 is intentionally set to 1.0.0 for some reason I’m unaware of or if this is just a global repo compatibility concern? Either way right now I’m forced to swap to apache-airflow for the reference but it would be nice to leverage the correct pip so that the site installs match what is in the image.

Stack trace for 2.0.0-5:

Summary

scheduler_1  | Traceback (most recent call last):
scheduler_1  |   File "/usr/local/bin/airflow", line 8, in <module>
scheduler_1  |     sys.exit(main())
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
scheduler_1  |     args.func(args)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
scheduler_1  |     return func(*args, **kwargs)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
scheduler_1  |     return f(*args, **kwargs)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 48, in upgradedb
scheduler_1  |     db.upgradedb()
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/astronomer/airflow/version_check/plugin.py", line 29, in run_before
scheduler_1  |     fn(*args, **kwargs)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 684, in upgradedb
scheduler_1  |     command.upgrade(config, 'heads')
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 294, in upgrade
scheduler_1  |     script.run_env()
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 481, in run_env
scheduler_1  |     util.load_python_file(self.dir, "env.py")
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
scheduler_1  |     module = load_module_py(module_id, path)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
scheduler_1  |     spec.loader.exec_module(module)
scheduler_1  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
scheduler_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 108, in <module>
scheduler_1  |     run_migrations_online()
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
scheduler_1  |     context.run_migrations()
scheduler_1  |   File "<string>", line 8, in run_migrations
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
scheduler_1  |     self.get_context().run_migrations(**kw)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 548, in run_migrations
scheduler_1  |     for step in self._migrations_fn(heads, self):
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 283, in upgrade
scheduler_1  |     return script._upgrade_revs(revision, rev)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 361, in _upgrade_revs
scheduler_1  |     for script in reversed(list(revs))
scheduler_1  |   File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
scheduler_1  |     self.gen.throw(type, value, traceback)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 194, in _catch_revision_errors
scheduler_1  |     compat.raise_(util.CommandError(resolution), from_=re)
scheduler_1  |   File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 294, in raise_
scheduler_1  |     raise exception
scheduler_1  | alembic.util.exc.CommandError: Can't locate revision identified by '2e42bb497a22'

kaxil · May 13, 2021, 1:06pm

For using Airflow/AC 2.0.2 image with the Astronomer Airflow chart, don’t forget to update both of the following values:

airflowVersion → set this to 2.0.2 when you run helm install command
defaultAirflowTag → set this to 2.0.2-buster

github.com

astronomer/airflow-chart/blob/master/values.yaml#L76-L83


      
          # Airflow webserver settings
          webserver:
            allowPodLogReading: false
            args: ["airflow", "webserver"]
            webserverConfig: |
              import os
              from airflow import configuration as conf
              from flask_appbuilder.security.manager import AUTH_REMOTE_USER

Regarding the following:

I’m curious if the transitive bind for ^2.0.0 is intentionally set to 1.0.0 for some reason I’m unaware of or if this is just a global repo compatibility concern?

That is not true, for example, installing astronomer-certified==2.0.2-2 install apache-airflow==1!2.0.2+astro.2 from our PIP repository: link

❯ PIP_EXTRA_INDEX_URL="https://pip.astronomer.io/simple" pip install 'astronomer-certified==2.0.2-2'
Looking in indexes: https://pypi.org/simple, https://pip.astronomer.io/simple
Collecting astronomer-certified==2.0.2-2
  Using cached https://pip.astronomer.io/simple/astronomer-certified/astronomer_certified-2.0.2.post2-py3-none-any.whl (4.7 MB)
Collecting astronomer-airflow-version-check~=1.0
  Using cached https://pip.astronomer.io/simple/astronomer-airflow-version-check/astronomer_airflow_version_check-1.0.7-py3-none-any.whl (17 kB)
Collecting apache-airflow==1!2.0.2+astro.2
  Downloading https://pip.astronomer.io/simple/apache-airflow/apache_airflow-1%212.0.2%2Bastro.2-py3-none-any.whl (4.7 MB)

mgoeke · May 13, 2021, 1:39pm

Thank you for the detailed response!

It looks like I was missing defaultAirflowTag as I had already been setting images.airflow.repository, images.airflow.tag and airflowVersion so I’ll try that out shortly.

Secondly, it looks like my post insinuated an incorrect claim regarding the transitive bind which I need to apologize for . I am still running into an issue with the solver even when leveraging 2.0.2-2:

[tool.poetry.dependencies]
apache-airflow-providers-databricks = "1.0.1"

[tool.poetry.dev-dependencies]
astronomer_certified = "2.0.2-2"

  Because astronomer-certified (2.0.2.post2) depends on apache-airflow (1)
   and apache-airflow-providers-databricks (1.0.1) depends on apache-airflow (>=2.0.0), astronomer-certified (2.0.2.post2) is incompatible with apache-airflow-providers-databricks (1.0.1).
  So, because airflow-libs depends on both apache-airflow-providers-databricks (1.0.1) and astronomer_certified (2.0.2-2), version solving failed.

I’m fairly new to Poetry so I’m guessing this is this is a nuance with the way it’s doing semver parsing of the 1!2.0.2+astro.2 that I’ll need to take a look into.

Again, thank you for your time on the response. Is this forum the best place to engage with questions like this or would it be better to leverage Git? Since some of these questions crossed multiple repo boundaries I didn’t want to clutter up the issues section.

kaxil · May 13, 2021, 2:55pm

Yeah, I have not used Poetry that much so not sure why it is failing to identify versions with epoch (1!2.0.2+astro.2) as it is valid in PEP 440: PEP 440 – Version Identification and Dependency Specification | peps.python.org
By any chance can you use pip instead of poetry?

tbh we don’t recommend poetry officially as Airflow project too for other reasons though (constraints vs requirements): https://airflow.apache.org/docs/apache-airflow/stable/installation.html :

While there are some successes with using other tools like poetry or pip-tools, they do not share the same workflow as pip - especially when it comes to constraint vs. requirements management. Installing via Poetry or pip-tools is not currently supported. If you wish to install airflow using those tools you should use the constraint files and convert them to appropriate format and workflow that your tool requires.

paola · May 13, 2021, 3:02pm

As for your question on Forum Post v GitHub Issue, I think this is the right way to do it @mgoeke! We monitor our forum frequently and help provide guidance as much as we can. Questions here are more easily searchable in Google and thus are better positioned to help other users running into the same problem. If we identify and validate a bug that needs to be addressed in airflow-chart, ap-airflow etc, our team can take care of creating a GH issue in the right place.

If you’re an Astronomer customer and ever run into a more time sensitive issue, however, you can always reach out to Astronomer Support as well.

mgoeke · May 13, 2021, 4:27pm

We are currently discussing hosted alternatives so I’m sure y’all will be one of the vendors we evaluate

Part of my efforts here are due diligence to understand what leveraging your platform would look like in comparison to other solutions like AWS MWAA.

mgoeke · May 13, 2021, 4:41pm

We definitely could and our org is currently looking into what toolchain we’re going to standardize on for multi language repo artifact generation. I am still coming up to speed with Python build toolchains and so far with Poetry I really like the ability to mimic Maven’s artifact scoping (test / provided / etc). I’d like the ability to have the provided dependencies (i.e. astronomer_certified, astronomer_airflow_scripts and astronomer-fab-security-manager) from your image still be version resolved / installed in our development venv with anything we define in our custom whl without having to vendor it.

It’s possible pip can do this already and I’m just not savvy to what it looks like?

Topic		Replies	Views
Is Airflow 2.3 available yet in astronomer cloud? Astronomer Nebula	3	1623	May 5, 2022
Airflow 1.10.10 on Astronomer Astronomer	3	2053	July 2, 2020
Astronomer install failure Astronomer airflow	0	670	July 19, 2023
Astronomer Cloud : Successful image push, code not updated Astronomer Nebula	4	3811	September 12, 2019
DAG Missing After Second Deploy Astronomer Nebula	1	7176	September 11, 2019

Issue with Astronomer Certified 2.0.2 and Airflow Helm Chart 0.19.0

Related Topics