No module named 'airflow.contrib.kubernetes' - deployment fails

Astronomer
1

I tried to create a basic astronomer project, as per: Create an Astronomer Project | Astronomer Cloud
Dockerfile has this line: FROM quay.io/astronomer/ap-airflow:2.2.3-onbuild
astro dev start and local instance has the example dag running!

When I try to deploy this very project to our production installation I get these errors and airflow doesn’t startup:

02/01 10:00:58Traceback (most recent call last): File “/usr/local/lib/python3.9/site-packages/airflow/settings.py”, line 450, in import_local_settings import airflow_local_settings File “/usr/local/airflow/config/airflow_local_settings.py”, line 2, in from airflow.contrib.kubernetes.pod import Pod ModuleNotFoundError: No module named ‘airflow.contrib.kubernetes’

Has anyone have any idea what might be wrong?

2

I’m getting the exact same error but with another DAG I’m wroking on… Were you able to solve it by any chance?

Broken DAG: [/usr/local/airflow/dags/cells_polygon_stores/cells_polygon_stores.py] Traceback (most recent call last):
  File "/usr/local/airflow/dags/cells_polygon_stores/cells_polygon_stores.py", line 34, in <module>
    from dags.utils.ml_operator_sup import MLOperator
  File "/usr/local/airflow/dags/utils/ml_operator_sup.py", line 5, in <module>
    from airflow.contrib.kubernetes.pod import Resources
ModuleNotFoundError: No module named 'airflow.contrib.kubernetes'
3

Might not be your problem.

In my case the problem was that our Astronomer installation had not supported Airflow 2.x. Once we created a new instance that does, everything works as it should.

4

Hi @alainmore & @biku did you add apache-airflow[kubernetes] to your requirements.txt before deploying?

This will pip install the package

5

I tried that but it did not help because my deployment environment wasn’t 2.x. With 2.x I did not need to add it to requirements.txt