Airflow 2.0 on AppleM1 Machine: `qemu: uncaught target signal 4 (Illegal Instructions) - core dumped`

Astro CLI
1

Error

If you’re running Airflow with the Astronomer CLI on an AppleM1 machine, you’re likely to get this error on $ astro dev start if your Dockerfile references an image for Airflow 2.0.0+ (e.g. FROM quay.io/astronomer/ap-airflow:2.0.0-buster-onbuild).

qemu: uncaught target signal 4 (Illegal Instructions) - core dumped

This is typically not an issue for users running Airflow 1.10.x.

Root Cause

Astronomer’s Airflow images install apache-airflow-providers-google by default (source code). This provider in turns installs google-cloud-bigquery[bqstorage] and google-cloud-bigquery[pandas], both of which require pyarrow.

The core issue here is an underlying bug with qemu, the upstream component that Docker uses for running Intel (amd64) containers on M1 (arm64) chips.

Solution

To successfully run Apache Airflow 2.0+ on Docker with an AppleM1 Mac, either:

  • Run apt uninstall pyarrow
  • Run apt uninstall apache-airflow-providers-google

If you’re on Astronomer, add the following to your Dockerfile:

RUN pip uninstall pyarrow -y
RUN pip uninstall apache-airflow-providers-google -y

Our team is exploring better ways to support Apple’s newest users, so expect improvements around this issue to be ongoing. If you have additional questions or insights to share, comment here or reach out to us.

May 2021 Update

Our team recently found that updating to Docker 3.3.2 fixes the issue above with qemu. If you’re running Docker 3.3.2, you can run $ astro dev start without any modification to your Dockefile.

If you get the following error, you can ignore it:

[2021-05-03 16:41:51,646] {providers_manager.py:299} WARNING - Exception when importing
 'airflow.providers.google.common.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package:
No module named 'airflow.providers.google.common.hooks.leveldb'
1 Like