Run-airflow-migration and wait-for-airflow-migrations

1

I am installing Astronomer Airflow using the Helm chart and have a question regarding start up. When I run the installation and watch the pods I can see the scheduler start but eventually crash. When I examine the logs of wait-for-airflow-migrations I can see that it times out:

[2021-04-26 20:19:06,234] {command_line.py:39} INFO - Waiting for migrations... 60 second(s)
Traceback (most recent call last):
  File "/usr/local/bin/airflow-migration-spinner", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/astronomer/migration_spinner/command_line.py", line 47, in main
    spinner(args.timeout)
  File "/usr/local/lib/python3.7/site-packages/astronomer/migration_spinner/command_line.py", line 35, in spinner
    "There are still unapplied migrations after {} seconds".format(timeout)
TimeoutError: There are still unapplied migrations after 60 seconds

When I watch the pods I never see run-airflow-migrations launch. When I look at the annotations for that job I can see that the helm.sh/hook is post-install,post-upgrade which indicates that its supposed to run after everything else is installed. So is wait-for-airflow-migrations actually waiting for run-airflow-migrations, but if so how does that reconcile with migration hooks?

One other thing, the database I am using was formerly used tied to a pre-2.0 Airflow installation. Could this be my issue?

2

You can ignore this. I rebuilt my chart and the problem went away. I must have introduced an error when I was tweaking the chart to conform with our Kubernetes cluster.

1 Like
3

Glad you’re no longer seeing this issue, @cmukai ! Let us know if you run into anything else, happy to help.

4

@paola Actually I discovered its a Terraform issue (the helm CLI works but the Terraform helm_release resource does not). I will file a bug report with them.

5

@cmukai if you don’t mind linking the bug report here, that’d be helpful! I’m also encountering the same error, trying to get Airflow 2 on my Kubernetes cluster via Terraform’s helm_release (I’m on the Helm provider >2.0), so I’m trying to figure out if the errors I’m seeing were the ones you were seeing.

I’ll give it a shot through the helm CLI.

6

@johntam I just did the switch this morning in my Terraform (use null_resource/local_exec instead of helm_release) and it works. But this is definitely sub-optimal since null_resource is not idempotent without me doing some scripting to check if the release exists or not.

7

Just tried this myself, @cmukai – you’re right, Terraform’s helm_release doesn’t succeed, and I did this via Helm CLI, and it deployed properly. So strange. I might try your null_resource/local_exec strategy.

I assume it was something like:

provisioner “local-exec” {
command = “helm install airflow foo/bar -f foo-bar-values.yaml”
}

If you link your Terraform bug report, happy to +1 or give some more data points for the Hashicorp team, might even dig into the bug myself since this might become a blocker for me.

8

Hi @johntam and @cmukai

I have the same issue and local-exec is not a good solution.
Here is the bug report: helm_release can not properly deploy airflow while helm cli does ¡ Issue #742 ¡ hashicorp/terraform-provider-helm ¡ GitHub

I have tried the following charts and both are affected:

9

Thanks for doing this. I got tied up with my day job so I didn’t get around to setting up a bug report.

1 Like
10

Hi @cmukai and @johntam

I did more investigation and everything comes down helm_release does not trigger the helm hooks when wait=true. This was raised here: Helm post-install hooks are not triggered when used by `helm_release` and `wait = true` ¡ Issue #683 ¡ hashicorp/terraform-provider-helm ¡ GitHub

This helm hooks runs the airflow db init command which creates the databased schema for airflow. This bug of helm-relese effects all of the airflow helm charts (official, astronomer, airflow-helm)

Possible solution set wait=false:

resource "helm_release" "airflow" {
  repository = "https://airflow-helm.github.io/charts"
  chart      = "airflow"
  name       = "airflow"
  version    = "8.0.9"
  namespace  = "airflow"
  wait       = false
}
1 Like
11

The reason why we are using the Astronomer Helm charts is because they support Airflow 2.0. For some reason it seems that the Airflow charts do not.

12

This appears to do the trick. Thanks a lot.

1 Like
13

This solution worked. Thank you!