Can I use the Airflow REST API to externally trigger a DAG?

We’re trying to trigger a DAG from an external source. I know that Airflow has an API that we can use to do so, but is there a best practice around this that you know of other than using a sensor?

Airflow does have a REST API being developed for external triggering but it’s still considered to be in the “experimental” stage (defined by the core Airflow contributors). For that reason, we wouldn’t recommend it as a production solution at the moment.

We’d suggest either creating a DAG that runs at a more frequent interval (possibly what the poke is set at) and skips downstream tasks if no file is found. If you need absolute real-time, it might be a job better suited for a lambda function.

Here’s what I wrote in our own Confluence page around using the REST API. It works pretty well. The links are broken because new users can only put 2 links in a post apparently.

Airflow exposes what they call an “experimental” REST API, which allows you to make HTTP requests to an Airflow endpoint to do things like trigger a DAG run. With Astronomer, you just need to create a service account to use as the token passed in for authentication. Here are the steps:

Create service account in Astronomer deployment. Navigate to https://app.astronomer.cloud/deployments and choose your deployment. Click “Service Accounts” and click “New Service Account”:

  • Note down the token, because you won’t be able to view it again.
  • Decide which endpoint to use. In this example, we’ll be using the dag_runs endpoint, which lets you trigger a DAG to run. For more information, the docs are available here: https ://airflow.apache.org/api.html
  • Using an entry in the Astronomer Forums (https ://forum.astronomer.io/t/hitting-the-airflow-api-through-astronomer/44), we start out with a generic cURL request:
  • curl -v -X POST https ://AIRFLOW_DOMAIN/api/experimental/dags/airflow_test_basics_hello_world/dag_runs -H -H 'Authorization: ’ -H ‘Cache-Control: no-cache’ -H ‘content-type: application/json’ -d ‘{}’
  • Replace the “AIRFLOW_DOMAIN” value with the value matching your deployment.
  • Replace the “” with the Service Account token obtained in step 1.
  • Replace “airflow_test_basiscs_hello_world” with the name of your DAG you want to trigger a run for. This is case sensitive. In our case, we’ll be using “customer_health_score”.
  • Our new cURL command is now: curl -v -X POST https ://AIRFLOW_DOMAIN/api/experimental/dags/customer_health_score/dag_runs -H ‘Authorization: XXXX’ -H ‘Cache-Control: no-cache’ -H ‘content-type: application/json’ -d ‘{}’
  • Please note the token is hidden for security.
  • This will successfully kick off a DAG run for the customer_health_score DAG with an execution_date value of NOW(), which is equivalent to clicking the “Play” symbol from the DAG main screen:
  • If you would like to choose a specific execution_date value to kick off the DAG for, we can pass that in with the Data parameter’s JSON value ("-d ‘{}’").
  • The string needs to be in the “YYYY-mm-DDTHH:MM:SS” format, for example: “2016-11-16T11:34:15”.
  • *More information on this StackOverflow article: https ://stackoverflow.com/questions/50121593/pass-parameters-to-airflow-experimental-rest-api-when-creating-dag-run
  • *More information on the source code here: https ://github.com/apache/airflow/blob/v1-9-stable/airflow/www/api/experimental/endpoints.py
  • Our new command becomes: curl -v -X POST https ://AIRFLOW_DOMAIN/api/experimental/dags/customer_health_score/dag_runs -H ‘Authorization: XXXX’ -H ‘Cache-Control: no-cache’ -H ‘content-type: application/json’ -d ‘{“execution_date”:“2019-03-05T08:30:00”}’

Because this is just a cURL command executed through the command line, this functionality can also be replicated in a scripting language like Python using the Requests library.

7 Likes

This is great. Thanks @joshuamoore-procore!

note, this no longer works with more recent versions of Astronomer, but it is a feature we are looking to add again soon.