Can you programmatically remove DAGs from the Airflow UI?

Airflow
1

What is the best practice for removing dags from your UI that were dynamically generated?

For instance, if I had code that generates dags dynamically from a database table and one of the dags were removed, it still exists in the UI (because the dag still exists in the database even though it’s no longer in the dagbag).

How do I remove these dags cleanly and automatically? Is the only way to manually delete it in the database?

1 Like
2

Hi @timc3406! Thanks for reaching out here. To answer your question, the easiest way to delete a DAG is to do both of the following:

  • Remove the DAG file from your project
  • Delete the DAG from the Airflow UI (there’s a red “X” at the right of the “Links” menu in the main DAGs Dashboard view)

Doing both of these items ensure that it does not show in the Airflow UI.

image
image2808×506 59.1 KB

A few additional notes:

  • You CAN leverage the Airflow CLI command airflow delete_dag, which you can run on Astronomer Enterprise by wrapping the Airflow CLI command around a kubectl exec command on your Scheduler or Webserver Pod
    • E.g. kubectl exec -it {scheduler-container] bash -c ..."
  • There is currently no add DAG or delete DAG endpoint exposed via the current API, but a restructured API is a milestone our team + the Airflow community is very much working towards that should help users programmatically delete dynamically generated DAGs in the future

Let me know if that answers your question :slightly_smiling_face:

3

Hey there,

My question is related to this topic. I generate DAGs automatically using a python script which selects configurations from BigQuery. Works like a charm, but when I disable one config for a dag, i can see in the webserver log that that DAG is no longer in the dagbag. But it doesn’t get deleted from the database, I guess, because it remains in the UI. Does it disappear after some time or do I also need to add some dag-delete code in my script? Hope to hear from you.

cheers,
Peter