Hi all,
I’m encountering an issue in Airflow. I’m running 1,000 DAG runs per minute, and while tasks are completing quickly, the DAG run itself remains in a running state for a few minutes even after all tasks have finished. It takes about 5-6 minutes before the DAG run finally switches to success, which causes other DAG runs to get stuck in the queue.
Does anyone know why the DAG run remains in running even though all tasks have completed?
I noticed in the Airflow UI that the dag_run statuses seem to update in batches, with all statuses changing at the same time. Is there a configuration we can update to reduce this wait time and have the status change immediately?
For reference, I’m using 5 schedulers.
airflow.cfg
max_active_runs_per_dag = 1000
max_active_tasks_per_dag = 2000
parallelism = 512
max_tis_per_query = 1000
max_dagruns_per_loop_to_schedule = 100