In our project ,there are thousands of Dags files which are generated dynamically. When there is a new dag file created inside “dags” directory then Airflow takes more than 30 minutes to load new dag file in UI and is ready to be triggered.
Kindly please help me improve this performance.
The first thing would be to check what your dag-dir-list-interval is configured to…it defaults to 5 minutes, so if it’s still at the default value this is probably caused by your existing dags taking a long time to load - there is a guage metric exposed dag_processing.total_parse_time which can tell you exactly how long your dags take to load.
As a general rule, to keep your dags loading fast, you should not make any external network calls during the load phase - not to an external service, nor (ideally) to the metadata database…so, no reading variables etc during load.