I’m looking for a visualization tool to see all the cross-DAGs dependencies we have in our codebase. I was reading that Lyft built an internal tool for this, but they haven’t released it to the public.
If you have any idea to accomplish this, let me know.
Hi @marcos.ortiz, Welcome to the Astronomer Community, Hope are you doing well!
Regarding your question to visualize the Cross-DAG dependencies you can leverage Airflow DAG Dependencies view.
In Airflow 2.1, a new cross-DAG dependencies view was added to the Airflow UI. This view shows all DAG dependencies in your Airflow environment as long as they are implemented using one of the following methods:
- Using dataset driven scheduling
- Using a TriggerDagRunOperator
- Using an ExternalTaskSensor
To view dependencies in the UI, go to Browse > DAG Dependencies or by click Graph within the Datasets tab.
You can Read more about it here - Cross-DAG dependencies | Astronomer Documentation
Hope this helps!
1 Like
Further, If you would like you can Integrate OpenLineage and Airflow with Marquez to get greater observability over your data pipelines.
Here’s the Astronomer Tutorial to Integrate OpenLineage and Airflow with Marquez - Integrate OpenLineage and Airflow with Marquez | Astronomer Documentation
1 Like
Great @iamakashladdha Let me try this. Thank you