Hi. I have a parent DAG that runs multiple subdags chained in a sequence. I run them by using TriggerDagRunOperator. It’s simple, when one dag is successfully completed, then the next in line is triggered. However, I have issues with subdags that fail, because then obviously parent dag fails, but when I fix the task within the failed subdag and the subdag is again in “running” status, the parent dag isn’t notified that subdag has been restored (either by fixing the failed task and clicking “Clear” to run it again or by marking it as “Success”, because the failure is false positive). Therefore, if the subdag is completed, the parent dag doesn’t see it and this TriggerDagRunOperator is still marked in red, even though this dag is completed successfully. Does anyone know how to fix it in code or where to click when clearing/marking success a task in a subdag, so that it has an effect on the parent dag as well? Thank you in advance.
Hey @MichalB
I assume when you say “SubDAG” you actually do not mean the deprecated SubDAG feature of Airflow and just the child DAGs of the Parent DAG.
To answer you question, each DAG is independent and won’t keep track of the child DAGRuns as you are expecting. This is because a DAG’s execution creates a DAGRun object. Once you have executed a DAG the details about it’s run are stored in the metadata.
If you want to clear the state of that task, you can either clear the task starting from the Parent Task itself or mark is as success later on.
If you encounter such scenarios regularly while supporting your DAGs, you could do this using Airflow’s REST API. If it is one-off scenario, then Airflow UI is the best way to do it (Browse → DAG Runs).
Hopefully this answers your question.
Thanks
Manmeet