I plan to set up a daily run for a certain DAG. I have used catchup=True and depends_on_past = True
For every dag run, I have used “prev_execution_date_success” as a date to pass into my DAG for execution of a certain script. Meaning, my dag has to pick all the data that has the date = “prev_execution_date_success”
This works if the DAG runs successfully everyday. However, when the DAG fails for any reason, and I manually correct it and run it. The “prev_execution_date_success” is not the correct date to pass to the script. The future runs will lose some days and data will be lost
So, any ideas on how to fix this or is there any other alternate way of doing this?