Setting up incremental jobs - Not working

Hi Team,

I plan to set up a daily run for a certain DAG. I have used catchup=True and depends_on_past = True

For every dag run, I have used “prev_execution_date_success” as a date to pass into my DAG for execution of a certain script. Meaning, my dag has to pick all the data that has the date = “prev_execution_date_success”

This works if the DAG runs successfully everyday. However, when the DAG fails for any reason, and I manually correct it and run it. The “prev_execution_date_success” is not the correct date to pass to the script. The future runs will lose some days and data will be lost

So, any ideas on how to fix this or is there any other alternate way of doing this?



It’s hard to understand your design intent. Can you describe in a sentence the meaning of the variable that you are using prev_execution_date_success for?