I’m trying to figure out how to best set up my dags for situations when we need to re-run them (even if the runs were successful as we have some data sources that can have unreliable & changing data sometimes). I was reading through some existing threads on backfilling data (Triggering past execution date through the Airflow UI - #11 by Alan, Backfilling guidelines, What's the best way to re-run a task in Airflow?) but I’m still unsure as to what the best approach is for my use case.
Are my options the only ones listed below?:
- Run the Airflow CLI’s backfill command on my local dev environment (Does Astro CLI support Airflow CLI commands? If so, how do I run Airflow CLI commands using Astro CLI?)
- Delete the task runs & dag runs on Airflow UI for the days I want to re-run the dag for. Then run the dag with a specified execution date for those days
- Clear the task runs on Airflow UI
- Set up a separate dag for backfilling/ re-running
These options feel a bit manual or effort-intensive with lots of redundancies when trying to re-run the task for a large time frame. So I’m hoping that maybe there are alternatives that I haven’t thought of.
Thanks in advance!