Hi folks,
New to airflow, trying to investigate a scenario and looking to know if there are functionalities in airflow to handle it.
We have 2 dags: dag_1 and dag_2.
dag_1: Looks into a sql table and any record for the query we have, picks the record(ex: some
value in column Id) and trigger dag_2 by passing it and dag_1 job is done and will
be keep running and looking for entries matching the query.
dag_2: Does some job with that provided arguments (i.e Id) and this might take some time to complete.
Now if dag_1 gets a new record in sql with same value (Id) as previous(which is valid scenario), it will trigger new dag_2 run even while previous run is in progress, this will mess up few things.
Is there are way we can configure that if there is a dag run on-going with same provided parameter(i.e id in this case), that we skip the new run or cancel the previous so the new one keeps going.
If provided parameter(i.e id) is different it can keep triggering/run the dag_2, this is only if the Id parameter passed is same as previous.
Only case it should skip or cancel is if the arguments passed in new run is same as previous which is still running. Any new dag trigger and there is no on-going run matching arguments are good to start.
**Id here means the column value in the sql record pass on as a parameter/arguments.
Thank You
Vishu