Hi,
can you please let me know how to ensure airflow singleton instance in env. for Example in UI there is option Number of runs to create dag instances and which is minimum 5 instances to be created.But i need only one instance to be required for it.
Thanks in Advance
RRR
Dear @RRR,
As you have guessed, there is an option to set number of runs. The DAG class has a parameter to limit number of active runs named max_active_runs. Here is where you can specify the number of concurrent dagruns that can be in the running state.
Here is an example to showcase the parameter in use.
from airflow.models import DAG
dag = DAG(
dag_id='my_dag',
max_active_runs=1,
schedule_interval='*/15 * * * *',
catchup=True
)
The example here is a DAG that is scheduled every 15 minutes. If your DAG takes two hours to run, airflow will not schedule another one until the current one finishes. Also note that catchup is set as True meaning that the dagrun will have the execution date following the current execution date according to the schedule interval.
More concretely, this means that if the current dagrun has an execution date of 2020-01-01T00:00:00+00:00, the next dagrun will have the execution date of 2020-01-01T00:15:00+00:00.
Thank you @Alan for quick response.I got solution already but I made geniric fix by changing below parameter in airflow.cfg file .Thank you to given this fix for specific to dag.
max_active_runs_per_dag = 1
And one more questionn that what is the significance role of Number of Runs in Dag UI. is it same with our discussing thread?
For clarity, I have included an screenshot of the UI element in question.
Number of runs here indicates how many previous runs to display starting from the base date.
Given the base date of 2020-01-01T00:00:00+00:00 with its schedule set to run every hour, and a history back to the beginning of 2019, we will see 25 runs under the Run dropdown instead of the full list of dagruns.
