How do I set automatic restarts to my Airflow scheduler?

virajparekh · December 6, 2018, 11:29pm

I want the scheduler to restart automatically after a given amount of time

virajparekh · December 6, 2018, 11:31pm

You can configure the run_duration as an env variable:

apache/airflow/blob/v1-9-stable/airflow/config_templates/default_airflow.cfg#L319


# listen (in seconds).
job_heartbeat_sec = 5


# The scheduler constantly tries to trigger new tasks (look at the
# scheduler section in the docs for more information). This defines
# how often the scheduler should run (in seconds).
scheduler_heartbeat_sec = 5


# after how much time should the scheduler terminate in seconds
# -1 indicates to run continuously (see also num_runs)
run_duration = -1


# after how much time a new DAGs should be picked up from the filesystem
min_file_process_interval = 0


dag_dir_list_interval = 300


# How often should stats be printed to the logs
print_stats_interval = 30


child_process_log_directory = {AIRFLOW_HOME}/logs/scheduler

In the Dockerfile, add the line:

ENV AIRFLOW__SCHEDULER__RUN_DURATION=3600

to get it to restart every hour.

Topic		Replies	Views
Scheduler stopping frequently Airflow	0	3279	December 22, 2020
Dag scheduler interval not working Airflow	1	1646	September 27, 2021
Airflow Scheduler not recognizing CRON with Month and/or Day of the month schedule Airflow	2	2630	July 13, 2021
Airflow Dag Run Airflow	1	1785	March 14, 2022
Is there a way to trigger entire airflow DAG to rerun if a single task failed? Airflow airflow	1	22	October 25, 2024

How do I set automatic restarts to my Airflow scheduler?

Related Topics