Airflow schedule not updating

Airflow
1

I created a DAG that will run on a weekly basis. Below is what I tried and it’s working as expected.

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator

SCHEDULE_INTERVAL = timedelta(weeks=1, seconds=00, minutes=00, hours=00)
default_args = {
    'depends_on_past': False,
    'retries': 0,
    'retry_delay': timedelta(minutes=2),
    'wait_for_downstream': True,
    'provide_context': True,
    'start_date': datetime(2020, 12, 20, hour=00, minute=00, second=00)
}

with DAG("DAG", default_args=default_args, schedule_interval=SCHEDULE_INTERVAL, catchup=True) as dag:
    t1 = BashOperator(
        task_id='dag_schedule',
        bash_command='echo DAG',
        dag=dag
    )

As per the schedule, it ran on the 27. As there is a change in requirement, Now I’m updating the start date to 30th instead of 27(My idea is to start the schedule from 30 and from there every week). When I change the schedule of the DAG i.e. start date from 27 to 30th. DAG is not picking as per the latest start date, not sure why? When I deleted the old DAG(as it is test DAG I deleted it, in prod I can’t delete it) and created the new DAG with the same name with the latest start date i.e. 30th it’s running as per the schedule.

2

We have been struggling with the scheduler and start_date and schedule_interval values. We couldn’t get the month and day of the month working using the CRON.

3

@rameshgangineni, the DAG didn’t create any dagruns because the dagrun for the current interval has already been created. Moving the start_date up won’t move the interval anymore since the future interval are determined by the first dagrun start_date.

It worked when you created a new DAG because there were no dagruns. So, when Airflow sees that there are no dagruns and the DAG has a time delta schedule_interval, it will create a new dagrun with the 30th as the start_date.

If you had deleted the dagruns or the DAG and have Airflow recreate the DAG, it would have had the same effect as recreating the DAG.

4

@matique Please create a new post with your specific issue and context.