I have a DAG running since few months and from last one week it’s behaving abnormal. i am running a bash operator which is executing a shell script and in shell script we have a hive query. no of retries set to 4 as below.
default_args = { ‘owner’: ‘airflow’, ‘depends_on_past’: False, ‘email’: [‘airflow@example.com’], ‘email_on_failure’: False, ‘email_on_retry’: False, ‘retries’: 4 , ‘retry_delay’: timedelta(minutes=5) }
i can see in the log that it’s triggering the hive query and loosing the heartbeats after some time(around 5 to 6 minutes) and going for the retry. Yarn is showing that query is not yet finished but airflow triggered the next run. now in the yarn 2 queries are running (one for the first run and second for the retry) for the same task.similarly this dag is triggering 5 queries(as retry is 4) for the same task and showing the failed status in the last. Interesting point is that the same dag was running fine from long time. also, this is the issue will all the dags related to hive in the production. today i upgraded to latest version of airflow v 1.10.9.
Did anyone have faced the similar issue?