Failed to fetch log file from worker on retry?

Airflow
1

We recently began encountering an unfamiliar error in (at least) one of our long-running dags:

Annotation 2021-12-30 084903
Annotation 2021-12-30 0849031083×241 13.9 KB

Note that this error is on the second attempt and did not occur on the first attempt. It looks like when the first attempt receives a SIGTERM, the retry just quits as soon as it can’t locate a new log file. Any guidance as to what’s happening here?

2

Can you share the log from attempt 1?

3

Annotation 2021-12-30 084903
Annotation 2021-12-30 0849031383×261 21.3 KB

I don’t know what’s causing this either, but for now I’m assuming it’s user error.

Also, I can get the dag to succeed if I simply clear and rerun the failed task. Sometimes it takes a couple of tries, but I can generally get it to work this way.

4

Can you check your scheduler logs for “Marked X SchedulerJob instances as failed” where X is an integer? Also, what’s your airflow version?

5

I think that the task instance is getting started twice by the scheduler and then the two clobber each other. Which is a bug that we’ve been dealing with and has two fixes, one in 2.2.2 and the other in 2.2.3

6

Ahh understood - that might be the case. It makes sense considering we’re running v 2.1.2 at the moment.

7

Yep, that’s it then.