Tasks that don't execute until they are cleared

Airflow
1

Hi All,

I’ve been struggling with this issue for a while with no success.
I have a DAG with about 40 tasks.
some of these tasks, although they are scheduled, don’t run until I clear them.
Any ideas for how to debug this issue?

2

Can you provide more context as to what happened?

Is this happening to other DAGs?
Is this happening to the same set of tasks in the DAG you mentioned?

What states were the tasks in before you cleared them?
Once the tasks are cleared, do they go through the normal task life cycle?

Did you check if the pool is full? Or if the workers are at full capacity if using CeleryExecutor?

I would enabled DEBUG level logs in the scheduler and check if the task instances are making it to the executor queue. The logs should also provide some general information about the task instances as they go through the task lifecycle.

1 Like
3

Hi Alan,
Thanks for your response.
I have only two dags running and they are identical.
These are DWH loading processes, we built a new DWH on a different DB and run both dags for a while.
This happens on both.
The funny thing is when I change the dag’s code and move the relations from one line to another
For Example:
I have four tasks A, B, C, and D
It was:
A>>B
B>>C this didn’t execute
C>>D this didn’t execute

changed to:
A>>B>>C
C >> D this doesnt execute

The task status is none and after I clear them through the normal task life cycle.

I’ll check all the points you mentioned and turn on DEBUG.

Thanks again