Hi,
I am using airflow with celery executor. As an example, I have a dag with 3 tasks.
task1 >> task2 >> task3
When my worker nodes = 1, I see that the tasks execute fine and in the right sequence
However, when I increase the worker nodes to 2, I see that task1 is picked by worker node 1 and task2 by worker node 2. In my use case, task2 must execute only after task1. I believed that the celery executor will understand this. My dag fails because of this
Can you please me understand what’s wrong and how do I fix this issue?
Thanks
Rathi