I am running a pretty simple workflow about sentiment analysis of tweets. I am comparing the 3 different executors: local, celery and Kubernetes. I noticed, that Kuberenetes is just really really slow. So why should I use Kubernetes executor? Is it more suitable for high computing workflows or more for simple workflows?
Btw, I have the free trial of Astronomer, so I can’t change the resources.
The reason why your dagrun may take longer is because Airflow will spin up and down pods for the execution of task instances. If your namespace does not have enough resources to accommodate the tasks concurrently, Airflow will have to wait until there is resources available.
I would recommend reading our guide on Airflow executors. It will explain the pros and cons of each executor.
I would suggest starting with CeleryExecutor if the majority of your tasks does not need a different environments to run in.
