HI Everybody,
We are using an old airflow version as you were using an htcondor cluster to run the different tasks.
Between the different executor, it seems most of the people are using Spark… Do you have any statistics what the user are using ?
If I understand well, it should change with Kubernetes executor, isn’t it?
Regards,
Thierry
Currently, we do not have any statistics on the distribution of users for the different executors.
Please take a look at the pros and cons of the Airflow Executors that we offer at Astronomer.
I am unaware of a spark executor or do you mean you have mostly tasks that run spark? If that is the case, I would recommend the KubernetesExexutors since you are able to use custom images/environments tailored for the task.