CeleryKubernetesExecutor in Astronomer?

Astronomer
1

Hi,
Airflow currently supports CeleryKubernetesExecutor, is there a plan to add that support in Astronomer too?

CeleryKubernetesExecutor gives you best of both worlds: quick response of CeleryExecutor for small tasks and flexibility of KubernetesExecutor for larger tasks.

My company is using Software version of Astrononomer.

2

Hey @karun.poudel! Thanks for posting. I know your question here is now a few months old, but we had a discussion internally about the CeleryKubernetes Executor that made me remember to reply.

Astro does not currently support the CeleryKubernetes Executor. Here’s a quick breakdown of how we’re thinking about it:

  • The executor on principle sounds really interesting to us. We very much want to deliver the user experience that you’re articulating, where you don’t pick an executor per Deployment — but rather a single executor that can support different types of execution modes and can be optimized per task: the quick response of CeleryExecutor for small tasks and flexibility of KubernetesExecutor for larger tasks.
  • The CeleryKubernetes executor has had lesser-known adoption in the Airflow community and has not been thoroughly tested or documented, so it would take some time for us to get to that level. It also wouldn’t be immediately compatible with our queue framework (See: Worker Queues) in Astro.
  • All of that to say, there’s certainly an opportunity for us to fill this gap, but it will take some time and is not currently committed to. In the meantime, we plan to release support for the Kubernetes Executor.

Hope this answers your question. Happy to chat through this anytime.