How would a user use the Kubernetes executor to delegate specific tasks in their DAG to its own Kubernetes node pool?
When you’re using the Kubernetes Executor, you can set the executor_config on specific tasks, which allows you to pass a collection of Kubernetes-specific params with the task when it runs. When the k8s executor makes the API call to Kubernetes that spins up a pod for the task, it will include these params.
If you’d like to specify node pools for a task, you can leverage the Kubernetes node affinity feature, which allows you to constrain which nodes your pod is able to run on via labels you place on the node.
If you’d like to achieve the same functionality for python code or arbitrary images, you can use the KubernetesPodOperator. You can pass the PodOperator the image you want to run and the Kubernetes specific params you’d like to include.
1 Like