Kubernetes Executor: How do I know what Kubernetes Pod my task is running on?

[paola]

In contrast to Celery or Local Executors, the Kubernetes Executor spins up a Pod in your Kubernetes Namespace for each and every task that needs to be executed. Once that task is completed, that pod spins down.

As you’re troubleshooting your DAGs on the Executor, you might ask, “How do I know what pod this one task is running on?” Here are some guidelines:

1. To grab the pods within your Kubernetes Namespace, switch to your deployment’s corresponding namespace:

   • kubens <NAMESPACE>

2. To list all running pods within your namespace, run kubectl get pods

   • You should see a pod for:
     • Your pg-bouncer, scheduler, statsd, and Webserver
   • AND an additional pod for each running task
     • Notice that the pod has dag_id and task_id within into the pod name itself

3. In the Airflow UI, go to your DAG > Graph/Tree View > Task Instances > Hostname

   • The pod name should be listed as the host name for that task