Airflow on Kubernetes

HI Guys

I have a architecture related question for Airflow on K8s. I have airflow running using a standard image, and the webserver and workers have the same image.

If I have a requirement to have a specific linux package or python package for the dags, meaning:

  • Dag1 needs linux Package with version 1.5
  • Dag2 needs linux Package with version 1.7 (hypothetical use case)

How can I accomplish this if I am using 1 image of airflow ?

#1) Is it better to use kubernetesPodOPerator for the dag that has out of blue requirement, like a specific package?

#2) Or, would it be better to use k8sPodOPerator for all Dags and install the pythonpackage(like say aws packate) and the required Linux package for each dag at the dag level ?

The disadvantage of #1 is that all package(say python packages) will be installed across all workers, regardless the need.

Any thoughts on this ?

Regards
Tanmay