HI Guys
I have a architecture related question for Airflow on K8s. I have airflow running using a standard image, and the webserver and workers have the same image.
If I have a requirement to have a specific linux package or python package for the dags, meaning:
- Dag1 needs linux Package with version 1.5
- Dag2 needs linux Package with version 1.7 (hypothetical use case)
How can I accomplish this if I am using 1 image of airflow ?
#1) Is it better to use kubernetesPodOPerator for the dag that has out of blue requirement, like a specific package?
#2) Or, would it be better to use k8sPodOPerator for all Dags and install the pythonpackage(like say aws packate) and the required Linux package for each dag at the dag level ?
The disadvantage of #1 is that all package(say python packages) will be installed across all workers, regardless the need.
Any thoughts on this ?
Regards
Tanmay