I have multiple integration running but for one integration when I extract zip file size of zip is 2-3 gbs, extract and upload each file to our storage.It shows Negsignal.SIGKILL. It was running fine a month ago but now its this error. The file is same always where it fails.
I am using Airflow for Azure Data Factory. When I run it like 1 more time it will get successfully run It seems to be issue with resources but I cant don’t know how much resouces airflow is using and how to increase it.
Is there a way to increase resources in DAG or will adding multiple workers solve the issue?or will I had to add concurrency in case of multiple works.
Dag code is simple we are using PythonOperator