On both my local sandbox and a trial instance on Astronomer Cloud, which both use the Local Executor, DAGs appear to start out on a temporary folder. I was having trouble running a BashOperator with a command like python path/to/script.py but was getting errors that the file did not exist. In the logs I noticed the following: [2021-08-10 03:50:15,052] {subprocess.py:52} INFO - Tmp dir root location: /tmp
I also stumbled upon someone on SO mentioning using the AIRFLOW_HOME environment variable. So I instead updated the bash command to be python ${AIRFLOW_HOME}/path/to/script.py and it worked.
So I’m wondering - should I expect that to happen regardless of executor type? Any reason why this happens?