Airflow BashOperator - cannot create directory

Airflow
1

I have a ‘create_dir.sh’ script in directory: \wsl.localhost\Ubuntu\home\username\dataeng\airflow\dags\bash_operator\scripts

#!/bin/bash
cd /home/username/dataeng/
mkdir test_dir

DAG file ‘bash_operator_create_directory.py’ is the following:

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "start_date": datetime(2022, 4, 3),
    "email": ["xxx.xxx@gmail.com"],
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5)
}

dag = DAG("bash_operator_create_directory", default_args=default_args, schedule_interval=timedelta(1))
task = BashOperator(task_id="create_directory", bash_command="/home/username/dataeng/airflow/dags/bash_operator/scripts/create_dir.sh ", dag=dag)

After running this task, it’s giving the following error message:

[2022-04-03, 08:48:38 UTC] {subprocess.py:74} INFO - Running command: ['bash', '-c', '/home/username/dataeng/***/dags/bash_operator/scripts/create_dir.sh ']
[2022-04-03, 08:48:38 UTC] {subprocess.py:85} INFO - Output:
[2022-04-03, 08:48:38 UTC] {subprocess.py:89} INFO - bash: /home/username/dataeng/***/dags/bash_operator/scripts/create_dir.sh: No such file or directory
[2022-04-03, 08:48:38 UTC] {subprocess.py:93} INFO - Command exited with return code 127

Then I modified BashOperator task to run mkdir command directly:

task = BashOperator(task_id="create_directory", bash_command="mkdir /home/username/dataeng/test_dir", dag=dag)

After running this task, it’s giving the following error message:

[2022-04-03, 07:51:04 UTC] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'mkdir /home/username/dataeng/test_dir']
[2022-04-03, 07:51:04 UTC] {subprocess.py:85} INFO - Output:
[2022-04-03, 07:51:04 UTC] {subprocess.py:89} INFO - mkdir: cannot create directory ‘/home/username/dataeng/test_dir’: No such file or directory
[2022-04-03, 07:51:04 UTC] {subprocess.py:93} INFO - Command exited with return code 1

Permissions seem to be fine for the create_dir.sh file:

-rwxrwxrwx 1 username root 55 Apr 3 10:27 dags/bash_operator/scripts/create_dir.sh

Any idea what’s wrong?

1 Like
2

How are you running Airflow on your machine?

If you’re running it it Docker or using Astro CLI, you may need to use the path in the container that looks something like /usr/local/airflow/dags rather than using the path on your host machine.

If so, I would try to change the DAG code to:

task = BashOperator(task_id="create_directory", bash_command="/usr/local/airflow/dags/bash_operator/scripts/create_dir.sh ", dag=dag)

and the bash script to:

#!/bin/bash
cd /usr/local/airflow/dags
mkdir test_dir
3

It’s Windows 11 machine. I was using Wsl, Ubuntu and Docker to set up the Airflow environment. And after that I was using the Airflow web UI to trigger a DAG. Now I changed the code as You mentioned but still getting the same error message:

#!/bin/bash
cd /usr/local/airflow/dags
mkdir test_dir

task = BashOperator(task_id="create_directory", bash_command="/usr/local/airflow/dags/bash_operator/scripts/create_dir.sh ", dag=dag)

[2022-04-07, 05:54:52 UTC] {subprocess.py:74} INFO - Running command: ['bash', '-c', '/usr/local/***/dags/bash_operator/scripts/create_dir.sh ']
[2022-04-07, 05:54:52 UTC] {subprocess.py:85} INFO - Output:
[2022-04-07, 05:54:52 UTC] {subprocess.py:89} INFO - bash: /usr/local/***/dags/bash_operator/scripts/create_dir.sh: No such file or directory
[2022-04-07, 05:54:52 UTC] {subprocess.py:93} INFO - Command exited with return code 127
4

Let’s try this:

#!/bin/bash
cd /opt/airflow/dags
mkdir test_dir

task = BashOperator(task_id="create_directory", bash_command="/opt/airflow/dags/bash_operator/scripts/create_dir.sh ", dag=dag)

For the official Airflow image, the default DAG directory will be /opt/airflow/dags.

5

Now it’s fine. Thank You! At first sight it was a little bit confusing that Airflow image and host machine run on different paths.

1 Like