New to Airflow- Need help with BashOperator

Airflow
1

Hi all, I am trying to execute a shell script using BashOperator. The DAG code looks as follows:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator

default_args = {
'owner': 'Airflow',
'depends_on_past': False,
'start_date': datetime(2020, 1, 14),
'email': ['my email is here'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 0,
'retry_delay': timedelta(minutes=5),
# 'queue': 'bash_queue',
# 'pool': 'backfill',
# 'priority_weight': 10,
# 'end_date': datetime(2016, 1, 1),
}

dag = DAG('lex-test', default_args=default_args, schedule_interval=timedelta(days=1))
t1 = BashOperator(
task_id='make-edge-files',
bash_command=' bash /project/lex/data-analysis/scripts/run-edge-creation.sh ',
dag=dag)

The file run-edge-creation.sh exists inside /project/lex/data-analysis/scripts/

It errors out saying

[2020-01-14 14:53:44,219] {bash_operator.py:105} INFO - Temporary script location: /scratch/airflowtmp1wtnzu3v/make-edge-files526zmgrw [2020-01-14 14:53:44,219] {bash_operator.py:115} INFO - Running command: /project/lex/data-analysis/scripts/run-edge-creation.sh [2020-01-14 14:53:44,228] {taskinstance.py:1088} ERROR - [Errno 2] No such file or directory: 'bash': 'bash

If I run this script through terminal, it executes fine. I am new to Airflow, so any pointers would be appreciated! Thank you so much everyone!

2

Hi @shreyapandit- apologies for the delay here!

You shouldn’t include bash in the bash_command arg that you’re passing to your DAG- removing that should fix your issue.

You can find some more documentation on the bash operator here.