Using Airflow Contrib operators in Astro docker

Astronomer
1

Hi ,
I downloaded Astro cli and started airflow as noted here : https://github.com/astronomer/astro-cli

When i try use the contrib operator ; it gives an error with import;

Broken DAG: [/usr/local/airflow/dags/list_s3_files.py] No module named 'airflow.contrib.operators.s3_copy_object_operator’

Are we not supposed to use the contrib operators?
this is code line in the dag :
“from airflow.contrib.operators.s3_copy_object_operator import S3CopyObjectOperator”

2

Hello ravi,
Try this instead:
from airflow.providers.amazon.aws.operators.s3_copy_object import S3CopyObjectOperator

It looks like the operators under the contrib folder have been deprecated and moved to the providers folder.

3

Hi Pegomes,
Now the error is " No module named ‘airflow.providers’"
my dag code is below :

from airflow import DAG:

from airflow.providers.amazon.aws.operators.s3_copy_object import S3CopyObjectOperator
from datetime import datetime, timedelta
default_args = {
‘owner’: ‘airflow’,
‘depends_on_past’: False,
‘start_date’: datetime(2020, 4, 14),
‘email_on_failure’: False,
‘email_on_retry’: False,
‘retries’: 1,
‘retry_delay’: timedelta(minutes=5),
}

dag = DAG(‘list_s3_objects’,
max_active_runs=3,
schedule_interval="@once",
default_args=default_args)

t1 = S3CopyObjectOperator(
task_id=‘list_s3_bucket_objects’,
source_bucket_key =“test_csv/out_1.csv”,

dest_bucket_key = "test_ravi/out_1.csv",
source_bucket_name= "some_bucketname1",
dest_bucket_name= "some_bucketname2",
aws_conn_id ='aws_default',
dag=dag

)

t1

4

Hi @ravi - what is inside of your Dockerfile ?

5

Thanks Viraj.
I received mail from Ben about updating cli version ; I will do that first and make sure the dockerfile is pointing to the latest airflow version and try again.
currently! the docker file was showing as :
FROM astronomerinc/ap-airflow:0.8.2-1.10.2-onbuild

6

That’s the issue - it’s pointing to an outdated version our image. Once you update the CLI, can you try changing the Dockerfile to:

FROM astronomerinc/ap-airflow:1.10.7-alpine3.10-onbuild

and rebuild the image (astro dev stop and astro dev start)

7

Hi,
I have removed all the images (docker)

  1. Attempt 1:
    downloaded latest build : Astro CLI Version: 0.12.0
    docker file is showing : FROM astronomerinc/ap-airflow:latest-onbuild
    with this error is "Broken DAG: [/usr/local/airflow/dags/list_s3_files.py] No module named ‘airflow.providers’"

  2. Attempt2: As indicated earlier(by Viraj) ;
    I have updated docker file to the version :
    " FROM astronomerinc/ap-airflow:1.10.7-alpine3.10-onbuild"
    and rebuilt image ( astro dev stop and astro dev start )
    and did the error is the same:

  3. Attempt3:
    a) Removed all the images and containers on docker;
    b) did astro dev start which downloaded the version “astronomerinc/ap-airflow:1.10.7-alpine3.10-onbuild” image
    Same Error "Broken DAG: [/usr/local/airflow/dags/list_s3_files.py] No module named ‘airflow.providers’"

    image
    image1127×130 6.79 KB

    Astro CLI Version: v0.12.0 (2020.03.09)
    Astro CLI Latest: v0.12.0 (2020.03.09)
    You are running the latest version.

8

Hi Ravi,

That airflow.providers file path is not avail in the version of the astronomer image you are running (it’s only on master branch for airflow right now)

from airflow.contrib.operators.s3_copy_object_operator import S3CopyObjectOperator
from airflow.contrib.operators.s3_delete_objects_operator import S3DeleteObjectsOperator

That matches up to the 1.10.7 file path in Airflow:

9

Hi viraj
thanks for the reply;
I changed it to contrib and it worked.

  1. Is Astronomer cloud version and Enterprise version supports “Providers” or “contrib operators”?
  2. how can i pass aws credentials file to docker(windows) ; did n’t find any documentation in forum and your start up guides;
    thanks
10

-You can use any operators on either cloud and enterprise; our support covers airflow itself, not the specific operator. You’ll just have to make sure you are importing them from where the operators exist on that particular branch (e.g. you are running airflow 1.10.7, in which the operators exist in the path airflow.contrib.operators...)

  • Are those separate creds you want to bake in? You can add files to the same directory as the Dockerfile and everything will be baked into the image
1 Like
11

Hi Viraj,
While developing with docker locally ;
we first sign up through okta which gets aws credentials for period of time and those credentails are stored locally on windows machine in under /usr/username/.aws.
how can i get this to map with docker ; dag can work with credentials and connect AWS?

12

Hi Ravi,

The entire directory will be built into your image when you build it, so you should be able to just put credentails inside of that directory. They will by default be found at /usr/loca/airflow

1 Like