Can I use an operator available in 1.10.2 if I'm running an earlier Airflow version?

#1

Let’s say you want to use the latest version of the DatabricksRunNowOperator available in Airflow 1.10.2, but you’re running Astronomer Cloud on Airflow 1.10.1.

If you don’t want to wait for an upgrade, you can always copy the operator (and any dependencies) into a local file in your plugins directory and reference from there.

0 Likes

#2

For example, if you wanted to use the GoogleCloudStorageToBigQueryOperator because the ability to autodetect schema was added in 1.10.2, you could do this

Create a file in the hook and plugin folder and give it a reasonable name.

image

Find the hook and operator you are interested in in github and copy the code into your hook and operator. Update the operator’s import statement to point to this new version of the hook. (I’ve commented out the old import)

# from airflow.contrib.hooks.bigquery_hook import BigQueryHook
from plugins.hooks.bigquery_hook_1_10_2 import BigQueryHook

Now these versions of the hook/operator will be available in your DAG, you just need to use the correct import statements like below.

from airflow import DAG
from plugins.operators.gcs_to_bq_1_10_2 import GoogleCloudStorageToBigQueryOperator

dag = DAG(
    dag_id='my_dag'
    schedule_interval="15 * * * *",
    catchup=False,
    start_date='2019-01-01'
)

gcs_to_bq = GoogleCloudStorageToBigQueryOperator(
            task_id='my_task_id',
            dag=dag,
            bucket='my_bucketGCS_BUCKET',
            source_objects=['my_source_key'],
            destination_project_dataset_table="my_dest_table",
            bigquery_conn_id='my_bq_conn',
            source_format='NEWLINE_DELIMITED_JSON',
            autodetect=True,
            write_disposition='APPEND'
        )
1 Like