Can I use an operator available in 1.10.2 if I'm running an earlier Airflow version?

Astronomer
1

Let’s say you want to use the latest version of the DatabricksRunNowOperator available in Airflow 1.10.2, but you’re running Astronomer Cloud on Airflow 1.10.1.

If you don’t want to wait for an upgrade, you can always copy the operator (and any dependencies) into a local file in your plugins directory and reference from there.

2

For example, if you wanted to use the GoogleCloudStorageToBigQueryOperator because the ability to autodetect schema was added in 1.10.2, you could do the following:

  1. Create a file in the hook and plugin folder. Give it a reasonable name.

image

  1. Find the hook and operator you are interested in in GitHub

  2. Copy the code into your hook and operator

  3. Update the operator’s import statement to point to this new version of the hook (I’ve commented out the old import):

# from airflow.contrib.hooks.bigquery_hook import BigQueryHook
from plugins.hooks.bigquery_hook_1_10_2 import BigQueryHook

Now, these versions of the hook/operator will be available in your DAG. From here, make sure to use the correct import statements like below:

from airflow import DAG
from plugins.operators.gcs_to_bq_1_10_2 import GoogleCloudStorageToBigQueryOperator

dag = DAG(
    dag_id='my_dag'
    schedule_interval="15 * * * *",
    catchup=False,
    start_date='2019-01-01'
)

gcs_to_bq = GoogleCloudStorageToBigQueryOperator(
            task_id='my_task_id',
            dag=dag,
            bucket='my_bucketGCS_BUCKET',
            source_objects=['my_source_key'],
            destination_project_dataset_table="my_dest_table",
            bigquery_conn_id='my_bq_conn',
            source_format='NEWLINE_DELIMITED_JSON',
            autodetect=True,
            write_disposition='APPEND'
        )
1 Like