Not able to connect to Mongo DB from Airflow

1

I have added the connection using airflow UI. but, when I do pymongo.MongoClient(connection_url). It doesn’t connect and dag failed.

2

Hi there @gaurav! Happy to help here. A few questions for you:

  • Can you share your Mongo DB URI, as described in Mongo’s documentation?
  • Can you verify that the values in the connection you set in the Airflow UI accurately correspond to the values in your URI?
  • Can you also verify how you’re calling that connection in your DAG?

Here’s a related forum post, in case it’s helpful: MongoDB Connection

1 Like
3

@paola

  1. My Mongo DB URI is correct.
  2. I have verified it on UI and also through CLI.
  3. The problem is while calling the connect. It is taking the conn_id and not the URI. How can I feed the URI or extract URI based on conn_id?
4

@gaurav When you’re calling the connection in your DAG, you should be calling the Conn Id (the name of the connection when you created it), not the connection URI. Your MongoDB URI is used to populate the fields required to create an Airflow connection to MongoDB, and another Connection URI can be abstracted from the Airflow connection itself, but you shouldn’t need to call either of those URIs in your DAG.

Here’s an example of a Fivetran connection called in a task definition within a DAG:

fivetran_sync_start = FivetranOperator(
    task_id='fivetran-task',
    fivetran_conn_id='fivetran_default',
    connector_id="{{ var.value.get('connector_id') }}",
    dag=dag
)

To connect to Mongo, assuming the name of your Connection in the Airflow UI is mongo_default, you’d have to set the following in your task definition:

mongo_conn_id='mongo_default'

Does that make sense? Here are some additional resources for reference:

Note: To lessen the strain on your Metadata DB, you can also call a connection via an Airflow Environment Variable. Airflow docs here.