Where is the best place to initialize 3rd party APIs (to "bootstrap" the Airflow environment)?

Airflow
1

I want to initialize an error tracking service (Rollbar) for use across my DAGs and the utility functions it uses. This involves initializing the client library with my API key before using it:

import rollbar
rollbar.init('my-api-token')

I thought about two ways to go about this:

  1. sticking this in airflow_local_settings.py, or
  2. adding this initialization code into the plugins folder

Neither of the above seem ideal to me, but I can’t find any documentation on what the best practices are around initializing client libraries in Airflow. How do you guys manage this?

Thanks in advance.

2

For anyone stumbling upon this post with the same dilemma, pyrollbar 0.15 has a RollbarHandler that you can attach to your logger, so I decided to go that route.

I made a shared helper function that’s something along these lines:

# logger.py
import logging

def get_logger(name):
    log = logging.getLogger(name)
    rh = RollbarHandler(access_token=<api_token>, environment='production', level=<log_level>)
    log.addHandler(rh)
    return log

Then, in my DAGfiles (or anywhere else):

import logger
log = logger.get_logger(__name__)

... # Business logic here

log.info('Some log message') # Logs to Airflow core logger as well as Rollbar.

It’s still not clear to me if there’s a single best “entrypoint” to the Airflow project, so I can’t speak to a generic solution, but for now, the solution above works for my use case.

1 Like