I want to initialize an error tracking service (Rollbar) for use across my DAGs and the utility functions it uses. This involves initializing the client library with my API key before using it:
I thought about two ways to go about this:
- sticking this in airflow_local_settings.py, or
- adding this initialization code into the plugins folder
Neither of the above seem ideal to me, but I can’t find any documentation on what the best practices are around initializing client libraries in Airflow. How do you guys manage this?
Thanks in advance.
For anyone stumbling upon this post with the same dilemma, pyrollbar 0.15 has a
RollbarHandler that you can attach to your logger, so I decided to go that route.
I made a shared helper function that’s something along these lines:
log = logging.getLogger(name)
rh = RollbarHandler(access_token=<api_token>, environment='production', level=<log_level>)
Then, in my DAGfiles (or anywhere else):
log = logger.get_logger(__name__)
... # Business logic here
log.info('Some log message') # Logs to Airflow core logger as well as Rollbar.
It’s still not clear to me if there’s a single best “entrypoint” to the Airflow project, so I can’t speak to a generic solution, but for now, the solution above works for my use case.