How to extract runtime of a DBT task via Airflow to then trigger a Datadog notification?

Airflow
1

Adding a question here on behalf of one of our Cloud customers -

Whilst we wait for Datadog support in Astronomer I am trying to extract the run time of a dbt task and report this in a datadog notification. I have tried several approaches without success. My current best guess is the following, but it results in an error with no explanation in Airflow UI. Can someone advise how to achieve the desired notification?

end_dd = DatadogNotification(
    task_id="dbt_end_notification",
    datadog_conn_id="datadog_default",
    msg_title=r'DBT hourly run for {{ execution_date.strftime("%d-%m-%Y %H:%M") }} has Completed Successfully',
    msg=f'Run time: { macros.datetime_diff_for_humans(dag_run.get_task_instance('run_dbt').start_date) } {EnvHandler["dbt_alert_channel"]}',
    msg_tags=get_datadog_tags(),
    dag=dag,
    )
1 Like
2

To update on this one, we went with a simple solution, which is just to use the alert time to indicate very long run times. It is an hourly job so if the alert comes in significantly after we are expecting it, someone can look into it.

end_dd = DatadogNotification(
    task_id="XXX_end_notification",
    msg_title="XXX hourly run has completed successfully",
    msg=f'{{ts}} run completed @channel}',
)
1 Like
3

Glad to hear you found a solution! Thanks for the update on this post @robmarkcole.

1 Like