Hi,
I am looking for some options to monitor my airflow jobs for,failure,failure logs etc using datadog.
I tried airflow jobs parsing logs in datadog,but that’s not very helpful.
I tried to use pip module airflow-metrics,but installation fails.
Please suggest The ways to monitor airflow.
Hi,
Do you mean with the Astronomer platform or just generic Airflow?
Andrew
Hi Andrew.Thanks for the reply…I mean the Apache airflow,the one provided open source.
OK, here at Astronomer we push the logs to elastic search and layer a Kibana dashboard on top of that. Metrics are scraped by Prometheus then pushed to Grafana. We don’t use datadog, so I’m afraid i’m not much help there. You may want to look into stasd https://airflow.apache.org/docs/stable/metrics.html for more details on how Airflow pushes out metrics