We use datadog to monitor much of our infrastructure. Often a sidecar datadog agent is able to provide a statsd daemon to receive metrics and forward them on.
Is there a way to do similar with our astronomer cloud deployment? Could we add another container to our deployment?
The alternative I am planning to go with if there is not already a known better approach is to use a custom entrypoint script that will spawn a datadog agent in the background. Something along these lines: https://docs.docker.com/config/containers/multi-service_container.
However, I generally avoid running two processes in a single container and hope there is a good alternative.
There is an official datadog integration for airflow:
I raised a support ticket to ask when this would be available for Astronomer and the reply (May 2020):
Unfortunately at this moment we do not have support for Datadog integration with Astronomer platform, but this is something we are looking to have in near future.
Perhaps with more interest they will raise the priority of this request.
Cheers
+1 on this topic. Would be really nice to see either a working config for the Airflow integration or a separate integration for Astronomer. If it’s already using statsd it seems like the metrics could be remapped per the Datadog Ariflow integration docs into the dogstatsd.