Is network or disk I/O severely limited?

Astronomer Astronomer Nebula
1

Hi all,

Today I have a dag with 1 task in it running on 1.10.12 with the k8s executor. My extra capacity is set to 1 CPU with 3.75 GB of RAM. I have a custom operator that installs a fairly complex package (DBT to be precise) at runtime, so that my users may specify their version. Downloading and installing the package takes almost 20 full minutes! The behavior has been consistent since we started using Astronomer Cloud 2 months ago, starting with 1.10.10.

I briefly looked at the metrics and it seems that I’m unable to see metrics for the running tasks. I am hoping to first confirm that this is true? Otherwise I am shooting blind trying to determine the issue with my task.

Either way I am hoping to confirm whether or not the network or disk I/O is severely limited? I will almost certainly just install specific supported versions of my packages at build time instead, but it would be nice to know what’s going on so that I may support the dynamic version install.

Thanks,
Joe Nap

2

Hmm. This seems to be something that would require us to dive deeper into your instance and check the state of your pods.

I would recommend submitting this issue through our Zendesk portal so you can provide more context/information to help us investigate.

As for the question, I am not aware of any limitation of network usage or disk i/o.

1 Like
3

Thanks Alan, and sorry for the slow response. As mentioned, I will work around it for now. Thanks for your help and I’ll explore further tech assistance if needed.