DAG Missing After Second Deploy

Astronomer Astronomer Nebula
1

I had a DAG running. I needed to hotfix it after some git issues, so I did, but fat-fingered my hotfix. So, when I deployed the first fix, I got an appropriate error telling me what was wrong with the DAG file. I went and hotfixed that (double-checked this time). Then I deployed again, but now the DAG I fixed has an error message saying “DAG <dag_name> seems to be missing”. I can’t seem to find any info on it.

I am not quite sure what to check now. Any suggestions?

2

Hi @jonpie! We ended up sorting this out via a private support ticket with our team, but for anyone else following - the issue here was rooted in an underprovisioned Webserver.

In this case,

  • Webserver for a deployment on Astronomer Cloud was configured to 2AU
  • An underprovisioned Webserver caused unexpected behavior in the Airflow UI, including a message that erroneously signaled a missing DAG (that was confirmed to be in the right directory)
  • Scaling the Webserver to 5AU via the Astronomer UI (and pushing a deploy via the CLI) resolved the issue

What we’re doing to improve the user experience as it related to managing Airflow’s Webserver on Astronomer:

  1. Adjusting the default Webserver configuration on Astronomer Cloud to 5AU
  2. Exposing Webserver/Scheduler/Worker logs in the Astronomer UI (previously unavailable)
  3. Improved documentation on recommended resource configurations on Astronomer at scale

These changes will be incorporated in our upcoming upgrade to Astronomer Cloud. If you’re seeing related issues, feel free to also reference this forum post for more detail on symptoms and other potential fixes.

To reach our technical support team, please reach out to us at support@astronomer.io or submit a request at support.astronomer.io