Unable to start airflow with astro dev start

Astro CLI
,
1

Hi all,
I am trying to follow along this data engineering tutorial and for that I need to use astro cli. Before using the astro dev start command, I had to enable Docker. In order to use docker, I make use of Colima, which includes docker. My docker specs can be seen below:

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.42 (downgraded from 1.43)
 Go version:        go1.21.3
 Git commit:        afdd53b4e3
 Built:             Thu Oct 26 07:06:42 2023
 OS/Arch:           darwin/amd64
 Context:           colima

Before running astro/airflow, I first activate colima, which in turns enables docker. After having done that, I run the astro dev start command, which enables the different containers for my project, and then finally says the following:

Airflow is starting up!

Error: there might be a problem with your project starting up. 
The webserver health check timed out after 1m0s but your project 
will continue trying to start. 
Run 'astro dev logs --webserver | --scheduler' for details.

Try again or use the --wait flag to increase the time out

I also tried to use the --wait flag and set it to 1 hour, but it just runs indefinitely, without any success. So I am not sure what is going here. Any ideas?

If needed, you can find my laptop specs below:

  Model Name:	MacBook Pro
  OS: MacOS BigSur (11.6.5) 
  Model Identifier:	MacBookPro11,3
  Processor Name:	Quad-Core Intel Core i7
  Processor Speed:	2,5 GHz
  Number of Processors:	1
  Total Number of Cores:	4
  L2 Cache (per Core):	256 KB
  L3 Cache:	6 MB
  Hyper-Threading Technology:	Enabled
  Memory:	16 GB

The Docker file is below:
FROM quay.io/astronomer/astro-runtime:9.4.0

If someone has any advice then please let me know!

2

Hey @reyrobs, Welcome to the Astro Community, Glad to see you using the Astro CLI and Sorry for the issue you are experiencing.

Looking at the Error provided, it seems that Colima containers don’t have sufficient memory assigned, and the Containers aren’t starting up.

Could you please try the following commands

-- 1. Make sure Colima is not already running
colima stop
colima delete

-- 2. Start Colima with Enough Resources
colima start --cpu 4 --memory 8

-- 3. Make sure the default context is set to colima
docker context use colima

-- 4. Run Astro Dev Start command to start the containers
astro dev start

I hope this works for you!

3

Wow that actually fixed it! I can now access Airflow Webserver at http://localhost:8080.
However, if I try to access localhost:5432/postgres, it says

This page isn’t working

**localhost** didn’t send any data.

ERR_EMPTY_RESPONSE

Do you have any idea about this by any chance?

Thanks for the fix anyhow :slight_smile:

4

Hi @reyrobs, Glad to know that Astro CLI worked for you. You can access Airflow UI using http://localhost:8080. The Empty Response that you are getting using Postgres should be okay and not a concern

You can execute the command astro dev ps to check the status of the containers. More here

In case you want to access the Postgres container you can exec into it using Terminal. For Docker, Go to Docker Container, Click on 3 Dots for Scheduler, and Open in the terminal.

image
image2066×856 102 KB

and use the command astro config get / set to get and set the configuration for Postgres if needs to be modified.

Further to access the running Postgres container you can create the following Airflow connection and use it to play around.

Connection ID - postgres_default
Connection Type - Postgres
Host - postgres
login -  postgres
password - postgres
port - 5432

I hope this helps!

5

Hello again,

Thanks for the detailed response. I unfortunately do not have access to Docker Desktop since my Macbook is quite old (2014) and doesnt support the Docker Desktop. I would have to do everything through the terminal. This is actually the reason why I used colima for docker, as it works on my laptop.

6

Hey @reyrobs, no worries running from the terminal should be good, it was just an example to access containers!

Develop your Astro project is a detailed guide that provides information about adding and organizing Astro project files using Astro CLI, including:

  • Adding DAGs
  • Adding Python and OS-level packages
  • Setting environment variables
  • Applying changes
  • Running on-build commands

Further, If needed you can use the Astro CLI command astro dev bash to run a bash command into a locally running container.

Also, here are other guides/docs which will be useful for you

With the above you should be all set running Airflow locally using Astro CLI!

I hope this helps!

1 Like
7

@iamakashladdha This was answered in detail. Welldone and Thank You.

1 Like
8

@iamakashladdha Thanks for the detailed response!

1 Like