I am trying to follow along this data engineering tutorial and for that I need to use astro cli. Before using the
astro dev start command, I had to enable Docker. In order to use docker, I make use of Colima, which includes docker. My docker specs can be seen below:
Client: Docker Engine - Community
API version: 1.42 (downgraded from 1.43)
Go version: go1.21.3
Git commit: afdd53b4e3
Built: Thu Oct 26 07:06:42 2023
Before running astro/airflow, I first activate colima, which in turns enables docker. After having done that, I run the
astro dev start command, which enables the different containers for my project, and then finally says the following:
Airflow is starting up!
Error: there might be a problem with your project starting up.
The webserver health check timed out after 1m0s but your project
will continue trying to start.
Run 'astro dev logs --webserver | --scheduler' for details.
Try again or use the --wait flag to increase the time out
I also tried to use the --wait flag and set it to 1 hour, but it just runs indefinitely, without any success. So I am not sure what is going here. Any ideas?
If needed, you can find my laptop specs below:
Model Name: MacBook Pro
OS: MacOS BigSur (11.6.5)
Model Identifier: MacBookPro11,3
Processor Name: Quad-Core Intel Core i7
Processor Speed: 2,5 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 6 MB
Hyper-Threading Technology: Enabled
Memory: 16 GB
The Docker file is below:
If someone has any advice then please let me know!
Hey @reyrobs, Welcome to the Astro Community, Glad to see you using the Astro CLI and Sorry for the issue you are experiencing.
Looking at the Error provided, it seems that Colima containers don’t have sufficient memory assigned, and the Containers aren’t starting up.
Could you please try the following commands
-- 1. Make sure Colima is not already running
-- 2. Start Colima with Enough Resources
colima start --cpu 4 --memory 8
-- 3. Make sure the default context is set to colima
docker context use colima
-- 4. Run Astro Dev Start command to start the containers
astro dev start
I hope this works for you!
Wow that actually fixed it! I can now access Airflow Webserver at http://localhost:8080.
However, if I try to access localhost:5432/postgres, it says
This page isn’t working
**localhost** didn’t send any data.
Do you have any idea about this by any chance?
Thanks for the fix anyhow
Hi @reyrobs, Glad to know that Astro CLI worked for you. You can access Airflow UI using http://localhost:8080. The Empty Response that you are getting using Postgres should be okay and not a concern
You can execute the command
astro dev ps to check the status of the containers. More here
In case you want to access the Postgres container you can exec into it using Terminal. For Docker, Go to Docker Container, Click on 3 Dots for Scheduler, and Open in the terminal.
and use the command astro config get / set to get and set the configuration for Postgres if needs to be modified.
Further to access the running Postgres container you can create the following Airflow connection and use it to play around.
Connection ID - postgres_default
Connection Type - Postgres
Host - postgres
login - postgres
password - postgres
port - 5432
I hope this helps!
Thanks for the detailed response. I unfortunately do not have access to Docker Desktop since my Macbook is quite old (2014) and doesnt support the Docker Desktop. I would have to do everything through the terminal. This is actually the reason why I used colima for docker, as it works on my laptop.
Hey @reyrobs, no worries running from the terminal should be good, it was just an example to access containers!
Develop your Astro project is a detailed guide that provides information about adding and organizing Astro project files using Astro CLI, including:
- Adding DAGs
- Adding Python and OS-level packages
- Setting environment variables
- Applying changes
- Running on-build commands
Further, If needed you can use the Astro CLI command astro dev bash to run a bash command into a locally running container.
Also, here are other guides/docs which will be useful for you
With the above you should be all set running Airflow locally using Astro CLI!
I hope this helps!
@iamakashladdha This was answered in detail. Welldone and Thank You.
@iamakashladdha Thanks for the detailed response!