Error trying to migrate existing project into Astronomer

Astronomer Astronomer Nebula
1

image
image1319×572 123 KB

I ran astro dev init from an existing repository, but cannot run astro dev start to launch the scheduler & webserver. I’m on the 14 day free trial and this is the first deployment I am trying to run. I had an existing airflow project & github repo, and ran these two commands from within the directory for my existing project.

I’m lost on how to resolve this error. Any help is appreciated

Edit: seems like an error on my end with pip and some python libraries, and not an astronomer error…

Edit2: it is not an error with pip. When I emptied the requirements.txt file, then astro dev start successfully ran… so it’s something with my requirements.txt, which looks like:

gcsfs==0.6.1
SQLAlchemy==1.3.15
google-cloud-bigquery
pymongo
flatten-dict
bigquery_schema_generator
backoff

Edit3: I’ve narrowed the issue down to the google-cloud-bigquery library. This error is not thrown when I remove google-cloudbigquery` from the requirements.txt file.

Edit4: I copy / pasted the packages here - https://github.com/astronomer/airflow-guides/blob/main/example_code/gcp/example_code/packages.txt, into the packages.txt file text. It seems to have maybe helped? Now, the terminal is sort of hanging though, providing the following errors after installing packages:

...
(24/26) Installing openblas-ilp64 (0.3.6-r0)
(25/26) Installing openblas-dev (0.3.6-r0)
(26/26) Installing python3-dev (3.7.7-r1)
Executing busybox-1.30.1-r3.trigger
OK: 581 MiB in 95 packages
Removing intermediate container 1615ce6eff46
 ---> Running in 1168f1029f9e
ERROR: google-cloud-container 2.1.0 has requirement grpc-google-iam-v1<0.13dev,>=0.12.3, but you'll have grpc-google-iam-v1 0.11.4 which is incompatible.
ERROR: google-api-python-client 1.12.3 has requirement httplib2<1dev,>=0.15.0, but you'll have httplib2 0.9.2 which is incompatible.
ERROR: astroid 2.4.2 has requirement lazy-object-proxy==1.4.*, but you'll have lazy-object-proxy 1.5.1 which is incompatible.
ERROR: apache-airflow 1.10.5 has requirement flask-appbuilder<2.0.0,>=1.12.5, but you'll have flask-appbuilder 2.3.2 which is incompatible.
1 Like
2
  1. What base image are you using in your Dockerfile
  2. Can you paste your error after Edit3?
3

When I removed google-cloud-platform, the webserver / scheduler launched successfully. I think I need to familiarize more with the files I need to include in the package.txt and requirements.txt files.

1 Like
4

Hey @nick! Sorry to hear you had trouble on $ astro dev init here. Glad you narrowed it down to google-cloud-platform. What were you looking to do? Were you able to add that package back in at all?

5

It looked like google-cloud-platform came installed in the build already. I removed google-cloud-platform from the requirements.txt file, and this solved our problem.