Define gcp connections in `airflow_settings.yaml`?

Astronomer Astronomer Nebula
1

I did this

    ....
    - conn_id: google_cloud_default
      conn_type: google_cloud_platform
      project_id: test_project
      keyfile_json: 'xxxxx'

then astro dev start without issues. I could see the connection google_cloud_default in Admin>Connections but somehow the “Project ID” field is not populated. Anything wrong with the yaml above? I was thinking I may have used a wrong key on “project_id”, but can’t be sure - this spec here isn’t super helpful either.

1 Like
2

Hi @shawnx! You should be able to add a conn_extra field and then the project_id under it. Here’s an example of what your airflow_settings.yaml might look like:

airflow:
  connections:
    - conn_id:
      conn_type:
      conn_host:
      conn_schema: 
      conn_login:
      conn_password:
      conn_port:
      conn_extra:
        extra__google_cloud_platform__project: test_project

Note: conn_extra should be a string and will be JSON encoded.

Here’s an example of how you might do it via the Airflow CLI:

❯ airflow connections --add --conn_id example_gcp --conn_type google_cloud_platform --conn_extra 'extra__google_cloud_platform__key_path=%2Fkeys%2Fkey.json&extra__google_cloud_platform__scope=https%3A%2F%[2Fwww.googleapis.com](http://2fwww.googleapis.com/)%2Fauth%2Fcloud-platform&extra__google_cloud_platform__project=airflow&extra__google_cloud_platform__num_retries=5'

successfully added `conn_id`=example_gcp : google_cloud_platform://:******@:

Want to give it a shot and let us know if this works?

3

Thanks paola, this snippet below doesn’t work.

airflow:
  connections:
    - conn_id: google_cloud_default
      conn_type: google_cloud_platform
      keyfile_json: '{....}'
      conn_extra:
        extra__google_cloud_platform__project: test_project

No error complaints on >astro dev start, and I can see the connection in airflow UI, but when I dig into the setting in “Admin > Connections”, I see “Project Id” still empty. Anything I did wrong?

Also, (maybe a n00b question) how did you get into airflow CLI and be able to run >airflow connections --add ...? Thanks

4

I’m wondering this same thing. I’m working through this right now, so any guidance would be greatly appreciated!

5

Hello, you need to put all as JSON under conn_extra.
Make sure you put the JSON between quotes and have double quotes inside the JSON.
If you put the keyfile_dict value, you also need to put it as JSON as one line- escape every quote (") and other characters like New Line (\n)

airflow:
  connections:
    - conn_id: "google_cloud_default"
      conn_type: "google_cloud_platform"
      scheme: "google-cloud-platform"
      conn_extra: '{"extra__google_cloud_platform__project":"gcp-project","extra__google_cloud_platform__scope": "https://www.googleapis.com/auth/cloud-platform","extra__google_cloud_platform__keyfile_dict":"{\"type\":\"service_account\", ... }"}'

Inside conn_extra the content looks like this, but it needs to be ONE LINE

"extra__google_cloud_platform__project":"gcp-project",
"extra__google_cloud_platform__scope": "https://www.googleapis.com/auth/cloud-platform",
"extra__google_cloud_platform__keyfile_dict":"{\"type\":\"service_account\", ... }"

The documentation is pretty bad, got the format from the code (here and here)