Define gcp connections in `airflow_settings.yaml`?

I did this

    - conn_id: google_cloud_default
      conn_type: google_cloud_platform
      project_id: test_project
      keyfile_json: 'xxxxx'

then astro dev start without issues. I could see the connection google_cloud_default in Admin>Connections but somehow the “Project ID” field is not populated. Anything wrong with the yaml above? I was thinking I may have used a wrong key on “project_id”, but can’t be sure - this spec here isn’t super helpful either.

1 Like

Hi @shawnx! You should be able to add a conn_extra field and then the project_id under it. Here’s an example of what your airflow_settings.yaml might look like:

    - conn_id:
        extra__google_cloud_platform__project: test_project

Note: conn_extra should be a string and will be JSON encoded.

Here’s an example of how you might do it via the Airflow CLI:

❯ airflow connections --add --conn_id example_gcp --conn_type google_cloud_platform --conn_extra 'extra__google_cloud_platform__key_path=%2Fkeys%2Fkey.json&extra__google_cloud_platform__scope=https%3A%2F%[]('

successfully added `conn_id`=example_gcp : google_cloud_platform://:******@:

Want to give it a shot and let us know if this works?

Thanks paola, this snippet below doesn’t work.

    - conn_id: google_cloud_default
      conn_type: google_cloud_platform
      keyfile_json: '{....}'
        extra__google_cloud_platform__project: test_project

No error complaints on >astro dev start, and I can see the connection in airflow UI, but when I dig into the setting in “Admin > Connections”, I see “Project Id” still empty. Anything I did wrong?

Also, (maybe a n00b question) how did you get into airflow CLI and be able to run >airflow connections --add ...? Thanks

I’m wondering this same thing. I’m working through this right now, so any guidance would be greatly appreciated!

Hello, you need to put all as JSON under conn_extra.
Make sure you put the JSON between quotes and have double quotes inside the JSON.
If you put the keyfile_dict value, you also need to put it as JSON as one line- escape every quote (") and other characters like New Line (\n)

    - conn_id: "google_cloud_default"
      conn_type: "google_cloud_platform"
      scheme: "google-cloud-platform"
      conn_extra: '{"extra__google_cloud_platform__project":"gcp-project","extra__google_cloud_platform__scope": "","extra__google_cloud_platform__keyfile_dict":"{\"type\":\"service_account\", ... }"}'

Inside conn_extra the content looks like this, but it needs to be ONE LINE

"extra__google_cloud_platform__scope": "",
"extra__google_cloud_platform__keyfile_dict":"{\"type\":\"service_account\", ... }"

The documentation is pretty bad, got the format from the code (here and here)