Can I integrate IAM roles with Astronomer?

Astronomer Astronomer Software (Enterprise)
1

As of Astronomer v0.15, IAM roles can now be appended to all pods within any individual Airflow Deployment on the platform.

Overview

IAM roles on AWS, GCP and other platforms are often used to manage the level of access a specific user (or object, or group of users) have to some resource (or set of resources). The resource in question could be an S3 bucket or Secret Backend, both of which are commonly used in tandem with Airflow and Astronomer and can now be configured to be accessible only to a subset of Kubernetes pods within your wider Astronomer cluster.

To support this functionality, each pod in an Astronomer namespace (each of which maps to an Airflow deployment) is launched by a Kubernetes Service Account. By annotating the service account with the role, all pods launched will inherit that role.

At a high-level, you can now:

  1. Create an IAM role defining access to the target service or set of services (e.g. AWS S3)
  2. Create or update an existing Astronomer deployment with that appended IAM role
  3. Confirm your deployment was annotated successfully by running kubectl describe po on any new pod created within that namespace

A few clarifying notes:

  • ALL Kubernetes pods within your Astronomer Deployment (Scheduler, Webserver, Workers) will assume the IAM role. There is currently no way to run more than 1 IAM role per deployment.
  • If you’d like your IAM role to apply to more than 1 deployment, you must annotate each deployment
  • You must use the Astro CLI to pass IAM role annotations
  • Only Workspace Admins can pass IAM role annotations

More specific guidelines below.

Pre-Requisites

  • The Astronomer CLI
  • Your IAM role arn
  • Workspace Admin access
  • Permission to run kubectl describe po in your cluster

Guidelines

  1. Create or update an Airflow Deployment with annotated IAM Role, inserting --cloud-role with the role-arn for the IAM role in question

    • Create: astro deployment create new-deployment-name123 --executor=celery --cloud-role={arn-role}
    • Update: astro deployment update new-deployment-name123 --cloud-role={arn-role}

  2. Confirm the role was passed successfully. You can either:

      1. Run a kubectl describe po/<pod> on any new pod created in your namespace and see Annotations within the output
      1. At the bottom of your config.yaml, you should see the role listed (source code here).
astronomer_houston=# select config from houston$default."Deployment";
                                   config
----------------------------------------------------------------------------
 {"serviceAccountAnnotations":{"eks.amazonaws.com/role-arn":"test-update"}}
(1 row)
2

Update that you can find our official documentation on IAM role integration here: https://www.astronomer.io/docs/ee-iam-integration/