Most AWS services in Airflow (such as the S3Hook) inherit from and build off of the common AWSHook to make connections, which just relies on typical access_keys/secrets.We generally see the aws_access_key_id and aws_secret_access_key used to access AWS resources.
Storing your Connections
You can either:
- Stored these together in a dict in the “Extras” field of your Connection (in the Airflow UI) OR
- You can put the
aws_access_key_idin the login field + theaws_secret_access_keyin the password field where youraws_secret_access_keywill be encrypted at REST but inaccessible from the UI (still present but not shown after initial “Save”).
In both cases, your connection will be [encrypted at REST](where they will be encrypted at REST ), but in the former the secret access key in “Extras” will remain visible in the UI.
Dynamically Updating Connections
Right now, theres’s no “out of the box” way to push connection credentials up to a Cloud deploy with Astronomer, but you’re free to do so locally with airflow_settings.yaml, which lets you define Airflow connections and variables when you run astro airflow start. More info here.
Generally, we see people manually create their connections through the Airflow UI, as they don’t tend to change too often.
We haven’t personally encountered anyone using KMS but it might be worth to check out the modified AWSkmsHook and AWS Secret Manager Hook to do so. To sync these connections to a vault or AWS Secrets manager, you could leverage these hooks to write a DAG that runs every hour or so, connects to that store and syncs up connections.
Note: You’ll then need to also modify the S3, EMR, etc. hooks to inherit from here rather than the typical AWS Hook.
Note: If you’re trying to connect to MySQL, Postgres or AWS Redshift, we pushed an update to those operators to allow IAM auth to those databases in Airflow 1.10.4. Airflow PR here and Jira Issue.