AWS SSM Parameter Store as Secrets Backend (Airflow 1.10.10+)

Airflow
1

Overview

Airflow 1.10.10 includes feature that allows you to use Environment Variables to sync Airflow Connections + Variables to secrets held in a few different secret backends, including Hashicorp Vault, GCP Secrets Manager and AWS Parameters Store.

On Astronomer’s 1.10.10 image, the following additional backends are included (not yet available in core Airflow’s 1.10.10 release):

  • AWS Secrets Manager
  • AWS Vault

Airflow Instructions: https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html#alternative-secrets-backend

AWS Parameter Store Example

1. Enable with Env Vars

ENV AIRFLOW__SECRETS__BACKEND =airflow.contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend

ENV AIRFLOW__SECRETS__BACKEND_KWARGS
=’{“connections_prefix”: “/airflow/connections”, “variables_prefix”: “/airflow/variables”}’

ENV AWS_DEFAULT_REGION=us-west-2

2A. Set your connection in AWS Parameter Store:

First, set connections_prefix as /airflow/connections. The connection itself has three parts:

  • Connection Name
  • Type
  • Value

param = {

    'Name': '/airflow/connections/example_postgres',

    'Type': 'String',

    'Value': 'postgresql://airflow:airflow@host:5432/airflow'
}

Note: SecureString Parameter Type is not yet supported, only String type.

2B. Supply AWS Profile

To authenticate, you can either supply a profile name to reference aws profile, e.g. defined in ~/.aws/config or set the following Env Vars:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

3. Airflow Hook to pull Connection

Given the example above, using the following in your DAG - BaseHook.get_connection(conn_id="example_postgres") should give a Connection object with:

  • host=host
  • port=5432
  • login=airflow
  • password=airflow
2

Hello Paola, thanks for posting this. I’ve been working on using AWS Secrets manager as the secrets backend. It seems that AWS Param Store and AWS Secrets Manager are similar … but also completely different.

I tried following these instructions for setting up AWS secrets manager .

No joy.

At the time of writing the link above has theSecretsManagerBackend class at airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

I am using astronomerinc/ap-airflow:1.10.10-buster-onbuild and I have the theSecretsManagerBackend class at airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend

I was able to retrieve secrets from AWS Secret Manager with these settings in the dockerfile:

ENV AWS_ACCESS_KEY_ID $AWS_ACCESS_KEY_ID
ENV AWS_SECRET_ACCESS_KEY $AWS_SECRET_ACCESS_KEY
ENV AWS_DEFAULT_REGION=eu-west-1
ENV AIRFLOW__SECRETS__BACKEND="airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend"
ENV AIRFLOW__SECRETS__BACKEND_KWARGS="{"connections_prefix": "/airflow/connections"}"

Within AWS Secrets Manager I stored my connections with name prefix = airflow/connections and with the value using the URI format (https://airflow.readthedocs.io/en/latest/howto/connection/index.html#connection-uri-format):

my-conn-type://my-login:my-password@my-host:my-port-number/my-schema?param1=val1&param2=val2

I retrieved secrets with:

myconn = BaseHook.get_connection(“secret-name”)
print(myconn.conn_type)
print(myconn.login)
print(myconn.schema)

1 Like
3

Hi @abazuaye! Thanks for posting this. Am I understanding correctly that you got AWS Secrets Manager to work then? If not, let me know what specific issues you’re running into and we can see if our team can help.

Also, we’ve updated our official Astronomer documentation with more specific instructions for Hashicorp Vault and AWS SSM Parameter Store here: https://www.astronomer.io/docs/secrets-backend/

Would love to add an “AWS Secrets Manager” section. If you can confirm that you got it to work with the instructions above, that’d be helpful :slightly_smiling_face:

1 Like
4

Hi @paola
yes, the config that I posted above is working for me in Astronomer cloud.
I am now able to retrieve connections from AWS Secret Manager
:+1:t5:

1 Like