Vault kv_engine_version 1 configuration is not working in Airflow

Airflow
1

Hello,
My Vault is on kv_engine_version is 1 . So I tried to configure that as follows in airflow.cfg:

backend_kwargs = {"url": "https://vproxy.us-east.**************.com", "kv_engine_version": 1, "auth_type": "approle", "role_id": "d48b422d-f2cb-****-7781-206104fa7367", "secret_id": "ac48613e-*****-8638-125a-dce1e60aa115", "mount_point": "cf/243d677f-dbe3-****-babc-970ce67ebf3b/secret"}

But when I run a DAG which is trying to pull a connection which is stored in vault, I am getting following error:

[2020-10-08 20:07:18,972] {taskinstance.py:900} INFO - Executing <Task(PythonOperator): vault_python_task> on 2020-10-09T00:02:06.044874+00:00
[2020-10-08 20:07:18,976] {standard_task_runner.py:53} INFO - Started process 34072 to run task
[2020-10-08 20:07:19,068] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: tutorial_v1.7.vault_python_task 2020-10-09T00:02:06.044874+00:00 [running]> MACC02Y42HGJG5J
[2020-10-08 20:07:19,449] {taskinstance.py:1145} ERROR - The conn_id s3_connect2 isn’t defined

…

raise AirflowException(“The conn_id {0} isn’t defined”.format(conn_id))
airflow.exceptions.AirflowException: The conn_id s3_connect2 isn’t defined

Note: s3_connect2 is present in vault and I am able to get that value via a python script using hvac library as airflow does internally

When I remove kv_engine_version = 1 and has following:
backend_kwargs = {"url": "https://vproxy.us-east.**************.com", "auth_type": "approle", "role_id": "d48b422d-f2cb-****-7781-206104fa7367", "secret_id": "ac48613e-*****-8638-125a-dce1e60aa115", "mount_point": "cf/243d677f-dbe3-****-babc-970ce67ebf3b/secret"}
Then I am getting following error:

[2020-10-09 10:04:37,011] {logging_mixin.py:112} INFO - [2020-10-09 10:04:37,010] {hashicorp_vault.py:196} INFO - Secret s3_connect2 not found in Path: connections/s3_connect2
[2020-10-09 10:04:37,144] {taskinstance.py:1145} ERROR - The conn_id s3_connect2 isn’t defined

Here I can see that its hitting hashicorp_vault.py but when I have kv_engine_version = 1 , its not even hitting hashicorp_vault.py . So can someone help me on what am I doing wrong?

Apache Airflow version : 1.10.10

Kubernetes version (if you are using kubernetes) (use kubectl version ): No, Local machine (MacOS)

Environment :

  • Cloud provider or hardware configuration :
  • OS (e.g. from /etc/os-release): Mac OS
  • Kernel (e.g. uname -a ):
  • Install tools :
  • Others :

What happened :
getting error for an existing vault secret key:

The conn_id s3_connect2 isn’t defined

What you expected to happen :

expected to return the connection from the vault for the given connection_id

1 Like
2

I got it resolved. But only one help I need is how do I store a ‘aws_default’ connection with Extra field in Vault?
Because when I debugged:
def get_conn_uri(self, conn_id):
# type: (str) -> Optional[str]
“”"
Get secret value from Vault. Store the secret in the form of URI

    :param conn_id: connection id
    :type conn_id: str
    """
    response = self._get_secret(self.connections_path, conn_id)
    return response.get("conn_uri") if response else None

if I dont have “conn_uri” - nothing seems work. So I am wondering how will we store a aws_default connection in vault?