"ERROR - Exception rendering Jinja template..." in Advanced dbt Tutorial

iolalog · March 2, 2023, 6:42pm

Hi,

I installed Advanced dbt Tutorial on Ubuntu 22.04.2 LTS. It starts, but displays the following errors:

mkdir: cannot create directory '/run/user/0': Permission denied
Authorization required, but no authorization protocol specified
Error: cannot open display: :1

Unable to open the webserver URL, please visit the following link: http://localhost:8080

After logging into the Airflow UI, I get the following error when running all 3 dags (full log in next post):

[2023-03-02, 18:28:12 UTC] {abstractoperator.py:417} ERROR - Exception rendering Jinja template for task 'dbt_seed', field 'env'. Template: {'DBT_USER': '{{ conn.***.login }}', 'DBT_ENV_SECRET_PASSWORD': '***', 'DBT_HOST': '{{ conn.***.host }}', 'DBT_SCHEMA': '{{ conn.***.schema }}', 'DBT_PORT': '{{ conn.***.port }}'}
Traceback (most recent call last):
...

Any ideas what goes wrong?

Regards,
Olav

iolalog · March 2, 2023, 6:43pm

*** Reading local file: /usr/local/airflow/logs/dag_id=dbt_advanced_dag/run_id=manual__2023-03-02T18:28:11.079842+00:00/task_id=dbt_seed/attempt=1.log
[2023-03-02, 18:28:11 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dbt_advanced_dag.dbt_seed manual__2023-03-02T18:28:11.079842+00:00 [queued]>
[2023-03-02, 18:28:11 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dbt_advanced_dag.dbt_seed manual__2023-03-02T18:28:11.079842+00:00 [queued]>
[2023-03-02, 18:28:11 UTC] {taskinstance.py:1362} INFO - 
--------------------------------------------------------------------------------
[2023-03-02, 18:28:11 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2023-03-02, 18:28:11 UTC] {taskinstance.py:1364} INFO - 
--------------------------------------------------------------------------------
[2023-03-02, 18:28:12 UTC] {abstractoperator.py:417} ERROR - Exception rendering Jinja template for task 'dbt_seed', field 'env'. Template: {'DBT_USER': '{{ conn.***.login }}', 'DBT_ENV_SECRET_PASSWORD': '***', 'DBT_HOST': '{{ conn.***.host }}', 'DBT_SCHEMA': '{{ conn.***.schema }}', 'DBT_PORT': '{{ conn.***.port }}'}
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 410, in _do_render_template_fields
    rendered_content = self.render_template(
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 487, in render_template
    return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 487, in <dictcomp>
    return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 474, in render_template
    return render_template_to_string(template, context)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 288, in render_template_to_string
    return render_template(template, cast(MutableMapping[str, Any], context), native=False)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 283, in render_template
    return "".join(nodes)
  File "<template>", line 12, in root
  File "/usr/local/lib/python3.9/site-packages/jinja2/sandbox.py", line 326, in getattr
    value = getattr(obj, attribute)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/context.py", line 112, in __getattr__
    self.var = Connection.get_connection_from_secrets(key)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 432, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `***` isn't defined
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING - Exception in thread Thread-1:
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING - Traceback (most recent call last):
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     self.run()
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/threading.py", line 917, in run
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     self._target(*self._args, **self._kwargs)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/listener.py", line 92, in on_running
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     task_instance_copy.render_templates()
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2193, in render_templates
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     rendered_task = self.task.render_template_fields(context)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 1192, in render_template_fields
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 75, in wrapper
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     return func(*args, session=session, **kwargs)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 410, in _do_render_template_fields
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     rendered_content = self.render_template(
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 487, in render_template
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 487, in <dictcomp>
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 474, in render_template
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     return render_template_to_string(template, context)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 288, in render_template_to_string
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     return render_template(template, cast(MutableMapping[str, Any], context), native=False)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 283, in render_template
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     return "".join(nodes)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "<template>", line 12, in root
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/jinja2/sandbox.py", line 326, in getattr
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     value = getattr(obj, attribute)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/utils/context.py", line 112, in __getattr__
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     self.var = Connection.get_connection_from_secrets(key)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 432, in get_connection_from_secrets
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING - airflow.exceptions.AirflowNotFoundException: The conn_id `***` isn't defined
[2023-03-02, 18:28:12 UTC] {listener.py:27} INFO - TaskInstance Details: dag_id=dbt_advanced_dag, task_id=dbt_seed, dagrun_id=manual__2023-03-02T18:28:11.079842+00:00, map_index=-1, run_start_date=2023-03-02 18:28:11.988804+00:00, try_number=1, job_id=9, op_classpath=airflow.operators.bash.BashOperator
[2023-03-02, 18:28:12 UTC] {taskinstance.py:1383} INFO - Executing <Task(BashOperator): dbt_seed> on 2023-03-02 18:28:11.079842+00:00
[2023-03-02, 18:28:12 UTC] {standard_task_runner.py:54} INFO - Started process 263 to run task
[2023-03-02, 18:28:12 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'dbt_advanced_dag', 'dbt_seed', 'manual__2023-03-02T18:28:11.079842+00:00', '--job-id', '9', '--raw', '--subdir', 'DAGS_FOLDER/dbt_advanced.py', '--cfg-path', '/tmp/tmpdlz5n3w7']
[2023-03-02, 18:28:12 UTC] {standard_task_runner.py:83} INFO - Job 9: Subtask dbt_seed
[2023-03-02, 18:28:12 UTC] {dagbag.py:525} INFO - Filling up the DagBag from /usr/local/airflow/dags/dbt_advanced.py
[2023-03-02, 18:28:12 UTC] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): dbt_seed>, model.jaffle_shop.stg_orders already registered for DAG: dbt_advanced_dag
[2023-03-02, 18:28:12 UTC] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): model.jaffle_shop.stg_orders>, dbt_seed already registered for DAG: dbt_advanced_dag
[2023-03-02, 18:28:12 UTC] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): dbt_seed>, model.jaffle_shop.stg_payments already registered for DAG: dbt_advanced_dag
[2023-03-02, 18:28:12 UTC] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): model.jaffle_shop.stg_payments>, dbt_seed already registered for DAG: dbt_advanced_dag
[2023-03-02, 18:28:12 UTC] {task_command.py:384} INFO - Running <TaskInstance: dbt_advanced_dag.dbt_seed manual__2023-03-02T18:28:11.079842+00:00 [running]> on host 9ae827b0cfe1
[2023-03-02, 18:28:12 UTC] {abstractoperator.py:417} ERROR - Exception rendering Jinja template for task 'dbt_seed', field 'env'. Template: {'DBT_USER': '{{ conn.***.login }}', 'DBT_ENV_SECRET_PASSWORD': '***', 'DBT_HOST': '{{ conn.***.host }}', 'DBT_SCHEMA': '{{ conn.***.schema }}', 'DBT_PORT': '{{ conn.***.port }}'}
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 410, in _do_render_template_fields
    rendered_content = self.render_template(
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 487, in render_template
    return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 487, in <dictcomp>
    return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 474, in render_template
    return render_template_to_string(template, context)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 288, in render_template_to_string
    return render_template(template, cast(MutableMapping[str, Any], context), native=False)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 283, in render_template
    return "".join(nodes)
  File "<template>", line 12, in root
  File "/usr/local/lib/python3.9/site-packages/jinja2/sandbox.py", line 326, in getattr
    value = getattr(obj, attribute)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/context.py", line 112, in __getattr__
    self.var = Connection.get_connection_from_secrets(key)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 432, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `***` isn't defined
[2023-03-02, 18:28:12 UTC] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1457, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1576, in _execute_task_with_callbacks
    task_orig = self.render_templates(context=context)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2193, in render_templates
    rendered_task = self.task.render_template_fields(context)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 1192, in render_template_fields
    self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 410, in _do_render_template_fields
    rendered_content = self.render_template(
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 487, in render_template
    return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 487, in <dictcomp>
    return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 474, in render_template
    return render_template_to_string(template, context)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 288, in render_template_to_string
    return render_template(template, cast(MutableMapping[str, Any], context), native=False)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 283, in render_template
    return "".join(nodes)
  File "<template>", line 12, in root
  File "/usr/local/lib/python3.9/site-packages/jinja2/sandbox.py", line 326, in getattr
    value = getattr(obj, attribute)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/context.py", line 112, in __getattr__
    self.var = Connection.get_connection_from_secrets(key)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 432, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `***` isn't defined
[2023-03-02, 18:28:12 UTC] {taskinstance.py:1401} INFO - Marking task as FAILED. dag_id=dbt_advanced_dag, task_id=dbt_seed, execution_date=20230302T182811, start_date=20230302T182811, end_date=20230302T182812
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING - Exception in thread Thread-2:
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING - Traceback (most recent call last):
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     self.run()
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/threading.py", line 917, in run
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     self._target(*self._args, **self._kwargs)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/listener.py", line 152, in on_failure
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     task_metadata = extractor_manager.extract_metadata(
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/extractors/manager.py", line 28, in extract_metadata
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     extractor = self._get_extractor(task)
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -   File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/extractors/manager.py", line 97, in _get_extractor
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING -     if task.task_id in self.extractors:
[2023-03-02, 18:28:12 UTC] {logging_mixin.py:117} WARNING - AttributeError: 'NoneType' object has no attribute 'task_id'
[2023-03-02, 18:28:12 UTC] {standard_task_runner.py:102} ERROR - Failed to execute job 9 for task dbt_seed (The conn_id `***` isn't defined; 263)
[2023-03-02, 18:28:12 UTC] {local_task_job.py:164} INFO - Task exited with return code 1
[2023-03-02, 18:28:12 UTC] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check

josh-fell · March 3, 2023, 7:17pm

Hi @iolalog!

From the task log it looks like you have not setup the Postgres connection that’s needed with the conn_id of “postgres”. This tutorial demonstrates, in part, how users can define an Airflow Connection and use it within an dbt context rather than storing connection credentials in a dbt project.

This repo specifically can work by connecting to the Postgres metadatabase. (Although of course not recommended to do so in a production environment or anywhere else outside of demo/learning situations.)

The Jinja templating piece is coming from how this particular DAG retrieves the Airflow Connection information to then pass to dbt environment variables:

DBT_ENV = {
    "DBT_USER": "{{ conn.postgres.login }}",
    "DBT_ENV_SECRET_PASSWORD": "{{ conn.postgres.password }}",
    "DBT_HOST": "{{ conn.postgres.host }}",
    "DBT_SCHEMA": "{{ conn.postgres.schema }}",
    "DBT_PORT": "{{ conn.postgres.port }}",
}

Airflow Connections are accessible through Jinja templates with the syntax {{ conn.conn_id.attribute }}. Check out this section specifically for accessing Airflow Connections in templates too.

iolalog · March 4, 2023, 9:18pm

Thanks for the information! I thought the purpose of the Advanced dbt Tutorial would be for me to test the Airflow / dbt interaction without doing configuration / setup myself. Would be good if it was as least mentioned in the readme.

Anyway, I logged into the container, installed dbt-postgres, added environment variables and ran dbt run fine.

In Airflow I setup the postgres connection and retried running the dbt_advanced_dag. Got past the previous issue, but a new one appears. Any ideas? I guess “Could not find adapter type *****!” is the relevant error.

*** Reading local file: /usr/local/airflow/logs/dag_id=dbt_advanced_dag/run_id=manual__2023-03-04T20:56:58.356442+00:00/task_id=dbt_seed/attempt=1.log
[2023-03-04, 20:56:58 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dbt_advanced_dag.dbt_seed manual__2023-03-04T20:56:58.356442+00:00 [queued]>
[2023-03-04, 20:56:58 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: dbt_advanced_dag.dbt_seed manual__2023-03-04T20:56:58.356442+00:00 [queued]>
[2023-03-04, 20:56:58 UTC] {taskinstance.py:1362} INFO - 
--------------------------------------------------------------------------------
[2023-03-04, 20:56:58 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2023-03-04, 20:56:58 UTC] {taskinstance.py:1364} INFO - 
--------------------------------------------------------------------------------
[2023-03-04, 20:56:59 UTC] {manager.py:67} WARNING - Unable to find an extractor. task_type=SerializedBaseOperator airflow_dag_id=dbt_advanced_dag task_id=dbt_seed airflow_run_id=manual__2023-03-04T20:56:58.356442+00:00 
[2023-03-04, 20:56:59 UTC] {factory.py:122} ERROR - Did not find openlineage.yml and OPENLINEAGE_URL is not set
[2023-03-04, 20:56:59 UTC] {factory.py:43} WARNING - Couldn't initialize transport; will print events to console.
[2023-03-04, 20:56:59 UTC] {console.py:22} INFO - {"eventTime": "2023-03-04T20:56:58.972272Z", "eventType": "START", "inputs": [], "job": {"facets": {"documentation": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/DocumentationJobFacet", "description": "A dbt wrapper for Airflow."}}, "name": "dbt_advanced_dag.dbt_seed", "namespace": "default"}, "outputs": [], "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "run": {"facets": {"airflow_runArgs": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet", "externalTrigger": true}, "airflow_version": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet", "airflowVersion": "2.4.0.dev1658+astro.1", "openlineageAirflowVersion": "0.14.1", "operator": "airflow.serialization.serialized_objects.SerializedBaseOperator", "taskInfo": {"_BaseOperator__from_mapped": false, "_BaseOperator__init_kwargs": {"task_id": "dbt_seed"}, "_BaseOperator__instantiated": true, "_dag": {"dag_id": "dbt_advanced_dag", "tags": []}, "_is_empty": false, "_log": "<Logger airflow.task.operators (INFO)>", "_operator_name": "BashOperator", "_task_module": "airflow.operators.bash", "_task_type": "BashOperator", "bash_command": "dbt seed --profiles-dir /usr/local/airflow/dbt --project-dir /usr/local/airflow/dbt", "depends_on_past": false, "do_xcom_push": true, "downstream_task_ids": "{'model.jaffle_shop.stg_payments', 'model.jaffle_shop.stg_orders', 'model.jaffle_shop.stg_customers'}", "email_on_failure": true, "email_on_retry": true, "env": {"DBT_ENV_SECRET_PASSWORD": "***", "DBT_HOST": "***", "DBT_PORT": "5432", "DBT_SCHEMA": "***", "DBT_TYPE": "***", "DBT_USER": "***"}, "executor_config": {}, "ignore_first_depends_on_past": true, "inlets": [], "operator_extra_links": [], "outlets": [], "owner": "airflow", "params": "{}", "pool": "default_pool", "pool_slots": 1, "priority_weight": 1, "queue": "default", "retries": 0, "retry_delay": "0:05:00", "retry_exponential_backoff": false, "start_date": "2020-12-23T00:00:00+00:00", "task_group": "<airflow.serialization.serialized_objects.SerializedTaskGroup object at 0x7f1d91049f70>", "task_id": "dbt_seed", "template_ext": [], "template_fields": [], "template_fields_renderers": {"bash_command": "bash", "env": "json"}, "trigger_rule": "all_success", "ui_color": "#f0ede4", "ui_fgcolor": "#000", "upstream_task_ids": "set()", "wait_for_downstream": false, "weight_rule": "downstream"}}, "nominalTime": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/NominalTimeRunFacet", "nominalStartTime": "2023-03-04T20:56:58.356442Z"}, "parent": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/ParentRunFacet", "job": {"name": "dbt_advanced_dag", "namespace": "default"}, "run": {"runId": "1f9d3c97-2e3c-39f5-8d26-897a4a81c0b1"}}, "parentRun": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/ParentRunFacet", "job": {"name": "dbt_advanced_dag", "namespace": "default"}, "run": {"runId": "1f9d3c97-2e3c-39f5-8d26-897a4a81c0b1"}}, "unknownSourceAttribute": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet", "unknownItems": [{"name": "SerializedBaseOperator", "properties": {"_BaseOperator__from_mapped": false, "_BaseOperator__init_kwargs": {"task_id": "dbt_seed"}, "_BaseOperator__instantiated": true, "_dag": "<<non-serializable: SerializedDAG>>", "_is_empty": false, "_log": "<<non-serializable: Logger>>", "_operator_name": "BashOperator", "_task_module": "airflow.operators.bash", "_task_type": "BashOperator", "bash_command": "dbt seed --profiles-dir /usr/local/airflow/dbt --project-dir /usr/local/airflow/dbt", "depends_on_past": false, "do_xcom_push": true, "downstream_task_ids": [], "email_on_failure": true, "email_on_retry": true, "env": {"DBT_ENV_SECRET_PASSWORD": "***", "DBT_HOST": "***", "DBT_PORT": "5432", "DBT_SCHEMA": "***", "DBT_TYPE": "***", "DBT_USER": "***"}, "executor_config": {}, "ignore_first_depends_on_past": true, "inlets": [], "operator_extra_links": [], "outlets": [], "owner": "airflow", "params": "<<non-serializable: ParamsDict>>", "pool": "default_pool", "pool_slots": 1, "priority_weight": 1, "queue": "default", "retries": 0, "retry_delay": "<<non-serializable: timedelta>>", "retry_exponential_backoff": false, "start_date": "<<non-serializable: DateTime>>", "task_group": "<<non-serializable: SerializedTaskGroup>>", "task_id": "dbt_seed", "template_ext": [], "template_fields": [], "template_fields_renderers": {"bash_command": "bash", "env": "json"}, "trigger_rule": "all_success", "ui_color": "#f0ede4", "ui_fgcolor": "#000", "upstream_task_ids": [], "wait_for_downstream": false, "weight_rule": "downstream"}, "type": "operator"}]}}, "runId": "13ffdb2e-2c83-4146-b13f-607829e59fb4"}}
[2023-03-04, 20:56:59 UTC] {listener.py:27} INFO - TaskInstance Details: dag_id=dbt_advanced_dag, task_id=dbt_seed, dagrun_id=manual__2023-03-04T20:56:58.356442+00:00, map_index=-1, run_start_date=2023-03-04 20:56:58.972272+00:00, try_number=1, job_id=13, op_classpath=airflow.operators.bash.BashOperator
[2023-03-04, 20:56:59 UTC] {taskinstance.py:1383} INFO - Executing <Task(BashOperator): dbt_seed> on 2023-03-04 20:56:58.356442+00:00
[2023-03-04, 20:56:59 UTC] {standard_task_runner.py:54} INFO - Started process 10181 to run task
[2023-03-04, 20:56:59 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'dbt_advanced_dag', 'dbt_seed', 'manual__2023-03-04T20:56:58.356442+00:00', '--job-id', '13', '--raw', '--subdir', 'DAGS_FOLDER/dbt_advanced.py', '--cfg-path', '/tmp/tmpq_53xggm']
[2023-03-04, 20:56:59 UTC] {standard_task_runner.py:83} INFO - Job 13: Subtask dbt_seed
[2023-03-04, 20:56:59 UTC] {dagbag.py:525} INFO - Filling up the DagBag from /usr/local/airflow/dags/dbt_advanced.py
[2023-03-04, 20:56:59 UTC] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): dbt_seed>, model.jaffle_shop.stg_orders already registered for DAG: dbt_advanced_dag
[2023-03-04, 20:56:59 UTC] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): model.jaffle_shop.stg_orders>, dbt_seed already registered for DAG: dbt_advanced_dag
[2023-03-04, 20:56:59 UTC] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): dbt_seed>, model.jaffle_shop.stg_payments already registered for DAG: dbt_advanced_dag
[2023-03-04, 20:56:59 UTC] {taskmixin.py:205} WARNING - Dependency <Task(BashOperator): model.jaffle_shop.stg_payments>, dbt_seed already registered for DAG: dbt_advanced_dag
[2023-03-04, 20:56:59 UTC] {task_command.py:384} INFO - Running <TaskInstance: dbt_advanced_dag.dbt_seed manual__2023-03-04T20:56:58.356442+00:00 [running]> on host b696daca4467
[2023-03-04, 20:56:59 UTC] {taskinstance.py:1590} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=dbt_advanced_dag
AIRFLOW_CTX_TASK_ID=dbt_seed
AIRFLOW_CTX_EXECUTION_DATE=2023-03-04T20:56:58.356442+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-03-04T20:56:58.356442+00:00
[2023-03-04, 20:56:59 UTC] {subprocess.py:63} INFO - Tmp dir root location: 
 /tmp
[2023-03-04, 20:56:59 UTC] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'dbt seed --profiles-dir /usr/local/airflow/dbt --project-dir /usr/local/airflow/dbt']
[2023-03-04, 20:56:59 UTC] {subprocess.py:86} INFO - Output:
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - e[0m20:57:01  Error importing adapter: No module named 'dbt.adapters.*****'
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - e[0m20:57:01  Encountered an error while reading profiles:
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - e[0m20:57:01    ERROR: Runtime Error
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   Credentials in profile "jaffle_shop", target "dev" invalid: Runtime Error
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     Could not find adapter type *****!
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - e[0m20:57:01  Defined profiles:
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - e[0m20:57:01   - jaffle_shop
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - e[0m20:57:01
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - For more information on configuring profiles, please consult the dbt docs:
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - https://docs.getdbt.com/docs/configure-your-profile
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - e[0m20:57:01  Encountered an error:
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - Runtime Error
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   Could not run dbt
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - e[0m20:57:01  Traceback (most recent call last):
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 62, in load_plugin
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     mod: Any = import_module("." + name, "dbt.adapters")
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     return _bootstrap._gcd_import(name[level:], package, level)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - ModuleNotFoundError: No module named 'dbt.adapters.*****'
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - During handling of the above exception, another exception occurred:
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - Traceback (most recent call last):
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 181, in _credentials_from_profile
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     cls = load_plugin(typename)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 207, in load_plugin
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     return FACTORY.load_plugin(name)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 68, in load_plugin
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     raise RuntimeException(f"Could not find adapter type {name}!")
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - dbt.exceptions.RuntimeException: Runtime Error
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   Could not find adapter type *****!
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - The above exception was the direct cause of the following exception:
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - Traceback (most recent call last):
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/task/base.py", line 108, in from_args
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     config = cls.ConfigType.from_args(args)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/config/runtime.py", line 226, in from_args
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     project, profile = cls.collect_parts(args)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/config/runtime.py", line 202, in collect_parts
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     profile = cls._get_rendered_profile(args, profile_renderer, profile_name)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/config/runtime.py", line 187, in _get_rendered_profile
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     return Profile.render_from_args(args, profile_renderer, profile_name)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 428, in render_from_args
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     return cls.from_raw_profiles(
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 394, in from_raw_profiles
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     return cls.from_raw_profile_info(
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 348, in from_raw_profile_info
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     credentials: Credentials = cls._credentials_from_profile(
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 187, in _credentials_from_profile
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     raise DbtProfileError(
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - dbt.exceptions.DbtProfileError: Runtime Error
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   Credentials in profile "jaffle_shop", target "dev" invalid: Runtime Error
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     Could not find adapter type *****!
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - The above exception was the direct cause of the following exception:
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - Traceback (most recent call last):
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/main.py", line 129, in main
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     results, succeeded = handle_and_check(args)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/main.py", line 191, in handle_and_check
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     task, res = run_from_args(parsed)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/main.py", line 218, in run_from_args
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     task = parsed.cls.from_args(args=parsed)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/task/base.py", line 185, in from_args
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     return super().from_args(args)
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/task/base.py", line 131, in from_args
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -     raise dbt.exceptions.RuntimeException("Could not run dbt") from exc
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - dbt.exceptions.RuntimeException: Runtime Error
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO -   Could not run dbt
[2023-03-04, 20:57:01 UTC] {subprocess.py:93} INFO - 
[2023-03-04, 20:57:01 UTC] {subprocess.py:97} INFO - Command exited with return code 2
[2023-03-04, 20:57:01 UTC] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/operators/bash.py", line 196, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 2.
[2023-03-04, 20:57:01 UTC] {taskinstance.py:1401} INFO - Marking task as FAILED. dag_id=dbt_advanced_dag, task_id=dbt_seed, execution_date=20230304T205658, start_date=20230304T205658, end_date=20230304T205701
[2023-03-04, 20:57:01 UTC] {manager.py:67} WARNING - Unable to find an extractor. task_type=SerializedBaseOperator airflow_dag_id=dbt_advanced_dag task_id=dbt_seed airflow_run_id=manual__2023-03-04T20:56:58.356442+00:00 
[2023-03-04, 20:57:01 UTC] {console.py:22} INFO - {"eventTime": "2023-03-04T20:57:01.774954Z", "eventType": "FAIL", "inputs": [], "job": {"facets": {}, "name": "dbt_advanced_dag.dbt_seed", "namespace": "default"}, "outputs": [], "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "run": {"facets": {"unknownSourceAttribute": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.14.1/integration/airflow", "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet", "unknownItems": [{"name": "SerializedBaseOperator", "properties": {"_BaseOperator__from_mapped": false, "_BaseOperator__init_kwargs": {"task_id": "dbt_seed"}, "_BaseOperator__instantiated": true, "_dag": "<<non-serializable: SerializedDAG>>", "_is_empty": false, "_log": "<<non-serializable: Logger>>", "_operator_name": "BashOperator", "_task_module": "airflow.operators.bash", "_task_type": "BashOperator", "bash_command": "dbt seed --profiles-dir /usr/local/airflow/dbt --project-dir /usr/local/airflow/dbt", "depends_on_past": false, "do_xcom_push": true, "downstream_task_ids": [], "email_on_failure": true, "email_on_retry": true, "env": {"DBT_ENV_SECRET_PASSWORD": "***", "DBT_HOST": "***", "DBT_PORT": "5432", "DBT_SCHEMA": "***", "DBT_TYPE": "***", "DBT_USER": "***"}, "executor_config": {}, "ignore_first_depends_on_past": true, "inlets": [], "operator_extra_links": [], "outlets": [], "owner": "airflow", "params": "<<non-serializable: ParamsDict>>", "pool": "default_pool", "pool_slots": 1, "priority_weight": 1, "queue": "default", "retries": 0, "retry_delay": "<<non-serializable: timedelta>>", "retry_exponential_backoff": false, "start_date": "<<non-serializable: DateTime>>", "task_group": "<<non-serializable: SerializedTaskGroup>>", "task_id": "dbt_seed", "template_ext": [], "template_fields": [], "template_fields_renderers": {"bash_command": "bash", "env": "json"}, "trigger_rule": "all_success", "ui_color": "#f0ede4", "ui_fgcolor": "#000", "upstream_task_ids": [], "wait_for_downstream": false, "weight_rule": "downstream"}, "type": "operator"}]}}, "runId": "13ffdb2e-2c83-4146-b13f-607829e59fb4"}}
[2023-03-04, 20:57:01 UTC] {standard_task_runner.py:102} ERROR - Failed to execute job 13 for task dbt_seed (Bash command failed. The command returned a non-zero exit code 2.; 10181)
[2023-03-04, 20:57:01 UTC] {local_task_job.py:164} INFO - Task exited with return code 1
[2023-03-04, 20:57:01 UTC] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check

Regards,
Olav

iolalog · March 4, 2023, 9:52pm

Found it . I had installed dbt-postgres (pip install dbt-postgres) in the webserver. Installed it in the scheduler and trigger servers as well. After that it worked.

Regards,
Olav