Deploy 2 dags based on DbtTaskGroup - getting a Dag Import Error

Astronomer
,
1

Hello, all!
I am trying to deploy on Google Composer two Dags, each one has a DbtTaskGroup operator and represents one dbt project. Both are structured very similar, but the name of the projects and profiles. It happens that one is deployed correctly and the other one gets a Dag Import Error.

Please look at the logs below, where I replace custom name with Uppercased Placeholders just to not be specific.

`cosmos.dbt.graph.CosmosLoadDbtException: Unable to run [‘/opt/python3.8/bin/dbt’, ‘ls’, ‘–output’, ‘json’, ‘–select’, ‘+staging’, ‘–project-dir’, ‘/tmp/tmp_0enldiw’, ‘–profiles-dir’, ‘/home/airflow/gcs/dags/PROJECT_NAME/.dbt’, ‘–profile’, ‘MY_PROFILE’, ‘–target’, ‘qa’] due to the error:
e[0m15:16:46 Running with dbt=1.5.9
e[0m15:16:48 Registered adapter: bigquery=1.5.7
e[0m15:16:48 Unable to do partial parsing because saved manifest not found. Starting full parse.
e[0m15:17:01 [e[33mWARNINGe[0m]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:

  • models.PROJECT_NAME.models.staging
    e[0m15:17:01 Found 48 models, 182 tests, 0 snapshots, 0 analyses, 1344 macros, 2 operations, 0 seed files, 18 sources, 0 exposures, 0 metrics, 0 groups
    {“name”: “TABLE_1”, “resource_type”: “model”, “package_name”: “DATASET_1”, “original_file_path”: “models/staging/TABLE_1.sql”, “unique_id”: “model.DATASET_1.TABLE_1”, “alias”: “TABLE_1”, “config”: {“enabled”: true, “alias”: null, “schema”: null, “database”: null, “tags”: , “meta”: {}, “group”: null, “materialized”: “view”, “incremental_strategy”: null, “persist_docs”: {}, “quoting”: {}, “column_types”: {}, “full_refresh”: null, “unique_key”: null, “on_schema_change”: “ignore”, “grants”: {}, “packages”: , “docs”: {“show”: true, “node_color”: null}, “contract”: {“enforced”: false}, “post-hook”: , “pre-hook”: }, “tags”: , “depends_on”: {“macros”: , “nodes”: [“source.DATASET_1.SOURCE_NAME.TABLE_1”]}}
    `

What looks also weird is that in the command line below the project-dir looks wrong, besides I have passed it trought the Operator

dbt_tg = DbtTaskGroup(
        group_id=GROUP_ID,
        project_config=ProjectConfig(DBT_PROJECT_PATH),

May someone help me please?

2

Hey @Jose

Apologies for the delay in replying. Could you please clarify a few things:

  • Airflow version
  • Cosmos version
  • Do you have two dbt projects within the same Airflow

Thanks
Manmeet