Packages.yml not installed

Cosmos
,
1

Hi all,

I’m having hard time setting up my first dbt-core taskgroup with Airflow. I’m using the virtualenv approach when running my dag:

dbt_tg = DbtTaskGroup(
    dbt_project_name="bq_test",
    dbt_root_path = "/opt/airflow/dbt",
    conn_id="google-cloud-" + ENV,
    execution_mode="virtualenv",
    operator_args={
        "project_dir": "/opt/airflow/dbt/bq_test",
        "py_system_site_packages": False,
        "py_requirements": ["dbt-core==1.5.3", "dbt-postgres==1.5.3", "dbt-bigquery", "pandas", "matplotlib", "seaborn"],
    },
    profile_args={ 
        "dataset" : "TestDataset",
        "project" : "project-test"
    },
)

The virtualenv gets created, but I keep getting the following error:

[2023-07-20, 19:52:48 UTC] {subprocess.py:66} INFO - Running command: ['/tmp/cosmos-venvrjf3b7d0/bin/dbt', 'run', '--models', 'distributors_staging', '--profile', 'bq_test', '--target', 'cosmos_target']
[2023-07-20, 19:52:48 UTC] {subprocess.py:77} INFO - Output:
[2023-07-20, 19:52:58 UTC] {subprocess.py:87} INFO - e[0m19:52:58  Running with dbt=1.5.3
[2023-07-20, 19:53:06 UTC] {subprocess.py:87} INFO - e[0m19:53:06  Registered adapter: bigquery=1.5.3
[2023-07-20, 19:53:06 UTC] {subprocess.py:87} INFO - e[0m19:53:06  Encountered an error:
[2023-07-20, 19:53:06 UTC] {subprocess.py:87} INFO - Compilation Error
[2023-07-20, 19:53:06 UTC] {subprocess.py:87} INFO -   dbt found 1 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_packages. Run "dbt deps" to install package dependencies.
[2023-07-20, 19:53:08 UTC] {subprocess.py:91} INFO - Command exited with return code 2
[2023-07-20, 19:53:08 UTC] {taskinstance.py:1847} ERROR - Task failed with exception

My packages.yml file is in the root of the project:

image
image323×541 8.66 KB

The content of the file is:

packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.1

Any hints would be appreciated why cosmos can not install the package despite finding and reading the packages.yml

2

Hey @gkerekes

Have you tried using the install_deps option within the dbt_args parameter?

For example:

dbt_tg = DbtTaskGroup(
    dbt_project_name="bq_test",
    dbt_root_path = "/opt/airflow/dbt",
    conn_id="google-cloud-" + ENV,
    execution_mode="virtualenv",
    operator_args={
        "project_dir": "/opt/airflow/dbt/bq_test",
        "py_system_site_packages": False,
        "py_requirements": ["dbt-core==1.5.3", "dbt-postgres==1.5.3", "dbt-bigquery", "pandas", "matplotlib", "seaborn"],
    },
    dbt_args={
        "install_deps": True,
    },
    profile_args={ 
        "dataset" : "TestDataset",
        "project" : "project-test"
    },
)

This will run dbt deps before running other dbt commands.

Let us know how it goes! :slight_smile:

Thanks
Manmeet

3

Hello, manmeet,

Thanks for the reply. I am facing the same problems described by gkerekes, the only difference is that I have more packages. When I go to the folder location, the files are there, so I’m not sure why cosmos cannot find them in the first place.

I tried to include the dbt_args you mentioned, but it seems it was not found in DbtTaskGroup. I thought I might be using the wrong version, but I double checked, and the version I have is astronomer-cosmos==1.2.5.

I am running cosmos inside a Docker image, as described here. I’ve built the custom image myself, and cosmos was added to the system.

In terms of dbt, this is the version I am using:
dbt-bigquery==1.7.2
dbt-core==1.7.3

Could you please help me out? Thanks!

4

I solved this somehow but can’t remember how… let me have a look

5 
with dag:
    dbt_tg = DbtTaskGroup(
            project_config=projectConfig,
            profile_config=profileConfig,
            execution_config=ExecutionConfig(
                dbt_executable_path="/opt/airflow/dbt_venv/bin/dbt",
            ),
            operator_args={
                "install_deps": True
            }
    )

this worked for me

2 Likes
6

Thanks! It worked for me as well :slight_smile: