How does CI/CD work with the Astronomer environment?

Astronomer Astronomer Software (Enterprise)
1

CI/CD can be easily setup within Astronomer by following the steps in this guide.

2

When I deploy with astro airflow deploy everything goes smoothly, but with Gitlab CI I keep getting an import problem with one of my custom operators.

Broken DAG: [/usr/local/airflow/dags/legacy/redshift/cleanup_datalake.py] No module named 'cleanup_plugin.operators'

My .gitlab-ci.yml is pretty simple:

image: docker:latest

variables:
  DOCKER_DRIVER: overlay2
  BASE_DOMAIN: astronomer.cloud
  DEPLOYMENT_NAME: (ommitted)

services:
  - docker:dind

Build and Push:
  script:
    - docker pull astronomerinc/ap-airflow:0.8.0-alpha.8-1.10.2-onbuild || true
    - docker login registry.$BASE_DOMAIN -u _ -p $API_KEY_SECRET # Add a new Gitlab Environment variables called $API_KEY_SECRET with your service account token
    - docker build --cache-from registry.$BASE_DOMAIN/$DEPLOYMENT_NAME/airflow:gitlab-$CI_COMMIT_SHORT_SHA -t registry.$BASE_DOMAIN/$DEPLOYMENT_NAME/airflow:gitlab-$CI_COMMIT_SHORT_SHA .
    - docker push registry.$BASE_DOMAIN/$DEPLOYMENT_NAME/airflow:gitlab-$CI_COMMIT_SHORT_SHA
4

Hi @edbizarro, sorry for the delay here!

Are there any more logs that you have from there? I’m not super familiar with Gitlab, but can you try using your script on deploying our sample dags?