Anyone interested in Github Actions CI/CD?

I’ve been checking out Github’s new Actions CI/CD solution, and looking at https://www.astronomer.io/docs/ci-cd/ it seems like it wouldn’t be too difficult to come up with a GH Action for Astronomer Airflow automated deployment.

Anyone tried this yet or interested in collaborating on a prototype GH action? Or maybe the Astronomer devs interested in working with a motivated user to craft one? I’ve got the GH Actions half in my head, but not as familiar with the Astronomer end of the CI/CD process.

Just putting out feelers.

1 Like

I’d be happy to help. never tried it myself, but just set up one for Git Lab. I bet they are pretty similar. From astro point of view, you basically just need to build your image, then push it to your docker registry. you authenticate with a service account token you create from the Orbit UI. In the doc yo posted, use the Gitlab as an example. It’s the most up to date.

also, once we get it nailed down, we’ll add it to that doc.

Yep, sounds pretty straight forward. And there are already a few published Github Action templates out there doing similar.

I’m currently using https://github.com/marketplace/actions/publish-docker , which pairs with another action to both 1) login to an ECR registry and 2) publish the image to it. This seems like a close enough pattern to follow.

I’ll let you know what I figure out.

1 Like

Love what you’re doing here, @andrewcstewart! Keep us posted.

Ok, turns out this is ridiculously easy to do using the following github action: https://github.com/marketplace/actions/publish-docker

Just create a new action in your repo at .github/workflows/main.yml with the following content:

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest    
    steps:
    - uses: actions/checkout@v1
    - name: Publish to Astronomer.io
      uses: elgohr/Publish-Docker-Github-Action@2.6
      with:
        name: infrared-photon-7780/airflow:ci-${{ github.sha }}
        username: _
        password: ${{ secrets.ASTRO_SERVICE_KEY }}
        registry: registry.astronomer.cloud

Replace infrared-photon-7780 with your deployment name. Also create an astronomer service account and store the key in your github repo’s secrets (as per https://help.github.com/en/articles/virtual-environments-for-github-actions#creating-and-using-secrets-encrypted-variables)

And that should be it!

4 Likes

I had to adjust the value in registry to:
registry.gcp0001.us-east4.astronomer.io
before I could successfully deploy.

1 Like

@jpuffer @andrewcstewart Just pushed this up to our CI/CD doc, with the updated registry listed: https://github.com/astronomer/docs/commit/64b89ae1e60f570845e70a81956dea481a2e8fc7

Appreciate both of your contributions to this - happy this works and can be shared with the rest of our customer base :slight_smile:

What about an integration test step? How can the CI assure that the deployment image isn’t broken?

Currently I’m just running $ pytest plus this smoke test: https://gist.github.com/criccomini/2862667822af7fae8b55682faef029a7

But wondering if there’s an astronomer-native solution I should be aware of too.

1 Like

what is the username here ? should we use it as it is i.e. username: _ or need to change ?

@ andrewcstewart I followed all your mentioned steps. All things worked fine here except one the DAG did not push to the Airflow and I am not able to see any DAG on Airflow.