Anyone interested in Github Actions CI/CD?

1

I’ve been checking out Github’s new Actions CI/CD solution, and looking at https://www.astronomer.io/docs/ci-cd/ it seems like it wouldn’t be too difficult to come up with a GH Action for Astronomer Airflow automated deployment.

Anyone tried this yet or interested in collaborating on a prototype GH action? Or maybe the Astronomer devs interested in working with a motivated user to craft one? I’ve got the GH Actions half in my head, but not as familiar with the Astronomer end of the CI/CD process.

Just putting out feelers.

1 Like
2

I’d be happy to help. never tried it myself, but just set up one for Git Lab. I bet they are pretty similar. From astro point of view, you basically just need to build your image, then push it to your docker registry. you authenticate with a service account token you create from the Orbit UI. In the doc yo posted, use the Gitlab as an example. It’s the most up to date.

3

also, once we get it nailed down, we’ll add it to that doc.

4

Yep, sounds pretty straight forward. And there are already a few published Github Action templates out there doing similar.

I’m currently using https://github.com/marketplace/actions/publish-docker , which pairs with another action to both 1) login to an ECR registry and 2) publish the image to it. This seems like a close enough pattern to follow.

I’ll let you know what I figure out.

1 Like
5

Love what you’re doing here, @andrewcstewart! Keep us posted.

6

Ok, turns out this is ridiculously easy to do using the following github action: https://github.com/marketplace/actions/publish-docker

Just create a new action in your repo at .github/workflows/main.yml with the following content:

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest    
    steps:
    - uses: actions/checkout@v1
    - name: Publish to Astronomer.io
      uses: elgohr/Publish-Docker-Github-Action@2.6
      with:
        name: infrared-photon-7780/airflow:ci-${{ github.sha }}
        username: _
        password: ${{ secrets.ASTRO_SERVICE_KEY }}
        registry: registry.astronomer.cloud

Replace infrared-photon-7780 with your deployment name. Also create an astronomer service account and store the key in your github repo’s secrets (as per https://help.github.com/en/articles/virtual-environments-for-github-actions#creating-and-using-secrets-encrypted-variables)

And that should be it!

3 Likes
7

I had to adjust the value in registry to:
registry.gcp0001.us-east4.astronomer.io
before I could successfully deploy.

1 Like
8

@jpuffer @andrewcstewart Just pushed this up to our CI/CD doc, with the updated registry listed: https://github.com/astronomer/docs/commit/64b89ae1e60f570845e70a81956dea481a2e8fc7

Appreciate both of your contributions to this - happy this works and can be shared with the rest of our customer base :slight_smile:

9

What about an integration test step? How can the CI assure that the deployment image isn’t broken?

Currently I’m just running $ pytest plus this smoke test: https://gist.github.com/criccomini/2862667822af7fae8b55682faef029a7

But wondering if there’s an astronomer-native solution I should be aware of too.

1 Like
10

what is the username here ? should we use it as it is i.e. username: _ or need to change ?

11

@ andrewcstewart I followed all your mentioned steps. All things worked fine here except one the DAG did not push to the Airflow and I am not able to see any DAG on Airflow.