How do I install something from my org's private github repo without exposing credentials?

I use private GitHub repos to store custom Python packages for my org. Is there a way for me to add those secrets to my Astronomer image without exposing credentials anywhere?

August 2022: Astronomer has updated documentation on this here: Install Python packages from private GitHub repositories.

2 Likes

Adding a note in here from our team. We’re working on integrating this with our CLI, but there is a current workaround for adding secrets to your Astronomer image that’s applicable for private git repos, ssh keys, etc.

Follow the instructions below, and feel free to follow our corresponding internal GitHub issue. We’ll keep it updated as we work towards a long-term solution integrated with our CLI.

1) Put the following in a file called Dockerfile.build:

FROM astronomerinc/ap-airflow:0.7.5 AS stage1
LABEL maintainer="Astronomer <humans@astronomer.io>"
ARG BUILD_NUMBER=-1
LABEL io.astronomer.docker=true
LABEL io.astronomer.docker.build.number=$BUILD_NUMBER
LABEL io.astronomer.docker.airflow.onbuild=true
# Install alpine packages
COPY packages.txt .
RUN cat packages.txt | xargs apk add --no-cache

FROM stage1 AS stage2
RUN mkdir -p /root/.ssh
ARG PRIVATE_RSA_KEY=""
ENV PRIVATE_RSA_KEY=${PRIVATE_RSA_KEY}
RUN echo "${PRIVATE_RSA_KEY}" >> /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN apk update && apk add openssh-client
RUN ssh-keyscan -H github.com >> /root/.ssh/known_hosts
# Install python packages
COPY requirements.txt .
RUN pip install --no-cache-dir -q -r requirements.txt

FROM stage1 AS stage3
# Copy requirements directory
COPY --from=stage2 /usr/lib/python3.6/site-packages/ /usr/lib/python3.6/site-packages/
ONBUILD COPY . .

2) Build the image with:

docker build -f Dockerfile.build --build-arg PRIVATE_RSA_KEY="$(cat ~/.ssh/id_rsa)" -t custom-ap-airflow .

3) Replace the contents of Dockerfile from:

FROM astronomerinc/ap-airflow:latest-onbuild

to

FROM custom-ap-airflow

4) Issue astro airflow start

2 Likes

Note: Make sure to replace v0.6.2 in the first line of your Dockerfile.build to match your current Astronomer version.

If you’re on Astronomer v0.7.5, you’ll want to run:

FROM astronomerinc/ap-airflow:0.7.5 AS stage1

Update

This forum post was made into a doc on Astronomer - https://www.astronomer.io/docs/build-from-private-repo/

Any and all feedback appreciated - feel free to still post questions here :slight_smile:

Hi,

I have followed the above mentioned steps and running into following error:

[stage1 5/3] RUN if grep -Eqx ‘apache-airflow\s*[=~>]{1,2}.’ requirements.txt; then echo >&2 “Do not upgrade by specifying ‘apache-airflow’ in your requirements.txt, change the base image instead!”; exit 1; fi; pip install --no-cache-dir -q -r requirements.txt:
#9 0.244 + grep -Eqx 'apache-airflow\s
[=~>]{1,2}.*’ requirements.txt
#9 0.246 + pip install --no-cache-dir -q -r requirements.txt
#9 0.657 ERROR: Error [Errno 2] No such file or directory: ‘git’ while executing command git version
#9 0.657 ERROR: Cannot find command ‘git’ - do you have ‘git’ installed and in your PATH?
#9 1.092 WARNING: You are using pip version 21.3.1; however, version 22.0.4 is available.
#9 1.092 You should consider upgrading via the ‘/usr/local/bin/python -m pip install --upgrade pip’ command.


Could you please advice.

1 Like

I also have this error.

It occurs when adding the suggested line to the requirements.txt file:

git+ssh://git@github.com/<your-github-organization-name>/<your-private-repository>.git

Is there a resolution?

EDIT:

This occurs when running astrocloud dev restart:

dev % astrocloud dev restart
[+] Running 4/0
 ⠿ Container dev_26a49a-webserver-1  Stopped                                                                                                                                                                                        0.0s
 ⠿ Container dev_26a49a-triggerer-1  Stopped                                                                                                                                                                                        0.0s
 ⠿ Container dev_26a49a-scheduler-1  Stopped                                                                                                                                                                                        0.0s
 ⠿ Container dev_26a49a-postgres-1   Stopped                                                                                                                                                                                        0.0s
[+] Building 5.0s (9/10)                                                                                                                                                                                                                 
 => [internal] load build definition from Dockerfile                                                                                                                                                                                0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                   0.0s
 => => transferring context: 34B                                                                                                                                                                                                    0.0s
 => [internal] load metadata for quay.io/astronomer/astro-runtime:5.0.1                                                                                                                                                             0.3s
 => [1/1] FROM quay.io/astronomer/astro-runtime:5.0.1@sha256:3f061f3755d457f11253e57cdc4610a3741ed77af87f60f36224df161ba154ce                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                                                                   0.0s
 => => transferring context: 886B                                                                                                                                                                                                   0.0s
 => CACHED [2/1] COPY packages.txt .                                                                                                                                                                                                0.0s
 => CACHED [3/1] RUN if [[ -s packages.txt ]]; then     apt-get update && cat packages.txt | xargs apt-get install -y --no-install-recommends     && apt-get clean     && rm -rf /var/lib/apt/lists/*;   fi                         0.0s
 => CACHED [4/1] COPY requirements.txt .                                                                                                                                                                                            0.0s
 => ERROR [5/1] RUN if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'apache-airflow' in your requirements.txt, change the base image instead!";  exit 1;   fi;   pi  4.6s
------                                                                                                                                                                                                                                   
 > [5/1] RUN if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'apache-airflow' in your requirements.txt, change the base image instead!";  exit 1;   fi;   pip install --no-cache-dir -q -r requirements.txt:                                                                                                                                                                                                       
#9 0.348 + grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt                                                                                                                                                                    
#9 0.392 + pip install --no-cache-dir -q -r requirements.txt                                                                                                                                                                             
#9 3.264   ERROR: Error [Errno 2] No such file or directory: 'git' while executing command git version                                                                                                                                   
#9 3.265 ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?
#9 4.420 WARNING: You are using pip version 21.3.1; however, version 22.1 is available.
#9 4.420 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/bash -o pipefail -e -u -x -c if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'apache-airflow' in your requirements.txt, change the base image instead!";  exit 1;   fi;   pip install --no-cache-dir -q -r requirements.txt]: exit code: 1
Error: command 'docker build -t dev_26a49a/airflow:latest failed: failed to execute cmd: exit status 1

Hi there @StevePny! Late reply here, but I’m sorry to hear you were running into trouble. We’ve updated our documentation on this over the past few months.

Can you try following the instructions here? Install Python packages from private GitHub repositories.

Let me know if that works.