How do I install something from my org's private github repo without exposing credentials?


#1

I use private GitHub repos to store custom Python packages for my org. Is there a way for me to add those secrets to my Astronomer image without exposing credentials anywhere?


My own python dependencies?
#2

Adding a note in here from our team. We’re working on integrating this with our CLI, but there is a current workaround for adding secrets to your Astronomer image that’s applicable for private git repos, ssh keys, etc.

Follow the instructions below, and feel free to follow our corresponding internal GitHub issue. We’ll keep it updated as we work towards a long-term solution integrated with our CLI.

1) Put the following in a file called Dockerfile.build:

FROM astronomerinc/ap-airflow:0.6.2 AS stage1
LABEL maintainer="Astronomer <humans@astronomer.io>"
ARG BUILD_NUMBER=-1
LABEL io.astronomer.docker=true
LABEL io.astronomer.docker.build.number=$BUILD_NUMBER
LABEL io.astronomer.docker.airflow.onbuild=true
# Install alpine packages
COPY packages.txt .
RUN cat packages.txt | xargs apk add --no-cache

FROM stage1 AS stage2
RUN mkdir -p /root/.ssh
ARG PRIVATE_RSA_KEY=""
ENV PRIVATE_RSA_KEY=${PRIVATE_RSA_KEY}
RUN echo "${PRIVATE_RSA_KEY}" >> /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN apk update && apk add openssh-client
RUN ssh-keyscan -H github.com >> /root/.ssh/known_hosts
# Install python packages
COPY requirements.txt .
RUN pip install --no-cache-dir -q -r requirements.txt

FROM stage1 AS stage3
# Copy requirements directory
COPY --from=stage2 /usr/lib/python3.6/site-packages/ /usr/lib/python3.6/site-packages/

2) Build the image with:

docker build -f Dockerfile.build --build-arg PRIVATE_RSA_KEY="$(cat ~/.ssh/id_rsa)" -t custom-ap-airflow .

3) Replace the contents of Dockerfile from:

FROM astronomerinc/ap-airflow:latest-onbuild

to

FROM custom-ap-airflow

4) Issue astro airflow start