I use private GitHub repos to store custom Python packages for my org. Is there a way for me to add those secrets to my Astronomer image without exposing credentials anywhere?
Adding a note in here from our team. Weâre working on integrating this with our CLI, but there is a current workaround for adding secrets to your Astronomer image thatâs applicable for private git repos, ssh keys, etc.
Follow the instructions below, and feel free to follow our corresponding internal GitHub issue. Weâll keep it updated as we work towards a long-term solution integrated with our CLI.
1) Put the following in a file called Dockerfile.build:
FROM astronomerinc/ap-airflow:0.7.5 AS stage1
LABEL maintainer="Astronomer <humans@astronomer.io>"
ARG BUILD_NUMBER=-1
LABEL io.astronomer.docker=true
LABEL io.astronomer.docker.build.number=$BUILD_NUMBER
LABEL io.astronomer.docker.airflow.onbuild=true
# Install alpine packages
COPY packages.txt .
RUN cat packages.txt | xargs apk add --no-cache
FROM stage1 AS stage2
RUN mkdir -p /root/.ssh
ARG PRIVATE_RSA_KEY=""
ENV PRIVATE_RSA_KEY=${PRIVATE_RSA_KEY}
RUN echo "${PRIVATE_RSA_KEY}" >> /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN apk update && apk add openssh-client
RUN ssh-keyscan -H github.com >> /root/.ssh/known_hosts
# Install python packages
COPY requirements.txt .
RUN pip install --no-cache-dir -q -r requirements.txt
FROM stage1 AS stage3
# Copy requirements directory
COPY --from=stage2 /usr/lib/python3.6/site-packages/ /usr/lib/python3.6/site-packages/
ONBUILD COPY . .
I have followed the above mentioned steps and running into following error:
[stage1 5/3] RUN if grep -Eqx âapache-airflow\s*[=~>]{1,2}.â requirements.txt; then echo >&2 âDo not upgrade by specifying âapache-airflowâ in your requirements.txt, change the base image instead!â; exit 1; fi; pip install --no-cache-dir -q -r requirements.txt: #9 0.244 + grep -Eqx 'apache-airflow\s[=~>]{1,2}.*â requirements.txt #9 0.246 + pip install --no-cache-dir -q -r requirements.txt #9 0.657 ERROR: Error [Errno 2] No such file or directory: âgitâ while executing command git version #9 0.657 ERROR: Cannot find command âgitâ - do you have âgitâ installed and in your PATH? #9 1.092 WARNING: You are using pip version 21.3.1; however, version 22.0.4 is available. #9 1.092 You should consider upgrading via the â/usr/local/bin/python -m pip install --upgrade pipâ command.
Hi there @StevePny! Late reply here, but Iâm sorry to hear you were running into trouble. Weâve updated our documentation on this over the past few months.