Adding dependencies to image

1

According to the customizing-your-image doc, if I add a folder in project root it will be added to docker image.

In the same way, you can add a folder of helper_functions (or any other files for your DAGs to use) to build into your image. To do so, just add the folder into your project directory and rebuild your image.

But this does not work, at least with CLI version 0.7.5.2.

Am I doing something wrong? I tried building with both astro airflow start and with docker build ..

2

This seems to work for me. Here are the steps I used. can you try the same?

mkdir test
cd test
astro airflow init
mkdir my_folder
touch my_folder/my_file.txt
astro airflow start
docker exec -it test_scheduler_1 bash

after these steps, i see my_file.txt in my_folder

3

Thanks for the reply @AndrewHarmon.

Clearly I am screwing something up.

I am trying to use python3.7 and using dockerfile below. The only changes are (1) switched alpine from 3.9 to 3.10 and (2) I replaced references to python 3.6 with 3.7.

Here’s my dockerfile.

Any advice?

#
# Copyright 2019 Astronomer Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

FROM alpine:3.10 as my-astro-3.7
LABEL maintainer="Astronomer <humans@astronomer.io>"

ARG BUILD_NUMBER=-1
LABEL io.astronomer.docker.build.number=$BUILD_NUMBER
LABEL io.astronomer.docker=true
LABEL io.astronomer.docker.module="airflow"
LABEL io.astronomer.docker.component="airflow"
LABEL io.astronomer.docker.airflow.version="1.10.3"

ARG ORG="astronomer"
ARG VERSION="1.10.3-1"
ARG SUBMODULES="all, statsd, elasticsearch"

ENV AIRFLOW_REPOSITORY="https://github.com/${ORG}/airflow"
ENV AIRFLOW_MODULE="git+${AIRFLOW_REPOSITORY}@${VERSION}#egg=apache-airflow[${SUBMODULES}]"
ENV AIRFLOW_HOME="/usr/local/airflow"
ENV PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1
ENV PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}${AIRFLOW_HOME}
ENV AIRFLOW_GPL_UNIDECODE="True"

ARG ASTRONOMER_USER="astro"
ARG ASTRONOMER_GROUP="astro"
ENV ASTRONOMER_USER=${ASTRONOMER_USER}
ENV ASTRONOMER_GROUP=${ASTRONOMER_GROUP}

RUN addgroup -S ${ASTRONOMER_GROUP} \
	&& adduser -S -G ${ASTRONOMER_GROUP} ${ASTRONOMER_USER}

# Install packages
RUN apk update \
	&& apk add --no-cache --virtual .build-deps \
		build-base \
		cyrus-sasl-dev \
		freetds-dev \
		freetype-dev \
		git \
		krb5-dev \
		libffi-dev \
		libxml2-dev \
		libxslt-dev \
		linux-headers \
		mariadb-dev \
		nodejs \
		nodejs-npm \
		postgresql-dev \
		python3-dev \
		tzdata \
	&& apk add --no-cache \
		bash \
		ca-certificates \
		cyrus-sasl \
		krb5-libs \
		mariadb-connector-c \
		postgresql \
		python3 \
		tini \
	&& update-ca-certificates \
	&& cp /usr/share/zoneinfo/UTC /etc/localtime \
	&& pip3 install --no-cache-dir --upgrade pip==19.1.1 \
	&& pip3 install --no-cache-dir --upgrade setuptools==39.0.1 \
	&& pip3 install --no-cache-dir cython \
	&& pip3 install --no-cache-dir numpy \
	&& pip3 install --no-cache-dir flask==1.0.0 \
	&& pip3 install --no-cache-dir "${AIRFLOW_MODULE}" \
	&& pip3 install --no-cache-dir "https://github.com/astronomer/astronomer-fab-securitymanager/releases/download/v1.1.2/astronomer_fab_security_manager-1.1.2-py3-none-any.whl" \
	&& cd /usr/lib/python3.7/site-packages/airflow/www_rbac \
	&& npm install \
	&& npm run build \
	&& rm -rf node_modules \
	&& apk del .build-deps \
	&& ln -sf /usr/bin/python3 /usr/bin/python \
	&& ln -sf /usr/bin/pip3 /usr/bin/pip

# Create logs directory so we can own it when we mount volumes
RUN mkdir -p ${AIRFLOW_HOME}/logs

# Copy entrypoint to root
COPY include/entrypoint /

# Copy cron scripts
COPY include/clean-airflow-logs /etc/periodic/15min/clean-airflow-logs

# Ensure our user has ownership to AIRFLOW_HOME
RUN chown -R ${ASTRONOMER_USER}:${ASTRONOMER_GROUP} ${AIRFLOW_HOME}

# Switch to AIRFLOW_HOME
WORKDIR ${AIRFLOW_HOME}

# Expose all airflow ports
EXPOSE 8080 5555 8793

# Run airflow with minimal init
ENTRYPOINT ["tini", "--", "/entrypoint"]


FROM my-astro-3.7
LABEL maintainer="Astronomer <humans@astronomer.io>"

ARG BUILD_NUMBER=-1
LABEL io.astronomer.docker=true
LABEL io.astronomer.docker.build.number=$BUILD_NUMBER
LABEL io.astronomer.docker.airflow.onbuild=true

# Install alpine packages
ONBUILD COPY packages.txt .
ONBUILD RUN cat packages.txt | xargs apk add --no-cache

# Install python packages
ONBUILD COPY requirements.txt .
ONBUILD RUN pip install --no-cache-dir -q -r requirements.txt

# Copy entire project directory
ONBUILD COPY . .
4

@AndrewHarmon this is very curious…

If I add an alias to the last build section in above dockerfile, it works.

So where this does not work:

FROM my-astro-3.7
...

This does:

FROM my-astro-3.7 as my-img
...
FROM my-img

I am not sure what’s going on here.

5

Hmm, i’m not entirely sure about the alias. Is there a reason you aren’t starting with one of our astronomer images? Also, technically Python 3.7 isn’t supported by Airflow yet. I think most of the core functionality is ok and most of the hooks/operators, but you may find a few issues as you test.

6

Is there a reason you aren’t starting with one of our astronomer images?

Because just changing alpine:3.9 to alpine:3.10 seemed easier than installing python 3.7 in existing image.

Re issues with 3.7, airflow has been working fine for me on python 3.7 for a few months now.