KubernetesPodOperator volume mount with windows base OS

Airflow
1

Hello, I’m running a dockerized Kubernetes pod within Azure cloud which needs a volume mount. It works fine outside of airflow, but with KubernetesPodOperator, the Volume mount is failing with the error ‘One of the request inputs is out of range.’. The error happens when using a Windows Host OS, since this container needs a Windows machine. I have another container wherein the same Volume mount code with KubernetesPodOperator works fine running on Linux host within Azure cloud. Clearly there seems a bug with KubernetesPodOperator for volume mounts with Windows base OS.

Versions : kubernetes 11.0.0. apache-airflow 2.2.2 apache-airflow-providers-cncf-kubernetes 2.2.0

/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:357: DeprecationWarning: The logging_level option in [core] has been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:357 DeprecationWarning: The logging_level option in [core] has been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:67 DeprecationWarning: This module is deprecated. Please use kubernetes.client.models.V1VolumeMount.
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:55 DeprecationWarning: This module is deprecated. Please use kubernetes.client.models.V1Volume.
Running <TaskInstance: image_extraction_pipeline.image_extraction manual__2021-12-10T21:19:48.885493+00:00 [queued]> on host imageextractionpipelineimageextraction.b97789b6333240d18ea81825
Traceback (most recent call last):
File “/home/airflow/.local/lib/python3.8/site-packages/azure/storage/blob/_container_client.py”, line 292, in create_container
return self._client.container.create( # type: ignore
File “/home/airflow/.local/lib/python3.8/site-packages/azure/storage/blob/_generated/operations/_container_operations.py”, line 134, in create
raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: Operation returned an invalid status ‘One of the request inputs is out of range.’

2

from datetime import datetime, timedelta
from airflow import DAG
from kubernetes.client import models as k8s
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

default_args = {
‘owner’: ‘airflow’,
‘depends_on_past’: False,
}

Define volume for Azure File Share

volume = k8s.V1Volume(
name=‘aks-file-share’,
azure_file=k8s.V1AzureFileVolumeSource(share_name=‘akswinshare’, secret_name=‘azure-file-share’, read_only=False))

Define volume_mount for Azure File Share

volume_mount = k8s.V1VolumeMount(
name=‘aks-file-share’, mount_path=‘e:’, sub_path=None)

Initialize DAG defaults

with DAG(‘image_extraction_pipeline’,
default_args=default_args,
description=‘Image extraction pipeline’,
schedule_interval=timedelta(days=1),
catchup=False,
start_date=datetime(2021, 11, 18)) as dag:

dag.doc_md = __doc__
image_extraction_task = KubernetesPodOperator(
    name="image-extraction",
    namespace='airflow',
    task_id='image_extraction',
    image='us.gcr.io/ridecell-1/bosch-image-extraction:v6',
    get_logs=True,
    volumes=[volume],
    volume_mounts=[volume_mount],
    node_selector={"kubernetes.io/os": "windows"}
)
# Initialize airflow task
image_extraction_task