KubernetesPodOperator seems to drop arguments

Airflow
1

I recently refactored my script to require a larger amount of command line arguments that get passed via the KubernetesPodOperator's arguments field. After this refactor, I noticed that two of the arguments were just getting… dropped. To test, I simply made the very first line of code in script.py print(sys.argv), and sure enough, the last two args in the list are just not there. Long story short-after a bit of debugging, it turns out that a max of five arguments are getting passed to the script by the KubernetesPodOperator. Anything beyond argument #5 gets mysteriously dropped.

We’re running Airflow 2.2.2 and Kubernetes manifest 2.1.0.

Here’s the instantiation of the KubernetesPodOperator:

the_dag_step = KubernetesPodOperator(
    task_id="the_task",
    name="the_name",
    arguments=[
        "/app/src/script.py",
        "--flag1",
        "arg1",
        "--flag2",
        "arg2",
        "--flag3",
        "arg3",
    ],
    dag=the_dag
)

In the case above, “–flag3” and “arg3” don’t make it to the script.

2

Can you test this workaround?

    arguments=[
        "/app/src/script.py",
        "--flag1 arg1 --flag2 arg2 --flag3 arg3",
    ],
3

We have tried this and it works as expected.

4

Can you share the full pip freeze without grep airflow? The kubernetespodoperator sends the args unchanged into a kubernetes V1Container object, so unclear why they would be cut off.

5 
[2022-04-26, 08:40:40 CDT] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'pip freeze']
[2022-04-26, 08:40:40 CDT] {subprocess.py:85} INFO - Output:
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - adal==1.2.7
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - aenum==3.1.5
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - airflow-exporter==1.5.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - alembic==1.7.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - amqp==5.0.6
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - anyio==3.3.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow==2.2.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-amazon==2.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-apache-livy==2.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-apache-spark==2.0.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-celery==2.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-cncf-kubernetes==2.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-docker==2.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-elasticsearch==2.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-ftp==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-google==6.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-grpc==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-hashicorp==2.1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-http==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-imap==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-jdbc==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-microsoft-azure==3.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-microsoft-mssql==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-mysql==2.1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-odbc==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-postgres==2.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-redis==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-sendgrid==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-sftp==2.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-slack==4.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-sqlite==2.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apache-airflow-providers-ssh==2.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - apispec==3.3.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - argcomplete==1.12.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - asn1crypto==1.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - attrs==20.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Authlib==0.15.5
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-batch==11.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-common==1.1.27
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-core==1.20.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-cosmos==4.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-datalake-store==0.0.52
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-identity==1.7.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-keyvault==4.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-keyvault-certificates==4.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-keyvault-keys==4.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-keyvault-secrets==4.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-kusto-data==0.0.45
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-mgmt-containerinstance==1.5.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-mgmt-core==1.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-mgmt-datafactory==1.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-mgmt-datalake-nspkg==3.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-mgmt-datalake-store==0.5.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-mgmt-nspkg==3.0.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-mgmt-resource==20.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-nspkg==3.0.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-storage-blob==12.8.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-storage-common==2.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - azure-storage-file==2.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Babel==2.9.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - backports.entry-points-selectable==1.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - bcrypt==3.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - beautifulsoup4==4.7.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - billiard==3.6.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - blinker==1.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - boto3==1.18.65
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - botocore==1.21.65
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - cachetools==4.2.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - cattrs==1.5.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - celery==5.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - certifi==2020.12.5
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - cffi==1.15.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - charset-normalizer==2.0.7
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - click==8.0.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - click-didyoumean==0.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - click-plugins==1.1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - click-repl==0.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - clickclick==20.10.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - cloudpickle==1.4.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - colorama==0.4.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - colorlog==4.8.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - commonmark==0.9.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - croniter==1.0.15
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - cryptography==3.4.8
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - dask==2021.6.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - decorator==5.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - defusedxml==0.7.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - dill==0.3.1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - distlib==0.3.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - distributed==2.19.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - dnspython==2.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - docker==5.0.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - docutils==0.16
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - elasticsearch==7.13.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - elasticsearch-dbapi==0.2.6
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - elasticsearch-dsl==7.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - email-validator==1.1.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - eventlet==0.32.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - filelock==3.3.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask==1.1.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask-AppBuilder==3.3.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask-Babel==2.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask-Caching==1.10.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask-JWT-Extended==3.25.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask-Login==0.4.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask-OpenID==1.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask-SQLAlchemy==2.5.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Flask-WTF==0.14.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - flower==1.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - fsspec==2021.11.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - future==0.18.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - gevent==21.8.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-ads==14.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-api-core==1.31.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-api-python-client==1.12.8
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-auth==1.35.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-auth-httplib2==0.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-auth-oauthlib==0.4.6
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-appengine-logging==1.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-audit-log==0.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-automl==2.5.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-bigquery==2.30.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-bigquery-datatransfer==3.4.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-bigquery-storage==2.10.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-bigtable==1.7.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-build==3.7.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-container==1.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-core==1.7.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-datacatalog==3.6.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-dataproc==3.1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-dlp==1.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-kms==2.10.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-language==1.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-logging==2.7.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-memcache==1.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-monitoring==2.6.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-os-login==2.5.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-pubsub==2.8.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-redis==2.4.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-secret-manager==1.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-spanner==1.19.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-speech==1.3.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-storage==1.42.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-tasks==2.7.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-texttospeech==1.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-translate==1.7.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-videointelligence==1.16.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-vision==1.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-cloud-workflows==1.5.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-crc32c==1.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - google-resumable-media==2.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - googleapis-common-protos==1.53.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - graphviz==0.18
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - greenlet==1.1.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - grpc-google-iam-v1==0.12.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - grpcio==1.42.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - grpcio-gcp==0.2.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - grpcio-tools==1.42.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - gunicorn==20.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - h11==0.12.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - HeapDict==1.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - httpcore==0.13.7
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - httplib2==0.19.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - httpx==0.19.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - humanize==3.12.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - hvac==0.11.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - idna==3.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - importlib-metadata==4.8.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - importlib-resources==5.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - inflection==0.5.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - iso8601==1.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - isodate==0.6.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - itsdangerous==1.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - JayDeBeApi==1.2.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Jinja2==2.11.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - jmespath==0.10.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - JPype1==1.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - json-merge-patch==0.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - jsonpath-ng==1.5.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - jsonschema==3.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - kombu==5.2.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - kubernetes==11.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - lazy-object-proxy==1.4.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - ldap3==2.9.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - libcst==0.3.21
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - locket==0.2.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - lockfile==0.12.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - lxml==4.6.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Mako==1.1.5
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - maprdb-python-client==1.1.5
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Markdown==3.3.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - MarkupSafe==1.1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - marshmallow==3.14.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - marshmallow-enum==1.5.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - marshmallow-oneofschema==3.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - marshmallow-sqlalchemy==0.26.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - msal==1.16.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - msal-extensions==0.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - msgpack==1.0.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - msrest==0.6.21
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - msrestazure==0.6.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - mypy-extensions==0.4.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - mysql-connector-python==8.0.27
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - mysqlclient==2.0.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - nox==2020.12.31
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - numpy==1.20.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - oauthlib==3.1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - ojai-python-api==1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - openapi-schema-validator==0.1.5
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - openapi-spec-validator==0.3.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - packaging==21.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pandas==1.3.4
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pandas-gbq==0.14.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - paramiko==2.8.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - partd==1.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pendulum==2.1.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - platformdirs==2.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - ply==3.11
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - portalocker==1.7.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - prison==0.2.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - prometheus-client==0.12.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - prompt-toolkit==3.0.22
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - proto-plus==1.18.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - protobuf==3.17.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - psutil==5.8.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - psycopg2-binary==2.9.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - py==1.11.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - py4j==0.10.9.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pyarrow==4.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pyasn1==0.4.8
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pyasn1-modules==0.2.8
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pycparser==2.21
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pydata-google-auth==1.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Pygments==2.10.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - PyJWT==1.7.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pymssql==2.2.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - PyNaCl==1.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pyodbc==4.0.32
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pyOpenSSL==20.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pyparsing==2.4.7
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pyrsistent==0.18.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pysftp==0.2.9
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pyspark==3.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - python-daemon==2.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - python-dateutil==2.8.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - python-http-client==3.3.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - python-ldap==3.3.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - python-nvd3==0.15.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - python-slugify==4.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - python3-openid==3.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pytz==2021.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - pytzdata==2020.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - PyYAML==5.4.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - redis==3.5.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - redshift-connector==2.0.901
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - requests==2.26.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - requests-oauthlib==1.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - retrying==1.3.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - rfc3986==1.5.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - rich==10.13.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - rsa==4.7.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - s3transfer==0.5.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - scramp==1.4.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - sendgrid==6.9.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - setproctitle==1.2.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - six==1.16.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - slack-sdk==3.11.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - sniffio==1.2.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - sortedcontainers==2.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - soupsieve==2.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - SQLAlchemy==1.3.24
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - SQLAlchemy-JSONField==1.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - sqlalchemy-redshift==0.8.8
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - SQLAlchemy-Utils==0.37.9
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - sshtunnel==0.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - starkbank-ecdsa==1.1.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - statsd==3.3.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - swagger-ui-bundle==0.0.9
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - tabulate==0.8.9
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - tblib==1.7.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - tenacity==8.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - termcolor==1.1.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - text-unidecode==1.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - toolz==0.11.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - tornado==6.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - typing-extensions==3.10.0.2
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - typing-inspect==0.7.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - unicodecsv==0.14.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - uritemplate==3.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - urllib3==1.26.7
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - vine==5.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - virtualenv==20.10.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - watchtower==1.0.6
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - wcwidth==0.2.5
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - websocket-client==1.2.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - Werkzeug==1.0.1
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - WTForms==2.3.3
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - zict==2.0.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - zipp==3.6.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - zope.event==4.5.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:89} INFO - zope.interface==5.4.0
[2022-04-26, 08:40:41 CDT] {subprocess.py:93} INFO - Command exited with return code 0
1 Like
6

I don’t think this is a problem with KPO. I tried to reproduce with this code using Airflow 2.2.5 and kubernetes provider 2.1.0.

from datetime import datetime, timedelta
from airflow.decorators import dag
from airflow.models import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
import kubernetes.client as k8s

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}


with DAG(
    'kube',
    default_args=default_args,
    description='Task Group Test',
    start_date=datetime(2021, 7, 1),
    schedule_interval=None
) as dag:
    task = KubernetesPodOperator(
        task_id="task",
        cluster_context="docker-desktop",
        namespace="default",
        image="alpine",
        name="some_task",
        cmds=["echo"],
        arguments= ["1", "2", "3", "4", "5", "6", "7"],
        in_cluster=False,
        reattach_on_restart=True,
        get_logs=True,
        resources=k8s.V1ResourceRequirements(
            limits={
                "cpu": "100m",
                "memory": "384Mi",
            },
            requests={
                "cpu": "100m",
                "memory": "384Mi",
            },
        ),
        config_file="/usr/local/airflow/include/kubeconfig"
    )

In the logs I see all 7 arguments:

[2022-04-27, 15:29:30 ] {pod_launcher.py:159} INFO - 1 2 3 4 5 6 7

I was also able to kubectl describe po <pod name> and see the following section of the result. I recommend doing the same on your kpo-launched pod to see if all the arguments you had passed are there.

Containers:
  base:
    Container ID:  docker://8936e13fb78601c1d01f151e4f58bcccc8fd02d3bee1b9cc9ec07d395c99d7bf
    Image:         alpine
    Image ID:      docker-pullable://alpine@sha256:4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
    Port:          <none>
    Host Port:     <none>
    Command:
      echo
    Args:
      1
      2
      3
      4
      5
      6
      7
7

If all the arguments are present in kubectl describe then the problem is with how your container is handling the arguments, not with kubernetes receiving/passing them in the first place.