Airflow not scheduling the jobs in celery architecture

Airflow
1

Hello Team,

Recently I have set up the celery architecture 1.10.14 in Dev and prod, same setup working fine in dev. In prod having the issue.

Issue:

  1. When triggering sample jobs , the scheduler not sending messages to rabbit MQ.
  2. Flower UI worker status is active online and broker is connected.
  3. RabbitMQ UI not showing any message update.

Below Scheduler log :

Starting Loop…
[2021-04-01 20:03:50,505] {scheduler_job.py:1446} DEBUG - Harvesting DAG parsing results
[2021-04-01 20:03:50,505] {dag_processing.py:658} DEBUG - Received message of type DagParsingStat
[2021-04-01 20:03:50,506] {dag_processing.py:658} DEBUG - Received message of type DagParsingStat
[2021-04-01 20:03:50,507] {dag_processing.py:658} DEBUG - Received message of type DagParsingStat
[2021-04-01 20:03:50,507] {dag_processing.py:658} DEBUG - Received message of type DagParsingStat
[2021-04-01 20:03:50,508] {scheduler_job.py:1448} DEBUG - Harvested 0 SimpleDAGs
[2021-04-01 20:03:50,508] {scheduler_job.py:1514} DEBUG - Heartbeating the executor
[2021-04-01 20:03:50,509] {base_executor.py:122} DEBUG - 0 running task instances
[2021-04-01 20:03:50,509] {base_executor.py:123} DEBUG - 0 in queue
[2021-04-01 20:03:50,509] {base_executor.py:124} DEBUG - 32 open slots
[2021-04-01 20:03:50,510] {base_executor.py:133} DEBUG - Calling the <class ‘airflow.executors.celery_executor.CeleryExecutor’> sync method
[2021-04-01 20:03:50,510] {celery_executor.py:256} DEBUG - No task to query celery, skipping sync
[2021-04-01 20:03:50,511] {scheduler_job.py:1460} DEBUG - Heartbeating the scheduler

Worker log :

[2021-04-01 19:53:49,484: DEBUG/MainProcess] pidbox received method revoked() [reply_to:{‘exchange’: ‘reply.celery.pidbox’, ‘routing_key’: ‘197d0100-dade-3962-ba0e-e9669919f562’} ticket:e5069a69-766c-4de6-9dc6-f87fe7fbfb52]
[2021-04-01 19:53:49,485: DEBUG/MainProcess] pidbox received method active() [reply_to:{‘exchange’: ‘reply.celery.pidbox’, ‘routing_key’: ‘5c56848f-f7a7-343e-aef6-5a803fb352ad’} ticket:8c69129b-3df1-45db-a12c-e26c969416e6]
[2021-04-01 19:53:53,289: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2021-04-01 19:53:53,290: INFO/MainProcess] Events of group {task} enabled by remote.
[2021-04-01 19:53:53,421: DEBUG/MainProcess] heartbeat_tick : for connection 9f80d8c88c2245a58f293d5332ea3dda
[2021-04-01 19:53:53,421: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 28/85, now - 28/178, monotonic - 24724845.34612785, last_heartbeat_sent - 24724805.343917303, heartbeat int. - 120.0 for connection 9f80d8c88c2245a58f293d5332ea3dda
[2021-04-01 19:53:58,291: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2021-04-01 19:54:03,291: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2021-04-01 19:54:08,290: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2021-04-01 19:54:13,288: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]

flower UI log
[I 210401 20:05:34 mixins:229] Connected to amqp://admin:**@histname:5672/airflow_host
[W 210401 20:05:39 control:49] ‘stats’ inspect method failed
[W 210401 20:05:39 control:49] ‘active_queues’ inspect method failed
[W 210401 20:05:39 control:49] ‘registered’ inspect method failed
[W 210401 20:05:39 control:49] ‘scheduled’ inspect method failed

2

dis you got any solution

3

Yeah, This issue has resolved.

Solutions - Some dependency package was not installed on worker hosts, after installing the packages and restarting the worker and scheduler works fine