Hi,
I am new to airflow, currently running tests on aws mwaa environment, we have 150 dags with 4 tasks in each , these are scheduled to run every 5mins. I am using instance type which has 4vcpus on aws, min workers-20 and max workers - 25. 5 schedulers are configured to run on the same instance type. Below are airflow configs we are using. With this configuration dags are running 4 instances per hour and actual dag runtime is below 50secs but rest of 14mins is taken by scheduler. can you please let me know how to tune the performance of my environment so that we can run a job every 5mins.
|Configuration option|Custom value|
|celery.sync_parallelism|1|
|celery.worker_autoscale|10,10|
|core.dag_concurrency|10|
|core.dags_are_paused_at_creation|False|
|core.lazy_load_plugins|False|
|core.max_active_runs_per_dag|1|
|core.parallelism|500|
|scheduler.dag_dir_list_interval|600|
|scheduler.min_file_process_interval|90|
|scheduler.parsing_processes|7|
|scheduler.processor_poll_interval|10|