It looks like by default celery will run a max of 16 jobs concurrently. Is it easy to increase that? By how much and at what cost?


#1

You can change the max concurrent jobs via the environment variable WORKER_CONCURRENCY (which you can add in the “Environment vars” section of your configuration) More information on this ENV VAR can be found here.

It should be noted that if you exceed the defaults for parallelism or dag_concurrency then you’ll also need to increase these so you don’t get bottlenecked by what airflow will schedule.

Changing these values won’t cost anything extra but may impact your worker/scheduler’s performance which might require you to scale the resources up. This is largely dependent on the nature of the tasks being run.