Hello and Happy Tuesday everyone!
Firstly, I am using Airflow 2.0.2. I have a use case where I have several tasks all running against the same server. These tasks are all independent and can run in parallel, however, in order to avoid overloading the server I would like to limit the number of concurrent tasks. As a task completes, a new task is chosen from the list and started.
As an example, I have the following json:
{
“concurrent-tasks”: 5,
“task_queue” : [
“task1”, “task2”, “task3”, “task4”, “task5”,
“task6”, “task7”, “task8”, “task9”, “taska”,
“taskb”, “taskc”, “taskd”, “taske”
]
}
Passing this json would fire the first 5 tasks. As a task finishes another task from task_queue would be selected and run the next available task. There are other features I’m thinking would be helpful including typical airflow type things like “retries” or “timeout”, etc.
I’m not finding anything like this, is there something out there that I’m missing?
Joe