We have to build an application which will have following flow.
Api will trigger a scan, scan will consist of several dataset and dataset will be consisting of several tools.
Scan will be dag and dataset will be dag also.
These tools needs to be run in perticular order and each tool will produce data and some of the tool will take parsed output of earlier tools as input. Many of these tools can be CPU and memory, network heavy.
Datasets will also needs to be run in particular order and will take output of earlier dataset as input.
We need around 1000 scans concurrently at the same time.