I’m having issues with scaling airflow over 700 tasks instances, using local executor and MySQL. The PIDs are getting killed with no other message. I am now trying with dask, and all seems to run with no errors. But now the dag runs aren’t being scheduled. It just sits there after being triggered.
I followed this:
I also had to set queues to None on the dag and run scheduler with the -do_pickle option.
Id like to try with dask and LSFCluster.