We needed input for the Airflow production setup, as we were able to accomplish a small POC on SQLite and a sequential executor as well as local executor in Airflow 2.5.0.
Well, we have almost 10 concurrent jobs, which we want to schedule in airflow.
We have almost 70 jobs in total, that will be totally pushed onto the system one after another.
Previously, our jobs were on Jenkins, which is used to schedule + execute these jobs. But, now we will go ahead with Airflow to schedule
Over and above, these all need to be automated for scheduling, whereas these jobs would be using the Docker instance for different projects with individual DAG's as their jobs for daily purposes.
We want to go-ahead with the AWS Ec2 instance as an airflow server with MySQL as our metadata back-end.
Well, we need input on the parameters:
6.a Parameters of airflow.cfg
6.b On infrastructure
6.c Any good practices
6.d Thumb rules
6.e Good experience
6.f Industry standard