Required Airflow multitenant setup procedure

Airflow
1

We are planning to set up the airflow in a multi-tenant posture.
Our stack is hosted on Kubernetes.
1- Kindly help us with the configuration steps to make the current stand-alone model into a multi-tenant model?
2- Additionally, we have planned to use a different RDBMS DB in place of Default POSTGRESQL DB. So need to know the steps to be performed for changing the configuration.

Thank You.

2

Hi @ramahal1,

  1. On your stand-alone model, you so have the Webserver and Scheduler + the PostgreSQL database?
  2. You have to check if the database that you want to use is compatible with the SQLAlchemy connector. But the Airflow documentation recommend to use MySQL and PostgreSQL. What database do you want to use?

To scale up the workers, you have to check for Celery which has to be installed on your Worker nodes.
You can also, put an other ‘master Webserver node’ and use a Loadbalancer in front of the webservers.
You can also have a Redis/RabbitMQ cluster for the Worker node part.
I don’t think that the airflow-scheduler service can be scale up, you have to scale it horizontally by increasing CPUs.

I hope, that these few lines will help you.
Tom

3

Thanks Tom. It helps. I will get back if in case of any queries.