When I used Airflow a couple of years ago, I was not that impressed. Especially I didn’t like it only supported a single scheduler and the Celery executor had to be used for scalability. I thought Step Functions would be simpler, cheaper and more scalable.
When I checked recently, Airflow 2.0 supports multiple schedulers and it looks much better. The scheduler can be horizontally scaled and, if we use the Docker/ECS/Kubernetes/Batch (or Lambda) Operator mainly, we just need the webserver, database and scheduler (with autoscaling), which is much more simpler.
Would you consider multiple scheduler support could end the reliance of Celery?