Let’s say we have some DAGs in a Github repo with CI/CD that runs something like astro deploy after tests pass. This should trigger a new deployment of Airflow, i.e. the image tag for that deployment will be updated. New webserver and scheduler Pods get initialized. This results in the Airflow UI being unavailable until the new deployments comes up.
First question: Am I understanding things correctly so far?
How would one go about creating / updating / deleting DAGs without having to redeploy a new webserver and scheduler, so users don’t experience UI downtime?
Sorry for slow response Lee - users don’t experience downtime if you use Celery w/ KEDA autoscaler enabled.
Restarting webserver + scheduler doesn’t affect running Celery tasks — they will finish out and report their status directly to the Airflow metadata database.
The UI downtime is brief, but DAG processing continues. We plan to build a new React UI after Airflow 2.0 which will come back up in seconds.
Thanks ryw. You say there is no downtime with Celery + KEDA. Can you help me understand why that is? Also, would there be downtime with the KubernetesExecutor?