Run the Airflow CLI’s backfill command on my local dev environment (Does Astro CLI support Airflow CLI commands? If so, how do I run Airflow CLI commands using Astro CLI?)
Delete the task runs & dag runs on Airflow UI for the days I want to re-run the dag for. Then run the dag with a specified execution date for those days
Clear the task runs on Airflow UI
Set up a separate dag for backfilling/ re-running
These options feel a bit manual or effort-intensive with lots of redundancies when trying to re-run the task for a large time frame. So I’m hoping that maybe there are alternatives that I haven’t thought of.
Run the Airflow CLI’s backfill command on my local dev environment (Does Astro CLI support Airflow CLI commands? If so, how do I run Airflow CLI commands using Astro CLI?)
Yes, this is possible through the Astro CLI to run airflow commands on your LOCAL deployment. You can use the run command.
astro dev run <airflow command>
For example, if you want to clear all your tasks in your DAG.
Delete the task runs & dag runs on Airflow UI for the days I want to re-run the dag for. Then run the dag with a specified execution date for those days
This is a valid way to rerun the DAG. Though the caveat is that catchup needs to set as True and all of the deleted dagruns are after the latest existing dagrun.
Clear the task runs on Airflow UI
This option sounds the most reasonable to me. Easiest to execute and you do not have alter the DAG. You only need to clear the tasks you want to rerun. This allows you to be more precise with the tasks you want to rerun too.
Set up a separate dag for backfilling/ re-running
Probably the most excessive route. You do not need another DAG just for rerunning tasks.
I think both option 1 and options 3 are equally easy to perform. Up to the preference of the individual.
Just a follow-up question on clearing the task runs. Can I use Astro CLI to clear the task runs on my astronomer-hosted airflow instance? Or is the CLI just for my local environment?
I see Paola’s answer back in 2019 that cloud users can run the airflow commands just in the local environment, but I want to check if that’s still the case: Can I use the Airflow CLI on Astronomer?
Unfortunately, Astro CLI is not able to run airflow command on Astronomer cloud deployments.
You will either need to upgrade to Airflow 2.0 and leverage the Stable API or install a plugin that allows you to perform those functions from the webserver. A few customer has used this plugin made by AnkurChoraywal
Hi Alan, do you know if that plugin works for Airflow 2.0?
When I navigate to the backfill tab, I get the following:
Python version: 3.7.10
Airflow version: 2.0.0+astro.3
Node: frozen-apastron-2617-webserver-78d9477796-6xchc
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/airflow/plugins/backfill/main.py", line 71, in list
rbac_authentication_enabled=rbac_authentication_enabled)
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 281, in render_template
template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
ctx.app,
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "/usr/local/airflow/plugins/templates/backfill_page.html", line 1, in top-level template code
{% extends 'airflow/master.html' %}
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 60, in get_source
return self._get_source_fast(environment, template)
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 89, in _get_source_fast
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: airflow/master.html
