I created a quick dag which creates dynamic tasks
(using Airflow v1.10.12 and Local Executor).
My Dag has a list like
tables_list = [‘table1’,‘table2’]
Then i would create a first task (dummy) and then generate bash operators
for every table in a list and use first dummy task as upstream task,
It works great on a first run - all tasks created properly.
Then I change the list to add a new table3:
tables_list = [‘table1’,‘table2’,‘table3’]
DAG runs again but I do not see table3 in the Graph or Tree view. I do see
table3 task under Task Instance View so it was generated. But if I click on
it, i would get an error like Task [dynamic_job_proto_v1.t_table3] doesn’t
seem to exist at the moment
Then I restarted the scheduler - same thing. New Dags would not show that
task.
then I restarted airflow webserver - this time I was able to see table3
task in views.
After that I removed table2 from my list and DAG ran again - same issue.
Table2 was still in views untill i restarted the webserver. After the
restart, table2 dissappered from previosly ran Dags which is bad because
now i cannot go back in history, cannot compare execution time etc.
Is it bug ? anyone faced same issue ? any solution for this ?