I’m pretty new to Airflow (2.0.1) and workflow engineering.
I need to create a dynamically scaled workflow in Airflow, which kind of works through variables usage.
Unfortunately I’m not able to end my workflow, since the trigger_rule ALL_DONE and also ONE_SUCCESS are not causing the task
process_storage_meta (which is joining the branches) to be triggered… The task is being skipped although one of the predecessors succeed. I really need to resume for logging even if all tasks are failing, so I assumed the rules should work…
not sure if it’s related to the dynamic branching or if I’m getting something wrong.
In addition, the task
process_storage_meta is a custom operator instance…
Would be great if someone could help, thanks in advance