My DAG is execute two sql queries that insert data into a table and then create one file from that table.
What I want to do is to split the table (using row_number) and for each rank, to create a different file.
I rank the data that was inserted to the table and according to that rank, I know to how many files to split. Since I don’t know how many files I’ll create in each execution, I initial the number to 1.
I need to add task that will wait until the data is inserted to the table and then refresh the DAG before start creating the files.
How do I do it?
Adding a sleeping task is not the best practice because I don’t know how much time it will take the DAG to be refreshed in each execution.