Hi, is it at all supported (or a good idea) to have a dag run which will delete itself from the db before it completes execution?
Here’s my situation…
I have a dag that is running constantly (by design) which is monitoring a directory waiting for a file to appear…it grabs it and processes it when it does. It all works great. I have it configured to max_active_runs=1 because, obviously, I don’t want a single file to get picked up by multiple parallel runs and be processed multiple times. The side-effect of this is if I have a past run that failed that I want to retry, I can’t retry it easily because of the current dagrun which is monitoring for new files…my retry attempt just waits indefinitely. SO to get around this I introduced a soft-fail in the file monitoring where if a file hasn’t appeared after 10 minutes it will end the dag run successfully and start over…so every 10 minutes my retry attempt will get a chance to execute. This is great, it works well…BUT, the side effect of this is that it generates a lot of noise in my dagrun list because there are a lot of empty dag runs…the dag run list is fairly important so I’d like to keep it clean and free from all this noise. So, this brings me back around to my main question…on these empty dagruns I would just like to delete the dagrun itself from the db, to keep my dagrun list clean…is it going to break anything horribly if the dagrun disappears from the db before it has even completed executing?
Thanks!