I’m looking for a best practice here:
I have a task that is essentially a Boolean. It goal is to look at 2 datasets and determines if the next task can be yet. Until the answer is Yes(or True) I need to keep running the task every X seconds until it is True or we hit some retry threshold.
When it returns true, then the workflow can continue.
What is the optimal way to do this kind of a thing in Airflow?
Hi @gregJ, thanks for reaching out!
What you’re describing sounds like a sensor.
Sensors are a special type of operator that are designed to wait for something to happen. They can be time-based, or waiting for a file, or an external event, but all they do is wait until something happens, and then succeed so their downstream tasks can run.
I’d recommend taking a look at our guide or OSS Airflow documentation.