I am working to automate a lot of tasks currently performed manually. The starting point is adding some rows in a Postgres database and then taking action based on them.
Does Airflow work well with detecting a new row in a DB as a dependency?
Yes, this is a perfect use case for an Airflow sensor.
You would want your “detecting task” to be a sensor to check at some “poke interval” for a condition to be met. If the condition is met, it will perform the next task(s).
You can read more about sensors here.
Sensors should be used carefully since they will take up a task slot while they are running (they inherit from the baseoperator class like all other Airlfow operators).
in Airflow 1.10.2 there is a new parameter for a sensor called mode. This can help prevent long running sensors from taking up a task spot.
mode: How the sensor operates.
Options are: { poke | reschedule }, default is poke.
When set to poke the sensor is taking up a worker slot for its
whole execution time and sleeps between pokes. Use this mode if the
expected runtime of the sensor is short or if a short poke interval
is requried.
When set to reschedule the sensor task frees the worker slot when
the criteria is not yet met and it’s rescheduled at a later time. Use
this mode if the expected time until the criteria is met is. The poke
inteval should be more than one minute to prevent too much load on
the scheduler.
2 Likes