Possible to return success if FileSensor reaches its timeout?

Airflow
1

I have a couple of dags that utilize FileSensors. In both cases, it’s pretty random when a file may or may not get dropped in the monitored directory.

Currently the dags run daily, and the FileSensor timeout is set to 1 day. If no files are dropped in the monitored directores at all that day, the dag returns a failure. It’s not a huge deal, but seeing all that red in the dag history isn’t the most desirable look.

My first question is: Is it possible to force a success return if the FileSensor hits its timeout?

My second question is: Is there a totally different and better way to implement this solution?

2
  1. I don’t think so.
  2. You could use a BranchPythonOperator downstream of your sensor. In that operator, check for your file’s existence (which basically checks if the previous task succeeded). If it does exist continue down the rest of your tasks. If it does not exist, branch off to a dummy operator. The net effect will be that if your file sensor times out, the rest of your tasks will be marked skipped instead of failed, and the dag as a whole will be marked success.
3
4

Hi @tthompson2!

Have you tried setting the soft_fail arg to True in your FileSensor task? The FileSensor inherits from the BaseSensorOperator which has a parameter called soft_fail. In the event of the sensor failing, the task will be marked as skipped (instead of failed) if soft_fail is True.

If your use case calls for the DAG to continue regardless if the FileSensor task finds a file, this might be a workaround as well.

1 Like
5

Studying this now, thanks!

6

Will give this a shot - I didn’t realize that argument was even an option. Thanks!