I’m writing files to the workers that I sometimes need to retrieve. How can I do that?

#1

As a general best practice when using the Celery Executor, we recommend avoiding writing a file to a worker (and retrieving it). Given the temporary nature of workers/their resources, all storage on workers should be treated as ephemeral.

If you’re looking to retrieve a file from that worker, we’d recommend considering defining an on_success or on_failure callback in your DAGs that writes your file to an external system (e.g. S3, GCS). Related forum post here.

0 Likes

I wrote a file to disk - why can't a downstream task find it?