Scheduling Celery Tasks with Big ETA

I am currently experimenting with future tasks in celery using ETA and the redis browser. One of the known issues with using the redis broker has to do with the visibility timeout :

If the task is not confirmed within the visibility timeout, the task will be redirected to another worker and completed.

This causes problems with ETA / countdown / redo tasks when the execution time exceeds the visibility timeout; in fact, if it does, it will run over and over in a loop.

Some tasks that I can imagine will have an ETA in a week / month scale. A setup visibility timeout

large enough to cover these tasks is probably unwise.

Are there any ways to handle these tasks with the redis broker? I know this question . Does the only option change brokers?

+3


source to share





All Articles