Clever Geek Handbook

Scheduling Celery Tasks with Big ETA

I am currently experimenting with future tasks in celery using ETA and the redis browser. One of the known issues with using the redis broker has to do with the visibility timeout :

If the task is not confirmed within the visibility timeout, the task will be redirected to another worker and completed.

This causes problems with ETA / countdown / redo tasks when the execution time exceeds the visibility timeout; in fact, if it does, it will run over and over in a loop.

Some tasks that I can imagine will have an ETA in a week / month scale. A setup visibility timeout

large enough to cover these tasks is probably unwise.

Are there any ways to handle these tasks with the redis broker? I know this question . Does the only option change brokers?

+3
python redis celery


source to share


No one has answered this question yet

See similar questions:

2
Schedule a task to be launched in the future (architecture)
2
Celery SQS + Job Duplication + SQS Visibility Timeout

or similar:

6
Celery did not queue task back to RabbitMQ after timeout
4
Celery restart tasks
2
Celery SQS + Job Duplication + SQS Visibility Timeout
1
If a celery worker dies hard, will the job be repeated?
1
What is different between task_reject_on_worker_lost and task_acks_late in Celery
1
The same task is executed multiple times
0
Celery task does not appear in queue after task.apply_async using countdown
0
Connect two separate projects via Celery
0
Celery related tasks are lost along with the worker, and the new worker does not pick up the task
0
Celery scheduled task lost from Redis


More articles:


All Articles
Loading...
X
Show
Funny
Dev
Pics