Celery: Are Duplicate Tasks Completed Automatically?

I am running (Django) Celery to schedule tasks for remote workers w1, w2, w3. Each of these workers has their own queue, from which they consume tasks allocated by the "scheduler", which is another bit celery task on the main server:

w1: q1
w2: q2
w3: q3

      

The scheduler schedules tasks based on the db check, that is, it will reconfigure the task with the same parameters if the db is not updated to match the task being performed. Therefore, if one or more queues accumulate, several tasks with the same parameters ("duplicates" from the point of view of my application) can be simultaneously in several queues.

I see some strange behavior: if there are duplicate tasks in several queues, if one of the queues starts its own task instance, just a few milliseconds before that, the rest of the duplicated tasks are "duplicated", so all of a sudden all tasks are executed in one and then the same time, even if they were concluded at different minutes from each other.

Is there any documentation or other link explaining this behavior? Is the behavior known, if so how do I disable it? I only need one instance of this task.

+3


source to share





All Articles