Celery: ensuring tasks are completed consistently

On a project I'm currently working on, I periodically release Celery tasks. These tasks are for specific clients, so there are tasks for example. clientA, for clientB and for clientC. There are some additional conditions:

• Tasks for the same client can never run in parallel.
• Tasks for the same client must run sequentially, that is, the order of the message queues.

The herring cookbook (see also this article ) shows a locking mechanism that ensures that one task can only be done one at a time. This mechanism can be easily adapted to ensure that a task for one client can only be performed one at a time. This satisfies the first condition.

The second condition is more difficult to ensure. Since the tasks are generated from different processes, I cannot use the task chaining. Perhaps I could modify the locking mechanism to repeat tasks while they are waiting for a lock, but that still cannot guarantee the order (due to the retry timeout, but also due to a race condition when acquiring a lock).

At the moment I have limited my concurrency to 1 to enforce order, although some of the tasks are time consuming and quite heavy.