How to configure execution of celery at the same time from the queue
In an environment with 8 cores, celery should be able to process 8 incoming tasks in parallel by default. But sometimes, when new tasks emerge, celery puts them into a lengthy process.
I played around with the default setting, allowing one worker to consume from one queue.
celery -A proj worker --loglevel=INFO --concurrency=8
I understand that one worker with concurrency of 8 is able to process 8 tasks from one queue in parallel?
What is the preferred way to set up celery to prevent this behavior described above?
source to share
Simply put, concurrency is the number of jobs running on the desktop. Prefetching is the number of jobs queued at the worker itself. You have 1 of 2 options here. The first is to set the prefetch multiplier to 1. This would mean that the worker would hold 8 additional jobs in the queue in your case. The second one I would recommend is to create two different queues for your short tasks and another one for your long tasks.
source to share