What is the optimal number of celery workers per kernel?

There is not much information in the celery docs about the optimal number of celery workers to use on a machine. I believe that by default celery creates one worker per machine core.

I know from experimentation that starting more celery workers on one main machine is definitely beneficial (default 1 worker due to 1 core). I'm looking for a threshold at which adding more workers has a slightly diminishing return - the optimal worker number per core. I am currently using tree celery with a demon celeryd config file having the following line:

CELERYD_NODES="worker1 worker2 worker3"

      

My intention is to create 3 PER CORE workers (so if I started a 4 core machine there would be 12 workers). Am I doing this right or is it only starting 3 workers regardless of the number of cores?

+3


source to share





All Articles