Python thread pool processor load

A typical Python thread pool will have a structure like this:

def run(self):
    while True:


So it looks like there is continuous monitoring of the task queue. How CPU intensive is the constant monitoring of the task queue?

Would it be better to introduce some sleep (a few milliseconds) to reduce the load on the processor? Thus, it is possible to stop monitoring the task queue for some time when all threads are busy and reduce the load on the processor.


source to share

2 answers

There is a 0.0%

cpu load while 1000 threads are locked on .get()

on my machine:

#!/usr/bin/env python
from __future__ import print_function
import os
import time
from threading import Thread
from Queue import Queue

try: import psutil # pip install psutil
except ImportError:
    psutil = None

def f(queue):
    while True:
        item = queue.get() # block until an item is available
        print("got %s" % (item,))
        break # end thread

# create threads
q = Queue()
threads = [Thread(target=f, args=(q,)) for _ in xrange(1000)]

# starts them
for t in threads:
    t.daemon = True # die with the program

# show cpu load while the threads are blocked on `queue.get()`
if psutil is None:
    print('Observe cpu load yourself (or install psutil and rerun the script)')
    time.sleep(10) # observe cpu load
    p = psutil.Process(os.getpid())
    for _ in xrange(10):
        print("cpu %s%%" % (p.get_cpu_percent(interval=0),))

# finish threads
for i in range(len(threads)):
    q.put_nowait(i) #note: queue is unlimited so there is no reason to wait

for t in threads: t.join() # wait for completion




Thus, it seems that there is continuous monitoring of the task queue.

It depends on which monitoring tool is right for you.


will block as needed until the item is available, so it depends on how the locking is implemented.

I have no references, but I think there must be a signal handler waiting to wake up, so I would say it is "asleep".



All Articles