Python thread pool processor load
A typical Python thread pool will have a structure like this:
def run(self):
while True:
z=self.some_task_queue.get()
do_work(z)
So it looks like there is continuous monitoring of the task queue. How CPU intensive is the constant monitoring of the task queue?
Would it be better to introduce some sleep (a few milliseconds) to reduce the load on the processor? Thus, it is possible to stop monitoring the task queue for some time when all threads are busy and reduce the load on the processor.
source to share
There is a 0.0%
cpu load while 1000 threads are locked on .get()
on my machine:
#!/usr/bin/env python
from __future__ import print_function
import os
import time
from threading import Thread
from Queue import Queue
try: import psutil # pip install psutil
except ImportError:
psutil = None
def f(queue):
while True:
item = queue.get() # block until an item is available
print("got %s" % (item,))
break # end thread
# create threads
q = Queue()
threads = [Thread(target=f, args=(q,)) for _ in xrange(1000)]
# starts them
for t in threads:
t.daemon = True # die with the program
t.start()
# show cpu load while the threads are blocked on `queue.get()`
if psutil is None:
print('Observe cpu load yourself (or install psutil and rerun the script)')
time.sleep(10) # observe cpu load
else:
p = psutil.Process(os.getpid())
for _ in xrange(10):
print("cpu %s%%" % (p.get_cpu_percent(interval=0),))
time.sleep(1)
# finish threads
for i in range(len(threads)):
q.put_nowait(i) #note: queue is unlimited so there is no reason to wait
for t in threads: t.join() # wait for completion
print('done')
source to share
Thus, it seems that there is continuous monitoring of the task queue.
It depends on which monitoring tool is right for you.
Queue.get()
will block as needed until the item is available, so it depends on how the locking is implemented.
I have no references, but I think there must be a signal handler waiting to wake up, so I would say it is "asleep".
source to share