Python multiprocessor combined with multithreading
I'm not sure if what I'm trying to do is valid practice, but here it goes: I need my program to be highly parallelized, so I thought I could do 2-3 processes and each process could have 2 -3 threads.
1) Is it possible? 2) Does this make sense? 3) This is my code, but it hangs when I try to join processes.
PQ = multiprocessing.Queue()
[...]
def node(self, files, PQ):
l1, l2 = self.splitList(files)
p1 = multiprocessing.Process(target=self.filePro, args=(l1,PQ,))
p2 = multiprocessing.Process(target=self.filePro, args=(l2,PQ,))
p1.daemon = True
p2.daemon = True
p1.start()
p2.start()
p1.join() # HANGS HERE
p2.join()
while 1:
if PQ.empty():
break
else:
print(PQ.get())
PQ.join()
def filePro(self,lst,PQ):
TQ = queue.Queue()
l1, l2 = self.splitList(lst)
t1 = threading.Thread(target=self.fileThr, args=('a',l1,TQ,))
t2 = threading.Thread(target=self.fileThr, args=('b',l2,TQ,))
t1.daemon = True
t2.daemon = True
t1.start()
t2.start()
t1.join()
t2.join()
while 1:
if TQ.empty():
break
else:
PQ.put(TQ.get())
TQ.task_done()
TQ.join()
def fileThr(self,id,lst,TQ):
while lst:
tmp_path = lst.pop()
if (not tmp_path[1]):
continue
for item in tmp_path[1]:
TQ.put(1)
TQ.join()
source to share
1) Is it possible?
Yes.
2) Is there any point in it?
Yes. But usually this is not the moment you are looking for.
First, almost every modern operating system uses a "flat" scheduler; there is no difference between 8 threads spread over 3 programs or 8 threads spread over 8 programs. *
* Some programs can gain significant benefit by carefully using intraprocess-only locks or other synchronization primitives in some places where you know you are only using threads from the same program, and of course avoiding shared memory in those places, but you will not get this benefit by evenly distributing your jobs across threads and threads between processes.
Second, even if you were using, say, old SunOS, in the default CPython interpreter, the Global Interpreter Lock (GIL) ensures that only one thread can run Python code at a time. If you're wasting time executing code from the C extension library that explicitly releases the GIL (for example, some NumPy features), streams can help, but otherwise they all end up serializing anyway.
The main case where threads and processes are useful together is where you work with both CPU bound and I / O bound. In this case, as a rule, one feeds the other. If the I / O is fed by the CPU, use one thread pool in the main process to handle the I / O, and then use the pool of worker processes to do the CPU work on the results. If it is the other way around, use a pool of worker processes to run the CPU, and then each worker process uses a pool of threads for I / O.
3) This is my code, but it hangs when I try to join processes.
It's very difficult to debug your code if you don't provide a minimal, complete, testable example .
However, I see one obvious problem.
You are trying to use TQ
as a producer-consumer queue with t1
and t2
as producers and parent filePro
as consumer. Your consumer doesn't call TQ.task_done()
until after t1.join()
and t2.join()
returns, which won't happen until these threads are done. But these producers won't finish because they're waiting for you to call TQ.task_done()
. So you have a dead end.
And since each of the main threads of your child processes are blocked, they never end, so they p1.join()
will block forever.
If you really want the main thread to wait until other threads are done before doing any work, you don't need the producer-consumer idiom; just let the kids do their job and go out without being called TQ.join()
and not worry about TQ.task_done()
parenting. (Note that you are already doing this correctly with PQ
.)
If, on the other hand, you want them to run in parallel, do not try join
child threads until you have finished the loop.
source to share