Can a callable be applied to a process pool?
Whenever I call a function Pool.apply_async
, I have to pass the function that this process will execute. I tried to pass in instead of a callable but didn't do anything. Is there a way to do this? Or should I create the pool myself from scratch?
The code looks like this:
import queue
class TaskThread(object):
def __init__(self):
#self.queue=queue.Queue()
def __call__(self):
print("in TaskThread.__call__")
#self.queue.put(1)
pool=multiprocessing.Pool(4)
task=TaskThread()
pool.apply_async(target=task)
Something like that.
source to share
The problem is that you didn't call get()
in AsyncResult
, the returned one apply_async
, and you didn't use pool.close
/ pool.join()
to wait until the child process is done before exiting the main process. Since all worker processes inside Pool
are daemons, they are terminated as soon as the main process exits. This means that your sample program exits (and takes its children with it) before the child process can print anything. You can fix this by calling .get()
in AsyncResult
or adding close()
/ join()
:
class TaskThread(object):
def __call__(self):
print("in TaskThread.__call__")
pool=multiprocessing.Pool(4)
task=TaskThread()
pool.apply_async(task)
pool.close()
pool.join()
Or:
class TaskThread(object):
def __call__(self):
print("in TaskThread.__call__")
pool=multiprocessing.Pool(4)
task=TaskThread()
result = pool.apply_async(task)
result.get()
Edit:
To convey the Queue
way you are trying you need to do something like this:
import multiprocessing
class TaskThread(object):
def __init__(self, manager):
self.queue = manager.Queue()
def __call__(self):
print("in TaskThread.__call__")
self.queue.put(1)
if __name__ == "__main__":
pool=multiprocessing.Pool(4)
m = multiprocessing.Manager()
task=TaskThread(m)
result = pool.apply_async(task)
result.get()
print(task.queue.get())
Output:
in TaskThread.__call__
1
source to share