Using join () for processes created using multiprocessing in python
I am using the multiprocessing Process class module to create multiple processes, these processes execute multiple script and then die. What I wanted was a timeout that should be applied to each process so that the process could die if the timeout failed . I am using join(timeout)
for process objects.
Since the join() function doesn;t kill the process, it just blocks the process until it finishes
Now my question is, are there any side effects when used join()
with timeout
.. like, will the processes be cleaned up automatically after the main process is gone? or should I kill these processes manually
I'm new to python and its multiprocessing module, please be patient.
My code that creates processes in a for loop ::
q = Queue()
jobs = [
Process(
target=get_current_value,
args=(q,),
kwargs=
{
'device': device,
'service_list': service_list,
'data_source_list': data_source_list
}
) for device in device_list
]
for j in jobs:
j.start()
for k in jobs:
k.join()
source to share
The argument timeout
simply tells join
how long to wait for completion Process
before giving up . If it timeout
expires, it Process
doesn't exit; the call is join
simply unblocked. If you want to finish your workers when the timeout is running out, you need to do it manually. You can use terminate
, as suggested by wRAR, to uncleanly close apps, or use some other signaling mechanism so that kids can disconnect to the end:
p = Process(target=worker, args=(queue,))
p.start()
p.join(50)
if p.isalive(): # join timed out without the process actually finishing
#p.terminate() # unclean shutdown
If you don't want to use terminate
, the alternative approach really depends on what the workers are doing. If they are consuming from the queue, you can use the sentinel signal:
def worker(queue):
for item in iter(queue.get, None): # None will break the loop
# Do normal work
if __name__ == "__main__":
queue = multiprocessing.Queue()
p = multiprocessing.Process(target=worker, args=(queue,))
p.start()
# Do normal work here
# Time to shut down
queue.put(None)
Or you can use Event
if you are doing some other operation in a loop:
def worker(event):
while not event.is_set():
# Do work here
if __name__ == "__main__":
event= multiprocessing.Event()
p = multiprocessing.Process(target=worker, args=(event,))
p.start()
# Do normal work here
# Time to shut down
event.set()
Usage terminate
can be fine though if your child processes are not using resources that could get corrupted if the process exits unexpectedly (like writing to a file or db or blocking). If you are just doing some calculations on the worker's side, terminate
it doesn't hurt anything.
source to share
join()
does nothing with the child process. If you really want to terminate a workflow in a non-clean way, you must use terminate()
(you must understand the consequences). If you want the children to be terminated when the main process is finished, you must set an attribute on them daemon
.
source to share