Python multiprocessing, ValueError: I / O operation on closed file
I have a problem with the Python multiprocessing package. Below is a simple sample code that illustrates my problem.
import multiprocessing as mp
import time
def test_file(f):
f.write("Testing...\n")
print f.name
return None
if __name__ == "__main__":
f = open("test.txt", 'w')
proc = mp.Process(target=test_file, args=[f])
proc.start()
proc.join()
When I run this I get the following error.
Process Process-1:
Traceback (most recent call last):
File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\Python27\lib\multiprocessing\process.py", line 114, in run
self.target(*self._args, **self._kwargs)
File "C:\Users\Ray\Google Drive\Programming\Python\tests\follow_test.py", line 24, in test_file
f.write("Testing...\n")
ValueError: I/O operation on closed file
Press any key to continue . . .
It looks like sometime the file descriptor is "lost" while creating a new process. Can someone please explain what's going on?
source to share
I've had similar problems in the past. Not sure if this is done in a multiprocessing module or if it open
sets the close-on-exec checkbox by default , but I know for sure that file descriptors open in the main process are closed in multiprocessing children.
The obvious work is to pass the filename as a parameter to the init function for the child process and open it once in each child (when using a pool) or pass it as a parameter to the target function and open / close each call. The former requires using a global file descriptor to store (not very good) - unless someone shows me how to avoid this :) - and the latter can take a performance hit (but can be used with multiprocessing.Process directly).
An example of the first:
filehandle = None
def child_init(filename):
global filehandle
filehandle = open(filename,...)
../..
def child_target(args):
../..
if __name__ == '__main__':
# some code which defines filename
proc = multiprocessing.Pool(processes=1,initializer=child_init,initargs=[filename])
proc.apply(child_target,args)
source to share