Why is it throwing "module object" has no XXX attribute "when I call apply_async from multiprocessing.
The code looks like this. When I copy and paste it into the cmd prompt it throws the 'module' object has no 'func' attribute , but when I save it as a .py file and execute python test.py
it just works fine.
import multiprocessing
import time
def func(msg):
for i in xrange(3):
print msg
time.sleep(1)
if __name__ == '__main__':
pool = multiprocessing.Pool(processes=4)
for i in xrange(5):
msg = "hello %d" %(i)
pool.apply_async(func, (msg, ))
pool.close()
pool.join()
print "Sub-process(es) done."
Can anyone explain to me the difference between a hint and a file when running Python code? Many thanks!
source to share
This is because on Windows, func
you need to pickle and send to the child process via IPC. In order for the child to be able to unpack func
, he must be able to import it from the parent module __main__
. When this happens in a normal Python script, the child can re-import your script, and it __main__
will contain all the functions declared at the top level of your script, so it works great. However, in the interactive interpreter, the functions that you have defined during the interpreter cannot be simply re-imported from the file as in a normal script, so they will not be in __main__
the child. This is clearer if you use multiprocessing.Process
directly to recreate the problem:
>>> def f():
... print "HI"
...
>>> import multiprocessing
>>> p = multiprocessing.Process(target=f)
>>> p.start()
>>> Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\python27\lib\multiprocessing\forking.py", line 381, in main
self = load(from_parent)
File "C:\python27\lib\pickle.py", line 1378, in load
return Unpickler(file).load()
File "C:\python27\lib\pickle.py", line 858, in load
dispatch[key](self)
File "C:\python27\lib\pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "C:\python27\lib\pickle.py", line 1126, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'f'
So it is clearer that the pickle
module cannot be found. If you add tracing to pickle.py
, you will see that it 'module'
refers to __main__
:
def load_global(self):
module = self.readline()[:-1]
name = self.readline()[:-1]
print("module {} name {}".format(module, name)) # I added this.
klass = self.find_class(module, name)
self.append(klass)
Reusing the same code again with this additional print statement gives this:
module multiprocessing.process name Process
module __main__ name f
< same traceback as before>
It's worth noting that this example actually works great on Posix platforms as it is os.fork()
used to spawn child processes, which means that any function defined before the created Pool
one will be available in the child __main__
. So, while the above example will work, this one will still fail because the working function is defined after creation Pool
(which means after being called os.fork()
):
>>> import multiprocessing
>>> p = multiprocessing.Pool(2)
>>> def f(a):
... print(a)
...
>>> p.apply(f, "hi")
Process PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 231, in _bootstrap
self.run()
File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 57, in worker
task = get()
File "/usr/lib64/python2.6/multiprocessing/queues.py", line 339, in get
return recv()
AttributeError: 'module' object has no attribute 'f'
source to share