Multiprocessing: pool and pickle error - pickle error: cannot pickle <type 'instancemethod'>: lookup for __builtin __ attribute. Instancemethod failed
I have two files:
x.py
class BF(object)
def __init__():
.
.
def add(self,z):
.
.
y.py
from y import BF
def FUNC((a,b,bf))
.
.
bf.add(x)
.
.
return bf
.
.
if __name__ == '__main__':
pool = multiprocessing.Pool(3)
for i in range(len(sl)):
bf_set.append(BF())
results = pool.map(FUNC,zip(sl, itertools.repeat(aa), bf_set))
I also tried to define BF inside FUNC, but I got it:
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
I've read several posts about related problems, but they have their own pool.map () inside the class, so the solutions cannot be applied to this problem (I think).
Any idea?
source to share
I mostly use what you have, but turn it into working code. No serialization problems if you are using dill
. I am using a fork multiprocessing
called pathos.multiprocessing
which uses dill
instead pickle
.
>>> def FUNC((a,b,bf)):
... z = a+b
... bf.add(z)
... return bf
...
>>> class BF(object):
... def add(self, z):
... self.z += z
... def __init__(self):
... self.z = 0
...
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> pool = Pool()
>>>
>>> f = BF()
>>> f.add(1)
>>> f.z
1
>>>
>>> FUNC((0,1,f))
<__main__.BF object at 0x10d387f50>
>>>
>>> FUNC((0,1,f)).z
2
>>>
>>> sl = [BF() for i in range(10)]
>>> results = pool.map(FUNC, zip(range(len(sl)), range(len(sl)), sl))
>>> [bf.z for bf in results]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
It works because it pathos
uses dill
which can serialize almost anything in python.
>>> import dill as pickle
>>> pickle.loads(pickle.dumps(bf.add))
<bound method BF.add of <__main__.BF object at 0x10d383950>>
>>> pickle.loads(pickle.dumps(BF.add))
<unbound method BF.add>
Get it pathos
and dill
at: https://github.com/uqfoundation
source to share