Multiprocessing: adding to 2 lists at the same time
I have this code:
from multiprocessing import Pool, Manager
import numpy as np
l = Manager().list()
def f(args):
a, b = args
l.append((a, b))
data = [(1,2), (3,4), (5,6)]
with Pool() as p:
p.map(f, data)
x, y = np.transpose(l)
# do something with x and y...
In reality, the data is an array with a lot of values, and the transpose operation is long and consumed.
I would like to add "a" and "b" directly to the x and y lists to avoid the transpose operation. It is important that the output is consistent with the data and looks like this: [[1,3,5], [2,4,6]]
What would be a sane way to do this?
+3
source to share
1 answer
Instead of trying to add from subprocesses, you can force the function to return values ββand add them to the main process; you don't need to worry about mutual access between subprocesses (also no need to use a manager).
from multiprocessing import Pool
def f(args):
a, b = args
# do something with a and b
return a, b
if __name__ == '__main__':
data = [(1,2), (3,4), (5,6)]
x, y = [], []
with Pool() as p:
for a, b in p.map(f, data): # or imap()
x.append(a)
y.append(b)
# do something with x and y
assert x == [1,3,5]
assert y == [2,4,6]
+4
source to share