RawArray from numpy array?
I want to use a numpy array for multiple processes. Processes only read data, so I want to avoid copying. I know how to do this, if I can start with multiprocessing.sharedctypes.RawArray
and then create a numpy array with numpy.frombuffer
. But what if I am initially assigned a numpy array? Is there a way to initialize RawArray with numpy array data without copying the data? Or is there another way to exchange data between processes without copying it?
source to share
I also have some of your requirements: a) given a large numpy array, b) need to share it between a bunch of processes c) read-only, etc. And for this I used something according to
mynparray = #initialize a large array from a file
shrarr_base_ptr = RawArray(ctypes.c_double, len*rows*cols)
shrarr_ptr = np.frombuffer(shrarr_base_ptr)
shrarr_ptr = mynparray
where in my case mynparray is 3-D. As for the actual exchange, I used the following style and it works so far.
inq1 = Queue()
inq2 = Queue()
outq = Queue()
p1 = Process(target = myfunc1, args=(inq1, outq,))
p1.start()
inq1.put((shrarr_ptr, ))
p2 = Process(target = myfunc2, args=(inq2, outq,))
p2.start()
inq2.put((shrarr_ptr,))
inq1.close()
inq2.close()
inq1.join_thread()
inq2.join_thread()
....
source to share
I'm not sure if this copies data from the inside out, but you can pass a flat array:
a = numpy.random.randint(1,10,(4,4)) >>> a array([[5, 6, 7, 7], [7, 9, 2, 8], [3, 4, 6, 4], [3, 1, 2, 2]]) b = RawArray(ctypes.c_long, a.flat) >>> b[:] [5, 6, 7, 7, 7, 9, 2, 8, 3, 4, 6, 4, 3, 1, 2, 2]
source to share