RawArray from numpy array?

I want to use a numpy array for multiple processes. Processes only read data, so I want to avoid copying. I know how to do this, if I can start with multiprocessing.sharedctypes.RawArray

and then create a numpy array with numpy.frombuffer

. But what if I am initially assigned a numpy array? Is there a way to initialize RawArray with numpy array data without copying the data? Or is there another way to exchange data between processes without copying it?

+3


source to share


2 answers


I also have some of your requirements: a) given a large numpy array, b) need to share it between a bunch of processes c) read-only, etc. And for this I used something according to

mynparray = #initialize a large array from a file
shrarr_base_ptr = RawArray(ctypes.c_double, len*rows*cols)
shrarr_ptr = np.frombuffer(shrarr_base_ptr)
shrarr_ptr = mynparray

      



where in my case mynparray is 3-D. As for the actual exchange, I used the following style and it works so far.

    inq1 = Queue()
    inq2 = Queue()  
    outq = Queue()
    p1 = Process(target = myfunc1, args=(inq1, outq,))
    p1.start()
    inq1.put((shrarr_ptr, ))
    p2 = Process(target = myfunc2, args=(inq2, outq,))
    p2.start()
    inq2.put((shrarr_ptr,))
    inq1.close()
    inq2.close()
    inq1.join_thread()
    inq2.join_thread()
    ....

      

+1


source


I'm not sure if this copies data from the inside out, but you can pass a flat array:



a = numpy.random.randint(1,10,(4,4))
>>> a
array([[5, 6, 7, 7],
       [7, 9, 2, 8],
       [3, 4, 6, 4],
       [3, 1, 2, 2]])

b = RawArray(ctypes.c_long, a.flat)
>>> b[:]
[5, 6, 7, 7, 7, 9, 2, 8, 3, 4, 6, 4, 3, 1, 2, 2]

      

-1


source







All Articles