Cython - defining 2d arrays
here is the cython code i am trying to optimize,
import cython
cimport cython
from libc.stdlib cimport rand, srand, RAND_MAX
import numpy as np
cimport numpy as np
def genLoans(int loanid):
cdef int i, j, k
cdef double[:,:,:] loans = np.zeros((240, 20, 1000))
cdef double[:,:] aggloan = np.zeros((240, 20))
for j from 0<=j<1000:
srand(loanid*1000+j)
for i from 0<=i<240:
for k from 0<=k<20:
loans[i,k,j] = rand()
###some other logics
aggloan[i,k] += loans[i,k,j]/1000
return aggloan
cython -a shows
I think when I try to initialize credits with zero array and aggloan numpy slows me down. However, I need to run 5000 credits. Just wondering if there are other ways to avoid using numpy when I define 3d / 2d and return arrays ...
source to share
The yellow part is due to the Numpy call where you are allocating the array. What you can do is pass these arrays as arguments to a function and reuse them from one to the other.
Also, I see that you are rewriting all the elements, so you require memory, write it in zeros and then enter your numbers. If you are sure that you are overwriting all elements, you can use np.empty
one that will not initialize variables.
Note. The Linux kernel has a specific way of allocating memory, initialized to 0, which is faster than any other value, and modern Numpy can use it, but it is still slower than
empty
:
In [4]: %timeit np.zeros((100,100))
100000 loops, best of 3: 4.04 ยตs per loop
In [5]: %timeit np.ones((100,100))
100000 loops, best of 3: 8.99 ยตs per loop
In [6]: %timeit np.empty((100,100))
1000000 loops, best of 3: 917 ns per loop
Last but not least, are you sure this is your bottleneck? I don't know what kind of processing you are doing, but yellow is the number of lines of C code, not the time. In any case, starting with the timings, using empty
should speed it up four times. If you want more, place the rest of your code in CR.
Edit:
Extending my second suggestion: your function signature could be
def genLoans(int loanid, cdef double[:,:,:] loans, cdef double[:,:] aggloan):
You initialize the arrays before your loop and just pass them over and over.
Anyway, on my machine (Linux Intel i5) it takes 9 ฮผs, so you are wasting 45ms in total. This is definitely not your bottleneck. Profile!
source to share