Clever Geek Handbook

NumPy speeds up 2D array setting elements

I am trying to efficiently index a 2D array in Python and the problem is that it is really slow.

Here's what I tried (simplified example):

xSize = veryBigNumber
ySize = veryBigNumber
a = np.ones((xSize,ySize))
N = veryBigNumber  
const = 1

for t in range(N):

  for i in range(xSize):
    for j in range(ySize):
      a[i,j] *= f(i,j)*const   # f(i,j) is an arbitrary function of i and j.

Now I would like to replace the nested loop with something more efficient. How to do it?

+3


source to share


3 answers


Your 2D array can be created using the following addition:

np.arange(200)[:,np.newaxis] + np.arange(200)

This type of vector operation is likely to be very fast:

>>> %timeit np.arange(200)[:,np.newaxis] + np.arange(200)
1000 loops, best of 3: 178 µs per loop


This method is not limited to adding. We can use the two arrays in the above operation as arguments to any generic function (usually abbreviated to ufunc).

For example:

>>> np.multiply(np.arange(5)[:,np.newaxis], np.arange(5))
array([[ 0,  0,  0,  0,  0],
       [ 0,  1,  2,  3,  4],
       [ 0,  2,  4,  6,  8],
       [ 0,  3,  6,  9, 12],
       [ 0,  4,  8, 12, 16]])

NumPy has built in ufuncs all the basic arithmetic operations and even more interesting ones. If you want a more exotic feature, NumPy lets you create your own ufunc .



Edit . To quickly explain the broadcasting going on in this method; you can think of it like this ...

np.arange(5)

creates a 1D array that looks like this:

array([0, 1, 2, 3, 4])

The code np.arange(5)[:,np.newaxis]

adds a second dimension (columns) to the range, creating this 2D array:

array([[0],
       [1],
       [2],
       [3],
       [4]])

To create the final 5x5 array using np.multiply

(although we could use any ufunc operation or binary arithmetic), NumPy takes 0

in the second array and changes it with each element, the first array makes a string like this:

[ 0,  0,  0,  0,  0]

Then it takes the second element in the second array 1

and multiplies it by the first array, creating this string:

[ 0,  1,  2,  3,  4]

This continues until we get the final 5x5 matrix.

+6


source


You can use indices

:

b=np.indices(a.shape)
a=b[0]+b[1]


Timings:

%%timeit
    ...: b=np.indices(a.shape)
    ...: c=b[0]+b[1]
1000 loops, best of 3: 370 µs per loop


%%timeit
for i in range(200):
  for j in range(200):
     a[i,j] = i + j

100 loops, best of 3: 10.4 ms per loop
+5


source


Since your output matrix a

is the elemental cardinality N of a matrix F

with elements f_ij = f(i,j) * const

, your code can simplify to

F = np.empty((xSize, ySize))
for i in range(xSize):
    for j in range(ySize):
        F[i,j] = f(i,j) * const

a = F ** n

For even more speed, you can trade the creation of the matrix F

with something more efficient, given that the function is f(i,j)

vectorized:

xmap, ymap = numpy.meshgrid(range(xSize), range(ySize))
F = f(xmap, ymap) * const
+1


source






More articles:


All Articles