How to keep numpy from translation while creating an array of objects from different shaped arrays
I'm trying to store a list of various massive arrays as an array dtype=object
using np.save
(I know I can just sort the list, but I'm really curious how to do that). If I do this:
import numpy as np
np.save('test.npy', [np.zeros((2, 2)), np.zeros((3,3))])
it works. But this:
np.save('test.npy', [np.zeros((2, 2)), np.zeros((2,3))])
Gives me an error:
ValueError: could not broadcast input array from shape (2,2) into shape (2)
I think np.save
converts the list to an array first, so I tried:
x=np.array([np.zeros((2, 2)), np.zeros((3,3))])
y=np.array([np.zeros((2, 2)), np.zeros((2,3))])
Which has the same effect (the first works, the second doesn't. The result x
behaves as expected:
>>> x.shape
(2,)
>>> x.dtype
dtype('O')
>>> x[0].shape
(2, 2)
>>> x[0].dtype
dtype('float64')
I also tried to force the dtype of the object:
np.array([np.zeros((2, 2)), np.zeros((2,3))], dtype=object)
No success. It seems that numpy is trying to pass an array with equal first dimension to a new array and realizes too late that their shape is different. Oddly enough, it worked at some point - so I'm really wondering what the difference is, and how to do it properly.
EDIT: I figured out which case it worked before: The only difference is that the numpy arrays in the list are of a different datatype. It works with dtype('<f8')
, but not with dtype('float64')
, I'm not even sure what the difference is.
EDIT 2: I found a very non-pythonic way to solve my problem, I am adding it here, maybe it helps to understand what I wanted to do:
array_list=np.array([np.zeros((2, 2)), np.zeros((2,3))])
save_array = np.empty((len(array_list),), dtype=object)
for idx, arr in enumerate(array_list):
save_array[idx] = arr
np.save('test.npy', save_array)
source to share
One of the first things it does np.save
is
arr = np.asanyarray(arr)
So yes, it is trying to turn your list into an array.
Constructing an array of objects from arrays or lists of arbitrary size is tricky. np.array(...)
tries to create as large a sized array as possible, even if trying to concatenate the inputs if possible. The surest way to do what you did is to create an array empty
and fill it.
A slightly more compact way of constructing an array of objects:
In [21]: alist = [np.zeros((2, 2)), np.zeros((2,3))]
In [22]: arr = np.empty(len(alist), dtype=object)
In [23]: arr[:] = alist
In [24]: arr
Out[24]:
array([array([[ 0., 0.],
[ 0., 0.]]),
array([[ 0., 0., 0.],
[ 0., 0., 0.]])], dtype=object)
Here are 3 scenarios:
Arrays that fit the shape are combined into a 3D array:
In [27]: np.array([np.zeros((2, 2)), np.zeros((2,2))])
Out[27]:
array([[[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.]]])
In [28]: _.shape
Out[28]: (2, 2, 2)
Arrays that do not match in the first dimension create an array of objects
In [29]: np.array([np.zeros((2, 2)), np.zeros((3,2))])
Out[29]:
array([array([[ 0., 0.],
[ 0., 0.]]),
array([[ 0., 0.],
[ 0., 0.],
[ 0., 0.]])], dtype=object)
In [30]: _.shape
Out[30]: (2,)
And an awkward intermediate case (which can even be described as a bug). The first measurements coincide, but in the second they don't):
In [31]: np.array([np.zeros((2, 2)), np.zeros((2,3))])
...
ValueError: could not broadcast input array from shape (2,2) into shape (2)
[ 0., 0.]])], dtype=object)
It's like it initialized the array (2,2,2)
and then found that (2,3) didn't fit. And the current logic does not allow it to back up and create an array of objects like it did in the previous script.
If you want to put two (2,2) arrays into an array of objects, you will have to use create and fill logic.
source to share