How to manipulate numpy arrays for use with ESRI arcpy.da.NumPyArrayToTable
ESRI provides access to moving data from tables to arrays and vice versa. I have a script that takes rewriting data from an api call and converts it to arrays, does some simple math, and then ideally outputs it to a table. To do the math, the array cannot be rec array. No combination of vstack, hstack or concatenate seemed to give a good result. I have resorted to creating individual 1-dimensional arrays as repeats and then using the merge function in np.lib.recfunctions.merge_arrays. of course there is a better way.
Returning ESRI from TableToNumPyArray:
>>> testArray
array([ (41039000100.0, 2628.0, 100.0, 2339.0, 135.0, 18.0, 22.0, 16.0, 25.0, 0.0, 92.0, 0.0, 92.0, 0.0, 92.0, 0.0, 92.0, 6.0, 9.0, 249.0, 90.0, 0.0, 92.0, 1, u'41039000100'),
...
dtype=[('Geo_id', '<f8'), ('TotalUnits', '<f8'), ('MOE_Total', '<f8'), >('Total_1_detached', '<f8'), ('MOE_Total_1_detached', '<f8'), ('Total_1_attached', >'<f8'), ('MOE_Total_1_attached', '<f8'), ('Total_2', '<f8'), ('MOE_Total_2', '<f8'), >('Total_3_or_4', '<f8'), ('MOE_Total_3_or_4', '<f8'), ('Total_5_to_9', '<f8'), >('MOE_Total_5_to_9', '<f8'), ('Total_10_to_19', '<f8'), ('MOE_Total_10_to_19', '<f8'), >('Total_20_to_49', '<f8'), ('MOE_Total_20_to_49', '<f8'), ('Total_50_or_more', '<f8'), >('MOE_Total_50_or_more', '<f8'), ('Total_Mobile_home', '<f8'), ('MOE_Total_Mobile_home', '<f8'), ('Total_Boat_RV_van_etc', '<f8'), ('MOE_Total_Boat_RV_van_etc', '<f8'), >('ObjectID', '<i4'), ('geo_id_t', '<U50')])
My code snippet looks like
try:
# Assign Geo_id array
Geo_id_array = B25008_001E_array[...,0]
Tpop_array = B25008_001E_array[...,1]
Tunits_array = B25024_001E_array[...,1]
# divide by sero is possible for real rowns and definite for the end-of-file
# tract, so convert nan in the HHsize_array to zero with nan_to_num
# HHsize_array = Tpop_array.view(np.float32)/Tunits_array.view(np.float32)
HHsize_array = Tpop_array/Tunits_array
HHsize_array = nan_to_num(HHsize_array)
# Table_array = array(vstack((Geo_id_array, Tpop_array, Tunits_array, HHsize_array)), dtype = ([('Geo_id', '|S13'), ('Tpop', np.int32), ('Tunits_array', np.int32), ('HHsize', np.float32)]))
# Table_array = np.hstack((Geo_id_array, Tpop_array, Tunits_array, HHsize_array))
Geo_id_recarray = np.array(Geo_id_array, dtype = ([('Geo_id', '|S13')]))
Tpop_recarray = np.array(Tpop_array, dtype = ([('Tpop', np.int32)]))
Tunits_recarray = np.array(Tunits_array, dtype = ([('Tunits_array', np.int32)]))
HHsize_recarray = np.array(HHsize_array, dtype = ([('HHsize', np.float32)]))
arrays = [Geo_id_recarray, Tpop_recarray, Tunits_recarray, HHsize_recarray]
MergedArray = np.lib.recfunctions.merge_arrays(arrays, usemask=False)
print
print
except Exception as e:
# If an error occurred, print line number and error message
import traceback, sys
tb = sys.exc_info()[2]
print "An error occured on line %i" % tb.tb_lineno
print str(e)
I would prefer to concatenate / concatenate / collect the arrays before structuring them. Thoughts?
source to share
You should be able to use structured arrays (you don't technically use recarrays ) to do "simple math". I'm not sure if you're showing the math you want to do, but for example if you want:
HHsize_array = Tpop_array/Tunits_array
But I don't want to have all these separate arrays, you could just do the math on the representations of the main (combined array), let's call it data
:
data['HHsize'] = data['Tpop']/data['Tunits']
where HHsize
, Tpop
and Tunits
are all the field names in one named structured array data
, so you have
>>> data.dtype
dtype([('Geo_id', '|S13'), ('Tpop', np.int32), ('Tunits_array', np.int32), ('HHsize', np.float32)])
source to share