Adding a datetime field to repeat
I am trying to add a date field (datetime64) to an existing recurrence - without much success. I can create a datetime field, but when I try to add it to the array of records, I get the error:
ValueError: Error parsing datetime string "?" in position 0
However, if I pass the data as int64, I can add it to that format without issue. (code shown below)
Does anyone know why this is not working?
(my end goal is to write the replay to a netcdf file, so an appropriate date and time format will also be helpful for this purpose)
I am using python 2.7.6.1, numpy 1.8.1
Thanks Rob
import numpy as np
import numpy.lib.recfunctions as rf
# ----- make a recarray ---------
dummy = np.arange(0,10)
datarray = np.core.records.fromarrays([dummy,dummy,dummy],names='a,b,c')
# ----- make some time data using datetime64 ---------
sec = np.arange(0,10)*1000
millisec = np.arange(0,10)
mytime = sec + millisec
mytime64 = mytime.astype('timedelta64[ms]')
basetime = np.datetime64('1990-01-01')
mydatetime = mytime64+basetime
# ----- convert time data to int64 ---------
idatetime = mydatetime.astype('int64');
#------ try and append to recarray ---------
# this works
datarray = rf.append_fields(datarray, 'iDateTime', data=idatetime)
# this doesnt
datarray = rf.append_fields(datarray, 'DateTime', data=mydatetime)
source to share
Tracking:
Traceback (most recent call last):
File "stack26739733.py", line 30, in <module>
datarray = rf.append_fields(datarray, 'DateTime', data=mydatetime, usemask=False, dtypes=mydatetime.dtype)
File "/usr/local/lib/python2.7/site-packages/numpy/lib/recfunctions.py", line 641, in append_fields
dtype=base.dtype.descr + data.dtype.descr)
File "/usr/local/lib/python2.7/site-packages/numpy/ma/extras.py", line 163, in masked_all
mask=np.ones(shape, make_mask_descr(dtype)))
File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 2644, in __new__
_data = ndarray.view(_data, cls)
File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 2800, in __array_finalize__
self._fill_value = _check_fill_value(None, self.dtype)
File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 402, in _check_fill_value
dtype=ndtype,)
ValueError: Error parsing datetime string "?" at position 0
So this append function creates a masked array ( ma
) and checks "fill_value" for the added "dtype". Apparently _check_fill_value
does not understand dtype datetime
. Looks like incompatibility between array mask and datetime. A bug report numpy
might be ok.
Here's a simple one, do it yourself:
dt1 = np.dtype(datarray.dtype.descr + mydatetime.dtype.descr) newarray = np.empty(datarray.shape, dtype=dt1) for n in datarray.dtype.names: newarray[n] = datarray[n] newarray['f3'] = mydatetime
I am creating an empty array with a combined dtype. Then I copy the data from the fields datarray
and mydatetime
across the field. Since the number of fields is usually quite small compared to shape
, this copy is pretty fast. I'm pretty sure the function rf
does the same.
'f3'
is the name of the added field by default. You can change this on creation dt1
.
Result:
array([(0, 0, 0, datetime.datetime(1990, 1, 1, 0, 0)),
(1, 1, 1, datetime.datetime(1990, 1, 1, 0, 0, 1, 1000)),
(2, 2, 2, datetime.datetime(1990, 1, 1, 0, 0, 2, 2000)),
...
(9, 9, 9, datetime.datetime(1990, 1, 1, 0, 0, 9, 9000))],
dtype=[('a', '<i4'), ('b', '<i4'), ('c', '<i4'), ('f3', '<M8[ms]')])
Including this newarray
in a masked array throws the same error _check_fill_value
.
np.ma.masked_array(newarray)
source to share