Pandas: Synchronizing dates in different columns with read_csv
I have an ascii file where dates are formatted like this:
Jan 20 2015 00:00:00.000
Jan 20 2015 00:10:00.000
Jan 20 2015 00:20:00.000
Jan 20 2015 00:30:00.000
Jan 20 2015 00:40:00.000
When uploading a file to pandas, each column above gets its own column in the pandas dataframe. I've tried the following options:
from pandas import read_csv
from datetime import datetime
df = read_csv('file.txt', header=None, delim_whitespace=True,
parse_dates={'datetime': [0, 1, 2, 3]},
date_parser=lambda x: datetime.strptime(x, '%b %d %Y %H %M %S'))
I am getting a couple of errors:
TypeError: <lambda>() takes 1 positional argument but 4 were given
ValueError: time data 'Jun 29 2017 00:35:00.000' does not match format '%b %d %Y %H %M %S'
I am confused because:
- I am passing a dict for
parse_dates
to parse different columns as one date. - I use:
%b
- abbreviated month name,%d
- day of month,%Y
year with century,%H
24 hour,%M
- minute and%S
- second
Does anyone see what I am doing wrong?
Edit:
I have tried date_parser=lambda x: datetime.strptime(x, '%b %d %Y %H:%M:%S')
which returnsValueError: unconverted data remains: .000
Edit 2:
I tried what @MaxU suggested in his update, but it was problematic because my original data is formatted like this:
Jan 1 2017 00:00:00.000 123 456 789 111 222 333
I'm only interested in the first 7 columns, so I import a file with the following:
df = read_csv(fn, header=None, delim_whitespace=True, usecols=[0, 1, 2, 3, 4, 5, 6])
Then to create a column with time and time information from the first 4 columns, I try:
df['datetime'] = to_datetime(df.ix[:, :3], format='%b %d %Y %H:%M:%S.%f')
However, this doesn't work because it to_datetime
expects "integer, float, string, datetime, list, tuple, 1-d array, Series" as the first argument and df.ix[:, :3]
returns a dataframe with the following format:
0 1 2 3
0 Jan 1 2017 00:00:00.000
How can I feed in each row of the first four columns a value to_datetime
to get one column datetimes
?
Edit 3:
I think I solved the second problem. I just use the following command and do everything when I read my file (I was just missing %f
to parse the last seconds):
df = read_csv(fileName, header=None, delim_whitespace=True,
parse_dates={'datetime': [0, 1, 2, 3]},
date_parser=lambda x: datetime.strptime(x, '%b %d %Y %H:%M:%S.%f'),
usecols=[0, 1, 2, 3, 4, 5, 6])
The whole reason I wanted to parse it manually, instead of letting pandas handle it like @MaxU, was suggesting to check if manual instruction would be faster - and it is! From my tests, the snippet above is about 5-6 times faster than allowing pandas to output the parsing for you.
source to share
Pandas (tested with version 0.20.1) is smart enough to do this for you:
In [4]: pd.read_csv(fn, sep='\s+', parse_dates={'datetime': [0, 1, 2, 3]})
Out[4]:
datetime
0 2015-01-20 00:10:00
1 2015-01-20 00:20:00
2 2015-01-20 00:30:00
3 2015-01-20 00:40:00
UPDATE: if all records are in the same format, you can try doing it like this:
df = pd.read_csv(fn, sep='~', names=['datetime'])
df['datetime'] = pd.to_datetime(df['datetime'], format='%b %d %Y %H:%M:%S.%f')
source to share