Pandas reading NULL as float NaN instead of str
Given the file:
$ cat test.csv
a,b,c,NULL,d
e,f,g,h,i
j,k,l,m,n
If the third column should be viewed as str
.
When I made a string function on the column, pandas
read NULL
str as a NaN
float:
>>> import pandas as pd
>>> df = pd.read_csv('test.csv', names=[0,1,2,3,4], dtype={0:str, 1:str, 2:str, 3:str, 4:str})
>>> df[3].apply(str.strip)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/site-packages/pandas/core/series.py", line 2355, in apply
mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas/_libs/src/inference.pyx", line 1569, in pandas._libs.lib.map_infer (pandas/_libs/lib.c:66440)
TypeError: descriptor 'strip' requires a 'str' object but received a 'float'
To check:
>>> for i in df[3]:
... print (type(i), i)
...
<class 'float'> nan
<class 'str'> h
<class 'str'> m
I initialized dtype
on initialization but somehow it worked out above.
How to force-fix the type of a specific column?
Is there a way to automatically find these abnormal floats NaN
and change and then return to the line 'NULL'
?
source to share
Works for me astype
:
df[3] = df[3].astype(str)
for i in df[3]:
print (type(i), i)
<class 'str'> nan
<class 'str'> h
<class 'str'> m
Another solution is to use keep_default_na=False
in read_csv
:
import pandas as pd
from pandas.compat import StringIO
temp=u"""a,b,c,NULL,d
e,f,g,h,i
j,k,l,m,n"""
#after testing replace 'StringIO(temp)' to 'filename.csv'
df = pd.read_csv(StringIO(temp), names=[0,1,2,3,4], keep_default_na=False)
print (df)
0 1 2 3 4
0 a b c NULL d
1 e f g h i
2 j k l m n
for i in df[3]:
print (type(i), i)
<class 'str'> NULL
<class 'str'> h
<class 'str'> m
Then you can use the parameter na_values
if you need to parse NaN
in numeric columns, but it must be different, for example. NA
:
import pandas as pd
from pandas.compat import StringIO
temp=u"""a,b,c,NULL,1
e,f,g,h,2
j,k,l,m,NA"""
#after testing replace 'StringIO(temp)' to 'filename.csv'
df = pd.read_csv(StringIO(temp), names=[0,1,2,3,4], keep_default_na=False, na_values=['NA'])
print (df)
0 1 2 3 4
0 a b c NULL 1.0
1 e f g h 2.0
2 j k l m NaN
for i in df[3]:
print (type(i), i)
<class 'str'> NULL
<class 'str'> h
<class 'str'> m
for i in df[4]:
print (type(i), i)
<class 'numpy.float64'> 1.0
<class 'numpy.float64'> 2.0
<class 'numpy.float64'> nan
source to share