Counting value changes in each column in dataframe in pandas ignoring NaN changes
I am trying to count the number of changes to a value in each column in a dataframe in pandas. The code I'm working fine, except for NaNs: If a column contains two subsequent NaNs, it counts as a value change, which I don't want. How can I avoid this?
I am doing the following (thanks to unutbu answer ):
import pandas as pd
import numpy as np
frame = pd.DataFrame({
'time':[1234567000 , np.NaN, np.NaN],
'X1':[96.32,96.01,96.05],
'X2':[23.88,23.96,23.96]
},columns=['time','X1','X2'])
print(frame)
changes = (frame.diff(axis=0) != 0).sum(axis=0)
print(changes)
changes = (frame != frame.shift(axis=0)).sum(axis=0)
print(changes)
returns:
time X1 X2
0 1.234567e+09 96.32 23.88
1 NaN 96.01 23.96
2 NaN 96.05 23.96
time 3
X1 3
X2 2
dtype: int64
time 3
X1 3
X2 2
dtype: int64
Instead, the results should be (note the change in the time column):
time 2
X1 3
X2 2
dtype: int64
+3
source to share
1 answer
change = (frame.fillna(0).diff() != 0).sum()
Output:
time 2
X1 3
X2 2
dtype: int64
NaN are "truthful" . Then change NaN to zero.
nan - nan = nan
nan != 0 = True
fillna(0)
0 - 0 = 0
0 != 0 = False
+3
source to share