Set intersection as columns in pandas
I have a df for example:
df=pd.DataFrame.from_items([('i', [set([1,2,3,4]), set([1,2,3,4]), set([1,2,3,4]),set([1,2,3,4])]), ('j', [set([2,3]), set([1]), set([4]),set([3,4])])])
so it looks like
>>> df
i j
0 {1, 2, 3, 4} {2, 3}
1 {1, 2, 3, 4} {1}
2 {1, 2, 3, 4} {4}
3 {1, 2, 3, 4} {3, 4}
I would like to calculate df.i.intersection (df.j) and assign it to column k. That is, I want this:
df['k']=[df.i.iloc[t].intersection(df.j.iloc[t]) for t in range(4)]
>>> df.k
0 {2, 3}
1 {1}
2 {4}
3 {3, 4}
Name: k, dtype: object
Is there a df.apply () for this? The actual df is millions of lines.
source to share
Working with set
s, list
and dict
in is a pandas
bit problematic because the best way to work with scalars is:
df['k'] = [x[0] & x[1] for x in zip(df['i'], df['j'])]
print (df)
i j k
0 {1, 2, 3, 4} {2, 3} {2, 3}
1 {1, 2, 3, 4} {1} {1}
2 {1, 2, 3, 4} {4} {4}
3 {1, 2, 3, 4} {3, 4} {3, 4}
df['k'] = [x[0].intersection(x[1]) for x in zip(df['i'], df['j'])]
print (df)
i j k
0 {1, 2, 3, 4} {2, 3} {2, 3}
1 {1, 2, 3, 4} {1} {1}
2 {1, 2, 3, 4} {4} {4}
3 {1, 2, 3, 4} {3, 4} {3, 4}
Solution with apply
:
df['k'] = df.apply(lambda x: x['i'].intersection(x['j']), axis=1)
print (df)
i j k
0 {1, 2, 3, 4} {2, 3} {2, 3}
1 {1, 2, 3, 4} {1} {1}
2 {1, 2, 3, 4} {4} {4}
3 {1, 2, 3, 4} {3, 4} {3, 4}
source to share
You can reproduce the specified intersection using the specified differences. The intersection between A and B is A minus elements from A that are not in B. (You can do this symmetrically with B).
This way you can use the dataframe sub method to manage the given differences:
df['k'] = df['i'].sub(df['i'].sub(df['j']))
# df['k'] = df['j'].sub(df['j'].sub(df['i'])) # equivalent
Which gives the expected output:
df
Out[11]:
i j k
0 {1, 2, 3, 4} {2, 3} {2, 3}
1 {1, 2, 3, 4} {1} {1}
2 {1, 2, 3, 4} {4} {4}
3 {1, 2, 3, 4} {3, 4} {3, 4}
source to share