Check the number of unique values ββin each column of the matrix in the spark
I have a csv file that is currently stored as a dataframe in spark
scala> df
res11: org.apache.spark.sql.DataFrame = [ο»Ώ2013-03-25 12:49:36.000: string, OES_PSI603_EC1: string, 250.3315__SI: string, 250.7027__SI: string, 251.0738__SI: string, 251.4448__SI: string, 251.8159__SI: string, 252.1869__SI: string, 252.5579__SIF: string, 252.9288__SI: string, 253.2998__SIF: string, 253.6707__SIF: string, 254.0415__CI2: string, 254.4124__CI2: string, 254.7832__CI2: string, 255.154: string, 255.5248__NO: string, 255.8955__NO: string, 256.2662__NO: string, 256.6369: string, 257.0075: string, 257.3782: string, 257.7488: string, 258.1193: string, 258.4899: string, 258.8604__NO: string, 259.2309__NO: string, 259.6013__NO: string, 259.9717__N2: string, 260.3421__N2: string, 260.7125__N2: string, 261.4531: string, 261.8234: string, 262.1937: string, 262.5639: string, 262.9341: s...
scala>
I want to count the number of unique items in each column. How can i do this?
+3
source to share
1 answer
You can use a function countDistinct
for each column.
For example, in pyspark:
df = spark.createDataFrame([ (1, 1), (1, 3), (2, 1), (3, 2), (3, 3) ], ["user_id", "genre_id"])
>>> df.show()
+-------+--------+
|user_id|genre_id|
+-------+--------+
| 1| 1|
| 1| 3|
| 2| 1|
| 3| 2|
| 3| 3|
+-------+--------+
>>> import pyspark.sql.functions as F
>>> df.select( [ F.countDistinct(cn).alias("c_{0}".format(cn)) for cn in df.columns ] ).show()
+---------+----------+
|c_user_id|c_genre_id|
+---------+----------+
| 3| 3|
+---------+----------+
+1
source to share