Spark Structtype to combine
I am using Spark 2.0.1 Scala 2.11
How do I provide a default value using coalesce
for the column in which StructType
?
Say ...
val ss = new StructType().add("x", IntegerType).add("y", IntegerType) val s = new StructType() .add("a", IntegerType) .add("b", ss) val d = Seq( Row(1, Row(1,2)), Row(2, Row(2,3)), Row(2, null) ) val rd = sc.parallelize(d) val df = spark.createDataFrame(rd, s)
Now df.select($"b").show
leads to
+-----+ | b | +-----+ |[1,2]| |[2,3]| | null| +-----+
My question is, how can I provide a default value (say [0,0]
) using coalesce
?
+3
source to share
1 answer
You can use the function struct
by passing two values lit(0)
named after the structure names you already have:
df.select(coalesce($"b", struct(lit(0).as("x"), lit(0).as("y"))))
.show()
// +---------------------------------------+
// |coalesce(b, struct(0 AS `x`, 0 AS `y`))|
// +---------------------------------------+
// | [1,2]|
// | [2,3]|
// | [0,0]|
// +---------------------------------------+
+3
source to share