KryoSerializer buffer overflow
I have a few Spark jobs that work fine in Spark 1.2, but not in 1.3 due to a KryoSerializer buffer overflow. I tried increasing spark.kryoserializer.buffer.max
to 20 GB and spark.kryoserializer.buffer
up to 2 GB. But it did not help. Am I using KryoSerializer incorrectly or am I not ready for prime time yet?
Through trial and error, it seems that the Cryo Serializer buffer is limited to 2GB. Not sure if this is by design or something?
+3
source to share
1 answer
My observation is correct: SPARK-6405 : Kryo's maximum buffer size limit to 2GB. Apparently the Databrick tech evangelist was unaware of this and told me to increase the buffer size.
+3
source to share