Spark SQL Unsupported Data Type TimestampType

I am new to sparks and scala. Read the text file and save its parquet file. For me, one of the fields I use is TimeStamp and its docs say spark1.1.0 supports java.util.TimeStamp.

The runtime error I get when saving to parquet files is

Exception on stream "main" java.lang.RuntimeException: Unsupported data type TimestampType in scala.sys.package $ .error (package.scala: 27) at org.apache.spark.sql.parquet.ParquetTypesConverter $$ anonfun $ fromDataType $ 2 .Apply (ParquetTypes.scala: 301)

Any recommendation is really visible.

thank

+3


source to share


1 answer


This is actually a known bug in versions prior to 1.3.0: https://issues.apache.org/jira/browse/SPARK-4987

The pull request is already complicated: https://github.com/apache/spark/pull/3820 , but it won't be available until 1.3.0 is released.



If you are in a hurry, you can build the main 1.3.0 branch locally, otherwise you will have to wait, but there is hope because RC2 Spark 1.3.0 started voting for the latest version yesterday, so if everything is ok with the candidate version, things can move pretty fast.

Hello,

+1


source







All Articles