SparkSql does not support date format

I tried using a date file with sparkSql but its not working

As in javaSparkSqlExample example

I tried to add datecoloumn dob

In Person class, I added

setter and getter for dob as Date

when trying to execute

SELECT dob,name,age,count(*) as totalCount FROM Person WHERE dob >= '1995-01-01' AND age <= '2014-02-01';

      

Also tried to use also in query instead of <= and> = also

/Volumes/Official/spark-1.0.2-bin-hadoop2$: bin/spark-submit --class "SimpleApp" --master local[4] try/simple-project/target/simple-project-1.0.jar 
Spark assembly has been built with Hive, including Datanucleus jars on classpath
2014-08-21 11:42:47.360 java[955:1903] Unable to load realm mapping info from SCDynamicStore
=== Data source: RDD ===
Exception in thread "main" scala.MatchError: class java.util.Date (of class java.lang.Class)

      

+3


source to share


1 answer


It is still pending, instead Date

you can use Timestamp

in the class Person

SPARK-2552

Spark SQL currently supports Timestamp but not Date.

We're going to wait a while until version 1.2.0.



More details:

  • Type: Upgrade Upgrade
  • Status: Open
  • Priority: Minor
  • Resolution: Unresolved
  • Affects version / s: 1.0.1
  • Fix Version / s: None
  • Component / s: SQL
  • Target version / s: 1.2.0
+3


source







All Articles