Clever Geek Handbook

Scala generic encoder for spark vessel class

How can I get this method to compile. Oddly enough, the implicit sparks are already imported.

def loadDsFromHive[T <: Product](tableName: String, spark: SparkSession): Dataset[T] = {
    import spark.implicits._
    spark.sql(s"SELECT * FROM $tableName").as[T]
  }

This is mistake:

Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.
[error]     spark.sql(s"SELECT * FROM $tableName").as[T]
+3


source to share


1 answer


According to the original code, for org.apache.spark.sql.SQLImplicits

you need the type type TypeTag

for your type for the implicit one to exist Encoder

:



import scala.reflect.runtime.universe.TypeTag
def loadDsFromHive[T <: Product: TypeTag](tableName: String, spark: SparkSession): Dataset[T] = ...
+11


source






More articles:


All Articles