The task is not serializable when using an object in the REPL

So another SO question asked me to try this:

object Foo{
  def f = 1
}

sc.parallelize(List(1)).map(x=>{
  val myF = Foo.f _
  x + myF()
}

      

Which works, but the following doesn't:

object Foo{
  def f = 1
  def run(rdd: org.apache.spark.rdd.RDD[Int]) = rdd.map(x=>{
    val myF = Foo.f _
    x + myF()
  }
}

Foo.run(sc.parallelize(List(1)))

      

I'll take a closer look at the serialization stack tomorrow when I can remove the clutter of the REPL output. But it must be the same. Why one of the ways doesn't scream and the other doesn't.

+3


source to share





All Articles