The task is not serializable when using an object in the REPL
So another SO question asked me to try this:
object Foo{
def f = 1
}
sc.parallelize(List(1)).map(x=>{
val myF = Foo.f _
x + myF()
}
Which works, but the following doesn't:
object Foo{
def f = 1
def run(rdd: org.apache.spark.rdd.RDD[Int]) = rdd.map(x=>{
val myF = Foo.f _
x + myF()
}
}
Foo.run(sc.parallelize(List(1)))
I'll take a closer look at the serialization stack tomorrow when I can remove the clutter of the REPL output. But it must be the same. Why one of the ways doesn't scream and the other doesn't.
+3
source to share
No one has answered this question yet