How do I access batteries in a facility outside of the location where they were identified?
I define the helper map function as a separate def in the helper object, and it doesn't "see" the drive defined earlier in the code. Sparks docs recommends to recommend keeping the "remote" functions inside the object, but how can I make it all work with these batteries?
object mainlogic{
val counter = sc.accumulator(0)
val data = sc.textFile(...)// load logic here
val myrdd = data.mapPartitionsWithIndex(mapFunction)
}
object helper{
def mapFunction(...)={
counter+=1 // not compiling
}
}
+3
source to share
1 answer
Something like this should be passed as a parameter, just like any other code:
object mainlogic{
val counter = sc.accumulator(0)
val data = sc.textFile(...)// load logic here
val myrdd = data.mapPartitionsWithIndex(mapFunction(counter, _, _))
}
object helper{
def mapFunction(counter: Accumulator[Int], ...)={
counter+=1 // not compiling
}
}
Be sure to remember the note from the docs:
For battery updates performed only within activities , Spark ensures that each task update on the battery is only applied once, meaning that restarted tasks will not update the value. In conversions, users should be aware that each task update can be applied multiple times if tasks or work steps are restarted.
+1
source to share