Is there a way to mimic the lowercase syntax of a higher-order R (binary) function inside spark or pyspark?
In R
I can write the following:
## Explicit Reduce(function(x,y) x*y, c(1, 2, 3)) # returns 6
However, I can also do it less explicitly with the following:
## Less explicit Reduce(`*`, c(1, 2, 3)) # also returns 6
In
pyspark
I could do the following:
rdd = sc.parallelize([1, 2, 3])
rdd.reduce(lambda a, b: a * b)
Question: Can you emulate the "shorthand" (less explicit) R syntax
Reduce('*', ...)
with
pyspark
or some anonymous function?
+3
source to share
1 answer
In R, you are using a binary function. The multiplication operator (as with all operators) is actually a binary function. A type
`*`(2, 3)
to see what I mean.
In Python, the equivalent for multiplication is operator.mul
.
So:
rdd = sc.parallelize([1, 2, 3]) rdd.reduce(operator.mul)
+4
source to share