Test is equal for RDD defining mock behavior with Mockito
I am writing a unit test for some java spark program with Mockito, I am having a problem when I am trying to determine the behavior of a mocked object method like:
when(mock.method(someRDD)).thenReturn(0);
Since RDD does not override equals () function, the mocked behavior only works with rdd passed in methods is the same reference of this "someRDD".
I wonder if there is a way to customize the "equals ()" behavior in Mockito in mocking methods? or maybe I should use a different frame scheme instead?
source to share
You can write your own ArgumentMatcher
to correlate between the passed argument and the expected one. Assuming this is just a straight forward comparison of RDD fields, you can use the Mockito built in refEq
, which uses reflection and just compares each field individually:
when(mock.method(refEq(someRDD))).thenReturn(0);
source to share
Have a look at https://github.com/holdenk/spark-testing-base for testing with spark rdds. Also, here's a tutorial on how to use it. http://www.jesse-anderson.com/2016/04/unit-testing-spark-with-java/
This will do what you are looking for. Typically the test will look like this:
JavaRDD<A> inputRdd = jsc().parallelizePairs(inputData);
JavaRDD<B> transformedRdd = someMethod(inputRdd);
JavaRDD<B> expectedRdd = jsc().parallelize(expectedData);
JavaRDDComparisons.assertRDDEquals(transformedRdd, expectedRdd);
However, please note that as far as I remember, there might be clashes with the contemplative version or other maven dependencies that you might be using. To get around this, I ran the tests in a separate project.
EDIT: if you are testing a couple of RDDs you need to add a tag like ...
ClassTag<Tuple2<K, V>> tag = scala.reflect.ClassTag$.MODULE$.apply(Tuple2.class);
JavaRDDComparisons.assertRDDEquals(JavaRDD.fromRDD(JavaPairRDD.toRDD(transformedRdd), tag),
JavaRDD.fromRDD(JavaPairRDD.toRDD(expectedRdd), tag));
source to share