Dstream apache spark input layout

I am trying to mock the input dstream while writing a unit test spark stream. I can mock the RDDs, but when I try to convert them to dstream, the dstream object becomes empty. I used the following code -

val lines = mutable.Queue[RDD[String]]()
val dstream = streamingContext.queueStream(lines)

// append data to DStream
lines += sparkContext.makeRDD(Seq("To be or not to be.", "That is the question."))

      

Any help regarding the same would be much appreciated.

+3
unit-testing apache-spark rdd spark-streaming dstream


source to share


No one has answered this question yet

Check out similar questions:

1
Why does textFileStream dstream give empty RDDs as if no files were being processed?
1
Pyspark Writing DStream data to Elasticsearch using saveAsNewAPIHadoopFile
1
Creating dstream programmatically in apache spark
0
Unable to iterate over a list of keys extracted from Dstream to List coverage when using spark stream with kafka
0
Single test cases in Spark Streaming
0
Spark Streaming - How to get results from foreachRDD function?
0
Testing Twitter with the Spark Streaming API
0
Apache Spark Streaming: how to compare 2 dataframes in 2 streams
0
Spark Stream Dstream RDD Fix
0
dstream [Double] for scala handling



All Articles
Loading...
X
Show
Funny
Dev
Pics