Is there a way to combine multiple waiting queues into one?

I have a spark stream that reads every 5 seconds from Kafka. Each read cycle creates a queue with InputSize = 10 entries (for example). After a while, I see a lot of games with the "in line" status. I would rather handle the large input size per batch. One way to achieve this is to increase the reading interval of the kafka from 5 seconds to a longer duration. But I wanted to see if there was a way to dynamically combine multiple queued packages into one big batch. For example: If my queues look like this:

Batch Input Time Size Scheduling (?) Processing Time (?) Output Operations: Continuity / General State 2017/05/14 17:48:15 4 entries - -
0/1 queued 2017/05/14 17:48: 10 6 entries - -
0/1 in the queue 2017/05/14 17:48:05 4 entries - -
0/1 in the queue

Is there a way to pragmatically combine three batches into one batch with input size = 14 records?

This is how I create a stream:

            JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
                    jssc,
                    String.class,
                    String.class,
                    StringDecoder.class,
                    StringDecoder.class,
                    kafkaParams,
                    topicsSet
            );

      

Thank you for your help.

+3
apache-spark spark-streaming


source to share


No one has answered this question yet

See similar questions:

2
Combining micropackages in Spark Streaming

or similar:

6
Limiting Kafka batch size when using Spark Streaming
4
Spark Streaming: Queued Microbatches
3
High read latency with spark flow 2.2.1 and kafka 0.10.0.1
2
Spark Streaming & # 8594; DStream.checkpoint vs. SparkStreaming.checkpoint
2
Spark streaming with kafka - checkpoint restart
1
How to run multiple batch spark flow tasks at the same time?
1
Fixed 21 second duration of spark jet burst under sparks 2.1, not under Spark 2.0
0
Spark Streaming from Apache Kafka
0
What does the Input Size column indicate in the web interface?



All Articles
Loading...
X
Show
Funny
Dev
Pics