Is there a way to combine multiple waiting queues into one?
I have a spark stream that reads every 5 seconds from Kafka. Each read cycle creates a queue with InputSize = 10 entries (for example). After a while, I see a lot of games with the "in line" status. I would rather handle the large input size per batch. One way to achieve this is to increase the reading interval of the kafka from 5 seconds to a longer duration. But I wanted to see if there was a way to dynamically combine multiple queued packages into one big batch. For example: If my queues look like this:
Batch Input Time Size Scheduling (?) Processing Time (?) Output Operations: Continuity / General State 2017/05/14 17:48:15 4 entries - -
0/1 queued 2017/05/14 17:48: 10 6 entries - -
0/1 in the queue 2017/05/14 17:48:05 4 entries - -
0/1 in the queue
Is there a way to pragmatically combine three batches into one batch with input size = 14 records?
This is how I create a stream:
JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
jssc,
String.class,
String.class,
StringDecoder.class,
StringDecoder.class,
kafkaParams,
topicsSet
);
Thank you for your help.
source to share
No one has answered this question yet
See similar questions:
or similar: