Is there a way to combine multiple waiting queues into one?

I have a spark stream that reads every 5 seconds from Kafka. Each read cycle creates a queue with InputSize = 10 entries (for example). After a while, I see a lot of games with the "in line" status. I would rather handle the large input size per batch. One way to achieve this is to increase the reading interval of the kafka from 5 seconds to a longer duration. But I wanted to see if there was a way to dynamically combine multiple queued packages into one big batch. For example: If my queues look like this:

Batch Input Time Size Scheduling (?) Processing Time (?) Output Operations: Continuity / General State 2017/05/14 17:48:15 4 entries - -
0/1 queued 2017/05/14 17:48: 10 6 entries - -
0/1 in the queue 2017/05/14 17:48:05 4 entries - -
0/1 in the queue

Is there a way to pragmatically combine three batches into one batch with input size = 14 records?

This is how I create a stream:

            JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
                    jssc,
                    String.class,
                    String.class,
                    StringDecoder.class,
                    StringDecoder.class,
                    kafkaParams,
                    topicsSet
            );

      

Thank you for your help.

+3


source to share





All Articles