Why is Kafka's performance slow?

I have one simple theme and one simple Kafka consumer and producer using the default configuration.

The program is very simple, I have two threads.

In producer, it keeps on sending 16 byte data.

And on the consumer side, he continues to receive.

I found the fact that the throughput for the manufacturer is about 10 MB / s is ok.

But the bandwidth for the consumer is only 0.2MB / s. I've disabled all debug logs, but that doesn't make it any better. The test is performed on the local machine. Does any body have an idea of ​​what is going wrong? Thank!

The code I used is below: Manufacturer:

KafkaProducer producer = new KafkaProducer(props);
int size = 16;
byte[] payload = new byte[size];
String key = "key";
Arrays.fill(payload, (byte) 1);
ProducerRecord record = new ProducerRecord("test",0,key.getBytes(),payload);
while(true){
producer.send(record);
}

      

Consumer:

Properties consumerProps = new Properties();
consumerProps.put("zookeeper.connect", "localhost:2181");
consumerProps.put("group.id", "test");
ConsumerConnector consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps));
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put("test", 1);
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get("test");
ConsumerIterator<byte[], byte[]> it = streams.get(0).iterator();
while(it.hasNext()){
    it.next().message();
}

      

+3


source to share


1 answer


Try to customize your consumers with the following properties.

  1. fetch.min.bytes

  2. fetch.max.wait.ms

  3. max.partition.fetch.bytes



Additionally, you can adjust the method timeout setting poll()

for bandwidth.

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

      

0


source







All Articles