Clever Geek Handbook

Exit the Kafka Console Consumer when all messages have been read

I know there must be a way to do this, but I cannot figure it out. I need to stop the kafka consumer as soon as I have read all messages from the queue.

Can anyone provide information on this?

+5


source to share


4 answers


You can pass the parameter: -consumer-timeout-ms with a value when starting the consumer, and it will throw an exception if no messages have been read in that time. For example, to stop a consumer if no new messages have been received in the last 2 seconds: kafka.consumer.ConsoleConsumer -consumer-timeout-ms 2000



You can see this and all other input options here

+6


source


You can use SimpleConsumerShell without the wait-at-logend option. See SystemTools-SimpleConsumerShell

For example:



./kafka-run-class.bat kafka.tools.SimpleConsumerShell --broker-list localhost:9092 --topic kafkademo --partition 0 --no-wait-at-logend
+3


source


Currently Kafka version 2.11-2.1.1 has a script called kafka-console-consumer.sh

.

He has a new flag: --timeout-ms

.

Essentially, this flag is the maximum time to wait before exiting when there is no new log to wait. This is in a millisecond time frame.

You can use this property to shutdown your console consumer after reading all messages.

+1


source


If you are not in the mood for using the Scala client, try kafkacat with a help -e

telling it to exit when the end of the section is reached.

eg. to destroy all messages from mitopic section 2 and then exit:

$ kafkacat -b mybroker -t mytopic -p 2 -o beginning -e

Or use the last 3000 posts and then exit:

$ kafkacat -b mybroker -t mytopic -p 2 -o -3000 -e
0


source






More articles:


All Articles