Who is responsible for maintaining delays?
Here are the Kafka Docs forpublic ConsumerRecords<K,V> poll(long timeout)
Get data for topics or sections specified using one of the subscribe / assign APIs. The error is that you did not subscribe to any topics or sections before polling the data. For each survey, the consumer will try to use the last absorbed bias as the starting bias and sample sequentially. The last consumed offset can be set manually via seek (TopicPartition, long) or automatically set as the last fixed offset for a signed topic list
My question is who (Broker or consumer or zookeper) is responsible for maintaining the offset and where is it stored (memory or disk)? Should the consumer store it in memory, will the consumer start reading it from the beginning, or should the consumer application persist to disk?
source to share
As stated in the Offsets and Consumer Positions section of the referenced docs, the offsets are stored by Kafka (the broker):
Kafka maintains a numerical offset for each entry in a section
In particular, it stores them in an "internal" consumer offset called "__consumer_offsets".
The "old consumer" api (deprecated in the upcoming v0.11) allows you to choose an offset in kafka or zookeeper.
Alternatively, you can store the offsets on the consumer side and always look for those offsets at startup if you so desire.
So, in general, depending on your consumer api and your preference, offsets can be stored in the broker or zookeeper and / or on the consumer side.
source to share