Marking the coordinator dead for the group (Kafka)
One of my services stopped working a few hours ago. I restarted the server, but this consumer process just isn't registering with Kafka. This is what appears when the consumer starts up:
2017-04-21 10:22:54.887 INFO 18036 --- [afka-consumer-1] o.a.k.c.c.internals.ConsumerCoordinator : Revoking previously assigned partitions [] for group mysql-conversions-group
2017-04-21 10:22:54.890 INFO 18036 --- [afka-consumer-1] o.s.k.l.KafkaMessageListenerContainer : partitions revoked:[]
2017-04-21 10:22:54.890 INFO 18036 --- [afka-consumer-1] o.a.k.c.c.internals.AbstractCoordinator : (Re-)joining group mysql-conversions-group
2017-04-21 10:22:54.915 INFO 18036 --- [afka-consumer-1] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator li819-138.members.linode.com:9092 (id: 2147483646 rack: null) dead for group mysql-conversions-group
I have 4 different consumer groups and they are working fine. Any thoughts?
UPDATE
This is actually strange. We had to restart another Consumer of ours, and after restarting, another consumer that worked fine the second time now logs the same error. Basically, we now have 2 consumer processes that cannot communicate with Kafka for some reason.
source to share
Answering my own question:
Finally it worked. This is not a broker problem or a consumer problem. What happened, we turned on the firewall on both of our Kafka brokers around 8:30 AM. Consumers who were already associated with Kafka worked fine. However, when one of these consumers was restarted, they were unable to connect to the coordinator node. We turned off the firewall of both Kafka brokers and the data was immediately transferred to consumers. This is a temporary ATM machine, but at least we have something.
source to share