How do I load the balance of a Kafka manual?
My version of kafka kafka_2.9.2-0.8.1.1
. I have two brokers in a cluster, 4 topics and each topic has 4 sections.
When I ran
sh kafka-topics.sh --describe --zookeeper rhost:2181
for all topics / sections, I see broker 1 as the leader.
How can I load the leader's balance?
For example, topic 1 and topic 2 have broker 1 as leader and topic 3 and topic 4 have broker 2 as leader.
source to share
You can use the Preferred Replica Leader Selection Tool :
sh kafka-preferred-replica-election.sh --zookeeper zklist
This ensures that the leading load across the brokers in the cluster is balanced.
source to share
The partitions should be automatically rebalanced as the default for the broker configuration parameter auto.leader.rebalance.enable
is true
. (see documentation )
However, by default, this rebalancing occurs every 5 minutes as defined by the parameter leader.imbalance.check.interval.seconds
. If you want this to happen more often, you will have to change this setting.
source to share
I know this is a little late, maybe you already have an answer. But to balance the leaders, you first need to make the brokers equally preferred between all sections, since the broker will be the preferred leader, he has two criteria. First it needs to be an insync resynthesis, secondly it needs to be the first item in the replica list (weird IMO). So if you have a small enough number of topics / sections, it will be easier, but you need to reassign the sections with the distribution of the first item (preferred copy) among all your brokers, then start using the preferred leader selection tool, which will be sure that the preferred leader is on really is a leader.
source to share