kafka某个broker节点未启动造成无法选举leader

问题描述：topic都是多分区，多副本，broker节点关闭后，由于某些情况，某个节点无法启动，其他节点都启动后，topic或部分分区仍不可用，造成无法选举leader

1. 创建topic

topic：topic-demo02 3分区，3副本

[xuhaixing@xhx151 cluster]$ kafka-topics.sh --zookeeper 192.168.94.151:2181/kafkaCluster --create --topic topic-demo02 --replication-factor 3 --partitions 3
Created topic topic-demo02.
[xuhaixing@xhx151 cluster]$ kafka-topics.sh --zookeeper 192.168.94.151:2181/kafkaCluster --describe --topic topic-demo02
Topic: topic-demo02 PartitionCount: 3   ReplicationFactor: 3    Configs: 
    Topic: topic-demo02 Partition: 0    Leader: 2   Replicas: 2,3,1 Isr: 2,3,1
    Topic: topic-demo02 Partition: 1    Leader: 3   Replicas: 3,1,2 Isr: 3,1,2
    Topic: topic-demo02 Partition: 2    Leader: 1   Replicas: 1,2,3 Isr: 1,2,3
​
​
复制代码

2. 停掉三个broker

​
[xuhaixing@xhx151 cluster]$ kafka-topics.sh --zookeeper 192.168.94.151:2181/kafkaCluster --describe --topic topic-demo02
Topic: topic-demo02 PartitionCount: 3   ReplicationFactor: 3    Configs: 
    Topic: topic-demo02 Partition: 0    Leader: 1   Replicas: 2,3,1 Isr: 1
    Topic: topic-demo02 Partition: 1    Leader: 1   Replicas: 3,1,2 Isr: 1
    Topic: topic-demo02 Partition: 2    Leader: 1   Replicas: 1,2,3 Isr: 1
​
复制代码

此时Leader与ISR集合并不是空，而是1。因为zookeeper中存储的信息是kafka节点维护的，随着最后一个kafka节点的宕机，没有kafka节点更新zookeeper中的信息，故保持不变。

3. 启动broker.id 为2，3的节点

[xuhaixing@xhx151 cluster]$ kafka-topics.sh --zookeeper 192.168.94.151:2181/kafkaCluster --describe --topic topic-demo02
Topic: topic-demo02 PartitionCount: 3   ReplicationFactor: 3    Configs: unclean.leader.election.enable=false
    Topic: topic-demo02 Partition: 0    Leader: none    Replicas: 2,3,1 Isr: 1
    Topic: topic-demo02 Partition: 1    Leader: none    Replicas: 3,1,2 Isr: 1
    Topic: topic-demo02 Partition: 2    Leader: none    Replicas: 1,2,3 Isr: 1
复制代码

此时Leader为null，Isr依旧为未启动的broker节点，三个节点，恢复两个节点，此topic目前还是不可用状态。

部分分区不可用复现

1.创建如下topic

[xuhaixing@xhx151 cluster]$ kafka-topics.sh --zookeeper 192.168.94.151:2181/kafkaCluster --describe --topic topic-demo03
Topic: topic-demo03 PartitionCount: 3   ReplicationFactor: 2    Configs: 
    Topic: topic-demo03 Partition: 0    Leader: 3   Replicas: 3,1   Isr: 3,1
    Topic: topic-demo03 Partition: 1    Leader: 1   Replicas: 1,2   Isr: 1,2
    Topic: topic-demo03 Partition: 2    Leader: 2   Replicas: 2,3   Isr: 2,3
​
复制代码

2.停掉3个broker节点

3.启动broker.id=2的节点

[xuhaixing@xhx151 bin]$ kafka-topics.sh --zookeeper 192.168.94.151:2181/kafkaCluster --describe --topic topic-demo03
Topic: topic-demo03 PartitionCount: 3   ReplicationFactor: 2    Configs: 
Topic: topic-demo03 Partition: 0    Leader: none    Replicas: 3,1   Isr: 3
Topic: topic-demo03 Partition: 1    Leader: 2   Replicas: 1,2   Isr: 2
Topic: topic-demo03 Partition: 2    Leader: 2   Replicas: 2,3   Isr: 2
复制代码

此时分区0，Leader变成了none， Isr变成了3

4.启动broker.id=1的节点

再次查看状态

[xuhaixing@xhx151 bin]$ kafka-topics.sh --zookeeper 192.168.94.151:2181/kafkaCluster --describe --topic topic-demo03
Topic: topic-demo03 PartitionCount: 3   ReplicationFactor: 2    Configs: 
Topic: topic-demo03 Partition: 0    Leader: none    Replicas: 3,1   Isr: 3
Topic: topic-demo03 Partition: 1    Leader: 1   Replicas: 1,2   Isr: 2,1
Topic: topic-demo03 Partition: 2    Leader: 2   Replicas: 2,3   Isr: 2
复制代码

分区0依旧不可用

原因分析

这跟broker端的一个配置项有关 unclean.leader.election.enable，该参数决定是否从非ISR集合中选举leader。因怕非ISR节点集合数据延迟较大丢数据，所以默认为false。

因此，若kafka broker全部停机后，某topic的leader节点未启动，该topic依旧不可用

官方文档解释：

unclean.leader.election.enable Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss.

Type: boolean Default: false Valid Values: Server Default Property: unclean.leader.election.enable Importance: medium

解决方案

1.修改所有broker配置文件：unclean.leader.election.enable=true，重启

2.动态修改broker配置，不用重启

[xuhaixing@xhx151 cluster]$ kafka-configs.sh --bootstrap-server 192.168.94.151:9094 --entity-type brokers --entity-name 2 --add-config unclean.leader.election.enable=true  --alter
Completed updating config for broker 2.
​
复制代码

可以再用--delete-config删除配置

[xuhaixing@xhx151 cluster]$ kafka-configs.sh --bootstrap-server 192.168.94.151:9094 --entity-type brokers --entity-name 2 --delete-config unclean.leader.election.enable  --alter
Completed updating config for broker 2.
​
复制代码

3.修改topic配置参数

[xuhaixing@xhx151 cluster]$ kafka-configs.sh --bootstrap-server 192.168.94.151:9093 --entity-type topics --entity-name topic-demo02  --add-config unclean.leader.election.enable=true  --alter
Completed updating config for topic topic-demo02.
复制代码

选举成功

[xuhaixing@xhx151 bin]$ kafka-topics.sh --zookeeper 192.168.94.151:2181/kafkaCluster --describe --topic topic-demo02
Topic: topic-demo02	PartitionCount: 3	ReplicationFactor: 3	Configs: unclean.leader.election.enable=true
Topic: topic-demo02	Partition: 0	Leader: 1	Replicas: 2,3,1	Isr: 1
Topic: topic-demo02	Partition: 1	Leader: 1	Replicas: 3,1,2	Isr: 1
Topic: topic-demo02	Partition: 2	Leader: 1	Replicas: 1,2,3	Isr: 1
复制代码

再删除配置

[xuhaixing@xhx151 cluster]$ kafka-configs.sh --bootstrap-server 192.168.94.151:9093 --entity-type topics --entity-name topic-demo02  --delete-config unclean.leader.election.enable  --alter
Completed updating config for topic topic-demo02.
复制代码

更多优质文章请关注公众号：程序员星星