kafka

【kafka】给多分区的topic的分区加1副本,变为2副本

2022-10-18  本文已影响0人  Bogon

三节点的 kafka集群,borker.id 分别为 0 1 2

  1. 查看 topic 的 分区分布
$  /path/to/bin/kafka-topics.sh  --zookeeper  xx.xx.xx.xx:2181   --topic  testTopic  --describe
Topic:testTopic    PartitionCount:16       ReplicationFactor:1     Configs:
        Topic: testTopic   Partition: 0    Leader: 1       Replicas: 1     Isr: 1
        Topic: testTopic   Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: testTopic   Partition: 2    Leader: 2       Replicas: 2     Isr: 2
        Topic: testTopic   Partition: 3    Leader: 1       Replicas: 1     Isr: 1
        Topic: testTopic   Partition: 4    Leader: 0       Replicas: 0     Isr: 0
        Topic: testTopic   Partition: 5    Leader: 2       Replicas: 2     Isr: 2
        Topic: testTopic   Partition: 6    Leader: 1       Replicas: 1     Isr: 1
        Topic: testTopic   Partition: 7    Leader: 0       Replicas: 0     Isr: 0
        Topic: testTopic   Partition: 8    Leader: 2       Replicas: 2     Isr: 2
        Topic: testTopic   Partition: 9    Leader: 1       Replicas: 1     Isr: 1
        Topic: testTopic   Partition: 10   Leader: 0       Replicas: 0     Isr: 0
        Topic: testTopic   Partition: 11   Leader: 2       Replicas: 2     Isr: 2
        Topic: testTopic   Partition: 12   Leader: 1       Replicas: 1     Isr: 1
        Topic: testTopic   Partition: 13   Leader: 0       Replicas: 0     Isr: 0
        Topic: testTopic   Partition: 14   Leader: 2       Replicas: 2     Isr: 2
        Topic: testTopic   Partition: 15   Leader: 1       Replicas: 1     Isr: 1

从输出可以看出, testTopic 为 16 分区, 按照 1 0 2 的顺序 分布在 3 节点上 。

如果给每个分区加1个副本 ,给 1 加个到 0 上,给 0 加个到 1 上,给2 加个到 1 上,讲究个公平分配。

  1. 根据 topic 的 分区分布 构造 increase-replication-factor.json 文件

$ cat increase-replication-factor.json

{"version":1, "partitions":[
{"topic":"testTopic","partition":0,"replicas":[0,1]},
{"topic":"testTopic","partition":1,"replicas":[0,2]},
{"topic":"testTopic","partition":2,"replicas":[1,2]},
{"topic":"testTopic","partition":3,"replicas":[0,1]},
{"topic":"testTopic","partition":4,"replicas":[0,2]},
{"topic":"testTopic","partition":5,"replicas":[1,2]},
{"topic":"testTopic","partition":6,"replicas":[0,1]},
{"topic":"testTopic","partition":7,"replicas":[0,2]},
{"topic":"testTopic","partition":8,"replicas":[1,2]},
{"topic":"testTopic","partition":9,"replicas":[0,1]},
{"topic":"testTopic","partition":10,"replicas":[0,2]},
{"topic":"testTopic","partition":11,"replicas":[1,2]},
{"topic":"testTopic","partition":12,"replicas":[0,1]},
{"topic":"testTopic","partition":13,"replicas":[0,2]},
{"topic":"testTopic","partition":14,"replicas":[1,2]},
{"topic":"testTopic","partition":15,"replicas":[0,1]}]
}
  1. 执行增加副本操作
$   /path/to/bin/kafka-reassign-partitions.sh --zookeeper xx.xx.xx.xx:2181  --reassignment-json-file increase-replication-factor.json  --execute
  1. 查看执行进度
$  /path/to/bin/kafka-reassign-partitions.sh --zookeeper xx.xx.xx.xx:2181  --reassignment-json-file  increase-replication-factor.json  --verify

Status of partition reassignment:
Reassignment of partition testTopic-15 completed successfully
Reassignment of partition testTopic-2 completed successfully
Reassignment of partition testTopic-7 completed successfully
Reassignment of partition testTopic-12 completed successfully
Reassignment of partition testTopic-4 completed successfully
Reassignment of partition testTopic-11 completed successfully
Reassignment of partition testTopic-10 completed successfully
Reassignment of partition testTopic-8 completed successfully
Reassignment of partition testTopic-5 completed successfully
Reassignment of partition testTopic-13 completed successfully
Reassignment of partition testTopic-1 completed successfully
Reassignment of partition testTopic-14 completed successfully
Reassignment of partition testTopic-9 completed successfully
Reassignment of partition testTopic-6 completed successfully
Reassignment of partition testTopic-0 completed successfully
Reassignment of partition testTopic-3 completed successfully

要熟练使用 Kafka 自带的 kafka-reassign-partitions.sh 脚本工具来完成对 topic 的分区分配、分区副本增加等操作。
该脚本有三个参数:

--generate:配合着 --topics-to-move-json-file 可以生成分区分配策略,该参数适用于分区多的情况
--execute: 配合着 --reassignment-json-file   可以执行分区分配策略
--verify:  配合着 --reassignment-json-file   可以检查分区分配进度

通过以上命令,既可以分配分区,也可以增加分区副本数,非常方便。

参考

kafka 迁移分区创建规则json
https://www.orchome.com/454#item-8

kafka命令大全
https://www.orchome.com/454#item-8

kafka集群中有多个offsets.topic.replication.factor?
https://cloud.tencent.com/developer/ask/sof/1540657/answer/2102855

kafka高可用失败问题 3broker,单杀一个broker就不能消费的问题探讨
https://www.orchome.com/805

上一篇下一篇

猜你喜欢

热点阅读