【kafka】kafka数据迁移、分区副本重分配

2022-05-16 本文已影响0人 Bogon

一、分区副本重分配

1 生成推荐配置脚本

关键参数--generate
在进行分区副本重分配之前,最好是用下面方式获取一个合理的分配文件;
编写move-json-file.json文件; 这个文件就是告知想对哪些topic进行重新分配的计算。

$ cat move-json-file.json

{
  "topics": [
    {"topic": "topic_name"}
  ],
  "version": 1
}

然后执行下面的脚本,--broker-list "0,1,2" 这个参数是你想要分配的Brokers。

$ sh bin/kafka-reassign-partitions.sh --zookeeper xx.xx.xx.xx:2181 --topics-to-move-json-file config/move-json-file.json --broker-list "0,1,2"   --generate

执行完毕之后会打印：

Current partition replica assignment

{"version":1,"partitions":[{"topic":"topic_name","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":3,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":8,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":5,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":6,"replicas":[0],"log_dirs":["any"]}]}

Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"topic_name","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":5,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":6,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":3,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":8,"replicas":[2],"log_dirs":["any"]}]}

Current partition replica assignment

{
    "version": 1,
    "partitions": [{
        "topic": "topic_name",
        "partition": 4,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 1,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 2,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 0,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 7,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 3,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 8,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 5,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 6,
        "replicas": [0],
        "log_dirs": ["any"]
    }]
}

Proposed partition reassignment configuration

{
    "version": 1,
    "partitions": [{
        "topic": "topic_name",
        "partition": 2,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 5,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 7,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 4,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 1,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 6,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 3,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 0,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 8,
        "replicas": [2],
        "log_dirs": ["any"]
    }]
}

需求注意的是，此时分区移动尚未开始，它只是告诉你当前的分配和建议。保存当前分配，以防你想要回滚它。

2. 执行json文件

关键参数--execute将上面得到期望的重新分配方式文件保存在一个json文件里面 reassignment-json-file.json

$ cat reassignment-json-file.json

{
    "version": 1,
    "partitions": [{
        "topic": "topic_name",
        "partition": 2,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 5,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 7,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 4,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 1,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 6,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 3,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 0,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 8,
        "replicas": [2],
        "log_dirs": ["any"]
    }]
}

$ sh bin/kafka-reassign-partitions.sh --zookeeper xx.xx.xx.xx:2181 --reassignment-json-file config/reassignment-json-file.json --execute

二、副本扩缩

kafka并没有提供一个专门的脚本来支持副本的扩缩, 不像kafka-topic.sh脚本一样,是可以扩分区的;
想要对副本进行扩缩,只能是曲线救国，利用kafka-reassign-partitions.sh来重新分配副本。

副本扩容

假设我们当前的情况是 3分区1副本,为了提供可用性,我想把副本数升到2;

计算副本分配方式
我们用 --generate 获取一下当前的分配情况,得到如下json

Current partition replica assignment

{
    "version": 1,
    "partitions": [{
        "topic": "topic_name",
        "partition": 4,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 1,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 2,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 0,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 7,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 3,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 8,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 5,
        "replicas": [2],
        "log_dirs": ["any"]
    }, {
        "topic": "topic_name",
        "partition": 6,
        "replicas": [0],
        "log_dirs": ["any"]
    }]
}