Redis 5 版本的高可用集群的水平扩展
2018-12-15 本文已影响306人
挂机的啊洋zzZ
1.jpg
Redis 5 版本的高可用集群的水平扩展
Redis系统介绍：
Redis的基础介绍与安装使用步骤：https://www.jianshu.com/p/2a23257af57b
Redis的基础数据结构与使用：https://www.jianshu.com/p/c95c8450c5b6
Redis核心原理：https://www.jianshu.com/p/4e6b7809e10a
Redis 5 之后版本的高可用集群搭建：https://www.jianshu.com/p/8045b92fafb2
Redis 5 版本的高可用集群的水平扩展：https://www.jianshu.com/p/6355d0827aea
Redis 5 集群选举原理分析：https://www.jianshu.com/p/e6894713a6d5
Redis 5 通信协议解析以及手写一个Jedis客户端：https://www.jianshu.com/p/575544f68615
Redis3.0以后的版本虽然有了集群功能，提供了比之前版本的哨兵模式更高的性能与可用性，但是集群的水平扩展却比较麻烦，今天就来带大家看看redis高可用集群如何做水平扩展，原始集群(见下图)由6个节点组成。
6个节点分布在一台机器上，采用三主三从的模式，以及进行水平新增的2个节点，一主一从。
实际应用中，最好用多台机器，比如说6个节点分布到3台机器上，redis在建立集群时为自动的将主从节点进行不同机器的分配，比如说：master-8001分布在192.168.5.100这台机器上，则它的slave-8004则不会在这台机器上，这是为了如果一台机器挂掉之后，还有其他的机器上的从节点进行替换master，以达到高可用的目的。
1.png
按之前的方法将集群进行启动。
分别启动6个节点：
/usr/local/redis/redis-5.0.2/src/redis-server /usr/local/redis-cluster/800*/redis.conf
查询启动情况：
ps -ef | grep redis
建立集群
/usr/local/redis/redis-5.0.2/src/redis-cli --cluster create --cluster-replicas 1 192.168.5.100:8001 192.168.5.100:8002 192.168.5.100:8003 192.168.5.100:8004 192.168.5.100:8005 192.168.5.100:8006
注意：如果之前redis集群给全部停掉了，这时候再建立集群时，会出现如下的情况
[ERR] Node 192.168.5.100:8001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
这个时候需要将每个节点下的这几个文件给删掉（测试情况删掉，实际应用不要删，这是备份文件以及节点信息，按实际的情况进行处理）：
appendonly.aof dump.rdb nodes-8001.conf
客户端连接8001端口的redis实例
/usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8001
查看集群状态
192.168.5.100:8001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:607
cluster_stats_messages_pong_sent:607
cluster_stats_messages_sent:1214
cluster_stats_messages_ping_received:602
cluster_stats_messages_pong_received:607
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1214
查看节点信息
192.168.5.100:8001> cluster nodes
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544881722939 6 connected
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544881722000 4 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544881721931 2 connected 5461-10922
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544881719000 3 connected 10923-16383
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 myself,master - 0 1544881719000 1 connected 0-5460
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544881720000 5 connected
这时候看一下这个信息
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544881721931 2 connected 5461-10922
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544881719000 3 connected 10923-16383
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 myself,master - 0 1544881719000 1 connected 0-5460
介绍一下redis的hash槽的概念
从上图可以看出，整个集群运行正常，三个master节点和三个slave节点
- 8001端口的实例节点存储0-5460这些hash槽
- 8002端口的实例节点存储5461-10922这些hash槽
- 8003端口的实例节点存储10923-16383这些hash槽
这三个master节点存储的所有hash槽组成redis集群的存储槽位，slave点是每个主节点的备份从节点，不显示存储槽位
Redis 集群中内置了 16384 个哈希槽，当需要在 Redis 集群中放置一个 key-value时，redis 先对 key 使用 crc16 算法算出一个结果，然后把结果对 16384 求余数
这样每个 key 都会对应一个编号在 0-16383 之间的哈希槽，redis 会根据节点数量大致均等的将哈希槽映射到不同的节点。
-
使用哈希槽的好处就在于可以方便的添加或移除节点。
-
当需要增加节点时，只需要把其他节点的某些哈希槽挪到新节点就可以了；
-
当需要移除节点时，只需要把移除节点上的哈希槽挪到其他节点就行了；
-
在这一点上，我们以后新增或移除节点的时候不用先停掉所有的 redis 服务。
用了哈希槽的概念，而没有用一致性哈希算法，不都是哈希么？这样做的原因是为什么呢？
Redis Cluster是自己做的crc16的简单hash算法，没有用一致性hash。Redis的作者认为它的crc16(key) mod 16384的效果已经不错了，虽然没有一致性hash灵活，但实现很简单，节点增删时处理起来也很方便。
为了动态增删节点的时候，不至于丢失数据么？
节点增删时不丢失数据和hash算法没什么关系，不丢失数据要求的是一份数据有多个副本。
还有集群总共有2的14次方，16384个哈希槽，那么每一个哈希槽中存的key 和 value是什么？
当你往Redis Cluster中加入一个Key时，会根据crc16(key) mod 16384计算这个key应该分布到哪个hash slot中，一个hash slot中会有很多key和value。你可以理解成表的分区，使用单节点时的redis时只有一个表，所有的key都放在这个表里；改用Redis Cluster以后会自动为你生成16384个分区表，你insert数据时会根据上面的简单算法来决定你的key应该存在哪个分区，每个分区里有很多key。
引用：https://blog.csdn.net/tianyeshiye/article/details/79600014
开始水平扩展
我们在原始集群基础上再增加一主(8007)一从(8008)，结构如下图
1.png
增加redis实例
在/usr/local/redis-cluster下创建8007和8008文件夹，并拷贝8001文件夹下的redis.conf文件到8007和8008这两个文件夹下
[root@localhost redis-cluster]# mkdir 8007 8008
[root@localhost redis-cluster]# cp 8001/redis.conf 8007
[root@localhost redis-cluster]# cp 8001/redis.conf 8008
按之前的方法修改8007、8008中redis.conf参数，修改完成后进行启动
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-server /usr/local/redis-cluster/8007/redis.conf
3193:C 15 Dec 2018 22:16:35.799 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3193:C 15 Dec 2018 22:16:35.799 # Redis version=5.0.2, bits=64, commit=00000000, modified=0, pid=3193, just started
3193:C 15 Dec 2018 22:16:35.799 # Configuration loaded
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-server /usr/local/redis-cluster/8008/redis.conf
3198:C 15 Dec 2018 22:16:41.682 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3198:C 15 Dec 2018 22:16:41.682 # Redis version=5.0.2, bits=64, commit=00000000, modified=0, pid=3198, just started
3198:C 15 Dec 2018 22:16:41.682 # Configuration loaded
查询启动情况：
ps -ef | grep redis
[root@localhost 8007]# ps -ef | grep redis
root 3066 1 0 21:37 ? 00:00:04 /usr/local/redis/redis-5.0.2/src/redis-server *:8001 [cluster]
root 3071 1 0 21:37 ? 00:00:05 /usr/local/redis/redis-5.0.2/src/redis-server *:8002 [cluster]
root 3076 1 0 21:37 ? 00:00:05 /usr/local/redis/redis-5.0.2/src/redis-server *:8003 [cluster]
root 3081 1 0 21:37 ? 00:00:05 /usr/local/redis/redis-5.0.2/src/redis-server *:8004 [cluster]
root 3086 1 0 21:37 ? 00:00:05 /usr/local/redis/redis-5.0.2/src/redis-server *:8005 [cluster]
root 3091 1 0 21:37 ? 00:00:05 /usr/local/redis/redis-5.0.2/src/redis-server *:8006 [cluster]
root 3194 1 0 22:16 ? 00:00:00 /usr/local/redis/redis-5.0.2/src/redis-server *:8007 [cluster]
root 3199 1 0 22:16 ? 00:00:00 /usr/local/redis/redis-5.0.2/src/redis-server *:8008 [cluster]
root 3205 2676 0 22:18 pts/0 00:00:00 grep --color=auto redis
这时候客户端连接8001端口的redis实例，查看节点信息，会发现并无8007、8008节点信息
/usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8001
192.168.5.100:8001> cluster nodes
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544883610414 6 connected
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544883607000 4 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544883611423 2 connected 5461-10922
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544883610000 3 connected 10923-16383
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 myself,master - 0 1544883610000 1 connected 0-5460
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544883609403 5 connected
那么开始配置8007为集群主节点
使用add-node命令新增一个主节点8007(master)，8007为新增节点，8001为已知存在节点，看到日志最后有"[OK] New node added correctly"提示代表新节点加入成功
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli --cluster add-node 192.168.5.100:8007 192.168.5.100:8001
>>> Adding node 192.168.5.100:8007 to cluster 192.168.5.100:8001
>>> Performing Cluster Check (using node 192.168.5.100:8001)
M: 662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006
slots: (0 slots) slave
replicates 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6
S: b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004
slots: (0 slots) slave
replicates 33206e9384297092b5b8a85c944f3564e5d983d7
M: 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005
slots: (0 slots) slave
replicates 662809cf2d5bb138912dea7fb1e452f6e0f149da
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.5.100:8007 to make it join the cluster.
[OK] New node added correctly.
查看集群状态,可以看到已知节点为7个，nodes中也显示了8007节点，但是现在没有hash槽分配到8007
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8001
192.168.5.100:8001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:7
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:2831
cluster_stats_messages_pong_sent:2805
cluster_stats_messages_sent:5636
cluster_stats_messages_ping_received:2799
cluster_stats_messages_pong_received:2831
cluster_stats_messages_meet_received:6
cluster_stats_messages_received:5636
192.168.5.100:8001> cluster nodes
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544883929000 6 connected
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544883930910 4 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544883929000 2 connected 5461-10922
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544883928000 3 connected 10923-16383
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 myself,master - 0 1544883928000 1 connected 0-5460
fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007@18007 master - 0 1544883927000 0 connected
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544883929897 5 connected
我们为新节点手工分配hash槽
使用redis-cli命令为8007分配hash槽，找到集群中的任意一个主节点(8001)，对其进行重新分片工作。
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli --cluster reshard 192.168.5.100:8001
>>> Performing Cluster Check (using node 192.168.5.100:8001)
M: 662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006
slots: (0 slots) slave
replicates 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6
S: b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004
slots: (0 slots) slave
replicates 33206e9384297092b5b8a85c944f3564e5d983d7
M: 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007
slots: (0 slots) master
S: 60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005
slots: (0 slots) slave
replicates 662809cf2d5bb138912dea7fb1e452f6e0f149da
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?
会询问要分多少个槽出来（1000）？分给哪个节点（8007）
How many slots do you want to move (from 1 to 16384)? 1000
What is the receiving node ID? fea53768189af3e3e4849038af13607f59ec84b0
然后有两种方式，一种是all，以将所有节点用作散列槽的源节点，一种是done，这种是你自己选择从哪个节点上拿出来节点分给8007
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
all是随机的，比如说我们要分出1000个，则3个主节点分别拿出333个，333个，334个节点分别8007，这里我们选择done，从8001拿1000个给8007
How many slots do you want to move (from 1 to 16384)? 1000
What is the receiving node ID? fea53768189af3e3e4849038af13607f59ec84b0
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: 662809cf2d5bb138912dea7fb1e452f6e0f149da
Source node #2: done
Ready to move 1000 slots.
Source nodes:
M: 662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
Destination node:
M: fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007
slots: (0 slots) master
Resharding plan:
Moving slot 0 from 662809cf2d5bb138912dea7fb1e452f6e0f149da
Moving slot 1 from 662809cf2d5bb138912dea7fb1e452f6e0f149da
Moving slot 2 from 662809cf2d5bb138912dea7fb1e452f6e0f149da
Moving slot 3 from 662809cf2d5bb138912dea7fb1e452f6e0f149da
.
.
.
Moving slot 996 from 662809cf2d5bb138912dea7fb1e452f6e0f149da
Moving slot 997 from 662809cf2d5bb138912dea7fb1e452f6e0f149da
Moving slot 998 from 662809cf2d5bb138912dea7fb1e452f6e0f149da
Moving slot 999 from 662809cf2d5bb138912dea7fb1e452f6e0f149da
Do you want to proceed with the proposed reshard plan (yes/no)? yes
.
.
.
Moving slot 995 from 192.168.5.100:8001 to 192.168.5.100:8007:
Moving slot 996 from 192.168.5.100:8001 to 192.168.5.100:8007:
Moving slot 997 from 192.168.5.100:8001 to 192.168.5.100:8007:
Moving slot 998 from 192.168.5.100:8001 to 192.168.5.100:8007:
Moving slot 999 from 192.168.5.100:8001 to 192.168.5.100:8007:
[root@localhost 8007]#
这时候我们再看一下节点信息
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8001
192.168.5.100:8001> cluster nodes
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544885201000 6 connected
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544885199000 4 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544885200000 2 connected 5461-10922
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544885201374 3 connected 10923-16383
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 myself,master - 0 1544885200000 1 connected 1000-5460
fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007@18007 master - 0 1544885200360 7 connected 0-999
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544885199351 5 connected
192.168.5.100:8001>
可以看到0-999已经分给8007了，而8001则从1000-5460
这时候我们配置8008为8007的从节点
添加从节点8008到集群中去并查看集群状态
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli --cluster add-node 192.168.5.100:8008 192.168.5.100:8001
>>> Adding node 192.168.5.100:8008 to cluster 192.168.5.100:8001
>>> Performing Cluster Check (using node 192.168.5.100:8001)
M: 662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001
slots:[1000-5460] (4461 slots) master
1 additional replica(s)
S: 194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006
slots: (0 slots) slave
replicates 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6
S: b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004
slots: (0 slots) slave
replicates 33206e9384297092b5b8a85c944f3564e5d983d7
M: 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007
slots:[0-999] (1000 slots) master
S: 60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005
slots: (0 slots) slave
replicates 662809cf2d5bb138912dea7fb1e452f6e0f149da
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.5.100:8008 to make it join the cluster.
[OK] New node added correctly.
查看节点状态
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8001
192.168.5.100:8001> cluster nodes
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544885494000 6 connected
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544885495000 4 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544885493850 2 connected 5461-10922
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544885496000 3 connected 10923-16383
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 myself,master - 0 1544885493000 1 connected 1000-5460
fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007@18007 master - 0 1544885496880 7 connected 0-999
71404f4e815c2e315926ac788389120f82029958 192.168.5.100:8008@18008 master - 0 1544885494857 0 connected
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544885496000 5 connected
192.168.5.100:8001>
可以看到8008是一个master节点，没有被分配任何的hash槽。
我们需要执行replicate命令来指定当前节点(从节点)的主节点id为哪个,首先需要连接新加的8008节点的客户端，然后使用集群命令进行操作，把当前的8008(slave)节点指定到一个主节点下
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8008
192.168.5.100:8008> cluster nodes
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544885646000 3 connected 10923-16383
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544885647000 1 connected
fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007@18007 master - 0 1544885647003 7 connected 0-999
71404f4e815c2e315926ac788389120f82029958 192.168.5.100:8008@18008 myself,master - 0 1544885643000 0 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544885646000 2 connected 5461-10922
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 master - 0 1544885648015 1 connected 1000-5460
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544885649022 3 connected
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544885646000 2 connected
192.168.5.100:8008> CLUSTER REPLICATE fea53768189af3e3e4849038af13607f59ec84b0
OK
192.168.5.100:8008> cluster nodes
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544885696000 3 connected 10923-16383
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544885697562 1 connected
fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007@18007 master - 0 1544885695543 7 connected 0-999
71404f4e815c2e315926ac788389120f82029958 192.168.5.100:8008@18008 myself,slave fea53768189af3e3e4849038af13607f59ec84b0 0 1544885693000 0 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544885692516 2 connected 5461-10922
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 master - 0 1544885694534 1 connected 1000-5460
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544885693528 3 connected
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544885696554 2 connected
192.168.5.100:8008>
扩展redis集群已经实现，下面进行删除节点
删除8008从节点
用del-node删除从节点8008，指定删除节点ip和端口，以及节点id
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli --cluster del-node 192.168.5.100:8008 71404f4e815c2e315926ac788389120f82029958
>>> Removing node 71404f4e815c2e315926ac788389120f82029958 from cluster 192.168.5.100:8008
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8007
192.168.5.100:8007> cluster nodes
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544885984990 2 connected
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544885983000 3 connected 10923-16383
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544885984000 2 connected 5461-10922
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544885983000 3 connected
fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007@18007 myself,master - 0 1544885981000 7 connected 0-999
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 master - 0 1544885983000 1 connected 1000-5460
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544885982000 1 connected
192.168.5.100:8007>
如上所示，8008这个slave节点已经移除，并且该节点的redis服务也已被停止
删除8007主节点
我们尝试删除之前加入的主节点8007，这个步骤相对比较麻烦一些，因为主节点的里面是有分配了hash槽的，所以我们这里必须先把8007里的hash槽放入到其他的可用主节点中去，然后再进行移除节点操作，不然会出现数据丢失问题(目前只能把master的数据迁移到一个节点上，暂时做不了平均分配功能)
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli --cluster reshard 192.168.5.100:8007
>>> Performing Cluster Check (using node 192.168.5.100:8007)
M: fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007
slots:[0-999] (1000 slots) master
S: 194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006
slots: (0 slots) slave
replicates 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6
M: 33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004
slots: (0 slots) slave
replicates 33206e9384297092b5b8a85c944f3564e5d983d7
M: 662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001
slots:[1000-5460] (4461 slots) master
1 additional replica(s)
S: 60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005
slots: (0 slots) slave
replicates 662809cf2d5bb138912dea7fb1e452f6e0f149da
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 1000
What is the receiving node ID? 662809cf2d5bb138912dea7fb1e452f6e0f149da
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: fea53768189af3e3e4849038af13607f59ec84b0
Source node #2: done
Ready to move 1000 slots.
Source nodes:
M: fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007
slots:[0-999] (1000 slots) master
Destination node:
M: 662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001
slots:[1000-5460] (4461 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 0 from fea53768189af3e3e4849038af13607f59ec84b0
Moving slot 1 from fea53768189af3e3e4849038af13607f59ec84b0
Moving slot 2 from fea53768189af3e3e4849038af13607f59ec84b0
.
.
.
Moving slot 995 from fea53768189af3e3e4849038af13607f59ec84b0
Moving slot 996 from fea53768189af3e3e4849038af13607f59ec84b0
Moving slot 997 from fea53768189af3e3e4849038af13607f59ec84b0
Moving slot 998 from fea53768189af3e3e4849038af13607f59ec84b0
Moving slot 999 from fea53768189af3e3e4849038af13607f59ec84b0
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 0 from 192.168.5.100:8007 to 192.168.5.100:8001:
Moving slot 1 from 192.168.5.100:8007 to 192.168.5.100:8001:
Moving slot 2 from 192.168.5.100:8007 to 192.168.5.100:8001:
.
.
.
Moving slot 995 from 192.168.5.100:8007 to 192.168.5.100:8001:
Moving slot 996 from 192.168.5.100:8007 to 192.168.5.100:8001:
Moving slot 997 from 192.168.5.100:8007 to 192.168.5.100:8001:
Moving slot 998 from 192.168.5.100:8007 to 192.168.5.100:8001:
Moving slot 999 from 192.168.5.100:8007 to 192.168.5.100:8001:
已经成功的把8007主节点的数据迁移到8001上去了，我们可以看一下现在的集群状态
[root@localhost 8007]# /usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8001
192.168.5.100:8001> cluster nodes
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544886422991 6 connected
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544886422000 4 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544886421000 2 connected 5461-10922
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544886424001 3 connected 10923-16383
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 myself,master - 0 1544886421000 8 connected 0-5460
fea53768189af3e3e4849038af13607f59ec84b0 192.168.5.100:8007@18007 master - 0 1544886422000 7 connected
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544886420966 8 connected
192.168.5.100:8001>
ok,那么删除8007节点
[root@localhost redis-cluster]# /usr/local/redis/redis-5.0.2/src/redis-cli --cluster del-node 192.168.5.100:8007 fea53768189af3e3e4849038af13607f59ec84b0
>>> Removing node fea53768189af3e3e4849038af13607f59ec84b0 from cluster 192.168.5.100:8007
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-cluster]# /usr/local/redis/redis-5.0.2/src/redis-cli -c -h 192.168.5.100 -p 8001
192.168.5.100:8001> cluster nodes
194a31057d2e098483bcd2ad01e1bba6a1af6a7d 192.168.5.100:8006@18006 slave 7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 0 1544886555484 6 connected
b0db47b7bbd3694596f293aa522488882e8fe647 192.168.5.100:8004@18004 slave 33206e9384297092b5b8a85c944f3564e5d983d7 0 1544886556495 4 connected
7b5f6aa6dcb4d5aca4a94e57ddeea6971b38bba6 192.168.5.100:8002@18002 master - 0 1544886555000 2 connected 5461-10922
33206e9384297092b5b8a85c944f3564e5d983d7 192.168.5.100:8003@18003 master - 0 1544886556000 3 connected 10923-16383
662809cf2d5bb138912dea7fb1e452f6e0f149da 192.168.5.100:8001@18001 myself,master - 0 1544886555000 8 connected 0-5460
60a0f7ced303374d8b36e7aa219cbcd4ff5b0caf 192.168.5.100:8005@18005 slave 662809cf2d5bb138912dea7fb1e452f6e0f149da 0 1544886554000 8 connected
192.168.5.100:8001>
