zookeeper

2021-08-04 本文已影响0人 SparkOnly

Zookeeper的核心是原子广播，这个机制保证了各个Server之间的同步。实现这个机制的协议叫做Zab协议（ZooKeeper Atomic Broadcast protocol）
ZAB协议定义了4个阶段：

选举(election)
选出哪台为leader
发现(discovery)、同步(sync)
主节点选出后，要做的恢复数据阶段
广播(broadcast)
leader和follower选出并同步好数据后，正常的主从同步阶段

三种角色

Leader 一个zk集群同一时间只会有一个实际工作的leader，它会发起并维护与各follower和observer间的心跳。所有写操作必须要通过leader完成再由leader将写操作广播给其他服务器
Follower 会响应leader的心跳。可直接处理并返回客户端的读请求，同时将写请求转发给leader处理，并且负责在leader处理写请求的时候对请求进行投票
Observer leader的Copy，不参与投票

节点状态

LOOKING：竞选Leader状态
LEADING：领导状态，当前Server即为选举出来的leader
FOLLOWING：跟随状态，同步leader状态，参与投票
OBSERVING：观察状态，同步leader状态，不参与投票

当服务启动或者在leader崩溃后，Zab就进入了恢复模式，当leader被选举出来，且大多数Server完成了和leader的状态同步以后，恢复模式就结束了。状态同步保证了leader和Server具有相同的系统状态。

FastLeaderElection

Zk的默认的选举算法为基于TCP的FastLeaderElection

electionEpoch，逻辑时钟，每执行一次leader选举，electionEpoch就会自增(logicalclock.incrementAndGet())，用来判断多个投票是否在同一轮的选举周期内（相当于Raft系统里的term）。
一般情况下，逻辑时钟是一样的，但是由于一些机器崩溃或网络断开，可能会出现不一致的情况。
例如5台机器，sid分别为1/2/3/4/5，逻辑时钟为0，依次启动后，开始选举，所有逻辑时钟自增为1，多次投票后，第3台为leader，5台的逻辑时钟都为1。
然后3崩溃了，zk需要重新选主，1/2/4/5逻辑时钟变为2。1重启后，逻辑时钟为1，小于其他节点，清除1的内部投票，重置逻辑时钟
peerEpoch，选举期间节点的peerEpoch不会变，选举结束后和leader同步时会根据leader的zxid取出高32位，更新自身节点的currentEpoch
广播选票时是使用proposedEpoch，proposedEpoch的初始值是peerEpoch，当新的投票胜出时，会使用胜出票的peerEpoch（推举的leader的epoch）。
myid，每个zk服务器的唯一ID
zxid，类似于关系型数据库的事务ID，用于标识一次更新操作的Proposal ID。为了保证顺序性，该zxid必须单调递增。实现中zxid是一个64位的数字，它高32位是epoch，表示选举纪元，用来标识leader关系是否改变，每次一个leader被选出来，它都会有一个新的epoch，标识当前属于那个leader的任期。低32位是自增的id

选举过程

每次选举都要把选举轮数(逻辑时钟)+1，防止不同轮次的选举互相干扰。
每个进入LOOKING状态的节点，会先清空投票，然后通过广播投票给自己，再把投票消息发给其他机器，同时也在接受其他节点的投票。

其他LOOKING状态的节点收到后：

首先判断选票是否有效。有效的选票接着处理
如果新的选票也是处于LOOKING状态：
2.1 如果选举轮数(n.electionEpoch) > 本地，证明自己的投票已过期。清空本地投票内容，更新投票轮数和结果为收到的内容。通知其他节点新的投票方案；
2.2 如果选举轮数 < 本地，忽略
2.3 如果选举轮数 == 本地，比较优先级。如果新的投票胜出本地投票，则更新本地投票，通知其他节点
优先级比较方法：
--1. 新的epoch(选票的peerEpoch)更高
--2. 新的epoch和当前epoch(节点peerEpoch或proposedEpoch)一样，但新的zxid更高
--3. 新的epoch和zxid都和当前一样，但server id更高
2.4 查询已收到的投票结果记录，如果有节点达到一半以上的投票数，则终止投票，选举结束，更新自身状态。
投票终止后，服务器更新自身状态。如果过半的票投给自己，则将自己的状态更新为LEADING，否则为FOLLOWING
如果新的选票是LEADING或FOLLOWING状态：
3.1 如果选举轮数一样，投入新票，判断投票结果，如果有节点达到一半以上的投票数，则终止投票，选举结束
3.2 如果不一致，则投新票到outofelection，判断outofelection里是否leader有过半选票，有的话结束投票

广播-主从同步

Zab: Broadcasting State Updates
有写操作时，如果是follower接收，会转到leader，保证写都是在主节点进行。Leader会先提议事务，收到过半回复后，再发提交。这里有通过“两阶段提交协议”来决定事务是否被commit执行。

当Leader收到写操作时，先本地生成PROPOSAL消息(带有zxid)，然后发给所有Follower节点
当Follower收到PROPOSAL消息，写到磁盘，成功后返回给Leader一个ACK，告知已经收到
Leader收到法定人数(quorum)的follower的ACK时，提交事务，再通知Follower提交事务。

Follower工作流程

Follower主要有四个功能：
向Leader发送请求（PING消息、REQUEST消息、ACK消息、REVALIDATE消息）；
接收Leader消息并进行处理；
接收Client的请求，如果为写请求，发送给Leader进行投票；
返回Client结果。

过半选举算法

QuorumMaj，针对某种状态，投票的成员超过所有可投票成员的一半时，就认为成功

5台服务器
服务器1启动,此时只有它一台服务器启动了,它发出去的Vote没有任何响应,所以它的选举状态一直是LOOKING状态；
服务器2启动,它与最开始启动的服务器1进行通信,互相交换自己的选举结果,由于两者都没有历史数据,所以id值较大的服务器2胜出,但是由于没有达到超过半数以上的服务器都同意选举它(这个例子中的半数以上是3),所以服务器1,2还是继续保持LOOKING状态.
服务器3启动,根据前面的理论，分析有三台服务器选举了它,服务器3成为服务器1,2,3中的老大,所以它成为了这次选举的leader.
服务器4启动,根据前面的分析，理论上服务器4应该是服务器1,2,3,4中最大的,但是由于前面已经有半数以上的服务器选举了服务器3,所以它只能接收当小弟的命了.
服务器5启动,同4一样,当小弟.
3台服务器
第二台启动的时候，已经有半数票都投给了服务器2，故第二台会成为leader

三台选举过程的日志

第一台的日志

给自己，节点2，节点3发送通知，宣称自己为leader
收到自己的通知，给自己投票。
FLE状态：id: 1, proposed id: 1, zxid: 0x0, proposed zxid: 0x0
Adding vote: from=1, proposed leader=1, proposed zxid=0x0, proposed election epoch=0x1

Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

收到节点2(LOOKING)的通知，投票给节点2。
FLE状态：id: 2, proposed id: 1, zxid: 0x0, proposed zxid: 0x0
修改状态：Updating proposal: 2 (newleader), 0x0 (newzxid), 1 (oldleader), 0x0 (oldzxid)

Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

发送通知给自己，节点2，节点3。推举节点2

Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 1 (recipient), 1 (myid), 0x0 (n.peerEpoch)
Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 2 (recipient), 1 (myid), 0x0 (n.peerEpoch)
Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 3 (recipient), 1 (myid), 0x0 (n.peerEpoch)

收到节点1(LOOKING)的通知，投票给节点2。
FLE状态：id: 2, proposed id: 2, zxid: 0x0, proposed zxid: 0x0
Adding vote: from=2, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1
Adding vote: from=1, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1

Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

更改自己状态为following，离开FLE，leader=2, zxid=0x0, my id=1, my state=FOLLOWING
收到节点3启动后发送的通知，宣称节点3为leader

Notification: my state:FOLLOWING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

发送新的通知给节点3，告知新leader

Sending new notification. My id =1 recipient=3 zxid=0x0 leader=2 config version = 0

#初始
09:45:59,243 [myid:1] - INFO  [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager$Listener$ListenerHandler@1066] - 1 is accepting connections now, my election bind port: localhost/127.0.0.1:3888
09:45:59,246 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1430] - LOOKING
09:45:59,246 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1357] - Initializing leader election protocol...
09:45:59,246 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@816] - Updating proposal: 1 (newleader), 0x0 (newzxid), -1 (oldleader), 0xffffffffffffffff (oldzxid)
09:45:59,247 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@945] - New election. My id = 1, proposed zxid=0x0
09:45:59,248 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@703] - Sending Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 1 (recipient), 1 (myid), 0x0 (n.peerEpoch)
09:45:59,248 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@703] - Sending Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 2 (recipient), 1 (myid), 0x0 (n.peerEpoch)
09:45:59,249 [myid:1] - DEBUG [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 1
09:45:59,249 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@703] - Sending Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 3 (recipient), 1 (myid), 0x0 (n.peerEpoch)
09:45:59,249 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
09:45:59,252 [myid:1] - DEBUG [WorkerSender[myid=1]:QuorumCnxManager@763] - Server 1 knows 2 already, it is in the lastCommittedView
09:45:59,252 [myid:1] - DEBUG [QuorumConnectionThread-[myid=1]-1:QuorumCnxManager@377] - Opening channel to server 2
09:45:59,252 [myid:1] - DEBUG [WorkerSender[myid=1]:QuorumCnxManager@763] - Server 1 knows 3 already, it is in the lastCommittedView
09:45:59,252 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@723] - id: 1, proposed id: 1, zxid: 0x0, proposed zxid: 0x0
09:45:59,253 [myid:1] - DEBUG [QuorumConnectionThread-[myid=1]-2:QuorumCnxManager@377] - Opening channel to server 3
09:45:59,253 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=1, proposed leader=1, proposed zxid=0x0, proposed election epoch=0x1
09:45:59,255 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-1:QuorumCnxManager@401] - Cannot open channel to 2 at election address localhost/127.0.0.1:3889
09:45:59,255 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-2:QuorumCnxManager@401] - Cannot open channel to 3 at election address localhost/127.0.0.1:3890
#第二台启动
09:46:13,673 [myid:1] - INFO  [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager$Listener$ListenerHandler@1071] - Received connection request from /127.0.0.1:54889
09:46:13,674 [myid:1] - DEBUG [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager@554] - Sync handling of connection request received from: /127.0.0.1:54889
09:46:13,675 [myid:1] - DEBUG [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager@613] - Initial message parsed by 1: InitialMessage{sid=2, electionAddr=[/127.0.0.1:3889]}
09:46:13,677 [myid:1] - DEBUG [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager$SendWorker@1179] - Address of remote peer: 2
09:46:13,678 [myid:1] - DEBUG [SendWorker:2:QuorumCnxManager$SendWorker@1263] - SendWorker thread started towards 2. myId: 1
09:46:13,678 [myid:1] - DEBUG [RecvWorker:2:QuorumCnxManager$RecvWorker@1379] - RecvWorker thread towards 2 started. myId: 1
09:46:13,680 [myid:1] - DEBUG [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 1
09:46:13,680 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
09:46:13,680 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@723] - id: 2, proposed id: 1, zxid: 0x0, proposed zxid: 0x0
09:46:13,680 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@816] - Updating proposal: 2 (newleader), 0x0 (newzxid), 1 (oldleader), 0x0 (oldzxid)
09:46:13,681 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@703] - Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 1 (recipient), 1 (myid), 0x0 (n.peerEpoch)
09:46:13,681 [myid:1] - DEBUG [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 1
09:46:13,681 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
09:46:13,681 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@703] - Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 2 (recipient), 1 (myid), 0x0 (n.peerEpoch)
09:46:13,682 [myid:1] - DEBUG [WorkerSender[myid=1]:QuorumCnxManager@744] - There is a connection already for server 2
09:46:13,682 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@703] - Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 3 (recipient), 1 (myid), 0x0 (n.peerEpoch)
09:46:13,682 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=2, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1
09:46:13,682 [myid:1] - DEBUG [WorkerSender[myid=1]:CircularBlockingQueue@84] - Queue is full. Discarding oldest element [count=1]: java.nio.HeapByteBuffer[pos=176 lim=176 cap=176]
09:46:13,682 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@723] - id: 2, proposed id: 2, zxid: 0x0, proposed zxid: 0x0
09:46:13,683 [myid:1] - DEBUG [WorkerSender[myid=1]:QuorumCnxManager@763] - Server 1 knows 3 already, it is in the lastCommittedView
09:46:13,683 [myid:1] - DEBUG [QuorumConnectionThread-[myid=1]-3:QuorumCnxManager@377] - Opening channel to server 3
09:46:13,683 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=1, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1
09:46:13,683 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-3:QuorumCnxManager@401] - Cannot open channel to 3 at election address localhost/127.0.0.1:3890
...
09:46:13,887 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@841] - I am a participant: 1
09:46:13,888 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@901] - Peer state changed: following
09:46:13,888 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@661] - About to leave FLE instance: leader=2, zxid=0x0, my id=1, my state=FOLLOWING
#此时已经选出leader
#第三台启动
16:53:25,102 [myid:1] - INFO  [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager$Listener$ListenerHandler@1071] - Received connection request from /127.0.0.1:60039
16:53:25,147 [myid:1] - DEBUG [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager@554] - Sync handling of connection request received from: /127.0.0.1:60039
16:53:25,158 [myid:1] - DEBUG [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager@613] - Initial message parsed by 1: InitialMessage{sid=3, electionAddr=[/127.0.0.1:3890]}
16:53:25,161 [myid:1] - DEBUG [ListenerHandler-localhost/127.0.0.1:3888:QuorumCnxManager$SendWorker@1179] - Address of remote peer: 3
16:53:25,214 [myid:1] - DEBUG [RecvWorker:3:QuorumCnxManager$RecvWorker@1379] - RecvWorker thread towards 3 started. myId: 1
16:53:25,214 [myid:1] - DEBUG [SendWorker:3:QuorumCnxManager$SendWorker@1263] - SendWorker thread started towards 3. myId: 1
16:53:25,220 [myid:1] - DEBUG [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 1
16:53:25,221 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:FOLLOWING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
16:53:25,221 [myid:1] - DEBUG [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@446] - Sending new notification. My id =1 recipient=3 zxid=0x0 leader=2 config version = 0

第二台的日志

初始化。并发送给节点1，自己，节点3通知，宣称自己为leader

Updating proposal: 2 (newleader), 0x0 (newzxid), -1 (oldleader), 0xffffffffffffffff (oldzxid)
Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 1 (recipient), 2 (myid), 0x0 (n.peerEpoch)
Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 2 (recipient), 2 (myid), 0x0 (n.peerEpoch)
Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 3 (recipient), 2 (myid), 0x0 (n.peerEpoch)

收到自己的通知，给自己投票
FLE状态：id: 2, proposed id: 2, zxid: 0x0, proposed zxid: 0x0
Adding vote: from=2, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1

Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

收到节点1(LOOKING)的投票，给节点1投票
FLE状态：id: 1, proposed id: 2, zxid: 0x0, proposed zxid: 0x0
Adding vote: from=1, proposed leader=1, proposed zxid=0x0, proposed election epoch=0x1

Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

收到节点1(LOOKING)的投票，给节点2投票
FLE状态：id: 2, proposed id: 2, zxid: 0x0, proposed zxid: 0x0
Adding vote: from=1, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1

Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

节点2赢得选举，变为主节点
leader=2, zxid=0x0, my id=2, my state=LEADING
收到节点3启动后发送的通知，宣称节点3为leader

Notification: my state:LEADING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

发送新通知给节点3，告知当前状态

Sending new notification. My id =2 recipient=3 zxid=0x0 leader=2 config version = 0

#初始
09:46:13,661 [myid:2] - INFO  [ListenerHandler-localhost/127.0.0.1:3889:QuorumCnxManager$Listener$ListenerHandler@1066] - 2 is accepting connections now, my election bind port: localhost/127.0.0.1:3889
09:46:13,664 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):QuorumPeer@1430] - LOOKING
09:46:13,664 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):QuorumPeer@1357] - Initializing leader election protocol...
09:46:13,665 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@816] - Updating proposal: 2 (newleader), 0x0 (newzxid), -1 (oldleader), 0xffffffffffffffff (oldzxid)
09:46:13,665 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@945] - New election. My id = 2, proposed zxid=0x0
09:46:13,666 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@703] - Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 1 (recipient), 2 (myid), 0x0 (n.peerEpoch)
09:46:13,666 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@703] - Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 2 (recipient), 2 (myid), 0x0 (n.peerEpoch)
09:46:13,670 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@763] - Server 2 knows 1 already, it is in the lastCommittedView
09:46:13,670 [myid:2] - DEBUG [QuorumConnectionThread-[myid=2]-1:QuorumCnxManager@377] - Opening channel to server 1
09:46:13,671 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@703] - Sending Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 3 (recipient), 2 (myid), 0x0 (n.peerEpoch)
09:46:13,671 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@763] - Server 2 knows 3 already, it is in the lastCommittedView
09:46:13,672 [myid:2] - DEBUG [QuorumConnectionThread-[myid=2]-2:QuorumCnxManager@377] - Opening channel to server 3
09:46:13,672 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 2
09:46:13,672 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
09:46:13,672 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@723] - id: 2, proposed id: 2, zxid: 0x0, proposed zxid: 0x0
09:46:13,672 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=2, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1
09:46:13,673 [myid:2] - DEBUG [QuorumConnectionThread-[myid=2]-1:QuorumCnxManager@394] - Connected to server 1 using election address: localhost/127.0.0.1:3888
09:46:13,673 [myid:2] - DEBUG [QuorumConnectionThread-[myid=2]-1:QuorumCnxManager@469] - startConnection (myId:2 --> sid:1)
09:46:13,674 [myid:2] - WARN  [QuorumConnectionThread-[myid=2]-2:QuorumCnxManager@401] - Cannot open channel to 3 at election address localhost/127.0.0.1:3890
java.net.ConnectException: Connection refused (Connection refused)
...
#此时第三个节点还未启动，根据节点1和节点2，可以选择出leader
09:46:13,675 [myid:2] - DEBUG [QuorumConnectionThread-[myid=2]-1:QuorumCnxManager@518] - Have larger server identifier, so keeping the connection: (myId:2 --> sid:1)
09:46:13,677 [myid:2] - DEBUG [QuorumConnectionThread-[myid=2]-1:QuorumCnxManager$SendWorker@1179] - Address of remote peer: 1
09:46:13,679 [myid:2] - DEBUG [SendWorker:1:QuorumCnxManager$SendWorker@1263] - SendWorker thread started towards 1. myId: 2
09:46:13,679 [myid:2] - DEBUG [RecvWorker:1:QuorumCnxManager$RecvWorker@1379] - RecvWorker thread towards 1 started. myId: 2
09:46:13,680 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 2
09:46:13,680 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
09:46:13,680 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@723] - id: 1, proposed id: 2, zxid: 0x0, proposed zxid: 0x0
09:46:13,680 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=1, proposed leader=1, proposed zxid=0x0, proposed election epoch=0x1
09:46:13,682 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 2
09:46:13,682 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
09:46:13,683 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@723] - id: 2, proposed id: 2, zxid: 0x0, proposed zxid: 0x0
09:46:13,683 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=1, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1
09:46:13,887 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):QuorumPeer@901] - Peer state changed: leading
09:46:13,888 [myid:2] - DEBUG [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2182)(secure=disabled):FastLeaderElection@661] - About to leave FLE instance: leader=2, zxid=0x0, my id=2, my state=LEADING
#此时节点3启动，告知最新状态
16:53:25,101 [myid:2] - INFO  [ListenerHandler-localhost/127.0.0.1:3889:QuorumCnxManager$Listener$ListenerHandler@1071] - Received connection request from /127.0.0.1:60040
16:53:25,144 [myid:2] - DEBUG [ListenerHandler-localhost/127.0.0.1:3889:QuorumCnxManager@554] - Sync handling of connection request received from: /127.0.0.1:60040
16:53:25,206 [myid:2] - DEBUG [ListenerHandler-localhost/127.0.0.1:3889:QuorumCnxManager@613] - Initial message parsed by 2: InitialMessage{sid=3, electionAddr=[/127.0.0.1:3890]}
16:53:25,209 [myid:2] - DEBUG [ListenerHandler-localhost/127.0.0.1:3889:QuorumCnxManager$SendWorker@1179] - Address of remote peer: 3
16:53:25,210 [myid:2] - DEBUG [RecvWorker:3:QuorumCnxManager$RecvWorker@1379] - RecvWorker thread towards 3 started. myId: 2
16:53:25,210 [myid:2] - DEBUG [SendWorker:3:QuorumCnxManager$SendWorker@1263] - SendWorker thread started towards 3. myId: 2
16:53:25,223 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 2
16:53:25,225 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LEADING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
16:53:25,225 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@446] - Sending new notification. My id =2 recipient=3 zxid=0x0 leader=2 config version = 0

节点3日志

初始化。并发送通知给自己以及节点1，节点2，宣称自己为leader

Updating proposal: 3 (newleader), 0x0 (newzxid), -1 (oldleader), 0xffffffffffffffff (oldzxid)
Sending Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 1 (recipient), 3 (myid), 0x0 (n.peerEpoch)
Sending Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 2 (recipient), 3 (myid), 0x0 (n.peerEpoch)
Sending Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 3 (recipient), 3 (myid), 0x0 (n.peerEpoch)

收到自己的投票，投给自己
FLE状态：id: 3, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
Adding vote: from=3, proposed leader=3, proposed zxid=0x0, proposed election epoch=0x1

Notification: my state:LOOKING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

收到节点2(LOOKING)的投票，投给节点2
FLE状态：id: 2, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
Adding vote: from=2, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1

Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

收到节点1(LOOKING)的投票，投给节点2
FLE状态：id: 2, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
Adding vote: from=1, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1

Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0

收到节点1(FOLLOWING)的投票，follow节点2

Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2, n.round:0x1, n.peerEpoch:0x1, n.zxid:0x0, message format version:0x2, n.config version:0x0

收到节点2(LEADING)的投票，leader

Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x1, n.peerEpoch:0x1, n.zxid:0x0, message format version:0x2, n.config version:0x0

变更状态为following
leader=2, zxid=0x0, my id=3, my state=FOLLOWING

16:53:25,050 [myid:3] - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):QuorumPeer@1430] - LOOKING
16:53:25,050 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):QuorumPeer@1357] - Initializing leader election protocol...
16:53:25,051 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@816] - Updating proposal: 3 (newleader), 0x0 (newzxid), -1 (oldleader), 0xffffffffffffffff (oldzxid)
16:53:25,051 [myid:3] - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@945] - New election. My id = 3, proposed zxid=0x0
16:53:25,052 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@703] - Sending Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 1 (recipient), 3 (myid), 0x0 (n.peerEpoch)
16:53:25,053 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@703] - Sending Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 2 (recipient), 3 (myid), 0x0 (n.peerEpoch)
16:53:25,059 [myid:3] - DEBUG [WorkerSender[myid=3]:QuorumCnxManager@763] - Server 3 knows 1 already, it is in the lastCommittedView
16:53:25,060 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-1:QuorumCnxManager@377] - Opening channel to server 1
16:53:25,060 [myid:3] - DEBUG [WorkerSender[myid=3]:QuorumCnxManager@763] - Server 3 knows 2 already, it is in the lastCommittedView
16:53:25,060 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@703] - Sending Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 3 (recipient), 3 (myid), 0x0 (n.peerEpoch)
16:53:25,060 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-2:QuorumCnxManager@377] - Opening channel to server 2
16:53:25,061 [myid:3] - DEBUG [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 3
16:53:25,061 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
16:53:25,061 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@723] - id: 3, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
16:53:25,061 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=3, proposed leader=3, proposed zxid=0x0, proposed election epoch=0x1
16:53:25,062 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-1:QuorumCnxManager@394] - Connected to server 1 using election address: localhost/127.0.0.1:3888
16:53:25,063 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-1:QuorumCnxManager@469] - startConnection (myId:3 --> sid:1)
16:53:25,063 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-2:QuorumCnxManager@394] - Connected to server 2 using election address: localhost/127.0.0.1:3889
16:53:25,063 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-2:QuorumCnxManager@469] - startConnection (myId:3 --> sid:2)
16:53:25,065 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-1:QuorumCnxManager@518] - Have larger server identifier, so keeping the connection: (myId:3 --> sid:1)
16:53:25,065 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-2:QuorumCnxManager@518] - Have larger server identifier, so keeping the connection: (myId:3 --> sid:2)
16:53:25,067 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-1:QuorumCnxManager$SendWorker@1179] - Address of remote peer: 1
16:53:25,067 [myid:3] - DEBUG [QuorumConnectionThread-[myid=3]-2:QuorumCnxManager$SendWorker@1179] - Address of remote peer: 2
16:53:25,068 [myid:3] - DEBUG [SendWorker:1:QuorumCnxManager$SendWorker@1263] - SendWorker thread started towards 1. myId: 3
16:53:25,068 [myid:3] - DEBUG [RecvWorker:1:QuorumCnxManager$RecvWorker@1379] - RecvWorker thread towards 1 started. myId: 3
16:53:25,068 [myid:3] - DEBUG [SendWorker:2:QuorumCnxManager$SendWorker@1263] - SendWorker thread started towards 2. myId: 3
16:53:25,068 [myid:3] - DEBUG [RecvWorker:2:QuorumCnxManager$RecvWorker@1379] - RecvWorker thread towards 2 started. myId: 3
16:53:25,213 [myid:3] - DEBUG [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 3
16:53:25,213 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
16:53:25,213 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@723] - id: 2, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
16:53:25,213 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=2, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1
16:53:25,216 [myid:3] - DEBUG [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 3
16:53:25,216 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0
16:53:25,216 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@723] - id: 2, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
16:53:25,216 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@1017] - Adding vote: from=1, proposed leader=2, proposed zxid=0x0, proposed election epoch=0x1
16:53:25,225 [myid:3] - DEBUG [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 3
16:53:25,225 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2, n.round:0x1, n.peerEpoch:0x1, n.zxid:0x0, message format version:0x2, n.config version:0x0
16:53:25,253 [myid:3] - DEBUG [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@358] - Receive new notification message. My id = 3
16:53:25,254 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@390] - Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x1, n.peerEpoch:0x1, n.zxid:0x0, message format version:0x2, n.config version:0x0
16:53:25,254 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@841] - I am a participant: 3
16:53:25,254 [myid:3] - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):QuorumPeer@901] - Peer state changed: following
16:53:25,254 [myid:3] - DEBUG [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2183)(secure=disabled):FastLeaderElection@661] - About to leave FLE instance: leader=2, zxid=0x0, my id=3, my state=FOLLOWING

数据一致性与paxos算法

原则：在一个分布式数据库系统中，如果各节点的初始状态一致，每个节点都执行相同的操作序列，那么他们最后能得到一个一致的状态
paxos解决的就是保证每个节点执行相同的操作序列
通过投票来对写操作进行全局编号，同一时刻，只有一个写操作被批准，同时并发的写操作要去争取选票，只有获得过半数选票的写操作才会被批准（所以永远只会有一个写操作得到批准），其他的写操作竞争失败只好再发起一轮投票，就这样，在日复一日年复一年的投票中，所有写操作都被严格编号排序。
编号严格递增，当一个节点接受了一个编号为100的写操作，之后又接受到编号为99的写操作（因为网络延迟等很多不可预见原因），它马上能意识到自己数据不一致了，自动停止对外服务并重启同步过程。任何一个节点挂掉都不会影响整个集群的数据一致性（总2n+1台，除非挂掉大于n台）

集群数目为奇数个

Leader选举算法采用了Paxos协议；
Paxos核心思想：当多数Server写成功，则任务数据写成功如果有3个Server，则两个写成功即可；如果有4或5个Server，则三个写成功即可。
Server数目一般为奇数（3、5、7）如果有3个Server，则最多允许1个Server挂掉；如果有4个Server，则同样最多允许1个Server挂掉由此，我们看出3台服务器和4台服务器的的容灾能力是一样的，所以为了节省服务器资源，一般我们采用奇数个数，作为服务器部署个数

zookeeper

三种角色

节点状态

FastLeaderElection

选举过程

广播-主从同步

Follower工作流程

过半选举算法

三台选举过程的日志

数据一致性与paxos算法

集群数目为奇数个

猜你喜欢

热点阅读