MongoDB高级应用（二）：集群复制
2017-08-29 本文已影响171人
四冶读史
1.概述
MongoDB复制是将数据同步到多个服务器的过程。复制提供了数据冗余备份，并在多个服务器上存储副本，提高了数据的可用性，保证数据的安全性，还允许您从硬件故障和服务中断中恢复数据。
MongoDB集群复制是将数据同时复制到2台及以上（建议3台及以上，因为2台，其中一台Primary宕机后，另外一台Secondary并不能担当Primary），一台充当Primary，其它的充当Secondary，当Primary宕机，其中的一个Secondary充当Primary，顶替宕机的Primary，而再次修复启动的Primary则是Secondary。也可以添加一个Arbiter（仲裁者），不做数据的读写，只做投票选举。这样可保证服务器具有高可用性。
2.原理
MongoDB复制至少需要两个节点，其中一个是主节点，负责处理客户端请求，其余的都是从节点，负责复制主节点上的数据。MongoDB各个节点常见的搭配方式为：一主一从、一主多从。
主节点将所有的操作记录在oplog中，从节点定时查看自己的oplog记录，将主节点在自己最新记录之后的所有写操作记录在自己副本数据上再执行一次，并记录oplog，从而保证从节点数据的一致性。
如果从节点宕机，当再次启动后，就会自动从oplog最后一个操作开始同步，同步完成后，将信息写入oplog，由于复制操作是先复制数据，复制完成后再写入oplog，有可能相同的操作会同步两份，不过MongoDB在设计之处就考虑到这个问题，将oplog的同一个操作执行多次，与执行一次的效果是一样的。
3.架构
Paste_Image.png
4.优点
1.安全性高
2.可用性高
3.自动恢复
4.分布式读取数据
5.易扩展
6.自动故障转移
等等
5.实现
测试目的
1.数据同步复制
2.自动故障转移
硬件准备
三台服务器，分别为10.11.1.58，10.11.1.66，10.11.1.59，任意一台为Primary，其它2台为Secondary
环境搭建
分别在三台服务器上创建日志文件、配置文件和数据库路径
mkdir -p /data/db
mkdir -p /data/log
mkdir -p /data/conf
编写启动文件，命名为replset.conf，存在在/data/conf下
port=27017
logpath=/data/log/mongod.log
dbpath=/data/db
pidfilepath=/data/db/mongod.pid
fork=true
oplogSize=2048
replSet=rs
启动服务
分别启动3台服务
/usr/local/mongodb/bin/mongod --config /data/conf/replset.conf
about to fork child process, waiting until server is ready for connections.
forked process: 5821
child process started successfully, parent exiting
任意登录其中一台服务器，以10.11.1.58为例：
/usr/local/mongodb/bin/mongo
MongoDB shell version v3.4.3
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.3
Server has startup warnings:
2017-03-31T14:43:25.787+0800 I CONTROL [initandlisten]
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten]
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten]
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten]
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-03-31T14:43:25.788+0800 I CONTROL [initandlisten]
>
查看数据
> show dbs
2017-03-31T14:45:05.441+0800 E QUERY [thread1] Error: listDatabases failed:{
"ok" : 0,
"errmsg" : "not master and slaveOk=false",
"code" : 13435,
"codeName" : "NotMasterNoSlaveOk"
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:62:1
shellHelper.show@src/mongo/shell/utils.js:761:19
shellHelper@src/mongo/shell/utils.js:651:15
@(shellhelp2):1:1
发现不可用
查看状态
> rs.status()
{
"info" : "run rs.initiate(...) if not yet done for the set",
"ok" : 0,
"errmsg" : "no replset config has been received",
"code" : 94,
"codeName" : "NotYetInitialized"
}
发现，需要通过rs.initiate()初始化
初始化
> cfg={_id:'rs',members:[{_id:0,host:'10.11.1.58:27017'}]}
{
"_id" : "rs",
"members" : [
{
"_id" : 0,
"host" : "10.11.1.58:27017"
}
]
}
> rs.initiate(cfg)
{ "ok" : 1 }
发现命令行已发生变化
rs:SECONDARY>
rs:PRIMARY>
再次查看状态
rs:PRIMARY> rs.status()
{
"set" : "rs",
"date" : ISODate("2017-03-31T06:56:14.748Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1490943373, 1),
"t" : NumberLong(1)
},
"appliedOpTime" : {
"ts" : Timestamp(1490943373, 1),
"t" : NumberLong(1)
},
"durableOpTime" : {
"ts" : Timestamp(1490943373, 1),
"t" : NumberLong(1)
}
},
"members" : [
{
"_id" : 0,
"name" : "10.11.1.58:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 769,
"optime" : {
"ts" : Timestamp(1490943373, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2017-03-31T06:56:13Z"),
"electionTime" : Timestamp(1490943032, 2),
"electionDate" : ISODate("2017-03-31T06:50:32Z"),
"configVersion" : 1,
"self" : true
}
],
"ok" : 1
}
rs:PRIMARY>
**添加Secondary**
rs:PRIMARY> rs.add('10.11.1.66:27017')
{ "ok" : 1 }
rs:PRIMARY> rs.add('10.11.1.59:27017')
{ "ok" : 1 }
rs:PRIMARY>
再次查看状态
rs:PRIMARY> rs.status()
{
"set" : "rs",
"date" : ISODate("2017-03-31T06:58:15.304Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1490943493, 1),
"t" : NumberLong(1)
},
"appliedOpTime" : {
"ts" : Timestamp(1490943493, 1),
"t" : NumberLong(1)
},
"durableOpTime" : {
"ts" : Timestamp(1490943493, 1),
"t" : NumberLong(1)
}
},
"members" : [
{
"_id" : 0,
"name" : "10.11.1.58:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 890,
"optime" : {
"ts" : Timestamp(1490943493, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2017-03-31T06:58:13Z"),
"electionTime" : Timestamp(1490943032, 2),
"electionDate" : ISODate("2017-03-31T06:50:32Z"),
"configVersion" : 3,
"self" : true
},
{
"_id" : 1,
"name" : "10.11.1.66:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 30,
"optime" : {
"ts" : Timestamp(1490943493, 1),
"t" : NumberLong(1)
},
"optimeDurable" : {
"ts" : Timestamp(1490943493, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2017-03-31T06:58:13Z"),
"optimeDurableDate" : ISODate("2017-03-31T06:58:13Z"),
"lastHeartbeat" : ISODate("2017-03-31T06:58:15.205Z"),
"lastHeartbeatRecv" : ISODate("2017-03-31T06:58:14.205Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "10.11.1.58:27017",
"configVersion" : 3
},
{
"_id" : 2,
"name" : "10.11.1.59:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 26,
"optime" : {
"ts" : Timestamp(1490943493, 1),
"t" : NumberLong(1)
},
"optimeDurable" : {
"ts" : Timestamp(1490943493, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2017-03-31T06:58:13Z"),
"optimeDurableDate" : ISODate("2017-03-31T06:58:13Z"),
"lastHeartbeat" : ISODate("2017-03-31T06:58:15.205Z"),
"lastHeartbeatRecv" : ISODate("2017-03-31T06:58:14.248Z"),
"pingMs" : NumberLong(1),
"syncingTo" : "10.11.1.58:27017",
"configVersion" : 3
}
],
"ok" : 1
}
rs:PRIMARY>
发现，后面添加进来的节点，是为Secondary
查看主节点连接状态
rs:PRIMARY> db.printReplicationInfo()
configured oplog size: 2048MB
log length start to end: 611secs (0.17hrs)
oplog first event time: Fri Mar 31 2017 14:50:32 GMT+0800 (CST)
oplog last event time: Fri Mar 31 2017 15:00:43 GMT+0800 (CST)
now: Fri Mar 31 2017 15:00:46 GMT+0800 (CST)
rs:PRIMARY>
查看从节点连接状态
rs:SECONDARY> rs.printSlaveReplicationInfo()
source: 10.11.1.66:27017
syncedTo: Fri Mar 31 2017 15:01:33 GMT+0800 (CST)
0 secs (0 hrs) behind the primary
source: 10.11.1.59:27017
syncedTo: Fri Mar 31 2017 15:01:33 GMT+0800 (CST)
0 secs (0 hrs) behind the primary
rs:SECONDARY>
至此MongoDB集群复制初步搭建完毕，下面就进行验证测试吧！
验证一：数据同步
在主库中写入书库，在从库中查看数据
主写：
rs:PRIMARY> use testdb
switched to db testdb
rs:PRIMARY> db.col.save({a:11})
WriteResult({ "nInserted" : 1 })
rs:PRIMARY> db.col.find()
{ "_id" : ObjectId("58ddffadcf63064602a4daf6"), "a" : 11 }
rs:PRIMARY>
从读：
rs:SECONDARY> show dbs
2017-03-31T15:06:04.948+0800 E QUERY [thread1] Error: listDatabases failed:{
"ok" : 0,
"errmsg" : "not master and slaveOk=false",
"code" : 13435,
"codeName" : "NotMasterNoSlaveOk"
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:62:1
shellHelper.show@src/mongo/shell/utils.js:761:19
shellHelper@src/mongo/shell/utils.js:651:15
@(shellhelp2):1:1
rs:SECONDARY>
怎么回事？"not master and slaveOk=false"
初始状态下，slave是没有读的权限？
为什么？
因为不知道用户需要将那个slave设置成读，故需要用户设置slave为可读
rs.slaveOk()
rs:SECONDARY> rs.slaveOk()
rs:SECONDARY> show dbs
admin 0.000GB
local 0.000GB
testdb 0.000GB
rs:SECONDARY> use testdb
switched to db testdb
rs:SECONDARY> db.col.find()
{ "_id" : ObjectId("58ddffadcf63064602a4daf6"), "a" : 11 }
rs:SECONDARY>
同样的方法，可验证其它的Secondary是否同步成功
10.11.1.66和10.11.1.59经本人验证同步是成功的！
至此，数据同步验证成功！
验证二：故障转移
也就是说，如果Primary宕机，其它Secondary会接替Primary工作并充当Priamary
停掉Primary
切换至admin数据库，并执行db.shutdownServer()
rs:PRIMARY> use admin
switched to db admin
rs:PRIMARY> db.shutdownServer()
server should be down...
2017-03-31T15:14:16.608+0800 I NETWORK [thread1] trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2017-03-31T15:14:16.839+0800 I NETWORK [thread1] Socket recv() Connection reset by peer 127.0.0.1:27017
2017-03-31T15:14:16.839+0800 I NETWORK [thread1] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:27017]
2017-03-31T15:14:16.839+0800 I NETWORK [thread1] reconnect 127.0.0.1:27017 (127.0.0.1) failed failed
>
成功停掉Primary
在从上查看状态（10.11.1.66）
rs:SECONDARY> rs.status()
{
"set" : "rs",
"date" : ISODate("2017-03-31T07:15:25.526Z"),
"myState" : 2,
"term" : NumberLong(2),
"syncingTo" : "10.11.1.59:27017",
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1490944525, 1),
"t" : NumberLong(2)
},
"appliedOpTime" : {
"ts" : Timestamp(1490944525, 1),
"t" : NumberLong(2)
},
"durableOpTime" : {
"ts" : Timestamp(1490944525, 1),
"t" : NumberLong(2)
}
},
"members" : [
{
"_id" : 0,
"name" : "10.11.1.58:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : {
"ts" : Timestamp(0, 0),
"t" : NumberLong(-1)
},
"optimeDurable" : {
"ts" : Timestamp(0, 0),
"t" : NumberLong(-1)
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2017-03-31T07:15:23.873Z"),
"lastHeartbeatRecv" : ISODate("2017-03-31T07:14:15.568Z"),
"pingMs" : NumberLong(0),
"lastHeartbeatMessage" : "Connection refused",
"configVersion" : -1
},
{
"_id" : 1,
"name" : "10.11.1.66:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 1931,
"optime" : {
"ts" : Timestamp(1490944525, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2017-03-31T07:15:25Z"),
"syncingTo" : "10.11.1.59:27017",
"configVersion" : 3,
"self" : true
},
{
"_id" : 2,
"name" : "10.11.1.59:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 1056,
"optime" : {
"ts" : Timestamp(1490944525, 1),
"t" : NumberLong(2)
},
"optimeDurable" : {
"ts" : Timestamp(1490944525, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2017-03-31T07:15:25Z"),
"optimeDurableDate" : ISODate("2017-03-31T07:15:25Z"),
"lastHeartbeat" : ISODate("2017-03-31T07:15:23.856Z"),
"lastHeartbeatRecv" : ISODate("2017-03-31T07:15:25.206Z"),
"pingMs" : NumberLong(0),
"electionTime" : Timestamp(1490944475, 1),
"electionDate" : ISODate("2017-03-31T07:14:35Z"),
"configVersion" : 3
}
],
"ok" : 1
}
rs:SECONDARY>
发现，之前的Primary变为不可达，而10.11.1.59充当Primary
至此，故障转移验证成功！
再将原来的Primary（10.11.1.58）启动，会是怎样？
rs:SECONDARY> rs.status()
{
"set" : "rs",
"date" : ISODate("2017-03-31T07:18:52.649Z"),
"myState" : 2,
"term" : NumberLong(2),
"syncingTo" : "10.11.1.59:27017",
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1490944735, 1),
"t" : NumberLong(2)
},
"appliedOpTime" : {
"ts" : Timestamp(1490944735, 1),
"t" : NumberLong(2)
},
"durableOpTime" : {
"ts" : Timestamp(1490944735, 1),
"t" : NumberLong(2)
}
},
"members" : [
{
"_id" : 0,
"name" : "10.11.1.58:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 13,
"optime" : {
"ts" : Timestamp(1490944735, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2017-03-31T07:18:55Z"),
"syncingTo" : "10.11.1.59:27017",
"configVersion" : 3,
"self" : true
},
{
"_id" : 1,
"name" : "10.11.1.66:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 12,
"optime" : {
"ts" : Timestamp(1490944735, 1),
"t" : NumberLong(2)
},
"optimeDurable" : {
"ts" : Timestamp(1490944735, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2017-03-31T07:18:55Z"),
"optimeDurableDate" : ISODate("2017-03-31T07:18:55Z"),
"lastHeartbeat" : ISODate("2017-03-31T07:18:51.953Z"),
"lastHeartbeatRecv" : ISODate("2017-03-31T07:18:52.312Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "10.11.1.59:27017",
"configVersion" : 3
},
{
"_id" : 2,
"name" : "10.11.1.59:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 12,
"optime" : {
"ts" : Timestamp(1490944735, 1),
"t" : NumberLong(2)
},
"optimeDurable" : {
"ts" : Timestamp(1490944735, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2017-03-31T07:18:55Z"),
"optimeDurableDate" : ISODate("2017-03-31T07:18:55Z"),
"lastHeartbeat" : ISODate("2017-03-31T07:18:51.954Z"),
"lastHeartbeatRecv" : ISODate("2017-03-31T07:18:51.789Z"),
"pingMs" : NumberLong(0),
"electionTime" : Timestamp(1490944475, 1),
"electionDate" : ISODate("2017-03-31T07:14:35Z"),
"configVersion" : 3
}
],
"ok" : 1
}
rs:SECONDARY>
果然，原来的Primary（10.11.1.58）是Secondary，并不是Primary
至此，MongoDB集群复制搭建完成！
那如何知道谁之Master、Primary呢？
有的人可能发现，用rs.status()
这个当然可以，可是当机器很多的时候，还是不爽不。
Primary:
rs:PRIMARY> rs.isMaster()
{
"hosts" : [
"10.11.1.58:27017",
"10.11.1.66:27017",
"10.11.1.59:27017"
],
"setName" : "rs",
"setVersion" : 3,
"ismaster" : true,
"secondary" : false,
"primary" : "10.11.1.59:27017",
"me" : "10.11.1.59:27017",
"electionId" : ObjectId("7fffffff0000000000000002"),
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1490945435, 1),
"t" : NumberLong(2)
},
"lastWriteDate" : ISODate("2017-03-31T07:30:35Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 1000,
"localTime" : ISODate("2017-03-31T07:30:42.044Z"),
"maxWireVersion" : 5,
"minWireVersion" : 0,
"readOnly" : false,
"ok" : 1
}
Secondary:
rs:SECONDARY> rs.isMaster()
{
"hosts" : [
"10.11.1.58:27017",
"10.11.1.66:27017",
"10.11.1.59:27017"
],
"setName" : "rs",
"setVersion" : 3,
"ismaster" : false,
"secondary" : true,
"primary" : "10.11.1.59:27017",
"me" : "10.11.1.66:27017",
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1490945545, 1),
"t" : NumberLong(2)
},
"lastWriteDate" : ISODate("2017-03-31T07:32:25Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 1000,
"localTime" : ISODate("2017-03-31T07:32:20.352Z"),
"maxWireVersion" : 5,
"minWireVersion" : 0,
"readOnly" : false,
"ok" : 1
}
是不是很不错！
删除Secondary
rs:SECONDARY> rs.remove('10.11.1.58:27017')
{
"ok" : 0,
"errmsg" : "replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the \"force\" argument to override",
"code" : 10107,
"codeName" : "NotMaster"
}
发现，Secondary是无法删除节点的
Primary是可以删除节点的
rs:PRIMARY> rs.remove('10.11.1.58:27017')
{ "ok" : 1 }
查看下
rs:PRIMARY> rs.status()
{
"set" : "rs",
"date" : ISODate("2017-03-31T07:37:22.314Z"),
"myState" : 1,
"term" : NumberLong(2),
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1490945835, 1),
"t" : NumberLong(2)
},
"appliedOpTime" : {
"ts" : Timestamp(1490945835, 1),
"t" : NumberLong(2)
},
"durableOpTime" : {
"ts" : Timestamp(1490945835, 1),
"t" : NumberLong(2)
}
},
"members" : [
{
"_id" : 1,
"name" : "10.11.1.66:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2363,
"optime" : {
"ts" : Timestamp(1490945835, 1),
"t" : NumberLong(2)
},
"optimeDurable" : {
"ts" : Timestamp(1490945835, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2017-03-31T07:37:15Z"),
"optimeDurableDate" : ISODate("2017-03-31T07:37:15Z"),
"lastHeartbeat" : ISODate("2017-03-31T07:37:20.818Z"),
"lastHeartbeatRecv" : ISODate("2017-03-31T07:37:21.809Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "10.11.1.59:27017",
"configVersion" : 4
},
{
"_id" : 2,
"name" : "10.11.1.59:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 3270,
"optime" : {
"ts" : Timestamp(1490945835, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2017-03-31T07:37:15Z"),
"electionTime" : Timestamp(1490944475, 1),
"electionDate" : ISODate("2017-03-31T07:14:35Z"),
"configVersion" : 4,
"self" : true
}
],
"ok" : 1
}
10.11.1.58已删除！
