MongoDB集群搭建
1 部署方案图
端口的规划：
mongos为 20000
config server 为 21000
shard1为 22001 ， shard2为22002， shard3为22003.
2 前期准备
2.1 服务器的准备
三台服务器f1: 12.112.1.236、f2: 12.112.1.237、f3: 12.112.1.238
(注：此ip为博主随意填写若有巧合请联系修改)
2.2 mogodb文件
使用版本：3.4.4
下载链接：https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel62-3.4.4.tgz
将下载的文件拷贝到f1、f2、f3三台服务器的/temp目录下
2.3 相应文件夹的创建
a) 将mongodb-linux-x86_64-rhel62-3.4.4.tgz解压到/app/data/server/mongodb-3.4.4
b) f1、f2、f3建立data 、conf、 config、logs、pid等文件夹
公共文件夹：
mkdir -p /app/data/server/mongodb-3.4.4/conf
mkdir -p /app/data/server/mongodb-3.4.4/data
mkdir -p /app/data/server/mongodb-3.4.4/logs
mkdir -p /app/data/server/mongodb-3.4.4/pid
mkdir -p /app/data/server/mongodb-3.4.4/config
f1创建文件夹：
mkdir -p /app/data/server/mongodb-3.4.4/shard11
mkdir -p /app/data/server/mongodb-3.4.4/shard21
mkdir -p /app/data/server/mongodb-3.4.4/shard31
mkdir -p /app/data/server/mongodb-3.4.4/data/shard11
mkdir -p /app/data/server/mongodb-3.4.4/data/shard21
mkdir -p /app/data/server/mongodb-3.4.4/data/shard31
f2创建文件夹：
mkdir -p /app/data/server/mongodb-3.4.4/shard12
mkdir -p /app/data/server/mongodb-3.4.4/shard22
mkdir -p /app/data/server/mongodb-3.4.4/shard32
mkdir -p /app/data/server/mongodb-3.4.4/data/shard12
mkdir -p /app/data/server/mongodb-3.4.4/data/shard22
mkdir -p /app/data/server/mongodb-3.4.4/data/shard32
f3创建文件夹：
mkdir -p /app/data/server/mongodb-3.4.4/shard13
mkdir -p /app/data/server/mongodb-3.4.4/shard23
mkdir -p /app/data/server/mongodb-3.4.4/shard33
mkdir -p /app/data/server/mongodb-3.4.4/data/shard13
mkdir -p /app/data/server/mongodb-3.4.4/data/shard23
mkdir -p /app/data/server/mongodb-3.4.4/data/shard33
2.4 mongdb-shard 分片配置
2.4.1 机器f1分片配置文件
机器f1分片配置/app/data/server/mongodb-3.4.4/config/shard11.conf:
dbpath= /app/data/server/mongodb-3.4.4/data/shard11
logpath= /app/data/server/mongodb-3.4.4/logs/shard11.log
pidfilepath= /app/data/server/mongodb-3.4.4/pid/shard11.pid
directoryperdb=true
logappend=true
replSet=shard1
port=22001
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile= /app/data/server/mongodb-3.4.4/key/keyfile
机器f1分片配置/app/data/server/mongodb-3.4.4/config/shard21.conf:
dbpath=/app/data/server/mongodb-3.4.4/data/shard21
logpath=/app/data/server/mongodb-3.4.4/logs/shard21.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/shard21.pid
directoryperdb=true
logappend=true
replSet=shard2
port=22002
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
机器f1分片配置/app/data/server/mongodb-3.4.4/config/shard31.conf:
dbpath=/app/data/server/mongodb-3.4.4/data/shard31
logpath=/app/data/server/mongodb-3.4.4/logs/shard31.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/shard31.pid
directoryperdb=true
logappend=true
replSet=shard3
port=22003
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
2.4.2 机器f2分片配置文件
机器 f2分片配置 shard12.conf:
dbpath=/app/data/server/mongodb-3.4.4/data/shard12
logpath=/app/data/server/mongodb-3.4.4/logs/shard12.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/shard12.pid
directoryperdb=true
logappend=true
replSet=shard1
port=22001
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
机器f2分片配置 shard22.conf
dbpath=/app/data/server/mongodb-3.4.4/data/shard22
logpath=/app/data/server/mongodb-3.4.4/logs/shard22.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/shard22.pid
directoryperdb=true
logappend=true
replSet=shard2
port=22002
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
机器f2分片配置 shard32.conf
dbpath=/app/data/server/mongodb-3.4.4/data/shard32
logpath=/app/data/server/mongodb-3.4.4/logs/shard32.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/shard32.pid
directoryperdb=true
logappend=true
replSet=shard3
port=22003
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
2.4.3 机器f3分片配置文件
机器f3分片配置 shard13.conf:
dbpath=/app/data/server/mongodb-3.4.4/data/shard13
logpath=/app/data/server/mongodb-3.4.4/logs/shard13.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/shard13.pid
directoryperdb=true
logappend=true
replSet=shard1
port=22001
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
机器f3分片配置 shard23.conf:
dbpath=/app/data/server/mongodb-3.4.4/data/shard23
logpath=/app/data/server/mongodb-3.4.4/logs/shard23.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/shard23.pid
directoryperdb=true
logappend=true
replSet=shard2
port=22002
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
机器f3分片配置 shard33.conf:
dbpath=/app/data/server/mongodb-3.4.4/data/shard33
logpath=/app/data/server/mongodb-3.4.4/logs/shard33.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/shard33.pid
directoryperdb=true
logappend=true
replSet=shard3
port=22003
fork=true
shardsvr=true
journal=true
oplogSize=64
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
2.4.4 mongdb-config配置
服务器f1 config.conf:
dbpath=/app/data/server/mongodb-3.4.4/config/
logpath=/app/data/server/mongodb-3.4.4/logs/config01.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/config01.pid
directoryperdb=true
logappend=true
port=21000
fork=true
replSet=cfgReplSet
configsvr=true
journal=true
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
服务器f2 config.conf:
dbpath=/app/data/server/mongodb-3.4.4/config/
logpath=/app/data/server/mongodb-3.4.4/logs/config02.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/config02.pid
directoryperdb=true
logappend=true
port=21000
fork=true
replSet=cfgReplSet
configsvr=true
journal=true
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
服务器f3 config.conf:
dbpath=/app/data/server/mongodb-3.4.4/config/
logpath=/app/data/server/mongodb-3.4.4/logs/config03.log
pidfilepath=/app/data/server/mongodb-3.4.4/pid/config03.pid
directoryperdb=true
logappend=true
port=21000
fork=true
replSet=cfgReplSet
configsvr=true
journal=true
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
2.4.5 mongdb-mongos配置
服务器f1、f2、f3均添加配置文件 mongos-route.conf:
configdb=cfgReplSet/ 12.112.1.236:21000, 12.112.1.237:21000,12.112.1.238:21000
pidfilepath=/app/data/server/mongodb-3.4.4/pid/route.pid
port=20000
logpath=/app/data/server/mongodb-3.4.4/logs/route.log
logappend=true
fork=true
#keyFile=/app/data/server/mongodb-3.4.4/key/keyfile
3 启动程序
启动顺序为 mogodb(shard) -> config - > mongos
3.1 shard分片启动
3.1.1.1 创建脚本文件
在f1，f2，f3的目录 /app/data/server/mongodb-3.4.4/bin 下建立 shard-start.sh 执行文件
f1脚本内容如下：
#!/bin/bash
>../logs/shard11.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard11.conf
>../logs/shard21.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard21.conf
>../logs/shard31.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard31.conf
f2脚本内容如下：
#!/bin/bash
>../logs/shard12.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard12.conf
>../logs/shard22.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard22.conf
>../logs/shard32.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard32.conf
f3脚本内容如下：
#!/bin/bash
>../logs/shard13.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard13.conf
>../logs/shard23.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard23.conf
>../logs/shard33.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard33.conf
3.1.1.2 授权
chmod 777 shard-start.sh
3.1.1.3 执行脚本，启动
./shard-start.sh
3.1.1.4 分片初始化
只需在机器f1执行：
分片1初始化：
cd /app/data/server/mongodb-3.4.4/bin
./mongo 12.112.1.236:22001/admin
db.runCommand(
{
"replSetInitiate":
{
"_id":"shard1",
"members":
[
{
"_id":1,
"host":"12.112.1.236:22001",priority:10
},
{
"_id":2,
"host":"12.112.1.237:22001",priority:8
},
{
"_id":3,
"host":"12.112.1.238:22001",arbiterOnly:true
}
]
}
});
分片2初始化：
cd /app/data/server/mongodb-3.4.4/bin
./mongo 12.112.1.236:22002/admin
db.runCommand(
{
"replSetInitiate":
{
"_id":"shard2",
"members":
[
{
"_id":1,
"host":"12.112.1.236:22002",priority:8
},
{
"_id":2,
"host":"12.112.1.237:22002",priority:10
},
{
"_id":3,
"host":"12.112.1.238:22002",arbiterOnly:true
}
]
}
});
分片3初始化：
cd /app/data/server/mongodb-3.4.4/bin
./mongo 12.112.1.236:22003/admin
db.runCommand(
{
"replSetInitiate":
{
"_id":"shard3",
"members":
[
{
"_id":1,
"host":"12.112.1.236:22003"
},
{
"_id":2,
"host":"12.112.1.237:22003"
},
{
"_id":3,
"host":"12.112.1.238:22003",arbiterOnly:true
}
]
}
});
3.1.1.5 移除分片（例如移除分片2）
db.runCommand(
{"removeshard" : "shard2/ 12.112.1.236:22002, 12.112.1.237:22002, 12.112.1.238:22002"}
);
3.2 config启动
3.2.1 创建脚本、授权、启动
在f1，f2，f3目录 /app/data/server/mongodb-3.4.4/bin 建 config-start.sh 执行文件
脚本文件内容如下：
#!/bin/bash
>../logs/config01.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/config.conf && tailf ../logs/config01.log
授权：chmod 777 ./config-start.sh
启动：./config-start.sh
3.2.2 config初始化
./mongo 12.112.1.236:21000/admin
仅需要12.112.1.236这一台执行
db.runCommand(
{
"replSetInitiate":
{
"_id":"cfgReplSet",
"members":
[
{
"_id":1,
"host":"12.112.1.236:21000"
},
{
"_id":2,
"host":"12.112.1.237:21000"
},
{
"_id":3,
"host":"12.112.1.238:21000"
}
]
}
});
3.2.3 config初始化后节点的查看
./mongo 12.112.1.236:21000/admin
rs.status()
结果如下：
{
"set" : "cfgReplSet",
"date" : ISODate("2018-12-01T10:06:13.988Z"),
"myState" : 1,
"term" : NumberLong(1),
"configsvr" : true,
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
},
"readConcernMajorityOpTime" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
},
"appliedOpTime" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
},
"durableOpTime" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
}
},
"members" : [
{
"_id" : 1,
"name" : "eshop-cache01:21000",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 13737,
"optime" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-12-01T10:06:11Z"),
"electionTime" : Timestamp(1543646459, 1),
"electionDate" : ISODate("2018-12-01T06:40:59Z"),
"configVersion" : 1,
"self" : true
},
{
"_id" : 2,
"name" : "eshop-cache02:21000",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 12326,
"optime" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
},
"optimeDurable" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-12-01T10:06:11Z"),
"optimeDurableDate" : ISODate("2018-12-01T10:06:11Z"),
"lastHeartbeat" : ISODate("2018-12-01T10:06:12.288Z"),
"lastHeartbeatRecv" : ISODate("2018-12-01T10:06:12.190Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "eshop-cache01:21000",
"configVersion" : 1
},
{
"_id" : 3,
"name" : "eshop-cache03:21000",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 12326,
"optime" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
},
"optimeDurable" : {
"ts" : Timestamp(1543658771, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-12-01T10:06:11Z"),
"optimeDurableDate" : ISODate("2018-12-01T10:06:11Z"),
"lastHeartbeat" : ISODate("2018-12-01T10:06:12.196Z"),
"lastHeartbeatRecv" : ISODate("2018-12-01T10:06:12.435Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "eshop-cache01:21000",
"configVersion" : 1
}
],
"ok" : 1
}
3.3 mongos启动
3.3.1 创建脚本、授权、启动
在f1，f2，f3目录 /app/data/server/mongodb-3.4.4/bin 建 route-start.sh 执行文件
脚本内容如下：
#!/bin/bash
>../logs/route.log && ./mongos -f /app/data/server/mongodb-3.4.4/conf/mongos-route.conf && tailf ../logs/route.log
授权：chmod 777 route-start.sh
启动：./route-start.sh
3.3.2 分片生效
服务器f1: ./mongo 127.0.0.1:20000/admin
串联路由服务器与分配副本集1
db.runCommand({
addshard : "shard1/ 12.112.1.236:22001, 12.112.1.237:22001, 12.112.1.238:22001"
});
串联路由服务器与分配副本集2
db.runCommand({
addshard : "shard2/ 12.112.1.236:22002, 12.112.1.237:22002, 12.112.1.238:22002"
});
串联路由服务器与分配副本集3
db.runCommand({
addshard : "shard3/ 12.112.1.236:22003, 12.112.1.237:22003, 12.112.1.238:22003"
});
#查看分片服务器的配置
db.runCommand( { listshards : 1 } );
结果如下：
{
"shards" : [
{
"_id" : "shard1",
"host" : "shard1/eshop-cache01:22001,eshop-cache02:22001",
"state" : 1
},
{
"_id" : "shard2",
"host" : "shard2/eshop-cache01:22002,eshop-cache02:22002",
"state" : 1
},
{
"_id" : "shard3",
"host" : "shard3/eshop-cache01:22003,eshop-cache02:22003",
"state" : 1
}
],
"ok" : 1
}
4 集群安全认证
4.1 key的生成
在f1 f2 f3机器 分别建立 key 文件夹
mkdir –p /app/data/server/mongodb-3.4.4/key
在文件夹key运行以下命令
>openssl rand -base64 753 >keyfile
>chmod 600 keyfile #重点 必须
cat /app/data/server/mongodb-3.4.4/key/keyfile
结果如下：
9DIO5b2DMBPgDm4G0519xbCf/nv08YFopN871tJDrxE8NAu+0IDygEfLaQCeHe3k
5c08TFs/CcibTh5+Bni9chJGaJjvolTP5wBiZhl1T/N0sMsmukOXLjFLeVtuILvY
BUs5m2O2g9n/q60M5P+Kh6kfZav13SrVuEwbJTxal2bzytHg+QqG7RD4nGMPXiME
Q7rrcxI5FiS86ZBNSI7MYHD4wHJlLjhLmtx3qivwa1c/RotL1Pcw3iFx5mTcLrpx
f8SLDOek9d/jvWw7132ny2WPYwSXMfOAQDpNAP4vAFQbLqllJKt5d0zBJVLi6sEi
y1762vlhMc7TihcxXIchQ+Jv2f8yeLN+e6IW4/sZmgvPGFKZOOnmhkyS4hLPcYVu
K0XVZ39Bba9foThL1uIeZPj9Wfu9WE4OeDB63NkzlaM7HfHRyeBaPx73No7fJGK9
lNyxToIUDdiW+xrjOuxA3ELSTVfhhf3/ZZinAKheQzi9pE6TuLBBOyMw/spUaUwh
ML9FCqdnBaxxztxKrT8h8hUKDeI0bllZk5S/51d5y81glQoivy6qr4hvRvvamgwg
BcaQyg7bPXdFzCnIJJtIyU69ofa+/jhA0LvfpxeO+03ngj7XzUgq6cUci7WznHhf
4cdRnutQDXdThenoB2A8KMaR2TpZTnYOVwi2wpjM6QVCRVYGW5pt6ZoxGF5NyLJa
3ME1wev9Uzrv2aEUVh/c8t6N06TZl9y3iIuUF73biu4ycuO23Bx44vWk1z31T42Y
PWut0TIBziPqjg8lm1sEnCN3BwgXDRJw1Ww0/c4ids52G9s+iLBIPoEaC5twExxc
TbG+mrYFNck2mprJZh5hywn2a5ugOZxdyQqa1noiDTa0TKVjY8wNQYi9osazELAb
l8zoxHnB3s3GcsST78TMcU2ucvMDg8W4Sg+kez9LaRdE7O1PoaIoSvzG5+OT0rCF
eLG8mTuaOHPCyn2gayIdOcH3tz8xGw6V7mVZUaNkV6H3
4.2 配置文件的修改
分别修改 服务器 f1、f2、f3所对应配置，conf目录 shard (分片) 、config（配置）、route（路由）下对对应的文件夹下 添加 keyFile 所对应路径
keyFile= /app/data/server/mongodb-3.4.4/key/keyfile
f1 服务器: config.conf route.conf shard11.conf shard21.conf shard31.conf
f2 服务器：config.conf route.conf shard12.conf shard22.conf shard32.conf
f3 服务器：config.conf route.conf shard13.conf shard23.conf shard33.conf
4.3 增加权限实例
admin 库中 增加用户权限
./mongo 127.0.0.1:20000/admin
db.createUser( { user: "kenny", pwd: "123456",roles: [ "userAdminAnyDatabase" ] } )
db.auth("kenny","123456")
5 分片的添加
5.1 文件夹的添加
将 keyFile屏蔽掉，不进行验证
f1 f2 f3机器建文件夹
f1机器建立文件夹在
/app/data/server/mongodb-3.4.4/data/shard41
f2机器建立文件夹在
/app/data/server/mongodb-3.4.4/data/shard42
f3机器建立文件夹在
/app/data/server/mongodb-3.4.4/data/shard43
Shard4 端口规划：22004
5.2 shard4配置
服务器f1分片配置 shard41.conf
dbpath= /app/data/server/mongodb-3.4.4/data/shard41
logpath= /app/data/server/mongodb-3.4.4/logs/shard41.log
pidfilepath= /app/data/server/mongodb-3.4.4/pid/shard41.pid
directoryperdb=true
logappend=true
replSet=shard4
port=22004
fork=true
shardsvr=true
journal=true
oplogSize=64
服务器f1分片配置 shard42.conf
dbpath= /app/data/server/mongodb-3.4.4/data/shard42
logpath= /app/data/server/mongodb-3.4.4/logs/shard42.log
pidfilepath= /app/data/server/mongodb-3.4.4/pid/shard42.pid
directoryperdb=true
logappend=true
replSet=shard4
port=22004
fork=true
shardsvr=true
journal=true
oplogSize=64
服务器f1分片配置 shard43.conf
dbpath= /app/data/server/mongodb-3.4.4/data/shard43
logpath= /app/data/server/mongodb-3.4.4/logs/shard43.log
pidfilepath= /app/data/server/mongodb-3.4.4/pid/shard43.pid
directoryperdb=true
logappend=true
replSet=shard4
port=22004
fork=true
shardsvr=true
journal=true
oplogSize=64
5.3 程序的启动
服务器f1:
>../logs/shard41.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard41.conf
服务器f2:
>../logs/shard42.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard42.conf
服务器f3:
>../logs/shard43.log && ./mongod -f /app/data/server/mongodb-3.4.4/conf/shard43.conf
shard4初始化
./mongo 127.0.0.1:22004/admin
db.runCommand(
{
"replSetInitiate":
{
"_id":"shard4",
"members":
[
{
"_id":1,
"host":"12.112.1.236:22004",priority:1
},
{
"_id":2,
"host":"12.112.1.237:22004",arbiterOnly:true
},
{
"_id":3,
"host":"12.112.1.238:22004",priority:8
}
]
}
});
5.4 查看分片状态
db.runCommand(
{ addshard : "shard4/12.112.1.236:22004, 12.112.1.237:22004, 12.112.1.238:22004"}
);
db.runCommand( { listshards : 1 } );
5.5 shard4生效
目前搭建了mongodb配置服务器、路由服务器，各个分片服务器，不过应用程序连接到 mongos 路由服务器并不能使用分片机制，还需要在程序里设置分片配置，让分片生效。
./mongo 127.0.0.1:20000/admin
串联路由服务器与分配副本集4
db.runCommand(
{ addshard : "shard4/12.112.1.236:22004, 12.112.1.237:22004, 12.112.1.238:22004"}
);
#查看分片服务器的配置
db.runCommand( { listshards : 1 } );
5.6 移除shard4
移除分片shard4 : db.runCommand( { removeShard: "shard4" } )
