MongoDB学习笔记（二）

2021-01-11 本文已影响0人 dev_winner

分片集群（Sharded Cluster）

分片（sharding）是一种跨多台机器分布数据的方法， MongoDB使用分片来支持具有非常大的数据集和高吞吐量操作的部署。将数据分散到不同的机器上，不需要功能强大的大型计算机就可以储存更多的数据，处理更多的负载。
具有大型数据集或高吞吐量应用程序的数据库系统会挑战单个服务器的容量，例如：高查询率会耗尽服务器的CPU容量。工作集大小大于系统的RAM会强调磁盘驱动器的I/O容量。
有两种解决系统增长的方法：垂直扩展和水平扩展。
- 垂直扩展意味着增加单个服务器的容量，例如使用更强大的CPU，添加更多的RAM或增加存储空间量。因为基于云的提供商基于可用的硬件配置具有硬性上限，所以垂直缩放有实际的最大值。
- 水平扩展意味着划分系统数据集并加载多个服务器，添加其它服务器以根据需要增加容量。虽然单个机器的总体速度或容量可能不高，但每台机器处理整个工作负载的子集，可能提供比单个高速大容量服务器更高的效率。扩展部署容量只需根据需添加额外的服务器，这可能比单个机器的高端硬件的总体成本更低。权衡是基础架构和部署维护的复杂性增加。
MongoDB支持通过分片进行水平扩展。
MongoDB分片群集包含以下组件：
- 分片（存储）：每个分片包含分片数据的子集，每个分片都可以部署为副本集。
- mongos （路由）：mongos充当查询路由器，在客户端应用程序和分片集群之间提供接口。
- config servers （"调度"的配置）：配置服务器存储群集的元数据。从MongoDB 3.4开始，必须将配置服务器部署为副本集（CSRS）。

分片集群中组件的交互

MongoDB在集合级别对数据进行分片，将集合数据分布在集群中的分片上。
分片集群架构目标：两个分片节点副本集（3+3）+一个配置节点副本集（3）+两个路由节点（2），共11个服务节点。

分片（存储）节点副本集的创建

第一套副本集

mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27018/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27018/data/db \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27018/conf \&
mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27118/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27118/data/db \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27118/conf \&
mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27218/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27218/data/db \&
mkdir -p /opt/mongodb/sharded_cluster/myshardrs01_27218/conf \&

新建一个配置文件：vim /opt/mongodb/sharded_cluster/myshardrs01_27018/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myshardrs01_27018/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myshardrs01_27018/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myshardrs01_27018/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27018
 replication:
    replSetName: myshardrs01
 sharding:
    clusterRole: shardsvr

对sharding.clusterRole值的说明：

值	描述
configsvr	Start this instance as a config server. The instance starts on port 27019 by default.
shardsvr	Start this instance as a shard. The instance starts on port 27018 by default.

注意：设置sharding.clusterRole需要mongod实例运行复制，要将实例部署为副本集成员，并设置replSetName副本集的名称。
新建一个配置文件：vim /opt/mongodb/sharded_cluster/myshardrs01_27118/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myshardrs01_27118/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myshardrs01_27118/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myshardrs01_27118/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27118
 replication:
    replSetName: myshardrs01
 sharding:
    clusterRole: shardsvr

新建一个配置文件：vim /opt/mongodb/sharded_cluster/myshardrs01_27218/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myshardrs01_27218/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myshardrs01_27218/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myshardrs01_27218/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27218
 replication:
    replSetName: myshardrs01
 sharding:
    clusterRole: shardsvr

启动第一套副本集：一主一副本一仲裁。

[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27018/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24008
child process started successfully, parent exiting
[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27118/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24054
child process started successfully, parent exiting
[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27218/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24097
child process started successfully, parent exiting
[root@dev mongodb]# ps -ef |grep mongod
root     24008     1  1 10:17 ?        00:00:01 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27018/conf/mongod.conf
root     24054     1  1 10:17 ?        00:00:01 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27118/conf/mongod.conf
root     24097     1  1 10:18 ?        00:00:00 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27218/conf/mongod.conf
root     24151 23883  0 10:19 pts/0    00:00:00 grep --color=auto mongod

初始化副本集和创建主节点：使用客户端命令连接任意一个节点，但尽量要连接主节点，并添加副本节点和仲裁节点：

[root@dev mongodb]# /opt/mongodb/bin/mongo --port 27018
> rs.initiate()
{
    "info2" : "no configuration specified. Using a default configuration for the set",
    "me" : "公网ip:27018",
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610419832, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610419832, 1)
}
myshardrs01:SECONDARY> 
myshardrs01:PRIMARY> rs.add("公网ip:27118")
{
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610420477, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610420477, 1)
}
myshardrs01:PRIMARY> rs.addArb("公网ip:27218")
{
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610420529, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610420529, 1)
}

第二套副本集

mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27318/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27318/data/db \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27318/conf \&
mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27418/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27418/data/db \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27418/conf \&
mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27518/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27518/data/db \&
mkdir -p /opt/mongodb/sharded_cluster/myshardrs02_27518/conf \&

新建一个配置文件：vim /opt/mongodb/sharded_cluster/myshardrs02_27318/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myshardrs02_27318/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myshardrs02_27318/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myshardrs02_27318/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27318
 replication:
    replSetName: myshardrs02
 sharding:
    clusterRole: shardsvr

新建一个配置文件：vim /opt/mongodb/sharded_cluster/myshardrs02_27418/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myshardrs02_27418/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myshardrs02_27418/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myshardrs02_27418/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27418
 replication:
    replSetName: myshardrs02
 sharding:
    clusterRole: shardsvr

新建一个配置文件：vim /opt/mongodb/sharded_cluster/myshardrs02_27518/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myshardrs02_27518/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myshardrs02_27518/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myshardrs02_27518/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27518
 replication:
    replSetName: myshardrs02
 sharding:
    clusterRole: shardsvr

启动第二套副本集：一主一副本一仲裁。

[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27318/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24173
child process started successfully, parent exiting
[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27418/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24216
child process started successfully, parent exiting
[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27518/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24259
child process started successfully, parent exiting
[root@dev mongodb]# ps -ef | grep mongod
root     24008     1  0 10:17 ?        00:00:05 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27018/conf/mongod.conf
root     24054     1  0 10:17 ?        00:00:05 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27118/conf/mongod.conf
root     24097     1  0 10:18 ?        00:00:05 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27218/conf/mongod.conf
root     24173     1  1 10:30 ?        00:00:01 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27318/conf/mongod.conf
root     24216     1  1 10:30 ?        00:00:01 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27418/conf/mongod.conf
root     24259     1  1 10:30 ?        00:00:00 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27518/conf/mongod.conf
root     24301 23883  0 10:31 pts/0    00:00:00 grep --color=auto mongod

初始化副本集和创建主节点：使用客户端命令连接任意一个节点，但尽量要连接主节点，并添加副本节点和仲裁节点：

[root@dev mongodb]# /opt/mongodb/bin/mongo --port 27318
> rs.initiate()
{
    "info2" : "no configuration specified. Using a default configuration for the set",
    "me" : "公网ip:27318",
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610420740, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610420740, 1)
}
myshardrs02:SECONDARY> 
myshardrs02:PRIMARY> rs.add("公网ip:27418")
{
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610420841, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610420841, 1)
}
myshardrs02:PRIMARY> rs.addArb("公网ip:27518")
{
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610420911, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610420911, 1)
}

配置节点副本集

mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27019/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27019/data/db \ &
mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27019/conf \&
mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27119/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27119/data/db \ &
mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27119/conf \&
mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27219/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27219/data/db \&
mkdir -p /opt/mongodb/sharded_cluster/myconfigrs_27219/conf \&

新建一个配置文件：vim /opt/mongodb/sharded_cluster/myconfigrs_27019/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myconfigrs_27019/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myconfigrs_27019/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myconfigrs_27019/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27019
 replication:
    replSetName: myconfigrs
 sharding:
    clusterRole: configsvr

新建一个配置文件：vim /opt/mongodb/sharded_cluster/myconfigrs_27119/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myconfigrs_27119/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myconfigrs_27119/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myconfigrs_27119/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27119
 replication:
    replSetName: myconfigrs
 sharding:
    clusterRole: configsvr

新建一个配置文件：vim /opt/mongodb/sharded_cluster/myconfigrs_27219/conf/mongod.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/myconfigrs_27219/logs/mongod.log
    logAppend: true
 storage:
    dbPath: /opt/mongodb/sharded_cluster/myconfigrs_27219/data/db
    journal:
        enabled: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/myconfigrs_27219/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27219
 replication:
    replSetName: myconfigrs
 sharding:
    clusterRole: configsvr

启动副本集：一主两副本。

[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27019/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24329
child process started successfully, parent exiting
[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27119/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24380
child process started successfully, parent exiting
[root@dev mongodb]# /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27219/conf/mongod.conf
about to fork child process, waiting until server is ready for connections.
forked process: 24431
child process started successfully, parent exiting
[root@dev mongodb]# ps -ef | grep mongod
root     24008     1  0 10:17 ?        00:00:09 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27018/conf/mongod.conf
root     24054     1  0 10:17 ?        00:00:09 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27118/conf/mongod.conf
root     24097     1  0 10:18 ?        00:00:09 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27218/conf/mongod.conf
root     24173     1  0 10:30 ?        00:00:05 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27318/conf/mongod.conf
root     24216     1  0 10:30 ?        00:00:05 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27418/conf/mongod.conf
root     24259     1  0 10:30 ?        00:00:05 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs02_27518/conf/mongod.conf
root     24329     1  1 10:44 ?        00:00:00 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27019/conf/mongod.conf
root     24380     1  1 10:45 ?        00:00:00 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27119/conf/mongod.conf
root     24431     1  2 10:45 ?        00:00:00 /opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27219/conf/mongod.conf
root     24483 23883  0 10:45 pts/0    00:00:00 grep --color=auto mongod

初始化副本集和创建主节点：使用客户端命令连接任意一个节点，但尽量要连接主节点，并添加两个副本节点：

[root@dev mongodb]# /opt/mongodb/bin/mongo --port 27019
> rs.initiate()
{
    "info2" : "no configuration specified. Using a default configuration for the set",
    "me" : "公网ip:27019",
    "ok" : 1,
    "$gleStats" : {
        "lastOpTime" : Timestamp(1610421160, 1),
        "electionId" : ObjectId("000000000000000000000000")
    },
    "lastCommittedOpTime" : Timestamp(0, 0),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610421160, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610421160, 1)
}
myconfigrs:SECONDARY> 
myconfigrs:PRIMARY> rs.add("公网ip:27119")
{
    "ok" : 1,
    "$gleStats" : {
        "lastOpTime" : {
            "ts" : Timestamp(1610421330, 1),
            "t" : NumberLong(1)
        },
        "electionId" : ObjectId("7fffffff0000000000000001")
    },
    "lastCommittedOpTime" : Timestamp(1610421330, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610421332, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610421330, 1)
}
myconfigrs:PRIMARY> rs.add("公网ip:27219")
{
    "ok" : 1,
    "$gleStats" : {
        "lastOpTime" : {
            "ts" : Timestamp(1610421343, 2),
            "t" : NumberLong(1)
        },
        "electionId" : ObjectId("7fffffff0000000000000001")
    },
    "lastCommittedOpTime" : Timestamp(1610421345, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610421345, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1610421343, 2)
}

路由节点的创建和操作

第一个路由节点的创建和连接

mkdir -p /opt/mongodb/sharded_cluster/mymongos_27017/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/mymongos_27017/conf

新建一个配置文件：vim /opt/mongodb/sharded_cluster/mymongos_27017/conf/mongos.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/mymongos_27017/logs/mongod.log
    logAppend: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/mymongos_27017/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27017
 sharding:
    configDB: myconfigrs/公网ip:27019,公网ip:27119,公网ip:27219

启动mongos服务，若启动失败，则查看 logs目录下的日志。

[root@dev mongodb]# /opt/mongodb/bin/mongos -f /opt/mongodb/sharded_cluster/mymongos_27017/conf/mongos.conf
about to fork child process, waiting until server is ready for connections.
forked process: 25188
child process started successfully, parent exiting

登录mongos客户端，若写数据则会报错，原因：通过路由节点操作，现只是连接了配置节点，还没有连接分片数据节点，需要添加第一套分片副本集和第二套分片副本集。
在路由节点上进行分片配置，添加分片的语法：sh.addShard("IP:Port")

[root@dev mongodb]# /opt/mongodb/bin/mongo --port 27017
mongos> show dbs
admin   0.000GB
config  0.000GB
mongos> use aadb
switched to db aadb
mongos> db.aa.insert({aa:"aa"})
WriteCommandError({
    "ok" : 0,
    "errmsg" : "unable to initialize targeter for write op for collection aadb.aa :: caused by :: Database aadb could not be created :: caused by :: No shards found",
    "code" : 70,
    "codeName" : "ShardNotFound",
    "operationTime" : Timestamp(1610430671, 2),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610430671, 2),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
})
mongos> sh.addShard("myshardrs01/公网ip:27018,公网ip:27118,公网ip:27218")
{
    "shardAdded" : "myshardrs01",
    "ok" : 1,
    "operationTime" : Timestamp(1610431326, 5),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610431326, 5),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
mongos> sh.addShard("myshardrs02/公网ip:27318,公网ip:27418,公网ip:27518")
{
    "shardAdded" : "myshardrs02",
    "ok" : 1,
    "operationTime" : Timestamp(1610431577, 3),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610431577, 3),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5ffd13a86c140163a85aa531")
  }
  shards:
        {  "_id" : "myshardrs01",  "host" : "myshardrs01/公网ip:27018,公网ip:27118",  "state" : 1 }
        {  "_id" : "myshardrs02",  "host" : "myshardrs02/公网ip:27318,公网ip:27418",  "state" : 1 }
  active mongoses:
        "4.4.3" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                39 : Success
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                myshardrs01 985
                                myshardrs02 39
                        too many chunks to print, use verbose if you want to force print

提示：若添加分片失败，则需要先手动移除分片、检查添加分片信息的正确性，再次添加分片。注意：若只剩下最后一个 shard，是无法被移除的；移除时会自动转移分片数据，需要一个时间过程。完成后，再次执行删除分片命令才能真正删除。

mongos> use admin
mongos> db.runCommand( { removeShard: "myshardrs02" } )

开启分片功能：①在mongos服务器上的articledb数据库配置sharding：sh.enableSharding("库名")；②对集合进行分片：sh.shardCollection(namespace, key, unique)，如：sh.shardCollection("库名.集合名", {"key":1})

参数	类型	描述
namespace	string	要分片共享的目标集合的命名空间，格式：`<database>.<collection>`
key	document	用作分片键的索引规范文档。shard键决定MongoDB如何在shard之间分发文档。除非集合为空，否则索引必须在shardCollection命令之前存在。若集合为空，则MongoDB在对集合进行分片之前创建索引，前提是支持片键的索引不存在。简单来说，其由包含字段和该字段的索引遍历方向的文档组成。
unique	boolean	当值为true的情况下，片键字段上会限制为唯一索引。哈希策略片键不支持唯一索引。默认是false。

对集合进行分片时，需要选择一个片键（Shard Key），shard key是每条记录都必须包含的，且建立了索引的单个字段或复合字段，MongoDB按照片键将数据划分到不同的数据块中，并将数据块均衡地分布到所有分片中。为了按照片键划分数据块，MongoDB使用基于哈希的分片方式（随机平均分配）或者基于范围的分片方式（数值大小分配）。
分片规则一：哈希策略。对于基于哈希的分片，MongoDB计算一个字段的哈希值，并用这个哈希值来创建数据块。在使用基于哈希分片的系统中，拥有"相近"片键的文档很可能不会存储在同一个数据块中，因此数据的分离性更好一些。如：使用nickname作为片键，根据其值的哈希值进行数据分片：sh.shardCollection("articledb.comment",{"nickname":"hashed"})。
分片规则二：范围策略。对于基于范围的分片，MongoDB按照片键的范围把数据分成不同部分。假设有一个数字的片键：想象一个从负无穷到正无穷的直线，每一个片键的值都在直线上画了一个点。MongoDB把这条直线划分为更短的不重叠的片段，并称之为数据块，每个数据块包含了片键在一定范围内的数据。在使用片键做范围划分的系统中，拥有"相近"片键的文档很可能存储在同一个数据块中，因此也会存储在同一个分片中。如：使用作者年龄字段作为片键，按照点赞数的值进行分片：sh.shardCollection("articledb.author",{"age":1})
注意：①一个集合只能指定一个片键。②一旦对一个集合分片，分片键和分片值就不可改变。如：不能给集合选择不同的分片键、不能更新分片键的值。③根据age索引进行分配数据。

mongos> sh.enableSharding("articledb")
{
    "ok" : 1,
    "operationTime" : Timestamp(1610432042, 8),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610432042, 8),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
mongos> sh.shardCollection("articledb.comment",{"nickname":"hashed"})
{
    "collectionsharded" : "articledb.comment",
    "collectionUUID" : UUID("f3676c7e-25be-4364-8216-5a8c0338512f"),
    "ok" : 1,
    "operationTime" : Timestamp(1610433360, 5),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610433360, 5),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
mongos> sh.shardCollection("articledb.author",{"age":1})
{
    "collectionsharded" : "articledb.author",
    "collectionUUID" : UUID("e993611c-f30c-4871-a488-dffbd56d5413"),
    "ok" : 1,
    "operationTime" : Timestamp(1610433415, 12),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1610433415, 12),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5ffd13a86c140163a85aa531")
  }
  shards:
        {  "_id" : "myshardrs01",  "host" : "myshardrs01/公网ip:27018,公网ip:27118",  "state" : 1 }
        {  "_id" : "myshardrs02",  "host" : "myshardrs02/公网ip:27318,公网ip:27418",  "state" : 1 }
  active mongoses:
        "4.4.3" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                512 : Success
  databases:
        {  "_id" : "articledb",  "primary" : "myshardrs02",  "partitioned" : true,  "version" : {  "uuid" : UUID("03cc6699-146c-46f3-b266-5f43b8d65a39"),  "lastMod" : 1 } }
                articledb.author
                        shard key: { "age" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                myshardrs02 1
                        { "age" : { "$minKey" : 1 } } -->> { "age" : { "$maxKey" : 1 } } on : myshardrs02 Timestamp(1, 0) 
                articledb.comment
                        shard key: { "nickname" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                myshardrs01 2
                                myshardrs02 2
                        { "nickname" : { "$minKey" : 1 } } -->> { "nickname" : NumberLong("-4611686018427387902") } on : myshardrs01 Timestamp(1, 0) 
                        { "nickname" : NumberLong("-4611686018427387902") } -->> { "nickname" : NumberLong(0) } on : myshardrs01 Timestamp(1, 1) 
                        { "nickname" : NumberLong(0) } -->> { "nickname" : NumberLong("4611686018427387902") } on : myshardrs02 Timestamp(1, 2) 
                        { "nickname" : NumberLong("4611686018427387902") } -->> { "nickname" : { "$maxKey" : 1 } } on : myshardrs02 Timestamp(1, 3) 
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                myshardrs01 512
                                myshardrs02 512
                        too many chunks to print, use verbose if you want to force print

基于范围的分片方式与基于哈希的分片方式性能对比：
- 前者提供了更高效的范围查询，给定一个片键的范围，分发路由可以很简单地确定哪个数据块存储了请求需要的数据，并将请求转发到相应的分片中。不过基于范围的分片会导致数据在不同分片上的不均衡，有时带来的消极作用会大于查询性能的积极
  作用。比如：若片键所在的字段是线性增长的，则一定时间内所有请求都会落到某个固定的数据块中，最终导致分布在同一个分片中。在这种情况下，一小部分分片承载了集群大部分的数据，系统并不能很好地进行扩展。
- 与此相比，后者以范围查询性能的损失为代价，保证了集群中数据的均衡。哈希值的随机性使数据随机分布在每个数据块中，因此也随机分布在不同分片中。但是正因为随机性，一个范围查询很难确定应该请求哪些分片。通常为了返回需要的结果，需要请求所有分片。
- 如无特殊情况，一般推荐使用Hash Sharding。使用_id作为片键是一个不错的选择，因为它本来就有的，所以可以使用数据文档_id的哈希作为片键。这个方案使得读和写都能够平均分布，且它能够保证每个文档都有不同的片键，所以数据块能够分得很精细。理想化的 shard key 可以让 documents 均匀地在集群中分布：

显示集群的详细信息：

mongos> db.printShardingStatus()

查看均衡器是否工作（需要重新均衡时系统才会自动启动，不用管它）：

mongos> sh.isBalancerRunning()
false

查看当前 Balancer状态：

mongos> sh.getBalancerState()
true

分片后插入数据测试

测试一（哈希规则）：登录mongs后，测试向comment集合插入1000条数据。提示：for插入语句是js的语法，因为mongo的shell是一个JavaScript的shell。注意：从路由上插入的数据必须包含片键，否则无法插入！

mongos> use articledb
switched to db articledb
mongos> for(var i=1;i<=1000;i++){db.comment.insert({_id:i+"",nickname:"BoBo"+i})}
WriteResult({ "nInserted" : 1 })
mongos> db.comment.count()
1000

分别登陆两个片的主节点，统计文档数量：
第一个分片副本集：/opt/mongodb/bin/mongo --port 27018

myshardrs01:PRIMARY> use articledb
switched to db articledb
myshardrs01:PRIMARY> db.comment.count()
507

第二个分片副本集：/opt/mongodb/bin/mongo --port 27318

myshardrs02:PRIMARY> use articledb
switched to db articledb
myshardrs02:PRIMARY> db.comment.count()
493

从结果可以看到，1000条数据近似均匀地分布到了2个shard上，其是根据片键的哈希值分配的。这种分配方式非常易于水平扩展：一旦数据存储需要更大空间，可以直接再增加分片即可，同时提升了性能。
使用db.comment.stats()查看单个集合的完整情况，mongos执行该命令可以查看该集合的数据分片的情况。
使用sh.status()查看本库内所有集合的分片信息。
测试二（范围规则）：登录mongs后，向comment集合插入12057条数据做测试：

mongos> use config
switched to db config
mongos> db.settings.save({ _id:"chunksize", value: 1 })
WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "chunksize" })
mongos> for(var i=1;i<=20000;i++){db.author.save({"name":"BoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBo"+i,"age":NumberInt(i%120)})}
mongos> db.author.count()
12057

分别登陆两个片的主节点，统计文档数量：
第一个分片副本集：/opt/mongodb/bin/mongo --port 27018

myshardrs01:PRIMARY> use articledb
switched to db articledb
myshardrs01:PRIMARY> db.author.count()
99

第二个分片副本集：/opt/mongodb/bin/mongo --port 27318

myshardrs02:PRIMARY> use articledb
switched to db articledb
myshardrs02:PRIMARY> db.author.count()
11958

若查看状态发现没有分片，则可能是以下原因造成的：①系统繁忙，正在分片中；②数据块（chunk）没有填满，默认的数据块尺寸（chunksize）是64M，填满后才会考虑向其它片的数据块填充数据。因此，为了方便测试，可以将其改为1M，操作如下：

use config
db.settings.save( { _id:"chunksize", value: 1 } )

第二个路由节点的创建和连接

mkdir -p /opt/mongodb/sharded_cluster/mymongos_27117/logs \ &
mkdir -p /opt/mongodb/sharded_cluster/mymongos_27117/conf

新建一个配置文件：vim /opt/mongodb/sharded_cluster/mymongos_27117/conf/mongos.conf

 systemLog:
    destination: file
    path: /opt/mongodb/sharded_cluster/mymongos_27117/logs/mongod.log
    logAppend: true
 processManagement:
    fork: true
    pidFilePath: /opt/mongodb/sharded_cluster/mymongos_27117/logs/mongod.pid
 net:
    bindIp: localhost,192.168.0.128
    port: 27117
 sharding:
    configDB: myconfigrs/公网ip:27019,公网ip:27119,公网ip:27219

启动mongos服务，若启动失败，则查看 logs目录下的日志。

[root@dev mongodb]# /opt/mongodb/bin/mongos -f /opt/mongodb/sharded_cluster/mymongos_27117/conf/mongos.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27783
child process started successfully, parent exiting

使用Compass 连接分片集群

若在搭建分片时有操作失败或配置有问题，需要重新来过，可以进行如下操作：
第一步：查询出所有测试服务节点的进程：ps -ef | grep mongo，根据列举的进程编号，依次中断进程：kill -2 进程编号。
第二步：清除所有节点的数据：

rm -rf /opt/mongodb/sharded_cluster/myconfigrs_27019/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/myconfigrs_27119/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/myconfigrs_27219/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/myshardrs01_27018/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/myshardrs01_27118/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/myshardrs01_27218/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/myshardrs02_27318/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/myshardrs02_27418/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/myshardrs02_27518/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/mymongos_27017/data/db/*.* \ &
rm -rf /opt/mongodb/sharded_cluster/mymongos_27117/data/db/*.*

第三步：查看或修改有问题的配置。
第四步：依次启动所有节点，不包括路由节点：

/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27018/conf/mongod.conf
/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27118/conf/mongod.conf
/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27218/conf/mongod.conf
/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27318/conf/mongod.conf
/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27418/conf/mongod.conf
/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myshardrs01_27518/conf/mongod.conf
/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27019/conf/mongod.conf
/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27119/conf/mongod.conf
/opt/mongodb/bin/mongod -f /opt/mongodb/sharded_cluster/myconfigrs_27219/conf/mongod.conf

第五步：对两个数据分片副本集和一个配置副本集进行初始化和相关配置。
第六步：检查路由mongos的配置，并启动mongos服务：

/opt/mongodb/bin/mongos -f /opt/mongodb/sharded_cluster/mymongos_27017/conf/mongos.conf
/opt/mongodb/bin/mongos -f /opt/mongodb/sharded_cluster/mymongos_27117/conf/mongos.conf

第七步：mongo登录mongos，在其上进行相关操作。

MongoDB学习笔记（二）

分片集群（Sharded Cluster）

分片（存储）节点副本集的创建

第一套副本集

第二套副本集

配置节点副本集

路由节点的创建和操作

第一个路由节点的创建和连接

分片后插入数据测试

第二个路由节点的创建和连接

猜你喜欢

热点阅读