RocketMQ安装配置与维护手册
2021-01-27 本文已影响0人
以梦为马_8467
准备工作
安装包准备
JAVA安装包
下载地址见https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html
RocketMQ安装包
下载地址见https://github.com/apache/rocketmq/releases
安装配置
| 组件 | 版本 |
|---|---|
| JDK | 8u241 |
| RocketMQ | 4.6.1 |
单机版
| 机器IP | 操作系统版本 | 安装目录 |
|---|---|---|
| 192.168.229.121 | CentOS-7.5 | /middleware |
注意事项
namesrv默认配置JVM内存为4G,broker默认配置JVM内存为8G,如需修改,可以修改rocketmq家目录下bin下的runserver.sh和runbroker.sh,如下图所示:
runserver.sh
image.png
runbroker.sh
image.png
安装步骤
- 安装配置JAVA环境
- 解压jdk压缩包到指定目录
[root@CentOS7 ~]# tar -xf jdk-8u241-linux-x64.tar.gz
[root@CentOS7 ~]# mv jdk1.8.0_241 /usr/local/
- 设置环境变量
[root@CentOS7 ~]# echo "export $JAVA_HOME=/usr/local/jdk1.8.0_241" >> /etc/profile
[root@CentOS7 ~]# echo "export $PATH=$JAVA_HOME/bin:$PATH >> /etc/profile
- 安装RocketMQ
拷贝rocketmq安装包到指定目录,此处为/middleware/rocketmq-4.6.1
- 启动namesrv
[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqnamesrv &
- 安装broker
修改broker配置文件
[root@CentOS7 rocketmq-4.6.1]# cat /middleware/rocketmq-4.6.1/conf/broker.conf
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerIP1=192.168.229.121
brokerId=0
deleteWhen=04
fileReservedTime=48
storePathRootDir=/middleware/rocketmq/store
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
namesrvAddr=192.168.229.121:9876
启动broker
[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqbroker -c /middleware/rocketmq-4.6.1/conf/broker.conf &
- 验证
[root@CentOS7 ~]# /middleware/rocketmq-4.6.1/bin/mqadmin cluster -n 192.168.229.121:9876
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
DefaultCluster broker-a 0 192.168.229.121:10911 V4_6_1 0.00(0,0ms) 0.00(0,0ms) 0 447650.86 1.0000
集群版
| 机器IP | 操作系统版本 | 安装目录 | 角色 |
|---|---|---|---|
| 192.168.229.121 | CentOS-7.5 | /middleware | 从 |
| 192.168.229.128 | CentOS-7.5 | /middleware | 主 |
注意事项
- namesrv默认配置JVM内存为4G,broker默认配置JVM内存为8G,修改方式同单机版修改方式
- 集群版默认提供了3个配置文件,分别是2主2从同步复制,2主2从异步复制和双主模式,实际部署可以根据自己的需要修改对应的配置文件
安装步骤
分别在2台机器上执行下列操作
- 安装JAVA环境
- 解压jdk压缩包到指定目录
[root@CentOS7 ~]# tar -xf jdk-8u241-linux-x64.tar.gz
[root@CentOS7 ~]# mv jdk1.8.0_241 /usr/local/
- 设置环境变量
[root@CentOS7 ~]# echo "export $JAVA_HOME=/usr/local/jdk1.8.0_241" >> /etc/profile
[root@CentOS7 ~]# echo "export $PATH=$JAVA_HOME/bin:$PATH >> /etc/profile
- 安装RocketMQ
拷贝rocketmq安装包到指定目录,此处为/middleware/rocketmq-4.6.1
- 启动namesrv
[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqnamesrv &
- 安装broker
修改broker配置文件
192.168.229.128-主broker的配置文件为
[root@cent7-2 ~]# cat /middleware/rocketmq-4.6.1/conf/broker-10-m.conf
brokerClusterName=cluster1
brokerIP1=192.168.229.128
brokerIP2=192.168.229.128
brokerName=broker-10
brokerId=0
deleteWhen=04
fileReservedTime=48
storePathRootDir=/middleware/rocketmq/store
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
namesrvAddr=192.168.229.128:9876;192.168.229.121:9876
192.168.229.121-从broker的配置文件为
[root@CentOS7 conf]# cat /middleware/rocketmq-4.6.1/conf/broker-10-s.conf
brokerClusterName=cluster1
brokerIP1=192.168.229.121
brokerIP2=192.168.229.121
brokerName=broker-10
brokerId=1
deleteWhen=04
fileReservedTime=48
storePathRootDir=/middleware/rocketmq/store
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
namesrvAddr=192.168.229.128:9876;192.168.229.121:9876
192.168.229.128启动broker
[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqbroker -c /middleware/rocketmq-4.6.1/conf/broker-10-m.conf &
192.168.229.121启动broker
[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqbroker -c /middleware/rocketmq-4.6.1/conf/broker-10-s.conf &
broker配置文件说明
| 参数值 | 说明 |
|---|---|
| brokerClusterName | 集群名称,一个集群内所有的broker机器该值配置相同 |
| brokerIP1 | broker服务监听IP |
| brokerIP2 | 存在broker主从时，在broker主节点上配置了brokerIP2的话,broker从节点会连接主节点配置的brokerIP2来同步,如果存在多块网卡,最好配置brokerIP1和brokerIP2,指定集群需要通信的IP地址 |
| brokerName | broker名称,主broker和从broker通过使用相同的Broker名称来表明相互关系 |
| brokerId | 一个主broker可以有多个从broker ，0表示是主，大于0表示从broker |
| deleteWhen | 删除过期消息文件的时间 |
| fileReservedTime | 保存消息的时长，单位是小时，自动删除超时的消息,与deleteWhen共用 |
| storePathRootDir | 消息文件的保存目录 |
| brokerRole | SYNC_MASTER: 同步复制,数据会同步到slave一份，master和slave均写入成功后才反馈给生产者写成功状态；ASYNC_MASTER: 异步复制,数据会异步的形式发送到slave一份,master写入成功后即反馈给生产者写成功状态；SLAVE: 从broker |
| flushDiskType | SYNC_FLUSH: 同步刷盘,消息刷到磁盘成功后才返回写成功状态；ASYNC_FLUSH: 异步刷盘,消息刷到pagecache后即返回写成功状态 |
| namesrvAddr | namesrv的地址 |
- 安装验证
[root@cent7-2 ~]# /middleware/rocketmq-4.6.1/bin/mqadmin clusterlist -n "192.168.229.121:9876;192.168.229.128:9876"
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
cluster1 broker-10 0 192.168.229.128:10911 V4_6_1 0.00(0,0ms) 0.00(0,0ms) 0 5.93 1.0000
cluster1 broker-10 1 192.168.229.121:10911 V4_6_1 0.00(0,0ms) 0.00(0,0ms) 0 5.93 1.0000
RocketMQ优化实践
内核参数优化
| 命令 | 说明 |
|---|---|
| echo 'ulimit -n 655350' >> /etc/profile | 设置当前用户可以打开的最大文件描述符数量,重启失效 |
| echo '* hard nofile 655350' >> /etc/security/limits.conf | 设置任一用户可以打开的最大文件描述符数量 |
| echo '* hard memlock unlimited' >> /etc/security/limits.conf | 最大锁定内存地址空间硬限制,避免内存被操作系统swap掉 |
| echo '* soft memlock unlimited' >> /etc/security/limits.conf | 最大锁定内存地址空间软限制,避免内存被操作系统swap掉 |
| echo 'deadline' > /sys/block/${DISK}/queue/scheduler | 设置磁盘的调度策略为deadline |
| sysctl -w vm.overcommit_memory=1 | 控制是否允许内存overcommit。设为1，则是允许。当应用申请内存时，系统都会认为存在足够的内存，准许申请 |
| sysctl -w vm.drop_caches=1 | 设置为1会释放page cache。这个操作是一次性的，在执行该命令时释放page cache，没有持续性作用 |
| sysctl -w vm.zone_reclaim_mode=0 | 该配置用于控制zone内存使用完之后的处理策略，设为0则禁止zone reclaim。对于大量使用缓存的应用来说，一般都需要禁止掉 |
| sysctl -w vm.max_map_count=655360 | RocketMQ使用MMAP映射文件到内存，因此需要设置映射文件的数量，避免MMAP操作失败 |
| sysctl -w vm.dirty_background_ratio=50 | 设置内存中的脏数据占比，当超过该值时，内核后台线程将数据脏数据刷入磁盘 |
| sysctl -w vm.dirty_ratio=50 | 设置内存的脏数据占比，当超过该值时，应用进程会主动将数据刷入磁盘 |
| sysctl -w vm.dirty_writeback_centisecs=360000 | 内核线程会定期启动将内存中的旧数据刷入磁盘，通过此参数可以控制启动间隔 |
| sysctl -w vm.page-cluster=3 | 设置一次读操作会加载几个数据页。 |
| sysctl -w vm.swappiness=1 | vm.swappiness用于控制使用系统swap空间比例，一般设置为0是禁止使用swap，设为1估计是某些系统不支持设置为0.RocketMQ的数据存储基于MMAP，大量使用内存，因此需要禁止swap，否则性能会急剧下降 |
broker配置优化
如果场景对消息可靠性要求比较严格,可以采取同步复制,同步刷盘模式,但是该种模式对性能影响比较大
主broker:
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
从broker:
brokerRole=SLAVE
flushDiskType=SYNC_FLUSH
如果追求性能,可以采取异步复制异步刷盘模式,该种模式性能最高,但是消息可靠性最差
主broker:
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
从broker:
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
在生产环境建议采用同步复制,异步刷盘模式,兼顾性能和消息可靠性
主broker:
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
从broker:
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
RocketMQ日常维护操作
查看集群状态
sh mqadmin clusterlist -n 'namesrv地址:端口'
[root@CentOS7 bin]# sh mqadmin clusterlist -n "192.168.229.121:9876;192.168.229.128:9876"
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
cluster1 broker-10 0 192.168.229.128:10911 V4_6_1 0.00(0,0ms) 0.00(0,0ms) 0 8.88 1.0000
cluster1 broker-10 1 192.168.229.121:10911 V4_6_1 0.00(0,0ms) 0.00(0,0ms) 0 8.88 1.0000
输出说明:
BID: broker的ID,0为主,其他的为从
Addr: broker的地址
Version: RocketMQ版本号
InTPS(LOAD): 生产者每秒钟发送的消息数量(发送线程池队列头部元素等待时间)
OutTPS(LOAD): 消费者每秒钟消费的消息数量(拉取线程池队列头部元素等待时间)
PCWait(ms): 缓存锁定时间
Hour: Broker启动时间
SPACE: commitLog磁盘占比
查看消费进度
sh mqadmin consumerProgress -g 'cid或者gid' -n 'namesrv地址:端口'
[root@CentOS7 bin]# sh mqadmin consumerprogress -g CID_PRODRES_CHANGE_OLD_W_E_CA
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
JM.Log:INFO Init JM logger with Slf4jLoggerFactory success, sun.misc.Launcher$ExtClassLoader@4b1210ee
JM.Log:INFO Log root path: /home/admin/logs/
JM.Log:INFO Set diamond-client log path: /home/admin/logs/diamond-client
#Topic #Broker Name #QID #Broker Offset #Consumer Offset #Diff #LastTime
%RETRY%CID_PRODRES_CHANGE_OLD_W_ broker11 0 616327 616326 1 2021-01-25 18:35:41
%RETRY%CID_PRODRES_CHANGE_OLD_W_ broker12 0 275051 275051 0 2021-01-25 18:35:28
PRODRES_CHANGE_OLD_W_E_CARD broker11 0 3379335 3379335 0 2021-01-25 18:35:37
PRODRES_CHANGE_OLD_W_E_CARD broker11 1 3379825 3379825 0 2021-01-25 18:35:13
PRODRES_CHANGE_OLD_W_E_CARD broker11 2 3381098 3381098 0 2021-01-25 18:34:57
PRODRES_CHANGE_OLD_W_E_CARD broker11 3 3381333 3381333 0 2021-01-25 18:35:25
PRODRES_CHANGE_OLD_W_E_CARD broker11 4 3378243 3378243 0 2021-01-25 18:34:50
PRODRES_CHANGE_OLD_W_E_CARD broker11 5 3380287 3380287 0 2021-01-25 18:35:30
PRODRES_CHANGE_OLD_W_E_CARD broker11 6 3379392 3379392 0 2021-01-25 18:35:17
PRODRES_CHANGE_OLD_W_E_CARD broker11 7 3376878 3376878 0 2021-01-25 18:32:45
PRODRES_CHANGE_OLD_W_E_CARD broker12 0 3339015 3339015 0 2021-01-25 18:35:32
PRODRES_CHANGE_OLD_W_E_CARD broker12 1 3340017 3340017 0 2021-01-25 18:34:11
PRODRES_CHANGE_OLD_W_E_CARD broker12 2 3339889 3339888 1 2021-01-25 18:34:28
PRODRES_CHANGE_OLD_W_E_CARD broker12 3 3336453 3336453 0 2021-01-25 18:34:32
PRODRES_CHANGE_OLD_W_E_CARD broker12 4 3340674 3340674 0 2021-01-25 18:35:25
PRODRES_CHANGE_OLD_W_E_CARD broker12 5 3337425 3337425 0 2021-01-25 18:34:42
PRODRES_CHANGE_OLD_W_E_CARD broker12 6 3340160 3340160 0 2021-01-25 18:35:05
PRODRES_CHANGE_OLD_W_E_CARD broker12 7 3338999 3338999 0 2021-01-25 18:34:30
Consume TPS: 0.6499999999999999
Diff Total: 2
输出说明:
Topic: topic名称
Qid: 队列id
Broker offset: broker消息偏移量
Cousumer offset: 消费者消费消息偏移量(Consumer offset-Broker offset为堆积量)
Diff: 每个队列的堆积量
Consume TPS: 该组消费者每秒钟消费的消息数量
Diff Total: 该组消费者总的堆积量
查看broker配置参数
sh mqadmin getBrokerConfig -b 'broker地址:端口'
[root@CentOS7 bin]# sh mqadmin getBrokerConfig -b 192.168.229.121:10911
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
============192.168.229.121:10911============
serverSelectorThreads = 3
brokerRole = SLAVE
serverSocketRcvBufSize = 131072
osPageCacheBusyTimeOutMills = 1000
shortPollingTimeMills = 1000
clientSocketRcvBufSize = 131072
clusterTopicEnable = true
brokerTopicEnable = true
autoCreateTopicEnable = true
maxErrorRateOfBloomFilter = 20
......
修改broker配置参数
禁止broker写入
sh mqadmin updateBrokerConfig -b 'broker地址:端口' -k brokerPermission -v 4
[root@CentOS7 bin]# sh mqadmin updateBrokerConfig -b 192.168.229.121:10911 -k brokerPermission -v 4
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
update broker config success, 192.168.229.121:10911
允许broker写入
sh mqadmin updateBrokerConfig -b 'broker地址:端口' -k brokerPermission -v 6
[root@CentOS7 bin]# sh mqadmin updateBrokerConfig -b 192.168.229.121:10911 -k brokerPermission -v 6
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
update broker config success, 192.168.229.121:10911
修改发送队列等待时间
sh mqadmin updateBrokerConfig -c '集群名称' -k waitTimeMillsInSendQueue -v '值'
[root@CentOS7 bin]# sh mqadmin updateBrokerConfig -c cluster1 -k waitTimeMillsInSendQueue -v 2700 -n "192.168.229.121:9876;192.168.229.128:9876"
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
update broker config success, 192.168.229.128:10911
查询消费者的通道分配情况
[root@CentOS7 bin]# sh mqadmin consumerstatus -g CID_PRODACT_KFK_CHARGE
-bash-4.1$ sh mqadmin consumerstatus -g CID_PRODRES_CHANGE_OLD_W_E_CA
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
JM.Log:INFO Init JM logger with Slf4jLoggerFactory success, sun.misc.Launcher$ExtClassLoader@4b1210ee
JM.Log:INFO Log root path: /home/admin/logs/
JM.Log:INFO Set diamond-client log path: /home/admin/logs/diamond-client
001 10.191.6.123@29201#-72148370#-2113662258#16854876802355494 V3_6_2_SNAPSHOT 1611649872403/10.191.6.123@29201#-72148370#-2113662258#16854876802355494
002 10.191.32.43@15595#-72148370#-2113662258#16856843059515866 V3_6_2_SNAPSHOT 1611649872403/10.191.32.43@15595#-72148370#-2113662258#16856843059515866
003 10.191.32.42@12626#-72148370#-2113662258#16853156057574522 V3_6_2_SNAPSHOT 1611649872403/10.191.32.42@12626#-72148370#-2113662258#16853156057574522
004 10.191.32.51@11868#-72148370#-2113662258#4147806882227359 V3_6_2_SNAPSHOT 1611649872403/10.191.32.51@11868#-72148370#-2113662258#4147806882227359
005 10.191.6.127@15336#-72148370#-2113662258#16851112540028361 V3_6_2_SNAPSHOT 1611649872403/10.191.6.127@15336#-72148370#-2113662258#16851112540028361
006 10.191.6.126@15109#-72148370#-2113662258#16851128433606829 V3_6_2_SNAPSHOT 1611649872403/10.191.6.126@15109#-72148370#-2113662258#16851128433606829
007 10.191.6.129@23882#-72148370#-2113662258#16600532628304138 V3_6_2_SNAPSHOT 1611649872403/10.191.6.129@23882#-72148370#-2113662258#16600532628304138
008 10.191.6.125@3824#-72148370#-2113662258#16854816388973399 V3_6_2_SNAPSHOT 1611649872403/10.191.6.125@3824#-72148370#-2113662258#16854816388973399
009 10.191.6.122@6276#-72148370#-2113662258#16854877719952652 V3_6_2_SNAPSHOT 1611649872403/10.191.6.122@6276#-72148370#-2113662258#16854877719952652
010 10.191.32.44@11632#-72148370#-2113662258#16856946745907441 V3_6_2_SNAPSHOT 1611649872403/10.191.32.44@11632#-72148370#-2113662258#16856946745907441
011 10.191.6.124@16279#-72148370#-2113662258#16854800476860500 V3_6_2_SNAPSHOT 1611649872403/10.191.6.124@16279#-72148370#-2113662258#16854800476860500
012 10.191.32.53@24030#-72148370#-2113662258#16856847451283931 V3_6_2_SNAPSHOT 1611649872403/10.191.32.53@24030#-72148370#-2113662258#16856847451283931
013 10.191.32.50@16649#-72148370#-2113662258#16853127473470245 V3_6_2_SNAPSHOT 1611649872403/10.191.32.50@16649#-72148370#-2113662258#16853127473470245
014 10.191.32.52@14362#-72148370#-2113662258#16856838700264761 V3_6_2_SNAPSHOT 1611649872403/10.191.32.52@14362#-72148370#-2113662258#16856838700264761
015 10.191.6.128@15048#-72148370#-2113662258#16525614251821957 V3_6_2_SNAPSHOT 1611649872403/10.191.6.128@15048#-72148370#-2113662258#16525614251821957
Same subscription in the same group of consumer
Rebalance OK
[root@CentOS7 bin]# cd 1611649872403/
[root@CentOS7 bin]# cat * | grep ProcessQueueInfo * |sed '1d'|awk '{print $1, $2, $3}'|sort -k2,3|grep -v "Broker Name"|awk -F[\#\:] '{print $1,$NF }'
10.191.32.42@12626 PRODRES_CHANGE_OLD_W_E_CARD broker11 0
10.191.32.42@12626 %RETRY%CID_PRODRES_CHANGE_OLD_W_E_CA broker11 0
10.191.32.42@12626 PRODRES_CHANGE_OLD_W_E_CARD broker11 1
10.191.32.43@15595 PRODRES_CHANGE_OLD_W_E_CARD broker11 2
10.191.32.44@11632 PRODRES_CHANGE_OLD_W_E_CARD broker11 3
10.191.32.50@16649 PRODRES_CHANGE_OLD_W_E_CARD broker11 4
10.191.32.51@11868 PRODRES_CHANGE_OLD_W_E_CARD broker11 5
10.191.32.52@14362 PRODRES_CHANGE_OLD_W_E_CARD broker11 6
10.191.32.53@24030 PRODRES_CHANGE_OLD_W_E_CARD broker11 7
10.191.32.43@15595 %RETRY%CID_PRODRES_CHANGE_OLD_W_E_CA broker12 0
10.191.6.122@6276 PRODRES_CHANGE_OLD_W_E_CARD broker12 0
10.191.6.123@29201 PRODRES_CHANGE_OLD_W_E_CARD broker12 1
10.191.6.124@16279 PRODRES_CHANGE_OLD_W_E_CARD broker12 2
10.191.6.125@3824 PRODRES_CHANGE_OLD_W_E_CARD broker12 3
10.191.6.126@15109 PRODRES_CHANGE_OLD_W_E_CARD broker12 4
10.191.6.127@15336 PRODRES_CHANGE_OLD_W_E_CARD broker12 5
10.191.6.128@15048 PRODRES_CHANGE_OLD_W_E_CARD broker12 6
10.191.6.129@23882 PRODRES_CHANGE_OLD_W_E_CARD broker12 7
重置消息位点
sh mqadmin resetOffsetByTime -g 'CID名称' -s now -t 'TOPIC名称'
如果已经进入retry队列，则需要通过以下命令，在broker或nameserv所在主机使用如下命令重置消费位点
sh mqadmin resetOffsetByTime -g 'CID名称' -s now -f true -t 'TOPIC名称'
