CentOS安装GlusterFS+Heketi
2019-05-28 本文已影响13人
老陕西
本文讲介绍CentOS7.5下面搭建GlusterFS集群+Heketi的详细部署步骤。
安装前配置
本实例安装三台GlusterFS集群和一台Gluster client机器。
为了在第一台主机上保留yum rpm包，在其它主机离线安装，需要配置yum保留cache：
sudo sed -i 's/keepcache=0/keepcache=1/g' /etc/yum.conf
安装GlusterFS集群
Gluster的安装包在centos-release-gluster源里面，CentOS7.5默认带的是GlusterFS 3.11版本，默认版本比较低，如果要安装更高的版本需要安装centos-release-gluster源，这个源中有4，5，6版本的Gluster包。
yum search centos-release-gluster
Loaded plugins: fastestmirror, langpacks
Determining fastest mirrors
* base: mirrors.aliyun.com
* extras: mirror.jdcloud.com
* updates: mirrors.nwsuaf.edu.cn
base | 3.6 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/4): base/7/x86_64/group_gz | 166 kB 00:00:00
(2/4): extras/7/x86_64/primary_db | 200 kB 00:00:00
(3/4): base/7/x86_64/primary_db | 6.0 MB 00:00:01
(4/4): updates/7/x86_64/primary_db | 5.0 MB 00:02:07
=================================================================== N/S matched: centos-release-gluster ===================================================================
centos-release-gluster-legacy.noarch : Disable unmaintained Gluster repositories from the CentOS Storage SIG
centos-release-gluster40.x86_64 : Gluster 4.0 (Short Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster41.noarch : Gluster 4.1 (Long Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster5.noarch : Gluster 5 packages from the CentOS Storage SIG repository
centos-release-gluster6.noarch : Gluster 6 packages from the CentOS Storage SIG repository
Name and summary matches only, use "search all" for everything.
Gluster版本如何选型
这个需要查看GusterFS的发布计划：
https://www.gluster.org/community/release-schedule
image.png
从这个计划中我们可以看出，EOL是EndOfLife的意思，就是不在维护和支持了，出现bug也不会修改了，所以一般不需要选择EOL的版本，当前已经发布并且维护的版本是4.1和5，其中4.1已经发布并且较长时间，比较稳定，所以本次选择4.1版本安装。
sudo yum install centos-release-gluster41.noarch -y
然后在第一台机器上安装gluster服务端，注意，要使用我们安装的源需要添加选项--enablerepo=centos-gluster*-test
sudo yum --enablerepo=centos-gluster*-test install glusterfs-server glusterfs-cli glusterfs-geo-replication -y
关闭防火墙
sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service
启动Gluster服务
sudo systemctl start glusterd.service
sudo systemctl enable glusterd.service
第一台Gluster服务器就安装完成，复制yum cache到其它节点：
scp -r /var/cache/yum/ root@centos3:/var/cache/
在其它节点选择从cache安装并启动
sudo yum -C -y install centos-release-gluster41.noarch
sudo yum -C -y --enablerepo=centos-gluster*-test install glusterfs-server glusterfs-cli glusterfs-geo-replication
sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service
sudo systemctl start glusterd.service
sudo systemctl enable glusterd.service
把三个节点的Gluster添加为一个集群
HOST_NAME_LIST="centos1,centos2,centos3"
awk 'BEGIN{split('"\"$HOST_NAME_LIST\""',arr,",");for(i in arr){cmd="sudo gluster peer probe "arr[i];print(cmd);system(cmd)}}'
sudo gluster peer probe centos1
peer probe: success. Probe on localhost not needed
sudo gluster peer probe centos2
peer probe: success.
sudo gluster peer probe centos3
peer probe: success.
[root@centos1 cent]# gluster peer status
Number of Peers: 2
Hostname: centos2
Uuid: 62f6ecbf-fec4-4004-aa5b-c69d2b792a19
State: Peer in Cluster (Connected)
Hostname: centos3
Uuid: dd1e84c4-e9c2-4b0a-8496-55941abb305a
State: Peer in Cluster (Connected)
安装Heketi
Heketi的安装包也是随着GlusterFS发布的，所以必须选择一致的Heketi进行安装，同样，我们从centos-gluster41的源中安装Heketi。
sudo yum --enablerepo=centos-gluster*-test install heketi heketi-client -y
配置Heketi配置文件，配置文件的含义可以去下面链接查看
https://www.cnblogs.com/netonline/p/10288219.html
## /etc/heketi/heketi.json
sudo echo '
{
"_port_comment": "Heketi Server Port Number",
"port": "8080",
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": true,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "a123"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "a123"
}
},
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": [
"Execute plugin. Possible choices: mock, ssh",
"mock: This setting is used for testing and development.",
" It will not send commands to any node.",
"ssh: This setting will notify Heketi to ssh to the nodes.",
" It will need the values in sshexec to be configured.",
"kubernetes: Communicate with GlusterFS containers over",
" Kubernetes exec api."
],
"executor": "ssh",
"_sshexec_comment": "SSH username and private key file information",
"sshexec": {
"keyfile": "/var/lib/heketi/id_rsa",
"user": "root",
"port": "22",
"fstab": "/etc/fstab"
},
"_kubeexec_comment": "Kubernetes configuration",
"kubeexec": {
"host" :"https://kubernetes.host:8443",
"cert" : "/path/to/crt.file",
"insecure": false,
"user": "kubernetes username",
"password": "password for kubernetes user",
"namespace": "OpenShift project or Kubernetes namespace",
"fstab": "Optional: Specify fstab file on node. Default is /etc/fstab"
},
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",
"_loglevel_comment": [
"Set log level. Choices are:",
" none, critical, error, warning, info, debug",
"Default is warning"
],
"_log": "logging to /var/log/message",
"loglevel" : "warning"
}
}
' | sudo tee /etc/heketi/heketi.json
定义每个节点上的存储结构topology
sudo echo '
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"centos1"
],
"storage": [
"centos1"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"centos2"
],
"storage": [
"centos2"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"centos3"
],
"storage": [
"centos3"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
}
]
}
]
}
' | sudo tee /etc/heketi/topology-cluster-1.json
配置SSH免密登录
sudo ssh-keygen -t rsa -q -f /var/lib/heketi/id_rsa -N ''
sudo chown heketi:heketi /var/lib/heketi/id_rsa*
sudo ssh-copy-id -i /var/lib/heketi/id_rsa root@centos1
sudo ssh-copy-id -i /var/lib/heketi/id_rsa root@centos2
sudo ssh-copy-id -i /var/lib/heketi/id_rsa root@centos3
启动Heketi服务
sudo systemctl start heketi.service
sudo systemctl enable heketi.service
sudo systemctl status heketi.service
通过Heketi命令初始化GlusterFS各节点的存储
heketi-cli --user admin --secret a123 --server http://127.0.0.1:8080 topology load --json /etc/heketi/topology-cluster-1.json
Creating cluster ... ID: 919157039f0aef6126dad5934d3fdb66
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node 192.168.43.88 ... ID: 61b13f9ebce7c382ef9bd43215017374
Adding device /dev/sdb ... OK
Creating node 192.168.43.152 ... ID: 0ff9b39ca465aefb777ad2b196a116a5
Adding device /dev/sdb ... OK
这里会生成cluster id，这个后面要用到。
创建一个2G的3副本的复制卷
alias heketi-cli='heketi-cli --server "http://127.0.0.1:8080" --user admin --secret a123'
heketi-cli volume create --size=2 --replica=3 --clusters=919157039f0aef6126dad5934d3fdb66
Name: vol_a34cffc6832123b9ffd795744c3eb2df
Size: 1
Volume Id: a34cffc6832123b9ffd795744c3eb2df
Cluster Id: 919157039f0aef6126dad5934d3fdb66
Mount: 192.168.43.152:vol_a34cffc6832123b9ffd795744c3eb2df
Mount Options: backup-volfile-servers=192.168.43.88
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distribute Count: 1
Replica Count: 3
Heketi会为每个卷起一个uuid的名字，可以通过heketi-cli volume list查看。
使用gluster命令也可以create volume，但是默认没有start，需要用gluster start命令启动volume.
使用Gluster FS存储
如果需要通过native方式使用GlusterFS，则需要在客户端机器上安装glusterfs-client包
sudo yum install centos-release-gluster41.noarch -y
sudo yum --enablerepo=centos-gluster*-test install glusterfs-client -y
通过mount命令就可以挂在GlusterFS的存储卷
sudo mount -t glusterfs centos1:vol_a34cffc6832123b9ffd795744c3eb2df /gfsmnt/
