LeoFS集群部署
1.概述
LeoFS，一个数据湖和网络的存储系统。
LeoFS是一个高度可用的、分布式的、最终一致的对象/ blob存储。如果您正在搜索一个能够以原生格式存储大量和各种原始数据的存储系统，那么LeoFS就适合于此。
LeoFS提供了高性价比的性能比。它允许您在Linux操作系统之上使用商品硬件构建LeoFS集群。即使在商品硬件上，LeoFS也会提供很好的性能。与其他存储相比，LeoFS需要一个较小的集群来实现相同的性能。LeoFS也很容易设置和操作。
由于其在Erlang / OTP功能上的出色设计，LeoFS提供了高可靠性。Erlang / OTP在生产系统中被使用多年，有9个nines(99.9999999%)可用，而LeoFS也不例外。不管软件错误或硬件故障发生在集群内部，一个LeoFS系统都会保持。
要是提供高可伸缩性。添加和删除节点是简单而快捷的，允许您在需要更改时快速反应。一个LeoFS集群可以被认为是弹性存储，你可以尽可能多地伸展它。
LeoFS由三个组件组成:LeoStorage、LeoGateway和LeoManager依赖于Erlang。
在使用rest - api或s3 - api时，LeoGateway可以处理任何客户端的http请求和http响应。此外，它已经构建在对象缓存机制、内存和磁盘缓存中。
LeoStorage可以处理、放置和删除对象以及元数据。此外，它还具有复制器、恢复器和排队机制，以保持运行存储节点并实现最终一致性。
LeoManager总是监视LeoGateway和LeoStorage节点。主要监测状态为节点状态和环形校验和，以实现高可用性和数据一致性。
本次部署采用手动部署，并未通过ansible工具部署。
根据LeoFS特性，部署一个集群，生产环境配置如下：
[manager]
10.x.y.13 M0@10.x.y.13 CPU 2 RAM 1024 DISK 20G master
10.x.y.24 M1@10.x.y.24 CPU 2 RAM 1024 DISK 20G slave
[storage]
10.x.y.14 S0@10.x.y.14 CPU 2 RAM 1024 DISK 100G
10.x.y.31 S1@10.x.y.31 CPU 2 RAM 1024 DISK 100G
10.x.y.49 S2@10.x.y.49 CPU 2 RAM 1024 DISK 100G
10.x.y.29 S3@10.x.y.29 CPU 2 RAM 1024 DISK 100G
10.x.y.15 S4@10.x.y.15 CPU 2 RAM 1024 DISK 100G
[gateway]
10.x.y.30 G0@10.x.y.30 CPU 2 RAM 1024 DISK 20G
10.x.y.16 G1@10.x.y.16 CPU 2 RAM 1024 DISK 20G
官网架构图
2.部署manager
从v1.3.3版本开始，可以将一些配置文件和日志文件指向目录树之外(/usr/local/leofs/<version>)，这样则简化了以后的升级，使升级变得简单，升级到新版本的过程就像放置leo_*.environment文件到新版本etc路径下一样简单，以leo_manager_0为例：
## Stop the process of LeoManager$ /usr/local/leofs/<old_version>/leo_manager_0/bin/leo_manager stop
## Overwrite the environment file
$cp /usr/local/leofs/<old_version>/leo_manager_0/etc/leo_manager.environment
/usr/local/leofs/<new_version>/leo_manager_0/etc/
## Restart the process of LeoManager$ /usr/local/leofs/<new_version>/leo_manager_0/bin/leo_manager start
部署master下载并安装leofs
cd /opt
wget https://github.com/leo-project/leofs/releases/download/1.3.4/leofs-1.3.4-1.erl-18.3.4.5.el7.x86_64.rpm
sudo rpm -ivh leofs-1.3.4-1.erl-18.3.4.5.el7.x86_64.rpm
检查安装
ls -l /usr/local/leofs/
total 4
drwxr-xr-x. 6 root root 4096 Aug 10 10:41 1.3.4
更改environment路径指向
vi /usr/local/leofs/1.3.4/leo_manager_0/etc/leo_manager.environment
# pick config file from fixed place
RUNNER_ETC_DIR=/etc/leofs/leo_manager_0
# store erlang.log.* and run_erl.log in this directory
RUNNER_LOG_DIR=/var/log/leofs/leo_manager_0
RUNNER_ETC_DIR存放配置文件的
RUNNER_LOG_DIR存放日志文件的
退出，保存。
创建配置文件存放路径，更改用户和组，并拷贝配置文件
mkdir -p /etc/leofs/leo_manager_0
cp /usr/local/leofs/1.3.4/leo_manager_0/etc/leo_manager.conf /etc/leofs/leo_manager_0
chown -R leofs:leofs /etc/leofs/
修改配置文件
vi /etc/leofs/leo_manager_0/leo_manager.conf
sasl.sasl_error_log = /var/log/leofs/leo_manager_0/sasl/sasl-error.log
sasl.error_logger_mf_dir = /var/log/leofs/leo_manager_0/sasl
mnesia.dir = /var/local/leofs/leo_manager_0/work/mnesia/10.x.y.13
queue_dir = /var/local/leofs/leo_manager_0/work/queue
log.erlang = /var/log/leofs/leo_manager_0/erlang
log.app = /var/log/leofs/leo_manager_0/app
log.member_dir = /var/log/leofs/leo_manager_0/ring
log.ring_dir = /var/log/leofs/leo_manager_0/ring
erlang.crash_dump = /var/log/leofs/leo_manager_0/erl_crash.dump
## Partner of manager's alias
manager.partner = M1@10.x.y.24
## A number of replicas
consistency.num_of_replicas = 3
## A number of replicas needed for a successful WRITE operation
consistency.write = 2
## A number of replicas needed for a successful READ operation
consistency.read = 1
## A number of replicas needed for a successful DELETE operation
consistency.delete = 2
## Name of the LeoFS's manager node
nodename = M0@10.x.y.13
上面需要注意的配置：
manager.partner：M1@10.x.y.24 // slave的名称
#A number of replicas
consistency.num_of_replicas = 3 // 存储份数，只有leo_manager_0有，启动后不可更改
#A number of replicas needed for a successful WRITE operation
consistency.write = 2 // 写成功数，只有leo_manager_0有，启动后可修改
#A number of replicas needed for a successful READ operation
consistency.read = 1 // 读成功数，只有leo_manager_0有，启动后可修改
#A number of replicas needed for a successful DELETE operation
consistency.delete = 2 // 删除成功数，只有leo_manager_0有，启动后可修改
#Name of the LeoFS's manager node
nodename = M0@10.x.y.13 // 节点名称
创建路径，更改用户和组
mkdir -p /var/log/leofs/leo_manager_0/sasl
mkdir -p /var/local/leofs/leo_manager_0/work
chown -R leofs:leofs /var/log/leofs/
chown -R leofs:leofs /var/local/leofs/
关闭防火墙
systemctl stop firewalld
systemtl disable firewalld
设置开机启动
vi /usr/lib/systemd/system/leofs-manager.service
[Unit]
Description=leofs-manager service
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/leofs/1.3.4/leo_manager_0/bin/leo_manager start
ExecReload=/usr/local/leofs/1.3.4/leo_manager_0/bin/leo_manager restart
ExecStop=/usr/local/leofs/1.3.4/leo_manager_0/bin/leo_manager stop
privateTmp=true
[Install]
WantedBy=multi-user.target
chmod 754 /usr/lib/systemd/system/leofs-manager.service
systemctl enable leofs-manager.service
安装nc工具
yum install -y nc
启动leo_manager_0
systemctl start leofs-manager.service
部署slaveslave部署和manager一样，只是少了如下4个配置项：
#A number of replicas
consistency.num_of_replicas = 3 // 存储份数，只有leo_manager_0有，启动后不可更改
#A number of replicas needed for a successful WRITE operation
consistency.write = 2 // 写成功数，只有leo_manager_0有，启动后可修改
#A number of replicas needed for a successful READ operation
consistency.read = 1 // 读成功数，只有leo_manager_0有，启动后可修改
#A number of replicas needed for a successful DELETE operation
consistency.delete = 2 // 删除成功数，只有leo_manager_0有，启动后可修改
3.部署storage
下载并安装leofs
cd /opt
wget https://github.com/leo-project/leofs/releases/download/1.3.4/leofs-1.3.4-1.erl-18.3.4.5.el7.x86_64.rpm
sudo rpm -ivh leofs-1.3.4-1.erl-18.3.4.5.el7.x86_64.rpm
检查安装
ls -l /usr/local/leofs/
total 4
drwxr-xr-x. 6 root root 4096 Aug 10 10:41 1.3.4
更改environment路径指向
vi /usr/local/leofs/1.3.4/leo_storage/etc/leo_storage.environment
# pick config file from fixed place
RUNNER_ETC_DIR=/etc/leofs/leo_storage
# store erlang.log.* and run_erl.log in this directory
RUNNER_LOG_DIR=/var/log/leofs/leo_storage
RUNNER_ETC_DIR存放配置文件的
RUNNER_LOG_DIR存放日志文件的
退出，保存。
创建配置文件存放路径，更改用户和组，并拷贝配置文件
mkdir -p /etc/leofs/leo_storage
chown -R leofs:leofs /etc/leofs/
修改配置文件
vi /etc/leofs/leo_storage/leo_storage.conf
sasl.sasl_error_log = /var/log/leofs/leo_storage/sasl/sasl-error.log
sasl.error_logger_mf_dir = /var/log/leofs/leo_storage/sasl
obj_containers.path = [/data/avs]
log.erlang = /var/log/leofs/leo_storage/erlang
log.app = /var/log/leofs/leo_storage/app
log.member_dir = /var/log/leofs/leo_storage/ring
log.ring_dir = /var/log/leofs/leo_storage/ring
queue_dir = /var/local/leofs/leo_storage/work/queue
leo_ordning_reda.temp_stacked_dir = "/var/local/leofs/leo_storage/work/ord_reda/"
erlang.crash_dump = /var/log/leofs/leo_storage/erl_crash.dump
## Name of Manager node(s)
managers = [M0@10.x.y.13, M1@10.x.y.24]
nodename = S0@10.x.y.14
以上需要注意的配置：
bj_containers.path = [/data/avs] ## 数据存储路径
managers = [M0@10.x.y.13, M1@10.x.y.24] ## manager服务器
创建路径，更改用户和组
mkdir -p /var/log/leofs/leo_storage/sasl
mkdir -p /var/local/leofs/leo_storage/work
mkdir -p /data/avs
chown -R leofs:leofs /var/log/leofs/
chown -R leofs:leofs /var/local/leofs/
chown -R leofs:leofs /data/avs/
关闭防火墙
systemctl stop firewalld
systemtl disable firewalld
设置开机启动
vi /usr/lib/systemd/system/leofs-storage.service
[Unit]
Description=leofs-storage service
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/leofs/1.3.4/leo_storage/bin/leo_storage start
ExecReload=/usr/local/leofs/1.3.4/leo_storage/bin/leo_storage restart
ExecStop=/usr/local/leofs/1.3.4/leo_storage/bin/leo_storage stop
privateTmp=true
[Install]
WantedBy=multi-user.target
chmod 754 /usr/lib/systemd/system/leofs-storage.service
systemctl enable leofs-storage.service
启动storage
systemctl start leofs-storage.service
其他storage节点同上
4.部署gateway
下载并安装leofs
cd /opt
wget https://github.com/leo-project/leofs/releases/download/1.3.4/leofs-1.3.4-1.erl-18.3.4.5.el7.x86_64.rpm
sudo rpm -ivh leofs-1.3.4-1.erl-18.3.4.5.el7.x86_64.rpm
检查安装
ls -l /usr/local/leofs/
total 4
drwxr-xr-x. 6 root root 4096 Aug 10 10:41 1.3.4
更改environment路径指向
vi /usr/local/leofs/1.3.4/leo_storage/etc/leo_storage.environment
# pick config file from fixed place
RUNNER_ETC_DIR=/etc/leofs/leo_storage
# store erlang.log.* and run_erl.log in this directory
RUNNER_LOG_DIR=/var/log/leofs/leo_storage
RUNNER_ETC_DIR存放配置文件的
RUNNER_LOG_DIR存放日志文件的
退出，保存。
创建配置文件存放路径，更改用户和组，并拷贝配置文件
mkdir -p /etc/leofs/leo_storage
chown -R leofs:leofs /etc/leofs/
修改配置文件
vi /etc/leofs/leo_gateway/leo_gateway.conf
sasl.sasl_error_log = /var/log/leofs/leo_gateway/sasl/sasl-error.log
sasl.error_logger_mf_dir = /var/log/leofs/leo_gateway/sasl
log.erlang = /var/log/leofs/leo_gateway/erlang
log.app = /var/log/leofs/leo_gateway/app
log.member_dir = /var/log/leofs/leo_gateway/ring
log.ring_dir = /var/log/leofs/leo_gateway/ring
cache.cache_disc_dir_data = /var/local/leofs/leo_gateway/cache/
datacache.cache_disc_dir_journal = /var/local/leofs/leo_gateway/cache/
journalqueue_dir = /var/local/leofs/leo_gateway/work/queue
erlang.crash_dump = /var/log/leofs/leo_gateway/erl_crash.dump
managers = [M0@10.x.y.13, M1@10.x.y.24]
nodename = G0@10.x.y.30
managers = [M0@10.x.y.13, M1@10.x.y.24] ## manager服务器
创建路径，更改用户和组
mkdir -p /var/log/leofs/leo_gateway/sasl
mkdir -p /var/local/leofs/leo_gateway/work
mkdir -p /var/local/leofs/leo_gateway/cache
chown -R leofs:leofs /var/log/leofs/
chown -R leofs:leofs /var/local/leofs/
关闭防火墙
systemctl stop firewalld
ystemtl disable firewalld
设置开机启动
vi /usr/lib/systemd/system/leofs-gateway.service
[Unit]
Description=leofs-gateway service
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/leofs/1.3.4/leo_gateway/bin/leo_gateway start
ExecReload=/usr/local/leofs/1.3.4/leo_gateway/bin/leo_gateway restart
ExecStop=/usr/local/leofs/1.3.4/leo_gateway/bin/leo_gateway stop
privateTmp=true
[Install]
WantedBy=multi-user.target
chmod 754 /usr/lib/systemd/system/leofs-gateway.service
systemctl enable leofs-gateway.service
启动gateway
systemctl start leofs-gateway.service
其他gateway节点同上
搭建好的最终状态
leofs-adm status
[System Confiuration]
-----------------------------------+----------
Item | Value
-----------------------------------+----------
Basic/Consistency level
-----------------------------------+----------
system version | 1.3.4
cluster Id | leofs_1
DC Id | dc_1
Total replicas | 3
number of successes of R | 2
number of successes of W | 2
number of successes of D | 2
number of rack-awareness replicas | 0
ring size | 2^128
-----------------------------------+----------
Multi DC replication settings
-----------------------------------+----------
[mdcr] max number of joinable DCs | 2
[mdcr] total replicas per a DC | 1
[mdcr] number of successes of R | 1
[mdcr] number of successes of W | 1
[mdcr] number of successes of D | 1
-----------------------------------+----------
Manager RING hash
-----------------------------------+----------
current ring-hash | 80c13546
previous ring-hash | 6623bd87
-----------------------------------+----------
[State of Node(s)]
-------+--------------------+--------------+----------------+----------------+----------------------------
type | node | state | current ring | prev ring | updated at
-------+--------------------+--------------+----------------+----------------+----------------------------
S | S0@10.x.y.14 | running | 80c13546 | 6623bd87 | 2017-08-22 10:45:09 +0000
S | S1@10.x.y.31 | running | 80c13546 | 6623bd87 | 2017-08-22 10:45:09 +0000
S | S2@10.x.y.49 | running | 80c13546 | 6623bd87 | 2017-08-22 10:45:09 +0000
S | S3@10.x.y.29 | running | 80c13546 | 6623bd87 | 2017-08-22 10:45:09 +0000
S | S4@10.x.y.15 | running | 80c13546 | 6623bd87 | 2017-08-23 02:30:53 +0000
G | G0@10.x.y.30 | running | 80c13546 | 6623bd87 | 2017-08-23 01:32:14 +0000
G | G1@10.x.y.16 | running | 80c13546 | 6623bd87 | 2017-08-23 01:41:56 +0000
-------+--------------------+--------------+----------------+----------------+----------------------------
5. 节点管理
5.1.add
当需要扩大集群的时候，添加节点start一个节点，在manager执行leofs-adm status，可看到该node state为attached，执行leofs-adm rebalance命令，可将该node加入集3群。
5.2.detach
当需要收缩集群的时候，删除节点在manager上执行leofs-adm detach nodename，可看到该node state为detached，执行leofs-adm rebalance命令，可将该node移除集群。
5.3.take over
更换节点1）删除节点2）添加节点3）执行leofs-adm rebalance命令
5.4.susqend
当需要对某个节点进行维护的时候，可挂起节点在manager上执行leofs-adm susqend nodename，可看到该node state为susqend
5.5.resume
当节点维护好后，需重新并入集群在manager上执行leofs-adm resume nodename，可看到该node state为running
参考地址：
简介http://leo-project.net/leofs/docs/index.html
环境配置文件http://leo-project.net/leofs/docs/admin/settings/environment_config/
leofs版本下载https://github.com/leo-project/leofs/releases
集群参数配置http://leo-project.net/leofs/docs/admin/settings/cluster/
节点操作http://leo-project.net/leofs/docs/admin/system_operations/flow/
systemctl使用https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/sect-Managing_Services_with_systemd-Services.html#sect-Managing_Services_with_systemd-Services-List
systemctl配置：http://www.jianshu.com/p/78e3b024fb8e
