ceph 分布式存储-文件存储(CephFS)搭建
2018-12-06 本文已影响7人
lihanglucien
1. 创建元数据服务器
1.1 安装mds
PG数量的预估 集群中单个池的PG数计算公式如下：PG 总数 = (OSD 数 * 100) / 最大副本数 / 池数 (结果必须舍入到最接近2的N次幂的值)
#ceph-deploy install mds_server
$ ceph-deploy install ceph-xxx-osd03.gz01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy install ceph-xxx-osd03.gz01
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] testing : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xcaa830>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] dev_commit : None
[ceph_deploy.cli][INFO ] install_mds : False
[ceph_deploy.cli][INFO ] stable : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] adjust_repos : True
[ceph_deploy.cli][INFO ] func : <function install at 0xc22050>
[ceph_deploy.cli][INFO ] install_mgr : False
[ceph_deploy.cli][INFO ] install_all : False
[ceph_deploy.cli][INFO ] repo : False
[ceph_deploy.cli][INFO ] host : ['ceph-xxx-osd03.gz01']
[ceph_deploy.cli][INFO ] install_rgw : False
[ceph_deploy.cli][INFO ] install_tests : False
[ceph_deploy.cli][INFO ] repo_url : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] install_osd : False
[ceph_deploy.cli][INFO ] version_kind : stable
[ceph_deploy.cli][INFO ] install_common : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] dev : master
[ceph_deploy.cli][INFO ] nogpgcheck : False
[ceph_deploy.cli][INFO ] local_mirror : None
[ceph_deploy.cli][INFO ] release : None
[ceph_deploy.cli][INFO ] install_mon : False
[ceph_deploy.cli][INFO ] gpg_url : None
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts ceph-xxx-osd03.gz01
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-xxx-osd03.gz01 ...
[ceph-xxx-osd03.gz01][DEBUG ] connected to host: ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] detect platform information from remote host
[ceph-xxx-osd03.gz01][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: CentOS Linux 7.2.1511 Core
[ceph-xxx-osd03.gz01][INFO ] installing Ceph on ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][INFO ] Running command: yum clean all
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository ceph is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Cleaning repos: base didi_update epel extras tmprepo updates
[ceph-xxx-osd03.gz01][DEBUG ] Cleaning up everything
[ceph-xxx-osd03.gz01][INFO ] Running command: yum -y install epel-release
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository ceph is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Determining fastest mirrors
[ceph-xxx-osd03.gz01][DEBUG ] 132 packages excluded due to repository priority protections
[ceph-xxx-osd03.gz01][DEBUG ] Package matching epel-release-7-9.noarch already installed. Checking for update.
[ceph-xxx-osd03.gz01][DEBUG ] Nothing to do
[ceph-xxx-osd03.gz01][INFO ] Running command: yum -y install yum-plugin-priorities
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository ceph is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Loading mirror speeds from cached hostfile
[ceph-xxx-osd03.gz01][DEBUG ] 132 packages excluded due to repository priority protections
[ceph-xxx-osd03.gz01][DEBUG ] Package yum-plugin-priorities-1.1.31-42.el7.noarch already installed and latest version
[ceph-xxx-osd03.gz01][DEBUG ] Nothing to do
[ceph-xxx-osd03.gz01][DEBUG ] Configure Yum priorities to include obsoletes
[ceph-xxx-osd03.gz01][WARNIN] check_obsoletes has been enabled for Yum priorities plugin
[ceph-xxx-osd03.gz01][INFO ] Running command: rpm --import https://download.ceph.com/keys/release.asc
[ceph-xxx-osd03.gz01][INFO ] Running command: yum remove -y ceph-release
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository ceph is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Resolving Dependencies
[ceph-xxx-osd03.gz01][DEBUG ] --> Running transaction check
[ceph-xxx-osd03.gz01][DEBUG ] ---> Package ceph-release.noarch 0:1-1.el7 will be erased
[ceph-xxx-osd03.gz01][DEBUG ] --> Finished Dependency Resolution
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Dependencies Resolved
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Package Arch Version Repository Size
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Removing:
[ceph-xxx-osd03.gz01][DEBUG ] ceph-release noarch 1-1.el7 installed 535
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Transaction Summary
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Remove 1 Package
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Installed size: 535
[ceph-xxx-osd03.gz01][DEBUG ] Downloading packages:
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction check
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction test
[ceph-xxx-osd03.gz01][DEBUG ] Transaction test succeeded
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction
[ceph-xxx-osd03.gz01][DEBUG ] Erasing : ceph-release-1-1.el7.noarch 1/1
[ceph-xxx-osd03.gz01][DEBUG ] warning: /etc/yum.repos.d/ceph.repo saved as /etc/yum.repos.d/ceph.repo.rpmsave
[ceph-xxx-osd03.gz01][DEBUG ] Verifying : ceph-release-1-1.el7.noarch 1/1
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Removed:
[ceph-xxx-osd03.gz01][DEBUG ] ceph-release.noarch 0:1-1.el7
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Complete!
[ceph-xxx-osd03.gz01][INFO ] Running command: yum install -y https://download.ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Examining /var/tmp/yum-root-Y2Avp6/ceph-release-1-0.el7.noarch.rpm: ceph-release-1-1.el7.noarch
[ceph-xxx-osd03.gz01][DEBUG ] Marking /var/tmp/yum-root-Y2Avp6/ceph-release-1-0.el7.noarch.rpm to be installed
[ceph-xxx-osd03.gz01][DEBUG ] Resolving Dependencies
[ceph-xxx-osd03.gz01][DEBUG ] --> Running transaction check
[ceph-xxx-osd03.gz01][DEBUG ] ---> Package ceph-release.noarch 0:1-1.el7 will be installed
[ceph-xxx-osd03.gz01][DEBUG ] --> Finished Dependency Resolution
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Dependencies Resolved
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Package Arch Version Repository Size
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Installing:
[ceph-xxx-osd03.gz01][DEBUG ] ceph-release noarch 1-1.el7 /ceph-release-1-0.el7.noarch 535
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Transaction Summary
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Install 1 Package
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Total size: 535
[ceph-xxx-osd03.gz01][DEBUG ] Installed size: 535
[ceph-xxx-osd03.gz01][DEBUG ] Downloading packages:
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction check
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction test
[ceph-xxx-osd03.gz01][DEBUG ] Transaction test succeeded
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction
[ceph-xxx-osd03.gz01][DEBUG ] Installing : ceph-release-1-1.el7.noarch 1/1
[ceph-xxx-osd03.gz01][DEBUG ] Verifying : ceph-release-1-1.el7.noarch 1/1
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Installed:
[ceph-xxx-osd03.gz01][DEBUG ] ceph-release.noarch 0:1-1.el7
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Complete!
[ceph-xxx-osd03.gz01][WARNIN] ensuring that /etc/yum.repos.d/ceph.repo contains a high priority
[ceph-xxx-osd03.gz01][WARNIN] altered ceph.repo priorities to contain: priority=1
[ceph-xxx-osd03.gz01][INFO ] Running command: yum -y install ceph ceph-radosgw
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Loading mirror speeds from cached hostfile
[ceph-xxx-osd03.gz01][DEBUG ] 133 packages excluded due to repository priority protections
[ceph-xxx-osd03.gz01][DEBUG ] Package matching 1:ceph-10.2.10-0.el7.x86_64 already installed. Checking for update.
[ceph-xxx-osd03.gz01][DEBUG ] Package matching 1:ceph-radosgw-10.2.10-0.el7.x86_64 already installed. Checking for update.
[ceph-xxx-osd03.gz01][DEBUG ] Nothing to do
[ceph-xxx-osd03.gz01][INFO ] Running command: ceph --version
[ceph-xxx-osd03.gz01][DEBUG ] ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1.2 拷贝ceph配置及秘钥
$ cd /etc/ceph/
$ ceph-deploy admin ceph-xxx-osd03.gz01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy admin ceph-xxx-osd03.gz01
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x2ac9b90>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : ['ceph-xxx-osd03.gz01']
[ceph_deploy.cli][INFO ] func : <function admin at 0x2a24c08>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] connected to host: ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] detect platform information from remote host
[ceph-xxx-osd03.gz01][DEBUG ] detect machine type
[ceph-xxx-osd03.gz01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
1.3 修改秘钥文件权限，保证有可读权限
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
1.4 创建元数据服务器
$ ceph-deploy --overwrite-conf mds create ceph-xxx-osd03.gz01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf mds create ceph-xxx-osd03.gz01
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : True
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xc61518>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mds at 0xc428c0>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] mds : [('ceph-xxx-osd03.gz01', 'ceph-xxx-osd03.gz01')]
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts ceph-xxx-osd03.gz01:ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] connected to host: ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] detect platform information from remote host
[ceph-xxx-osd03.gz01][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.2.1511 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-xxx-osd03.gz01][DEBUG ] create path if it doesn't exist
[ceph-xxx-osd03.gz01][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-xxx-osd03.gz01 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-xxx-osd03.gz01/keyring
[ceph-xxx-osd03.gz01][INFO ] Running command: systemctl enable ceph-mds@ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-xxx-osd03.gz01.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-xxx-osd03.gz01][INFO ] Running command: systemctl start ceph-mds@ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][INFO ] Running command: systemctl enable ceph.target
2. 存储池
2.1 创建存储池数据
#ceph osd pool create cephfs_data <pg_num>
$ ceph osd pool create cephfs_data 128
pool 'cephfs_data' created
2.2 创建存储池元数据
#ceph osd pool create cephfs_metadata <pg_num>
$ ceph osd pool create cephfs_data 128
pool 'cephfs_metadata' created
2.3 查看存储池
$ ceph osd lspools
1 rbd,2 test_data,3 test_metadata,5 test,6 benmark_test,7 .rgw.root,8 default.rgw.control,9 default.rgw.meta,10 default.rgw.log,11 default.rgw.buckets.index,12 web-services,13 test_pool,15 cephfs_data,16 cephfs_metadata
3. 文件系统
3.1 创建文件系统
#ceph fs new <fs_name> cephfs_metadata cephfs_data
$ ceph fs new test_fs cephfs_metadata cephfs_data
new fs with metadata pool 16 and data pool 15
3.2 查看文件系统
$ ceph fs ls
name: test_fs, metadata pool: test_metadata, data pools: [test_data ]
4. mds状态
4.1 查看mds状态
$ ceph mds stat
test_fs-1/1/1 up [test_fs:0]=ceph-bench-osd00=up:active}
说明：
-
e8: e标识epoch，8是epoch号 -
tstfs-1/1/1 up:tstfs是cephfs名字，后面的三个1分别是[mds_map.in/mds_map.up/mds_map.max_mds](http://mds_map.in/mds_map.up/mds_map.max_mds)，up是cephfs状态 -
{[tstfs:0]=mds-daemon-1=up:active}:[tstfs:0]指tstfs的rank 0，mds-daemon-1是服务tstfs的mds daemon name，up:active是cephfs的状态为 up & active - test_fs-1是active的，它的mds daemon为ceph-xxx-osd03.gz01
- 又添加一个新的mds daemon后，它会处于standby状态，若前两个mds daemon出问题，它会顶替上去
5. 内核驱动挂载ceph文件系统
5.1 查看秘钥
$ cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQBpxadZjXLcBxAArU8dYn75aAZYfdGdogYRYg==
5.2 创建目录
$ sudo mkdir /mnt/mycephfs
5.3 挂载文件系统
#指定用户名、密钥
#sudo mount -t {ip-address-of-monitor}:6789 /mycephfs -o name=xxxx,secret=xxxx
$ sudo mount -t ceph 10.1.178.32:6789:/ /mnt/mycephfs -o name=admin,secret=AQBpxadZjXLcBxAArU8dYn75aAZYfdGdogYRYg==
$ df -h
10.1.178.32:6789:/ 201T 1.9T 199T 1% /mnt/mycephfs
6. 用户空间挂载ceph文件系统
6.1 安装ceph-fuse
$ yum install ceph-fuse
6.2 创建目录
$ sudo mkdir /mnt/mycephfs-fuse
6.3 挂载文件系统
#Ceph 存储集群默认要求认证，需指定相应的密钥环文件
#sudo ceph-fuse -k keyring -m {ip-address-of-monitor}:6789 /mycephfs
$ sudo ceph-fuse -k /etc/ceph/ceph.client.admin.keyring -m 10.1.178.32:6789 /mnt/mycephfs-fuse
2017-11-15 14:26:45.323695 7fef1e1e2ec0 -1 init, newargv = 0x7fef29cb68a0 newargc=11ceph-fuse[3300741]:
starting ceph client
$ df -h
ceph-fuse 201T 1.9T 199T 1% /mnt/mycephfs-fuse
