Ceph实践之rbd
image
1.创建image
- 数据存储池保护策略为副本
命令格式：
rbd create {replicated_pool_name}/{image_name} --size {image_size}
或
rbd create {image_name} --size {image_size} --pool {replicated_pool_name}
eg：
rbd create data/image --size=1024M
rbd create image4 --size 1M --pool data
- 数据数据池保护策略为纠删码
ceph Luminous支持image的数据池为纠删码，rbd将imge数据存储在纠删码池中，但image头和元数据仍然需要进入副本存储池（所以需要2个存储池，副本存储池做元数据存储池）
命令格式：
rbd create {replicated_pool}/{image_name} --size {image_size} --data-pool {erasure_pool}
或
rbd create {image_name} --pool {replicated_pool} --size {image_size} --data-pool {image_name}
eg:(data为副本，ec为纠删码)
rbd create data/image_ec --size 1024 --data-pool ec
或
rbd create image_ec1 --pool data --size 1024 --data-pool ec
在后续的一次实践中发现使用上述命令遇到创建image失败，报错形如：librbd::image::CreateRequest: 0x564b123bcef0 handle_validate_overwrite: pool missing required overwrite support
rbd: create error: (22) Invalid argument
研究后发现，Luminous支持image的数据池为纠删码，必须设置纠删码存储池的allow_ec_overwrites属性为true
问题解决如下：
[root@node81 test]#
[root@node81 test]# rbd create rbd_meta/image_ec --size 1024 --data-pool rbd_data
2018-12-12 02:00:50.926325 7f7033fff700 -1 librbd::image::CreateRequest: 0x564b123bcef0 handle_validate_overwrite: pool missing required overwrite support
rbd: create error: (22) Invalid argument
[root@node81 test]#
[root@node81 test]#
[root@node81 test]# ceph osd pool set rbd_data allow_ec_overwrites true
set pool 4 allow_ec_overwrites to true
[root@node81 test]#
[root@node81 test]#
[root@node81 test]# rbd create rbd_meta/image_ec --size 1024 --data-pool rbd_data
[root@node81 test]#
[root@node81 test]#
[root@node81 test]# rbd info rbd_meta/image_ec
rbd image 'image_ec':
size 1GiB in 256 objects
order 22 (4MiB objects)
data_pool: rbd_data
block_name_prefix: rbd_data.3.1edeb6b8b4567
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
flags:
create_timestamp: Wed Dec 12 02:02:03 2018
[root@node81 test]#
2.查看image信息
- 查看某个存储池下所有image信息，可选参数-l会例举一些扩展信息（快照、克隆、大小等）
命令格式：
rbd ls [ -l ] {pool_name}
eg:
rbd ls -l data
输出形如：
[root@node81 ~]# rbd ls -l data
NAME SIZE PARENT FMT PROT LOCK
clone1 1GiB 2
image 1GiB 2
image@snap 1GiB 2
image2 1GiB 2
image3 1022MiB 2 excl
image3@1 1022MiB 2
image4 1MiB 2
image_ec 1GiB 2
image_ec1 1GiB 2
[root@node81 ~]#
- 查看某个image详情(大小)
命令格式：
rbd info {pool_name}/{image_name}
eg:
rbd info data/image2
输出形如：
[root@node81 ~]# rbd info data/image2
rbd image 'image2':
size 1GiB in 256 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.647b6b8b4567
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
create_timestamp: Wed Nov 14 02:54:50 2018
[root@node81 ~]#
[root@node81 ~]#
3.删除image
命令形式：
rbd rm {pool_name}/{image_name}
例如：
rbd rm data/image3
注意：
如果该image还有快照信息，删除image会失败，形如：
[root@node81 ~]# rbd rm data/image3
2018-11-15 02:08:27.845097 7f8dd2ffd700 -1 librbd::image::RemoveRequest: 0x55f34ff35090 check_image_snaps: image has snapshots - not removing
Removing image: 0% complete...failed.
rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed.
[root@node81 ~]#
快照
快照（snapshot）是镜像（image）在某个时间点只读副本，可在以后用作恢复
1.创建快照
命令格式：
rbd snap create {pool_name}/{image_name}@{snapshot_name}
eg:
为data存储池下的image镜像创建一个名为snap的快照
rbd snap create data/image@snap
2.查看快照信息
可通过rbd info查看快照的包括大小、创建时间等信息
命令格式：
rbd info {pool_name}/{image_name}@{snapshot_name}
eg:
rbd info data/image@snap
输出形如：
[root@node81 ~]# rbd info data/image@snap
rbd image 'image':
size 1GiB in 256 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.63b06b8b4567
format: 2
features: layering, exclusive-lock
flags:
create_timestamp: Wed Nov 14 02:54:18 2018
protected: False
[root@node81 ~]#
3.删除快照
命令形式:
rbd snap rm {pool_name}/{image_name}@{snapshot_name}
例如：
rbd snap rm data/image3@1
输出形如：
[root@node81 ~]# rbd snap rm data/image3@1
Removing snap: 100% complete...done.
[root@node81 ~]#
注意：如果一个快照存在子依赖关系，删除快照会失败，形如：
[root@node81 ~]# rbd children data/image@snap
data/image_clone
[root@node81 ~]#
[root@node81 ~]# rbd snap rm data/image@snap
Removing snap: 0% complete...failed.
rbd: snapshot 'snap' is protected from removal.
2018-11-15 02:19:11.942321 7fd574251d40 -1 librbd::Operations: snapshot is protected
[root@node81 ~]#
[root@node81 ~]# rbd children data/image@snap
data/image_clone
[root@node81 ~]#
[root@node81 ~]# rbd snap rm data/image@snap
Removing snap: 0% complete...failed.
rbd: snapshot 'snap' is protected from removal.
2018-11-15 02:19:11.942321 7fd574251d40 -1 librbd::Operations: snapshot is protected
[root@node81 ~]#
[root@node81 ~]# rbd snap unprotect data/image@snap
2018-11-15 02:21:57.264547 7fb0c7fff700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [1da406b8b4567] in pool 'data'
2018-11-15 02:21:57.264869 7fb0c7fff700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
2018-11-15 02:21:57.264899 7fb0c7fff700 -1 librbd::SnapshotUnprotectRequest: 0x55e25d19e7f0 should_complete_error: ret_val=-16
2018-11-15 02:21:57.293915 7fb0c7fff700 -1 librbd::SnapshotUnprotectRequest: 0x55e25d19e7f0 should_complete_error: ret_val=-16
rbd: unprotecting snap failed: (16) Device or resource busy
[root@node81 ~]#
[root@node81 ~]# rbd flatten data/image_clone
Image flatten: 100% complete...done.
[root@node81 ~]# rbd snap rm data/image@snap
Removing snap: 0% complete...failed.
rbd: snapshot 'snap' is protected from removal.
2018-11-15 02:22:51.594912 7f5ae715cd40 -1 librbd::Operations: snapshot is protected
[root@node81 ~]#
[root@node81 ~]# rbd snap unprotect data/image@snap
[root@node81 ~]#
[root@node81 ~]#
[root@node81 ~]# rbd snap rm data/image@snap
Removing snap: 100% complete...done.
[root@node81 ~]#
4.回滚快照
命令形式：
rbd snap rollback {pool_name}/{image_name}@{snapshot_name}
例如：
rbd snap rollback data/image@snap
输出：
[root@node81 ~]# rbd snap rollback data/image@snap
Rolling back to snapshot: 100% complete...done.
[root@node81 ~]#
- 记实践过程中回滚快照不成功情况
问题出现步骤：创建镜像->映射->格式化为xfs->挂载文件系统->写入数据->创建快照->回滚快照
回滚快照时出现如下报错：
[root@node81 h]# rbd snap rollback data/image@snap_volume1
Rolling back to snapshot: 0% complete...failed.
rbd: rollback failed: (30) Read-only file system
[root@node81 h]#
出现这种问题的原因是在映射镜像时exclusive-lock（独占锁）特性的问题导致的
克隆
1.创建克隆实践
在创建克隆之前，必须将快照保护起来（快照默认是没有保护的状态），否则会报错，如：
[root@node81 ~]# rbd clone data/image@snap data/image_clone
2018-11-14 22:52:40.996878 7f6505f35d40 -1 librbd::image::CloneRequest: parent snapshot must be protected
rbd: clone error: (22) Invalid argument
[root@node81 ~]#
保护快照
命令：
rbd protect {pool_name}/{image_name}@{snap_name}
例如：
rbd snap protect data/image@snap
输出：
[root@node81 ~]# rbd snap protect data/image@snap
[root@node81 ~]# rbd info data/image@snap
rbd image 'image':
size 1GiB in 256 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.63b06b8b4567
format: 2
features: layering, exclusive-lock
flags:
create_timestamp: Wed Nov 14 02:54:18 2018
protected: True //默认为False
[root@node81 ~]#
[root@node81 ~]#
创建克隆
ceph Luminous支持image的数据池为纠删码，rbd将imge数据存储在纠删码池中，但image头和元数据仍然需要进入副本存储池（所以需要2个存储池，副本存储池做元数据存储池）
--克隆目标是副本存储池
命令：rbd clone {pool_name}/{image_name}@{snapshot_name} {replicated_pool}/{dest_image_name}
例如：
rbd clone data/image@snap data/image_clone
输出：
[root@node81 ~]# rbd clone data/image@snap data/image_clone
[root@node81 ~]# rbd ls -l data
NAME SIZE PARENT FMT PROT LOCK
clone1 1GiB 2
image 1GiB 2
image@snap 1GiB 2 yes
image2 1GiB 2
image3 1022MiB 2 excl
image3@1 1022MiB 2
image4 1MiB 2
image_clone 1GiB data/image@snap 2
image_ec 1GiB 2
image_ec1 1GiB 2
[root@node81 ~]#
--克隆目标是纠删码存储池
命令格式：
rbd clone {replicate_pool}/{image_name}@{snapshot_name} {replicated_pool}/{dest_image} --data-pool {erasure_pool}
例如：
data为副本存储池，ec为纠删码存储池
rbd clone data/image@snap data/image_clone1 --data-pool ec
输出：
[root@node81 ~]# rbd clone data/image@snap data/image_clone1 --data-pool ec
[root@node81 ~]# rbd ls -l data
NAME SIZE PARENT FMT PROT LOCK
clone1 1GiB 2
image 1GiB 2
image@snap 1GiB 2 yes
image2 1GiB 2
image3 1022MiB 2 excl
image3@1 1022MiB 2
image4 1MiB 2
image_clone 1GiB data/image@snap 2
image_clone1 1GiB data/image@snap 2
image_ec 1GiB 2
image_ec1 1GiB 2
[root@node81 ~]#
依赖问题
从一个快照创建克隆后，生成的克隆卷会依赖快照的信息，查看一个快照的"孩子"：
命令形式：
rbd children {pool_name}/{image_name}@{snapshot_name}
例如：
rbd children data/image@snap
输出形如：
[root@node81 ~]# rbd children data/image@snap
data/image_clone
data/image_clone1
[root@node81 ~]#
要想一个克隆卷不再依赖于某个快照（解除这种依赖关系），需要将这个快照扁平化
命令形式：
rbd flatten {pool_name}/{clone_image}
例如：
rbd flatten data/image_clone
扁平化后再来查看，此依赖，如下：
[root@node81 ~]# rbd children data/image@snap
data/image_clone
data/image_clone1
[root@node81 ~]#
[root@node81 ~]#
[root@node81 ~]# rbd flatten data/image_clone
Image flatten: 100% complete...done.
[root@node81 ~]#
[root@node81 ~]# rbd children data/image@snap
data/image_clone1
[root@node81 ~]#
取消快照保护
对快照创建克隆后，建议取消快照的保护，防止下一次保护一个已保护的快照报错，注意克隆一个快照但未解除快照与克隆之间的依赖时去取消一个快照的保护(扁平化)，此操作会失败。
保护一个已保护的快照，失败形如：
[root@node81 ~]# rbd snap protect data/image@snap
rbd: snap is already protected
[root@node81 ~]#
取消快照保护,命令形式：
rbd snap unprotect {pool_name}/{image_name}@{snapshot_name}
例如：
[root@node81 ~]# rbd children data/image@snap
data/clone
[root@node81 ~]# rbd snap unprotect data/image@snap
2018-11-15 00:46:49.348735 7fbbfeffd700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [192aa6b8b4567] in pool 'data'
2018-11-15 00:46:49.348976 7fbbfeffd700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
2018-11-15 00:46:49.349014 7fbbfeffd700 -1 librbd::SnapshotUnprotectRequest: 0x556bb1e5e850 should_complete_error: ret_val=-16
2018-11-15 00:46:49.369881 7fbbfeffd700 -1 librbd::SnapshotUnprotectRequest: 0x556bb1e5e850 should_complete_error: ret_val=-16
rbd: unprotecting snap failed: (16) Device or resource busy
[root@node81 ~]#
[root@node81 ~]# rbd flatten data/clone
Image flatten: 100% complete...done.
[root@node81 ~]#
[root@node81 ~]# rbd snap unprotect data/image@snap
[root@node81 ~]#
映射
- 映射一个镜像
命令形式：
rbd map {pool_name}/{image_name}
例如：
rbd map data/image
在实践的过程中如下错误，映射失败
[root@node81 ~]#
[root@node81 ~]# rbd map data/image1
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable data/image1 object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@node81 ~]#
出现这种错误的原因是系统kernel不支持块设备镜像的一些特性，所以映射失败
禁用掉不支持的特性：
rbd feature disable {pool_name}/{image_name} {some_features}
例如：
[root@node81 ~]# rbd feature disable data/image1 object-map, fast-diff, deep-flatten
[root@node81 ~]#
[root@node81 ~]# rbd info data/image1
rbd image 'image1':
size 10MiB in 3 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.1d7456b8b4567
format: 2
features: layering, exclusive-lock
flags:
create_timestamp: Thu Nov 15 04:47:08 2018
[root@node81 ~]# rbd map data/image1
/dev/rbd2
[root@node81 ~]#
- 查看映射关系
命令：
rbd showmapped
返回形如：
[root@node81 ~]# rbd showmapped
id pool image snap device
0 data image - /dev/rbd0
1 data image1 - /dev/rbd1
[root@node81 ~]#
- 取消映射关系
rbd unmap {pool_name}/{image_name}
返回形如：
[root@node81 ~]# rbd showmapped
id pool image snap device
0 data image - /dev/rbd0
1 data image1 - /dev/rbd1
[root@node81 ~]#
[root@node81 ~]# rbd unmap data/image1
[root@node81 ~]#
[root@node81 ~]# rbd showmapped
id pool image snap device
0 data image - /dev/rbd0
