spdk（bdev层） + fio测试nvme 设备的性能

2018-03-01 本文已影响239人 Stansosleepy

1、背景

spdk ：是一个基于dpdk的存储开发kit，这里主要利用了它提供的用户态nvme driver —— spdk链接
fio: io测试工具，提供丰富的参数，可以构造复杂的io pattern

在spdk中，定义了一层虚拟的block device层，在这一层提供了若干的功能：

A pluggable module API for implementing block devices that interface with different types of block storage devices.
Driver modules for NVMe, malloc (ramdisk), Linux AIO, virtio-scsi, Ceph RBD, and more.
An application API for enumerating and claiming SPDK block devices and then performing operations (read, write, unmap, etc.) on those devices.
Facilities to stack block devices to create complex I/O pipelines, including logical volume management (lvol) and partition support (GPT).
Configuration of block devices via JSON-RPC and a configuration file.
Request queueing, timeout, and reset handling.
Multiple, lockless queues for sending I/O to block devices.

spdk中提供了一个bdev的应用，将这一层的api打包成一个fio的ioengine，供fio进行测试

2、使用步骤

2.1下载dpdk、spdk、fio并解压

用最新的版本即可

2.2编译fio

cd fio_dir
./configure
make&&make install

2.3编译dpdk

vim <dpdk_dir>/config/defconfig_x86_64-native-linuxapp-gcc
#增加一行
EXTRA_CFLAGS=-fPIC
#回到<dpdk_dir>编译dpdk
make install T=x86_64-native-linuxapp-gcc DESTDIR=.

2.4编译spdk

cd <spdk_dir>
./configure --with-fio=/root/Downloads/fio-fio-3.3/ --with-dpdk=/root/Downloads/dpdk-17.11/x86_64-native-linuxapp-gcc
#修改<spdk_dir>/CONFIG文件
CONFIG_FIO_PLUGIN=y
FIO_SOURCE_DIR=fio的目录
#编译spdk
make DPDK_DIR=/root/Downloads/dpdk-17.11/x86_64-native-linuxapp-gcc

2.5unbind nvme driver替换为vfio

cd <spdk_dir>/scripts
sh setup.sh

2.6 配置bdev的conf文件

这个conf文件会设置一个虚拟块设备的各种参数，在spdk中<spdk_dir>/etc/spdk/中有几个例子，主要看nvme部分

[Nvme]
# NVMe Device Whitelist
# Users may specify which NVMe devices to claim by their transport id.
# See spdk_nvme_transport_id_parse() in spdk/nvme.h for the correct format.
# The second argument is the assigned name, which can be referenced from
# other sections in the configuration file. For NVMe devices, a namespace
# is automatically appended to each name in the format <YourName>nY, where
# Y is the NSID (starts at 1).
#将某个pcie总线上的设备“定义”成一个块设备Nvme0，这里的总线地址可以用lspci查询
TransportId "trtype:PCIe traddr:0000:00:00.0" Nvme0
TransportId "trtype:PCIe traddr:0000:01:00.0" Nvme1
# The number of attempts per I/O when an I/O fails. Do not include
# this key to get the default behavior.
RetryCount 4
# Timeout for each command, in seconds. If 0, don't track timeouts.
Timeout 0
# Action to take on command time out. Only valid when Timeout is greater
# than 0. This may be 'Reset' to reset the controller, 'Abort' to abort
# the command, or 'None' to just print a message but do nothing.
# Admin command timeouts will always result in a reset.
ActionOnTimeout None
# Set how often the admin queue is polled for asynchronous events.
# Units in microseconds.
AdminPollRate 100000
# Disable handling of hotplug (runtime insert and remove) events,
# users can set to Yes if want to enable it.
# Default: No
HotplugEnable No

2.7 编辑fio的jobfile，有几个特别的要求

ioengine=spdk_bdev
spdk_conf=path/to/bdev.conf 引入上一步中bdev设备的配置文件
filename就是上一个配置文件中对应某个pci设备的那个nvme设备名

[global]
ioengine=spdk_bdev
spdk_conf=/root/Downloads/spdk-18.01/examples/bdev/fio_plugin/bdev.conf.in
thread=1
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=60
iodepth=128
rw=randread
bs=4k
[test]
numjobs=1
filename=Nvme0n1

2.8使用fio执行测试

LD_PRELOAD=/root/Downloads/spdk-18.01/examples/bdev/fio_plugin/fio_plugin /root/Downloads/fio-fio-3.3/fio example_config.fio

3、注意

spdk做性能测试时，对每个namespace会绑定一个lcore，所以fio的thread只能等于1
fio测试random的io时，需要设置norandommap=1 ，防止fio的random map影响性能