IB 网卡安全驱动和虚拟化流程
2022-04-21 本文已影响0人
河码匠
一、安装驱动
1. 安装
tar -zxvf MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu18.04-x86_64.tgz
cd MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu18.04-x86_64
./mlnxofedinstall --force
2. 启动
/etc/init.d/openibd restart
/etc/init.d/opensmd restart
mst start
mst status
二、虚拟化
1. 确认物理机系统是否启动 IOMMU
dmesg | grep -E "DMAR|IOMMU"
2. 修改 grub 启动 amd_iommu=on iommu=pt
vim /etc/default/grub
GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt"
# 更新
update-grub
3. 启动 SR-IOV
mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
4. 查看设备状态
mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4119_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:60:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 00
5. 查看 IB 网卡配置文件
mlxconfig -d /dev/mst/mt4119_pciconf0 q
Device #1:
----------
Device type: ConnectX5
Name: N/A
Description: N/A
Device: /dev/mst/mt4119_pciconf0
Configurations: Next Boot
MEMIC_BAR_SIZE 0
MEMIC_SIZE_LIMIT _256KB(1)
HOST_CHAINING_MODE DISABLED(0)
HOST_CHAINING_DESCRIPTORS Array[0..7]
HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7]
FLEX_PARSER_PROFILE_ENABLE 0
FLEX_IPV4_OVER_VXLAN_PORT 0
ROCE_NEXT_PROTOCOL 254
NON_PREFETCHABLE_PF_BAR False(0)
NUM_PF_MSIX_VALID True(1)
NUM_OF_VFS 16
SRIOV_EN True(1)
......
其中
SRIOV_EN ：启动虚拟化
NUM_OF_VFS： 虚拟化网卡的数量
6. 设置 虚拟化网卡数量
mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=24
7. 重启系统，然后从新启动 mst start
8. 修改 /sys/class/infiniband/mlx5_0/device/sriov_numvfs
echo 16 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs
9. 查看当前 IB 网卡 在 lspci 中数量
lspci -D | grep Mellanox
0000:60:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
10. 查看 IB 网卡当前状态
ibstat
CA 'mlx5_0'
CA type: MT4119
Number of ports: 1
Firmware version: 16.23.1020
Hardware version: 0
Node GUID: 0xb88303ffff95a97c
System image GUID: 0xb88303ffff95a97c
Port 1:
State: Down
Physical state: Disabled
Rate: 10
Base lid: 65535
LMC: 0
SM lid: 0
Capability mask: 0x2651e848
Port GUID: 0xb88303ffff95a97c
Link layer: InfiniBand
11. 查看 IB 网卡固件中各项参数是否正确
- VF 数量
cat /sys/class/net/ib0/device/sriov_totalvfs
16
# 如果您没有看到此参数，则 amd_iommu=on 未添加到 grub 文件中。
- 获取此设备上的当前 VF 数
cat /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs
0
cat /sys/class/net/ib0/device/sriov_numvfs
0
cat /sys/class/net/ib0/device/mlx5_num_vfs
0
12. 再次查看 IB 网卡在 lspci 中数量。可以看到已经虚拟出来的 IB 网卡
lspci -D | grep Mellanox
0000:60:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
0000:60:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.5 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.6 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.7 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.5 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.6 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.7 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:02.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
13. 查看 IB 网卡使用情况
ibdev2netdev -v
0000:60:00.0 mlx5_0 (MT4119 - IL293304JDV0) HPE InfiniBand EDR 100Gb 1-port 841QSFP28 Adapter fw 16.23.1020 port 1 (DOWN ) ==> ib0 (Down)
0000:60:00.1 mlx5_1 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:01.2 mlx5_10 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:01.3 mlx5_11 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:01.4 mlx5_12 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:01.5 mlx5_13 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:01.6 mlx5_14 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:01.7 mlx5_15 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:02.0 mlx5_16 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:00.2 mlx5_2 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:00.3 mlx5_3 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:00.4 mlx5_4 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:00.5 mlx5_5 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:00.6 mlx5_6 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:00.7 mlx5_7 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:01.0 mlx5_8 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
0000:60:01.1 mlx5_9 (MT4120 - NA) fw 16.23.1020 port 1 (DOWN ) ==> ib1 (Down)
