万事有商量，论沟通的重要性 (NFS篇)
2019-05-08 本文已影响0人
hynoor
主流的网络存储协议中，几乎都有一部叫做协商 (Negotiate), 其重要作用就是把连接双方的各自的情况告知对方，以便找出一个双方都满意的方式来传输数据，这好比男女结婚前会谈一阵子朋友，双方对彼此了解以后，才能更为和谐的相处，不然就会出现婚姻的危机。
主流的NAS网络存储协议在我看来就是NFS和SMB(CIFS), 双方的协商过程都叫做Negotiate, 这篇文章的目的就是要‘偷听’这两种协议分别在‘谈朋友’的过程中的悄悄话。
先说NFS, 以最新的4.1版本为例， 首先要确定协商(Negotiate)发生在哪个阶段，通过RFC知道，协商一般会在mount 的时候（还有就是在NFS服务器重启后），
二话不说，开始mount抓包吧
[root@centos ~] tshark -t ud -Y "ip.addr==<server_ip>" -r mount-nfs4-1.pcap >>mount-nfs4-1.txt; vim mount-nfs4-1.txt
871 11:52:29.533571 client -> server TCP 74 881 → 2049 [SYN] Seq=0 Win=26880 Len=0 MSS=8960 SACK_PERM=1 TSval=1968097466 TSecr=0 WS=128
872 11:52:29.558365 server -> client TCP 74 2049 → 881 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1686911269 TSecr=1968097466 WS=128
873 11:52:29.558557 client -> server TCP 66 881 → 2049 [ACK] Seq=1 Ack=1 Win=26880 Len=0 TSval=1968097491 TSecr=1686911269
874 11:52:29.558608 client -> server NFS 110 V4 NULL Call
876 11:52:29.585548 server -> client TCP 66 2049 → 881 [ACK] Seq=1 Ack=45 Win=29056 Len=0 TSval=1686911276 TSecr=1968097491
877 11:52:29.585626 server -> client NFS 94 V4 NULL Reply (Call In 874)
878 11:52:29.585657 client -> server TCP 66 881 → 2049 [ACK] Seq=45 Ack=29 Win=26880 Len=0 TSval=1968097518 TSecr=1686911276
879 11:52:29.585978 client -> server NFS 362 V4 Call EXCHANGE_ID
882 11:52:29.610601 server -> client NFS 238 V4 Reply (Call In 879) EXCHANGE_ID
883 11:52:29.610929 client -> server NFS 322 V4 Call CREATE_SESSION
884 11:52:29.634930 server -> client NFS 194 V4 Reply (Call In 883) CREATE_SESSION
885 11:52:29.635147 client -> server NFS 222 V4 Call RECLAIM_COMPLETE
886 11:52:29.659248 server -> client NFS 158 V4 Reply (Call In 885) RECLAIM_COMPLETE
887 11:52:29.659499 client -> server NFS 230 V4 Call SECINFO_NO_NAME
888 11:52:29.683580 server -> client NFS 258 V4 Reply (Call In 887) SECINFO_NO_NAME
889 11:52:29.690901 client -> server NFS 242 V4 Call PUTROOTFH | GETATTR
898 11:52:29.715565 server -> client NFS 378 V4 Reply (Call In 889) PUTROOTFH | GETATTR
899 11:52:29.715809 client -> server NFS 302 V4 Call GETATTR FH: 0x12166020
900 11:52:29.739614 server -> client NFS 234 V4 Reply (Call In 899) GETATTR
先看summary, #871 - #873 很明显是TCP层的三次握手(顺便说一句，NFS 从version4 开始支持吃TCP的传输方式)， 接下来是一个NFS NULL的call，用具检测NFS服务是否在线，接下来便要开始协商和建立会话了, 这期间主要会经历4步
- Frame #879 EXCHANGE_ID
- Frame #883 CREATE_SESSION
- Frame #885 RECLAIM_COMPLETE
- Frame #887 SECINFO_NO_NAME
- Frame #889 PUTROOTFH
EXCHANGE_ID
这一步就是告知对方自己的名字，就行连个刚见面的商人互相交换名片一样，我们展开这一帧开仔细看看他们之间说了什么：
Network File System, Ops(1): EXCHANGE_ID
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 1
Operations (count: 1): EXCHANGE_ID
Opcode: EXCHANGE_ID (42)
eia_clientowner
verifier: 0x5cc7bb2c294e6346 # 提供给服务器一个verifier，用于服务器生成Client ID
Data: <DATA>
length: 48
contents: <DATA>
flags: 0x00000101, EXCHGID4_FLAG_BIND_PRINC_STATEID, EXCHGID4_FLAG_SUPP_MOVED_REFER
0... .... .... .... .... .... .... .... = EXCHGID4_FLAG_CONFIRMED_R: Not set
.0.. .... .... .... .... .... .... .... = EXCHGID4_FLAG_UPD_CONFIRMED_REC_A: Not set
.... .... .... .0.. .... .... .... .... = EXCHGID4_FLAG_USE_PNFS_DS: Not set
.... .... .... ..0. .... .... .... .... = EXCHGID4_FLAG_USE_PNFS_MDS: Not set
.... .... .... ...0 .... .... .... .... = EXCHGID4_FLAG_USE_NON_PNFS: Not set
.... .... .... .... .... ...1 .... .... = EXCHGID4_FLAG_BIND_PRINC_STATEID: Set
.... .... .... .... .... .... .... ..0. = EXCHGID4_FLAG_SUPP_MOVED_MIGR: Not set
.... .... .... .... .... .... .... ...1 = EXCHGID4_FLAG_SUPP_MOVED_REFER: Set
eia_state_protect: SP4_NONE (0)
eia_client_impl_id
Implementor DNS domain name(nii_domain): kernel.org
length: 10
contents: kernel.org
fill bytes: opaque data
Implementation product name(nii_name): Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64
length: 75
contents: Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64
fill bytes: opaque data
Build timestamp(nii_date)
seconds: 0
nseconds: 0
[Main Opcode: EXCHANGE_ID (42)]
可以看到这一步客户端会告诉服务器一串验证码（verifier），自己的Linux 内核版本，自己的DNS信息等等等，而服务器收到后会告知其为此客户端分配的Client ID 这个信息会存在服务器作为此客户端在服务器的唯一标识符，与客户端提供的verifier关联在一起
Network File System, Ops(1): EXCHANGE_ID
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4_OK (0)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 1)
Opcode: EXCHANGE_ID (42)
Status: NFS4_OK (0)
clientid: 0x5c6a7d06a9000080
seqid: 0x00000001
flags: 0x00020000, EXCHGID4_FLAG_USE_PNFS_MDS
0... .... .... .... .... .... .... .... = EXCHGID4_FLAG_CONFIRMED_R: Not set
.0.. .... .... .... .... .... .... .... = EXCHGID4_FLAG_UPD_CONFIRMED_REC_A: Not set
.... .... .... .0.. .... .... .... .... = EXCHGID4_FLAG_USE_PNFS_DS: Not set
.... .... .... ..1. .... .... .... .... = EXCHGID4_FLAG_USE_PNFS_MDS: Set
.... .... .... ...0 .... .... .... .... = EXCHGID4_FLAG_USE_NON_PNFS: Not set
.... .... .... .... .... ...0 .... .... = EXCHGID4_FLAG_BIND_PRINC_STATEID: Not set
.... .... .... .... .... .... .... ..0. = EXCHGID4_FLAG_SUPP_MOVED_MIGR: Not set
.... .... .... .... .... .... .... ...0 = EXCHGID4_FLAG_SUPP_MOVED_REFER: Not set
eia_state_protect: SP4_NONE (0)
eir_server_owner
minor ID: 1
major ID: <DATA>
length: 44
contents: <DATA>
server scope: <DATA>
length: 3
contents: <DATA>
fill bytes: opaque data
eir_server_impl_id
Implementor DNS domain name(nii_domain): <EMPTY>
length: 0
contents: <EMPTY>
Implementation product name(nii_name): cycle_test_src_0
length: 16
contents: cycle_test_src_0
Build timestamp(nii_date)
seconds: 0
nseconds: 0
[Main Opcode: EXCHANGE_ID (42)]
CREATE_SESSION
这一步就是最重要的客户端发起建立会话请求，通过此请求告知会话中使用的BLOCK SIZE等信息，会话一旦建立就会一直保存到umount或者服务器重启为止：
Network File System, Ops(1): CREATE_SESSION
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 1
Operations (count: 1): CREATE_SESSION
Opcode: CREATE_SESSION (43)
clientid: 0x5c6a7d06a9000080
seqid: 0x00000001
csa_flags: 0x00000003, CREATE_SESSION4_FLAG_PERSIST, CREATE_SESSION4_FLAG_CONN_BACK_CHAN
.... .... .... .... .... .... .... ...1 = CREATE_SESSION4_FLAG_PERSIST: Set
.... .... .... .... .... .... .... ..1. = CREATE_SESSION4_FLAG_CONN_BACK_CHAN: Set
.... .... .... .... .... .... .... .0.. = CREATE_SESSION4_FLAG_CONN_RDMA: Not set
csa_fore_chan_attrs
hdr pad size: 0
max req size: 1049620
max resp size: 1049480
max resp size cached: 3416
max ops: 8
max reqs: 64
csa_back_chan_attrs
hdr pad size: 0
max req size: 4096
max resp size: 4096
max resp size cached: 0
max ops: 2
max reqs: 1
cb_program: 0x40000000
flavor: 1
stamp: 693003078
machine name: drmcyc-dengh2-1-00.cec.lab.emc.com
length: 34
contents: drmcyc-dengh2-1-00.cec.lab.emc.com
fill bytes: opaque data
uid: 0
gid: 0
[Main Opcode: CREATE_SESSION (43)]
服务器回复，其中最重要的便是Session ID：04000000067d6a5c7856341204000000
Network File System, Ops(1): CREATE_SESSION
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4_OK (0)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 1)
Opcode: CREATE_SESSION (43)
Status: NFS4_OK (0)
sessionid: 04000000067d6a5c7856341204000000
seqid: 0x00000001
csr_flags: 0x00000002, CREATE_SESSION4_FLAG_CONN_BACK_CHAN
.... .... .... .... .... .... .... ...0 = CREATE_SESSION4_FLAG_PERSIST: Not set
.... .... .... .... .... .... .... ..1. = CREATE_SESSION4_FLAG_CONN_BACK_CHAN: Set
.... .... .... .... .... .... .... .0.. = CREATE_SESSION4_FLAG_CONN_RDMA: Not set
csr_fore_chan_attrs
hdr pad size: 0
max req size: 1049620
max resp size: 1049480
max resp size cached: 3416
max ops: 8
max reqs: 64
csr_back_chan_attrs
hdr pad size: 0
max req size: 4096
max resp size: 4096
max resp size cached: 0
max ops: 2
max reqs: 1
[Main Opcode: CREATE_SESSION (43)]
RECLAIM_COMPLETE
这一步先忽略，以后在高可靠性的文章里再详细的解释，目前忽略掉不影响我们理解。
SECINFO_NO_NAME
客户将不使用任何认证用户访问NFS共享目录。
PUTROOTFH
获取NFS根目录信息，客户端会保存这些信息以便随后的文件访问。
