思科N9K交换机配置BGP-EVPN
2019-04-02 本文已影响0人
ljyfree
测试环境
topo描述
- 两台N9K(A和B)的eth1/2互联
- A的eth1/48连接服务器的端口配置IP 20.1.1.2/24
- B的eth1/48连接服务器的端口配置IP 20.1.1.3/24
配置
- 配置要点
- A和B建立IBGP
- Loopback1用来做RP
- Loopback2用来做VTEP和BGP的source-interface
- route-target全部auto，自动生成
- 对称式IRB
- 分布式网关
- A的配置
fabric forwarding anycast-gateway-mac 0002.0002.0002
ip pim rp-address 10.10.10.10 group-list 224.0.0.0/4
ip pim ssm range 239.0.0.0/8
ip pim anycast-rp 10.10.10.10 1.1.1.1
vlan 1-3,11-3000,3900-3901
vlan 200
vn-segment 20000
vlan 3900
name l3-vni-vlan-for-tenant-1
vn-segment 39000
vrf context evpn-tenant-1
vni 39000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
interface Vlan200
no shutdown
vrf member evpn-tenant-1
ip address 20.1.1.1/24
fabric forwarding mode anycast-gateway
interface Vlan3900
no shutdown
vrf member evpn-tenant-1
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback2
member vni 20000
suppress-arp
mcast-group 239.1.1.1
member vni 39000 associate-vrf
interface Ethernet1/2
ip address 12.12.12.1/24
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/48
switchport
switchport access vlan 200
speed 10000
no shutdown
interface loopback1
ip address 10.10.10.10/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
interface loopback2
ip address 1.1.1.1/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
router ospf 100
router-id 1.1.1.1
router bgp 65535
router-id 1.1.1.1
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
neighbor 2.2.2.2
remote-as 65535
update-source loopback2
address-family ipv4 unicast
address-family l2vpn evpn
send-community
send-community extended
vrf evpn-tenant-1
address-family ipv4 unicast
advertise l2vpn evpn
evpn
vni 20000 l2
rd auto
route-target import auto
route-target export auto
- B的配置，基本上就是将1.1.1.1和2.2.2.2进行互换
注意点
- 一定要确保loopback和vlan-if是up的！
检查
- 以A为例，查看相关表项状态和内容
# show nve inter
interface internal
N9K-C93180YC-EX# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: 00f6.63ca.933b
Host Learning Mode: Control-Plane
Source-Interface: loopback2 (primary: 1.1.1.1, secondary: 0.0.0.0)
# show nve peer
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 2.2.2.2 Up CP 00:35:42 003a.9c39.ede7
# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 20000 239.1.1.1 Up CP L2 [200] SA
nve1 39000 n/a Up CP L3 [evpn-tenant-1]
# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 35, Local Router ID is 1.1.1.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:32967 (L2VNI 20000)
*>i[2]:[0]:[0]:[48]:[001b.21ba.a7ef]:[0]:[0.0.0.0]/216
2.2.2.2 100 0 i
*>l[2]:[0]:[0]:[48]:[90e2.ba88.b3cb]:[0]:[0.0.0.0]/216
1.1.1.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[001b.21ba.a7ef]:[32]:[20.1.1.3]/272
2.2.2.2 100 0 i
*>l[2]:[0]:[0]:[48]:[90e2.ba88.b3cb]:[32]:[20.1.1.2]/272
1.1.1.1 100 32768 i
Route Distinguisher: 2.2.2.2:32967
*>i[2]:[0]:[0]:[48]:[001b.21ba.a7ef]:[0]:[0.0.0.0]/216
2.2.2.2 100 0 i
*>i[2]:[0]:[0]:[48]:[001b.21ba.a7ef]:[32]:[20.1.1.3]/272
2.2.2.2 100 0 i
Route Distinguisher: 1.1.1.1:5 (L3VNI 39000)
*>i[2]:[0]:[0]:[48]:[001b.21ba.a7ef]:[32]:[20.1.1.3]/272
2.2.2.2 100 0 i
# show ip arp suppression-cache detail
Flags: + - Adjacencies synced via CFSoE
L - Local Adjacency
R - Remote Adjacency
L2 - Learnt over L2 interface
PS - Added via L2RIB, Peer Sync
RO - Dervied from L2RIB Peer Sync Entry
Ip Address Age Mac Address Vlan Physical-ifindex Flags Remote Vtep Addrs
20.1.1.2 00:15:16 90e2.ba88.b3cb 200 Ethernet1/48 L
20.1.1.3 00:35:06 001b.21ba.a7ef 200 (null) R 2.2.2.2
- 此时从两台服务器，都可以ping通20.1.1.1(分布式网关)，也能ping通彼此
- 两台服务器上看网关20.1.1.1的mac，都是设置的anycast-gateway-mac 00:02:00:02:00:02
# arp -a
? (20.1.1.1) at 00:02:00:02:00:02 [ether] on p2p1
场景验证
将VxLAN报文通过镜像到监控口进行解析
场景一:同租户下同网段通信
- 服务器20.1.1.2和20.1.1.3可以ping通
- tcpdump获取信息
- VxLAN报文的VNI=20000
- ICMP原始报文的source/dest mac就是两台服务器的端口mac
09:58:17.473711 00:3a:9c:39:ed:e7 > 00:f6:63:ca:93:3b, ethertype IPv4 (0x0800), length 148: 2.2.2.2.28364 > 1.1.1.1.4789: VXLAN, flags [I] (0x08), vni 20000
00:1b:21:ba:a7:ef > 90:e2:ba:88:b3:cb, ethertype IPv4 (0x0800), length 98: 20.1.1.3 > 20.1.1.2: ICMP echo request, id 26526, seq 1, length 64
09:58:17.473804 00:f6:63:ca:93:3b > 00:3a:9c:39:ed:e7, ethertype IPv4 (0x0800), length 148: 1.1.1.1.34227 > 2.2.2.2.4789: VXLAN, flags [I] (0x08), vni 20000
90:e2:ba:88:b3:cb > 00:1b:21:ba:a7:ef, ethertype IPv4 (0x0800), length 98: 20.1.1.2 > 20.1.1.3: ICMP echo reply, id 26526, seq 1, length 64
场景2：同租户下不同网段之间通信
- 对交换机B的配置做一些修改
interface Ethernet1/48
switchport
switchport access vlan 201
speed 10000
no shutdown
- 对交换机B的配置做一些增加
vlan 201
vn-segment 20001
interface Vlan201
vrf member evpn-tenant-1
ip address 20.1.2.1/24
fabric forwarding mode anycast-gateway
no shutdown
interface nve1
member vni 20001
suppress-arp
mcast-group 239.1.1.1
evpn
vni 20001 l2
rd auto
route-target import auto
route-target export auto
- 交换机A上的配置与B类似，虽然A上并没有20.1.2.0/24这个网段，但还是要配置vlan201这个网关
- 与交换机A相连的服务器A配置，添加路由
# ip route add 20.1.2.0/24 via 20.1.1.1
- 与交换机B相连的服务器B配置，修改IP，添加路由
# ifconfig p2p1 20.1.2.3/24
# ip route add 20.1.1.0/24 via 20.1.2.1
- 两台服务器可以ping通，抓VxLAN的报文发现
- VxLAN报文的VNI=39000，是本租户的L3-VNI
- ICMP原始报文的source/dest mac已经被替换成两台交换机的route mac
10:23:59.629647 00:3a:9c:39:ed:e7 > 00:f6:63:ca:93:3b, ethertype IPv4 (0x0800), length 148: 2.2.2.2.42646 > 1.1.1.1.4789: VXLAN, flags [I] (0x08), vni 39000
00:3a:9c:39:ed:e7 > 00:f6:63:ca:93:3b, ethertype IPv4 (0x0800), length 98: 20.1.2.3 > 20.1.1.2: ICMP echo request, id 27067, seq 1, length 64
10:23:59.629694 00:f6:63:ca:93:3b > 00:3a:9c:39:ed:e7, ethertype IPv4 (0x0800), length 148: 1.1.1.1.9550 > 2.2.2.2.4789: VXLAN, flags [I] (0x08), vni 39000
00:f6:63:ca:93:3b > 00:3a:9c:39:ed:e7, ethertype IPv4 (0x0800), length 98: 20.1.1.2 > 20.1.2.3: ICMP echo reply, id 27067, seq 1, length 64
- 在交换机A上evpn中可以看到20.1.2.3这台主机和它的mac
# show bgp l2vpn evpn
...
*>i[2]:[0]:[0]:[48]:[001b.21ba.a7ef]:[32]:[20.1.2.3]/272
2.2.2.2 100 0 i
...
不同租户下网络通信
和本主题没有直接关联，实现方法一般是:
- NAT后通过vGW通信，或者
- 服务器在本地通过bridge或是封装成VxLAN来决定是本租户通信还是和其它租户通信
一些心得
专门开一章聊聊聊一些配置细节，包括上面的配置示例中由RT auto隐藏的一些细节
同网段跨交换机主机通信
- 一个租户对应一个VRF
- 配置中需要有接入VLAN，需要
- 关联L2-VNI
- 绑定到VRF
- BGP的VRF下，需要通告evpn路由
- interface VLAN作为分布式网关，配置网关地址
- 接入VLAN的L2-VNI，设置RT
- import RT需要和其它节点的export RT相同
- export RT需要和其它节点的import RT相同
不同网段跨交换机主机通信
- 按照我的理解，这里L3-VNI需要一个VLAN，其实是为了将VNI和VRF联系起来
- VLAN下对应一个L3-VNI
- VLAN有自己的VLAN interface(不需要配置IP)
- VLAN interface是VRF的member
- 感觉“白白”浪费了一个VLAN
- L3-VNI的RT
- import RT需要和其它节点的export RT相同
- export RT需要和其它节点的import RT相同
- 还要把上面原样来一份evpn
- 接入VLAN和L2-VPN的配置和上面相同，但要注意
- 因为是“分布式网关”，即便在某台交换机下没有对应网段的主机，也还是需要将所有的接入VLAN的网关地址配置上
- L2-VNI要有一个export RT和L3-VNI的import RT相同,反之不需要
