NEUTRON: VROUTER HA机制的实现

2019-12-26 本文已影响0人圣地亚哥_SVIP

neutron vrouter属于单点，为了保障高可用，利用keepalived实现VPC GW及EIP的漂移。本文中，介绍如何利用keepalived实现vrouter的高可用。

neutron-ha

HA配置

vrouter有四类接口：

loopback口
网关： VPC的网关
EIP: floating ip的接口
HA: keepalived track的口，具有独立的VPC网络

HA IP信息:

169.254.192.1/18
169.254.192.2/18
VIP: 169.254.0.1/24

NETWORK2上添加vrouter

在Blog已经完成一个network1上vrouter的创建和测试

首先Down Network1上vrouter的网关及外部接口：

# ip netns exec qrouter-123 ip link set dev qr-123 down
# ip netns exec qrouter-123 ip link set dev qg-123 down

NETWORK2:

添加vrouter及设置外部网络，network2复用之前COMPUTE2：

# ovs-vsctl add-port br-int qr-123 -- set Interface qr-123 type=internal -- set port qr-123 tag=22
# ip netns add qrouter-123
# ip link set dev qr-123 netns qrouter-123
# ip netns exec qrouter-123 ip link set qr-123 up
# ip netns exec qrouter-123 ip addr add 192.168.10.1/24 dev qr-123

外部连接的参数，参考上一篇Blog：

Out Vlan: 1102
Local Vlan: 60
CIDR： 18.18.18.0/24

添加本地br-ex-biz网桥：

# ovs-vsctl add-br br-ex-biz
# ovs-vsctl add-port br-ex-biz ex-biz--int -- set interface ex-biz--int type=patch -- set interface ex-biz--int options:peer=int--ex-biz
# ovs-vsctl add-port br-int int--ex-biz -- set interface int--ex-biz type=patch -- set interface int--ex-biz options:peer=ex-biz--int

添加qg-xxx端口：

# ovs-vsctl add-port br-int qg-123 -- set Interface qg-123 type=internal -- set port qg-123 tag=60
# ip link set dev qg-123 netns qrouter-123
# ip netns exec qrouter-123 ip link set qg-123 up
# ip netns exec qrouter-123 ip addr add 18.18.18.10/24 dev qg-123

流表添加：

# ovs-ofctl add-flow br-ex-biz 'cookie=0x79, table=0, priority=4,in_port=1,dl_vlan=60 actions=mod_vlan_vid:1102,NORMAL'
# ovs-ofctl add-flow br-ex-biz 'cookie=0x79, table=0, priority=2,in_port=1 actions=drop'
# ovs-ofctl add-flow br-ex-biz 'cookie=0x79, table=0, priority=0 actions=NORMAL'
此处in_port为：int--ex-biz
# ovs-ofctl add-flow br-int 'cookie=0x79,table=0, priority=3,in_port=5,dl_vlan=1102 actions=mod_vlan_vid:60,NORMAL'
# ovs-ofctl add-flow br-int 'cookie=0x79,table=0, priority=2,in_port=5 actions=drop'

设置外部端口(此处主要用于测试，实际中外部网络的网关应在交换机中)：

# ip netns add outer
# ovs-vsctl add-port br-ex-biz outer-123 -- set Interface outer-123 type=internal
  
# ip link set dev outer-123 netns outer
# ip netns exec outer ip link set outer-123 up
# ip netns exec outer ip addr add 18.18.18.1/24 dev outer-123
  
# ovs-vsctl set port outer-123 tag=1102

NAT功能添加：

# ip netns exec qrouter-123 iptables -t nat -A POSTROUTING  -o qg-123 -j SNAT --to-source 18.18.18.10
# ip netns exec qrouter-123 iptables -t nat -A POSTROUTING -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 18.18.18.10
# ip netns exec qrouter-123 iptables -t mangle -I PREROUTING -i qg-123 -j MARK --set-xmark 0x2/0xffff
  
# ip netns exec qrouter-123 route add -net Default gw 18.18.18.1
# ip netns exec test route add -net 0.0.0.0 gw 192.168.10.1
# ip netns exec qrouter-123 sysctl -w net.ipv4.ip_forward=1

测试此vrouter的功能是否正常：

# ip netns exec test ping 18.18.18.1
PING 18.18.18.1 (18.18.18.1) 56(84) bytes of data.
64 bytes from 18.18.18.1: icmp_seq=1 ttl=63 time=0.805 ms
64 bytes from 18.18.18.1: icmp_seq=2 ttl=63 time=0.073 ms
64 bytes from 18.18.18.1: icmp_seq=3 ttl=63 time=0.064 ms
64 bytes from 18.18.18.1: icmp_seq=4 ttl=63 time=0.062 ms
^C
--- 18.18.18.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.062/0.251/0.805/0.319 ms

vrouter功能验证正常。

清理vrouter中的IP信息：

NETWORK2:

# ip netns exec qrouter-123 ip addr del 18.18.18.10/24 dev qg-123
# ip netns exec qrouter-123 ip addr del 192.168.10.1/24 dev qr-123

NETWORK1：

# ip netns exec qrouter-123 ip addr del 18.18.18.10/24 dev qg-123
# ip netns exec qrouter-123 ip addr del 192.168.10.1/24 dev qr-123
# ip netns exec qrouter-123 ip link set dev qr-123 up
# ip netns exec qrouter-123 ip link set dev qg-123 up

创建HA的网络

参数：

vni: 0x43
network1,local vlan: 113
network2,local vlan: 123

添加HA接口：

NETWORK1：

# ovs-vsctl add-port br-int ha-123 -- set interface ha-123 type=internal -- set port ha-123 tag=113
# ip link set dev ha-123 netns qrouter-123
# ip netns exec qrouter-123 ip link set dev ha-123 up
# ip netns exec qrouter-123 ip addr add 169.254.192.1/18 dev ha-123

NETWORK2:

# ovs-vsctl add-port br-int ha-123 -- set interface ha-123 type=internal -- set port ha-123 tag=123
# ip link set dev ha-123 netns qrouter-123
# ip netns exec qrouter-123 ip link set dev ha-123 up
# ip netns exec qrouter-123 ip addr add 169.254.192.2/18 dev ha-123

添加流表

NETWORK1：

# ovs-ofctl add-flow br-tun 'cookie=0x79,  table=4,  priority=1,tun_id=0x43 actions=mod_vlan_vid:113,resubmit(,10)'
# ovs-ofctl add-flow br-tun 'cookie=0x79, table=22, dl_vlan=113 actions=strip_vlan,set_tunnel:0x43,output:8'

NETWORK2：

# ovs-ofctl add-flow br-tun 'cookie=0x79,  table=4,  priority=1,tun_id=0x43 actions=mod_vlan_vid:123,resubmit(,10)'
# ovs-ofctl add-flow br-tun 'cookie=0x79, table=22, dl_vlan=123 actions=strip_vlan,set_tunnel:0x43,output:7'

HA网络的连通性

Network2:

# ip netns exec qrouter-123 ping 169.254.192.1
PING 169.254.192.1 (169.254.192.1) 56(84) bytes of data.
64 bytes from 169.254.192.1: icmp_seq=1 ttl=64 time=2.74 ms
64 bytes from 169.254.192.1: icmp_seq=2 ttl=64 time=0.488 ms
64 bytes from 169.254.192.1: icmp_seq=3 ttl=64 time=0.485 ms
64 bytes from 169.254.192.1: icmp_seq=4 ttl=64 time=0.481 ms
^C
--- 169.254.192.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 0.481/1.050/2.746/0.979 ms

配置keepalived

系统配置：

# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf
# sysctl -p

# iptables -I INPUT -p vrrp -j ACCEPT
# service iptables save

配置如下，/home/keepalived/keepalived.conf：

vrrp_instance VR_1 {
    state BACKUP
    interface ha-123
    virtual_router_id 24
    priority 50
    garp_master_delay 60
    nopreempt
    advert_int 2
    track_interface {
        ha-123
    }
    virtual_ipaddress {
        169.254.0.1/24 dev ha-123
    }
    virtual_ipaddress_excluded {
        192.168.10.1/24 dev qr-123
        18.18.18.10/24 dev qg-123
    }
    virtual_routes {
        0.0.0.0/0 via 18.18.18.1 dev qg-123
    }
}

运行keepalived进程：

# ip netns exec qrouter-123 keepalived -P -f /home/keepalived/keepalived.conf

HA测试

NETWORK1：

#  ip netns exec qrouter-123 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
154: qr-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 46:b2:c3:95:d6:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::44b2:c3ff:fe95:d620/64 scope link 
       valid_lft forever preferred_lft forever
156: qg-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether de:86:a9:7b:b0:40 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc86:a9ff:fe7b:b040/64 scope link 
       valid_lft forever preferred_lft forever
158: qr-124: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 66:c7:ae:94:b9:3a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::64c7:aeff:fe94:b93a/64 scope link 
       valid_lft forever preferred_lft forever
161: ha-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 52:27:5d:9c:3c:83 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.1/18 scope global ha-123
       valid_lft forever preferred_lft forever
    inet6 fe80::5027:5dff:fe9c:3c83/64 scope link 
       valid_lft forever preferred_lft forever

NETWORK2：

# ip netns exec qrouter-123 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
95: qr-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 82:a7:97:58:91:7a brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.1/24 scope global qr-123
       valid_lft forever preferred_lft forever
    inet6 fe80::80a7:97ff:fe58:917a/64 scope link 
       valid_lft forever preferred_lft forever
97: qg-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 3e:44:c1:35:a7:cf brd ff:ff:ff:ff:ff:ff
    inet 18.18.18.10/24 scope global qg-123
       valid_lft forever preferred_lft forever
    inet6 fe80::3c44:c1ff:fe35:a7cf/64 scope link 
       valid_lft forever preferred_lft forever
99: ha-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether b2:f4:a3:4c:27:82 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.2/18 scope global ha-123
       valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-123
       valid_lft forever preferred_lft forever
    inet6 fe80::b0f4:a3ff:fe4c:2782/64 scope link 
       valid_lft forever preferred_lft forever

如上，我们可以看出，网关IP及外部IP都只起在其中一个节点上。

连通性如下：

PING 网关：
# ip netns exec test ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.799 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.084 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=0.091 ms
64 bytes from 192.168.10.1: icmp_seq=4 ttl=64 time=0.073 ms
^C
--- 192.168.10.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.073/0.261/0.799/0.310 ms

PING外部网络的网关：
# ip netns exec test ping 18.18.18.1
PING 18.18.18.1 (18.18.18.1) 56(84) bytes of data.
64 bytes from 18.18.18.1: icmp_seq=1 ttl=63 time=0.860 ms
64 bytes from 18.18.18.1: icmp_seq=2 ttl=63 time=0.080 ms
64 bytes from 18.18.18.1: icmp_seq=3 ttl=63 time=0.066 ms
^C
--- 18.18.18.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.066/0.335/0.860/0.371 ms

测试IP漂移：

NETWORK1：

关闭ha口
#ip netns exec qrouter-123 ip link set dev ha-123 down

查看NETWORK2的VROUTER:

# ip netns exec qrouter-123 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
95: qr-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 82:a7:97:58:91:7a brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.1/24 scope global qr-123
       valid_lft forever preferred_lft forever
    inet6 fe80::80a7:97ff:fe58:917a/64 scope link 
       valid_lft forever preferred_lft forever
97: qg-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 3e:44:c1:35:a7:cf brd ff:ff:ff:ff:ff:ff
    inet 18.18.18.10/24 scope global qg-123
       valid_lft forever preferred_lft forever
    inet6 fe80::3c44:c1ff:fe35:a7cf/64 scope link 
       valid_lft forever preferred_lft forever
99: ha-123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether b2:f4:a3:4c:27:82 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.2/18 scope global ha-123
       valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-123
       valid_lft forever preferred_lft forever
    inet6 fe80::b0f4:a3ff:fe4c:2782/64 scope link 
       valid_lft forever preferred_lft forever

可以发现网关IP，及外部网络IP都漂移至NETWORK2上，测试一下连通性：

# ip netns exec test ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.465 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.062 ms
^C
--- 192.168.10.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.062/0.263/0.465/0.202 ms
[root@youngster keepalived]# ip netns exec test ping 18.18.18.1
PING 18.18.18.1 (18.18.18.1) 56(84) bytes of data.
64 bytes from 18.18.18.1: icmp_seq=1 ttl=63 time=0.428 ms
64 bytes from 18.18.18.1: icmp_seq=2 ttl=63 time=0.065 ms
64 bytes from 18.18.18.1: icmp_seq=3 ttl=63 time=0.062 ms
^C
--- 18.18.18.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.062/0.185/0.428/0.171 ms

keepalived配置说明：

vrrp_instance VR_1 {
    当前节点在此虚拟路由器上的初始状态
    state BACKUP
    设置实例绑定的网卡 VRRP心跳包从哪块网卡发出
    interface ha-123
    当前虚拟路由器的惟一标识，范围是0-255
    virtual_router_id 1
    当前主机在此虚拟路径器中的优先级；范围1-254
    priority 50
    当切为主状态后多久更新ARP缓存
    garp_master_delay 60
    不抢占
    nopreempt
    检查间隔，2s VRRP心跳包的发送周期
    advert_int 2
    监控网卡
    track_interface {
        ha-123
    }
    设置VIP
    virtual_ipaddress {
        169.254.0.1/24 dev ha-123
    }
    virtual_ipaddress_excluded {
        192.168.10.1/24 dev qr-123
        18.18.18.10/24 dev qg-123
    }
    默认路由
    virtual_routes {
        0.0.0.0/0 via 18.18.18.1 dev qg-123
    }
}

NEUTRON: VROUTER HA机制的实现

HA配置

NETWORK2上添加vrouter

创建HA的网络

HA测试

keepalived配置说明：

猜你喜欢

热点阅读