Ubuntu+iptables软路由诡异失效的问题
2019-04-16 本文已影响0人
我要牛肉面面
TL;DR
PCI设备变更 -> 网卡名称改变 -> 数据包不匹配iptables规则，无法转发 -> issue
背景
Ubuntu 16.04.5， kernel 4.4.0-140
Libvirt自带软路由+网桥，nat模式，开机自动配置iptables
路由WAN口为enp3s0，LAN端网桥名为virbr0，之后手动通过brctl命令添加物理LAN口
详细配置见另一篇文章
问题描述
内网DHCP正常，但无法转发数据包，内网ping外网提示无法打开端口。
问题排查
软路由ping内外网都没问题，确定为软路由自身问题。
[user@localhost ~]$ ping 192.168.122.190 -c 4
PING 192.168.122.190 (192.168.122.190) 56(84) bytes of data.
64 bytes from 192.168.122.190: icmp_seq=1 ttl=128 time=0.589 ms
64 bytes from 192.168.122.190: icmp_seq=2 ttl=128 time=0.642 ms
64 bytes from 192.168.122.190: icmp_seq=3 ttl=128 time=0.719 ms
64 bytes from 192.168.122.190: icmp_seq=4 ttl=128 time=0.662 ms
--- 192.168.122.190 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.589/0.653/0.719/0.046 ms
[user@localhost ~]$ ping www.baidu.com -c 4
PING www.a.shifen.com (119.75.217.26) 56(84) bytes of data.
64 bytes from 119.75.217.26: icmp_seq=1 ttl=50 time=1.56 ms
64 bytes from 119.75.217.26: icmp_seq=2 ttl=50 time=1.47 ms
64 bytes from 119.75.217.26: icmp_seq=3 ttl=50 time=1.59 ms
64 bytes from 119.75.217.26: icmp_seq=4 ttl=50 time=1.44 ms
--- www.a.shifen.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.443/1.520/1.595/0.061 ms
iptables -L看规则，filter和nat表都未见异常。
[root@localhost ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 192.168.32.123 192.168.122.0/24
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
[root@localhost ~]# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
RETURN all -- 192.168.122.0/24 base-address.mcast.net/24
RETURN all -- 192.168.122.0/24 255.255.255.255
MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24
tshark抓ICMP包，发现无法转发。
[root@localhost ~]# tshark -f icmp -i any
Running as user "root" and group "root". This could be dangerous.
tshark: Lua: Error during loading:
/usr/share/wireshark/init.lua:32: dofile has been disabled due to running Wireshark as superuser. See https://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user.
Capturing on 'any'
1 0.000000000 192.168.122.1 → 192.168.122.190 ICMP 96 Destination unreachable (Port unreachable)
2 0.000006704 192.168.122.1 → 192.168.122.190 ICMP 96 Destination unreachable (Port unreachable)
3 1.060752767 192.168.122.1 → 192.168.122.190 ICMP 92 Destination unreachable (Port unreachable)
4 1.060758520 192.168.122.1 → 192.168.122.190 ICMP 92 Destination unreachable (Port unreachable)
5 3.556631036 192.168.122.1 → 192.168.122.190 ICMP 92 Destination unreachable (Port unreachable)
6 3.556638393 192.168.122.1 → 192.168.122.190 ICMP 92 Destination unreachable (Port unreachable)
7 4.415358663 192.168.122.1 → 192.168.122.190 ICMP 96 Destination unreachable (Port unreachable)
8 4.415363261 192.168.122.1 → 192.168.122.190 ICMP 96 Destination unreachable (Port unreachable)
9 6.006074581 192.168.122.1 → 192.168.122.190 ICMP 92 Destination unreachable (Port unreachable)
10 6.006081615 192.168.122.1 → 192.168.122.190 ICMP 92 Destination unreachable (Port unreachable)
无意间通过ifconfig发现网卡名称改变为enp2s0，且后面原名为enp5s0的PCI网卡名称变为enp4s0。
[user@localhost ~]$ ifconfig
enp2s0 Link encap:以太网 硬件地址 ??:??:??:??:??:??
inet 地址:192.168.32.122 广播:192.168.32.255 掩码:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 跃点数:1
接收数据包:7503 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:2504 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1000
接收字节:1146987 (1.1 MB) 发送字节:319633 (319.6 KB)
中断:25 基本地址:0x6000
enp4s0 Link encap:以太网 硬件地址 ??:??:??:??:??:??
UP BROADCAST RUNNING MULTICAST MTU:1500 跃点数:1
接收数据包:1794 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:1900 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1000
接收字节:163712 (163.7 KB) 发送字节:202771 (202.7 KB)
lo Link encap:本地环回
inet 地址:127.0.0.1 掩码:255.0.0.0
inet6 地址: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 跃点数:1
接收数据包:866 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:866 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1
接收字节:106525 (106.5 KB) 发送字节:106525 (106.5 KB)
virbr0 Link encap:以太网 硬件地址 ??:??:??:??:??:??
inet 地址:192.168.122.1 广播:192.168.122.255 掩码:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 跃点数:1
接收数据包:1664 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:1195 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1000
接收字节:124214 (124.2 KB) 发送字节:133241 (133.2 KB)
lspci发现独显缺失。
[user@localhost ~]$ lspci
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation B85 Express LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
03:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 10)
05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
回想到之前尝试给G3258超频时拔掉了显卡供电。
关机，重新插上显卡供电，显卡占据PCI 01:00.0的位置，网卡占据03:00.0和05:00.0的位置：
[user@localhost ~]$ lspci
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation B85 Express LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970]
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 10)
06:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
网卡名称也变回了enp3s0和enp5s0，问题解决。
[user@localhost ~]$ ifconfig
enp3s0 Link encap:以太网 硬件地址 ??:??:??:??:??:??
inet 地址:192.168.32.122 广播:192.168.32.255 掩码:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 跃点数:1
接收数据包:1366 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:759 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1000
接收字节:227359 (227.3 KB) 发送字节:85406 (85.4 KB)
中断:26 基本地址:0x6000
enp5s0 Link encap:以太网 硬件地址 ??:??:??:??:??:??
UP BROADCAST RUNNING MULTICAST MTU:1500 跃点数:1
接收数据包:519 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:331 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1000
接收字节:67400 (67.4 KB) 发送字节:85487 (85.4 KB)
lo Link encap:本地环回
inet 地址:127.0.0.1 掩码:255.0.0.0
inet6 地址: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 跃点数:1
接收数据包:231 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:231 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1
接收字节:26272 (26.2 KB) 发送字节:26272 (26.2 KB)
virbr0 Link encap:以太网 硬件地址 ??:??:??:??:??:??
inet 地址:192.168.122.1 广播:192.168.122.255 掩码:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 跃点数:1
接收数据包:345 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:179 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1000
接收字节:45392 (45.3 KB) 发送字节:63595 (63.5 KB)
另外也可以在Libvirt的网络配置文件里修改<forward dev='enp2s0'>，之后通过
virsh net-destroy default
virsh net-start default
拆除并重建网络。
