WAN NIC for pfSense VM keeps dropping connection

spencerh

Member
May 18, 2021
2
0
6
32
I have a Proxmox system running a pfSense VM that I use as my router. I have two Intel 82574L NICs, one for WAN, one for LAN. Seemingly at random, the NIC that's used as the WAN interface will drop and reacquire the link every few seconds for a few minutes. The NICs are setup as dedicated bridges, and pfSense is using the virtio drivers inside the VM. pfSense doesn't seem to see any of the drops. I tried running `ethtool -K vmbr1 tx off gso off` and upgrading to the 5.11 kernel, both of which didn't solve the issue. The LAN interface never seems to drop. Any idea why this might be happening? Is there any other information I can provide to help diagnose the issue? I'm pulling my hair out and my roommates are getting frustrated with their inability to work/game (understandably) so if anyone has any insight into why this might be happening I'd be very appreciative.

dmesg output:
Code:
[Tue May 18 11:02:08 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Down
[Tue May 18 11:02:08 2021] vmbr1: port 1(enp8s0) entered disabled state
[Tue May 18 11:02:12 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Tue May 18 11:02:12 2021] vmbr1: port 1(enp8s0) entered blocking state
[Tue May 18 11:02:12 2021] vmbr1: port 1(enp8s0) entered forwarding state
[Tue May 18 11:02:12 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Down
[Tue May 18 11:02:13 2021] vmbr1: port 1(enp8s0) entered disabled state
[Tue May 18 11:02:16 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Tue May 18 11:02:16 2021] vmbr1: port 1(enp8s0) entered blocking state
[Tue May 18 11:02:16 2021] vmbr1: port 1(enp8s0) entered forwarding state
[Tue May 18 11:02:21 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Down
[Tue May 18 11:02:21 2021] vmbr1: port 1(enp8s0) entered disabled state
[Tue May 18 11:02:24 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Tue May 18 11:02:24 2021] vmbr1: port 1(enp8s0) entered blocking state
[Tue May 18 11:02:24 2021] vmbr1: port 1(enp8s0) entered forwarding state
[Tue May 18 11:02:31 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Down
[Tue May 18 11:02:31 2021] vmbr1: port 1(enp8s0) entered disabled state
[Tue May 18 11:02:34 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Tue May 18 11:02:34 2021] vmbr1: port 1(enp8s0) entered blocking state
[Tue May 18 11:02:34 2021] vmbr1: port 1(enp8s0) entered forwarding state
[Tue May 18 11:02:57 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Down
[Tue May 18 11:02:57 2021] vmbr1: port 1(enp8s0) entered disabled state
[Tue May 18 11:03:01 2021] e1000e 0000:08:00.0 enp8s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Tue May 18 11:03:01 2021] vmbr1: port 1(enp8s0) entered blocking state
[Tue May 18 11:03:01 2021] vmbr1: port 1(enp8s0) entered forwarding state

pveversion -v output:
Code:
proxmox-ve: 6.4-1 (running kernel: 5.11.17-1-pve)
pve-manager: 6.4-6 (running version: 6.4-6/be2fa32c)
pve-kernel-5.11: 7.0-1~bpo10
pve-kernel-5.4: 6.4-2
pve-kernel-helper: 6.4-2
pve-kernel-5.11.17-1-pve: 5.11.17-1~bpo10
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-2
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.6-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-4
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-3
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

ethtool -k vmbr1 output:
Code:
Features for vmbr1:
rx-checksumming: off [fixed]
tx-checksumming: off
   tx-checksum-ipv4: off [fixed]
   tx-checksum-ip-generic: off
   tx-checksum-ipv6: off [fixed]
   tx-checksum-fcoe-crc: off [fixed]
   tx-checksum-sctp: off [fixed]
scatter-gather: on
   tx-scatter-gather: on
   tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
   tx-tcp-segmentation: off [requested on]
   tx-tcp-ecn-segmentation: off [requested on]
   tx-tcp-mangleid-segmentation: off [requested on]
   tx-tcp6-segmentation: off [requested on]
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [requested on]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: on
tx-sctp-segmentation: off [requested on]
tx-esp-segmentation: on
tx-udp-segmentation: off [requested on]
tx-gso-list: off [requested on]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
 
Don’t run the ethtool on vmbrX interfaces if the real physical (enp8s0) is the one that is breaking the link.

Use `ethtool enp8s0`

a couple more things to note on link breakage at Ethernet/802.11 data link:

1. Longer or bad cable (inspect entire length)
2. bad connector
3. router/switch is failing
4. DHCP server hijacking IP address
5. Duplicate MAC fighting
 
Last edited:
Yeah the cable will be the next thing I'll be replacing.

The router is being virtualized on the Proxmox instance; enp8s0 is the WAN interface of the virtualized pfSense instance.

/etc/network/interfaces
Code:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp11s0 inet manual

iface enp10s0 inet manual

iface enp12s0 inet manual

iface enp13s0 inet manual

iface enp2s0 inet manual

iface enp2s0d1 inet manual

iface enp7s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.0.6/16
        gateway 10.10.0.1
        bridge-ports enp7s0
        bridge-stp off
        bridge-fd 0
#Main VM Bridge

auto vmbr1
iface vmbr1 inet manual
#       bridge-ports enp10s0
        bridge-ports enp8s0
        bridge-stp off
        bridge-fd 0
#pfSense WAN

auto vmbr2
iface vmbr2 inet manual
#       bridge-ports enp11s0
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
#pfSense LAN

ethtool -k enp8s0
Code:
Features for enp8s0:
rx-checksumming: on
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off [requested on]
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!