This is weird.. I had gone through lots of pain to get an IBM 4-port 1Gbit copper ethernet NIC working in ProxMox and after much effort I finally saw the ports in the GUI.
I have 2 ports (enp3s0f0, enp3s0f1) set to a bond called bond0 with LACP 802.3ad and my switch has tagged VLANs on those ports. I added bond0 to bridge vmbr1 and this was up and working. Next I had a third port enp4s0f0 directly plugged into a port on my NAS. Just a simple connection. Ports were up, traffic was flowing and working.
Then some minutes later, without explanation, all ports are down, and no traffic is flowing. The ports show up in the GUI but whenever I try to change network configuration, I now get an error
and when I try to bring interfaces up, now it claims it doesn't recognize the devices, even though they are clearly visible in the GUI. I don't understand what is happening here. Can anyone help me? I've had nothing but problems with this 4-port NIC and I'm about to give up and either buy another card or just go to USB Ethernet adapters, much as I don't like that option. It's a standard Intel chipset and this card worked fine under Windows Server 2022. ¯\_(ツ)_/¯
All the interfaces are now missing from "ip link" whereas they were there before.
My interfaces file:
Relevant dmesg:
I have 2 ports (enp3s0f0, enp3s0f1) set to a bond called bond0 with LACP 802.3ad and my switch has tagged VLANs on those ports. I added bond0 to bridge vmbr1 and this was up and working. Next I had a third port enp4s0f0 directly plugged into a port on my NAS. Just a simple connection. Ports were up, traffic was flowing and working.
Then some minutes later, without explanation, all ports are down, and no traffic is flowing. The ports show up in the GUI but whenever I try to change network configuration, I now get an error
Code:
command 'ifreload -a' failed: exit code 1
Bash:
proxmox:~# ifup -a --force
warning: bond0: attribute bond-min-links is set to '0'
warning: enp3s0f0: interface not recognized - please check interface configuration
warning: enp3s0f1: interface not recognized - please check interface configuration
warning: enp4s0f0: interface not recognized - please check interface configurationAll the interfaces are now missing from "ip link" whereas they were there before.
My interfaces file:
Bash:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet manual
#1
auto enp3s0f0
iface enp3s0f0 inet manual
#Port 1 (right-most)
auto enp3s0f1
iface enp3s0f1 inet manual
#Port 2 (2nd from right)
auto enp4s0f0
iface enp4s0f0 inet manual
mtu 9000
#Port 3 (2nd from left)
iface enp4s0f1 inet manual
#Port 4 (left-most)
iface bond0 inet manual
bond-slaves enp3s0f0 enp3s0f1
bond-miimon 100
bond-mode 802.3ad
#LAG to Switch
auto vmbr0
iface vmbr0 inet static
address 192.168.<REDACTED>.66/24
gateway 192.168.<REDACTED>.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#Onboard NIC - LAN
auto vmbr2
iface vmbr2 inet static
address 192.168.<REDACTED>2/24
bridge-ports enp4s0f0
bridge-stp off
bridge-fd 0
mtu 9000
#To QNAP NAS
auto vmbr1
iface vmbr1 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094Relevant dmesg:
Bash:
proxmox:~# dmesg | grep 0000:04
[ 0.571235] pci 0000:04:00.0: [8086:10bc] type 00 class 0x020000
[ 0.571252] pci 0000:04:00.0: reg 0x10: [mem 0x910a0000-0x910bffff]
[ 0.571261] pci 0000:04:00.0: reg 0x14: [mem 0x91080000-0x9109ffff]
[ 0.571269] pci 0000:04:00.0: reg 0x18: [io 0x3020-0x303f]
[ 0.571295] pci 0000:04:00.0: reg 0x30: [mem 0x91060000-0x9107ffff pref]
[ 0.571359] pci 0000:04:00.0: PME# supported from D0 D3hot
[ 0.571443] pci 0000:04:00.1: [8086:10bc] type 00 class 0x020000
[ 0.571460] pci 0000:04:00.1: reg 0x10: [mem 0x91040000-0x9105ffff]
[ 0.571468] pci 0000:04:00.1: reg 0x14: [mem 0x91020000-0x9103ffff]
[ 0.571477] pci 0000:04:00.1: reg 0x18: [io 0x3000-0x301f]
[ 0.571503] pci 0000:04:00.1: reg 0x30: [mem 0x91000000-0x9101ffff pref]
[ 0.571566] pci 0000:04:00.1: PME# supported from D0 D3hot
[ 0.696252] pci_bus 0000:04: resource 0 [io 0x3000-0x3fff]
[ 0.696254] pci_bus 0000:04: resource 1 [mem 0x91000000-0x910fffff]
[ 0.697542] pci 0000:04:00.0: Adding to iommu group 1
[ 0.697548] pci 0000:04:00.1: Adding to iommu group 1
[ 1.831452] e1000e 0000:04:00.0: enabling device (0100 -> 0102)
[ 1.831596] e1000e 0000:04:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[ 1.991240] e1000e 0000:04:00.0 eth3: (PCI Express:2.5GT/s:Width x4) e4:1f:13:fa:0c:df
[ 1.991243] e1000e 0000:04:00.0 eth3: Intel(R) PRO/1000 Network Connection
[ 1.991323] e1000e 0000:04:00.0 eth3: MAC: 0, PHY: 4, PBA No: E39336-005
[ 1.991347] e1000e 0000:04:00.1: enabling device (0100 -> 0102)
[ 1.991492] e1000e 0000:04:00.1: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[ 2.155197] e1000e 0000:04:00.1 eth4: (PCI Express:2.5GT/s:Width x4) e4:1f:13:fa:0c:de
[ 2.155203] e1000e 0000:04:00.1 eth4: Intel(R) PRO/1000 Network Connection
[ 2.155283] e1000e 0000:04:00.1 eth4: MAC: 0, PHY: 4, PBA No: E39336-005
[ 2.202983] e1000e 0000:04:00.1 enp4s0f1: renamed from eth4
[ 2.238907] e1000e 0000:04:00.0 enp4s0f0: renamed from eth3
[ 26.619971] e1000e 0000:04:00.0: Some CPU C-states have been disabled in order to enable jumbo frames
[ 29.143633] e1000e 0000:04:00.0 enp4s0f0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 30.266883] e1000e 0000:04:00.0 enp4s0f0: NIC Link is Down
[ 32.923697] e1000e 0000:04:00.0 enp4s0f0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 32.924245] vmbr2: port 1(enp4s0f0) entered blocking state
[ 32.924660] vmbr2: port 1(enp4s0f0) entered forwarding state
[ 33.571224] e1000e 0000:03:00.0 enp3s0f0: NIC Link is Down
[ 33.571369] bond0: (slave enp3s0f0): Releasing backup interface
[ 34.999176] e1000e 0000:03:00.1 enp3s0f1: NIC Link is Down
[ 34.999567] bond0: (slave enp3s0f1): Removing an active aggregator
[ 34.999586] bond0: (slave enp3s0f1): Releasing backup interface
[ 35.743213] e1000e 0000:04:00.0 enp4s0f0: NIC Link is Down
[ 35.743369] vmbr2: port 1(enp4s0f0) entered disabled state
[ 35.743640] device enp4s0f0 left promiscuous mode
[ 35.743652] vmbr2: port 1(enp4s0f0) entered disabled state
[ 37.980755] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
[ 38.020643] vfio-pci 0000:04:00.1: enabling device (0000 -> 0003)