i40e Network driver Error, NICs are down

maxprox

maxprox

Renowned Member
Proxmox Subscriber
Aug 23, 2011
423
60
93
Germany - Nordhessen
fair-comp.de
Hi,

today, 2020-08-04, after the kernel upgrade, from initrd.img-5.4.41-1-pve to initrd.img-5.4.44-2-pve, and a reboot,
all NIC of the "DMI: Supermicro Super Server/X11SPM-F, BIOS 3.2 10/18/2019 " are down.
There are two RJ45 1Gbit/s NICs (and not one 10 GBit NIC).
As in this thread: https://forum.proxmox.com/threads/error-i40e_aq_rc_enospc-forcing-overflow-promiscuous-on-pf.62875/
I get the Errors in the syslog:
Code: 
...
Aug  4 08:52:08 fcpro kernel: [  369.378911] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC adding RX filters on PF, promiscuous mode forced on
Aug  4 08:52:08 fcpro kernel: [  369.379532] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF
Aug  4 08:52:08 fcpro kernel: [  369.379638] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF
Aug  4 08:52:08 fcpro kernel: [  369.379794] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC adding RX filters on PF, promiscuous mode forced on
Aug  4 08:52:08 fcpro kernel: [  369.380390] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF
Aug  4 08:52:08 fcpro kernel: [  369.380496] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF
...
The Network driver i40e is version 2.8.20:
Code: 
 # modinfo i40e 
filename:       /lib/modules/5.4.44-2-pve/kernel/drivers/net/ethernet/intel/i40e/i40e.ko
version:        2.8.20-k

(The newest Intel driver is version i40e-2.12.6/ from here: https://downloadcenter.intel.com/do...twork-Connections-Under-Linux-?product=139349 I did not install it until now)

After a login with the IPMI an a network restart via
Code: 
  systemctl restart networking.service
the network Interfaces are up with an IP-Address and working.


Is this a Bug?
Or is it needed to install the newest Intel driver?

regards,
maxprox

Code: 
proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-1
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
pve-zsync: 2.0-3
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

the simple network setup:

Code: 
 cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
	address 192.168.88.100/24
	gateway 192.168.88.20
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
	bridge-ports eno2
	bridge-stp off
	bridge-fd 0
 
Last edited:
Just to be sure that you hunt down the correct problem:
AFAIK i40e is a driver only for 10G (and 40G) NICs not 1G ones

please post the output of :
Code: 
lspci -nnk
ip link
cat /etc/network/interfaces

one possible cause could be that the 'predictable network names' of the nics changed, which broke the network configuration
 
Thanks for the reply

yes both point I also reflected, but I can not see a changing of the NIC names, before and after the upgrade: eno1 and eno2
and lspci show me that with this board also this driver is in use, have a look at the last 7 lines.
Our Supermicro board only have two RJ45 1GBit/s NICs, not more (and one IPMI)

Code: 
root@fcpro /opt/i40e # lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
    Subsystem: Super Micro Computer Inc Sky Lake-E DMI3 Registers (X11SPM-TF) [15d9:095d]
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
    Kernel driver in use: ioatdma
    Kernel modules: ioatdma
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
    Kernel driver in use: ioatdma
    Kernel modules: ioatdma
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
    Kernel driver in use: ioatdma
    Kernel modules: ioatdma
...
10.000 characters and no more ;-)
...
Kernel driver in use: skx_uncore
64:0b.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:0000]
    Kernel driver in use: skx_uncore
64:0c.7 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:2047] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:0000]
64:0d.0 System peripheral [0880]: Intel Corporation Sky Lake-E DECS Channel 2 [8086:2048] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E DECS Channel 2 [8086:0000]
64:0d.1 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 2 [8086:2049] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LM Channel 2 [8086:0000]
64:0d.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LMS Channel 2 [8086:0000]
    Kernel driver in use: skx_uncore
64:0d.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:0000]
b2:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1A [8086:2030] (rev 04)
    Kernel driver in use: pcieport
b2:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E VT-d [8086:0000]
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036]
b2:0e.0 Performance counters [1101]: Intel Corporation Sky Lake-E KTI 0 [8086:2058] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E KTI 0 [8086:0000]
    Kernel driver in use: skx_uncore
b2:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E UPI Registers [8086:2059] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E UPI Registers [8086:0000]
b2:0f.0 Performance counters [1101]: Intel Corporation Sky Lake-E KTI 0 [8086:2058] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E KTI 0 [8086:0000]
    Kernel driver in use: skx_uncore
b2:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E UPI Registers [8086:2059] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E UPI Registers [8086:0000]
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
    Kernel driver in use: skx_uncore
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
    Kernel driver in use: skx_uncore
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M2PCI Registers [8086:0000]
...
b5:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:37cc] (rev 09)
    Subsystem: Super Micro Computer Inc Device [15d9:0000]
b5:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 1GbE [8086:37d1] (rev 09)
    Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 1GbE [15d9:37d1]
    Kernel driver in use: i40e
    Kernel modules: i40e
b5:00.3 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 1GbE [8086:37d1] (rev 09)
    Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 1GbE [15d9:37d1]
    Kernel driver in use: i40e
    Kernel modules: i40e
 
Last edited:
I also do not understand, that the only thing I have to do is a restart of the network service, then the NICs are up and working correct
Code: 
 systemctl restart networking.service

EDIT:
The only thing which we changes on this proxmox server except the upgrade is to activate the new proxmox-backup for testing it.
 
Last edited:
maxprox said:
The only thing which we changes on this proxmox server except the upgrade is to activate the new proxmox-backup for testing it.
Click to expand...
* does the issue persist if you boot into an older kernel?
* how many vlans do you have configured on that machine (including ones only used in guests)
* is the proxmox backup server maybe reachable over a new vlan (which you added to the node and thus maybe went over the limit of the NIC)?

Oh - and thanks for sharing the lspci output - learned that there are 1Gb NICs which use the i40e driver :)
 
Stoiko Ivanov said:
1:* does the issue persist if you boot into an older kernel?
2:* how many vlans do you have configured on that machine (including ones only used in guests)
3:* is the proxmox backup server maybe reachable over a new vlan (which you added to the node and thus maybe went over the limit of the NIC)?

Oh - and thanks for sharing the lspci output - learned that there are 1Gb NICs which use the i40e driver :)
Click to expand...

1: we have to test it later the server is in production use
2: YES we have some vlans configured, and have changed it now to outside from proxmox to the router and vlan-Switch, so that just now neither proxmox nor one vm see vlan's
3: there is no vlan (this proxmox is the backup client, the other one (VPN - Internet) is the proxmox backup server)

maby tonight I will test a reboot without any vlan settings ... I will report
best regards,
maxprox
 
  • Like
Reactions: Stoiko Ivanov
@maxprox This is not related but I am currently having a similar issue. Apologies for talking about XCP-ng here but network interface on Intel 10G card keeps flapping and eventually loses connection. I have to reboot the host. Going to get a replacement card. I wish this is a hardware issue because if not, I don't know what to do.

Eoin
 
You must log in or register to reply here.
Top Bottom