i40e Network driver Error, NICs are down

maxprox

maxprox

Renowned Member
Proxmox Subscriber
Aug 23, 2011
420
53
93
Germany - Nordhessen
fair-comp.de
Hi,

today, 2020-08-04, after the kernel upgrade, from initrd.img-5.4.41-1-pve to initrd.img-5.4.44-2-pve, and a reboot,
all NIC of the "DMI: Supermicro Super Server/X11SPM-F, BIOS 3.2 10/18/2019 " are down.
There are two RJ45 1Gbit/s NICs (and not one 10 GBit NIC).
As in this thread: https://forum.proxmox.com/threads/error-i40e_aq_rc_enospc-forcing-overflow-promiscuous-on-pf.62875/
I get the Errors in the syslog:
Code: 
...
Aug  4 08:52:08 fcpro kernel: [  369.378911] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC adding RX filters on PF, promiscuous mode forced on
Aug  4 08:52:08 fcpro kernel: [  369.379532] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF
Aug  4 08:52:08 fcpro kernel: [  369.379638] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF
Aug  4 08:52:08 fcpro kernel: [  369.379794] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC adding RX filters on PF, promiscuous mode forced on
Aug  4 08:52:08 fcpro kernel: [  369.380390] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF
Aug  4 08:52:08 fcpro kernel: [  369.380496] i40e 0000:b5:00.3: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF
...
The Network driver i40e is version 2.8.20:
Code: 
 # modinfo i40e 
filename:       /lib/modules/5.4.44-2-pve/kernel/drivers/net/ethernet/intel/i40e/i40e.ko
version:        2.8.20-k

(The newest Intel driver is version i40e-2.12.6/ from here: https://downloadcenter.intel.com/do...twork-Connections-Under-Linux-?product=139349 I did not install it until now)

After a login with the IPMI an a network restart via
Code: 
  systemctl restart networking.service
the network Interfaces are up with an IP-Address and working.


Is this a Bug?
Or is it needed to install the newest Intel driver?

regards,
maxprox

Code: 
proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-1
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
pve-zsync: 2.0-3
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

the simple network setup:

Code: 
 cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
	address 192.168.88.100/24
	gateway 192.168.88.20
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
	bridge-ports eno2
	bridge-stp off
	bridge-fd 0
 
Last edited:
Just to be sure that you hunt down the correct problem:
AFAIK i40e is a driver only for 10G (and 40G) NICs not 1G ones

please post the output of :
Code: 
lspci -nnk
ip link
cat /etc/network/interfaces

one possible cause could be that the 'predictable network names' of the nics changed, which broke the network configuration
 
Thanks for the reply

yes both point I also reflected, but I can not see a changing of the NIC names, before and after the upgrade: eno1 and eno2
and lspci show me that with this board also this driver is in use, have a look at the last 7 lines.
Our Supermicro board only have two RJ45 1GBit/s NICs, not more (and one IPMI)

Code: 
root@fcpro /opt/i40e # lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
    Subsystem: Super Micro Computer Inc Sky Lake-E DMI3 Registers (X11SPM-TF) [15d9:095d]
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
    Kernel driver in use: ioatdma
    Kernel modules: ioatdma
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
    Kernel driver in use: ioatdma
    Kernel modules: ioatdma
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers [8086:0000]
    Kernel driver in use: ioatdma
    Kernel modules: ioatdma
...
10.000 characters and no more ;-)
...
Kernel driver in use: skx_uncore
64:0b.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:0000]
    Kernel driver in use: skx_uncore
64:0c.7 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:2047] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:0000]
64:0d.0 System peripheral [0880]: Intel Corporation Sky Lake-E DECS Channel 2 [8086:2048] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E DECS Channel 2 [8086:0000]
64:0d.1 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 2 [8086:2049] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LM Channel 2 [8086:0000]
64:0d.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LMS Channel 2 [8086:0000]
    Kernel driver in use: skx_uncore
64:0d.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:0000]
b2:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1A [8086:2030] (rev 04)
    Kernel driver in use: pcieport
b2:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E VT-d [8086:0000]
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036]
b2:0e.0 Performance counters [1101]: Intel Corporation Sky Lake-E KTI 0 [8086:2058] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E KTI 0 [8086:0000]
    Kernel driver in use: skx_uncore
b2:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E UPI Registers [8086:2059] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E UPI Registers [8086:0000]
b2:0f.0 Performance counters [1101]: Intel Corporation Sky Lake-E KTI 0 [8086:2058] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E KTI 0 [8086:0000]
    Kernel driver in use: skx_uncore
b2:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E UPI Registers [8086:2059] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E UPI Registers [8086:0000]
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
    Kernel driver in use: skx_uncore
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers [8086:0000]
    Kernel driver in use: skx_uncore
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M2PCI Registers [8086:0000]
...
b5:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:37cc] (rev 09)
    Subsystem: Super Micro Computer Inc Device [15d9:0000]
b5:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 1GbE [8086:37d1] (rev 09)
    Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 1GbE [15d9:37d1]
    Kernel driver in use: i40e
    Kernel modules: i40e
b5:00.3 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 1GbE [8086:37d1] (rev 09)
    Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 1GbE [15d9:37d1]
    Kernel driver in use: i40e
    Kernel modules: i40e
 
Last edited:
I also do not understand, that the only thing I have to do is a restart of the network service, then the NICs are up and working correct
Code: 
 systemctl restart networking.service

EDIT:
The only thing which we changes on this proxmox server except the upgrade is to activate the new proxmox-backup for testing it.
 
Last edited:
maxprox said:
The only thing which we changes on this proxmox server except the upgrade is to activate the new proxmox-backup for testing it.
Click to expand...
* does the issue persist if you boot into an older kernel?
* how many vlans do you have configured on that machine (including ones only used in guests)
* is the proxmox backup server maybe reachable over a new vlan (which you added to the node and thus maybe went over the limit of the NIC)?

Oh - and thanks for sharing the lspci output - learned that there are 1Gb NICs which use the i40e driver :)
 
Stoiko Ivanov said:
1:* does the issue persist if you boot into an older kernel?
2:* how many vlans do you have configured on that machine (including ones only used in guests)
3:* is the proxmox backup server maybe reachable over a new vlan (which you added to the node and thus maybe went over the limit of the NIC)?

Oh - and thanks for sharing the lspci output - learned that there are 1Gb NICs which use the i40e driver :)
Click to expand...

1: we have to test it later the server is in production use
2: YES we have some vlans configured, and have changed it now to outside from proxmox to the router and vlan-Switch, so that just now neither proxmox nor one vm see vlan's
3: there is no vlan (this proxmox is the backup client, the other one (VPN - Internet) is the proxmox backup server)

maby tonight I will test a reboot without any vlan settings ... I will report
best regards,
maxprox
 
  • Like
Reactions: Stoiko Ivanov
@maxprox This is not related but I am currently having a similar issue. Apologies for talking about XCP-ng here but network interface on Intel 10G card keeps flapping and eventually loses connection. I have to reboot the host. Going to get a replacement card. I wish this is a hardware issue because if not, I don't know what to do.

Eoin
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back