Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF

For me issue kinda went away at some point with kernel upgrade, and I've just got plenty of that messages in syslog during bootup. I've also tried to upgrade X710 firmware a month ago with no improvement.
 
I manually loaded the driver under kernel 6.8.
Starting from 6.11, the issue disappeared on its own. I haven’t tested 6.14.
 
Also had the problem, with the system also going stale (backups to NFS storage blocked..)
Code:
journalctl -b-1
PVE-Backup: error fetching datastores - 500 Can't connect <---- Good times!

Code:
dmesg | less
i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF <--- Snazzy! (PF=Physical Function (FYI))

Code:
dmesg -Hl err -f kern
i40e 0000:03:00.0: Invalid VF Identifier 0 <------ Hmmm. "Invalid" Indeed! (VF=Virtual Function)

... who is this troubled 0000:3:00.0 I wonder...
Code:
ls -l /sys/class/net/
enp3s0f0np0 -> ../../devices/pci0000:00/0000:00:06.2/0000:03:00.0/net/enp3s0f0np0
Ahhh the ole "VM-Private" IF w/ vmbr2 - super!

I was only able to reproduce the problem with cards that have firmware version 9.40.
Code:
ethtool -i enp3s0f0np0
driver: i40e
version: 6.8.12-10-pve
firmware-version: 9.40 0x8000ecd6 0.0.0 <------- Lucky Me!

What is enabled/disabled??
Code:
ethtool -k enp3s0f1np1 | grep rx-vlan
rx-vlan-offload: on
rx-vlan-filter: on [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]

I've had similar problems and I've changed the interfaces file to what you have here, with only difference being:
Code:
bridge-vids 2-4094
And today one of the the servers decided to spew..

Code:
cat /etc/network/interfaces
iface vmbr2 inet static
bridge-vlan-aware yes
bridge-vids 2-4094 <----- Hmm when did I set that?

I updated /etc/network/interfaces as below, and it solved the problem...

Code:
...
        bridge-vids 10 11 12 ...
        offload-rx-vlan-filter off
&

Oh that Readme from Intel looks like a sexy bit of late night reading - Joy!

Once I wake up 5 times while reading, then get that hot new v2.27.8 driver slapped on and, get that config corrected, I'll report back. Thank you everyone from 2020 until today for sharing your findings <3
 
  • Like
Reactions: itNGO and xenon96
Code:
journalctl -b-1
PVE-Backup: error fetching datastores - 500 Can't connect <---- Good times!

Code:
dmesg | less
i40e 0000:03:00.0: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on PF <--- Snazzy! (PF=Physical Function (FYI))

Code:
dmesg -Hl err -f kern
i40e 0000:03:00.0: Invalid VF Identifier 0 <------ Hmmm. "Invalid" Indeed! (VF=Virtual Function)

... who is this troubled 0000:3:00.0 I wonder...
Code:
ls -l /sys/class/net/
enp3s0f0np0 -> ../../devices/pci0000:00/0000:00:06.2/0000:03:00.0/net/enp3s0f0np0
Ahhh the ole "VM-Private" IF w/ vmbr2 - super!


Code:
ethtool -i enp3s0f0np0
driver: i40e
version: 6.8.12-10-pve
firmware-version: 9.40 0x8000ecd6 0.0.0 <------- Lucky Me!

What is enabled/disabled??
Code:
ethtool -k enp3s0f1np1 | grep rx-vlan
rx-vlan-offload: on
rx-vlan-filter: on [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]



Code:
cat /etc/network/interfaces
iface vmbr2 inet static
bridge-vlan-aware yes
bridge-vids 2-4094 <----- Hmm when did I set that?


&


Oh that Readme from Intel looks like a sexy bit of late night reading - Joy!

Once I wake up 5 times while reading, then get that hot new v2.27.8 driver slapped on and, get that config corrected, I'll report back. Thank you everyone from 2020 until today for sharing your findings <3
Any update on this?
 
Any update on this?
Yes! Sorry I got distracted..

I think the error is solved simply by changing the VLAN IDs setting (under each node's Networking --> vmbr*) from 2-4096 to whatever you're actually using.
So long as you don't have hundreds of VLANs...
If you do, you need to disable VLAN offloading to the NIC and instead use the CPU.
The 700 series NICs can only handle so many VLANs.

I gave it all an update
Bash:
apt update && apt full-upgrade -y && apt autoremove -y && apt clean
I also updated the NIC driver...

Bash:
wget https://github.com/intel/ethernet-linux-i40e/releases/download/v2.28.6/i40e-2.28.6.tar.gz

tar zxf i40e-2.28.6.tar.gz

rm i40e-2.28.6.tar.gz

cd i40e-2.28.6/src/

make install

update-initramfs -u

reboot

modinfo i40e

... and then the firmware of the NIC

Bash:
apt install unzip -y

wget https://downloadmirror.intel.com/853788/700Series_NVMUpdatePackage_v9_54.zip

unzip 700Series_NVMUpdatePackage_v9_54.zip

tar zxf 700Series_NVMUpdatePackage_v9_54_Linux.tar.gz

rm 700Series_NVMUpdatePackage_v9_54_EFI.zip
rm 700Series_NVMUpdatePackage_v9_54_ESX.tar.gz
rm 700Series_NVMUpdatePackage_v9_54_FreeBSD.tar.gz
rm 700Series_NVMUpdatePackage_v9_54_Windows.zip
rm 700Series_NVMUpdatePackage_v9_54.zip
rm 700Series_NVMUpdatePackage_v9_54_Linux.tar.gz

cd 700Series/Linux_x64

chmod 755 nvmupdate64e

./nvmupdate64e

Select your NIC
01
Make a backup..
Y

<any key>

reboot

Everything is all good now!
 
Hi Everyone,

I am also facing these issues and tried to find a solution but it seems to cause issues. Maybe someone can help.

As advices I updated to the latest version
Code:
root@pve1:~# uname -r
6.8.12-15-pve

This is my network card
Code:
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

This is my network config
Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

iface enxbe3af2b6059f inet manual

iface eno2 inet manual


iface enp1s0f0np0 inet manual
post-up /sbin/ethtool -K enp1s0f0np0 rxvlan off
post-up /sbin/ethtool -K enp1s0f0np0  rx-vlan-offload off

iface enp1s0f1np1 inet manual
post-up /sbin/ethtool -K enp1s0f1np1 rxvlan off
post-up /sbin/ethtool -K enp1s0f1np1  rx-vlan-offload off

#auto vmbr10
#iface vmbr10 inet static
#       address 172.16.98.50/24
#       gateway 172.16.98.254
#       bridge-ports eno2
#       bridge-stp off
#       bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp1s0f0np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4096

auto vmbr0.150
iface vmbr0.150 inet static
        address 172.20.150.10/24
        gateway 172.20.150.254


Everything works as expected until I switch on the VLAN Awareness. (I need that in my network)
As I want to be flexible adding VLANs I have to use the whole range --> 2-4096 (Maybe I can leave >1500 out but same error)

So I understood that when using this I have to disable the VLAN Offloading and therefore added the Post up commands in the config.

So long as you don't have hundreds of VLANs...
If you do, you need to disable VLAN offloading to the NIC and instead use the CPU.

I did that rebooted the system but then see this error in the log:
Code:
Sep 24 16:13:12 pve1 kernel: i40e 0000:01:00.0 enp1s0f0np0: entered allmulticast mode
Sep 24 16:13:12 pve1 networking[1206]: error: enp1s0f0np0: failed to set vid `{2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,

---> Goes up to 4096 ... Just don't want to mess up the Post here.

So I am really uncertain what to do know. Any ideas or suggestions?

Thanks Alex
 
Last edited:
Hi Everyone,

I am also facing these issues and tried to find a solution but it seems to cause issues. Maybe someone can help.

As advices I updated to the latest version
Code:
root@pve1:~# uname -r
6.8.12-15-pve

This is my network card
Code:
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

This is my network config
Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

iface enxbe3af2b6059f inet manual

iface eno2 inet manual


iface enp1s0f0np0 inet manual
post-up /sbin/ethtool -K enp1s0f0np0 rxvlan off
post-up /sbin/ethtool -K enp1s0f0np0  rx-vlan-offload off

iface enp1s0f1np1 inet manual
post-up /sbin/ethtool -K enp1s0f1np1 rxvlan off
post-up /sbin/ethtool -K enp1s0f1np1  rx-vlan-offload off

#auto vmbr10
#iface vmbr10 inet static
#       address 172.16.98.50/24
#       gateway 172.16.98.254
#       bridge-ports eno2
#       bridge-stp off
#       bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp1s0f0np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4096

auto vmbr0.150
iface vmbr0.150 inet static
        address 172.20.150.10/24
        gateway 172.20.150.254


Everything works as expected until I switch on the VLAN Awareness. (I need that in my network)
As I want to be flexible adding VLANs I have to use the whole range --> 2-4096 (Maybe I can leave >1500 out but same error)

So I understood that when using this I have to disable the VLAN Offloading and therefore added the Post up commands in the config.



I did that rebooted the system but then see this error in the log:
Code:
Sep 24 16:13:12 pve1 kernel: i40e 0000:01:00.0 enp1s0f0np0: entered allmulticast mode
Sep 24 16:13:12 pve1 networking[1206]: error: enp1s0f0np0: failed to set vid `{2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,

---> Goes up to 4096 ... Just don't want to mess up the Post here.

So I am really uncertain what to do know. Any ideas or suggestions?

Thanks Alex
1. Your interface configuration looks very strange to me. I think that it needs to be re-done.
2. Flexibility to assign a new VLAN is good, but do you really need 4094 options? How many do you really have?
3. VLAN Hardware Offload should be set in the actual hardware NIC/Interface's settings, not the OS. Usually you want hardware offloading for throughput.... but these Intel 710 interfaces are not Beasts - they are cheap, that's why everything has them ;-) If you truly need thousands of VLANs you should buy a MUCH better NIC.
 
Hi,

thank you very much for your quick answer.

1. Your interface configuration looks very strange to me. I think that it needs to be re-done.

Can you please share which lines look strange? That would help a lot. If you mean the post-up then OK I already removed that but the rest should be OK

2. Flexibility to assign a new VLAN is good, but do you really need 4094 options? How many do you really have?
No :-) Of course not. With flexibility I mean sometimes I use higher number for VLANs etc. IT's a test Plattform. That's the only reason. All in All it is not more than 20-30. I can limit that if Hardware does not allow more for sure.

3. VLAN Hardware Offload should be set in the actual hardware NIC/Interface's settings, not the OS. Usually you want hardware offloading for throughput.... but these Intel 710 interfaces are not Beasts - they are cheap, that's why everything has them ;-) If you truly need thousands of VLANs you should buy a MUCH better NIC.
As I said - for testing :-) But good to find out some things here. In serious environments I fully agree - not a good idea :-)

So if you have some more feedback with the config that would help.

Thanks