PVE 6.0-7 + ixgbe firmware errors

For people who don't think this is causing any issues, I have 2 ports bridged and goes:
192.168.2.100 <=> 192.168.2.1 <=> 192.168.2.200
With B being the machine causing problems.

From 192.168.2.200 a Win10 box before update:
E:\data\iperf3>iperf3.exe -c 192.168.2.1
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.30 GBytes 2.84 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.30 GBytes 2.84 Gbits/sec receiver

E:\data\iperf3>iperf3.exe -c 192.168.2.100
Connecting to host 192.168.2.100, port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 280 MBytes 235 Mbits/sec sender
[ 4] 0.00-10.00 sec 280 MBytes 235 Mbits/sec receiver
2.8Gb going to the interface, 235**Mb** going through it, or an order of magnitude less.

After updating driver:
E:\data\iperf3>iperf3.exe -c 192.168.2.100
Connecting to host 192.168.2.100, port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.37 GBytes 2.90 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.37 GBytes 2.90 Gbits/sec receiver

Going through the bridge is the same speed as going to the bridge node. Thanks so much for these instructions!
 
  • Like
Reactions: stark
Just a note that this does not build with Proxmox 6.2 and the errors are back. The most recent driver from Intel is only tested up to kernel 5.3 not 5.4. There's a few compile errors that I'm trying to work through.
 
Last edited:
DougW said:
Just a note that this does not build with Proxmox 6.1 and the errors are back. The most recent driver from Intel is only tested up to kernel 5.3 not 5.4. There's a few compile errors that I'm trying to work through.
Click to expand...

You mean that it doesn't build with Proxmox 6.2? Yeah, that's unfortunate but we'll have to wait on Intel to update their driver to work with kernel 5.4.

I'm hoping that it won't be too long. Until then I'm going to force myself to put up with the errors.
 
stark said:
You mean that it doesn't build with Proxmox 6.2? Yeah, that's unfortunate but we'll have to wait on Intel to update their driver to work with kernel 5.4.

I'm hoping that it won't be too long. Until then I'm going to force myself to put up with the errors.
Click to expand...

Yes, I updated my post.
 
DougW said:
Let me know how it works, between that and WSL2 the iperf3 report up thread went from 3g to 10g
Click to expand...
That's a pretty amazing increase. I did a quick iperf2 test before and after and the results were similar ~ 9.86Gb/sec for each one. No massive increase for me.. :( :)
 
Last edited:
Sorry to bump an old thread, this issue looks to be back on the latest kernel + driver:

Bash: 
root@proxmox1:~# pveversion
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.11-7-pve)

Bash: 
root@proxmox1:~# ethtool -i ens1f0
driver: ixgbe
version: 5.19.9
firmware-version: 0x800001a5
expansion-rom-version:
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

dmesg is showing:

Bash: 
...
[ 1523.104064] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
[ 1525.120072] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
[ 1527.136088] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
...

This is with a Dell branded X520-T2
Anyone else seeing this?
 
Last edited:
Had that happen sometimes and it be cleared by a reseat / wipe down of the PCIe pins with isopropyl alcohol.

Give that a try and see if it clears up.
 
Aterfax said:
Had that happen sometimes and it be cleared by a reseat / wipe down of the PCIe pins with isopropyl alcohol.

Give that a try and see if it clears up.
Click to expand...
It looks like TrueNAS team found this is likely a bug in the Intel firmware not setting a bit after boot: https://ixsystems.atlassian.net/browse/NAS-121174

There's a edit they did to the Intel driver to suppress the messages after the first one here: https://github.com/truenas/linux/pull/81/commits/b42356c9b0419af14df988c9970e667e91cc075a
 
Had a quick read of that earlier today, sounds like they silenced the repeated warnings rather than fixing a bug per se. Seems unclear if this is a red herring error warning or indicative of any real issues (probably worth you doing some iperf3 tests to see if you have any poor performance.)

Edit: probably also worth double checking the error warnings are only appearing in the latest kernel too and looking through the git history for the driver to look for anything obvious.

It would seem a sensible patch to include in Proxmox, but won't address the root cause of the firmware error (whether this is a red herring or not.)
 
Last edited:
I have the same issue with latest proxmox 6.8.4-3
I will try the fix, but with the latest drivers found here
I will reply back if the errors have gone away
 
imasonaz said:
Sorry to bump an old thread, this issue looks to be back on the latest kernel + driver:

Bash: 
root@proxmox1:~# pveversion
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.11-7-pve)

Bash: 
root@proxmox1:~# ethtool -i ens1f0
driver: ixgbe
version: 5.19.9
firmware-version: 0x800001a5
expansion-rom-version:
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

dmesg is showing:

Bash: 
...
[ 1523.104064] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
[ 1525.120072] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
[ 1527.136088] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
...

This is with a Dell branded X520-T2
Anyone else seeing this?
Click to expand...
I am seeing exactly the same flood of warnings with two X520-T2 cards on an old Dell R710 running PVE 8.3.4 and kernel 6.8.12.
Code: 
root@pve1:~# pveversion
pve-manager/8.3.4/65224a0f9cd294a3 (running kernel: 6.8.12-8-pve)
root@pve1:~# dmesg | tail -4
[18376.827853] ixgbe 0000:06:00.1: Warning firmware error detected FWSM: 0x00000000
[18377.147846] ixgbe 0000:06:00.0: Warning firmware error detected FWSM: 0x00000000
[18377.467811] ixgbe 0000:09:00.1: Warning firmware error detected FWSM: 0x00000000
[18377.659743] ixgbe 0000:09:00.0: Warning firmware error detected FWSM: 0x00000000
root@pve1:~# ethtool -i enp6s0f0
driver: ixgbe
version: 6.8.12-8-pve
firmware-version: 0x800001a5
expansion-rom-version:
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@pve1:~# ethtool -i enp9s0f0
driver: ixgbe
version: 6.8.12-8-pve
firmware-version: 0x80000135
expansion-rom-version:
bus-info: 0000:09:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
 
You must log in or register to reply here.
Top Bottom