OK. I'm not totally sure. Once you compile it and if it all goes fine, then you should be good. Try a reboot again and see if it registers on your system.
PVE 6.0-7 + ixgbe firmware errors
- Thread starter dm15
- Start date
For people who don't think this is causing any issues, I have 2 ports bridged and goes:
192.168.2.100 <=> 192.168.2.1 <=> 192.168.2.200
With B being the machine causing problems.
From 192.168.2.200 a Win10 box before update:
E:\data\iperf3>iperf3.exe -c 192.168.2.1
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.30 GBytes 2.84 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.30 GBytes 2.84 Gbits/sec receiver
E:\data\iperf3>iperf3.exe -c 192.168.2.100
Connecting to host 192.168.2.100, port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 280 MBytes 235 Mbits/sec sender
[ 4] 0.00-10.00 sec 280 MBytes 235 Mbits/sec receiver
2.8Gb going to the interface, 235**Mb** going through it, or an order of magnitude less.
After updating driver:
E:\data\iperf3>iperf3.exe -c 192.168.2.100
Connecting to host 192.168.2.100, port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.37 GBytes 2.90 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.37 GBytes 2.90 Gbits/sec receiver
Going through the bridge is the same speed as going to the bridge node. Thanks so much for these instructions!
192.168.2.100 <=> 192.168.2.1 <=> 192.168.2.200
With B being the machine causing problems.
From 192.168.2.200 a Win10 box before update:
E:\data\iperf3>iperf3.exe -c 192.168.2.1
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.30 GBytes 2.84 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.30 GBytes 2.84 Gbits/sec receiver
E:\data\iperf3>iperf3.exe -c 192.168.2.100
Connecting to host 192.168.2.100, port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 280 MBytes 235 Mbits/sec sender
[ 4] 0.00-10.00 sec 280 MBytes 235 Mbits/sec receiver
2.8Gb going to the interface, 235**Mb** going through it, or an order of magnitude less.
After updating driver:
E:\data\iperf3>iperf3.exe -c 192.168.2.100
Connecting to host 192.168.2.100, port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.37 GBytes 2.90 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.37 GBytes 2.90 Gbits/sec receiver
Going through the bridge is the same speed as going to the bridge node. Thanks so much for these instructions!
You mean that it doesn't build with Proxmox 6.2? Yeah, that's unfortunate but we'll have to wait on Intel to update their driver to work with kernel 5.4.
I'm hoping that it won't be too long. Until then I'm going to force myself to put up with the errors.
Let me know how it works, between that and WSL2 the iperf3 report up thread went from 3g to 10g
That's a pretty amazing increase. I did a quick iperf2 test before and after and the results were similar ~ 9.86Gb/sec for each one. No massive increase for me..
Last edited:
Looks like the Intel ixgbe has been updated to support kernels through 5.6:
https://sourceforge.net/projects/e1000/files/ixgbe stable/5.7.1/
https://sourceforge.net/projects/e1000/files/ixgbe stable/5.7.1/
Sorry to bump an old thread, this issue looks to be back on the latest kernel + driver:
dmesg is showing:
This is with a Dell branded X520-T2
Anyone else seeing this?
Bash:
root@proxmox1:~# pveversion
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.11-7-pve)
Bash:
root@proxmox1:~# ethtool -i ens1f0
driver: ixgbe
version: 5.19.9
firmware-version: 0x800001a5
expansion-rom-version:
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yesdmesg is showing:
Bash:
...
[ 1523.104064] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
[ 1525.120072] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
[ 1527.136088] ixgbe 0000:07:00.0: Warning firmware error detected FWSM: 0x00000000
...This is with a Dell branded X520-T2
Anyone else seeing this?
Last edited:
It looks like TrueNAS team found this is likely a bug in the Intel firmware not setting a bit after boot: https://ixsystems.atlassian.net/browse/NAS-121174
There's a edit they did to the Intel driver to suppress the messages after the first one here: https://github.com/truenas/linux/pull/81/commits/b42356c9b0419af14df988c9970e667e91cc075a
Had a quick read of that earlier today, sounds like they silenced the repeated warnings rather than fixing a bug per se. Seems unclear if this is a red herring error warning or indicative of any real issues (probably worth you doing some iperf3 tests to see if you have any poor performance.)
Edit: probably also worth double checking the error warnings are only appearing in the latest kernel too and looking through the git history for the driver to look for anything obvious.
It would seem a sensible patch to include in Proxmox, but won't address the root cause of the firmware error (whether this is a red herring or not.)
Edit: probably also worth double checking the error warnings are only appearing in the latest kernel too and looking through the git history for the driver to look for anything obvious.
It would seem a sensible patch to include in Proxmox, but won't address the root cause of the firmware error (whether this is a red herring or not.)
Last edited:
I am seeing exactly the same flood of warnings with two X520-T2 cards on an old Dell R710 running PVE 8.3.4 and kernel 6.8.12.
Code:
root@pve1:~# pveversion
pve-manager/8.3.4/65224a0f9cd294a3 (running kernel: 6.8.12-8-pve)
root@pve1:~# dmesg | tail -4
[18376.827853] ixgbe 0000:06:00.1: Warning firmware error detected FWSM: 0x00000000
[18377.147846] ixgbe 0000:06:00.0: Warning firmware error detected FWSM: 0x00000000
[18377.467811] ixgbe 0000:09:00.1: Warning firmware error detected FWSM: 0x00000000
[18377.659743] ixgbe 0000:09:00.0: Warning firmware error detected FWSM: 0x00000000
root@pve1:~# ethtool -i enp6s0f0
driver: ixgbe
version: 6.8.12-8-pve
firmware-version: 0x800001a5
expansion-rom-version:
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@pve1:~# ethtool -i enp9s0f0
driver: ixgbe
version: 6.8.12-8-pve
firmware-version: 0x80000135
expansion-rom-version:
bus-info: 0000:09:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yesIn the meantime I upgraded to PVE 9.0.6 running kernel 6.14.11 but the "
Warning firmware error detected FWSM" messages from the in-tree ixgbe kernel module kept coming.
Code:
root@pve1:~# pveversion && journalctl --since today | fgrep 'Warning firmware error detected FWSM' | tail -2
pve-manager/9.0.6/49c767b70aeb6648 (running kernel: 6.14.11-1-pve)
Sep 08 01:02:08 pve1 kernel: ixgbe 0000:09:00.0: Warning firmware error detected FWSM: 0x00000000
Sep 08 01:02:08 pve1 kernel: ixgbe 0000:06:00.0: Warning firmware error detected FWSM: 0x00000000-Here are the references I used:
https://www.paulcostan.com/2024/09/how-to-fix-proxmox-pve-kernel-ixgbe-warning-firmware-error-detected-fwsm/
https://github.com/intel/ethernet-linux-ixgbe/ (see README > To manually build the driver)
Fast track on PVE 9.0.6:
Code:
# apt install pve-headers build-essential
# wget https://github.com/intel/ethernet-linux-ixgbe/releases/download/v6.2.5/ixgbe-6.2.5.tar.gz
# tar xvpf ixgbe-6.2.5.tar.gz
# make install
# modinfo ixgbe | head -2 # check version
# systemctl rebootUpdate: PVE 9.1 kernel 6.17.4-2-pve requires ixgbe-6.3.4 or later. With older driver versions you'll run into a build error.
Just thought I'd chime in as I hadn't been looking at the console of my proxmox server very closely but was doing some stuff today where I noticed the page was just getting the same firmware error message listed that this thread addressed. Running pve-manager 9.1.6, kernel 6.17.13-1-pve here (pveversion) and I had done the fix previously, but along the way with updates I guess it was lost or overwritten. It worked though! As of now the server has been up for a few hours and I haven't seen a single error and I was getting them every few seconds before I applied the fix. Anyway...
I used the ixgbe-6.3.4 that Stefan pointed to just above me and followed the directions to build, install, and assign the correct drivers. I didn't do the embedded reboot, but mainly because that NIC isn't my main network interface to proxmox so I was able to use more conventional ways to ensure everything was working. Just to recap and give someone else something to work from, here's what I issued for commands to fix things:
Create directory to work in and download the release I did:
That's it! After these steps were complete I did reboot the server just to ensure it all had a clean start and I could monitor for any errors. Now that it's been a few hours I feel confident saying that this fixed my issue.
Also I'll add that I ran this a few times just to check in on current driver being used and to ensure the new one was in use now:
Thank you very much everyone that contributed to dealing with this issue! I'm lucky that I don't use this as my primary interface for anything as it's really just a backup way to access proxmox as well as some failover iSCSI pathing that could be used by it.
I used the ixgbe-6.3.4 that Stefan pointed to just above me and followed the directions to build, install, and assign the correct drivers. I didn't do the embedded reboot, but mainly because that NIC isn't my main network interface to proxmox so I was able to use more conventional ways to ensure everything was working. Just to recap and give someone else something to work from, here's what I issued for commands to fix things:
Create directory to work in and download the release I did:
Enable the use of the new driver for the target NICs, which is enp10 on my system but this WILL differ for you:
This runs two commands, unloads the current driver for the ixgbe interface and then immediately loads the new one
That's it! After these steps were complete I did reboot the server just to ensure it all had a clean start and I could monitor for any errors. Now that it's been a few hours I feel confident saying that this fixed my issue.
Also I'll add that I ran this a few times just to check in on current driver being used and to ensure the new one was in use now:
Thank you very much everyone that contributed to dealing with this issue! I'm lucky that I don't use this as my primary interface for anything as it's really just a backup way to access proxmox as well as some failover iSCSI pathing that could be used by it.
Last edited: