(7.1) Syslog( /var/log/messages ) worthless

[kromberg]

With a 7.1 I have a 10gb NIC writing out 10,000s of these messages. Can these messages be turned off or is there something that can be done to fix what it is complaining about:


Dec 12 13:48:15 thor kernel: [112353.329938] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:15 thor kernel: [112353.937934] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:17 thor kernel: [112355.345946] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:17 thor kernel: [112355.953927] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:19 thor kernel: [112357.361924] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:19 thor kernel: [112357.969921] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:21 thor kernel: [112359.377884] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:21 thor kernel: [112359.985901] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:23 thor kernel: [112361.393917] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:23 thor kernel: [112362.005902] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:25 thor kernel: [112363.409931] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:25 thor kernel: [112364.017895] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:27 thor kernel: [112365.425901] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:28 thor kernel: [112366.033879] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:29 thor kernel: [112367.441883] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:30 thor kernel: [112368.049876] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:31 thor kernel: [112369.457887] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:32 thor kernel: [112370.065880] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:33 thor kernel: [112371.473892] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:34 thor kernel: [112372.081861] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:35 thor kernel: [112373.493858] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:36 thor kernel: [112374.097893] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:37 thor kernel: [112375.505878] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:38 thor kernel: [112376.113865] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:39 thor kernel: [112377.521867] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:40 thor kernel: [112378.129896] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:41 thor kernel: [112379.537870] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:42 thor kernel: [112380.145877] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:43 thor kernel: [112381.553882] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:44 thor kernel: [112382.161826] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:45 thor kernel: [112383.569861] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:46 thor kernel: [112384.181817] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:47 thor kernel: [112385.585818] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:48 thor kernel: [112386.193848] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:49 thor kernel: [112387.601836] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:50 thor kernel: [112388.209857] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:51 thor kernel: [112389.617839] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:52 thor kernel: [112390.225865] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:53 thor kernel: [112391.637848] ixgbe 0000:88:00.0: Warning firmware error detected FWSM: 0x00000000 Dec 12 13:48:54 thor kernel: [112392.241835] ixgbe 0000:88:00.1: Warning firmware error detected FWSM: 0x00000000

 

[fabian]

I'd try updating the firmware of your NIC - seems like some models shipped with buggy firmware that triggers this check in the kernel..

 

[kromberg]

OK, I have looked around a bit and I can not find the latest firmware for the card: intel 82599. Looking at the loaded firmware, it does look like it is an older version:

root@thor:/usr/src# ethtool -i ens6f0 | grep firmware
firmware-version: 0x800001a5

If I can find the firmware and tool to update it, is it possible to get the kernel source so that I can modify the ixgbe module/driver to turn off the message?

 

[fabian]

the kernel sources are available of course:

https://git.proxmox.com/?p=pve-kernel.git;a=summary

 

[kromberg]

Thanks. I will do that as a last resort as modifying the kernel driver make things more complicated. I am really hoping to track down the latest firmware for the card.

 

[notnulled]

Same problem here ...

[104817.649137] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104819.665172] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104821.681392] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104823.697239] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104825.713273] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104827.729308] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104829.745338] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104831.761375] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104833.777408] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104835.793445] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104837.809473] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104839.825507] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104841.841540] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104843.857588] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104845.873609] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104847.889644] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104849.905672] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104851.921707] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104853.937741] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104855.953824] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104857.973984] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104859.986058] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104862.001867] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104864.017900] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104866.033943] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104868.050152] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104870.066200] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104872.082198] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104874.098063] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104876.114094] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104878.130125] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104880.146157] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104882.162189] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104884.178220] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104886.194252] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104888.210285] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104890.226317] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[104892.242349] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x0000000

Same firmware...

root@pxzen:~# ethtool -i enp5s0f0 | grep firmware
firmware-version: 0x800001a5
root@pxzen:~#

as work-around is download an compile manually the driver from here https://sourceforge.net/projects/e1000/files/ixgbe stable/

 

[kromberg]

I am going to try the latest driver next. Looking at the source code, it looks like it still has the same check and will continue to fill up the log with the warning messages because the card still has the McBroken firmware. I can modify the driver not to log the warning, but with each kernel update/upgrade, you have to rebuild the modified driver.

I think I have found the latest firmware for the card. preboot_26.6.tar.gz I have yet to update a NICs firmware, but it does not look too bad:
1) unpack the tarball
2) cd APPS/BootUtil/Linux_x64
3) chmod +x bootutil64e
4) ./bootutil64e # to list out the ports
5) bootutil64e -UP=PXE -NIC=X BootIMG.FLB

Still researching if I need to supply the -UP switch and/or -NIC options. Next maintenance window I have I am going to give it a try.