PVE 6.0-7 + ixgbe firmware errors

[dm15]

Since yesterday I am getting the following error spam in kernel messages:

[44650.577580] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[44650.741545] ixgbe 0000:05:00.1: Warning firmware error detected FWSM: 0x00000000
[44652.593549] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[44652.753532] ixgbe 0000:05:00.1: Warning firmware error detected FWSM: 0x00000000
[44654.609580] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[44654.769600] ixgbe 0000:05:00.1: Warning firmware error detected FWSM: 0x00000000
[44656.625582] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[44656.785601] ixgbe 0000:05:00.1: Warning firmware error detected FWSM: 0x00000000
[44658.641640] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000
[44658.801585] ixgbe 0000:05:00.1: Warning firmware error detected FWSM: 0x00000000
[44660.657626] ixgbe 0000:05:00.0: Warning firmware error detected FWSM: 0x00000000

The NICs (05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)) randomly hang without any state changes, `ip link set ... up` brings them back without reboot.

The current PVE ixgbe driver looks a little bit outdated

# ethtool -i enp5s0f0
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x000118b3

because the lastest version is 5.6.3, which is far away from being actual.

Also got the following errors on the login screen today



I think all those indicate that ixgbe driver included in the current kernel is outdated (5.0.21-2-pve #1 SMP PVE 5.0.21-6 (Fri, 27 Sep 2019 17:17:02 +0200) x86_64 GNU/Linux)

Is there any possibility you can ship an updated ixgbe driver within next update?

related: These NICs on CentOS with the same ixgbe driver version have same behaviour - https://bugs.centos.org/view.php?id=16495

p.s. just noticed that this could be a wrong forum to report this and 'Networking and firewall' would be more appropriate, moderators please move my topic if you feel like

 

[dm15]

I've just compiled the latest version of ixgbe driver from the Intel download site, installed it and rebooted the server, will report here if the firmware errors return.

# ethtool -i enp5s0f0
driver: ixgbe
version: 5.6.3
firmware-version: 0x18b30001
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

 

[stark]

Since yesterday I am getting the following error spam in kernel messages:




The NICs (05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)) randomly hang without any state changes, `ip link set ... up` brings them back without reboot.

The current PVE ixgbe driver looks a little bit outdated



because the lastest version is 5.6.3, which is far away from being actual.

Also got the following errors on the login screen today

View attachment 12067

I think all those indicate that ixgbe driver included in the current kernel is outdated (5.0.21-2-pve #1 SMP PVE 5.0.21-6 (Fri, 27 Sep 2019 17:17:02 +0200) x86_64 GNU/Linux)

Is there any possibility you can ship an updated ixgbe driver within next update?

related: These NICs on CentOS with the same ixgbe driver version have same behaviour - https://bugs.centos.org/view.php?id=16495

p.s. just noticed that this could be a wrong forum to report this and 'Networking and firewall' would be more appropriate, moderators please move my topic if you feel like

I get these errors as well. Though, for me, they don't appear to be causing any issues. I just filter them out so they don't make it to my logfiles and just put up with the syslog spam.

 

[stark]

I went ahead and compiled the driver from source and the errors went away. Have you thought about installing DKMS to keep the driver current with each new kernel release?

 

[homozavrus]

i Have the same error on this network-card. but in my case, the interfaces and Ceph monitors fall.

 

[dm15]

i Have the same error on this network-card. but in my case, the interfaces and Ceph monitors fall.

Yeah was my case as well. Compiling & installing this driver resolves the problem, proofed by me (reply #2) & stark (reply #4)

Have you thought about installing DKMS to keep the driver current with each new kernel release?

No, I did it on my production machine and was afraid installing with DKMS would break something... I will probably test it on a non-production machine and report back.

 

[vertoz]

I've just compiled the latest version of ixgbe driver from the Intel download site, installed it and rebooted the server, will report here if the firmware errors return.

Hi, Could you help me with the steps for this activity, we've got the same issue. Thank you.

 

[dthompson]

I'm also seeing these same issues on my own card since updating to the latest 6.1 release from 5.4 yesterday on my production servers.

 

[dthompson]

This is what I did to get around the issue:

1.) cd /tmp
2.) Download the link from above (also here)
3.) tar -xvpf ixgbe-5.6.5.tar.gz
4.) cd ixgbe-5.6.5/src
5.) apt install pve-headers build-essential
6.) make install

As per the read me notes, the driver gets installed in the following location:
/lib/modules/5.3.13-1-pve/updates/drivers/net/ethernet/intel/ixgbe

7.) modinfo ixgbe
8.) modprobe ixgbe [parameter=port1_value,port2_value]
9.) rmmod ixgbe; modprobe ixgbe
10.) ifdown <NETWORK Interface>

In my case, it was the following on one of my servers:
ifdown enp3s0f0
then
ifup enp3s0f0

Repeat for your other Nic(s) that are also having the same issue

11.) update-initramfs -u

Reboot and all should be good and the error messages should be gone from the syslog. This was done on a fairly old super micro server that I have with a PCI Intel Ethernet Controller X540 card. I have of the these servers and this fix as supplied be the previous posters solved me issue.

I think that once you install the build-essentials and the pve-headers that they have to be kept manually updated after each update on the base system but someone from the Proxmox team, or someone who knows more about this than me might be able to help more.

Its fairly straight forward to do. All I did before I started was migrated my VM's over to other servers in case something went wrong and once it all came back up, I migrated them back.

 

[dthompson]

I'm not seeing this issue pop up again since running an update to my node. I now have

pveversion 
pve-manager/6.1-7/13e58d5e (running kernel: 5.3.13-2-pve)

This error is now popping up on my 2 nodes:
ixgbe 0000:03:00.0: Warning firmware error detected FWSM: 0x00000000

Trying to run the make again provides me with this error:

common.mk:84: *** Kernel header files not in any of the expected locations.
common.mk:85: *** Install the appropriate kernel development package, e.g.
common.mk:86: *** kernel-devel, for building kernel modules and try again.  Stop.

I'm sure this is a fairly easy fix, but I'm not sure what it is I need to get the proper kernel environment installed for this in order to resolve the issue. My previous solution is no longer working for this.

Thanks

 

[dthompson]

Never mind, I found the solution to my issue. After running an update to the hosts, I needed to reboot the hosts to see all the new header information.

Sometimes I'm amazed that I have the brain capacity to tie my own shoes given how stupid I am.

 

[AlexSSP]

Hi! You fix this issue? I want to update ixgbe driver, but i've got the same issue. Thank you.

 

[dthompson]

Yes. See post #9 in this thread for the resolution.

 

[AlexSSP]

I try it, but after 6 step i've got
common.mk:84: *** Kernel header files not in any of the expected locations. common.mk:85: *** Install the appropriate kernel development package, e.g. common.mk:86: *** kernel-devel, for building kernel modules and try again. Stop

 

[dthompson]

I try it, but after 6 step i've got
common.mk:84: *** Kernel header files not in any of the expected locations. common.mk:85: *** Install the appropriate kernel development package, e.g. common.mk:86: *** kernel-devel, for building kernel modules and try again. Stop

Did you install install this:

pve-headers build-essential

You might need to reboot your host once you have done this in order for it to see them

 

[AlexSSP]

Oh, yes.

build-essential is already the newest version (12.6).
pve-headers is already the newest version (6.1-2).

But after make install still

common.mk:84: *** Kernel header files not in any of the expected locations.
common.mk:85: *** Install the appropriate kernel development package, e.g.
common.mk:86: *** kernel-devel, for building kernel modules and try again. Stop.

 

[dthompson]

Oh, yes.

But after make install still

Did you try and run a

apt dist-upgrade

to see if that helps you?

 

[AlexSSP]

apt dist-upgrade

It's help me. Thank you.
But now I have a new problem - no link

ethtool ens3f0
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no

ethtool -i ens3f0
driver: ixgbe
version: 5.6.5
firmware-version: 0x800000cb
expansion-rom-version:
bus-info: 0000:19:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

ethtool -m ens3f0
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x00 0x00 0x00 0x06 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Transceiver type : FC: Multimode, 50um (M5)
Encoding : 0x03 (NRZ)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 300m
Length (62.5um) : 300m
Length (Copper) : 0m
Length (OM3) : 300m
Laser wavelength : 850nm
Vendor name : FIBO
Vendor OUI : 00:00:00
Vendor PN : FT-S10-M8503LD
Vendor rev :
Option values : 0x00 0x1a
Option : RX_LOS implemented
Option : TX_FAULT implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : SO19101910
Date code : 191013
Optical diagnostics support : Yes
Laser bias current : 6.552 mA
Laser output power : 0.4149 mW / -3.82 dBm
Receiver signal average optical power : 0.0001 mW / -40.00 dBm
Module temperature : 21.96 degrees C / 71.52 degrees F
Module voltage : 3.3311 V
Alarm/warning flags implemented : Yes
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : On
Laser rx power high warning : Off
Laser rx power low warning : On
Laser bias current high alarm threshold : 15.000 mA
Laser bias current low alarm threshold : 0.000 mA
Laser bias current high warning threshold : 12.000 mA
Laser bias current low warning threshold : 1.000 mA
Laser output power high alarm threshold : 1.2589 mW / 1.00 dBm
Laser output power low alarm threshold : 0.1259 mW / -9.00 dBm
Laser output power high warning threshold : 1.0000 mW / 0.00 dBm
Laser output power low warning threshold : 0.1585 mW / -8.00 dBm
Module temperature high alarm threshold : 75.00 degrees C / 167.00 degrees F
Module temperature low alarm threshold : -5.00 degrees C / 23.00 degrees F
Module temperature high warning threshold : 70.00 degrees C / 158.00 degrees F
Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
Module voltage high alarm threshold : 3.6000 V
Module voltage low alarm threshold : 3.0000 V
Module voltage high warning threshold : 3.5000 V
Module voltage low warning threshold : 3.1000 V
Laser rx power high alarm threshold : 1.2589 mW / 1.00 dBm
Laser rx power low alarm threshold : 0.0431 mW / -13.66 dBm
Laser rx power high warning threshold : 1.0000 mW / 0.00 dBm
Laser rx power low warning threshold : 0.0794 mW / -11.00 dBm

This optical cables is definitely working. This SFP+ is supported on the switch too, but no link detected.

 

[dthompson]

I'm not sure, but you might need different firmware that what I used for my 10GB Cards. The firmware that I pointed to was for a RJ-45 10GB Intel card.

The supported cards are:
The driver is compatible with devices based on the following:


* Intel(R) Ethernet Controller 82598
* Intel(R) Ethernet Controller 82599
* Intel(R) Ethernet Controller X520
* Intel(R) Ethernet Controller X540
* Intel(R) Ethernet Controller x550
* Intel(R) Ethernet Controller X552
* Intel(R) Ethernet Controller X553

 

[AlexSSP]

This driver is compatible with my device

lspci | grep net
19:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
19:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)