[SOLVED] Supermicro AOC-STGN-i1S 10G SFP+ E157872 (Intel x520-DA1) doesn't work with proxmox (kernel 5.15.30)

[nk1]

Hi guys,

I just bought the above NIC (basically the intel x520) and can't get it to work with proxmox 7.1-7 (kernel 5.15.30)

lspci:

04:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Network Connection (rev 01)
08:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

dmesg | grep ixgbe
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.30-1-pve root=/dev/mapper/pve-root ro ixgbe.allow_unsupported_sfp=1 quiet intel_iommu=on iommu=pt
[ 0.718320] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.30-1-pve root=/dev/mapper/pve-root ro ixgbe.allow_unsupported_sfp=1 quiet intel_iommu=on iommu=pt
[ 3.813768] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 3.813772] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 3.874588] ixgbe 0000:04:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 3.874644] ixgbe 0000:04:00.0: Reload the driver after installing a supported module.


root@vdi:/tmp/ixgbe-5.12.5/src# modinfo ixgbe
filename: /lib/modules/5.15.30-1-pve/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
license: GPL v2
description: Intel(R) 10 Gigabit PCI Express Network Driver
author: Intel Corporation, <linux.nics@intel.com>
srcversion: 03B2CCD01D2EFB0B41F5150
alias: pci:v00008086d000015E5sv*sd*bc*sc*i*
alias: pci:v00008086d000015E4sv*sd*bc*sc*i*
alias: pci:v00008086d000015CEsv*sd*bc*sc*i*
alias: pci:v00008086d000015C8sv*sd*bc*sc*i*
alias: pci:v00008086d000015C7sv*sd*bc*sc*i*
alias: pci:v00008086d000015C6sv*sd*bc*sc*i*
alias: pci:v00008086d000015C4sv*sd*bc*sc*i*
alias: pci:v00008086d000015C3sv*sd*bc*sc*i*
alias: pci:v00008086d000015C2sv*sd*bc*sc*i*
alias: pci:v00008086d000015AEsv*sd*bc*sc*i*
alias: pci:v00008086d000015ACsv*sd*bc*sc*i*
alias: pci:v00008086d000015ADsv*sd*bc*sc*i*
alias: pci:v00008086d000015ABsv*sd*bc*sc*i*
alias: pci:v00008086d000015B0sv*sd*bc*sc*i*
alias: pci:v00008086d000015AAsv*sd*bc*sc*i*
alias: pci:v00008086d000015D1sv*sd*bc*sc*i*
alias: pci:v00008086d00001563sv*sd*bc*sc*i*
alias: pci:v00008086d00001560sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Asv*sd*bc*sc*i*
alias: pci:v00008086d00001557sv*sd*bc*sc*i*
alias: pci:v00008086d00001558sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Fsv*sd*bc*sc*i*
alias: pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias: pci:v00008086d00001528sv*sd*bc*sc*i*
alias: pci:v00008086d000010F8sv*sd*bc*sc*i*
alias: pci:v00008086d0000151Csv*sd*bc*sc*i*
alias: pci:v00008086d00001529sv*sd*bc*sc*i*
alias: pci:v00008086d0000152Asv*sd*bc*sc*i*
alias: pci:v00008086d000010F9sv*sd*bc*sc*i*
alias: pci:v00008086d00001514sv*sd*bc*sc*i*
alias: pci:v00008086d00001507sv*sd*bc*sc*i*
alias: pci:v00008086d000010FBsv*sd*bc*sc*i*
alias: pci:v00008086d00001517sv*sd*bc*sc*i*
alias: pci:v00008086d000010FCsv*sd*bc*sc*i*
alias: pci:v00008086d000010F7sv*sd*bc*sc*i*
alias: pci:v00008086d00001508sv*sd*bc*sc*i*
alias: pci:v00008086d000010DBsv*sd*bc*sc*i*
alias: pci:v00008086d000010F4sv*sd*bc*sc*i*
alias: pci:v00008086d000010E1sv*sd*bc*sc*i*
alias: pci:v00008086d000010F1sv*sd*bc*sc*i*
alias: pci:v00008086d000010ECsv*sd*bc*sc*i*
alias: pci:v00008086d000010DDsv*sd*bc*sc*i*
alias: pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias: pci:v00008086d000010C8sv*sd*bc*sc*i*
alias: pci:v00008086d000010C7sv*sd*bc*sc*i*
alias: pci:v00008086d000010C6sv*sd*bc*sc*i*
alias: pci:v00008086d000010B6sv*sd*bc*sc*i*
depends: mdio,dca,xfrm_algo
retpoline: Y
intree: Y
name: ixgbe
vermagic: 5.15.30-1-pve SMP mod_unload modversions
parm: max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated) (uint)
parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint)
parm: debugebug level (0=none,...,16=all) (int)

When I tried to compile intel driver, I have a bunch of errors:

root@vdi:/tmp/ixgbe-5.12.5/src# make
*** The target kernel has CONFIG_MODULE_SIG_ALL enabled, but
*** the signing key cannot be found. Module signing has been
*** disabled for this build.
make[1]: Entering directory '/usr/src/linux-headers-5.15.30-1-pve'
CC [M] /tmp/ixgbe-5.12.5/src/ixgbe_ethtool.o
/tmp/ixgbe-5.12.5/src/ixgbe_ethtool.c:4732:19: error: initialization of ‘int (*)(struct net_device *, struct ethtool_coalesce *, struct kernel_ethtool_coalesce *, struc t netlink_ext_ack *)’ from incompatible pointer type ‘int (*)(struct net_device *, struct ethtool_coalesce *)’ [-Werror=incompatible-pointer-types]
4732 | .get_coalesce = ixgbe_get_coalesce,
| ^~~~~~~~~~~~~~~~~~
/tmp/ixgbe-5.12.5/src/ixgbe_ethtool.c:4732:19: note: (near initialization for ‘ixgbe_ethtool_ops.get_coalesce’)
/tmp/ixgbe-5.12.5/src/ixgbe_ethtool.c:4733:19: error: initialization of ‘int (*)(struct net_device *, struct ethtool_coalesce *, struct kernel_ethtool_coalesce *, struc t netlink_ext_ack *)’ from incompatible pointer type ‘int (*)(struct net_device *, struct ethtool_coalesce *)’ [-Werror=incompatible-pointer-types]
4733 | .set_coalesce = ixgbe_set_coalesce,
| ^~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:285: /tmp/ixgbe-5.12.5/src/ixgbe_ethtool.o] Error 1
make[1]: *** [Makefile:1875: /tmp/ixgbe-5.12.5/src] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.15.30-1-pve'
make: *** [Makefile:99: default] Error 2



root@vdi:/tmp/ixgbe-5.12.5/src# pveversion
pve-manager/7.1-7/df5740ad (running kernel: 5.15.30-1-pve)

ip a only shows the 4 built-in nics
root@vdi:/tmp/ixgbe-5.12.5/src# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 00:25:90:fe:01:24 brd ff:ff:ff:ff:ff:ff
altname enp8s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:25:90:fe:01:25 brd ff:ff:ff:ff:ff:ff
altname enp8s0f1
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:25:90:fe:01:26 brd ff:ff:ff:ff:ff:ff
altname enp8s0f2
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:25:90:fe:01:27 brd ff:ff:ff:ff:ff:ff
altname enp8s0f3
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:25:90:fe:01:24 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.199/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::225:90ff:fefe:124/64 scope link
valid_lft forever preferred_lft forever


Can someone help me out?

Thanks
nk1

 

[Stoiko Ivanov]

3.874588] ixgbe 0000:04:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.

parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint)

as the output of `dmesg` says - the ixgbe driver (in some versions) is quite picky about the SFP modules you use in that card:
* either use a supported SFP+ module
* or add a /etc/modprobe.d/ixgbe.conf file with the following:

options ixgbe allow_unsupported_sfp=1

(and recreate the initramfs and reload the module)
(no need for the out-of-tree driver)

I hope this helps!

 

[mvs]

It is strange that you get unsupported SFP+ error on Supermicro card. This check is usually disabled on OEM cards.

You can modify OEM flag in EEPROM with this tool
https://gist.github.com/ixs/dbaac42730dea9bd124f26cbd439c58e

 

[nk1]

as the output of `dmesg` says - the ixgbe driver (in some versions) is quite picky about the SFP modules you use in that card:
* either use a supported SFP+ module
* or add a /etc/modprobe.d/ixgbe.conf file with the following:

options ixgbe allow_unsupported_sfp=1

(and recreate the initramfs and reload the module)
(no need for the out-of-tree driver)

I hope this helps!

Yes I did all of that.

root@vdi:/etc/modprobe.d# cat ixgbe.conf
options ixgbe allow_unsupported_sfp=1

At the end of my /etc/default/grub file also has
GRUB_CMDLINE_LINUX="ixgbe.allow_unsupported_sfp=1"

depmod -a
update-initramfs -u and everything else, multiple reboot etc.

The nic just doesn't show up in the interfaces file. The real issue is that I can't compile a nother version of the driver from intel site.

 

[nk1]

It is strange that you get unsupported SFP+ error on Supermicro card. This check is usually disabled on OEM cards.

You can modify OEM flag in EEPROM with this tool
https://gist.github.com/ixs/dbaac42730dea9bd124f26cbd439c58e

How do I use that? it needs an interface and I can't tell what the interface my nic is.

"ip a" doesn't show the nic at all.

 

[Stoiko Ivanov]

Yes I did all of that.

root@vdi:/etc/modprobe.d# cat ixgbe.conf
options ixgbe allow_unsupported_sfp=1

At the end of my /etc/default/grub file also has
GRUB_CMDLINE_LINUX="ixgbe.allow_unsupported_sfp=1"

sorry - I overread that part in your initial post.

if you have physical access to the machine - try removing the SFP+ module and unload+load the driver
(this should make the interface show up in `ip link` output)

do you get any more information if you load the module with debug set to 16?
(modprobe ixgbe debug=16 allow_unsupported_sfp=1)?

The real issue is that I can't compile a nother version of the driver from intel site.

This does happen now and then if the out-of-tree driver published by intel is not updated to be compatible with newer kernels
(which is why we recommend to work with the in-tree module)

 

[nk1]

sorry - I overread that part in your initial post.

if you have physical access to the machine - try removing the SFP+ module and unload+load the driver
(this should make the interface show up in `ip link` output)

do you get any more information if you load the module with debug set to 16?
(modprobe ixgbe debug=16 allow_unsupported_sfp=1)?



This does happen now and then if the out-of-tree driver published by intel is not updated to be compatible with newer kernels
(which is why we recommend to work with the in-tree module)

I do have physical access to the machine. I'll try removing the transceiver from the NIC and will do the unload/reload the driver later tonight.

Right now when I do
rmmod ixgbe
then
modprobe ixgbe debug=16 allow_unsupported_sfp=1

nothing changes, I still get the same as before for dmesg | grep ixgbe

"ip link" still doesn't show the interface

thanks

 

[mvs]

How do I use that? it needs an interface and I can't tell what the interface my nic is.

"ip a" doesn't show the nic at all.

You need to detach unsupported SFP modules from your NIC and reload ixgbe kernel module (or reboot server).
You can also try to boot older kernel.

 

[nk1]

You need to detach unsupported SFP modules from your NIC and reload ixgbe kernel module (or reboot server).
You can also try to boot older kernel.

Got it. I understand your thinking now. The SPF+ module (transceiver) may not be compatible with the NIC preventing the NIC from showing up. Will try tonight.

Thanks.

 

[nk1]

You need to detach unsupported SFP modules from your NIC and reload ixgbe kernel module (or reboot server).
You can also try to boot older kernel.

Update,

after pulling the SPF+ module, unload (rmmod ixgbe) and load the module (modprobe ixgbe debug=16 allow_unsupported_sfp=1), the nic shows up in ip link.

root@vdi:/etc/network# ethtool -i enp4s0
driver: ixgbe
version: 5.15.30-1-pve
firmware-version: 0x800004f4
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Checking for SPF+ support gives the "unlock" status (fd).

root@vdi:/etc/network# ethtool -e enp4s0 offset 0x58 length 1
Offset Values
------ ------
0x0058: fd

plug the Spf+ module back in and I am back to square one.

 

[nk1]

sorry - I overread that part in your initial post.

if you have physical access to the machine - try removing the SFP+ module and unload+load the driver
(this should make the interface show up in `ip link` output)

do you get any more information if you load the module with debug set to 16?
(modprobe ixgbe debug=16 allow_unsupported_sfp=1)?



This does happen now and then if the out-of-tree driver published by intel is not updated to be compatible with newer kernels
(which is why we recommend to work with the in-tree module)

Please see my reply to mvs

Removing the SPF+ module and the nic shows up properly. Plug it back in and am I back to square one.

 

[mvs]

Checking for SPF+ support gives the "unlock" status (fd).

Ok, your card has no vendor lock for SFP modules. But SFP module can still be uncompatible with Ethernet. It could be for Fiber Channel or some other technology. Is there any modell number on the label?

I have a spare x520-DA* card at work and will try to run it with 5.15.30-1-pve kernel and different SFP modules.

 

[nk1]

Ok, your card has no vendor lock for SFP modules. But SFP module can still be uncompatible with Ethernet. It could be for Fiber Channel or some other technology. Is there any modell number on the label?

I have a spare x520-DA* card at work and will try to run it with 5.15.30-1-pve kernel and different SFP modules.

After trying everything and nothing works, I unplug my HP DAC and plug in a no-name cable. Everything works with the no-name cable (still with an HP switch).

I tested that HP cable with a mellamox x-2 NIC and it works also so it’s just incompatibility issue.