[SOLVED] intel x553 SFP+ ixgbe no go on PVE8

V

vesalius

Renowned Member
Aug 19, 2020
408
115
63
These quit working with the shift to linux kernel 6.*. I reported initially here, but no luck with any of the PVE8 kernels either. I am currently using the last 5.15.108-1-pve kernel successfully otherwise software is latest pve8. Bringing this up again because I have been following along with similar complaints from a VyOS router forum and so far nothing has worked for those reporting issues there either. Latest suspicion is that something is broken in the ixgbe driver in all linux Kernels 6.* including Ubuntu 23.10 with latest stable 6.5.7.

Is anyone here using a x553-based intel card SFP+ successfully with pve8? Hoping if the vyos folks can build a custom working driver Proxmos would consider doing the same or we wait for intel then the slow distribution back into the Ubuntu kernel.
 
No response from anyone at @proxmox on this one, likely too small of an affected installed base so far. Have seen this Linux kernel commit suggested as the culprit.
ixgbe: Manual AN-37 for troublesome link partners for X550 SFI
https://git.kernel.org/pub/scm/linu...3&id=565736048bd5f9888990569993c6b6bfdf6dcb6d
Click to expand...

An amazon employee states reverting this commit and recompiling the kernel allows their similar network hardware to use the current in-tree 6.1 ixgbe driver. Otherwise as stated in the VyOS forum thread linked above compiling the linux kernel with the out-of-tree intel ixgbe driver 5.19.6 works too.
 

How to download and install ixgbe driver on Ubuntu or Debian​

Commands below are what I adapted for proxmox from that page to install the latest out of tree intel ixgbe driver and fix this issue for now. Mainly need to install proxmox-default-headers instead of linux-headers-$(uname -r)


Code: 
sudo apt-get install proxmox-default-headers build-essential
sudo apt-get install gcc make ( I restarted after this step or the later commands dependent on the headers failed)
cd /tmp
wget https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.19.9/ixgbe-5.19.9.tar.gz
tar xvfvz ixgbe-5.19.9.tar.gz
cd ixgbe-5.19.9/src
make
modinfo ./ixgbe.ko
sudo rmmod ixgbe
sudo modprobe dca (modules listed may vary for you. look in depends: section in the above modinfo ./ixgbe.ko output are all loaded first.)
sudo insmod ./ixgbe.ko
sudo make install
nano /etc/modules (If you want ixgbe driver to be loaded automatically upon boot, you can add ixgbe)

So Far, I can't get dkms to build and install ixgbe OOT automatically. I tried using the page linked below:
How to build a kernel module with DKMS on Linux
 
Last edited:
  • Like
Reactions: mcraven, jrm523 and Pheckphul
I built Intel's driver using DKMS and it works, so it's the current in-tree driver that fails.

Here's the dkms.conf I'm using:

MAKE="BUILD_KERNEL=${kernelver} make -C src/ KERNELDIR=/lib/modules/${kernelver}/build" CLEAN="make -C src/ clean" BUILT_MODULE_NAME=ixgbe BUILT_MODULE_LOCATION=src/ DEST_MODULE_LOCATION="/updates" PACKAGE_NAME=ixgbe-dkms PACKAGE_VERSION=5.19.9
 
  • Like
Reactions: mcraven, Pheckphul and vesalius
vesalius said:

How to download and install ixgbe driver on Ubuntu or Debian​

Commands below are what I adapted for proxmox from that page to install the latest out of tree intel ixgbe driver and fix this issue for now. Mainly need to install pve-headers instead of linux-headers-$(uname -r)


Code: 
sudo apt-get install pve-headers build-essential
sudo apt-get install gcc make ( I restarted after this step or the later commands dependent on the headers failed)
cd /tmp
wget https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.19.9/ixgbe-5.19.9.tar.gz
tar xvfvz ixgbe-5.19.9.tar.gz
cd ixgbe-5.19.9/src
make
modinfo ./ixgbe.ko
sudo rmmod ixgbe
sudo modprobe dca (modules listed may vary for you. look in depends: section in the above modinfo ./ixgbe.ko output are all loaded first.)
sudo insmod ./ixgbe.ko
sudo make install
nano /etc/modules (If you want ixgbe driver to be loaded automatically upon boot, you can add ixgbe)

So Far, I can't get dkms to build and install ixgbe OOT automatically. I tried using the page linked below:
How to build a kernel module with DKMS on Linux
Click to expand...
I wanted to stop by and tell you that this worked great for me. I am not a Linux pro by any means and I appreciate you taking the time to post a solution to the issue.
 
  • Like
Reactions: aarononeal and vesalius
vesalius said:

How to download and install ixgbe driver on Ubuntu or Debian​

Commands below are what I adapted for proxmox from that page to install the latest out of tree intel ixgbe driver and fix this issue for now. Mainly need to install proxmox-default-headers instead of linux-headers-$(uname -r)
Click to expand...

Same... I was struggling with this and you post was spot on. BIG THANKS.

Before finding this, I was trying to add the "allow_unsupported_sfp" option to my setup. This didn't help until I followed your instructions about manually putting the OOT driver in place. My next step will be to try to get DKMS working so I don't lose the driver when there is a kernel update.

Small question: my SFP link is working now but, after the driver install, I didn't remove all the "allow_unsupported_sfp" stuff. Is it needed?

Any trick to getting the DKMS working? Like the previous, I'm dangerous (but learning) in Linux... :)
 
So with the help of @aarononeal and the How to build a kernel module with DKMS on Linux webpage I have DKMS autoinstall working with the last kernel upgrade.

Code: 
apt-get install proxmox-default-headers build-essential dkms gcc make
cd /tmp
wget https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.19.9/ixgbe-5.19.9.tar.gz
tar xvfvz ixgbe-5.19.9.tar.gz -C /usr/src
nano /usr/src/ixgbe-5.19.9/dkms.conf

Copy the below into the dkms.conf file then save.
MAKE="BUILD_KERNEL=${kernelver} make -C src/ KERNELDIR=/lib/modules/${kernelver}/build"
CLEAN="make -C src/ clean"
PACKAGE_NAME="ixgbe-dkms"
PACKAGE_VERSION="5.19.9"
BUILT_MODULE_NAME="ixgbe"
BUILT_MODULE_LOCATION=src/
DEST_MODULE_LOCATION="/kernel/drivers/net/ethernet/intel/ixgbe/"
AUTOINSTALL="yes"
Click to expand...

Code: 
sudo dkms add ixgbe/5.19.9
sudo dkms build ixgbe/5.19.9
sudo dkms install ixgbe/5.19.9

After this DKMS should work to autoinstall with new kernels.
 
Last edited:
  • Like
Reactions: Kingneutron and Cha0s
vesalius said:
After this DKMS should work to autoinstall with new kernels.
Click to expand...

I'll try to set this up too. Many thanks.

Now that the driver is working. Is there a resource for setting up any options? I'm getting some very unexpected throughput with iperf3 testing. I've tried 10Gb SFP and DAC. If I run iperf3 server on my NAS (different box) I get 9.3Gbps from C3758 client to my NAS. But, if I reverse and do the "server" for iperf3 on the C3758, the same test only gets me ~4Gbps.

I'd love any insights! Cheers.
 
Last edited:
> If I run iperf3 server on my NAS (different box) I get 9.3Gbps from C3758 client to my NAS. But, if I reverse and do the "server" for iperf3 on the C3758, the same test only gets me ~4Gbps

Try MTU 9000 and parallel ' -P 2 ' or higher on iperf3
 
Kingneutron said:
FYI I tried the fix(es) listed in this thread and it still did not work for me on kernel 6.5.13-1-pve.

If anyone wants to take a look at the script and see if I did anything wrong I can try again, but for now have reverted back to kernel 5.15.136-1-pve

https://github.com/kneutron/ansitest/blob/master/proxmox/proxmox-fix-10gig-driver-compile.sh

I did verify that the new compiled driver loaded after reboot; 10Gbit nics still did not connect.
Click to expand...
Comment out rmmod ixgbe from the end of your script and try again. Don't think you should be running that at all with dkms. Or you can just try to run the commands from here at the CLI because with DKMS you should not ever have to run that script more than once.
 
Anyone affected by this try 8.2 yet ... either natively to see if the default driver for 6.8.4 works or with DKMS of the 5.19.9 intel driver?
 
  • Like
Reactions: Kingneutron
If I get some time and inclination $soon, I may try it - but it's a pain (and probably a few hours of lost time) if it doesn't fix the 4x 10Gbit ports on my primary standalone PVE (Qotom firewall appliance.) That box is running my pihole/squid VM for the whole house and the DHCP + ntp timeservice VMs for 2.5Gbit and 10Gbit networks.

Will probably try installing 8.2 to a USB3 SSD so it doesn't impact my internal nvme install. Probably take the opportunity to expand my usb3 hub from 4 ports to 8 as well.
 
Nope, ixgbe driver still not fixed in kernel 6.5.13 OR 6.8.4. Had to boot all the way back to 5.15.136 again and put all kernel packages on apt-mark hold. Building it from source also failed for 6.8.4.

Dear proxmox developers, PLEASE FIX THIS ASAP, I have 4x " Intel X553 10 GbE SFP+ " ports hardwired to the motherboard of my Qotom firewall appliance, it's not like I can swap them out!

I've upgraded in-place starting from a PVE 7.x install to 8.1 and now 8.2, and I still have to run an old-ass kernel just to get 10Gbit! How many more months are you guys going to let this driver issue drag on??
 
I have been using all the 6.5.* kernels with DKMS autoinstalling ixgbe-5.19.9 drivers without issue following what I posted above. Could not manually get 5.19.9 to build for 6.8.* series though. Today Intel releases 5.20.3, but that fails with DKMS as well. Went back and pinned to 6.5.13 for now.

Code: 
DKMS make.log for ixgbe-5.20.3 for kernel 6.8.4-2-pve (x86_64)
Thu Apr 25 11:20:16 PM CDT 2024
make: Entering directory '/var/lib/dkms/ixgbe/5.20.3/build/src'
*** The target kernel has CONFIG_MODULE_SIG_ALL enabled, but
*** the signing key cannot be found. Module signing has been
*** disabled for this build.
make[1]: Entering directory '/usr/src/linux-headers-6.8.4-2-pve'
  CC [M]  /var/lib/dkms/ixgbe/5.20.3/build/src/ixgbe_main.o
/var/lib/dkms/ixgbe/5.20.3/build/src/ixgbe_main.c: In function ‘ixgbe_clean_rx_irq’:
/var/lib/dkms/ixgbe/5.20.3/build/src/ixgbe_main.c:2389:17: error: implicit declaration of function ‘xdp_do_flush_map’; did you mean ‘xdp_do_flush’? [-Werror=implicit-function-declaration]
 2389 |                 xdp_do_flush_map();
      |                 ^~~~~~~~~~~~~~~~
      |                 xdp_do_flush
cc1: some warnings being treated as errors
make[3]: *** [scripts/Makefile.build:243: /var/lib/dkms/ixgbe/5.20.3/build/src/ixgbe_main.o] Error 1
make[2]: *** [/usr/src/linux-headers-6.8.4-2-pve/Makefile:1926: /var/lib/dkms/ixgbe/5.20.3/build/src] Error 2
make[1]: *** [Makefile:240: __sub-make] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-6.8.4-2-pve'
make: *** [Makefile:100: default] Error 2
make: Leaving directory '/var/lib/dkms/ixgbe/5.20.3/build/src'
 
Same issue here, had this all working with a Qotom box with DAC cables, decided to update Proxmox this evening, now can't get the 10GBe links back up, and can't recompile the driver now.
Switched over to a spare 1Gb port for now, and it's annoying that this never seems to just work with every update I've done.
 
  • Like
Reactions: Kingneutron
Well I fixed all the issues with the source in 5.20.3 with what I believe are sane alternatives, got the driver loaded, but still fail to get a carrier detected. I think it's somehow ignoring the allow unsupported SFPs option now. If I get some more time this weekend I'll see if I can force that to be overridden, unless someone has a better idea of where to look.

Interestingly, one of the functions of XDP appears to be missing (trace_xdp_exception), though all the other definitions are found. I was referring to this presentation which has the matching prototype that's in this code:
https://netdevconf.info/0x14/pub/slides/10/add-xdp-on-driver.pdf

The missing case happens after the default, and the next option is to drop the packet, so I went ahead and commented out the exception which should just drop anything that wasn't handled earlier (based on my ~20 minute lesson on how XDP works).

modinfo output:

Code: 
root@pve:/usr/src# modinfo ixgbe
filename:       /lib/modules/6.8.4-2-pve/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        5.20.3
license:        GPL
description:    Intel(R) 10GbE PCI Express Linux Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     4182AFF2D8E04B4B40F1955
alias:          pci:v00008086d000015E5sv*sd*bc*sc*i*
...
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        dca
retpoline:      Y
name:           ixgbe
vermagic:       6.8.4-2-pve SMP preempt mod_unload modversions
parm:           IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
parm:           InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)
parm:           MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm:           DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
parm:           RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
parm:           VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable (1 queue) 2-16 enable (default=8) (array of int)
parm:           max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
parm:           VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int)
parm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)
parm:           LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm:           LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
parm:           LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
parm:           LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
parm:           LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
parm:           FdirPballoc:Flow Director packet buffer allocation level:
                        1 = 8k hash filters or 2k perfect filters
                        2 = 16k hash filters or 4k perfect filters
                        3 = 32k hash filters or 8k perfect filters (array of int)
parm:           AtrSampleRate:Software ATR Tx packet sample rate (array of int)
parm:           FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm:           MDD:Malicious Driver Detection: (0,1), default 1 = on (array of int)
parm:           LRO:Large Receive Offload (0,1), default 0 = off (array of int)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)
parm:           dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int)
parm:           vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int)

Still busted:
Code: 
root@pve:/usr/src# ip link show eno1
22: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 20:7c:14:f3:55:21 brd ff:ff:ff:ff:ff:ff
    altname enp11s0f0
 
I can't help with this but I'm stuck too. I'm new and was following the posts above. I didn't catch these posted issues and manually did an upgrade to PVE 8.2. I've got my system booting from a previous 6.5 kernel but I'm kind of stuck. :( Can I get some help to back out of the DKMS fails on the 8.2 upgrade process so I can get the kernel to build without the broken driver and the install to complete cleanly?
Code: 
Building module:
Cleaning build area...
BUILD_KERNEL=6.8.4-2-pve make -C src/ KERNELDIR=/lib/modules/6.8.4-2-pve/build......(bad exit status: 2)
Error! Bad return status for module build on kernel: 6.8.4-2-pve (x86_64)
Consult /var/lib/dkms/ixgbe/5.19.9/build/make.log for more information.
Error! One or more modules failed to install during autoinstall.
Refer to previous errors for more information.
dkms: autoinstall for kernel: 6.8.4-2-pve failed!

Errors were encountered while processing:
 proxmox-kernel-6.8.4-2-pve-signed
 proxmox-kernel-6.8
 proxmox-default-kernel
 proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom