intel x553 SFP+ ixgbe no go on PVE8

vesalius

Active Member
Aug 19, 2020
374
96
33
These quit working with the shift to linux kernel 6.*. I reported initially here, but no luck with any of the PVE8 kernels either. I am currently using the last 5.15.108-1-pve kernel successfully otherwise software is latest pve8. Bringing this up again because I have been following along with similar complaints from a VyOS router forum and so far nothing has worked for those reporting issues there either. Latest suspicion is that something is broken in the ixgbe driver in all linux Kernels 6.* including Ubuntu 23.10 with latest stable 6.5.7.

Is anyone here using a x553-based intel card SFP+ successfully with pve8? Hoping if the vyos folks can build a custom working driver Proxmos would consider doing the same or we wait for intel then the slow distribution back into the Ubuntu kernel.
 
No response from anyone at @proxmox on this one, likely too small of an affected installed base so far. Have seen this Linux kernel commit suggested as the culprit.

An amazon employee states reverting this commit and recompiling the kernel allows their similar network hardware to use the current in-tree 6.1 ixgbe driver. Otherwise as stated in the VyOS forum thread linked above compiling the linux kernel with the out-of-tree intel ixgbe driver 5.19.6 works too.
 

How to download and install ixgbe driver on Ubuntu or Debian

Commands below are what I adapted for proxmox from that page to install the latest out of tree intel ixgbe driver and fix this issue for now. Mainly need to install proxmox-default-headers instead of linux-headers-$(uname -r)


Code:
sudo apt-get install proxmox-default-headers build-essential
sudo apt-get install gcc make ( I restarted after this step or the later commands dependent on the headers failed)
cd /tmp
wget https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.19.9/ixgbe-5.19.9.tar.gz
tar xvfvz ixgbe-5.19.9.tar.gz
cd ixgbe-5.19.9/src
make
modinfo ./ixgbe.ko
sudo rmmod ixgbe
sudo modprobe dca (modules listed may vary for you. look in depends: section in the above modinfo ./ixgbe.ko output are all loaded first.)
sudo insmod ./ixgbe.ko
sudo make install
nano /etc/modules (If you want ixgbe driver to be loaded automatically upon boot, you can add ixgbe)

So Far, I can't get dkms to build and install ixgbe OOT automatically. I tried using the page linked below:
How to build a kernel module with DKMS on Linux
 
Last edited:
I built Intel's driver using DKMS and it works, so it's the current in-tree driver that fails.

Here's the dkms.conf I'm using:

MAKE="BUILD_KERNEL=${kernelver} make -C src/ KERNELDIR=/lib/modules/${kernelver}/build" CLEAN="make -C src/ clean" BUILT_MODULE_NAME=ixgbe BUILT_MODULE_LOCATION=src/ DEST_MODULE_LOCATION="/updates" PACKAGE_NAME=ixgbe-dkms PACKAGE_VERSION=5.19.9
 

How to download and install ixgbe driver on Ubuntu or Debian

Commands below are what I adapted for proxmox from that page to install the latest out of tree intel ixgbe driver and fix this issue for now. Mainly need to install pve-headers instead of linux-headers-$(uname -r)


Code:
sudo apt-get install pve-headers build-essential
sudo apt-get install gcc make ( I restarted after this step or the later commands dependent on the headers failed)
cd /tmp
wget https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.19.9/ixgbe-5.19.9.tar.gz
tar xvfvz ixgbe-5.19.9.tar.gz
cd ixgbe-5.19.9/src
make
modinfo ./ixgbe.ko
sudo rmmod ixgbe
sudo modprobe dca (modules listed may vary for you. look in depends: section in the above modinfo ./ixgbe.ko output are all loaded first.)
sudo insmod ./ixgbe.ko
sudo make install
nano /etc/modules (If you want ixgbe driver to be loaded automatically upon boot, you can add ixgbe)

So Far, I can't get dkms to build and install ixgbe OOT automatically. I tried using the page linked below:
How to build a kernel module with DKMS on Linux
I wanted to stop by and tell you that this worked great for me. I am not a Linux pro by any means and I appreciate you taking the time to post a solution to the issue.
 

How to download and install ixgbe driver on Ubuntu or Debian

Commands below are what I adapted for proxmox from that page to install the latest out of tree intel ixgbe driver and fix this issue for now. Mainly need to install proxmox-default-headers instead of linux-headers-$(uname -r)

Same... I was struggling with this and you post was spot on. BIG THANKS.

Before finding this, I was trying to add the "allow_unsupported_sfp" option to my setup. This didn't help until I followed your instructions about manually putting the OOT driver in place. My next step will be to try to get DKMS working so I don't lose the driver when there is a kernel update.

Small question: my SFP link is working now but, after the driver install, I didn't remove all the "allow_unsupported_sfp" stuff. Is it needed?

Any trick to getting the DKMS working? Like the previous, I'm dangerous (but learning) in Linux... :)
 
So with the help of @aarononeal and the How to build a kernel module with DKMS on Linux webpage I have DKMS autoinstall working with the last kernel upgrade.

Code:
apt-get install proxmox-default-headers build-essential dkms gcc make
cd /tmp
wget https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.19.9/ixgbe-5.19.9.tar.gz
tar xvfvz ixgbe-5.19.9.tar.gz -C /usr/src
nano /usr/src/ixgbe-5.19.9/dkms.conf

Copy the below into the dkms.conf file then save.
MAKE="BUILD_KERNEL=${kernelver} make -C src/ KERNELDIR=/lib/modules/${kernelver}/build"
CLEAN="make -C src/ clean"
PACKAGE_NAME="ixgbe-dkms"
PACKAGE_VERSION="5.19.9"
BUILT_MODULE_NAME="ixgbe"
BUILT_MODULE_LOCATION=src/
DEST_MODULE_LOCATION="/kernel/drivers/net/ethernet/intel/ixgbe/"
AUTOINSTALL="yes"

Code:
sudo dkms add ixgbe/5.19.9
sudo dkms build ixgbe/5.19.9
sudo dkms install ixgbe/5.19.9

After this DKMS should work to autoinstall with new kernels.
 
Last edited:
  • Like
Reactions: Cha0s

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!