[SOLVED] intel x553 SFP+ ixgbe no go on PVE8
- Thread starter vesalius
- Start date
Proxmox released proxmox-kernel-signed-6.8.12-1 to the no-subscription repository, which sources Ubuntu-6.8.0-43.43 (where ubuntu finally reverts the ixgbe 553 breakage from back in Linux Kernel 6.1).
Try it out, installed and works after reboot without the OOT intel IXGBE installed for me. Have not done any iperf.
Try it out, installed and works after reboot without the OOT intel IXGBE installed for me. Have not done any iperf.
Supported link modes: 10000baseT/Full
Do you want to use 1G SFP modules or those MultiGig SFP+ RJ45 Modules? They usually have a 10G SFP+ interface and an integrated 2 port switch to support 100M/1G/2.5G/5G/10G Ethernet
A 1G SFP fiber module. My ISP only provides 1GB (hackable to 2.5gb), and I was hoping to be able to connect it directly, but as of now the card wont recognize the module, presumably because the card can't negotiate (or be hard coded to) 1/2.5GB
I may try to upgrade this weekend. I've been traveling for work and don't like tinkering when I'm not guaranteed to be around the "home lab". Any chance you have tried an perf tests? what module version is part of this new pve kernel?
After replacing my Brocade ICX-7250 with a Cisco 3064PQ, everything worked fine. The link is stable, although it's not quite 10Gbps (9.8/8ish) in both directions. I guess the problem was a combination of the Brocade and the NIC, regardless of the SFP used.
Thanks for the heads-up. Trying to upgrade to kernel 6.8.12-1-pve broke veeam, which I then uninstalled, and now it's failing to build the ixgbe driver. I previously posted where it looked like it was working but apparently it had actually booted to the old 5.15 kernel
Code:
Building module:
Cleaning build area...
BUILD_KERNEL=6.8.12-1-pve make -C src/ KERNELDIR=/lib/modules/6.8.12-1-pve/build.......(bad exit status: 2)
Error! Bad return status for module build on kernel: 6.8.12-1-pve (x86_64)
Consult /var/lib/dkms/ixgbe/5.19.9/build/make.log for more information.
Error! One or more modules failed to install during autoinstall.
Refer to previous errors for more information.
dkms: autoinstall for kernel: 6.8.12-1-pve failed!
run-parts: /etc/kernel/postinst.d/dkms exited with return code 11
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.8.12-1-pve-signed.postinst line 20.
dpkg: error processing package proxmox-kernel-6.8.12-1-pve-signed (--configure):
installed proxmox-kernel-6.8.12-1-pve-signed package post-installation script subprocess returned error exit status 2
Last edited:
You should not need to build the ixgbe OOT driver with 6.8.12-1, it should just work out of the box. use
proxmox-boot-tool kernel pin 6.8.12-1-pve --next-boot to force trial the new kernel on the next boot only. I am curious if you have pinned the old 5.15 kernel?
and the DKMS error message states
that means you are likely having the same issue I was with a zombie 5.19.9 build attempt. see my post in this thread https://forum.proxmox.com/threads/intel-x553-sfp-ixgbe-no-go-on-pve8.135129/post-688845
Excellent - thanks again for the fix, it worked. Can confirm this is FINALLY fixed out-of-the-box in kernel 6.8.12-1-pve for
Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
For anyone running the Qotom firewall mini-pc*, (which is an amazing deal now for under $400) this is welcome news. However, I am not pleased with proxmox due to the amount of time it took to fix this OBVIOUS REGRESSION - it's been most of a year, and this has affected THOUSANDS of installs.
* https://www.amazon.com/gp/product/B0CJLK9GZV/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
...And it also broke veeam. I'm ok, I uninstalled it and still have fsarchiver, will probably try reinstalling it again in a month or so.
This is with MTU=9000 , point-to-point with a DAC cable:
[ lenovo-520-lmde (scrn=) ]
~ $ iperf3 -c qotom-proxmox-10g
Connecting to host qotom-proxmox-10g, port 5201
[ 5] local 172.16.10.44 port 46766 connected to 172.16.10.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.16 GBytes 9.94 Gbits/sec 1 1.93 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 9.86 Gbits/sec 0 1.95 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 9.84 Gbits/sec 0 2.00 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 9.83 Gbits/sec 0 2.00 MBytes
[ 5] 4.00-5.00 sec 1.13 GBytes 9.74 Gbits/sec 0 2.24 MBytes
[ 5] 5.00-6.00 sec 1.14 GBytes 9.81 Gbits/sec 0 2.47 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 9.86 Gbits/sec 0 2.47 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 9.86 Gbits/sec 0 3.02 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 9.84 Gbits/sec 0 3.02 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec 0 4.52 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.86 Gbits/sec 1 sender
[ 5] 0.00-9.97 sec 11.5 GBytes 9.88 Gbits/sec receiver
$ iperf3 -c qotom-proxmox-10g -R
Connecting to host qotom-proxmox-10g, port 5201
Reverse mode, remote host qotom-proxmox-10g is sending
[ 5] local 172.16.10.44 port 60904 connected to 172.16.10.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 4 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec receiver
So to be clear this is/was a main linux kernel ixgbe regression affecting all distributions dating back to kernel 6.1. It was only reverted in 6.9 and backported to LTS Kernels 6.6 and 6.1 in May 2024. So Proxmox (actually Ubuntu) is only 2-3 months behind on back porting this fix to 6.8. See: https://forum.proxmox.com/threads/intel-x553-sfp-ixgbe-no-go-on-pve8.135129/post-690100
No doubt this was really frustrating for those of us affected for the last couple of years.
I have just installed a fresh copy of PVE 8 and upgraded to the latest 6.8.12-1 kernel. However, it does not work for my hardware. I am using an a2sdi-h-tp4f motherboard that comes with intel atom C3958. the network interface is X553. I guess I will have to try manually install the new ixgbe driver instead.
Additional things I have tried:
1. compiled the 5.21.5 x553 driver from intel and it did not work.
2. Installed a fresh copy of PVE7, it does not work either.
To make sure that this is specific to debian based linux, I have installed my pfsense on the same hardware and the x553 driver works flawlessly.
Additional things I have tried:
1. compiled the 5.21.5 x553 driver from intel and it did not work.
2. Installed a fresh copy of PVE7, it does not work either.
To make sure that this is specific to debian based linux, I have installed my pfsense on the same hardware and the x553 driver works flawlessly.
Last edited:
Just wanted to add a similar issue seen on a Supermicro server which uses the "Ethernet Controller 10-Gigabit X540-AT2" as four onboard NICs on the motherboard.
The first NIC seems to work, although with 1Gb only. Connections to the other ports don't seem to work. Not sure if its a driver issue or physical issue yet.
The new Proxmox installation is only a few weeks old.
The first NIC seems to work, although with 1Gb only. Connections to the other ports don't seem to work. Not sure if its a driver issue or physical issue yet.
The new Proxmox installation is only a few weeks old.
I am using the Supermicro X10SDV-4C-TLN2F motherboard, with the X552/x557-AT network card, installed with Proxmox VE version 8.3.0 and kernel version 6.8.12-5. I recompiled the driver following the methods provided by everyone above, but the issue seems not to be resolved. I noticed a strange phenomenon: when I power on for the first time, the system starts up and my ens3 network card does not work, while the IPMI network card works fine. After I set poweroff in IPMI and then set poweron, my ens3 network card starts working properly. 