[SOLVED] intel x553 SFP+ ixgbe no go on PVE8

ixgbe stable / 5.20.9 released.

5.20.9 builds and installs without error for me on 6.8.8-1-pve, BUT the SFP+ ports are still down after a reboot into this kernel/driver combo. Don't have time to investigate much, hopefully others have better luck.

reverted back to 6.5.13-5-pve/ixgbe-5.19.9 working combo
 
Last edited:
I'm about to offer a bug bounty for this after my next paycheck - $50 Cash App to whoever gets this driver working reliably for kernel 6.5. I'm still stuck on 5.15 but upgraded PVE in-place to 8.2.4 bc the 10Gbit ports are non-replaceable.


0b:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
 
Last edited:
https://www.intel.com/content/www/u...ethernet-network-connections-under-linux.html

Intel just released a driver update 5.20.9
Real world improvements:
1. It compiles without error (make)
2. It install fine (make install)

I updated to proxmox 6.8.8 and I had to re-compile and install 5.20.9, but after that it at least works.
I haven't been able to get 10gbps, but I get plenty of speed for my needs.
I get about 1gbps for the record.
 
Last edited:
Yeah that's not a real fix. I have 4x10Gbit ports and the kernel needs to work out of the box at full 10Gbit speed without jumping thru hoops.
 
Maybe I should call it a bandaid. However, all 4 ports do work, where as before I had to modify the source code to get it to even compile. Unfortunately I don't have the paid update license so I am stuck at 6.8.8 kernel wise and don't know if you can upgrade the kernel to 6.9.x manually.
 
cybernard said:
Maybe I should call it a bandaid. However, all 4 ports do work, where as before I had to modify the source code to get it to even compile. Unfortunately I don't have the paid update license so I am stuck at 6.8.8 kernel wise and don't know if you can upgrade the kernel to 6.9.x manually.
Click to expand...
I've been pinned at 6.5.13-5-pve for a while now... "patiently" waiting for some dust to settle and to see a somewhat reasonable update happen. My idea of "reasonable" is kernel support for these 10G ports and operating at 10G. :)

Are you suggesting that 6.9.x via subscription does this?

If so, I may subscribe and move on! I've been planning to subscribe anyway but I was on-the-fence because of the lack of support for this HW.
 
  • Like
Reactions: Kingneutron
A basic subscription gets you updates whether or not they fix the issue I don't know since I am not subscribed either.

I have been thinking about it
 
1719677511241.png
1719677568530.png

I am using the even new 5.20.9 ixgbe driver from intel and proxmox 6.8.8.2 (just available today)
I can't guarantee you will get the same results.

The combination of 5.20.9 and 6.8.8.2 results in 2 warnings about unused functions, but still compiles. I have not noticed any adverse consequences.

Make you do download and extract driver 5.20.9 before updating.
Then make sure you have the necessary packages to compile before rebooting.

apt install gcc make buildtools-esentials

You can do a make and not do the make install to make sure it compiles as you may not have access to the network until you have 5.20.9 installed.

After rebooting you will have to compile and install 5.20.9

Do a uname -r to verify 6.8.8.2 is active, not just installed

You will need to make and make install every time you upgrade the kernel until the problem is actually fixed upstream.



make
make install

Reboot again
Test

Others should probably test it first.
 
cybernard said:
I am using the even new 5.20.9 ixgbe driver from intel and proxmox 6.8.8.2 (just available today)
I can't guarantee you will get the same results.
Click to expand...

This is a welcome post... I'm hoping others can test and post too. I will likely try this weekend as I'm currently pinned on 6.5.13 kernel.

You seem to be getting proper performance. Curious... an earlier post indicated successful compile but only 1Gbps perf?
 
Proxmox just released version 6.8.8.2 today and I combined that with intel driver version 5.20.9

Previous test was with 6.8.4.2 and then 6.8.8.1, but I could not test against 6.8.8.2 because it was not available.
Also 5.20.9 was only recently made available.

Note: I did have a power outage, and all my gear was rebooted. I have no idea if that had any effect, but it shouldn't have.
 
Last edited:
Kingneutron said:
Yeah that's not a real fix. I have 4x10Gbit ports and the kernel needs to work out of the box at full 10Gbit speed without jumping thru hoops.
Click to expand...
I think you misunderstand the whole concept of Linux.

NO ONE IS RESPONSIBLE FOR YOUR EXPERIENCE. if "someone" didnt do it for you, "someone" isnt responsible for it to you unless you commissioned them to do so. and I know I'd like to get paid for my time and expertise before accepting any such responsibility.

also, "Proxmox" isnt responsible for drivers for every combination of hardware. If you have hardware that isnt functioning, I'd try any resources from the product manufacturer (Intel?,) Kernel.org, or even the distro (debian) as drivers are more directly their concern.
 
cybernard said:
Proxmox just released version 6.8.8.2 today and I combined that with intel driver version 5.20.9

Previous test was with 6.8.4.2 and then 6.8.8.1, but I could not test against 6.8.8.2 because it was not available.
Also 5.20.9 was only recently made available.

Note: I did have a power outage, and all my gear was rebooted. I have no idea if that had any effect, but it shouldn't have.
Click to expand...

Confirming clean compile of 5.20.9 on 6.8.8-2. x553 link is up... Commencing iperf testing now.

EDIT: getting over 9.0Gbps via iperf! Success!
 
Last edited:
alexskysilk said:
I think you misunderstand the whole concept of Linux.

NO ONE IS RESPONSIBLE FOR YOUR EXPERIENCE. if "someone" didnt do it for you, "someone" isnt responsible for it to you unless you commissioned them to do so. and I know I'd like to get paid for my time and expertise before accepting any such responsibility.

also, "Proxmox" isnt responsible for drivers for every combination of hardware. If you have hardware that isnt functioning, I'd try any resources from the product manufacturer (Intel?,) Kernel.org, or even the distro (debian) as drivers are more directly their concern.
Click to expand...

Kindly stop shouting. My $DAYJOB is wanting to get off vmware esxi and we are considering proxmox for a dev lab. The responsibility for fixing this is on PROXMOX, as they don't use Debian kernels and have been aimlessly waiting for a fix from upstream.

I have to tell my boss that this stupid-ass fkg BUG has been open for over 8 MONTHS without a fix from the company! That directly affects the decision on what we want/are able to use in the datacenter going forward, because A) it needs to be supported to a point where I can be confident with running it; and B) I did mention offering a bug bounty/fix for cash.
 
Kingneutron said:
The responsibility for fixing this is on PROXMOX
Click to expand...
I was trying to disabuse you of that notion. keep complaining though, I suppose you might feel better.

Kingneutron said:
I have to tell my boss that this stupid-ass fkg BUG has been open for over 8 MONTHS without a fix from the company!
Click to expand...
You have two choices. you either get a grip on what Proxmox is, what Linux is, and what Proxmox GMBH is providing you. Or dont. either way, the only thing that changes is your expectations; even if we were to assume the Proxmox staff would be obliged to help with this issue, I dont see a subscriber tag by your name.
Kingneutron said:
That directly affects the decision on what we want/are able to use in the datacenter going forward
Click to expand...
Oh no doubt. see above.
 
Dude I have no idea what your deal is, but I've been a Linux sysadmin since the late 90's and I'm in charge of technology procurement and buildout.

If you have nothing helpful to contribute as far as actually fixing this issue, stop spamming the thread with your opinions.
 
How did this ticket get so off topic? It was only about the ixgbe drivers working in proxmox. Anyway, the problem seems fixed so I am leaving the conversation.
 
I recently picked up one of the qotom 3758R's 1U systems featured on ServeTheHome not too long ago. So far I've tried the following combinations in iperf3:

KernelDriverServer/ClientResult
6.5stockclient9.4 Gbit/s
6.5stockserver10Mbit/s
6.55.19.9client9.2 Gbit/s
6.55.19.9server10Mbit/s
6.8.8.25.20.9client9.5 Gbit/s
6.8.8.25.20.9server10Mbit/s

I've found if I change the MTU from 9000 to 1500 I can get around 1.5 Gbit/s as the server in iperf3 but overall performance is still terrible for what I'm expecting/wanting.

From reading through the thread it seems this is still unresolved. Does anyone know of a time/combination of drivers where this actually did work for 10 Gbit bidirectional? I might be able to spend some time bisecting the issue if I can get pointed in that direction.
 
mcraven said:
Confirming clean compile of 5.20.9 on 6.8.8-2. x553 link is up... Commencing iperf testing now.

EDIT: getting over 9.0Gbps via iperf! Success!
Click to expand...
Crazy the variability, this combo is not work for me at all. No link after reboot.
 
cybernard said:
Did you do a make AND
make install

Then reboot?
Click to expand...
I'll assume this was directed at me. Yes, I did. I haven't been able to find a combination that achieves ~10/~10 Gbps in both directions. Usually incoming is fine but outgoing is ridiculously low which is just... odd. It implies the optics are fine, the ones I have are just generic 10GTeks and that it's a definite driver issue, which is also backed up by the fact older kernels seem to have worked mostly fine. I'm half tempted to install RHEL on the machine and benchmark it to see if the issue goes away. That'd at least give me a starting place for figuring out what changed.
 
You must log in or register to reply here.
Top Bottom
Back