intel x553 SFP+ ixgbe no go on PVE8
- Thread starter vesalius
- Start date
Done. And... yes, this allowed the 6.8 kernel install to complete. Thanks. This is a dirt simple solution and I should have thought of it. I was trying to temporarily pause the DKMS service, set the ixgbe 5.19.9 auto-install option to "no", etc. but nothing I tried cleared the road to allow the kernel to build.
I'm now pinned to the 6.5 kernel. Again, I was drafting on the work in this forum topic to get the x553 10G ports working before. I'm hoping there is a new fix shortly or that Proxmox gets around to properly supporting this NIC. When I purchased this device, I didn't realize that these NICs were a problem for Proxmox.
Is the best path to get 5.19.9 to compile under the new kernel or move ahead to 5.20.3?
Again, big thanks.
Same goes for me. Kinda frustrating, but I don't think there will be a fix any time soon.
So I did a bit more digging, but still no luck with this. I did find that the driver is properly detecting the DAC cable without complaints, but it's still stuck with the NO CARRIER message. This past week I ordered a pair of Intel-branded FTLX8571D3BCV-IT. I confirmed that it also detects these properly and does turn on the laser, but they will not connect either. So the problem does seem to be somewhere other than SFP detection. I tried one of the 1G SFP ports on my box and it also won't link up, so it seems all SFPs on this model NIC are just bricked with this new kernel.
Just sticking with 1G for now because it doesn't really affect what I'm currently running, but all those fancy ports I paid for are currently unusable.
Just sticking with 1G for now because it doesn't really affect what I'm currently running, but all those fancy ports I paid for are currently unusable.
While this affects proxmox, it is not just a proxmox issue, it is a general Linux kernel driver issue, that affects every distribution using Linux kernel 6.* and above, so Ubuntu, debian, suse, etc ....
you can pin the last proxmox 6.5 kernel and build/install the 5.20.3 driver there to regain use of your 10g ports.
Last edited:
Just some info for the party: Intel's ixgbe driver is a few versions behind Linux kernel, stilling using removed kernel APIs so it won't compile on newest 6.8 kernel. xdp_do_flush_map in this case, by the look of things from other people's error message.
The in-tree ixgbe seems to lagging behind more (feature and bugfix wise) as it is showing multiple bugs that were fixed on Intel's version.
For anyone who cares enough, you can make money patches by switching to the new xdp_do_flush, which is how they (the kernel guys) made the in-tree ixgbe compile with kernel 6.8
Disclaimer: I don't work for Intel, I don't have any x553 laying around and I'm not running anything on kernel 6.8 Also it has been VERY long time since the last time I wrote any kernel/driver code so I'm not responsible for any consequences by doing that money patch. Refer to Intel/Linux kernel working groups for further information.
The in-tree ixgbe seems to lagging behind more (feature and bugfix wise) as it is showing multiple bugs that were fixed on Intel's version.
For anyone who cares enough, you can make money patches by switching to the new xdp_do_flush, which is how they (the kernel guys) made the in-tree ixgbe compile with kernel 6.8
Disclaimer: I don't work for Intel, I don't have any x553 laying around and I'm not running anything on kernel 6.8 Also it has been VERY long time since the last time I wrote any kernel/driver code so I'm not responsible for any consequences by doing that money patch. Refer to Intel/Linux kernel working groups for further information.
Last edited:
Just took a look at 5.20.3 driver from Intel and it appears they already added money patch for kernel 6.8, so this version should compile with kernel 6.8 and there is no need to pin 6.5
Anyone care enough to try it?
Basically they replaced all xdp_do_flush_map to xdp_do_flush and retained backward compatibility through the following macro.
C:
/* NEED_XDP_DO_FLUSH
*
* Upstream commit 1d233886dd90 ("xdp: Use bulking for non-map XDP_REDIRECT
* and consolidate code paths") replaced xdp_do_flush_map with xdp_do_flush
* and 7f04bd109d4c ("net: Tree wide: Replace xdp_do_flush_map() with
* xdp_do_flush()") cleaned up related code.
*/
#ifdef NEED_XDP_DO_FLUSH
static inline void xdp_do_flush(void)
{
xdp_do_flush_map();
}
#endif /* NEED_XDP_DO_FLUSH */
Last edited:
I pinned the 6.5 kernel and installed the 5.19.9 and 5.20.3 drivers, which sadly didn't work either. The link is up while booting but dissapears when Proxmox is booted.
Must be a different issue than what I am experiencing then, those out of tree intel drivers work fine to resurrect my SFP+ ports when installed in the Proxmox 6.5 kernel.
Unless I am misunderstanding your suggestions, I did try already and the build errored out. See https://forum.proxmox.com/threads/intel-x553-sfp-ixgbe-no-go-on-pve8.135129/post-657758
No freaking idea why but, its true that intel still makes call to
xdp_do_flush_map in 5.20.3They actually have a check in place to see if
xdp_do_flush is present: gen NEED_XDP_DO_FLUSH if fun xdp_do_flush absent in "$fh"But never call it anywhere. (What is this? Half baked patch that, pretend it's trying to fix things?)
To me this looks like a bug.
Also I'm seeing 5.19.9 listed as "Latest" in Intel's site but sourceforge has 5.20.3 released April 26. Did Intel pull 5.20.3 from their website, or they just don't give a shit about their own sites?
https://www.intel.com/content/www/u...ethernet-network-connections-under-linux.html
Last edited:
Pinned to 6.5.x like everybody else... considering moving from 5.19.9 to 5.20.3 on the 6.5.x kernel, for now...
That said, x553 is not new... any value in considering an older OOT driver version? I assume ixgbe.ko is included in the 6.8.x kernel... what version?
That said, x553 is not new... any value in considering an older OOT driver version? I assume ixgbe.ko is included in the 6.8.x kernel... what version?
This is with the newest version 6.8.4-3
So I downloaded 5.20.3 from source forge and extracted it.
mkdir /src
cd /src
Get the tar file from here.
https://sourceforge.net/projects/e1000/files/ixgbe stable/5.20.3/
tar xvf ixgbe-5.20.3.tar.gz
installed gcc make buildtools-esentials
went to compile and got errors
There are roughly 10 errors we will have to address.
5 instances in a row of replacing strlcpy with strncpy because the compiler wanted me to do it.
on lines that it errors on replace xdp_do_flush_map with xdp_do_flush
In the ethtool.c
There will be 2 further errors
The last 2 parts of a structure, now I am sure it will come up later, but I use // to comment both of them out.
compiled again
a few more of the above xdp_do_flush_map rename xdp_do_flush
finally it compiles
make
make install
for luck I added
vi etc/modprobe.d/ixgbe.conf
options ixgbe allow_unsupported_sfp=1
rebooted
dhclient -v eno1
BING
I had an IP address and could ping things.
I hope someone creates an official patch so we don't have to dig around in the weds like this.
Remarking out 2 entries in the ethtool.c files seems especially dodgy, but it works for me.
-------
I got 5.19.9 to compile, but there were 2 more lines I had to comment out, and seemed even more sketchy to do so.
So I downloaded 5.20.3 from source forge and extracted it.
mkdir /src
cd /src
Get the tar file from here.
https://sourceforge.net/projects/e1000/files/ixgbe stable/5.20.3/
tar xvf ixgbe-5.20.3.tar.gz
installed gcc make buildtools-esentials
went to compile and got errors
There are roughly 10 errors we will have to address.
5 instances in a row of replacing strlcpy with strncpy because the compiler wanted me to do it.
on lines that it errors on replace xdp_do_flush_map with xdp_do_flush
In the ethtool.c
There will be 2 further errors
The last 2 parts of a structure, now I am sure it will come up later, but I use // to comment both of them out.
compiled again
a few more of the above xdp_do_flush_map rename xdp_do_flush
finally it compiles
make
make install
for luck I added
vi etc/modprobe.d/ixgbe.conf
options ixgbe allow_unsupported_sfp=1
rebooted
dhclient -v eno1
BING
I had an IP address and could ping things.
I hope someone creates an official patch so we don't have to dig around in the weds like this.
Remarking out 2 entries in the ethtool.c files seems especially dodgy, but it works for me.
-------
I got 5.19.9 to compile, but there were 2 more lines I had to comment out, and seemed even more sketchy to do so.
Last edited:
Looks like you made the same decisions I did, except for 'xdp_do_flush_map rename xdp_do_flush'. I'll give that a try this week.
I don't think the variables commented out in ethtool.c matter for this NIC, but that's just a half-educated guess.
On kernel 6.8.4-3, I've managed to compile ixgbe 5.20.3... module is untested yet though.
Confirming:
1) replace all "xdp_do_flush_map" with "xdp_do_flush"
2) in ixgbe_eththool.c:
- comment out 2x erroring struc lines
- replace "strlcpy" with "strscpy" in function def
I may not get to test right away as I'm traveling and I don't want to play around remote. Any others that take a swing, please post results of getting x553 NIC to come up at 10G. Comments on issues/performance welcome!
Confirming:
1) replace all "xdp_do_flush_map" with "xdp_do_flush"
2) in ixgbe_eththool.c:
- comment out 2x erroring struc lines
- replace "strlcpy" with "strscpy" in function def
I may not get to test right away as I'm traveling and I don't want to play around remote. Any others that take a swing, please post results of getting x553 NIC to come up at 10G. Comments on issues/performance welcome!
Went ahead and gave it another go. Confirmed link is up after a reboot. Note that you have to do a reboot, just inserting the updated module does not seem to bring it to life (I think that was my original issue).
Code:
root@pve:~# ethtool eno1
Settings for eno1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yesWhich DAC/Transceiver are you using? I tried multiple transceiver and DACs and still didn't get it to work
Nevermind. The DACS and the transceivers worked flawlessly. I still didn't manage to get it working with proxmox, but I passed the PCIe Device to my pfSense VM and it worked right out of the box.
Either way I'm using this one: https://www.amazon.com/gp/product/B00WHS3NCA/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
10Gtek SFP+ DAC Twinax Cable, 10G SFP+ to SFP+ Direct Attach Copper Patch Passive Cable for Cisco SFP-H10GB-CU1M, Meraki MA-CBL-TA-1M, Ubiquiti UniF, D-Link, Fortinet, 1-Meter(3.3ft)
Transceivers I picked up are Intel-branded FTLX8571D3BCV-IT.
One clue it was Proxmox is that the link comes up while booting/in BIOS and then the link goes down as Proxmox loads the network driver.