[SOLVED] intel x553 SFP+ ixgbe no go on PVE8

vesalius said:
Just doing a mv /usr/src/ixgbe-5.19.9/dkms.conf /usr/src/ixgbe-5.19.9/dkms.conf.bak should be enough. Then you can install the 6.8 kernel, though you will lose function of the sfp+ ports unless you pin the 6.5 kernel with the DKMS 5.19.9 driver.
Click to expand...
Done. And... yes, this allowed the 6.8 kernel install to complete. Thanks. This is a dirt simple solution and I should have thought of it. I was trying to temporarily pause the DKMS service, set the ixgbe 5.19.9 auto-install option to "no", etc. but nothing I tried cleared the road to allow the kernel to build.

I'm now pinned to the 6.5 kernel. Again, I was drafting on the work in this forum topic to get the x553 10G ports working before. I'm hoping there is a new fix shortly or that Proxmox gets around to properly supporting this NIC. When I purchased this device, I didn't realize that these NICs were a problem for Proxmox.

Is the best path to get 5.19.9 to compile under the new kernel or move ahead to 5.20.3?

Again, big thanks.
 
  • Like
Reactions: cr4ckDe and vesalius
mcraven said:
I'm hoping there is a new fix shortly or that Proxmox gets around to properly supporting this NIC. When I purchased this device, I didn't realize that these NICs were a problem for Proxmox.
Click to expand...

Same goes for me. Kinda frustrating, but I don't think there will be a fix any time soon.
 
So I did a bit more digging, but still no luck with this. I did find that the driver is properly detecting the DAC cable without complaints, but it's still stuck with the NO CARRIER message. This past week I ordered a pair of Intel-branded FTLX8571D3BCV-IT. I confirmed that it also detects these properly and does turn on the laser, but they will not connect either. So the problem does seem to be somewhere other than SFP detection. I tried one of the 1G SFP ports on my box and it also won't link up, so it seems all SFPs on this model NIC are just bricked with this new kernel.
Just sticking with 1G for now because it doesn't really affect what I'm currently running, but all those fancy ports I paid for are currently unusable.
 
mcraven said:
When I purchased this device, I didn't realize that these NICs were a problem for Proxmox.
Click to expand...
While this affects proxmox, it is not just a proxmox issue, it is a general Linux kernel driver issue, that affects every distribution using Linux kernel 6.* and above, so Ubuntu, debian, suse, etc ....

Plocmstart said:
So I did a bit more digging, but still no luck with this. I did find that the driver is properly detecting the DAC cable without complaints, but it's still stuck with the NO CARRIER message. This past week I ordered a pair of Intel-branded FTLX8571D3BCV-IT. I confirmed that it also detects these properly and does turn on the laser, but they will not connect either. So the problem does seem to be somewhere other than SFP detection. I tried one of the 1G SFP ports on my box and it also won't link up, so it seems all SFPs on this model NIC are just bricked with this new kernel.
Just sticking with 1G for now because it doesn't really affect what I'm currently running, but all those fancy ports I paid for are currently unusable.
Click to expand...

you can pin the last proxmox 6.5 kernel and build/install the 5.20.3 driver there to regain use of your 10g ports.
 
Last edited:
Just some info for the party: Intel's ixgbe driver is a few versions behind Linux kernel, stilling using removed kernel APIs so it won't compile on newest 6.8 kernel. xdp_do_flush_map in this case, by the look of things from other people's error message.

The in-tree ixgbe seems to lagging behind more (feature and bugfix wise) as it is showing multiple bugs that were fixed on Intel's version.

For anyone who cares enough, you can make money patches by switching to the new xdp_do_flush, which is how they (the kernel guys) made the in-tree ixgbe compile with kernel 6.8

Disclaimer: I don't work for Intel, I don't have any x553 laying around and I'm not running anything on kernel 6.8 Also it has been VERY long time since the last time I wrote any kernel/driver code so I'm not responsible for any consequences by doing that money patch. Refer to Intel/Linux kernel working groups for further information.
 
Last edited:
  • Like
Reactions: vesalius
vesalius said:
you can pin the last proxmox 6.5 kernel and build/install the 5.20.3 driver there to regain use of your 10g ports.
Click to expand...
Just took a look at 5.20.3 driver from Intel and it appears they already added money patch for kernel 6.8, so this version should compile with kernel 6.8 and there is no need to pin 6.5

Anyone care enough to try it?

Basically they replaced all xdp_do_flush_map to xdp_do_flush and retained backward compatibility through the following macro.
C: 
/* NEED_XDP_DO_FLUSH
 *
 * Upstream commit 1d233886dd90 ("xdp: Use bulking for non-map XDP_REDIRECT
 * and consolidate code paths") replaced xdp_do_flush_map with xdp_do_flush
 * and 7f04bd109d4c ("net: Tree wide: Replace xdp_do_flush_map() with
 * xdp_do_flush()") cleaned up related code.
 */
#ifdef NEED_XDP_DO_FLUSH
static inline void xdp_do_flush(void)
{
    xdp_do_flush_map();
}
#endif /* NEED_XDP_DO_FLUSH */
 
Last edited:
vesalius said:
you can pin the last proxmox 6.5 kernel and build/install the 5.20.3 driver there to regain use of your 10g ports.
Click to expand...
I pinned the 6.5 kernel and installed the 5.19.9 and 5.20.3 drivers, which sadly didn't work either. The link is up while booting but dissapears when Proxmox is booted.
 
cr4ckDe said:
I pinned the 6.5 kernel and installed the 5.19.9 and 5.20.3 drivers, which sadly didn't work either. The link is up while booting but dissapears when Proxmox is booted.
Click to expand...
Must be a different issue than what I am experiencing then, those out of tree intel drivers work fine to resurrect my SFP+ ports when installed in the Proxmox 6.5 kernel.
 
zzz09700 said:
Just took a look at 5.20.3 driver from Intel and it appears they already added money patch for kernel 6.8, so this version should compile with kernel 6.8 and there is no need to pin 6.5

Anyone care enough to try it?

Basically they replaced all xdp_do_flush_map to xdp_do_flush and retained backward compatibility through the following macro.
C: 
/* NEED_XDP_DO_FLUSH
 *
 * Upstream commit 1d233886dd90 ("xdp: Use bulking for non-map XDP_REDIRECT
 * and consolidate code paths") replaced xdp_do_flush_map with xdp_do_flush
 * and 7f04bd109d4c ("net: Tree wide: Replace xdp_do_flush_map() with
 * xdp_do_flush()") cleaned up related code.
 */
#ifdef NEED_XDP_DO_FLUSH
static inline void xdp_do_flush(void)
{
    xdp_do_flush_map();
}
#endif /* NEED_XDP_DO_FLUSH */
Click to expand...
Unless I am misunderstanding your suggestions, I did try already and the build errored out. See https://forum.proxmox.com/threads/intel-x553-sfp-ixgbe-no-go-on-pve8.135129/post-657758
 
vesalius said:
Unless I am misunderstanding your suggestions, I did try already and the build errored out. See https://forum.proxmox.com/threads/intel-x553-sfp-ixgbe-no-go-on-pve8.135129/post-657758
Click to expand...
No freaking idea why but, its true that intel still makes call to xdp_do_flush_map in 5.20.3
They actually have a check in place to see if xdp_do_flush is present: gen NEED_XDP_DO_FLUSH if fun xdp_do_flush absent in "$fh"
But never call it anywhere. (What is this? Half baked patch that, pretend it's trying to fix things?)

To me this looks like a bug.
Also I'm seeing 5.19.9 listed as "Latest" in Intel's site but sourceforge has 5.20.3 released April 26. Did Intel pull 5.20.3 from their website, or they just don't give a shit about their own sites?
https://www.intel.com/content/www/u...ethernet-network-connections-under-linux.html
 
Last edited:
  • Like
Reactions: Kingneutron
Pinned to 6.5.x like everybody else... considering moving from 5.19.9 to 5.20.3 on the 6.5.x kernel, for now...

That said, x553 is not new... any value in considering an older OOT driver version? I assume ixgbe.ko is included in the 6.8.x kernel... what version?
 
This is with the newest version 6.8.4-3
So I downloaded 5.20.3 from source forge and extracted it.

mkdir /src
cd /src
Get the tar file from here.
https://sourceforge.net/projects/e1000/files/ixgbe stable/5.20.3/

tar xvf ixgbe-5.20.3.tar.gz

installed gcc make buildtools-esentials

went to compile and got errors

There are roughly 10 errors we will have to address.

5 instances in a row of replacing strlcpy with strncpy because the compiler wanted me to do it.

on lines that it errors on replace xdp_do_flush_map with xdp_do_flush

In the ethtool.c
There will be 2 further errors
The last 2 parts of a structure, now I am sure it will come up later, but I use // to comment both of them out.

compiled again

a few more of the above xdp_do_flush_map rename xdp_do_flush

finally it compiles
make
make install

for luck I added

vi etc/modprobe.d/ixgbe.conf
options ixgbe allow_unsupported_sfp=1


rebooted
dhclient -v eno1


BING
I had an IP address and could ping things.


I hope someone creates an official patch so we don't have to dig around in the weds like this.

Remarking out 2 entries in the ethtool.c files seems especially dodgy, but it works for me.

-------
I got 5.19.9 to compile, but there were 2 more lines I had to comment out, and seemed even more sketchy to do so.
 
Last edited:
cybernard said:
This is with the newest version 6.8.4-3
So I downloaded 5.20.3 from source forge and extracted it.

mkdir /src
cd /src
Get the tar file from here.
https://sourceforge.net/projects/e1000/files/ixgbe stable/5.20.3/

tar xvf ixgbe-5.20.3.tar.gz

installed gcc make buildtools-esentials

went to compile and got errors

There are roughly 10 errors we will have to address.

5 instances in a row of replacing strlcpy with strncpy because the compiler wanted me to do it.

on lines that it errors on replace xdp_do_flush_map with xdp_do_flush

In the ethtool.c
There will be 2 further errors
The last 2 parts of a structure, now I am sure it will come up later, but I use // to comment both of them out.

compiled again

a few more of the above xdp_do_flush_map rename xdp_do_flush

finally it compiles
make
make install

for luck I added

vi etc/modprobe.d/ixgbe.conf
options ixgbe allow_unsupported_sfp=1


rebooted
dhclient -v eno1


BING
I had an IP address and could ping things.


I hope someone creates an official patch so we don't have to dig around in the weds like this.

Remarking out 2 entries in the ethtool.c files seems especially dodgy, but it works for me.

-------
I got 5.19.9 to compile, but there were 2 more lines I had to comment out, and seemed even more sketchy to do so.
Click to expand...
Looks like you made the same decisions I did, except for 'xdp_do_flush_map rename xdp_do_flush'. I'll give that a try this week.
I don't think the variables commented out in ethtool.c matter for this NIC, but that's just a half-educated guess.
 
On kernel 6.8.4-3, I've managed to compile ixgbe 5.20.3... module is untested yet though.

Confirming:
1) replace all "xdp_do_flush_map" with "xdp_do_flush"
2) in ixgbe_eththool.c:
- comment out 2x erroring struc lines
- replace "strlcpy" with "strscpy" in function def

I may not get to test right away as I'm traveling and I don't want to play around remote. Any others that take a swing, please post results of getting x553 NIC to come up at 10G. Comments on issues/performance welcome!
 
mcraven said:
On kernel 6.8.4-3, I've managed to compile ixgbe 5.20.3... module is untested yet though.

Confirming:
1) replace all "xdp_do_flush_map" with "xdp_do_flush"
2) in ixgbe_eththool.c:
- comment out 2x erroring struc lines
- replace "strlcpy" with "strscpy" in function def

I may not get to test right away as I'm traveling and I don't want to play around remote. Any others that take a swing, please post results of getting x553 NIC to come up at 10G. Comments on issues/performance welcome!
Click to expand...
Went ahead and gave it another go. Confirmed link is up after a reboot. Note that you have to do a reboot, just inserting the updated module does not seem to bring it to life (I think that was my original issue).

Code: 
root@pve:~# ethtool eno1
Settings for eno1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
 
  • Like
Reactions: avneeshsud and Kingneutron
Plocmstart said:
Went ahead and gave it another go. Confirmed link is up after a reboot. Note that you have to do a reboot, just inserting the updated module does not seem to bring it to life (I think that was my original issue).
Click to expand...

Which DAC/Transceiver are you using? I tried multiple transceiver and DACs and still didn't get it to work
 
cr4ckDe said:
Which DAC/Transceiver are you using? I tried multiple transceiver and DACs and still didn't get it to work
Click to expand...

Nevermind. The DACS and the transceivers worked flawlessly. I still didn't manage to get it working with proxmox, but I passed the PCIe Device to my pfSense VM and it worked right out of the box.
 
cr4ckDe said:
Nevermind. The DACS and the transceivers worked flawlessly. I still didn't manage to get it working with proxmox, but I passed the PCIe Device to my pfSense VM and it worked right out of the box.
Click to expand...
Either way I'm using this one: https://www.amazon.com/gp/product/B00WHS3NCA/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

10Gtek SFP+ DAC Twinax Cable, 10G SFP+ to SFP+ Direct Attach Copper Patch Passive Cable for Cisco SFP-H10GB-CU1M, Meraki MA-CBL-TA-1M, Ubiquiti UniF, D-Link, Fortinet, 1-Meter(3.3ft)​


Transceivers I picked up are Intel-branded FTLX8571D3BCV-IT.

One clue it was Proxmox is that the link comes up while booting/in BIOS and then the link goes down as Proxmox loads the network driver.
 
Anybody willing to make a real step by step guide to make this work with Proxmox 8.2?
I have tried so many guides and they are old or error out. And here it seems people got it working with the latest drivers and kernel?
 
You must log in or register to reply here.
Top Bottom
Back