10Gbps performance differences on Proxmox 6.2 vs Debian Live USB 10.5

[newing]

I have recently been setting up 10Gbps for two Proxmox nodes. I have found that when using Proxmox 6.2 on both machines, the lower powered machine is only capable of sustaining speeds of around 5.8Gbps when on the receiving side (`iperf3 -s`) of an iperf test. When that host is on the sending side, the speed is consistently around 9.4Gbps.

At first I would have possibly attributed the difference to the hardware of the lower powered machine, but then I ran the same tests using a Debian Live 10.5 USB on both hosts. With that setup, the same hardware was able to sustain speeds of around 8Gbps and 9.4Gbps, so around a 2Gbps improvement on the receiving side of the lower powered host.

One thing I noticed when using the Proxmox install, during only one of the iperf runs, the bitrate started at around 9.4 and then quickly fell to its normal 5.8Gbps.

I have tried several options I've found online to get the speeds on the Proxmox install to match that of the plain Debian install, but have not been able to get quite as high of speeds.

Setting txqueuelen to 10000 (`ip link set dev eno3 txqueuelen 10000`) improves things slightly, as does setting both the rx/tx ring params to 4096 (`ethtool -G eno3 rx 4096 tx 4096`).

Iv'e tried changing the tx, rx, tso, gso with ethtool, but any changes I've made there have only decreased the bitrate. Currently I'm using the following ethtool config: `ethtool -K eno3 tx on rx on tso on gso on lro off`

I checked the output of `ethtool -k` on both Proxmox and Debian Live and they were identical. Also worth noting that in both cases, LRO was disabled.

The hardware I'm using, in case it matters:
Node 1: Supermicro X11SSH-F motherboard with Xeon E3-1275 CPU, and an X520-DA2
Node 2: A2SDI-H-TP4F motherboard with integrated Intel Atom C3958 and integrated SoC 10Gbps SFP+
10Gbps SFP+ switch

Is there anything else I should try, or any other settings I should compare in the two setups to match the bitrate from the Debian Live USB on the Proxmox install?

Thanks in advance for any help!

 

[mira]

Can you try a current Ubuntu 20.04 Live version? PVE uses the Ubuntu 5.4 kernel and there might have been changes to the driver or somewhere else that result in lower speeds.

 

[newing]

Thanks for the tip, Mira. I had thought PVE used Debian hence the choice to use the Debian live USB.

Had to find some time to get that all set up, but it does seem like the kernel difference could be at least contributing to this. Using the Ubuntu 20.04 Live USB, I was seeing similar 6Gbps speeds on the receiving side of the lower powered machine.

Given that, is there anything else I can try here, or am I just stuck with the slower speeds if I want to use the proxmox distro?

 

[mira]

You could try booting older kernels in PVE. Start with 5.3 and if it still has the same issue, boot kernel 5.0.
You can install the older kernels with apt install pve-kernel-5.3 (latest 5.3 one) and apt install pve-kernel-5.0.

This can help narrow down when the issue first appeared. Which exact NIC model are you using?

 

[newing]

Thanks again for the response. I will have to look into booting into an older kernel.

On the receiving side where I am seeing the slower speeds, I am using the motherboard-integrated SFP+ from the Intel Atom C3000 SoC. The exact motherboard I am using is here: https://www.supermicro.com/en/products/motherboard/A2SDi-H-TP4F. On the sending side I am using an X520-DA2.

 

[mira]

Which drivers are in use? You can find out by running lspci -k on both hosts.

 

[newing]

I have not yet had time to check different kernel versions, but I am starting to think that the issue doesn't actually have anything to do with the distro actually. Oddly it seems like it might actually be that the sending machine (the slightly higher powered one) is not able to send at the higher bitrate when iperf3 is ran from within a VM.

I was able to actually get around 9.41Gbps from host A to host B and 8.7Gbps from host B to host A with all the updated settings and not using a VM on either side but when host B is a VM, the bitrate goes down to around 7Gbps.

Checking the driver versions, I see that both are using ixgbe:

06:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553/X557-AT 10GBASE-T (rev 11)
    Subsystem: Super Micro Computer Inc Ethernet Connection X553/X557-AT 10GBASE-T
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe
06:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553/X557-AT 10GBASE-T (rev 11)
    Subsystem: Super Micro Computer Inc Ethernet Connection X553/X557-AT 10GBASE-T
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe
07:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
    Subsystem: Super Micro Computer Inc Ethernet Connection X553 10 GbE SFP+
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe
07:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
    Subsystem: Super Micro Computer Inc Ethernet Connection X553 10 GbE SFP+
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe

06:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553/X557-AT 10GBASE-T (rev 11)
    Subsystem: Super Micro Computer Inc Ethernet Connection X553/X557-AT 10GBASE-T
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe
06:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553/X557-AT 10GBASE-T (rev 11)
    Subsystem: Super Micro Computer Inc Ethernet Connection X553/X557-AT 10GBASE-T
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe
07:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
    Subsystem: Super Micro Computer Inc Ethernet Connection X553 10 GbE SFP+
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe
07:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
    Subsystem: Super Micro Computer Inc Ethernet Connection X553 10 GbE SFP+
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe

Both machines are running the same install of Proxmox which includes the ixgbe driver version of 5.1.0-k. It seems that the newest version of that driver is now 5.8.1. I don't have any experience with updating linux drivers. Should I try building and installing the newer version of the driver on both machines? How can I revert if it doesn't work out?

Thank you again