Horrible network performance with Atom C3758 + Intel X557-AT2

[LionelHutz]

I am testing a migration from ESXi to Proxmox.

I currently have a lab environment setup on ancient hardware/software.
Ancient setup is a SuperMicro A1SAi motherboard with Intel Atom C2750. Hypervisor is VMWare ESXi 5.5.
Onboard 1G NICs are driven by "igb" driver version 5.0.5.1. Add-on 10G NIC (Intel 82598EB-AT2) is driven by "ixgbe" version 3.7.13.7.14iov-NAPI.
The SLOWEST path in my setup is from LAN to Internet cable modem, which is reached on a 10G switch port, but the service is rate-limited to 300Mbps.
When I run a SpeedTest.net transfer through a virtual firewall hosted on this, I achieve >= 300Mbps (the ISP allows bursting and the typical performance is around 350Mbps).
This is to be expected, as 300Mbps on a NIC easily in range of something like an Atom C2750.
NOTE: I also regularly do transfers between internal VLANs through this firewall at around 5+Gbps, but this isn't even important for what I'm about to complain about...

I did a fresh install of Proxmox VE 8.2 onto a newer system with a SuperMicro A2SDi-H-TF motherboard at BIOS version 1.8. (CPU is Intel Atom C3758).
The kernel recognizes the onboard 10G NICs as device 8086:15c8 (Intel Ethernet X557-AT2) and binds the ixgbe driver (version 6.8.4-2-pve) to them.
Ethtool reports 10G link on the NICS. An FTP performance test between Proxmox and a LAN system shows transfer rates over 1Gbps, so I know the 10G link is being utilized.
NOTE: I know I'm bottlenecked to disk transfer rates with this kind of test. I'm not trying to establish the maximum performance, I'm just trying to prove we should be able to go at least 300Mbps on a guest. There are forum articles about people having problems with the X557 optical fiber links not linking up. That's not what I'm using. This is 10G copper and linking is not a problem.

Things get interesting when I try bringing a VM into this mix.
The default network bridge setup is used - vmbr0 is defined as using the one active NIC (eno1).
One virtual system is installed - a CheckPoint firewall running guest OS with kernel 2.6.18-92.
This guest is setup with identical parameters to its ESXi counterpart.
Machine type is "q35", vIOMMU is set to "VirtIO", CPU type is "host".
When I try the SpeedTest.net transfer through this system, the results are quite odd:
VirtIO virtual NIC: 16Mbps.
E1000e virtual NIC: 64Mbps.
VMXNet3 virtual NIC: 132Mbps.
(Note the binary progression here - I'm a bit surprised VMXNet3 isn't 128Mbps).

These results are confusing - particularly that VirtIO is the slowest in the group, but also that Proxmox on newer CPU/memory with a newer NIC can't match virtualization performance on something a decade older.
I have fiddled with processor type, machine type, and virtio type, but nothing seems to meaningfully change these numbers.
The ethernet driver packet counters aren't showing errors on guest or host. I just seem to have some phantom latency stealing performance.
I'd love to move over to Proxmox as a paying customer but can't see doing that when I'm immediately confronted with guest performance like this.
There isn't even contention on the Proxmox server at this point. This is one server running one VM trying to do the LEAST challenging thing my environment is setup for.
And, anticipating the question, no: I can't upgrade the guest OS kernel.
Also: No, I don't have a 1G NIC I can install to see if this is specifically X557 related. Just seems like given the raw FTP performance to the host OS, the guest performance should be able to "approach it", not be 2 orders of magnitude under it.

Anyone have anything to suggest? Something seems horribly wrong here and the performance numbers on the virtual NICs are hinting at some sub-optimal time-slicing going on in Proxmox.

 

[LionelHutz]

After extensive testing with a raw KVM implementation under Fedora 40 using several guest kernel versions, I've come to the conclusion the title of my post is just not-applicable to the real problem, so I wanted to post back some findings for future readers of the initial post.

My performance problem has nothing to do with CPU or NIC or even Proxmox. This is simply poor network performance between the host and guest VMs running VirtIO driver when the guest kernel is ancient. As I moved the guest kernel to more and more recent versions, the performance likewise improved (not shocking, of course, but I was surprised at just how bad it was under old 2.x kernels) . Unfortunately, my particular application is an old "network appliance" that bundles the kernel in with the application under a license, so I've got no freedom to update to a newer kernel without paying money for a license upgrade - which is basically the same problem I was trying to avoid with upgrading the ESXi. I do note that the kernel of ESXi operates more efficiently with guests running VMXNet3 NICs than the ProxMox kernel delivers to the same guests, but that's to be expected, given the driver origins. Unfortunately, the VMXNet3 performance is just too erratic to be acceptable.

With these findings, it's making the most sense to just ditch my appliance firewall for OpnSense and take advantage of the additional performance boost of virtual function adapters. Having setup a quick demo of this under Fedora 40, I got all the performance I expected, so I believe a ProxMox hosting would be equally performant.