I'm extremely confused about interaction of network hardware checksum settings on the Proxmox host side and BSD/OPNsense guest side with virtio.

[marcosscriven]

I'm having difficulty getting good LAN and WAN performance at the same time on a virtualised opnsense firewall.

I'm unsure if it's a Realtek problem, an Proxmox/KVM problem, or a FreeBSD/OPNsense problem (or combination thereof)

What I know is that if I enable CRC in the OPNsense VM, then:

* LAN traffic improves markedly (from 700mbit to 940mbit)
* VM CPU usage drops to a quarter
* LAN -> WAN traffic crawls to a halt

Obviously I want the improved LAN speed without killing the WAN.

On the Promox side, I have two identical NICs:

driver: r8152
version: v1.12.12
firmware-version: rtl8153a-4 v2 02/07/20

The wan has its own NIC connected to a virtual bridge, and the LAN is connected to the default virtual bridge used by all VMs.

Does the NIC hardware have to support the same checksum settings as that advertised in the virtual bridge and virtio NIC?

Can anyone help with this please?

Note some references I found on this:

* https://pve.proxmox.com/wiki/PfSense_Guest_Notes
* https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059
* https://www.reddit.com/r/PFSENSE/comments/842unp/having_an_issue_with_virtualized_pfsense_speeds/

 

[bobmc]

well this is interesting...
I had been running pfsense with the virtio nic emulation and network interface settings as per the recommendations from Netgate and I have a 10Gb hardware nic on vmbr0 which all my VM's use for LAN including pfSense. It's been running like this for a couple of years and I've had no performance concerns.

After reading your post, I ran some iperf tests on the LAN side of pfSense and was surprised to see I was only getting between 1Gb/s and 2Gb/s - not at all what I was expecting.

However, after enabling all the 'offload' options (checksum, TCP segmentation and large receive) speeds jumped to 8Gb/s from pfSense to a physical Windows client and up to 14Gb/s between pfSense and the Proxmox Host. CPU load is a little higher but remains below 30% and my WAN speed is normal at 200Mb/s

Given that pfSense and OPNSense are closely related I would expect similar results.

 

[marcosscriven]

and my WAN speed is normal at 200Mb/s

Interesting! In my case WAN speed plummeted, which is a known thing apparently: https://docs.netgate.com/pfsense/en...ble-hardware-checksums-with-proxmox-ve-virtio

I wonder how it works for you?

 

[bobmc]

Are you using virtio or realtek drivers?

 

[marcosscriven]

virtio

 

[bobmc]

so the main differences are that my WAN nic is an Intel 82576 (where yours is a realtek) and I am running pfSense+ which is still on FreeBSD 12.3 (where you are on OpnSense which is based on FreeBSD 13)

I did find this post https://forum.opnsense.org/index.php?topic=16531.0

which might be worth trying (you'll need to install ethtool)