Linux Bridge + VirtIO Performance: Ubuntu vs FreeBSD

[mcraven]

I've seen many posts about this but many seem dated...

I have a PVE node running on a C3758R CPU and I've created a Linux Bridge that is bound to a 10Gbps X553 port. On this box, I have 3x VMs and they are all sharing this bridge to my LAN:
1) Ubuntu Server 22.04.4 LTS (x86-64-v2-AES CPU + virtio enet)
2) OPNsense v24.1 [FreeBSD 13.2] (x86-64-v2-AES CPU + virtio enet)
3) pfSense 2.7.2 [FreeBSD 14.0] (x86-64-v2-AES CPU + virtio enet)

To test, I run an iperf server out on my LAN.... iperf from these VMs to the test server gives me ~8Gbps from the Ubuntu machine but both the FreeBSD VMs only give me ~700Mbps. Same bridge... same node... only difference seems to be Linux vs FreeBSD. Is the VirtIO adaptor on the FreeBSD VMs throttled by the firewalling activity???

Can anybody point me to a recent resource or offer some suggestions. I've tried most of the ideas I've found in other posts, including disabling hardware offloading, and enabling multi-queue on the VirtoIO adaptors.

Help would be appreciated.

 

[_gabriel]

https://docs.netgate.com/pfsense/en...ble-hardware-checksums-with-proxmox-ve-virtio

 

[mcraven]

https://docs.netgate.com/pfsense/en...ble-hardware-checksums-with-proxmox-ve-virtio

this step was followed... for some reason, the VirtIO adaptor under FreeBSD seems to be throttled to under 1Gbps. I even tried setting up the iperf3 server on the Ubuntu server and just running the test within the PVE node:

Ubuntu VM <--> VirtIO <--> Linux Bridge (PVE) <--> VirtIO <--> FreeBSD VM [result: <1Gbps]

but, if I go from either of these VMs to an iperf3 server that is elsewhere in my LAN (over a 10Gbps wired link):
Ubuntu VM [result: 9Gbps]
FreeBSD VM [result: <1Gbps]

 

[_gabriel]

what about ubuntu VM to another ubuntu VM ?
to get your internal virtio nic speed bounded to your CPU.
iperf3 -P is CPU multithreaded only since version 3.16.

 

[mcraven]

what about ubuntu VM to another ubuntu VM ?

DONE! Both tests are run within the same PVE node

Ubuntu VM <--> VirtIO <--> Linux Bridge (PVE) <--> VirtIO <--> FreeBSD VM [result: <1Gbps] <==== OPNsense 24.1
Ubuntu VM <--> VirtIO <--> Linux Bridge (PVE) <--> VirtIO <--> Ubuntu VM [result: ~8-9Gbps]

 

[_gabriel]

in OPNsense/pfSense, have you double checked if "hardware checksum offload" is disabled ?

 

[mcraven]

in OPNsense/pfSense, have you double checked if "hardware checksum offload" is disabled ?

I'll double check but I believe the answer is yes. That said, I did it via GUI only... Is there a better way or a command line way to verify?

 

[_gabriel]

OK,
So, perhaps Atom C3758R can handle 10 Gb/s only with offloading some process, which missing in a virtualized FreeBSD.
Try with a Linux Firewall VM like ipfire.

Also, someone do some tests :
https://github.com/opnsense/src/issues/163