Linux Bridge + VirtIO Performance: Ubuntu vs FreeBSD

M

mcraven

New Member
Jan 17, 2024
23
3
3
I've seen many posts about this but many seem dated...

I have a PVE node running on a C3758R CPU and I've created a Linux Bridge that is bound to a 10Gbps X553 port. On this box, I have 3x VMs and they are all sharing this bridge to my LAN:
1) Ubuntu Server 22.04.4 LTS (x86-64-v2-AES CPU + virtio enet)
2) OPNsense v24.1 [FreeBSD 13.2] (x86-64-v2-AES CPU + virtio enet)
3) pfSense 2.7.2 [FreeBSD 14.0] (x86-64-v2-AES CPU + virtio enet)

To test, I run an iperf server out on my LAN.... iperf from these VMs to the test server gives me ~8Gbps from the Ubuntu machine but both the FreeBSD VMs only give me ~700Mbps. Same bridge... same node... only difference seems to be Linux vs FreeBSD. Is the VirtIO adaptor on the FreeBSD VMs throttled by the firewalling activity???

Can anybody point me to a recent resource or offer some suggestions. I've tried most of the ideas I've found in other posts, including disabling hardware offloading, and enabling multi-queue on the VirtoIO adaptors.

Help would be appreciated.
 
Last edited:
_gabriel said:
https://docs.netgate.com/pfsense/en...ble-hardware-checksums-with-proxmox-ve-virtio
Click to expand...
this step was followed... for some reason, the VirtIO adaptor under FreeBSD seems to be throttled to under 1Gbps. I even tried setting up the iperf3 server on the Ubuntu server and just running the test within the PVE node:

Ubuntu VM <--> VirtIO <--> Linux Bridge (PVE) <--> VirtIO <--> FreeBSD VM [result: <1Gbps]

but, if I go from either of these VMs to an iperf3 server that is elsewhere in my LAN (over a 10Gbps wired link):
Ubuntu VM [result: 9Gbps]
FreeBSD VM [result: <1Gbps]
 
what about ubuntu VM to another ubuntu VM ?
to get your internal virtio nic speed bounded to your CPU.
iperf3 -P is CPU multithreaded only since version 3.16.
 
Last edited:
_gabriel said:
what about ubuntu VM to another ubuntu VM ?
Click to expand...

DONE! Both tests are run within the same PVE node

Ubuntu VM <--> VirtIO <--> Linux Bridge (PVE) <--> VirtIO <--> FreeBSD VM [result: <1Gbps] <==== OPNsense 24.1
Ubuntu VM <--> VirtIO <--> Linux Bridge (PVE) <--> VirtIO <--> Ubuntu VM [result: ~8-9Gbps]
 
_gabriel said:
in OPNsense/pfSense, have you double checked if "hardware checksum offload" is disabled ?
Click to expand...
I'll double check but I believe the answer is yes. That said, I did it via GUI only... Is there a better way or a command line way to verify?
 
Last edited:
You must log in or register to reply here.
Top Bottom