"Best" virtual adapter

[HelliR]

Hello,

i just migrated all my VMs from VMWare to PVE successfully, smoothly. I'm very impressed.

While my Windows and Linux machines work fine with the VirtIO network adapter my FreeBSD 14 don't. SSH, SMTP, HTTP, ... work fine but OpenVPN does not like the apapter at all, constant connection drops and very slow performance. Same with DHCP.

I switched to 1000e which immediately worked flawless. DIgging around I found out that the VMNET3 adapter not only works on VMWare but also on Proxmox. I thought this is something VMWare-specific?!

Now, that I came from VMWare and the VMNET3 was running flawless I tried it out and it also works great with FreeBSD.

Question:

- Is there anything known about issues with VirtIO and FreeBSD on Proxmox?
- Am I correct that after the VirtIO VMNet3 is the best choice in regards of performance?

Thank you!

 

[jauling]

I'm new to Proxmox and FreeBSD. I am playing with OPNsense 25.1, which runs on FreeBSD 14.2. I have got 2 other VMs running on my Proxmox host, both running Ubuntu 24.04. I'm virtualizing OPNsense primarily in hopes that it will alleviate single threaded pain of PPPOE on my WAN interface.

Just for funsies, I was testing iperf3 performance between all the VMs, as well as between the VMs and the Proxmox host. With the OPNsense VM I also tried all 5 different NIC hardware (E1000, E1000E, VirtIO, Realtek, vmxnet3), as well as trying iperf3 from single user mode, in hopes that bypassing firewall/routing services would make a difference. None of it mattered. For the OPNsense VM, all the NIC hardware performed horribly compared to VirtIO, like no contest. I saw Mbps throughput, sometimes even double digit Mbps numbers using iperf3 to the Proxmox host when using the other 4 NIC models with my OPNsense VM.

Here are some iperf3 results I found. OPNsense and Ubuntu 24.04 VMs running VirtIO network hardware. Not sure if it matters, but I saw around 40-45Gbps when running iperf3 -c localhost on each. All VMs and Proxmox on the same /24 network. Proxmox 8.3.3.

Proxmox -> OPNsenseVM: 3.3 Gbps
OPNsenseVM -> Proxmox: 4.7 Gbps
Proxmox -> UbuntuVM1: 20.2 Gbps
UbuntuVM1 -> UbuntuVM2: 16 Gbps
OPNsenseVM -> UbuntuVM1: 4.7 Gbps
UbuntuVM1 -> OPNsenseVM: 2.9 Gbps

I've just downloaded FreeBSD 14.2 and FreeBSD 13.4 qcow2 images, so I'll give those a whirl in the next few days and see if it makes a difference.

FWIW, I don't run 10 Gbit Ethernet at home, so maybe I shouldn't care so much. My internet connection is also just 1Gbit symmetric. But I would hope that my router/firewall would not be the bottleneck.

 

[mcraven]

Proxmox -> OPNsenseVM: 3.3 Gbps
OPNsenseVM -> Proxmox: 4.7 Gbps
Proxmox -> UbuntuVM1: 20.2 Gbps
UbuntuVM1 -> UbuntuVM2: 16 Gbps
OPNsenseVM -> UbuntuVM1: 4.7 Gbps
UbuntuVM1 -> OPNsenseVM: 2.9 Gbps

Any more learning here? I've just moved my setup to Proxmox v9 and I'm using OPNsense 25.7 in a VM... I'm currently passing PCIe ethernet adaptors to the VM but this makes my VM tethered to that host when, ideally, I'd prefer to be using a virtual adaptor. I am running 2.5Gb and 10Gb links... while overkill (for now), I'd like to get the OPNsense VM setup to (future proof) maximize performance AND be using a virtual adaptor...

I too tried some of the above iperf tests and had similar results... my tests were are a bit stale now but I've been hoping the FreeBSD and Proxmox updates might have brought new FreeBSD performance...

*sigh*

 

[cwt]

There are several options to improve performance.

- CPU type: host (especially AES-NI is important for VPN)
- VirtIO with multiqueue (4 or 8)
- dedicated VMBR for WAN
- disable PVE firewall on all OPNSense vNICs
- no nested NAT
- disable checksum offload & tso within VM
- optimize MTU/MSS clamping for VPNs

 

[meyergru]

Any more learning here? I've just moved my setup to Proxmox v9 and I'm using OPNsense 25.7 in a VM... I'm currently passing PCIe ethernet adaptors to the VM but this makes my VM tethered to that host when, ideally, I'd prefer to be using a virtual adaptor. I am running 2.5Gb and 10Gb links... while overkill (for now), I'd like to get the OPNsense VM setup to (future proof) maximize performance AND be using a virtual adaptor...

I too tried some of the above iperf tests and had similar results... my tests were are a bit stale now but I've been hoping the FreeBSD and Proxmox updates might have brought new FreeBSD performance...

*sigh*

There is a tutorial on how to correctly use vtnet adapters in OpnSense here.