FreeBSD + Virtio => Slow (5GBit/s)

[Thalhammer]

Hi,
I know this has come up a bunch of times, but nothing metioned anywhere seems to work for me.
I do have Hardware offload in freebsd disabled and I had it disabled on both bridge and nic of the host, but it did not change anything.
No matter what I do I can't get more than about 4-5Gbit/s to or from any FreeBSD VM i have.
For comparison, an ubuntu vm on the same host also with virtio does ~30Gbit/s without breaking a sweat.
Any other idea I could try ?

 

[fwcheck]

You could try jumbo-frames. Are you using a firewall (like pf) on FreeBSD ? Is the result CPU bound (are you using 100% cpu ?)?
Have you tried to set network-card-Features ? As far as i remember this could be due to LRO, TSO etc. not set on Freebsd.

 

[Thalhammer]

I tried various different freebsd installations, first I noticed it on pfsense, but I dont care about that one cause 5GBit/s is way over my internet.
The one I care about is FreeNAS cause my raid array is easily capable of outperforming this speed even without caching.
But for reference I also tried a brand new stock install of FreeBSD (Only thing I did after install was installing iperf) and all of them show exactly the same behaviour. ~4-5GBit/s and comparatively high cpu usage by kvm. I dont think that its cpu bound though, cause still neither of the cores hits 100% load.
If I do iperf (from proxmox) to/from a freebsd vm I have two cores at around 70% (+15% on the one core running iperf on the host. While doing the same with a ubuntu server vm results in about 6x the troughput at lower absolute cpu load. However If I open top on the freebsd machine during the test it shows almost no load, which leads me to believe that the time is somehow spent inside kvm emulating what should be very low overhead virtio. Both proxmox and Freebsd are running the latest available version.

I can rule out TSO/GSO... cause I literally did a whole spreadsheet of tests in both directions with various combinations of enabled/disabled on both proxmox and the client only to find out that it did not have any effect at all.
I can't try out jumbo-frames cause the nic is bridged to my home network and not all devices support them.

I disabled all firewall features on both proxmox and as far as I know freebsd which also did not change anything.