[SOLVED] opnsense 10gbit performance and throughput limitation

[johen]

HI
I want to enhance my home server infrastructure with an advance firewall solution based on opnsense, pfsense or ipfire in a virtualized enviorment based on proxmox.
As this would allow my do this in a power and cost effiecent way, while still allow me to utilizing the 10G connection from my home server to my working PC.
To me the virtualized approach based on proxmox is here a good compormise for all aspects...

For this I deployed a test infrastructure to run some benchmarks on my home server (an i9 9900T) - but the results raises some questions.

What suprises me is that from the VM Opnsense but also an ipfire VM the max iperf thorughtput was ~4bit/s but from another linux a tuxedo test VM ~29GBit/s.
The CPU load (8cores where assigned) of the Opnsense and ipfire was still low, so I'm not sure where this limitation comes from?
As another test I used then a ubuntu VM and configured it for IPforward and then also for NAT and suprisingly here the throughput from LAN to WAN was ~20Gbit/s...

Is this really a limitation from opnsense and ipfire?
I read that Opnsense is capable of much higher throughput even when routing between two networks WAN-LAN and not as in my case if they are connected to the same network?
So I'm even not sure if this is actually a proxmox topic but any thoughts are welcome

Many thanks in advance!
best regards
Jochen

 

[johen]

[ 5] 89.00-90.00 sec 579 MBytes 4.86 Gbits/sec 1 1.33 MBytes
[ 5] 90.00-91.00 sec 596 MBytes 5.00 Gbits/sec 0 1.61 MBytes
[ 5] 91.00-92.00 sec 472 MBytes 3.96 Gbits/sec 31 1.40 MBytes
[ 5] 92.00-93.00 sec 561 MBytes 4.71 Gbits/sec 0 1.67 MBytes
[ 5] 93.00-94.00 sec 589 MBytes 4.94 Gbits/sec 0 1.91 MBytes
[ 5] 94.00-95.00 sec 536 MBytes 4.50 Gbits/sec 206 1.14 MBytes
[ 5] 95.00-96.00 sec 564 MBytes 4.73 Gbits/sec 0 1.45 MBytes
[ 5] 96.00-97.00 sec 590 MBytes 4.95 Gbits/sec 0 1.72 MBytes
[ 5] 97.00-98.00 sec 596 MBytes 5.00 Gbits/sec 101 1.39 MBytes
[ 5] 98.00-99.00 sec 564 MBytes 4.73 Gbits/sec 0 1.66 MBytes
[ 5] 99.00-100.00 sec 578 MBytes 4.84 Gbits/sec 0 1.90 MBytes
[ 5] 100.00-101.00 sec 576 MBytes 4.83 Gbits/sec 0 2.12 MBytes
[ 5] 101.00-102.00 sec 550 MBytes 4.61 Gbits/sec 393 1.67 MBytes
[ 5] 102.00-103.00 sec 590 MBytes 4.95 Gbits/sec 0 1.91 MBytes
[ 5] 102.00-103.00 sec 590 MBytes 4.95 Gbits/sec 0 1.91 MBytes

what I can see when routing through opnsense is that retries happens, which I think points to that packages are dropped...
total CPU load is 12-14%

 

[johen]

I can see that one core is utlilized 77-80% for interrupts..

I also tried the following and added for the vmbrX
post-up ethtool -K vmbrX tx off gso off
and also mtu 9000
but didn't make a difference

 

[johen]

For those who might end here as well
Good news I'm now able to achieve in my virtualized test environment ~21Gbit/s throughput between LAN <-> WAN
There are basically two points which I have identified:
1) Make use of multi core -> enabling multiqueue for the network virtio interface of my opnsense VM
For my 8 cores I configured 4 queues each

2) Make sure all nodes have configured MTU = 9000 - jumbo frames - it looks like the automatic selection is not always working...

Beside this to utilize the multiqueue I also discovered that I need to run iperf3 with -P option for parallel threads
I also reach 10gbit/s also with MTU size 1500

I will make more tests also switching back to linux bridge, because I have tried many different things... if maybe there is something else beside those two points.... UPDATE: Multiqueue also works fine with linux bridges

cheers
Jochen

FYI

CPU load with iperf3 -P 2


iperf3 -P 4