Much higher CPU utilization when NIC is virtualized comparing to pass-through

[Sandbo]

Lately I am trying to optimize the OpenVPN transfer performance of my VM router inside PVE.

I have a 2-port NIC (Intel X550-T2, but the infrastructures are 1 Gbps) and they act as the WAN (port1) and LAN (port2) for my router VM (running ClearOS, a CentOS variant).
I tried two configurations:
1. Creating two Linux bridges, and assign the port1 and port2 to each bridge respectively. Then these two bridges are added as network interfaces (using Virtio driver) to the router VM. Finally, the LAN port connects to a physical switch and other computers in the LAN network.
This is what I read from the pfsense virtualized router guide.

2. Passing the 2-port NIC directly to the router VM and have the same configuration as above. I also created a Linux bridge with no port attached, this

In particular, I am interested in the performance with OpenVPN of the router VM. So with two other computers within LAN, I connect one computer to the OpenVPN server setup by the router VM, and have all traffic routed through it.
Then I setup a iperf3 server on a LAN computer A, and try to connect from the other computer B whose traffic goes through the OpenVPN server.

First of all, for some reason, with either of the above configurations I could only get 260 Mbps of throughput (maybe bottleneck by the CPU of the client). Then I try to observe the CPU utilization of the router VM using htop within the VM. As OpenVPN is a single-threaded application, I can see it clearly by checking how much a single core is being utilized.

With config 1, I am seeing 78% utilization with the above throughput.
With config 2, I am seeing only 35% utilization with the same throughput as config 1.

Is this normal to have such a big difference? Or is there any thing I can do further with config 1 to optimize the performance? Thanks.

 

[aaron]

I don't have much experience regarding the performance for these particular setups.

But regarding the OVPN throughput: What is the CPU config of the VM? Do you see the "aes" flag in the output of "cat /proc/cpuinfo" in the VM?

If that is not there you might be able to get better throughput by changing the VMs CPU to something newer (or even host if you don't need to migrate it to another node) that supports AES.

 

[Sandbo]

I don't have much experience regarding the performance for these particular setups.

But regarding the OVPN throughput: What is the CPU config of the VM? Do you see the "aes" flag in the output of "cat /proc/cpuinfo" in the VM?

If that is not there you might be able to get better throughput by changing the VMs CPU to something newer (or even host if you don't need to migrate it to another node) that supports AES.

Thanks for the ideas.
I was also reading up on what the CPU type does so I switched from KVM (which afaik is emulating Pentinum 4 and lacks a lot of flags) to host for all my VMs, including the router at the moment.

Checking the cpuinfo, it throws:

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca

which does contain the aes flag.

But it is actually possible that I was using other CPU type back when I was doing the throughput test, I may need to go back and test it again making sure host was used. (I guess I was using EPYC at some point, as I have an AMD Athlon 200GE)