pfSense VM - very slow network throughput

[pederko]

The host is Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2 Sockets).

If there is a CPU issue I can surely assign more CPU to the pfsense VM, but the CPU usage is very low on pfsense when I do iperf3 testing between VM and proxmox.

UPDATE: Added some more CPU to pfsense VM - still the same.

From VM -> Proxmox: (iperf3 -c 192.168.1.2)
[ 5] 0.00-10.00 sec 2.76 GBytes 2.37 Gbits/sec 25 sender
[ 5] 0.00-10.00 sec 2.76 GBytes 2.37 Gbits/sec receiver

From VM -> Proxmox Reverse (iperf3 -c 192.168.1.2 -R)
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver

 

[Ballistic]

Virtualisation overhead. You're not going to get linespeed.

What speed are you getting between VM's over a bridge?

 

[pederko]

Between 2 Linux VM's using same bridge I get:

From WebServer VM -> TrueNAS VM (iperf3 -c 192.168.1.3)
[ 5] 0.00-10.00 sec 13.6 GBytes 11.7 Gbits/sec 0 sender
[ 5] 0.00-10.04 sec 13.6 GBytes 11.6 Gbits/sec receiver

From WebServer VM -> TrueNAS VM Reverse (iperf3 -c 192.168.1.3 -R)
[ 5] 0.00-10.04 sec 16.1 GBytes 13.7 Gbits/sec 1 sender
[ 5] 0.00-10.00 sec 16.1 GBytes 13.8 Gbits/sec receiver

What's the difference between 2 Linux VM (TrueNAS & Webserver) and TrueNAS -> pfsense or WebServer -> pfsense
Between 2 Linux VM I get over 10 Gbit/s, but when trying with pfsense the speed is much lower...

 

[Ballistic]

I tested/compared for you.

On an Intel NUC with i5, I don't have a VM so i cannot test identical. But i do have an LXC container.
LXC -> LXC = 38Gbit/sec
PFsense VM -> LXC = 2.95Gbit/sec

Stepping it up to my big boy environment; A dedicated PFsense machine with a Xeon E-2236 and as Proxmox machine a Xeon Silver 4314 with Intel nics. Everything on the VM side is Virtio.
LXC -> LXC over bridge = 46.9Gbit/sec
PFsense physical -> LXC = 4.8Gbit/sec
PFsense physical -> Windows VM = 4.3Gbit/sec

LXC -> LXC, routed and inspected(Suricata) via PFsense = 7.4Gbit/sec (55%+ PFsense CPU usage!)
LXC -> Internet, routed, inspected(suricata) and NAT'ed via PFsense = 6.5Gbit/sec (55%+ PFsense CPU usage!)

I"m no expert but my best guess is that Pfsense is optimzed for forwarding traffic. Not for host traffic. So don't judge the max performance of the machine on an iperf test.

So;
1: Don't run virtualised if you expect every bit of performance
2: Get a fast high frequency CPU (single core performance)
3: Don't test TO PFsense, Do test THROUGH PFsense

 

[Urbaman]

Hi,

I have the same problem on my pfSense VM.

• 2,5G fiber from ISP
• Proxmox gets 2G throughtput if tested directly on vmbr0
• Simple Ubuntu VM gets 2G throughtput if tested directly on vmbr0

So it seems not to be hardware related?

• pfSense gets capped at roughly 0,5G on VirtIO NIC on vmbr0
• I have 8 (2x4) CPUs and 24G RAM on pfSense, so it seems resources are not the problem
• Promxox though pfSense is obviously capped to 0,5G
• Other 10G Nics on the Proxmox machine(s), both from the same card and form other cards, sign 9G with iperf through a 10G switch

Only problem seems to be with pfSense somehow.