pfSense VM - very slow network throughput

[stuartbh]

ProxMox users, et alia:

I have a VM running the latest community edition version of pfSense on the latest version of ProxMox on an HP T620 with 16GB of RAM, 4 x AMD GX-415GA, and the internal Ethernet device is a RTL8111/8168/8411 device (the driver in use is the VirtIO driver).

The mechanism that I used for testing the VM and hosts involved herein was iperf3 (iperf 3.12 on a GD8200 laptop using a GiG-E Ethernet port connected full duplex @ 1Gbps).

Initially I noticed that the VMs network throughput was lower than I expected it should be, though I also had a FreeIPA VM running on the HP T620 as well. Follows hereupon an instantiation of the experimental results I obtained.

In order to assure that I am not merely measuring the residual bandwidth of what pfSense may otherwise be using for its normative routing work I looked at the graph on the summary page for the pfSense VM and ran my iPerf3 test at a time whence my network traffic in and out of the pfSense VM was relatively quiet; the foregoing notwithstanding, the issue remained persistent.

My original deployment of pfSense was to run it directly on the T620, however, that left me with WAN disconnects all the time which ended the moment I installed ProxMox on the T620 and virtualized pfSense. I do not remember what the performance was like running pfSense bare metal, but I do not recall it being much better.

Normally my T620 runs the pfSense VM and a FreeIPA VM.

Experiment 1:

I migrated the pfSense VM to an IBM x3650 M3 box I have that runs ProxMox (in my cluster) whereupon the pfSense VM was the only VM running on that server under ProxMox and the performance was as such:

iPerf3 to the host: 947 Mbits/second
iPerf3 to the VM: 833 Mbits/second

Experiment 2:
I moved the FreeIPA VM off of the HP T620 and left only the pfSense VM running on it and observed the follow performance as such:

iPerf3 to the host: 935 Mbits/second
iPerf3 to the VM: 290 Mbits/second (on average, as over a few runs I got between 260-335Mbits)

In the end it may just be that the T620 is not good for running pfSense on and I'll use it for running other VMs where network performance is far less of an issue (like FreeIPA or other VMs with low network traffic). That said, I'd like to at least identify the problem and a potential solution if practicable, but if no solution is viable then I can run pfSense on a different server for sure.

I welcome the comments and advice of anyone that can provision input regarding the instant matter.

Stuart

 

[_gabriel]

have you set vcpu type to host ?

 

[stuartbh]

_Gabriel,

Well, no, my vCPU is set to KVM64 so that the machine can migrate betwixt one AMD based system I have and several Intel based systems I have (the HP T620 is AMD based). I can certainly exact an experiment and try this change to see what occurs (likely later tonight when I have time). Honestly, I have had challenges getting reasonable network performance out of the HP T620 even when I ran pfSense bare metal, so it may just be that this Realtek chip set with whatever firmware it has is just crap, and if so, so be it, I'll use it for something less network intensive.

(from my ProxMox instances running on the HP T620):
$ sudo lspci -v -s 01:00.0
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
    Subsystem: Hewlett-Packard Company RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
    Flags: bus master, fast devsel, latency 0, IRQ 27
    I/O ports at e000
    Memory at fea00000 (64-bit, non-prefetchable) [size=4K]
    Memory at d0800000 (64-bit, prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [70] Express Endpoint, MSI 01
    Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
    Capabilities: [d0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Virtual Channel
    Capabilities: [160] Device Serial Number xx-xx-xx-xx-xx-xx-xx-xx <- redacted
    Capabilities: [170] Latency Tolerance Reporting
    Kernel driver in use: r8169
    Kernel modules: r8169

From pfSense:

sudo pciconf -l -b -v virtio_pci3@pci0:0:18:0
virtio_pci3@pci0:0:18:0:    class=0x020000 card=0x00011af4 chip=0x10001af4 rev=0x00 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio network device'
    class      = network
    subclass   = ethernet
    bar   [10] = type I/O Port, range 32, base 0xd300, size 32, enabled
    bar   [14] = type Memory, range 32, base 0x81440000, size 4096, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xc04000c000, size 16384, enabled

So from the pfSense perspective it sees a VirtIO based network card, thus I would imagine that any driver deficiency (bugulance) would either have to be the Linux driver or the pfSense VirtIO driver it is using. However, the sub standard performance is not shown when I run iPerf3 against the ProxMox vmbr0 Ethernet port only when run against the pfSense bridged Ethernet port.

The easiest solution is to run the pfSense VM on one of my NUCs or x3650 boxes that offers far greater throughput with Intel based NICs. Having the T620 as simply "a third/fourth cluster member" is fine and there are some things like Fedora CoreOS that can run fine on it and handle their tasks despite the lower network bandwidth on the Realtek NIC.

I think in the future I will try to use only Intel processors and network chip sets for systems I intend to have within my ProxMox cluster, or at least if I have AMD systems, create a cluster of just AMD systems and a different cluster for just Intel based systems.

Stuart

 

[Spoonman2002]

Try set nic hardware offloading to OFF in pfSense.
(disable all offloading options for the nic, three tickboxes)

 

[stuartbh]

Spoonman2002,

In precedence to any changes or reboots, I measured 300Mbits/second throughput for send and also for receive. I then disabled "Hardware Checksum Offloading" (the other two were already disabled), saved the change and rebooted. In succession to the reboot I measured about the same throughput. Well, it was worth a try anyway!

I do most certainly appreciate the suggestion and it was totally worth a try!!

I will probably try a USB3 based Ethernet adapter I have (just for fun) as the simple solution is to run the VM on a server where it gets close to 975Mbits/second honestly. But, clearly, there is something different about that Realtek interface, the T620, the Linux drivers, the pfSense (FreeBSD) drivers, something?!?


Stuart

 

[_gabriel]

report with cpu type to host. just a try.

 

[stuartbh]

Gabriel,

This evening I did indeed exact this experiment of changing the pfSense's VM vCPU to "host" (from "kvm64") and I did not see any material differential in the measurements taken via iPerf3 using the same parameters. As such, the average bit rate was about 300Mbits/second or so. That said, Computer Science (like any other science) requires experimental data to rule out a hypothesis.

I think my next experiment is to try an USB3 Ethernet adapter in one of the USB3 ports on the T620 and see what results I get. If that proves to have a close to 975Mbits/second bit rate then this does suggest the issue is in some manner related to the Realtek chip set and driver combination I suppose.

This does lead me to think that maybe whatever the issue is it might be something specific to the HP T620. If that is the case, then it is understandable as the T620 was designed to be a workstation and not a router or server, so are my expectations to high? Maybe.


Stuart

 

[stuartbh]

Gabriel, et alia:

My T620 also has a FreeIPA VM running on it and for kicks and giggles I installed iperf3 thereupon this morning. Succeeding that, whence I ran the iperf3 client (on my GD8200) against the FreeIPA VM I was getting bit rates averaging 930-940Mbits/second. Thus, I realize I am resigned to accept that whatever the issue is with poor throughput on the pfSense VM on the T620 it is related to something within the environment on that VM specifically.

Stuart

 

[Ballistic]

Also running into this issue. It seems it happens since PFsense 2.6. Same problem in 2.7.
2.5 was fine.

iperf over LAN to pfsense = gbit
internet directly on the ISP router but through my network/vlan = 900down/100up
internet behind PFsense NAT = ~270down/9up

Have not found the problem yet.

NIC hardware offload all disabled
Host CPU (core i5)

 

[_gabriel]

have you tried 4x vcpu ?
have you tried hw option virtio nic multiqueue to 4 ?

 

[stuartbh]

Gabriel, Ballistic, et alia:

Whereas there has been a rather significant intercession of time since the last moment I posted regarding the issue subject to this epistle, it is worthy of notation that I am now running ProxMox 8.0.4 and pfSense 2.7.0 software releases. Moreover, the OpenWRT version of both my wireless routers has been upgraded as well. The significance of this is not remarkable as I made no effort to track the lackluster network performance relative to the release cycle of any of these operating environments (ProxMox, pfSense, OpenWRT, etc...).

However, the foregoing notwithstanding, it is interesting to note that the issue has indeed evaporated from my environment. All I can do now is enunciate the configuration as I currently see it. Taking into account that the pfSense VM is currently running upon an Intel NUC it becomes requisite to move the VM back to the HP T620 in order to obtain new metrics via iperf3. Additionally, please note that pfSense is using the network model of "virtio (paravirtualized)" within its VM configuration.

In succession to having moved the pfSense VM back to the HP T620 the metrics instantiated by taking new iperf3 measurements seem to suggest that the issue is reappearing. Thus, one is left only to conclude that there is some incompatibility with the HP T620 (i.e. embedded chipset) and the drives executing that cause this issue to exist.

I hope this information is useful as you work to troubleshoot the issues present in the instant case before us.

Stuart

 

[Ballistic]

I have 2 identical scenarios.
Intel NUC Celeron 2820 -> Proxmox8 -> PFsense 2.7 = OK. I reach the speed of the WAN connection (100/30mbit)
Intel NUC i5 8259U -> Proxmox8 -> PFsense 2.7 = Not-OK. WAN is 1000/100 but only reaching 270/9mbit when traffic is NAT'ed

Nothing is different between these systems.
Hardware/settings of the VM are identical (host cpu, no balloon, virtio disk & nic, os other, no qemu-guest-agent)

I have added the max amount of vcpu's. No change
Added 2gb of ram (4 total). No change
Added multiqeue nic maching the number of cores. No change

 

[stuartbh]

Ballistic,

Interesting indeed!

When I move the pfSense virtual machine to either one of my IBM x3650 servers (M1 or M3) or one of my NUCs (NUC7i7DNHE or NUC7i7BNB) it runs fine just not on the HP T620.

The only thing I blamed it on, was the HP T620 being AMD, based, whereas the other systems were all Intel-based. I think the HP T620 has a Realtek network chip in it.

I presume you have the updated current BIOS versions for the NUCs?

What version of pfSense are you running? The new 2.7 version is using a much later version of FreeBSD. I have not tested running my upgraded pfSense on the HP T620 after upgrading to 2.7 version.

Stuart

 

[Ballistic]

Good point on the bios. The system had a 1000 days uptime so the bios is at least that old. I'll give the update a go.

Both systems originally started on Pfsense 2.5 and have upgraded them to 2.6 and 2.7 along the way.
Never noticed the issue on 2.5. Once we noticed the speed problem, we where running 2.6 already but no 100% guarantee yet that it's related. Because as stated; 2.6/2.7 on the Celeron works fine but not on the i5.

 

[Ballistic]

Allright. Issue found!

On the original 2.5 setup, there was a traffic limiter of 300/10 configured on a guest VLAN interface. Worked as designed.
Somewhere during the update to 2.6, this limiter somehow shifted to the WAN interface!

I'm saying somehow because I don't know how. The limiter was configured under Firewall->Traffic Shaper->Limiters but no interface or rule was present anymore where these limiters where applied.
Upon deleting the limiter, i lost all connectivity to the box and had to go on-site. This happened BEFORE applying the deletion!! When I came on-site, everything looked fine but no internet connectivity. I was presented with the green "Apply to take effect" button. After pressing it, everything came back online and got speeds of 800/100mbit again

 

[pederko]

Hi guys! Sorry for hijacking this post - but I have some kind of same issue.

Setup: Proxmox 8.0.3 and Pfsense 2.7.0 (tried 2.6.0 as well...)
PCI Passthrough of 1 network port (10 GBE) to Pfsense used as WAN adapter.
Bridged the second port (10GBE) to Pfsense used an LAN adapter.

iperf3 between proxmox and pfsense gives about 2.17 Gbit/s.
My WAN connection (connected to fiber ONT with 10Gbps link) has a 4 Gbps internet speed.
With 2.17 Gbit/s between pfsense and proxmox I'm not able to use the whole capacity from my ISP.

Does anyone know if there are any speed limitations when using pfsense in VM?

(For the record - when I connect fiber ONT directly to NIC that isn't passthrough I get 4 Gbit upload/download from ISP)

 

[_gabriel]

vm use Linux bridge which depend on cpu.
What's your host cpu ?

 

[Ballistic]

Like Gabriel mentioned, everything is running on CPU. Not a specialized ASIC packet handler. Don't expect to reach multi gbit speeds easily.

Even my fastest pfsense box which is not virtualized "only" does about 5.5Gbit/sec sustained (Xeon E-2236, Chelsio T520-CR)

 

[_gabriel]

Here I get (will update with newer cpus ):
with iperf3 daemon on the PVE host then running iperf3 -R from the guest, PVE sending data.
The reverse (VM send to PVE host) is even speedy (like 10-20 Gbits/s)
E5-2620 v0 @ 2 GHz : PVE > WinVM = 2,7 Gbits/s E5-2609 v0 @ 2,4 GHz : PVE>WinVM = 5 Gbits/s E3-1220 v3 @3.10GHz : PVE -> WinVM = 7,3 Gbits/s i7-6700HQ Laptop @2.6 GHz : PVE > WinVM = 10 Gbits/s

 

[Ballistic]

You are not NAT'ing the way you are testing.