Proxmox pfsense vm extreme low throughput

[abhishek.akt]

Hello

I migrated to proxmox ve over a year now from hyperv and I just love the product. I have small home system running one single proxmox on Intel Core i5 with 16 GB RAM.

I have firewall and few other vms running.

Everything was working fine before I moved to pfsense firewall. With Proxmox, pfsense throughput dropped quite badly. I have gone through Proxmox and Netgate suggested settings and turned off all Hardware offloading. It has fixed the upload but still have slow internet speed. I have 300 Mbps up down link and I get around 150 with pfsense. PFSense vm has 2 cores and 4 GB RAM. Hardware usage are quite low.

I also tried spinning fresh PFSense VM and even tried PFSENSE developer build 2.5 but same problem.

I have Intel Gigabit nic. I tried using all network adapter options from Proxmox VirtIO, Intel E1000, VMware and even realteck one but no help.

I also tried other forum post suggesting disable TX on vmbr0 and actual ethernet port in /etc/network/interfaces but no help.

I can tell its not Proxmox issue as if I spin up Untangle vm, I get full throughput. I read somewhere FreeBSD does not work well with proxmox. When I use pfsense in dedicated hardware or Hyperv, it works just fine. I guess it has to do with the FreeBSD.

Here's my iperf result between pfsense vm and ubuntu vm connected to vmbr0 which shows it is indeed problem with pfsense vm:

ubuntu@zm:~$ iperf3 -c asa
Connecting to host asa, port 5201
[ 4] local 10.12.47.43 port 43808 connected to 10.12.47.1 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 19.7 MBytes 165 Mbits/sec 0 452 KBytes
[ 4] 1.00-2.00 sec 32.4 MBytes 272 Mbits/sec 0 755 KBytes
[ 4] 2.00-3.00 sec 35.0 MBytes 293 Mbits/sec 0 823 KBytes
[ 4] 3.00-4.00 sec 32.2 MBytes 270 Mbits/sec 0 823 KBytes
[ 4] 4.00-5.00 sec 36.2 MBytes 305 Mbits/sec 0 923 KBytes
[ 4] 5.00-6.00 sec 43.7 MBytes 366 Mbits/sec 0 929 KBytes
[ 4] 6.00-7.00 sec 35.5 MBytes 298 Mbits/sec 0 977 KBytes
[ 4] 7.00-8.00 sec 35.9 MBytes 301 Mbits/sec 0 977 KBytes
[ 4] 8.00-9.00 sec 33.5 MBytes 281 Mbits/sec 0 977 KBytes
[ 4] 9.00-10.04 sec 31.2 MBytes 253 Mbits/sec 0 977 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.04 sec 335 MBytes 280 Mbits/sec 0 sender
[ 4] 0.00-10.04 sec 333 MBytes 278 Mbits/sec receiver


iperf between two ubuntu vm's connected to same vmbr0 switch give better results:

ubuntu@nextcloud:~$ iperf3 -c zm
Connecting to host zm, port 5201
[ 4] local 10.12.47.50 port 42630 connected to 10.12.47.43 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 419 MBytes 3.51 Gbits/sec 0 3.15 MBytes
[ 4] 1.00-2.00 sec 437 MBytes 3.67 Gbits/sec 1 3.15 MBytes
[ 4] 2.00-3.00 sec 422 MBytes 3.55 Gbits/sec 1 3.15 MBytes
[ 4] 3.00-4.00 sec 526 MBytes 4.40 Gbits/sec 0 3.15 MBytes
[ 4] 4.00-5.00 sec 522 MBytes 4.39 Gbits/sec 0 3.15 MBytes
[ 4] 5.00-6.00 sec 474 MBytes 3.97 Gbits/sec 0 3.15 MBytes
[ 4] 6.00-7.00 sec 314 MBytes 2.64 Gbits/sec 1 3.15 MBytes
[ 4] 7.00-8.00 sec 258 MBytes 2.16 Gbits/sec 0 3.15 MBytes
[ 4] 8.00-9.00 sec 412 MBytes 3.46 Gbits/sec 1 3.15 MBytes
[ 4] 9.00-10.00 sec 439 MBytes 3.68 Gbits/sec 0 3.15 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 4.13 GBytes 3.54 Gbits/sec 4 sender
[ 4] 0.00-10.00 sec 4.12 GBytes 3.54 Gbits/sec receiver


iperf from Proxmox host to ubuntu vm works well as expected:

root@proxmox:~# iperf3 -c zm
Connecting to host zm, port 5201
[ 5] local 10.12.47.10 port 42520 connected to 10.12.47.43 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 756 MBytes 6.34 Gbits/sec 1 3.13 MBytes
[ 5] 1.00-2.00 sec 905 MBytes 7.59 Gbits/sec 0 3.13 MBytes
[ 5] 2.00-3.00 sec 729 MBytes 6.11 Gbits/sec 0 3.13 MBytes
[ 5] 3.00-4.00 sec 834 MBytes 6.99 Gbits/sec 1 3.13 MBytes
[ 5] 4.00-5.00 sec 849 MBytes 7.12 Gbits/sec 0 3.13 MBytes
[ 5] 5.00-6.00 sec 625 MBytes 5.24 Gbits/sec 5 3.13 MBytes
[ 5] 6.00-7.00 sec 638 MBytes 5.35 Gbits/sec 0 3.13 MBytes
[ 5] 7.00-8.00 sec 860 MBytes 7.21 Gbits/sec 0 3.13 MBytes
[ 5] 8.00-9.00 sec 882 MBytes 7.40 Gbits/sec 0 3.13 MBytes
[ 5] 9.00-10.00 sec 810 MBytes 6.80 Gbits/sec 0 3.13 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.70 GBytes 6.62 Gbits/sec 7 sender
[ 5] 0.00-10.00 sec 7.70 GBytes 6.62 Gbits/sec receiver


iperf from Promox host to pfsense gets slow results:

root@proxmox:~# iperf3 -c asa
Connecting to host asa, port 5201
[ 5] local 10.12.47.10 port 34060 connected to 10.12.47.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 46.6 MBytes 391 Mbits/sec 223 184 KBytes
[ 5] 1.00-2.00 sec 59.3 MBytes 497 Mbits/sec 79 341 KBytes
[ 5] 2.00-3.00 sec 53.6 MBytes 449 Mbits/sec 53 426 KBytes
[ 5] 3.00-4.00 sec 43.9 MBytes 369 Mbits/sec 0 495 KBytes
[ 5] 4.00-5.00 sec 42.6 MBytes 357 Mbits/sec 95 544 KBytes
[ 5] 5.00-6.00 sec 42.7 MBytes 358 Mbits/sec 0 601 KBytes
[ 5] 6.00-7.00 sec 49.5 MBytes 415 Mbits/sec 90 646 KBytes
[ 5] 7.00-8.00 sec 46.2 MBytes 388 Mbits/sec 43 691 KBytes
[ 5] 8.00-9.00 sec 51.2 MBytes 430 Mbits/sec 48 742 KBytes
[ 5] 9.00-10.00 sec 35.0 MBytes 294 Mbits/sec 0 778 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 471 MBytes 395 Mbits/sec 631 sender
[ 5] 0.00-10.01 sec 467 MBytes 391 Mbits/sec receiver



I am not sure what could be the problem. I know lot of people use pfsense on proxmox so would appreciate if anyone can help.

Thanks

 

[wolfgang]

Hi,

please send the config of the VM.

 

[abhishek.akt]

@wolfgang Here's the config


root@proxmox:~# cat /etc/pve/qemu-server/130.conf
bootdisk: virtio0
cores: 2
cpu: host
ide2: none,media=cdrom
memory: 4096
name: pfsense
net0: virtio=AA:1E:31:27:6B:29,bridge=vmbr0,firewall=1,tag=100
net1: virtio=86:4C:25:40:18:C1,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=56f25d68-0dee-4b4a-8494-60ea6aa0c9a8
sockets: 1
startup: order=1
virtio0: local-lvm:vm-130-disk-0,size=40G
vmgenid: b86993af-9c0b-4cab-995d-8657a8c64724

 

[wolfgang]

I have tested it here and there are have all the same throughput.
But try to disable the firewall at the vNIC and restart the VM.

 

[abhishek.akt]

Sorry for the late response. I tried disabling firewall on both virtio nics and tried changing multiqueue but no help.

Just to add, I am running OVS if it helps.

 

[wolfgang]

We do not recommend OVS.
OVS is a leftover from the past where not everything works with Linux bridges.
The myth that OVS is faster is only true if you use it in combination with DPDK.

There are some age cases where you need OVS instead of Linux bridges.

Why do you need it?

If you do not really need it I would recommend you switch to Linux Bridges.

 

[abhishek.akt]

I wanted to do port mirroring and flows and the time I moved to OVS, it wasn't natively supported on Linux brige not sure current status.

However I don't think it will make any major difference in performance unless I am missing something.

I have identical setup at my friend's place running Linux bridges with same slow speed issue

 

[fwcheck]

Have you tried to set
Hardware Checksum Offloading
[X] Disable hardware checksum offload

I have the same problems with pfsense throughput without disabling the hardware checksum. Throughput is < 100 Kbit otherwise.

 

[t-ice]

I am facing similar issue where on 100 mbps internet uplink I get 40/50 mbps upload/download respectively.
Running similar setup, pfsense in proxmox, using linux bridge. I've disabled hw offload with no luck, played around with few other settings but no improvement.
Did anyone managed to fix this?

 

[DICKHILL]

Have you tried to set
Hardware Checksum Offloading
[X] Disable hardware checksum offload

I have the same problems with pfsense throughput without disabling the hardware checksum. Throughput is < 100 Kbit otherwise.

To be honest, I haven't read the thread!
I were just trying to find a solution to the slow network/internet speed on VMs behind a pfSense..
Disabling "Hardware Checksum Offloading" helped me and worked on the fly.. (Haven't rebooted yet!)
As soon i saved the configuration, my tested DL speeds flew from ~5MB/s to almost 800MB/s (I have a 1000/500 Mbit line)

 

[Jonte]

Hey,
I've exactly the same problem. Any news on this?

 

[ghostface]

I am also having this issue

Spoiler: pveversion -v

proxmox-ve: 7.3-1 (running kernel: 6.1.2-1-pve)
pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4)
pve-kernel-6.1: 7.3-2
pve-kernel-helper: 7.3-2
pve-kernel-5.15: 7.3-1
pve-kernel-6.1.2-1-pve: 6.1.2-1
pve-kernel-5.19.7-2-pve: 5.19.7-2
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.2-1
proxmox-backup-file-restore: 2.3.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-2
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.7-pve3

I am running proxmox on an Intel Celeron N5105 machine with 4*Intel i226-V 2.5G NICs.
On proxmox, I am hosting pfsense 2.6 (in use as my firewall) and also LCX for test purposes.
For pfsense, I used virtIO bridges due to pci pass-through somehow is not working properly.

I tried multiple cables with the same result:
- When I run iperf3 from a VM to Proxmox host, things are fast as expected (around 2.98 Gbits/sec)
- When I run iperf3 from a VM/Proxmox host/wired computer towards the internet, I constantly get 105 Mbits/sec (ISP supports 1 Gbit/s)
- When I run iperf3 from a wired computer to Proxmox host, I get around 941 Mbits/sec

Disable "Hardware Checksum Offloading" did not help. Any suggestions?

 

[timbckto]

I think I remember hearing that pfSense doesn't have an updated kernel to work well with the i225-V or the i226. Have you tried OPNsense to see if that helps the throughput?

 

[ghostface]

Thanks for the reply! I am using virtIO as network devices. Does it still matter then, if i226-v are not yet supported? (They will be supported in pfsense 2.7.0)

UPDATE: I updated to pfsense 2.7.0 to test but this didn't help and I am still at the same slow speed.

 

[ghostface]

UPDATE: I switched to OPNsense and experienced the same issue (I did get a max speed of 105 Mbit/s on my WAN interface) for my N5105 unit with 4 x 226-v (https://de.aliexpress.com/item/1005004627244223.html). First, I thought that something might be wrong with the negotiation of the speed between the ISP provided modem (UPC) and the box but turned out, that the following tunables (System: Settings: Tunables) did the trick:

vm.pmap.pti = 0
hw.ibrs_disable = 1
net.isr.maxthreads= 1
net.inet.rss.enabled=1
hw.pci.honor_msi_blacklist="0"

In addition, I installed the os-qemu-guest-agent plugin and enabled it with the following tunable:

virtio_console_load=YES

I did not yet test but do highly assume that this would also solve my issue I had on pfSense.

What is weird however:

When I run iperf3 on the opnSense VM against any server on the internet, I get 105 Mbit/s max.



When I run https://www.speedtest.net/ from a connected device or a docker container, I get full speed:



Do you know why?

 

[Ballistic]

Did not help for me on PFsense.
900/100mbit internet line
9xx iperf test from LAN to PFsense VM
However, NAT performance horrible at 270mbit down, 10mbit up