Virtualized pfSense unusable upload speed

[Scar_UY]

I apologize beforehand, I know this has been asked a thousand times already here and everywhere else, but I've tried all the suggestions on those posts with no effect, so maybe I'm missing something, here's my case:

- Gigabyte GA-J1900N-D3V (2x Realtek RTL8111/8168/8411 I know! Bad Idea, but it's what the client has and in this country it's almost impossible to get another dual lan SFF PC)
- pve-manager/7.2-7/d0dd0e85
- pfSense 2.6.0 on this vm conf:

boot: order=virtio0
cores: 4
memory: 4096
meta: creation-qemu=6.2.0,ctime=1654987697
name: pfSense
net0: virtio=AE:36:FE:16:01:BB,bridge=vmbr0
net1: virtio=32:61:DF:13:40:2F,bridge=vmbr1
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=a7696147-b320-4b87-aa5e-ba58ae085340
sockets: 1
startup: order=1,up=90
vga: qxl
virtio0: HDD1:100/vm-100-disk-0.qcow2,size=16G

I've disabled:
Hardware checksum offload
Hardware TCP segmentation offload
Hardware large receive offload

Restarted everything several times, deleted the vm, reinstalled it as per https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html, double checked that physical MACs are different from virtual MACs, enabled, restarted and disabled those three and restarted again and doesn't make a difference at all.

The connection is 120 Mbps download 10 Mbps upload, downloads are excellent, but upload speed is ridiculous, 0.6 Mbps at best. I've tried with a linksys router and the speed is as expected, so the issue it's the realtek-proxmox-pfSense combo.

I'm very much aware this setup is far from ideal, but I don't need much from it, reaching 5 Mbps uploads would be enough, but I'm getting 10% of that, does anyone have another suggestion?

Thank you very much in advance for your time reading this.

 

[bobmc]

First observation I would make is that you have allocated 4 cpu cores to a single VM when the host only has 4 cores to begin with, so in that case the host and VM are competing with each other for system resources. Also, how much RAM does the host have? pfSense runs quite well on 1cpu and 1gb of RAM in a typical home environment.

Finally, take note of the recommendation to experiment with disabling the

TCP Segmentation Offload

and

Hardware Large Receive Offload

settings under System->Advanced in the pfsense menus.

 

[Scar_UY]

First observation I would make is that you have allocated 4 cpu cores to a single VM when the host only has 4 cores to begin with, so in that case the host and VM are competing with each other for system resources. Also, how much RAM does the host have? pfSense runs quite well on 1cpu and 1gb of RAM in a typical home environment.

Finally, take note of the recommendation to experiment with disabling the

TCP Segmentation Offload

and

Hardware Large Receive Offload

settings under System->Advanced in the pfsense menus.

The ram and cores assigned are as a result of many tests, originally had only 1 core and 2 Gb of ram, system has 8 Gb, changing the amount of cores and ram up to 4 Gb changed nothing, and as I mentioned in my post "TCP Segmentation Offload" and "Hardware Large Receive Offload" are disabled (also tried enabling them [unchecking] restart and disabling them again) with no impact on performance either way.
Any other ideas?
Has anyone else had that terrible results with realtek? I've seen here people having "only" 25 Mbps throughput, I'll be happy with 5...

 

[datdenkikniet]

Have you tried this test from within proxmox by running speedtest over that interface from the proxmox installation itself? It may be interesting to see if that makes a difference, or not. Not sure if it will yield any useful results, but who knows?

 

[Neobin]

Give CPU-type host for the VM a try.

Otherwise results of an bare-metal pfSense installation (on another free drive only to test with for example) on that hardware would be interesting.

 

[Scar_UY]

Have you tried this test from within proxmox by running speedtest over that interface from the proxmox installation itself? It may be interesting to see if that makes a difference, or not. Not sure if it will yield any useful results, but who knows?

I don't think I understand what or how you want me to test, I wasn't aaware that proxmos had any way to run speedtest from/to the vms, if so I'd gladly try if you can give me some indication on how to do it.

 

[Scar_UY]

Give CPU-type host for the VM a try.

Otherwise results of an bare-metal pfSense installation (on another free drive only to test with for example) on that hardware would be interesting.

I had tried that already, did it again now, still no difference:

boot: order=virtio0
cores: 2
cpu: host
memory: 4096
meta: creation-qemu=6.2.0,ctime=1654987697
name: pfSense
net0: virtio=AE:36:FE:16:01:BB,bridge=vmbr0
net1: virtio=32:61:DF:13:40:2F,bridge=vmbr1
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=a7696147-b320-4b87-aa5e-ba58ae085340
sockets: 1
startup: order=1,up=90
vga: qxl
virtio0: HDD1:100/vm-100-disk-0.qcow2,size=16G

I'm hours away from the site, but I'll try a bare metal install as soon as I can get there (maybe on the weekend).

 

[Scar_UY]

Have you tried this test from within proxmox by running speedtest over that interface from the proxmox installation itself? It may be interesting to see if that makes a difference, or not. Not sure if it will yield any useful results, but who knows?

Just in case your suggestion was to run speedtest from pfsense itself, I did, and the results are:

Shell Output - speedtest-cli
Retrieving speedtest.net configuration...
Testing from Administracion Nacional de Telecomunicaciones (200.125.25.46)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Telecom Personal (Vicente Lopez) [216.03 km]: 25.322 ms
Testing download speed................................................................................
Download: 139.46 Mbit/s
Testing upload speed......................................................................................................
Upload: 5.97 Mbit/s

Which would be acceptable, so it is as the documentation says "the virtual machine will not properly pass traffic" which should be solved disabling Hardware Checksums, which are disabled, at least in the web gui.
Is there a way to check if those are actually disabled using the command line? It doesn't make any sense otherwise...

I run the test several times after that, but I was never able to get those speeds, second best was

Testing download speed................................................................................
Download: 149.52 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.43 Mbit/s

But average was more like

Testing download speed................................................................................
Download: 149.52 Mbit/s
Testing upload speed......................................................................................................
Upload: 1.67 Mbit/s

If I run the same test with --no-download, I get about the same as when I try it from a client on the network

Testing upload speed......................................................................................................
Upload: 0.59 Mbit/s

I don't know if this information is of any help, I'm open to new Ideas

 

[Neobin]

To get more informations for troubleshooting and to maybe rule out some things, I would run some iperf3 tests:

• From pfSense-VM to PVE-host and vice versa.
• From pfSense-VM to another client/workstation on the LAN and vice versa.
• From PVE-host to another client/workstation on the LAN and vice versa.

You can change the direction with the -R option on the client-side iperf3 or exchange the server-side with the client-side.

pfSense: https://docs.netgate.com/pfsense/en/latest/packages/iperf.html
Debian: https://manpages.debian.org/bullseye/iperf3/iperf3.1.en.html
Windows: https://iperf.fr/iperf-download.php#windows

 

[datdenkikniet]

I don't think I understand what or how you want me to test, I wasn't aaware that proxmos had any way to run speedtest from/to the vms, if so I'd gladly try if you can give me some indication on how to do it.

What @Neobin said, plus from proxmox to some external (i.e. over the internet) site! There's also a CLI for speedtest.net, if you're feeling adventurous.

That way you may be able to figure out "where" the problem exists in the line of PfSense <-> Proxmox Bridge <-> The Interwebz

 

[Scar_UY]

To get more informations for troubleshooting and to maybe rule out some things, I would run some iperf3 tests:

• From pfSense-VM to PVE-host and vice versa.
• From pfSense-VM to another client/workstation on the LAN and vice versa.
• From PVE-host to another client/workstation on the LAN and vice versa.

You can change the direction with the -R option on the client-side iperf3 or exchange the server-side with the client-side.

pfSense: https://docs.netgate.com/pfsense/en/latest/packages/iperf.html
Debian: https://manpages.debian.org/bullseye/iperf3/iperf3.1.en.html
Windows: https://iperf.fr/iperf-download.php#windows

Living and learning! here are the result of said tests:

• From pfSense-VM to PVE-host = average 403 Mbits/sec <_> average 431 Mbits/sec
• From pfSense-VM to a Windows-VM on the same PVE-host = average 204 Mbits/sec <_> average 225 Mbits/sec
• From pfSense-VM to a Windows-LAN = average 415 Mbits/sec <_> average 425 Mbits/sec
• From PVE-host to another client/workstation on the LAN and vice versa = 938 Mbits/sec <_> average 937 Mbits/sec

 

[Scar_UY]

What @Neobin said, plus from proxmox to some external (i.e. over the internet) site! There's also a CLI for speedtest.net, if you're feeling adventurous.

That way you may be able to figure out "where" the problem exists in the line of PfSense <-> Proxmox Bridge <-> The Interwebz

Yes, I did those tests with speedtest-cli, results are above.

 

[Neobin]

So the complete internal/LAN-side is at least no bottleneck for the internet connection.

So what in my opinion is left, is the pfSense-firewalling from LAN to WAN and/or something on the WAN-side, but only in upload direction.

Do you use a pure modem in front of the pfSense for the connection to the ISP? And the pfSense does establish the connection over the modem to the ISP?
Maybe it needs some specific settings for the connection and/or the WAN-interface? But why would this affect only the upload? Could it even be?
That is unfortunately out of my knowledge and therefore completely wild guesses, sorry.

 

[Scar_UY]

So the complete internal/LAN-side is at least no bottleneck for the internet connection.

So what in my opinion is left, is the pfSense-firewalling from LAN to WAN and/or something on the WAN-side, but only in upload direction.

Do you use a pure modem in front of the pfSense for the connection to the ISP? And the pfSense does establish the connection over the modem to the ISP?
Maybe it needs some specific settings for the connection and/or the WAN-interface? But why would this affect only the upload? Could it even be?
That is unfortunately out of my knowledge and therefore completely wild guesses, sorry.

"pfSense-firewalling from LAN to WAN" was my first thougth as said in the documentation, but every post I saw about it was way over 20 Mbps, not 1 Mbps.
The ISP side of things is a pure modem and we have a fixed ip address (no ppoe or anything).
This week I'll get my hands on it and will try non-virtualized installation amongst other things, and report back, maybe I can find something useful to someone else.

 

[Scar_UY]

This is my thanks to bobmc, datdenkikniet and Neobin fot their answers, and like Dr House says, "Everybody lies".
Like I said this "server" was at a remote site I had no acces for a while, the local IT assured me all cables where checked and that they properly measured the connection directly from the modem and with another router and got the expected results, turned out, they were lying! Eventually I made the time to go in person, plug another router, then my laptop directly to the modem and upload was just as shitty, called the provider, they changed some poorly patched optic fiber on the street and voila, I get perfect speeds regardless of the device I connect.

So, it wasn't a proxmox problem, it wasn't a hardware problem, it wasn't a pfsense problem, it was a laziness problem....

Sorry I wasted your time, at least I learned a few new things, and here are some speed test for consolation:

Retrieving speedtest.net configuration...
Testing from Administracion Nacional de Telecomunicaciones (200.125.25.46)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Telecom Personal (Vicente Lopez) [216.03 km]: 26.213 ms
Testing download speed................................................................................
Download: 148.49 Mbit/s
Testing upload speed......................................................................................................
Upload: 19.48 Mbit/s

 

[edarrigo]

I have solved! YES! After a week on internet forums, pfsense communities and more, I tried unchecking an advanced option. First I tried changing the VM's eth type from WMNEXT to E1000, and all the others I found on the internt that don't work for me, on my VM pfsense community edition v.2.6.0 (2.5.2 was also affwcted) on ESXi 7.x I DISABLED:
packet filtering in the System-Advanced-Firewall menu and NAT.
I hope it works for all other same cases as well

 

[Neobin]

I have solved! YES! After a week on internet forums, pfsense communities and more, I tried unchecking an advanced option. First I tried changing the VM's eth type from WMNEXT to E1000, and all the others I found on the internt that don't work for me, on my VM pfsense community edition v.2.6.0 (2.5.2 was also affwcted) on ESXi 7.x I DISABLED:
packet filtering in the System-Advanced-Firewall menu and NAT.
I hope it works for all other same cases as well

Disabling the firewall is most likely not what most people want...

PS.: Not sure, if this is a troll or maybe even a bot...

 

[edarrigo]

No bot or troll!
Sorry, effectively I did not delve into my environment in detail. I have disabled all packet filtering because I am already behind a corporate MPLS firewall, and my only need was a Captive Portal by WiFi. Wi-fi network is behind corporate fw yet.

 

[Neobin]

No bot or troll!
Sorry, effectively I did not delve into my environment in detail. I have disabled all packet filtering because I am already behind a corporate MPLS firewall, and my only need was a Captive Portal by WiFi. Wi-fi network is behind corporate fw yet.

Then, sorry for the assumption.

But without any detailed context, it sounded pretty weird, if not even suspicious, to suggest to disable the firewall as a general solution.

 

[edarrigo]

Ok, but the "unchek" disable only packet filtering feature, do other fw rule and services work the same?